Skip to content

feat(site): news - V4-Pro in the field, what practitioners report #18

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
lroolle wants to merge 1 commit into from
Closed

feat(site): news - V4-Pro in the field, what practitioners report #18

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 83 additions & 0 deletions site/build.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1384,6 +1384,89 @@ def jstr(s):
live API where that is possible; the in-terminal feed is
<code>ds docs changelog</code>.</p>

<h2 id="field-reports">2026-08-13 &middot; V4-Pro in the field: what practitioners report</h2>
<p>The <a href="{{root}}bench/">launch table</a> is DeepSeek's own. This is what
people running <code>deepseek-v4-pro-0813</code> in real harnesses said in the
first day, mostly in the
<a href="https://news.ycombinator.com/item?id=49274600">Hacker News launch
thread</a> (900+ points). It is anecdote, not measurement, but it is consistent
anecdote, and it lines up with the sober reading on the bench page.</p>

<h3>Pro thinks less and says less than Flash</h3>
<p>The clearest repeated observation is the split between the two models. One
reviewer running both through OpenRouter in a pi harness put it plainly: Flash
<q>makes a lot more initial mistakes, and then has to re-check stuff, and
produces much more output compared to Pro &hellip; the output volume is often 5x
more &hellip; saying something wrong &hellip; and then &lsquo;Wait, let me
re-check:&rsquo;</q> before it lands on the right answer
(<a href="https://news.ycombinator.com/item?id=49275429">pixelesque</a>). Pro
reaches the conclusion more directly. If you are paying for output tokens and
wall-clock, that difference is the argument for Pro on the hard steps and Flash
on the bounded ones, which is the <a href="{{root}}bench/#practice">route-by-role</a>
advice from the other direction.</p>

<h3>The benchmarks and a real task disagree, and the harness is why</h3>
<p>The most useful counterweight was a like-for-like test: scan a repo and
produce a single <code>docker-compose</code> to deploy behind Caddy, with
wildcard certs provisioned outside, some ports already taken, and Postgres
built in. Run on <code>deepseek-v4-pro-0813</code> and on
<code>gpt-5.6-terra-high</code>, the verdict was <q>this one had few issues.
terra: none</q>, with the note that the gap opens up as the project stops being
simple (<a href="https://news.ycombinator.com/item?id=49276546">freakynit</a>,
<a href="https://news.ycombinator.com/item?id=49276751">repro</a>). The
immediate reply from another user was the load-bearing caveat:
<q>DS V4 is harness sensitive</q>
(<a href="https://news.ycombinator.com/item?id=49276768">scrlk</a>). A score is a
model plus a harness, and a cheap model with a good harness (self-verification
tools, an <code>AGENTS.md</code> it rereads) closes a lot of the gap the raw
number shows.</p>

<h3>The economics are the story, but only if you cache right</h3>
<p>This is where the field agrees with the <a href="{{root}}bench/#kill-line">kill
line</a>. One practitioner built a cost simulation from their own pi sessions
and found that once caching is counted, <code>deepseek-v4-pro</code> comes out
<em>cheaper</em> than <code>gpt-5.6-luna</code>, on the strength of the cache-read
price (<a href="https://news.ycombinator.com/item?id=49277160">taosx</a>). The
catch, stated bluntly by another: a 50% cache-hit rate is a misconfigured
harness, an agentic loop should see 99%+, and DeepSeek's cache pricing is low
enough that you should <q>never change history &hellip; even more so with deepseek
because their cache hit pricing is so low</q> (no stripping old thinking tokens,
no rewriting tool results to save context)
(<a href="https://news.ycombinator.com/item?id=49280319">p1necone</a>). The 138x
cache-hit lever on the <a href="{{root}}cost/">cost page</a> is only real if the
prefix actually stays put.</p>

<h3>Against the other open models</h3>
<p>On GLM-5.2, one report from OpenCode work on an old Unix-clone codebase:
<q>GLM ended up being far slower, and far more expensive, for approximately the
same results &hellip; There was never a problem that GLM could solve that DS
couldn't solve, faster, and significantly cheaper</q>, with the honest coda that
DS <q>isn't as good as GPT or Claude &hellip; but it's fast, and pretty darned
effective</q> (<a href="https://news.ycombinator.com/item?id=49276824">spijdar</a>,
who also runs a 3-bit quant locally at ~15 tok/s). Against Qwen3.8-max, released
the same day, the read was comparable capability at much lower price, with no
reason to reach for Qwen unless you need vision
(<a href="https://news.ycombinator.com/item?id=49275709">parsimo2010</a>).</p>

<h3>The trust question: is the endpoint even serving the real model?</h3>
<p>One thread ties straight to the entry below. An open-weights user asked to be
able to <q>prove to me &hellip; that this output was generated by FP8 DeepSeek V4
Pro 0813. Not some cheaper quantization</q>
(<a href="https://news.ycombinator.com/item?id=49277965">Phemist</a>), and others
noted they pin to a single provider for deepseek specifically. The model will
not tell you which checkpoint or precision it is, and its self-description is
worthless for it. The signals that hold up are the ones you can check without
asking it: the endpoint's model list and the docs version cell, which is exactly
the point of the <a href="#no-rollback">rollback question</a> below.</p>

<h3>The bottom line from the field</h3>
<p>It matches the bench page. V4-Pro-0813 sits in the frontier-adjacent cluster,
not clear of it; a stronger closed model still lands the hardest task in fewer
turns; and the reason to run it anyway is that cheap cached input makes retries,
review passes and parallel workers affordable. Route by role, keep the prefix
stable, and measure successful-task cost rather than the per-token price or the
launch chart.</p>

<h2 id="no-rollback">2026-08-13 &middot; No, V4-Pro did not roll back</h2>
<p>Ask <code>deepseek-v4-pro</code> what model it is and it may tell you it is
GPT-4o. A model that answers with a competitor's name looks like a swap, and
Expand Down
83 changes: 83 additions & 0 deletions site/news/index.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,89 @@ <h1>News</h1>
live API where that is possible; the in-terminal feed is
<code>ds docs changelog</code>.</p>

<h2 id="field-reports">2026-08-13 &middot; V4-Pro in the field: what practitioners report</h2>
<p>The <a href="../bench/">launch table</a> is DeepSeek's own. This is what
people running <code>deepseek-v4-pro-0813</code> in real harnesses said in the
first day, mostly in the
<a href="https://news.ycombinator.com/item?id=49274600">Hacker News launch
thread</a> (900+ points). It is anecdote, not measurement, but it is consistent
anecdote, and it lines up with the sober reading on the bench page.</p>

<h3>Pro thinks less and says less than Flash</h3>
<p>The clearest repeated observation is the split between the two models. One
reviewer running both through OpenRouter in a pi harness put it plainly: Flash
<q>makes a lot more initial mistakes, and then has to re-check stuff, and
produces much more output compared to Pro &hellip; the output volume is often 5x
more &hellip; saying something wrong &hellip; and then &lsquo;Wait, let me
re-check:&rsquo;</q> before it lands on the right answer
(<a href="https://news.ycombinator.com/item?id=49275429">pixelesque</a>). Pro
reaches the conclusion more directly. If you are paying for output tokens and
wall-clock, that difference is the argument for Pro on the hard steps and Flash
on the bounded ones, which is the <a href="../bench/#practice">route-by-role</a>
advice from the other direction.</p>

<h3>The benchmarks and a real task disagree, and the harness is why</h3>
<p>The most useful counterweight was a like-for-like test: scan a repo and
produce a single <code>docker-compose</code> to deploy behind Caddy, with
wildcard certs provisioned outside, some ports already taken, and Postgres
built in. Run on <code>deepseek-v4-pro-0813</code> and on
<code>gpt-5.6-terra-high</code>, the verdict was <q>this one had few issues.
terra: none</q>, with the note that the gap opens up as the project stops being
simple (<a href="https://news.ycombinator.com/item?id=49276546">freakynit</a>,
<a href="https://news.ycombinator.com/item?id=49276751">repro</a>). The
immediate reply from another user was the load-bearing caveat:
<q>DS V4 is harness sensitive</q>
(<a href="https://news.ycombinator.com/item?id=49276768">scrlk</a>). A score is a
model plus a harness, and a cheap model with a good harness (self-verification
tools, an <code>AGENTS.md</code> it rereads) closes a lot of the gap the raw
number shows.</p>

<h3>The economics are the story, but only if you cache right</h3>
<p>This is where the field agrees with the <a href="../bench/#kill-line">kill
line</a>. One practitioner built a cost simulation from their own pi sessions
and found that once caching is counted, <code>deepseek-v4-pro</code> comes out
<em>cheaper</em> than <code>gpt-5.6-luna</code>, on the strength of the cache-read
price (<a href="https://news.ycombinator.com/item?id=49277160">taosx</a>). The
catch, stated bluntly by another: a 50% cache-hit rate is a misconfigured
harness, an agentic loop should see 99%+, and DeepSeek's cache pricing is low
enough that you should <q>never change history &hellip; even more so with deepseek
because their cache hit pricing is so low</q> (no stripping old thinking tokens,
no rewriting tool results to save context)
(<a href="https://news.ycombinator.com/item?id=49280319">p1necone</a>). The 138x
cache-hit lever on the <a href="../cost/">cost page</a> is only real if the
prefix actually stays put.</p>

<h3>Against the other open models</h3>
<p>On GLM-5.2, one report from OpenCode work on an old Unix-clone codebase:
<q>GLM ended up being far slower, and far more expensive, for approximately the
same results &hellip; There was never a problem that GLM could solve that DS
couldn't solve, faster, and significantly cheaper</q>, with the honest coda that
DS <q>isn't as good as GPT or Claude &hellip; but it's fast, and pretty darned
effective</q> (<a href="https://news.ycombinator.com/item?id=49276824">spijdar</a>,
who also runs a 3-bit quant locally at ~15 tok/s). Against Qwen3.8-max, released
the same day, the read was comparable capability at much lower price, with no
reason to reach for Qwen unless you need vision
(<a href="https://news.ycombinator.com/item?id=49275709">parsimo2010</a>).</p>

<h3>The trust question: is the endpoint even serving the real model?</h3>
<p>One thread ties straight to the entry below. An open-weights user asked to be
able to <q>prove to me &hellip; that this output was generated by FP8 DeepSeek V4
Pro 0813. Not some cheaper quantization</q>
(<a href="https://news.ycombinator.com/item?id=49277965">Phemist</a>), and others
noted they pin to a single provider for deepseek specifically. The model will
not tell you which checkpoint or precision it is, and its self-description is
worthless for it. The signals that hold up are the ones you can check without
asking it: the endpoint's model list and the docs version cell, which is exactly
the point of the <a href="#no-rollback">rollback question</a> below.</p>

<h3>The bottom line from the field</h3>
<p>It matches the bench page. V4-Pro-0813 sits in the frontier-adjacent cluster,
not clear of it; a stronger closed model still lands the hardest task in fewer
turns; and the reason to run it anyway is that cheap cached input makes retries,
review passes and parallel workers affordable. Route by role, keep the prefix
stable, and measure successful-task cost rather than the per-token price or the
launch chart.</p>

<h2 id="no-rollback">2026-08-13 &middot; No, V4-Pro did not roll back</h2>
<p>Ask <code>deepseek-v4-pro</code> what model it is and it may tell you it is
GPT-4o. A model that answers with a competitor's name looks like a swap, and
Expand Down