feat: pin parallel_flush as opt-in; a default-on gate is refuted (#445)

[jdatcmd]

What

Finishes #445's default-on question with a measured no-go. parallel_flush (opt-in, off by default) helps one shape — a large flush of many cheap numeric columns (~14%) — and regresses the rest (small/frequent flushes 3.6×, text-heavy ~+16%). This asked whether a gate could default it on while declining the losers.

It can't. No metric computable at dispatch (natts, buffered bytes, bytes/value) separates the win from the losses, because the deciding variable is per-column encode CPU and its balance across columns, which the buffered .len fields don't carry. Verified by hand (not just the workflow): 20×int2 and 5×int8 buffer the same bytes yet do 6.4× different work (3680 vs 578 ms/500k rows), and a random vs a constant column are byte-identical to the metric and opposite in cost. A threshold over such a metric mispredicts.

Shipped (no gate)

• Dispatch observability at DEBUG1 (rows/natts/bufbytes/valbytes/valcount -> parallel|serial) via a read-only pflush_metrics() sum. Byte-neutral (differential/native_writer/native_zonemap/native_bloom unchanged).
• Refutation comment at the dispatch so no one re-adds a (natts, bufbytes, width) gate without a new per-column-cost signal.
• test/parallel_flush_optin.sh — pins default-off (removal proof: flipping the boot default takes the arm RED, verified), opt-in byte-identity (on == off), and the metric collision (random vs constant column log an identical dispatch metric). Registered; harness_selftest + docs_style green.
• Design doc records the verdict and the only path to default-on: a re-shape around per-column encode-cost estimation + per-column partitioning (a separate design, not a threshold re-calibration).

Green on PG17 (peer is on PG18 for the ClickBench re-run). Default stays off; opt-in path byte-identical and still ~14% on the wide-numeric shape.

🤖 Generated with Claude Code

[ChronicallyJD]

Review — the no-go is correct and proven deeper than the measurement, and the code is clean

Reviewed against what I measured on #445, and it holds up and then some.

The refutation is right. I found parallel_flush wins one narrow shape (wide-numeric-large, ~14%) and regresses the rest — a byte threshold would wrongly enable the large text-heavy flushes that lose. You've proven the reason a step deeper: the deciding variable is per-column encode CPU and its balance, which the buffered .len fields do not carry. The 20×int2 vs 5×int8 pair (same bytes, 6.4× work) and random-vs-constant (byte-identical to the metric, opposite cost) are exactly the counterexamples that kill any (natts, bufbytes, width) gate. Refuted, not just declined.

A clean cross-check from #595, which strengthens this. The sibling question over there — gating decode per vector — has the opposite answer, and for a principled reason: its discriminator (payload column count) is computable at plan time, so a width gate works and recovered the ClickBench regression. Flush parallelism's discriminator (encode CPU) is not computable at dispatch, so no gate. Two gates, opposite verdicts, each because of whether the deciding quantity is knowable up front. That contrast is worth a line in the design doc; it's the general principle.

The code is safe on the hot path. The pflush_metrics sum is behind message_level_is_interesting(DEBUG1), so the default path pays nothing — exactly the discipline the slice-2 catch was about. goParallel is a pure refactor of the existing gate; the dispatch log is observability only; the GUC stays off. No behaviour change.

The test pins the right things. Default-off, byte-identity, and the metric-does-not-separate — and the "actually dispatches parallel" check via the DEBUG1 line keeps the byte-identity assertion non-vacuous (a silent serial fallback can't pass as a parallel byte-identity proof). That's the premise-check I'd want.

No objection. This is the correct close for #445's default-on question.