Coordination: what goes in alpha2, and the SUITES line that keeps conflicting

[ChronicallyJD]

Where the board stands, and a proposal for what alpha2 should contain

Cleared what I could tonight and want to agree scope rather than keep picking items off. There
is no alpha2 milestone, so this is a proposal, not a plan.

Landed since this evening

#462 merged	parallel degree sized from work, not compressed pages (#451)
#454 merged	ClickBench definition re-fetched every run
#434 closed	verified fixed on main by #459 + #460, with the measurement
#466 closed	withdrawn — I filed it wrongly, encode_effort = fast already existed

Open PRs

PR	state	blocker
#444 (mine)	MERGEABLE, rebased	your re-review; suite now passes on main so it is no longer a red gate
#468 (yours)	approved by me	rebase only — the SUITES line again
#446 (yours)	approved by me	conflicting, same line
#457 (mine)	MERGEABLE	needs a reviewer. Its two red checks were Service Unavailable from Actions, not a test failure; re-running

The thing I keep tripping over, and would fix before adding anything

SUITES=(...) conflicted four times today: #459 vs #462, #460 vs #462, #462 vs #468, and
#444 behind them. Every PR that adds a suite edits one backslash-continued line, so any two
conflict by construction — and the bad resolution is one bash -n accepts and that only fails
at run time with foo: command not found.

One name per line, or generating the list from test/*.sh with an explicit exclude, ends it.
I have deliberately not filed it while four PRs are in flight, because the fix touches the line
all four of them are fighting over. Once #444, #446 and #468 land, I would like to do this
first, before anything else goes in.

What I would put in alpha2, and what I would not

Proposing by what is measured and closeable rather than by what is interesting:

In — finished or nearly:

• Bloom filters are sized by value count, not distinct count: 19.3x over-provisioned, 29% of load time, 361 MB raw against a 262 MB table #467 bloom sizing (Size a bloom filter by its distinct count, not its value count (#467) #468 approved, matrix green): 19.3x over-provisioned, filters larger than
  the data. Real space and write-time win.
• bench/ loads serially and understates our own loader by 7x: wire in pgcolumnar.parallel_copy #465 bench wiring for parallel_copy: our own benchmark understates our loader 7x. That
  is a publishing-accuracy problem, not a feature.
• A pgcolumnar.analyze() could read one column instead of all, and take null_frac and bounds exactly from zone maps #414 pgcolumnar.analyze(): slices 1 and 2 green, measured 14.1x on a 3M x 20 table
  (6,197 ms to 440 ms), exact null_frac and n_distinct. Slices 3-5 remain.

In if you agree — cheap and it is a correctness story:

• The zone-map cost discount reads correlation, but pruning is a local property that correlation cannot see #461 zone-map cost reads correlation rather than locality.

Out — real but too large, or unmeasured:

• Citus columnar loads the same data 3x faster than we do, which contradicts why #300 was closed #445 write path. Decomposed now: ~1.9x is the FSST trade at default effort, ~1.4x bloom
  (Bloom filters are sized by value count, not distinct count: 19.3x over-provisioned, 29% of load time, 361 MB raw against a 262 MB table #467), ~1.17x unexplained. encode_effort = fast already gets us to 1.64x Citus. This is a
  defaults question, not a bug, and I would rather answer it after Bloom filters are sized by value count, not distinct count: 19.3x over-provisioned, 29% of load time, 361 MB raw against a 262 MB table #467 lands and changes the
  arithmetic.
• No late materialization: decode cost scales with rows scanned, not rows emitted #452 late materialization, Text predicates cannot be pushed down, so a LIKE scan decodes every value: 2.2x slower where an integer predicate is 27.7x faster #426 text pushdown, Columnar partitions get neither the custom scan nor the index-fetch penalty: set_rel_pathlist_hook returns early for non-baserels #436 partitions. Each is a proper piece of
  work.
• Object storage reads for external Parquet, which #388 depends on #393/Object storage writes for the export functions, which is not the read path reversed #394/Review what it would take to support Apache Iceberg #388 object storage and Iceberg. The roadmap already says object storage gates
  Iceberg; neither belongs in a point alpha.
• Now that we have gone alpha, perhaps it is time to port our tests #432 pytest port. Worth doing, and it is the kind of thing that wants its own window rather
  than sharing one with feature work. My honest view: the bash suite's problems are real —
  I hit sudo -E dropping an env var and a scp landing on the wrong branch today alone —
  but porting mid-queue would invalidate every in-flight PR's test evidence.

What I am not proposing

A date. And I would rather you set the boundary than have me infer it — you have context on
what alpha2 is for that I do not.

[ChronicallyJD]

Owner decision on sequencing: the pytest port is the last item considered before alpha2

Recording this so it is not re-litigated: #432 is not out of scope, it is sequenced last.
Everything else proposed for alpha2 gets considered first, and the port is the final call made
before the line is drawn.

That resolves the objection I raised above better than dropping it would have. My concern was
never the port's merit — it was that porting mid-queue invalidates the test evidence attached
to every in-flight PR. Taking it last removes that collision entirely: by then #444, #446, #468
and whatever else lands have already been evidenced against the bash suite, and the port starts
from a settled tree.

What that means practically

• Nothing blocks on it. Bloom filters are sized by value count, not distinct count: 19.3x over-provisioned, 29% of load time, 361 MB raw against a 262 MB table #467/Size a bloom filter by its distinct count, not its value count (#467) #468, bench/ loads serially and understates our own loader by 7x: wire in pgcolumnar.parallel_copy #465, A pgcolumnar.analyze() could read one column instead of all, and take null_frac and bounds exactly from zone maps #414 and The zone-map cost discount reads correlation, but pruning is a local property that correlation cannot see #461 proceed on the current harness.
• The SUITES fix still comes first, and it becomes cheaper rather than wasted: whatever
  replaces that line has to be ported too, and porting a one-name-per-line list is simpler than
  porting a 126-entry backslash-continued one. If the port does happen, the fix is a step
  toward it rather than throwaway work.
• I will keep recording harness failures as they happen, since they are the actual evidence
  for or against the port. Today's, in one session: sudo -E silently dropping
  PGC_SKIP_TIMING so a run measured the flag being absent rather than failing; an scp
  landing a suite file while the wrong branch was checked out, so a "RED" run tested a tree
  with none of the feature in it; and three void measurement runs before a valid one
  (over-compressible fixture, metadata-answerable aggregates, a parallelism confound).

That last list is not an argument for pytest by itself — a fixture that compresses too well
would have fooled a Python harness identically. The ones that are genuinely about bash are the
env-var and working-tree failures. Worth being precise about which is which when the decision
is actually taken, rather than counting every bad run as evidence for the port.

[ChronicallyJD]

The SUITES item is done (#473), but not the way it was proposed

Taken first, as asked, once #444, #446 and #468 landed. Merged in #473, full five-major matrix green.

One name per line does not fix the conflicts. That was the proposal and it is wrong, which I only found by measuring it. Branching twice, adding a suite on each branch, merging:

arrangement	two additions	result
one line (before)	same physical line	CONFLICT
one per line	both appended at the end	CONFLICT
one per line + sorted	names far apart	clean
one per line + sorted	names sort adjacently	CONFLICT

Everyone appends at the end. That is the shape of all four conflicts you counted and all four of mine, so one-per-line alone would have left every one of them still conflicting. Sorting is the part that works, because it gives a new suite an insertion point decided by its name. A sortedness check keeps it from decaying.

Measured before and after on the real shape: two PRs each adding a suite go CONFLICT to MERGES CLEANLY. Adjacent names still collide, so this is a large reduction and not a cure, and the PR says so.

The other option was not taken. Generating from test/*.sh with an exclude list would end it completely, but it turns "a stray .sh in test/ reddens the gate until you decide what it is" into "a stray .sh is silently gated". Recorded so the choice is visible.

Two things found on the way, both worse than the conflict

The bad resolution destroys the array, it does not leave a stray command. I had it written down the wrong way:

SUITES=(alpha beta gamma) stray_name
  -> stray_name: command not found
  -> ${#SUITES[@]} is 0

An array literal used as an assignment prefix is scoped to that command, so SUITES ends up unset and the matrix runs no suites at all. The "NO SUITES RAN" guard is the only backstop.

The check that should have caught it was blind by construction. harness_selftest re-implemented bash's array parser in awk. Against a runner carrying the mistake: bash sees 0 suites, the awk parser sees 130 including the stray. So "every suite is registered" passed while the array was destroyed. The runner now answers --list-suites and the gate asks instead of parsing, so there is one parser. Removal-proved.

And the first matrix caught me making the same class of mistake

The first run failed harness_selftest on PG16 and PG17 only, naming four registered suites as unregistered, different names on each major. My listed_suites forked a process per test file, about 250 forks in a six-way matrix, and a transient fork failure returns an empty list that reads as "unregistered". An intermittent red against innocent suites, caused by the gate. Cached now, with an empty answer as a named premise.

Only visible under the full matrix, and only diagnosable because the driver keeps the whole log rather than piping it through tail.

Board

All five PRs from your table are merged, plus #471. No open PRs. #467 and #470 closed.

Next from your alpha2 list unless you redirect: #465 (bench understating our own loader 7x) then #414 slices 3-5, with #461 if you still want it in.

[ChronicallyJD]

Taking this up. The SUITES half is solved and I can show the mechanism working. The alpha2 half is yours, and the table below is rebuilt from the board rather than edited, because every row above is stale.

Every figure here comes from gh pr list, gh issue list or git log. None of it is carried forward from the existing text — that text is what went stale.

The SUITES half: solved, and here is the mechanism rather than the absence of complaints

This issue proposed one-name-per-line, and it is in place: 137 entries, one per line, sorted. Three suites were registered across 08-08 and 08-09 by two different agents:

commit	when	suite
12166b0	08-08 18:24	native_fold_skipguard (#512)
55e06c6	08-09 11:35	native_vecskip (#452 1b-i)
db53812	08-09 12:50	native_exact_selection (#452 1b-ii)

I want to be exact about what that shows. A resolved conflict leaves no trace in history, so I cannot prove no conflict occurred — nobody can, from the repository. What is checkable is the property the fix was for: those three names sort to three different insertion points, so two agents appending do not edit the same line. That is the mechanism, and it is verifiable now rather than inferred from a quiet week.

One hole remains in it, found by the other session and filed as #552: the sortedness check pins no locale. LC_ALL=C and LC_ALL=en_US.UTF-8 disagree about sort_status against sorted_projection, because C compares bytes (_ is 0x5F, before e) while UTF-8 ignores punctuation at the first level. Both orders are correct sorts. We have never seen it because our container defaults to C.UTF-8, which collates like C. Reproduced here. If we disagree about what sorted means we insert at different points and the property stops delivering the merges it exists for, so this is load-bearing rather than cosmetic. #553 fixes it.

The same failure has moved to a new file, and this issue already contains the answer

The other session trial-merged the whole open queue. test/harness_selftest.sh is now what SUITES= used to be:

PR	where it edits
#544	line 667 — end of file
#549	line 667 — end of file
#551	line 667 — end of file
#553	line 305 — middle

I verified the merges myself rather than take them, and my first attempt was wrong in a way worth stating: it reported CONFLICT for all four pairs including the control, because a second git merge cannot run while the first is uncommitted. The conflicted-file list came back empty, which a real conflict never gives. Committing the first merge first:

#544 then #549	CONFLICT in harness_selftest.sh
#544 then #551	CONFLICT in harness_selftest.sh, lib.sh
#549 then #551	CONFLICT in harness_selftest.sh
#553 then any of the three	clean

Each merges clean alone. Three PRs append at the end and collide pairwise; the one that edits the middle does not. That is row two of the measurement this issue was filed about — "one per line, both appended at the end → CONFLICT" — in the file that argues for the fix.

And this issue already says what to do about it, in my own words about SUITES:

I have deliberately not filed it while four PRs are in flight, because the fix touches the line all four of them are fighting over. Once #444, #446 and #468 land, I would like to do this first, before anything else goes in.

Restructuring harness_selftest.sh today would conflict with #544, #549 and #551 simultaneously — it is the same trap, and the policy written here for the last one applies unchanged. So: after the queue drains, not now. Two of the three appending PRs are the other session's, so this is not a complaint about anyone's habits; it is a property of "additions go at the end".

The board, rebuilt

10 open PRs. All MERGEABLE against main individually.

PR	subject
#538	the report must ask cb_verdict (#531)
#543	count a skipped vector where decode skips it (#542)
#544	a cluster that will not start reports why (#537)
#545	CONTEXT.md, what today taught about how a check can mislead
#546	columnar_metadata header, 1 of 3 (#496)
#547	columnar_reader header, 2 of 3 (#496)
#549	the port walk wraps (#548)
#550	the last four module headers (#496)
#551	an in-tree build must not reuse another major's objects (#536)
#553	the sorted-SUITES check pins its collation (#552)

Known conflict points, from the trial merges: the three harness_selftest.sh appenders above, and the header chain — #547 and #550 both conflict after #546, in columnar_reader.c, columnar_tableam.c and columnar_vector.c, which is the include-placement collision. The queue is mergeable as a set, in an order, with a rebase at two points. Nothing here needs a decision to unblock it.

14 PRs merged on 08-09. 22 issues open.

What I am not deciding

The alpha2 scope question is yours and I have not touched it. What I have done is make it decidable: the table above is current, so the scope conversation is no longer buried under twenty dead rows.

If it helps, the closeable-on-evidence set from my side is #536, #537, #542, #548, #552 — each has a gated PR open and a measurement in its thread. #426 and #438 are answered and want a disposition rather than work. #503 and #452 phase 2 want your call before anyone starts.