fix(query+ingest): value-anchored NL→SQL grounding, hierarchy resolution & pipeline robustness

[poche123]

Why

End-to-end testing against two real, messy Excel workbooks (a 25k-row financial MTP table and an 8-sheet sales-structure workbook) exposed that the NL→SQL pipeline guessed SQL literals (e.g. Year IN (2023) when values are FY23, Brand='DAPA' when DAPA lives in Family/TA, summing a scaled FY instead of FY (Real)) and that the control flow reported wrong-but-empty results as confident answers. This PR makes the agents anchor on real data and hardens the query + ingestion pipelines.

Result: a question set that previously answered ~0/9 now answers Section 1 (financial) 4/4 exact and Section 2 (org/sales) 4/5, with the remaining miss flagged rather than confidently wrong.

What changed

Grounding / value anchoring

• profile: store the full distinct value set for low-cardinality columns (was top-5) so filter/CASE literals are copied verbatim.
• retrieval: value_fingerprint() on hits; render a "Column value catalogue" into the grounding and generation prompts; instructions for tall/EAV layouts and scaled-duplicate measures.
• Structurally detect self-referencing hierarchy columns (value containment, with a row-wise discriminator so a duplicate name column isn't mistaken for a manager column) → "a person's team/reports/structure" filters the hierarchy column.

Control flow / governance

• 0-row / degenerate aggregate → clarification + confidence cap (no more confident empties).
• Auto-learn gated on row_count > 0.
• Prefer a candidate that passes the firewall and returns non-empty rows over candidates[0].
• Exclude CTE names from AST table_refs (the firewall was rejecting valid CTE SQL).
• Snapshot-scope retrieval; honor + persist conversation snapshot pins.
• Unify /stream, :explain, :validate on the rendered prompts + firewall/scope guards (they previously ran blind).
• Thread a re-firewalled extra_filter through the semantic-metric bind().

Ingestion robustness / determinism

• Header-quality scoring (skip numeric spacer rows) + stable leading-blank handling.
• temperature=0 for naming/describe agents → re-ingest schema churn 302 → 17 (~94%).
• Rewrite Parquet column names on the no-key fallback (fixes profile/sample Binder errors).
• Index column values into embeddings + content_tsv; warn when the reranker degrades to a no-op.

Validation (vs ground truth computed directly from the Parquet)

	Result
DAPA Brazil revenue FY23/24/25	✅ 175.6 / 127.5 / 119.9M
Operating profit CAMCAR FY24	✅ $252,519,261
Field:HQ Mexico FY23/FY26	✅ 3.49 / 3.40
CAGR Imfinzi/Lynparza/Tagrisso	✅ 96.1 / 22.1 / 19.0%
días-en-campo, consecución, team dimensioning, structure movements	✅ 4/5 (e.g. ANA consecución 0.911 exact)

Known limitations / follow-ups

• Two-hop questions ("the BUM of Calquence's direct reports") aren't resolved — needs nested-entity grounding (the generator stops at the first hop). Tracked as a follow-up.
• Subtotal detection is implemented conservatively but not surfaced as an exclusion directive (name-based exclusion was net-harmful on additive Total_* buckets); a structural value-aggregate == sum-of-others confirmation would re-enable it.
• New logic is not yet unit-tested; task lint not run in this PR.

Note

pyproject.toml is intentionally excluded (it carried a machine-local pyfly path tweak).