G2 S2b: remove X5 Retail Hero loader, retire at-scale benchmark, de-brand repo-wide#156
Merged
Merged
Conversation
Delete warehouse/agentflow/dv2/loaders/x5_retail_hero/ entirely (no successor generator, per the S2a benchmark-fate decision) and its dedicated test tests/unit/test_x5_retail_hero_loader.py. Relocate the vault-generic row models (VaultRow, Hub*, Link*, SatOrderHeader, SatOrderPricing, SatLinkOrderProduct) from the deleted loader's schemas.py into a neutral warehouse/agentflow/dv2/loaders/vault_rows.py, since PostgresVaultWriter's own tests (test_dv2_postgres_ingestion.py) are the only coverage for that class and must keep running. Surgical edits: - test_dv2_postgres_ingestion.py: re-point fixtures at vault_rows, drop the loader-sink tests (_open_sink/_DryRunSink/_PostgresSink/_ClickHouseSink, which die with the loader). - test_dv2_supplier_reference.py: replace the X5-loader hash-equality pin with self-contained known-vector digests for vault_mapping's own MD5 canonicalisation. - test_dv2_business_vault_ddl.py / test_dv2_postgres_ddl.py: drop the negative "x5__ not in raw" regression guard (the string itself violates the no-X5-anywhere mandate); keep the positive "mp__ in raw" check. De-brand spec.yaml: "1c (X5 Retail Hero)" -> "1c (branch order feed)" across all 5 sat_order_header__1c__* entries, then regenerate the 10 dependent satellite DDL files (5 ClickHouse + 5 PostgreSQL) via generate_satellites.py so the generated header comments come out de-X5'd too. Part of the G2 audit follow-up (see x5-benchmark-decision.md for the full disposition list; deleted in this step's final commit).
Remove all remaining "X5" / "Retail Hero" / retired at-scale row-count
mentions (45.8M, 8.06M, 402K) outside CHANGELOG.md, per the project owner's
override that the dataset must not appear anywhere else in the project.
Load-test harness comments (run-bench.sh, job.yaml) and README get the
ready-made de-branded rationale text from x5-benchmark-decision.md §3;
docs/dv2-multi-branch/load-test-baseline.md is rewritten entirely with the
seed-scale baseline + a "Retired at-scale capture" section per §4 (full
report stays in this file's git history).
Overlap de-branding beyond the load-test scope:
- docs/domain.md §5.4, docs/generator-spec.md: reword the x5__ -> mp__
record_source rename entry and GS1 docstring note without the literal
Kaggle/X5 strings.
- docs/dv2-multi-branch/schema_dv2.md: drop the "X5 Retail Hero / Lenta
BigTarget" naming-convention credit and rewrite the source-data table to
point at the synthetic seed / mp__ feed instead of external Kaggle data.
- docs/dv2-multi-branch/demo_evidence.md: "X5-era" flags -> "retired-seed-era";
de-brand the legend-reset paragraph and the retired-capture callouts while
keeping their meaning.
- docs/operations/{aws-oidc-setup,openssf-security-posture}.md,
docs/perf/vault-pii-governance-pg-verify-2026-07-0{2,3}.md: reword stray
"X5" mentions in ops/perf docs the original audit's file list missed.
- infrastructure/dv2/clickhouse-sts.yaml, infrastructure/dv2/dbt/dbt-run-job.yaml,
warehouse/agentflow/dv2/{dbt,business_vault}/*: de-brand resource-tuning
rationale comments (ClickHouse memory/merge config, dbt profile threads,
the bv_order_canonical_mat staged loader) the same way as the load-test
comments — keep the engineering rationale, lose the brand and exact
at-scale row counts.
- warehouse/agentflow/dv2/{README.md,loaders/pg_vault_writer.py,
postgres_oltp/README.md,reference/{README.md,load_postgres.py,vault_mapping.py}}:
reword Python/README references to "the X5 loader" now that it is deleted
— point instead at the still-live reference loader / vault_rows models /
"the other DV2 loaders", and describe vault_mapping's MD5 convention on
its own terms instead of by comparison to the removed loader.
Dependency discoveries beyond the S2a decision doc's file list: dv2/README.md's
hash-key gotcha note, generator-spec.md's stale gs1.py docstring pointer,
clickhouse-sts.yaml's benchmark-tuning comments, dbt-run-job.yaml's thread
comment, postgres_oltp/README.md and reference/README.md's loader mentions,
and the two dated pg-verify perf transcripts — none were in the original
audit's or S2a's file list.
…S2b) CHANGELOG.md is the one sanctioned place "X5 Retail Hero" may still appear, per the project owner's override executed in this step.
The S2a decision doc's disposition list is fully executed: loader package removed, benchmark retired as historical, dependent tests fixed, and every overlap file de-branded. This process document intentionally contained the string "X5" for its own instructions; deleting it in this final commit is what makes the repo-wide "no X5 outside CHANGELOG.md" gate pass. The decision record survives in this PR and in git history.
DORA Metrics
MTTR note: No failed mainline CI runs in the selected window. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Executes the S2a decision doc's (
x5-benchmark-decision.md, deleted in thefinal commit) disposition list in full, per the project owner's override:
the "X5 Retail Hero" dataset (a real external grocery retailer's public
Kaggle dataset) must not appear anywhere in this project going forward,
except as a historical mention in
CHANGELOG.md.warehouse/agentflow/dv2/loaders/x5_retail_hero/entirely (nosuccessor generator) and its dedicated test. Vault-generic row models
needed by
PostgresVaultWriter's only test coverage(
test_dv2_postgres_ingestion.py) are relocated to a neutralloaders/vault_rows.py.orders", 2026-06-07) as historical. The harness
(
infrastructure/dv2/load-test/) stays fully runnable and gating againstthe synthetic demo seed;
docs/dv2-multi-branch/load-test-baseline.mdisrewritten to the seed-scale baseline + a "Retired at-scale capture"
section (full report preserved in this file's git history).
test_dv2_supplier_reference.py's X5-loaderhash-equality pin → self-contained known-vector MD5 digests;
test_dv2_business_vault_ddl.py/test_dv2_postgres_ddl.pydrop thenegative
"x5__" not in DDLguard (the literal string would itselfviolate the no-X5-anywhere rule) and keep the positive
"mp__" in DDLcheck.
"1c (X5 Retail Hero)"→"1c (branch order feed)") and regenerate its 10 dependent satellite DDL files.mention repo-wide — dbt config,
docs/domain.md§5.4,schema_dv2.md,demo_evidence.md,generator-spec.md, ops/perf docs,clickhouse-sts.yaml,dbt-run-job.yaml, business_vault files, DV2loader/reference Python docstrings. Several of these files were not in
the original G2 audit's or the S2a decision doc's file list (dependency
discoveries — see judgment calls below).
[Unreleased]entry added — the one sanctioned place"X5 Retail Hero" may still appear.
x5-benchmark-decision.mddeleted in the final commit (its owndisposition, once executed).
Verification
ruff check src/ tests/ scripts/— cleanruff format --check src/ tests/ scripts/— clean (361 files)mypy src/ --ignore-missing-imports— clean (124 files)pytesttargeted (test_dv2_postgres_ingestion.py,test_dv2_supplier_reference.py,test_dv2_business_vault_ddl.py,test_dv2_postgres_ddl.py) — 105 passedpytest tests/unit/ tests/property/(broad) — 1659 passed, 2 failed;both failures (
test_sdk_wheel_contains_py_typed_marker,test_distribution_version_matches_sdk_version) are pre-existing localenvironment gaps (missing
hatchlingbuild backend, stale installed SDKversion) in files this PR never touches (
sdk/,tests/unit/test_sdk_packaging.py,tests/unit/test_version.py) — unrelated to this change.x5case-insensitive outsideCHANGELOG.mdreturns only expected false positives —docker-compose.e2e.yml(
-Xmx512m),docs/operations/aws-oidc-setup.md(X509...), and 4 binaryscreenshots/videos with coincidental byte matches.
45.8/8.06M/402K/ "Retail Hero" outside CHANGELOG: clean except one legitimate45.8QPS value inload-test-baseline.md's seed-scale results table(not a row count; copied verbatim from the decision doc's ready-made
content).
origin/main(dc23318, S1's currency/locale sweep,PR fix(g2-s1): sweep residual USD/generic-catalog text to kitchen/RUB legend #155) — zero file overlap, all checks re-run post-rebase.
Judgment calls worth a sanity check
docs/dv2-multi-branch/schema_dv2.md's "Связь с реальными source-data"table previously pointed at three external Kaggle datasets (X5 Retail
Hero + Lenta BigTarget); rewrote it to point at the synthetic seed /
mp__feed instead of just deleting the X5 rows, since the table's whole premise
(external prototype-data provenance) no longer describes the project.
(
clickhouse-sts.yaml's benchmark-tuning comments,dv2/README.md'shash-key gotcha pointer,
generator-spec.md's stalegs1.pydocstringnote, the two dated
docs/perf/vault-pii-governance-pg-verify-2026-07-0{2,3}.mdtranscripts,
postgres_oltp/README.mdandreference/README.md's loadermentions) — treated the same as the decision doc's explicit list (keep
the engineering rationale/meaning, lose the brand and exact at-scale
numbers).
load-test/README.md's scenario table had two other stale issues beyondthe decision doc's ask (filenames missing
_adhocsuffix, and05_line_items_reach's description still saiduniqExactwhen the actualquery already uses
uniq()) — fixed both while touching that table.🤖 Generated with Claude Code