Skip to content

G2 S2b: remove X5 Retail Hero loader, retire at-scale benchmark, de-brand repo-wide#156

Merged
brownjuly2003-code merged 4 commits into
mainfrom
feat/g2-s2b-x5-removal
Jul 6, 2026
Merged

G2 S2b: remove X5 Retail Hero loader, retire at-scale benchmark, de-brand repo-wide#156
brownjuly2003-code merged 4 commits into
mainfrom
feat/g2-s2b-x5-removal

Conversation

@brownjuly2003-code

Copy link
Copy Markdown
Owner

Summary

Executes the S2a decision doc's (x5-benchmark-decision.md, deleted in the
final commit) disposition list in full, per the project owner's override:
the "X5 Retail Hero" dataset (a real external grocery retailer's public
Kaggle dataset) must not appear anywhere in this project going forward,
except as a historical mention in CHANGELOG.md.

  • Delete warehouse/agentflow/dv2/loaders/x5_retail_hero/ entirely (no
    successor generator) and its dedicated test. Vault-generic row models
    needed by PostgresVaultWriter's only test coverage
    (test_dv2_postgres_ingestion.py) are relocated to a neutral
    loaders/vault_rows.py.
  • Retire the at-scale load-test benchmark ("45.8M line items / 8.06M
    orders", 2026-06-07) as historical. The harness
    (infrastructure/dv2/load-test/) stays fully runnable and gating against
    the synthetic demo seed; docs/dv2-multi-branch/load-test-baseline.md is
    rewritten to the seed-scale baseline + a "Retired at-scale capture"
    section (full report preserved in this file's git history).
  • Surgical test fixes: test_dv2_supplier_reference.py's X5-loader
    hash-equality pin → self-contained known-vector MD5 digests;
    test_dv2_business_vault_ddl.py / test_dv2_postgres_ddl.py drop the
    negative "x5__" not in DDL guard (the literal string would itself
    violate the no-X5-anywhere rule) and keep the positive "mp__" in DDL
    check.
  • De-brand spec.yaml ("1c (X5 Retail Hero)""1c (branch order feed)") and regenerate its 10 dependent satellite DDL files.
  • De-brand every remaining "X5" / "Retail Hero" / at-scale row-count
    mention repo-wide
    — dbt config, docs/domain.md §5.4,
    schema_dv2.md, demo_evidence.md, generator-spec.md, ops/perf docs,
    clickhouse-sts.yaml, dbt-run-job.yaml, business_vault files, DV2
    loader/reference Python docstrings. Several of these files were not in
    the original G2 audit's or the S2a decision doc's file list (dependency
    discoveries — see judgment calls below).
  • CHANGELOG [Unreleased] entry added — the one sanctioned place
    "X5 Retail Hero" may still appear.
  • x5-benchmark-decision.md deleted in the final commit (its own
    disposition, once executed).

Verification

  • ruff check src/ tests/ scripts/ — clean
  • ruff format --check src/ tests/ scripts/ — clean (361 files)
  • mypy src/ --ignore-missing-imports — clean (124 files)
  • pytest targeted (test_dv2_postgres_ingestion.py,
    test_dv2_supplier_reference.py, test_dv2_business_vault_ddl.py,
    test_dv2_postgres_ddl.py) — 105 passed
  • pytest tests/unit/ tests/property/ (broad) — 1659 passed, 2 failed;
    both failures (test_sdk_wheel_contains_py_typed_marker,
    test_distribution_version_matches_sdk_version) are pre-existing local
    environment gaps (missing hatchling build backend, stale installed SDK
    version) in files this PR never touches (sdk/, tests/unit/test_sdk_packaging.py,
    tests/unit/test_version.py) — unrelated to this change.
  • Grep-clean (git-tracked files only): x5 case-insensitive outside
    CHANGELOG.md returns only expected false positives — docker-compose.e2e.yml
    (-Xmx512m), docs/operations/aws-oidc-setup.md (X509...), and 4 binary
    screenshots/videos with coincidental byte matches. 45.8 / 8.06M /
    402K / "Retail Hero" outside CHANGELOG: clean except one legitimate
    45.8 QPS value in load-test-baseline.md's seed-scale results table
    (not a row count; copied verbatim from the decision doc's ready-made
    content).
  • Rebased cleanly onto origin/main (dc23318, S1's currency/locale sweep,
    PR fix(g2-s1): sweep residual USD/generic-catalog text to kitchen/RUB legend #155) — zero file overlap, all checks re-run post-rebase.

Judgment calls worth a sanity check

  • docs/dv2-multi-branch/schema_dv2.md's "Связь с реальными source-data"
    table previously pointed at three external Kaggle datasets (X5 Retail
    Hero + Lenta BigTarget); rewrote it to point at the synthetic seed / mp__
    feed instead of just deleting the X5 rows, since the table's whole premise
    (external prototype-data provenance) no longer describes the project.
  • Several files outside the S2a decision doc's list needed de-branding
    (clickhouse-sts.yaml's benchmark-tuning comments, dv2/README.md's
    hash-key gotcha pointer, generator-spec.md's stale gs1.py docstring
    note, the two dated docs/perf/vault-pii-governance-pg-verify-2026-07-0{2,3}.md
    transcripts, postgres_oltp/README.md and reference/README.md's loader
    mentions) — treated the same as the decision doc's explicit list (keep
    the engineering rationale/meaning, lose the brand and exact at-scale
    numbers).
  • load-test/README.md's scenario table had two other stale issues beyond
    the decision doc's ask (filenames missing _adhoc suffix, and
    05_line_items_reach's description still said uniqExact when the actual
    query already uses uniq()) — fixed both while touching that table.

🤖 Generated with Claude Code

JuliaEdom added 4 commits July 6, 2026 02:15
Delete warehouse/agentflow/dv2/loaders/x5_retail_hero/ entirely (no successor
generator, per the S2a benchmark-fate decision) and its dedicated test
tests/unit/test_x5_retail_hero_loader.py.

Relocate the vault-generic row models (VaultRow, Hub*, Link*, SatOrderHeader,
SatOrderPricing, SatLinkOrderProduct) from the deleted loader's schemas.py
into a neutral warehouse/agentflow/dv2/loaders/vault_rows.py, since
PostgresVaultWriter's own tests (test_dv2_postgres_ingestion.py) are the only
coverage for that class and must keep running. Surgical edits:

- test_dv2_postgres_ingestion.py: re-point fixtures at vault_rows, drop the
  loader-sink tests (_open_sink/_DryRunSink/_PostgresSink/_ClickHouseSink,
  which die with the loader).
- test_dv2_supplier_reference.py: replace the X5-loader hash-equality pin
  with self-contained known-vector digests for vault_mapping's own MD5
  canonicalisation.
- test_dv2_business_vault_ddl.py / test_dv2_postgres_ddl.py: drop the
  negative "x5__ not in raw" regression guard (the string itself violates
  the no-X5-anywhere mandate); keep the positive "mp__ in raw" check.

De-brand spec.yaml: "1c (X5 Retail Hero)" -> "1c (branch order feed)" across
all 5 sat_order_header__1c__* entries, then regenerate the 10 dependent
satellite DDL files (5 ClickHouse + 5 PostgreSQL) via generate_satellites.py
so the generated header comments come out de-X5'd too.

Part of the G2 audit follow-up (see x5-benchmark-decision.md for the full
disposition list; deleted in this step's final commit).
Remove all remaining "X5" / "Retail Hero" / retired at-scale row-count
mentions (45.8M, 8.06M, 402K) outside CHANGELOG.md, per the project owner's
override that the dataset must not appear anywhere else in the project.

Load-test harness comments (run-bench.sh, job.yaml) and README get the
ready-made de-branded rationale text from x5-benchmark-decision.md §3;
docs/dv2-multi-branch/load-test-baseline.md is rewritten entirely with the
seed-scale baseline + a "Retired at-scale capture" section per §4 (full
report stays in this file's git history).

Overlap de-branding beyond the load-test scope:
- docs/domain.md §5.4, docs/generator-spec.md: reword the x5__ -> mp__
  record_source rename entry and GS1 docstring note without the literal
  Kaggle/X5 strings.
- docs/dv2-multi-branch/schema_dv2.md: drop the "X5 Retail Hero / Lenta
  BigTarget" naming-convention credit and rewrite the source-data table to
  point at the synthetic seed / mp__ feed instead of external Kaggle data.
- docs/dv2-multi-branch/demo_evidence.md: "X5-era" flags -> "retired-seed-era";
  de-brand the legend-reset paragraph and the retired-capture callouts while
  keeping their meaning.
- docs/operations/{aws-oidc-setup,openssf-security-posture}.md,
  docs/perf/vault-pii-governance-pg-verify-2026-07-0{2,3}.md: reword stray
  "X5" mentions in ops/perf docs the original audit's file list missed.
- infrastructure/dv2/clickhouse-sts.yaml, infrastructure/dv2/dbt/dbt-run-job.yaml,
  warehouse/agentflow/dv2/{dbt,business_vault}/*: de-brand resource-tuning
  rationale comments (ClickHouse memory/merge config, dbt profile threads,
  the bv_order_canonical_mat staged loader) the same way as the load-test
  comments — keep the engineering rationale, lose the brand and exact
  at-scale row counts.
- warehouse/agentflow/dv2/{README.md,loaders/pg_vault_writer.py,
  postgres_oltp/README.md,reference/{README.md,load_postgres.py,vault_mapping.py}}:
  reword Python/README references to "the X5 loader" now that it is deleted
  — point instead at the still-live reference loader / vault_rows models /
  "the other DV2 loaders", and describe vault_mapping's MD5 convention on
  its own terms instead of by comparison to the removed loader.

Dependency discoveries beyond the S2a decision doc's file list: dv2/README.md's
hash-key gotcha note, generator-spec.md's stale gs1.py docstring pointer,
clickhouse-sts.yaml's benchmark-tuning comments, dbt-run-job.yaml's thread
comment, postgres_oltp/README.md and reference/README.md's loader mentions,
and the two dated pg-verify perf transcripts — none were in the original
audit's or S2a's file list.
…S2b)

CHANGELOG.md is the one sanctioned place "X5 Retail Hero" may still appear,
per the project owner's override executed in this step.
The S2a decision doc's disposition list is fully executed: loader package
removed, benchmark retired as historical, dependent tests fixed, and every
overlap file de-branded. This process document intentionally contained the
string "X5" for its own instructions; deleting it in this final commit is
what makes the repo-wide "no X5 outside CHANGELOG.md" gate pass. The
decision record survives in this PR and in git history.
@github-actions

github-actions Bot commented Jul 5, 2026

Copy link
Copy Markdown

DORA Metrics

  • Window: last 30 days
  • Branch: main
  • Deployment frequency: 83 total / 19.37 per week
  • Lead time for changes: avg 0.56h / median 0.0h
  • Change failure rate: 56.63% (47/83)
  • MTTR: n/a across 0 incident(s)

MTTR note: No failed mainline CI runs in the selected window.

@brownjuly2003-code brownjuly2003-code merged commit 56e10f6 into main Jul 6, 2026
19 checks passed
@brownjuly2003-code brownjuly2003-code deleted the feat/g2-s2b-x5-removal branch July 6, 2026 00:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants