diff --git a/CHANGELOG.md b/CHANGELOG.md index 82907b73..61376838 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,57 @@ All notable changes to AgentFlow are documented in this file. ## [Unreleased] +### Removed — X5 Retail Hero loader deleted; at-scale benchmark retired as historical (G2 S2b, 2026-07-05) + +- **Deleted `warehouse/agentflow/dv2/loaders/x5_retail_hero/`** in full + (`loader.py`, `mappers.py`, `schemas.py`, `branch_distributor.py`, + `README.md`, `requirements.txt`) and its dedicated test + `tests/unit/test_x5_retail_hero_loader.py`. Per the project owner's + override, the "X5 Retail Hero" dataset — a real external grocery + retailer's public Kaggle dataset that the demo's synthetic legend never + needed — must not appear anywhere in the project going forward; this + changelog entry is the one place that name may still be written. No + successor bulk generator is built (S2a decision, formerly + `x5-benchmark-decision.md`, deleted in this same step once executed): + demo-scale raw-vault data is fully covered by + `warehouse/agentflow/dv2/synthetic_seed.sql` + + `satellite_seed_all_branches.sql`, the kitchen live generator + (`src/ingestion/`), and `reference/load_postgres.py`. +- **At-scale load-test benchmark retired as historical.** The 2026-06-07 + ClickHouse capture (tens of millions of raw-vault rows, loaded from the + now-deleted dataset) is no longer presented as a current baseline. The + load-test harness (`infrastructure/dv2/load-test/`) stays fully runnable + and gating against the synthetic demo seed unchanged (including the + `P99_MS_POINT=250` budget, whose queueing rationale is preserved in + `job.yaml`'s comments). `docs/dv2-multi-branch/load-test-baseline.md` now + documents only the seed-scale baseline; the retired capture's three + engineering findings (`customer_360` sort key, `uniq()` vs `uniqExact()`, + the point-budget queueing analysis) are summarized there and the full + report is preserved in this file's git history. +- **Test surgery**: the vault-generic row models used by both + `tests/unit/test_dv2_postgres_ingestion.py` (the only coverage for + `PostgresVaultWriter`) and the deleted loader now live in + `warehouse/agentflow/dv2/loaders/vault_rows.py`. The loader-sink tests + (`_open_sink` / `_DryRunSink` / `_PostgresSink` / `_ClickHouseSink`) are + removed along with the loader. `tests/unit/test_dv2_supplier_reference.py`'s + hash-equality pin now checks `vault_mapping`'s own MD5 canonicalisation + against precomputed known-vector digests instead of importing the deleted + loader; `test_dv2_business_vault_ddl.py` / `test_dv2_postgres_ddl.py` drop + the "`x5__` not in DDL" regression guard (the check string itself would + violate the no-X5-anywhere rule) and keep the positive "`mp__` in DDL" + assertion. +- **De-branded** every remaining reference across `spec.yaml` (and its 10 + generated satellite DDL files), the dbt config (`sources.yml`, `README.md`, + `profiles.example.yml`), `docs/domain.md`, `docs/dv2-multi-branch/{schema_dv2,demo_evidence}.md`, + `docs/generator-spec.md`, `docs/operations/{aws-oidc-setup,openssf-security-posture}.md`, + `docs/perf/vault-pii-governance-pg-verify-2026-07-0{2,3}.md`, + `infrastructure/dv2/{clickhouse-sts.yaml,dbt/dbt-run-job.yaml,load-test/*}`, + and the DV2 loader/reference Python docstrings (`pg_vault_writer.py`, + `vault_mapping.py`, `reference/README.md`, `reference/load_postgres.py`, + `postgres_oltp/README.md`, `dv2/README.md`) — no "X5" / "Retail Hero" / + retired at-scale row-count strings ("45.8M", "8.06M", "402K") survive + outside this file and git history. + ### Added — Operational serving split decided; ops-surfaces spec (ADR 0011, 2026-07-03) - **New [ADR 0011](docs/decisions/0011-ops-serving-split.md)** — the design diff --git a/docs/domain.md b/docs/domain.md index d743b9dc..43b7f254 100644 --- a/docs/domain.md +++ b/docs/domain.md @@ -208,7 +208,7 @@ renames. The two exceptions are listed in §5.4. | Change | Scope | Status | | ------ | ----- | ------ | -| `x5__*` → `mp__*` record_source (+ governance SQL, officer probes, admission tests) | The prefix carried the name of the Kaggle seed dataset (X5 Retail Hero) that the demo loader replays as transaction history. Under the legend it is the **consolidated marketplace feed**, and the prefix says so. Dataset attribution stays in the loader README | **Done** (B2) | +| Former external-dataset record_source prefix → `mp__*` (+ governance SQL, officer probes, admission tests) | The prefix carried the name of a retired public benchmark dataset that the demo loader once replayed as transaction history. Under the legend it is the **consolidated marketplace feed**, and the prefix says so. The dataset and its loader have since been removed from the codebase entirely (see CHANGELOG.md) | **Done** (B2) | | Demo value repin: currencies to `RUB` (primary), `AED`/`KZT` in branch stories; demo revenue/counts consistent with §1–2 | `contracts/entities/order.yaml` currency examples, NL demo answers, seeded `ORD-*` rows | Planned (data phase, after the generator spec) | Vocabulary guardrails for all public docs: the company is an **own-brand / diff --git a/docs/dv2-multi-branch/demo_evidence.md b/docs/dv2-multi-branch/demo_evidence.md index 28037103..1294e87b 100644 --- a/docs/dv2-multi-branch/demo_evidence.md +++ b/docs/dv2-multi-branch/demo_evidence.md @@ -14,14 +14,15 @@ Docker: (schema → 8 hubs → 8 links → 48 satellites → `03_business_vault.sql`) → `postgres/governance/01..04.sql`. -> **Legend reset (2026-07-03).** The demo no longer models an X5 retail -> contractor. It is an own-brand ("private-label smart-kitchen") importer: -> China-manufactured goods, RU HQ, bimodal channel economy (money in -> wholesale, order-count on marketplaces), five branches across three -> jurisdictions (msk / spb / ekb + dxb + ala). All prior "X5 Retail Hero / -> 8.06M orders / USD" figures are retired. The synthetic demo seed is now -> **2,500 customers / 10,000 orders / 160 SKU / 14,853 line items, priced in -> ₽** — see `docs/domain.md` and `docs/generator-spec.md` for the model. +> **Legend reset (2026-07-03).** The demo no longer models a bulk-retail +> contractor built on a public benchmark dataset. It is an own-brand +> ("private-label smart-kitchen") importer: China-manufactured goods, RU HQ, +> bimodal channel economy (money in wholesale, order-count on marketplaces), +> five branches across three jurisdictions (msk / spb / ekb + dxb + ala). All +> prior at-scale order-count / USD figures are retired. The synthetic demo +> seed is now **2,500 customers / 10,000 orders / 160 SKU / 14,853 line +> items, priced in ₽** — see `docs/domain.md` and `docs/generator-spec.md` +> for the model. > **⚠ Infra sections (§1–3, §9–15) are pending re-capture on the kind > cluster (Mac stand).** Topology, workload pinning, PVCs, the MinIO cold @@ -30,8 +31,8 @@ Docker: > kind cluster; the standalone WSL-CH / Windows-PG split used for this > re-capture cannot reproduce cross-engine networking or Kubernetes. Their > **mechanisms are volume- and legend-independent** and stand as previously -> demonstrated, but the **row counts printed in those blocks reflect the -> retired X5-era volumes** and must not be read as current. They are flagged +> demonstrated, but the **row counts printed in those blocks reflect +> retired-seed-era volumes** and must not be read as current. They are flagged > inline and are the Mac-tail of this step (see `plan_endgame_02_07_26.md`). ## Governance verify_live — both engines green on the new seeds @@ -47,8 +48,9 @@ against the freshly-built stands: The PII boundary holds in every SQL shape on both engines; row policies scope officers to their own jurisdiction; the msk demo seed now spans `1c__msk`, -`pg_ops__msk` and `mp__msk` record sources (the `x5__` convention was retired -in B2). The current CH suite defines 29 probes (earlier revisions cited 32); +`pg_ops__msk` and `mp__msk` record sources (the legacy marketplace-seed +prefix was retired in B2). The current CH suite defines 29 probes (earlier +revisions cited 32); the count is whatever the checked-in script asserts — every probe passes. ## 1. Cluster topology — `kubectl get nodes --show-labels` @@ -114,7 +116,7 @@ sources; full generation in `spec.yaml` + `generate_satellites.py`). ## 5. Multi-branch distribution proof -The retired X5 seed spread orders 40/25/15/10/10 across branches. The current +The retired at-scale seed spread orders 40/25/15/10/10 across branches. The current legend does **not** — every marketplace and e-com order is fulfilled from the msk hub (`mp__` is msk-only), and branch identity lives in the dealer / B2B / PII layers, not in the marketplace order stream. Order distribution by @@ -157,8 +159,8 @@ branch, count() FROM rv.hub_order GROUP BY branch ORDER BY 2 DESC"`: This is the **demo-scale synthetic seed** (10k orders), not a load benchmark. The multi-million-row throughput characterisation is a separate artifact ([`load-test-baseline.md`](load-test-baseline.md)) and is not part of the -synthetic demo evidence; the earlier "1.1 s over 8.06M X5 rows" block is -retired with the X5 legend. +synthetic demo evidence; the earlier multi-million-row latency capture is +retired with the 2026-07-03 legend reset. ## 7. Line items reach @@ -248,7 +250,7 @@ verify_live matrix above). > ⚠ **Kind-cluster section — pending Mac re-capture.** MinIO + the ClickHouse > `s3()` CronJobs need the shared-network cluster; the standalone stand cannot -> reproduce them. Row counts below are X5-era and await refresh on the new +> reproduce them. Row counts below are retired-seed-era and await refresh on the new > seeds. The mechanism (native `s3()` write + read-back, PII-free source > selection) is unchanged. @@ -273,7 +275,7 @@ cloud-provider secret takes its place. > ⚠ **Kind-cluster section — pending Mac re-capture.** The bridge needs > ClickHouse and Postgres on one network; the standalone split (CH in WSL, > Postgres on the Windows host loopback) cannot reach across. Row counts and -> the `Dasha/Egor/Fedor`-style names below are X5-era and await refresh on the +> the `Dasha/Egor/Fedor`-style names below are retired-seed-era and await refresh on the > new kitchen-legend seeds. The code path (Postgres → CH `Engine=PostgreSQL()` > live read-through → raw_vault → business_vault) is unchanged. @@ -315,7 +317,7 @@ bash infrastructure/dv2/bootstrap.sh # idempotent rebuild on the kind cluster ## 12. Argo Workflows orchestration > ⚠ **Kind-cluster section — pending Mac re-capture.** Timings/counts below are -> X5-era. DAG ordering (hub → link → satellite → cold-offload) is enforced by +> retired-seed-era. DAG ordering (hub → link → satellite → cold-offload) is enforced by > dependencies, not clock-time; that property is legend-independent. `infrastructure/dv2/argo/` deploys Argo Workflows plus a `dv2-refresh` @@ -332,7 +334,7 @@ are never out of sync with the warm tier. ## 13. dbt mart layer > ⚠ **Kind-cluster section — pending Mac re-capture.** The dbt-in-Kubernetes -> Job and its per-branch row counts are X5-era. The three marts + 12 tests are +> Job and its per-branch row counts are retired-seed-era. The three marts + 12 tests are > legend-independent in structure; the numbers await refresh on the new seeds. `warehouse/agentflow/dv2/dbt/` ships three materialized marts and 12 data @@ -347,7 +349,7 @@ mart) — the same 12/5/20% rates verified live in §8. > ⚠ **Kind-cluster section — pending Mac re-capture.** MaterializedPostgreSQL > consumes the Postgres WAL via logical replication and needs both engines on > one network with `wal_level=logical`; the standalone split cannot reproduce -> it. Contents are X5-era. +> it. Contents are retired-seed-era. The pull-based `oltp_live` bridge is replaced by a single `oltp_cdc` ClickHouse database backed by `MaterializedPostgreSQL`, consuming the Postgres @@ -359,7 +361,7 @@ rows in raw_vault. ## 15. Per-branch CDC fan-out -> ⚠ **Kind-cluster section — pending Mac re-capture.** Contents are X5-era. +> ⚠ **Kind-cluster section — pending Mac re-capture.** Contents are retired-seed-era. Operational reality wants a single branch to be pausable, re-snapshotable and rotatable without touching another branch's stream. ClickHouse 25.5+ rejects a diff --git a/docs/dv2-multi-branch/load-test-baseline.md b/docs/dv2-multi-branch/load-test-baseline.md index 34146ee7..3930950b 100644 --- a/docs/dv2-multi-branch/load-test-baseline.md +++ b/docs/dv2-multi-branch/load-test-baseline.md @@ -1,90 +1,22 @@ # DV2.0 ClickHouse load test — baseline -Captured **2026-06-07** against the `hq-demo` kind cluster on the iMac demo -host, **real X5 Retail Hero data**: 8.06M orders / 45.8M line items across 5 -branches (`rv.*` raw vault ≈ 3.5 GiB on disk, `rv.bv_order_canonical_mat` -≈ 538 MiB). Harness: [`infrastructure/dv2/load-test/`](../../infrastructure/dv2/load-test/) -— `clickhouse-benchmark` run as a Kubernetes Job in namespace `dv2`, driving +Captured **2026-06-02** against the `hq-demo` kind cluster on the iMac demo +host, **synthetic demo seed** (~10K orders across 5 branches). Harness: +[`infrastructure/dv2/load-test/`](../../infrastructure/dv2/load-test/) — +`clickhouse-benchmark` run as a Kubernetes Job in namespace `dv2`, driving the in-cluster `clickhouse` service over the native protocol (9000). -VM at capture time: 6 GiB RAM, 2 vCPU; ClickHouse pod limit 5 Gi. Server-level -memory accounting is disabled on this stack (see `clickhouse-sts.yaml` — the -2026-06-07 saga); the per-query cap and the cgroup limit guard the pod. -Serving scenarios sweep concurrency 1 / 4 / 8 × 60 iterations; raw-vault -`adhoc` scenarios run **c=1 × 2 iterations** — at X5 a single recompute runs -for minutes (below), so a concurrency sweep would measure nothing but a queue. +VM at capture time: 6 GiB RAM, 2 vCPU; ClickHouse pod limit 5 Gi. Budgets at +capture time: point p99 ≤ 200 ms (since moved to 250 ms — see finding 3 +below), heavy p99 ≤ 1000 ms, adhoc p99 ≤ 2000 ms (adhoc is informational — +reported, never gates). -## Results (X5, 2026-06-07 — final gating run) +> **Refresh pending.** This capture predates the 2026-07-03 legend reset +> (kitchen-appliance wholesaler seed). Re-run `apply.sh` on the Mac demo +> host against the current seed and replace the table below; harness +> mechanics and budgets are unchanged. -| Scenario | Class | c=1 p99 | c=4 p99 | c=8 p99 | c=8 QPS | Verdict | -|----------|-------|--------:|--------:|--------:|--------:|---------| -| `01_branch_pnl_adhoc` — live `bv_order_canonical` recompute | adhoc | **238 s** | — | — | — | INFO | -| `02_top_products_adhoc` — `lnk_order_product` ⋈ `hub_order` top-N | adhoc | **102 s** | — | — | — | INFO | -| `03_customer360_point` — single-row mart lookup by `customer_bk` | point | 42 ms | 100 ms | 197 ms | 44.3 | PASS | -| `04_returns_velocity` — mart aggregation | heavy | 108 ms | 92 ms | 141 ms | 62* | PASS | -| `05_line_items_reach_adhoc` — 45.8M-row join + `uniq` | adhoc | **294 s** | — | — | — | INFO | -| `06_branch_pnl_mart` — **materialized** `marts.branch_pnl` | heavy | 20 ms | 72 ms | 293 ms | 41.3 | PASS | - -Budgets: point p99 ≤ 250 ms, heavy p99 ≤ 1000 ms, adhoc p99 ≤ 2000 ms (adhoc -is informational — reported, never gates). `LOAD TEST: PASS`. - -\* between-run variance on this shared 2-vCPU host is ~2× on sibling cells -(e.g. 04 c=8 p99 ranged 141→396 ms across four captures the same day); single -cells are indicative, the gate verdicts were stable across runs once the real -findings below were fixed. - -## What the X5 run surfaced (three real findings) - -**1. `customer_360` had the wrong sort key for its serving pattern.** The -mart was `ORDER BY (branch, customer_hk)`, but its point query — and the -realistic serving access — looks up by **`customer_bk`**, which was not in -the key at all: every lookup full-scanned the 402K-row mart. At the synthetic -800 rows this was invisible; at X5 it was p99 250–468 ms vs the 200 ms point -budget. Fixed in the dbt model: `ORDER BY (customer_bk, branch)` + -`index_granularity 1024` (a point lookup now reads one ~1K-row granule) -→ p99 42 / 100 / 197 ms at c=1/4/8. - -**2. Exact distinct at X5 is not an interactive query on this host.** -`05_line_items_reach` used `uniqExact(order_hk)` over 45.8M rows — the state -holds every one of 8M keys and needs > 3 GiB even with spill; it OOM'd its -per-query cap at every setting tried. Switched to `uniq()` (HyperLogLog, -~KB per group, ~1% error): completes in ~5 min as a capacity reference. -The exact-count use case belongs in a mart or an offline job. - -**3. The point budget is a queueing budget at c=8 on 2 vCPU.** With the sort -key fixed, c=1 p99 is 42 ms but c=8 p50 is ~127 ms with QPS plateaued from -c=4 — past the box's concurrency design point, p99 measures the queue, not -the data path. Budget moved 200 → 250 ms with the rationale recorded in -`load-test/job.yaml`; the pre-fix full-scan regression (276–468 ms at c=1/4) -would still fail it. - -## Architecture takeaway — the X5 numbers ARE the pitch - -The same business question (branch P&L): - -- **materialized mart** (`06`): p99 **20 ms** at c=1, still ≤ 293 ms with 8 - concurrent clients; -- **live raw-vault recompute** (`01`): **238 seconds** for a single client — - a ~10,000× gap, and it only runs at all with hand-tuned spill settings. - -At the 10K synthetic seed this gap was 3–9×; real volume widened it by three -orders of magnitude. Materialization is the serving path; the live view is an -exploration/debug convenience. The speed comes from **materialization**, not -from dbt: `marts.*` are physical `MergeTree` tables sorted for their access -pattern, the expensive UNION ALL + argMax SCD2 collapse + joins are paid once -at build time — by the staged loader for `bv_order_canonical_mat` -(`warehouse/agentflow/dv2/business_vault/load_bv_order_canonical_mat.sh`) and -by dbt for the marts (dependency DAG, tests, docs). dbt contributes -reproducibility and testability; the latency comes from the physical layout. - -The mart build itself tells the same story: against the live view the three -dbt marts OOM'd four runs in a row at X5; against the materialized canonical -they build in **4–14 seconds each** with 12/12 tests green on the same 5Gi -pod. - -## Previous capture (2026-06-02, synthetic 10K seed) - -Kept for contrast; budgets then: point ≤ 200 ms. +## Results (synthetic seed, 2026-06-02) | Scenario | c=1 p99 | c=4 p99 | c=8 p99 | c=8 QPS | Verdict | |----------|--------:|--------:|--------:|--------:|---------| @@ -95,9 +27,43 @@ Kept for contrast; budgets then: point ≤ 200 ms. | `05_line_items_reach` | 275 ms | 397 ms | 310 ms | 45.8 | PASS | | `06_branch_pnl_mart` | 61 ms | 68 ms | 97 ms | 91.7 | PASS | -That capture's two findings (pod memory limit was the ceiling, not the VM; +Scenarios `02` and `05` have since been reclassified as informational +`*_adhoc` scenarios (c=1 only); the next capture will report them that way. +This capture's two findings (pod memory limit was the ceiling, not the VM; the live 5-join view needs frugal join settings under load) are preserved in -git history and remain true — the X5 capture extends both. +git history. + +Even at seed scale the mart-vs-live-view boundary is measurable: the same +business question (branch P&L) answers 3–9× faster off the materialized +`marts.branch_pnl` (`06`) than through the live `rv.bv_order_canonical` +recompute (`01`), and the gap widens superlinearly with volume. The +materialized mart is the serving path; the live view is an exploration/debug +convenience. + +## Retired at-scale capture (2026-06-07) + +A one-off capture against a bulk public seed dataset (tens of millions of +raw-vault rows; dataset and loader retired with the 2026-07-03 legend reset) +ran on this same harness. The full report lives in the git history of this +file. Its three engineering findings are permanent and survive in the +current code: + +1. **`customer_360` had the wrong sort key for its serving pattern.** The + mart was `ORDER BY (branch, customer_hk)` while its point query looks up + by `customer_bk` — invisible at seed scale, a full mart scan at volume + (p99 250–468 ms vs the then-200 ms budget). Fixed in the dbt model: + `ORDER BY (customer_bk, branch)` + `index_granularity 1024` (a point + lookup reads one ~1K-row granule). +2. **Exact distinct is not an interactive query at volume.** + `05_line_items_reach` originally used `uniqExact(order_hk)`; the state + OOM'd its per-query cap at every setting tried. Switched to `uniq()` + (HyperLogLog, ~KB per group, ~1 % error). The exact-count use case + belongs in a mart or an offline job. +3. **The point budget is a queueing budget at c=8 on 2 vCPU.** Past the + box's concurrency design point, p99 measures the queue, not the data + path. The point budget moved 200 → 250 ms with the rationale recorded in + `load-test/job.yaml`; the pre-fix full-scan regression (276–468 ms) + would still fail it. ## Re-run / refresh diff --git a/docs/dv2-multi-branch/schema_dv2.md b/docs/dv2-multi-branch/schema_dv2.md index 385f9031..e97410df 100644 --- a/docs/dv2-multi-branch/schema_dv2.md +++ b/docs/dv2-multi-branch/schema_dv2.md @@ -4,7 +4,7 @@ Multi-source / multi-branch DWH own-brand импортёра малой бытовой техники (легенда: [`domain.md`](../domain.md)). 5 локаций (MSK / SPB / EKB / DXB / ALA). Источники: 1С (УТ/ЗУП), Битрикс24, WMS, Excel-логистика (контейнерные манифесты), собственный D2C-сайт, API маркетплейсов (ВБ/Озон). -Naming convention основана на публичных данных и engineering-материалах крупных ритейлеров (X5 Retail Hero, Lenta BigTarget, Магнит). DDL-шаблоны — адаптация Celestinfo + Tampere DV2.0 automation patterns под ClickHouse. +Naming convention основана на публичных engineering-материалах крупных ритейлеров (retail Data Vault case studies). DDL-шаблоны — адаптация Celestinfo + Tampere DV2.0 automation patterns под ClickHouse. ## Naming conventions @@ -178,15 +178,20 @@ Idempotent re-loading через `hash_diff` — если satellite-payload не - **PIT tables:** для критичных pull-запросов с фиксированной датой - **Effectivity satellites:** open/close intervals для SCD2-tracking на links -## Связь с реальными source-data +## Связь с текущими source-data -| Hub/Sat | Прототип-данные | Where to get | +| Hub/Sat | Данные | Где взять | |---|---|---| -| `hub_customer` + `sat_customer_personal__*` | X5 Retail Hero `clients.csv` | Kaggle | -| `hub_product` + `sat_product_catalog__*` | X5 Retail Hero `products.csv` | Kaggle | -| `lnk_order_customer` + `sat_lnk_order_product__*` | X5 Retail Hero `purchases.csv` (45.8 млн строк) | Kaggle | -| `sat_customer_loyalty__*` | Lenta BigTarget loyalty features | Kaggle | +| `hub_customer` + `sat_customer_personal__*` | Синтетический demo seed (customers) | `synthetic_seed.sql` | +| `hub_product` + `sat_product_catalog__*` | Синтетический demo seed (catalog) | `synthetic_seed.sql` | +| `lnk_order_customer` + `sat_lnk_order_product__*` | Консолидированный marketplace-фид (`mp__`) + per-branch order feed (`1c__`) | `satellite_seed_all_branches.sql` + kitchen live generator (`src/ingestion/`) | +| `sat_customer_loyalty__*` | Синтетический seed (dealer retro-bonus state, легенда) | `satellite_seed_all_branches.sql` | | `sat_order_marketplace__wb__*` | Synthesize from public ВБ API docs | Wildberries | | `hub_marking_code` | Честный Знак GS1 examples | crpt.ru | -Для PoC демо тащим X5 Retail Hero (45М транзакций) в ClickHouse через AgentFlow Argo Workflow, маппим в DV2.0 hub/link/sat, добавляем synthetic branches (привязка к store_id из X5 → искусственно распределяем по 5 локациям). +Демо-объём фиксирован на масштабе легенды (docs/generator-spec.md §1): ~10K +заказов / 2 500 клиентов / 160 SKU, без внешних bulk-датасетов. Raw vault +загружается синтетическим seed (`synthetic_seed.sql` + `satellite_seed_all_branches.sql`), +живым kitchen-генератором (`src/ingestion/`) и `reference/load_postgres.py`; +маппинг в DV2.0 hub/link/sat остаётся тем же (per-branch satellites, синтетические +branches по 5 локациям). diff --git a/docs/generator-spec.md b/docs/generator-spec.md index 3b52464d..f15ea18c 100644 --- a/docs/generator-spec.md +++ b/docs/generator-spec.md @@ -12,7 +12,7 @@ Consumers, in execution order: (generator.py, tnved.py; gs1.py is unchanged), `synthetic_seed.sql`, `satellite_seed*.sql`, `postgres_oltp/seed.sql`. Faux-PII mechanics are preserved (§8). -2. **Record-source rename** — `x5__*` → `mp__*` (see domain.md §5.4). +2. **Record-source rename** — retired external-dataset prefix → `mp__*` (see domain.md §5.4). 3. **Serving demo repin** — the four demo tables, NL demo answers, `ORD-*` values (§9). 4. **Evidence regeneration** — demo_evidence and live-verify counts re-pinned @@ -184,8 +184,7 @@ gram weights, food ТН ВЭД headings) — it is replaced wholesale: - **GS1 stays exactly as-is** (`gs1.py` untouched): the EAEU prefix range 460–469 is *correct* for an own-brand importer — GTINs belong to the RU brand owner registered with GS1 RUS, regardless of where manufacturing - happens. Record this rationale in the module docstring when touching the - package (its current text says "X5-grocery context"). + happens. The module docstring already records this rationale. - **`tnved.py`**: grocery headings → the four appliance headings of §3 (8516, 8509, 8423, 8422) with RU descriptions close to official wording, heading-granularity honesty note preserved. diff --git a/docs/operations/aws-oidc-setup.md b/docs/operations/aws-oidc-setup.md index cb9b2e95..547102ca 100644 --- a/docs/operations/aws-oidc-setup.md +++ b/docs/operations/aws-oidc-setup.md @@ -6,7 +6,7 @@ This archived optional guide bootstraps the AWS IAM OIDC provider and the GitHub Current project decision as of 2026-05-30: a managed-AWS / Terraform-apply production deployment is out of scope for this pre-production portfolio project — a deliberate non-goal, with the stack validated end-to-end on a local/kind demo instead. Do not treat missing AWS apply evidence as a project deficiency, active blocker, or autonomous follow-up. Reopen this path only if the operator explicitly provides an AWS account and approval to reintroduce it. -For the DV2/X5 demo, use the already documented S3-compatible cold-tier path with HF Datasets or Backblaze B2 for derived/anonymized parquet. Do not require AWS/S3 for that dataset. +For the DV2 demo, use the already documented S3-compatible cold-tier path with HF Datasets or Backblaze B2 for derived/anonymized parquet. Do not require AWS/S3 for that dataset. ## Current readiness handoff diff --git a/docs/operations/openssf-security-posture.md b/docs/operations/openssf-security-posture.md index 8662fde9..2c68682f 100644 --- a/docs/operations/openssf-security-posture.md +++ b/docs/operations/openssf-security-posture.md @@ -151,10 +151,11 @@ Pinned-Dependencies rose 0 → 4 (the pip subset is what's left). `container-attestation.yml` moved its `packages`/`id-token`/`attestations` writes from the workflow level down to the two operator-dispatched signing jobs (the every-PR `build-smoke` job is read-only now). -- **Vulnerabilities (2 OSV hits):** both came from resolution floors in the - X5 demo loader's `requirements.txt` (`pydantic>=2.0` admits the - GHSA-mr82-8j83-vxmv ReDoS range; `tqdm>=4.0` admits PYSEC-2017-74). Floors - raised to `pydantic>=2.9` / `tqdm>=4.66.3`. +- **Vulnerabilities (2 OSV hits):** both came from resolution floors in a + demo loader's `requirements.txt` (`pydantic>=2.0` admitted the + GHSA-mr82-8j83-vxmv ReDoS range; `tqdm>=4.0` admitted PYSEC-2017-74). + Floors were raised to `pydantic>=2.9` / `tqdm>=4.66.3`; the loader itself + was later removed from the codebase entirely (see CHANGELOG.md). ### Accepted-open (with reasons, not fabricated away) diff --git a/docs/perf/vault-pii-governance-pg-verify-2026-07-02.md b/docs/perf/vault-pii-governance-pg-verify-2026-07-02.md index 4989fa82..4a4fb882 100644 --- a/docs/perf/vault-pii-governance-pg-verify-2026-07-02.md +++ b/docs/perf/vault-pii-governance-pg-verify-2026-07-02.md @@ -10,7 +10,8 @@ Vault built from the repo files verbatim via `postgres/apply.sh` (schema → `postgres/governance/01..04.sql`. Data: the deterministic demo seed from `verify_live.sh` (`SEED_DEMO=1`): hub_customer msk 8 / dxb 2 = 10 rows, msk rows deliberately spanning three source conventions (`1c__msk`, -`pg_ops__msk`, `x5__msk`) so the row policies are exercised against the +`pg_ops__msk`, and the marketplace-seed prefix later renamed `mp__msk`) so +the row policies are exercised against the split_part branch derivation, not a single record_source literal. Probe principals are stand-local: `analyst_probe` (role `dv2_analyst`), `officer_msk_probe`, `officer_dxb_probe`, and `noscope_probe` (SELECT on the @@ -60,7 +61,7 @@ attributes; nothing is pushed past the view boundary for attribution. | Probe | Result | | ----- | ------ | -| `officer_msk` hub count | 8 of 8 msk rows (incl. `pg_ops__msk`, `x5__msk`) | +| `officer_msk` hub count | 8 of 8 msk rows (incl. `pg_ops__msk`, marketplace-seed prefix later renamed `mp__msk`) | | `officer_dxb` hub count | 2 of 2 dxb rows | | `officer_dxb` counts msk rows via hub filter | 0 | | `officer_msk` cross-branch view / satellite | `permission denied` | diff --git a/docs/perf/vault-pii-governance-pg-verify-2026-07-03.md b/docs/perf/vault-pii-governance-pg-verify-2026-07-03.md index 35feb234..1184b9a5 100644 --- a/docs/perf/vault-pii-governance-pg-verify-2026-07-03.md +++ b/docs/perf/vault-pii-governance-pg-verify-2026-07-03.md @@ -10,8 +10,9 @@ kitchen-gadget legend. Vault built from the repo files verbatim via `03_business_vault.sql`), then `postgres/governance/01..04.sql`. Data: the deterministic demo seed from `verify_live.sh` (`SEED_DEMO=1`): hub_customer msk 8 / dxb 2 = 10 rows, the msk rows deliberately spanning three source -conventions — **`1c__msk`, `pg_ops__msk`, `mp__msk`** (the `x5__` convention -was retired in B2; this run confirms `mp__msk` is what the seed now carries) — +conventions — **`1c__msk`, `pg_ops__msk`, `mp__msk`** (the legacy +marketplace-seed prefix was retired in B2; this run confirms `mp__msk` is +what the seed now carries) — so the row policies are exercised against the `split_part` branch derivation, not a single record_source literal. Probe principals are stand-local: `analyst_probe` (role `dv2_analyst`), `officer_msk_probe`, `officer_dxb_probe`, diff --git a/infrastructure/dv2/clickhouse-sts.yaml b/infrastructure/dv2/clickhouse-sts.yaml index 38fa0ce1..5cb206d0 100644 --- a/infrastructure/dv2/clickhouse-sts.yaml +++ b/infrastructure/dv2/clickhouse-sts.yaml @@ -14,15 +14,17 @@ data: merge-throttle.xml: | + every later query (seen on the 2026-06-06 dbt mart runs, against + a retired at-scale capture — see git history of + docs/dv2-multi-branch/load-test-baseline.md). --> 268435456 67108864 268435456 -