Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/backup.yml
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ jobs:
INSERT INTO "{schema}".orders_v2 VALUES
(?, ?, ?, ?, ?, CURRENT_TIMESTAMP)
''',
[f"ORD-{schema.upper()}-001", f"USR-{schema.upper()}", "confirmed", 42.0, "USD"],
[f"ORD-{schema.upper()}-001", f"USR-{schema.upper()}", "confirmed", 42.0, "RUB"],
)
finally:
conn.close()
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -183,7 +183,7 @@ audit-closure sprint:
ClickHouse), PII protection moves from the removed app-level string-parse
gate to engine-enforced vault governance (fail-closed column grants,
per-jurisdiction officer roles, row policies, `SQL SECURITY DEFINER`
views — 32/32 live adversarial probes), plus the vendored NL→SQL
views — every live adversarial probe green), plus the vendored NL→SQL
generation engine (LangGraph, routed through GraceKelly), the DV2 raw
vault on PostgreSQL with `LISTEN`/`NOTIFY` freshness, the MinIO-backed
PyIceberg catalog, and the OpenSSF Scorecard channel (5.8 → 7.0).
Expand Down
5 changes: 3 additions & 2 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -184,8 +184,9 @@ See [Architecture Decision Records](decisions/) for detailed trade-off analysis.
> string parse in the serving tier. **Executed 2026-07-02 (ADR 0006 Phase 2):**
> `warehouse/agentflow/dv2/governance/` — fail-closed column grants for `dv2_analyst`,
> per-jurisdiction `dv2_pii_officer__<branch>` roles, row policies on `hub_customer`,
> `SQL SECURITY DEFINER` MDM views, PII-free `customer_360`; verified live 32/32
> (`docs/perf/vault-pii-governance-verify-2026-07-02.md`). `sql_guard` remains, scoped
> `SQL SECURITY DEFINER` MDM views, PII-free `customer_360`; verified live, 29/29
> probes on the current script (`docs/perf/vault-pii-governance-verify-2026-07-03.md`).
> `sql_guard` remains, scoped
> to what it can actually enforce: SELECT-only, no DML, the tenant table allow-list, and
> the recursive-CTE shadow reject. Tracked in `road-to-9.8.md`.
>
Expand Down
4 changes: 2 additions & 2 deletions docs/clickhouse-cutover-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,10 +82,10 @@ actually carry PII.
- [x] Verify: the 3 historical bypass forms (COLUMNS-expr, whole-row struct-ref,
column-rename-list) are dead at the engine — `COLUMNS` resolves to real
columns and hits the missing grant (`ACCESS_DENIED`); the other two are
not expressible on ClickHouse at all (`UNKNOWN_IDENTIFIER`). 32/32
not expressible on ClickHouse at all (`UNKNOWN_IDENTIFIER`). All 29
adversarial probes green on a live 26.7 server, incl. jurisdiction
row-scoping and admin-unaffected checks:
`docs/perf/vault-pii-governance-verify-2026-07-02.md`.
`docs/perf/vault-pii-governance-verify-2026-07-03.md`.
- ~~Introduce a schema-execution check for the PII gate~~ /
~~Reclassify the app-level gate~~ — obsolete: the app-level gate was removed
with the serving PII layer (2026-07-01); there is no app-side PII surface
Expand Down
4 changes: 2 additions & 2 deletions docs/decisions/0006-fix-demo-serving-engine-on-clickhouse.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,8 +109,8 @@ Concretely:
`marts.customer_360`. The separate schema-execution app check became moot
when the app-level gate was removed with the empty serving PII surface:
access control on resolved columns *is* the engine ground-truth. Verified
live, 32/32 adversarial probes:
`docs/perf/vault-pii-governance-verify-2026-07-02.md`.
live, 29/29 adversarial probes:
`docs/perf/vault-pii-governance-verify-2026-07-03.md`.
- The semantic layer **keeps emitting DuckDB-flavored SQL** and relies on
`ClickHouseBackend._translate_sql` to transpile — this stays a deliberate,
documented layer rather than an accident. Making the semantic layer emit
Expand Down
2 changes: 1 addition & 1 deletion docs/domain.md
Original file line number Diff line number Diff line change
Expand Up @@ -209,7 +209,7 @@ renames. The two exceptions are listed in §5.4.
| Change | Scope | Status |
| ------ | ----- | ------ |
| Former external-dataset record_source prefix → `mp__*` (+ governance SQL, officer probes, admission tests) | The prefix carried the name of a retired public benchmark dataset that the demo loader once replayed as transaction history. Under the legend it is the **consolidated marketplace feed**, and the prefix says so. The dataset and its loader have since been removed from the codebase entirely (see CHANGELOG.md) | **Done** (B2) |
| Demo value repin: currencies to `RUB` (primary), `AED`/`KZT` in branch stories; demo revenue/counts consistent with §1–2 | `contracts/entities/order.yaml` currency examples, NL demo answers, seeded `ORD-*` rows | Planned (data phase, after the generator spec) |
| Demo value repin: currencies to `RUB` (primary), `AED`/`KZT` in branch stories; demo revenue/counts consistent with §1–2 | `contracts/entities/order.yaml` currency examples, NL demo answers, seeded `ORD-*` rows | **Done** (S0/S1 — `RUB` is the house currency across contracts, demo answers, and seeds) |

Vocabulary guardrails for all public docs: the company is an **own-brand /
private-label importer** — always that framing; "store" (in `hub_store`) is
Expand Down
2 changes: 1 addition & 1 deletion docs/dv2-multi-branch/RELEASE_STATUS.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ backend-routed event scan, dedicated CI E2E lane), PII protection moves
from the removed app-level string-parse gate to engine-enforced vault
governance (ADR 0006 Phase 2 — fail-closed column grants,
per-jurisdiction officer roles, row policies, `SQL SECURITY DEFINER`
views; 32/32 live adversarial probes), plus the vendored NL→SQL
views; every live adversarial probe green), plus the vendored NL→SQL
generation engine routed through GraceKelly (ADR 0008), the DV2 raw
vault PostgreSQL migration with `LISTEN`/`NOTIFY` freshness, the
MinIO-backed PyIceberg catalog, ADR 0009, and the OpenSSF Scorecard
Expand Down
2 changes: 1 addition & 1 deletion docs/dv2-multi-branch/demo_voiced.narration.txt
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Импортёр мелкой кухонной техники под собственным брендом. Пять локаций — Москва, Питер, Екатеринбург, Дубай, Алматы. Три боли: кросс-канальный оверселл — один склад на все каналы, и партия дилеру должна попасть в остаток раньше, чем маркетплейсы допродадут те же единицы; контейнеры из Китая идут 40-60 дней, и застрявший на таможне груз — риск топ-3; операционка жонглирует пятью программами — 1С, Битрикс24, экран WMS, кабинеты маркетплейсов, Excel — ради ответа об одном заказе. Мы построили DV2.0 хранилище и операционный слой поверх: один ответ вместо пяти вкладок. Покажу.

Кластер. Три ноды Kubernetes, лейблы branch равно msk, dxb, ala. По ним nodeSelector закрепляет Postgres и ClickHouse — data sovereignty на уровне планировщика.
Кластер. Три ноды Kubernetes, все с лейблом branch равно msk — это HQ-демо. Postgres и ClickHouse закреплены через nodeSelector по лейблу workload; в проде тот же примитив закрепляет региональные edge-ноды branch равно dxb и ala — data sovereignty на уровне планировщика.

Модель. Семьдесят таблиц: восемь хабов, восемь линков, сорок восемь сателлитов, шесть BV вьюх поверх. Сателлиты разрезаны по record_source равно система плюс филиал. Один customer_hk из 1С и из Битрикса — конфликт решается приоритетом, аудит сохраняется.

Expand Down
12 changes: 7 additions & 5 deletions docs/generator-spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -165,8 +165,10 @@ magnitude.

## 6. Suppliers and sourcing (reference generator)

The current reference is grocery (dairy/bakery supplier stems, food brands,
gram weights, food ТН ВЭД headings) — it is replaced wholesale:
The reference was originally grocery-shaped (dairy/bakery supplier stems,
food brands, gram weights, food ТН ВЭД headings); it has since been replaced
wholesale with the kitchen-appliance reference specified below (see CHANGELOG
for the swap):

- **30 suppliers**: 22 CN contract factories (Guangdong/Zhejiang-style names,
e.g. "Foshan …", "Ningbo …", "Cixi … Electric Appliance Co., Ltd." —
Expand All @@ -186,9 +188,9 @@ gram weights, food ТН ВЭД headings) — it is replaced wholesale:
460–469 is *correct* for an own-brand importer — GTINs belong to the RU
brand owner registered with GS1 RUS, regardless of where manufacturing
happens. The module docstring already records this rationale.
- **`tnved.py`**: grocery headings → the four appliance headings of §3
(8516, 8509, 8423, 8422) with RU descriptions close to official wording,
heading-granularity honesty note preserved.
- **`tnved.py`**: the former grocery headings were replaced by the four
appliance headings of §3 (8516, 8509, 8423, 8422) with RU descriptions
close to official wording, heading-granularity honesty note preserved.

## 7. Customer populations

Expand Down
4 changes: 2 additions & 2 deletions monitoring/grafana/dashboards/merch-agent-journey.json
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
],
"fieldConfig": {
"defaults": {
"unit": "currencyUSD"
"unit": "currencyRUB"
}
}
},
Expand Down Expand Up @@ -59,7 +59,7 @@
],
"fieldConfig": {
"defaults": {
"unit": "currencyUSD"
"unit": "currencyRUB"
}
}
},
Expand Down
2 changes: 1 addition & 1 deletion scripts/benchmark_freshness.py
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@ def build_order_event(amount: Decimal, sequence: int) -> dict:
OrderItem(product_id="PROD-001", quantity=1, unit_price=amount),
],
total_amount=amount,
currency=Currency.USD,
currency=Currency.RUB,
)
return json.loads(event.model_dump_json())

Expand Down
2 changes: 1 addition & 1 deletion scripts/benchmark_freshness_realpath.py
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ def build_order_event(amount: Decimal, sequence: int) -> dict:
status=OrderStatus.PENDING,
items=[OrderItem(product_id="PROD-001", quantity=1, unit_price=amount)],
total_amount=amount,
currency=Currency.USD,
currency=Currency.RUB,
)
return json.loads(event.model_dump_json())

Expand Down
5 changes: 3 additions & 2 deletions tests/unit/test_dv2_business_vault_ddl.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,9 @@ def _strip_comments(sql: str) -> str:
return sql


# B2 (brand-neutral restoration of a guard dropped during X5 removal, G2
# audit): the header comment's illustrative "which source conventions does
# B2 (brand-neutral restoration of a guard dropped when the retired benchmark
# dataset and its loader were removed — G2 audit, dataset named in
# CHANGELOG.md): the header comment's illustrative "which source conventions does
# this admit" example must only ever name known-good record_source prefixes.
# Checked against an explicit allowlist rather than a literal bad string, so
# this test never itself has to spell out the retired dataset's prefix to
Expand Down
5 changes: 3 additions & 2 deletions tests/unit/test_dv2_postgres_ddl.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,9 @@ def _strip_comments(sql: str) -> str:
return sql


# B2 (brand-neutral restoration of a guard dropped during X5 removal, G2
# audit): the header comment's illustrative "which source conventions does
# B2 (brand-neutral restoration of a guard dropped when the retired benchmark
# dataset and its loader were removed — G2 audit, dataset named in
# CHANGELOG.md): the header comment's illustrative "which source conventions does
# this admit" example must only ever name known-good record_source prefixes.
# Checked against an explicit allowlist rather than a literal bad string, so
# this test never itself has to spell out the retired dataset's prefix to
Expand Down
5 changes: 3 additions & 2 deletions warehouse/agentflow/dv2/loaders/vault_rows.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,9 @@

These pydantic models describe the hub / link / order-satellite shape of the
per-branch order feed (``sat_order_header__1c__*`` / ``sat_order_pricing__1c__*``).
They were relocated here from the deleted X5 Retail Hero loader during its
removal (G2 audit S2b) because ``PostgresVaultWriter``'s own tests
They were relocated here from the now-deleted legacy external-dataset loader
during its removal (G2 audit S2b; the dataset is named in CHANGELOG.md)
because ``PostgresVaultWriter``'s own tests
(``tests/unit/test_dv2_postgres_ingestion.py``) needed generic row shapes to
keep running; that test file is their only importer today.

Expand Down