Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion config/contracts/order.v1.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,12 @@ fields:
- name: total_amount
type: float
required: true
unit: USD
unit: RUB
- name: currency
type: enum
required: true
values:
- RUB
- USD
- EUR
- GBP
Expand Down
3 changes: 2 additions & 1 deletion config/contracts/order.v2.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,12 @@ fields:
- name: total_amount
type: float
required: true
unit: USD
unit: RUB
- name: currency
type: enum
required: true
values:
- RUB
- USD
- EUR
- GBP
Expand Down
4 changes: 2 additions & 2 deletions docs/dataflow.html
Original file line number Diff line number Diff line change
Expand Up @@ -360,10 +360,10 @@ <h1><b>AgentFlow</b> — движение данных</h1>
];

/* DV2 demo lane */
const DV2={x:40,w:2280,y:1226,title:"Контур DV2.0 — Data Vault 2.0 на реальных данных X5",
const DV2={x:40,w:2280,y:1226,title:"Контур DV2.0 — Data Vault 2.0 на синтетических данных",
chip:"демо · kind-кластер · хост 8 GB",
flow:[
{t:"X5 Retail Hero CSV",c:"45.8M строк",d:"clients · products · purchases (4.5 GB), Kaggle-датасет"},
{t:"Синтетический генератор",c:"кухонная легенда · ₽",d:"clients · products · purchases · детерминир. seed, масштабируемый объём"},
{t:"loader.py",c:"pandas + clickhouse-driver",d:"чанки 50K · PartsThrottle (--max-active-parts) — backpressure по system.parts"},
{t:"ClickHouse",c:"25.5 · k8s",d:"Raw Vault: хабы · линки · сателлиты (ReplacingMergeTree, ветки __1c__/__bitrix__); merge-throttle в config.d"},
{t:"Business Vault",c:"SQL-вьюхи",d:"bv_order_canonical (argMax-дедуп) · bv_customer_mdm × 5 веток"},
Expand Down
10 changes: 6 additions & 4 deletions docs/domain.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,8 +149,10 @@ requirements:
the ops layer.
3. **Kill-the-five-programs triage.** Where is order X? Which orders are stuck
between confirmation and shipment longer than the stage SLA? Which failed
events need a manual decision? These map to the three planned ops surfaces:
**Order 360 timeline**, **stuck-orders worklist**, and **exception inbox**.
events need a manual decision? These map to the three ops surfaces —
**Order 360 timeline**, **stuck-orders worklist**, and **exception inbox** —
now live (`GET /v1/entity/order/{id}/timeline`, `/v1/ops/stuck-orders`,
`/v1/ops/exceptions`).

## 5. Reinterpretation dictionary

Expand Down Expand Up @@ -224,10 +226,10 @@ the operational-layer roadmap; everything else is live API surface.
| ------- | ------------------ | ------- |
| **Owner / CEO** | "Revenue today vs yesterday? Orders during the НГ peak? Is AOV holding after the price move?" | `GET /v1/metrics/revenue` · `order_count` · `avg_order_value`; `POST /v1/query` (NL: "top products this week") |
| **B2B account manager** | "What has dealer X ordered this quarter? Are they on track for the retro-bonus threshold? Which of my contacts has a birthday before March 8?" | `GET /v1/entity/user/{id}`; `POST /v1/query`; vault: `bv_customer_mdm`, `sat_customer_loyalty__bitrix__*` |
| **E-com / marketplace manager** | "Site conversion this week? Are WB orders flowing or did the feed break? Top SKUs by orders during the sale event?" | `GET /v1/metrics/conversion_rate` · `active_sessions` · `error_rate`; `GET /v1/analytics/top-entities`; `POST /v1/query` |
| **E-com / marketplace manager** | "Site conversion this week? Are WB orders flowing or did the feed break? Top SKUs by orders during the sale event?" | `GET /v1/metrics/conversion_rate` · `active_sessions` · `error_rate`; `GET /v1/admin/analytics/top-entities` (admin-key); `POST /v1/query` |
| **Operations manager** | "Everything about ORD-20260404-1001 — now, in one place. Which orders are stuck pre-shipment past SLA? What failed overnight and needs a human?" | `GET /v1/entity/order/{id}`; `/v1/alerts` (+history/test); `/v1/deadletter` (+replay/dismiss); `GET /v1/entity/order/{id}/timeline`; `GET /v1/ops/stuck-orders`; `GET /v1/ops/exceptions` (+stats/acknowledge/resolve) |
| **Category / procurement manager** | "Which SKUs sell fastest per branch? What is on hand vs reserved? When does the next container land?" | `GET /v1/search`; `GET /v1/entity/product/{id}`; `POST /v1/query`; vault: `sat_product_stock__wms__*`; container ETA — **planned** (today: `excel__*` manifests) |
| **Data engineer / analyst** | "Which events move this metric? What is the contract and its staleness budget? What changed between contract v3 and v4? Are we inside SLO?" | `GET /v1/catalog`; `/v1/contracts/*` (+versions/diff/validate); `/v1/lineage/*`; `/v1/slo`; `/v1/analytics/*` |
| **Data engineer / analyst** | "Which events move this metric? What is the contract and its staleness budget? What changed between contract v3 and v4? Are we inside SLO?" | `GET /v1/catalog`; `/v1/contracts/*` (+versions/diff/validate); `/v1/lineage/*`; `/v1/slo`; `/v1/admin/analytics/*` (admin-key) |
| **AI agent / integration** | Any of the above, programmatically — the agent is one consumer, not the product | Python/TS SDKs, MCP/LangChain/LlamaIndex integrations over the same API: `POST /v1/query` + `/v1/query/explain`, `/v1/entity/*`, `POST /v1/batch`, `/v1/webhooks`, `/v1/stream` |
| **PII officer (per jurisdiction)** | "Who can read dealer-contact PII in `dxb`? Prove RU data never crosses the border" | Not REST: DV2 governance — jurisdiction-scoped officer roles, column-limited analyst grants, per-branch row policies (`warehouse/agentflow/dv2/governance/`, ClickHouse and Postgres variants) |

Expand Down
3 changes: 2 additions & 1 deletion docs/dv2-multi-branch/demo_evidence.md
Original file line number Diff line number Diff line change
Expand Up @@ -170,7 +170,8 @@ SELECT count() FROM rv.lnk_order_product; -- 14853
basket profile: marketplace and D2C orders are predominantly single-item,
while the B2B dealer orders carry multi-line baskets, pulling the mean just
above 1. `hub_product` holds **160 SKU**; `hub_marking_code` holds **12,160**
per-unit Chestny ZNAK marking codes (issued / in-circulation / withdrawn).
Chestny ZNAK marking codes — **160** SKU-level GTIN templates + **12,000**
per-unit codes (issued / in-circulation / withdrawn).

## 8. Business Vault — populated views with MDM conflict resolution

Expand Down
4 changes: 2 additions & 2 deletions scripts/generate_contracts.py
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ class ContractSpec:
"values": [status.value for status in OrderStatus],
},
"total_amount": {
"unit": "USD",
"unit": "RUB",
},
"currency": {
"type": "enum",
Expand Down Expand Up @@ -125,7 +125,7 @@ class ContractSpec:
"values": [status.value for status in OrderStatus],
},
"total_amount": {
"unit": "USD",
"unit": "RUB",
},
"currency": {
"type": "enum",
Expand Down
45 changes: 29 additions & 16 deletions src/ingestion/producers/event_producer.py
Original file line number Diff line number Diff line change
Expand Up @@ -46,17 +46,24 @@ class ProducerConfig(BaseSettings):
model_config = {"env_prefix": "PRODUCER_"}


# Live catalog = the 10 pinned kitchen-appliance SKUs of the serving demo
# (generator-spec.md §9, mirrored row-for-row in the duckdb/clickhouse seeds):
# EN names, §3 category slugs, RUB retail prices, and each SKU's seeded
# stock. PROD-001 (kettle) is the deliberately out-of-stock bestseller — the
# oversell/freshness story needs it, so generate_product emits this stock
# verbatim rather than randomising it.
# (id, name, category slug, price ₽, stock_quantity)
PRODUCT_CATALOG = [
("PROD-001", "Wireless Headphones", "electronics", Decimal("79.99")),
("PROD-002", "Running Shoes", "footwear", Decimal("129.99")),
("PROD-003", "Coffee Maker", "kitchen", Decimal("49.99")),
("PROD-004", "Mechanical Keyboard", "electronics", Decimal("149.99")),
("PROD-005", "Yoga Mat", "fitness", Decimal("34.99")),
("PROD-006", "Backpack", "accessories", Decimal("89.99")),
("PROD-007", "Water Bottle", "fitness", Decimal("24.99")),
("PROD-008", "Desk Lamp", "home", Decimal("44.99")),
("PROD-009", "Bluetooth Speaker", "electronics", Decimal("59.99")),
("PROD-010", "Sunglasses", "accessories", Decimal("119.99")),
("PROD-001", "Electric Kettle 1.7L 2200W", "kettles", Decimal("2190.00"), 0),
("PROD-002", "Air Fryer Grill 5.5L", "grills", Decimal("5490.00"), 58),
("PROD-003", "Immersion Blender Set 800W", "blenders", Decimal("2490.00"), 203),
("PROD-004", "Stand Mixer 5L Planetary", "mixers", Decimal("6990.00"), 37),
("PROD-005", "Drip Coffee Maker 1.2L", "coffee", Decimal("3490.00"), 94),
("PROD-006", "Waffle Maker Double", "multibakers", Decimal("2290.00"), 142),
("PROD-007", "Mini Chopper 500ml", "choppers", Decimal("1490.00"), 315),
("PROD-008", "Cold-Press Juicer", "juicers", Decimal("4490.00"), 72),
("PROD-009", "Digital Kitchen Scale 5kg", "scales", Decimal("990.00"), 421),
("PROD-010", "Vacuum Sealer Compact", "vacuum-dry", Decimal("3290.00"), 167),
]

PAGES = ["/", "/products", "/cart", "/checkout", "/account", "/search"]
Expand Down Expand Up @@ -111,7 +118,7 @@ def generate_order() -> tuple[str, OrderEvent]:
status=OrderStatus.PENDING,
items=items,
total_amount=total,
currency=Currency.USD,
currency=Currency.RUB,
)
return "orders.raw", event

Expand All @@ -126,8 +133,10 @@ def generate_payment(order_id: str | None = None) -> tuple[str, PaymentEvent]:
payment_id=f"PAY-{_uuid()[:8]}",
order_id=oid,
user_id=f"USR-{random.randint(10000, 99999)}",
amount=Decimal(str(round(random.uniform(20, 500), 2))),
currency=Currency.USD,
# ₽-scale single-appliance retail range (§3 RRC band 790–7990),
# deliberately below the 10k–25k bimodality dead-zone (§12 #4).
amount=Decimal(str(round(random.uniform(790, 7990), 2))),
currency=Currency.RUB,
method=random.choice(list(PaymentMethod)),
status="initiated",
)
Expand Down Expand Up @@ -155,6 +164,7 @@ def generate_click() -> tuple[str, ClickstreamEvent]:

def generate_product() -> tuple[str, ProductEvent]:
product = random.choice(PRODUCT_CATALOG)
stock_quantity = product[4]
event = ProductEvent(
event_id=_uuid(),
event_type=EventType.PRODUCT_UPDATED,
Expand All @@ -164,9 +174,12 @@ def generate_product() -> tuple[str, ProductEvent]:
name=product[1],
category=product[2],
price=product[3],
currency=Currency.USD,
in_stock=random.random() > 0.1,
stock_quantity=random.randint(0, 500),
currency=Currency.RUB,
# Emit the SKU's seeded stock so the out-of-stock bestseller (PROD-001)
# stays out of stock even under the live feed — the oversell story
# depends on it, and random stock would clobber the seeded витрина.
in_stock=stock_quantity > 0,
stock_quantity=stock_quantity,
)
return "products.cdc", event

Expand Down
11 changes: 8 additions & 3 deletions src/ingestion/schemas/events.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,11 @@ class CdcSource(StrEnum):


class Currency(StrEnum):
# RUB is the house currency of the kitchen-appliance importer legend
# (docs/domain.md §1, generator-spec.md §10); the live producer and every
# serving sink default to it. USD/EUR/GBP stay valid enum members so
# historical fixtures and cross-border edge cases still parse.
RUB = "RUB"
USD = "USD"
EUR = "EUR"
GBP = "GBP"
Expand Down Expand Up @@ -96,7 +101,7 @@ class OrderEvent(BaseEvent):
status: OrderStatus
items: list[OrderItem] = Field(..., min_length=1)
total_amount: Decimal = Field(..., gt=0)
currency: Currency = Currency.USD
currency: Currency = Currency.RUB

@field_validator("total_amount")
@classmethod
Expand All @@ -121,7 +126,7 @@ class PaymentEvent(BaseEvent):
order_id: str = Field(..., pattern=r"^ORD-\d{8}-\d{4,}$")
user_id: str
amount: Decimal = Field(..., gt=0)
currency: Currency = Currency.USD
currency: Currency = Currency.RUB
method: PaymentMethod
status: str = Field(..., pattern=r"^(initiated|completed|failed|refunded)$")
failure_reason: str | None = None
Expand All @@ -142,6 +147,6 @@ class ProductEvent(BaseEvent):
name: str = Field(..., min_length=1, max_length=500)
category: str
price: Decimal = Field(..., ge=0)
currency: Currency = Currency.USD
currency: Currency = Currency.RUB
in_stock: bool
stock_quantity: int = Field(..., ge=0)
2 changes: 1 addition & 1 deletion src/processing/clickhouse_sink.py
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ def upsert_order(self, event: dict) -> None:
"user_id": event["user_id"],
"status": event["status"],
"total_amount": float(event["total_amount"]),
"currency": event.get("currency", "USD"),
"currency": event.get("currency", "RUB"),
"created_at": datetime.fromisoformat(event["timestamp"]),
}
],
Expand Down
6 changes: 3 additions & 3 deletions src/processing/iceberg_sink.py
Original file line number Diff line number Diff line change
Expand Up @@ -264,7 +264,7 @@ def _normalize_order(self, record: dict[str, Any]) -> dict[str, Any]:
"user_id": str(record["user_id"]),
"status": str(record["status"]),
"total_amount": float(record["total_amount"]),
"currency": str(record.get("currency", "USD")),
"currency": str(record.get("currency", "RUB")),
"item_count": int(
derived.get(
"item_count",
Expand All @@ -291,7 +291,7 @@ def _normalize_payment(self, record: dict[str, Any]) -> dict[str, Any]:
"order_id": str(record["order_id"]),
"user_id": str(record["user_id"]),
"amount": float(record["amount"]),
"currency": str(record.get("currency", "USD")),
"currency": str(record.get("currency", "RUB")),
"method": str(record["method"]),
"status": str(record["status"]),
"risk_score": (float(derived["risk_score"]) if "risk_score" in derived else None),
Expand Down Expand Up @@ -331,7 +331,7 @@ def _normalize_inventory(self, record: dict[str, Any]) -> dict[str, Any]:
"name": str(record["name"]),
"category": str(record["category"]),
"price": float(record["price"]),
"currency": str(record.get("currency", "USD")),
"currency": str(record.get("currency", "RUB")),
"in_stock": bool(record["in_stock"]),
"stock_quantity": int(record["stock_quantity"]),
"created_at": self._coerce_timestamp(record.get("timestamp")),
Expand Down
4 changes: 2 additions & 2 deletions src/processing/local_pipeline.py
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ def _ensure_tables(conn: duckdb.DuckDBPyConnection) -> None:
user_id VARCHAR,
status VARCHAR,
total_amount DECIMAL(10,2),
currency VARCHAR DEFAULT 'USD',
currency VARCHAR DEFAULT 'RUB',
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
)
""")
Expand Down Expand Up @@ -327,7 +327,7 @@ def _upsert_order(conn: duckdb.DuckDBPyConnection, event: dict) -> None:
event["user_id"],
event["status"],
float(event["total_amount"]),
event.get("currency", "USD"),
event.get("currency", "RUB"),
datetime.fromisoformat(event["timestamp"]),
],
)
Expand Down
6 changes: 3 additions & 3 deletions tests/unit/test_event_schemas.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ def test_order_valid_minimal(sample_order_event) -> None:

event = OrderEvent.model_validate(payload)

assert event.currency is Currency.USD
assert event.currency is Currency.RUB
assert event.total_amount == Decimal("209.97")
assert event.items[0].product_id == "PROD-001"

Expand Down Expand Up @@ -50,13 +50,13 @@ def test_order_rejects_negative_amount(sample_order_event) -> None:
)


def test_order_currency_defaults_to_usd(sample_order_event) -> None:
def test_order_currency_defaults_to_rub(sample_order_event) -> None:
payload = dict(sample_order_event)
payload.pop("currency")

event = OrderEvent.model_validate(payload)

assert event.currency is Currency.USD
assert event.currency is Currency.RUB


def test_order_rejects_unknown_currency(sample_order_event) -> None:
Expand Down
4 changes: 2 additions & 2 deletions warehouse/agentflow/dv2/business_vault/bv_order_canonical.sql
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,8 @@ WITH
UNION ALL
SELECT order_hk, order_date, channel, order_status, total_amount, load_ts
FROM rv.sat_order_header__bitrix__ala WHERE is_deleted = 0
-- 1C / X5 Retail Hero order headers (per-branch), so real X5 volume
-- flows through to bv_order_canonical and the branch_pnl mart.
-- 1C order headers (per-branch), so the full order volume flows
-- through to bv_order_canonical and the branch_pnl mart.
UNION ALL
SELECT order_hk, order_date, channel, order_status, total_amount, load_ts
FROM rv.sat_order_header__1c__msk WHERE is_deleted = 0
Expand Down
9 changes: 7 additions & 2 deletions warehouse/agentflow/dv2/dbt/models/marts/branch_pnl.sql
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,14 @@ SELECT
sum(toFloat64(subtotal_amount)) AS net_revenue,
sum(toFloat64(discount_amount)) AS discounts,
sum(toFloat64(shipping_cost)) AS shipping,
countIf(order_status = 'returned') AS returned_orders,
-- The legend has no dedicated 'returned' status (§2 vocabulary is
-- pending/confirmed/shipped/delivered/cancelled). Its terminal-negative
-- bucket 'cancelled' carries both cancellations and marketplace returns
-- (§2 "cancel/return allowance"; satellite_seed.sql — "marketplace cancels
-- dominate the last bucket"), so returns are measured off 'cancelled'.
countIf(order_status = 'cancelled') AS returned_orders,
sumIf(toFloat64(total_amount),
order_status = 'returned') AS returned_value,
order_status = 'cancelled') AS returned_value,
round(sum(toFloat64(tax_amount)) /
nullIf(sum(toFloat64(subtotal_amount)), 0),
4) AS effective_tax_rate
Expand Down
9 changes: 6 additions & 3 deletions warehouse/agentflow/dv2/dbt/models/marts/customer_360.sql
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{#
customer_bk first in the sort key: the mart's serving pattern is a point
lookup by business key (load-test 03_customer360_point). At X5 scale the
lookup by business key (load-test 03_customer360_point). At benchmark scale the
old (branch, customer_hk) key meant every bk lookup full-scanned the mart:
p99 250-470 ms vs the 200 ms point budget.

Expand Down Expand Up @@ -49,9 +49,12 @@ order_agg AS (
sum(toFloat64(total_amount)) AS lifetime_value,
min(order_date) AS first_order_dt,
max(order_date) AS last_order_dt,
countIf(order_status = 'returned') AS returned_orders,
-- Returns off the 'cancelled' bucket: no dedicated 'returned' status
-- exists in the §2 vocabulary; 'cancelled' folds cancellations and
-- marketplace returns together (satellite_seed.sql).
countIf(order_status = 'cancelled') AS returned_orders,
sumIf(toFloat64(total_amount),
order_status = 'returned') AS returned_value
order_status = 'cancelled') AS returned_value
FROM {{ source('rv', 'bv_order_canonical_mat') }}
WHERE customer_hk != toFixedString('', 16)
GROUP BY customer_hk, branch
Expand Down
12 changes: 8 additions & 4 deletions warehouse/agentflow/dv2/dbt/models/marts/returns_velocity.sql
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,17 @@ SELECT
channel AS channel,
toStartOfWeek(order_date) AS week,
count() AS orders,
countIf(order_status = 'returned') AS returned_orders,
round(countIf(order_status = 'returned') * 1.0 /
-- Returns are measured off the 'cancelled' bucket: the legend has no
-- dedicated 'returned' status (§2), and its terminal-negative bucket folds
-- cancellations and marketplace returns together (§2 "cancel/return
-- allowance"; satellite_seed.sql — "marketplace cancels dominate").
countIf(order_status = 'cancelled') AS returned_orders,
round(countIf(order_status = 'cancelled') * 1.0 /
count(), 4) AS return_rate,
sumIf(toFloat64(total_amount),
order_status = 'returned') AS returned_value,
order_status = 'cancelled') AS returned_value,
sumIf(toFloat64(tax_amount),
order_status = 'returned') AS returned_tax_unrecovered
order_status = 'cancelled') AS returned_tax_unrecovered
FROM {{ source('rv', 'bv_order_canonical_mat') }}
WHERE order_date IS NOT NULL
AND channel IS NOT NULL
Expand Down
Loading