diff --git a/docs/ADRs/013_sampled_forecast_wire_contract.md b/docs/ADRs/013_sampled_forecast_wire_contract.md index ada9e70..1594502 100644 --- a/docs/ADRs/013_sampled_forecast_wire_contract.md +++ b/docs/ADRs/013_sampled_forecast_wire_contract.md @@ -1044,6 +1044,28 @@ record execution progress against it. (maintainer: "fix this through the pipeline"): lifting the `<17` ceiling at its source is filed as pipeline-core#280; when it lifts, the fixture gets one planned re-baseline, coordinated, never a drive-by.** +- **2026-07-27 — run-0 attempt OOM-killed; both memory legs rebuilt (fix + campaign complete same-day; expert-review-governed).** The first live delivery + attempt was killed at 23.8 GB (31 GB host). Two stacked causes, both fixed: + **(A) the delivery now STREAMS one target at a time** — `resolve_run` pins + shard file_ids cheaply (race-free fetch-by-id), `TargetLease.load()` + fetches → verifies → curates → coverage-checks per target, the sink releases + each frame after its shards; measured on the real published run: + **4.73 GB peak, 108 shards, 5 min, zero store calls** (interlock held). + **(B) the historical path is pandas-free** (#126): actuals fetched as a + `views_frames.FeatureFrame` (the frame path's FIRST production consumer — + C-40's gate lifted), artifact built by `unfao/historical.py` via pyarrow — + reader-level parity with a legacy characterization golden proven through + faoapi's own reader semantics. Two ghosts found in the legacy artifact and + deliberately exorcised (faoapi reader verified safe on both): junk `row`/`col` + columns that would have served as bogus targets, and C-40's single-element + list-in-cell values (now scalars; their reader normalizes both identically). + Also closed: contract-mode `_save` had shipped no historical at all (S7 gap) — + actuals now ride beside the wire behind the same interlock. Upstream during + the same incident: the store's SDK JSON re-serialization surfaced + (pipeline-core #310 / C-217) — manifests are never hash-verified, rule now + pinned in docstrings. vpp PRs #127–#129; the third run-0 attempt is expected + at ~5 GB total. - **2026-07-24 — run-0 pre-flight: declared-region curation added at the anti-corruption layer.** The real run-0 payload carries the full model grid (64,818 cells) including exactly the 76 declared GAUL-uncovered exclusions diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index 5235ef1..7433454 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -4,7 +4,7 @@ |-------------------|--------------------------------------| | Project | views-postprocessing | | Owner | Dylan Pinheiro / PRIO MD&D Team | -| Last Updated | 2026-07-19 | +| Last Updated | 2026-07-27 | | Total Concerns | 55 | | Open Concerns | 25 | | Resolved Concerns | 30 | @@ -438,6 +438,8 @@ Tier 2: structural fragility under the realistic change of wiring to global, wit `UNFAOPostProcessorManager` subclasses **two concrete** pipeline-core base classes (`PostprocessorManager`, `ForecastingModelManager`) and **interleaves infrastructure** (env reading, `AppwriteConfig` construction, `DatastoreModule`, path resolution) with the FAO **business logic** (GAUL enrichment, the 9-column null gate) inside the lifecycle hooks. Consequences: (a) the FAO logic cannot be instantiated or unit-tested without the full framework + Appwrite env + viewser; (b) **pandas cannot leave the delivery path** because the inherited data loader and `PGMDataset` are pandas — gated on pipeline-core's own DataFrame retirement; (c) **SDP exposure** — heavy *inheritance* coupling to a pipeline-core that is itself unstable (mid-migration), so upstream changes break far from their cause (cf. C-27, C-29); (d) it's the repo's only composition-over-inheritance violation. The dependency itself is correct (`unfao.py` genuinely *is* a pipeline-core postprocessor) — the issue is its **blast radius**. Mitigation (does **not** fight the Template-Method framework): keep the subclass as a **thin shell** but extract `enrich` + `validate` + the 9-column contract into a pipeline-core-free core object the manager *calls*, and wrap the Appwrite I/O behind a small delivery-sink adapter (DIP). This makes the FAO logic testable standalone and insulates it from pipeline-core churn. +**Update 2026-07-27 — the pandas gate (consequence b) has LIFTED, and the mitigation shape largely landed via ADR-013:** pipeline-core shipped its frame-native fetch (`get_feature_frame`, tested, previously zero consumers), and this repo became its **first production consumer** (#126 / PR #129): the historical path now fetches a `views_frames.FeatureFrame` and builds the artifact pandas-free (`unfao/historical.py`), reader-parity-proven against a legacy characterization golden. The forecast path is frame-native end-to-end (wire/ + streaming leases, PRs #115–#129). Pandas remains ONLY on the legacy forecast branch (kept until run 0 proves the contract path live; its deletion is the named post-run-0 follow-up) and in the retired-in-place `enrichment.py`/`extraction.py` legacy seams. The double-inheritance shell (consequences a/c/d) still stands — the residual scope of this entry. + **Priority raised by the samples/uncertainty requirement (2026-06-27).** The delivery is moving from point estimates to **predictions-with-uncertainty (S samples per cell)**. This makes the pandas gate (consequence b) materially worse, not just cosmetic: - **views-frames** stores a distribution as a native contiguous `(N, S)` float32 array (sample axis always explicit; point = S=1). **pandas `PGMDataset`** stores it as **object-dtype list-in-cell** — each of N cells holds a separate length-S numpy array (`_ViewsDataset._convert_to_arrays` / `_check_prediction_samples` in pipeline-core `data/handlers.py`). - Cost of the object-dtype representation scales ~linearly with S: memory blow-up (this is pipeline-core's own OOM, **#181/#189** — "~18 GB, kills runs" off list-in-cell DataFrames), an encode/decode tax at every parquet/API boundary (faoapi's inverse `_convert_to_arrays`), and a silent `np.resize` pad on mismatched sample counts (feature path).