Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions docs/ADRs/013_sampled_forecast_wire_contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -1044,6 +1044,28 @@ record execution progress against it.
(maintainer: "fix this through the pipeline"): lifting the `<17` ceiling at its
source is filed as pipeline-core#280; when it lifts, the fixture gets one
planned re-baseline, coordinated, never a drive-by.**
- **2026-07-27 — run-0 attempt OOM-killed; both memory legs rebuilt (fix
campaign complete same-day; expert-review-governed).** The first live delivery
attempt was killed at 23.8 GB (31 GB host). Two stacked causes, both fixed:
**(A) the delivery now STREAMS one target at a time** — `resolve_run` pins
shard file_ids cheaply (race-free fetch-by-id), `TargetLease.load()`
fetches → verifies → curates → coverage-checks per target, the sink releases
each frame after its shards; measured on the real published run:
**4.73 GB peak, 108 shards, 5 min, zero store calls** (interlock held).
**(B) the historical path is pandas-free** (#126): actuals fetched as a
`views_frames.FeatureFrame` (the frame path's FIRST production consumer —
C-40's gate lifted), artifact built by `unfao/historical.py` via pyarrow —
reader-level parity with a legacy characterization golden proven through
faoapi's own reader semantics. Two ghosts found in the legacy artifact and
deliberately exorcised (faoapi reader verified safe on both): junk `row`/`col`
columns that would have served as bogus targets, and C-40's single-element
list-in-cell values (now scalars; their reader normalizes both identically).
Also closed: contract-mode `_save` had shipped no historical at all (S7 gap) —
actuals now ride beside the wire behind the same interlock. Upstream during
the same incident: the store's SDK JSON re-serialization surfaced
(pipeline-core #310 / C-217) — manifests are never hash-verified, rule now
pinned in docstrings. vpp PRs #127–#129; the third run-0 attempt is expected
at ~5 GB total.
- **2026-07-24 — run-0 pre-flight: declared-region curation added at the
anti-corruption layer.** The real run-0 payload carries the full model grid
(64,818 cells) including exactly the 76 declared GAUL-uncovered exclusions
Expand Down
4 changes: 3 additions & 1 deletion reports/technical_risk_register.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
|-------------------|--------------------------------------|
| Project | views-postprocessing |
| Owner | Dylan Pinheiro / PRIO MD&D Team |
| Last Updated | 2026-07-19 |
| Last Updated | 2026-07-27 |
| Total Concerns | 55 |
| Open Concerns | 25 |
| Resolved Concerns | 30 |
Expand Down Expand Up @@ -438,6 +438,8 @@ Tier 2: structural fragility under the realistic change of wiring to global, wit

`UNFAOPostProcessorManager` subclasses **two concrete** pipeline-core base classes (`PostprocessorManager`, `ForecastingModelManager`) and **interleaves infrastructure** (env reading, `AppwriteConfig` construction, `DatastoreModule`, path resolution) with the FAO **business logic** (GAUL enrichment, the 9-column null gate) inside the lifecycle hooks. Consequences: (a) the FAO logic cannot be instantiated or unit-tested without the full framework + Appwrite env + viewser; (b) **pandas cannot leave the delivery path** because the inherited data loader and `PGMDataset` are pandas — gated on pipeline-core's own DataFrame retirement; (c) **SDP exposure** — heavy *inheritance* coupling to a pipeline-core that is itself unstable (mid-migration), so upstream changes break far from their cause (cf. C-27, C-29); (d) it's the repo's only composition-over-inheritance violation. The dependency itself is correct (`unfao.py` genuinely *is* a pipeline-core postprocessor) — the issue is its **blast radius**. Mitigation (does **not** fight the Template-Method framework): keep the subclass as a **thin shell** but extract `enrich` + `validate` + the 9-column contract into a pipeline-core-free core object the manager *calls*, and wrap the Appwrite I/O behind a small delivery-sink adapter (DIP). This makes the FAO logic testable standalone and insulates it from pipeline-core churn.

**Update 2026-07-27 — the pandas gate (consequence b) has LIFTED, and the mitigation shape largely landed via ADR-013:** pipeline-core shipped its frame-native fetch (`get_feature_frame`, tested, previously zero consumers), and this repo became its **first production consumer** (#126 / PR #129): the historical path now fetches a `views_frames.FeatureFrame` and builds the artifact pandas-free (`unfao/historical.py`), reader-parity-proven against a legacy characterization golden. The forecast path is frame-native end-to-end (wire/ + streaming leases, PRs #115–#129). Pandas remains ONLY on the legacy forecast branch (kept until run 0 proves the contract path live; its deletion is the named post-run-0 follow-up) and in the retired-in-place `enrichment.py`/`extraction.py` legacy seams. The double-inheritance shell (consequences a/c/d) still stands — the residual scope of this entry.

**Priority raised by the samples/uncertainty requirement (2026-06-27).** The delivery is moving from point estimates to **predictions-with-uncertainty (S samples per cell)**. This makes the pandas gate (consequence b) materially worse, not just cosmetic:
- **views-frames** stores a distribution as a native contiguous `(N, S)` float32 array (sample axis always explicit; point = S=1). **pandas `PGMDataset`** stores it as **object-dtype list-in-cell** — each of N cells holds a separate length-S numpy array (`_ViewsDataset._convert_to_arrays` / `_check_prediction_samples` in pipeline-core `data/handlers.py`).
- Cost of the object-dtype representation scales ~linearly with S: memory blow-up (this is pipeline-core's own OOM, **#181/#189** — "~18 GB, kills runs" off list-in-cell DataFrames), an encode/decode tax at every parquet/API boundary (faoapi's inverse `_convert_to_arrays`), and a silent `np.resize` pad on mismatched sample counts (feature path).
Expand Down
Loading