un_fao: retire the pandas historical path — deliver frame-native end-to-end (fixes run-0 OOM)

[Polichinel]

Summary

Make the un_fao postprocessor's historical/actuals path pandas-free, so the FAO delivery runs entirely on views-frames. This is the root fix for the run-0 OOM and the gate that unblocks shipping the global-land FAO forecast.

Why (the incident)

Delivering the global-land forecast (region=land_gaul, 64,742 cells × ~438 months = 28,356,996 rows) OOM-killed the postprocessor at ~23.8 GB:

Out of memory: Killed process (python) total-vm:32146116kB, anon-rss:23778224kB

Root cause: _read_historical_data fetches the actuals as a pandas DataFrame (get_data() → read_dataframe() → PGMDataset). The region scope is correct and proven (datafactory served exactly 64,742 cells); the wall is pandas.

Decisive context: pipeline-core's pandas-free fetch (get_feature_frame → views_frames.FeatureFrame) is fully built and tested but has zero production consumers (a pipeline-core falsification test says the frame path is "capability-on-paper, exercised by zero real runs"). This issue makes the FAO product the first real consumer.

The forecast side is already frame-native — extend that seam to the actuals

The wire_contract path is already pandas-free for the forecast (wire/, delivery/, frame_extraction.py, track_a_source.py — several modules explicitly forbid importing pandas). We extend the same seam to the historical/actuals side and retire the pandas historical path — we do not migrate PGMDataset, we bypass it exactly as the forecast path already does.

Scope

1. Fetch actuals as a FeatureFrame. In unfao.py:_read_historical_data (~:88-104), call self._data_loader.get_feature_frame(...) and read cached_frame_path instead of get_data()/read_dataframe()/PGMDataset. Consume the views_frames.FeatureFrame directly. (Requires the descriptor to declare data_format: feature_frame — that companion change is staged in views-models postprocessors/un_fao/configs/config_queryset.py; land the two together.)
2. Route actuals through the existing frame-native seam the forecast already uses: frame_extraction, the pyarrow GAUL join in wire/sidecar.py:build_sidecar, and delivery/coverage. Retire PGMDataset, enrichment.py (pandas merge), and the pandas extraction.py seam for the historical path.
3. Build the three named frame-native gaps (self-documented in frame_extraction.py:14-21):
   • frame-native drop_months_above (the _clip_observed_history month clip; S7/S7 — historical inbound read_dataframe/PGMDataset retirement #92),
   • frame-native unmapped_cell_count (provenance; S4/S4 — enrichment.py: numpy/pyarrow keyed gather (drop pandas merge) #89),
   • a frame-native historical GAUL enrichment — adopt the sidecar/consumer-injected-mapping pattern (geography beside values), respecting that geography is deliberately not in views-frames (cross_level_align takes a consumer-supplied mapping).
4. Confirm the fetch works off-the-shelf: un_fao's historical queryset is a frame-capable datafactory descriptor (source: views-datafactory; FRAME_CAPABLE_SOURCES gate at feature_frame_path.py:160) — expected yes; verify.

Acceptance criteria

• un_fao runs global-land (land_gaul, 64,742) end-to-end without OOM (peak RSS well under host RAM).
• The coverage gate passes at 64,742 for both the forecast and the actuals companion frame.
• grep -rn "import pandas" on the un_fao delivery path (unfao/, delivery/, wire/) returns nothing.
• The forecast lands in unfao_bucket; python -m tools.liveness shows unfao_delivery fresh.

Coordination

• Companion change in views-models (config_queryset.py data_format: feature_frame + region land_gaul, README) — must land in lockstep.
• Part of the platform-wide "pandas out of the FAO product path" effort (see the companion issues in views-faoapi, views-frames, views-pipeline-core).
• This is the critical path: run-0 ships as soon as this + the views-models companion land.

Filed from the views-models seat during run-0 (2026-07-27), with a full three-repo code sweep behind it. Receipts available.

[Polichinel]

Epic: pandas out of the FAO product path (run-0, 2026-07-27). This is the critical path — run-0 ships once this + the views-models companion land.

• views-models companion (config data_format: feature_frame + region=land_gaul): staged in this session's checkout (lands in lockstep).
• Enabled by pipeline-core Make the feature_frame path the real default; advance get_data retirement (#267) views-pipeline-core#314 (frame-fetch seam / get_data retirement).
• Sibling: views-platform/views-faoapi#242 (API side), Add a dense-grid fill primitive (unblocks pandas-free FAO ingestion) views-frames#203 (fill primitive).

[Polichinel]

vpp seat review (2026-07-27): ACCEPT with four amendments. All three factual claims verified against the repos: get_feature_frame exists + tested (pipeline-core dataloaders.py:1596, two test modules); FRAME_CAPABLE_SOURCES = {"datafactory"} gate at feature_frame_path.py:47/:160 — un_fao's source qualifies; the views-models companion is staged (config_queryset.py: data_format: feature_frame, REGION = land_gaul). Strategic claim confirmed: vpp #92 (S7) is no longer blocked — the C-40 gate lifted when pipeline-core shipped the frame fetch; this issue makes the FAO product its first real consumer.

Amendments before build:

1. Historical wire artifact stays schema-identical (critical). ADR-013 §4.1/§8 pins the historical artifact as unchanged pandas-parquet on the wire. Pandas leaves the construction, not the artifact: add an AC — schema parity (column set, names, dtypes incl. the 9 GAUL columns) between the new pyarrow-built artifact and the legacy output, verified against faoapi's historical ingest expectations.
2. Scope the no-pandas grep. The legacy forecast path (and its pandas import in managers/unfao.py) survives until run 0 proves the contract path live; its deletion is a named post-run-0 follow-up, not part of this issue.
3. Complementary memory fix, forecast side: per-target streaming through wire/ (resolve manifests cheaply; fetch→gate→shard→release one target at a time) — in progress on the vpp side; both fixes land. Combined peak: ~24 GB → ~3–4 GB.
4. Closeout bookkeeping: absorb S7 — historical inbound read_dataframe/PGMDataset retirement #92 + the historical half of S4 — enrichment.py: numpy/pyarrow keyed gather (drop pandas merge) #89 (supersede comments), update [Tracking] Push pandas to the seams — execution checklist #93, update register C-40 (gate lifted; first frame-path consumer live).

Sequencing: vpp builds against the staged views-models descriptor; the two land in lockstep per the issue. Run 0 reruns after both + the streaming fix merge.

🤖 Generated with Claude Code