diff --git a/CHANGELOG.md b/CHANGELOG.md index 257ca4f..eb34896 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,34 @@ All notable changes to `views-frames` are documented here. The format is based o [Keep a Changelog](https://keepachangelog.com/), and this project adheres to [Semantic Versioning](https://semver.org/) as governed in `GOVERNANCE.md`. +## [1.10.0] — 2026-07-27 + +**The dense-grid fill primitive (ADR-026) — unblocks pandas-free FAO ingestion (#203).** +Additive MINOR; `CONFORMANCE_FLOOR` stays `1.0.0`. + +### Added +- **`frame.reindex_fill(other, *, fill_value)`** on all three sibling frames + (`PredictionFrame`/`FeatureFrame`/`TargetFrame`, WET per ADR-011): align to `other`'s + rows with **no** superset requirement — present rows pass through **bit-exact**, absent + rows get the caller's `fill_value` broadcast across the trailing axes (`NaN` legal; + keyword-only and required — no silent default, ADR-009). The result's index **is** + `other`; metadata (and `feature_names`) preserved. Owns the `-1`-sentinel scatter once: + a consumer hand-rolling `values[pos]` silently picks the *last* row for absent cells. + Inherits the C-21 row-uniqueness stance (unique rows assumed in *self*; duplicate + target rows allowed and repeat). +- **`SpatioTemporalIndex.cartesian(times, units, level)`**: the dense product-index + constructor — every `(time, unit)` combination in canonical **time-major** order, from + **explicit arrays only** (deriving them, e.g. "units of the last time step", is + consumer policy). Fails loud on duplicated input values (a duplicated product input + manufactures duplicate rows → undefined same-level joins, C-21). +- **`assert_reindex_fill_law`** in the published conformance suite (ADR-016): result + index equals the target row-for-row; present rows bit-exact; absent rows equal the + fill (NaN-safe); on a superset frame the fill degenerates to `reindex`. +- Consumer note: faoapi's `dense_grid.py` can now delegate (its last-step-entity rule + + C-87 dropped-entity check stay consumer-side) and drop its pandas implementation + (faoapi #242). Densification allocates the full dense buffer — a deliberate, costly + act at grid scale (documented on both symbols). + ## [1.9.0] — 2026-07-24 **The tower-tip MAP reads the top floor (ADR-019 Amendment 3): `tip_mass` 0.5 → 0.25.** diff --git a/CLAUDE.md b/CLAUDE.md index 4cb0293..bb8ab08 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ at the **root of the platform dependency DAG**, plus two sibling operation packa in the same wheel. numpy only; depends on nothing internal; every other repo depends *toward* it. -> **Status:** **released — v1.8.x on PyPI**, public API **frozen since v1.0.0** +> **Status:** **released — v1.10.x on PyPI**, public API **frozen since v1.0.0** > (ADR-018; everything after is additive, `CONFORMANCE_FLOOR` stays `1.0.0`). > Consumers install `views-frames` and validate against the published conformance > suite (`views_frames.conformance`, ADR-016). See `CHANGELOG.md` for the release @@ -90,7 +90,7 @@ CI additionally gates 100% line+branch coverage ## Governance -Constitutional ADRs 000–010, project ADRs 011–025, CICs for every non-trivial +Constitutional ADRs 000–010, project ADRs 011–026, CICs for every non-trivial surface (7 active incl. the package-level `Summarize.md` and `Reconcile.md`), contributor protocols, and standards live in `docs/`. The technical risk register (`reports/technical_risk_register.md`) is the curated concern/decision log. Run diff --git a/README.md b/README.md index 969540d..07545bb 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ > containers (`FeatureFrame`, `PredictionFrame`, and their anticipated siblings) > that every other repo depends on and that depends on nothing internal. > -> **Status:** **v1.9.0 — frozen API, published to PyPI** (frozen since v1.0.0, ADR-018; the +> **Status:** **v1.10.0 — frozen API, published to PyPI** (frozen since v1.0.0, ADR-018; the > v1.1 surface is > purely additive — the coherent posterior summary, ADR-019; v1.2.0 rebuilt the tower > `outside-in`, C-44; v1.3.0 makes the tower summary distribution-agnostic — no magnitude @@ -18,7 +18,10 @@ > ADR-025 immutability-by-convention, the adversarial red-test batch; v1.9.0 moves the > tower-tip MAP to the **top floor** of the published tower — `tip_mass` 0.25 with the > **MAP-containment law** in the conformance suite, ADR-019 Amendment 3 — and adds the -> `research/figures/` tower-figure toolkit). This +> `research/figures/` tower-figure toolkit; v1.10.0 adds the **dense-grid fill** +> primitive — `reindex_fill(other, *, fill_value)` on all three frames + +> `SpatioTemporalIndex.cartesian` + the published `assert_reindex_fill_law`, ADR-026, +> unblocking pandas-free FAO ingestion). This > README is the design > bible; the contract it specifies is realised in `src/views_frames/` (index, frames, > io, conformance suite) plus the `src/views_frames_summarize/` sibling package @@ -314,7 +317,12 @@ genuinely reused core. Build this once: `(time, unit)` **at a single `SpatialLevel`**. **This is the label-alignment that today drags pandas back in** — pred↔actual join, partial-overlap evaluation, same-level reindex. This alignment logic lives in the leaf - unconditionally. + unconditionally. The dense-grid pair rides the same join (ADR-026): + `cartesian(times, units, level)` builds the time-major product index + (explicit arrays only — deriving them is consumer policy), and the frames' + `reindex_fill(other, *, fill_value)` aligns with **no** superset requirement, + filling absent rows (present rows bit-exact; law-pinned by + `assert_reindex_fill_law`). - **Cross-level operations (`cross_level_align`) — protocol here, data injected.** The cm↔pgm **cross-level join** (country↔grid) is **not** a same-axis set op; it is a one-to-many lookup against a `priogrid→country` mapping that is **injected** diff --git a/docs/ADRs/026_dense_grid_fill_primitive.md b/docs/ADRs/026_dense_grid_fill_primitive.md new file mode 100644 index 0000000..bd0402e --- /dev/null +++ b/docs/ADRs/026_dense_grid_fill_primitive.md @@ -0,0 +1,83 @@ +# ADR-026: Dense-grid fill is a leaf primitive (`reindex_fill` + `cartesian`) + +**Status:** Accepted +**Date:** 2026-07-27 +**Deciders:** VIEWS platform maintainers +**Consulted:** expert-code-review (2026-07-27, design review of issue #203) +**Informed:** views-faoapi (ingestion, #242), views-pipeline-core, views-postprocessing + +--- + +## Context + +faoapi's ingestion still builds its dense forecast grid with pandas, and its docstring +names the reason explicitly: *"views-frames has no fill primitive"* +(`views-faoapi/.../forecast/ingestion/dense_grid.py:3`). That is the last hard blocker on +a pandas-free FAO ingestion edge (faoapi #242; filed here as #203). The operation itself — +align a sparse frame to a complete `(time × unit)` target index and fill absent cells — +is **same-level structural alignment**, the responsibility category `SpatioTemporalIndex` +already owns (`select`/`reindex`/`searchsorted`/`intersect`). It is not an adapter, not +geography, and not policy: it touches only values + identifier arrays. + +The machinery half-exists: `searchsorted` returns **-1 for absent rows** (F4-hardened for +the empty index), and frame-level `reindex` is `select(searchsorted(other))` behind a +fail-loud superset guard. What is missing is the materialization of the -1 positions as a +fill — and that missing piece is a genuine silent-corruption hazard when consumers +hand-roll it: `values[pos]` with a -1 in `pos` silently selects the **last row** (numpy +negative indexing) for every absent cell. + +## Decision + +Two additive symbols (MINOR, v1.10.0; `CONFORMANCE_FLOOR` stays `1.0.0`): + +1. **`frame.reindex_fill(other, *, fill_value)`** on all three sibling frames (WET per + ADR-011). Align to `other`'s rows with **no superset requirement**: present rows come + through **bit-exact**; absent rows get `fill_value` broadcast across the trailing + axes. `fill_value` is **keyword-only and required** — no silent default (ADR-009); the + caller states its policy (`0.0` vs `NaN`) aloud. The result's index **is** `other` + (immutable, shared); metadata (and `feature_names`) are preserved. The C-21 + row-uniqueness stance is inherited unchanged: unique rows assumed in *self*, + duplicates allowed in the target (a `cross_level_align` product is a legal target). + +2. **`SpatioTemporalIndex.cartesian(times, units, level)`** — the dense product-index + constructor, **time-major** order as contract (dense indices are canonical across + consumers). **Explicit arrays only**: any rule for deriving them (faoapi's + "units of the last time step" + dropped-entity check, their C-87) is consumer policy + and never lives in the leaf. **Fails loud on duplicated inputs** — a duplicated + product input manufactures duplicate `(time, unit)` rows, which make every same-level + join undefined (C-21); in a product constructor that is always a caller bug. + +3. **`assert_reindex_fill_law`** joins the published conformance suite (ADR-016): result + index equals the target row-for-row; present rows bit-exact; absent rows equal the + fill (NaN-safe); on a superset frame the fill degenerates to `reindex`. + +## Alternatives considered + +- **A `fill_value=None` kwarg on the frozen `reindex`** — rejected: one symbol, two + contracts (fail-loud vs fill) is the mode-switch shape D-09 (#113) already rejected; + under the ADR-018 freeze it could never be untangled. +- **Index-level only (consumers scatter themselves)** — rejected: it pushes the -1 + sentinel into every consumer *untested*; the wrap-to-last-row failure is silent wrong + data, the class of bug this leaf exists to remove. +- **Decline (keep fill consumer-side)** — rejected: the need is receipted + (`dense_grid.py:3`), blocks faoapi #242, and per-consumer numpy reimplementations + would each carry the hazard above. +- **A derivation rule in `cartesian`** (e.g. "observed times × last-step units") — + rejected permanently: that complects policy with structure (the C-52 accretion guard + names exactly this camel's nose). If a "derive the dense grid" convenience is ever + demanded, it belongs consumer-side. + +## Consequences + +- faoapi's `dense_grid.py` can delegate: build its entity set + C-87 check (policy, + stays theirs), call `cartesian`, then `reindex_fill` — and drop pandas (verified in + faoapi #242, acted on by the maintainers, never from this repo). +- Densification **allocates the full dense buffer** (`other.n_rows × …`): a full-pgm + grid runs to tens of millions of rows, and at large S the values buffer alone can + reach memory-breaking size. This is inherent to densifying, stated in both + docstrings, and tracked in the register — the leaf does not guess a size guard + (policy). +- Scalar fill only, deliberately minimal: a per-row/vector fill would be a future + additive overload if a consumer ever receipts the need. +- The frozen surface grows by three symbols that can never be removed short of a MAJOR + (ADR-018) — accepted on the receipted need and the silent-corruption argument. diff --git a/docs/ADRs/README.md b/docs/ADRs/README.md index c288b05..abbb806 100644 --- a/docs/ADRs/README.md +++ b/docs/ADRs/README.md @@ -11,7 +11,7 @@ ADRs are divided into: 1. **Constitutional ADRs (000–009)** — foundational architectural rules. 2. **Governance ADRs (010)** — the technical risk register. -3. **Project-Specific ADRs (011–025)** — the ratified contract, estimator, sibling-package, +3. **Project-Specific ADRs (011–026)** — the ratified contract, estimator, sibling-package, and freeze/immutability decisions. --- @@ -51,10 +51,10 @@ Together, these define the invariant layer of the system. --- -## Project-Specific ADRs (011–025) — the ratified project decisions +## Project-Specific ADRs (011–026) — the ratified project decisions **011–016** ratify the six founding contract decisions from the design bible (README §13a, -all Accepted 2026-06-21); **017–025** are the post-v1 decisions — sibling packages, the API +all Accepted 2026-06-21); **017–026** are the post-v1 decisions — sibling packages, the API freeze, estimators, contract homes, and the immutability convention (Accepted 2026-06-22 through 2026-07; ADR-024 is a design-direction ADR with implementation deferred). Each cites the risk-register IDs it resolves. @@ -74,6 +74,7 @@ the risk-register IDs it resolves. - **ADR-023** — [Forecast reconciliation is a sibling package (`views_frames_reconcile`)](023_reconciliation_is_a_sibling_package.md). Adds a third sibling package (mirrors ADR-017): the numpy-only, `views_frames`-only forecast reconciler (`reconcile_proportional`/`ReconciliationModule` — make pgm grid forecasts sum to cm country totals per draw), relocated **WET** from views-postprocessing and proven bit-identical. Charter: frame-reconciliation algorithms only; **never fetch the mapping** (injected as arrays, like `cross_level_align`); no IO/scoring/plotting/foreign `views_*`. Import-DAG `views_frames_reconcile → {views_frames}`; additive/MINOR (v1.7.0), `CONFORMANCE_FLOOR` stays 1.0.0. **Amended 2026-06-27 (#143, v1.8.0):** the sample-count contract — a point cm (`sample_count == 1`) is broadcast across the grid's draws; the equal-count path stays bit-exact. - **ADR-024** — [Principled joint probabilistic reconciliation — design direction & deferral](024_principled_joint_reconciliation_design.md). Design-only (no code): per-draw `proportional` is an information-losing approximation that pairs grid-draw `s` with country-draw `s` across **independently-trained** models with no shared draw identity. Names the **draw-identity contract** principled reconciliation requires (shared ensemble / copula coupling), fixes the upgrade as a **future sibling module** (probabilistic reconciliation / MinT — never a change to `proportional`), and **defers** implementation until a consumer needs calibrated joint tails *and* the country model can supply the identity/coupling. Records C-62; corrects the ambiguous `proportional.py` "C-37" reference (views-frames C-37 is unrelated/resolved; the lineage is views-postprocessing C-37). - **ADR-025** — [Value-buffer immutability is by convention; only the index is enforced](025_value_buffer_immutability_by_convention.md). Corrects the "immutable value objects" contract to match the code: only the index (`time`/`unit`) is `setflags(write=False)`-enforced; the frame **value buffer is immutable by convention** (mutating `.values` in place is unsupported and may corrupt buffer-sharing frames), left writeable to preserve zero-copy / `mmap` (C-07). The `setflags`-enforce on `.values` would be "tightening an invariant" on a frozen-surface member (a MAJOR, GOVERNANCE/ADR-018) for a hole nothing exercises, so it is recorded as a **deferred MAJOR-rider** (added free on the next MAJOR), not done now. Resolves C-63; docs-only, `CONFORMANCE_FLOOR` stays 1.0.0. +- **ADR-026** — [Dense-grid fill is a leaf primitive](026_dense_grid_fill_primitive.md). Adds `frame.reindex_fill(other, *, fill_value)` (align with **no** superset requirement; present rows bit-exact, absent rows filled; `fill_value` keyword-only + required) on all three siblings, `SpatioTemporalIndex.cartesian(times, units, level)` (dense product index, time-major, explicit arrays only, fails loud on duplicated inputs), and the published `assert_reindex_fill_law`. Closes #203 (faoapi's "views-frames has no fill primitive", the last pandas blocker on FAO ingestion, faoapi #242); rejects a derivation rule and a `reindex` mode-kwarg (D-09/C-52 precedents). Additive MINOR (v1.10.0); `CONFORMANCE_FLOOR` stays 1.0.0. --- diff --git a/docs/CICs/FeatureFrame.md b/docs/CICs/FeatureFrame.md index 9da00f9..7f8548d 100644 --- a/docs/CICs/FeatureFrame.md +++ b/docs/CICs/FeatureFrame.md @@ -3,8 +3,8 @@ **Status:** Active **Owner:** VIEWS platform maintainers -**Last reviewed:** 2026-07-02 -**Related ADRs:** ADR-001, ADR-008, ADR-011, ADR-012, ADR-013 +**Last reviewed:** 2026-07-27 +**Related ADRs:** ADR-001, ADR-008, ADR-011, ADR-012, ADR-013, ADR-026 > Implemented in v0.1.0 (`src/views_frames/feature_frame.py`). This contract governs > that implementation. @@ -44,6 +44,10 @@ array `y_features (N, F, S)` float32 aligned to a `SpatioTemporalIndex`, carryin - Row ops return new frames preserving `feature_names`: `select(positions | mask)` and `reindex(other)` — the latter raises unless this index is a superset of `other`; selection **copies** the selected `values`. +- `reindex_fill(other, *, fill_value)` (ADR-026): the dense-grid companion — no superset + requirement; present rows bit-exact, absent rows filled (`fill_value` keyword-only, + required, broadcast across `(F, S)`); `feature_names`/metadata preserved; law-pinned + by `assert_reindex_fill_law`. --- diff --git a/docs/CICs/PredictionFrame.md b/docs/CICs/PredictionFrame.md index e623213..8a32b33 100644 --- a/docs/CICs/PredictionFrame.md +++ b/docs/CICs/PredictionFrame.md @@ -3,8 +3,8 @@ **Status:** Active **Owner:** VIEWS platform maintainers -**Last reviewed:** 2026-07-02 -**Related ADRs:** ADR-001, ADR-008, ADR-011, ADR-012, ADR-013, ADR-017 +**Last reviewed:** 2026-07-27 +**Related ADRs:** ADR-001, ADR-008, ADR-011, ADR-012, ADR-013, ADR-017, ADR-026 > Implemented in v0.1.0 (`src/views_frames/prediction_frame.py`), relocated from > views-pipeline-core with numpy-only validation — the source imports pandas, so the @@ -43,6 +43,11 @@ ensemble samples `y_pred (N, S)` float32 aligned to a `SpatioTemporalIndex`. - Row ops return new frames: `select(positions | mask)` and `reindex(other)` — the latter raises unless this frame's index is a superset of `other`. Selection **copies** the selected `values` (only structural/metadata ops share the buffer). +- `reindex_fill(other, *, fill_value)` (ADR-026) is the dense-grid companion: aligns to + `other` with **no** superset requirement — present rows pass through **bit-exact**, + absent rows get the caller's `fill_value` (keyword-only, required; no silent default, + ADR-009). Allocates the full dense buffer; assumes unique rows in *self* (C-21); + law-pinned by `assert_reindex_fill_law` in the published conformance suite. - Sample-axis reduction (collapse/MAP/HDI) is **not** a method here — it lives in `views_frames_summarize` (ADR-017). The frame exposes the structural `sample_count`/ `is_sample` only. diff --git a/docs/CICs/SpatioTemporalIndex.md b/docs/CICs/SpatioTemporalIndex.md index 5ddfd60..bd2a827 100644 --- a/docs/CICs/SpatioTemporalIndex.md +++ b/docs/CICs/SpatioTemporalIndex.md @@ -3,8 +3,8 @@ **Status:** Active **Owner:** VIEWS platform maintainers -**Last reviewed:** 2026-06-24 -**Related ADRs:** ADR-001, ADR-002, ADR-009, ADR-013, ADR-014, ADR-015 +**Last reviewed:** 2026-07-27 +**Related ADRs:** ADR-001, ADR-002, ADR-009, ADR-013, ADR-014, ADR-015, ADR-026 > Implemented in v0.1.0 (`src/views_frames/index.py`). This contract governs that > implementation. @@ -43,6 +43,11 @@ label-alignment that today drags pandas back into the hot path. - Guarantees **same-level** set operations over `(time, unit)` are pure-numpy and deterministic: `intersect`, `reindex` / `searchsorted`, `is_superset_of`, `argsort`, and `select` (rows at integer positions or a boolean mask). +- Offers the dense product-index constructor `cartesian(times, units, level)` + (ADR-026): every `(time, unit)` combination in **time-major** order, from **explicit + arrays only** (a derivation rule — e.g. "units of the last time step" — is consumer + policy, never the leaf's). Fails loud on duplicated input values: a duplicated + product input manufactures duplicate rows and undefined joins (register C-21). - Exposes **cross-level** remap as an injected-mapping protocol — `cross_level_align` and the columnar `cross_level_align_arrays` (register C-26); see §2. - **Row-uniqueness stance (register C-21):** duplicate `(time, unit)` rows are *allowed* diff --git a/docs/CICs/TargetFrame.md b/docs/CICs/TargetFrame.md index 00956eb..ef18d29 100644 --- a/docs/CICs/TargetFrame.md +++ b/docs/CICs/TargetFrame.md @@ -3,8 +3,8 @@ **Status:** Active **Owner:** VIEWS platform maintainers -**Last reviewed:** 2026-07-02 -**Related ADRs:** ADR-001, ADR-008, ADR-011, ADR-012, ADR-013 +**Last reviewed:** 2026-07-27 +**Related ADRs:** ADR-001, ADR-008, ADR-011, ADR-012, ADR-013, ADR-026 > Implemented in v0.1.0 (`src/views_frames/target_frame.py`). This contract governs > that implementation. @@ -39,7 +39,10 @@ boundary array-native, replacing the pandas actuals the eval adapter takes today (writeable for zero-copy — in-place `.values` mutation is unsupported; ADR-025 / C-63). - Carries a typed `metadata` header (ADR-013) and the same row/metadata surface as the sibling frames: `with_metadata`, `select(positions | mask)`, `reindex(other)` (raises - unless this index is a superset of `other`); `sample_count == 1`, `is_sample == False`. + unless this index is a superset of `other`), and the dense-grid companion + `reindex_fill(other, *, fill_value)` (ADR-026 — no superset requirement; present rows + bit-exact, absent rows filled; law-pinned by `assert_reindex_fill_law`); + `sample_count == 1`, `is_sample == False`. - The role (ground truth, single realized value) is explicit so line-graph / eval code can treat it distinctly from sampled frames. diff --git a/pyproject.toml b/pyproject.toml index f781b53..492c8cd 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "views-frames" -version = "1.9.0" +version = "1.10.0" description = "The VIEWS platform data-contract layer: immutable array+identifier frames (numpy only, root of the dependency DAG)." authors = [ { name = "Simon Polichinel von der Maase", email = "simmaa@prio.org" }, diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index 22a8c81..692fe3c 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -4,9 +4,9 @@ |-------------------|--------------------------------------| | Project | views-frames | | Owner | VIEWS platform maintainers | -| Last Updated | 2026-07-02 | -| Total Concerns | 67 | -| Open Concerns | 12 | +| Last Updated | 2026-07-27 | +| Total Concerns | 68 | +| Open Concerns | 13 | | Resolved Concerns | 55 | | Disagreements | 12 | @@ -35,7 +35,9 @@ > and formalised by ADRs 011–016, all of which merged and shipped/froze in **v1.0.0** > (ADR-018) — they are now in **Resolved Concerns**. C-01/C-08/C-12 are resolved-by-decision > and persist only as **frozen-invariant guards** (their triggers protect the frozen scope). -> The **12 currently open** concerns fall into five clusters plus a cross-cutting theme +> The **13 currently open** concerns fall into five clusters plus a cross-cutting theme +> (and one standalone operational item — **C-71**, the dense-fill grid-scale allocation +> awareness shipped with ADR-026, 2026-07-27) > (detailed under *Causal clusters* in Register Conventions): **(1) summarize-estimator > coherence (#89)** — C-32, C-34, C-43, C-57 (the under-determined frozen MAP/bimodality > estimators); **(2) reconcile method + governance** — C-58, C-62 (+ D-12): a pragmatic @@ -85,7 +87,7 @@ Note the estimator is **already semi-parametric**: the `zero_mass_threshold` rul **Latent today** (the leaf publishes nothing — hence Tier 2, not 1), but it is **silent, directional output incorrectness for any consumer that adopts it expecting parity**. Resolution path: estimator-design effort tracked in **#89** (a distributional assumption *or* `n`-adaptive smoothing + floor; **not** merely a better tie-break; SemVer decision required). See C-24 (resolved), C-25, C-33. -**Mitigation shipped (2026-06-23, ADR-019; redesigned 2026-06-24, C-44) — not a full resolution; stays open.** `tower_point` ships as an **unbinned, median-based** point estimator (the median of the configurable **`tip_mass`** floor, default 0.5 — the shorth), so it carries **none** of the lowest-index histogram tie-break's directional bias, and — reading a *mass-aware* floor rather than the degenerate 2-sample 5% floor — it is now also **robust to minority duplicated draws** (C-44). Scored against a *non-circular analytic-mode* oracle (the active families only — zero-mode families have no analytic continuous mode), it ties/beats `map_estimate` on clean active cells **at the production sample size n=1024**; at **n=128 the two are mixed** (the tip wins on some families, loses on others — see `research/map_hdi/point_pass.py`), so this is a mitigation at production `n`, not a guaranteed win at the low-`n` regime where the bias bites hardest. `bimodality` flags the multimodal cells where any single mode is ill-defined (with its own recall caveat — see C-34). **Residual:** `map_estimate` itself is unchanged (frozen, ADR-018) — a naïve adopter can still step on it (now with a documented better path, `tower_point`); and `tower_point` uses a **fixed** `tip_mass` (50%) floor, so it is **not** the consistency-guaranteed convergent mode this entry calls for. That remains **#89**. +**Mitigation shipped (2026-06-23, ADR-019; redesigned 2026-06-24, C-44) — not a full resolution; stays open.** `tower_point` ships as an **unbinned, median-based** point estimator (the median of the configurable **`tip_mass`** floor, default 0.5 — the shorth), so it carries **none** of the lowest-index histogram tie-break's directional bias, and — reading a *mass-aware* floor rather than the degenerate 2-sample 5% floor — it is now also **robust to minority duplicated draws** (C-44). Scored against a *non-circular analytic-mode* oracle (the active families only — zero-mode families have no analytic continuous mode), it ties/beats `map_estimate` on clean active cells **at the production sample size n=1024**; at **n=128 the two are mixed** (the tip wins on some families, loses on others — see `research/map_hdi/point_pass.py`), so this is a mitigation at production `n`, not a guaranteed win at the low-`n` regime where the bias bites hardest. `bimodality` flags the multimodal cells where any single mode is ill-defined (with its own recall caveat — see C-34). **Residual:** `map_estimate` itself is unchanged (frozen, ADR-018) — a naïve adopter can still step on it (now with a documented better path, `tower_point`); and `tower_point` uses a **fixed** `tip_mass` floor (0.25 — the top-quartile floor since ADR-019 Amendment 3, 2026-07-24; originally the 0.5 shorth), so it is **not** the consistency-guaranteed convergent mode this entry calls for. That remains **#89**. --- @@ -237,6 +239,21 @@ C-63 was resolved by **correcting the contract** (ADR-025): the value buffer is --- +### C-71: dense-grid fill allocates the full dense buffer — grid-scale memory footgun + +| Field | Value | +|-------|-------| +| ID | C-71 | +| Tier | 3 | +| Source | expert-code-review (2026-07-27, the #203 design review), Nygard lens; shipped with ADR-026 (v1.10.0). | +| Trigger | When a consumer densifies at grid scale — a full-pgm `cartesian` target (~259k cells × months ≈ tens of millions of rows) fed to `reindex_fill` on a sampled frame — check the buffer arithmetic first: the dense values buffer is `target.n_rows × (trailing axes) × 4` bytes (at S=1000, hundreds of GB). Same check applies to `cartesian` itself (eager `T × U` identifier allocation). | +| Location | `src/views_frames/index.py` (`cartesian`); `src/views_frames/{prediction_frame,feature_frame,target_frame}.py` (`reindex_fill` — `np.full` of the full dense shape). | +| Cross-refs | **ADR-026** (the decision: the leaf documents the cost, never guesses a size guard — that would be policy), C-21 (the uniqueness stance `reindex_fill` inherits and `cartesian`'s duplicate-input `ValueError` protects), C-22/C-25 (the memory-bounded precedent on the *estimator* side — deliberately not applied here: densification's output *is* the allocation). | + +The fill primitive makes densification a one-liner, which is the point (#203, faoapi #242) — and also the hazard: the allocation that faoapi's pandas implementation made visible (an explicit `MultiIndex.from_product` + concat) is now behind one method call. The cost is **inherent to densifying** (the output *is* the dense buffer), not an implementation choice, so the leaf's controls are documentation (both docstrings state the cost) and this entry. The failure is **loud** (`MemoryError`/OOM-kill), not silent — hence Tier 3, an operational-awareness item, not a correctness risk. If a consumer legitimately needs bounded-memory densification (block-wise fill-and-stream), that is a future additive design, receipted first. + +--- + ## Disagreements ### D-01: `SpatioTemporalIndex` domain-purity fork (where does cross-level alignment live?) @@ -949,7 +966,7 @@ A spatial-forecasting showcase with no spatial display under-serves the audience |-------|-------| | ID | C-21 | | Resolved | 2026-06-21 (v0.3.0, PR #65) | -| Resolution | Documented the stance on `SpatioTemporalIndex` (duplicates allowed — `cross_level_align` makes them; same-level joins assume uniqueness) + added `has_unique_rows()` for consumers that need the guarantee. No construction-time behaviour change. | +| Resolution | Documented the stance on `SpatioTemporalIndex` (duplicates allowed — `cross_level_align` makes them; same-level joins assume uniqueness) + added `has_unique_rows()` for consumers that need the guarantee. No construction-time behaviour change. **2026-07-27 (ADR-026, v1.10.0):** the stance now also covers `reindex_fill` (assumes unique rows in *self*; duplicate target rows allowed) and is *enforced* at the one place a duplicate is always a caller bug — `cartesian` raises on duplicated input values. See C-71. | --- diff --git a/src/views_frames/conformance/__init__.py b/src/views_frames/conformance/__init__.py index f58b0b9..8a41282 100644 --- a/src/views_frames/conformance/__init__.py +++ b/src/views_frames/conformance/__init__.py @@ -37,6 +37,7 @@ "assert_frame_contract", "assert_frame_envelope", "assert_index_alignment_laws", + "assert_reindex_fill_law", ] @@ -136,6 +137,50 @@ def assert_index_alignment_laws(index_a: Any, index_b: Any) -> None: assert np.array_equal(index_a.unit[pos], index_a.unit), "searchsorted self-identity" +def assert_reindex_fill_law(frame: Any, target: Any, fill_value: float) -> None: + """Assert the dense-grid fill law (ADR-026) for ``frame`` against ``target``. + + ``reindex_fill`` aligns a frame to a target index with **no** superset + requirement, filling absent rows. The law: + + - the result's index equals ``target`` row-for-row (time, unit, level); + - every target row present in ``frame`` comes through **bit-exact**; + - every absent row equals ``fill_value`` exactly (NaN-safe comparison); + - when every target row is present, ``reindex_fill`` degenerates to + ``reindex`` (filling a superset frame adds nothing). + + Args: + frame: a frame exposing ``reindex_fill``/``reindex``/``index``/``values``. + target: a ``SpatioTemporalIndex`` (same level) to densify against. + fill_value: the fill for absent rows (``NaN`` is legal). + + Raises: + AssertionError: a law is violated. + """ + _require_assertions() + filled = frame.reindex_fill(target, fill_value=fill_value) + assert filled.index.level == target.level, "reindex_fill must keep the level" + assert np.array_equal(filled.index.time, target.time), ( + "reindex_fill result index must equal the target (time)" + ) + assert np.array_equal(filled.index.unit, target.unit), ( + "reindex_fill result index must equal the target (unit)" + ) + pos = frame.index.searchsorted(target) + found = pos >= 0 + assert np.array_equal( + filled.values[found], frame.values[pos[found]], equal_nan=True + ), "reindex_fill must pass present rows through bit-exact" + expected = np.full_like(filled.values[~found], np.float32(fill_value)) + assert np.array_equal(filled.values[~found], expected, equal_nan=True), ( + "reindex_fill must set every absent row to fill_value" + ) + if bool(found.all()): + assert np.array_equal( + filled.values, frame.reindex(target).values, equal_nan=True + ), "reindex_fill on a superset frame must equal reindex" + + def assert_cross_level_alignment_law( index: Any, mapping: Any, target_level: Any ) -> None: diff --git a/src/views_frames/feature_frame.py b/src/views_frames/feature_frame.py index 8d2f2c4..34a5893 100644 --- a/src/views_frames/feature_frame.py +++ b/src/views_frames/feature_frame.py @@ -154,6 +154,37 @@ def reindex(self, other: SpatioTemporalIndex) -> FeatureFrame: ) return self.select(self._index.searchsorted(other)) + def reindex_fill( + self, other: SpatioTemporalIndex, *, fill_value: float + ) -> FeatureFrame: + """Align to ``other``'s rows, filling rows absent from this frame. + + The dense-grid companion to :meth:`reindex` (ADR-026): every row of + ``other`` present in this frame comes through **bit-exact**; a row + absent from it gets ``fill_value``, broadcast across the feature and + sample axes. There is no superset requirement — that is the difference + from the fail-loud ``reindex``. The result's index **is** ``other`` + (immutable, shared); ``feature_names`` and metadata are preserved. + + Same-level join semantics: assumes unique ``(time, unit)`` rows in + **this** frame (register C-21); ``other`` may repeat rows. Allocates + the full ``(other.n_rows, F, S)`` buffer — densifying a large grid is + a deliberate, costly act. A ``NaN`` fill is legal here, but the + summarize estimators fail loud on NaN draws by design (ADR-017). + + Raises: + ValueError: ``other`` is at a different ``SpatialLevel``. + """ + pos = self._index.searchsorted(other) + found = pos >= 0 + out = np.full( + (other.n_rows, *self._values.shape[1:]), + np.float32(fill_value), + dtype=np.float32, + ) + out[found] = self._values[pos[found]] + return FeatureFrame(out, other, self._feature_names, self._metadata) + # ---- persistence -------------------------------------------------------- def save(self, directory: Path | str) -> None: diff --git a/src/views_frames/index.py b/src/views_frames/index.py index a2871d9..57c534f 100644 --- a/src/views_frames/index.py +++ b/src/views_frames/index.py @@ -43,9 +43,7 @@ def __init__( level: SpatialLevel, ) -> None: if not isinstance(level, SpatialLevel): - raise TypeError( - f"level must be a SpatialLevel, got {type(level).__name__}" - ) + raise TypeError(f"level must be a SpatialLevel, got {type(level).__name__}") is_array = isinstance(time, np.ndarray) and time.ndim >= 1 n = int(time.shape[0]) if is_array else -1 validate_identifiers({"time": time, "unit": unit}, n_rows=n) @@ -56,6 +54,65 @@ def __init__( self._unit.setflags(write=False) self._level = level + # ---- construction ------------------------------------------------------- + + @classmethod + def cartesian( + cls, + times: IntArray, + units: IntArray, + level: SpatialLevel, + ) -> SpatioTemporalIndex: + """The dense ``(time × unit)`` product index, in time-major order. + + The dense-grid constructor (ADR-026): every combination of the given + ``times`` and ``units``, ordered time-major (all units of the first time, + then all units of the next), so dense indices are canonical across + consumers. The inputs are **explicit arrays only** — any rule for + deriving them (e.g. "the units present in the last time step") is + consumer policy and never lives in the leaf. + + Fails loud on duplicated ``times`` or ``units``: a duplicated product + input manufactures duplicate ``(time, unit)`` rows, which make every + same-level join undefined (register C-21) — always a caller bug here. + + Allocates ``len(times) × len(units)`` rows eagerly; densifying a large + grid is a deliberate, costly act (a full-pgm dense grid runs to tens of + millions of rows). + + Args: + times: 1-D integer array of unique time identifiers. + units: 1-D integer array of unique unit identifiers. + level: The ``SpatialLevel`` of the produced index. + + Raises: + TypeError: ``times``/``units`` is not an integer-dtype numpy array, + or ``level`` is not a ``SpatialLevel``. + ValueError: an input array is not 1-D or contains duplicate values. + """ + for name, arr in (("times", times), ("units", units)): + if not isinstance(arr, np.ndarray): + raise TypeError( + f"cartesian {name} must be a numpy array, got {type(arr).__name__}" + ) + if not np.issubdtype(arr.dtype, np.integer): + raise TypeError( + f"cartesian {name} must be an integer dtype, got {arr.dtype}" + ) + if arr.ndim != 1: + raise ValueError(f"cartesian {name} must be 1-D, got ndim={arr.ndim}") + if np.unique(arr).shape[0] != arr.shape[0]: + raise ValueError( + f"cartesian {name} contains duplicate values; a duplicated " + "product input manufactures duplicate (time, unit) rows, " + "which make same-level joins undefined (register C-21)" + ) + return cls( + time=np.repeat(times, units.shape[0]), + unit=np.tile(units, times.shape[0]), + level=level, + ) + # ---- core surface ------------------------------------------------------- @property @@ -109,9 +166,11 @@ def _keys(self) -> NDArray[np.int64]: @staticmethod def _row_view(keys: NDArray[np.int64]) -> NDArray[np.void]: """View each ``(time, unit)`` row as a single void scalar for set ops.""" - return np.ascontiguousarray(keys).view( - np.dtype((np.void, keys.dtype.itemsize * keys.shape[1])) - ).reshape(-1) + return ( + np.ascontiguousarray(keys) + .view(np.dtype((np.void, keys.dtype.itemsize * keys.shape[1]))) + .reshape(-1) + ) def _require_same_level(self, other: SpatioTemporalIndex) -> None: if self._level != other._level: @@ -180,9 +239,7 @@ def has_unique_rows(self) -> bool: rows = self._row_view(self._keys()) return bool(len(np.unique(rows)) == rows.shape[0]) - def select( - self, indexer: IntArray | NDArray[np.bool_] - ) -> SpatioTemporalIndex: + def select(self, indexer: IntArray | NDArray[np.bool_]) -> SpatioTemporalIndex: """A new index of the rows at integer positions **or** a boolean mask. The row-selection primitive the frame-level ``select``/``reindex`` build on: @@ -298,9 +355,7 @@ def _remap( sorted_rows = map_rows[order] self_rows = self._row_view(self._keys()) - pos = np.clip( - np.searchsorted(sorted_rows, self_rows), 0, len(sorted_rows) - 1 - ) + pos = np.clip(np.searchsorted(sorted_rows, self_rows), 0, len(sorted_rows) - 1) found = sorted_rows[pos] == self_rows if not bool(found.all()): miss = int(np.argmax(~found)) diff --git a/src/views_frames/prediction_frame.py b/src/views_frames/prediction_frame.py index 912be83..c842307 100644 --- a/src/views_frames/prediction_frame.py +++ b/src/views_frames/prediction_frame.py @@ -118,6 +118,37 @@ def reindex(self, other: SpatioTemporalIndex) -> PredictionFrame: ) return self.select(self._index.searchsorted(other)) + def reindex_fill( + self, other: SpatioTemporalIndex, *, fill_value: float + ) -> PredictionFrame: + """Align to ``other``'s rows, filling rows absent from this frame. + + The dense-grid companion to :meth:`reindex` (ADR-026): every row of + ``other`` present in this frame comes through **bit-exact**; a row + absent from it gets ``fill_value``, broadcast across the sample axis. + There is no superset requirement — that is the difference from the + fail-loud ``reindex``. The result's index **is** ``other`` (immutable, + shared); metadata is preserved. + + Same-level join semantics: assumes unique ``(time, unit)`` rows in + **this** frame (register C-21); ``other`` may repeat rows. Allocates + the full ``(other.n_rows, S)`` buffer — densifying a large grid is a + deliberate, costly act. A ``NaN`` fill is legal here, but the + summarize estimators fail loud on NaN draws by design (ADR-017). + + Raises: + ValueError: ``other`` is at a different ``SpatialLevel``. + """ + pos = self._index.searchsorted(other) + found = pos >= 0 + out = np.full( + (other.n_rows, *self._values.shape[1:]), + np.float32(fill_value), + dtype=np.float32, + ) + out[found] = self._values[pos[found]] + return PredictionFrame(out, other, self._metadata) + # ---- persistence -------------------------------------------------------- def save(self, directory: Path | str) -> None: diff --git a/src/views_frames/target_frame.py b/src/views_frames/target_frame.py index 2f3c00f..2a83e25 100644 --- a/src/views_frames/target_frame.py +++ b/src/views_frames/target_frame.py @@ -113,6 +113,36 @@ def reindex(self, other: SpatioTemporalIndex) -> TargetFrame: ) return self.select(self._index.searchsorted(other)) + def reindex_fill( + self, other: SpatioTemporalIndex, *, fill_value: float + ) -> TargetFrame: + """Align to ``other``'s rows, filling rows absent from this frame. + + The dense-grid companion to :meth:`reindex` (ADR-026): every row of + ``other`` present in this frame comes through **bit-exact**; a row + absent from it gets ``fill_value``. There is no superset requirement — + that is the difference from the fail-loud ``reindex``. The result's + index **is** ``other`` (immutable, shared); metadata is preserved. + + Same-level join semantics: assumes unique ``(time, unit)`` rows in + **this** frame (register C-21); ``other`` may repeat rows. Allocates + the full ``(other.n_rows, 1)`` buffer — densifying a large grid is a + deliberate, costly act. A ``NaN`` fill is legal here, but the + summarize estimators fail loud on NaN draws by design (ADR-017). + + Raises: + ValueError: ``other`` is at a different ``SpatialLevel``. + """ + pos = self._index.searchsorted(other) + found = pos >= 0 + out = np.full( + (other.n_rows, *self._values.shape[1:]), + np.float32(fill_value), + dtype=np.float32, + ) + out[found] = self._values[pos[found]] + return TargetFrame(out, other, self._metadata) + # ---- persistence -------------------------------------------------------- def save(self, directory: Path | str) -> None: diff --git a/tests/test_reindex_fill.py b/tests/test_reindex_fill.py new file mode 100644 index 0000000..2be709e --- /dev/null +++ b/tests/test_reindex_fill.py @@ -0,0 +1,306 @@ +"""Dense-grid fill (ADR-026): `SpatioTemporalIndex.cartesian` + `reindex_fill`. + +Closes the faoapi ingestion gap (#203 — "views-frames has no fill primitive"): +materialize a complete (time × unit) grid from a sparse frame without pandas. +The fill rides the existing same-level join (`searchsorted` returns -1 for +absent rows) and owns the -1 sentinel once — a consumer hand-rolling +`values[pos]` would silently pick the *last* row for absent cells. +""" + +from __future__ import annotations + +import numpy as np +import pytest + +from views_frames import ( + FeatureFrame, + FrameMetadata, + PredictionFrame, + SpatialLevel, + SpatioTemporalIndex, + TargetFrame, +) +from views_frames.conformance import assert_reindex_fill_law + + +def _index(times, units, level=SpatialLevel.PGM): + return SpatioTemporalIndex( + np.array(times, dtype=np.int64), np.array(units, dtype=np.int32), level + ) + + +def _pf(times, units, s=3, meta=None): + idx = _index(times, units) + values = np.arange(len(times) * s, dtype=np.float32).reshape(len(times), s) + return PredictionFrame(values, idx, meta) + + +def _times(*vals): + return np.array(vals, dtype=np.int64) + + +def _units(*vals): + return np.array(vals, dtype=np.int32) + + +# --- SpatioTemporalIndex.cartesian ------------------------------------------- + + +def test_cartesian_product_is_time_major(): + idx = SpatioTemporalIndex.cartesian( + _times(1, 2), _units(10, 11, 12), SpatialLevel.PGM + ) + assert idx.n_rows == 6 + assert np.array_equal(idx.time, [1, 1, 1, 2, 2, 2]) + assert np.array_equal(idx.unit, [10, 11, 12, 10, 11, 12]) + assert idx.level is SpatialLevel.PGM + + +def test_cartesian_preserves_input_dtypes(): + idx = SpatioTemporalIndex.cartesian(_times(1), _units(10), SpatialLevel.CM) + assert idx.time.dtype == np.int64 + assert idx.unit.dtype == np.int32 + + +def test_cartesian_rows_are_unique(): + idx = SpatioTemporalIndex.cartesian( + _times(1, 2, 3), _units(10, 11), SpatialLevel.PGM + ) + assert idx.has_unique_rows() + + +def test_cartesian_empty_times_yields_empty_index(): + idx = SpatioTemporalIndex.cartesian( + np.array([], dtype=np.int64), _units(10, 11), SpatialLevel.PGM + ) + assert idx.n_rows == 0 + + +def test_cartesian_non_array_raises_type_error(): + with pytest.raises(TypeError, match="times must be a numpy array"): + SpatioTemporalIndex.cartesian([1, 2], _units(10), SpatialLevel.PGM) + + +def test_cartesian_non_integer_dtype_raises_type_error(): + with pytest.raises(TypeError, match="units must be an integer dtype"): + SpatioTemporalIndex.cartesian(_times(1), np.array([10.0]), SpatialLevel.PGM) + + +def test_cartesian_non_1d_raises(): + with pytest.raises(ValueError, match="times must be 1-D"): + SpatioTemporalIndex.cartesian( + np.array([[1, 2]], dtype=np.int64), _units(10), SpatialLevel.PGM + ) + + +def test_cartesian_duplicate_times_raise(): + # a duplicated product input manufactures duplicate rows → undefined joins + # (register C-21); always a caller bug, so it fails loud. + with pytest.raises(ValueError, match="times contains duplicate"): + SpatioTemporalIndex.cartesian(_times(1, 1), _units(10), SpatialLevel.PGM) + + +def test_cartesian_duplicate_units_raise(): + with pytest.raises(ValueError, match="units contains duplicate"): + SpatioTemporalIndex.cartesian(_times(1), _units(10, 10), SpatialLevel.PGM) + + +def test_cartesian_bad_level_raises_type_error(): + with pytest.raises(TypeError, match="level must be a SpatialLevel"): + SpatioTemporalIndex.cartesian(_times(1), _units(10), "pgm") + + +# --- frame.reindex_fill: sparse → dense -------------------------------------- + + +def test_reindex_fill_sparse_to_dense(): + # rows (1,10) and (2,11) exist; the dense 2×2 grid gains (1,11) and (2,10). + frame = _pf([1, 2], [10, 11]) + target = SpatioTemporalIndex.cartesian( + _times(1, 2), _units(10, 11), SpatialLevel.PGM + ) + filled = frame.reindex_fill(target, fill_value=0.0) + assert filled.n_rows == 4 + # present rows bit-exact + assert np.array_equal(filled.values[0], frame.values[0]) # (1,10) + assert np.array_equal(filled.values[3], frame.values[1]) # (2,11) + # absent rows filled + assert np.array_equal(filled.values[1], np.zeros(3, dtype=np.float32)) + assert np.array_equal(filled.values[2], np.zeros(3, dtype=np.float32)) + assert filled.values.dtype == np.float32 + + +def test_reindex_fill_result_index_is_target(): + frame = _pf([1], [10]) + target = _index([1, 2], [10, 10]) + filled = frame.reindex_fill(target, fill_value=0.0) + assert filled.index is target # immutable, shared — zero-copy identity + + +def test_reindex_fill_nan_fill_is_legal(): + frame = _pf([1], [10]) + target = _index([1, 2], [10, 10]) + filled = frame.reindex_fill(target, fill_value=float("nan")) + assert np.array_equal(filled.values[0], frame.values[0]) + assert np.isnan(filled.values[1]).all() + + +def test_reindex_fill_superset_equals_reindex(): + # idempotence: with every target row present, filling adds nothing. + frame = _pf([1, 2, 3], [10, 10, 10]) + target = _index([3, 1], [10, 10]) + filled = frame.reindex_fill(target, fill_value=-99.0) + assert np.array_equal(filled.values, frame.reindex(target).values) + + +def test_reindex_fill_empty_self_fills_everything(): + frame = _pf([], []) + target = _index([1, 2], [10, 10]) + filled = frame.reindex_fill(target, fill_value=7.0) + assert np.array_equal(filled.values, np.full((2, 3), 7.0, dtype=np.float32)) + + +def test_reindex_fill_empty_target_yields_empty_frame(): + frame = _pf([1], [10]) + filled = frame.reindex_fill(_index([], []), fill_value=0.0) + assert filled.n_rows == 0 + assert filled.values.shape == (0, 3) + + +def test_reindex_fill_duplicate_target_rows_repeat(): + # cross_level_align legitimately produces duplicate-row targets; positions + # simply repeat. + frame = _pf([1], [10]) + target = _index([1, 1], [10, 10]) + filled = frame.reindex_fill(target, fill_value=0.0) + assert np.array_equal(filled.values[0], filled.values[1]) + + +def test_reindex_fill_preserves_metadata(): + meta = FrameMetadata(model="m") + frame = _pf([1], [10], meta=meta) + filled = frame.reindex_fill(_index([1, 2], [10, 10]), fill_value=0.0) + assert filled.metadata.model == "m" + + +def test_reindex_fill_level_mismatch_raises(): + frame = _pf([1], [10]) + with pytest.raises(ValueError, match="same-level"): + frame.reindex_fill(_index([1], [10], level=SpatialLevel.CM), fill_value=0.0) + + +def test_feature_frame_reindex_fill(): + idx = _index([1, 2], [10, 11]) + values = np.arange(2 * 2 * 3, dtype=np.float32).reshape(2, 2, 3) + frame = FeatureFrame(values, idx, ["a", "b"]) + target = SpatioTemporalIndex.cartesian( + _times(1, 2), _units(10, 11), SpatialLevel.PGM + ) + filled = frame.reindex_fill(target, fill_value=0.0) + assert filled.values.shape == (4, 2, 3) + assert filled.feature_names == ["a", "b"] + assert np.array_equal(filled.values[0], values[0]) + assert np.array_equal(filled.values[1], np.zeros((2, 3), dtype=np.float32)) + + +def test_target_frame_reindex_fill(): + idx = _index([1, 2], [10, 11]) + frame = TargetFrame(np.array([[1.0], [2.0]], dtype=np.float32), idx) + target = SpatioTemporalIndex.cartesian( + _times(1, 2), _units(10, 11), SpatialLevel.PGM + ) + filled = frame.reindex_fill(target, fill_value=0.0) + assert filled.values.shape == (4, 1) + assert np.array_equal( + filled.values[:, 0], np.array([1.0, 0.0, 0.0, 2.0], dtype=np.float32) + ) + + +# --- the published conformance law ------------------------------------------- + + +def test_fill_law_holds_for_builtin_frames(): + frames = [ + _pf([1, 2], [10, 11]), + FeatureFrame( + np.ones((2, 2, 3), dtype=np.float32), _index([1, 2], [10, 11]), ["a", "b"] + ), + TargetFrame(np.ones((2, 1), dtype=np.float32), _index([1, 2], [10, 11])), + ] + target = SpatioTemporalIndex.cartesian( + _times(1, 2), _units(10, 11), SpatialLevel.PGM + ) + for frame in frames: + assert_reindex_fill_law(frame, target, 0.0) + + +def test_fill_law_holds_with_nan_fill(): + frame = _pf([1, 2], [10, 11]) + target = SpatioTemporalIndex.cartesian( + _times(1, 2), _units(10, 11), SpatialLevel.PGM + ) + assert_reindex_fill_law(frame, target, float("nan")) + + +def test_fill_law_covers_the_superset_degeneracy(): + # every target row present → the reindex-equality arm of the law runs. + frame = _pf([1, 2], [10, 10]) + assert_reindex_fill_law(frame, _index([2], [10]), 0.0) + + +# The law's job is to REJECT a lying implementation; each structural reject gets a +# direct adversarial test (register C-51 — branch coverage does not exercise an +# `assert`'s raise-path). + + +class _Lying: + """Wraps a real frame but corrupts what `reindex_fill` returns.""" + + def __init__(self, frame, corrupt): + self._frame = frame + self._corrupt = corrupt + self.index = frame.index + self.values = frame.values + + def reindex_fill(self, other, *, fill_value): + return self._corrupt(self._frame, other, fill_value) + + def reindex(self, other): + return self._frame.reindex(other) + + +def test_fill_law_catches_wrong_fill_value(): + frame = _pf([1, 2], [10, 11]) + target = SpatioTemporalIndex.cartesian( + _times(1, 2), _units(10, 11), SpatialLevel.PGM + ) + lying = _Lying(frame, lambda f, o, fv: f.reindex_fill(o, fill_value=fv + 1.0)) + with pytest.raises(AssertionError, match="absent row"): + assert_reindex_fill_law(lying, target, 0.0) + + +def test_fill_law_catches_corrupted_present_rows(): + frame = _pf([1, 2], [10, 11]) + target = SpatioTemporalIndex.cartesian( + _times(1, 2), _units(10, 11), SpatialLevel.PGM + ) + + def corrupt(f, o, fv): + good = f.reindex_fill(o, fill_value=fv) + return PredictionFrame(good.values + 1.0, good.index, good.metadata) + + with pytest.raises(AssertionError, match="bit-exact"): + assert_reindex_fill_law(_Lying(frame, corrupt), target, 0.0) + + +def test_fill_law_catches_wrong_result_index(): + frame = _pf([1, 2], [10, 11]) + target = _index([1, 2], [10, 11]) + + def corrupt(f, o, fv): + good = f.reindex_fill(o, fill_value=fv) + reversed_pos = np.arange(o.n_rows, dtype=np.intp)[::-1] + return good.select(reversed_pos) + + with pytest.raises(AssertionError, match="equal the target"): + assert_reindex_fill_law(_Lying(frame, corrupt), target, 0.0) diff --git a/uv.lock b/uv.lock index 24a03de..bd62752 100644 --- a/uv.lock +++ b/uv.lock @@ -3274,7 +3274,7 @@ wheels = [ [[package]] name = "views-frames" -version = "1.9.0" +version = "1.10.0" source = { editable = "." } dependencies = [ { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" },