refactor: reference payloads become test fixtures, not package data (#162)

.github/workflows/ci.yml

@@ -145,6 +145,31 @@ jobs:
             print(f'{wheel}: py.typed present')
         "
 
+    # Reference captures are test data. They shipped in both wheels until 3.1.0
+    # -- not through any packaging declaration, but because a directory inside a
+    # package directory ships -- and no runtime path ever read them. Nothing in
+    # the manifests would object to that happening again, so the built artifact
+    # is where it has to be asserted. Every wheel, not the two known ones: an
+    # adapter added under packages/ is covered the day it exists.
+    - name: Verify no wheel ships reference payloads
+      run: |
+        python -c "
+        import glob, posixpath, sys, zipfile
+        wheels = glob.glob('dist/*.whl')
+        if not wheels:
+            sys.exit('::error::no wheels were built')
+        for wheel in wheels:
+            names = zipfile.ZipFile(wheel).namelist()
+            carried = [
+                n for n in names
+                if 'reference_payloads' in n.split('/')
+                or posixpath.basename(n) in ('homie_schema.json', 'parent_child_tree.json')
+            ]
+            if carried:
+                sys.exit(f'::error::{wheel} ships test data: {carried}. Reference captures are fixtures under tests/reference_payloads; a directory inside a package directory ships whether or not the manifest names it.')
+            print(f'{wheel}: no reference payloads')
+        "
+
     # The configuration entry-point discovery exists to support, and the one
     # nothing else in CI exercises: the bootstrap wheel installed with no
     # adapter present. It must import, and it must fail by name rather than

.github/workflows/peer-drift.yml

@@ -193,7 +193,7 @@ jobs:
         print("```bash")
         print(f"# in a checkout of the emitter at v{latest}, from its own environment")
         print("uv run python ../span-panel-api/scripts/capture_parent_child_reference.py \\")
-        print("  ../span-panel-api/packages/schema-1/src/span_panel_api_schema_1/reference_payloads/parent_child_tree.json")
+        print("  ../span-panel-api/tests/reference_payloads/parent_child_tree.json")
         print("```\n")
         print("The script refuses until `peers.ebus-panel-sim.version` and `.commit` in")
         print("`spec_lock.json` name the release you captured from, which is what keeps the")

.pre-commit-config.yaml

@@ -87,7 +87,13 @@ repos:
           # types rather than silently reporting import-not-found.
           - ebus-sdk>=0.19.0
         args: ['--config-file=pyproject.toml']
-        exclude: '^src/span_panel_api/generated_client/.*|scripts/.*|tests/.*|docs/.*|examples/.*|\..*_cache/.*|dist/.*|venv/.*'
+        # `tests/reference_payloads/` is the one thing under tests/ this hook does
+        # check. Its five accessors were type-checked as package data until 3.1.0
+        # and are read by twenty test modules; moving them out of `src/` should not
+        # have quietly cost them `--strict`. `pyproject.toml`'s own `exclude` still
+        # names tests/, which is fine — that flag governs directory discovery, and
+        # a file passed by name is checked regardless.
+        exclude: '^src/span_panel_api/generated_client/.*|scripts/.*|tests/(?!reference_payloads/).*|docs/.*|examples/.*|\..*_cache/.*|dist/.*|venv/.*'
 
   # Pylint for code quality
   - repo: https://github.com/pycqa/pylint

CHANGELOG.md

@@ -142,6 +142,17 @@ install of the adapter distribution does not, and a 1.0.0 adapter against this b
   would have meant a caller passing both got system trust while believing it had pinned the panel CA — a security control that appears to be on and is off. When both are supplied, a dedicated client is built for the call and closed after it. The cost is
   named rather than hidden: those calls lose the injected client's connection pool, timeout and header policy. Acceptable because every caller here is bootstrap — registration, detection, schema, FQDN, status — a handful of calls per config entry.
 
+### Removed
+
+- **BREAKING: `span_panel_api.reference_payloads` is gone, and the wheel no longer carries `homie_schema.json`.** The captured `GET /api/v2/homie/schema` response is a fixture of this repository's test suite now, at
+  `tests/reference_payloads/homie_schema.json`, read through `homie_schema()` / `homie_schema_types()` there. Anyone importing the module from an installed distribution has to vendor the bytes instead — and should record which release they were taken from,
+  asserting that against `importlib.metadata.version("span-panel-api")`, so a pin that moves past a stale copy fails loudly rather than checking declarations against a schema no panel runs. That version claim is available to any consumer without a
+  checkout, which is what makes vendoring safe and is the whole reason this can be removed.
+
+  It shipped in the first place to spare consumers a copy that goes stale in silence, which was a real problem badly solved: no runtime path ever read the file, so every install of both distributions paid for test data it could not use, and the import
+  surface committed each distribution to a promise it never meant to make. Nothing declared the payloads as package data — a directory inside a package directory ships whether or not a manifest names it, which is exactly why this was easy to miss.
+  `tests/test_packaging.py` now fails if a capture reappears inside a shipped package, and CI asserts the same against every built wheel. See #162.
+
 ## [3.0.1]
 
 ### Fixed

DEVELOPMENT.md

@@ -78,17 +78,17 @@ now records the simulator commit its output came from, for the same reason `spec
 
 Two scripts, one per producer, and neither is run automatically — a capture is a deliberate act.
 
-| Artifact                                                          | Producer         | Script                                      |
-| ----------------------------------------------------------------- | ---------------- | ------------------------------------------- |
-| `tests/fixtures/flat_wire.json`                                   | `simulator`      | `scripts/capture_flat_reference.py`         |
-| `packages/schema-1/.../reference_payloads/parent_child_tree.json` | `ebus-panel-sim` | `scripts/capture_parent_child_reference.py` |
+| Artifact                                          | Producer         | Script                                      |
+| ------------------------------------------------- | ---------------- | ------------------------------------------- |
+| `tests/fixtures/flat_wire.json`                   | `simulator`      | `scripts/capture_flat_reference.py`         |
+| `tests/reference_payloads/parent_child_tree.json` | `ebus-panel-sim` | `scripts/capture_parent_child_reference.py` |
 
 Both run from the **producer's** environment rather than this one — each producer caps a dependency this repo installs above — and both substitute the transport rather than reassembling the emitter, because a capture taken through different wiring than a
 real panel uses is a capture of the wiring. Point them at a checkout with `SIMULATOR_DIR` / `PANEL_SIM_DIR`.
 
 `capture_parent_child_reference.py` goes one step further than documenting its producer: it reads the release it is a capture of out of `spec_lock.json` (`peers.ebus-panel-sim.version`) and **refuses to write** when the installed package disagrees. The pin
-therefore has exactly one home, and re-capturing against a newer emitter is a two-place change made together — that peer block, and the provenance section of `reference_payloads/README.md`. That is what stops the bytes and the claim about them drifting
-apart, and the drift is not hypothetical: it is how a producer defect in `$settable` on a locked relay reached about thirty test files across two repositories with no conformance gate objecting.
+therefore has exactly one home, and re-capturing against a newer emitter is a two-place change made together — that peer block, and the provenance section of `tests/reference_payloads/README.md`. That is what stops the bytes and the claim about them
+drifting apart, and the drift is not hypothetical: it is how a producer defect in `$settable` on a locked relay reached about thirty test files across two repositories with no conformance gate objecting.
 
 Its input is committed too, as `scripts/reference_panel.yaml`, pinned as `peers.ebus-panel-sim.manifest` — a capture whose input is not in the tree is the same class of problem as one whose producer is not recorded. That manifest is a synthetic `example-*`
 panel that mirrors the emitter's own `examples/forty_tab_minimal.yaml` key for key and marks its two deliberate divergences at the head of the file: spec-legal shed priorities in place of a value the emitter degrades to `UNKNOWN`

README.md

@@ -527,26 +527,11 @@ The `PanelCapability` flag enum advertises transport features at runtime:
 
 ## Reference Payloads
 
-Captures of what a panel actually serves, shipped as package data so a consumer can check its own assumptions against real bytes without vendoring a copy that silently goes stale:
+Captures of what a panel actually serves — the `GET /api/v2/homie/schema` document and a full 40-space parent/child retained-topic tree — live in [`tests/reference_payloads/`](tests/reference_payloads/README.md), with their provenance.
 
-```python
-from span_panel_api.reference_payloads import homie_schema, homie_schema_types
-
-document = homie_schema()        # the captured GET /api/v2/homie/schema response
-types = homie_schema_types()     # its `types` map, typed as HomieSchemaTypes
-```
-
-`homie_schema_types()` returns exactly what `span_panel_api_schema_0.field_metadata.build_field_metadata` accepts, so building real adapter metadata to compare against is two lines and no file handling.
-
-The parent/child device tree is the schema_1 counterpart and ships from that adapter, with the parser that can interpret it:
-
-```python
-from span_panel_api_schema_1.reference_payloads import devices_from_tree, parent_child_tree
-
-devices = devices_from_tree(parent_child_tree())
-```
-
-Each payload carries the version of the release it shipped in. Pin a version and you read the bytes that version was written against.
+**They are fixtures of this repository, not package data.** Until 3.1.0 they sat inside the two source packages and were therefore carried in the wheels, which no runtime path ever read. `span_panel_api.reference_payloads` and
+`span_panel_api_schema_1.reference_payloads` no longer exist; a consumer that was importing them should vendor the bytes it needs and record the release it took them from, asserting that against `importlib.metadata.version(...)` so a moved pin that outruns
+the copy fails loudly instead of testing against a schema no panel runs.
 
 ## Project Structure
 
@@ -567,7 +552,6 @@ src/span_panel_api/          # distribution: span-panel-api (no parser)
 ├── phase_validation.py      # Electrical phase utilities
 ├── protocol.py              # PEP 544 protocols, SchemaAdapter, PanelCapability flags
 ├── schema_drift.py          # Reporting a panel that outruns what we can read
-├── reference_payloads/      # Captured GET /api/v2/homie/schema, shipped as package data
 └── mqtt/
     ├── __init__.py
     ├── async_client.py      # NullLock + AsyncMQTTClient (HA core pattern)
@@ -585,7 +569,7 @@ packages/schema-1/           # distribution: span-panel-api-schema-1
 ├── spec/                    # eBus capability catalogs, byte-copied; checked against, never parsed
 └── src/span_panel_api_schema_1/
                              # Parent/child parser: ControllerRoutes, snapshot mapper,
-                             # adoption, catalog validator, spec_lock.json, reference payloads
+                             # adoption, catalog validator, spec_lock.json
 ```
 
 ## Development

packages/schema-1/CHANGELOG.md

@@ -84,6 +84,16 @@ Requires `span-panel-api` **3.1.0 or newer**, and the two must be upgraded toget
   handling them is a contract obligation regardless of what hardware emits — and `test_every_catalogued_priority_is_carried_through_rather_than_repaired` is now parametrised over the catalog's declared format rather than over whatever the capture happens
   to contain. Contract obligations come from the catalog; representativeness comes from the capture. A value added upstream reaches that test the moment the catalog is re-vendored.
 
+### Removed
+
+- **BREAKING: `span_panel_api_schema_1.reference_payloads` is gone, and the wheel no longer carries `parent_child_tree.json`.** The 40-space retained-topic capture, and the `parent_child_tree()` / `device_from_topics()` / `devices_from_tree()` replay that
+  reads it, are fixtures of the repository's test suite now, at `tests/reference_payloads/schema_one.py`. The helpers stay beside the capture wherever it lives, for the reason they were written: a tree is not directly usable, and separating them would put
+  the same twelve lines of replay in each of the modules that read it.
+
+  It was package data for a good reason — a vendored copy has no version and goes stale in silence — but the reason does not survive inspection: no runtime path here reads the capture, so every install paid 40 KB for test data, and detecting staleness
+  never needed the file, only the version claim. A consumer that vendors the bytes and asserts its recorded source release against `importlib.metadata.version("span-panel-api-schema-1")` gets the loud failure the package data was protecting, with no
+  checkout and no dependency on this distribution's test fixtures. `spec_lock.json`'s `peers.ebus-panel-sim.produces.tree` names the new location, and a test resolves it so the record cannot outlive the file. See #162.
+
 ## [1.0.0]
 
 First release as a standalone distribution, and the first parser for the parent/child data model. Requires `span-panel-api` 3.0.0 or newer, and `ebus-sdk` `>=0.19,<0.24`.

packages/schema-1/README.md

@@ -50,15 +50,7 @@ returns `None`, and an entity disappears.
 
 ## Reference payloads
 
-A retained-topic capture of a full 40-space parent/child panel ships as package data, with the replay that turns it back into devices:
+A retained-topic capture of a full 40-space parent/child panel, and the replay that turns it back into devices, are fixtures of the repository's test suite at `tests/reference_payloads/`.
 
-```python
-from span_panel_api_schema_1.reference_payloads import devices_from_tree, parent_child_tree
-
-devices = devices_from_tree(parent_child_tree())
-```
-
-It ships here rather than from the bootstrap because a retained topic tree is only interpretable by the parser that speaks its vocabulary, and the eBus SDK is this distribution's dependency alone. `devices_from_tree` takes the tree rather than reading it,
-so a consumer can filter the capture first — dropping the BESS to model a panel that has none — and still build devices the same way. The bootstrap ships the schema document it fetches; see `span_panel_api.reference_payloads`.
-
-Each payload carries the version of the release it shipped in. Pin a version and you read the bytes that version was written against.
+**`span_panel_api_schema_1.reference_payloads` no longer exists.** It was package data until 1.1.0 — carried in this wheel because it sat inside the package directory, though no runtime path read it. A consumer that was importing it should vendor the bytes
+it needs and record the release it took them from, asserting that against `importlib.metadata.version("span-panel-api-schema-1")` so a moved pin that outruns the copy fails loudly instead of testing against a tree no panel publishes.

packages/schema-1/src/span_panel_api_schema_1/spec_lock.json

@@ -35,7 +35,7 @@
       "capture_script": "scripts/capture_parent_child_reference.py",
       "manifest": "scripts/reference_panel.yaml",
       "produces": {
-        "tree": "packages/schema-1/src/span_panel_api_schema_1/reference_payloads/parent_child_tree.json"
+        "tree": "tests/reference_payloads/parent_child_tree.json"
       }
     }
   },

scripts/capture_parent_child_reference.py

@@ -1,16 +1,17 @@
 """Capture the parent/child emitter's retained surface, without a broker.
 
-Produces `packages/schema-1/src/span_panel_api_schema_1/reference_payloads/
-parent_child_tree.json`, the schema_1 reference tree that ships as package data
-and that fifteen test modules here replay through `devices_from_tree`.
+Produces `tests/reference_payloads/parent_child_tree.json`, the schema_1
+reference tree that fifteen test modules here replay through `devices_from_tree`.
+A repository fixture, not package data: it sat inside the adapter's package
+directory until 3.1.0 and was carried in the wheel for it, and no consumer of
+either distribution reads it at runtime.
 
 Run it from the **emitter's** environment, not this one — it imports
 `ebus_panel_sim`, which caps `ebus-sdk` below the version this repo installs:
 
     cd ../distribution-enclosure-simulator
     uv run python ../span-panel-api/scripts/capture_parent_child_reference.py \\
-        ../span-panel-api/packages/schema-1/src/span_panel_api_schema_1/\\
-reference_payloads/parent_child_tree.json
+        ../span-panel-api/tests/reference_payloads/parent_child_tree.json
 
 `PANEL_SIM_DIR` overrides where the checkout is looked for; it defaults to a
 `distribution-enclosure-simulator` directory beside this repo. Passing no output
@@ -475,11 +476,11 @@ def main() -> None:
     if PRODUCER_VERSION != expected:
         raise SystemExit(
             f"{SIM} is ebus-panel-sim {PRODUCER_VERSION}, and spec_lock.json records the reference "
-            f"tree as a capture of {expected}. Capturing anyway would put bytes in the wheel that "
-            "the lockfile attributes to a release that did not make them. Move the checkout to the "
-            "pinned release, or take the new capture deliberately: update peers.ebus-panel-sim's "
+            f"tree as a capture of {expected}. Capturing anyway would put bytes in this repository "
+            "that the lockfile attributes to a release that did not make them. Move the checkout to "
+            "the pinned release, or take the new capture deliberately: update peers.ebus-panel-sim's "
             "version, tag and commit in spec_lock.json, and the provenance section of "
-            "reference_payloads/README.md, in the same change."
+            "tests/reference_payloads/README.md, in the same change."
         )
 
     profile = Profile(MANIFEST)

scripts/reference_panel.yaml

@@ -2,7 +2,7 @@
 #
 # Read by `scripts/capture_parent_child_reference.py`, which drives the eBus
 # emitter (`ebus-panel-sim`) with it and records the retained topics as
-# `packages/schema-1/src/span_panel_api_schema_1/reference_payloads/parent_child_tree.json`.
+# `tests/reference_payloads/parent_child_tree.json`.
 # Recorded in `spec_lock.json` as `peers.ebus-panel-sim.manifest`, so the capture's
 # input is pinned the same way its producer is: a capture whose input is not in the
 # tree is the same class of problem as one whose producer version is not written down.

scripts/verify_reconnect.py

@@ -39,7 +39,7 @@
         --broker-host 127.0.0.1 --broker-port 1883 --no-tls \
         --data-model-version 1.0 \
         --adapter span_panel_api_schema_1:SchemaOneAdapter \
-        --seed packages/schema-1/src/span_panel_api_schema_1/reference_payloads/parent_child_tree.json
+        --seed tests/reference_payloads/parent_child_tree.json
 
 Exits non-zero if any check fails.
 """