feat(contracts): verify registry-fetch smoke check + update vendored $id to resolvable Pages URL#9
Merged
Merged
Conversation
- RED phase: test_registry_check.py covers all-200 success, one-404 failure, network-unavailable distinction, HTTP 5xx errors, and CLI exit-code paths for the not-yet-implemented cas_evals.registry_check - REQ-1.4.11
- GREEN phase: check_registry_urls() performs live GETs against the Pages registry, raising RegistryCheckError naming the first failing path on non-200, HTTPError, or network-unavailable conditions - main() is the python -m cas_evals.registry_check CLI entry point, checking index.json, v0.1/manifest.json, v0.1/common.schema.json, and v0.1/evaluation-result.schema.json by default; exits non-zero on failure - Follows reference_product.py's urllib pattern: capped response reads (MAX_RESPONSE_BYTES), typed exception-to-domain-error translation - REQ-1.4.11
…$id to resolvable Pages URL - Update vendored v0.1.0 common/evaluation-result schema $id values from the dead schemas.coding-autopilot.dev namespace to the resolvable Pages registry URL, matching the portfolio-wide convention from cas-contracts plan 32-01 - Recompute sha256 in provenance.json for the re-vendored files - Update verify_vendored_contract()'s two hardcoded $id equality checks to match the new value - Wire a new registry-smoke CI job invoking cas_evals.registry_check against the live Pages registry, isolated from the offline verify matrix so a transient network failure doesn't block unrelated merges - Regenerate releases/v0.2.0/manifest.json: its provenanceDigest is a sha256 of provenance.json, which changed as a direct consequence of the re-vendor above (Rule 3 auto-fix: this is a blocking downstream effect of the provenance.json edit required by this task, not new scope) - REQ-1.4.11
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
1 task
OgeonX-Ai
added a commit
that referenced
this pull request
Jul 8, 2026
* docs(36-02): append freshness footer to README * docs(36-03): add docs/wiki tree (Home, Architecture, Operations, Decisions) - Architecture.md documents the evidence gate flow (offline + live reference-product modes) and the in-progress registry-fetch smoke check (PR #9, companion to cas-contracts PR #18) - Operations.md verified CLI/gate/CI commands - Decisions.md indexes phase history and open PRs --------- Co-authored-by: Kim Harjamäki <kim.harjamaki@prosimo.fi>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wires cas-evals as the consumer-verification repo for REQ-1.4.11 (schema registry publishing). Two additive changes:
$idexpectations inverify_vendored_contract()and the re-vendoredvendor/cas-contracts/v0.1.0/*.schema.jsoncopies (+ recomputedprovenance.jsonsha256) to the new resolvable Pages registry URL convention (https://coding-autopilot-system.github.io/cas-contracts/registry/...), matching cas-contracts#18.cas_evals.registry_checkmodule (stdliburllib, no new deps) + unit tests + a dedicatedregistry-smokeCI job that performs live HTTP GETs against the Pages registry and asserts HTTP 200, proving the registry actually resolves — something the existing offline vendored-contract check cannot prove by design.Link to cas-contracts#18
This PR is a companion to Coding-Autopilot-System/cas-contracts#18 (
fix/registry-resolvable-id), which rewrites all 22 schema$idvalues in cas-contracts from the deadschemas.coding-autopilot.devnamespace to the live Pages registry URL.The vendored
$idupdate in this PR is independent of whether #18 has merged — the vendored copy undervendor/cas-contracts/v0.1.0/is a local, deliberately re-vendored snapshot edited to match the portfolio-wide$idconvention; it does not fetch from cas-contracts at build or test time.The new
registry-smokeCI job also does not depend on #18 merging: it checks paths (index.json,v0.1/manifest.json,v0.1/common.schema.json,v0.1/evaluation-result.schema.json) that already resolve 200 today via the existingpages.ymlpublisher from prior phases. Merging #18 only changes the$idfield text embedded inside those already-resolving files — it does not change which URLs exist. Verified locally:python -m cas_evals.registry_checkexits 0 with all four paths returning 200 against the live registry today, pre-merge of #18.Deferred post-merge verification (cannot be completed until cas-contracts#18 merges)
python -m cas_evals.registry_checkafter #18 merges and confirms continued 200s (expected no-op in observed behavior — paths already resolve, only the$idfield text inside those files changes).https://coding-autopilot-system.github.io/cas-contracts/registry/v0.1/common.schema.jsondirectly and confirm its$idfield now reads the new github.io value (proves the merged PR's content actually deployed, not just that the path still 200s).verify_vendored_contract()'s expected$idand the live-fetched schema's$idagree post-merge (both were updated to the same convention in this milestone, so they should match once #18's content is live).What changed
src/cas_evals/registry_check.py: new module —check_registry_urls(base_url, paths, timeout_seconds=10)andmain(argv)CLI entry point (python -m cas_evals.registry_check), followingreference_product.py's urllib pattern (capped reads viaMAX_RESPONSE_BYTES, typed exception translation toRegistryCheckError).tests/test_registry_check.py: unit tests covering all-200, one-404 (HTTPError), network-unavailable (URLError), and CLI exit-code paths (mockingurlopen).src/cas_evals/contracts.py:verify_vendored_contract()'s two hardcoded$idequality checks updated to the new Pages registry URL.vendor/cas-contracts/v0.1.0/{common,evaluation-result}.schema.json:$idfield rewritten to the new convention; this changes the file bytes and thus the sha256.vendor/cas-contracts/v0.1.0/provenance.json:sha256fields recomputed to match the re-vendored files.releases/v0.2.0/manifest.json:provenanceDigestregenerated — it is a sha256 ofprovenance.json, which changed as a direct, mechanical consequence of the re-vendor above (not new scope; downstream digest consistency required by the existing release-manifest invariant)..github/workflows/ci.yml: newregistry-smokejob, isolated from the offlineverifymatrix (separate job, not a blocking step) so a transient network failure doesn't block unrelated merges. SHA-pinned actions,permissions: contents: read,timeout-minutes: 5, matching the repo's existing CI conventions.Test plan
python -m unittest discover -s tests -v— 37/37 pass, including the 6 newtest_registry_checktests and the existingtest_contracts.pysuite (now expecting the new$id).python -m cas_evals.registry_check— exits 0, all four paths return 200 against the live Pages registry today.Not merging this PR per environment constraint — PR-only, leaving open for review.