From 9d390e9ef1b88aafa2293648566dbe6b616115a7 Mon Sep 17 00:00:00 2001 From: Felix Weinberger Date: Wed, 5 Aug 2026 14:45:38 +0000 Subject: [PATCH] Freeze conformance requirements per spec revision, run each at its wire There is no way to answer "which scenarios must my implementation pass to conform to spec revision X". The suite accumulates scenarios continuously, so what an implementation is measured against keeps growing after the revision shipped, and it can drop below 100% while standing still. --spec-version does not answer it: it filters on which revision a scenario targets, not on when the scenario became a requirement, so a scenario merged after a release is indistinguishable from one that existed at it. Add requirements/.yaml naming the scenarios a revision requires, and --requirements on the client, server, list and tier-check commands to run exactly that set. Passing no flag keeps today's behaviour of running everything, so the suite stays the place completeness lives. The revision also fixes the wire version its scenarios speak, which is the substance rather than a label: the dated revisions through 2025-11-25 use the stateful initialize handshake and 2026-07-28 is stateless with per-request _meta, and scenarios emit different checks under each. A scenario belonging to two revisions therefore has to run once under each, and one run does not cover the other. tier-check takes several revisions at once and every one of them must pass for Tier 1. Ship two sets. 2026-07-28 is a true snapshot, generated from 0.2.0-alpha.10 which was published the day before that revision shipped. 2025-11-25 could not be frozen the same way and its header says so: the release current on that date had no spec-version concept at all, so it is a reconstruction from the same referee rather than a contemporaneous record. Scenarios a revision runs without scoring go under not_scored with the leg that runs them and why they do not count: extensions are optional per SEP-1730, and a scenario added after the revision shipped is one no implementation could have been passing. Both run and both appear in the report, so a failing extension stays visible without moving a pass rate or the exit code. Scenarios pending in the source release appear nowhere, since the suite does not enforce them either. Requirement sets cover the client and server roles only. An authorization section is rejected, because the specification puts authorization-server implementation beyond its own scope. A run targets one revision, having one exit code and one baseline; tier-check aggregates. A child run that cannot happen at all, such as a set naming a scenario this build no longer has, is reported as not measured rather than reconciled into a zero score against the implementation. --- .claude/skills/mcp-sdk-tier-audit/README.md | 6 +- .claude/skills/mcp-sdk-tier-audit/SKILL.md | 16 +- README.md | 128 ++++++- package.json | 3 +- requirements/2025-11-25.yaml | 113 ++++++ requirements/2026-07-28.yaml | 130 +++++++ src/index.ts | 278 ++++++++++++-- src/requirements.test.ts | 130 +++++++ src/requirements.ts | 211 +++++++++++ .../checks/test-conformance-results.ts | 188 ++++++++-- src/tier-check/index.ts | 77 +++- src/tier-check/output.ts | 348 +++++++++++++----- src/tier-check/types.ts | 28 ++ 13 files changed, 1472 insertions(+), 184 deletions(-) create mode 100644 requirements/2025-11-25.yaml create mode 100644 requirements/2026-07-28.yaml create mode 100644 src/requirements.test.ts create mode 100644 src/requirements.ts diff --git a/.claude/skills/mcp-sdk-tier-audit/README.md b/.claude/skills/mcp-sdk-tier-audit/README.md index 77fc8cf8..afb003f8 100644 --- a/.claude/skills/mcp-sdk-tier-audit/README.md +++ b/.claude/skills/mcp-sdk-tier-audit/README.md @@ -95,11 +95,13 @@ The skill lives in `.claude/skills/` in this repo, so if you open [Claude Code]( 3. Run the skill: ``` -/mcp-sdk-tier-audit [client-cmd] +/mcp-sdk-tier-audit [client-cmd] [--requirements ] ``` Pass the client command as the third argument to include client conformance testing. If omitted, client conformance is skipped and noted as a gap in the report. +**Pass `--requirements` with every revision the SDK claims**, comma-separated. Each revision's scenarios run at that revision's own wire version, and all of them must pass for Tier 1: the dated revisions through `2025-11-25` use the stateful initialize handshake while `2026-07-28` is stateless, so a scenario belonging to both has to work on both and one run does not cover the other. It also means a scenario added to the suite after a revision shipped cannot fail an SDK that had no opportunity to adopt it. Without the flag, scoring uses the suite as it stands today, which is not a tier claim. See [Conformance Requirements](../../../README.md#conformance-requirements), and run `conformance list --requirements 2025-11-25,2026-07-28` to see both sets. + **TypeScript SDK example:** ```bash @@ -107,7 +109,7 @@ Pass the client command as the third argument to include client conformance test cd ~/src/mcp/typescript-sdk && npm run test:conformance:server:run # Terminal 2: run the audit (from the conformance repo) -/mcp-sdk-tier-audit ~/src/mcp/typescript-sdk http://localhost:3000/mcp "npx tsx ~/src/mcp/typescript-sdk/test/conformance/src/everythingClient.ts" +/mcp-sdk-tier-audit http://localhost:3000/mcp "node --import tsx /test/conformance/src/everythingClient.ts" --requirements 2025-11-25,2026-07-28 ``` **Python SDK example:** diff --git a/.claude/skills/mcp-sdk-tier-audit/SKILL.md b/.claude/skills/mcp-sdk-tier-audit/SKILL.md index 5da0fbde..6cd0fc14 100644 --- a/.claude/skills/mcp-sdk-tier-audit/SKILL.md +++ b/.claude/skills/mcp-sdk-tier-audit/SKILL.md @@ -5,7 +5,7 @@ description: >- Produces tier classification (1/2/3) with evidence table, gap list, and remediation guide. Works for any official MCP SDK (TypeScript, Python, Go, C#, Java, Kotlin, PHP, Swift, Rust, Ruby). -argument-hint: ' [client-cmd] [--branch ]' +argument-hint: ' [client-cmd] [--requirements ] [--branch ]' --- # MCP SDK Tier Audit @@ -43,6 +43,7 @@ Extract from the user's input: - **local-path**: absolute path to the SDK checkout (e.g. `~/src/mcp/typescript-sdk`) - **conformance-server-url**: URL where the SDK's everything server is already running (e.g. `http://localhost:3000/mcp`) - **client-cmd** (optional): command to run the SDK's conformance client (e.g. `npx tsx test/conformance/src/everythingClient.ts`). If not provided, client conformance tests are skipped and noted as a gap in the report. +- **requirements** (optional): spec revisions to score against, comma-separated, e.g. `--requirements 2025-11-25,2026-07-28`. Each revision's scenarios run at that revision's wire version, and every listed revision must pass for Tier 1. Scores the SDK against exactly the scenarios that revision required when it shipped, rather than everything the suite carries today. Prefer it whenever the question is "does this SDK conform to revision X". Without it, scoring uses today's suite, which can fail an SDK for a scenario added after it shipped. Run `conformance list --requirements ` to see the set. - **branch** (optional): Git branch to check on GitHub (e.g. `--branch fweinberger/v1x-governance-docs`). If not provided, derive from the local checkout's current branch: `cd && git rev-parse --abbrev-ref HEAD`. This is passed to the tier-check CLI so that policy signal file checks use the correct branch instead of the repo's default branch. The first two arguments are required. If either is missing, ask the user to provide it. @@ -61,11 +62,20 @@ The `tier-check` CLI handles all deterministic checks — server conformance, cl npm run --silent tier-check -- \ --repo \ --branch \ + --requirements \ --conformance-server-url \ --client-cmd '' \ --output json ``` +Omit `--requirements` only if the user did not name a revision. When it is set the +scorecard reports `requirements_revision`, both pass rates count exactly the +scenarios that revision requires, and anything run but not scored carries a +`notScoredReason` of `extension` or `added-after-release`. `requirements_revisions` +lists every revision scored, and each detail carries the `revision` it came from. Quote the revision +alongside any conformance number, and report the not-scored failures separately +rather than folding them into the score or omitting them. + If no client-cmd was detected, omit the `--client-cmd` flag (client conformance will be skipped). The `--branch` flag should always be included (derived from the local checkout if not explicitly provided). The CLI output includes server conformance pass rate, client conformance pass rate (with per-spec-version breakdown), issue triage compliance, P0 resolution times, label taxonomy, stable release status, policy signal files, and spec tracking gap. Parse the JSON output to feed into Step 4. @@ -117,8 +127,8 @@ Combine the deterministic scorecard (from the CLI) with the evaluation results ( ### Tier 1 requires ALL of: -- Server conformance test pass rate == 100% (date-versioned scenarios only; `draft` and `extension` are informational and not scored) -- Client conformance test pass rate == 100% (date-versioned scenarios only; `draft` and `extension` are informational and not scored) +- Server conformance test pass rate == 100% across every requirement set given to `--requirements` (each run at its own wire), otherwise of date-versioned scenarios only +- Client conformance test pass rate == 100%, on the same basis - Issue triage compliance >= 90% within 2 business days - All P0 bugs resolved within 7 days - Stable release >= 1.0.0 with no pre-release suffix diff --git a/README.md b/README.md index c320f481..6dc18d55 100644 --- a/README.md +++ b/README.md @@ -66,6 +66,7 @@ npx @modelcontextprotocol/conformance client --command "" --scen - `--suite` - Run a suite of tests in parallel: `all`, `core`, `extensions`, `backcompat`, `auth`, `metadata`, `draft` (scenarios targeting the in-progress draft spec), or `sep-835` - `--spec-version ` - Filter scenarios by spec version (e.g., `2025-11-25`, `2026-07-28`; `draft` is accepted as an alias for the current draft identifier). The draft version selects the latest dated release plus any draft-only scenarios. When omitted, the version is inferred from the scenario's spec applicability (draft-only scenarios run at the draft version, everything else at the latest dated release); an explicitly requested version outside a scenario's applicability window skips the scenario (exit 0) unless `--force` is passed - `--force` - Run a scenario even if it is not applicable at the requested `--spec-version` +- `--requirements ` - Run exactly what a spec revision requires, frozen at its release (see [Conformance Requirements](#conformance-requirements)) - `--expected-failures ` - Path to YAML baseline file of known failures (see [Expected Failures](#expected-failures)) - `--timeout` - Timeout in milliseconds (default: 30000) - `--verbose` - Show verbose output @@ -83,6 +84,7 @@ npx @modelcontextprotocol/conformance server --url [--scenario ] - `--url` - URL of the server to test - `--scenario ` - Test scenario to run (e.g., "server-initialize"). Runs all available scenarios by default - `--suite ` - Suite to run: "active" (default; excludes pending and draft-spec scenarios), "all", "draft" (scenarios targeting the in-progress draft spec), or "pending" +- `--requirements ` - Run exactly what a spec revision requires, frozen at its release (see [Conformance Requirements](#conformance-requirements)) - `--expected-failures ` - Path to YAML baseline file of known failures (see [Expected Failures](#expected-failures)) - `--verbose` - Show verbose output @@ -116,6 +118,73 @@ Scenarios that exchange no instrumented wire traffic (see issue #418) emit neither check. Like any other check, `wire-schema-valid` can be baselined via the expected-failures file. +## Conformance Requirements + +`--suite` and `--spec-version` describe the suite as it is today. Neither answers +"which scenarios did I need to pass to conform to the spec released on +2026-07-28", because the suite keeps growing: a scenario merged after a revision +ships still carries that revision's applicability tag, so it is +indistinguishable from one that existed at release. + +A requirement set answers that question. Each `requirements/.yaml` +names the scenarios a revision requires, for the two roles the specification +defines: an MCP server acting as an OAuth resource server, and an MCP client +acting as an OAuth client. It deliberately covers no authorization-server +scenarios, because the specification puts authorization server implementation +beyond its own scope, so those scenarios serve people deploying an authorization +server rather than implementations of MCP itself. + +**Scenarios run at their revision's wire version.** That is the point of a +per-revision set, not a label on it: the dated revisions through `2025-11-25` +use the stateful initialize handshake and `2026-07-28` is stateless with +per-request `_meta`, and a scenario emits different checks under each. A +scenario belonging to both revisions must therefore be run twice, once under +each set. Passing it on one wire says nothing about the other: + +```bash +# what does conforming to 2026-07-28 actually require? +npx @modelcontextprotocol/conformance list --requirements 2026-07-28 + +# run exactly that +npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp --requirements 2026-07-28 +``` + +`--requirements` replaces `--suite`, `--spec-version` and `--scenario`, since the +set already names every scenario that runs and the revision fixes the wire they +run at. Without it nothing changes: the default is still to run everything, which +is where completeness lives. + +`tier-check` takes several at once, and every one of them must pass for Tier 1: + +```bash +npx @modelcontextprotocol/conformance tier-check --repo \ + --conformance-server-url http://localhost:3000/mcp \ + --requirements 2025-11-25,2026-07-28 +``` + +Only scenarios a revision actually requires decide the exit code and the pass +rate. Anything run without being scored is reported separately and cannot fail +the run. + +Requirement sets are frozen, and `not_scored` holds what a revision runs and +reports without counting. Two reasons qualify, and the report names which +applies: + +| Reason | Meaning | +| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | +| `extension` | Optional by definition. SEP-1730: "Experimental features and protocol extensions (such as Tasks and MCP Apps) are not required for any tier." | +| `added-after-release` | The scenario did not exist when the revision shipped, so no implementation could have been passing it. | + +Both still run, so a failing extension stays visible in the report; neither moves +the pass rate. Promoting an entry into the required lists is a deliberate, +reviewable change, which is how the suite grows without retroactively failing +anyone. + +This is the project's contract and lives in this repository. It is the opposite +of an [expected-failures](#expected-failures) baseline, which lives in an +implementation's own repository and records what that implementation knows it +fails. A baselined failure is still a failure against a requirement set. + ## Expected Failures SDKs that don't yet pass all conformance tests can specify a baseline of known failures. This allows running conformance tests in CI without failing, while still catching regressions. @@ -326,26 +395,67 @@ Clones are cached under `.sdk-under-test/` and reused (fetched) on subsequent ru ## SDK Tier Assessment -The `tier-check` subcommand evaluates an MCP SDK repository against [SEP-1730](https://github.com/modelcontextprotocol/modelcontextprotocol/issues/1730) (the SDK Tiering System): +The `tier-check` subcommand evaluates an MCP SDK repository against [SEP-1730](https://github.com/modelcontextprotocol/modelcontextprotocol/issues/1730) (the SDK Tiering System). There are two ways to run it, and they answer different questions. + +### 1. The CLI, for the deterministic half + +Conformance pass rates, issue triage, P0 resolution, labels, releases, policy files. No AI, no judgment, reproducible. ```bash -# Without conformance tests (fastest) gh auth login -npm run --silent tier-check -- --repo modelcontextprotocol/typescript-sdk --skip-conformance -# With conformance tests (start the everything server first) -npm run --silent tier-check -- \ +# fastest: everything except conformance +npx @modelcontextprotocol/conformance tier-check \ + --repo modelcontextprotocol/typescript-sdk --skip-conformance + +# with conformance, scored against the revisions the SDK claims. +# start the SDK's everything server first, in its own terminal. +npx @modelcontextprotocol/conformance tier-check \ --repo modelcontextprotocol/typescript-sdk \ - --conformance-server-url http://localhost:3000/mcp + --conformance-server-url http://localhost:3000/mcp \ + --client-cmd '' \ + --requirements 2025-11-25,2026-07-28 ``` -For a full AI-assisted assessment with remediation guide, use Claude Code: +Omit `--client-cmd` and the client leg is skipped and reported as a gap. Omit +`--requirements` and scoring falls back to the suite as it stands today, which is +not what you want for a tier claim; see [Conformance Requirements](#conformance-requirements). + +### 2. The skill, for the whole assessment + +The CLI cannot judge documentation coverage, dependency policy or roadmap quality, and those decide the tier as much as conformance does. The [`mcp-sdk-tier-audit`](.claude/skills/mcp-sdk-tier-audit/README.md) skill runs the CLI, adds those evaluations, and writes a full report with a remediation plan. In Claude Code, from a checkout of this repo: ``` -/mcp-sdk-tier-audit +/mcp-sdk-tier-audit '' --requirements 2025-11-25,2026-07-28 ``` -See [`.claude/skills/mcp-sdk-tier-audit/README.md`](.claude/skills/mcp-sdk-tier-audit/README.md) for full documentation. +The server must already be running and stay up for the whole audit. Expect a few minutes. + +### Reading the result + +``` +Scored against 2025-11-25 and 2026-07-28, each run at its own wire version. + + Server 67/67 required scenarios (100%) + Client 50/50 required scenarios (100%) + + Not scored (8 run, 4 failing, no effect on tier): + ✗ auth/dpop (extension) + ✗ json-schema-2020-12-preservation (added-after-release) + +Tier 1 Blockers: + • triage + • p0_resolution +``` + +- **Required scenarios** are the only ones that move the number. `67/67` spans every revision listed: a scenario belonging to both runs once per revision, on that revision's wire, and both have to pass. +- **Not scored** ran and is reported so you can see it, but cannot fail the tier. `extension` means optional by definition; `added-after-release` means the scenario did not exist when that revision shipped. A failure here is information, not a blocker. +- **Not measured** is different from either, and means the run could not happen at all, e.g. a requirement set naming a scenario this build no longer has. Treat it as a broken invocation, never as an SDK failure. +- **Tier 1 blockers** lists every requirement short of Tier 1. Conformance absent from that list means the SDK met every requirement each listed revision imposes. + +The exit code follows the same rule: it reflects required scenarios only, so an implementation that meets a revision's requirements exits 0 even with failing extensions. + +In `--output json`, `passed` / `failed` / `total` describe the scored set and so always agree with `pass_rate`; anything run without being scored is counted separately under `not_scored`. ## Architecture diff --git a/package.json b/package.json index e59d9f04..f6243540 100644 --- a/package.json +++ b/package.json @@ -27,7 +27,8 @@ "prepare": "npm run build" }, "files": [ - "dist" + "dist", + "requirements" ], "bin": { "conformance": "dist/index.js" diff --git a/requirements/2025-11-25.yaml b/requirements/2025-11-25.yaml new file mode 100644 index 00000000..addf4271 --- /dev/null +++ b/requirements/2025-11-25.yaml @@ -0,0 +1,113 @@ +# Conformance requirements for the 2025-11-25 specification revision. +# +# This file is the canonical answer to "which scenarios must my implementation pass +# to conform to 2025-11-25". +# +# Anchor, and READ THIS BEFORE TRUSTING IT AS A SNAPSHOT: unlike the 2026-07-28 set, +# this one could not be frozen at its own ship date. The release current on 2025-11-25 +# was 0.1.7 (published 2025-11-20), which had no --spec-version flag and no concept of +# which revision a scenario belonged to, so there is nothing to snapshot. This set is +# instead derived from @modelcontextprotocol/conformance@0.2.0-alpha.10 filtered to +# 2025-11-25, and therefore contains scenarios written after 2025-11-25 shipped. It is +# frozen from here on, but it is a reconstruction rather than a contemporaneous record. +# +# `server` and `client` are named for the subcommand that runs them, and list what +# conformance to this revision requires. Scenarios run at THIS revision's wire +# version: the dated revisions through 2025-11-25 use the stateful initialize +# handshake and 2026-07-28 is stateless with per-request _meta, so a scenario that +# applies to both must be run once under each and one run does not cover the other. +# +# There is deliberately no authorization-server section: the MCP specification puts +# authorization-server implementation beyond its own scope, so those scenarios serve +# people deploying an authorization server, not implementations of MCP itself. +# +# `not_scored` is run and reported but never counts toward a pass rate; each entry +# says why. `extension` is optional by definition (SEP-1730: "Experimental features +# and protocol extensions ... are not required for any tier"). `added-after-release` +# did not exist when this revision shipped, so no implementation could have been +# passing it. Promoting an entry into the lists above is a deliberate, reviewable +# change. +# +# Scenarios that were PENDING in the source release appear nowhere: SEP-1730 scores +# "applicable required tests" only, and a pending scenario is one the suite itself +# does not enforce. Fourteen were pending at 0.2.0-alpha.10, including all ten +# io.modelcontextprotocol/tasks extension scenarios, json-schema-2020-12, +# server-sse-polling, http-header-validation and http-custom-header-server-validation. + +server: + - server-initialize + - logging-set-level + - ping + - completion-complete + - tools-list + - tools-call-simple-text + - tools-call-image + - tools-call-audio + - tools-call-embedded-resource + - tools-call-mixed-content + - tools-call-with-logging + - tools-call-error + - tools-call-with-progress + - tools-call-sampling + - tools-call-elicitation + - elicitation-sep1034-defaults + - server-sse-multiple-streams + - elicitation-sep1330-enums + - resources-list + - resources-read-text + - resources-read-binary + - resources-templates-read + - resources-subscribe + - resources-unsubscribe + - prompts-list + - prompts-get-simple + - prompts-get-with-args + - prompts-get-embedded-resource + - prompts-get-with-image + - dns-rebinding-protection + +client: + - initialize + - tools_call + - elicitation-sep1034-client-defaults + - sse-retry + - auth/metadata-default + - auth/metadata-var1 + - auth/metadata-var2 + - auth/metadata-var3 + - auth/basic-cimd + - auth/scope-from-www-authenticate + - auth/scope-from-scopes-supported + - auth/scope-omitted-when-undefined + - auth/scope-step-up + - auth/scope-retry-limit + - auth/token-endpoint-auth-basic + - auth/token-endpoint-auth-post + - auth/token-endpoint-auth-none + - auth/pre-registration + +not_scored: + - scenario: auth/client-credentials-jwt + leg: client + reason: extension + - scenario: auth/client-credentials-basic + leg: client + reason: extension + - scenario: auth/enterprise-managed-authorization + leg: client + reason: extension + - scenario: auth/dpop + leg: client + reason: extension + - scenario: auth/dpop-nonce + leg: client + reason: extension + - scenario: auth/wif-jwt-bearer + leg: client + reason: extension + - scenario: server-session-lifecycle + leg: server + reason: added-after-release + - scenario: json-schema-2020-12-preservation + leg: client + reason: added-after-release diff --git a/requirements/2026-07-28.yaml b/requirements/2026-07-28.yaml new file mode 100644 index 00000000..f893673d --- /dev/null +++ b/requirements/2026-07-28.yaml @@ -0,0 +1,130 @@ +# Conformance requirements for the 2026-07-28 specification revision. +# +# This file is the canonical answer to "which scenarios must my implementation pass +# to conform to 2026-07-28". It is FROZEN: the lists below were fixed when the +# revision shipped and must not be edited afterwards. An implementation is measured +# against the suite as it stood when it was expected to conform, not against whatever +# the suite has accumulated since. +# +# Anchor: @modelcontextprotocol/conformance@0.2.0-alpha.10, published 2026-07-27, the +# release current when this revision shipped. That release could express spec-version +# applicability, so this set is a faithful snapshot of what was required at ship. +# +# `server` and `client` are named for the subcommand that runs them, and list what +# conformance to this revision requires. Scenarios run at THIS revision's wire +# version: the dated revisions through 2025-11-25 use the stateful initialize +# handshake and 2026-07-28 is stateless with per-request _meta, so a scenario that +# applies to both must be run once under each and one run does not cover the other. +# +# There is deliberately no authorization-server section: the MCP specification puts +# authorization-server implementation beyond its own scope, so those scenarios serve +# people deploying an authorization server, not implementations of MCP itself. +# +# `not_scored` is run and reported but never counts toward a pass rate; each entry +# says why. `extension` is optional by definition (SEP-1730: "Experimental features +# and protocol extensions ... are not required for any tier"). `added-after-release` +# did not exist when this revision shipped, so no implementation could have been +# passing it. Promoting an entry into the lists above is a deliberate, reviewable +# change. +# +# Scenarios that were PENDING in the source release appear nowhere: SEP-1730 scores +# "applicable required tests" only, and a pending scenario is one the suite itself +# does not enforce. Fourteen were pending at 0.2.0-alpha.10, including all ten +# io.modelcontextprotocol/tasks extension scenarios, json-schema-2020-12, +# server-sse-polling, http-header-validation and http-custom-header-server-validation. + +server: + - server-stateless + - completion-complete + - tools-list + - tools-call-simple-text + - tools-call-image + - tools-call-audio + - tools-call-embedded-resource + - tools-call-mixed-content + - tools-call-error + - tools-call-with-progress + - server-sse-multiple-streams + - resources-list + - resources-read-text + - resources-read-binary + - resources-templates-read + - sep-2164-resource-not-found + - prompts-list + - prompts-get-simple + - prompts-get-with-args + - prompts-get-embedded-resource + - prompts-get-with-image + - dns-rebinding-protection + - caching + - input-required-result-basic-elicitation + - input-required-result-basic-sampling + - input-required-result-basic-list-roots + - input-required-result-request-state + - input-required-result-multiple-input-requests + - input-required-result-multi-round + - input-required-result-missing-input-response + - input-required-result-non-tool-request + - input-required-result-result-type + - input-required-result-unsupported-methods + - input-required-result-tampered-state + - input-required-result-capability-check + - input-required-result-ignore-extra-params + - input-required-result-validate-input + +client: + - tools_call + - request-metadata + - auth/metadata-default + - auth/metadata-var1 + - auth/metadata-var2 + - auth/metadata-var3 + - auth/basic-cimd + - auth/scope-from-www-authenticate + - auth/scope-from-scopes-supported + - auth/scope-omitted-when-undefined + - auth/scope-step-up + - auth/scope-retry-limit + - auth/token-endpoint-auth-basic + - auth/token-endpoint-auth-post + - auth/token-endpoint-auth-none + - auth/pre-registration + - auth/resource-mismatch + - auth/offline-access-scope + - auth/offline-access-not-supported + - auth/authorization-server-migration + - auth/iss-supported + - auth/iss-not-advertised + - auth/iss-supported-missing + - auth/iss-wrong-issuer + - auth/iss-unexpected + - auth/iss-normalized + - auth/metadata-issuer-mismatch + - sep-2322-client-request-state + - http-standard-headers + - http-custom-headers + - http-invalid-tool-headers + - json-schema-ref-no-deref + +not_scored: + - scenario: auth/client-credentials-jwt + leg: client + reason: extension + - scenario: auth/client-credentials-basic + leg: client + reason: extension + - scenario: auth/enterprise-managed-authorization + leg: client + reason: extension + - scenario: auth/dpop + leg: client + reason: extension + - scenario: auth/dpop-nonce + leg: client + reason: extension + - scenario: auth/wif-jwt-bearer + leg: client + reason: extension + - scenario: json-schema-2020-12-preservation + leg: client + reason: added-after-release diff --git a/src/index.ts b/src/index.ts index 788e4700..9c99753e 100644 --- a/src/index.ts +++ b/src/index.ts @@ -44,6 +44,15 @@ import { } from './schemas'; import type { AuthorizationServerOptions } from './schemas'; import { withWireRecorder } from './validation/wire-schema'; +import { + filterScenariosByRequirements, + Leg, + listRequirementRevisions, + loadRequirements, + notScoredScenarios, + RequirementSet, + scoredScenarios +} from './requirements'; import { loadExpectedFailures, evaluateBaseline, @@ -59,6 +68,123 @@ import packageJson from '../package.json'; // The `client` command tests Scenario objects (which test clients), // and the `server` command tests ClientScenario objects (which test servers). // This matches the inverted naming in scenarios/index.ts. +/** + * Resolve `--requirements`. A requirement set replaces suite and spec-version + * selection: it already names exactly the scenarios that revision requires. + */ +function resolveRequirements( + revision: string | undefined, + conflicts: { specVersion?: string; suiteFromCli?: boolean; scenario?: string } +): RequirementSet | undefined { + if (revision === undefined) return undefined; + // An explicitly passed empty value is a script whose variable did not expand. + // Ignoring it would silently score against the whole suite, which is the + // failure this flag exists to prevent. + if (revision === '') { + console.error( + '--requirements needs a revision, such as 2026-07-28. Run `conformance list` to see which are available.' + ); + process.exit(1); + } + const combined = conflicts.specVersion + ? '--spec-version' + : conflicts.suiteFromCli + ? '--suite' + : conflicts.scenario + ? '--scenario' + : undefined; + if (combined) { + console.error( + `--requirements cannot be combined with ${combined}: a requirement set already fixes which scenarios run.` + ); + process.exit(1); + } + try { + return loadRequirements(revision); + } catch (error) { + console.error(error instanceof Error ? error.message : String(error)); + process.exit(1); + } +} + +/** + * Exit status under a requirement set. Scenarios the set runs without scoring + * (extensions, post-release additions) are reported but must not fail the run: + * otherwise the documented command red-flags an implementation that meets every + * requirement the revision actually imposes. + */ +function requirementsExitCode( + requirements: RequirementSet, + leg: Leg, + results: { scenario: string; checks: ConformanceCheck[] }[] +): number { + const scored = new Set(scoredScenarios(requirements, leg)); + const unscored = results.filter((r) => !scored.has(r.scenario)); + if (unscored.length > 0) { + const failing = unscored.filter((r) => + r.checks.some((c) => c.status === 'FAILURE') + ); + console.log( + `\nNot scored for ${requirements.revision}: ${unscored.length} scenario(s) run, ${failing.length} failing. These do not affect conformance.` + ); + for (const r of unscored) { + const why = notScoredScenarios(requirements, leg).find( + (e) => e.scenario === r.scenario + ); + const failed = r.checks.some((c) => c.status === 'FAILURE'); + console.log( + ` ${failed ? '\u2717' : '\u2713'} ${r.scenario} (${why?.reason ?? 'not scored'})` + ); + } + } + const scoredFailed = results + .filter((r) => scored.has(r.scenario)) + .some((r) => r.checks.some((c) => c.status === 'FAILURE')); + return scoredFailed ? 1 : 0; +} + +/** Print one revision's requirement set. `list` is display-only, so it can show several. */ +function listOneRequirementSet(revision: string, specVersion?: string): void { + const requirements = resolveRequirements(revision, { specVersion })!; + const legs: [string, string[]][] = [ + ['Server scenarios (test against a server)', requirements.server], + ['Client scenarios (test against a client)', requirements.client] + ]; + const total = legs.reduce((n, [, names]) => n + names.length, 0); + console.log( + `Required for ${requirements.revision} (${total} scenarios, frozen; run at the ${requirements.revision} wire):\n` + ); + for (const [title, names] of legs) { + if (names.length === 0) continue; + console.log(`${title}:`); + names.forEach((name) => console.log(` - ${name}`)); + console.log(''); + } + if (requirements.notScored.length > 0) { + console.log('Run and reported, but never scored:'); + for (const reason of ['extension', 'added-after-release'] as const) { + const entries = requirements.notScored.filter((e) => e.reason === reason); + if (entries.length === 0) continue; + console.log(` ${reason} (${entries.length}):`); + entries.forEach((e) => console.log(` - ${e.scenario} [${e.leg}]`)); + } + console.log(''); + } +} + +function requiredScenariosOrExit( + allScenarios: string[], + requirements: RequirementSet, + command: Leg +): string[] { + try { + return filterScenariosByRequirements(allScenarios, requirements, command); + } catch (error) { + console.error(error instanceof Error ? error.message : String(error)); + process.exit(1); + } +} + function filterScenariosBySpecVersion( allScenarios: string[], version: SpecVersion, @@ -109,18 +235,36 @@ program '--force', 'Run a scenario even if it is not applicable at the requested --spec-version' ) + .option( + '--requirements ', + 'Run exactly the scenarios a spec revision requires, frozen at its release (e.g. 2026-07-28). Replaces --suite and --spec-version' + ) .option('--verbose', 'Show verbose output') - .action(async (options) => { + .action(async (options, cmd) => { try { const timeout = parseInt(options.timeout, 10); const verbose = options.verbose ?? false; const outputDir = options.outputDir; - const specVersionFilter = options.specVersion - ? resolveSpecVersion(options.specVersion) - : undefined; + const requirements = resolveRequirements(options.requirements, { + specVersion: options.specVersion, + suiteFromCli: cmd.getOptionValueSource('suite') === 'cli', + scenario: options.scenario + }); + // A requirement set decides two different things, and only the first is + // its scenario list. The revision it names is also the wire version those + // scenarios must speak: 2025-11-25 and 2026-07-28 are different protocols + // (stateful initialize handshake vs stateless per-request _meta), and a + // scenario emits different checks under each. Leaving this unset ran the + // right set against LATEST_SPEC_VERSION, which is a different wire than + // the one named on the flag. + const specVersionFilter = requirements + ? resolveSpecVersion(requirements.revision) + : options.specVersion + ? resolveSpecVersion(options.specVersion) + : undefined; // Handle suite mode - if (options.suite) { + if (options.suite || options.requirements !== undefined) { if (!options.command) { console.error('--command is required when using --suite'); process.exit(1); @@ -138,23 +282,37 @@ program listAuthScenarios().filter((name) => name.startsWith('auth/scope-')) }; - const suiteName = options.suite.toLowerCase(); - if (!suites[suiteName]) { - console.error(`Unknown suite: ${suiteName}`); - console.error(`Available suites: ${Object.keys(suites).join(', ')}`); - process.exit(1); - } - - let scenarios = suites[suiteName](); - if (specVersionFilter) { - scenarios = filterScenariosBySpecVersion( - scenarios, - specVersionFilter, + let scenarios: string[]; + let selection: string; + if (requirements) { + scenarios = requiredScenariosOrExit( + listScenarios(), + requirements, 'client' ); + selection = `requirements ${requirements.revision}`; + } else { + const suiteName = options.suite.toLowerCase(); + if (!suites[suiteName]) { + console.error(`Unknown suite: ${suiteName}`); + console.error( + `Available suites: ${Object.keys(suites).join(', ')}` + ); + process.exit(1); + } + + scenarios = suites[suiteName](); + if (specVersionFilter) { + scenarios = filterScenariosBySpecVersion( + scenarios, + specVersionFilter, + 'client' + ); + } + selection = `${suiteName} suite`; } console.log( - `Running ${suiteName} suite (${scenarios.length} scenarios) in parallel...\n` + `Running ${selection} (${scenarios.length} scenarios) in parallel...\n` ); const results = await Promise.all( @@ -169,7 +327,9 @@ program timeout, outputDir, specVersionFilter, - options.force ?? false + // a requirement set decides membership, so its choice outranks a + // scenario's own applicability window at the pinned revision + options.force || Boolean(requirements) ) ); return { @@ -261,7 +421,17 @@ program process.exit(baselineResult.exitCode); } - process.exit(totalFailed > 0 || totalWarnings > 0 ? 1 : 0); + process.exit( + requirements + ? requirementsExitCode( + requirements, + 'client', + results.map((r) => ({ scenario: r.scenario, checks: r.checks })) + ) + : totalFailed > 0 || totalWarnings > 0 + ? 1 + : 0 + ); } // Require either --scenario or --suite @@ -369,17 +539,35 @@ program '--force', 'Run a scenario even if it is not applicable at the requested --spec-version' ) + .option( + '--requirements ', + 'Run exactly the scenarios a spec revision requires, frozen at its release (e.g. 2026-07-28). Replaces --suite and --spec-version' + ) .option('--verbose', 'Show verbose output (JSON instead of pretty print)') - .action(async (options) => { + .action(async (options, cmd) => { try { // Validate options with Zod const validated = ServerOptionsSchema.parse(options); const verbose = options.verbose ?? false; const outputDir = options.outputDir; - const specVersionFilter = options.specVersion - ? resolveSpecVersion(options.specVersion) - : undefined; + const requirements = resolveRequirements(options.requirements, { + specVersion: options.specVersion, + suiteFromCli: cmd.getOptionValueSource('suite') === 'cli', + scenario: options.scenario + }); + // A requirement set decides two different things, and only the first is + // its scenario list. The revision it names is also the wire version those + // scenarios must speak: 2025-11-25 and 2026-07-28 are different protocols + // (stateful initialize handshake vs stateless per-request _meta), and a + // scenario emits different checks under each. Leaving this unset ran the + // right set against LATEST_SPEC_VERSION, which is a different wire than + // the one named on the flag. + const specVersionFilter = requirements + ? resolveSpecVersion(requirements.revision) + : options.specVersion + ? resolveSpecVersion(options.specVersion) + : undefined; // If a single scenario is specified, run just that one if (validated.scenario) { @@ -421,8 +609,16 @@ program // Run scenarios based on suite const suite = options.suite?.toLowerCase() || 'active'; let scenarios: string[]; + let selection = `${suite} suite`; - if (suite === 'all') { + if (requirements) { + scenarios = requiredScenariosOrExit( + listClientScenarios(), + requirements, + 'server' + ); + selection = `requirements ${requirements.revision}`; + } else if (suite === 'all') { scenarios = listClientScenarios(); } else if (suite === 'active' || suite === 'core') { // 'core' is an alias for 'active' - tier 1 requirements @@ -439,7 +635,7 @@ program process.exit(1); } - if (specVersionFilter) { + if (specVersionFilter && !requirements) { scenarios = filterScenariosBySpecVersion( scenarios, specVersionFilter, @@ -448,7 +644,7 @@ program } console.log( - `Running ${suite} suite (${scenarios.length} scenarios) against ${validated.url}\n` + `Running ${selection} (${scenarios.length} scenarios) against ${validated.url}\n` ); const allResults: { scenario: string; checks: ConformanceCheck[] }[] = @@ -464,7 +660,10 @@ program validated.url, scenarioName, outputDir, - specVersionFilter + specVersionFilter, + // a requirement set decides membership, so its choice outranks a + // scenario's own applicability window at the pinned revision + Boolean(requirements) ) ); allResults.push({ scenario: scenarioName, checks: result.checks }); @@ -502,7 +701,13 @@ program process.exit(baselineResult.exitCode); } - process.exit(totalFailed > 0 ? 1 : 0); + process.exit( + requirements + ? requirementsExitCode(requirements, 'server', allResults) + : totalFailed > 0 + ? 1 + : 0 + ); } } catch (error) { if (error instanceof ZodError) { @@ -700,7 +905,9 @@ program.addCommand(createTraceabilityCommand()); // List scenarios command program .command('list') - .description('List available test scenarios') + .description( + `List available test scenarios. Requirement sets available: ${listRequirementRevisions().join(', ') || 'none'}` + ) .option('--client', 'List client scenarios') .option('--server', 'List server scenarios') .option('--authorization', 'List authorization server scenarios') @@ -708,11 +915,22 @@ program '--spec-version ', 'Filter scenarios by spec version (cumulative for date versions)' ) + .option( + '--requirements ', + 'List exactly what a spec revision requires, frozen at its release' + ) .action((options) => { const specVersionFilter = options.specVersion ? resolveSpecVersion(options.specVersion) : undefined; + if (options.requirements !== undefined) { + for (const rev of String(options.requirements).split(',')) { + listOneRequirementSet(rev.trim(), options.specVersion); + } + return; + } + if ( options.server || (!options.client && !options.server && !options.authorization) diff --git a/src/requirements.test.ts b/src/requirements.test.ts new file mode 100644 index 00000000..5ba01e64 --- /dev/null +++ b/src/requirements.test.ts @@ -0,0 +1,130 @@ +import { describe, expect, it } from 'vitest'; +import { + filterScenariosByRequirements, + listRequirementRevisions, + loadRequirements, + notScoredScenarios, + scenariosToRun, + scoredScenarios +} from './requirements'; +import { listClientScenarios, listScenarios } from './scenarios'; + +const revisions = listRequirementRevisions(); + +describe('requirement sets', () => { + it('ships at least one revision', () => { + expect(revisions.length).toBeGreaterThan(0); + }); + + describe.each(revisions)('%s', (revision) => { + const requirements = loadRequirements(revision); + + // A requirement set is frozen, so a name it asks for that this build does + // not have means a scenario was renamed or removed out from under a shipped + // revision. That silently shrinks what conformance means, so it fails here. + it.each([ + ['server', requirements.server, listClientScenarios()], + ['client', requirements.client, listScenarios()] + ])( + 'every %s scenario exists in this build', + (_leg, required, available) => { + expect(required.filter((name) => !available.includes(name))).toEqual( + [] + ); + } + ); + + it('lists no scenario twice within a leg', () => { + for (const leg of [requirements.server, requirements.client]) { + expect(new Set(leg).size).toBe(leg.length); + } + }); + + it('keeps not-scored entries out of the required lists', () => { + const required = new Set([ + ...requirements.server, + ...requirements.client + ]); + const alsoRequired = requirements.notScored + .map((e) => e.scenario) + .filter((name) => required.has(name)); + expect(alsoRequired).toEqual([]); + }); + + it('runs the not-scored entries alongside the required ones', () => { + for (const leg of ['server', 'client'] as const) { + const run = scenariosToRun(requirements, leg); + const scored = scoredScenarios(requirements, leg); + const extra = notScoredScenarios(requirements, leg).map( + (e) => e.scenario + ); + expect(run).toEqual([...scored, ...extra]); + expect(extra.every((name) => !scored.includes(name))).toBe(true); + } + }); + }); +}); + +describe('loadRequirements', () => { + it('rejects anything that is not a revision date', () => { + expect(() => loadRequirements('../../etc/passwd')).toThrow( + /Invalid requirements revision/ + ); + expect(() => loadRequirements('latest')).toThrow( + /Invalid requirements revision/ + ); + }); + + it('rejects a date that is not a spec revision, since it also names the wire', () => { + expect(() => loadRequirements('1999-01-01')).toThrow( + /Unknown spec revision: 1999-01-01/ + ); + }); + + it('names the available revisions when a known one has no set', () => { + expect(() => loadRequirements('2025-03-26')).toThrow( + /No requirement set for 2025-03-26/ + ); + }); +}); + +describe('filterScenariosByRequirements', () => { + const requirements = loadRequirements(revisions[0]); + + it('returns what the revision runs, not the suite it was given', () => { + const selected = filterScenariosByRequirements( + listClientScenarios(), + requirements, + 'server' + ); + expect(selected).toEqual(scenariosToRun(requirements, 'server')); + }); + + it('runs the not-scored entries but keeps them out of the scored set', () => { + const selected = filterScenariosByRequirements( + listClientScenarios(), + requirements, + 'server' + ); + const extensions = notScoredScenarios(requirements, 'server').map( + (e) => e.scenario + ); + expect(extensions.length).toBeGreaterThan(0); + expect(extensions.every((name) => selected.includes(name))).toBe(true); + expect( + extensions.some((name) => + scoredScenarios(requirements, 'server').includes(name) + ) + ).toBe(false); + }); + + it('fails loudly when the build is missing a required scenario', () => { + expect(() => + filterScenariosByRequirements( + ['nothing-it-asks-for'], + requirements, + 'server' + ) + ).toThrow(/does not provide/); + }); +}); diff --git a/src/requirements.ts b/src/requirements.ts new file mode 100644 index 00000000..1f76dc93 --- /dev/null +++ b/src/requirements.ts @@ -0,0 +1,211 @@ +import { existsSync, readdirSync, readFileSync } from 'fs'; +import { dirname, join } from 'path'; +import { fileURLToPath } from 'url'; +import { parse as parseYaml } from 'yaml'; +import { ALL_SPEC_VERSIONS } from './scenarios'; + +/** + * A frozen requirement set for one specification revision: the scenarios an + * implementation must pass to conform to that revision, fixed when it shipped. + * + * This is the project's contract, not an implementation's configuration. It is + * the opposite of an expected-failures baseline, which lives in an + * implementation's own repository and records what that implementation knows it + * fails. A baselined failure is still a failure against a requirement set. + */ +export interface RequirementSet { + /** Revision this set belongs to, e.g. `2026-07-28`. */ + revision: string; + /** Scenarios `conformance server` runs against a server implementation. */ + server: string[]; + /** Scenarios `conformance client` runs against a client implementation. */ + client: string[]; + /** + * Scenarios run and reported alongside the required ones but never counted + * toward a pass rate. Two reasons qualify, and the report names which: + * an extension is optional by definition, and a scenario added after the + * revision shipped is one no implementation could have been passing. + */ + notScored: NotScored[]; +} + +export type NotScoredReason = 'extension' | 'added-after-release'; + +export interface NotScored { + scenario: string; + leg: Leg; + reason: NotScoredReason; + note?: string; +} + +/** + * Roles a requirement set covers. The MCP specification defines two: an MCP + * server acting as an OAuth resource server, and an MCP client acting as an + * OAuth client. Authorization-server implementation is explicitly beyond the + * specification's scope, so those scenarios are not a conformance requirement + * for anything the spec defines and have no place in a requirement set. + */ +export type Leg = 'client' | 'server'; + +const LEGS: Leg[] = ['client', 'server']; +const REASONS: NotScoredReason[] = ['extension', 'added-after-release']; + +/** Requirement sets ship with the package; see the `files` entry in package.json. */ +function requirementsDir(): string { + return join(dirname(fileURLToPath(import.meta.url)), '..', 'requirements'); +} + +export function listRequirementRevisions(): string[] { + const dir = requirementsDir(); + if (!existsSync(dir)) return []; + return readdirSync(dir) + .filter((f) => f.endsWith('.yaml')) + .map((f) => f.replace(/\.yaml$/, '')) + .sort(); +} + +function asNameList(value: unknown, field: string, revision: string): string[] { + if (value === undefined) return []; + if (!Array.isArray(value) || value.some((v) => typeof v !== 'string')) { + throw new Error( + `requirements/${revision}.yaml: "${field}" must be a list of scenario names` + ); + } + return value as string[]; +} + +export function loadRequirements(revision: string): RequirementSet { + // A revision names a bundled file. Arbitrary paths are deliberately not + // accepted: a requirement set is the project's contract, so an implementation + // under test must not be able to supply its own. + if (!/^[0-9]{4}-[0-9]{2}-[0-9]{2}$/.test(revision)) { + // A run has one exit code and one expected-failures baseline, so it targets + // one revision. tier-check is the aggregator and takes several at once. + throw new Error( + revision.includes(',') + ? `A run targets one revision at a time, because it has one exit code and one baseline. Run each of "${revision}" separately, or pass them together to \`tier-check --requirements\`.` + : `Invalid requirements revision: ${revision}. Expected a date such as 2026-07-28.` + ); + } + + // The revision is also the wire version its scenarios run at, so it has to be + // a protocol version this build knows, not merely a date-shaped string. + if (!(ALL_SPEC_VERSIONS as readonly string[]).includes(revision)) { + throw new Error( + `Unknown spec revision: ${revision}. Known revisions: ${ALL_SPEC_VERSIONS.join(', ')}` + ); + } + + const path = join(requirementsDir(), `${revision}.yaml`); + if (!existsSync(path)) { + const known = listRequirementRevisions(); + throw new Error( + `No requirement set for ${revision}.` + + (known.length + ? ` Available: ${known.join(', ')}` + : ' No requirement sets are bundled with this release.') + ); + } + + const parsed = parseYaml(readFileSync(path, 'utf-8')) ?? {}; + const notScoredRaw = parsed.not_scored ?? []; + if (!Array.isArray(notScoredRaw)) { + throw new Error( + `requirements/${revision}.yaml: "not_scored" must be a list` + ); + } + + if (parsed.authorization !== undefined) { + throw new Error( + `requirements/${revision}.yaml: requirement sets cover the client and server roles only. ` + + 'Authorization-server implementation is beyond the scope of the MCP specification, so it is not a conformance requirement.' + ); + } + + return { + revision, + server: asNameList(parsed.server, 'server', revision), + client: asNameList(parsed.client, 'client', revision), + notScored: notScoredRaw.map((entry: unknown) => { + const e = entry as NotScored; + if (!e || typeof e.scenario !== 'string') { + throw new Error( + `requirements/${revision}.yaml: each not_scored entry needs a "scenario" name` + ); + } + if (!LEGS.includes(e.leg)) { + throw new Error( + `requirements/${revision}.yaml: ${e.scenario} needs a "leg" of ${LEGS.join(', ')}` + ); + } + if (!REASONS.includes(e.reason)) { + throw new Error( + `requirements/${revision}.yaml: ${e.scenario} needs a "reason" of ${REASONS.join(', ')}` + ); + } + return { + scenario: e.scenario, + leg: e.leg, + reason: e.reason, + note: e.note + }; + }) + }; +} + +/** Scenarios that count toward the pass rate for a leg. */ +export function scoredScenarios( + requirements: RequirementSet, + leg: Leg +): string[] { + return requirements[leg]; +} + +/** Scenarios run and reported for a leg but excluded from the pass rate. */ +export function notScoredScenarios( + requirements: RequirementSet, + leg: Leg +): NotScored[] { + return requirements.notScored.filter((entry) => entry.leg === leg); +} + +/** Everything a leg runs: what it scores, plus what it only reports. */ +export function scenariosToRun( + requirements: RequirementSet, + leg: Leg +): string[] { + return [ + ...scoredScenarios(requirements, leg), + ...notScoredScenarios(requirements, leg).map((e) => e.scenario) + ]; +} + +/** + * Narrow a scenario list to the revision's requirements. + * + * Names the requirement set asks for that this build does not have are an + * error: the set is frozen, so a missing name means the suite renamed or + * removed a scenario out from under it, and silently scoring a smaller set + * would understate what conformance means. + */ +export function filterScenariosByRequirements( + allScenarios: string[], + requirements: RequirementSet, + leg: Leg +): string[] { + const scored = new Set(scoredScenarios(requirements, leg)); + const available = new Set(allScenarios); + const wanted = scenariosToRun(requirements, leg); + // Only the scored names are a contract violation when absent. A not-scored + // entry can legitimately postdate the build being run, which is the whole + // reason it is listed separately. + const missing = wanted.filter( + (name) => !available.has(name) && scored.has(name) + ); + if (missing.length > 0) { + throw new Error( + `requirements/${requirements.revision}.yaml lists ${leg} scenarios that this build does not provide: ${missing.join(', ')}` + ); + } + return wanted.filter((name) => available.has(name)); +} diff --git a/src/tier-check/checks/test-conformance-results.ts b/src/tier-check/checks/test-conformance-results.ts index ba16be21..289506b3 100644 --- a/src/tier-check/checks/test-conformance-results.ts +++ b/src/tier-check/checks/test-conformance-results.ts @@ -10,6 +10,12 @@ import { listClientScenariosForSpec, getScenarioSpecVersions } from '../../scenarios'; +import { + notScoredScenarios, + RequirementSet, + scenariosToRun, + scoredScenarios +} from '../../requirements'; import { ConformanceCheck, DRAFT_PROTOCOL_VERSION, @@ -115,7 +121,14 @@ function stripTimestamp(dirName: string): string { function reconcileWithExpected( result: ConformanceResult, expectedScenarios: string[], - resultPrefix?: string + resultPrefix?: string, + /** + * Names that count toward the pass rate. When a requirement set is in play it + * decides this directly, so extensions and post-release additions are run and + * reported but never scored. Undefined keeps the spec-version heuristic. + */ + scoredNames?: Set, + notScoredReasons?: Map ): ConformanceResult { const reportedNames = new Set( result.details.map((d) => { @@ -127,38 +140,56 @@ function reconcileWithExpected( }) ); - // Attach specVersion to existing detail entries + // Normalise to the canonical scenario name. Results live in per-run + // directories (`server-ping-`), and reporting that spelling means + // a reported name does not match the same scenario in `list` or in a + // requirement set. for (const detail of result.details) { let name = stripTimestamp(detail.scenario); if (resultPrefix) { name = name.replace(new RegExp(`^${resultPrefix}-`), ''); } + detail.scenario = name; detail.specVersions = getScenarioSpecVersions(name); + const reason = notScoredReasons?.get(name); + if (reason) detail.notScoredReason = reason; } for (const expected of expectedScenarios) { if (!reportedNames.has(expected)) { result.failed++; result.total++; + const reason = notScoredReasons?.get(expected); result.details.push({ scenario: expected, passed: false, checks_passed: 0, checks_failed: 0, - specVersions: getScenarioSpecVersions(expected) + specVersions: getScenarioSpecVersions(expected), + ...(reason ? { notScoredReason: reason } : {}) }); } } - // pass_rate only counts tier-scoring scenarios (date-versioned, not draft/extension). - // passed/failed/total reflect ALL scenarios for full reporting; pass_rate and status - // reflect only tier-scoring scenarios for tier logic. - const tierDetails = result.details.filter((d) => - isTierScoring(d.specVersions) - ); + // passed/failed/total describe the SCORED set, so they agree with pass_rate and + // with the report. Counting every scenario here made the object self- + // contradictory (passed 56, total 64, pass_rate 1) and invited 56/64 to be + // quoted as the conformance rate. Anything run without being scored is + // reported under not_scored instead. + const tierDetails = scoredNames + ? result.details.filter((d) => scoredNames.has(d.scenario)) + : result.details.filter((d) => isTierScoring(d.specVersions)); const tierPassed = tierDetails.filter((d) => d.passed).length; const tierTotal = tierDetails.length; + const unscored = result.details.filter((d) => !tierDetails.includes(d)); + result.passed = tierPassed; + result.failed = tierTotal - tierPassed; + result.total = tierTotal; + result.not_scored = { + total: unscored.length, + failed: unscored.filter((d) => !d.passed).length + }; result.pass_rate = tierTotal > 0 ? tierPassed / tierTotal : 0; result.status = result.pass_rate >= 1.0 @@ -170,13 +201,78 @@ function reconcileWithExpected( return result; } +/** + * Run the conformance CLI as a child. A non-zero exit is normal when scenarios + * fail, so it is not itself an error; what matters is whether the child got far + * enough to write results. Returns the child's stderr when it did not. + */ +function unmeasured(error: string): ConformanceResult { + return { + status: 'fail', + pass_rate: 0, + passed: 0, + failed: 0, + total: 0, + details: [], + error + }; +} + +function runChild(args: string[]): string | undefined { + try { + execFileSync(process.execPath, args, { + stdio: ['pipe', 'pipe', 'pipe'], + timeout: 120_000 + }); + return undefined; + } catch (error) { + const e = error as { stderr?: Buffer; signal?: string }; + const stderr = e.stderr?.toString().trim(); + return stderr && stderr.length > 0 + ? stderr.split('\n').slice(-3).join(' ') + : e.signal + ? `conformance run terminated (${e.signal})` + : undefined; + } +} + /** * Run server conformance tests by shelling out to the conformance CLI. */ +/** Merge per-revision results so a single pass rate spans every revision claimed. */ +export function mergeByRevision( + parts: { revision: string; result: ConformanceResult }[] +): ConformanceResult { + if (parts.length === 1) return parts[0].result; + const errored = parts.find((p) => p.result.error); + if (errored) + return unmeasured(`${errored.revision}: ${errored.result.error}`); + const details = parts.flatMap((p) => + p.result.details.map((d) => ({ ...d, revision: p.revision })) + ); + const scored = details.filter((d) => !d.notScoredReason); + const passed = scored.filter((d) => d.passed).length; + const unscored = details.filter((d) => d.notScoredReason); + const pass_rate = scored.length > 0 ? passed / scored.length : 0; + return { + status: pass_rate >= 1 ? 'pass' : pass_rate >= 0.8 ? 'partial' : 'fail', + pass_rate, + passed, + failed: scored.length - passed, + total: scored.length, + not_scored: { + total: unscored.length, + failed: unscored.filter((d) => !d.passed).length + }, + details + }; +} + export async function checkConformance(options: { serverUrl?: string; skip?: boolean; specVersion?: SpecVersion; + requirements?: RequirementSet; }): Promise { if (options.skip || !options.serverUrl) { return { @@ -198,17 +294,30 @@ export async function checkConformance(options: { '-o', outputDir ]; - if (options.specVersion) { + if (options.requirements) { + args.push('--requirements', options.requirements.revision); + } else if (options.specVersion) { args.push('--spec-version', options.specVersion); } - try { - execFileSync(process.execPath, args, { - stdio: ['pipe', 'pipe', 'pipe'], - timeout: 120_000 - }); - } catch { - // Non-zero exit is expected when tests fail — results are still in outputDir + const failure = runChild(args); + + const parsedServer = parseOutputDir(outputDir); + if (failure && parsedServer.total === 0) return unmeasured(failure); + + if (options.requirements) { + return reconcileWithExpected( + parsedServer, + scenariosToRun(options.requirements, 'server'), + 'server', + new Set(scoredScenarios(options.requirements, 'server')), + new Map( + notScoredScenarios(options.requirements, 'server').map((e) => [ + e.scenario, + e.reason + ]) + ) + ); } const activeScenarios = new Set(listActiveClientScenarios()); @@ -218,11 +327,7 @@ export async function checkConformance(options: { ) : [...activeScenarios]; - return reconcileWithExpected( - parseOutputDir(outputDir), - expectedScenarios, - 'server' - ); + return reconcileWithExpected(parsedServer, expectedScenarios, 'server'); } /** @@ -232,6 +337,7 @@ export async function checkClientConformance(options: { clientCmd?: string; skip?: boolean; specVersion?: SpecVersion; + requirements?: RequirementSet; }): Promise { if (options.skip || !options.clientCmd) { return { @@ -250,27 +356,41 @@ export async function checkClientConformance(options: { 'client', '--command', options.clientCmd, - '--suite', - 'all', '-o', outputDir ]; - if (options.specVersion) { - args.push('--spec-version', options.specVersion); + if (options.requirements) { + args.push('--requirements', options.requirements.revision); + } else { + args.push('--suite', 'all'); + if (options.specVersion) { + args.push('--spec-version', options.specVersion); + } } - try { - execFileSync(process.execPath, args, { - stdio: ['pipe', 'pipe', 'pipe'], - timeout: 120_000 - }); - } catch { - // Non-zero exit is expected when tests fail — results are still in outputDir + const failure = runChild(args); + + const parsedClient = parseOutputDir(outputDir); + if (failure && parsedClient.total === 0) return unmeasured(failure); + + if (options.requirements) { + return reconcileWithExpected( + parsedClient, + scenariosToRun(options.requirements, 'client'), + undefined, + new Set(scoredScenarios(options.requirements, 'client')), + new Map( + notScoredScenarios(options.requirements, 'client').map((e) => [ + e.scenario, + e.reason + ]) + ) + ); } const expectedScenarios = options.specVersion ? listScenariosForSpec(options.specVersion) : listScenarios(); - return reconcileWithExpected(parseOutputDir(outputDir), expectedScenarios); + return reconcileWithExpected(parsedClient, expectedScenarios); } diff --git a/src/tier-check/index.ts b/src/tier-check/index.ts index 1cd32c19..60ed6ad2 100644 --- a/src/tier-check/index.ts +++ b/src/tier-check/index.ts @@ -2,7 +2,8 @@ import { Command } from 'commander'; import { Octokit } from '@octokit/rest'; import { checkConformance, - checkClientConformance + checkClientConformance, + mergeByRevision } from './checks/test-conformance-results'; import { checkLabels } from './checks/labels'; import { checkTriage } from './checks/triage'; @@ -14,6 +15,7 @@ import { computeTier } from './tier-logic'; import { formatJson, formatMarkdown, formatTerminal } from './output'; import { TierScorecard } from './types'; import { resolveSpecVersion } from '../scenarios'; +import { loadRequirements, RequirementSet } from '../requirements'; function parseRepo(repo: string): { owner: string; repo: string } { const parts = repo.split('/'); @@ -53,6 +55,10 @@ export function createTierCheckCommand(): Command { '--spec-version ', 'Only run conformance scenarios for this spec version' ) + .option( + '--requirements ', + 'Score against the frozen requirement set for a spec revision (e.g. 2026-07-28), so the SDK is measured against the suite as it stood when that revision shipped' + ) .action(async (options) => { const { owner, repo } = parseRepo(options.repo); let token = options.token || process.env.GITHUB_TOKEN; @@ -61,6 +67,32 @@ export function createTierCheckCommand(): Command { ? resolveSpecVersion(options.specVersion) : undefined; + let requirements: RequirementSet[] | undefined; + if (options.requirements !== undefined) { + if (specVersion) { + console.error( + '--requirements cannot be combined with --spec-version: a requirement set already fixes which scenarios run.' + ); + process.exit(1); + } + try { + // Several revisions may be claimed at once. Tier 1 then means every + // one of them passes, because a scenario shared by two revisions has + // to work on both wires, and one run does not cover the other. + requirements = String(options.requirements) + .split(',') + .map((r) => r.trim()) + .filter(Boolean) + .map(loadRequirements); + if (requirements.length === 0) { + throw new Error('--requirements needs at least one revision'); + } + } catch (error) { + console.error(error instanceof Error ? error.message : String(error)); + process.exit(1); + } + } + if (!token) { // Try to get token from GitHub CLI try { @@ -97,19 +129,41 @@ export function createTierCheckCommand(): Command { files, specTracking ] = await Promise.all([ - checkConformance({ - serverUrl: options.conformanceServerUrl, - skip: options.skipConformance, - specVersion - }).then((r) => { + (requirements + ? Promise.all( + requirements.map((req) => + checkConformance({ + serverUrl: options.conformanceServerUrl, + skip: options.skipConformance, + requirements: req + }).then((result) => ({ revision: req.revision, result })) + ) + ).then(mergeByRevision) + : checkConformance({ + serverUrl: options.conformanceServerUrl, + skip: options.skipConformance, + specVersion + }) + ).then((r) => { console.error(' ✓ Server Conformance'); return r; }), - checkClientConformance({ - clientCmd: options.clientCmd, - skip: options.skipConformance || !options.clientCmd, - specVersion - }).then((r) => { + (requirements + ? Promise.all( + requirements.map((req) => + checkClientConformance({ + clientCmd: options.clientCmd, + skip: options.skipConformance || !options.clientCmd, + requirements: req + }).then((result) => ({ revision: req.revision, result })) + ) + ).then(mergeByRevision) + : checkClientConformance({ + clientCmd: options.clientCmd, + skip: options.skipConformance || !options.clientCmd, + specVersion + }) + ).then((r) => { console.error(' ✓ Client Conformance'); return r; }), @@ -157,6 +211,7 @@ export function createTierCheckCommand(): Command { branch: options.branch || null, timestamp: new Date().toISOString(), version: release.version, + requirements_revisions: requirements?.map((r) => r.revision) ?? null, checks, implied_tier }; diff --git a/src/tier-check/output.ts b/src/tier-check/output.ts index ab0d2fbe..99a0d888 100644 --- a/src/tier-check/output.ts +++ b/src/tier-check/output.ts @@ -105,6 +105,95 @@ function formatRate(cell: Cell): string { return `${cell.passed}/${cell.total} (${Math.round((cell.passed / cell.total) * 100)}%)`; } +/** + * Under a requirement set the spec-version matrix is the wrong picture: the set + * already names exactly what is scored, so splitting it by applicability tag and + * filing part of it as "informational" would contradict the score. + */ +/** + * A scenario excluded from scoring is excluded under every revision that runs it, + * so listing it once per revision just pads the report. Keep one row, and treat it + * as failing if it failed anywhere. + */ +function dedupeNotScored( + entries: T[] +): T[] { + const byName = new Map(); + for (const e of entries) { + const seen = byName.get(e.scenario); + if (!seen) byName.set(e.scenario, e); + else if (seen.passed && !e.passed) byName.set(e.scenario, e); + } + return [...byName.values()]; +} + +function requirementsSummary(scorecard: TierScorecard): string[] { + const revisions = scorecard.requirements_revisions ?? []; + const rows: [string, ConformanceResult][] = [ + ['Server', scorecard.checks.conformance as ConformanceResult], + ['Client', scorecard.checks.client_conformance as ConformanceResult] + ]; + const lines = [ + '', + `Scored against the ${revisions.join(' and ')} requirement set${revisions.length > 1 ? 's' : ''}, each run at its own wire version.`, + '', + '| | Required | Passed | |', + '|---|---|---|---|' + ]; + for (const [label, result] of rows) { + if (result.error) { + lines.push(`| ${label} | — | — | not measured |`); + continue; + } + if (result.status === 'skipped') { + lines.push(`| ${label} | — | — | skipped |`); + continue; + } + lines.push( + `| ${label} | ${result.total} | ${result.passed} | ${Math.round(result.pass_rate * 100)}% |` + ); + } + + // A requirement set is small and every entry is named, so a bare count is not + // actionable: say which ones failed. + const failed = rows.flatMap(([label, result]) => + result.details + .filter((d) => !d.passed && !d.notScoredReason) + .map((d) => `${label.toLowerCase()}: ${d.scenario}`) + ); + if (failed.length > 0) { + lines.push(''); + lines.push(`_Failing (${failed.length}):_`); + lines.push(''); + failed.forEach((f) => lines.push(`- ${f}`)); + } + + // Run and reported, but deliberately outside the score. + const notScored = dedupeNotScored( + rows.flatMap(([label, result]) => + result.details + .filter((d) => d.notScoredReason) + .map((d) => ({ ...d, leg: label.toLowerCase() })) + ) + ); + if (notScored.length > 0) { + const failing = notScored.filter((d) => !d.passed); + lines.push(''); + lines.push( + `_Not scored (${notScored.length} run, ${failing.length} failing). These do not affect the tier:_` + ); + lines.push(''); + lines.push('| Scenario | Leg | Result | Why not scored |'); + lines.push('|---|---|---|---|'); + for (const d of notScored) { + lines.push( + `| ${d.scenario} | ${d.leg} | ${d.passed ? 'pass' : 'fail'} | ${d.notScoredReason} |` + ); + } + } + return lines; +} + export function formatJson(scorecard: TierScorecard): string { return JSON.stringify(scorecard, null, 2); } @@ -118,6 +207,10 @@ export function formatMarkdown(scorecard: TierScorecard): string { lines.push(`**Repo**: ${scorecard.repo}`); if (scorecard.branch) lines.push(`**Branch**: ${scorecard.branch}`); if (scorecard.version) lines.push(`**Version**: ${scorecard.version}`); + if (scorecard.requirements_revisions) + lines.push( + `**Requirements**: ${scorecard.requirements_revisions.join(', ')} (frozen; each run at its own wire)` + ); lines.push(`**Timestamp**: ${scorecard.timestamp}`); lines.push(''); lines.push('## Check Results'); @@ -129,50 +222,54 @@ export function formatMarkdown(scorecard: TierScorecard): string { c.client_conformance as ConformanceResult ); - // Tier-scoring matrix - lines.push(''); - lines.push(`| | ${TIER_SPEC_VERSIONS.join(' | ')} | All* |`); - lines.push(`|---|${TIER_SPEC_VERSIONS.map(() => '---|').join('')}---|`); + if (scorecard.requirements_revisions) { + lines.push(...requirementsSummary(scorecard)); + } else { + // Tier-scoring matrix + lines.push(''); + lines.push(`| | ${TIER_SPEC_VERSIONS.join(' | ')} | All* |`); + lines.push(`|---|${TIER_SPEC_VERSIONS.map(() => '---|').join('')}---|`); - const mdRows: [string, MatrixRow][] = [ - ['Server', matrix.server], - ['Client: Core', matrix.clientCore], - ['Client: Auth', matrix.clientAuth] - ]; + const mdRows: [string, MatrixRow][] = [ + ['Server', matrix.server], + ['Client: Core', matrix.clientCore], + ['Client: Auth', matrix.clientAuth] + ]; + + for (const [label, row] of mdRows) { + lines.push( + `| ${label} | ${TIER_SPEC_VERSIONS.map((v) => formatCell(row.cells.get(v))).join(' | ')} | ${formatRate(row.tierUnique)} |` + ); + } - for (const [label, row] of mdRows) { + lines.push(''); lines.push( - `| ${label} | ${TIER_SPEC_VERSIONS.map((v) => formatCell(row.cells.get(v))).join(' | ')} | ${formatRate(row.tierUnique)} |` + '_* unique scenarios — a scenario may apply to multiple spec versions_' ); - } - - lines.push(''); - lines.push( - '_* unique scenarios — a scenario may apply to multiple spec versions_' - ); - // Informational matrix (draft/extension) - const hasInfoMd = mdRows.some(([, row]) => - INFO_SPEC_VERSIONS.some((v) => { - const cell = row.cells.get(v); - return cell && cell.total > 0; - }) - ); - if (hasInfoMd) { - lines.push(''); - lines.push('_Informational (not scored for tier):_'); - lines.push(''); - lines.push(`| | ${INFO_SPEC_VERSIONS.join(' | ')} |`); - lines.push(`|---|${INFO_SPEC_VERSIONS.map(() => '---|').join('')}`); - for (const [label, row] of mdRows) { - const hasData = INFO_SPEC_VERSIONS.some((v) => { + // Informational matrix (draft/extension) + const hasInfoMd = mdRows.some(([, row]) => + INFO_SPEC_VERSIONS.some((v) => { const cell = row.cells.get(v); return cell && cell.total > 0; - }); - if (!hasData) continue; - lines.push( - `| ${label} | ${INFO_SPEC_VERSIONS.map((v) => formatCell(row.cells.get(v))).join(' | ')} |` - ); + }) + ); + if (hasInfoMd) { + lines.push(''); + lines.push('_Informational (not scored for tier):_'); + lines.push(''); + lines.push(`| | ${INFO_SPEC_VERSIONS.join(' | ')} |`); + lines.push(`|---|${INFO_SPEC_VERSIONS.map(() => '---|').join('')}`); + for (const [label, row] of mdRows) { + const hasData = INFO_SPEC_VERSIONS.some((v) => { + const cell = row.cells.get(v); + return cell && cell.total > 0; + }); + if (!hasData) continue; + lines.push( + `| ${label} | ${INFO_SPEC_VERSIONS.map((v) => formatCell(row.cells.get(v))).join(' | ')} |` + ); + } } } lines.push(''); @@ -228,79 +325,142 @@ export function formatTerminal(scorecard: TierScorecard): void { console.log(`Repo: ${scorecard.repo}`); if (scorecard.branch) console.log(`Branch: ${scorecard.branch}`); if (scorecard.version) console.log(`Version: ${scorecard.version}`); + if (scorecard.requirements_revisions) + console.log( + `Requires: ${scorecard.requirements_revisions.join(', ')} (frozen, each at its own wire)` + ); console.log(`Timestamp: ${scorecard.timestamp}\n`); console.log(`${COLORS.BOLD}Conformance:${COLORS.RESET}\n`); - // Conformance matrix - const matrix = buildConformanceMatrix( - c.conformance as ConformanceResult, - c.client_conformance as ConformanceResult - ); - - const vw = 10; // column width for version cells - const lw = 14; // label column width - const tw = 16; // total column width - const rp = (s: string, w: number) => s.padStart(w); - const lp = (s: string, w: number) => s.padEnd(w); - - // Tier-scoring matrix (date-versioned specs only) - console.log( - ` ${COLORS.DIM}${lp('', lw + 2)} ${TIER_SPEC_VERSIONS.map((v) => rp(v, vw)).join(' ')} ${rp('All*', tw)}${COLORS.RESET}` - ); + if (scorecard.requirements_revisions) { + console.log( + ` ${COLORS.DIM}Scored against ${scorecard.requirements_revisions.join(' and ')}, each run at its own wire version.${COLORS.RESET}\n` + ); + const reqRows: [string, ConformanceResult][] = [ + ['Server', c.conformance as ConformanceResult], + ['Client', c.client_conformance as ConformanceResult] + ]; + for (const [label, result] of reqRows) { + if (result.status === 'skipped') { + console.log(` ${label.padEnd(8)} skipped`); + continue; + } + if (result.error) { + console.log( + ` ${label.padEnd(8)} ${COLORS.RED}not measured${COLORS.RESET}: ${result.error}` + ); + continue; + } + console.log( + ` ${label.padEnd(8)} ${result.passed}/${result.total} required scenarios (${Math.round(result.pass_rate * 100)}%)` + ); + } + const failedReq = reqRows.flatMap(([label, result]) => + result.details + .filter((d) => !d.passed && !d.notScoredReason) + .map((d) => `${label.toLowerCase()}: ${d.scenario}`) + ); + if (failedReq.length > 0) { + console.log( + `\n ${COLORS.RED}Failing (${failedReq.length}):${COLORS.RESET}` + ); + failedReq.forEach((f) => + console.log(` ${COLORS.RED}\u2717${COLORS.RESET} ${f}`) + ); + } + const notScoredReq = dedupeNotScored( + reqRows.flatMap(([, result]) => + result.details.filter((d) => d.notScoredReason) + ) + ); + if (notScoredReq.length > 0) { + const failing = notScoredReq.filter((d) => !d.passed); + console.log( + `\n ${COLORS.DIM}Not scored (${notScoredReq.length} run, ${failing.length} failing, no effect on tier):${COLORS.RESET}` + ); + for (const d of notScoredReq) { + const mark = d.passed + ? `${COLORS.GREEN}\u2713${COLORS.RESET}` + : `${COLORS.DIM}\u2717${COLORS.RESET}`; + console.log( + ` ${mark} ${d.scenario} ${COLORS.DIM}(${d.notScoredReason})${COLORS.RESET}` + ); + } + } + console.log(''); + } else { + // Conformance matrix + const matrix = buildConformanceMatrix( + c.conformance as ConformanceResult, + c.client_conformance as ConformanceResult + ); - const rows: [string, MatrixRow, CheckStatus | null, boolean][] = [ - ['Server', matrix.server, c.conformance.status, true], - ['Client: Core', matrix.clientCore, null, false], - ['Client: Auth', matrix.clientAuth, null, false] - ]; + const vw = 10; // column width for version cells + const lw = 14; // label column width + const tw = 16; // total column width + const rp = (s: string, w: number) => s.padStart(w); + const lp = (s: string, w: number) => s.padEnd(w); - for (const [label, row, status, bold] of rows) { - const icon = status ? statusIcon(status) + ' ' : ' '; - const b = bold ? COLORS.BOLD : ''; - const r = bold ? COLORS.RESET : ''; + // Tier-scoring matrix (date-versioned specs only) console.log( - ` ${icon}${b}${lp(label, lw)}${r} ${TIER_SPEC_VERSIONS.map((v) => rp(formatCell(row.cells.get(v)), vw)).join(' ')} ${b}${rp(formatRate(row.tierUnique), tw)}${r}` + ` ${COLORS.DIM}${lp('', lw + 2)} ${TIER_SPEC_VERSIONS.map((v) => rp(v, vw)).join(' ')} ${rp('All*', tw)}${COLORS.RESET}` ); - } - // Client total line (tier-scoring only) - const clientTierTotal: Cell = { - passed: - matrix.clientCore.tierUnique.passed + matrix.clientAuth.tierUnique.passed, - total: - matrix.clientCore.tierUnique.total + matrix.clientAuth.tierUnique.total - }; - console.log( - ` ${statusIcon(c.client_conformance.status)} ${COLORS.BOLD}${lp('Client Total', lw)}${COLORS.RESET} ${' '.repeat(TIER_SPEC_VERSIONS.length * (vw + 1) - 1)} ${COLORS.BOLD}${rp(formatRate(clientTierTotal), tw)}${COLORS.RESET}` - ); - console.log( - `\n ${COLORS.DIM}* unique scenarios — a scenario may apply to multiple spec versions${COLORS.RESET}` - ); + const rows: [string, MatrixRow, CheckStatus | null, boolean][] = [ + ['Server', matrix.server, c.conformance.status, true], + ['Client: Core', matrix.clientCore, null, false], + ['Client: Auth', matrix.clientAuth, null, false] + ]; - // Informational matrix (draft/extension) — only if there are any - const hasInfo = rows.some(([, row]) => - INFO_SPEC_VERSIONS.some((v) => { - const cell = row.cells.get(v); - return cell && cell.total > 0; - }) - ); - if (hasInfo) { - console.log(`\n Informational (not scored for tier):\n`); + for (const [label, row, status, bold] of rows) { + const icon = status ? statusIcon(status) + ' ' : ' '; + const b = bold ? COLORS.BOLD : ''; + const r = bold ? COLORS.RESET : ''; + console.log( + ` ${icon}${b}${lp(label, lw)}${r} ${TIER_SPEC_VERSIONS.map((v) => rp(formatCell(row.cells.get(v)), vw)).join(' ')} ${b}${rp(formatRate(row.tierUnique), tw)}${r}` + ); + } + + // Client total line (tier-scoring only) + const clientTierTotal: Cell = { + passed: + matrix.clientCore.tierUnique.passed + + matrix.clientAuth.tierUnique.passed, + total: + matrix.clientCore.tierUnique.total + matrix.clientAuth.tierUnique.total + }; console.log( - ` ${COLORS.DIM}${lp('', lw + 2)} ${INFO_SPEC_VERSIONS.map((v) => rp(v, vw)).join(' ')}${COLORS.RESET}` + ` ${statusIcon(c.client_conformance.status)} ${COLORS.BOLD}${lp('Client Total', lw)}${COLORS.RESET} ${' '.repeat(TIER_SPEC_VERSIONS.length * (vw + 1) - 1)} ${COLORS.BOLD}${rp(formatRate(clientTierTotal), tw)}${COLORS.RESET}` ); - for (const [label, row, , bold] of rows) { - const hasData = INFO_SPEC_VERSIONS.some((v) => { + console.log( + `\n ${COLORS.DIM}* unique scenarios — a scenario may apply to multiple spec versions${COLORS.RESET}` + ); + + // Informational matrix (draft/extension) — only if there are any + const hasInfo = rows.some(([, row]) => + INFO_SPEC_VERSIONS.some((v) => { const cell = row.cells.get(v); return cell && cell.total > 0; - }); - if (!hasData) continue; - const b = bold ? COLORS.BOLD : ''; - const r = bold ? COLORS.RESET : ''; + }) + ); + if (hasInfo) { + console.log(`\n Informational (not scored for tier):\n`); console.log( - ` ${b}${lp(label, lw)}${r} ${INFO_SPEC_VERSIONS.map((v) => rp(formatCell(row.cells.get(v)), vw)).join(' ')}` + ` ${COLORS.DIM}${lp('', lw + 2)} ${INFO_SPEC_VERSIONS.map((v) => rp(v, vw)).join(' ')}${COLORS.RESET}` ); + for (const [label, row, , bold] of rows) { + const hasData = INFO_SPEC_VERSIONS.some((v) => { + const cell = row.cells.get(v); + return cell && cell.total > 0; + }); + if (!hasData) continue; + const b = bold ? COLORS.BOLD : ''; + const r = bold ? COLORS.RESET : ''; + console.log( + ` ${b}${lp(label, lw)}${r} ${INFO_SPEC_VERSIONS.map((v) => rp(formatCell(row.cells.get(v)), vw)).join(' ')}` + ); + } } } console.log(`\n${COLORS.BOLD}Repository Health:${COLORS.RESET}\n`); diff --git a/src/tier-check/types.ts b/src/tier-check/types.ts index eacbcbe6..5016a266 100644 --- a/src/tier-check/types.ts +++ b/src/tier-check/types.ts @@ -12,12 +12,32 @@ export interface ConformanceResult extends CheckResult { passed: number; failed: number; total: number; + /** + * Scenarios run without being scored, kept out of passed/failed/total so those + * always agree with pass_rate. + */ + not_scored?: { total: number; failed: number }; + /** + * Set when the conformance run could not be performed at all, e.g. the child + * CLI rejected a requirement set naming a scenario this build lacks. Without + * this the empty result reconciles into "every scenario failed", which blames + * the implementation for a broken invocation. + */ + error?: string; details: Array<{ scenario: string; passed: boolean; checks_passed: number; checks_failed: number; specVersions?: ScenarioSpecTag[]; + /** + * Why this scenario ran but did not count toward the pass rate. Set only + * under a requirement set, which decides scoring by name rather than by + * spec-version tag. + */ + notScoredReason?: string; + /** Which revision's run produced this, when several were scored. */ + revision?: string; }>; } @@ -71,6 +91,14 @@ export interface TierScorecard { branch: string | null; timestamp: string; version: string | null; + /** + * Spec revisions whose frozen requirement sets were scored, if any were used. + * Every listed revision must pass for Tier 1: a scenario shared by two + * revisions must work on both wires, and one run does not cover the other. + * When set, the pass rates count exactly the scenarios that revision + * required when it shipped, rather than everything the suite carries today. + */ + requirements_revisions: string[] | null; checks: { conformance: ConformanceResult; client_conformance: ConformanceResult;