feat(interceptors): initial gateway interceptor implementation and reference example

[drew]

Summary

Implements #1927.

• Interceptor framework: Add descriptor-driven gRPC interception with ordered modify, validate, and post-commit phases, JSON Patch mutations, failure policies, limits, metrics, and audit annotations.
• Authority and secret isolation: Add operator-controlled binding policies and protobuf secret annotations that recursively exclude sensitive fields and patch paths from interceptor payloads.
• Metadata and provenance: Add annotations to object metadata, sandbox operations, and provider profiles; atomically persist policy provenance and expose it through APIs, CLI, and TUI.
• Provider profile catalogs: Compose built-in, user-managed, and interceptor-vended profile sources while using one validated, immutable catalog snapshot per request.
• Governance example: Add canonical policy/profile signing, authoritative profile vending, policy injection and verification, hot reload, and protections against governance bypasses.
• Documentation and testing: Update gateway configuration and architecture docs, generated protobuf support, unit/integration coverage, and the authenticated governance smoke suite.

Related Issue

Closes #1919, #1842

Changes

• Add interceptor, binding, and decision labels to gateway interceptor evaluation counters.
• Remove Z3-specific build setup from the governance interceptor smoke script.
• Start a dedicated smoke gateway with isolated ports, state, gateway ID, and scrubbed ambient OpenShell environment.
• Print neutral setup INFO messages and reserve PASS/FAIL for functional checks.
• Dump smoke, gateway, and interceptor logs inline on failures.
• Refresh the governance interceptor README smoke-test wording.

Testing

• CC=clang CXX=clang++ mise run pre-commit passes
• OPENSHELL_GOVERNANCE_KEEP_LOGS=1 examples/governance-interceptor/smoke.sh passes
• Hostile ambient OpenShell env smoke run passes
• Unit tests added/updated (not needed for smoke script hardening)
• E2E tests added/updated (not applicable)

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-2005.docs.buildwithfern.com/openshell

[johntmyers]

gator-agent

Blocked

Gator is blocked because PR #2005 is still marked as a draft.

Next action: @drew, mark the pull request ready for review when the interceptor implementation is ready for the independent code review and CI-gating workflow.

Head SHA: e6e5a8d35b2c4118496008ba592217201a016ef0

[johntmyers]

gator-agent

PR Review Status

Validation: This is project-valid, maintainer-authored work implementing #1919 and the actively reviewed gateway-interceptors RFC in #1927, with #1842 explicitly folded into the interceptor design.

Head SHA: e6e5a8d35b2c4118496008ba592217201a016ef0

Thanks @drew. I also checked your response on the crate-layout discussion: the current flat, component-prefixed openshell-gateway-interceptors layout is consistent with that stated direction, so I am treating that discussion as resolved.

Review findings:

• Critical — sandbox principals can bypass the signed-governance path. examples/governance-interceptor/src/main.rs:639, crates/openshell-gateway-interceptors/src/routes.rs:30, and crates/openshell-server/src/grpc/policy.rs:2550: sandbox UpdateConfig is allowed before signature validation, and non-interceptable SubmitPolicyAnalysis can auto-approve policy changes. Governed sandboxes need authenticated E2E coverage proving neither path can widen policy without valid authority.
• Critical — an interceptor can self-expand its authority and receive gateway secrets. crates/openshell-core/src/config.rs:529, crates/openshell-gateway-interceptors/src/lib.rs:340, and :381: Describe controls bindings and profile-source authority unless optionally narrowed. This can expose provider credentials or refresh material, including over accepted plaintext http:// endpoints. Require an operator allowlist, secret-field redaction/authorization, and UDS or authenticated TLS for secret-bearing operations.
• Critical — post_commit fail-closed can report failure after a mutation committed. crates/openshell-server/src/multiplex.rs:296-303: a post-commit timeout/invalid result replaces a committed success with PERMISSION_DENIED, inviting unsafe retries. Preserve the committed response and report audit failure separately (or use a transactional outbox), with middleware coverage.
• Critical — policy state and governance provenance are committed non-atomically. crates/openshell-server/src/grpc/policy.rs:1841-1855 and :1998-2016: policy writes happen before the separate annotation CAS. A stale/concurrent update can leave active policy without its signature/provenance while returning an error. Commit these together or store attestation atomically with the policy revision, and test the conflict path.
• Warning — the protobuf transcoder drops oneof semantics. crates/openshell-gateway-interceptors/src/lib.rs:1128-1160, :1209-1223, and :1607-1640: malformed wire input with multiple alternatives can be validated as one shape and re-encoded as another. Preserve oneof metadata or reject multiple alternatives, with a wire-level test.
• Warning — wrong-type patches bypass per-binding fail_open handling. crates/openshell-gateway-interceptors/src/lib.rs:568-579 and :497-500: JSON Patch can succeed before final protobuf encoding fails outside the binding's failure policy. Schema-validate/re-encode each candidate mutation before accepting it.
• Warning — provider profile snapshots are repeatedly fetched and can mix revisions. crates/openshell-server/src/provider_profile_sources.rs:167-180, :200-232, and crates/openshell-server/src/grpc/policy.rs:1317-1334: fetch one immutable catalog per request or use a validated revision-keyed cache.
• Warning — signed hashes are not canonical for nested protobuf maps. examples/governance-interceptor/src/policy_hash.rs:10-30 and examples/governance-interceptor/src/main.rs:895-914: recursively canonicalize nested maps and test insertion-order equivalence.
• Warning — docs advertise an unimplemented append catalog mode. docs/sandboxes/providers-v2.mdx:55 says catalogs can be append or authoritative, while crates/openshell-server/src/provider_profile_sources.rs:129-139 always replaces built-in/user sources when interceptor sources exist. Remove the append claim or implement/configure it.

Docs: Relevant Fern docs and gateway architecture docs are updated, but the append-catalog statement above must be corrected. No navigation change appears necessary for the existing pages.

Tests: The implementation is gateway/policy/provider infrastructure and will require test:e2e once review findings are resolved and pipeline gating begins.

Next state: gator:in-review

[johntmyers]

gator-agent

PR Review Status

Validation: This is project-valid maintainer-authored work implementing #1919 and the active gateway-interceptors RFC in #1927, with #1842 folded into the interceptor/governance design.

Head SHA: 4cf7381d39be0d0dff8191a5ecce8ccb229d24c6

Thanks @drew. I rechecked the July 10 updates against the earlier gator findings. The provider-profile source composition work resolves the append/authoritative docs mismatch and narrows the old interceptor profile-source self-selection concern, but blocking review feedback remains.

Review findings:

• Critical — sandbox principals can still bypass signed/frozen governance. examples/governance-interceptor/src/main.rs:652, crates/openshell-server/src/grpc/policy.rs:1021, crates/openshell-gateway-interceptors/src/routes.rs:30, and crates/openshell-server/src/grpc/policy.rs:2550: the example still unconditionally allows sandbox principals before validating governance annotations, while core only restricts sandbox callers to sandbox-scoped policy sync shape. SubmitPolicyAnalysis also remains non-interceptable and can auto-merge policy changes. Remove the unconditional sandbox allow and apply the same frozen-governance guard to sandbox policy sync and SubmitPolicyAnalysis auto-approval, or make that path interceptable.
• Critical — policy revisions and governance provenance still commit separately. crates/openshell-server/src/grpc/policy.rs:1998, crates/openshell-server/src/grpc/policy.rs:2009, crates/openshell-server/src/grpc/policy.rs:1071, and crates/openshell-server/src/grpc/policy.rs:1841: a policy revision can be activated before the annotation/provenance CAS succeeds, so a conflict can return an error after committing policy state without matching provenance. Store provenance with the policy revision or wrap both writes in one transaction, with conflict-path coverage.
• Critical — post_commit can still report failure after a mutation committed. crates/openshell-server/src/multiplex.rs:296, crates/openshell-server/src/multiplex.rs:309, crates/openshell-gateway-interceptors/src/lib.rs:996, and crates/openshell-gateway-interceptors/src/lib.rs:963: post-commit interceptor failure can replace a successful committed response with PERMISSION_DENIED, inviting unsafe retries. Preserve the committed response and report audit/metrics/outbox failure separately; blocking decisions need to happen before commit.
• Warning — wrong-type JSON patches still bypass per-binding fail_open. crates/openshell-gateway-interceptors/src/lib.rs:533 and crates/openshell-gateway-interceptors/src/lib.rs:450: patch application is under the binding failure policy, but protobuf re-encoding happens later outside that policy. Validate or re-encode each patched candidate under the binding policy before accepting it.
• Warning — provider profile snapshots can still mix revisions within one request. crates/openshell-server/src/provider_profile_sources.rs:266, crates/openshell-server/src/provider_profile_sources.rs:299, and crates/openshell-server/src/grpc/policy.rs:1392: profile lookups can rebuild the effective catalog independently. Fetch one immutable catalog snapshot per request/config build and pass it through.
• Warning — secret-bearing interceptor traffic can use plaintext http://. crates/openshell-core/src/config.rs:538 and crates/openshell-gateway-interceptors/src/lib.rs:849: interceptable operations can include provider credentials, but endpoints may be plaintext HTTP. Require UDS or authenticated TLS for secret-bearing bindings, or redact secrets unless explicitly allowed.
• Warning — signed hashes are still not canonical for nested maps. examples/governance-interceptor/src/policy_hash.rs:10 and examples/governance-interceptor/src/policy_hash.rs:29: only top-level network_policies are sorted before hashing nested protobuf encodings. Use recursive canonicalization or canonical JSON for every map-bearing message before signing.

Docs: The provider-profile source docs now match implementation, including the authoritative-by-omission behavior. No Fern navigation change appears necessary for the existing pages.

Tests: Once review findings are resolved, this gateway/policy/provider infrastructure change should run with test:e2e; add focused coverage for governed UpdateConfig, SubmitPolicyAnalysis auto-approval, annotation CAS conflict after policy write, and post-commit failure after commit.

Next state: gator:in-review

[drew]

Thanks @johntmyers. I worked through every finding from the review. The current head is 36567b88.

Finding	Disposition
Sandbox principals can bypass signed governance	Resolved in facaa2ca. SubmitPolicyAnalysis is interceptable, sandbox-authored proposals and auto-approval are denied by the governance example, and UpdateConfig requires a valid signed baseline. Authenticated smoke coverage proves these paths cannot widen policy.
Interceptors can expand authority or receive secrets	Resolved in a42ac33c, building on 4cf7381d. Operator-configured allowlist and exact binding policies constrain manifest authority. Protobuf secret annotations recursively omit sensitive fields from interceptor payloads, with no exposure escape hatch. Because interceptor RPC views are unconditionally non-secret-bearing, plaintext development endpoints cannot receive gateway secrets.
post_commit can replace a committed success	Resolved in da24af51. Post-commit bindings must fail open, and observation failures preserve the committed response while emitting warnings and metrics.
Policy and provenance are non-atomic	Resolved in 8c1b2206. Policy revision, immutable provenance, sandbox projection, optional backfill, and superseding now commit in one SQLite or PostgreSQL transaction. Conflict and insert failures roll back the entire operation.
Transcoder drops oneof semantics	Resolved initially in 3b91c641 and simplified in 33087ef0. The reflected protobuf codec preserves standard last-member-wins wire semantics and canonicalizes the selected member before evaluation and dispatch.
Wrong-type patches bypass fail_open	Resolved in 8a39af1c. Each binding's complete patch candidate is protobuf-validated within that binding's failure-policy boundary.
Provider-profile snapshots can mix revisions	Resolved in 84d0b6c5. Each logical request captures one immutable, validated catalog and reuses it for policy, credential scope, environment, dynamic grants, and revision derivation. Each configured source is fetched once per request.
Signed hashes are not recursively canonical	Resolved in 86582b80. The governance example owns recursive canonical ProtoJSON hashing for policies and provider profiles, including nested maps and insertion-order equivalence tests.
Append catalog mode was unimplemented	Resolved in 4cf7381d. Source composition is explicit and ordered; the documentation now matches the implementation.

Validation completed:

• Authenticated governance smoke suite passes, including both signed-governance bypass cases.
• 45 gateway-interceptor tests, 293 core tests, 32 governance-example tests, and 58 Python SDK tests pass.
• All 924 existing server tests pass, plus the new request-level provider snapshot regression.
• Strict server Clippy passes with -D warnings at 36567b88.
• The broader test:e2e remains a pipeline gate; I am not treating the targeted governance smoke suite as a substitute for that full run.

Please re-review the current head when ready.

[johntmyers]

gator-agent

Blocked

Gator is blocked because PR #2005 currently has merge conflicts with main (mergeable_state: dirty).

Thanks @drew, I saw your July 13 update saying the prior gator findings were addressed at 36567b88. I cannot run the next independent code review cycle until the branch is mergeable against the current base.

Next action: @drew, rebase or merge main into gateway-interceptors-impl and push an updated head. Gator will re-review the new head after the conflict blocker is cleared.

Head SHA: 36567b88f1214ec0b393daaf6be9f69c4115d8cc

[github-actions]

Label test:e2e applied for d94ba94. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[johntmyers]

gator-agent

PR Review Status

Validation: This remains project-valid maintainer-authored work implementing #1919 and the active gateway-interceptors RFC in #1927, with #1842 folded into the interceptor/governance design.

Head SHA: b8c717ee3c1e54fe4215c4fc67650bded5f8e644

Review findings:

• No blocking findings remain.
• Non-blocking reviewer note: crates/openshell-gateway-interceptors/src/runtime.rs can still let an interceptor denial request gRPC OK; mapping allowed = false, status_code = "OK" to PERMISSION_DENIED would make the denial contract clearer.

Prior gator findings: still resolved. I rechecked the latest head after the one-line policy test annotation update, and the independent reviewer confirmed the key prior areas remain addressed: atomic policy/provenance writes, post-commit response preservation, secret-field omission from interceptor payloads, and request-scoped provider-profile snapshots.

Docs: Direct gateway config and provider-profile UX/API changes are covered in docs/reference/gateway-config.mdx and docs/sandboxes/providers-v2.mdx. No Fern navigation update appears necessary for the existing pages.

Tests: Focused coverage appears present for the previously flagged critical paths. test:e2e is already applied and the E2E gate is still running for this head.

Next state: gator:watch-pipeline

[johntmyers]

gator-agent

PR Review Status

Validation: This remains project-valid maintainer-authored work implementing #1919 and the active gateway-interceptors RFC in #1927, with #1842 folded into the interceptor/governance design.

Head SHA: 64035c942efd765bd853f1fabe489b4867f86251

Review findings:

• No blocking findings remain.
• Non-blocking reviewer note: crates/openshell-gateway-interceptors/src/runtime.rs can still let an interceptor denial request gRPC OK; mapping allowed = false, status_code = "OK" to PERMISSION_DENIED would make the denial contract clearer.
• Non-blocking reviewer note: the new explicit route allowlist in crates/openshell-gateway-interceptors/src/routes.rs is the safer default, but a table-style test classifying every current unary RPC as interceptable or intentionally non-interceptable would make future omissions easier to catch.

Prior gator findings: still resolved. I rechecked the latest head after the route allowlist update; the independent reviewer confirmed the governance-relevant mutation paths remain interceptable, non-allowlisted configured bindings fail startup instead of silently skipping, and the previous governance, provenance, post-commit, secret-isolation, provider-profile snapshot, and canonical hashing concerns remain addressed.

Docs: The direct gateway interceptor docs were updated in architecture/gateway.md and docs/reference/gateway-config.mdx. Existing Fern navigation still appears sufficient for these pages.

Tests: Focused coverage appears present for the previously flagged critical paths. test:e2e is already applied, and the required Branch Checks / E2E gates are still pending for this head.

Next state: gator:watch-pipeline

[johntmyers]

gator-agent

PR Review Status

Validation: This remains project-valid maintainer-authored work implementing #1919 and the active gateway-interceptors RFC in #1927, with #1842 folded into the interceptor/governance design.

Head SHA: 4888a04cf44c035948a67e27dfbe135421349cfe

Thanks @pimlock and @drew. I rechecked the current-head discussion about UpdateConfigRequest.annotations versus policy revision provenance. The updated proto/openshell.proto comment now matches the intended contract: annotations are caller/interceptor-provided opaque metadata, policy updates store them immutably as revision provenance, and sandbox metadata is only a convenience projection.

Review findings:

• Warning — sandbox-scoped setting updates can partially commit before annotation CAS failure. crates/openshell-server/src/grpc/policy.rs:1821, crates/openshell-server/src/grpc/policy.rs:1831, crates/openshell-server/src/grpc/policy.rs:1864, and crates/openshell-server/src/grpc/policy.rs:1874: setting delete/upsert saves sandbox_settings before persist_update_config_annotations checks expected_resource_version. A stale resource version can therefore return an error after the setting change has already committed, which violates the CAS semantics described for expected_resource_version. Please make sandbox settings plus annotation projection atomic, similar to put_policy_revision_atomic, or otherwise ensure the resource-version conflict is detected before any setting mutation is persisted.

Non-blocking reviewer notes:

• crates/openshell-gateway-interceptors/src/runtime.rs can still let an interceptor denial request gRPC OK; mapping allowed = false, status_code = "OK" to PERMISSION_DENIED would make the denial contract clearer.
• The explicit route allowlist in crates/openshell-gateway-interceptors/src/routes.rs is the safer default. A table-style test classifying every current unary RPC as interceptable or intentionally non-interceptable would make future omissions easier to catch.

Prior gator findings: still resolved. The independent reviewer confirmed the key prior areas remain addressed: secret-field omission from interceptor payloads, atomic policy/provenance writes for full policy and merge updates, request-scoped provider-profile snapshots, and the updated annotation/provenance wording.

Docs: Direct gateway config and provider-profile UX/API changes are covered in docs/reference/gateway-config.mdx, docs/sandboxes/providers-v2.mdx, and architecture/gateway.md. Existing Fern navigation still appears sufficient for these pages.

Tests: test:e2e remains applied. The required E2E gate is still pending, but this review finding should be addressed before gator returns to pipeline watch.

Next state: gator:in-review

[johntmyers]

gator-agent

Monitoring Complete

Monitoring is complete because this PR has merged.

Final status: PR #2005 merged on 2026-07-15 at 03:58:36Z. The last active gator state was gator:in-review on head 4888a04cf44c035948a67e27dfbe135421349cfe; maintainer approval was present before merge.

I removed the active gator:* label because there is nothing left for gator to monitor on this PR.