feat(jsonb): storage-only public.eql_v3_json + rename searchable domain to _search (CIP-3512)#397
Open
freshtonic wants to merge 5 commits into
Open
feat(jsonb): storage-only public.eql_v3_json + rename searchable domain to _search (CIP-3512)#397freshtonic wants to merge 5 commits into
freshtonic wants to merge 5 commits into
Conversation
…omain to _search
Every scalar EQL type shipped a storage-only / encryption-only variant under
its bare name, but json had none — its only domain required a full searchable
SteVec document. public.eql_v3_json is now the storage-only domain (a plain
{v,i,c} envelope with a native-jsonb operator firewall, the JSON analogue of
the scalar storage domains), and the searchable SteVec document moves to
public.eql_v3_json_search. public.eql_v3_jsonb_entry and eql_v3.query_jsonb
are unchanged.
Threaded through the catalog (storage json is a Shape::Scalar domain in the
jsonb family), codegen, hand-written bindings (new Json struct), regenerated
TS/JSON schemas, a generated real-ciphertext storage fixture with tests, and
the v2->v3 conversion path. Ships as a patch (3.0.1) with an upgrade note
(U-007); EQL v3 has no production users yet.
CIP-3512
Contributor
|
Warning Review limit reached
Next review available in: 27 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (17)
📝 WalkthroughWalkthroughThe v3 JSONB model now separates storage-only Changesv3 JSONB domain split
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related issues
Possibly related PRs
Suggested reviewers: Sequence Diagram(s)sequenceDiagram
participant Client
participant PostgreSQL
participant JsonSearchDomain
participant JsonStorageDomain
Client->>PostgreSQL: submit encrypted JSON expression
PostgreSQL->>JsonSearchDomain: resolve searchable payload operators
JsonSearchDomain-->>Client: return SteVec query or extracted value
PostgreSQL->>JsonStorageDomain: resolve storage-only payload operator
JsonStorageDomain-->>Client: raise operator not supported
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (3)
tests/sqlx/tests/v3_uninstall_tests.rs (1)
145-160: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick winExtend uninstall/install validation to the new storage-only domain.
The split introduces two distinct public JSON domains, but these tests only validate
public.eql_v3_json_search.
tests/sqlx/tests/v3_uninstall_tests.rs#L145-L160: include bothpublic.eql_v3_jsonandpublic.eql_v3_json_searchin the installed public-domain expectations.tests/sqlx/tests/v3_uninstall_tests.rs#L211-L228: add an application column typed aspublic.eql_v3_jsonand insert a valid{v,i,c}payload.tests/sqlx/tests/v3_uninstall_tests.rs#L296-L298: assert that the storage-only column type, and its round-trip value, survive uninstall.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/sqlx/tests/v3_uninstall_tests.rs` around lines 145 - 160, Extend tests/sqlx/tests/v3_uninstall_tests.rs at lines 145-160 to expect both public.eql_v3_json and public.eql_v3_json_search. At lines 211-228, add an application column using public.eql_v3_json and insert a valid {v,i,c} payload. At lines 296-298, assert that this storage-only column retains its type and round-trip value after uninstall.tests/sqlx/src/fixtures/validation.rs (1)
145-153: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winTest coverage gap:
public.eql_v3_jsonno longer asserted as allowlisted.The rename swapped the assertion for
public.eql_v3_json_searchin place ofpublic.eql_v3_json, butpublic.eql_v3_jsonis still a distinct, live allowlist entry (storage-only fixtures use it pertargets_forinv3_convert.rs). Add back thepublic.eql_v3_jsonassertion alongside the new one so both allowlisted JSONB tokens are covered.✅ Proposed fix
assert!(ColumnType::try_from("jsonb").is_ok()); + assert!(ColumnType::try_from("public.eql_v3_json").is_ok()); assert!(ColumnType::try_from("public.eql_v3_json_search").is_ok());🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/sqlx/src/fixtures/validation.rs` around lines 145 - 153, Update column_type_accepts_allowlisted_tokens to assert that ColumnType::try_from("public.eql_v3_json") succeeds, while retaining the existing public.eql_v3_json_search assertion so both allowlisted JSONB tokens are covered.tests/sqlx/tests/v3_public_surface_tests.rs (1)
99-116: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winAdd
public.eql_v3_jsontouser_domain_names()
scalar_families()doesn’t include the non-scalarjsonbfamily, so this list now coverseql_v3_json_searchandeql_v3_jsonb_entrybut skips the storage-onlyeql_v3_jsondomain. Add it here so both public-surface placement checks keep covering the new column type.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/sqlx/tests/v3_public_surface_tests.rs` around lines 99 - 116, The user_domain_names function must also include the storage-only public.eql_v3_json domain. Extend its explicit JSON domain list alongside eql_v3_json_search and eql_v3_jsonb_entry, preserving the existing scalar-family collection and sorting behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/eql-domains/src/spec.rs`:
- Around line 28-36: Update full_name so the literal-name exception applies only
when family_name is "jsonb" and self.name is "json" or "json_search"; preserve
the existing family+suffix naming path for all other families.
In `@docs/upgrading/v3.0.md`:
- Line 16: Update the `eql_v3_json` rename entry in `packages/eql/CHANGELOG.md`
to link to the U-007 section
`docs/upgrading/v3.0.md#u-007-json-storage-only-and-searchable-document-rename`,
matching the existing changelog link style.
In `@src/v3/jsonb/blockers.sql`:
- Around line 27-285: Regenerate the bundled SQL assets so every corresponding
operator and function binding uses public.eql_v3_json_search instead of
public.eql_v3_json. Apply the same rename throughout
crates/eql-bindings/sql/cipherstash-encrypt.sql and
packages/eql/sql/cipherstash-encrypt.sql, covering the blocker definitions from
jsonb_blocked_exists through jsonb_blocked_concat_rhs and the related
functions.sql changes; preserve all signatures and behavior.
---
Outside diff comments:
In `@tests/sqlx/src/fixtures/validation.rs`:
- Around line 145-153: Update column_type_accepts_allowlisted_tokens to assert
that ColumnType::try_from("public.eql_v3_json") succeeds, while retaining the
existing public.eql_v3_json_search assertion so both allowlisted JSONB tokens
are covered.
In `@tests/sqlx/tests/v3_public_surface_tests.rs`:
- Around line 99-116: The user_domain_names function must also include the
storage-only public.eql_v3_json domain. Extend its explicit JSON domain list
alongside eql_v3_json_search and eql_v3_jsonb_entry, preserving the existing
scalar-family collection and sorting behavior.
In `@tests/sqlx/tests/v3_uninstall_tests.rs`:
- Around line 145-160: Extend tests/sqlx/tests/v3_uninstall_tests.rs at lines
145-160 to expect both public.eql_v3_json and public.eql_v3_json_search. At
lines 211-228, add an application column using public.eql_v3_json and insert a
valid {v,i,c} payload. At lines 296-298, assert that this storage-only column
retains its type and round-trip value after uninstall.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 5b3095fe-c531-438a-920a-77521aa46269
⛔ Files ignored due to path filters (6)
packages/eql/src/generated/schema-manifest.tsis excluded by!**/generated/**packages/eql/src/generated/schema/v3/eql_v3_json.jsonis excluded by!**/generated/**packages/eql/src/generated/schema/v3/eql_v3_json_search.jsonis excluded by!**/generated/**packages/eql/src/generated/v3/Json.tsis excluded by!**/generated/**packages/eql/src/generated/v3/SteVecDocument.tsis excluded by!**/generated/**packages/eql/src/generated/v3/index.tsis excluded by!**/generated/**
📒 Files selected for processing (46)
.changeset/json-storage-only-domain.md.gitignorecrates/eql-bindings/bindings/v3/Json.tscrates/eql-bindings/bindings/v3/SteVecDocument.tscrates/eql-bindings/schema/v3/eql_v3_json.jsoncrates/eql-bindings/schema/v3/eql_v3_json_search.jsoncrates/eql-bindings/src/from_v2/mod.rscrates/eql-bindings/src/from_v2/target.rscrates/eql-bindings/src/v3/inventory.rscrates/eql-bindings/src/v3/jsonb.rscrates/eql-bindings/src/v3/payload.rscrates/eql-bindings/tests/catalog_parity.rscrates/eql-bindings/tests/domain_payload.rscrates/eql-bindings/tests/from_v2.rscrates/eql-bindings/tests/v3_conformance.rscrates/eql-codegen/src/bindings.rscrates/eql-codegen/src/dump.rscrates/eql-domains/src/lib.rscrates/eql-domains/src/spec.rscrates/eql-domains/src/tests.rsdocs/upgrading/v3.0.mdsrc/v3/jsonb/blockers.sqlsrc/v3/jsonb/functions.sqlsrc/v3/jsonb/operators.sqlsrc/v3/jsonb/storage_blockers.sqlsrc/v3/jsonb/types.sqltests/sqlx/fixtures/FIXTURE_SCHEMA.mdtests/sqlx/snapshots/eql_v3_public_surface.txttests/sqlx/src/fixtures/eql_plaintext.rstests/sqlx/src/fixtures/mod.rstests/sqlx/src/fixtures/v3_convert.rstests/sqlx/src/fixtures/v3_doc_integer.rstests/sqlx/src/fixtures/v3_json_storage.rstests/sqlx/src/fixtures/v3_ste_vec.rstests/sqlx/src/fixtures/validation.rstests/sqlx/src/jsonb_entry.rstests/sqlx/tests/encrypted_domain/jsonb_entry.rstests/sqlx/tests/generate_all_fixtures.rstests/sqlx/tests/payload_schema_tests.rstests/sqlx/tests/v3_json_storage_tests.rstests/sqlx/tests/v3_jsonb_bindings_tests.rstests/sqlx/tests/v3_jsonb_operator_surface_tests.rstests/sqlx/tests/v3_jsonb_tests.rstests/sqlx/tests/v3_privilege_tests.rstests/sqlx/tests/v3_public_surface_tests.rstests/sqlx/tests/v3_uninstall_tests.rs
…ily; link U-007 Address PR review (CodeRabbit): - full_name: guard the json/json_search verbatim-name exception with family_name == "jsonb" so an unrelated future family with a colliding bare domain name still takes the normal family+suffix join (mirrors the guard in every_domain_name_starts_with_its_family_name). - changeset: cross-link the U-007 upgrade note from the changelog entry body.
The Clean-DB v3 install smoke (tasks/test/clean_install_v3.sh) still drove the
searchable SteVec document surface through public.eql_v3_json — inserting an sv
payload and running ->/@>/to_ste_vec_query on it — which now belong to
public.eql_v3_json_search (and the storage-only public.eql_v3_json CHECK rejects
sv payloads). Point the searchable block at _search, and add a storage-only
smoke: the {v,i,c} CHECK accepts a ciphertext-only payload, rejects a SteVec
document, and the native-jsonb firewall raises for the storage domain too.
This directory was outside the rename sweep's src/crates/tests/packages scope.
Fixes the "Validate (Postgres 17)" CI failure on #397. CIP-3512.
The searchable encrypted-JSON surface is now public.eql_v3_json_search; public.eql_v3_json is the new storage-only / encryption-only domain. Update the reference and tutorial docs that described the searchable surface under the old name, add storage-only mentions to json-support.md and proxy-configuration.md, and refresh the splinter allowlist descriptions for the ->/->>/@>/to_ste_vec_query functions (which now take a public.eql_v3_json_search arg).
Contributor
|
|
Contributor
Author
|
@tobyhede but |
tobyhede
added a commit
that referenced
this pull request
Jul 15, 2026
The storage-domain tests used inline {v,i,c} payloads, which technically violate the project rule that tests must use real ciphertexts, never synthetic blobs. Port PR #397's real-ciphertext approach (adapted to this branch's generated architecture and `json` module naming): - New `v3_json_storage` fixture: encrypts JSON through cipherstash-client with `.storage_only()` (no index) into a real {v,i,c} envelope, column-typed `public.eql_v3_json` so the domain CHECK runs at load. - New `v3_json_storage_tests` suite: a real client payload parses into the generated `Json` binding; `@>` on a real fixture row raises the firewall; the storage/search CHECKs are mutually exclusive on real data. - Fix `v3_convert::targets_for`: an unindexed jsonb fixture now converts to the storage-only `eql_v3_json` target (was unconditionally `eql_v3_json_search`), with a unit test pinning the bare {v,i,c} envelope. - Wire-up: allowlist `public.eql_v3_json`, register the fixture module + generation, gitignore the generated .sql. The inline jsonb_check tests are kept — they cover what fixtures can't (malformed-envelope rejection, the full blocked-operator matrix, and the path-operator fall-through characterization).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Every scalar EQL type shipped a storage-only / encryption-only variant under its bare name (
public.eql_v3_integeretc.), butjsonhad none — its only domain required a full searchable SteVec document, so there was no way to encrypt a JSON blob purely for storage. This closes that gap and alignsjsonwith the scalar convention that the bare name is the storage-only variant.public.eql_v3_jsonis now the storage-only / encryption-only domain: a plain{v, i, c}envelope with a native-jsonb operator firewall (raises on every operator, mirroring the scalar storage domains).public.eql_v3_json_searchis the searchable SteVec document — unchanged in behaviour, just renamed frompublic.eql_v3_json(all@>/<@/->/->>operators, theeql_v3.query_jsonbcast, and per-leafhm/opterms move with it).public.eql_v3_jsonb_entryandeql_v3.query_jsonbare unchanged.Fixes CIP-3512.
How it threads through the stack
src/v3/jsonb/): new storage domain +storage_blockers.sqlfirewall; searchable surface renamed to_search.eql-domains): storage json modelled as aShape::Scalardomain in the (still non-scalar)jsonbfamily; document domain renamed tojson_search;full_name/rust_struct_namemappings updated.eql-bindings): new hand-writtenJsonstruct;SteVecDocumentrepointed to_search; regenerated inventory/payload/TS/JSON Schema;from_v2conversion path updated (storage json resolves to a scalar target, SteVec doc tojson_search).v3_json_storage, via the.storage_only()path) + a dedicated SQLx suite; regenerated the public-surface golden snapshot (net change is exactly the 13 document ops moving to_search).Versioning
Ships as a patch (3.0.1), not a major — EQL v3 has no production users yet. Changeset (
patchon@cipherstash/eql) included;sync-lockstep-versionspropagates 3.0.1 toeql-bindingsand the bundled SQL at release time. Upgrade note U-007 added todocs/upgrading/v3.0.md(pre-release adopters change searchable JSON columns topublic.eql_v3_json_search).Test status
Rust crates (
eql-domains,eql-codegen,eql-bindings) and the affected SQLx suites (surface, jsonb bindings/operators, payload schema, privilege, uninstall, and the new storage suite) are green. Verified behaviour on a live Postgres: both CHECKs are mutually exclusive by payload shape, and the firewall raisesoperator … is not supported for public.eql_v3_json.Two caveats:
v3_jsonb_testsfail locally — the known SteVec fixture selector-drift (pinned constants match CI workspace material, not local); deliberately not re-pinned. They use thev3_ste_vecdocument fixture and are independent of this change.Follow-ups (tracked on CIP-3512)
Release EQL +
eql-bindings, then updateprotectjs-ffi,stack, and the stack adapters (Drizzle / Prisma / Supabase) for the renamed JSON domains.Summary by CodeRabbit
New Features
{v, i, c}envelope.Bug Fixes
Documentation