Refactor Collection Table Matcher by amorton · Pull Request #2499 · stargate/data-api

amorton · 2026-06-11T02:10:40Z

What this PR does:

Refactor the ColectionTableMatcher into the SuperShreddingTablePredicate , and create a standard way to define what is in a SuperShredding table, via the SuperShreddingBuilder. We can then use this builder to create different components we need, such as predciate and cql statements to create the table.

This PR sets a foundation by putting the definition of super shredding in one place, and building a way to test / verify we are creating CQL we expect. With the idea that above that low level CQL testing verification we run tests against the tableMetadata etc.

This PR only updates the table matcher, next PR's will update how CreateCollectionOperation and related classes work to make a super shredding table.

Checklist

Changes manually tested
Automated Tests added/updated
Documentation added/updated
CLA Signed: DataStax CLA

…-backward-compatibility

makes CreateCollectionBackwardCompatibilityIntegrationTest work

amorton · 2026-06-11T02:15:24Z


  public static String errFmt(DataType dataType) {
-    return nullSafe(dataType, d -> d.asCql(true, true));
+    return nullSafe(dataType, d -> d.asCql(false, true));


change to not force frozen when it was not needed

amorton · 2026-06-11T02:18:03Z

-    /** Physical table column name that stores the lexical content. */
-    String LEXICAL_INDEX_COLUMN_NAME = "query_lexical_value";
-  }
-


Moved all names of columns to be in SuperShreddingMetadata

github-actions · 2026-06-12T02:43:51Z

📈 Unit Test Coverage Delta vs Main Branch

Metric	Value
Main Branch	52.50%
This PR	52.97%
Delta	🟢 +0.47%
✅ Coverage improved!

github-actions · 2026-06-12T02:43:53Z

Unit Test Coverage Report

Overall Project	52.97% `-0.4%`	🍏
Files changed	83.04%	🍏

File	Coverage
SuperShreddingMetadataBuilder.java	100%	🍏
ColumnMetadataPredicate.java	100%	🍏
CollectionIndexUsage.java	100%	🍏
ExtendedVectorType.java	100%	🍏
DocumentConstants.java	100%	🍏
SuperShreddingCQL.java	98.86% `-1.14%`	🍏
SuperShreddingCQLBuilder.java	95.18% `-4.82%`	🍏
AllCollectionFilter.java	91.8%	🍏
SuperShreddingMetadata.java	89.18% `-10.82%`	🍏
CqlIdentifierUtil.java	86.67%	🍏
SuperShreddingBinding.java	85.88% `-14.12%`	🍏
SuperShreddingBuilder.java	84.16% `-15.84%`	🍏
FindCollectionOperation.java	80% `-0.4%`	❌
MapCollectionFilter.java	77.66% `-3.9%`	❌
CQLSAIIndex.java	71.53% `-5.11%`	🍏
CreateCollectionOperation.java	68.76%	🍏
ErrorFormatters.java	65.36%	🍏
StringUtil.java	60% `-5.71%`	🍏
ApiIndexFunction.java	58.44% `-23.38%`	❌
SuperShreddingTablePredicate.java	42.57% `-57.43%`	❌
FindCollectionsCollectionOperation.java	22.46%	🍏
InCollectionFilter.java	13.47% `-4.15%`	❌
CollectionSchemaObject.java	8.82% `-1.05%`	❌
SchemaObjectFactory.java	6.25% `-1.47%`	❌
SuperShreddingPredicateBuilder.java	0%	❌
MatchCollectionFilter.java	0% `-11.48%`	❌
MetadataDBTask.java	0% `-2.2%`	❌

github-actions · 2026-06-12T03:03:15Z

📉 Integration Test Coverage Delta vs Main Branch (dse69-it)

Metric	Value
Main Branch	72.50%
This PR	71.43%
Delta	🔴 -1.07%
⚠️ Coverage decreased

github-actions · 2026-06-12T03:03:17Z

Integration Test Coverage Report (dse69-it)

Overall Project	71.43% `-1.56%`	🍏
Files changed	34.38%	❌

File	Coverage
CollectionIndexUsage.java	100%	🍏
ExtendedVectorType.java	100%	🍏
FindCollectionsCollectionOperation.java	94.2%	🍏
AllCollectionFilter.java	91.8%	🍏
MetadataDBTask.java	90.11%	🍏
FindCollectionOperation.java	88.48% `-0.13%`	🍏
SchemaObjectFactory.java	87.5%	🍏
ErrorFormatters.java	86.79%	🍏
DocumentConstants.java	78.57%	🍏
CreateCollectionOperation.java	75.29%	🍏
CQLSAIIndex.java	73.72% `-14.6%`	❌
MapCollectionFilter.java	68.79%	🍏
ApiIndexFunction.java	64.94% `-35.06%`	❌
CollectionSchemaObject.java	63.38%	🍏
CqlIdentifierUtil.java	62.22% `-8.89%`	❌
InCollectionFilter.java	59.33%	🍏
SuperShreddingMetadata.java	56.96% `-43.04%`	❌
ColumnMetadataPredicate.java	42.44% `-57.56%`	❌
SuperShreddingTablePredicate.java	40.56% `-59.44%`	❌
StringUtil.java	25.71% `-34.29%`	❌
SuperShreddingBinding.java	15.88% `-84.12%`	❌
SuperShreddingCQLBuilder.java	0%	❌
SuperShreddingBuilder.java	0%	❌
SuperShreddingCQL.java	0%	❌
SuperShreddingMetadataBuilder.java	0%	❌
SuperShreddingPredicateBuilder.java	0%	❌
MatchCollectionFilter.java	0% `-11.48%`	❌

github-actions · 2026-06-14T20:33:43Z

📉 Integration Test Coverage Delta vs Main Branch (hcd-it)

Metric	Value
Main Branch	73.84%
This PR	72.75%
Delta	🔴 -1.09%
⚠️ Coverage decreased

github-actions · 2026-06-14T20:33:44Z

Integration Test Coverage Report (hcd-it)

Overall Project	72.75% `-1.55%`	🍏
Files changed	34.7%	❌

File	Coverage
CollectionIndexUsage.java	100%	🍏
ExtendedVectorType.java	100%	🍏
FindCollectionOperation.java	94.3%	🍏
FindCollectionsCollectionOperation.java	94.2%	🍏
AllCollectionFilter.java	91.8%	🍏
MetadataDBTask.java	90.11%	🍏
SchemaObjectFactory.java	87.5%	🍏
ErrorFormatters.java	86.79%	🍏
DocumentConstants.java	78.57%	🍏
CreateCollectionOperation.java	78.33%	🍏
CQLSAIIndex.java	73.72% `-14.6%`	❌
MapCollectionFilter.java	68.79%	🍏
CollectionSchemaObject.java	65.92%	🍏
ApiIndexFunction.java	64.94% `-35.06%`	❌
CqlIdentifierUtil.java	62.22% `-8.89%`	❌
InCollectionFilter.java	59.33%	🍏
SuperShreddingMetadata.java	56.96% `-43.04%`	❌
ColumnMetadataPredicate.java	42.44% `-57.56%`	❌
SuperShreddingTablePredicate.java	40.56% `-59.44%`	❌
MatchCollectionFilter.java	39.34%	🍏
StringUtil.java	25.71% `-34.29%`	❌
SuperShreddingBinding.java	15.88% `-84.12%`	❌
SuperShreddingCQLBuilder.java	0%	❌
SuperShreddingBuilder.java	0%	❌
SuperShreddingCQL.java	0%	❌
SuperShreddingMetadataBuilder.java	0%	❌
SuperShreddingPredicateBuilder.java	0%	❌

clun

Here are some findings and hopefully helpful new classes as a review

clun · 2026-06-15T14:25:37Z

+  protected final DataType type;
+
+  protected ColumnMetadataPredicate(CqlIdentifier name, DataType type) {
+    // no null checks in the ctor, so a subclass can fully override if they want to.


ok subclasses can fully override but they would also have to override the ToString or you raise a NPE.

i would not make name nor type Objects.requireNonNull() in the toString() or still add the test in the ctor

udpated the toString() - when it calls errFmt those functions handle null so no need to check there, they will print null.

removed code duplication by moving the null check into the super class, there are only 2 direct subclasses that need to worry about the checkNulls param

clun · 2026-06-15T14:32:29Z

+  private final List<ColumnMetadataPredicate> strictMatch;
+
+  // A def that represents the rules used by the old `CollectionTableMatcher`
+  private static final SuperShreddingBinding BACKWARDS_COMPAT =


SuperShreddingBinding.builder().build()

It seems you are putting all default builder values, now i would understand you want to make them explicitely visible at this level

not sure if this is a half finished comment.

The BackwardsCompat is there to capture past behaviour for the no param ctor to use for backwards compat.

So the new builder code to create the Predicate is only used by tests, until we start changing the tests. Such as also tests for indexes.

The goal in this PR was to change implementation, but leave behaviour the same.

clun · 2026-06-15T15:39:41Z

+ *
+ * <p>This class used to be called <code>CollectionTableMatcher</code>
+ *
+ * <p><b>NOTE:</b> As of June 2026, there is no check the indexes are valid, this will be future


Got it, but let me see if i can create a baby index validator after the column valudation in SchemaObjectFactory

The predicate checks columns but not indexes. A collection with correct columns but missing/broken SAI indexes passes the predicate and is treated as healthy -- downstream queries then fail with confusing CQL errors.

I've prototyped a SuperShreddingIndexValidator on branch clun/review-2499-index-validator:

SuperShreddingIndexValidator -- validates SAI indexes after the predicate passes, returns a ValidationResult with missing/present/unexpected indexes

SuperShreddingIndexValidatorTest -- 13 tests covering healthy tables, missing required/optional indexes, unexpected indexes, and warn logging

TableMetadataTestUtil additions -- removeIndex, removeAllIndexes, clearAllIndexes, addIndex

SchemaObjectFactory sketch -- shows where to wire validate() after IS_COLLECTION_PREDICATE.test() (old code kept as comment)

Branch: clun/review-2499-index-validator (based on this PR's branch)

thanks, will have a look after we get this one in. For this PR I wanted to keep the logic of the predicate the same but change implementation - so all previously passed tests keep passing.

Future work can expand the logic in the predicate

Hazel-Datastax and others added 30 commits May 4, 2026 17:39

fix and testes

b025ad1

refactor

5fd7368

formate

a2a0527

fix

57f2931

fix dse it

ab1cbdc

Merge branch 'main' into hazel/fix-lexical-rerank-disable-then-enable…

38a014b

…-backward-compatibility

fix the string

e6f317a

change to 2 methods

303b8b8

refactor

59bcd72

Merge branch 'main' into hazel/fix-lexical-rerank-disable-then-enable…

a169874

…-backward-compatibility

refactor

d1d0875

WIP- initial fix

3238b21

makes CreateCollectionBackwardCompatibilityIntegrationTest work

missed

2b28a27

Fix unit tests

a5c2849

fmt

38adaea

test fixes

7d64637

test fixes

14f9c93

handle disabled features

d70191b

bug fixes

55767b7

test fixes

d0c3e76

test fixes

1e44aa8

fmt

b082ea7

test fixes

1d9cb6d

pr review

ccbc30a

test fixes

28c387d

test fixes for escaping

a84a077

code tidy

01ba971

code tidy

c5bb15d

tidy, unit tests SchemaHolderAndFactoryTest

422f4c2

Merge branch 'main' into ajm/fix-lexical-rerank-compat

0feb770

amorton added 6 commits June 3, 2026 12:02

WIP

61eb5a0

WIP

dfe3e61

WIP - test for 2 builders work

b793528

WIP code tidy

22ed397

missing file

a9c8058

Merge branch 'main' into ajm/refactor-collection-table-matcher

466b6d3

amorton requested a review from a team as a code owner June 11, 2026 02:10

amorton marked this pull request as draft June 11, 2026 02:11

removed bob things

0b6df9d

amorton commented Jun 11, 2026

View reviewed changes

amorton added 2 commits June 11, 2026 16:44

code tidy pre PR

cc17f7a

code tidy

0e8dc95

amorton marked this pull request as ready for review June 14, 2026 20:07

clun reviewed Jun 15, 2026

View reviewed changes

fix from PR

e34aeea

amorton changed the title ~~DRAFT - Refactor Collection Table Matcher~~ Refactor Collection Table Matcher Jun 17, 2026

amorton added 3 commits June 17, 2026 13:51

Cleanup from review

22dff88

Merge branch 'main' into ajm/refactor-collection-table-matcher

1b1d958

PR fixes

1d5db50

amorton mentioned this pull request Jun 17, 2026

Refactor for tests associated with Create Collection #2512

Open

4 tasks

erichare self-requested a review June 17, 2026 19:56

erichare approved these changes Jun 17, 2026

View reviewed changes

tatu-at-datastax approved these changes Jun 17, 2026

View reviewed changes

Conversation

amorton commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📈 Unit Test Coverage Delta vs Main Branch

Uh oh!

github-actions Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Unit Test Coverage Report

Uh oh!

github-actions Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📉 Integration Test Coverage Delta vs Main Branch (dse69-it)

Uh oh!

github-actions Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integration Test Coverage Report (dse69-it)

Uh oh!

github-actions Bot commented Jun 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📉 Integration Test Coverage Delta vs Main Branch (hcd-it)

Uh oh!

github-actions Bot commented Jun 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integration Test Coverage Report (hcd-it)

Uh oh!

clun left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

amorton commented Jun 11, 2026 •

edited

Loading

github-actions Bot commented Jun 12, 2026 •

edited

Loading

github-actions Bot commented Jun 12, 2026 •

edited

Loading

github-actions Bot commented Jun 12, 2026 •

edited

Loading

github-actions Bot commented Jun 12, 2026 •

edited

Loading

github-actions Bot commented Jun 14, 2026 •

edited

Loading

github-actions Bot commented Jun 14, 2026 •

edited

Loading