Story REPL.1: single source of truth for SQL keywords (highlighter/completer/parser) — elasticsql

[fupelaqu]

Story REPL.1 — Single source of truth for SQL keywords (#161)

Epic REPL (REPL hardening), Layer 0. Closes #161.

What

The REPL highlighter (28 hardcoded words) and completer (82 hardcoded words) had drifted from the parser, which actually has FOUR keyword surfaces (TokenRegex objects, keyword("…") literals, type/literal parser regexes, private reservedKeywords). LIMIT and 53 other parser keywords were not colourised.

• NEW sql/SQLKeywords — curated registry referencing the token objects themselves (compile-checked) + curated statementWords/typeWords/literalWords/timeUnitPluralWords; exposes allWords, highlightedWords (mechanical filters only: word-shaped, length ≥ 2, no geo distance units) and compoundPhrases.
• NEW sql/SQLKeywordsSpec — anti-drift source scans, now TWO-WAY: every keyword("…") literal and Expr("…") token word must be in the registry, and every statementWords entry must be literal-backed (registry cannot rot). Scans tolerate scalafmt wraps and digit-bearing literals; reservedKeywords scraped read-only (AD-5).
• NEW core/ReplKeywords — the ONE set both REPL components consume: SQLKeywords.highlightedWords ++ clearly-labelled REPL-only extraWords (INTERSECT/EXPLAIN/BULK/CONDITION/ACTION/GEO/TRANSFORM — kept per PD-2, disjointness test-guarded).
• ReplHighlighter/ReplCompleter rewired to ReplKeywords.all; highlight/complete bodies unchanged byte-for-byte; compoundKeywords extended (PARTITION/FULL/CROSS/NULLS/IS).
• NEW core/ReplKeywordsSpec — registry superset, extras disjointness, the issue's 54-word list pinned verbatim, legacy 28-word regression pin, highlight-style and completer-candidate behaviour tests.
• Version 0.20.1 → 0.20.2-SNAPSHOT (epic kickoff).

Adding a keyword from now on

• New clause/function token → TokenRegex object per convention and add it to SQLKeywords.clauseTokens/functionTokens (forgetting fails the Expr-scan test).
• New DDL/DML word → keyword("…") and SQLKeywords.statementWords (two-way scan enforces both directions).
• New literal-value token (extends Value[...] with TokenRegex) → add to SQLKeywords.literalTokens by hand (scan blind spot, documented).
• REPL-only suggestion → ReplKeywords.extraWords (kept disjoint from the registry by test).

Candidate follow-up (not in scope)

The highlighter's green string-literal branch is dead at baseline: split("\\b") fragments 'John' so '[^']*' never matches a real quoted string. Behaviour pinned as-is (AC 4 = no regression); making strings green requires a tokenizer change → separate issue.

Verification

• sbt "+ sql/test" "+ core/test" — sql 385, core 619, green on 2.12.20 AND 2.13.16; all anti-drift scans executed (not cancelled)
• sbt scalafmtAll, sbt headerCheck — clean
• Zero parser-side changes (ParserSpec untouched and green); Repl.scala wiring unchanged
• Code review (3-layer adversarial): 4 patch findings applied (test-scan hardening + comment fix), 0 unresolved HIGH/MED

🤖 Generated with Claude Code

Part of epic #169 (REPL Hardening).