Add searchable fallback for unicode61 tokens longer than 1000 characters

[Widthdom]

Summary

CodeIndex now detects the FTS5 unicode61 1000-character token limit and reports fts_token_too_long, but the affected token remains unsearchable through cdidx search.

Please add a bounded search fallback or recovery index for content inside very long unbroken tokens. This is a follow-up to #1606: that issue successfully made the limitation visible; this request is about making the content searchable.

Current behavior

With cdidx v1.42.0, indexing a repository containing a generated/minified HTML report succeeds. Running cdidx validate reports:

fts_token_too_long doc/samples/diff_report.html:1410
Line 1410 contains an FTS5 unicode61 token longer than 1000 characters;
that token is not searchable through FTS

The diagnostic is useful and accurate, but users and coding agents still cannot discover a known term when it occurs only inside that long token via the normal search workflow.

Reproduction

cdidx index .
cdidx validate
cdidx search "<a term that occurs only inside the reported long token>"

A realistic input is a generated or minified HTML/JavaScript line containing an unbroken token longer than 1000 characters.

Expected behavior

At least one supported search path should recover bounded matches inside the long token without requiring the entire token as the query. Normal unicode61 behavior and performance for ordinary source files should remain unchanged.

Possible directions

• Index bounded overlapping chunks for only the tokens that exceed the FTS5 limit.
• Maintain a small fallback column/table for long-token recovery.
• Fall back to a bounded literal scan of candidate chunks when the normal FTS query returns no match.
• If recovery is intentionally out of scope, make the diagnostic and no-result hint point to a supported command such as cdidx find that can search the affected content.

Please add a regression fixture with a >1000-character minified HTML or JavaScript token and verify that a short literal from its middle can be found.

Environment

• CodeIndex: v1.42.0 (commit 35067ab)
• Platform: Windows
• Observed in a generated HTML report fixture

Related

• FTS5 unicode61 silently drops tokens longer than 1000 chars #1606 added detection/documentation for the FTS5 token-length limit. This issue requests search recovery after that diagnostic.

[Widthdom]

Work started on branch fix-issue5080.