fix(sqlite): translate antlr rune offsets to byte offsets - #4492
Conversation
antlr's InputStream stores the source as a []rune and reports every token position as a rune index. The rest of sqlc treats query offsets as byte offsets into the original source string (source.Pluck slices the raw SQL, source.Mutate applies parameter and star-expansion edits). For ASCII input the two coincide, so the mismatch was invisible. Any multi-byte rune in the source (e.g. an em-dash in a comment or string literal) shifts every later byte offset, which truncated the generated SQL constant and, in queries using sqlc.arg() or `*`, applied edits at the wrong positions and produced corrupt SQL. Translate antlr's rune indices to byte offsets in the SQLite parser, so that no rune-based offset ever escapes into the compiler: build a rune-index -> byte-offset table for the source and route the statement's StmtLocation/StmtLen and every AST node's Location through it (via a single cc.pos helper). Postgres and MySQL are unaffected since they receive byte offsets from libpg_query and TiDB. Add a regression test covering a query with a multi-byte rune in a comment.
|
Carrying the same fix in a fork, and this PR matches what we ended up with. Two The test doesn't cover the convert.go half. Revert just this line in parse.go: and TestParseNonASCIIOffsets still passes. It only asserts through source.Pluck, Those two are incomplete, which is the other note: node Locations stay rune The convert.go half is also where the silent failure lives. Truncation breaks That reads ResTarget.Location. I have a test that catches it. It checks each location by slicing the source at |
Summary
Fixes a bug where SQLite code generation produced truncated or corrupt
output whenever the source
.sqlfile contained a non-ASCII character(em-dash, accented letters, etc.).
The bug
Given a query with a multi-byte character in a comment (here the
éin"café", which is 2 bytes in UTF-8):
sqlc generated a truncated SQL constant, dropping the trailing
?:In queries that use
sqlc.arg()orSELECT *, the misaligned offsetscaused edits to land at the wrong byte positions and produced SQL that
fails to parse (e.g.
SEid ...). There was no error at generate time —the corruption was silent.
Root cause
antlr's
InputStreamstores the input as a[]runeand reports alltoken positions as rune indices. The rest of sqlc, shared with the
Postgres and MySQL engines, treats query offsets as byte offsets
into the original source:
source.Pluckslices the raw SQL for each statementsource.Mutateapplies parameter / star-expansion editsFor ASCII, rune index == byte offset, so the bug never surfaced. A
multi-byte rune (em-dash = 3 bytes, 1 rune) makes every later rune
index lag its true byte offset, truncating the statement slice and
shifting every edit. Postgres and MySQL are unaffected because they get
byte offsets from libpg_query and TiDB.
The fix
Confined to the SQLite engine. We build a rune-index → byte-offset table
for the source and translate offsets at the point they are produced, so
a rune-based offset never escapes the parser:
parse.goconverts the statement'sStmtLocation/StmtLenandthreads the table into the converter (
cc.convertPos).convert.goroutes every nodeLocationthrough a singlecc.poshelper instead of using the raw antlr token offset.
By the time the AST leaves the parser, all offsets are byte offsets,
matching the invariant the other engines already satisfy. The
translation lives where the offsets originate, so there is no separate
post-processing pass and no reflection.
Testing
internal/engine/sqlite/parse_test.go(failsbefore the fix, passes after).
sqlc generateend-to-end on the query above and on a hardercase (multi-byte string literal before
sqlc.arg()params +SELECT *); both are correct after the fix, and the latter producedcorrupt SQL before it.