feat(parser): match Python SQLGlot data type parsing

[fc-anjos]

Motivation

parse_data_type interprets standalone type strings, including types supplied to schema-aware qualification and lineage. Real database schemas contain dialect-specific spellings that are not always recognized.

For example, character varying is a standard PostgreSQL spelling for VARCHAR. On current master, this call returns a parse error:

parse_data_type("character varying", Dialect::Postgres)

A single type-parsing failure also prevents an FFI mapping schema from being constructed, so qualification and lineage cannot use an otherwise valid schema.

Currently, sql-glot-rust has no dedicated parser for standalone data types. parse_data_type delegates to the type grammar inside the general SQL parser. That grammar handles a fixed set of common types, but it does not apply comprehensive dialect-specific type mappings. It also has no explicit fallback for qualified user-defined type names.

Python SQLGlot already defines both behaviors through DataType.from_str:

from sqlglot import exp

postgres_type = exp.DataType.from_str(
    "character varying",
    dialect="postgres",
)
assert postgres_type.this is exp.DType.VARCHAR

custom_type = exp.DataType.from_str(
    "app.custom_type",
    dialect="postgres",
    udt=True,
)
assert custom_type.this is exp.DType.USERDEFINED

The dialect argument controls how SQL type names are interpreted. The udt (user-defined type) argument controls what happens when the input is not a recognized built-in type. It defaults to False, which produces an error, while True preserves a valid type name as a user-defined type.

This PR introduces a dedicated standalone type parser that follows these semantics across the dialects shared with Python SQLGlot. Both Rust entry points take an explicit dialect. parse_data_type performs strict parsing, while parse_data_type_with_udt additionally preserves an otherwise unrecognized valid type name as a user-defined type.

Summary

• add a dedicated dialect-aware data type parser modeled on Python SQLGlot's DataType.from_str(..., udt:)
• represent all 131 Python DType variants with their canonical serialized names
• preserve recognized dialect type SQL and user-defined types instead of collapsing them to UNKNOWN
• use the same parser for semantic FFI mapping schemas so valid dialect types and user-defined types do not invalidate the schema
• add a generated Python parity corpus covering every shared dialect plus focused grammar and user-defined type cases

Reference behavior

The parity fixture is generated from Python SQLGlot at commit 6b12cb8444816377fe7eb8b27d492fc117e3409b.

It contains 3,681 cases across ANSI and the 29 named dialects shared by the Python and Rust implementations. Dialect support for user-defined types follows Python SQLGlot's definitions at that commit.

The fixture uses normalized JSON Lines with one metadata record followed by one case per line. Normal Rust tests remain self-contained and do not require Python.

The parity tool provides explicit commands for writing and verifying the fixture:

python scripts/python_type_parity.py generate \
  --sqlglot /path/to/sqlglot \
  --fixture tests/fixtures/python_type_parity.jsonl

python scripts/python_type_parity.py verify \
  --sqlglot /path/to/sqlglot \
  --fixture tests/fixtures/python_type_parity.jsonl

The verify command regenerates the corpus from the supplied Python SQLGlot checkout and fails if it differs from the committed fixture. A separate Rust test records the user-defined type policy for every shared dialect.

The fixture asserts the resulting canonical DataTypeKind name, or that parsing fails. It does not claim complete AST parity for parameters, nested arguments, preserved spelling, or generated SQL.

Public API

parse_data_type(sql, dialect)
parse_data_type_with_udt(sql, dialect)

Python exposes user-defined type handling as an optional udt=False argument on DataType.from_str. Rust does not support optional function arguments, and changing the existing two-argument parse_data_type function would break its callers. The existing function therefore remains the strict form, equivalent to Python's default. The with_udt variant always enables the fallback. Both entry points use the same parser and require an explicit dialect.

For semantic schema input, the internal caller chooses whether to enable the fallback from the dialect's user-defined type policy.

Test strategy: open for discussion

This draft uses a committed fixture generated from a pinned Python SQLGlot checkout. Before proposing it upstream, I would appreciate feedback on what should constitute the long-term Rust contract.

Strategy	Committed tests	Role of Python	What normal cargo test enforces
1. Generated parity oracle (current draft)	The normalized 3,681-case JSON Lines fixture and its Rust test harness	Defines the expected type-kind/error result at a pinned commit	Exact agreement with every committed Python result
2. Manually maintained Rust cases	Ordinary, reviewed Rust test tables for canonical types, dialect aliases and overrides, user-defined type policy, grammar families, and FFI behavior	Informs the initial cases but is not part of the repository or test workflow	Only the explicit behavior maintained by this project
3. Rust cases plus a separate differential audit	The same manually maintained Rust cases as strategy 2	An optional or scheduled tool compares Rust with a pinned Python checkout and reports differences for review	The explicit Rust contract; reported Python differences are not automatically normative

Strategy 2 most closely follows the existing style in this repository, including the Rust-owned cases in tests/test_transpile.rs. It would require expanding the focused tests in this draft before removing the generated fixture; the current representative cases are not an equivalent replacement.

Strategy 3 adds broader drift discovery without turning every incidental Python classification into a permanent Rust API guarantee. It carries additional tooling and maintenance compared with strategy 2.

Strategy 1 provides the broadest immediate regression gate and requires no Python installation during normal Rust CI, but it commits a generated artifact and makes the pinned Python classifications normative for this surface.

Validation

• cargo fmt --all -- --check
• cargo test --features cli
• python scripts/python_type_parity.py verify against the pinned Python SQLGlot checkout
• generated Python parity fixture: 3,681 cases
• explicit user-defined type policy coverage for all 30 shared dialects
• semantic FFI regression coverage for dialect types, arrays, and user-defined types
• existing transpilation and doctest suites

cargo clippy --all-targets --features cli -- -D warnings still reports warnings that are also present on master with Rust 1.97. The files and ranges changed by this PR do not introduce additional clippy diagnostics.

[fc-anjos]

Hello @yigalrozenberg!

Looking for your input on this one, do you think this feature and API in general and the testing strategy in particular makes sense given the long term goals of the project, or should I adapt it to something else before marking it as ready for review?

Thank you!