Expose schema-aware semantic analysis through the C API

[fc-anjos]

Expose schema-aware semantic analysis through the C API

Summary

• add C ABI entry points for column qualification, scope analysis, and requested output-column lineage;
• accept the existing JSON AST plus an ordered schema transport matching MappingSchema's table paths;
• parse schema type strings with the SQL parser rather than maintaining a second type grammar;
• return explicit JSON views of the Rust semantic result types;
• cover qualification, CTE scopes, and output lineage through the public FFI.

Motivation

Language bindings currently expose parsing, generation, and transpilation, but cannot use the schema-aware optimizer APIs already implemented by sql-glot-rust. Consumers consequently need to maintain a second native extension to call MappingSchema, qualify_columns, build_scope, and lineage.

These entry points keep the same JSON boundary used by the existing parse and generate functions. They expose the existing Rust concepts directly rather than adding a consumer-specific combined analyzer. Python sqlglot also supports requesting every top-level output lineage in one call; the Rust API currently builds lineage for one requested output column, and this C ABI keeps that single-result contract.

Schema JSON uses explicit identifier paths and ordered columns:

{
  "tables": [
    {
      "path": ["catalog", "schema", "cards"],
      "columns": [
        {"name": "id", "data_type": "BIGINT"},
        {"name": "amount", "data_type": "DECIMAL(18, 4)"}
      ]
    }
  ]
}

Paths are arrays so quoted identifiers containing dots are not confused with catalog or schema separators. Columns are arrays so wildcard expansion retains schema definition order. Type strings use the parser's data-type grammar, including lengths, precision, scale, timezone modifiers, arrays, and dialect-specific types.

The parser entry point is also exposed as parse_data_type(sql, dialect) -> Result<DataType>, analogous to Python sqlglot's DataType.from_str, so other Rust callers do not need to embed a type inside a synthetic statement.

The scope transport uses stable lowercase scope_type and tagged source objects ({"kind":"table", ...} / {"kind":"scope", ...}) instead of serializing Debug output.

API

char *sqlglot_qualify_columns(
  const char *ast_json,
  const char *schema_json,
  const char *dialect
);

char *sqlglot_build_scope(const char *ast_json);

char *sqlglot_lineage(
  const char *column,
  const char *ast_json,
  const char *schema_json,
  const char *config_json
);

sqlglot_lineage requires a non-NULL output column and returns one lineage graph. An all-output lineage operation would require a separate Rust API and C entry point so this function's input and result shapes remain stable.

All successful calls return heap-allocated JSON and use the existing sqlglot_free ownership contract. Invalid input or semantic errors return NULL, matching the existing C API.

Verification

cargo fmt --check is clean, and the full test suite and doctests pass with cargo test --features cli.

[yigalrozenberg]

Thank you, @fc-anjos! 🙏

This is a great addition — exposing the schema-aware semantic APIs (qualify_columns, build_scope, lineage) plus a standalone parse_data_type through the C ABI closes a real gap for language bindings that previously had to maintain a second native extension. I especially appreciated the careful design: reusing the parser's data-type grammar instead of a second type grammar, the path-array schema transport so quoted identifiers with dots aren't split, stable tagged source objects instead of Debug output, and keeping the existing sqlglot_free ownership contract with NULL on invalid input.

Verified locally before merging:

• Full test suite green with cargo test --features cli (all bins + doctests, 0 failures), including your new FFI coverage for qualification, CTE scopes, and output lineage — no regressions.
• cargo fmt --check clean; no new clippy diagnostics in the added code.
• cargo publish --dry-run clean.

Merged into master and released as v0.10.25, now live on crates.io: https://crates.io/crates/sqlglot-rust/0.10.25

Thanks again for the thorough PR and clear write-up! 🚀