Databricks: support CREATE TABLE USING, MAP<K, V>, LONG, DIV and the pipe operator - #2425
Databricks: support CREATE TABLE USING, MAP<K, V>, LONG, DIV and the pipe operator#2425shuvamk wants to merge 2 commits into
Conversation
… BIGINT
Databricks SQL is built on Spark SQL, but DatabricksDialect never got
three of the flags SparkSqlDialect sets, so these fail on Databricks and
parse on Spark:
CREATE TABLE t (id BIGINT) USING DELTA
CREATE TABLE t (m MAP<STRING, INT>)
with "Expected: end of statement, found: USING at Line: 1, Column: 28"
and "Expected: ',' or ')' after column definition, found: < at Line: 1,
Column: 22".
CREATE TABLE t (id LONG) parses, but builds DataType::Custom("LONG")
instead of DataType::BigInt(None). Databricks documents the type as
{ BIGINT | LONG }.
Set supports_create_table_using, supports_long_type_as_bigint and
supports_map_literal_with_angle_brackets on DatabricksDialect, mirroring
src/dialect/spark.rs. No parser change and no other dialect is affected.
Regression tests in tests/sqlparser_databricks.rs; all three fail with
src/dialect/databricks.rs reverted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
|
||
| databricks().verified_stmt("CREATE TABLE IF NOT EXISTS t (id BIGINT) USING PARQUET"); | ||
|
|
||
| assert!(all_dialects_where(|d| !d.supports_create_table_using()) |
There was a problem hiding this comment.
Consider adding the analogous assert also for the other flags you have set
|
|
||
| databricks().verified_stmt("CREATE TABLE IF NOT EXISTS t (id BIGINT) USING PARQUET"); | ||
|
|
||
| assert!(all_dialects_where(|d| !d.supports_create_table_using()) |
There was a problem hiding this comment.
Also, I believe that since this assertion regards all dialects except databricks, it should not be in the databricks tests, but in common.
| fn supports_long_type_as_bigint(&self) -> bool { | ||
| true | ||
| } | ||
|
|
There was a problem hiding this comment.
While you are adding support for Spark flags in Databricks, I believe you should also add supports_pipe_operator and parse_infix for DIV.
| #[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))] | ||
| pub struct DatabricksDialect; | ||
|
|
||
| impl Dialect for DatabricksDialect { |
There was a problem hiding this comment.
I believe there is some argument to be made that, since several of these properties need to be kept aligned between Spark and Databricks, it may be desirable to call directly Spark methods in these methods instead of duplicating the scalar value.
Address review feedback on apache#2425. The flags Databricks shares with Spark now call the Spark methods instead of repeating the literal, so the two dialects cannot drift apart silently. `RedshiftSqlDialect` already delegates to `PostgreSqlDialect` this way. Two more Spark capabilities are documented for Databricks SQL and were missing: SELECT 10 div 3 SELECT * FROM t |> WHERE x > 1 |> SELECT x AS y Both failed with `No infix parser for token Word(... keyword: DIV })` and `Expected: end of statement, found: |`. `div` is documented at https://docs.databricks.com/aws/en/sql/language-manual/functions/div and pipeline syntax at https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-qry-pipeline (Databricks SQL / Databricks Runtime 16.2 and above). The cross-dialect assertion moves from the Databricks tests to `tests/sqlparser_common.rs`, joined by the analogous assertions for the `MAP<K, V>` and `LONG` flags. Dialects that do not set `supports_map_literal_with_angle_brackets` reject `MAP<STRING, INT>` with two different messages, so that one is checked per dialect as in `parse_create_table_exclude_constraint`. Dialects that do not set `supports_long_type_as_bigint` parse `LONG` as a custom type rather than erroring, so the assertion there is that the statement round-trips unchanged. `parse_pipeline_operator_negative_tests` asserted that `|> CALL 123invalid` fails with one message across every pipe-enabled dialect. Databricks sets `supports_numeric_prefix`, so it reads `123invalid` as an identifier and fails later, on the missing parentheses. The input is now `|> CALL 123`, which fails identically on all of them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
shuvamk
changed the title
|
Thanks — all four addressed in the follow-up commit.
On delegating the flags: I converted the ones Spark and Databricks share, but left The cross-dialect asserts moved to Title and description updated for the added scope. |
2 participants
Databricks SQL is built on Spark SQL, but
DatabricksDialectwas missing flagsSparkSqlDialectsets. These fail onDatabricksDialectand parse onSparkSqlDialect:CREATE TABLE t (id LONG)parses, but as aDataType::Custom, notDataType::BigInt(None).divand pipeline syntax are documented for Databricks SQL:https://docs.databricks.com/aws/en/sql/language-manual/functions/div
https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-qry-pipeline
The methods shared with Spark delegate to
SparkSqlDialectrather than repeating the value, the same shapeRedshiftSqlDialectuses forPostgreSqlDialect.parse_pipeline_operator_negative_testsnow uses|> CALL 123instead of123invalid: enabling the pipe operator brings Databricks into that assertion, andsupports_numeric_prefixmakes it fail at a different point. Both inputs fail identically on the dialects already covered.Tests are in
tests/sqlparser_databricks.rsandtests/sqlparser_common.rs; the five Databricks ones fail withsrc/dialect/databricks.rsreverted. TheAGENTS.mdpre-commit checks are clean locally.