[EPIC] Implement all Spark date/time expressions

[andygrove]

What is the problem the feature request solves?

This epic tracks support for the full set of Spark date/time expressions in Comet. Each expression below has a checkbox indicating whether it is implemented today. In-flight work is linked to the relevant PR. Versions in parentheses indicate the Spark version that introduced the expression; unmarked entries are present in all supported Spark versions (3.4 / 3.5 / 4.0 / 4.1).

The canonical list is derived from FunctionRegistry.scala in Spark 4.0.1 and Spark 4.1.1 (the // datetime functions section), augmented with the TimeType expressions defined in timeExpressions.scala (4.1+).

Date / Timestamp functions

• AddMonths (add_months) - feat: expand date/time expression support using codegen dispatcher #4417
• CurrentDate (current_date, curdate) - typically constant-folded by Spark before plan rewriting; only relevant when constant folding is disabled (fix: Support scalar inputs for datetime expressions when constant folding is disabled #3448)
• CurrentTimestamp (current_timestamp) - typically constant-folded; see above
• Now (now) - typically constant-folded; see above
• CurrentTimeZone (current_timezone)
• LocalTimestamp (localtimestamp)
• DateDiff (datediff, date_diff)
• DateAdd (date_add, dateadd)
• DateFormatClass (date_format)
• DateSub (date_sub)
• DayOfMonth (day, dayofmonth)
• DayOfYear (dayofyear)
• FromUnixTime (from_unixtime)
• FromUTCTimestamp (from_utc_timestamp)
• Hour (hour)
• LastDay (last_day)
• Minute (minute)
• Month (month)
• MonthsBetween (months_between) - feat: expand date/time expression support using codegen dispatcher #4417
• NextDay (next_day)
• Quarter (quarter)
• Second (second)
• ParseToTimestamp (to_timestamp) - partial via feat: add support for GetTimestamp, parse_to_date expressions #3467
• ParseToTimestampNTZ (to_timestamp_ntz)
• ParseToTimestampLTZ (to_timestamp_ltz)
• ParseToDate (to_date) - feat: add support for GetTimestamp, parse_to_date expressions #3467
• ToUnixTimestamp (to_unix_timestamp) - feat: expand date/time expression support using codegen dispatcher #4417
• ToUTCTimestamp (to_utc_timestamp)
• TruncDate (trunc)
• TruncTimestamp (date_trunc)
• UnixTimestamp (unix_timestamp)
• DayOfWeek (dayofweek)
• WeekDay (weekday)
• DayName (dayname) (3.5+)
• WeekOfYear (weekofyear)
• Year (year)
• MakeDate (make_date)
• MakeTimestamp (make_timestamp) - feat: expand date/time expression support using codegen dispatcher #4417
• MakeTimestampNTZ (make_timestamp_ntz) - [Feature] Support Spark expression: make_timestamp_ntz #3108
• MakeTimestampLTZ (make_timestamp_ltz)
• TryMakeTimestamp (try_make_timestamp) - [Feature] Support Spark expression: try_make_timestamp #3117
• TryMakeTimestampNTZ (try_make_timestamp_ntz)
• TryMakeTimestampLTZ (try_make_timestamp_ltz)
• MonthName (monthname) (4.0+)
• Extract (extract(field FROM source))
• DatePart (date_part, datepart)
• DateFromUnixDate (date_from_unix_date)
• UnixDate (unix_date)
• SecondsToTimestamp (timestamp_seconds)
• MillisToTimestamp (timestamp_millis) - feat: expand date/time expression support using codegen dispatcher #4417
• MicrosToTimestamp (timestamp_micros) - feat: expand date/time expression support using codegen dispatcher #4417
• UnixSeconds (unix_seconds) - feat: expand date/time expression support using codegen dispatcher #4417
• UnixMillis (unix_millis) - feat: expand date/time expression support using codegen dispatcher #4417
• UnixMicros (unix_micros) - feat: expand date/time expression support using codegen dispatcher #4417
• ConvertTimezone (convert_timezone)

Interval construction

• MakeInterval (make_interval) - [Feature] Support Spark expression: make_interval #3099
• MakeDTInterval (make_dt_interval) - feat: Support Spark expression: make_dt_interval #4338
• MakeYMInterval (make_ym_interval) - [Feature] Support Spark expression: make_ym_interval #3100
• TryMakeInterval - [Feature] Support Spark expression: try_make_interval #3103

Timestamp arithmetic (related catalyst expressions)

• TimestampAdd (timestampadd) - [Feature] Support Spark expression: timestamp_add #3113
• TimestampAddInterval - [Feature] Support Spark expression: timestamp_add_interval #3114
• TimestampAddYMInterval - [Feature] Support Spark expression: timestamp_add_ym_interval #3115
• TimestampDiff (timestampdiff) - [Feature] Support Spark expression: timestamp_diff #3116
• SubtractDates - [Feature] Support Spark expression: subtract_dates #3094
• SubtractTimestamps - [Feature] Support Spark expression: subtract_timestamps #3112
• DateTimeSub - [Feature] Support Spark expression: datetime_sub #3134

Time-window expressions

• TimeWindow (window)
• SessionWindow (session_window)
• WindowTime (window_time) - feat: Support Spark expression window_time #3732 (via PreciseTimestampConversion)

TimeType expressions (Spark 4.1+)

See umbrella issue #4288 for the full TimeType implementation plan.

• ToTime (to_time) - default format only
• TryToTime (try_to_time) - default format only
• MakeTime (make_time)
• CurrentTime (current_time) - typically constant-folded
• HoursOfTime (extract(HOUR FROM t)) - [Feature] Support Spark expression: hours_of_time #3126, feat: Support Spark expression hours_of_time #3615
• MinutesOfTime (extract(MINUTE FROM t)) - feat: Support Spark expression minutes_of_time #3447
• SecondsOfTime (extract(SECOND FROM t)) - [Feature] Support Spark expression: seconds_of_time #3129, feat: Support Spark expression seconds_of_time #3618
• SecondsOfTimeWithFraction
• TimeFromSeconds (time_from_seconds)
• TimeFromMillis (time_from_millis)
• TimeFromMicros (time_from_micros)
• TimeToSeconds (time_to_seconds)
• TimeToMillis (time_to_millis)
• TimeToMicros (time_to_micros)
• TimeAddInterval
• SubtractTimes
• TimeDiff (time_diff) - [Feature] Support Spark expression: time_diff #3122
• TimeTrunc (time_trunc) - [Feature] Support Spark expression: time_trunc #3123

Related work

• Implement JVM UDFs for all date/time expressions #4311 - Implement JVM UDFs for all date/time expressions (delivery strategy: route gaps through the codegen dispatcher for Spark-compatible defaults, native implementations remain opt-in)
• Implement TimeType support #4288 - TimeType support epic (covers infrastructure: shuffle, sort, min/max, casts)
• fix: Support scalar inputs for datetime expressions when constant folding is disabled #3448 - Scalar inputs for datetime expressions when constant folding is disabled (covers current_date/current_timestamp/now regression cases)
• date_trunc incorrect results in non-UTC timezone #2649 - date_trunc incorrect results in non-UTC timezone
• hour/minute/second expressions incorrectly apply timezone conversion to TimestampNTZ inputs #3180 - hour/minute/second incorrect timezone handling for TimestampNTZ

Describe the potential solution

No response

Additional context

No response