[FLINK-40300][python] Fix literal expression in Python Table API

[auroflow]

What is the purpose of the change

Creating a PyFlink literal with lit(value) and lit(value, data_type) requires the value to first cross the Py4J boundary and then pass Flink's Java-side literal validation. The current implementation can fail at either stage:

1. datetime.date, datetime.time, datetime.datetime, and datetime.timedelta are not supported by the Py4J protocol directly. Therefore, literals declared as DATE, TIME, TIMESTAMP, TIMESTAMP_LTZ, or day-time INTERVAL fail before the Java Table API is invoked.

2. Python numeric values are implicitly converted by Py4J into a limited set of Java boxed classes. A Python int becomes Integer when it fits into 32 bits and Long otherwise, while a Python float always becomes Double. Consequently:

   • TINYINT expects Byte but receives Integer;
   • SMALLINT expects Short but receives Integer;
   • BIGINT expects Long but receives Integer for values within the 32-bit range;
   • FLOAT expects Float but receives Double.

   These values are rejected by ValueLiteralExpression.validateValueDataType().

3. Without an explicit data type, Python lists and tuples reach Java as generic containers. Java literal inference requires concrete Java arrays to infer the element type, particularly for empty array.array values and nested arrays.

The same issue affected nested values in arrays, maps, multisets, and rows.

This change converts Python-only values before transport, normalizes boxed numeric values in Java, and creates the literal in the same JVM call so that Py4J cannot convert the normalized value back to a Python primitive. The goal is to align the functionality of PyFlink Table API lit with Java Table API lit.

Brief change log

• Route Table API and DataFrame API literals, with or without an explicit data type, through one Java bridge.
• Convert Python temporal and composite values into representations that can cross Py4J.
• Preserve Java literal inference by materializing Python sequences as concrete Java arrays without duplicating inference in Python.
• Convert boxed numeric values and year-month interval month counts inside the same JVM call.
• Recursively validate and construct plannable ARRAY, MAP, and ROW expressions.

Verifying this change

This change added tests and can be verified as follows:

• Added PythonTableUtilsTest coverage for Py4J numeric representations, year-month intervals, Java array inference, typed nulls, and invalid composite literals.
• Added Table API integration tests covering inferred and explicitly typed scalar and constructed literals, including nested and empty arrays, maps, rows, temporal values, and intervals. The tests exercise planning and execution.
• Extended DataFrame API tests to cover all supported inferred Python literal types, explicitly typed scalar and constructed values, typed nulls, and invalid value/type combinations.
• Added DataFrame integration coverage that collects inferred and explicitly typed scalar, array, map, and struct literals.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changed class annotated with @Public(Evolving): no
• The serializers: no
• The runtime per-record code paths (performance sensitive): no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? not applicable

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

Generated-by: OpenAI Codex (GPT-5)

[flinkbot]

CI report:

• 7256a9c Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build