Is your feature request related to a problem or challenge?
A distributed query engine plans on one node and executes on others, so physical
plans must survive serialization. project_with_partition injects a
PartitionExpr into the projection for partitioned writes, and that expression
cannot make the trip.
It wraps a PartitionValueCalculator, which is a live non-serializable state. The two
things that are serializable (the PartitionSpec and the table schema) are
consumed at construction and not retained, so nothing on the expression says
what produced it.
The spec alone is not enough to rebuild the calculator: PartitionSpec stores
only spec_id and fields, referring to columns by source_id. The schema is
what resolves those ids to real columns and determines the partition type.
Reaching for the schema elsewhere at decode time isn't possible either:
PhysicalExtensionCodec::try_encode_expr / try_decode_expr receive only the
expression, with no tab
Describe the solution you'd like
Retain both inputs on PartitionExpr and expose them:
try_new(partition_spec, table_schema) replacing the private
new(calculator, spec), constructing the calculator internally so the
retained inputs and the calculator cannot drift apart.
partition_spec() and table_schema() accessors.
Additive: project_with_partition keeps its signature, and new was private,
so no existing caller changes.
Willingness to contribute
I can contribute to this feature independently
Is your feature request related to a problem or challenge?
A distributed query engine plans on one node and executes on others, so physical
plans must survive serialization.
project_with_partitioninjects aPartitionExprinto the projection for partitioned writes, and that expressioncannot make the trip.
It wraps a
PartitionValueCalculator, which is a live non-serializable state. The twothings that are serializable (the
PartitionSpecand the table schema) areconsumed at construction and not retained, so nothing on the expression says
what produced it.
The spec alone is not enough to rebuild the calculator:
PartitionSpecstoresonly
spec_idandfields, referring to columns bysource_id. The schema iswhat resolves those ids to real columns and determines the partition type.
Reaching for the schema elsewhere at decode time isn't possible either:
PhysicalExtensionCodec::try_encode_expr/try_decode_exprreceive only theexpression, with no tab
Describe the solution you'd like
Retain both inputs on
PartitionExprand expose them:try_new(partition_spec, table_schema)replacing the privatenew(calculator, spec), constructing the calculator internally so theretained inputs and the calculator cannot drift apart.
partition_spec()andtable_schema()accessors.Additive:
project_with_partitionkeeps its signature, andnewwas private,so no existing caller changes.
Willingness to contribute
I can contribute to this feature independently