Skip to content

PartitionExpr cannot be reconstructed from its public API #3002

Description

@NoahKusaba

Is your feature request related to a problem or challenge?

A distributed query engine plans on one node and executes on others, so physical
plans must survive serialization. project_with_partition injects a
PartitionExpr into the projection for partitioned writes, and that expression
cannot make the trip.

It wraps a PartitionValueCalculator, which is a live non-serializable state. The two
things that are serializable (the PartitionSpec and the table schema) are
consumed at construction and not retained, so nothing on the expression says
what produced it.

The spec alone is not enough to rebuild the calculator: PartitionSpec stores
only spec_id and fields, referring to columns by source_id. The schema is
what resolves those ids to real columns and determines the partition type.

Reaching for the schema elsewhere at decode time isn't possible either:
PhysicalExtensionCodec::try_encode_expr / try_decode_expr receive only the
expression, with no tab

Describe the solution you'd like

Retain both inputs on PartitionExpr and expose them:

  • try_new(partition_spec, table_schema) replacing the private
    new(calculator, spec), constructing the calculator internally so the
    retained inputs and the calculator cannot drift apart.
  • partition_spec() and table_schema() accessors.

Additive: project_with_partition keeps its signature, and new was private,
so no existing caller changes.

Willingness to contribute

I can contribute to this feature independently

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions