chore(python): build the Python client against DataFusion 55 - #2385
Draft
andygrove wants to merge 1 commit into
Draft
chore(python): build the Python client against DataFusion 55#2385andygrove wants to merge 1 commit into
andygrove wants to merge 1 commit into
Conversation
The Rust workspace moved to DataFusion 55, but `python/` is a separate Cargo workspace that was still pinned to the DataFusion 54 crates and to the crates.io `ballista` 54.0.0 release. datafusion-python has no DataFusion 55 release yet, so point the dependency at the bump branch until 55.0.0 lands on crates.io. That in turn forces the `ballista` crates back onto path dependencies: the published 54.0.0 crates are built against DataFusion 54 and cannot be mixed with a DataFusion 55 datafusion-python, since `pyballista` hands DataFusion `DataFrame` values straight to `PyDataFrame::new`. Also bump pyo3 to 0.29 to match datafusion-python, and drop the two `concurrent_tasks` hacks in `cluster.rs` now that the field they were waiting on is available as `ExecutorProcessConfig::vcores`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
N/A - follow-up to the DataFusion 55 upgrade of the Rust workspace.
Rationale for this change
python/is a separate Cargo workspace from the rest of the repo, so the DataFusion 55 upgrade did not reach it. The Python client is still built against the DataFusion 54 crates and against the crates.ioballista54.0.0 release, which means the wheel we ship no longer matches the engine inmain.datafusion-python has not published a DataFusion 55 release yet, so this PR tracks the bump branch at https://github.com/andygrove/datafusion-python.git until 55.0.0 is on crates.io.
That choice forces the
ballistacrates back onto path dependencies. The published 54.0.0 crates are built against DataFusion 54, andpyballistapasses a DataFusionDataFrameproduced by a BallistaSessionContextstraight intodatafusion_python::dataframe::PyDataFrame::new, so the two sides have to agree on the DataFusion version. This restores the arrangement that was in place before #2277.What changes are included in this PR?
pyballistacrate version to 55.0.0.datafusion-pythonat thebump-datafusion-55.0.0branch of https://github.com/andygrove/datafusion-python.git, and pindatafusion/datafusion-prototo=55.0.0.ballista,ballista-core,ballista-executorandballista-schedulerback to path dependencies on the workspace crates.pyo3from 0.28 to 0.29 so it matches the version datafusion-python is built with.datafusionPython dependency inpyproject.tomlfrom==54to==55.concurrent_taskshacks inpython/src/cluster.rs. They carried aTODOwaiting for the rename, andExecutorProcessConfig::vcoresis now available.python/Cargo.lock.Verified locally in the
python/workspace:cargo check --locked,cargo clippy --locked --all-targets -- -D warningsandcargo fmt --checkall pass.Two things are deliberately left for a follow-up, both blocked on the datafusion-python release:
python/uv.lockis unchanged, becauseuv lockcannot resolvedatafusion==55while it is absent from PyPI. Every job that runsuv syncwill fail until datafusion-python 55.0.0 is published, at which point the lockfile can be regenerated.datafusion-pythongit dependency should become a=55.0.0crates.io pin once that release exists.This is why the PR is a draft.
Are there any user-facing changes?
The Python client will require
datafusion==55instead ofdatafusion==54. There are no API changes to the client itself.