You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Linux Spark CI currently starts from generic Rust images and installs additional build tools in each job. Repeated setup and dependency downloads add overhead and expose builds to transient network failures before tests run. See the Spark SQL workflow and setup action.
We could improve reliability and reduce setup time by publishing dedicated CI images containing the required toolchains and third-party dependencies.
Proposed scope:
Preinstall pinned versions of Rust, JDK, clang, protoc, the Maven wrapper distribution, and required Rust components.
Prepopulate Maven dependencies/plugins and SBT’s Coursier/Ivy caches. Configure cache paths explicitly so they remain accessible under GitHub Actions’ container mounts and runtime home directory.
Start with one Linux Spark/JDK combination, then expand based on measured results.
Publish images only from trusted main builds. Rebuild when toolchain or dependency manifests change, plus periodically for security updates.
Pin workflows to image digests and retain the previous working image for rollback.
Exclude Comet binaries and locally built/SNAPSHOT artifacts from the image. Every PR must build its own code; artifacts can still be shared between jobs within that run.
Before expanding adoption, validate that a fresh container can run a representative build without external network access once the required sources are available. Compare image-pull time, setup time, total CI duration, and download-related failures against the current workflow.
Linux Spark CI currently starts from generic Rust images and installs additional build tools in each job. Repeated setup and dependency downloads add overhead and expose builds to transient network failures before tests run. See the Spark SQL workflow and setup action.
We could improve reliability and reduce setup time by publishing dedicated CI images containing the required toolchains and third-party dependencies.
Proposed scope:
mainbuilds. Rebuild when toolchain or dependency manifests change, plus periodically for security updates.Before expanding adoption, validate that a fresh container can run a representative build without external network access once the required sources are available. Compare image-pull time, setup time, total CI duration, and download-related failures against the current workflow.