Skip to content

ci: Prebuild Linux CI images with toolchains and dependencies #5490

Description

@pingzh

Linux Spark CI currently starts from generic Rust images and installs additional build tools in each job. Repeated setup and dependency downloads add overhead and expose builds to transient network failures before tests run. See the Spark SQL workflow and setup action.

We could improve reliability and reduce setup time by publishing dedicated CI images containing the required toolchains and third-party dependencies.

Proposed scope:

  • Preinstall pinned versions of Rust, JDK, clang, protoc, the Maven wrapper distribution, and required Rust components.
  • Prepopulate Maven dependencies/plugins and SBT’s Coursier/Ivy caches. Configure cache paths explicitly so they remain accessible under GitHub Actions’ container mounts and runtime home directory.
  • Start with one Linux Spark/JDK combination, then expand based on measured results.
  • Publish images only from trusted main builds. Rebuild when toolchain or dependency manifests change, plus periodically for security updates.
  • Pin workflows to image digests and retain the previous working image for rollback.
  • Exclude Comet binaries and locally built/SNAPSHOT artifacts from the image. Every PR must build its own code; artifacts can still be shared between jobs within that run.
  • Retain bounded download retries for new dependencies and cache misses, complementing the work in ci: extend Maven setup and retry dependency downloads #5489.

Before expanding adoption, validate that a fresh container can run a representative build without external network access once the required sources are available. Compare image-pull time, setup time, total CI duration, and download-related failures against the current workflow.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions