refactor: simplify build scripts required to produce binaries for packaging

[olivercalder]

Problem Statement

Currently, the build process for producing binary artifacts for packaging varies a lot for each format, including the .deb, snap, and docker image.

This variation includes:

• Whether to build with bundled z3 (the .deb produced by tasks/scripts/package-deb-install.sh does not, but the .deb produced by CI does)
• Whether to use zigbuild, which adds complexity for the purpose of cross-compilation and static musl builds (AFAICT)
• Whether to target gnu or musl (this varies between binaries in the docker image, for example, and varies between the local .deb and snap)

Another issue is that there's currently no script to prepare the binaries locally to be built into a snap package, nor the RPM package. It is assumed that CI workflows build the binaries and then those are arranged in another CI job to be consumed by snapcraft or the other package building tools. However, this makes iterating on and locally testing the packaging for the snap and rpm difficult.

Here's a table summarizing the differences in the way binaries are built for different packages:

Binary	Snap (CI)	Deb (CI)	Deb (local)	RPM (CI)	Docker (CI/local)
openshell	musl (zig), bundled-z3	musl (zig), bundled-z3	gnu (native), no bundled-z3	musl (zig), bundled-z3	—
openshell-gateway	gnu.2.28 (cargo-zigbuild), bundled-z3	gnu.2.28 (cargo-zigbuild), bundled-z3	gnu (native), no bundled-z3	gnu.2.28 (cargo-zigbuild), bundled-z3	gnu (native, no --target) or gnu.2.28 (cargo-zigbuild if cross-compiling), bundled-z3
openshell-sandbox	gnu (native), no bundled-z3	—	—	—	musl (cargo-zigbuild if available, else native), no bundled-z3
openshell-driver-vm	—	gnu.2.28 (cargo-zigbuild), no bundled-z3	gnu (native), no bundled-z3	—	—

This relates somewhat to #2204, though the focus there is more on standardizing testing, while I want the focus here to be on unifying package building between CI and local development.

Proposed Design

We should have a consistent list of scripts which can be invoked either in CI or locally (assuming the host is a glibc Linux machine, like the CI Ubuntu runners). That way, developers on most Linux distributions can build Linux packages natively, the same way that CI does. And developers on MacOS can do the same within a Linux VM or their container environment of choice.

I propose several sets of scripts, one set for each package target:

• one script to build the binaries required for that package and produces the paths at which those binaries were written
  • scripts of the form: tasks/scripts/build_binaries_<package>.sh
• one script to build the package, given paths to the binaries, and produces the path to the resulting package artifact
  • scripts of the form: tasks/scripts/build_package_<package>.sh
• one wrapper script which calls the build scripts and gets the paths to the binaries, then calls the package script with those binary paths, producing the path to the resulting package artifact
  • scripts of the form: tasks/scripts/package_<package>.sh (open to ideas on naming here)
• one test script which takes the path to a package artifact, installs it, and runs basic smoke tests as appropriate
  • scripts of the form: tasks/scripts/test_package_<package>.sh

These scripts should live under tasks/scripts/ and modify/replace the existing scripts where relevant, such as package-deb-install.sh and package-deb.sh, and ideally some of the docker scripts (though some are publishing/testing scripts).

Then, we can remove as much of this duplicated logic from the CI workflows as possible, and instead invoke the same scripts in CI as we can locally, with minimal additional setup.

Alternatives Considered

I considered having shared scripts like the existing stage-prebuilt-binaries.sh which can be used for multiple packaging formats. Then, we could theoretically build every binary for every package format, and then choose which to include in any given package. However, I think this results in unnecessary complexity:

• Some scripts do additional complicated setup of non-native cargo targets, zig/zigbuild for statically-built musl and non-native glibc ABIs
  • This is not necessary for all packaging formats, e.g. the musl builds are only relevant (AFAICT) on docker images or hosts which don't have glibc, such as alpine
  • For the snap, deb, and rpm, gnu target is always okay, so any additional complexity (e.g. zigbuild) is not necessary
• Sometimes the same binary needs to be built in different ways for different packaging formats, with different setup/configuration beforehand, so building a single blessed binary for each piece of openshell is not viable
  • Encoding this logic into the scripts (e.g. build this binary for docker vs for snap) would add lots of additional complexity

I also considered whether it would be worth having a single script each for "build binaries", "build package", and "test package", where for each you may specify which package format you'd like to work with. However, as above, I think this would introduce lots of additional logic into the scripts, particularly in situations where different packaging formats vary dramatically in terms of the amount of setup and complexity required.

Agent Investigation

No response

Checklist

• I've reviewed existing issues and the architecture docs
• This is a design proposal, not a "please build this" request