migrate from bun-compile to scriptc (91MB -> 3.2MB) - #56
Draft
ryoppippi wants to merge 13 commits into
Draft
Conversation
scriptc compiles TypeScript to a native executable with no JavaScript engine in the binary. It is a build tool, so it belongs in devDependencies — keeping it out of `dependencies` keeps it out of the published runtime closure and out of package.nix's --production install.
scriptc type-checks with the real TypeScript compiler but forces `lib` to
es2025 and `types` to empty, and lowers only part of the standard library. Bring
the sources inside that subset without changing behaviour under Bun.
DOM values
linkedom runs in the embedded engine, so its nodes arrive as dynamic values
and scriptc refuses to validate one against a structural interface — a
hand-written `Element` type is not an option. Nodes stay untyped (lib/dom.ts's
DomNode) and only leaf values are narrowed at the point of use. Spreading a
dynamic collection has no lowering, hence toArray/toStrings; Map keys are
limited to strings and numbers and weak collections are unavailable, so
identity memoization tags each node with a serial number (nodeId).
Bun APIs -> node builtins
Bun.file/stdin/CryptoHasher/FileSink become node:fs, node:crypto and
readFileSync(0). fs/promises has no rename or open lowering, so the fetch
cache writes its entry with the final mode instead of writing-then-renaming,
and -o buffers the (already --max-bytes-bounded) body and writes it once. The
property that matters survives: a failed transfer leaves the old file alone.
Module graph
Text imports (`with { type: 'text' }`) are a Bun bundler feature: the
agent-context payload is generated into a TypeScript module instead. JSON
default imports do lower (baked as literals at compile time), named ones do
not. toTsv moves to lib/tsv.ts so the compiled graph excludes the unwired jq
surface in lib/query.ts.
Hand-rolled replacements
util.parseArgs, Object.keys/entries/fromEntries and Array unshift/reverse have
no lowering.
Out-of-bounds reads
scriptc bounds-checks indexed reads instead of answering undefined, so code
that is correct JS aborts at runtime — after compiling cleanly. Every
optional slot is now read behind an explicit length test: argv/positional
destructuring, the expression parser's token lookahead, the BOM sniff on an
empty (HEAD) body, the table grid, and argv[index - 1].
The recursive-descent parser in lib/expr.ts is restructured from nested
closures over a captured `pos` to top-level functions over an explicit cursor:
mutually recursive inner declarations sharing mutable outer state compiled but
aborted at runtime.
`bun run build:scriptc [out]` stages the sources into .scriptc-build and
compiles them with --dynamic, which embeds the JavaScript engine that linkedom
and fetch run in (~620KB).
Two host primitives shrink in a compiled binary and cannot be recovered at
runtime, because scriptc refuses the calls at COMPILE time and neither
`globalThis` nor `process` can be aliased into a dynamic value to reach them
another way:
* TextDecoder has no lowering beyond the default utf-8, so decoding goes
through Buffer's encodings — UTF-8, both UTF-16 byte orders and latin1.
Legacy codecs (shift_jis, windows-125x, ...) fall back to UTF-8 with a note.
* process.stdout.write lowers only for a single string, so there is no
completion callback and no byte writes.
A runtime branch therefore cannot work, and the staging step exists so
lib/platform.ts can be swapped for lib/platform.scriptc.ts before the compiler
sees it. Bun keeps the full-fidelity implementation, unchanged.
`bun run gen` regenerates src/agent-context.gen.ts from agent-context.txt.
Requires cmake and clang on PATH for the first build (the engine archive is
cached in node_modules afterwards), and Node to run the compiler itself:
typescript@7's synchronous RPC channel reads `stdout._handle.fd` off a spawned
child, which Bun does not expose.
`nix build .#ax-scriptc` produces the natively compiled binary; `.#ax` and `.#default` still build src/index.ts run under bun. The derivation uses stdenv rather than stdenvNoCC (scriptc drives a C toolchain), and takes zlib and curl as real build inputs: the generated C links the host's copies, and the vendored ones under @scriptc/runtime serve the cross-compile path only. nodejs is a build input because the compiler CLI does not run under Bun. Everything it compiles is vendored, so the build stays offline. It is deliberately out of `checks` — it compiles the embedded engine from source, which costs minutes per system — and built explicitly in CI instead. The dev shell gains cmake so `bun run build:scriptc` works inside it, plus clang on Linux. Darwin uses the Xcode toolchain: the nix clang wrapper does not find the macOS SDK headers the engine needs.
A `native` job in CI builds with scriptc and requires byte-identical output against the same sources run under bun across 16 invocations — parse, markdown, outline, locate, table, --where, JSON, envelope, paging and the error paths. A --version check alone would exercise almost none of the compiled surface. nix.yml builds .#ax-scriptc and runs it; its timeout grows to 45 minutes to cover compiling the embedded engine. The test job also now asserts src/agent-context.gen.ts is in sync with agent-context.txt, the same way nix/bun.nix is checked against bun.lock.
The payload is one long string literal, so the assignment needs to sit on its own line to fit oxfmt's printWidth. CI checks `bun run gen` output for both formatting and drift, so the generator has to produce the formatted shape directly.
`bun run build` now compiles with scriptc, and `.#ax` / `.#default` build that binary instead of installing src plus node_modules to run under bun. The separate ax-scriptc package folds back into package.nix, so there is one `ax`. `bun build --compile` is gone. Its output was 61MB against 2.5MB here, and keeping both meant maintaining two runtimes' worth of behaviour — the source already had to stay inside scriptc's subset. BREAKING CHANGE: the nix package installs a compiled executable rather than a bun shebang script, so bun is no longer in its runtime closure. Building it now needs a C toolchain (cmake, clang) and compiles the embedded JavaScript engine from vendored source, which takes minutes — `nix flake check` included.
CI ran only on pushes to main, so a branch got no signal until a PR existed; it now runs on every push, with a concurrency group so a pushed PR branch does not build twice. The compile job becomes a five-way matrix (Linux x64/arm64, macOS arm64/x64, Windows x64). It cannot be a cross-compile sweep from one runner: a --dynamic build is host-native, because the embedded engine archive is built per target. Release builds the same matrix and publishes what it produced. Windows is best-effort in both workflows — scriptc has not ported the socket stack (net/http/https/tls), which is where ax's fetch lives. It stays in the matrix so the gap is visible rather than assumed, and the release fails only if a platform the installer or the Homebrew formula resolves to is missing. The smoke test moves to .github/scripts/smoke.sh so the release matrix runs the same 26 comparisons before publishing a binary, and takes AX from the environment to check a renamed artifact.
scriptc has not ported the socket stack (net/http/https/tls), which is where ax's fetch lives, so a scriptc Windows binary could not do the one thing ax is for. Windows compiles with `bun build --compile` instead; the sources are identical and the bun path keeps the full-fidelity lib/platform.ts (TextDecoder labels, the stdout drain) that the compiled builds trade away. Every matrix leg is now expected to pass, so continue-on-error is gone and the release requires all five assets again.
pnpm 11 refuses to install until every dependency with a build script is listed under `allowBuilds`, so a bare `pnpm install` failed on bun2nix with ERR_PNPM_IGNORED_BUILDS. The workspace file records that decision. The lockfile is regenerated against the current manifest (scriptc moved to devDependencies), so it agrees with bun.lock rather than describing the older layout. Note that bun.lock stays the source of truth: nix/bun.nix is generated from it, and CI installs with bun. Both lockfiles now need updating together.
The macos-13 image is retired, so the job sat unassigned rather than failing — every other matrix leg finished while it waited for a runner that no longer exists.
Contributor
Author
|
Draft because I have no idea whether you want this at all — happy to un-draft if it looks interesting, and equally happy to close it. Full disclosure: this is AI slop. It works — CI is green on all five platforms and the compiled binary produces byte-identical output to The one bit I think has value regardless of the outcome: scriptc bounds-checks out-of-range array reads instead of evaluating to |
typeOf exists because `typeof` answers 'object' for both null and arrays, so it
normalises to the names jq uses — which is also what its error messages print
("cannot index array with ..."). Returning a bare `string` meant every
`typeOf(x) === 'object'` comparison would accept a typo silently and just never
match.
The union includes the non-JSON arms (undefined, bigint, symbol, function)
because the input is `unknown`; a JSON document never produces them, and
spelling them out keeps the body a plain `return typeof v` with no behaviour
change.
ryoppippi
changed the title
scriptc shells out to `clang` by name. The default Linux stdenv provides gcc and nothing called clang, so the derivation died with `spawn clang ENOENT` while compiling the embedded engine's vendored mbedtls. Darwin's stdenv is already clang, which is why this only showed up on the Linux runner. nix.yml also now runs on any branch push, not just pushes to main — the workflow could not be exercised before opening a PR, which is exactly how this reached CI unverified.
ryoppippi
changed the title
ryoppippi
changed the title
ryoppippi
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Compiles
axto a native executable with scriptc instead ofbun build --compile. No bundled runtime, so the binary drops by 25–28×.Size
bun --compileIt is slower, though —
--version16.7 ms → 35.6 ms,--md22.2 ms → 37.0 ms (darwin-arm64, hyperfine).--dynamicembeds a JS engine that linkedom andfetchrun inside, interpreted, while bun starts fast and JITs the DOM work. The win is size only.What it costs
src/lib/platform.tskeeps bun's full-fidelity implementation;platform.scriptc.tsis the reduced one, swapped in at build time. These calls are refused at compile time and no dynamic escape hatch reaches them, so a runtime branch isn't possible.TextDecoderlabel--bodybytes-ostill writes bytes untouched)-owritesrenamehas no lowering)The 20 MB / 30 s guards, the streaming capped read, and
-kall survive.Windows keeps
bun build --compile: scriptc hasn't ported the socket stack (net/http/https/tls), which is where ax's fetch lives. It's therefore the only build with none of the degradation above. Release builds one runner per target — a--dynamicbinary is host-native, so the old single-runner--targetsweep isn't possible.The port
scriptc type-checks with the real TypeScript compiler but forces
libto es2025 andtypesto empty, and lowers only part of the stdlib. So: DOM nodes stay untyped (a dynamic value can't be validated against a structural interface, so no hand-writtenElement),WeakMapand objectMapkeys are out, andutil.parseArgs,Object.keys/entries/fromEntriesandArray#unshift/reverseare hand-rolled. Text imports are a bun bundler feature, so the agent-context payload is generated into a TS module (bun run gen, drift-checked in CI).The one thing worth knowing before anyone else tries this:
Correct JS compiles cleanly and then aborts at runtime — including code that is type-correct under
noUncheckedIndexedAccess, where TS models exactly that| undefined. Seven sites: three argv/token destructurings, the--whereparser's lookahead, the BOM sniff on an empty (HEAD) body, the table grid's sparse cells, andargv[index - 1].Verification
.github/scripts/smoke.shrequires byte-identical output againstbun src/index.tsacross 26 invocations — every output mode plus the error paths. All five platforms reportpass=26 fail=0, and the release matrix runs it again before publishing.Also green:
tsc --noEmit,oxfmt --check,bun test(147),nix build, and the network paths (fetch report,--body,-I, cache/--fresh,-m,--max-bytes,-obyte-identical) checked locally against bun.CI now runs on every push instead of only pushes to
main.