Replace llama.cpp with ONNX Runtime for reranking

[felladrin]

Description

Currently, the reranker spawns llama-server, so the Docker image has to compile llama.cpp from source in a dedicated builder stage. On a fresh environment that stage alone takes 168s (measured with --no-cache), and it needs build-essential, cmake and ccache in the image.

This PR runs the same model in-process through onnxruntime-node, whose binaries ship prebuilt for every supported platform. The builder stage is gone, and no compiler is needed at all.

The model is unchanged (jina-reranker-v1-tiny-en), only the engine differs. That was the constraint: it's the best reranker I found for this task, and it works surprisingly well on non-English results too.

What changed

Area	Before	After
Engine	llama-server child process, GGUF Q8_0	onnxruntime-node in-process, ONNX fp32
Tokenizer	Handled by llama.cpp	@huggingface/tokenizers (zero dependencies, ~8kB)
Docker	Builder stage compiling llama.cpp	No builder stage
Service management	Spawn, port 8012, /health polling, warmup POST, 5s restart loop, SIGTRAP/SIGILL handling	Load the session, release it on close
Local development	Needed llama-server on PATH	Nothing to install
Model on disk	36MB	131MB

server/rerankerService.ts keeps the same exports (rerank, getRerankerStatus, startRerankerService, stopRerankerService, sanitizeUnicodeSurrogates), so rankSearchResults.ts, searchEndpointServerHook.ts and statusEndpointServerHook.ts are untouched. Two signature details for reviewers:

• stopRerankerService is now async (it awaits the session release). rerankerServiceHook is the only caller and now logs a rejection instead of leaving a floating promise.
• getRerankerModelPath is no longer exported. It was only used inside the module, and there are now three files to resolve rather than one.

Ranking quality

I compared both engines on identical inputs, feeding the scores through the real rankSearchResults before comparing.

Case	Spearman	Top-1	p@3 llama.cpp	p@3 ONNX
English / factual	0.976	identical	1.00	1.00
Portuguese / recipe	0.976	identical	0.67	0.67
Portuguese / technical	0.976	identical	1.00	0.67
English / ambiguous ("jaguar")	1.000	identical	1.00	1.00

Top-1 matched in every case. The one p@3 difference is a single adjacent swap in the nginx case.

The Portuguese behavior carries over, and the clearest evidence is that both engines make the same mistakes: on "como fazer pão de queijo caseiro" both rank a pizza recipe second, and both bury the English "Cheese Bread Recipe (Brazilian Pão de Queijo)". Same weights, same quirks.

Note: ONNX logits spread wider than llama.cpp's compressed scores, so the standard-deviation filter keeps slightly fewer results (4-6 of 8 where llama.cpp kept 5-7). Ordering is unaffected. I tried squashing the logits through a sigmoid to match the retention, but it only helped in 1 of the 4 cases, so I left the raw scale.

Why fp32

• q8 measurably degrades ranking on a 33M-parameter model (Spearman drops to 0.90, and p@3 drops in 2 of the 4 cases).
• fp16 fails to load in onnxruntime-node (a graph fusion error in SimplifiedLayerNormFusion).

So fp32 it is, at the cost of a larger file on disk.

Performance

Measured at the production result count (30 documents):

Configuration	Median
llama.cpp, CPU, --threads 1 (what we run today)	397ms
llama.cpp, CPU, 8 threads	88ms
ONNX, CPU, batched (the fallback path)	81ms
ONNX, WebGPU, batched (the default where available)	32ms

Not a regression on either path: even the CPU fallback is about 4x faster than what we run today, because --threads 1 left cores idle and ONNX Runtime uses them by default. Cold start also drops to 0.16s once the model is cached.

These are medians on an unpinned laptop, so treat them as ±15%. The two ONNX rows are from one run with the batching this PR ships, so they are comparable to each other; the llama.cpp rows were measured separately.

Event-loop blocking

onnxruntime-node wraps a synchronous native call in setImmediate (see run() in js/node/lib/backend.ts), so inference blocks the event loop for its whole duration. Scoring 30 documents in one batch blocked it for 78ms with zero timer ticks.

Documents are now scored in batches of 10, which yields between calls and caps the stall at ~27ms for about 4% more wall time. Scores are bit-identical either way, since padding is per batch but the attention mask excludes it.

This is the one real trade-off against the old design: llama-server ran in its own process, so reranking never blocked Node. If the remaining ~27ms ever matters, moving inference to a worker thread would remove it entirely.

GPU

The session requests ["webgpu", "cpu"], with nothing to configure. WebGPU is about 3x faster than CPU (24ms vs 77ms for 30 documents) and matches CPU numerically to 1e-6 with identical ordering, so there's no reason to make it opt-in. Hosts with no usable GPU provider (Linux arm64, or any host without a GPU) fall back to CPU rather than failing to load.

Worth noting: ONNX Runtime's WebGPU provider here is native, compiled into the onnxruntime-node binary. It is not the browser API, so it needs neither a browser nor Deno.

CoreML and CUDA are deliberately left out. CoreML is 2.8x slower than CPU for this model's dynamic shapes, and the CUDA binaries aren't bundled with the package.

ONNX Runtime's startup warnings are also silenced (it warns on every launch that it assigned shape operators to CPU, which is expected), matching the quiet startup llama-server had.

Building without a git repository

docker compose up --build failed from a git worktree, where .git is a file pointing at a gitdir outside the build context. Git resolves the repository before running any subcommand, so even git config --global --add safe.directory exited 128 and killed the build.

The commit hash is optional build metadata, so the build now tolerates its absence and prints a warning instead of failing. helper-git-hash already returned an empty string on failure, so nothing else needed changing.

Consequences when no repository is available:

• The build logs WARNING: no usable git repository in the build context, ....
• /status reports build.gitCommit as "".
• The version in the menu shows v2026.7.28 instead of v2026.7.28+0d0fe9a. appVersion now omits the separator rather than rendering a dangling 2026.7.28+.
• Nothing else changes: the AI Horde client agent string just carries the shorter version.

Builds from a normal checkout are unaffected and still embed the hash.

Dependencies

onnxruntime-node + @huggingface/tokenizers, which adds 263 lines to the lockfile. I also prototyped this with @huggingface/transformers and it produced bit-identical scores, but it pulls sharp (image processing the reranker never touches) for 51 packages, 381MB and 4 advisories, against 26 packages, 214MB and 2 advisories here.

Both remaining advisories are adm-zip, reached only through onnxruntime-node's install script, which downloads the binaries that aren't bundled (the CUDA EP). On a default CPU install it exits early and never opens an archive.

Documentation

docs/reranking.md is largely rewritten (service lifecycle, execution providers, batching, why fp32, how to run the integration test). Smaller updates remove the llama-server references from README.md (the architecture diagram node), agents.md, docs/overview.md, docs/quick-start.md, docs/development-commands.md, docs/glossary.md and docs/configuration.md.

The llama.cpp mentions left in README.md and docs/ai-integration.md are deliberate: those describe llama.cpp as an OpenAI-compatible backend users can point MiniSearch at, which is unrelated to reranking. @wllama/wllama (browser-side inference) is untouched.

Type of Change

• Bug fix
• New feature
• Documentation
• Other (refactor, build, chore)

Checklist

• npm run lint passes
• Tests pass (npm run test), with tests added where it made sense

server/rerankerService.integration.test.ts loads the real model and asserts ranking quality on the English and Portuguese fixtures. It downloads ~130MB, so it's excluded from the default suite:

npx vitest run --config vitest.integration.config.ts

How to test

1. npm run lint and npm run test (288 tests).
2. npx vitest run --config vitest.integration.config.ts to exercise the real model (6 tests). Delete server/models first if you want to check the download path.
3. docker build -t minisearch . and confirm there's no compile step. It also works from a git worktree now.
4. Run the container and search for something in Portuguese, then confirm /status reports the reranker as available.

Security, performance, or breaking changes? Expand if relevant.

No breaking changes for users: no new required configuration, and the model still downloads on first start.

Performance: reranking gets faster (397ms to 81ms on CPU, 32ms on WebGPU, at 30 results), but inference now blocks the event loop for up to ~27ms per search, where before it ran in a separate process. See the sections above.

Security: adds 2 high advisories, both adm-zip via onnxruntime-node, on a code path a default CPU install doesn't reach. In exchange it drops the git/cmake/build-essential toolchain and a from-source clone of llama.cpp from the image build.

GPU is used automatically where ONNX Runtime has a working provider, with a CPU fallback everywhere else. No new configuration.