Conversation
…own push queue, campaign state Co-Authored-By: Virgil <virgil@lethean.io>
Benchmarks for the pure per-request helpers the chat interceptor pays on every stateless turn: conversationKey (hashes the retained prefix — twice per turn), conversationTurnSplit, messagesCarryMedia, normaliseRole. Realistic shallow (2-turn) and deep (20-turn) transcripts. Co-Authored-By: Virgil <virgil@lethean.io>
conversationKey assembled the whole retained prefix into an un-presized core.Builder, paying the builder's doubling-growth reallocation on every turn (a deep conversation is hashed twice per turn — acquire + sleep). pprof -alloc_objects put the entire flat allocation at the msg.Content WriteString. Presize with a computed upper-bound byte budget (len(Role) is an upper bound for the normalised role; Grow is a capacity hint) so the prefix assembles in one allocation. Byte-identical key. Deep (20-turn): 34776 B/op 13 allocs -> 9536 B/op 2 allocs Shallow (2-turn): 984 B/op 5 allocs -> 576 B/op 2 allocs Co-Authored-By: Virgil <virgil@lethean.io>
…ant contracts Benches the per-token and per-load CPU surface of the backend-agnostic model root: MatNT reference matmul, foldNormBiasOne, NormalizeWrapperNames, DeriveLayers, the arch/assistant registries, QuantConfig parse+lookup, ProbeModelTypes, ValidateRequired, Assemble. All pure-Go, no model load. Co-Authored-By: Virgil <virgil@lethean.io>
mux: the per-token streaming wire encoders — writeResponseDeltaFrame, writeOllamaChatFrame/GenerateFrame and the shared appendJSONStringHTML escaper (safe + escaped + long deltas), plus the per-request idWithPrefix. Locks in the documented zero-per-token-allocation profile (all frame writers measure 0 allocs/op). admin: cacheEntryLabelsFrom, the per-request cache-entries label filter. Co-Authored-By: Virgil <virgil@lethean.io>
Benches the Qwen 3.6 hybrid's per-token CPU surface: SwiGLU MLP, MoE routing (forward/swigluExpert/topKIndices), full-attention Forward + rmsNormHead + rotary, the gated-delta mixer adapter, the bf16 boundary conversions, and the per-load tensorF32 widen + resolveKinds. Pure Go, no model load, no device GEMM. Co-Authored-By: Virgil <virgil@lethean.io>
…ne (#377) The stateless text lane opened a fresh session per request: every turn of a multi-turn chat re-prefilled the whole history (per-turn wall 3.1s -> 4.4s over ten turns on e2b) and paid a maxLen-sized KV alloc+free round trip — while llama-server's slot cache stays flat, and two complete LCP reuse implementations already existed in-engine with no serve-path caller. TextModel now keeps ONE resident session when the engine declares PromptReuseCapableModel; stream() prefills through PrefillTokensCached, which reuses the longest shared prefix of the resident ids (prompt + generated reply — cachedIDs is maintained by every decode path) and prefills only the divergent suffix. llama slot-cache parity: single slot, TryLock (busy -> fresh path), volatile. Ring safety is the hard rule: sliding layers keep bounded rings, so any rollback past a wrapped ring (pos > slidingWindow) resumes attention over rows the discarded tail destroyed — those calls degrade to a cold PrefillTokens, token-identical either way. Gated both ways in TestPrefillTokensCached_RingSafety_Ugly. Stand-down guards: conversation continuity installed (it owns caching; KV never budgeted twice), LTHN_PROMPT_REUSE=0 kill switch, capability probe at the model level (no throwaway session opens). engine suite 181 green, engine/metal suite green; reuse gates: turn-boundary append reuses prompt+reply rows with token-identical continuation vs a cold session. Co-Authored-By: Virgil <virgil@lethean.io>
Hoist the per-token MoE routing scratch out of the token loop: the top-k index buffer, the expert hidden buffer (sized to the widest expert), and the expert output buffer are now allocated once per forward call, not once per (token, expert). swigluExpertInto / topKInto are the buffer-reusing cores; swigluExpert and topKIndices stay as allocating wrappers for their existing tests. Output is byte-identical (each buffer is fully overwritten per use) — the 16 composed tests stay green. FLAT alloc profile confirms the 5 remaining allocs are all once-per-call setup (out/probs/idx/hbuf/eo), none per-token. benchtime=20x, D=1024, 8 experts FF=1408, top-2 + shared: before 8689277 ns/op 53376 B/op 9 allocs/op after 9219715 ns/op 20608 B/op 5 allocs/op Co-Authored-By: Virgil <virgil@lethean.io>
The existing ThinkingExtractor benches feed single tokens / short blocks, so they never exercised the cumulative content/thinking totals folded across a whole answer. These stream a realistic 200-token plain answer and a channel-reasoning stream token-by-token through one extractor then Flush — the honest per-response unit, and what exposed the O(n^2) accumulation. Co-Authored-By: Virgil <virgil@lethean.io>
…600->10/op ThinkingExtractor accumulated the cumulative Content()/Thinking() totals with `e.content += text` on EVERY token — quadratic over a stream (each token recopies the whole growing answer). It also paid a second, per-drain lazy strings.Builder for the returned delta, duplicating every write. Fix: make content/thinking core.Builder fields (amortised O(n) growth) and drop the per-drain delta builder entirely — drain records each builder's length on entry and returns the tail it grew by (tailFrom), a zero-copy view that stays valid across later reallocs. Deltas and Content()/Thinking() are byte-identical; all 702 package tests pass, incl. the plain-token alloc-budget guard (single-token floor stays 2, now 1 for the delta path). Process over a 200-token plain answer: 124704 B/op 600 allocs -> 4472 B/op 10 allocs Reasoning stream: 10512 B/op 175 allocs -> 1136 B/op 11 allocs Existing short benches also improved (PlainTokenShort 40->8 B/op, 2->1 alloc). Co-Authored-By: Virgil <virgil@lethean.io>
Benches the Qwen 3.5/3.6 gated delta-rule linear-attention recurrence at decode (L=1, carried state) and short prefill (fresh state). Pure Go, no load. Co-Authored-By: Virgil <virgil@lethean.io>
Benches the RWKV-7 WKV7F32 recurrence (decode + prefill), the full time-mix BlockForwardF32, and the host-default projMatMul seam. Pure Go, no load. Co-Authored-By: Virgil <virgil@lethean.io>
Benches the Qwen 3.6 GatedDeltaForwardF32 decode block, Config.Arch + InferFromWeights per-load derivation, and the registered arch parser via the model registry. Pure Go, no load. Co-Authored-By: Virgil <virgil@lethean.io>
Benches the per-token bf16<->f32 seam conversions, the host-default projMatMul, and the per-weight tensorF32 widen at load. Pure Go, no load. Co-Authored-By: Virgil <virgil@lethean.io>
runtimememory_bench_test.go and device_bench_test.go bench the observability/maintenance dispatch — RuntimeMemoryUsage, ClearRuntimeCache, BackendDeviceInfo — behind the existing fake reporter/provider backends (no GPU). Hit + miss paths; all measure 0 allocs/op, locking the registry-lookup + type-assert dispatch cost. The other four root files in scope (thinking.go, speculative.go, session_contract.go, kvstate.go) are pure type/interface declarations with no executable work — no benches. Co-Authored-By: Virgil <virgil@lethean.io>
…-ring rule prepareAssistantPrompt (the MTP pair serve lane) and GenerateCached / GenerateCachedSampledEach rolled the session boundary back to the shared prefix with no ring check. Sliding layers keep bounded rings: landing row p overwrites the slot of row p-w, so a rollback past a wrapped ring (pos > slidingWindow) resumes attention over window rows the discarded tail already destroyed — silent wrong context on multi-turn pair serving whenever the resent history diverges from the resident ids (thinking-on chats diverge at every reply, the thought channel is stripped from what clients resend). All three sites now apply the rule session_prompt_reuse.go established: a rollback past a wrapped ring degrades to a cold full prefill — token-identical, just uncached. Gated by TestPrepareAssistantPromptRingSafety (wrapped-ring divergence decodes token-identical to a fresh session); assistant suite green. Co-Authored-By: Virgil <virgil@lethean.io>
…row needs LTHN_PROMPT_REUSE=0 The energy table's third row is now 'pure replay (no cache of any kind)' with the reproduction note; the draft gains the stateless-is-not-the- punching-bag paragraph (turn-10 wall 4.4s -> 3.1s, flat, ring-safe, stands down under continuity). Wake note carries #377 closure + the two bench-agent worktrees to merge. Co-Authored-By: Virgil <virgil@lethean.io>
anthropic_stream: the streaming event encoders — AppendContentBlockDelta (the per-token content_block_delta hot path, safe + escaped), the tool input_json_delta, and the per-stream message_start/message_delta wrappers. All the per-token/per-stream builders measure 0 allocs/op (caller-owns-buf, off the reflect path). tools: RenderToolDeclarations over a realistic multi-tool Claude Code set. Ollama's chunkenc/unmarshal and anthropic's jsondec are already covered by the package-level bench files. Co-Authored-By: Virgil <virgil@lethean.io>
…e benches Fills the per-file bench gaps left uncovered by safetensors_bench_test.go: values.go DecodeFloat32/EncodeFloat32 (the whole-file codec, distinct from index.go's DecodeFloatData), write.go subsetHeaderEncoded + appendJSONInt64 (the pure header emit, distinct from the file-writing WriteSubset), and the NEON (darwin/arm64) + scalar float16SliceToFloat32 slice conversion. index.go is deliberately not given a bench file: its symbols are already benched in safetensors_bench_test.go + header_parse_bench_test.go (no duplication). Co-Authored-By: Virgil <virgil@lethean.io>
pathWithinDir is the traversal-escape gate on every hot-swap /reload request. Benches the within / escape / equal branches; the equal fast path is 0 allocs, the PathRel branches carry the core.PathRel allocation (path logic, not a hot-loop trap). Closes the admin package's bench gap — the other admin files are HTTP handlers and boot-time helpers (MachineHash, JSON I/O), not pure-CPU-benchable in isolation. Co-Authored-By: Virgil <virgil@lethean.io>
scorer.go (the in-process lem-scorer adapter that rides alongside every completed turn) was the one pipeline symbol the package bench left uncovered — pipeline_bench_test.go covers New/Complete/the adapters but not Score. Benches the typical pair, response-only and empty cases. The typical pair is ~229us / 99 allocs, dominated by lek.ScorePair + JSONMarshal (both outside this module's scope) — the adapter's own overhead is a lastUser read + one string copy + the map; no in-scope trap. types.go is pure interface declarations; wired.go's adapters are covered by pipeline_bench. Co-Authored-By: Virgil <virgil@lethean.io>
Benches the GGUF pure-CPU surface: Quantize per format (all nine kernels via the public entry), the Q8_0/Q4_0 dequantise-to-F16 load path, the quant classification (NormalizeQuantType/quantBits/quantFamily/buildGGUFTensorInfos/ inferGGUFQuantization), the format dispatch, the metadata coercers, and the pure header-prep (ggufQuantizeMetadata/assignGGUFTensorOffsets). Synthetic, no file. quantize_kernels.go is covered via Quantize (no duplicate direct-kernel benches); metadata.go + tensors_mmap_*.go need a .gguf fixture. Co-Authored-By: Virgil <virgil@lethean.io>
Two per-request traps killed with the new bench coverage watching: openai ThinkingExtractor 124704->4472 B/op, 600->10 allocs (quadratic cumulative concat + doubled per-drain builder, on EVERY streamed token); continuity conversationKey 34776->9536 B/op, 13->2 allocs (presized). Bench files land for continuity, compat, openai full-stream thinking, root dispatch, anthropic stream/tools, admin pathWithinDir, pipeline scorer; the rest of the census was already benched at package level (verified per file, no filler duplicates). Co-Authored-By: Virgil <virgil@lethean.io>
…ffer Quantize called AppendQuantize(format, nil, values), so the kernel grew the output from a nil slice block by block (~log2(nblocks) reallocations). The packed length is deterministic — (values/blockSize)*bytesPerBlock — so pre-size dst to it and AppendQuantize fills it in one allocation. Byte-identical output and identical errors (unknown format / non-block-multiple length leave dst nil and defer to AppendQuantize's check); the 112 gguf tests stay green. FLAT alloc profile confirms the one remaining alloc is the pre-sized output buffer. benchtime, 131072 f32 values (512 K-super-blocks / 4096 _0 blocks): Q8_0 before 685470 B 23 allocs/op after 139266 B 1 alloc/op Q4_K before 286702 B 20 allocs/op after 73922 B 1 alloc/op Q2_K before 212914 B 19 allocs/op after 49253 B 1 alloc/op Co-Authored-By: Virgil <virgil@lethean.io>
Quantize/AppendQuantize live in quantize_dispatch.go and resolveGGUFQuantizeFormat/
ggufQuantizeLayout/ValidationSummary in quantize.go — swap the two bench files'
contents so each {file}_bench_test.go benches its own file's symbols. No coverage
change; corrects the per-file mapping only.
Co-Authored-By: Virgil <virgil@lethean.io>
Benches the model-merge pure-CPU surface: linearMerge/slerpMerge/normalizedWeights (the per-element merge kernels), compareTensorEntries/compareCosine (per-tensor delta stats), the equalFold/containsFold/hasSuffixFold/clampFloat64 helpers, and the SamePath/SamePathResolved overwrite guards. Synthetic, no file (Packs/prepare/ indexSources + HashFile need real safetensors, benched via the merge path). Co-Authored-By: Virgil <virgil@lethean.io>
Benches gemma3's arch derivation (sliding/global layer schedule + dual RoPE bases) and the shape-inference dim recovery. Synthetic, no file. register.go is init-only (parser benched via the arch load path); mistral's YaRNInvFreqs is already benched in its config_bench_test.go (not duplicated). Co-Authored-By: Virgil <virgil@lethean.io>
Benches the per-record state-store codec: encodeRecordHeader/decodeRecordHeader (the 24-byte on-disk header, zero-alloc by design) and the hand-rolled JSON path (extractRecordURI/appendJSONString/appendJSONField/jsonUnescape) the store uses instead of reflection. Pure Go, no file. The file-backed store ops (put/resolve/ index/store_mmap/helpers) are I/O and exercised via the store path. Co-Authored-By: Virgil <virgil@lethean.io>
Benches BuildCalibrationPlan (config normalise + sample select + token count) and boundedCalibrationTokenCount — the native calibration metadata work preceding the SignRound quantise. QuantizeWeights/PackQuantizedWeights/Dequantize are already benched in autoround_bench_test.go (not duplicated). Synthetic, no file. Co-Authored-By: Virgil <virgil@lethean.io>
Benches inferGemma4HeadDim (sliding vs global head dim from q_proj rows) and inferGemma4PerLayerInputSize (PLE-tower width) — the flagship's don't-guess dim recovery at load. Config.Arch + Assemble are already benched in the gemma4 bench files (not duplicated). Synthetic tensor set, no file. Co-Authored-By: Virgil <virgil@lethean.io>
… mlx workload replay, fp16 A/B, sustained-clock arm Five acquittals banked tonight, each with a runnable instrument: rotating outputs (no WAW serialisation: 0.95x), the mlx wheel's own metallib through our dispatch (identical 22 TFLOPS — binary acquitted), float16 vs bfloat16 (1.09x — emulation tax acquitted), sustained 1.3s burns (22.8 flat — clock ramp acquitted), and their exact spied workload priced on our dispatches (1.50s vs their 0.74s wall at a uniform ~22 TFLOPS). The standing contradiction: the same process that ceilings at 20.5 TFLOPS on a clean 32-op qmm burst (their runtime, measured tonight) retired 33 TFLOP of spied qmm calls in 0.74s inside the real forward. Next probe is a Metal capture of their prefill (or the QQMatmul / compile-fusion hypothesis) — not a guess. Co-Authored-By: Virgil <virgil@lethean.io>
The missing brick: go-inference has the whole-model quantise pipeline (gguf.QuantizeModelPack, autoround) and the READER for the MLX native format (model.QuantConfig + engine/metal affine_qmv), but nothing WROTE the packed-uint32 + bf16 scales/biases format the engine loads. This closes the loop — "hand me a bf16, I make the quant variations". QuantizeTensor implements MLX's affine derivation exactly (reverse- engineered from and byte-verified against mlx-community's own snapshots): per inDim-group, the larger-magnitude edge is anchored to an exact code q0 = round(edge / ((wmax-wmin)/nBins)); scale = edge/q0 (signed negative when the max-edge dominates); bias = edge (0 when q0==0); codes = clamp(round((w-bias)/scale), 0, nBins) round-half-away-from-zero. All float32 — Go's native float32 reproduces the Metal kernel bit-for-bit. Layout: packed U32 [outDim, inDim*bits/32] LSB-first, scales/biases BF16 [outDim, inDim/groupSize]. Bits 2/4/8 (32%bits==0); 3/5/6 refused (no reference to verify their cross-word layout — refusing beats wrong bytes). ConvertSnapshot streams a whole bf16 model dir → a new quantised dir: eligible tensors (2-D, inDim%groupSize==0) quantised, everything else (norms, non-aligned matrices) passed through wide, config.json gains the quantization + quantization_config blocks, tokenizer/template sidecars copied. One source tensor in memory at a time (header planned from shapes first), so peak memory is bounded by the largest tensor, not the model. Eligibility rule verified against the gemma-4-12B 4-bit snapshot: 332 quantised, 292 passed through, ZERO disagreement with mlx_lm.convert. DequantizeTensor (w = scale*q + bias) added as the round-trip inverse. Co-Authored-By: Virgil <virgil@lethean.io>
ORACLE VERDICT: BYTE-IDENTICAL. TestByteOracle quantises a sample of tensors from the cached mlx-community/gemma-4-12B-it-bf16 snapshot and compares packed/scales/biases byte-for-byte against the 4-bit snapshot mlx_lm.convert produced from it. All five samples match exactly: q_proj v_proj gate_proj down_proj embed_tokens -> 1,658,880 bytes compared, ZERO divergence. So QuantizeTensor reproduces mlx_lm.convert; the format-authoring loop is verified against the reader's own producer. No drift to characterise. The test reads only a slice of leading rows per tensor via the safetensors ReadAt path (never the 12B model into RAM) and is env-gated on the two snapshot dirs (LEM_MLX_BF16_DIR / LEM_MLX_4BIT_DIR, or the HF cache), skipping cleanly when absent — like the repo's other real-model tests. Unit coverage (no model needed): synthetic round-trip within the group error bound across bits 2/4/8 and group sizes 32/64 incl. groupSize==inDim; reconstruction-stability under re-quantise (codes are a fixed point; scale can move one bf16 ULP — byte-exact idempotence deliberately NOT claimed, distinct from the oracle's byte-exactness against mlx); constant-group eps path; shape contract; and the rejection paths (3/5/6-bit, non-dividing group, length mismatch). Loadability (TestConvertSnapshot_ToyLoadable): the converter runs on a tiny synthetic 2-layer model and the output is accepted by the real config-parse/probe layer — model.ProbeDirArch reads it, the injected quantization block parses + Validate()s through model.QuantConfig, the emitted tensors carry the loader's expected U32/BF16 dtypes+shapes, the sidecar is copied, and dequant reproduces the source within the bound. Co-Authored-By: Virgil <virgil@lethean.io>
The user-facing verb the quantiser needed. Flag surface: lem quant <src-model-dir> [-bits 4] [-group-size 64] [-o <out-dir>] [-gguf <FORMAT>] Default lane is MLX group-affine (mlxaffine.ConvertSnapshot) at the given bits/group — the native format the engine loads. -gguf <FORMAT> switches to the existing GGUF whole-model pipeline (gguf.QuantizeModelPack: q4_k_m, q8_0, q5_k, q6_k, …). Default output dir mirrors the mlx_lm convention: <src>-<bits>bit (or <src>-gguf-<format>), with a trailing -bf16/-f32 tag on the source name stripped first. Per-tensor progress lines (quantise/copy) to stderr; a final before->after summary (tensor count, quantised vs passthrough, in/out sizes) to stdout. A two-phase parse lets the <src-model-dir> positional sit before OR after the flags (Go's flag stops at the first non-flag), matching the documented `quant <src> [-flags]` shape. Registered in main.go under a new "Convert" section (minimal diff). Smoked through the built binary on a synthetic model: - MLX 4-bit: 10 tensors (3 quantised, 1 passthrough), 7184 B -> 2032 B - MLX 8-bit: same set, 7184 B -> 3824 B (width flows through) - GGUF q8_0: 3 tensors -> model.gguf via the existing pipeline - both flag orderings, and the reject paths (no arg / bad bits / no shards) Co-Authored-By: Virgil <virgil@lethean.io>
…uant) The competitor-parity verb: lem quant <src> [-bits 4] [-group-size 64] [-o dir] [-gguf FORMAT]. The new model/quant/mlxaffine package writes the MLX affine format the engine itself loads — algorithm reverse-engineered from real input/output pairs and BYTE-IDENTICAL to mlx_lm.convert (oracle: q/v/gate/down/embed tensors from the cached 12B bf16 vs mlx-community's own 4bit conversion — 1,658,880 bytes, zero divergence; the pinned derivation: per-group larger-magnitude edge anchored to an exact code, signed scale, fp32 arithmetic exactly — fp64 diverges). Eligibility verified against the reference with zero disagreement (2-D + inDim%group==0; embeddings quantised; norms pass wide). bits 2/4/8 byte-exact; 3/5/6 refused (no reference to verify cross-word packing). -gguf lane rides the existing QuantizeModelPack. Output loadable (probe-gated), config.json quantization block + mlx-compat alias, sidecars copied. Streams one tensor at a time. Co-Authored-By: Virgil <virgil@lethean.io>
…lised 45-vs-22 mystery, saved traces Co-Authored-By: Virgil <virgil@lethean.io>
The deployment-owned filter on model OUTPUT: term/pattern rules with redact/refuse actions, loaded from a JSON file. The engine ships the MECHANISM, never a taxonomy — no built-in term lists, only what a deployment declares. Complements the welfare guard (inbound, the model adjudicates) with the house rules on output (outbound, the deployer adjudicates). A policy compiles ONCE: a bad regexp, an empty term, a refuse rule with no message, a term carrying a pattern-only window, or an out-of-range window is rejected at LOAD — never at serve time. Pattern matches are bounded by a window (per-rule / file / DefaultWindow) so the streaming enforcer can establish a hold-back bound; an empty-matching pattern is rejected (its bound cannot be established). Term matching folds ASCII case; the first-byte dispatch table + hold-back bound are precomputed at load for the streaming enforcer that follows. go test ./serving/policy/ — 21 passed (Compile Good/Empty/Bad×13, HoldBack term/window, Load Good/Bad). go build ./... clean; go test ./serving/... ./cmd/... — 2064 passed. Co-Authored-By: Virgil <virgil@lethean.io>
…model_type allow-list LoadTokenModelDir reached the composed hybrid loader via a hardcoded model_type switch (qwen3_5/qwen3_6/qwen3_5_moe/…). Replace the composed arm with model.LoadComposedDir — the neutral ArchSpec.Composed lookup — so a checkpoint whose model_type registers a Composed hook loads with zero engine edits (a future qwenX is a model-package init(), not a change here). mamba2 keeps its by-name branch (it registers no ArchSpec), and a narrow fallback preserves the historical composed types the registry does not yet cover (qwen3_6/qwen3_6_moe/composed/hybrid), so every type the old switch loaded still loads identically. Pin the routing contract at the model level (runnable without the metallib): every registered composed model_type resolves through LoadComposedDir, and a non-composed model_type declines so the caller falls through to the transformer loader. Co-Authored-By: Virgil <virgil@lethean.io>
…ction
Groundwork for #380 (O(N^2) per-token re-concat in the streamed reply path).
Introduces the bounded-window detection primitives WITHOUT yet wiring them into
the live handlers, and pins them to the pre-optimisation semantics via a
differential fuzz — so the fix that follows is verified byte-identical before it
lands.
New primitives (unwired this commit; serveStreaming / GenerateStream still run
the old inline loops):
* openai.contentStreamer — cumulative strings.Builder + a (maxMarkerLen-1)
rescan window over the tool-call open marker and the request's stop set.
* serving.streamStopWindow / maxStopLen — the same window over the adapter's
stop-truncated stream.
Window-bound derivation: a stop sequence or the 12-byte <|tool_call> marker can
only newly COMPLETE within the last (maxMarkerLen-1) bytes of already-emitted
content plus the new token — anything ending inside the already-emitted prefix
would have been detected (and broken / entered-tool on) at an earlier token, so
the loop never needs to rescan there. maxMarkerLen = max(len(ToolCallOpenMarker),
longest stop); a larger-than-needed window stays correct (no marker can sit
wholly inside it — that would have fired earlier).
Oracle = the exact old code path (full `emittedContent + contentDelta` concat,
whole-candidate scans) replicated verbatim. The differential fuzz drives both
paths and requires byte-identical per-token emitted slices, tool/stop boundaries,
final content, inTool state, callback sequence and source-token consumption:
* handler: 4078 every-boundary splits (15 fixtures) + 20000 random streams
* adapter: 1452 every-boundary splits (9 fixtures) + 20000 random streams
Adversarial: markers/stops split at every byte boundary (incl. mid-rune), overlapping
marker prefixes, marker-dense text, unicode + invalid-UTF8 straddling. Native
go-fuzz entry points (FuzzContentStreamer, FuzzStreamStopWindow) seeded for -fuzz.
Benches (oracle vs windowed, 2K/8K-token replies) added to measure the collapse
the wiring commit realises.
Gate: go build ./... ; go test ./serving/provider/openai/ ./serving/ (1101 pass).
Co-Authored-By: Virgil <virgil@lethean.io>
…e hybrid registration composed now registers top-level model_types (qwen3_5 / qwen3_5_moe / qwen3_next) through ArchSpec.Composed, but builtin did not import it, so a serve composition that only imported builtin left the registry entry inert — the hybrids resolved only in binaries that happened to link engine/metal's own composed import. Blank-import composed here and correct the stale "mixer/component packages carry no top-level model_type" comment (composed does now; deltanet/rwkv7 remain component-only; mamba2 is reached by the backend's SSM branch, not this registry). Co-Authored-By: Virgil <virgil@lethean.io>
The hard part done right. Matches span token boundaries, so the enforcer holds back the minimal disputable tail — HoldBack() bytes = max(longest term, largest pattern window) — and settles only positions whose full match window has arrived. A settled decision is byte-identical to running the policy over the whole text at once; the held tail is re-examined when the next chunk arrives, or flushed by Close at end of stream. Byte-exactness: a chunk with no match and no match-prefix in its tail is emitted as the SAME string (no copy) — terms dispatch by ASCII-folded first byte, so ordinary text withholds nothing and streams straight through. redact replaces the matched span; refuse ends the reply with the rule's message (finish stays clean) and swallows everything after. The audit Event carries the rule index + action ONLY — never the matched content, which the deployment may consider sensitive. Longest-match wins; equal-length ties resolve to the earliest rule. go test ./serving/policy/ — 37 passed, incl. TestPolicy_Enforcer_ Differential (1000 random chunkings × 4 texts, and the refuse variant) proving byte-identity across every boundary split, byte-at-a-time BoundarySpanning, case-fold, precedence, post-refuse swallow. bench (no-match hot path, b.ReportAllocs): BenchmarkPolicy_Enforcer_NoMatch 227.9 ns/op 0 B/op 0 allocs/op BenchmarkPolicy_Enforcer_NoMatch_Pattern 2530 ns/op 112 B/op 1 allocs/op go build ./... clean; go test ./serving/... ./cmd/... — 2080 passed. Co-Authored-By: Virgil <virgil@lethean.io>
…th (#380)
Wires the contentStreamer / streamStopWindow primitives (landed with their
differential-fuzz oracle in the prior commit) into the live handlers, replacing
the O(N^2) per-token re-scan of the whole accumulated reply.
serveStreaming (openai/handler.go):
before: candidate := emittedContent + contentDelta // full concat per token
core.Index(candidate, ToolCallOpenMarker) // full scan per token
firstStopSequenceCut(candidate, stops) // full scan per token
after: out := cs.step(contentDelta) // append + bounded window scan
GenerateStream / ChatStream (adapter.go):
before: full.WriteString(tok.Text)
truncated := applyStopSequences(full.String(), stops) // full scan per token
after: streamStopWindow(seq, stops, cb) // bounded window scan
Detection now scans only the last (maxMarkerLen-1) bytes of already-emitted
content plus the new token — the sole region a stop sequence or the tool marker
can newly complete in. Emitted bytes, tool/stop boundaries, finish reasons,
tool-call extraction, thought deltas and callback sequences are byte-identical
(24078 handler + 21452 adapter differential-fuzz cases, all existing package
tests green unchanged).
Bench (oracle = old path, windowed = new; -benchtime=50x/30x -benchmem):
contentStreamer 2K: 11324496 -> 46584 B/op (243x), 2048 -> 16 allocs, 1.99ms -> 49us
contentStreamer 8K: 184349672 -> 211706 B/op (871x), 8197 -> 21 allocs, 27.7ms -> 159us
streamStopWindow 2K: 20.0ms -> 76us/op (264x) [bytes flat: the O(N^2) was the scan]
streamStopWindow 8K: 294.2ms -> 170us/op (1736x)
The N-scaling is quadratic on the old path (2K->8K: contentStreamer B/op 11.3MB->184MB
= 16x for 4x tokens) and linear on the new (46KB->212KB = 4.5x).
Gate: go build ./... ; go vet + go test ./serving/provider/openai/ ./serving/ (1101 pass).
Co-Authored-By: Virgil <virgil@lethean.io>
… falls (#380) serveStreaming rebuilt candidate = emittedContent + contentDelta and re-scanned the full reply per streamed token; adapter GenerateStream/ ChatStream full-scanned per token with bytes flat. Detection needs only the last maxMarkerLen-1 bytes plus the delta (a marker completing earlier would have fired earlier — derivation in the commit chain); absolute positions map through a windowStart offset so every emitted slice is byte-identical. Oracle-first: the verbatim old path retained as a differential reference; 24,078 + 21,452 fuzz cases (every-byte boundary splits incl. mid-rune, overlapping prefixes, invalid UTF-8) require identical emissions, boundaries, callbacks, and consumption. contentStreamer 8K: 184,349,672 -> 211,706 B/op (871x), 27.7ms -> 159us; adapter stop-window 8K: 294ms -> 170us (1736x). N-scaling confirms quadratic -> linear. Native fuzz entry points seeded for CI. Co-Authored-By: Virgil <virgil@lethean.io>
WrapResolver decorates the resolver so every resolved model's Chat stream runs through a fresh policy Enforcer. It composes OUTERMOST — after the welfare wrap — so the policy enforces on the final tokens the deployment would otherwise emit. A refuse stops consuming the inner stream (the model stops generating); a clean stream forwards byte-for-byte; end-of-stream flushes the held tail. One audit line per enforcement (rule index + action, never the matched content). `lem serve -policy <file>` loads and compiles the policy up front: absent means no layer and zero overhead; a load failure is FATAL at boot — RunServe returns before binding the listener, so a deployer who asked for a filter never silently serves without it. Verified with the built binary: `-policy <bad>` exits 1 with "serving.RunServe: outbound policy <path> — refusing to serve unguarded: policy.Compile: rule #0: compile pattern "[": error parsing regexp ..." and never binds; `-policy <good>` logs "serve: outbound policy ON — 3 rule(s), hold-back 256B; redact/refuse on model output, audited per enforcement; -policy disables" after the welfare line (outermost). go test ./serving/policy/ — 42 passed (incl. WrapResolver Redact/Refuse/ Passthrough across split tokens, ResolveError propagation, Audit no-leak). go build ./... clean; go vet clean; go test ./serving/... ./cmd/... — 2085 passed. Co-Authored-By: Virgil <virgil@lethean.io>
…dels metalBackend.LoadModel type-asserted *NativeTokenModel and failed every other loaded model, so a composed hybrid (Qwen 3.6's host-f32 gated-delta / full-attention stack) loaded but could not be served — loadComposed TokenModel's result never reached an inference.TextModel. Wrap it: a composedTextModel adapts the composed model.SessionModel to engine .TokenModel (an engine.Session bridge that delegates decode wholesale to the composed model's own tested token loop, model.GenerateSampledWith StopTokensTransformEach — no decode logic re-rolled; the recurrent state has no portable KV snapshot, so the capture methods report that boundary honestly), and metalBackend.LoadModel returns it through engine.NewText Model exactly like the native path. The wrap DECLARES its chat dialect (engine.ChatTemplateDeclarer): ChatML for the Qwen model_types (config-driven via the new composed.ChatMLDialect — keyed on config.json's model_type, so a future qwenX is ChatML with zero edits), the gemma fallback otherwise. A declared template wins over engine .TextModel's tokenizer detection, so a Qwen checkpoint frames <|im_start|>/<|im_end|> turns with the <think>\n\n</think>\n\n no-think block even though its tokenizer carries no <|turn> marker. Runnable proof (no metallib): ChatMLDialect selection, and ChatML render + precedence (declared beats a tokenizer that carries <|turn>) through engine.NewTextModel, mirroring engine/chat_template_test.go's fake idiom. The engine/metal wrap itself compiles here but its live serving path needs the metallib (TestMain gates the package), so it is not runtime-exercised in this worktree. Co-Authored-By: Virgil <virgil@lethean.io>
Deployer house-rules on model output — the welfare guard's outbound sibling with the authority flipped. Mechanism only, no shipped taxonomy: term rules (ASCII case-insensitive, first-byte dispatch, 0 allocs on the no-match hot path) and pattern rules (compiled at load, bounded window, unbounded/empty-matching patterns rejected at load), actions redact/refuse. Streaming enforcement settles only positions whose full match window has arrived — byte-identical to whole-text application under a 1000-chunking differential fuzz. Composed OUTERMOST above the welfare wrap; policy load failure is fatal before bind; audit lines carry rule index + action, never the matched content. G2 mediated self-rewrite stays designed-next (#378). Co-Authored-By: Virgil <virgil@lethean.io>
…ry routing + ChatML declaration (#379) The wiring agent found the real gap: metalBackend.LoadModel hard-asserted *NativeTokenModel, so composed models (the whole Qwen hybrid family) were never servable end-to-end. A composedTextModel wrap now bridges ComposedTokenModel into engine.NewTextModel via a session that delegates decode to composed's own tested loop; KV snapshot/restore reports its boundary honestly (recurrent state has no portable snapshot yet — multi-turn -state is the follow-up). The hardcoded version allow-list yields to the ArchSpec.Composed registry (qwen3_7 loads with zero engine edits; narrow fallback kept for four historical ids the registry doesn't cover). model/builtin blank-imports composed so serve binaries carry the registration. ChatML declared config-driven (qwen-prefix predicate) through ChatTemplateDeclarer — declared template beats tokenizer detection (precedence golden), gemma fallback untouched. 3251 tests green in-branch; the composed live-serve path needs a metallib runtime smoke on this box (bridge is thin plumbing over composed's tested primitives). Co-Authored-By: Virgil <virgil@lethean.io>
…l parity mechanism Co-Authored-By: Virgil <virgil@lethean.io>
…ill chunks — e2b pp8K 3190->6777 tok/s (#381) The mlx prefill-parity mechanism, ported: on gemma4 E-family, the trailing 20-of-35 KV-shared layers own no cache rows, and a non-final chunk's outputs feed only that chunk's unread boundary hidden — so their compute is dead on every chunk except the last (mlx-lm's per-chunk cache-state eval lets lazy DCE prune exactly these layers; #381 traced its 2-3x prefill lead to this). The port: sharedLayerSuffixStart validates a clean non-owner suffix at state build (interleaved shapes disable the skip); prefillRetainedTokensBatchedDenseChunks arms prefillSkipToLayer for non-final chunks only; the batched pass bounds its layer loop there. Owner layers still land every KV row; the final chunk and all decode/verify passes run the full stack. Token-identical by construction and by receipt. LTHN_PREFILL_SKIP_SHARED=0 restores full-stack chunks. Receipts (e2b 4bit, M3 Ultra, build/dist metallib): pp8K (7225 tok): 3190 -> 6777 tok/s (wall 2.265s -> 1.066s, 2.12x) pp62K (56712 tok): 2323 -> 5833 tok/s (wall 24.41s -> 9.72s, 2.51x) 32-token greedy continuations byte-identical skip-on vs skip-off at both depths. TestArchSessionPrefillChunksSkipSharedSuffix: serial-vs-chunked byte identity over a kv-shared fixture, both lanes, plus next-token step. v1 scope: token-ids chunk lane only (the embeddings/bidir chunk lane keeps the full stack); the PLE slab is still gathered full-width per chunk — both are follow-up cuts. Chunk-width retune under the new balance is open (#381). Co-Authored-By: Virgil <virgil@lethean.io>
…uard (#379) composedEngineSession is the registry bridge for composed (hybrid-stack) archs — "composed" names an arch class, like Arch, not a model, so it meets the guard's neutrality bar. The full metal_runtime sweep is green again with this entry. Co-Authored-By: Virgil <virgil@lethean.io>
… bank the stale-metallib trap Co-Authored-By: Virgil <virgil@lethean.io>
…ip — e2b pp8K 6777->8049 tok/s (#381) With the skip armed, only the FINAL chunk pays the full layer stack — but the absorb policy was handing it up to wide+w/2 rows (1081 at pp8K) when only its LAST row's hidden is ever read. batchedDensePrefillChunkLenSkip splits a minimal window-aligned boundary chunk off the end instead: the last partial window, raised to batchedDenseSkipFinalFloor (32 — above the recorded-ICB decline at 16 and at the q8 fold gate, so no lane falls back). pp8K's full-stack span drops 1081 -> 57 rows. Receipts (e2b 4bit, on top of 473c242's skip): pp8K: 6777 -> 8049 tok/s (wall 1.066s -> 0.898s; 3190 pre-campaign = 2.52x) pp32K: 7008 -> 7195 pp62K: 5833 -> 6040 pp118K: 4249 -> 4305 32-token greedy continuations byte-identical skip-on vs skip-off at 8K + 62K. Full metal_runtime sweep green. Chunk-width sweep re-receipted under the skip: 2048 stays the peak (1024: 6453, 3072: 6402, 4096: 4986 at 8K). Co-Authored-By: Virgil <virgil@lethean.io>
Co-Authored-By: Virgil <virgil@lethean.io>
…t gather — e2b pp8K 8049->8528 tok/s (#381) Two cuts to the per-chunk PLE tower build, the largest host-side span of a skipped-chunk prefill (104ms of an 8K wall): - lthn_ple_gather_rows_quant: ONE dispatch dequant-gathers all K rows (the K-loop of per-token gathers paid encode+launch per token). The bf16 twin gains lthn_ple_gather_rows_bf16_pfx with the row STRIDE split from the gathered width. Old kernels stay — a stale metallib falls back to the loop / full width. - Bounded slab: a skipped chunk (#381) reads only the owner layers' gate slices, so pleSlabFor now allocates the layer-major prefix and the batch builders derive the bound from the slab length (closures unchanged) — gather width, projection rows (a prefix of projW), elementwise, relayout and both copies all run at 15/35 of the width. When the bounded qmm_t instantiation is missing (QAT) the builder computes full width and copies the prefix — identical bytes either way. Sub-span instruments (pleSlab.ensure/stage/gpu/out) stay in. Receipts (e2b 4bit, on top of c257ff0): pleSlab host span @8K: 104.1ms -> 47.1ms steady-state (gpu 37.1 / out 4.1 / stage 3.5) pp8K: 8049 -> 8528 tok/s (wall 0.847s) pp32K: 7195 -> 7556 pp62K: 6040 -> 6285 pp118K: 4305 -> 4437 32-token greedy continuations byte-identical skip-on vs skip-off at 8K + 62K. Full metal_runtime sweep green (PLE parity suite exercises the new gather). Co-Authored-By: Virgil <virgil@lethean.io>
Co-Authored-By: Virgil <virgil@lethean.io>
…blocks the host (#381) The quant PLE builder committed its command buffer and WAITED so the host could copy the slab out — which the pass then uploaded straight back. Both sides of that round-trip are dead: the pass's own command buffer follows on the SAME queue, so the GPU orders the read after the build with no host involvement at all. perLayerInputsBatchQuantEncode is the shared core (encode + commit, optional wait); IntoSlab keeps the wait+copy contract for existing callers, the new perLayerInputsBatchQuantDevice returns the layer-major tensor's buffer committed-not-waited at exactly the chunk's layer bound. pleSlabForPrefill resolves device-first; the pass binds prefillPLESlabDevice directly instead of pleSlabBuffer(hostSlab). Single-buffered scratch stays race-free: the host stages the next chunk only after the pass's wait, which covers the builder's buffer too. Host-slab path unchanged for QAT-without-bounded-qmm, small K, and stale metallibs. Receipts (e2b 4bit, on top of a0364b5): pleSlab host span @8K: 47.1ms -> 3.1ms (encode-only; wait/copy-out/upload gone) pp8K: 8528 -> 8708 tok/s (wall 0.830s) pp32K: 7556 -> 7681 pp62K: 6285 -> 6359 pp118K: 4437 -> 4469 32-token greedy continuations byte-identical skip-on vs skip-off at 8K + 62K. Full metal_runtime sweep green. Co-Authored-By: Virgil <virgil@lethean.io>
Co-Authored-By: Virgil <virgil@lethean.io>
…op dies, e2b pp8K 8708->9016 tok/s (#381) The last host-side per-token work in a prefill chunk was the embedding dequant loop (embedInto per token — 17.6ms of an 8K wall). The quant PLE builder's command buffer now gathers the K main-embed rows too (the same rows kernel, dModel width), the projection reads them in place of host-staged hidden rows, and the batched pass takes the same buffer as its input rows (directInputRows + i·rowBytes offsets) — only the token ids ever cross to the GPU. perLayerInputBatchDevice drops the embs parameter entirely; prefillInputsDevice is the one-call device-first seam, and every fallback (bf16 sessions, stale metallib, small K, QAT-without-bounded-qmm) keeps the host embed loop + host slab unchanged. Receipts (e2b 4bit, on top of cfc84d5): embed host span @8K: 17.6ms -> 0 (inputsDev 0.8ms encode-only) pp8K: 8708 -> 9016 tok/s (wall 0.801s) pp32K: 7681 -> 7849 pp62K: 6359 -> 6474 pp118K: 4469 -> 4522 32-token greedy continuations byte-identical skip-on vs skip-off at 8K + 62K, and byte-identical to the pre-gather build's output. Full metal_runtime sweep green. Night cumulative: pp8K 3,190 -> 9,016 (2.83x); mlx-lm true-wall gap 1.11x. Co-Authored-By: Virgil <virgil@lethean.io>
…2.83x (#381) Co-Authored-By: Virgil <virgil@lethean.io>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces several important updates across the documentation and codebase, focusing on improved benchmarking, enhanced documentation for recent engine features and campaigns, and significant performance and feature upgrades in the ngram drafter and welfare guard. The changes also clarify the energy measurement methodology for different engine modes and update command-line options for the welfare guard.
Documentation and Feature Updates:
Major engine and campaign documentation:
docs/handover.md: Added comprehensive session and campaign logs, including performance wins, folder retirements, welfare guard deployment, and details on the prompt reuse mechanism and Qwen model integration.docs/release-v0.12.0-DRAFT.md: Clarified energy measurement methodology for "pure replay" (no cache) and stateless prompt reuse, and expanded on the welfare guard's mediation process, including the newlem_endrung and its persistence requirements. [1] [2] [3] [4]docs/examples/energy-ab/README.md: Updated instructions for measuring energy in replay mode to reflect the new default prompt reuse, requiring the kill switch for cache-less measurements.Engine and Benchmarking Improvements:
Benchmarking and performance:
go/decode/generate/generate_bench_test.go: Added a comprehensive set of benchmarks for prompt prefix handling, image data URL decoding, kv storage encoding, and cache mode handling, improving coverage and enabling allocs/B-op tracking.go/decode/ngram/ngram.go: Refactored the ngram drafter's core lookup algorithm to scan only once, anchored on the suffix's last token, reducing work by up to maxNgram× on misses and ensuring byte-identical output with significantly less scanning. [1] [2]Serve command and welfare guard:
go/cmd/lem/serve.go: Added a-welfareflag (default ON) to enable or disable the per-turn welfare guard, and wired the flag into the engine options. [1] [2]Summary of most important changes:
Documentation and Feature Clarifications
lem_endrung and its persistence, with live deployment details and audit trails. [1] [2] [3]Performance and Benchmarking
Serve Command Enhancements
-welfareflag to control the welfare guard at serve time, defaulting to ON. [1] [2]