Conversation
…del_tuning.md AGENTS.md was 443 lines, 88% of it the Presets knowledge base, loaded into every session regardless of task. The rationale is now read on demand via a Reference table, with an always-on Traps section carrying the prohibitions whose violation is a silent OOM or silent corruption. Content moved verbatim; verified line-for-line against the previous revision.
Condenses Non-obvious behavior to one-line rules, each pointing at the section that now carries its reasoning. Consolidates the three hardcoded vendor paths, previously documented in three separate places, into one post-bump checklist. Notes in the pr-code-review checklist that its restatement of these rules is deliberate, so a review pass stays self-contained.
AGENTS.md has required scratch files under .tmp/sessions/<session-id>/ without anything supplying the id, so the existing session directories use three different ad-hoc naming schemes. Claude Code gets a SessionStart hook; OpenCode gets a plugin on experimental.chat.system.transform. The hook uses Windows PowerShell to parse the stdin payload rather than jq, which is not present on this platform. It is invoked through cmd.exe, where it is verified working; the bash on PATH here is WSL and cannot reach either PowerShell binary. The plugin needs no package.json because it imports nothing. Also stops tracking .claude/settings.local.json, which is machine-local.
Ports the routing-plus-gauntlet retro from the fertilizer_management repo so the new docs/ layout has a maintainer. Routing targets are this repo's files, and the table records that traps live in three places by design so a candidate proposes the edit to all of them at once. The process-worked gauntlet test carries a scope note: with no tests, linter or CI here, the only automated gates are the build and llama.cpp's own startup validation, so the test kills far fewer candidates than it does upstream.
Measured against the last 40 commits: presets/ is touched by 24, the llama.cpp submodule pointer moves in 6, rebuild_llama.cpp.ps1 is edited in 2, examples/ in none. Non-obvious behavior was spending always-on context on build-configuration detail at that bottom rate, so the five summaries whose full text already sits in docs/build_system.md are dropped; the Reference trigger already names every one. Adds the three examples scripts that were never named anywhere, points at README.md Usage as the end-user inventory instead of restating flags, and records which scripts actually answer Get-Help. Also removes the gguf_dump.py bullet that restated its own table row.
gguf-py's _apply_over_grouped_rows collected every 16-row group in a Python list and merged them with one np.concatenate(..., out=out). That keeps all groups alive at once and costs an extra full copy of the result, and it degrades with tensor size instead of staying linear. Converting Qwen3.8-Flash-Next, the ffn_down_exps tensors (512 x 2560 x 640 f32) ran at 13.2 MB/s while the gate/up_exps layout (512 x 640 x 2560) managed 348.8 MB/s for the same byte count, the only difference being 4x fewer groups. Writing each group straight into out, and sizing groups by bytes so one group stays in L2 at any row length, gives a uniform ~500 MB/s and takes the per-tensor pipeline from 126.7 s to 4.7 s. What is left is the torch.stack upcast, which this does not touch. A second hunk drops a np.uint64 widening in BF16.quantize_blocks, worth a further 21%. The nan fixup directly above it caps n, so the add cannot overflow uint32. Both hunks are byte-for-byte identical to upstream. Verified against the C implementation through gguf-py/tests/test_quants.py (25 types, 60 exact matches, output unchanged apart from a shifted warning line number), across all 24 registered quant types in both quantize and dequantize directions, and for the uint32 kernel exhaustively over all 2^32 float32 bit patterns. The fix ships as ./patches/0001-gguf-py-write-row-groups-directly.patch because each build resets the submodule to origin/master. The new step applies patches after the -version / PR checkout and throws on failure, so an upstream move cannot silently leave an unpatched, slow tree. git apply --3way implies --index, so the step unstages afterwards to keep the submodule showing the same plain worktree modification as the OpenBLAS shim.
The patch bullet claimed the old code cost an extra full copy of the result. It does not: np.concatenate(..., out=out) and the slice assignment that replaced it each copy every element exactly once, and both allocate one array per group. What the patch removes is peak intermediate memory, which equalled the whole output, plus the cache miss on re-reading groups that had gone cold. Restated accordingly, and the degradation is now tied to the group count rather than to tensor size, which is what the gate-up versus down comparison actually isolates. The 160 KiB group budget is computed from the input row bytes, so the dequantize direction produces a group several times larger than that. Noted, because the old wording promised L2 residency in both directions. The uint32 overflow argument now carries its bounds instead of asserting the conclusion: the NaN fixup leaves n at most 0xFFFF0000 with a zero low half, everything else is capped by -inf at 0xFF800000, so n + 0x7fff + 1 stays at or below 0xFFFF8000. The C reference gets the same guarantee from an early return that the vectorised form cannot use. A new bullet records that this is not the per_layer_token_embd OOM fix. That table reaches _apply_over_grouped_rows one shard at a time through LazyChunkedTensor, so the function never sees more than about 1.6 GB of it, and peak RSS there was solved by 53c0f624a and a510c82e1. a510c82e1 also landed after the 2.2x figure was measured, so that number needs re-measuring before it is quoted.
…A build flags v22.1 moved the reasoning-effort default from xhigh to medium, so the Qwen3.8-27B entry, which relied on the template default, was silently downgraded at the bump. It now pins xhigh explicitly. model_tuning.md records the new level semantics: medium is the level that injects no instruction text, unknown levels now map down to medium rather than up to xhigh, and a think-tag typed inside a message can override the kwarg. Adds why xhigh is kept rather than lowered, and why --reasoning-budget is the guard rail for it. Qwen publishes no per-level benchmarks and the whole mechanism is one injected sentence, 123 rendered chars against 332. The failure that moved the upstream default to medium was a truncation artifact: with no budget, xhigh ran 26000 tokens and returned empty content because truncation landed inside the think block, while the same rig at xhigh with a 1500-token budget returned more working code than medium did. Records the standalone mtp-Qwen3.8-27B-Q8_0.gguf sidecar as a second remedy for the Q4_0 blk.64 head, and why it is not taken: load_mtp comes from the type list rather than from a draft path, so the target keeps loading its own head and an external sidecar double-pays. Corrects the Muse Glimmer claim that no bundled template exists. models/templates/muse-glimmer.jinja is a test fixture mirroring the GGUF-embedded template, so not pinning it is deliberate rather than forced. build_system.md: the missing GGML_CUDA_FA_ALL_QUANTS failure is a silent placement of FLASH_ATTN_EXT on the CPU backend, not the GGML_ABORT, because supports_op consults the support query first. Also documents -DGGML_SCHED_MAX_COPIES=1 and what it keeps this repo out of. Stale llama.cpp line references refreshed throughout. The vendor/llama.cpp pointer is left uncommitted: the worktree is on the PR 27742 head, which is not reachable from origin/master, and every pointer this repo has committed so far is a master commit.
…-host requirements Qwen3.8-Flash-Next is arch qwen4exp, not a variant of the qwen35 family, so almost none of the Qwen3.8-27B tuning carries over. 48 blocks split 12 full-attention layers with QSA block-sparse attention over an indexer cache and 36 gated-delta-net layers, 512 experts with 10 used, native context 262144, qwen3vl_merger projector. The local IQ4_XS is 90.635 GiB and decomposes as 26.822 GiB per_layer_token_embd, 60.938 GiB routed experts at 1.270 GiB per layer, and 2.542 GiB of everything else. This is the one entry in the tier that must use load-mode = mmap. The 26.822 GiB n-gram hash table is created with TENSOR_READ_LAZY and the loader gates that flag on use_mmap, which every non-mmap load mode clears. A token gathers 16 of the table's 320 million rows, so mmap keeps the resident working set in the hundreds of MiB while dio reads and holds all 26.822 GiB for nothing. no-host = true is part of the same mechanism rather than an independent choice: without it the CPU buffer list is prepended with CUDA_Host, the chosen buffer type then fails the is_default_buft test and the mmap aliasing branch is skipped entirely, so every CPU-resident tensor goes through cudaMallocHost and the load dies as the same misleading CUDA OOM that DeepSeek hit. The table itself is unoffloadable: PER_LAYER_TOKEN_EMBD is classified LAYER_INPUT and input tensors are pinned to the CPU list regardless of -ngl, so -ngl is not the lever and the only question is how many expert layers survive beside the KV cache. fit handles that at sub-layer granularity and its overflow pattern matches this arch's _exps names, but it is blind twice over: it assumes host memory is unlimited and never consults the CPU slot once a GPU is present, and it measures with load mode NONE so its host figure counts the table non-lazily. Neither matters on this box, but they are why the split cannot be checked against fit's own host accounting. cache-type-k stays at q8_0 because it also types the QSA indexer cache, whose K is what top_k ranks blocks on, so cheapening it changes which tokens are attended rather than only their values. Two thirds of the indexer allocation is dead weight: hybrid_idx sets n_embd_head_k_full but not n_embd_head_v_full, so is_mla is false, a 256-wide V is allocated, and the graph only ever calls cpy_k and get_k. Dropping cache-type-v to q4_0 would recover 1152 MiB at 262144 with the cost paid only by the 12 real attention layers; it is left for a later measured pass. Unlike deepseek4 the two types may legally differ here. There is no MTP head, so ngram-mod is the only speculative type. IMRoPE forces n_pos_per_embd to 4, which makes get_can_shift false, so the server force-disables context shift and cache reuse with two warnings that are expected on startup; speculative rollback then goes through checkpoints, which is what makes ctx-checkpoints load-bearing rather than an optimisation. no-mmproj-offload = true because fit measures the language model alone and CLIP allocates after the split is committed, the exact silent-OOM window presets.md warns about. The chat template pin is the same file as Qwen3.8-27B's, byte for byte at 8952 bytes, so the three template defects and their fix carry over unchanged, along with reasoning-effort = xhigh. The vendored template renders the vision placeholders, so pinning it costs nothing on the image path. The vendor/llama.cpp pointer is bumped to b10660, the merge commit of upstream 27742 and reachable from origin/master, which is what makes this preset loadable at all. presets/README.md carried two stale claims from the v22.1 bump: that the vendored template defaults to xhigh and that Qwen3.8-27B leaves reasoning-effort unset. Both are corrected. Not yet verified against a running server: the build was still linking when this landed. The weight decomposition and the KV arithmetic are exact, but fit's chosen split, the QSA compute buffer at 262144, the per-checkpoint size and throughput are all unmeasured, so fit-target = 3072 and ctx-checkpoints = 32 are carried over from the DeepSeek entry rather than fitted to this model.
…ceiling The entry loaded and ran on the first try, so this replaces the arithmetic-only figures from the previous commit with measured ones: on a 24463 MiB card at ctx-size 262144, parallel 1, q8_0 K and V and CLIP on the CPU, the child sits at 20174 MiB used with 3964 MiB free and generates 19.87 t/s at short context. The free figure tracks fit-target 3072 plus roughly 0.9 GiB of slack, so the margin is neither wrong nor obviously over-provisioned and stays as it is. parallel goes to 4 because it is nearly free here. kv-unified makes n_ctx_seq equal n_ctx, so the four slots share the 262144-cell pool rather than each being handed one; the KV cache does not change size and the only VRAM the extra slots add is three more recurrent rows, 336.7 MiB against 3964 free. A single long conversation can still occupy the whole pool. ctx-checkpoints drops from 32 to 8 in the same breath, because raising parallel without that would have quietly reserved 14.3 GiB of host RAM. A checkpoint on this arch is not the cheap thing it is on DeepSeek: checkpoints are written with PARTIAL_ONLY, that flag skips the attention cache and the indexer cache, and what is left is the entire 112.219 MiB recurrent state rather than a 14.5 MiB window. Checkpoints are also per slot, so the budget is parallel times ctx-checkpoints times 112 MiB. 8 holds it at the same ~3.6 GiB that parallel 1 with 32 checkpoints would have cost. A 1M cell pool is not reachable on this card and the reason is arithmetic rather than tuning. Going from 262144 to 1048576 takes the KV cache from 4488 to 17952 MiB at q8_0, plus 13464, against 3964 MiB free and at most ~9 GiB recoverable by moving every remaining expert layer to the CPU, and the QSA bias tensors are shaped [n_kv, n_tokens] so the compute buffer grows fourfold on top of that. Even a config that squeezed in would be a throughput regression, since zero expert layers on the GPU is strictly worse than the current split. parallel is not a way around it either: under kv-unified four slots of 262144 is 1048576 cells at identical cost. The only route to a 1M pool would be q4_0 K and V, and cache-type-k is the one value that should not drop because it also types the indexer whose K ranks attention blocks. Also records that a large pool needs no override-kv on this model, unlike Muse Glimmer: n_ctx_train is already 262144, so the server caps each slot there and the capping warning is expected. The server-context line references for that cap were stale in presets.md and are refreshed.
…v22.4, and restate 1.41.0 as a release ctx-size goes to 524288 with parallel 2. Under kv-unified n_ctx_seq equals n_ctx while the server still caps each slot at n_ctx_train, so this is two slots that can each independently reach the model's native 262144, rather than four slots contending for one 262144 pool. It needs no override-kv because n_ctx_train is already 262144. It is not free, and the cost is invisible in "VRAM used" because fit always fills to the fit-target margin. llama-fit-params gives the fixed cost per n_ctx_seq directly: 1234 context and 573 compute at 65536, 4600 and 1821 at 262144, 9088 and 3497 at 524288, 18064 and 6829 at 1048576, plus 112 MiB per sequence beyond the first. So the shipped config's fixed cost is 12697 MiB against 6758 for 262144 with parallel 4, and that ~5900 MiB comes straight out of GPU-resident expert layers, roughly halving them from ~8.3 of 48 to ~4.4. Measured: 19950 MiB used, 4513 free, 14.90 t/s on a 400-token prose completion. The docs record that a single conversation is capped at 262144 either way, so on a single-user workload the larger pool is paid for and unused. The chat template submodule moves from 9f14778 to e649070. The pointer had never left v22 despite the CHANGELOG claiming a v22.1 bump and model_tuning.md describing v22.1 semantics that were not checked out; upstream had reached v22.4. Because every entry pins reasoning-effort explicitly, the default-effort move to medium cannot reach us: rendering both versions across plain chat, system prompt, thinking-off, multi-turn-with-thinking, vision and both tool-argument wire formats is byte-identical wherever the level is pinned. The single behavioural delta is an empty think block before a historical tool call whose assistant message carried no reasoning, which is deliberate upstream token alignment with what Qwen itself generates, and is suppressed by round-tripping reasoning_content, thinking, message.reasoning or an inline think block. Verified end-to-end through minja rather than only in jinja2: a tool call is emitted and parsed, and the follow-up turn returns finish_reason stop with correct synthesis for dict arguments, JSON-string arguments, and with reasoning present. 1.41.0 is unreleased, so the CHANGELOG and docs now state what is in it rather than what happened between iterations. The last released template was v19, so the net change is v19 to v22.4 and the intermediate v22 and v22.1 references are gone. Bullets that corrected or revised something introduced within 1.41.0 itself are removed rather than recorded, since no reader ever saw the intermediate state: the parallel 4 step, the gguf-py row-group cause correction, and both presets README corrections. Date set to today. Also documents that the desktop runs on the Intel iGPU, which is why the NVIDIA card reports 23151 of 24462 MiB free and fit-target 3072 is covering CUDA and WDDM scratch rather than a compositor.
…n its load-mode section The fit-target section restated docs/model_tuning.md:445-459 - the cudaMemGetInfo snapshot at t=0, the absent WDDM and framebuffer allowance, and the untracked CUDA VMM scratch, cuBLAS workspace and graph instances - in a second file, with the same citations and no link between them, so the two would have drifted apart at the first change. The one fact it added, that the desktop runs on the Intel iGPU and the card therefore reports 23151 of 24462 MiB free, does not change what a reader does. model_tuning.md already records the measured consequence: 1368 MiB left for untracked consumers at the default margin against 3497 at fit-target 3072, and one extra expert layer on the CPU as the price. It was written to make a commit message true rather than because the documentation needed it. Removing it surfaced two defects in the adjacent load-mode section, both introduced when the Qwen3.8-Flash-Next entry landed. The section still opened with "All entries use load-mode = dio" immediately above the bullet explaining that Qwen3.8-Flash-Next must not, and it put the per-token PLE gather at 8 rows where it is 16: ple_n_heads is (ngram_size - 1) * heads_per_ngram, 2 * 8, at src/models/qwen4exp.cpp:64. model_tuning.md already carried the corrected figure, so the two files disagreed.
_apply_over_grouped_rows split rows into groups of 16 rows, so the group's byte size followed the row width. The replacement grows the group until it holds 160 KiB of input but never goes below 16 rows, leaving np.array_split and np.concatenate(..., out=out) untouched. Full scale on Windows with numpy 2.2.6, a 640-wide f32 row goes 129.4 s -> 4.8 s (27.0x); across an 18-point width ladder from 128 to 16384 the patched form is flat at ~1.13-1.18 s where upstream ranges 1.12-5.13 s. The cause is the Windows low-fragmentation heap and the trigger is the retained result, not the working buffer. Microsoft documents that the LFH does not serve allocations above about 16 KiB; once the retained group result crosses that, it interleaves with the next group's transient temporaries, fragments the general heap, and per-allocation cost grows roughly with the square of the live block count. Measured 3.7 us/alloc flat from 1 K to 65 K live blocks at 8 KiB retained, against 13 -> 526 us/alloc over the same range at 32 KiB. The cliff was located by varying the result-to-temporary size ratio and watching it move with the retained size, staying pinned to the documented ceiling. A pure allocator microbenchmark with no gguf code reproduces it, and the same script on glibc is flat with no band at any size. The 16-row floor is deliberate. Without it the budget asks for fewer than 16 rows once a f32 row exceeds 2560 elements, which regresses Linux by about 1.2x at widths 4096-8192, reproduced on WSL1 and on a Debian 13 / kernel 6.12 / glibc 2.41 VM at repeats 7 and 15. The floor makes the change a no-op for every width the budget cannot improve, so it cannot be slower than upstream anywhere. The price is Windows throughput at wide rows: 7.57 s against 4.79 s unfloored at 5120-wide full scale. Three alternatives were measured and rejected. A larger fixed row count only relocates the fault, since the pathology follows rows x width x itemsize: 64 rows fixes 640-wide and breaks 160-wide. Writing each group straight into out regresses 2.2x on numpy 1.26.4, which upstream CI pins, and 2.3x on glibc for wide rows. Bounding the retained block to 16 KiB directly, which the mechanism suggests, loses 1.18-1.24x to the input budget at every scale. The previous patch also dropped a np.uint64 widening in BF16.quantize_blocks. That hunk is worth 1.41x on Windows and 1.84x on glibc but is a separate concern touching a line upstream added on purpose in #7843, so it is no longer carried here. Verified byte-for-byte against the C implementation through gguf-py/tests/test_quants.py (25 types, 60 exact matches, 0 mismatches), across 10 type and width combinations in both directions on Windows/numpy 2.2.6, Windows/numpy 1.26.4 and glibc/numpy 2.2.6, and over 15 adversarial shapes covering a single row, row counts below one group, prime row counts that leave a remainder, and rows up to 262144 wide. Assisted-by: Claude Opus 4.6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.