Skip to content

Prebuilt: pin the penalties sampler fix (#95) - #96

Merged
danielhanchen merged 1 commit into
masterfrom
pin-penalties-sampler-95
Aug 11, 2026
Merged

Prebuilt: pin the penalties sampler fix (#95)#96
danielhanchen merged 1 commit into
masterfrom
pin-penalties-sampler-95

Conversation

@danielhanchen

Copy link
Copy Markdown
Member

Pins c26185b from #95 into the nightly prebuild set.

#95 stops llama_sampler_penalties_apply from probing the frequency map once per candidate. It now walks token_count (at most penalty_last_n entries) and indexes cur_p directly, with a guard that falls back to the old scan when an earlier sampler has reordered or dropped candidates.

This matters for the prebuilds because our own Studio presets enable it: the qwen3.5 and qwen3.6 family profiles set presence_penalty: 1.5, and on a 248320 token vocab the old scan cost about 11% of generation throughput.

Qwen3.6-27B-UD-Q4_K_XL, 400 tokens, temperature 0, median of 4, throughput from llama.cpp's own /metrics:

presence_penalty build server t/s
1.5 stock 130.29
1.5 patched 146.66

Output is unchanged: 16 stock vs patched comparisons across 4 prompts, both penalty settings, greedy and temperature 0.8 with a fixed seed, all byte identical. tests/test-sampling passes.

Per the notes in pr-set.json the entry stays listed until the change lands upstream, since merging it into fork master alone would drop it from the mix.

Pins c26185b from #95 so the nightly prebuilds carry it. The pin stays listed
until the change lands upstream, since merging it into fork master would drop
it from the mix.
@danielhanchen
danielhanchen merged commit 9e5c7de into master Aug 11, 2026
1 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant