Skip to content

Prebuilt: move the #95 pin to the identity-guard commit - #98

Open
danielhanchen wants to merge 1 commit into
masterfrom
repin-95-identity-guard
Open

Prebuilt: move the #95 pin to the identity-guard commit#98
danielhanchen wants to merge 1 commit into
masterfrom
repin-95-identity-guard

Conversation

@danielhanchen

Copy link
Copy Markdown
Member

#95 gained a second commit, so the pin has to move with it.

The first version of the patch checked only that each penalized token sat at its own index in cur_p. An array holding the same id twice then took the by-index path and penalized one of the two entries, where the original scan penalized both. It now requires the full identity layout, data[i].id == i for every i, which also rules out duplicates.

A differential fuzz over 33600 cases found it: 209 mismatches against the first version, all on the duplicate-id layout, and 0 against this one. The same harness run against the stock library also reports 0, which is the check that the reference implementation is faithful.

Repins from e2e842a to 3db8cb5 so the nightly builds the fixed version.

Throughput with the stricter guard, both builds from b10359, Qwen3-0.6B-Q4_K_M (151936 vocab), 400 tokens, temperature 0, median of 5:

penalty stock patched delta
presence_penalty 1.5 467.29 583.09 +24.8%
frequency_penalty 0.8 440.94 586.39 +33.0%
repeat_penalty 1.15 443.18 595.42 +34.4%

Output is unchanged: 60/60 byte identical across 4 prompts, 5 penalty combinations, greedy and two seeded sampled settings.

#95 gained a second commit: the first version only checked that each penalized
token sat at its own index, so a cur_p holding the same id twice penalized one
entry where the scan penalized both. It now requires the full identity layout.

Repins to 3db8cb5 so the nightly builds the fixed version.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant