Skip to content

bitnet: honor relu2 hidden_activation from GGUF (fixes microsoft/BitNet#602) - #26751

Open
msrovani wants to merge 1 commit into
ggml-org:masterfrom
msrovani:fix/bitnet-relu2
Open

bitnet: honor relu2 hidden_activation from GGUF (fixes microsoft/BitNet#602)#26751
msrovani wants to merge 1 commit into
ggml-org:masterfrom
msrovani:fix/bitnet-relu2

Conversation

@msrovani

@msrovani msrovani commented Aug 8, 2026

Copy link
Copy Markdown

Fixes microsoft/BitNet#602. The official bitnet-b1.58-2B-4T declares hidden_act: relu2 but the bitnet arch hardcodes LLM_FFN_SILU in build_ffn, producing wrong-but-finite logits on every backend.

Changes:

  • llama-model.cpp: add "relu2" -> LLM_FFN_RELU_SQR to LLM_FFN_OP_TYPES_FROM_STRING
  • models/bitnet.cpp: read .hidden_activation in load_arch_hparams (mirroring modern-bert), defaulting to LLM_FFN_SILU for backward compatibility with existing GGUF files; use hparams.llm_ffn_op in the graph

LLM_FFN_RELU_SQR with LLM_FFN_PAR gating produces exactly relu2(gate) * up, matching the HF BitnetMLP (act_fn(gate) * up).

@msrovani
msrovani requested a review from CISC as a code owner August 8, 2026 04:41
@github-actions github-actions Bot added the model Model specific label Aug 8, 2026
@ggml-gh-bot

ggml-gh-bot Bot commented Aug 8, 2026

Copy link
Copy Markdown

Hi @msrovani, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

  • PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

model Model specific

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bitnet-b1.58-2B-4T runs FFN with SiLU instead of relu2 - wrong logits on every backend

1 participant