Skip to content

fix: restore torch_dtype when loading layerwise reranker model - #1589

Open
St4r4x wants to merge 1 commit into
FlagOpen:masterfrom
St4r4x:fix/layerwise-reranker-torch-dtype
Open

fix: restore torch_dtype when loading layerwise reranker model#1589
St4r4x wants to merge 1 commit into
FlagOpen:masterfrom
St4r4x:fix/layerwise-reranker-torch-dtype

Conversation

@St4r4x

@St4r4x St4r4x commented Aug 17, 2026

Copy link
Copy Markdown

Root cause

Closes #1564.

In FlagEmbedding/finetune/reranker/decoder_only/layerwise/load_model.py, both calls to LayerWiseMiniCPMForCausalLM.from_pretrained() in get_model() had their torch_dtype argument commented out, while attn_implementation="flash_attention_2" is still passed unconditionally whenever model_args.use_flash_attn is set:

model = LayerWiseMiniCPMForCausalLM.from_pretrained(
    model_args.model_name_or_path,
    trust_remote_code=model_args.trust_remote_code,
    # torch_dtype=torch.float16 if training_args.fp16 else torch.bfloat16,
    attn_implementation = "flash_attention_2" if model_args.use_flash_attn else None,
    ...
)

Without torch_dtype, the model loads in fp32 by default. modeling_minicpm_reranker.py's MiniCPMFlashAttention2.forward() only casts hidden states away from fp32 by falling back to self.q_proj.weight.dtype — which is also fp32 in this case, so the cast is a no-op and the flash-attn kernel then receives fp32 tensors. Flash Attention 2 only supports fp16/bf16 inputs, so this crashes as soon as attention runs during finetuning with --use_flash_attn.

Note the commented-out line already referenced training_args.fp16, but get_model() only ever received model_args as a parameter — training_args was never in scope, so simply uncommenting it would raise a NameError.

Fix

  • Thread training_args through get_model() (and its single caller in runner.py).
  • Restore torch_dtype=torch.float16 if training_args.fp16 else torch.bfloat16 in both from_pretrained() calls, matching the dtype-selection idiom already used by this repo's inference-side loaders (e.g. FlagEmbedding/inference/reranker/decoder_only/layerwise.py).

Also restored two pre-existing unused imports (AutoModelForCausalLM in load_model.py, os in runner.py) that a local lint auto-fix on my end stripped as an incidental side effect while editing these files — flagging this explicitly for transparency, in case a maintainer would rather those be removed in a separate/dedicated cleanup instead.

Validation

I don't have a GPU + real model weights handy to run this specific finetuning path end-to-end, so I validated by code inspection instead:

  • Confirmed AbsRerankerTrainingArguments subclasses transformers.TrainingArguments, so .fp16/.bf16 are valid attributes once training_args is passed in.
  • Confirmed via git blame that the torch_dtype line has been commented out since this file's original commit, and that a later commit (19edba7b) switched use_flash_attention_2=True/False to attn_implementation="flash_attention_2" without addressing the dtype issue.
  • Confirmed get_model() in this file has exactly one caller (runner.py), so the added training_args parameter doesn't break other call sites.
  • Traced MiniCPMFlashAttention2.forward() to confirm the fp32-input crash mechanism described above.
  • The exact same commented-out torch_dtype pattern also exists in the sibling loaders finetune/reranker/decoder_only/base/load_model.py, finetune/embedder/decoder_only/base/load_model.py, and finetune/embedder/decoder_only/icl/load_model.py — I scoped this PR to the file named in 这行注释掉会导致微调模型时flashattention2报错(fla2只支持fp16、bf16,关还关不掉),赶紧修复了吧。 #1564 only, to keep the change minimal and focused.

LayerWiseMiniCPMForCausalLM.from_pretrained() had its torch_dtype
argument commented out while attn_implementation="flash_attention_2"
was still passed unconditionally whenever use_flash_attn is set. With
no torch_dtype, weights load in fp32, and Flash Attention 2 only
supports fp16/bf16 inputs, so training crashes as soon as attention
runs.

Restore the dtype selection based on training_args.fp16/bf16, and
thread training_args through get_model() since it previously only
received model_args.

Fixes FlagOpen#1564
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

这行注释掉会导致微调模型时flashattention2报错(fla2只支持fp16、bf16,关还关不掉),赶紧修复了吧。

1 participant