fix: restore torch_dtype when loading layerwise reranker model - #1589
Open
St4r4x wants to merge 1 commit into
Open
fix: restore torch_dtype when loading layerwise reranker model#1589St4r4x wants to merge 1 commit into
St4r4x wants to merge 1 commit into
Conversation
LayerWiseMiniCPMForCausalLM.from_pretrained() had its torch_dtype argument commented out while attn_implementation="flash_attention_2" was still passed unconditionally whenever use_flash_attn is set. With no torch_dtype, weights load in fp32, and Flash Attention 2 only supports fp16/bf16 inputs, so training crashes as soon as attention runs. Restore the dtype selection based on training_args.fp16/bf16, and thread training_args through get_model() since it previously only received model_args. Fixes FlagOpen#1564
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause
Closes #1564.
In
FlagEmbedding/finetune/reranker/decoder_only/layerwise/load_model.py, both calls toLayerWiseMiniCPMForCausalLM.from_pretrained()inget_model()had theirtorch_dtypeargument commented out, whileattn_implementation="flash_attention_2"is still passed unconditionally whenevermodel_args.use_flash_attnis set:Without
torch_dtype, the model loads in fp32 by default.modeling_minicpm_reranker.py'sMiniCPMFlashAttention2.forward()only casts hidden states away from fp32 by falling back toself.q_proj.weight.dtype— which is also fp32 in this case, so the cast is a no-op and the flash-attn kernel then receives fp32 tensors. Flash Attention 2 only supports fp16/bf16 inputs, so this crashes as soon as attention runs during finetuning with--use_flash_attn.Note the commented-out line already referenced
training_args.fp16, butget_model()only ever receivedmodel_argsas a parameter —training_argswas never in scope, so simply uncommenting it would raise aNameError.Fix
training_argsthroughget_model()(and its single caller inrunner.py).torch_dtype=torch.float16 if training_args.fp16 else torch.bfloat16in bothfrom_pretrained()calls, matching the dtype-selection idiom already used by this repo's inference-side loaders (e.g.FlagEmbedding/inference/reranker/decoder_only/layerwise.py).Also restored two pre-existing unused imports (
AutoModelForCausalLMinload_model.py,osinrunner.py) that a local lint auto-fix on my end stripped as an incidental side effect while editing these files — flagging this explicitly for transparency, in case a maintainer would rather those be removed in a separate/dedicated cleanup instead.Validation
I don't have a GPU + real model weights handy to run this specific finetuning path end-to-end, so I validated by code inspection instead:
AbsRerankerTrainingArgumentssubclassestransformers.TrainingArguments, so.fp16/.bf16are valid attributes oncetraining_argsis passed in.git blamethat thetorch_dtypeline has been commented out since this file's original commit, and that a later commit (19edba7b) switcheduse_flash_attention_2=True/Falsetoattn_implementation="flash_attention_2"without addressing the dtype issue.get_model()in this file has exactly one caller (runner.py), so the addedtraining_argsparameter doesn't break other call sites.MiniCPMFlashAttention2.forward()to confirm the fp32-input crash mechanism described above.torch_dtypepattern also exists in the sibling loadersfinetune/reranker/decoder_only/base/load_model.py,finetune/embedder/decoder_only/base/load_model.py, andfinetune/embedder/decoder_only/icl/load_model.py— I scoped this PR to the file named in 这行注释掉会导致微调模型时flashattention2报错(fla2只支持fp16、bf16,关还关不掉),赶紧修复了吧。 #1564 only, to keep the change minimal and focused.