[Feature] Multimodal (image+text) DFlash training via server-side capture (Qwen3.5)

[curnane-lab]

Motivation

Image-text DFlash training is a long-standing request (#583, #461) and a high-priority item on the 2026 roadmap (#374: "General VLM training"). The pre-#678 HF-backend stack had community VLM prototypes (#585, #495, #505); after the unified-runtime consolidation (#678), training runs exclusively on server-side capture, and none of those prototypes apply anymore.

This PR brings multimodal DFlash training to the current architecture end-to-end: conversations with images are expanded client-side, captured by a patched SGLang server (aux hidden states and mRoPE position ids written straight to Mooncake), and trained with 3D mRoPE positions in the DFlash draft. Text-only runs are byte-identical to before.

Modifications

• Contracts/providers (algorithms/dflash/providers.py, common/providers.py): DFlash registers a STREAMING multimodal FeatureContract ({input_ids, loss_mask, hidden_states, position_ids}) and a ServerStreamingProvider with a VLM ServerInputAdapter (the modality seam already present in common/providers.py). ServerCaptureLayout/ServerCaptureSchema gain position_ids_feature.
• Data (data/vlm_preprocessing.py, common/vlm_input.py): ShareGPT-style JSONL with an optional image / image_path / images field (path or base64; images takes the single list element). The target's own chat template renders the image into the first user turn; the target's HF processor expands the image region in id space, yielding expanded input_ids/loss_mask (trainer + passthrough) and collapsed request_input_ids + data:-URI image_data (capture request — accepted by all supported SGLang builds). One image per sample (v1); text-only samples can mix in the same run. Unreadable images and multi-image samples fail loudly (ImageDataError), never a silent text-only downgrade.
• Capture (inference/adapters/server_capture.py, training/disaggregated.py, launch_plan.py, patches/sglang/v0.5.14/spec-capture.patch): the capture request's features map gains position_ids; the regenerated patch writes the request's mRoPE positions (1, L, 3) int64 into Mooncake from req.multimodal_inputs.mrope_positions (arange fallback for text prompts). Managed capture servers set SGLANG_MM_AVOID_RETOKENIZE=1 for input_modality=multimodal so the server re-expands placeholders in id space with zero retokenization drift (for externally managed servers via server_urls, set it in the server environment manually — noted in the recipe comments); client/server expansion mismatch fails loudly at the existing seq-len check. On Ascend hosts the capture servers default to --mm-attention-backend ascend_attn (fused vision attention) for non-text modalities — the sdpa vision backend materializes [heads, N, N] scores and OOMs on large images; model.sglang_mm_attention_backend overrides.
• Draft model (modeling/draft/dflash.py, ported from [Feature] VLM DFlash Training: Multi-Model Support for Qwen3-VL / Qwen3.5 / Qwen3.6 #585 by @zyk42, authorship preserved): partial rotation in apply_rotary_pos_emb (rotary_dim < head_dim, for Qwen3.5/3.6 partial_rotary_factor=0.25) and Qwen3InterleavedMultiRotaryEmbedding selected by rope_scaling.mrope_interleaved.
• Target handling (modeling/target/target_utils.py, algorithms/model_providers.py): auto-select model.language_model.embed_tokens.weight for VLM targets; language-model depth via the text_config fallback for draft config auto-generation.
• Training (common/dflash_family_{data,model}.py, training/strategies/base.py): build_vlm_collator pads position_ids; OnlineDFlashModel.forward accepts optional (B, S, 3) position ids and gathers 3D mRoPE positions for context + anchor-offset draft slots; DFlashTrainStrategy passes the tensor through.
• Recipe/config: examples/configs/qwen3.5-4b-vl-dflash-disaggregated.yaml + configs/qwen3.5-4b-vl-dflash.json (mirrors the Qwen3.5-4B target geometry: head_dim 256, mrope_section [11,11,10], partial_rotary_factor 0.25).
• Tests/docs: new tests/test_algorithms/test_dflash_multimodal.py; text-only assertions in test_builtin_providers / test_server_only_online / test_unified_feature_reachability / test_launch_topology updated for the new modality and recipe; status doc docs/advanced_features/vlm_dflash.md.

Scope notes (deliberate): DFlash only (Domino/DSpark can mirror the same registration later); one image per sample; offline (precomputed) multimodal capture not included; #585's wider config set (Qwen3-VL-8B/30B-A3B, Qwen3.5-9B/35B-A3B) deferred to a follow-up once more targets are validated.

Related Issues

• Addresses [Feature] Support image-text multimodal input for DFlash training, similar to Eagle3 #583
• Related: [Feature] [RFC] DFlash Training Adaptation for Qwen VL Models #461 (DFlash × Qwen-VL RFC), Development Roadmap 2026 Q1 #374 (2026 Q1 roadmap — VLM support), [Question]What factors contribute to Eagle3's reduced acceleration performance on VLM architectures compared to traditional LLMs? #352 (VLM speedup tuning)
• Builds on [Feature] VLM DFlash Training: Multi-Model Support for Qwen3-VL / Qwen3.5 / Qwen3.6 #585 (draft model + configs, author preserved via cherry-pick)

Accuracy Test

• End-to-end (Ascend NPU, first live run — PASSED): Qwen3.5-4B target + VLM DFlash draft, disaggregated managed run (5 capture servers + 5 trainer ranks, 32k context). Training runs end-to-end with healthy numerics (curves below). Two compatibility issues found during bring-up are fixed in this PR (data-URI image transport for older SGLang image loaders; ascend_attn as the default vision attention backend on NPU — the sdpa vision backend OOMs on large images).

  Data & training configuration

  Item	Value
  Dataset	CC-OCR, 7,058 samples (image + OCR conversations)
  Responses	regenerated with the Qwen3.5-4B target model
  Used for training	5,156 samples (after the 32k expansion filter)
  Target / draft	Qwen3.5-4B / VLM DFlash (configs/qwen3.5-4b-vl-dflash.json, 5 layers, block_size 16)
  Topology	10× Ascend NPU: 5 capture servers (tp=1 each) + 5 trainer ranks (disaggregated, Mooncake transport)
  max_length	32,768 (32k context)
  batch / accumulation	2 / 4, producer_lease 4
  lr	6e-4 cosine, warmup 4%, num_anchors 512, loss_decay_gamma 7

  

  Training curves of the NPU end-to-end run (~1.2k steps): draft loss decays from ~5.7 to ~1, draft token accuracy rises from ~0.10 to ~0.65+, grad_norm stays bounded (0.4–1.1), and lr follows the cosine decay. accuracy_denom (masked label tokens per logged window) oscillates with the image/text batch composition — expected for mixed VLM batches.

• Unit/integration (CPU): registration parity and provider gates, request/payload construction, image-expansion math, images-field resolution and loud image errors, data-URI transport, collator, golden topology/recipe tests — tests/test_algorithms/test_dflash_multimodal.py plus updated test_config suites pass (87 passed, 2 skipped locally; the only failure in tests/test_config is test_recipe_readme, which fails on upstream main as well — unrelated pre-existing issue).

• Patch integrity: the regenerated spec-capture.patch applies cleanly both ways to pristine sglang v0.5.14 (git apply --check / --reverse --check), and stacks cleanly with the Ascend companion patch.

Benchmark & Profiling

• Training-side (this PR): the NPU run above is the first end-to-end validation; draft loss/accuracy curves are healthy (see figure).
• Serving-side accept length / speedup for the trained VLM draft: pending (SGLang DFLASH serving with mRoPE VLM draft; companion inference-side work is tracked separately).
• Reference numbers from [Feature] VLM DFlash Training: Multi-Model Support for Qwen3-VL / Qwen3.5 / Qwen3.6 #585 (HF stack, author-validated): Qwen3-VL-30B-A3B-Thinking, 278K target-regenerated samples, 5-layer draft, block_size=8 → accept length 3.52, +35.8% inference speedup (4x RTX 5090, TP=4, SGLang 0.5.12).

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results. (Training-side NPU validation done; serving benchmark pending.)
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://sgl-fru7574.slack.com/archives/C09784E3EN6 to discuss your PR.

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.