Skip to content

Add DeepSeek-V4 H20 benchmark data to simulation - #16

Closed
xuqianmamba wants to merge 1 commit into
alibaba:mainfrom
xuqianmamba:agent/dsv4-h20-bench-data
Closed

Add DeepSeek-V4 H20 benchmark data to simulation#16
xuqianmamba wants to merge 1 commit into
alibaba:mainfrom
xuqianmamba:agent/dsv4-h20-bench-data

Conversation

@xuqianmamba

Copy link
Copy Markdown

What changed

  • add measured H20 TP8 DeepSeek-V4-Pro CSVs for C4/C128 attention, mHC and
    MXFP4 MoE in bench_data/dsv4
  • parse DeepSeek-V4 compression ratios and derive an architecture fingerprint
    from the model config
  • add exact table lookup and an end-to-end DeepSeek-V4 simulator path in
    main.py
  • add focused tests for config normalization, exact-shape rejection,
    grouped-layer aggregation and the packaged H20 data

Why

PR #15 adds a standalone DeepSeek-V4 sparse-prefill kernel benchmark, but the
simulator still has no DeepSeek-V4 model path and cannot consume measured V4
component latencies. Treating V4 as generic MLA/MHA would also mix C4 and C128
layers and silently reuse incompatible MFU data.

This change makes V4 simulation table-driven. It sums measured per-layer
C4/C128 attention, mHC and MoE latencies according to the checkpoint's layer
layout, then adds communication separately using InferSim's existing
communication model.

Measurement scope

The committed rows were collected on one node with 8 NVIDIA H20 GPUs using
SGLang 0.5.15:

  • prefill: batch size 1, query length 4096
  • decode: batch size 128, query length 1, past length 5120
  • attention: dsv4, FP8 weights, fp8_e4m3 KV, FP32 compression state
  • MoE: packed FP4 experts with flashinfer_mxfp4
  • architecture fingerprint: dc8a15770f7f8647

Lookup is intentionally exact. Unsupported fingerprints, TP/world sizes,
backends, dtypes or serving shapes raise MissingDSV4BenchmarkData instead of
extrapolating.

Validation

  • python3 -B -m unittest discover -s tests -v (4 tests)
  • python3 -m py_compile config/model_config.py kernel_sim/dsv4.py models/deepseek_v4_model.py main.py tests/test_deepseek_v4_simulator.py
  • CLI smoke test covering both prefill and decode table lookup
  • Black 24.10.0, isort 5.13.2 and Ruff 0.14.10 checks
  • git diff --check

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@xuqianmamba

Copy link
Copy Markdown
Author

Closing in favor of updating #15 to follow InferSim existing bench_data CSV conventions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants