Skip to content

Add an opt-in ZeRO-1/2 gradient norm fast path - #8331

Draft
yh0903 wants to merge 1 commit into
deepspeedai:masterfrom
yh0903:yh0903/zero1-optimizer-fastpath
Draft

Add an opt-in ZeRO-1/2 gradient norm fast path#8331
yh0903 wants to merge 1 commit into
deepspeedai:masterfrom
yh0903:yh0903/zero1-optimizer-fastpath

Conversation

@yh0903

@yh0903 yh0903 commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add zero_optimization.compute_grad_norm, defaulting to true
  • allow ZeRO Stage 1/2 GPU optimizers to skip an unused global gradient norm while preserving overflow detection
  • skip the gradient multiply when clipping is disabled and the effective loss scale is exactly 1
  • return None from get_global_grad_norm() when norm computation is explicitly disabled
  • fail fast for gradient clipping, optimizer offload, ZenFlow, unsupported ZeRO stages, and checkpoint-restored clipping

Safety contract

Finite/overflow checking is unchanged. This does not remove the required non-finite scan; it only removes work with no mathematical consumer. The default path is unchanged.

Performance

Qwen3-30B-A3B, 48 layers, EP16 on 2x8 H100, sequence length 1024, BF16, ZeRO-1, activation checkpointing enabled:

Metric Baseline Fast path Change
Median step 987.92 ms 966.67 ms -2.2%
Optimizer update 84.93 ms 59.82 ms -29.6%
Global norm 20.30 ms 0 ms removed
Identity unscale 5.09 ms 0.03 ms removed

The paired block used a benchmark-only control that restores the pre-change multiply-by-one path. Peak allocated and reserved memory were unchanged. Two additional rotated blocks isolating the opt-in norm change were both positive (+3.3% and +0.6% E2E); their optimizer-phase saving was stable at about 20 ms, while full-step variance remained larger.

Testing Done

  • H100 targeted suite: 19 passed, covering Stage 1/2 update parity, overflow skip behavior, get_global_grad_norm(), clipping/offload/ZenFlow guards, checkpoint clipping, and identity/non-identity scaling
  • changed-file yapf, check-torchdist, license, codespell, and other pre-commit hooks
  • flake8 5.0.4 under Python 3.12; the local pre-commit Python 3.14 environment fails inside pyflakes before linting because ast.Str was removed
  • canonical paired executions: fdf729ed8adac4ebea4b, feb0641f09c1d42fcb7b, ff508156ab98444ddb2f

Preserve overflow detection while allowing ZeRO-1/2 GPU optimizers to skip an unused global norm, and avoid identity gradient scaling when clipping is disabled.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: yh0903 <helloyu0903@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant