A layered systems study of dispatch, replay, orchestration, and persistent execution in iterative LLM decode.
This repository consolidates multiple experimental phases that isolate and quantify different layers of runtime overhead in LLM serving.
Iterative LLM decode is often assumed to be compute-bound.
In practice, performance evolves in layers:
T_total = T_dispatch + T_replay + T_orchestration + T_compute
Optimizing one layer reveals the next dominant bottleneck.
This study empirically measures each transition.
Location: phase1_dispatch/
- Baseline decode was dispatch-bound.
- CUDA Graph eliminated kernel launch overhead.
- ~15× CPU reduction observed.
- Replay became dominant after dispatch removal.
Model transition:
T_total = T_compute + T_replay + T_orchestration
Location: phase2_replay/
- Replay cost measured (~2.8 µs per step).
- Multi-step capture reduced replay frequency.
- Optimal capture granularity identified.
- Throughput scaled sublinearly with steps per replay.
Refined model:
T_total(K) = (N/K)T_replay + NT_compute + T_orchestration
Replay matters, but only until compute dominates.
Location: phase3_persistent/
- Eliminated replay boundary entirely.
- Host loop removed.
- Persistent kernel achieved ~2.1M steps/sec in ultra-light regime.
- Under heavier compute (hidden=256), compute became dominant (~153k steps/sec).
Final model:
Persistent execution removes replay, leaving compute as the primary limiter.
The full decode stack can be decomposed into:
- Dispatch layer
- Replay layer
- Host orchestration layer
- Compute layer
Regime dominance:
| Regime | Dominant Layer |
|---|---|
| Pre-Graph | Dispatch |
| Graph | Replay |
| Multi-Step | Replay + Compute |
| Persistent | Compute |
| Real Transformer Decode | Compute |
Performance is layered.
- Small-batch decode can be dispatch-bound.
- CUDA Graph effectively removes dispatch.
- Replay cost is measurable and non-trivial.
- Multi-step capture improves throughput but has diminishing returns.
- Persistent kernels eliminate replay but do not overcome compute limits.
- Runtime optimization must be systemic, not isolated.
For real LLM serving:
- Kernel micro-optimizations are insufficient alone.
- Runtime control flow matters.
- Replay granularity affects throughput.
- Persistent execution only helps when replay dominates.
- Compute ultimately sets the upper bound.
- Single GPU (SM75)
- No distributed inference
- No swap/preemption
- Simplified decode workloads
- Focused on runtime layering, not model accuracy
docs/ → consolidated report and figures
phase1_dispatch/ → dispatch and graph study
phase2_replay/ → replay scaling experiments
phase3_persistent/ → persistent kernel implementation
utils/ → shared utilities
LLM decode performance evolves in layers.
Eliminating dispatch reveals replay. Eliminating replay reveals compute. Each layer must be understood in isolation.
Runtime engineering for LLM serving requires systemic analysis.
MIT