feat: add Mooncake GPU-direct capture materialization

[Boreas618]

Summary

• add a Mooncake GPU-direct FeatureStore for nvlink, nvlink_intra, and rdma
• materialize server-owned capture descriptors directly into trainer CUDA tensors
• preserve CPU-only producer operation and report zero host tensor payload bytes
• retain consumed buffers until optimizer-boundary acknowledgement and release them through the SGLang control endpoint
• expose mooncake_gpu_direct in config and launch planning while preserving the existing Mooncake Store backend
• avoid Mooncake batch reads on multi-node NVLink because its API documentation records a correctness caveat for that path

Companion SGLang draft: sgl-project/sglang#35577

Hardware benchmark

Run on 2026-08-21 UTC. The fixed target was SGLang 60e519391d59 on 8x NVIDIA H200 (driver 580.105.08), split into two TP4 Qwen3.8-27B-FP8 replicas. A second H200 host ran the consumer on one GPU. Capture used Mooncake RDMA, DSpark taps [5, 19, 33, 47, 61], and a 32 GiB target arena per replica.

The repository base does not contain the GPU-direct FeatureStore, so the same protocol cannot run there. The baseline is the PR's first functional implementation (1f5a05bfb8b0) and the post-change checkout is PR head (9ce0b698c98c). The baseline used the PR's first live RDMA benchmark harness with only its later-only producer_ordered_publish keyword removed; production code was unchanged. Each value is the median of two completed 512-sample runs with max_length=8192, concurrency 64, and ingest batch 128.

CLI-equivalent controlled comparison

Both checkouts used lease 1, prefetch 0, shared replica routing, shuffled prompts, and serial materialization.

Metric	Baseline	PR head	Change
Capture throughput (samples/s)	55.4706	64.3044	+15.93%
RDMA materialization (samples/s)	117.2539	120.3692	+2.66%
Serial pipeline throughput (samples/s)	37.6540	41.4005	+9.95%
Pipeline elapsed (s)	13.5976	12.3676	-9.05%
Durable cleanup (s)	0.6229	0.6383	+2.46%

The baseline harness does not emit a pipeline metric, so its serial pipeline value is conservatively derived as 512 / (capture_elapsed + materialize_elapsed). The PR-head value uses the harness's measured pipeline interval.

Tuned PR-head path

The tuned path enabled lease 8, prompt prefetch 4, least-tokens replica routing, length-bucketed requests, batch cleanup, and overlapped capture/materialization.

Metric	Baseline native path	Tuned PR head	Change
Capture throughput (samples/s)	55.4706	67.2924	+21.31%
RDMA materialization (samples/s)	117.2539	120.6860	+2.93%
Pipeline throughput (samples/s)	37.6540	61.9381	+64.49%
Pipeline elapsed (s)	13.5976	8.2663	-39.21%
Durable cleanup (s)	0.6229	0.0056	-99.10%

Every completed run produced and removed all 512 samples, transferred 40,503,867,376 bytes, and reported zero host payload bytes. After each tuned run, both SGLang control endpoints reported no resident or reserved samples, published_samples == released_samples, and the full 32 GiB arena free.

The 1024-sample stress case was excluded from throughput statistics because the harness intentionally retains every sample until the final durable acknowledgement; its retained set exceeds the aggregate 64 GiB target capacity.

Builds, unit tests, integration tests, and CI were not run as part of this hardware benchmark.