Feature/metrics inspector - #444
Draft
klement wants to merge 11 commits into
Draft
Conversation
Add a GET /metrics endpoint modeled on llama.cpp's exporter, gated behind the new network.enable_metrics config option (default False) and served without API key auth in the Prometheus text exposition format. A MetricsManager singleton accumulates process-lifetime counters (prompt and generation tokens, cached tokens, processing seconds, request count) from handle_finish_chunk, the single choke point every completed generation flows through. Throughput gauges and the in-flight/deferred request gauges are computed live at scrape time, the latter read from the exllamav3 generator's existing num_active_jobs()/num_pending_jobs(). The response also carries the Process-Start-Time-Unix header. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Expose three new gauges computed live at scrape time from the exllamav3 generator's page table: - kv_cache_usage_ratio / kv_cache_tokens: instantaneous KV load, measured over pages referenced by in-flight jobs (unreferenced pages may hold reusable prefixes but are evictable, so they count as headroom). Names kept verbatim from llama.cpp's exporter. - kv_cache_max_tokens: total KV cache token capacity. A new _live_kv_cache() helper mirrors _live_request_counts(), reading the page table off the sync generator and returning zeros when no model is loaded or the backend exposes no page table. page_size is derived from the generator rather than importing exllamav3's PAGE_SIZE, keeping metrics.py backend-agnostic. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ats to /metrics Broaden the /metrics endpoint toward vLLM's widely-adopted metric set, using per-request timings and drafter tallies the exllamav3 backend already computes but previously discarded after logging. Prefix cache: add the vLLM-idiomatic prefix_cache_queries / prefix_cache_hits token counters rather than a ready-made ratio gauge, so the hit ratio is computed at query time with rate() and reflects recent behavior rather than a process-lifetime average. Latency and size histograms: add request_queue_time_seconds, request_prefill_time_seconds, request_decode_time_seconds, time_to_first_token_seconds (queue + prefill), e2e_request_latency_seconds, request_prompt_tokens and request_generation_tokens. These expose tail latency (p95/p99) that the existing average-throughput gauges cannot. A minimal _Histogram helper accumulates cumulative buckets, sum and count and renders the standard _bucket/_sum/_count lines; bucket boundaries are taken from vLLM. record_generation now also receives queue_time from handle_finish_chunk. Speculative decoding: expose drafter effectiveness, with naming following vLLM's spec-decode metric set so its dashboards work after a prefix swap. Counters: spec_decode_num_draft_tokens_total (accepted + rejected, since exllamav3 rejects every draft position after the last accepted one, making the sum the number of tokens proposed), spec_decode_num_accepted_tokens_total, spec_decode_num_decode_steps_total and spec_decode_requests_total. Only requests served with a drafter contribute, so a mixed workload cannot dilute the acceptance rate; a drafted request that happened to propose nothing still counts, which is why the tally is distinguished from None rather than zero. Gauges summarize the spec decode counters for a scrape without a query language: spec_decode_draft_acceptance_rate (per drafted token), spec_decode_mean_accepted_length (per decode step) and spec_decode_tokens_per_step, which adds the target model's own token and is therefore the decode speedup factor over running without a drafter. A spec_decode_acceptance_rate histogram records the per-request distribution over buckets spanning [0, 1], since a lifetime average hides variance across prompts. exllamav3 does not count draft rounds, but every decode step emits exactly one token from the target model with the accepted drafts riding on top of it, so gen_tokens - accepted recovers the step count. The first token of a request comes out of prefill rather than a decode step, so this overcounts steps by up to one per request, slightly understating mean accepted length. Per-position acceptance (vLLM's accept-by-draft-index) is still absent; it needs the backend change the existing TODO in handle_finish_chunk refers to. Output validated against the prometheus_client text parser. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
prompt_tokens_seconds divided prompt_tokens_total by prompt_seconds_total,
which is a biased estimator that decays as prefix cache reuse accumulates. The
backend times prefill as a single span per request, and that span cannot be
split into time spent on cached versus newly processed tokens, so a cache hit
takes tokens out of the numerator while the lookup, page allocation and
per-chunk overhead it still paid for stay in the denominator. A request served
almost entirely from cache contributes near-zero tokens and a non-zero
duration.
Measured on a 27B model at 4096 chunk size. One cold 70665-token prompt
prefilled at 1609 T/s, then 50 requests reusing that exact prefix contributed
590 tokens over 1.85s between them, an effective 319 T/s. The gauge fell from
1609 to 1557 over those 50 requests and converges on the marginal figure under
sustained reuse, which is the regime an agent or multi-turn chat workload runs
in permanently.
No replacement gauge is added, because prefill throughput cannot be estimated
honestly from production traffic. Sampling only requests that computed enough
tokens for the roughly 30ms of fixed per-request overhead to vanish does remove
the bias, and gating on a full chunk of new tokens would bound the error near
1%. But a server fronting a harness with a stable system prefix may see exactly
one qualifying request in its lifetime, and that one carries the autotuning
pass: the first cold request of a session measured 1619.9 T/s against a steady
state of ~1646 T/s over the next four, a 1.6% penalty a lifetime average never
sheds. An estimator pinned to its single worst sample is not an improvement on
a biased one.
The counters remain, so a windowed rate is still available and is the figure
worth putting on a dashboard. It divides by wall clock rather than by a
per-request span, so cache hits cannot skew it:
rate(tabbyapi:prompt_tokens_total[5m])
For prefill speed as a benchmark number, exllamav3's eval/perf.py and the
per-request log line both control their own conditions.
predicted_tokens_seconds is kept. Decode time has no cached-token analogue to
skew it, so generation tokens over decode seconds is the quantity it claims.
Also stop rounding prefill time to 0.01s before accumulating it.
handle_finish_chunk rounds for the log line, and feeding that rounded value into
prompt_seconds_total and the prefill histogram added several percent of
quantization noise on short prefills. The log line is unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
n_tokens_max reports the largest prompt seen but has no counterpart for the largest completion, so the peak generation length is not readable anywhere. The request_generation_tokens histogram cannot supply it. An extreme is by definition the top sample, which sits above every percentile the histogram can report: with 345 requests, p99 is only the third largest, so a single long completion among many short ones is invisible. Once a sample lands in the final bucket its magnitude is lost entirely, since buckets record counts rather than values. Observed on a server whose traffic was dominated by short completions: mean 35, p50 15, p90 19, p99 189, while two requests had in fact generated over 1000 and over 5000 tokens. Nothing in the exposed metrics showed either figure. tokens_predicted_max is a counter for the same reason n_tokens_max is: it only ever rises. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The backend can hold cache pages evicted from VRAM in system RAM, so a request that yields its cache to another one can resume without prefilling its context again. memory.sysmem_kv_cache turns that on. It is off by default, and nothing about it is observable once it is. A restore is already counted as a prefix cache hit, since the generator does not distinguish a page found in VRAM from one read back over PCIe. The new kv_offload series are that breakdown: usage and RAM committed as gauges, restored tokens as the reuse counter to take against cached_tokens_total, and stores over evictions to show a budget below the working set. There is deliberately no transfer rate gauge, for the same reason there is no prefill one: a lifetime average divides by wall clock that includes every interval with no transfers at all. The byte counters are exposed so a windowed rate can be taken at query time; every transfer moves exactly one whole slot, so they are exact rather than sampled. cold_allocs is published because the backend pins its slabs ahead of demand on a background thread, and a store that outruns it pins synchronously at roughly 2.5 GB/s on the generator's own thread. That stall is otherwise invisible. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HtCocGn1iMp6kqMb7BncpW
Hybrid models interleave full-attention layers, whose state is the paged K/V cache, with linear-attention layers, whose state is a single evolving tensor that cannot be indexed by position. The backend checkpoints the latter to system RAM at page boundaries, and prompt reuse is capped at the longest prefix that has both valid K/V pages and a matching checkpoint. That cap is why these series matter. If the checkpoint for a prefix is gone, its K/V pages are unusable however well the cache held them, so the two RAM budgets have to be sized against each other rather than independently. recurrent_capped_tokens_total is the figure that says which way to move: tokens that had valid K/V and were re-prefilled anyway. With KV offloading enabled they were also read back over PCIe before being discarded. The eviction breakdown says whether the recurrent budget is the one at fault. A stranded checkpoint had already lost the pages it anchors and could never have been resumed, so dropping it is free; one dropped while its anchor page was still cached is the drop that becomes a capped token later. The mirror case, where the K/V cache is the one under pressure and evicting a page strands the checkpoint, is counted by the page table and published alongside it. recurrent_checkpoint_bytes is published because it is the unit the budget is spent in. A checkpoint is indivisible, so max_bytes over checkpoint_bytes is how many prefixes can be resumed at all, and on a 27B hybrid that is a little over a hundred at the default size. All series read zero on a pure transformer, which has no recurrent layers to checkpoint, and the cap is still reported if the cache object is unavailable, since losing that figure would hide the failure it exists to show. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HtCocGn1iMp6kqMb7BncpW
The token buckets were a fixed 1-2-5 ladder running to 1M regardless of what the loaded model could accept, and the comment claimed they came from another exporter for dashboard compatibility. That exporter does not use a fixed ladder; it builds one from the model's context length, so the ladder here was both wrong and diverging from the thing it cited. It was not merely imprecise. Above 200k the only boundaries were 500k and 1M, so a server answering 200k-token prompts put every sample a few thousand above the floor of a 300k-wide bucket. Interpolation assumes samples are spread across the bucket they landed in, so it reported a p99 of 493k against a largest-ever prompt of 206k -- 2.4x too high, and above any prompt the server had ever seen. Rebuilt from a 262144 context the same distribution reads 1.27x, and with the client-side clamp to the published peak it lands exactly. One boundary past the ladder is added at max_seq_len itself. The usual construction stops at the last mantissa below the limit, which leaves the range between there and the real limit with no bucket: at 262144 the ladder ends at 200000 and the top 24% falls into +Inf. Buckets are sized on model load, when the context length is finally known. Changing them discards those two histograms, since counts against a different ladder cannot be carried over, so an unchanged ladder is left alone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UrNJgoqU78RwaCxT6NuThy
The phase counters say how the server's time divides between prefill and decode, but not what either costs against a real interval. Without an origin for wall clock there is no way to turn prompt_seconds_total into "the engine was busy 11% of the time", which is what says whether the server is saturated or whether the split is being read off a handful of requests. The name and semantics are the conventional process-level ones, so the usual dashboards pick it up without being told about it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HtCocGn1iMp6kqMb7BncpW
Author
|
Here only two top commits are relevant - adding inspect_metrics.py script consuming /metrics endpoint by #443 |
The exposition format is meant for a scrape, not for a person. Reading it by eye means holding the counter pairs in your head -- hits against queries, restored against written, capped against cached -- and it is the pairs, not the individual numbers, that say whether anything is wrong. Standing up Prometheus and Grafana to answer "is the offload cache paying for itself" is not a proportionate response on a single-box server. So this renders one scrape as the ratios worth looking at, with a note where a figure implies an action. Where the number cannot say anything on its own the note says so instead: a saturated cache is the steady state of a cache in use, not a fault, so the churn warning is keyed on whether the pages written out are ever read back rather than on eviction count. Likewise recurrent evictions are split into the ones that were free (the checkpoint had already lost its anchor pages) and the ones that cost a re-prefill. Percentiles are held to the observed peak, since a histogram's top bucket is open-ended and quantile interpolation across it will otherwise report a p99 above any request the server has ever seen. Stdlib only, so it runs anywhere the server does, and --watch/--json are there for a terminal left open beside a load test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HtCocGn1iMp6kqMb7BncpW
A parallelism layout is a trade between the two phases: tensor parallel buys decode and gives up prefill, pipeline parallel the reverse. Which one is right is not a property of the hardware, it is a property of the workload, and the figure that decides it is the share of time each phase takes on this server with these prompts. That trade turns out to be one number. Cutting the time in a phase by some fraction saves that fraction of its share, so one percent off the dominant phase pays for exactly (its share / the other's) percent onto the other. At a 99.8/0.2 split that is 638% -- prefill is worth six hundred times what decode is, and a layout change that trades them is almost free. At an even split it is 1%, and no trade is worth making. Printing the ratio answers the question directly rather than making the reader derive it from two percentages. The ratio is a marginal reading and overstates a large move, so the exact break-even is printed underneath for two concrete prefill factors: the speed the other phase may fall to before the swap stops paying. Queue time is shown but kept out of the split, since a request waiting is the engine working on another one and counting it as a phase would double-count. Utilization is against wall clock and clamped, because summed request time exceeds the interval it ran in as soon as there is any concurrency, and a number above 100% reads as a bug rather than as the overlap it is. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HtCocGn1iMp6kqMb7BncpW
klement
force-pushed
the
feature/metrics-inspector
branch
from
August 1, 2026 16:12
108fb33 to
11d1f8c
Compare
klement
marked this pull request as draft
August 1, 2026 16:12
Author
|
Need to dogfood this a bit more, marking draft for now. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Is your pull request related to a problem? Please describe.
A clear and concise description of what the problem is. You can also link to an existing issue.
Why should this feature be added?
An explanation of why the feature should be added. Please be as specific as possible to help us understand the reasoning.
Examples
Examples of the feature in action and its significance compared to not having the feature.
Additional context
Add any other context or screenshots about the pull request here.