Skip to content

feat(exemplar): store and query metric exemplars - #253

Open
tdakkota wants to merge 6 commits into
mainfrom
feat/exemplar-projection
Open

feat(exemplar): store and query metric exemplars#253
tdakkota wants to merge 6 commits into
mainfrom
feat/exemplar-projection

Conversation

@tdakkota

@tdakkota tdakkota commented Aug 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Metric exemplars end to end: OTLP conversion, storage, query, and retention. WriteMetrics persists the exemplars its points carry, and they read back under the metric's own label matchers.

Validated against the real consumer — see Verified against oteldb below.

Why a record signal

An exemplar is record-shaped: zero or more per data point, variable-width payload (trace id, span id, filtered attributes). The metrics engine is a dense (SeriesID, ts, float64) store whose merge and downsample paths assume exactly one row per point, so a ragged sidecar there would fight every invariant. The record engine already handles this shape for logs, traces, and profile samples.

An exemplar stream is its metric series

The stream id is the point's metric.Identity.SeriesID, byte for byte — and it is not recomputed. Project takes an already-projected *metric.Batch and reads the id the metrics path resolved:

p.b.Stream = mb.IDs[i]

So the invariant holds by construction rather than by two implementations agreeing. It also means one pass over the batch with the tenant resolved once, which fits WriteMetrics — everything there already happens inside a single metric.Project callback.

What that identity buys, with no exemplar-specific machinery: the postings index, tenant routing, and cluster shard placement all work already. A matcher set that selects metric series selects exactly the matching exemplar streams, which is what /api/v1/query_exemplars needs.

One emit is one point — within a metric, OTLP data points have distinct attribute sets, so a point is a stream. Points with no exemplars never reach the engine: a cheap structural scan short-circuits before the projection pass, so a producer that emits no exemplars pays nothing.

The record engine had to start indexing Series.Attributes

The one change reaching outside the new package, and it was not in the plan — the E2E tests caught it.

recordengine indexed only Resource.Attributes and Scope.Attributes. Logs and traces keep per-record attributes in a column, and profiles deliberately fold their type labels into the resource, so the identity's third attribute set had never carried anything. But metric.Identity.ToSeries() puts __name__ and the data-point attributes exactly there — so service.name matched while __name__ and http.route, the labels you actually filter exemplars by, resolved nothing.

The fix indexes all three sets, with a new KeyScopeSeries. It is a no-op for the other record signals. Folding the metric labels into the resource instead would also have worked, but it would break the stream-hash-equals-SeriesID property the whole design rests on.

Arguably a latent bug regardless of exemplars: the engine's identity type has three attribute sets and it was indexing two.

⚠️ Conflicts with #154. That PR extends the same KeyScope bitset with KeyScopeIndexed and persists a scope byte per key in keys.bin footer v2. Both PRs claim the same next bit value for different meanings, and one of them ends up on disk. Whichever merges second must renumber — worth coordinating rather than resolving at merge time.

Adding a signal means auditing the hardcoded switches

The generic record path covers less than it looks. recover() and walReplayerFor() dispatch on a hardcoded four-way switch over /metrics, /logs, /traces, /profiles — neither knew /exemplars, so a reopened store served no exemplars at all; the engine was never created. Auditing every profileEngineSnapshot call site turned up four more: Close (WAL handles leaked, unflushed data lost), Reset, the WAL-sync loop, and the maintenance loop — so exemplar parts would never have been flushed or compacted in the background either.

None of this is caught by go build. A reopen test is what surfaced it, and it is in the suite now.

Schema

column kind codec bloom
value int64 T64 —
trace_id bytes CodecBytesRaw Equality
span_id bytes CodecBytesRaw —
attrs bytes Dict Attrs

trace_id is near-unique by construction (one exemplar is one sampled request), so a dictionary is pure overhead — the same conclusion the trace-column work reached. Its equality bloom is what prunes a trace-to-metrics lookup.

value is the one wart: the record engine has no float kind, so the float64 bit pattern rides in an int64 column via explicit EncodeValue/DecodeValue. Lossless, but it forfeits compression (T64 has nothing to crop in an exponent field) and keeps exemplars out of the float-precision recompression policy. Helpers rather than a bare cast so the reinterpretation is named, testable, and has exactly one place to delete. #251 tracks the proper fix.

OTLP conversion

pdataconv.AppendMetrics now carries number-point exemplars through, and its return becomes Dropped{Points, Exemplars} — a breaking change to the optional bridge.

The split is the point: Points feeds OTLP partial-success, meaning data the producer should retry, while a dropped exemplar only degrades correlation. One combined number would tell a producer to resend metrics because an exemplar didn't fit. Drop accounting, each with a test:

case Points Exemplars
histogram / exp-histogram exemplars 0 n
value-less point with exemplars 1 n (no series left to hang off)
value-less exemplar on a good point 0 1

Exemplars on decomposed metrics are rejected, not stored. Classic decomposition splits a histogram point into _count/_sum/_bucket{le}, leaving an exemplar with no unambiguous home. The Prometheus convention would pick the _bucket whose le covers the value, but committing to that before native histograms exist bakes the choice into the stored shape. #252 holds the decision. Summary points carry no exemplars in OTLP at all — verified against the pdata API, not assumed.

Retention

tenant.Retention.ExemplarMaxAge, resolved through a new Retention.AgeFor(signal).

The age cutoff is now per-signal; the byte budget deliberately is not. MaxBytes bounds what a node stores, which is not a per-signal question — splitting it would mean N budgets that don't compose. MaxAge is, because an exemplar's value ends with the trace it points at: a trace store is usually kept far shorter than metrics, so on a shared MaxAge exemplars outlive their referents and become dangling pointers. Values above MaxAge are allowed; the budgets are independent.

Threading the signal through retainFrom means Admin.Compact/Admin.Retention honour it too, not just the maintenance loop.

Surface

  • signal.Exemplar — fifth signal, appended (the enum is wire-stable).
  • Storage.ExemplarFetcher(tenants...) — the read seam. Same matchers as the metric query.
  • Storage.ExemplarsForTrace(ctx, tenant, traceID) — completes the correlation triangle with Trace and LogsForTrace. Rides the trace_id equality bloom; within a surviving part it is a column scan, not an index lookup.
  • metric.NumberPoint.Exemplars + AddExemplar(), following the existing resettable-arena discipline. AddPoint now retains the exemplar backing array across Reset so the zero-alloc rebuild cycle still holds.

Exemplar writes are best-effort: their counts stay out of the returned Accepted, which reports data points. Losing an exemplar degrades correlation; losing a sample corrupts a time series. A failure is counted and logged, never propagated.

Verified against oteldb

Wired locally into go-faster-style embedder oteldb (via a replace, not committed) to check the API against its real consumer. oteldb's storagebackend carried:

// The storage engine does not store exemplars yet, so this returns an empty querier.

Implementing ExemplarQuerier against this branch took ~90 lines and needed no storage-side changes — ExemplarFetcher plus the existing PushableMatchers/PromLabels/MatchesAll helpers covered it. Tests there drive the full producer path (pdata → ConsumeMetrics → pdataconv → storage → ExemplarQuerier → Prometheus exemplar.QueryResult) and pass, with trace_id rendered as hex and filtered attributes on the exemplar labels. oteldb's own TestStorageBackend suites stay green.

That work lands separately once this is released.

Note for anyone reproducing: go.work does not work across these two modules — unioning the graphs makes github.com/knadh/koanf ambiguous (storage pulls the v1.5.0 monorepo, oteldb the split modules). Use a replace in oteldb/go.mod.

Testing

14 exemplar-specific tests across signal/exemplar (unit + a value round-trip fuzz target), the root facade (write/query, __name__ selection, ExemplarsForTrace, no-exemplar fast path, flush survival, reopen), pdataconv (conversion + the three drop cases), and retention. The retention test was checked with a negative control — setting ExemplarMaxAge = MaxAge makes it fail.

go test ./..., go vet, golangci-lint run ./..., go mod tidy and go generate all clean. Rebased on main (post-#99).

Follow-ups

🤖 Generated with Claude Code

@tdakkota tdakkota added the enhancement New feature or request label Aug 10, 2026
@codecov

codecov Bot commented Aug 10, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 79.91803% with 49 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
cluster.go 14.28% 18 Missing ⚠️
storage.go 74.50% 8 Missing and 5 partials ⚠️
exemplars.go 87.27% 3 Missing and 4 partials ⚠️
admin.go 0.00% 2 Missing ⚠️
cluster/read.go 33.33% 2 Missing ⚠️
efficiency.go 0.00% 1 Missing and 1 partial ⚠️
recordengine/keys.go 0.00% 1 Missing and 1 partial ⚠️
signal/signal.go 50.00% 2 Missing ⚠️
otlp/pdataconv/pdataconv.go 97.95% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@github-actions

github-actions Bot commented Aug 10, 2026 •
edited
Loading

Copy link
Copy Markdown

📊 Golden benchmarks

Warning

2 benchmarks regressed and 0 improved across the golden set.
Largest regression: write/concurrent bytes/op +19.2%.

benchstat base → head · linux/amd64 · AMD Ryzen 5 3600 6-Core Processor · n=6/benchmark · ~ = not significant (p≥0.05) · changes under ±5% treated as noise

⏱️ Time — sec/op (lower is better)

Benchmark base head Δ
write/head 16.29ms 16.22ms −0.4% ⚪
write/flush 31.02ms 30.20ms −2.6% ⚪
write/concurrent 26.01ms 25.97ms −0.2% ⚪
read/fetch_all 2.70ms 2.78ms +3.0% ⚪
read/fetch_all_release 2.25ms 2.28ms +1.5% ⚪
read/fetch_recent 1.79ms 1.84ms +2.7% ⚪
query/promql_count_cpu_cores 689.13µs 684.75µs −0.6% ⚪
query/promql_full_scan_count 638.59µs 621.88µs −2.6% ⚪
query/promql_cpu_usage_range 2.23ms 2.28ms +2.4% ⚪
density 50.19ms 58.56ms +16.7% 🔴
logs/write_flush 1.70ms 1.68ms −1.0% ⚪
logs/merge 16.57ms 16.44ms −0.8% ⚪
geomean +1.39%

🚀 Throughput — B/s (higher is better)

Benchmark base head Δ
write/head 93.7MiB/s 94.1MiB/s +0.4% ⚪
write/flush 49.2MiB/s 50.5MiB/s +2.7% ⚪
write/concurrent 58.7MiB/s 58.8MiB/s +0.2% ⚪
read/fetch_all 564.9MiB/s 548.2MiB/s −3.0% ⚪
read/fetch_all_release 679.6MiB/s 669.7MiB/s −1.5% ⚪
logs/write_flush 83.2MiB/s 84.1MiB/s +1.0% ⚪
logs/merge 68.4MiB/s 69.0MiB/s +0.8% ⚪
geomean +0.09%

🚀 Throughput — Mpoints/s (higher is better)

Benchmark base head Δ
write/head 7.59 7.45 −1.9% ⚪
write/flush 3.22 3.31 +2.7% ⚪
write/concurrent 7.21 6.37 −11.6% 🔴
geomean -3.79%

📦 Bytes per op — B/op (lower is better)

Benchmark base head Δ
write/head 11.0MiB 11.0MiB ≈0% ⚪
write/flush 46.5MiB 46.2MiB −0.8% ⚪
write/concurrent 71.3MiB 84.9MiB +19.2% 🔴
read/fetch_all 2.3MiB 2.3MiB ≈0% ⚪
read/fetch_all_release 623.9KiB 624.1KiB ≈0% ⚪
read/fetch_recent 777.9KiB 778.0KiB ≈0% ⚪
query/promql_count_cpu_cores 407.6KiB 407.2KiB −0.1% ⚪
query/promql_full_scan_count 399.2KiB 399.1KiB ≈0% ⚪
query/promql_cpu_usage_range 484.9KiB 484.7KiB ≈0% ⚪
density 76.7MiB 88.7MiB +15.7% 🔴
logs/write_flush 1.4MiB 1.4MiB ≈0% ⚪
logs/merge 42.4MiB 42.5MiB +0.2% ⚪
geomean +2.66%

♻️ Allocations — allocs/op (lower is better)

Benchmark base head Δ
write/head 5.9k 5.9k ≈0% ⚪
write/flush 29.5k 30.8k +4.1% ⚪
write/concurrent 10.5k 10.7k +2.4% ⚪
read/fetch_all 4.4k 4.4k ≈0% ⚪
read/fetch_all_release 2.4k 2.4k ≈0% ⚪
read/fetch_recent 4.4k 4.4k ≈0% ⚪
query/promql_count_cpu_cores 1.3k 1.3k ≈0% ⚪
query/promql_full_scan_count 1.3k 1.3k ≈0% ⚪
query/promql_cpu_usage_range 1.7k 1.7k ≈0% ⚪
density 50.1k 50.1k ≈0% ⚪
logs/write_flush 13.0k 13.0k ≈0% ⚪
logs/merge 33.7k 33.7k ≈0% ⚪
geomean +0.54%

🔢 Rows per op — rows/op (informational)

Benchmark base head Δ
read/fetch_all 100.0k 100.0k ≈0% ⚪
read/fetch_all_release 100.0k 100.0k ≈0% ⚪
read/fetch_recent 10.0k 10.0k ≈0% ⚪
geomean +0.00%

🗜️ Density — B/point (lower is better)

Benchmark base head Δ
density 1.498 1.498 ≈0% ⚪
geomean +0.00%
Raw benchstat table 
goos: linux
goarch: amd64
pkg: github.com/oteldb/storage
cpu: AMD Ryzen 5 3600 6-Core Processor              
                                       │  base.txt   │              head.txt               │
                                       │   sec/op    │    sec/op     vs base               │
Golden/write/head-12                     16.29m ± 3%   16.22m ± 11%        ~ (p=0.240 n=6)
Golden/write/flush-12                    31.02m ± 3%   30.20m ±  2%   -2.64% (p=0.004 n=6)
Golden/write/concurrent-12               26.01m ± 7%   25.97m ±  5%        ~ (p=0.699 n=6)
Golden/read/fetch_all-12                 2.701m ± 8%   2.783m ±  4%        ~ (p=0.240 n=6)
Golden/read/fetch_all_release-12         2.245m ± 1%   2.278m ±  2%        ~ (p=0.093 n=6)
Golden/read/fetch_recent-12              1.795m ± 2%   1.843m ±  3%        ~ (p=0.065 n=6)
Golden/query/promql_count_cpu_cores-12   689.1µ ± 2%   684.7µ ±  2%        ~ (p=0.310 n=6)
Golden/query/promql_full_scan_count-12   638.6µ ± 1%   621.9µ ±  2%   -2.62% (p=0.026 n=6)
Golden/query/promql_cpu_usage_range-12   2.230m ± 1%   2.284m ±  1%   +2.41% (p=0.009 n=6)
Golden/density-12                        50.19m ± 4%   58.56m ±  3%  +16.66% (p=0.002 n=6)
Golden/logs/write_flush-12               1.703m ± 2%   1.685m ±  2%        ~ (p=0.394 n=6)
Golden/logs/merge-12                     16.57m ± 9%   16.44m ± 11%        ~ (p=0.818 n=6)
geomean                                  4.915m        4.983m         +1.39%

                                 │   base.txt    │              head.txt               │
                                 │      B/s      │      B/s       vs base              │
Golden/write/head-12               93.67Mi ±  3%   94.06Mi ± 13%       ~ (p=0.240 n=6)
Golden/write/flush-12              49.21Mi ±  3%   50.53Mi ±  2%  +2.68% (p=0.004 n=6)
Golden/write/concurrent-12         58.66Mi ±  8%   58.76Mi ±  4%       ~ (p=0.699 n=6)
Golden/read/fetch_all-12           564.9Mi ±  8%   548.2Mi ±  5%       ~ (p=0.240 n=6)
Golden/read/fetch_all_release-12   679.6Mi ±  1%   669.7Mi ±  2%       ~ (p=0.093 n=6)
Golden/logs/write_flush-12         83.25Mi ±  2%   84.12Mi ±  2%       ~ (p=0.394 n=6)
Golden/logs/merge-12               68.41Mi ± 10%   68.96Mi ± 10%       ~ (p=0.818 n=6)
geomean                            128.9Mi         129.0Mi        +0.09%

                           │  base.txt   │             head.txt              │
                           │  Mpoints/s  │ Mpoints/s   vs base               │
Golden/write/head-12         7.589 ±  2%   7.447 ± 4%   -1.87% (p=0.015 n=6)
Golden/write/flush-12        3.225 ±  3%   3.312 ± 2%   +2.68% (p=0.004 n=6)
Golden/write/concurrent-12   7.212 ± 12%   6.374 ± 6%  -11.61% (p=0.015 n=6)
geomean                      5.609         5.397        -3.79%

                                       │   base.txt    │               head.txt               │
                                       │     B/op      │     B/op       vs base               │
Golden/write/head-12                     10.96Mi ± 13%   10.96Mi ± 34%        ~ (p=0.084 n=6)
Golden/write/flush-12                    46.52Mi ±  5%   46.16Mi ±  3%        ~ (p=0.900 n=6)
Golden/write/concurrent-12               71.26Mi ±  3%   84.92Mi ±  5%  +19.17% (p=0.002 n=6)
Golden/read/fetch_all-12                 2.320Mi ±  0%   2.321Mi ±  0%   +0.02% (p=0.024 n=6)
Golden/read/fetch_all_release-12         623.9Ki ±  0%   624.1Ki ±  0%        ~ (p=0.154 n=6)
Golden/read/fetch_recent-12              777.9Ki ±  0%   778.0Ki ±  0%        ~ (p=0.574 n=6)
Golden/query/promql_count_cpu_cores-12   407.6Ki ±  0%   407.2Ki ±  0%        ~ (p=0.240 n=6)
Golden/query/promql_full_scan_count-12   399.2Ki ±  0%   399.1Ki ±  0%        ~ (p=0.180 n=6)
Golden/query/promql_cpu_usage_range-12   484.9Ki ±  0%   484.7Ki ±  0%   -0.04% (p=0.026 n=6)
Golden/density-12                        76.67Mi ±  0%   88.75Mi ±  0%  +15.75% (p=0.002 n=6)
Golden/logs/write_flush-12               1.365Mi ±  0%   1.365Mi ±  0%        ~ (p=0.818 n=6)
Golden/logs/merge-12                     42.38Mi ±  4%   42.47Mi ±  4%        ~ (p=0.699 n=6)
geomean                                  3.909Mi         4.013Mi         +2.66%

                                       │  base.txt   │              head.txt               │
                                       │  allocs/op  │  allocs/op   vs base                │
Golden/write/head-12                     5.937k ± 4%   5.939k ± 7%       ~ (p=0.097 n=6)
Golden/write/flush-12                    29.54k ± 0%   30.75k ± 0%  +4.11% (p=0.002 n=6)
Golden/write/concurrent-12               10.49k ± 9%   10.74k ± 5%       ~ (p=0.403 n=6)
Golden/read/fetch_all-12                 4.443k ± 0%   4.443k ± 0%       ~ (p=1.000 n=6) ¹
Golden/read/fetch_all_release-12         2.443k ± 0%   2.443k ± 0%       ~ (p=1.000 n=6) ¹
Golden/read/fetch_recent-12              4.443k ± 0%   4.443k ± 0%       ~ (p=1.000 n=6) ¹
Golden/query/promql_count_cpu_cores-12   1.278k ± 0%   1.278k ± 0%       ~ (p=0.455 n=6)
Golden/query/promql_full_scan_count-12   1.301k ± 0%   1.301k ± 0%       ~ (p=1.000 n=6) ¹
Golden/query/promql_cpu_usage_range-12   1.654k ± 0%   1.654k ± 0%       ~ (p=1.000 n=6)
Golden/density-12                        50.14k ± 0%   50.14k ± 0%  +0.00% (p=0.002 n=6)
Golden/logs/write_flush-12               12.98k ± 0%   12.98k ± 0%       ~ (p=0.591 n=6)
Golden/logs/merge-12                     33.74k ± 0%   33.73k ± 0%       ~ (p=1.000 n=6)
geomean                                  6.467k        6.502k       +0.54%
¹ all samples are equal

                                 │  base.txt   │              head.txt               │
                                 │   rows/op   │   rows/op    vs base                │
Golden/read/fetch_all-12           100.0k ± 0%   100.0k ± 0%       ~ (p=1.000 n=6) ¹
Golden/read/fetch_all_release-12   100.0k ± 0%   100.0k ± 0%       ~ (p=1.000 n=6) ¹
Golden/read/fetch_recent-12        10.00k ± 0%   10.00k ± 0%       ~ (p=1.000 n=6) ¹
geomean                            46.42k        46.42k       +0.00%
¹ all samples are equal

                  │  base.txt  │           head.txt            │
                  │  B/point   │  B/point    vs base           │
Golden/density-12   1.498 ± 0%   1.498 ± 0%  ~ (p=1.000 n=6) ¹
¹ all samples are equal

tdakkota and others added 3 commits August 10, 2026 15:15
@tdakkota @claude
feat(exemplar): add exemplar record schema and projection
862ce72 
Exemplars are record-shaped, so they get a record-engine schema rather than a
column on the dense metrics engine. The stream id is the point's metric
SeriesID, read from the already-projected metric batch rather than recomputed,
so metric and exemplar identity cannot drift.

Projection only — nothing stores an exemplar yet.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@tdakkota @claude
feat(exemplar): store and query exemplars end to end
e2fe9c8 
Adds the signal.Exemplar signal, the exemplars record engine, and the facade:
WriteMetrics now stores the exemplars its points carry (best-effort, counts
kept out of the OTLP partial-success tally), ExemplarFetcher reads them back
under the metric's own matchers, and ExemplarsForTrace completes the
trace-correlation triangle.

recordengine now postings-indexes Series.Attributes alongside resource and
scope. Exemplar identity is the metric series, whose labels live there, so
without it a metric selector resolved no exemplar stream. Empty, hence a
no-op, for logs/traces/profiles.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@tdakkota @claude
fix(exemplar): wire exemplars into recovery, WAL replay, and lifecycle
5593c40 
recover() and walReplayerFor() dispatch on a hardcoded per-signal switch that
did not know /exemplars, so a reopened store served no exemplars at all. Close,
Reset, WAL sync, the maintenance loop, and recordEngineSnapshot had the same
gap.

Adds a reopen test asserting a metric selector still resolves exemplar streams
after restart — the head path alone never exercised the rebuilt index.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@tdakkota
tdakkota force-pushed the feat/exemplar-projection branch from 00845e2 to 5593c40 Compare August 10, 2026 12:16
@tdakkota tdakkota changed the title feat(exemplar): add exemplar record schema and projection feat(exemplar): store and query metric exemplars Aug 10, 2026
@github-actions

github-actions Bot commented Aug 10, 2026 •
edited
Loading

Copy link
Copy Markdown

📊 Query engine benchmarks (oteldb)

Warning

3 benchmarks regressed and 0 improved across the golden set.
Largest regression: GoldenLogQL/needle latency +11.6%.

benchstat base → head · linux/amd64 · AMD Ryzen 5 3600 6-Core Processor · n=10/benchmark · ~ = not significant (p≥0.05) · changes under ±5% treated as noise

⏱️ Time — sec/op (lower is better)

Benchmark base head Δ
GoldenLogQL/full_scan 172.85ms 173.66ms +0.5% ⚪
GoldenLogQL/select_service 3.88ms 3.85ms −0.7% ⚪
GoldenLogQL/select_multi_stream 4.94ms 5.06ms +2.5% ⚪
GoldenLogQL/select_regexp 4.69ms 4.83ms +3.0% ⚪
GoldenLogQL/line_filter 4.28ms 4.45ms +4.0% ⚪
GoldenLogQL/line_filter_negated 20.68ms 20.36ms −1.5% ⚪
GoldenLogQL/label_filter 3.97ms 3.96ms −0.2% ⚪
GoldenLogQL/json_parser 26.47ms 27.33ms +3.3% ⚪
GoldenLogQL/logfmt_parser 30.71ms 31.34ms +2.0% ⚪
GoldenLogQL/needle 603.65µs 673.61µs +11.6% 🔴
GoldenLogQL/limit_backward 2.63ms 2.89ms +9.8% 🔴
GoldenLogQL/metric_count_by_level 5.87ms 6.35ms +8.2% 🔴
GoldenLogQL/metric_rate_by_service 85.63ms 87.66ms +2.4% ⚪
GoldenProfileQL/select_merge_profile/all_services 13.81ms 13.94ms +1.0% ⚪
GoldenProfileQL/select_merge_profile/single_pod 1.18ms 1.21ms +2.3% ⚪
GoldenProfileQL/select_merge_profile/regex_service 7.34ms 7.48ms +1.9% ⚪
GoldenProfileQL/select_merge_profile/alloc_space 13.72ms 13.96ms +1.8% ⚪
GoldenProfileQL/select_merge_profile/recent_window 3.54ms 3.69ms +4.3% ⚪
GoldenProfileQL/profile_types 37.08µs 38.00µs +2.5% ⚪
GoldenProfileQL/label_names 43.27µs 44.07µs +1.9% ⚪
GoldenProfileQL/label_values/pod 46.45µs 47.81µs +2.9% ⚪
GoldenProfileQL/profile_resolver 124.51µs 127.73µs +2.6% ⚪
GoldenTraceQL/by_trace_id 772.06µs 802.12µs +3.9% ⚪
GoldenTraceQL/scan_all 11.14ms 11.41ms +2.4% ⚪
GoldenTraceQL/by_service 10.47ms 10.84ms +3.5% ⚪
GoldenTraceQL/by_name 9.88ms 10.15ms +2.7% ⚪
GoldenTraceQL/attr_route 1.38ms 1.45ms +4.9% ⚪
GoldenTraceQL/attr_status_code 2.14ms 2.23ms +4.0% ⚪
GoldenTraceQL/status_error 2.08ms 2.13ms +2.7% ⚪
GoldenTraceQL/kind_server 10.90ms 10.96ms +0.5% ⚪
GoldenTraceQL/duration_gt 10.16ms 10.36ms +2.0% ⚪
GoldenTraceQL/attr_and_status 2.24ms 2.31ms +3.1% ⚪
GoldenTraceQL/descendant 12.36ms 12.40ms +0.4% ⚪
GoldenTraceQL/root_name 12.26ms 12.23ms −0.2% ⚪
GoldenTraceQL/root_service_name 11.89ms 12.03ms +1.2% ⚪
GoldenTraceQL/sibling 11.20ms 11.28ms +0.7% ⚪
GoldenTraceQL/child 11.11ms 11.12ms +0.1% ⚪
GoldenTraceQL/pushdown/status_code 823.47µs 847.36µs +2.9% ⚪
GoldenTraceQL/pushdown/kind 1.51ms 1.52ms +0.7% ⚪
GoldenTraceQL/pushdown/name 961.31µs 980.94µs +2.0% ⚪
GoldenTraceQL/pushdown/attr_route 866.22µs 874.60µs +1.0% ⚪
GoldenTraceQL/pushdown/attr_status_code 862.32µs 879.30µs +2.0% ⚪
GoldenTraceQL/pushdown/attr_and_status 824.96µs 851.34µs +3.2% ⚪
geomean +2.52%

🚀 Throughput — B/s (higher is better)

Benchmark base head Δ
GoldenLogQL/full_scan 44.9MiB/s 44.7MiB/s −0.5% ⚪
GoldenLogQL/select_service 225.2MiB/s 226.7MiB/s +0.7% ⚪
GoldenLogQL/select_multi_stream 864.0MiB/s 843.3MiB/s −2.4% ⚪
GoldenLogQL/select_regexp 744.2MiB/s 722.8MiB/s −2.9% ⚪
GoldenLogQL/line_filter 204.2MiB/s 196.3MiB/s −3.9% ⚪
GoldenLogQL/line_filter_negated 42.2MiB/s 42.9MiB/s +1.5% ⚪
GoldenLogQL/label_filter 219.9MiB/s 220.4MiB/s +0.2% ⚪
GoldenLogQL/json_parser 33.0MiB/s 32.0MiB/s −3.2% ⚪
GoldenLogQL/logfmt_parser 25.1MiB/s 24.6MiB/s −2.0% ⚪
GoldenLogQL/needle 6.90GiB/s 6.18GiB/s −10.4% 🔴
GoldenLogQL/limit_backward 1.58GiB/s 1.44GiB/s −8.9% 🔴
GoldenLogQL/metric_count_by_level 726.9MiB/s 672.0MiB/s −7.6% 🔴
GoldenLogQL/metric_rate_by_service 49.8MiB/s 48.6MiB/s −2.3% ⚪
GoldenProfileQL/select_merge_profile/all_services 36.2MiB/s 35.9MiB/s −0.9% ⚪
GoldenProfileQL/select_merge_profile/single_pod 13.2MiB/s 12.9MiB/s −2.3% ⚪
GoldenProfileQL/select_merge_profile/regex_service 34.0MiB/s 33.4MiB/s −1.8% ⚪
GoldenProfileQL/select_merge_profile/alloc_space 36.4MiB/s 35.8MiB/s −1.8% ⚪
GoldenProfileQL/select_merge_profile/recent_window 35.3MiB/s 33.8MiB/s −4.1% ⚪
GoldenTraceQL/by_trace_id 1.5MiB/s 1.4MiB/s −3.5% ⚪
GoldenTraceQL/scan_all 51.8MiB/s 50.6MiB/s −2.4% ⚪
GoldenTraceQL/by_service 55.1MiB/s 53.3MiB/s −3.4% ⚪
GoldenTraceQL/by_name 58.4MiB/s 56.9MiB/s −2.6% ⚪
GoldenTraceQL/attr_route 418.2MiB/s 398.8MiB/s −4.6% ⚪
GoldenTraceQL/attr_status_code 269.1MiB/s 258.8MiB/s −3.8% ⚪
GoldenTraceQL/status_error 277.7MiB/s 270.3MiB/s −2.7% ⚪
GoldenTraceQL/kind_server 52.9MiB/s 52.7MiB/s −0.5% ⚪
GoldenTraceQL/duration_gt 56.8MiB/s 55.7MiB/s −2.0% ⚪
GoldenTraceQL/attr_and_status 257.5MiB/s 249.7MiB/s −3.0% ⚪
GoldenTraceQL/descendant 46.7MiB/s 46.5MiB/s −0.3% ⚪
GoldenTraceQL/root_name 47.1MiB/s 47.2MiB/s +0.2% ⚪
GoldenTraceQL/root_service_name 48.5MiB/s 48.0MiB/s −1.2% ⚪
GoldenTraceQL/sibling 51.5MiB/s 51.1MiB/s −0.7% ⚪
GoldenTraceQL/child 52.0MiB/s 51.9MiB/s −0.1% ⚪
GoldenTraceQL/pushdown/status_code 700.7MiB/s 680.9MiB/s −2.8% ⚪
GoldenTraceQL/pushdown/kind 383.1MiB/s 380.3MiB/s −0.7% ⚪
GoldenTraceQL/pushdown/name 600.2MiB/s 588.2MiB/s −2.0% ⚪
GoldenTraceQL/pushdown/attr_route 666.1MiB/s 659.7MiB/s −1.0% ⚪
GoldenTraceQL/pushdown/attr_status_code 669.1MiB/s 656.2MiB/s −1.9% ⚪
GoldenTraceQL/pushdown/attr_and_status 699.4MiB/s 677.8MiB/s −3.1% ⚪
geomean -2.46%

📦 Bytes per op — B/op (lower is better)

Benchmark base head Δ
GoldenLogQL/full_scan 131.3MiB 131.3MiB ≈0% ⚪
GoldenLogQL/select_service 6.5MiB 6.5MiB ≈0% ⚪
GoldenLogQL/select_multi_stream 10.2MiB 10.2MiB ≈0% ⚪
GoldenLogQL/select_regexp 9.3MiB 9.3MiB +0.1% ⚪
GoldenLogQL/line_filter 3.8MiB 3.8MiB ≈0% ⚪
GoldenLogQL/line_filter_negated 44.9MiB 44.9MiB ≈0% ⚪
GoldenLogQL/label_filter 3.8MiB 3.8MiB +0.1% ⚪
GoldenLogQL/json_parser 47.3MiB 47.3MiB ≈0% ⚪
GoldenLogQL/logfmt_parser 55.3MiB 55.3MiB ≈0% ⚪
GoldenLogQL/needle 1.1MiB 1.1MiB −0.1% ⚪
GoldenLogQL/limit_backward 9.2MiB 9.2MiB −0.1% ⚪
GoldenLogQL/metric_count_by_level 4.1MiB 4.1MiB ≈0% ⚪
GoldenLogQL/metric_rate_by_service 94.8MiB 94.8MiB ≈0% ⚪
GoldenProfileQL/select_merge_profile/all_services 9.2MiB 9.2MiB ≈0% ⚪
GoldenProfileQL/select_merge_profile/single_pod 1.7MiB 1.7MiB −0.1% ⚪
GoldenProfileQL/select_merge_profile/regex_service 5.3MiB 5.3MiB −0.1% ⚪
GoldenProfileQL/select_merge_profile/alloc_space 9.2MiB 9.2MiB ≈0% ⚪
GoldenProfileQL/select_merge_profile/recent_window 2.5MiB 2.5MiB +0.3% ⚪
GoldenProfileQL/profile_types 21.5KiB 21.5KiB ≈0% ⚪
GoldenProfileQL/label_names 23.1KiB 23.1KiB ≈0% ⚪
GoldenProfileQL/label_values/pod 26.7KiB 26.7KiB ≈0% ⚪
GoldenProfileQL/profile_resolver 181.6KiB 181.6KiB ≈0% ⚪
GoldenTraceQL/by_trace_id 840.1KiB 840.6KiB +0.1% ⚪
GoldenTraceQL/scan_all 17.7MiB 17.7MiB ≈0% ⚪
GoldenTraceQL/by_service 14.6MiB 14.6MiB ≈0% ⚪
GoldenTraceQL/by_name 13.9MiB 13.9MiB ≈0% ⚪
GoldenTraceQL/attr_route 1.5MiB 1.5MiB ≈0% ⚪
GoldenTraceQL/attr_status_code 2.5MiB 2.5MiB ≈0% ⚪
GoldenTraceQL/status_error 2.4MiB 2.4MiB ≈0% ⚪
GoldenTraceQL/kind_server 15.6MiB 15.6MiB ≈0% ⚪
GoldenTraceQL/duration_gt 13.9MiB 13.9MiB ≈0% ⚪
GoldenTraceQL/attr_and_status 2.5MiB 2.5MiB ≈0% ⚪
GoldenTraceQL/descendant 16.4MiB 16.4MiB ≈0% ⚪
GoldenTraceQL/root_name 17.6MiB 17.6MiB ≈0% ⚪
GoldenTraceQL/root_service_name 17.5MiB 17.5MiB ≈0% ⚪
GoldenTraceQL/sibling 14.9MiB 14.9MiB ≈0% ⚪
GoldenTraceQL/child 14.9MiB 14.9MiB ≈0% ⚪
GoldenTraceQL/pushdown/status_code 988.6KiB 988.5KiB ≈0% ⚪
GoldenTraceQL/pushdown/kind 2.5MiB 2.5MiB ≈0% ⚪
GoldenTraceQL/pushdown/name 1.3MiB 1.3MiB ≈0% ⚪
GoldenTraceQL/pushdown/attr_route 950.2KiB 950.4KiB ≈0% ⚪
GoldenTraceQL/pushdown/attr_status_code 984.6KiB 985.0KiB ≈0% ⚪
GoldenTraceQL/pushdown/attr_and_status 915.3KiB 915.6KiB ≈0% ⚪
geomean +0.01%

♻️ Allocations — allocs/op (lower is better)

Benchmark base head Δ
GoldenLogQL/full_scan 1.02M 1.02M ≈0% ⚪
GoldenLogQL/select_service 21.6k 21.6k ≈0% ⚪
GoldenLogQL/select_multi_stream 22.2k 22.2k ≈0% ⚪
GoldenLogQL/select_regexp 22.1k 22.1k ≈0% ⚪
GoldenLogQL/line_filter 20.3k 20.3k ≈0% ⚪
GoldenLogQL/line_filter_negated 111.0k 111.0k ≈0% ⚪
GoldenLogQL/label_filter 23.0k 23.0k ≈0% ⚪
GoldenLogQL/json_parser 182.9k 182.9k ≈0% ⚪
GoldenLogQL/logfmt_parser 185.4k 185.4k ≈0% ⚪
GoldenLogQL/needle 513 513 ≈0% ⚪
GoldenLogQL/limit_backward 3.1k 3.1k ≈0% ⚪
GoldenLogQL/metric_count_by_level 29.4k 29.4k ≈0% ⚪
GoldenLogQL/metric_rate_by_service 654.6k 654.6k ≈0% ⚪
GoldenProfileQL/select_merge_profile/all_services 157.4k 157.4k ≈0% ⚪
GoldenProfileQL/select_merge_profile/single_pod 6.0k 6.0k ≈0% ⚪
GoldenProfileQL/select_merge_profile/regex_service 79.2k 79.2k ≈0% ⚪
GoldenProfileQL/select_merge_profile/alloc_space 157.4k 157.4k ≈0% ⚪
GoldenProfileQL/select_merge_profile/recent_window 40.5k 40.5k ≈0% ⚪
GoldenProfileQL/profile_types 589 589 ≈0% ⚪
GoldenProfileQL/label_names 840 840 ≈0% ⚪
GoldenProfileQL/label_values/pod 847 847 ≈0% ⚪
GoldenProfileQL/profile_resolver 755 755 ≈0% ⚪
GoldenTraceQL/by_trace_id 629 629 ≈0% ⚪
GoldenTraceQL/scan_all 73.3k 73.3k ≈0% ⚪
GoldenTraceQL/by_service 69.1k 69.1k ≈0% ⚪
GoldenTraceQL/by_name 67.2k 67.2k ≈0% ⚪
GoldenTraceQL/attr_route 2.5k 2.5k ≈0% ⚪
GoldenTraceQL/attr_status_code 7.8k 7.8k ≈0% ⚪
GoldenTraceQL/status_error 7.7k 7.7k ≈0% ⚪
GoldenTraceQL/kind_server 70.7k 70.7k ≈0% ⚪
GoldenTraceQL/duration_gt 67.2k 67.2k ≈0% ⚪
GoldenTraceQL/attr_and_status 7.9k 7.9k ≈0% ⚪
GoldenTraceQL/descendant 74.7k 74.7k ≈0% ⚪
GoldenTraceQL/root_name 73.7k 73.7k ≈0% ⚪
GoldenTraceQL/root_service_name 73.6k 73.6k ≈0% ⚪
GoldenTraceQL/sibling 71.8k 71.8k ≈0% ⚪
GoldenTraceQL/child 71.8k 71.8k ≈0% ⚪
GoldenTraceQL/pushdown/status_code 401 401 ≈0% ⚪
GoldenTraceQL/pushdown/kind 661 661 ≈0% ⚪
GoldenTraceQL/pushdown/name 419 419 ≈0% ⚪
GoldenTraceQL/pushdown/attr_route 487 487 ≈0% ⚪
GoldenTraceQL/pushdown/attr_status_code 401 401 ≈0% ⚪
GoldenTraceQL/pushdown/attr_and_status 363 363 ≈0% ⚪
geomean -0.00%
Raw benchstat table 
goos: linux
goarch: amd64
pkg: github.com/oteldb/oteldb/internal/storagebackend
cpu: AMD Ryzen 5 3600 6-Core Processor              
                                                      │  base.txt   │               head.txt               │
                                                      │   sec/op    │    sec/op     vs base                │
GoldenLogQL/full_scan-12                                172.9m ± 1%    173.7m ± 1%        ~ (p=0.853 n=10)
GoldenLogQL/select_service-12                           3.878m ± 1%    3.853m ± 1%        ~ (p=0.105 n=10)
GoldenLogQL/select_multi_stream-12                      4.936m ± 2%    5.057m ± 3%        ~ (p=0.075 n=10)
GoldenLogQL/select_regexp-12                            4.695m ± 2%    4.834m ± 4%        ~ (p=0.052 n=10)
GoldenLogQL/line_filter-12                              4.277m ± 2%    4.449m ± 2%   +4.04% (p=0.000 n=10)
GoldenLogQL/line_filter_negated-12                      20.68m ± 3%    20.36m ± 1%   -1.51% (p=0.007 n=10)
GoldenLogQL/label_filter-12                             3.972m ± 2%    3.964m ± 1%        ~ (p=0.796 n=10)
GoldenLogQL/json_parser-12                              26.47m ± 3%    27.33m ± 3%   +3.27% (p=0.005 n=10)
GoldenLogQL/logfmt_parser-12                            30.71m ± 3%    31.34m ± 1%   +2.04% (p=0.015 n=10)
GoldenLogQL/needle-12                                   603.7µ ± 1%    673.6µ ± 1%  +11.59% (p=0.000 n=10)
GoldenLogQL/limit_backward-12                           2.632m ± 2%    2.890m ± 2%   +9.80% (p=0.000 n=10)
GoldenLogQL/metric_count_by_level-12                    5.866m ± 1%    6.346m ± 1%   +8.18% (p=0.000 n=10)
GoldenLogQL/metric_rate_by_service-12                   85.63m ± 1%    87.66m ± 1%   +2.37% (p=0.000 n=10)
GoldenProfileQL/select_merge_profile/all_services-12    13.81m ± 1%    13.94m ± 2%        ~ (p=0.218 n=10)
GoldenProfileQL/select_merge_profile/single_pod-12      1.181m ± 2%    1.208m ± 2%   +2.34% (p=0.004 n=10)
GoldenProfileQL/select_merge_profile/regex_service-12   7.345m ± 1%    7.481m ± 1%   +1.86% (p=0.001 n=10)
GoldenProfileQL/select_merge_profile/alloc_space-12     13.72m ± 1%    13.96m ± 1%   +1.78% (p=0.000 n=10)
GoldenProfileQL/select_merge_profile/recent_window-12   3.542m ± 1%    3.694m ± 1%   +4.29% (p=0.000 n=10)
GoldenProfileQL/profile_types-12                        37.08µ ± 1%    38.00µ ± 1%   +2.48% (p=0.000 n=10)
GoldenProfileQL/label_names-12                          43.27µ ± 1%    44.07µ ± 1%   +1.85% (p=0.000 n=10)
GoldenProfileQL/label_values/pod-12                     46.45µ ± 0%    47.81µ ± 1%   +2.91% (p=0.000 n=10)
GoldenProfileQL/profile_resolver-12                     124.5µ ± 0%    127.7µ ± 1%   +2.59% (p=0.000 n=10)
GoldenTraceQL/by_trace_id-12                            772.1µ ± 1%    802.1µ ± 1%   +3.89% (p=0.000 n=10)
GoldenTraceQL/scan_all-12                               11.14m ± 3%    11.41m ± 1%   +2.45% (p=0.003 n=10)
GoldenTraceQL/by_service-12                             10.47m ± 2%    10.84m ± 1%   +3.51% (p=0.000 n=10)
GoldenTraceQL/by_name-12                                9.879m ± 1%   10.147m ± 3%   +2.71% (p=0.019 n=10)
GoldenTraceQL/attr_route-12                             1.380m ± 1%    1.447m ± 2%   +4.87% (p=0.000 n=10)
GoldenTraceQL/attr_status_code-12                       2.144m ± 1%    2.229m ± 2%   +3.97% (p=0.002 n=10)
GoldenTraceQL/status_error-12                           2.077m ± 2%    2.134m ± 1%   +2.74% (p=0.000 n=10)
GoldenTraceQL/kind_server-12                            10.90m ± 1%    10.96m ± 4%        ~ (p=0.393 n=10)
GoldenTraceQL/duration_gt-12                            10.16m ± 1%    10.36m ± 2%   +2.01% (p=0.023 n=10)
GoldenTraceQL/attr_and_status-12                        2.241m ± 2%    2.311m ± 4%        ~ (p=0.052 n=10)
GoldenTraceQL/descendant-12                             12.36m ± 2%    12.40m ± 3%        ~ (p=0.631 n=10)
GoldenTraceQL/root_name-12                              12.26m ± 2%    12.23m ± 1%        ~ (p=1.000 n=10)
GoldenTraceQL/root_service_name-12                      11.89m ± 3%    12.03m ± 2%        ~ (p=0.631 n=10)
GoldenTraceQL/sibling-12                                11.20m ± 2%    11.28m ± 2%        ~ (p=0.123 n=10)
GoldenTraceQL/child-12                                  11.11m ± 2%    11.12m ± 2%        ~ (p=0.853 n=10)
GoldenTraceQL/pushdown/status_code-12                   823.5µ ± 1%    847.4µ ± 2%   +2.90% (p=0.000 n=10)
GoldenTraceQL/pushdown/kind-12                          1.506m ± 1%    1.517m ± 1%        ~ (p=0.436 n=10)
GoldenTraceQL/pushdown/name-12                          961.3µ ± 1%    980.9µ ± 1%   +2.04% (p=0.000 n=10)
GoldenTraceQL/pushdown/attr_route-12                    866.2µ ± 0%    874.6µ ± 1%   +0.97% (p=0.000 n=10)
GoldenTraceQL/pushdown/attr_status_code-12              862.3µ ± 1%    879.3µ ± 1%   +1.97% (p=0.000 n=10)
GoldenTraceQL/pushdown/attr_and_status-12               825.0µ ± 1%    851.3µ ± 0%   +3.20% (p=0.000 n=10)
geomean                                                 3.360m         3.444m        +2.52%

                                                      │   base.txt   │               head.txt               │
                                                      │     B/s      │     B/s       vs base                │
GoldenLogQL/full_scan-12                                44.88Mi ± 1%   44.67Mi ± 1%        ~ (p=0.853 n=10)
GoldenLogQL/select_service-12                           225.2Mi ± 1%   226.7Mi ± 1%        ~ (p=0.105 n=10)
GoldenLogQL/select_multi_stream-12                      864.0Mi ± 2%   843.3Mi ± 3%        ~ (p=0.075 n=10)
GoldenLogQL/select_regexp-12                            744.2Mi ± 2%   722.8Mi ± 4%        ~ (p=0.052 n=10)
GoldenLogQL/line_filter-12                              204.2Mi ± 2%   196.3Mi ± 2%   -3.88% (p=0.000 n=10)
GoldenLogQL/line_filter_negated-12                      42.25Mi ± 3%   42.90Mi ± 1%   +1.53% (p=0.007 n=10)
GoldenLogQL/label_filter-12                             219.9Mi ± 2%   220.4Mi ± 1%        ~ (p=0.796 n=10)
GoldenLogQL/json_parser-12                              33.00Mi ± 3%   31.96Mi ± 3%   -3.16% (p=0.005 n=10)
GoldenLogQL/logfmt_parser-12                            25.09Mi ± 3%   24.59Mi ± 1%   -2.01% (p=0.015 n=10)
GoldenLogQL/needle-12                                   6.898Gi ± 1%   6.182Gi ± 1%  -10.39% (p=0.000 n=10)
GoldenLogQL/limit_backward-12                           1.582Gi ± 2%   1.441Gi ± 2%   -8.92% (p=0.000 n=10)
GoldenLogQL/metric_count_by_level-12                    726.9Mi ± 1%   672.0Mi ± 1%   -7.56% (p=0.000 n=10)
GoldenLogQL/metric_rate_by_service-12                   49.80Mi ± 1%   48.65Mi ± 1%   -2.32% (p=0.000 n=10)
GoldenProfileQL/select_merge_profile/all_services-12    36.21Mi ± 1%   35.87Mi ± 2%        ~ (p=0.196 n=10)
GoldenProfileQL/select_merge_profile/single_pod-12      13.23Mi ± 2%   12.93Mi ± 2%   -2.27% (p=0.004 n=10)
GoldenProfileQL/select_merge_profile/regex_service-12   34.04Mi ± 1%   33.42Mi ± 1%   -1.83% (p=0.001 n=10)
GoldenProfileQL/select_merge_profile/alloc_space-12     36.45Mi ± 1%   35.81Mi ± 1%   -1.75% (p=0.000 n=10)
GoldenProfileQL/select_merge_profile/recent_window-12   35.29Mi ± 1%   33.84Mi ± 1%   -4.11% (p=0.000 n=10)
GoldenTraceQL/by_trace_id-12                            1.493Mi ± 2%   1.440Mi ± 1%   -3.51% (p=0.000 n=10)
GoldenTraceQL/scan_all-12                               51.81Mi ± 3%   50.57Mi ± 1%   -2.38% (p=0.003 n=10)
GoldenTraceQL/by_service-12                             55.12Mi ± 2%   53.25Mi ± 1%   -3.39% (p=0.000 n=10)
GoldenTraceQL/by_name-12                                58.40Mi ± 1%   56.86Mi ± 3%   -2.64% (p=0.019 n=10)
GoldenTraceQL/attr_route-12                             418.2Mi ± 1%   398.8Mi ± 2%   -4.64% (p=0.000 n=10)
GoldenTraceQL/attr_status_code-12                       269.1Mi ± 1%   258.8Mi ± 2%   -3.82% (p=0.002 n=10)
GoldenTraceQL/status_error-12                           277.7Mi ± 2%   270.3Mi ± 1%   -2.67% (p=0.000 n=10)
GoldenTraceQL/kind_server-12                            52.91Mi ± 1%   52.65Mi ± 4%        ~ (p=0.404 n=10)
GoldenTraceQL/duration_gt-12                            56.80Mi ± 2%   55.68Mi ± 2%   -1.96% (p=0.022 n=10)
GoldenTraceQL/attr_and_status-12                        257.5Mi ± 2%   249.7Mi ± 4%        ~ (p=0.052 n=10)
GoldenTraceQL/descendant-12                             46.69Mi ± 2%   46.53Mi ± 3%        ~ (p=0.631 n=10)
GoldenTraceQL/root_name-12                              47.08Mi ± 2%   47.17Mi ± 1%        ~ (p=0.986 n=10)
GoldenTraceQL/root_service_name-12                      48.53Mi ± 3%   47.95Mi ± 2%        ~ (p=0.643 n=10)
GoldenTraceQL/sibling-12                                51.50Mi ± 2%   51.14Mi ± 3%        ~ (p=0.123 n=10)
GoldenTraceQL/child-12                                  51.95Mi ± 2%   51.89Mi ± 2%        ~ (p=0.837 n=10)
GoldenTraceQL/pushdown/status_code-12                   700.7Mi ± 1%   680.9Mi ± 2%   -2.82% (p=0.000 n=10)
GoldenTraceQL/pushdown/kind-12                          383.1Mi ± 1%   380.3Mi ± 1%        ~ (p=0.436 n=10)
GoldenTraceQL/pushdown/name-12                          600.2Mi ± 1%   588.2Mi ± 1%   -2.00% (p=0.000 n=10)
GoldenTraceQL/pushdown/attr_route-12                    666.1Mi ± 0%   659.7Mi ± 1%   -0.96% (p=0.000 n=10)
GoldenTraceQL/pushdown/attr_status_code-12              669.1Mi ± 1%   656.2Mi ± 1%   -1.93% (p=0.000 n=10)
GoldenTraceQL/pushdown/attr_and_status-12               699.4Mi ± 1%   677.8Mi ± 0%   -3.10% (p=0.000 n=10)
geomean                                                 125.8Mi        122.7Mi        -2.46%

                                                      │   base.txt   │               head.txt                │
                                                      │     B/op     │     B/op      vs base                 │
GoldenLogQL/full_scan-12                                131.3Mi ± 0%   131.3Mi ± 0%       ~ (p=0.971 n=10)
GoldenLogQL/select_service-12                           6.530Mi ± 0%   6.526Mi ± 0%       ~ (p=0.218 n=10)
GoldenLogQL/select_multi_stream-12                      10.16Mi ± 0%   10.16Mi ± 0%       ~ (p=0.796 n=10)
GoldenLogQL/select_regexp-12                            9.280Mi ± 0%   9.291Mi ± 0%       ~ (p=1.000 n=10)
GoldenLogQL/line_filter-12                              3.754Mi ± 0%   3.754Mi ± 0%       ~ (p=0.739 n=10)
GoldenLogQL/line_filter_negated-12                      44.88Mi ± 0%   44.89Mi ± 0%       ~ (p=0.165 n=10)
GoldenLogQL/label_filter-12                             3.787Mi ± 0%   3.790Mi ± 0%       ~ (p=0.063 n=10)
GoldenLogQL/json_parser-12                              47.35Mi ± 0%   47.33Mi ± 0%       ~ (p=0.579 n=10)
GoldenLogQL/logfmt_parser-12                            55.26Mi ± 0%   55.26Mi ± 0%       ~ (p=0.684 n=10)
GoldenLogQL/needle-12                                   1.078Mi ± 0%   1.076Mi ± 0%       ~ (p=0.579 n=10)
GoldenLogQL/limit_backward-12                           9.210Mi ± 0%   9.204Mi ± 0%       ~ (p=0.393 n=10)
GoldenLogQL/metric_count_by_level-12                    4.065Mi ± 0%   4.065Mi ± 0%       ~ (p=0.436 n=10)
GoldenLogQL/metric_rate_by_service-12                   94.82Mi ± 0%   94.82Mi ± 0%       ~ (p=0.870 n=10)
GoldenProfileQL/select_merge_profile/all_services-12    9.184Mi ± 0%   9.184Mi ± 0%       ~ (p=0.739 n=10)
GoldenProfileQL/select_merge_profile/single_pod-12      1.680Mi ± 0%   1.678Mi ± 0%       ~ (p=0.481 n=10)
GoldenProfileQL/select_merge_profile/regex_service-12   5.337Mi ± 0%   5.334Mi ± 0%       ~ (p=0.063 n=10)
GoldenProfileQL/select_merge_profile/alloc_space-12     9.184Mi ± 0%   9.187Mi ± 0%       ~ (p=0.066 n=10)
GoldenProfileQL/select_merge_profile/recent_window-12   2.486Mi ± 0%   2.494Mi ± 1%  +0.34% (p=0.015 n=10)
GoldenProfileQL/profile_types-12                        21.52Ki ± 0%   21.52Ki ± 0%       ~ (p=1.000 n=10) ¹
GoldenProfileQL/label_names-12                          23.14Ki ± 0%   23.14Ki ± 0%       ~ (p=1.000 n=10) ¹
GoldenProfileQL/label_values/pod-12                     26.73Ki ± 0%   26.73Ki ± 0%       ~ (p=1.000 n=10) ¹
GoldenProfileQL/profile_resolver-12                     181.6Ki ± 0%   181.6Ki ± 0%       ~ (p=1.000 n=10) ¹
GoldenTraceQL/by_trace_id-12                            840.1Ki ± 0%   840.6Ki ± 0%       ~ (p=0.052 n=10)
GoldenTraceQL/scan_all-12                               17.70Mi ± 0%   17.70Mi ± 0%       ~ (p=0.481 n=10)
GoldenTraceQL/by_service-12                             14.60Mi ± 0%   14.60Mi ± 0%       ~ (p=0.631 n=10)
GoldenTraceQL/by_name-12                                13.88Mi ± 0%   13.88Mi ± 0%       ~ (p=0.853 n=10)
GoldenTraceQL/attr_route-12                             1.498Mi ± 0%   1.498Mi ± 0%       ~ (p=0.853 n=10)
GoldenTraceQL/attr_status_code-12                       2.470Mi ± 0%   2.469Mi ± 0%       ~ (p=0.123 n=10)
GoldenTraceQL/status_error-12                           2.386Mi ± 0%   2.386Mi ± 0%       ~ (p=0.796 n=10)
GoldenTraceQL/kind_server-12                            15.57Mi ± 0%   15.57Mi ± 0%  -0.01% (p=0.029 n=10)
GoldenTraceQL/duration_gt-12                            13.88Mi ± 0%   13.88Mi ± 0%       ~ (p=0.481 n=10)
GoldenTraceQL/attr_and_status-12                        2.508Mi ± 0%   2.508Mi ± 0%       ~ (p=0.853 n=10)
GoldenTraceQL/descendant-12                             16.40Mi ± 0%   16.40Mi ± 0%       ~ (p=0.089 n=10)
GoldenTraceQL/root_name-12                              17.63Mi ± 0%   17.63Mi ± 0%       ~ (p=1.000 n=10)
GoldenTraceQL/root_service_name-12                      17.52Mi ± 0%   17.52Mi ± 0%       ~ (p=0.971 n=10)
GoldenTraceQL/sibling-12                                14.86Mi ± 0%   14.86Mi ± 0%       ~ (p=0.353 n=10)
GoldenTraceQL/child-12                                  14.89Mi ± 0%   14.89Mi ± 0%       ~ (p=0.971 n=10)
GoldenTraceQL/pushdown/status_code-12                   988.6Ki ± 0%   988.5Ki ± 0%       ~ (p=0.315 n=10)
GoldenTraceQL/pushdown/kind-12                          2.467Mi ± 0%   2.466Mi ± 0%       ~ (p=0.075 n=10)
GoldenTraceQL/pushdown/name-12                          1.281Mi ± 0%   1.281Mi ± 0%       ~ (p=0.971 n=10)
GoldenTraceQL/pushdown/attr_route-12                    950.2Ki ± 0%   950.4Ki ± 0%  +0.02% (p=0.043 n=10)
GoldenTraceQL/pushdown/attr_status_code-12              984.6Ki ± 0%   985.0Ki ± 0%  +0.04% (p=0.002 n=10)
GoldenTraceQL/pushdown/attr_and_status-12               915.3Ki ± 0%   915.6Ki ± 0%       ~ (p=0.063 n=10)
geomean                                                 4.005Mi        4.005Mi       +0.01%
¹ all samples are equal

                                                      │  base.txt   │               head.txt               │
                                                      │  allocs/op  │  allocs/op   vs base                 │
GoldenLogQL/full_scan-12                                1.025M ± 0%   1.025M ± 0%       ~ (p=0.614 n=10)
GoldenLogQL/select_service-12                           21.62k ± 0%   21.62k ± 0%       ~ (p=0.554 n=10)
GoldenLogQL/select_multi_stream-12                      22.25k ± 0%   22.25k ± 0%       ~ (p=0.743 n=10)
GoldenLogQL/select_regexp-12                            22.13k ± 0%   22.13k ± 0%       ~ (p=0.890 n=10)
GoldenLogQL/line_filter-12                              20.26k ± 0%   20.26k ± 0%       ~ (p=1.000 n=10)
GoldenLogQL/line_filter_negated-12                      111.0k ± 0%   111.0k ± 0%       ~ (p=0.344 n=10)
GoldenLogQL/label_filter-12                             23.05k ± 0%   23.05k ± 0%       ~ (p=0.132 n=10)
GoldenLogQL/json_parser-12                              182.9k ± 0%   182.9k ± 0%       ~ (p=0.695 n=10)
GoldenLogQL/logfmt_parser-12                            185.4k ± 0%   185.4k ± 0%       ~ (p=0.422 n=10)
GoldenLogQL/needle-12                                    513.0 ± 0%    513.0 ± 0%       ~ (p=1.000 n=10)
GoldenLogQL/limit_backward-12                           3.114k ± 0%   3.113k ± 0%       ~ (p=0.280 n=10)
GoldenLogQL/metric_count_by_level-12                    29.42k ± 0%   29.42k ± 0%       ~ (p=1.000 n=10) ¹
GoldenLogQL/metric_rate_by_service-12                   654.6k ± 0%   654.6k ± 0%       ~ (p=0.507 n=10)
GoldenProfileQL/select_merge_profile/all_services-12    157.4k ± 0%   157.4k ± 0%       ~ (p=0.582 n=10)
GoldenProfileQL/select_merge_profile/single_pod-12      6.043k ± 0%   6.043k ± 0%       ~ (p=0.350 n=10)
GoldenProfileQL/select_merge_profile/regex_service-12   79.25k ± 0%   79.25k ± 0%       ~ (p=0.120 n=10)
GoldenProfileQL/select_merge_profile/alloc_space-12     157.4k ± 0%   157.4k ± 0%       ~ (p=0.232 n=10)
GoldenProfileQL/select_merge_profile/recent_window-12   40.49k ± 0%   40.49k ± 0%  +0.00% (p=0.032 n=10)
GoldenProfileQL/profile_types-12                         589.0 ± 0%    589.0 ± 0%       ~ (p=1.000 n=10) ¹
GoldenProfileQL/label_names-12                           840.0 ± 0%    840.0 ± 0%       ~ (p=1.000 n=10) ¹
GoldenProfileQL/label_values/pod-12                      847.0 ± 0%    847.0 ± 0%       ~ (p=1.000 n=10) ¹
GoldenProfileQL/profile_resolver-12                      755.0 ± 0%    755.0 ± 0%       ~ (p=1.000 n=10) ¹
GoldenTraceQL/by_trace_id-12                             629.0 ± 0%    629.0 ± 0%       ~ (p=1.000 n=10) ¹
GoldenTraceQL/scan_all-12                               73.26k ± 0%   73.26k ± 0%       ~ (p=1.000 n=10) ¹
GoldenTraceQL/by_service-12                             69.11k ± 0%   69.11k ± 0%       ~ (p=1.000 n=10) ¹
GoldenTraceQL/by_name-12                                67.17k ± 0%   67.17k ± 0%       ~ (p=1.000 n=10) ¹
GoldenTraceQL/attr_route-12                             2.515k ± 0%   2.515k ± 0%       ~ (p=1.000 n=10) ¹
GoldenTraceQL/attr_status_code-12                       7.819k ± 0%   7.818k ± 0%  -0.01% (p=0.011 n=10)
GoldenTraceQL/status_error-12                           7.705k ± 0%   7.705k ± 0%       ~ (p=1.000 n=10) ¹
GoldenTraceQL/kind_server-12                            70.70k ± 0%   70.70k ± 0%       ~ (p=1.000 n=10)
GoldenTraceQL/duration_gt-12                            67.20k ± 0%   67.20k ± 0%       ~ (p=0.303 n=10)
GoldenTraceQL/attr_and_status-12                        7.855k ± 0%   7.855k ± 0%       ~ (p=1.000 n=10) ¹
GoldenTraceQL/descendant-12                             74.74k ± 0%   74.74k ± 0%       ~ (p=1.000 n=10)
GoldenTraceQL/root_name-12                              73.67k ± 0%   73.67k ± 0%       ~ (p=0.650 n=10)
GoldenTraceQL/root_service_name-12                      73.62k ± 0%   73.62k ± 0%       ~ (p=1.000 n=10)
GoldenTraceQL/sibling-12                                71.84k ± 0%   71.84k ± 0%       ~ (p=1.000 n=10) ¹
GoldenTraceQL/child-12                                  71.85k ± 0%   71.85k ± 0%       ~ (p=1.000 n=10) ¹
GoldenTraceQL/pushdown/status_code-12                    401.0 ± 0%    401.0 ± 0%       ~ (p=1.000 n=10) ¹
GoldenTraceQL/pushdown/kind-12                           661.0 ± 0%    661.0 ± 0%       ~ (p=1.000 n=10) ¹
GoldenTraceQL/pushdown/name-12                           419.0 ± 0%    419.0 ± 0%       ~ (p=1.000 n=10)
GoldenTraceQL/pushdown/attr_route-12                     487.0 ± 0%    487.0 ± 0%       ~ (p=1.000 n=10) ¹
GoldenTraceQL/pushdown/attr_status_code-12               401.0 ± 0%    401.0 ± 0%       ~ (p=1.000 n=10) ¹
GoldenTraceQL/pushdown/attr_and_status-12                363.0 ± 0%    363.0 ± 0%       ~ (p=1.000 n=10) ¹
geomean                                                 13.25k        13.25k       -0.00%
¹ all samples are equal

tdakkota and others added 3 commits August 10, 2026 15:31
@tdakkota @claude
feat(pdataconv): convert exemplars, reject them on decomposed metrics
79cad0d 
AppendMetrics now carries number-point exemplars into the internal batch, so an
OTel-Go embedder's exemplars reach storage instead of being discarded at the
bridge.

Its return becomes a Dropped{Points, Exemplars}: a dropped exemplar only
degrades trace correlation and must not inflate the point count an OTLP
partial-success response reports. Exemplars on histogram and
exponential-histogram points are counted there and dropped — classic
decomposition gives them no unambiguous home series (#252). Summary points
carry no exemplars in OTLP.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@tdakkota @claude
docs(signal): note pdataconv Dropped split and exemplar conversion
8a793ea 
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@tdakkota @claude
feat(tenant): add ExemplarMaxAge, a per-signal retention override
64a51ff 
The age cutoff is now resolved per signal (Retention.AgeFor), so exemplars can
expire ahead of the samples they hang off. They are a sampled debugging aid
whose value ends with the trace they point at, and a trace store is usually
kept far shorter than metrics — left on the shared MaxAge they outlive their
referents.

The byte budget stays tenant-wide: it bounds what the node stores, which is not
a per-signal question.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@tdakkota