fix(precompute_engine): Use value_column from labels at ingest#436
Merged
milindsrivastava1997 merged 4 commits intoJun 25, 2026
Merged
Conversation
Contributor
Author
|
parse series labels once per sample; warn on non-numeric value_column Follow-up to the value_column ingest fix. Two improvements in
|
milindsrivastava1997
requested changes
Jun 24, 2026
milindsrivastava1997
approved these changes
Jun 24, 2026
milindsrivastava1997
deleted the
432-fix-precompute-value_column-ignored-at-ingest
branch
June 25, 2026 02:35
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #432. The streaming config's
value_columnwas ignored at ingest time, so every aggregation consumed the default wire scalar (e.g.pkt_len) regardless of what column it was configured to aggregate. The visible symptom:COUNT(DISTINCT dstip)made the HLL sketch estimate the cardinality ofpkt_leninstead ofdstip, diverging sharply from the ClickHouse baseline (collapsing to ≈1 distinct value per group).Root cause
The wire format collapses each sample to a single scalar plus a set of labels carried in the series key. At ingest,
apply_sample()always fed that wire scalar straight into the accumulator and never consultedconfig.value_column. For HLLCOUNT(DISTINCT dstip),dstiparrives as a label, not as the wire value, so it never reached the sketch.Changes
resolve_sample_value(series_key, wire_val, config)(new, inprecompute_engine/worker.rs): ifconfig.value_columnnames a column present among the series labels, parse that label's value tof64and aggregate it; otherwise fall back to the wire value.apply_sample(): resolves the value via the helper before updating both keyed and non-keyed accumulators.Why this is general and safe
pkt_lenare never transmitted as labels, so SUM / quantile / top-k transparently keep using the wire scalar — no behavior change for existing aggregations.dstip) get substituted, which is exactly the broken case.aggregation_id, so nothing about lookup/routing changes.Known limitation (follow-up)
Non-numeric distinct targets (e.g.
proto) can't be parsed tof64; they log a debug message and fall back to the wire value. A byte/string hashing path for non-numericCOUNT(DISTINCT)is left as a follow-up.Tests
resolve_sample_value_uses_label_when_value_column_presentresolve_sample_value_falls_back_when_column_not_a_labelresolve_sample_value_none_value_column_uses_wire_valueresolve_sample_value_non_numeric_label_falls_backhll_counts_distinct_value_column_not_wire_value— worker integration test: two samples, samesrcip, differentdstip, identicalpkt_len; asserts the closed window's HLL estimate is ≈2 (would be 1 with the bug).Test plan
cargo test -p query_engine_rust resolve_sample_valuecargo test -p query_engine_rust hll_counts_distinctCOUNT(DISTINCT dstip) GROUP BY srcipbetween ASAP (:8088/clickhouse/query) and ClickHouse (:8123) over the same window — ASAP cardinalities should track ClickHouse within HLL error (~1–2% at precision 14) instead of collapsing to ≈1.