Skip to content

fix(precompute_engine): Use value_column from labels at ingest#436

Merged
milindsrivastava1997 merged 4 commits into
mainfrom
432-fix-precompute-value_column-ignored-at-ingest
Jun 25, 2026
Merged

fix(precompute_engine): Use value_column from labels at ingest#436
milindsrivastava1997 merged 4 commits into
mainfrom
432-fix-precompute-value_column-ignored-at-ingest

Conversation

@akanksha-akkihal

Copy link
Copy Markdown
Contributor

Summary

Fixes #432. The streaming config's value_column was ignored at ingest time, so every aggregation consumed the default wire scalar (e.g. pkt_len) regardless of what column it was configured to aggregate. The visible symptom: COUNT(DISTINCT dstip) made the HLL sketch estimate the cardinality of pkt_len instead of dstip, diverging sharply from the ClickHouse baseline (collapsing to ≈1 distinct value per group).

Root cause

The wire format collapses each sample to a single scalar plus a set of labels carried in the series key. At ingest, apply_sample() always fed that wire scalar straight into the accumulator and never consulted config.value_column. For HLL COUNT(DISTINCT dstip), dstip arrives as a label, not as the wire value, so it never reached the sketch.

Changes

  • resolve_sample_value(series_key, wire_val, config) (new, in precompute_engine/worker.rs): if config.value_column names a column present among the series labels, parse that label's value to f64 and aggregate it; otherwise fall back to the wire value.
  • apply_sample(): resolves the value via the helper before updating both keyed and non-keyed accumulators.

Why this is general and safe

  • Numeric value columns like pkt_len are never transmitted as labels, so SUM / quantile / top-k transparently keep using the wire scalar — no behavior change for existing aggregations.
  • Only label-present distinct targets (e.g. dstip) get substituted, which is exactly the broken case.
  • The store is keyed by aggregation_id, so nothing about lookup/routing changes.

Known limitation (follow-up)

Non-numeric distinct targets (e.g. proto) can't be parsed to f64; they log a debug message and fall back to the wire value. A byte/string hashing path for non-numeric COUNT(DISTINCT) is left as a follow-up.

Tests

  • resolve_sample_value_uses_label_when_value_column_present
  • resolve_sample_value_falls_back_when_column_not_a_label
  • resolve_sample_value_none_value_column_uses_wire_value
  • resolve_sample_value_non_numeric_label_falls_back
  • hll_counts_distinct_value_column_not_wire_value — worker integration test: two samples, same srcip, different dstip, identical pkt_len; asserts the closed window's HLL estimate is ≈2 (would be 1 with the bug).

Test plan

  • cargo test -p query_engine_rust resolve_sample_value
  • cargo test -p query_engine_rust hll_counts_distinct
  • E2E: replay netflow, then compare COUNT(DISTINCT dstip) GROUP BY srcip between ASAP (:8088/clickhouse/query) and ClickHouse (:8123) over the same window — ASAP cardinalities should track ClickHouse within HLL error (~1–2% at precision 14) instead of collapsing to ≈1.

@akanksha-akkihal

Copy link
Copy Markdown
Contributor Author

parse series labels once per sample; warn on non-numeric value_column

Follow-up to the value_column ingest fix. Two improvements in apply_sample:

  • Parse the series labels at most once and share the map between value
    resolution and aggregated-key extraction. Previously the keyed path
    parsed labels twice per sample, and the non-keyed path with a
    value_column parsed once even when not needed. Now labels are parsed
    only when the accumulator is keyed or a value_column is set, removing
    redundant allocations from the per-sample ingest hot path.
  • Surface non-numeric value_column targets (e.g. COUNT(DISTINCT proto))
    as a warn-once instead of a per-sample debug log, since falling back to
    the wire value silently produces incorrect results. A process-global
    AtomicBool guard keeps the warning off the hot path.

resolve_sample_value and extract_aggregated_key_from_series now take the
pre-parsed label map; tests updated accordingly.

Comment thread asap-query-engine/src/precompute_engine/worker.rs Outdated
@milindsrivastava1997
milindsrivastava1997 merged commit 5cafc2b into main Jun 25, 2026
10 of 11 checks passed
@milindsrivastava1997
milindsrivastava1997 deleted the 432-fix-precompute-value_column-ignored-at-ingest branch June 25, 2026 02:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(precompute): value_column ignored at ingest - COUNT(DISTINCT col) estimates cardinality of the wrong column

2 participants