Skip to content

Estimate mode: CMS/CountSketch top-k output needs a heavy-hitter tracker #2

Description

@zzylol

Context

The estimate output mode (transmit_sketch = false, added in the msgpack-wire-format work) emits typed metric rows instead of sketch bytes:

  • DDSketch / KLL → one Gauge per configured quantile (quantile label)
  • HLL → one cardinality Gauge

CMS and CountSketch emit nothing — their FrequencySketch::top_k() returns an empty slice (src/sketches/cms.rs, src/sketches/countsketch.rs).

Why

CMS / CountSketch are hash-counter matrices: they answer "estimate the count for a given key" but do not record which keys were seen, so they cannot enumerate the top-K highest-frequency keys.

What's needed

Attach a heavy-hitter tracker (Space-Saving / a Top-K heap) that maintains candidate hot keys at insert time, then have top_k(k) read it. asap_sketchlib already has building blocks (e.g. CountL2HH, and the hh_keys channel on the delta types) that could back this.

Until then, estimate mode is quantile + cardinality only; frequency sketches stay on the empty default (documented in Sketch::estimate).

Scope

  • asap-precompute-rs: wire a tracker into CMSWrapper / CountSketchWrapper, implement top_k, map to EstimatePoint { labels: [("key", …)], value } (see the docs/df-processor-topology.md "OTel convention" row).
  • Possibly a small asap_sketchlib addition for the tracker.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions