feat(msgpack): add ASAPv1 self-describing envelope to wire format

[GordonYuanyc]

Overview

Adds the ASAPv1 envelope to every serialized sketch binary, making the wire
format self-describing and carrying the hash metadata needed by out-of-process
consumers.

[ b"ASAPv1" | version:u8 | kind_id_len:u8 | kind_id:bytes | metadata_len:u32_be | metadata:msgpack | msgpack_payload ]

• b"ASAPv1" — 6-byte ASCII sentinel, encoded as 0x41 0x53 0x41 0x50 0x76 0x31
• version — envelope layout version; currently 0x02
• kind_id_len + kind_id — variable-length sketch discriminant
• metadata_len + metadata — compact MessagePack metadata block

The sketch payload structure is unchanged; metadata is wrapper-level.

Metadata

Current metadata is a compact MessagePack array in this field order:

[
  metadata_version,
  hash_spec_present,
  hash_profile_id,
  hash_algorithm,
  seed_list,
  canonical_seed_index,
  matrix_seed_index,
  hydra_seed_index,
  univmon_bottom_layer_seed_index,
  seed_derivation,
  input_encoding
]

For the standard ProjectASAP hash profile:

• metadata_version = 1
• hash_profile_id = "projectasap.xxh3.seedlist.v1"
• hash_algorithm = "xxh3_64_128"
• seed_list = [0xcafe3553, 0xade3415118, 0x8cc70208, 0x2f024b2b, 0x451a3df5, 0x6a09e667, 0xbb67ae85, 0x3c6ef372, 0xa54ff53a, 0x510e527f, 0x9b05688c, 0x1f83d9ab, 0x5be0cd19, 0xcbbb9d5d, 0x629a292a, 0x9159015a, 0x152fecd8, 0x67332667, 0x8eb44a87, 0xdb0c2e0d]
• canonical_seed_index = 5
• matrix_seed_index = 0
• hydra_seed_index = 6
• univmon_bottom_layer_seed_index = 19
• seed_derivation = "seed_list_index_wrap"
• input_encoding = "projectasap.input.v1"

Decoders validate both the metadata contents and that the metadata matches the
decoded kind_id.

Breaking change

Bytes produced before this PR (bare msgpack, or earlier draft ASAPv1 wrapper
without metadata) will be rejected. No backward-compatibility shim; callers must
re-serialize stored blobs.

Verification

• cargo test msgpack
• cargo test

Go mirror

sketchlib-go PR #68 is updated in lock-step.

[milindsrivastava1997]

One thought - a 2 digit hex magic ID restricts us to 256 IDs. While this seems reasonable for the short term, it's not out of the question that the number of sketches may go beyond that. We can make it 3 digits, or if we want to ensure complete future proofness (at the cost of some complexity), the magic ID can be prepended by a number that denotes the number of digits in the magic ID. Are there any references in other frameworks as to how this is handled?

[GordonYuanyc]

I agree that a single-byte sketch ID gives us an unnecessary 256-ID ceiling.

Prometheus and OTAP both use lightweight metadata before the raw binary payload so the receiver can interpret the following bytes from the bytes themselves. Prometheus stores a chunk encoding before each chunk payload; OTAP wraps raw Arrow IPC bytes with schema_id/type metadata.

For us, a thin wrapper around the existing MessagePack payload should be enough:

[ "ASK1" | version:u8 | kind_id_len:u8 | kind_id:bytes | msgpack_payload ]

"ASK1" identifies this as an ASAP sketch binary. version is for future wrapper-layout changes. kind_id_len + kind_id avoids the 256-ID limit while keeping small IDs compact. kind_id should be canonical unsigned big-endian with no leading zero bytes.

[GordonYuanyc]

With this wrapper, the serialized bytes are self-describing enough for the deserializer to determine which sketch decoder to use.

[milindsrivastava1997]

Sounds good. This encoding scheme makes sense.

[milindsrivastava1997]

This might also need changes to https://github.com/ProjectASAP/asap_sketchlib/blob/main/proto/sketchlib.proto?

[GordonYuanyc]

Protobuf is intentionally left untouched. protobuf's oneof already handles type discrimination the same way ASK1 does for msgpack, so there's nothing to add.
This PR pairs with ProjectASAP/sketchlib-go#68 which applies the same ASK1 change on the Go side.

[milindsrivastava1997]

Is there a description of what the sketch payload is, for each sketch family?

[GordonYuanyc]

Is there a description of what the sketch payload is, for each sketch family?

Not yet. The payload that can work for cross-language ser/de needs modification.

On the other hand, something that works currently is under the portable subdirectory.

[GordonYuanyc]

The PR is marked as draft, instead of close for one reason: it's easier to find reference for possible metadata and envelope design in PR description.

That means, for self-describing, cross-language ser/de, this PR will be broken into several small PRs, where each PR will contain one sketch algorithm.

Portable and payload will be redesigned and will gradually be unified into one place.

[milindsrivastava1997]

Sure. I would say though, it is better to put the envelope and metadata description in a design doc markdown file and add that to the repo. Similary, for the serialization formats for each sketch family (whether it supports cross-language serde or not)