Skip to content

test(roles): exercise the disaggregated ingest and query pools - #2

Draft
tdakkota wants to merge 1 commit into
claude/private-backend-durabilityfrom
claude/role-pools
Draft

test(roles): exercise the disaggregated ingest and query pools#2
tdakkota wants to merge 1 commit into
claude/private-backend-durabilityfrom
claude/role-pools

Conversation

@tdakkota

Copy link
Copy Markdown

Stacked on #1 — review that first.

Adds the disaggregated topology to the suite: the same three storage nodes, plus a stateless
odbingest pool (spec.ingest) and a stateless odbselect pool (spec.query), each a Deployment
behind its own Service, each scaling independently.

roles_test.go writes through the ingest pool's Service and reads back through the query
pool's, for all three signals. That crossing is the point of the split, and it is the one thing a
symmetric suite structurally cannot check: when one process does both, a write-then-read proves
nothing about the boundary between them. It also asserts the pools stay out of the storage ring —
a pool pod that joined it would own shards whose data it does not have — and that losing a pod from
each pool costs capacity and nothing else.

Depends on (all unmerged — draft until they land)

Changes

  • Ref-pinned sibling checkouts. OTELDB_REF / OPERATOR_REF (with OTELDB_REPO /
    OPERATOR_REPO and E2E_SRC_CACHE) make the suite clone a sibling repo at a git ref and build
    that, instead of the local working tree. Empty ref keeps today's behaviour, so the fast local loop
    is untouched; a set ref lets this run against unmerged branches now and main after, without
    editing anyone's checkout. The clone is blobless and cached between runs.
  • Extra build targets. ./cmd/odbingest and ./cmd/odbselect go through the same fast path as
    ./cmd/oteldb — host go build, single-COPY image, kind load. They need separate images
    (separate binaries), which is also why "odbselect ships in no released image" is a non-issue here.
  • manifests/role-pools.yaml — a merge patch rather than a second cluster manifest, so the
    storage half of the topology under test stays defined in exactly one place. Two replicas per pool:
    the smallest count that still puts a Service in front of more than one pod.
  • E2E_ROLE_POOLS gates the whole thing, default off, so make test on main is unchanged
    while the dependencies are unmerged. make test-roles runs it with the branch refs prefilled.

Ports, as verified against the PRs

The ingest pool folds OTLP/HTTP, remote write and health onto one in-pod listener (:19291), and
its Service republishes that as 4318 as well, so an OTLP/HTTP client needs no special-casing; gRPC
keeps its own listener on 4317. The query pool serves each read API on its own listener and
publishes each enabled one (9090 / 3100 / 3200 / 4040); health (13133) is a container port only, so
the specs do not reach for it. One correction to the ports I was given: the pools' self-metrics port
is 9464, not 8090 — 8090 is oteldb's own admin API bind (oteldb/operator#13), which is what
internal/nodeadmin from #1 talks to.

Three pre-existing bugs the pools exposed

  • The fast image installed each binary only at /<bin>. The storage pods work off the ENTRYPOINT
    and the manager Deployment asks for /manager, so nothing had noticed; the pools' Deployments
    exec the release path /usr/local/bin/<bin> and died with "no such file or directory" before
    logging anything. The image now installs both.
  • WaitRollout raced the operator: it ran kubectl rollout status right after applying the CR, and
    that command fails NotFound rather than waiting for the object to appear. It usually won the race.
    It now waits for existence first.
  • make deploy leaves a reused cluster running the previous manager binary — the image tag never
    changes, so the pod template is identical and nothing restarts even though kind load replaced
    the image underneath. Harmless on a fresh cluster, silently wrong on the fast loop, and actively
    misleading now that OPERATOR_REF can switch which operator is being built. The manager is now
    restarted onto the just-loaded image.

Testing

Run on kind: 11/11 specs pass with E2E_ROLE_POOLS=1, OTELDB_REF=claude/1263-odbselect and an
operator built from claude/role-query-group with oteldb/operator#16's privateBackend field
applied on top (the branch does not carry it, and #1's durability spec requires it). Both pools come
up, a trace/log/metric written to oteldb-ingest reads back through oteldb-query, the ring stays
at three members, and killing a pod from each pool recovers. Default mode (pools off, what make test does on main) is 7/7 with the 4 role specs skipped.

Deploy an odbingest pool (spec.ingest) and an odbselect pool (spec.query) alongside the storage
nodes, then write through the ingest pool's Service and read back through the query pool's for all
three signals. That crossing is the point of the split and the one thing a symmetric suite cannot
check: with one process doing both, a write-then-read proves nothing about the boundary. Also
asserts the pools stay out of the storage ring and that losing a pod from each costs only capacity.

Sibling repos can now be pinned to a git ref (OTELDB_REF / OPERATOR_REF), cloned into a cache
instead of the local tree, so this runs against the unmerged branches now and main later. odbingest
and odbselect go through the same fast build path as oteldb.

Three fixes the pools exposed, all pre-existing: the fast image installed binaries only at /<bin>,
but the pools' Deployments exec the release path /usr/local/bin/<bin>; WaitRollout raced the
operator creating the workload, since `kubectl rollout status` fails NotFound rather than waiting;
and `make deploy` leaves a reused cluster running the previous manager, because the image tag never
changes. Gated behind E2E_ROLE_POOLS until oteldb/operator#14, #15 and oteldb/oteldb#1266 land.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant