test(roles): exercise the disaggregated ingest and query pools - #2
Draft
tdakkota wants to merge 1 commit into
Draft
test(roles): exercise the disaggregated ingest and query pools#2tdakkota wants to merge 1 commit into
tdakkota wants to merge 1 commit into
Conversation
Deploy an odbingest pool (spec.ingest) and an odbselect pool (spec.query) alongside the storage nodes, then write through the ingest pool's Service and read back through the query pool's for all three signals. That crossing is the point of the split and the one thing a symmetric suite cannot check: with one process doing both, a write-then-read proves nothing about the boundary. Also asserts the pools stay out of the storage ring and that losing a pod from each costs only capacity. Sibling repos can now be pinned to a git ref (OTELDB_REF / OPERATOR_REF), cloned into a cache instead of the local tree, so this runs against the unmerged branches now and main later. odbingest and odbselect go through the same fast build path as oteldb. Three fixes the pools exposed, all pre-existing: the fast image installed binaries only at /<bin>, but the pools' Deployments exec the release path /usr/local/bin/<bin>; WaitRollout raced the operator creating the workload, since `kubectl rollout status` fails NotFound rather than waiting; and `make deploy` leaves a reused cluster running the previous manager, because the image tag never changes. Gated behind E2E_ROLE_POOLS until oteldb/operator#14, #15 and oteldb/oteldb#1266 land.
tdakkota
force-pushed
the
claude/role-pools
branch
from
August 17, 2026 23:02
1c5b2fe to
38fe309
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1 — review that first.
Adds the disaggregated topology to the suite: the same three storage nodes, plus a stateless
odbingestpool (spec.ingest) and a statelessodbselectpool (spec.query), each a Deploymentbehind its own Service, each scaling independently.
roles_test.gowrites through the ingest pool's Service and reads back through the querypool's, for all three signals. That crossing is the point of the split, and it is the one thing a
symmetric suite structurally cannot check: when one process does both, a write-then-read proves
nothing about the boundary between them. It also asserts the pools stay out of the storage ring —
a pool pod that joined it would own shards whose data it does not have — and that losing a pod from
each pool costs capacity and nothing else.
Depends on (all unmerged — draft until they land)
claude/role-node-groups—spec.ingestclaude/role-query-group—spec.query, based on #14claude/1263-odbselect— thecmd/odbselectbinaryspec.cluster.privateBackend, which test(durability): assert shared-nothing part replication and disk-loss recovery #1 needsChanges
OTELDB_REF/OPERATOR_REF(withOTELDB_REPO/OPERATOR_REPOandE2E_SRC_CACHE) make the suite clone a sibling repo at a git ref and buildthat, instead of the local working tree. Empty ref keeps today's behaviour, so the fast local loop
is untouched; a set ref lets this run against unmerged branches now and
mainafter, withoutediting anyone's checkout. The clone is blobless and cached between runs.
./cmd/odbingestand./cmd/odbselectgo through the same fast path as./cmd/oteldb— hostgo build, single-COPY image,kind load. They need separate images(separate binaries), which is also why "odbselect ships in no released image" is a non-issue here.
manifests/role-pools.yaml— a merge patch rather than a second cluster manifest, so thestorage half of the topology under test stays defined in exactly one place. Two replicas per pool:
the smallest count that still puts a Service in front of more than one pod.
E2E_ROLE_POOLSgates the whole thing, default off, somake testonmainis unchangedwhile the dependencies are unmerged.
make test-rolesruns it with the branch refs prefilled.Ports, as verified against the PRs
The ingest pool folds OTLP/HTTP, remote write and health onto one in-pod listener (
:19291), andits Service republishes that as 4318 as well, so an OTLP/HTTP client needs no special-casing; gRPC
keeps its own listener on 4317. The query pool serves each read API on its own listener and
publishes each enabled one (9090 / 3100 / 3200 / 4040); health (13133) is a container port only, so
the specs do not reach for it. One correction to the ports I was given: the pools' self-metrics port
is 9464, not 8090 — 8090 is oteldb's own admin API bind (oteldb/operator#13), which is what
internal/nodeadminfrom #1 talks to.Three pre-existing bugs the pools exposed
/<bin>. The storage pods work off the ENTRYPOINTand the manager Deployment asks for
/manager, so nothing had noticed; the pools' Deploymentsexec the release path
/usr/local/bin/<bin>and died with "no such file or directory" beforelogging anything. The image now installs both.
WaitRolloutraced the operator: it rankubectl rollout statusright after applying the CR, andthat command fails NotFound rather than waiting for the object to appear. It usually won the race.
It now waits for existence first.
make deployleaves a reused cluster running the previous manager binary — the image tag neverchanges, so the pod template is identical and nothing restarts even though
kind loadreplacedthe image underneath. Harmless on a fresh cluster, silently wrong on the fast loop, and actively
misleading now that
OPERATOR_REFcan switch which operator is being built. The manager is nowrestarted onto the just-loaded image.
Testing
Run on kind: 11/11 specs pass with
E2E_ROLE_POOLS=1,OTELDB_REF=claude/1263-odbselectand anoperator built from
claude/role-query-groupwith oteldb/operator#16'sprivateBackendfieldapplied on top (the branch does not carry it, and #1's durability spec requires it). Both pools come
up, a trace/log/metric written to
oteldb-ingestreads back throughoteldb-query, the ring staysat three members, and killing a pod from each pool recovers. Default mode (pools off, what
make testdoes on main) is 7/7 with the 4 role specs skipped.