Skip to content

feat(api): add an optional stateless odbselect query node group - #15

Draft
tdakkota wants to merge 2 commits into
claude/role-node-groupsfrom
claude/role-query-group
Draft

feat(api): add an optional stateless odbselect query node group#15
tdakkota wants to merge 2 commits into
claude/role-node-groupsfrom
claude/role-query-group

Conversation

@tdakkota

Copy link
Copy Markdown
Contributor

Stacked on #14 — base is claude/role-node-groups, review that first. Closes the query half of #11.

Adds spec.query, the read mirror of #14's spec.ingest: an optional *QuerySpec deploying a pool
of stateless odbselect query nodes so query capacity scales independently of storage.

Grounded in cmd/odbselect/config.go, app.go and api.go on oteldb's claude/1263-odbselect
(oteldb/oteldb#1266), where the config is composed from the shared internal/config blocks.

The structural difference from ingest

odbingest serves OTLP/HTTP, remote write and health on one listener. odbselect does not — it
serves four query APIs on four listeners plus health on a fifth:

block default bind published as
prometheus :9090 prom-http
loki :3100 loki-http
tempo :3200 tempo-http
pyroscope :4040 pyroscope
health :13133 not published — probed

Three decisions follow.

One Service, several ports

<name>-query publishes every enabled API plus 8090 self-metrics. The four APIs are one pod set
behind one selector and each already has its own port, so <name>-query:3100 is exactly as
addressable as a hypothetical <name>-query-loki:3100 would be. A Service per API would add four
DNS names, four objects to prune, and — at type: LoadBalancer — four load balancers, for nothing.
The health listener is deliberately not published: it is what the probes hit, not a client API.

No per-API binds in the CRD; spec.signals is the off switch

QuerySpec exposes no bind fields. The four APIs map 1:1 onto the four signals spec.signals
already toggles, and serving an API for a signal the cluster does not store can only answer empty —
so a second, independent toggle would only let a user express a pool that disagrees with its
cluster. A disabled signal renders bind: "-" (odbselect's own disable convention) and drops the
port from the container and the Service. validateSignals already rejects all-disabled, which lines
up with odbselect's own "every query API is disabled: odbselect would serve nothing".

The "-" is required, not cosmetic: Config.setDefaults runs over every API block
unconditionally, so an omitted block gets its default bind and is served anyway. Omitting is not
disabling. Pinned by TestRenderQueryConfigDisabledSignal.

The binds are reserved from extraConfig — the one place this goes further than #14

spec.ingest.extraConfig reserves only cluster. spec.query.extraConfig reserves cluster and
prometheus.bind, loki.bind, tempo.bind, pyroscope.bind, health.bind.

That is a consequence of the four-listener shape, not a change of mind. For odbingest, a moved bind
breaks the one listener the probes use, so the pod never goes Ready and the mistake is loud. For
odbselect, moving one bind leaves the other four listeners serving, the health listener answering,
and the pod Ready — while the Service port the operator published points at nothing. That is the
same silent-misconfiguration class the ring argument is built on, so it gets the same treatment.

Everything else in a block stays open, which is the point: prometheus.max_samples,
loki.max_sample_rows, per-listener auth, shutdown_timeout are all still reachable.

Carried over from #14

  • Optional pointer stanza. Absent ⇒ no pool, and rendering is byte-identical to today.
  • Deployment, not StatefulSet: holds nothing, not a ring member, no PVC, no stable identity. No HPA
    object — a user's own HPA/KEDA can target apps/v1 Deployment <name>-query already.
  • Ring rendered from the shared ringConfig() with cluster reserved.
    TestIngestRingMatchesStorageRing is generalized to TestRoleRingsMatch, asserting all three
    roles agree key for key on etcd/rf/shards_per_tenant/root. A mismatch does not fail, it
    resolves a different owner set.
  • The storage selector and the existing client Service are untouched (selectors are immutable;
    repointing a live endpoint is a data-path change). TestQuerySelectorIsDisjointFromOtherRoles
    pins that no role's selector matches another role's pods.
  • Dropping spec.query prunes its Deployment, Service and ConfigMap.
  • status.queryReplicas / queryReadyReplicas. The phase still tracks the storage nodes.

Small shared extractions along the way: roleProbe(path, portName), statelessPodEnv (odbingest's
and odbselect's pod env were identical), bindAll(port), and constants for the repeated env-var and
annotation names.

Known caveat, not solved here

odbselect is not in any released image. Its release-packaging commit lives on
claude/1263-odbselect and ships when oteldb/oteldb#1266 merges — the same situation odbingest is
in on #14. The operator side is correct regardless (the container runs /usr/local/bin/odbselect
out of the cluster image, spec.query.image overrides it), but the pool cannot start until oteldb
ships the binary. Noted in the README, the QuerySpec godoc and the sample CR.

Verification

make lint (0 issues), make test (envtest, controller package at 83.1%) and go build ./... all
pass. Generated manifests and deepcopy are in a separate commit.

🤖 Generated with Claude Code

spec.query is the read half of spec.ingest: an optional *QuerySpec that deploys a Deployment of
stateless odbselect pods reading through the ring. Absent means no pool and byte-identical
rendering to today; the storage StatefulSet's selector and the client Service are untouched.

Unlike odbingest's single shared listener, odbselect serves four query APIs on four listeners plus
health on 13133. One <name>-query Service publishes them all — one pod set, one selector, a port
each — and the probes hit the health listener rather than a data port. spec.signals decides which
APIs are served: a disabled signal renders odbselect's bind: "-", which is required because
odbselect defaults any API block it does not find.

The ring comes from the shared ringConfig() helper with the cluster block reserved from
extraConfig, and the listener binds are reserved too: odbselect keeps serving its other listeners
if one moves, so a rebind leaves a published Service port pointing at nothing while the pod stays
Ready. TestRoleRingsMatch now pins that all three roles render the same ring key for key.

odbselect is not in any released image yet; its packaging ships with oteldb/oteldb#1266.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant