Skip to content

feat(api): publish the admin API on an opt-in Service, and make self-metrics serve - #21

Draft
tdakkota wants to merge 2 commits into
claude/ingest-tenantfrom
claude/admin-service
Draft

feat(api): publish the admin API on an opt-in Service, and make self-metrics serve#21
tdakkota wants to merge 2 commits into
claude/ingest-tenantfrom
claude/admin-service

Conversation

@tdakkota

Copy link
Copy Markdown
Contributor

Closes two related gaps left by #12 and #13.

1. The admin API is served but unreachable

oteldb registers the admin API unconditionally (cmd/oteldb/admin.go: "It is always
registered"
) and config.Admin.SetDefaults binds :8090 whether or not the config declares the
block. So every storage pod is already serving it — build info, health, runtime and storage
statistics, MaintainNow / CompactNow, and the per-stream cost report — and no Service routed to
it.

Before #13 it was reachable by accident: it was published under a port named metrics. #13 moved
self-metrics to 9464 and took the accidental exposure with it. It went from reachable-by-accident to
unreachable, and neither state was a decision.

spec.admin — a separate <name>-admin Service, absent by default.

spec:
  admin:
    service:
      type: ClusterIP   # the default

Why separate and opt-in rather than a port on the client Service:

  • runAction triggers the engine's maintenance and compaction passes, and getStreamCosts is
    documented upstream as the heaviest call the storage engine exposes — it decodes every
    accounted byte column of every live part. That must not ride the Service PromQL and OTLP share,
    where an ingress or a broad NetworkPolicy picks it up by default.
  • Exposure and reachability stay independent: admin can be ClusterIP while the client Service is a
    LoadBalancer, with its own annotations (auth proxy, internal-only LB).
  • A distinct app.kubernetes.io/component: admin label lets a NetworkPolicy or ServiceMonitor select
    it without matching client traffic.

Toggling it does not roll the StatefulSet. The Service targets 8090 by number rather than
declaring a matching container port: the listener exists regardless, so a container port would
document nothing new while changing the pod template. TestAdminDoesNotRollPods pins that, and that
spec.admin renders no config at all.

The API has no auth of its own beyond spec.extraConfig's auth block — the README says to treat
the Service as privileged and restrict it with a NetworkPolicy.

2. Port 9464 was published and served nothing

#12 established that OTEL_EXPORTER_PROMETHEUS_HOST/PORT are set but OTEL_METRICS_EXPORTER never
is. go-faster/sdk's autometer defaults that variable to otlp (getEnvOr(envName, expOTLP)) and
only constructs the Prometheus reader — and with it the /metrics server — when it names
prometheus. So the port was published on all three Services and dead.

This sets OTEL_METRICS_EXPORTER=prometheus on storage, ingest and query pods.

Behaviour change. Self-metrics previously left the pods over OTLP only, to the SDK's default
endpoint. They are now scraped from 9464 instead. Anyone collecting oteldb's self-metrics over
OTLP needs to point a scrape at the metrics port. Called out in the README under Self-metrics.

Unlike spec.admin this one is unconditional and does roll the pods — it is a fix, not a stanza:
the current state is a published port that cannot answer.

Stack

Based on #20 (spec.ingest.tenant) → #19 (spec.policy.ec) → #17 (spec.cluster.privateBackend) →
#15 (spec.query) → #14 (spec.ingest). This PR depends on #14/#15 only for statelessPodEnv,
which is where the exporter variable lands for the two stateless pools; the rest is stack ordering.

This is the tip of the stack — e2e runs here.

…-metrics

The admin API is served in every storage pod (bind defaults to :8090, and the block
defaults unconditionally) but nothing routed to it. Before #13 it was reachable by
accident, under a port named "metrics"; moving self-metrics to 9464 took that with it.
spec.admin now publishes it on a dedicated <name>-admin Service, absent by default: the
API triggers MaintainNow/CompactNow and StreamCosts, so it must not ride the client
Service. The Service targets 8090 by number, leaving the pod template untouched.

Also sets OTEL_METRICS_EXPORTER=prometheus. go-faster/sdk defaults it to otlp and only
starts the /metrics server when it names prometheus, so 9464 was published and dead. This
is a behaviour change: self-metrics previously left over OTLP only.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant