feat(api): publish the admin API on an opt-in Service, and make self-metrics serve

README.md

@@ -162,6 +162,7 @@ for a fuller example including the S3 backend.
 | `policy.recompress` | `{after, level}`. Rewrites fully-cold parts with a higher-ratio Zstandard profile. Decode-transparent and lossless. |
 | `policy.ec` | `{data, parity, after}`. Erasure-codes fully-cold parts across `data+parity` nodes instead of RF full copies — `{4,2}` is 1.5x the logical bytes for two tolerated node losses, against 3x for RF=3. See [Erasure coding](#erasure-coding). |
 | `service.type` / `annotations` | Client Service exposing the query/ingest APIs. |
+| `admin` | Optional `<name>-admin` Service publishing oteldb's admin API (`8090`). Absent ⇒ no Service; the API is still served inside every storage pod. See [Admin API](#admin-api). |
 | `resources`, `nodeSelector`, `affinity`, `tolerations`, `topologySpreadConstraints`, `podSecurityContext`, `securityContext`, `podAnnotations`, `podLabels`, `serviceAccountName` | Standard pod scheduling/security knobs. |
 | `extraConfig` | Arbitrary raw oteldb config **deep-merged** over the generated config — for fields the CRD does not model (auth, prometheus tuning, …). Nested objects merge key by key (`storage.policy` does not wipe `storage.backend`); operator-owned paths are [reserved](#reserved-extraconfig-paths). |
 
@@ -278,7 +279,56 @@ Pyroscope and `9464` self-metrics, dropping any whose signal is disabled. `odbse
 its own listener, so these are published one-to-one; its health listener (`13133`) is probed, not
 published.
 
-`8090` is deliberately not exposed: it is oteldb's admin API bind, not a self-metrics endpoint.
+`8090` is oteldb's admin API bind, not a self-metrics endpoint. It is served in every storage pod
+whether or not anything publishes it, and it is deliberately kept off the client Service — set
+`spec.admin` to publish it on a Service of its own. See [Admin API](#admin-api).
+
+### Self-metrics
+
+Every pod exports its own metrics on `9464` in Prometheus format. The operator sets
+`OTEL_METRICS_EXPORTER=prometheus` alongside `OTEL_EXPORTER_PROMETHEUS_HOST/PORT`, because
+go-faster/sdk defaults that variable to `otlp` and only starts the `/metrics` server when it names
+`prometheus` — without it the published port served nothing.
+
+> **Behaviour change.** Self-metrics previously left the pods over OTLP only (to the SDK's default
+> endpoint), and `9464` was published but dead. They are now scraped from `9464` instead. If you
+> were collecting oteldb's self-metrics via OTLP, point a scrape at the `metrics` port on the client,
+> ingest and query Services.
+
+### Admin API
+
+oteldb's admin API (`8090`) reports build info, health, runtime and storage statistics, and it can
+trigger the engine's maintenance and compaction passes and the per-stream storage-cost attribution
+report — documented upstream as *the heaviest call the storage engine exposes*, since it decodes
+every accounted byte column of every live part.
+
+`spec.admin` publishes it on a dedicated `<name>-admin` Service. It is **opt-in and absent by
+default**:
+
+```yaml
+spec:
+  admin:
+    service:
+      type: ClusterIP   # the default
+```
+
+Three notes:
+
+- **It is a separate Service, not a port on the client Service.** An endpoint that can start a
+  compaction should not ride the Service that PromQL and OTLP share, where an ingress or a broad
+  NetworkPolicy picks it up by default. A separate Service also keeps exposure and reachability
+  independent: admin can stay `ClusterIP` while the client Service is a `LoadBalancer`, and it
+  carries its own annotations.
+- **Enabling it does not roll the pods.** oteldb registers the admin server unconditionally
+  (`cmd/oteldb/admin.go`), and its bind defaults to `:8090` whether or not the config declares the
+  block — so the listener already exists. The Service targets the port by number and the pod
+  template is untouched.
+- **It has no auth of its own.** Treat the Service as privileged and restrict it with a
+  NetworkPolicy, or add oteldb's auth via `spec.extraConfig`.
+
+Before this, the admin port was *accidentally* reachable: it was published under a port named
+`metrics`, and moving self-metrics to `9464` removed that exposure without anyone deciding it
+should go. Neither state was a decision; `spec.admin` is.
 
 ## Getting Started
 
@@ -318,4 +368,5 @@ Controller layout under `internal/controller/`:
 - `resources.go` — builds the ConfigMap, headless + client Services, and StatefulSet.
 - `ingest.go` / `query.go` — the stateless `odbingest` and `odbselect` pools: config, Service,
   Deployment.
+- `admin.go` — the opt-in admin API Service.
 - `naming.go` — names, labels, ports.

api/v1alpha1/oteldbcluster_types.go

@@ -113,6 +113,12 @@ type OtelDBClusterSpec struct {
 	// +optional
 	Service ServiceSpec `json:"service,omitempty"`
 
+	// Admin optionally publishes oteldb's admin API on a Service of its own. Absent — the default —
+	// publishes nothing: the API is still served inside every storage pod, but no Service routes to
+	// it.
+	// +optional
+	Admin *AdminSpec `json:"admin,omitempty"`
+
 	// Resources are the compute resources for each oteldb container.
 	// +optional
 	Resources corev1.ResourceRequirements `json:"resources,omitempty"`
@@ -778,6 +784,33 @@ type LimitsSpec struct {
 	MaxPartSize *resource.Quantity `json:"maxPartSize,omitempty"`
 }
 
+// AdminSpec publishes oteldb's admin API on a dedicated <name>-admin Service.
+//
+// The API is not optional inside the pod: oteldb registers it unconditionally and its bind defaults
+// to :8090, so every storage node is already serving it. What was missing is a way to reach it —
+// before the self-metrics port moved to 9464 it happened to be published under a port named
+// "metrics", and moving that port took the accidental exposure with it. Neither state was a
+// decision; this field is.
+//
+// It is a separate Service, and opt-in, because of what the API can do: it triggers the engine's
+// maintenance and compaction passes, and serves the stream-cost attribution report — documented
+// upstream as the heaviest call the storage library exposes, decoding every accounted byte column
+// of every live part. None of that belongs on the client Service that PromQL and OTLP share, where
+// an ingress or a broad NetworkPolicy would pick it up by default.
+//
+// A separate Service also keeps the exposure decision separate from the reachability decision: the
+// admin Service can stay ClusterIP while the client Service is a LoadBalancer, and it can carry its
+// own annotations (auth proxy, internal-only load balancer) without touching client traffic.
+//
+// oteldb has no auth on the admin API beyond the global spec.extraConfig auth block, so treat this
+// Service as privileged and restrict it with a NetworkPolicy.
+type AdminSpec struct {
+	// Service configures the <name>-admin Service. Its default type is ClusterIP, which is the
+	// intended shape: the admin API is an operator tool, not a client-facing endpoint.
+	// +optional
+	Service ServiceSpec `json:"service,omitempty"`
+}
+
 // ServiceSpec configures the client-facing Service.
 type ServiceSpec struct {
 	// Type of the client Service.

config/crd/bases/db.oteldb.io_oteldbclusters.yaml

@@ -57,6 +57,32 @@ spec:
           spec:
             description: spec defines the desired state of OtelDBCluster
             properties:
+              admin:
+                description: |-
+                  Admin optionally publishes oteldb's admin API on a Service of its own. Absent — the default —
+                  publishes nothing: the API is still served inside every storage pod, but no Service routes to
+                  it.
+                properties:
+                  service:
+                    description: |-
+                      Service configures the <name>-admin Service. Its default type is ClusterIP, which is the
+                      intended shape: the admin API is an operator tool, not a client-facing endpoint.
+                    properties:
+                      annotations:
+                        additionalProperties:
+                          type: string
+                        description: Annotations added to the client Service.
+                        type: object
+                      type:
+                        default: ClusterIP
+                        description: Type of the client Service.
+                        enum:
+                        - ClusterIP
+                        - NodePort
+                        - LoadBalancer
+                        type: string
+                    type: object
+                type: object
               affinity:
                 description: |-
                   Affinity for oteldb pods. When empty, the operator applies a soft anti-affinity that

internal/controller/admin.go

@@ -0,0 +1,65 @@
+/*
+Copyright 2026.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package controller
+
+import (
+	corev1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/util/intstr"
+
+	dbv1alpha1 "github.com/oteldb/operator/api/v1alpha1"
+)
+
+func adminEnabled(cr *dbv1alpha1.OtelDBCluster) bool { return cr.Spec.Admin != nil }
+
+// buildAdminService publishes oteldb's admin API, which every storage pod already serves on
+// portAdmin, on a Service of its own.
+//
+// It targets the port by number rather than by name on purpose. oteldb registers the admin server
+// unconditionally (cmd/oteldb/admin.go: "It is always registered", and its bind block defaults to
+// :8090 whether or not the config declares it), so the listener exists regardless of this Service.
+// Declaring a matching container port would therefore document nothing the pod does not already do,
+// while changing the pod template — which would roll the whole StatefulSet on a change that only
+// adds a Service. Toggling spec.admin stays a Service-only operation.
+func buildAdminService(cr *dbv1alpha1.OtelDBCluster) *corev1.Service {
+	n := namesFor(cr)
+	svcType := cr.Spec.Admin.Service.Type
+	if svcType == "" {
+		svcType = corev1.ServiceTypeClusterIP
+	}
+
+	return &corev1.Service{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      n.adminService(),
+			Namespace: cr.Namespace,
+			// The component label separates this from the client Service, so a ServiceMonitor or a
+			// NetworkPolicy can select the admin endpoint without matching client traffic.
+			Labels:      roleCommonLabels(cr, appName, valAdmin),
+			Annotations: cr.Spec.Admin.Service.Annotations,
+		},
+		Spec: corev1.ServiceSpec{
+			Type:     svcType,
+			Selector: selectorLabels(cr),
+			Ports: []corev1.ServicePort{{
+				Name:       portNameAdmin,
+				Port:       portAdmin,
+				TargetPort: intstr.FromInt32(portAdmin),
+				Protocol:   corev1.ProtocolTCP,
+			}},
+		},
+	}
+}

internal/controller/admin_test.go

@@ -0,0 +1,121 @@
+/*
+Copyright 2026.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package controller
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/require"
+	corev1 "k8s.io/api/core/v1"
+
+	dbv1alpha1 "github.com/oteldb/operator/api/v1alpha1"
+)
+
+func TestAdminDisabledByDefault(t *testing.T) {
+	require.False(t, adminEnabled(testCluster()),
+		"the admin API must not be published unless spec.admin asks for it")
+}
+
+// The admin port must never reach the client Service: that is where PromQL and OTLP traffic lands,
+// and the admin API can trigger maintenance and the stream-cost scan.
+func TestClientServiceExcludesAdminPort(t *testing.T) {
+	for _, svc := range []*corev1.Service{
+		buildClientService(testCluster()),
+		buildPeerService(testCluster()),
+	} {
+		for _, p := range svc.Spec.Ports {
+			require.NotEqual(t, int32(portAdmin), p.Port, "%s must not publish the admin API", svc.Name)
+			require.NotEqual(t, portNameAdmin, p.Name, "%s must not publish the admin API", svc.Name)
+		}
+	}
+}
+
+func TestBuildAdminService(t *testing.T) {
+	cr := testCluster()
+	cr.Spec.Admin = &dbv1alpha1.AdminSpec{}
+
+	svc := buildAdminService(cr)
+	require.Equal(t, "obs-admin", svc.Name)
+	require.Equal(t, "monitoring", svc.Namespace)
+	require.Equal(t, corev1.ServiceTypeClusterIP, svc.Spec.Type,
+		"the admin API is an operator tool, so it must default to a cluster-internal Service")
+	require.Equal(t, selectorLabels(cr), svc.Spec.Selector,
+		"the admin API is served by the storage pods")
+	require.Equal(t, "admin", svc.Labels["app.kubernetes.io/component"],
+		"the component label must separate the admin endpoint from client traffic")
+
+	require.Len(t, svc.Spec.Ports, 1)
+	port := svc.Spec.Ports[0]
+	require.Equal(t, portNameAdmin, port.Name)
+	require.EqualValues(t, portAdmin, port.Port)
+	require.EqualValues(t, portAdmin, port.TargetPort.IntVal,
+		"the admin port is targeted by number: oteldb always serves it, so declaring a container "+
+			"port would roll the StatefulSet for no behaviour change")
+}
+
+func TestBuildAdminServiceOverrides(t *testing.T) {
+	cr := testCluster()
+	cr.Spec.Admin = &dbv1alpha1.AdminSpec{
+		Service: dbv1alpha1.ServiceSpec{
+			Type:        corev1.ServiceTypeNodePort,
+			Annotations: map[string]string{"a": "b"},
+		},
+	}
+
+	svc := buildAdminService(cr)
+	require.Equal(t, corev1.ServiceTypeNodePort, svc.Spec.Type)
+	require.Equal(t, map[string]string{"a": "b"}, svc.Annotations)
+}
+
+// Enabling spec.admin must not touch the pod template: a Service is all that changes.
+func TestAdminDoesNotRollPods(t *testing.T) {
+	off := testCluster()
+	on := testCluster()
+	on.Spec.Admin = &dbv1alpha1.AdminSpec{}
+
+	require.Equal(t,
+		buildStatefulSet(off, "hash").Spec.Template,
+		buildStatefulSet(on, "hash").Spec.Template,
+	)
+
+	cfgOff, err := renderConfig(off, off.Spec.Etcd.Endpoints)
+	require.NoError(t, err)
+	cfgOn, err := renderConfig(on, on.Spec.Etcd.Endpoints)
+	require.NoError(t, err)
+	require.Equal(t, cfgOff, cfgOn, "spec.admin renders no config: oteldb serves the API either way")
+}
+
+// 9464 is published on all three Services, so the exporter that serves it has to be selected:
+// go-faster/sdk defaults OTEL_METRICS_EXPORTER to "otlp" and never starts the /metrics server.
+func TestSelfMetricsExporterIsPrometheus(t *testing.T) {
+	cr := testCluster()
+	cr.Spec.Ingest = &dbv1alpha1.IngestSpec{}
+	cr.Spec.Query = &dbv1alpha1.QuerySpec{}
+
+	for name, env := range map[string][]corev1.EnvVar{
+		"storage":  podEnv(cr),
+		"stateles": statelessPodEnv(cr),
+	} {
+		got := map[string]string{}
+		for _, e := range env {
+			got[e.Name] = e.Value
+		}
+		require.Equal(t, "prometheus", got[envMetricsExporter], "%s pods", name)
+		require.Equal(t, bindAllHost, got[envPrometheusHost], "%s pods", name)
+		require.Equal(t, "9464", got[envPrometheusPort], "%s pods", name)
+	}
+}