Skip to content

feat: rework metrics#1872

Merged
gianlucam76 merged 1 commit into
projectsveltos:mainfrom
gianlucam76:metrics
Jul 14, 2026
Merged

feat: rework metrics#1872
gianlucam76 merged 1 commit into
projectsveltos:mainfrom
gianlucam76:metrics

Conversation

@gianlucam76

Copy link
Copy Markdown
Member

Replace the per-cluster metric-name histograms (cluster identity baked into the Prometheus metric name via Namespace, producing one never-unregistered metric per cluster) and the two unlabeled global histograms with a single properly labeled reconcile_duration_seconds histogram. Add outcome and fleet-adoption visibility that didn't exist before, and ship the RBAC that was missing for anyone actually trying to scrape /metrics.

Metrics (all projectsveltos_ prefixed):

  • reconcile_duration_seconds (histogram): cluster_type, cluster_namespace, cluster_name, feature. Buckets extended to 600s to catch slow Helm installs that used to fall into +Inf with no visibility.
  • reconcile_operations_total (counter): cluster_type, cluster_namespace, cluster_name, feature. Counts every reconcile attempt.
  • reconcile_outcome_total (counter): cluster_type, cluster_namespace, cluster_name, feature, status, profile_kind, profile_namespace, profile_name. Terminal success/failure only, answers "is add-on X failing on N clusters."
  • matching_clusters (gauge): profile_kind, profile_namespace, profile_name. Clusters currently matching a ClusterProfile/Profile selector.
  • reconcile_consecutive_failures (gauge): cluster_type, cluster_namespace, cluster_name, feature, profile_kind, profile_namespace, profile_name. Mirrors ClusterSummary.status.featureSummaries[].consecutiveFailures.
  • reconcile_last_success_timestamp_seconds (gauge): same labels as above. Unix timestamp of last Provisioned/Removed outcome.
  • total_drifts (counter): cluster_type, cluster_namespace, cluster_name, feature, profile_kind, profile_namespace, profile_name. Existing metric, gained the profile_* labels.

Also:

  • Fix metric name prefix:
  • Enable auth-proxy RBAC resources (metrics-reader ClusterRole, Service, proxy-role/binding) in config/rbac/kustomization.yaml. Previously /metrics was protected by kube-rbac-proxy with no way to grant a scraper access without hand-authoring the RBAC yourself.

Replace the per-cluster metric-name histograms (cluster identity baked into
the Prometheus metric name via Namespace, producing one never-unregistered
metric per cluster) and the two unlabeled global histograms with a single
properly labeled reconcile_duration_seconds histogram. Add outcome and
fleet-adoption visibility that didn't exist before, and ship the RBAC that
was missing for anyone actually trying to scrape /metrics.

Metrics (all `projectsveltos_` prefixed):
- **reconcile_duration_seconds** (histogram): cluster_type, cluster_namespace,
  cluster_name, feature. Buckets extended to 600s to catch slow Helm
  installs that used to fall into +Inf with no visibility.
- **reconcile_operations_total** (counter): cluster_type, cluster_namespace,
  cluster_name, feature. Counts every reconcile attempt.
- **reconcile_outcome_total** (counter): cluster_type, cluster_namespace,
  cluster_name, feature, status, profile_kind, profile_namespace,
  profile_name.  Terminal success/failure only, answers "is add-on X
  failing on N clusters."
- **matching_clusters** (gauge): profile_kind, profile_namespace, profile_name.
  Clusters currently matching a ClusterProfile/Profile selector.
- **reconcile_consecutive_failures** (gauge): cluster_type, cluster_namespace,
  cluster_name, feature, profile_kind, profile_namespace, profile_name.
  Mirrors ClusterSummary.status.featureSummaries[].consecutiveFailures.
- **reconcile_last_success_timestamp_seconds** (gauge): same labels as above.
  Unix timestamp of last Provisioned/Removed outcome.
- **total_drifts** (counter): cluster_type, cluster_namespace, cluster_name,
  feature, profile_kind, profile_namespace, profile_name. Existing metric,
  gained the profile_* labels.

Also:
- Fix metric name prefix:
- Enable auth-proxy RBAC resources (metrics-reader ClusterRole, Service,
  proxy-role/binding) in config/rbac/kustomization.yaml.
  Previously /metrics was protected by kube-rbac-proxy with
  no way to grant a scraper access without hand-authoring the RBAC yourself.
@gianlucam76 gianlucam76 merged commit 99652f8 into projectsveltos:main Jul 14, 2026
19 of 20 checks passed
@gianlucam76 gianlucam76 deleted the metrics branch July 14, 2026 09:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant