docs: add RFC 0006 for K8sGPT LLM-powered cluster monitoring by mani-thumu · Pull Request #6 · RADAR-base/rfcs

mani-thumu · 2026-05-29T16:32:28Z

Title: RFC 0006: K8sGPT LLM-Powered Kubernetes Cluster Monitoring

Summary
This RFC proposes adopting the k8sgpt-operator to integrate LLM-powered cluster analysis into our Kubernetes monitoring workflow. Currently, engineers must manually correlate raw metrics and events to diagnose failures — k8sgpt automates this by continuously scanning the cluster and emitting human-readable Result objects with root-cause explanations and remediation suggestions via Slack.

The RFC covers: architecture and control flow, Amazon Bedrock (IRSA) as the preferred LLM backend, S3 caching, Slack notifications, phased rollout plan, LLM cost model, and security/RBAC considerations. Manager and team sign-off on LLM spend is flagged as a required open decision before production rollout.

Checklist

Linked pre-RFC discussion issue
Used file name rfcs/platform/0006-k8sgpt-llm-cluster-monitoring.md and updated RFC front matter
Filled all required sections (Summary, Motivation, Design, Security, Migration, Testing)
Considered alternatives and documented open questions
Applied appropriate status label (status: draft)
Applied area label (area: platform)

yatharthranjan

Thanks Mani, looks very nice and right way forward in radar-base. Just added some questions.

yatharthranjan · 2026-07-24T13:50:25Z

+  interval: 2m
+```
+
+> **Auth note:** No API key secret is needed for Amazon Bedrock. Auth is handled via IRSA — the k8sgpt `ServiceAccount` is annotated with the IAM role ARN that has `bedrock:InvokeModel` permission.


presume this would require explicit configuration to be enabled ?

yatharthranjan · 2026-07-24T13:50:40Z

+    enabled: true
+    model: anthropic.claude-3-haiku-20240307-v1:0  # Claude Haiku via Bedrock
+    backend: amazonbedrock
+    region: us-west-2  # match your EKS cluster region


does this require any setup prior on amazon bedrock?

yatharthranjan · 2026-07-24T13:51:44Z

+
+> **Auth note:** No API key secret is needed for Amazon Bedrock. Auth is handled via IRSA — the k8sgpt `ServiceAccount` is annotated with the IAM role ARN that has `bedrock:InvokeModel` permission.
+
+> **How `interval` works:** `interval: 2m` means k8sgpt scans the cluster every 2 minutes. A sink notification (e.g. Slack) is only sent when a `Result` is **first created** or when its content **changes** — not on every scan. A persistently broken pod will generate one alert, not one every 2 minutes.


instead of just once, can a recurring issue alert every once in a while (like once a day or once a week)

Checked the operator source — k8sgpt can't do periodic re-alerts natively. Notifications are edge-triggered: it hashes each finding, and if a pod stays broken with the same error the Result is marked historical and isn't re-sent.
No repeat-interval option exists. (interval is just scan frequency — bumping it to 24h would only delay detection, not create reminders.) For "nudge again daily/weekly while still open," we'd do it downstream — Alertmanager repeat_interval via the CloudEvents sink, or a scheduled digest of open Result CRs into Slack.

yatharthranjan · 2026-07-24T13:57:06Z

+
+| Backend | Auth mechanism | Notes |
+|---------|---------------|-------|
+| Amazon Bedrock | IRSA (IAM Role for Service Account) | No static credentials needed — auth via native AWS IAM |


LiteLLM please, that can be single point, it can route to bedrock if needed

one question would be cost of data egress to the litellm endpoint, but should not cost much

Yes, if we use Bedrock: model access granted for that account+region, model available in-region, and IRSA. But since we're going LiteLLM-primary, Bedrock is now the optional downstream route and those creds live on the LiteLLM side — covered in the new Prerequisites section.

Done — reworked to LiteLLM as the primary backend, routing to vLLM (and Bedrock if needed). Egress is negligible (few KB, edge-triggered) and I've updated the security section since traffic no longer stays within AWS. All in the RFC now.

yatharthranjan · 2026-07-24T13:59:49Z

+### Caching
+
+k8sgpt deduplicates analysis results using a local cache keyed on resource identity + error hash. For persistent cache needs aligned with our AWS infrastructure, the operator supports:
+- Amazon S3 bucket (preferred — fits existing AWS setup)


will this be slow for caching?

No - AFAIK S3 won't be a bottleneck. Caching is off the critical path (background monitoring at 5m), it's one small object op per finding, and a ~tens-of-ms S3 GET is trivial next to the LLM call it avoids.

this will be worth reconsidering whether we need S3 at all now. The reason for a persistent remote cache is to avoid re-paying LLM cost across pod restarts, but since we moved the primary model to self-hosted vLLM (no per-token cost), a cache miss just re-runs inference for free. For a single cluster I'd suggest the built-in local/Interplex cache and dropping the S3 dependency + IAM — simpler with no real downside.

yatharthranjan · 2026-07-24T14:12:23Z

+
+_Bedrock pricing based on on-demand rates as of 2026. Actual cost depends on anomaly volume and output length._
+
+For a typical cluster, daily LLM spend is in the order of cents. Costs spike only when there are many simultaneous new issues or if the Log analyzer is enabled. Model selection (Haiku vs Sonnet) and spend approval require manager and team sign-off before production rollout.


yeah would expect at least sonnet for this

Sonnet isn't self-hostable on vLLM (Claude weights aren't open), so either the vLLM-served model needs to hit that capability bar, or we point k8sgpt's LiteLLM route at Bedrock Claude Sonnet (per-token cost). Left as an open question for the team to decide.

yatharthranjan · 2026-07-24T14:13:08Z

+k8sgpt sends resource metadata and error messages (e.g., pod names, container statuses, event messages) to the configured LLM API. It does **not** send pod environment variables, secret values, or volume contents. Sensitive field scrubbing is enabled by default and configurable.
+
+**Threat model:**
+- *Credential exposure:* Amazon Bedrock with IRSA requires no static API keys — credentials are handled via IAM role federation. Ensure the IAM role is scoped to `bedrock:InvokeModel` only and the IRSA annotation is set on the k8sgpt `ServiceAccount`.


yes agree. initially let's just focus on litellm

yes, agreed

yatharthranjan · 2026-07-24T14:15:22Z

+Open questions
+--------------
+- **Model selection and spend approval:** Amazon Bedrock is the chosen backend. Decision needed on Claude Haiku vs. Claude Sonnet (see cost model in Operational considerations) with manager and team sign-off before production rollout. A Bedrock spend budget alert in AWS Cost Explorer is recommended.
+- **Analysis interval tuning:** `2m` is a starting point; production load and token usage may require adjustment after the staging validation week.


i thought this was mentioned as 5m?

this sounds good for me, updated to 5m

yatharthranjan · 2026-07-24T14:15:50Z

+- **Model selection and spend approval:** Amazon Bedrock is the chosen backend. Decision needed on Claude Haiku vs. Claude Sonnet (see cost model in Operational considerations) with manager and team sign-off before production rollout. A Bedrock spend budget alert in AWS Cost Explorer is recommended.
+- **Analysis interval tuning:** `2m` is a starting point; production load and token usage may require adjustment after the staging validation week.
+- **Slack channel strategy:** Dedicated `#k8sgpt-findings` channel vs. routing to existing `#cluster-alerts`? High-volume findings may warrant a separate channel with digest/summarization.
+- **Multi-cluster scope:** Do we want to evaluate monitoring across multiple EKS clusters (via remote kubeconfig injection) in a follow-up RFC once the single-cluster rollout stabilizes?


don't think so, each cluster can deploy its own k8sgpt operator

Agreed — no central multi-cluster monitoring; each cluster deploys its own operator. Dropped it from Open Questions and updated the migration section to state the per-cluster model. Resolved.

yatharthranjan · 2026-07-24T14:16:03Z

+--------------
+- **Model selection and spend approval:** Amazon Bedrock is the chosen backend. Decision needed on Claude Haiku vs. Claude Sonnet (see cost model in Operational considerations) with manager and team sign-off before production rollout. A Bedrock spend budget alert in AWS Cost Explorer is recommended.
+- **Analysis interval tuning:** `2m` is a starting point; production load and token usage may require adjustment after the staging validation week.
+- **Slack channel strategy:** Dedicated `#k8sgpt-findings` channel vs. routing to existing `#cluster-alerts`? High-volume findings may warrant a separate channel with digest/summarization.


def dedicated channel

…cope

…sis.interval; 5m; per-cluster scope

…channel) for RFC 0006

docs: add RFC 0006 for K8sGPT LLM-powered cluster monitoring

23c7dd1

mani-thumu requested review from afolarin, baixiac, mpgxvii, pvannierop and yatharthranjan as code owners May 29, 2026 16:32

github-actions Bot added rfc RFC document and related discussion area: platform Platform-wide concerns, infra, deployment labels May 29, 2026

ManiDeepakThumu added 3 commits May 29, 2026 17:35

chore: re-trigger CI

9887a2f

docs: update RFC 0006 for K8sGPT LLM-powered cluster monitoring

92a4302

docs: remove unnecessary AI backends and standardise on AWS Bedrock

e0aa0c2

yatharthranjan added the status: draft RFC is in draft and under review label Jul 24, 2026

yatharthranjan reviewed Jul 24, 2026

View reviewed changes

docs: rework RFC 0006 for LiteLLM backend, 5m interval, per-cluster s…

70dce19

…cope

github-actions Bot requested a review from yatharthranjan July 29, 2026 15:45

ManiDeepakThumu added 3 commits July 30, 2026 12:12

docs: rework RFC 0006 for LiteLLM backend; fix interval to spec.analy…

1511d4a

…sis.interval; 5m; per-cluster scope

docs: update RFC 0006 from stage validation and review feedback

769c85b

docs: resolve caching (local/Interplex default) and Slack (dedicated …

b14812c

…channel) for RFC 0006


		> Auth note: No API key secret is needed for Amazon Bedrock. Auth is handled via IRSA — the k8sgpt `ServiceAccount` is annotated with the IAM role ARN that has `bedrock:InvokeModel` permission.

		> How `interval` works: `interval: 2m` means k8sgpt scans the cluster every 2 minutes. A sink notification (e.g. Slack) is only sent when a `Result` is first created or when its content changes — not on every scan. A persistently broken pod will generate one alert, not one every 2 minutes.


		_Bedrock pricing based on on-demand rates as of 2026. Actual cost depends on anomaly volume and output length._

		For a typical cluster, daily LLM spend is in the order of cents. Costs spike only when there are many simultaneous new issues or if the Log analyzer is enabled. Model selection (Haiku vs Sonnet) and spend approval require manager and team sign-off before production rollout.

Uh oh!

Conversation

mani-thumu commented May 29, 2026 • edited by yatharthranjan Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yatharthranjan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mani-thumu Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mani-thumu commented May 29, 2026 •

edited by yatharthranjan

Loading

mani-thumu Jul 29, 2026 •

edited

Loading