Skip to content

cli.md

Latest commit

 

History

History
146 lines (104 loc) · 4.53 KB

cli.md

File metadata and controls

146 lines (104 loc) · 4.53 KB
Raw

llm-observability CLI

llm-observability is the Go CLI for this repository. It manages the LLM application and observability layer that runs on top of the k3s-nvidia-edge base layer.

Architecture Boundary

  • k3s-nvidia-edge owns the local Linux, k3s, k3s containerd, NVIDIA runtime, GPU Operator, NVIDIA device plugin, DCGM exporter, Node Feature Discovery, RuntimeClass/nvidia, and nvidia.com/gpu validation layer.
  • llm-observability-stack owns Ollama, Open WebUI, Open WebUI Redis, OpenTelemetry Collector, optional Prometheus/Grafana, optional LangChain/FastAPI proxy, benchmark tooling, notebooks, and local model configuration.
  • The llm-observability CLI imports reusable base workflows from github.com/Edge-Computing-LLM/k3s-nvidia-edge/pkg/edgebase.
  • The CLI does not import k3s-nvidia-edge/internal/....
  • This is an incremental step toward a future organization-level CLI spanning Edge-Computing-LLM repositories.

Build

From this repository:

go build -o bin/llm-observability ./cmd/llm-observability

During local sibling-repo development, go.mod uses:

replace github.com/Edge-Computing-LLM/k3s-nvidia-edge => ../k3s-nvidia-edge

That keeps the long-term import path clean while allowing local changes in k3s-nvidia-edge/pkg/edgebase to be tested immediately. When k3s-nvidia-edge publishes a version tag containing pkg/edgebase, this can be changed to a normal tagged requirement.

Commands

bin/llm-observability doctor
bin/llm-observability install --profile geforce-940m-k3s --skip-base --yes
bin/llm-observability status
bin/llm-observability validate
bin/llm-observability benchmark --model gemma3-1b-it-gguf-local --runs 3
bin/llm-observability uninstall --yes
bin/llm-observability uninstall --with-base --yes
bin/llm-observability print-commands --profile geforce-940m-k3s

Profiles

The CLI maps profile names to existing values files:

Profile Values file
geforce-940m-k3s values.geforce-940m-k3s.yaml
enterprise-pilot-k3s values.enterprise-pilot-k3s.yaml
cpu-k3s values.cpu-k3s.yaml
local-k3s-example values.local-k3s.example.yaml
local-k3s values.local-k3s.yaml
full-stack-nvidia values.full-stack-nvidia.example.yaml
validation-k3s values.validation-k3s.yaml

You can also pass a values file directly:

bin/llm-observability install --profile values.geforce-940m-k3s.yaml --yes

Additional overrides are supported:

bin/llm-observability install \
  --profile geforce-940m-k3s \
  --values values.local-k3s.yaml \
  --set ollamaModel.gguf.hostPath=/path/to/models \
  --yes

For NVIDIA profiles, the CLI adds Helm safeguards so this chart does not redeploy the base GPU layer:

--set gpu-operator.enabled=false
--set nvidia-device-plugin.enabled=false
--set dcgm-exporter.enabled=false

Recommended Local Command

For the current Xubuntu 24 + k3s + NVIDIA GPU setup where k3s-nvidia-edge is already healthy:

bin/llm-observability install --profile geforce-940m-k3s --skip-base --yes
bin/llm-observability validate

If the base layer is not installed yet:

bin/llm-observability install --profile geforce-940m-k3s --with-base --yes

By default, install and uninstall commands are dry-run unless --yes is provided.

Validation

validate checks:

  • base NVIDIA readiness for GPU profiles
  • Helm release status
  • Ollama service, deployment, model list, and loaded model state
  • Open WebUI statefulset and service
  • Open WebUI Redis deployment when present
  • OpenTelemetry Collector deployment and service
  • pod readiness in the namespace
  • CUDA/offload evidence in Ollama logs for GPU profiles

The base CUDA validation pod from edgebase is dry-run unless --yes is provided, matching the safety behavior of k3s-nvidia-edge.

Benchmark

The benchmark command wraps the existing Python client:

bin/llm-observability benchmark \
  --model gemma3-1b-it-gguf-local \
  --runs 3 \
  --prompt "Explain edge GPU observability in one sentence." \
  --output artifacts/benchmark-local.json

It starts a temporary kubectl port-forward to svc/ollama and then runs benchmarks/ollama_benchmark.py.

Uninstall

Uninstall only the LLM stack:

bin/llm-observability uninstall --yes

Keep the namespace:

bin/llm-observability uninstall --keep-namespace --yes

Also uninstall the base GPU layer:

bin/llm-observability uninstall --with-base --yes

The base layer is never removed unless --with-base is explicitly passed.