Skip to content

OPERATIONS-RUNBOOK.md

Latest commit

 

History

History
190 lines (133 loc) · 4.8 KB

OPERATIONS-RUNBOOK.md

File metadata and controls

190 lines (133 loc) · 4.8 KB
Raw

Operations Runbook

This runbook collects the day-0 and day-1 commands you are most likely to need while operating llm-observability-stack on a local k3s machine.

1. Before You Change Anything

Check current state:

kubectl get all -n llm-observability
kubectl get svc -n llm-observability
helm list -n llm-observability
git status --short

2. Build and Refresh Local Images

Rebuild langchain-demo

./hack/build-local-image.sh langchain-demo 0.1.1 ./langchain-demo
./hack/import-local-image-to-k3s.sh langchain-demo 0.1.1
kubectl rollout restart deploy/langchain-demo -n llm-observability
kubectl rollout status deploy/langchain-demo -n llm-observability

Rebuild python-toolbox

./hack/build-local-image.sh python-toolbox 0.2.0 ./python-toolbox
./hack/import-local-image-to-k3s.sh python-toolbox 0.2.0
kubectl rollout restart deploy/python-toolbox -n llm-observability
kubectl rollout status deploy/python-toolbox -n llm-observability

3. Install, Upgrade, and Roll Back

Install or upgrade

helm upgrade --install llm-observability-stack . \
  -n llm-observability --create-namespace \
  -f values.enterprise-pilot-k3s.yaml \
  --set kube-prometheus-stack.crds.enabled=false

Use a private values.local-k3s.yaml only when your host paths, secrets, or service exposure differ. Keep ollama.persistentVolume.size aligned with any existing ollama PVC; k3s local-path storage does not resize that claim in place.

Inspect release values

helm get values llm-observability-stack -n llm-observability -a

Review history and rollback

helm history llm-observability-stack -n llm-observability
helm rollback llm-observability-stack <REVISION> -n llm-observability

Uninstall

helm uninstall llm-observability-stack -n llm-observability

4. Health Checks

Basic workload checks

kubectl get pods -n llm-observability -o wide
kubectl get svc -n llm-observability
kubectl get pvc -n llm-observability
kubectl top pods -n llm-observability

Service-specific logs

kubectl logs -n llm-observability deploy/langchain-demo --tail=100
kubectl logs -n llm-observability deploy/ollama --tail=100
kubectl logs -n llm-observability statefulset/open-webui --tail=100
kubectl logs -n llm-observability deploy/python-toolbox --tail=100

5. Access Internal APIs

kubectl port-forward -n llm-observability svc/ollama 11434:11434
kubectl port-forward -n llm-observability svc/langchain-demo 8000:8000

Use these for:

  • direct Ollama API tests
  • LangChain proxy notebook cells
  • OpenTelemetry traced proxy requests from the host

6. Jupyter Notebook Operations

Launch:

PYTHON_BIN="${PYTHON_BIN:-python3.11}"
cd jupyter-notebooks
"${PYTHON_BIN}" -m jupyter lab

Useful pairings:

  • 01 before any major change
  • 07 when validating python-toolbox
  • 09 when validating cluster networking

If notebook cells fail:

  • confirm the required port-forwards
  • confirm the release is installed
  • confirm the toolbox pod is running
  • confirm OpenTelemetry environment variables for tracing notebooks

7. In-Cluster Toolbox Checks

Open a shell:

kubectl exec -it -n llm-observability deploy/python-toolbox -- bash

Run helper scripts:

kubectl exec -it -n llm-observability deploy/python-toolbox -- python /workspace/examples/service_dns_check.py
kubectl exec -it -n llm-observability deploy/python-toolbox -- python /workspace/examples/ollama_smoke.py
kubectl exec -it -n llm-observability deploy/python-toolbox -- python /workspace/examples/redis_ping.py

8. GPU Checks

watch -n 0.5 nvidia-smi
kubectl get pods -n nvidia-device-plugin
kubectl get nodes -o json | jq '.items[0].status.allocatable'

If scheduling is failing, inspect the node plugin runtime behavior and confirm the resource name requested in values.enterprise-pilot-k3s.yaml or your private local override.

9. Troubleshooting Patterns

open-webui works but notebooks fail

  • likely missing kubectl port-forward for ollama and/or langchain-demo

langchain-demo is unhealthy

  • inspect logs
  • confirm the local image tag imported into k3s matches the chart values
  • confirm OpenTelemetry env inputs are valid when tracing is enabled

ollama is healthy but model calls fail

  • verify GGUF host path mounts
  • verify Modelfile render values
  • verify the model exists through /api/tags

python-toolbox is present but scripts are missing

  • rebuild/import the local python-toolbox image
  • restart the toolbox deployment

10. Cleanup and Hygiene

Do not commit:

  • values.local-k3s.yaml
  • notebook checkpoint directories
  • generated notebook assets
  • rendered manifests
  • secret material
  • large model binaries

Before publishing:

git status --short
helm lint .
pytest -q tests