Skip to content

[unbounded-system] Operator owns CRD lifecycle and config; harden migration#464

Merged
plombardi89 merged 3 commits into
feature/unbounded-systemfrom
operator-config-bootstrap
Jul 13, 2026
Merged

[unbounded-system] Operator owns CRD lifecycle and config; harden migration#464
plombardi89 merged 3 commits into
feature/unbounded-systemfrom
operator-config-bootstrap

Conversation

@plombardi89

@plombardi89 plombardi89 commented Jul 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

Follow-up to #386 that was meant to land there. Targets the feature/unbounded-system integration branch as part of umbrella #388.

Makes unbounded-operator self-sufficient for CRD lifecycle, moves environment-specific settings into a ConfigMap, and hardens reconciliation and legacy migration against rollout, ownership, readiness, and configuration races.

Operator owns CRDs

  • Add operator.BootstrapCRDs: at startup the operator server-side applies all 12 CRDs embedded in the machina and net manifests before starting the manager.
  • Bound the complete bootstrap operation, including CRD applies and establishment, with a four-minute timeout.
  • Add a startup probe whose budget exceeds the complete bootstrap timeout so liveness cannot restart the operator mid-bootstrap.
  • kubectl unbounded install no longer applies CRDs or exposes --skip-crds; the breaking CLI change is documented.
  • install waits for a complete operator rollout, rejecting the old-ReplicaSet-available/new-pod-failing upgrade case, and then waits for CRD establishment.
  • The Site reconcile loop skips CRD objects because CRDs are owned exclusively by startup bootstrap.

Config via ConfigMap

  • Add unbounded-operator-config, surfaced to the Deployment through envFrom.
  • --api-server-endpoint and --reap-legacy-resources default from UNBOUNDED_API_SERVER_ENDPOINT and UNBOUNDED_REAP_LEGACY_RESOURCES; explicit flags retain precedence.
  • Invalid UNBOUNDED_REAP_LEGACY_RESOURCES values now fail startup rather than silently enabling destructive migration.
  • Apply the operator ConfigMap before the Deployment to prevent a new pod from starting with stale environment values.
  • Stamp the resolved config hash on the operator pod template so endpoint changes trigger a rollout in both install and rendered-manifest workflows.
  • Restore UNBOUNDED_OPERATOR_API_SERVER_ENDPOINT to the Make/render path and make endpoint forwarding and YAML rendering safe for shell- and YAML-sensitive values.

Machina config propagation

  • Merge the configured API endpoint into existing or migrated machina-config while preserving all other custom settings and comments.
  • Hash the exact stored config.yaml and stamp it on the Machina Deployment so configuration changes roll Machina, which reads its subPath-mounted config only at startup.
  • Watch machina-config changes and enqueue Sites without re-enabling status-only Site reconciliation.
  • During migration, apply the configured endpoint to copied legacy Machina config.
  • Reap legacy Machina only after the replacement Deployment has the current config hash and a completed rollout.

Reconciliation reliability

  • Preserve per-component reconciliation errors while still publishing all Site status conditions.
  • Return joined component and status-patch errors so controller-runtime rate-limits and retries transient failures even with GenerationChangedPredicate filtering status-only updates.

Migration hardening

  • Detect legacy per-site Metalman by deterministic Deployment name or canonical/deprecated Site label.
  • Preserve the legacy --dhcp-auto-interface setting in the translated Site; malformed or conflicting values fail closed.
  • Gate Metalman reaping on creation of every corresponding replacement Deployment without requiring readiness, avoiding host-port cutover deadlock.
  • Require storage replacements to be current and ready. A zero-desired per-Site DaemonSet is accepted only when no Node InternalIP matches that Site node CIDRs; matching but not-yet-labeled Nodes continue to block reaping.
  • Make the net controller update SiteNodeSlice ownership even when slice contents are unchanged.
  • Delete the legacy net-group Site CRD only after the new net controller is Available and every existing SiteNodeSlice has the exact current v1alpha3 Site UID/GVK owner reference.
  • Warn and emit an Event when a drained legacy namespace still contains non-operator workloads before namespace deletion.
  • Add the missing read-only RBAC for StatefulSets, Nodes, and SiteNodeSlices, plus events.k8s.io write access for the modern event recorder.
  • Reconcile Sites on generation changes only, avoiding full net/machina re-application on status-only updates.
  • Retarget embedded namespace references in service-account usernames and flag values, not only exact-value matches.

Tests

Added or extended coverage for:

  • all embedded CRDs, establishment polling, and apply-phase timeout cancellation;
  • startup probe budget and operator ConfigMap-before-Deployment ordering;
  • complete Deployment rollout detection and endpoint-driven rollout hashes;
  • shell/YAML-sensitive endpoint rendering;
  • strict environment parsing and explicit flag precedence;
  • reconciliation error retries and joined status-patch errors;
  • Machina config preservation, endpoint updates, hashes, and rollout gates;
  • Metalman detection and DHCP mode preservation;
  • zero-node storage readiness and unconverged-node blocking;
  • SiteNodeSlice owner migration and legacy CRD deletion gates;
  • StatefulSet foreign-workload detection and rendered RBAC.

The kind-based operator reaper e2e now validates the migration against a real Kubernetes API server, including preserved Metalman mode and the Machina config rollout gate.

Verification

  • make fmt
  • make build
  • make lint
  • make test
  • race tests for all modified Go packages
  • go test -tags=e2e ./e2e/operator/... -run TestOperatorReaperMigration -count=1 -v

@plombardi89 plombardi89 requested a review from a team as a code owner July 13, 2026 17:04
@plombardi89
[unbounded-system] Operator owns CRD lifecycle and config; harden mig…
4071d64 
…ration

Make the unbounded-operator self-sufficient for CRD lifecycle and move
environment-specific settings off Deployment args into a ConfigMap, then
harden the legacy migration gates.

Operator-owned CRDs:
- Add operator.BootstrapCRDs: at startup the operator server-side applies
  every CRD embedded in the machina and net manifests and waits for each to
  become Established, before the manager starts (the typed Site informer
  cannot sync until the Site CRD is served). Idempotent on every start.
- `kubectl unbounded install` no longer applies CRDs or takes --skip-crds;
  it bootstraps the operator and waits for its rollout, then for the CRDs.
  The reconcile loop skips CRD objects when applying net manifests.

Config via ConfigMap:
- Add deploy/unbounded-operator/04-configmap.yaml.tmpl
  (unbounded-operator-config), surfaced to the Deployment via envFrom.
  --api-server-endpoint and --reap-legacy-resources now default from
  UNBOUNDED_API_SERVER_ENDPOINT and UNBOUNDED_REAP_LEGACY_RESOURCES (an
  explicit flag still overrides). install writes api-server-endpoint into
  the ConfigMap instead of a Deployment arg.

Migration hardening:
- Detect legacy per-site metalman by Deployment name or canonical/deprecated
  site label, not a single label match.
- Gate metalman reaping on the per-site replacement Deployment existing
  (presence gate, like net: hostNetwork host-port ownership).
- Require storage DaemonSets to actually schedule (DesiredNumberScheduled >= 1
  and observed generation current) before reaping the legacy supervisor, so a
  zero-desired DaemonSet no longer reads as ready.
- Delete the legacy net-group Site CRD only after the new net controller is
  Available, so the GC does not transiently drop SiteNodeSlices mid-cutover.
- Warn and emit an Event when a drained legacy namespace still holds
  non-operator workloads before the whole-namespace delete.
- Reconcile Sites on generation changes only, so net/machina manifests are not
  re-applied on status-only Site updates.
- Retarget the namespace as a substring (SA usernames, flag values), not just
  on exact-value equality.

Tests: CRD bootstrap, env-var flag defaults, metalman detection/presence gate,
storage readiness gates, net-controller CRD gate, foreign-workload detection;
extend the upgrade e2e to assert the operator (not install) established and
owns the Site CRD and that the operator-config ConfigMap is present.
@plombardi89 plombardi89 force-pushed the operator-config-bootstrap branch from de4a233 to 4071d64 Compare July 13, 2026 17:05
@plombardi89
[unbounded-system] Address install/operator review findings
a97cbf6 
Fixes from PR review of the operator CRD-lifecycle/config work.

1. install no longer reports success against an unavailable operator.
   waitForOperator gated on AvailableReplicas>=desired, which the old
   ReplicaSet satisfies while a new crash-looping pod surges in during an
   upgrade. Replace it with a full rollout-complete check
   (ObservedGeneration current and Replicas == UpdatedReplicas ==
   AvailableReplicas == desired), the same condition `kubectl rollout status`
   waits on.

2. Liveness no longer kills the operator mid-bootstrap. BootstrapCRDs runs
   before the manager starts the health server, so /healthz is unserved for up
   to crdEstablishedTimeout (~2m) while liveness (10s + 3x15s) would restart at
   ~40s. Add a startupProbe with a 150s budget (30 x 5s) that exceeds the
   bootstrap timeout and suppresses liveness until the health server binds.

3. Re-install with a changed endpoint now rolls the operator. install wrote the
   endpoint only into the ConfigMap, but envFrom is read once at pod start and
   the pod template was unchanged, so no rollout occurred. Stamp a config-hash
   annotation (sha256 of the endpoint) onto the Deployment pod template; a
   changed endpoint changes the template and triggers a rollout that the
   strengthened waitForOperator waits on.

4. The make/GitOps render path can set the endpoint again. Restore
   .APIServerEndpoint into the operator-config ConfigMap and re-add the
   UNBOUNDED_OPERATOR_API_SERVER_ENDPOINT Makefile variable / --set. The
   Deployment template stamps the same config-hash (sprig sha256sum) so raw
   `kubectl apply` of re-rendered manifests also rolls on an endpoint change.

--skip-crds stays removed (CRDs are operator-owned); documented as a breaking
change in the CLI reference.

Tests: rollout-complete table + waitForOperator upgrade/complete cases;
config-hash stamped and endpoint-sensitive; BootstrapCRDs applies all 12
embedded CRDs and blocks until Established; a render test asserting the
ConfigMap endpoint and the matching Deployment hash annotation stay in sync.
@plombardi89
[unbounded-system] Harden operator reconciliation and migration
7bddcf7
@plombardi89 plombardi89 mentioned this pull request Jul 13, 2026
Open
jveski
jveski approved these changes Jul 13, 2026
View reviewed changes
Comment thread deploy/unbounded-operator/03-configmap.yaml.tmpl
# cannot be derived in-cluster. Set this before enabling machina provisioning.
# Rendered from .APIServerEndpoint so make/GitOps deploys can set it via
# --set APIServerEndpoint=; `kubectl unbounded install` overrides it in place.
UNBOUNDED_API_SERVER_ENDPOINT: {{ default "" .APIServerEndpoint | quote }}

@jveski jveski Jul 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We discover this from kube-public in metalman, might be worth aligning on. I'm indifferent as to which approach we take as long as it's consistent.

@plombardi89 plombardi89 merged commit fd5ae30 into feature/unbounded-system Jul 13, 2026
11 of 12 checks passed
@plombardi89 plombardi89 deleted the operator-config-bootstrap branch July 13, 2026 23:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jveski jveski jveski approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@plombardi89 @jveski