[unbounded-system] Integration: single namespace + operator-driven migration#388
Open
plombardi89 wants to merge 16 commits into
Open
[unbounded-system] Integration: single namespace + operator-driven migration#388plombardi89 wants to merge 16 commits into
plombardi89 wants to merge 16 commits into
Conversation
Integration branch tracking doc for the namespace-consolidation effort: the problem, the integration-branch strategy (keep main releasable), the PR-A/PR-B/PR-C decomposition, the migration workflow, and a living status checklist updated as each chunk opens and merges.
…title convention
…ace (#372) * build: add unified UNBOUNDED_NAMESPACE feeding all component namespaces Introduce a single UNBOUNDED_NAMESPACE (default unbounded-system) that every per-component *_NAMESPACE var derives from, so overriding it moves all components at once while individual components can still be overridden. Re-point MACHINA/MACHINE_OPS/ORCA/NET/GANTRY/UNBOUNDED_STORAGE_SUPERVISOR namespace vars and fix the hardcoded -n unbounded-kube in machina-run. * deploy: default machina/machine-ops/orca/storage-supervisor namespace to unbounded-system Flip the template default from {{ default "unbounded-kube" .Namespace }} to unbounded-system across the machina, machine-ops, orca, and storage-supervisor manifest templates, plus the orca cachestore/cluster service DNS defaults. Genuine kube-system/kube-public RBAC reads are unchanged. * net: default namespace to unbounded-system Add a default to the net manifest templates (previously bare {{ .Namespace }}) so they render into unbounded-system, and align the controller/webhook Go defaults (leader-election lease namespace, POD_NAMESPACE fallbacks) and the kubectl install constant to match. Genuine kube-system reads (kube-proxy, extension-apiserver-authentication) are unchanged. * gantry: templatize manifests onto unbounded-system namespace Convert the gantry deploy manifests from hardcoded gantry-system plain YAML to *.yaml.tmpl that default to unbounded-system, add a gantry-manifests render target (rendered/ gitignored), and update the gantry test fixtures, e2e references, and README. Runtime namespace was already sourced from the Downward API, so no Go constant change is needed. * inventory: templatize manifests, fix machina-system orphan, default to unbounded-system Convert the inventory manifests from plain .yaml (which already held template syntax but were never rendered) to proper *.yaml.tmpl, default their namespace to unbounded-system, and replace the non-Go-template {{ SSL_MODE }} / {{ PASSWORD }} tokens with real template keys. Fix the aggregator Service, which was pinned to a stray machina-system namespace, to co-locate with its Deployment. Add an inventory-manifests render target and rework hack/inventory-dev/local.sh to render-then-apply. * go: default hardcoded component namespaces to unbounded-system Update the remaining hardcoded namespace constants and flag defaults (machina SSH-secret namespace, machine-ops leader-election/credential secret, kubectl install + deploy-pxe namespaces, metalman bootstrap SA), fix site_init to use the machina namespace constant instead of a hardcoded literal, and resolve metalman's leader-election lease namespace from POD_NAMESPACE (injected via the Downward API by deploy-pxe) so the lease and its RBAC stay co-located. Product/repo-name references (Azure/unbounded-kube, 'unbounded-kube machines/sites') are unchanged. Update affected test fixtures. * hack: point scripts, smoke checks, and examples at unbounded-system Update orca install/smoke scripts, the storage-mesh config generator, the release core-namespaces-ready smoke check, the metalman lease cleanup, the orca config example DNS names, and the agent e2e-kind harness namespace references to the unified unbounded-system namespace. Resource/binary/label names (unbounded-net-controller, unbounded-net-gateway, ...) and product/repo references are left unchanged. * docs: document unbounded-system namespace + add migration runbook Update guides and reference docs to the unified unbounded-system namespace (machina/metalman SSH/PXE namespaces, net kubectl/exec namespace flags, net leader-election default), preserving the net.unbounded-kube.io API domain and product-name references. Add designs/namespace-unification.md describing the configuration model and a forward-only migration runbook for existing clusters. * test/ci: finalize unbounded-system unification and fix stragglers Point the orca stable garage template default and the gantry configmap regression test at unbounded-system / the renamed *.yaml.tmpl, update the from-source aks-ssh-quickstart workflow and component READMEs, and align the orca manifests render test. Verified via make fmt, lint (0 issues), test (green), and build. Note: .github/workflows/release-upgrade.yaml still references the old namespaces; it deploys released artifacts and implements cross-version upgrade semantics, so it must be updated when the first unbounded-system release is cut (see designs/namespace-unification.md). * refactor: centralize the default namespace in internal/unbounded Add internal/unbounded.SystemNamespace as the Go-side source of truth for the default install namespace, and point the production default/fallback sites and the descriptive local consts (machinaNamespace, unboundedCNINamespace, deployPXENamespace, SecretNamespaceUnboundedSystem, BootstrapSANamespace, orcadev defaultNamespace, net/machine-ops/metalman leader-election defaults) at it. Update the tests coupled to those defaults to reference the const instead of the literal. Add a drift-guard test that renders each component's namespace template with no Namespace override and asserts the template default matches unbounded.SystemNamespace, so the Go const and the manifest templates cannot silently diverge. The Makefile UNBOUNDED_NAMESPACE and the template literals remain the render/deploy source of truth (they cannot reference Go); the drift-guard keeps them aligned. * fix: catch stragglers still pointing at old namespaces Three call sites were missed by the unbounded-system unification: - aks-ssh-quickstart workflow: the ACR pull-secret step created the secret and patched the unbounded-net-node ServiceAccount in the unbounded-net namespace. site init now installs net into unbounded-system, so the patch (and create) hard-failed under set -euo pipefail. Point both at unbounded-system. - kubectl unbounded net: the namespace auto-detection probed unbounded-net and kube-system but not unbounded-system, so net resources could not be located without an explicit --namespace. Add unbounded.SystemNamespace as the primary candidate, keeping the old names as legacy fallbacks. - deploy/gantry/README: the Files table and apply order still referenced the raw *.yaml files and applied them directly, but they are now *.yaml.tmpl rendered via make gantry-manifests into rendered/. Update the docs and the broken overlay link. * docs: drop namespace-unification design doc The design doc is not needed in designs/. The one-time manual cluster cutover steps it carried are tracked in the follow-up issue instead, so no operational knowledge is lost. * hack: add namespace migration script for legacy clusters Existing clusters installed unbounded across split namespaces (machina/metalman in unbounded-kube, net in unbounded-net). Consolidating onto unbounded-system requires a one-time move because namespaces cannot be renamed in place, and we want every cluster off the old namespaces rather than lingering on the split layout. Add hack/scripts/migrate-namespace.sh, an idempotent, sequenced migrator for machina + metalman + net: - copies operator-owned state (Secrets, the machina-config ConfigMap, and metalman PXE Deployments) into unbounded-system, skipping auto-managed and regenerable resources (SA tokens, the net serving cert); - redeploys the components there from the released manifests, supplied either by --release TAG (downloaded from GitHub via curl) or an extracted --manifests-dir for air-gapped use; - rewrites the cluster-scoped CR secret references that enforce a namespace (MachineOperationCredential auth secretRef, Machine pxe.redfish passwordRef); - decommissions the old namespaces only after the new components are healthy, so a failure mid-run leaves a recoverable cluster and re-running completes. Cluster-scoped CRs/CRDs are left untouched; gantry/inventory are out of scope. There is a brief net data-plane interruption during the cutover (old and new net cannot run in parallel), which is inherent to the move. Add hack/smoke-namespace-migration.py, a kind-based e2e that stages a neutralized legacy install plus operator state and asserts the consolidated end-state across four scenarios (happy + idempotency, dry-run, resume, release-download), wired into a path-triggered smoke workflow. Also fix a rename straggler in the `site` command help text (unbounded-kube -> unbounded). * namespace: address review - operator-owned migration + machina ns + docs - Remove the standalone migrate-namespace.sh, its kind smoke test, and the smoke workflow: migration from the legacy split namespaces is now handled by unbounded-operator, so the shell path is no longer needed. - Resolve machina's SSH-secret namespace at runtime from POD_NAMESPACE (injected via the Downward API on the machina Deployment) instead of a compile-time constant, and use it for both getSecretValue and the informer cache scope. Add a --namespace flag to 'kubectl unbounded machine register' so the written Secret and its Machine ref land in the namespace machina runs in. This makes a non-default install namespace line up end-to-end. - Point the net day-2 operations docs (pods, node agents, leases, serving cert, endpoints, NetworkPolicies) at unbounded-system, where net now runs. make fmt/lint/test/build all green. * namespace: centralize namespace resolution in unbounded.SystemNamespace() Replace the SystemNamespace const with a POD_NAMESPACE-aware helper so every component resolves its runtime namespace the same way: - internal/unbounded: SystemNamespace() returns POD_NAMESPACE when set (injected via the Downward API), else the unexported systemNamespace default. This folds the scattered os.Getenv("POD_NAMESPACE")-with-fallback logic (net controller, metalman lease, machina secret lookup) into one helper. - Convert the compile-time namespace consts that derived from it to package vars (they now read POD_NAMESPACE at init: client tools see the default, in-cluster components see their pod namespace). - Drop machina's SecretNamespaceUnboundedSystem in favor of the helper for the SSH-secret read and informer cache scope. - Drift-guard test becomes an internal test comparing template defaults to the unexported systemNamespace. - Makefile documents that a non-default namespace lines up end to end and that 'machine register --namespace' is needed when installing to one. Deployments keep injecting POD_NAMESPACE, so a non-default install namespace works end to end. make fmt/lint/test/build all green. * namespace: inline unbounded.SystemNamespace() instead of alias vars Remove the local namespace aliases that just wrapped the helper (machinaNamespace, deployPXENamespace, unboundedCNINamespace, BootstrapSANamespace, orcadev defaultNamespace) and call unbounded.SystemNamespace() directly at each use site. Now that SystemNamespace() is itself the resolver, the aliases added indirection with no value.
…unded-system; PR-C now migration-only
… and released-cluster migration (#386) * Introduce unbounded-operator on the unbounded-system namespace Add the top-level unbounded-operator and reshape Site into the object it reconciles, running on the unified unbounded-system namespace. Components install at the operator's own version into unbounded-system, so per-site image and namespace overrides are gone: net and machina are cluster singletons and metalman/storage are site-scoped. Site API: - Move Site from net.unbounded-cloud.io/v1alpha1 to the cluster-scoped unbounded-cloud.io/v1alpha3 (machina) group. spec.components now covers only the operator-managed workloads (machina, metalman, storage) plus per-component status; net is not a component. SiteComponentSpec is just {enabled} (image/namespace overrides removed). The CRD moves with it. - Retarget the net controller, validating webhook, unboundednet client, net-node agent, and the 'net create' path to the new group/version; split RBAC and the webhook rules accordingly. Operator: - cmd/unbounded-operator + internal/operator: a controller-runtime manager that reconciles Site into component workloads by server-side applying the embedded rendered manifests. net is deployed as a cluster singleton whenever any Site exists; machina as a singleton whenever any Site enables it; metalman is a per-site Deployment (with POD_NAMESPACE) plus its shared RBAC; storage is a per-site DaemonSet node-selected on net.unbounded-cloud.io/site. Config is reduced to the metalman image (version-matched at render) and the machina apiServerEndpoint. - Deploy assets under deploy/unbounded-operator and an image build. kubectl plugin: - Replace the imperative install paths with a single 'install' command that bootstraps CRDs + the operator; 'site init' writes declarative Site resources. Drop --enable-net and the per-component image-override install flags along with the net: block in the site template. Namespace: the operator, its defaults, and the kubectl install paths all resolve to unbounded.SystemNamespace(); operator manifests render into unbounded-system and the drift-guard covers the operator template. make fmt/lint/test/build all green. * Migrate existing clusters onto unbounded-system via the operator Add the operator-driven migration that moves a pre-redesign cluster (machina, metalman, and storage in unbounded-kube; net in unbounded-net; net-group Sites) onto the unified unbounded-system namespace and the machina-group Site API. It runs as an always-on, leader-elected manager Runnable gated by --reap-legacy-resources (default on); there is no one-shot subcommand and no standalone shell migrator. Each idempotent pass: 0. Translates every pre-redesign net-group Site (sites.net.unbounded-cloud.io) into the machina-group Site the operator reconciles, copying the networking spec verbatim and inferring spec.components from the running legacy workloads: storage on every translated Site, machina on the cluster Site, and metalman where a per-site metalman Deployment is detected. Existing machina-group Sites are never clobbered. 1. Copies non-regenerable state (Secrets, the machina-config and unbounded-storage-config ConfigMaps) into unbounded-system, stripping server-managed metadata, never overwriting existing target copies, and skipping regenerable/auto-managed secrets (net serving cert, SA tokens, Helm release secrets). 2. Rewrites the namespace embedded in cluster-scoped secret references (Machine.spec.pxe.redfish.passwordRef, MachineOperationCredential spec.auth.secretRef). 3. Reaps the operator-owned resources left in each legacy namespace once the new workloads are in place (label-scoped; net last; empty-footprint components skipped). Storage is gated on the per-site unbounded-storage-supervisor-<site> DaemonSets. Net is gated only on the new net workloads being CREATED (not Ready): the old and new net-node DaemonSets both use hostNetwork and cannot occupy the same node host ports, so the old net must be reaped to let the new net schedule (a brief net data-plane gap during cutover is expected and accepted). 4. Once drained, deletes the legacy net-group Site CRD and the legacy namespaces. Translating the Sites first gives the operator Sites to reconcile (bringing up the new components, including the net singleton that re-labels nodes for storage node-selection) before the reaper drains the old namespaces. Wire the reaper into the operator manager and broaden the operator ClusterRole to read legacy net-group Sites, translate/create machina-group Sites, rewrite Machine/MachineOperationCredential secret refs, install the component RBAC (escalate + bind, since the net/machina Roles grant permissions the operator does not itself hold), reap by label (deletecollection), and delete the drained legacy resources, namespaces, and Site CRD. Testing. Two complementary reaper e2e's: - e2e/operator (Go, //go:build e2e): drives the LegacyReaper in-process against a kind apiserver with staged inert legacy workloads and Ready targets; fast, per-PR, covers the reaper logic incl. storage/metalman reaping. - hack/operator-upgrade-e2e/e2e.py (standalone Python, run by the operator-upgrade-e2e workflow on dispatch + nightly): stands up kind with a local registry, installs the real released v0.1.19 multi-namespace version via that release's `kubectl unbounded site init --manage-cni-plugin=false`, then upgrades via this tree's `kubectl unbounded install` and asserts the reaper migrates net + machina for real (real datapath). Storage (RDMA) and metalman (PXE) can't run in vanilla kind and are excluded there. The driver is self-contained so it runs the same locally and in CI. To build the operator image with registry-hosted component images, images/unbounded-operator/ Containerfile gains NET_CONTROLLER_IMAGE/NET_NODE_IMAGE/MACHINA_IMAGE build-args (default empty; release builds unchanged). The faithful upgrade e2e surfaced three real fixes included here: the operator RBAC escalate/bind (component RBAC install), the net cutover host-port gating above, and deletecollection for the reaper's label-scoped deletes. Each now has a fast per-PR regression guard so it is caught without waiting for the nightly e2e: TestReapOnceReapsNetWhenNewNetNotReady + TestNetTargetsPresent (net gated on creation, not readiness), and TestOperatorClusterRoleAllowsComponentRBACInstall + TestOperatorClusterRoleGrantsReaperDeletes (operator RBAC has escalate/bind and deletecollection for every reapableKinds() type). make fmt/lint/test/build all green; both the in-process and the faithful released-version upgrade e2e pass on kind. * [unbounded-system] Address PR386 review, deprecate net site label, harden migration Address the review findings on PR386, deprecate the net-group site label, and fix the issues surfaced by running the faithful upgrade e2e. Review findings (operator / migration / kubectl / release): - Reconcile is declarative: disabling a component or deleting a Site tears down its workloads (singleton teardown, per-site cleanup, Site finalizer). - Per-site storage gets a per-site ConfigMap the DaemonSet mounts; metalman and storage node-select the canonical site label. - Reaper: gate storage reaping on every storage-enabled Site's DaemonSet; copy Machine cloud-init ConfigMaps and repoint their refs; fold legacy storage config into the translated Site; upsert machina-config so migrated config wins over reconcile defaults. SiteNodeSlice ownerRef uses the v1alpha3 GVK. - Operator namespace is authoritative (--namespace + POD_NAMESPACE); install wires it through. - kubectl: `site init --enable-storage` applies to the remote Site. - Release: build/sign the unbounded-operator image; storage supervisor tag defaults to $(VERSION) (no :latest) with a guard test for embedded :latest. Deprecate net.unbounded-cloud.io/site in favor of unbounded-cloud.io/site: - Net dual-writes both node labels and reads canonical-first with fallback. - Storage / metalman / kube-proxy node-select the canonical label; the managed kube-proxy DaemonSet is recreated on the immutable-selector change. - ValidatingAdmissionPolicy allows the net SA to write the canonical label. - Docs updated. Migration bugs found by the faithful upgrade e2e: - Operator ClusterRole grants update/patch on sites (for the Site finalizer) and on net-group sites (to clear the protection finalizer so the legacy CRD can delete). - Reaper clears finalizers on abandoned legacy net Sites before deleting the legacy Site CRD, avoiding a deletion deadlock on clusters with assigned nodes. - e2e: cluster Site nodeCidr is computed to cover the kind nodes so labeling is exercised; verify asserts the node-label dual-write. Make hostNetwork controller Deployments restart cleanly: - net-controller and per-site metalman use RollingUpdate maxSurge=0 so the old pod frees its host ports before the new one starts; e2e asserts a clean net-controller rollout restart. make build / lint / test pass; the faithful v0.1.19 -> operator upgrade e2e passes on kind. * [unbounded-system] Harden site lifecycle and label migration * [unbounded-system] Make storage config ConfigMap-owned * [unbounded-system] Address review nits: MachinaComponentSpec and Site status conditions Wrap the machina component spec in a dedicated MachinaComponentSpec type (mirroring MetalmanComponentSpec/StorageComponentSpec) so machina-specific fields can be added later without an API break. Replace Site.status.components (a map of custom SiteComponentStatus) with a standard []metav1.Condition. The operator now publishes one condition per component (NetReady, MachinaReady, MetalmanReady, StorageReady) so callers can `kubectl wait --for=condition=NetReady site/<name>`. Regenerated deepcopy and the Site CRD. * [unbounded-system] Fix migration sharp edges from review - Reaper self-deletion guard: the migration reaper filters the target namespace out of its legacy-namespace set so installing with --namespace unbounded-kube (or unbounded-net) can no longer drain and delete the namespace it just migrated into. `kubectl unbounded install` now also rejects a legacy namespace up front. The legacy namespace names and an IsLegacyNamespace helper move to internal/unbounded as the single source of truth shared by the CLI guard and the reaper. - release-upgrade workflow: wait on and summarize the operator-managed workloads in unbounded-system instead of the removed unbounded-kube / unbounded-net namespaces, and wait for each workload to exist before checking rollout status (they are now created asynchronously by the operator). - CLI docs: correct the install --namespace default to unbounded-system and drop the removed --net-namespace, image, and --enable-net flags from cli.md and the ssh / existing-cluster guides. - site init: fix swapped cluster node/pod CIDR validation error messages. Adds regression tests for each fix.
…ration (#464) * [unbounded-system] Operator owns CRD lifecycle and config; harden migration Make the unbounded-operator self-sufficient for CRD lifecycle and move environment-specific settings off Deployment args into a ConfigMap, then harden the legacy migration gates. Operator-owned CRDs: - Add operator.BootstrapCRDs: at startup the operator server-side applies every CRD embedded in the machina and net manifests and waits for each to become Established, before the manager starts (the typed Site informer cannot sync until the Site CRD is served). Idempotent on every start. - `kubectl unbounded install` no longer applies CRDs or takes --skip-crds; it bootstraps the operator and waits for its rollout, then for the CRDs. The reconcile loop skips CRD objects when applying net manifests. Config via ConfigMap: - Add deploy/unbounded-operator/04-configmap.yaml.tmpl (unbounded-operator-config), surfaced to the Deployment via envFrom. --api-server-endpoint and --reap-legacy-resources now default from UNBOUNDED_API_SERVER_ENDPOINT and UNBOUNDED_REAP_LEGACY_RESOURCES (an explicit flag still overrides). install writes api-server-endpoint into the ConfigMap instead of a Deployment arg. Migration hardening: - Detect legacy per-site metalman by Deployment name or canonical/deprecated site label, not a single label match. - Gate metalman reaping on the per-site replacement Deployment existing (presence gate, like net: hostNetwork host-port ownership). - Require storage DaemonSets to actually schedule (DesiredNumberScheduled >= 1 and observed generation current) before reaping the legacy supervisor, so a zero-desired DaemonSet no longer reads as ready. - Delete the legacy net-group Site CRD only after the new net controller is Available, so the GC does not transiently drop SiteNodeSlices mid-cutover. - Warn and emit an Event when a drained legacy namespace still holds non-operator workloads before the whole-namespace delete. - Reconcile Sites on generation changes only, so net/machina manifests are not re-applied on status-only Site updates. - Retarget the namespace as a substring (SA usernames, flag values), not just on exact-value equality. Tests: CRD bootstrap, env-var flag defaults, metalman detection/presence gate, storage readiness gates, net-controller CRD gate, foreign-workload detection; extend the upgrade e2e to assert the operator (not install) established and owns the Site CRD and that the operator-config ConfigMap is present. * [unbounded-system] Address install/operator review findings Fixes from PR review of the operator CRD-lifecycle/config work. 1. install no longer reports success against an unavailable operator. waitForOperator gated on AvailableReplicas>=desired, which the old ReplicaSet satisfies while a new crash-looping pod surges in during an upgrade. Replace it with a full rollout-complete check (ObservedGeneration current and Replicas == UpdatedReplicas == AvailableReplicas == desired), the same condition `kubectl rollout status` waits on. 2. Liveness no longer kills the operator mid-bootstrap. BootstrapCRDs runs before the manager starts the health server, so /healthz is unserved for up to crdEstablishedTimeout (~2m) while liveness (10s + 3x15s) would restart at ~40s. Add a startupProbe with a 150s budget (30 x 5s) that exceeds the bootstrap timeout and suppresses liveness until the health server binds. 3. Re-install with a changed endpoint now rolls the operator. install wrote the endpoint only into the ConfigMap, but envFrom is read once at pod start and the pod template was unchanged, so no rollout occurred. Stamp a config-hash annotation (sha256 of the endpoint) onto the Deployment pod template; a changed endpoint changes the template and triggers a rollout that the strengthened waitForOperator waits on. 4. The make/GitOps render path can set the endpoint again. Restore .APIServerEndpoint into the operator-config ConfigMap and re-add the UNBOUNDED_OPERATOR_API_SERVER_ENDPOINT Makefile variable / --set. The Deployment template stamps the same config-hash (sprig sha256sum) so raw `kubectl apply` of re-rendered manifests also rolls on an endpoint change. --skip-crds stays removed (CRDs are operator-owned); documented as a breaking change in the CLI reference. Tests: rollout-complete table + waitForOperator upgrade/complete cases; config-hash stamped and endpoint-sensitive; BootstrapCRDs applies all 12 embedded CRDs and blocks until Established; a render test asserting the ConfigMap endpoint and the matching Deployment hash annotation stay in sync. * [unbounded-system] Harden operator reconciliation and migration
…eate races (#473) Set SiteNodeSlice owner references to blockOwnerDeletion:false (and accept nil/false while repairing true) so the controller no longer needs sites/finalizers RBAC to write slices. Guard every slice, status, and finalizer write with a live GET plus an expected-UID check so a Site that is deleted and recreated under the same name is never mutated with stale data, and delete extra or orphaned slices with UID + ResourceVersion preconditions. Preserve foreign metadata by mutating a deep copy of the live slice instead of rebuilding it, propagate errors up the reconcile chain so a failed pass re-dirties and retries rather than silently dropping work, aggregate per-site status/finalizer errors, count gateway nodes in the Site status, convert node string slices to []interface{} for valid unstructured content, clean up orphaned slices whose Site is gone, and wire SiteNodeSlice informer events to mark slices dirty.
… harden the legacy reaper (#474) Route every destructive migration read through an uncached APIReader and make namespaceExists fail closed, so the reaper never reaps based on stale cache data. createIfAbsent now Gets into a freshly zeroed object and compares the complete payload before treating an existing target as migrated, closing a silent data-loss path where the real client merges Get results into the desired object's maps. copyConfigMapByName refuses to leave a dangling cloud-init reference, translateSite requires spec-equivalence and is re-verified live before the legacy Site CRD is deleted, and the machina/net/ storage readiness gates re-check ResourceVersion + full-payload hash to avoid TOCTOU reaping during an in-flight rollout. Replace the config.yaml-only hash with configMapPayloadHash over the complete Data + BinaryData, gate per-site storage config, require UpdatedNumberScheduled on DaemonSet readiness, broaden the foreign-workload audit to pods, RCs, RSs, jobs, and cronjobs (fail closed, skipping controller-owned and terminating legacy descendants), copy the Secret Immutable field, and fix the inverted SiteNodeSlice owner-reference predicate that otherwise stalled migration. On the reconciler side, preserve a live unbounded-net-config instead of force-applying embedded defaults, stamp the config hash on both net workloads, watch all managed ConfigMaps and singleton workloads, reconcile retained net/machina without recreating them from nothing, and patch machina/storage config with optimistic locking. Adds the foreign-workload audit RBAC grants. The reaper migration e2e (reaper_e2e_test.go) travels with this change: it now stamps the full-payload config hash on the staged target workloads and drives a restricted-RBAC SiteController owner-repair, exercising the new gate behavior.
…install repair (#475) Add a leader-elected CRDMaintainer that re-applies and re-establishes the complete owned CRD set on an interval, so a deleted or drifted CRD is repaired without waiting for a pod restart. Transient maintenance failures are logged and retried on the next interval; only a run of consecutive failures (MaxFailures, default 3) stops the manager, letting the pod restart after any terminating CRD has finished deleting. Introduce the exported RequiredCRDNames as the single source of truth (all 12 CRDs) and wait on the full set at bootstrap; reject CRDs carrying a deletionTimestamp as established in both the operator and the install client so a terminating CRD is never treated as healthy. kubectl unbounded install now preflights/waits on the complete RequiredCRDNames set, hashes the full operator ConfigMap payload (not just the endpoint) so a change to UNBOUNDED_REAP_LEGACY_RESOURCES rolls the operator, preserves the live reaper value via strconv.ParseBool and fails closed on an invalid value before applying, and repairs missing CRDs against an already-healthy operator via an idempotent repair token that only rolls when CRDs are unhealthy and the prior rollout is complete. Pin the operator Deployment to the Recreate strategy with rollingUpdate nulled so rollout completion is unambiguous, template the reap flag, and thread it through the Makefile and render tests.
) test: faithful upgrade e2e coverage and net paths for the operator e2e workflow Extend the faithful upgrade e2e (hack/operator-upgrade-e2e/e2e.py) to assert byte-for-byte preservation of the migrated unbounded-net-config payload and its config hash on both net workloads, that a BinaryData-only change rolls both workloads via the managed ConfigMap watch, and that an identical same-release reinstall repairs a deleted CRD by rotating the repair token and restarting the operator without changing its image or config hash. Add the net controller and net CRD paths to the operator reaper e2e workflow triggers so future changes to internal/net/controller and deploy/net/crd exercise the migration e2e. The reaper migration e2e itself now lives with the reaper change (the operator config/migration PR); the upgrade e2e here runs on demand and nightly.
Integrate 48 commits of main into the operator/unified-namespace umbrella branch ahead of PR #388. Conflict resolutions: - Makefile: union both target sets (unbounded-operator/inventory + machine-ops/ playpen/agent-artifacts). Keep the branch's `./...` test/build/vulncheck recipes rather than main's $(GO_PACKAGES) so deploy/ render tests still run. - .github/workflows/release.yaml: metalman moved to its own release job on main; keep the new unbounded-operator entry in the component-images matrix. - cmd/kubectl-unbounded/app/deploy_pxe.go: keep deleted (the standalone deploy-pxe command was removed; the operator manages PXE via Site components.metalman). See follow-up re: main's #389 --default-netboot-image. - cmd/metalman/README.md: drop the obsolete `site deploy-pxe` flags section. - deploy/gantry/README.md: keep the templated (*.yaml.tmpl / unbounded-system) docs; add main's node-config.yaml row. - docs/content/reference/machina-crd.md, hack/smoke-metalman.py, hack/agent/e2e-kind/e2e.py: take main's newer content and re-apply the namespace unification / operator additions (ASCII hyphens per AGENTS.md). Apply the unbounded-kube/unbounded-net -> unbounded-system namespace unification to main's new operational content (nightly.yaml, release-upgrade.yaml, setup-nightly-cluster.sh) so the post-deploy checks match where the unified manifests and hack/orca/deploy-integration.sh now install components. make build / lint / test pass; go generate ./api/... is a no-op (deepcopy/CRDs consistent); eBPF generate requires llvm in CI.
Collaborator
Author
Follow-ups from the
|
…info (#480) * build: sanitize the git-derived VERSION for Docker image tags git describe can surface a nearest tag containing a slash (e.g. agent-artifacts/v20260710), which is invalid in a Docker image reference and breaks local/e2e image builds ("invalid reference format") and the image refs in rendered manifests. Add VERSION_TAG (VERSION with '/' replaced by '-') and use it for every image tag, while keeping VERSION intact for the embedded version string (ldflags) and release artifact paths. Unifies the existing one-off PLAYPEN_TAG sanitization. * operator: auto-discover the API server endpoint from kube-public/cluster-info The operator no longer requires UNBOUNDED_API_SERVER_ENDPOINT to be configured. At startup it resolves the endpoint from the standard kube-public/cluster-info ConfigMap (the server field of the embedded kubeconfig); an explicit --api-server-endpoint / $UNBOUNDED_API_SERVER_ENDPOINT override still wins. With neither an override nor a discoverable value the operator fails hard, since machina and metalman cannot function without an endpoint. kubectl unbounded install stops force-populating the endpoint: it writes a value only when --api-server-endpoint is given, otherwise preserving any existing value and dropping the kubeconfig-host fallback so auto-discovery takes effect. Extract the cluster-info reader shared with metalman into a new internal/clusterinfo package and migrate internal/metalman/commands to it. The operator's existing cluster-wide configmaps RBAC already permits reading kube-public/cluster-info, so no RBAC change is required. * test: fix layered HTTP metalman smoke to use unbounded-system namespace The layered DHCP-free HTTP smoke test (smoke-metalman-http.py) still resolved the metalman-controller ServiceAccount kubeconfig from the legacy unbounded-kube namespace, while the rendered metalman RBAC now creates that ServiceAccount in unbounded-system. The token request therefore failed with a non-zero exit before metalman could start, breaking the layered-http-smoke job whenever the metalman smoke workflow ran. Reuse the shared smoke.METALMAN_NAMESPACE constant (unbounded-system) so the HTTP smoke stays in lockstep with the PXE smoke and the namespace rename.
This was referenced Jul 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
unbounded-operatoris the future of installing and upgrading Unbounded. Instead of applying a stack of per-component manifests spread across several namespaces, you install one operator. It reconciles every Unbounded component into a single, configurableunbounded-systemnamespace, driven bySitecustom resources, and it automatically migrates existing clusters off the legacy split-namespace layout - copying the state that cannot be regenerated and reaping the old namespaces once the new workloads are healthy.This is the umbrella PR that lands the whole
unbounded-systemeffort onmain.Why
Unbounded historically installed across several namespaces (
unbounded-kubefor machina/metalman/storage,unbounded-netfor net, plus orca/inventory/machine-ops/gantry). The split is historical, not meaningful: it fragments RBAC, discovery, NetworkPolicy, and day-2 ops, and leaves no single namespace that represents "Unbounded" on a cluster. This consolidates everything onto one configurable namespace and provides a safe, automated path to move existing clusters onto it (Kubernetes namespaces cannot be renamed in place, so a deliberate migration is required).What it does
unbounded-system. The namespace is configurable and POD_NAMESPACE-aware end to end (internal/unbounded.SystemNamespace()).kubectl unbounded installdeploys the operator; the operator owns CRD lifecycle and installs components at its own version. Components are declared per location via theSiteAPI rather than hand-applied manifests.Architecture
api/machina/v1alpha3/site_types.go): cluster-scopedSitein the machina group carries the networking spec plusspec.components(machina / metalman / storage). Networking (unbounded-net) is not a component; it is a cluster singleton the operator runs whenever any Site exists.cmd/unbounded-operator,internal/operator):bootstrap.go).SiteReconciler: renders and owns component workloads from embedded manifests intounbounded-system(reconciler.go).LegacyReaper: the migration routine (migrate.go).config_map.go,machina_config.go).kube-public/cluster-infodiscovery (internal/clusterinfo).Migration, in order (LegacyReaper)
sites.net.unbounded-cloud.iointosites.unbounded-cloud.io, copying the networking spec verbatim and inferringspec.componentsfrom the running legacy workloads. Existing machina-group Sites are never clobbered.machina-config, per-Site storage config) intounbounded-system; skip regenerable/auto-managed secrets (net serving cert, SA tokens, Helm release secrets).passwordRef, MachineOperationCredentialsecretRef) and copy each Machine's cloud-init user-data ConfigMap intounbounded-system, repointing the reference.Reviewer's guide
This diff is large (~230 files); start here, in order:
api/machina/v1alpha3/site_types.go- the Site + components contract everything is built on.internal/operator/reconciler.go- how components are rendered/owned; net singleton vs per-Site components; site-label node-selection (canonical vs deprecated label).internal/operator/migrate.go- the reaper; ordering + per-component health gate are the safety-critical parts. Tests:migrate_test.go,migrate_gates_test.go.internal/operator/bootstrap.go- operator-owned CRD install/maintenance.cmd/unbounded-operator/main.go- wiring: CRD bootstrap -> manager -> SiteReconciler -> reaper; endpoint resolution.internal/unbounded/unbounded.go- namespace resolution + the drift guard keeping Go/Makefile/templates in agreement.cmd/kubectl-unbounded/app/install*.go,site_init.go- the consolidated install/site UX.deploy/unbounded-operator/**and the retargeteddeploy/{net,machina,...}manifests/embeds.internal/net/**,api/net/v1alpha1/**(Site type move, controller/client namespace + label changes).Testing
internal/unbounded/*_test.go).e2e/operator/reaper_e2e_test.go,//go:build e2e): drives the reaper against a kind apiserver with staged legacy workloads; run byoperator-e2e-kind.hack/operator-upgrade-e2e/e2e.py): installs the last released multi-namespace version, upgrades via this tree'skubectl unbounded install, and asserts the operator migrates net + machina for real; run byoperator-upgrade-e2e(nightly + dispatch).make build/make lint/make testgreen.