Run ate-api-server in HA mode#538
Open
Julian Gutierrez Oschmann (juli4n) wants to merge 5 commits into
Open
Conversation
Julian Gutierrez Oschmann (juli4n)
force-pushed
the
replicas
branch
from
July 25, 2026 17:54
803cf91 to
bf29da6
Compare
We have a few places where we need to do port-forwarding to Pods behind a service. Consolidate this logic under `internal/portforward`.
This demo is broken now that we require authn to ate-apiserver. The fix is not trivial as it requires the actor to be able to authenticate, plus it doesn't really adds much.
Holding state in memory means that we cannot scale to more than one replica. Change the mock implementation to be stateless. It doesn't really need to track state anyways.
A ClusterIP Service load-balances per TCP connection, not per RPC, but gRPC multiplexes many RPCs over one long-lived HTTP/2 connection, so a client that dials once (e.g. ate-controller, atenet) gets pinned to whichever single pod that connection was routed to, for its entire lifetime. Make the "api" Service headless so DNS returns one A record per ready pod instead of a VIP, and switch clients to a "dns:///" dial target with the round_robin load balancing policy so RPCs actually spread across every replica.
Julian Gutierrez Oschmann (juli4n)
force-pushed
the
replicas
branch
from
July 25, 2026 18:08
bf29da6 to
e93b02c
Compare
This is the minimum HA setup we can run. Configure rolling update policy to recreate replicas one by one, make before break. Also configure PDBs so that we never lose more than one replica during pod eviction events.
Julian Gutierrez Oschmann (juli4n)
force-pushed
the
replicas
branch
from
July 25, 2026 18:35
e93b02c to
321dcdb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR updates
ate-apiserverdeployment to run with 2 replicas.Some changes were needed to enable this, and are separate in a series of independent commits.
kubectl-atebinary was not checking for Pod readiness. With multiple replicas, there is a risk of picking a not ready Pod and failing the request.agent-secretdemo: This demo is broken now thatate-apiserverrequires authn. I don't see an easy way to fix it, and the demo was not really adding much value.ate-apiserverdeployment toreplicas: 2: Additionally, configure the deployment rolling update so that we update one instance at a time (make before break) and PDB so that we never lose more than one replica during eviction events.Part of #181