Migrate stellar-core Ingress to Gateway API HTTPRoute via a scalable nginx proxy by Jonathan-Eid · Pull Request #405 · stellar/supercluster

Jonathan-Eid · 2026-07-09T17:09:59Z

What

Migrate the per-mission stellar-core Ingress → Gateway API HTTPRoute, fronted by a scalable per-mission nginx proxy:

Replace the per-pod ExternalName/ClusterIP services + fat multi-rule Ingress with one nginx proxy (Deployment + Service + ConfigMap) behind a single-rule HTTPRoute. The proxy demuxes /<pod>/core|history to the pod's in-cluster DNS, so the route needs one rule regardless of node count.
Proxy replicas auto-scale ceil(nodes/64), capped by --http-proxy-replicas (default 10, floor 1), with a readiness wait before the driver routes through it.
Convert the parallel-catchup job-monitor Ingress → HTTPRoute.
Bump KubernetesClient 15 → 17 (+ typed HTTPRoute models in CSLibrary/GatewayApi.cs).
Rename internal ingress* identifiers → route* (CLI flag strings kept for Jenkinsfile compat); drop the now-dead --ingress-class.

Why

The old ExternalName-per-pod + nginx-ingress design overloaded CoreDNS (per Improve ingress handling #197) — the ingress controller re-resolving N ExternalNames + NXDOMAIN during pod startup. The proxy creates zero ExternalName services.
HTTPRoute has a hard 16-rule cap; a one-rule-per-pod route breaks past ~8 nodes. The single-rule proxy route is unaffected at any scale.

Testing

Build + unit tests (11/11) green.
Validated live on a test cluster: SimplePayment, HistoryTestnetRecentCatchup, ComplexTopology (7 nodes), SlowNodesNetwork (10), a 3-way parallel batch (survey + slow-nodes + complex, isolated per-nonce), VersionMix/DBUpgrade parallel, HistoryGenerateAndCatchup (sequential multi-network), and SimulatePubnetMixedLoad provisioning at 91 nodes → proxy auto-scaled to 2 replicas.

🤖 Generated with Claude Code

Replace the nginx Ingress (ToIngress) with an HTTPRoute (ToHttpRoute) attached to the shared traefik-gateway-private gateway. Per-pod /<pod>/core|history PathPrefix matches with a ReplacePrefixMatch URLRewrite (equivalent to the old rewrite-target). - Add typed Gateway API models (GatewayApiModels) in CSLibrary; official KubernetesClient has no Gateway API types. - Create/track/delete + orphan-sweep the HTTPRoute via the custom-object API. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Highest version before the 18.x model-constructor breaking change (parameterized ctors removed). Builds clean + tests pass with no code changes; 18/19 would need a repo-wide constructor migration, left as separate work. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Traefik's Gateway provider rejects ExternalName services as HTTPRoute backends ('type ExternalName is not supported'), unlike the nginx Ingress path. Select the target pod via its statefulset.kubernetes.io/pod-name label instead. Works for both gateway and ingress; verified live on ssc-test (core /info + protocol upgrade route through the gateway). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The generated object is now an HTTPRoute, not an Ingress; name it <nonce>-stellar-core-http-route.

…e rename Replace the per-pod ClusterIP services + fat multi-rule HTTPRoute with a single per-mission nginx proxy (Deployment + Service + ConfigMap) behind a one-rule HTTPRoute. The proxy demuxes /<pod>/core|history to the pod's in-cluster DNS name, so the route needs one rule regardless of node count -- fixing the HTTPRoute 16-rule cap and the CoreDNS amplification of the old ExternalName-per-pod design. Replica count auto-scales ceil(nodes/64), capped by --http-proxy-replicas (default 10, floor 1), with a readiness wait before the driver routes through it. Rename internal ingress* identifiers -> route* (CLI flag strings kept for Jenkinsfile compatibility); drop the now-dead --ingress-class. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

socket-security · 2026-07-09T17:10:24Z

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	kubernetesclient@15.0.1 ⏵ 17.0.14		⁺²

View full report

Copilot

Pull request overview

Migrates Supercluster’s per-mission stellar-core HTTP exposure from per-pod Ingress + ExternalName services to a Gateway API HTTPRoute that fronts a scalable per-mission nginx proxy, and updates related mission/job-monitor routing and Kubernetes client dependencies.

Changes:

Replace per-pod service + multi-rule Ingress with an nginx proxy (ConfigMap/Deployment/Service) behind a single-rule HTTPRoute, plus a readiness wait before routing through the proxy.
Migrate parallel-catchup job-monitor from Ingress to HTTPRoute and adjust Helm values and mission set-options accordingly.
Add typed Gateway API models (GatewayApiModels) and bump KubernetesClient from 15.x to 17.x; rename ingress-centric fields/identifiers to route-centric equivalents and add --http-proxy-replicas.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
src/MissionParallelCatchup/parallel_catchup_helm/values.yaml	Switch job-monitor routing config from ingress fields to Gateway API parent refs + path prefix.
src/MissionParallelCatchup/parallel_catchup_helm/templates/job_monitor.yaml	Replace Ingress with `HTTPRoute` and URLRewrite-based prefix stripping.
src/FSLibrary/StellarSupercluster.fs	Create proxy resources + `HTTPRoute` and wait for proxy readiness before proceeding.
src/FSLibrary/StellarOrphanSweep.fs	Switch orphan sweeping from Ingress to `HTTPRoute` via `GenericClient`.
src/FSLibrary/StellarNetworkCfg.fs	Introduce route/proxy naming and labels; rename ingress host fields to route host fields.
src/FSLibrary/StellarNamespaceContent.fs	Track and delete `HTTPRoute` resources instead of Ingress in namespace cleanup.
src/FSLibrary/StellarMissionContext.fs	Replace ingress-related mission context fields with route fields and add `httpProxyReplicas`.
src/FSLibrary/StellarKubeSpecs.fs	Add proxy ConfigMap/Deployment/Service generation and new `HTTPRoute` creation logic.
src/FSLibrary/StellarCoreHTTP.fs	Update Host header + URL construction to use route host fields instead of ingress host fields.
src/FSLibrary/StellarCoreCfg.fs	Add constants for Gateway target and proxy config file naming/ports.
src/FSLibrary/MissionHistoryPubnetParallelCatchupV2.fs	Update Helm set-option key from `monitor.path` to `monitor.path_prefix`.
src/FSLibrary/FSLibrary.fsproj	Bump KubernetesClient dependency to 17.0.14.
src/FSLibrary.Tests/Tests.fs	Update `MissionContext` test construction for route fields + proxy replicas.
src/CSLibrary/GatewayApi.cs	Add typed POCO models for `HTTPRoute`/`HTTPRouteList` used with `GenericClient`.
src/CSLibrary/CSLibrary.csproj	Bump KubernetesClient dependency to 17.0.14.
src/App/Program.fs	Remove `--ingress-class`, rename options to route fields, and add `--http-proxy-replicas` (while keeping ingress-* flag strings).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- waitProxyReady: treat null Deployment.Status as 0 ready (status is populated asynchronously; don't NPE mission startup). - Orphan sweep: also reap legacy Ingresses (pre-migration orphans), alongside HTTPRoutes. - job_monitor HTTPRoute: omit spec.hostnames when monitor.hostname is empty (avoids invalid hostnames: [""]). - Clarify --ingress-{internal-domain,external-host,external-port} help text to describe the gateway route (flag names kept for Jenkinsfile compat). - Run fantomas over the changed files (fixes the failing format check). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Replace the nginx-ingress setup with gateway-api CRDs + traefik as the Gateway API controller + a traefik-gateway-private Gateway (matching CfgVal), so the migrated BootAndSync smoke mission (which creates an HTTPRoute) works. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…o IngressClass) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…amespace) Restore the configurability the old --ingress-class flag provided: instead of hardcoding traefik-gateway-private/traefik in CfgVal, take --gateway-name and --gateway-namespace (same defaults). Lets each cluster / CI point the route at whits own gateway. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The migrated BootAndSync creates an HTTPRoute, so CI needs a gateway-api env. Traefik-in-k3s proved a version-compat maze (bundled-traefik conflict, chart value drift, missing TLSRoute CRD, CEL/k8s-version mismatches). Envoy Gateway ships matching CRDs + controller in one chart and programs cleanly. Recipe validated on local k3d (k3s 1.31, bundled traefik disabled, envoy-gateway v1.2.6, Gateway 'ssc-gateway'): BootAndSync brought up 3 nodes and synced, driver reaching them via the gateway -> proxy. Uses the new --gateway-name / --gateway-namespace flags. Also bump checkout@v2->v4, setup-dotnet@v1->v4. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Use 'use ... = new GenericClient(...)' so the client is disposed and the FS0760 IDisposable warning is cleared. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The previous 'use gc = new GenericClient' disposed the shared mission kube client (GenericClient wraps it), NPEing the next API call. Use 'let ... = new': clears FS0760 via the 'new' keyword without disposing the shared client. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

sisuresh · 2026-07-10T17:23:51Z

Posting some claude review comments

Route-programming latency isn't awaited (low-med). waitProxyReady (StellarSupercluster.fs:983) waits for the proxy pod, but not for the gateway to program the HTTPRoute. There's a window where the pod is Ready but the gateway still 404s. The comment says the wait exists "so startup doesn't spend its retry budget on 502s" — yet route-programming 404s can still eat that budget. Consider also polling the HTTPRoute's Accepted/Programmed status condition, or note explicitly that mission retries cover it.

Proxy Deployment has no readiness probe (low). With no probe, ReadyReplicas flips to 1 the moment the container process starts — before nginx is guaranteed to be listening. A one-line readinessProbe: tcpSocket: {port: 80} would make waitProxyReady's guarantee real instead of relying on downstream retries.

forceCleanNamespace isn't guarded like the startup sweep (low). The per-kind list() runs outside sweepKind's try/catch (StellarOrphanSweep.fs:31), so on a cluster lacking the gateway-api CRD the HTTPRoute list throws and aborts the remaining kinds (Job/DaemonSet). The auto-sweep is fine; only the explicit recovery verb is exposed.

"Auto-scale" is compute-once (nit). Replicas are fixed at creation from MaxPeerCount (ceil(n/64), clamped [1,cap]) — no live HPA. Fine for this model, but "auto-scale" in the comment/PR text slightly oversells it.

Resolver shim fails hard if resolv.conf has no nameserver (nit). sed s/RESOLVER/$R/ with empty $R yields an invalid resolver directive → CrashLoop → waitProxyReady fails after ~2 min. Effectively impossible in-cluster, but a fallback IP or fail-fast message would be cheap insurance.

One operational dependency to confirm before merge: the production traefik-gateway-private Gateway (ns traefik) must have a listener whose allowedRoutes.namespaces admits all mission namespaces (validated in ns jonathan, but parallel-catchup and other namespaces should be confirmed). If it's restricted by namespace/label selector, routes silently won't attach.

- Add tcpSocket:80 readinessProbe to the http-proxy Deployment so ReadyReplicas reflects nginx actually listening (waitProxyReady's guarantee is now real). - waitProxyReady: after pod-ready, best-effort wait for the HTTPRoute to be Accepted by the gateway (non-fatal), closing the pod-ready -> route-programmed 404 window. Adds minimal HTTPRoute.Status model. - Orphan sweep: guard sweepKind's list() in try/catch so a missing CRD on a cluster skips that kind instead of aborting the whole sweep. - http-proxy resolver shim: fail fast if /etc/resolv.conf has no nameserver. - Reword proxy replica comment (scales with node count at creation, not a live HPA). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.

The job-monitor chart values didn't propagate --gateway-name/--gateway-namespace, so its HTTPRoute always used the values.yaml defaults (traefik-gateway-private/ traefik) and would attach to the wrong Gateway on any cluster using a different one. Mirror the core route: pass monitor.gateway_name/gateway_namespace from the mission context. Verified on ssc-test: job-monitor route parentRef followed --gateway-name/--gateway-namespace. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Dismissing so Graydon can take a look

graydon

I'm .. cautiously optimistic that this ought to work. I don't really understand why the previous config was generating so much DNS traffic relative to this one, but if you prefer this operational configuration I won't really complain.

I'd like to add a doc explaining the flow of traffic with a diagram (that doesn't need to block merging this, I can write it up) but more importantly I think we need some kind of solution to the prometheus exporter. Possibly it's already solved? If so, could you say how? Does the prometheus scraper just find all the services exposing that port automatically somehow?

graydon · 2026-07-10T22:20:35Z

+    location ~ ^/([^/]+)/core(?:/(.*))?$ {
+        proxy_pass http://$1.%s:%d/$2$is_args$args;
+    }
+
+    location ~ ^/([^/]+)/history(?:/(.*))?$ {
+        proxy_pass http://$1.%s:%d/$2$is_args$args;
+    }
+}
+"""
+                fqdn
+                CfgVal.httpPort
+                fqdn
+                CfgVal.historyPort


I'm not sure if (before this PR) the prometheus scraper hits the services directly, or hits the ExternalNames, or hits the ingress somehow. I think it hits the ExternalNames, which you're removing, so .. maybe it needs to be proxied through here somehow? Or maybe something else? See CfgVal.prometheusExporterPort and the way it gets added to the ExternalName ports in the pre-state of this PR.

graydon · 2026-07-10T22:21:39Z

-                if self.missionContext.exportToPrometheus then
-                    Array.append ports [| V1ServicePort(name = "prom-exp", port = CfgVal.prometheusExporterPort) |]
-                else


This bit has to get handled .. somehow. I don't quite know where/how, but it's pretty important that we continue being able to scrape supercluster runs to prometheus.

Jonathan-Eid and others added 6 commits July 2, 2026 14:35

Rename IngressName -> HttpRouteName; drop -ingress from the object name

f7871ce

The generated object is now an HTTPRoute, not an Ingress; name it <nonce>-stellar-core-http-route.

Convert parallel-catchup job-monitor Ingress -> HTTPRoute

ab49e35

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot AI review requested due to automatic review settings July 9, 2026 17:10

Copilot started reviewing on behalf of Jonathan-Eid July 9, 2026 17:10 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Jonathan-Eid and others added 13 commits July 9, 2026 13:20

Drop --ingress-class from CI workflow + docs (flag removed)

a357372

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

CI: bump k3s to v1.28.5 (traefik chart needs k8s >=1.25)

d17c2af

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

CI: isolate our traefik from k3s bundled traefik (fullnameOverride, n…

1ba2221

…o IngressClass) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Dispose the gateway GenericClient in the HTTPRoute orphan sweep

12290dd

Use 'use ... = new GenericClient(...)' so the client is disposed and the FS0760 IDisposable warning is cleared. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Refactor comments in NetworkCfg to improve clarity and remove redundancy

a8e1a13

Strip trailing whitespace so fantomas --check passes

52611e5

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Drop redundant comments in orphan sweep

b6d0096

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Merge branch 'main' into jonathan/ingress-to-gateway-httproute

67242f4

Jonathan-Eid requested a review from Copilot July 10, 2026 18:06

Copilot started reviewing on behalf of Jonathan-Eid July 10, 2026 18:06 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Comment thread src/FSLibrary/StellarKubeSpecs.fs

Comment thread src/FSLibrary/MissionHistoryPubnetParallelCatchupV2.fs

Comment thread .github/workflows/build-and-test.yml

sisuresh previously approved these changes Jul 10, 2026

View reviewed changes

graydon requested changes Jul 10, 2026

View reviewed changes

graydon mentioned this pull request Jul 11, 2026

add ingress/gateway doc #407

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Migrate stellar-core Ingress to Gateway API HTTPRoute via a scalable nginx proxy#405

Migrate stellar-core Ingress to Gateway API HTTPRoute via a scalable nginx proxy#405
Jonathan-Eid wants to merge 21 commits into
stellar:mainfrom
Jonathan-Eid:jonathan/ingress-to-gateway-httproute

Jonathan-Eid commented Jul 9, 2026 •

edited

Loading

Uh oh!

socket-security Bot commented Jul 9, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sisuresh commented Jul 10, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

graydon left a comment

Uh oh!

graydon Jul 10, 2026

Uh oh!

graydon Jul 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

Jonathan-Eid commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Testing

Uh oh!

socket-security Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sisuresh commented Jul 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

graydon left a comment

Choose a reason for hiding this comment

Uh oh!

graydon Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

graydon Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Jonathan-Eid commented Jul 9, 2026 •

edited

Loading

socket-security Bot commented Jul 9, 2026 •

edited

Loading