feat(gatewayapi): calico-system policy for namespaced data-plane proxies - #4970
feat(gatewayapi): calico-system policy for namespaced data-plane proxies#4970electricjesus wants to merge 3 commits into
Conversation
d3d69a5 to
311699b
Compare
311699b to
fe034ab
Compare
electricjesus
changed the title
| // not calico-system, so they need their own policy in the calico-system tier. EnvoyProxy | ||
| // stamps gateway.envoyproxy.io/owning-gateway-name on every proxy pod, so use it as the selector. | ||
| ProxyPolicyName = networkpolicy.CalicoComponentPolicyPrefix + "envoy-gateway-proxy" | ||
| EnvoyProxyPolicySelector = "has(gateway.envoyproxy.io/owning-gateway-name)" |
There was a problem hiding this comment.
Could we set some calico labels on these objects and use those for policy?
Related: do the envoy related components have standard labels that we set in utils/component.go? If not,that is a gap.
There was a problem hiding this comment.
did both, thanks.
The selector keys off a label we own now. Proxy pods get k8s-app: calico-gateway-api-proxy, stamped through the EnvoyProxy pod spec (ensureGatewayProxyLabel), and calico-system.envoy-gateway-proxy selects on that instead of gateway.envoyproxy.io/owning-gateway-name. Controller and certgen keep the labels they had, just set from one place now (setGatewayComponentLabel).
On your component.go question: you were right, there was a gap, and it's fixed here too.
setStandardSelectorAndLabels does set the app.kubernetes.io/* labels on any object with a CR, so the controller Deployment, the certgen Job and the EnvoyProxy all get those. But the pod-template pass below it only handles Deployment/DaemonSet/Prometheus/Alertmanager, so Jobs hit the default case and return early and their pods miss the lot. Not an envoy thing, it's all four Jobs we render. Added *batchv1.Job to that switch.
Two bits worth flagging on that one. It labels the pod template only, since a Job's selector belongs to the job controller and is immutable, so we can't do the Spec.Selector thing the Deployment case does. And existing Jobs don't get rewritten, because mergeState compares Jobs on container images and pod-template annotations rather than labels. No churn on upgrade, but it does mean a Job already on a cluster keeps its old template until something recreates it. Seemed like the better trade than recreating four Jobs on every upgrade, though shout if you'd rather it were immediate..
The proxy is the one case the labeller genuinely can't reach. It derives the label from the object name, so we'd get envoy-gateway rather than a stable calico-gateway-api-*, and the proxy pods are created by the EG controller at runtime so we never render them to label. The EnvoyProxy pod spec is the only hook we own. That's why the k8s-app labelling still sits in the gateway render, with a comment saying why.
One bit of luck: the pod-template pass only sets k8s-app when it's empty, so our explicit labels survive it.
ed7bc96 to
c549093
Compare
Since deploy.type=GatewayNamespace (tigera#4690) the data-plane envoy proxies run in each Gateway's own namespace, not calico-system. The only calico-system gateway policy selects the controller/certgen pods in calico-system, so the proxies match nothing and have no policy punching through a default-deny tier in the namespaces they now run in. Add a GlobalNetworkPolicy selecting the EG proxy pods (label gateway.envoyproxy.io/owning-gateway-name) so it covers every Gateway namespace with no re-render: DNS + xDS(18000)/Wasm(18002) egress to the controller in calico-system, and all inbound TCP so a managed Gateway serves traffic out of the box under a default-deny tier. Backend egress is left to the user, matching the controller policy.
Per review: instead of selecting the data-plane proxy pods on Envoy Gateway's gateway.envoyproxy.io/owning-gateway-name label, which we do not own and which could change upstream without notice, stamp our own k8s-app=calico-gateway-api-proxy label on the proxy pods through the EnvoyProxy pod spec and point the calico-system-tier proxy policy at that. Any user-supplied pod labels from a custom EnvoyProxy are kept.
Put the Calico-owned k8s-app labeling for all three envoy-gateway components in one documented place. The operator-rendered controller and certgen pods go through setGatewayComponentLabel; the runtime-created proxy pods, which the operator never renders, go through the EnvoyProxy pod spec (ensureGatewayProxyLabel). Documents why this lives in the gateway render and not the standard labeler in component.go, which keys off the object name and cannot reach a runtime-created pod.
f50b72f to
1d372cd
Compare
… cover tigera#2873 replaced the obsolete tigera-gateway namespace exclusion with a reference NetworkPolicy that users apply in each Gateway namespace. That policy allows DNS and the gateway controller hop itself. With tigera/operator#4970 the operator renders calico-system.envoy-gateway-proxy, which already allows inbound traffic to the proxy plus proxy egress to DNS and to the controller on 18000 and 18002. The reference policy would duplicate rules the operator now owns. What the operator's policy deliberately does not cover is the backend hop. It ends its egress rules with a Pass so the user keeps control of which backends the proxy may reach. Under default deny that hop stays denied, and the gateway returns 503 after the upstream connect timeout. So the step now covers only the user's part: - proxy egress to the backend, and backend ingress from the proxy. Both are needed, because default deny applies to the backend pod too. - on Enterprise, egress to the Kubernetes API server, for the log collector that runs in the proxy pod. The example selects proxies by k8s-app == "calico-gateway-api-proxy", the Calico-owned label that #4970 stamps on the proxy pod template.
There was a problem hiding this comment.
Pull request overview
This PR updates the Gateway API rendering to ensure Envoy Gateway data-plane proxy pods (which run in per-Gateway namespaces in namespaced mode) are covered by a calico-system-tier policy, so they can function under namespace default-deny.
Changes:
- Add a
GlobalNetworkPolicy(calico-system.envoy-gateway-proxy) that selects proxy pods via a Calico-stampedk8s-applabel and allows required proxy ingress/egress under default-deny. - Stamp a Calico-owned
k8s-applabel onto proxy pods via theEnvoyProxypod spec, and refactor controller/certgen label stamping into a helper. - Extend rendering tests to expect the new
GlobalNetworkPolicyand verify the proxy label is stamped.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| pkg/render/gatewayapi/gateway_api.go | Adds proxy GlobalNetworkPolicy and stamps/standardizes k8s-app labeling for controller/certgen/proxy selection. |
| pkg/render/gatewayapi/gateway_api_test.go | Updates expectations for the new proxy policy and verifies proxy pod labeling behavior. |
Suppressed comments (1)
pkg/render/gatewayapi/gateway_api.go:892
- This comment says any user-supplied pod labels are preserved, but the function unconditionally overwrites "k8s-app" (which could also be user-supplied on a custom EnvoyProxy). Clarifying that only non-"k8s-app" labels are preserved would avoid confusion.
// ensureGatewayProxyLabel stamps the same Calico-owned k8s-app label on the data-plane
// proxy pods, via the EnvoyProxy pod spec, so gatewayAPIProxyPolicy can select them.
// Any user-supplied pod labels carried over from a custom EnvoyProxy are preserved.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| proxyPolicy, err := rtest.GetResourceOfType[*v3.GlobalNetworkPolicy](objsToCreate, ProxyPolicyName, "") | ||
| Expect(err).NotTo(HaveOccurred()) | ||
| Expect(proxyPolicy.Spec.Tier).To(Equal("calico-system")) | ||
| Expect(proxyPolicy.Spec.Selector).To(Equal(EnvoyProxyPolicySelector)) | ||
| Expect(proxyPolicy.Spec.Selector).NotTo(ContainSubstring("owning-gateway-name"), | ||
| "proxy policy must select by our Calico label, not Envoy Gateway's owning-gateway-name") |
| // setGatewayComponentLabel stamps the Calico-owned k8s-app label on an operator-rendered | ||
| // envoy-gateway component's pod template (controller, certgen). Existing template labels | ||
| // are preserved. |
Description
Bug fix.
Since #4690 we run a single envoy-gateway controller in
calico-systemwithdeploy.type=GatewayNamespace. The data-plane proxies now run in each Gateway's own namespace, not incalico-system.We render one gateway policy in the
calico-systemtier:calico-system.envoy-gateway. It lives incalico-systemand selects the controller and certgen pods. The proxies run elsewhere and match no policy in that tier. When a Gateway namespace runs a default-deny tier, the proxy there has nothing to let its traffic through. This is the same kind of miss as the conformance MetalLB pool that stayed pinned totigera-gateway(projectcalico/calico#13095).This adds a
GlobalNetworkPolicy,calico-system.envoy-gateway-proxy, that selects the proxy pods byk8s-app == 'calico-gateway-api-proxy'. The operator stamps that Calico-owned label on the proxy pods through the EnvoyProxy pod spec, so the policy keys off a label we control rather than Envoy Gateway's owngateway.envoyproxy.io/owning-gateway-name, which we do not own and which could change upstream. A GNP covers any Gateway namespace, including ones created later, with no re-render. It allows:calico-system. The proxy dials the controller (seeenvoyproxy/gatewayinternal/infrastructure/kubernetes/proxy/resource.goandinternal/xds/bootstrap). 18001 is the ratelimit path, not a proxy path, so it is left out.The same Calico-owned labelling now covers all three envoy-gateway components (controller, certgen, proxy) from one documented place in the gateway render. That
k8s-applabel is not set through the standard labeller inpkg/controller/utils/component.goon purpose: the labeller keys off the object name, and the proxy pods are created by the envoy-gateway controller at runtime, so the operator never renders them to label.How I tested it
I built this operator image and ran it on an OSS master cluster (eBPF, namespaced mode). I put a Gateway, an HTTPRoute, and an nginx backend in a normal user namespace, scoped a Calico default-deny to the proxy pod, and curled the proxy from inside the cluster.
I also tried the narrower ingress idea (allow only 19001, then Pass). Under default-deny it times out: listener traffic falls through to the user's deny and the Gateway stops serving. So allowing all inbound TCP is the right default. The cost is that an Allow is terminal in this tier, so a user cannot narrow proxy ingress with their own policy. Scaling the operator back up showed it reverts any drift on the GNP and the Gateway recovers.
I re-ran the whole repro after switching the selector to the Calico-owned
k8s-applabel (OSS master, eBPF, namespaced mode). The operator stampsk8s-app: calico-gateway-api-proxyon the proxy pod, the rendered GNPcalico-system.envoy-gateway-proxyselects it, and the table holds: baseline 200; under a Gateway-namespace default-deny the proxy returns 503 (reachable and routing, only the backend hop denied); allowing proxy to backend gives 200. Scaling the operator down and deleting the GNP drops the proxy to a full timeout, and scaling it back up re-renders the GNP and the Gateway recovers. One thing to watch when testing this by hand: the 503 lands after Envoy's ~10s upstream timeout, so a curl timeout under 10s reads as a connect failure even though the proxy is reachable..Rebased onto master on 2026-08-19. Clean rebase, no conflicts.
go build ./...passes andpkg/render/gatewayapiis 34 of 34 green.One thing to know
The GNP covers the proxy's own needs: DNS, the control-plane link, and ingress. It does not open egress to backends, because backends are arbitrary user workloads. Its egress rules end in a
Pass, so the decision falls to the user's own tier and they keep control of which backends the proxy may reach. A user who runs default-deny in a Gateway namespace has to allow the proxy to reach their backend themselves. Until they do, the proxy is up and configured but returns 503 on the upstream. This matches how the controller policy already works.The docs side of that is tigera/docs#2802, which trims the reference policy on the create-an-ingress-gateway page down to the part the user still owns.
What moved out of this PR
An earlier push carried a second change that added the standard
app.kubernetes.io/*labels to the pod templates of operator-rendered Jobs. It came out of Rene's review question aboutcomponent.go. I pulled it out on its own branch, because it turned out not to be labels-only: making a Job's pod template non-nil also makes the shared tail incomponent.gostampoperator.tigera.io/host-networkedon any hostNetwork pod template. The AWS security-group setup Job runs hostNetwork, and thepodiprecoverycontroller deletes labelled hostNetwork pods whosestatus.podIPsdo not match the node. That is a real side effect and it deserves its own review rather than riding along here.Release Note
For PR author
make gen-files(n/a, no API change).make gen-versions(n/a).