Skip to content

feat: Redesign gateway CRDs around DSR/Maglev datapath #17

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
privateip merged 1 commit into from
Aug 19, 2026
Merged

feat: Redesign gateway CRDs around DSR/Maglev datapath #17

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 16 additions & 34 deletions api/v1alpha1/egresspolicy_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,12 @@ import (
)

// NetworkEgressPolicy enables internet egress for a single tenant
// VPC/VPCAttachment, served by the shared hyperconverged gateway engine's
// masquerade (SNAT/PAT) datapath. Unlike NetworkRule, it carries no
// VIP/backend/port: egress is on or off for a (vpcRef, vpcAttachmentRef)
// pair, existence-implies-enabled, not a per-flow rule — because the
// destination of an egress flow is an arbitrary internet address, not a
// pre-configured backend list.
// VPC/VPCAttachment, served by the sharded, stateful galactic-nat66 tier
// (see NAT66Shard). Unlike NetworkRule, it carries no VIP/backend/port:
// egress is on or off for a (vpcRef, vpcAttachmentRef) pair,
// existence-implies-enabled, not a per-flow rule — because the destination
// of an egress flow is an arbitrary internet address, not a pre-configured
// backend list.
//
// It is namespaced (deployed to galactic-system) and tenant-writable; like
// NetworkRule, vpcRef/vpcAttachmentRef are opaque string identifiers because
Expand All @@ -20,21 +20,22 @@ import (
// accepted — see the Accepted condition.
//
// Presence of an accepted NetworkEgressPolicy resolves only *enablement*
// (should this tenant reach the egress datapath at all) — a routing-layer
// decision (does the tenant's VRF have a default route toward the shared
// egress_sid locator), not a per-packet datapath lookup. *Isolation*
// (preventing two tenants with colliding ULA source addresses from
// colliding in the egress connection table) is a separate, datapath-level
// concern resolved by tagging each flow with the tenant/VRF identifier
// carried in the egress_sid locator's own Argument bits, not by anything in
// this spec.
// (should this tenant's VRF get a default route toward the shared NAT66
// tier at all) — unlike this type's original design (superseded), there is
// no single "assigned gateway node" to compute or pin: any NAT66Shard may
// serve any tenant's flow, chosen by the shard-placement consistent-hash
// ring (internal/maglev, keyed on (tenant VRFID, backend, destination) —
// see NAT66Shard's doc comment), not by a per-tenant node assignment stored
// here. *Isolation* (preventing two tenants with colliding ULA source
// addresses from colliding in the egress connection table) is a separate,
// datapath-level concern resolved by tagging each flow with the VRFID
// carried in the tenant's own SRv6 Argument, not by anything in this spec.
//
// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:resource:scope=Namespaced,shortName=netegress
// +kubebuilder:printcolumn:name="VPC",type="string",JSONPath=".spec.vpcRef"
// +kubebuilder:printcolumn:name="VPC-ATTACHMENT",type="string",JSONPath=".spec.vpcAttachmentRef"
// +kubebuilder:printcolumn:name="ASSIGNED-NODE",type="string",JSONPath=".status.assignedGatewayNode"
// +kubebuilder:printcolumn:name="AGE",type="date",JSONPath=".metadata.creationTimestamp"
type NetworkEgressPolicy struct {
metav1.TypeMeta `json:",inline"`
Expand Down Expand Up @@ -71,25 +72,6 @@ type NetworkEgressPolicyStatus struct {
// +optional
ObservedGeneration int64 `json:"observedGeneration,omitempty"`

// AssignedGatewayNode is the name of the NetworkGateway-backed gateway
// node this policy's tenant should route egress traffic through,
// mirroring NetworkRule's own status.primaryNode field and computed
// the same way: assigned_node = hash(vpcRef) % <gateway node count>
// (design plan §4.5 — a tenant's egress node and its primary ingress
// node are the same node, by design, so both fields are computed by
// the identical AssignPrimaryNode function). The controller consuming
// this CRD sets this field exactly once, at creation.
//
// This value must never be silently recomputed by a reconciler once
// set, for the exact same reason NetworkRuleStatus.PrimaryNode's own
// doc comment gives: recomputing it on a later reconcile can flip
// which node a tenant's egress traffic routes through and cause an
// avoidable traffic flap; a reconciler that observes a stale or
// removed node here must surface that via a condition instead of
// overwriting the value.
// +optional
AssignedGatewayNode string `json:"assignedGatewayNode,omitempty"`

// Conditions contains the standard conditions for this resource,
// including Accepted (see AcceptedReasonOwnershipVerified /
// AcceptedReasonOwnershipDenied in rule_types.go, reused as-is here).
Expand Down
92 changes: 9 additions & 83 deletions api/v1alpha1/egresspolicy_types_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,73 +128,14 @@ func TestNetworkEgressPolicyFieldNames(t *testing.T) {
}
}

// TestNetworkGatewayEgressAddressFieldName verifies NetworkGatewayStatus's
// new EgressAddress field round-trips under the JSON key "egressAddress"
// and stays independently settable from SRv6Address (a gateway node may
// have one, both, or neither populated).
func TestNetworkGatewayEgressAddressFieldName(t *testing.T) {
gw := newTestGateway()
gw.Status.EgressAddress = "2001:db8:ffff::1"

data, err := json.Marshal(gw)
if err != nil {
t.Fatalf("Marshal: %v", err)
}

var m map[string]any
if err := json.Unmarshal(data, &m); err != nil {
t.Fatalf("Unmarshal: %v", err)
}

status, ok := m["status"].(map[string]any)
if !ok {
t.Fatalf("status not found or wrong type: %v", m["status"])
}
if v, ok := status["egressAddress"]; !ok || v != "2001:db8:ffff::1" {
t.Errorf("expected status.egressAddress=%q, got %v", "2001:db8:ffff::1", status["egressAddress"])
}
if v, ok := status["sRv6Address"]; !ok || v != "2001:db8:1::1" {
t.Errorf("expected status.sRv6Address unaffected, got %v", v)
}
}

// TestNetworkGatewayEgressSIDFieldName verifies NetworkGatewayStatus's new
// EgressSID field round-trips under the JSON key "egressSID" (design plan
// §3.1/§4.3) and stays independently settable from SRv6Address/
// EgressAddress.
func TestNetworkGatewayEgressSIDFieldName(t *testing.T) {
gw := newTestGateway()
gw.Status.EgressAddress = "2001:db8:ffff::1"
gw.Status.EgressSID = "2001:db8:eeee::1"

data, err := json.Marshal(gw)
if err != nil {
t.Fatalf("Marshal: %v", err)
}

var m map[string]any
if err := json.Unmarshal(data, &m); err != nil {
t.Fatalf("Unmarshal: %v", err)
}

status, ok := m["status"].(map[string]any)
if !ok {
t.Fatalf("status not found or wrong type: %v", m["status"])
}
if v, ok := status["egressSID"]; !ok || v != "2001:db8:eeee::1" {
t.Errorf("expected status.egressSID=%q, got %v", "2001:db8:eeee::1", status["egressSID"])
}
if v, ok := status["egressAddress"]; !ok || v != "2001:db8:ffff::1" {
t.Errorf("expected status.egressAddress unaffected, got %v", v)
}
}

// TestNetworkEgressPolicyAssignedGatewayNodeFieldName verifies the new
// AssignedGatewayNode field round-trips under the JSON key
// "assignedGatewayNode" (design plan §4.5).
func TestNetworkEgressPolicyAssignedGatewayNodeFieldName(t *testing.T) {
// TestNetworkEgressPolicyStatusHasNoAssignedGatewayNode is a regression
// test: the earlier design pinned a policy to a single gateway node's
// masquerade datapath. The sharded galactic-nat66 tier has no such fixed
// assignment — any NAT66Shard may serve any tenant's flow, chosen by the
// shard-placement consistent-hash ring, not a per-tenant node stored here.
func TestNetworkEgressPolicyStatusHasNoAssignedGatewayNode(t *testing.T) {
orig := newTestEgressPolicy()
orig.Status.AssignedGatewayNode = "gw-node-a"
orig.Status.Conditions = []metav1.Condition{{Type: ConditionTypeAccepted}}

data, err := json.Marshal(orig)
if err != nil {
Expand All @@ -210,22 +151,7 @@ func TestNetworkEgressPolicyAssignedGatewayNodeFieldName(t *testing.T) {
if !ok {
t.Fatalf("status not found or wrong type: %v", m["status"])
}
if v, ok := status["assignedGatewayNode"]; !ok || v != "gw-node-a" {
t.Errorf("expected status.assignedGatewayNode=%q, got %v", "gw-node-a", status["assignedGatewayNode"])
}
}

// TestNetworkEgressPolicyDeepCopyIncludesAssignedGatewayNode extends
// TestNetworkEgressPolicyDeepCopy to cover the new field: mutating the copy
// must not affect the original.
func TestNetworkEgressPolicyDeepCopyIncludesAssignedGatewayNode(t *testing.T) {
orig := newTestEgressPolicy()
orig.Status.AssignedGatewayNode = "gw-node-a"

dup := orig.DeepCopy()
dup.Status.AssignedGatewayNode = "gw-node-b"

if orig.Status.AssignedGatewayNode != "gw-node-a" {
t.Errorf("AssignedGatewayNode mutated: got %q", orig.Status.AssignedGatewayNode)
if _, ok := status["assignedGatewayNode"]; ok {
t.Errorf("unexpected assignedGatewayNode field present in status: %v", status)
}
}
81 changes: 17 additions & 64 deletions api/v1alpha1/gateway_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,33 +4,29 @@ import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// NetworkGateway defines an XDP ingress NAT+LB gateway engine instance bound
// to a single dedicated gateway-role node. Exactly one NetworkGateway exists
// per gateway node (spec.targetRef.name is the Kubernetes node name),
// mirroring the BGPRouter node-scoped root object pattern. NetworkRule
// resources are assigned to a NetworkGateway via status.primaryNode.
// NetworkGateway marks a single dedicated gateway-role node as running the
// Maglev/DSR consistent-hash L4 load-balancer engine. Exactly one
// NetworkGateway exists per gateway node (spec.targetRef.name is the
// Kubernetes node name), mirroring the BGPRouter node-scoped root object
// pattern. NetworkRule resources are served by every NetworkGateway in the
// namespace equally (anycast — see NetworkRuleStatus's doc comment); this
// object's only job is to identify which nodes participate at all and
// surface each node's engine health via Conditions.
//
// There is no tunnel overlay in this design (an earlier Geneve-based
// approach was superseded before this type shipped): the gateway's XDP
// program does Full-NAT (DNAT the VIP to a backend Pod's address, SNAT the
// client's source to status.sRv6Address) and pushes an SRv6 uSID outer
// header addressed to the backend's worker node directly, so return traffic
// (addressed to status.sRv6Address) arrives back at this same gateway node
// over the ordinary SRv6 fabric — no compute-node encap agent, no tunnel
// endpoint to publish. status.sRv6Address is advertised into BGP the same
// way any workload prefix is (a BGPAdvertisement naming it, /128, Argument
// 0 — the value PR #740 reserves and forbids registering into any tenant
// VRF, guaranteeing it never collides with a real tenant's Argument), so
// every other node learns a real kernel SEG6 route to it for free through
// the existing EVPN pipeline.
// This design does no address rewriting on the load-balancing path at all
// (DSR — Direct Server Return): the gateway's XDP program picks a backend
// via consistent hashing on the flow's 5-tuple and pushes an SRv6 uSID outer
// header addressed to the backend's worker node directly, untouched
// otherwise. The backend node answers the client directly (see
// ServiceVIPBinding) — reply traffic never re-enters this gateway node, so
// unlike the Full-NAT design this type originally described, a gateway node
// has no SNAT source address of its own to publish and nothing analogous to
// sRv6Address/egressAddress/egressSID belongs on this status anymore.
//
// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:resource:scope=Namespaced,shortName=netgw
// +kubebuilder:printcolumn:name="TARGET",type="string",JSONPath=".spec.targetRef.name"
// +kubebuilder:printcolumn:name="SRV6-ADDRESS",type="string",JSONPath=".status.sRv6Address"
// +kubebuilder:printcolumn:name="EGRESS-ADDRESS",type="string",JSONPath=".status.egressAddress"
// +kubebuilder:printcolumn:name="EGRESS-SID",type="string",JSONPath=".status.egressSID"
// +kubebuilder:printcolumn:name="AGE",type="date",JSONPath=".metadata.creationTimestamp"
type NetworkGateway struct {
metav1.TypeMeta `json:",inline"`
Expand All @@ -53,49 +49,6 @@ type NetworkGatewayStatus struct {
// +optional
ObservedGeneration int64 `json:"observedGeneration,omitempty"`

// SRv6Address is this gateway node's own SRv6-reachable IPv6 address,
// used as the Full-NAT SNAT source for every ingress flow this node
// translates. Backend Pods' replies are naturally routed back to it
// over the ordinary SRv6 fabric (the same mechanism that routes any
// other node's traffic), where this node's XDP program decapsulates
// and un-NATs them using its own conn_table — there is no separate
// tunnel endpoint or overlay device to publish. Populated by the
// engine once it has computed the address (a uFMT 48+16 uSID over this
// node's own BGPRouter locator/node-ID, at the reserved Argument 0)
// and advertised it into BGP.
// +optional
// +kubebuilder:validation:XValidation:rule="self == '' || isIP(self)",message="sRv6Address must be a valid IPv6 address"
SRv6Address string `json:"sRv6Address,omitempty"`

// EgressAddress is this gateway node's own publicly-routable IPv6
// address, used as the masquerade SNAT source for every egress flow
// this node translates on behalf of tenant VPC backends reaching the
// internet. Unlike SRv6Address (reachable only within the SRv6 fabric),
// this address must additionally be reachable from the public internet
// — an eBGP/uplink-peering concern outside this API. Operator-supplied
// via GALACTIC_GATEWAY_EGRESS_ADDRESS; there is no in-cluster
// derivation mechanism yet, the same gap SRv6Address itself has today.
// A gateway node not offering egress leaves this field empty.
// +optional
// +kubebuilder:validation:XValidation:rule="self == '' || isIP(self)",message="egressAddress must be a valid IPv6 address"
EgressAddress string `json:"egressAddress,omitempty"`

// EgressSID is this gateway node's own egress_sid uSID *locator*
// (design plan §3.1) — the reserved Argument range's Block+Node-ID
// portion tenant VRF default routes encapsulate toward. Unlike
// EgressAddress (a plain, publicly-routable address, no uSID
// structure), this is a real uSID: other nodes need a kernel route to
// it before they can install a SEG6 encap route naming it as the
// destination (the same reason SRv6Address is advertised into BGP),
// so this is published and advertised the same way SRv6Address/
// EgressAddress already are. Operator-supplied via
// GALACTIC_GATEWAY_EGRESS_SID; a gateway node not offering egress
// leaves this field empty, always paired with EgressAddress (both
// set, or neither).
// +optional
// +kubebuilder:validation:XValidation:rule="self == '' || isIP(self)",message="egressSID must be a valid IPv6 address"
EgressSID string `json:"egressSID,omitempty"`

// Conditions contains the standard conditions for this resource.
//
// +listType=map
Expand Down
28 changes: 10 additions & 18 deletions api/v1alpha1/gateway_types_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,6 @@ func newTestGateway() *NetworkGateway {
Spec: NetworkGatewaySpec{
TargetRef: TargetRef{Kind: "Node", Name: "gw-node-a"},
},
Status: NetworkGatewayStatus{
SRv6Address: "2001:db8:1::1",
},
}
}

Expand All @@ -30,15 +27,11 @@ func TestNetworkGatewayDeepCopy(t *testing.T) {
dup := orig.DeepCopy()

dup.Spec.TargetRef.Name = "gw-node-b"
dup.Status.SRv6Address = "2001:db8:1::2"
dup.Status.Conditions = append(dup.Status.Conditions, metav1.Condition{Type: ConditionTypeReady})

if orig.Spec.TargetRef.Name != "gw-node-a" {
t.Errorf("TargetRef.Name mutated: got %q", orig.Spec.TargetRef.Name)
}
if orig.Status.SRv6Address != "2001:db8:1::1" {
t.Errorf("SRv6Address mutated: got %q", orig.Status.SRv6Address)
}
if len(orig.Status.Conditions) != 0 {
t.Errorf("Conditions mutated: got %v", orig.Status.Conditions)
}
Expand Down Expand Up @@ -73,9 +66,6 @@ func TestNetworkGatewayJSONRoundTrip(t *testing.T) {
if got.Spec.TargetRef != orig.Spec.TargetRef {
t.Errorf("TargetRef: got %+v, want %+v", got.Spec.TargetRef, orig.Spec.TargetRef)
}
if got.Status.SRv6Address != orig.Status.SRv6Address {
t.Errorf("SRv6Address: got %q, want %q", got.Status.SRv6Address, orig.Status.SRv6Address)
}
if len(got.Status.Conditions) != 1 {
t.Fatalf("Conditions len: got %d, want 1", len(got.Status.Conditions))
}
Expand All @@ -95,8 +85,10 @@ func TestNetworkGatewayListDeepCopy(t *testing.T) {
}
}

// TestNetworkGatewayFieldNames verifies the JSON keys for spec/status fields
// match the CRD schema.
// TestNetworkGatewayFieldNames verifies the JSON keys for spec fields match
// the CRD schema, and that no per-node SNAT-source fields survive from the
// superseded Full-NAT design — a DSR gateway node rewrites nothing, so it
// has no address of its own to publish here.
func TestNetworkGatewayFieldNames(t *testing.T) {
orig := newTestGateway()

Expand All @@ -118,11 +110,11 @@ func TestNetworkGatewayFieldNames(t *testing.T) {
t.Errorf("expected spec.targetRef field, got %v", spec)
}

status, ok := m["status"].(map[string]any)
if !ok {
t.Fatalf("status not found or wrong type: %v", m["status"])
}
if v, ok := status["sRv6Address"]; !ok || v != "2001:db8:1::1" {
t.Errorf("expected status.sRv6Address=%q, got %v", "2001:db8:1::1", status["sRv6Address"])
if status, ok := m["status"].(map[string]any); ok {
for _, removed := range []string{"sRv6Address", "egressAddress", "egressSID"} {
if _, ok := status[removed]; ok {
t.Errorf("unexpected superseded field %q present in status: %v", removed, status)
}
}
}
}
Loading
Loading