[Bug] 2-networking-a-fedramp-high: NVA return-route generation only routes the FIRST tenant subnet per environment ([0] index) — additional tenant subnets get no NVA return route and lose ALL internet egress

[JohnHales]

Bug Description

In fast/stages-aw/2-networking-a-fedramp-high/nva.tf, the NVA cloud-config's trusted-side ("landing") return routes are built from a per-environment comprehension that selects only the FIRST tenant subnet of each environment:

locals {
  routing_config = [
    { name = "dmz", enable_masquerading = true, routes = [var.gcp_ranges.gcp_dmz_primary] },
    { name = "landing", routes = [for k, v in var.envs_folders : module.env-spoke-vpc[k].subnets[
        "${var.regions.primary}/${[for s in try(var.subnets[lower(k)], []) : s.name if s.tenant != null][0]}"
      ].ip_cidr_range] },
  ]
}

The trailing […][0] means each environment contributes exactly ONE tenant subnet CIDR to the NVA's return routes. But var.subnets is typed map(list(object(... name, ip_cidr_range, tenant = optional(string) ...))) — a LIST of subnets per network — so an environment can legitimately hold multiple tenant subnets. Any tenant subnet that is not the first in its environment's list receives no return route on the NVA.

The NVA (the simple-nva cloud-config, enable_masquerading = true on the DMZ interface) masquerades outbound workload traffic on its untrusted/DMZ interface and forwards it correctly. But when the reply returns, the NVA looks up the original client IP, finds no trusted-side route for that subnet, falls through to its own default route, and sends the reply back out the DMZ/untrusted interface toward the internet gateway instead of back to the trusted spoke. The connection hangs. Net effect for VMs in the un-routed subnet: name resolution still works (the metadata resolver needs no egress) but 100% of real internet egress (TCP and ICMP) times out — a silent, hard-to-diagnose failure.

Environment and Deployment Context

• Stellar Engine Version/Commit: main (fast/stages-aw/2-networking-a-fedramp-high/nva.tf, local.routing_config "landing" routes)
• Deployment Type:
  • US Region Restricted (e.g., Access Policy constraint)
  • FedRAMP Medium
  • FedRAMP High
  • DoD IL4
  • DoD IL5
  • Stand-alone / Custom
• FAST Stage (if applicable):
  • Stage 0 (Bootstrap)
  • Stage 1 (Resource Management)
  • Stage 2 (Network Creation)
  • Stage 3 (Security and Audit)
• Affected Component: fast/stages-aw/2-networking-a-fedramp-high/nva.tf — local.routing_config, the landing routes comprehension ([for s in try(var.subnets[lower(k)], []) : s.name if s.tenant != null][0]); the value is consumed by module.nva-cloud-config (modules/cloud-config-container/simple-nva) and pushed to both NVAs as their trusted-side static routes.

Steps to Reproduce

1. Deploy an SE FRH landing zone whose prod environment spoke has TWO tenant subnets — e.g. a base tenant subnet (10.1.0.0/24) and a second tenant subnet (10.1.1.0/24), both with tenant set.
2. Bring up a VM in the SECOND subnet (10.1.1.x), no external IP.
3. From that VM: curl -m 15 https://www.google.com and ping -c2 8.8.8.8 — both time out; DNS still resolves (metadata resolver at 169.254.169.254).
4. On either NVA: ip route get 10.1.1.4 → resolves via <dmz-gw> dev eth0 (the default/untrusted interface), NOT toward the trusted spoke; ip route shows landing return routes only for each environment's FIRST tenant subnet (10.1.0.0/24, 10.2.0.0/24, 10.3.0.0/24) and none for 10.1.1.0/24.
5. Add the missing route on both NVAs: sudo ip route replace 10.1.1.0/24 via <landing-gw> dev eth1 — egress immediately works (HTTP 200, ping 0% loss). This confirms the NVA return-route omission is the sole cause.

Expected Behavior

The NVA is given return routes for ALL tenant subnets in every environment, so every tenant subnet's VMs egress correctly through the NVA + Cloud NAT.

Actual Behavior

Only the first tenant subnet per environment gets a return route; VMs in any additional tenant subnet lose all internet egress (silently — DNS still resolves), while the NVA misroutes their return traffic out its untrusted interface.

Relevant Logs and Errors

# from a VM in the SECOND tenant subnet (10.1.1.4):
$ curl -sS -m 15 https://www.google.com
curl: (28) Connection timed out after 15000 milliseconds
$ ping -c2 8.8.8.8
2 packets transmitted, 0 received, 100% packet loss

# on the NVA:
$ ip route get 10.1.1.4
10.1.1.4 via 10.0.0.1 dev eth0 src 10.0.0.20      # <- out the DMZ/default iface, WRONG WAY
$ ip route | grep -E '10\.[123]\.'
10.1.0.0/24 via 10.0.1.1 dev eth1
10.2.0.0/24 via 10.0.1.1 dev eth1
10.3.0.0/24 via 10.0.1.1 dev eth1                 # <- no route for 10.1.1.0/24

Expected/Suggested Fix

Replace the single-subnet [0] pick with a flatten over ALL tenant subnets per environment:

routes = flatten([for k, v in var.envs_folders : [
  for s in try(var.subnets[lower(k)], []) :
    module.env-spoke-vpc[k].subnets["${var.regions.primary}/${s.name}"].ip_cidr_range
  if s.tenant != null
]])

This routes every tenant subnet back through the NVA, so adding a second tenant subnet no longer silently breaks its egress.

Additional Context

Verified live on an SE FRH deployment: before the fix, egress from the subnet was 100% broken; adding the missing NVA return route on both NVAs restored it (confirmed HTTP 200 + a Cloud NAT public egress IP + ping 0% loss). Belongs with the other subnet-handling gaps in the gemini ingress path (#105 hardcoded LB subnet, #111 fragile network resolution), but this one is upstream in Stage-2 networking, not the gemini blueprint. NOTE: the ephemeral ip route replace fix does NOT survive an NVA reboot / MIG replacement / networking-stage redeploy — the durable fix is the source change above (or a persistent route baked into the NVA cloud-config).