Skip to content

[Bug] Standard Brownfield auto-discovers a nonexistent CMEK keyring (${Cap_Env}-${Tenant}-keyring) with no override prompt, writing a broken kms_key_id that fails the gemini-stage-0 apply #132

Description

@JohnHales

Bug Description

In gemini-enterprise/deploy.sh, standard Brownfield (option 2) derives the CMEK keyring name as ${Cap_Env}-${Tenant}-keyring (e.g. Prod-<tenant>-keyring) in the tenant iac-core project and looks up the Gemini resources key there (US_KEYRING_NAME set ~Line 618). On a Stellar Engine landing zone the key actually lives in the prod-us keyring under <prefix>-prod-sec-core-0, not the derived keyring, so keyrings describe misses it, deploy.sh prints "US Keyring not found" and leaves CMEK_US_RESOURCES_KEY blank (~Lines 639–641). The three interactive CMEK override prompts (Enter CMEK Project ID / US Multi-Region Keyring ID / US Gemini Resources Key ID) exist only inside the elif IS_CUSTOM (Custom Brownfield, option 3) branch (~Lines 675, L699–706), so standard Brownfield gives the operator no way to correct the value. With the resources key blank, the data-store step later auto-creates the key on the wrong keyring (if [[ -z "$CMEK_US_RESOURCES_KEY" ]], ~Line 1937): kms keys create returns NOT_FOUND, the Discovery Engine service-agent grant fails, cmekConfig registration returns HTTP 400, and the generated gemini-stage-0/terraform.tfvars carries the wrong kms_key_id, so the subsequent automatic terraform apply fails.

Environment and Deployment Context

  • Stellar Engine Version/Commit: main @ 3728fc98
  • Deployment Type:
    • US Region Restricted (e.g., Access Policy constraint)
    • FedRAMP Medium
    • FedRAMP High
    • DoD IL4
    • DoD IL5
    • Stand-alone / Custom
  • Affected Component: blueprints/fedramp-high/gemini-enterprise/deploy.shdiscover_infrastructure() CMEK block: keyring derive (~Line 618), "US Keyring not found" blank (~Lines 639–641), Custom-only override prompts (~Lines 675, L699–706), data-store auto-create (~Line 1937), cmekConfig registration (~Lines 1960–1989)
  • Terraform Version: deploy.sh pins Terraform 1.12.2 (tfenv)

Steps to Reproduce

  1. On a healthy SE FRH landing zone, from <prefix>-<env>-<tenant>-main-0, run ./deploy.sh; choose Brownfield (2), then Step 1.
  2. When prompted, enable CMEK for the data store.
  3. Observe deploy.sh derive keyring <env>-<tenant>-keyring in the iac-core project, print "US Keyring not found", and leave the resources key blank — no override prompt appears (those are Custom-Brownfield only).
  4. deploy.sh auto-creates the key on the wrong keyring → NOT_FOUND; the DE service-agent grant fails; cmekConfig registration returns 400; the wrong kms_key_id is written to terraform.tfvars; the automatic apply then fails on it.

Expected Behavior

Standard Brownfield should discover the real SE key (projects/<prefix>-prod-sec-core-0/locations/us/keyRings/prod-us/cryptoKeys/gemini-enterprise) — or expose the CMEK override prompts in standard Brownfield — so the data store is encrypted with the sovereign key without hand-editing tfvars.

Actual Behavior

On a fresh SE tenant the CMEK step silently targets a nonexistent keyring, the data store cannot be CMEK-encrypted, and the apply fails until the operator manually corrects gemini-stage-0/terraform.tfvars.

Relevant Logs and Errors

CMEK for Data Stores requested. Ensuring key exists...
  Target Project: <prefix>-prod-<tenant>-iac-core-0
  Keyring: Prod-<tenant>-keyring
  Creating Key 'gemini-enterprise'...
ERROR: (gcloud.kms.keys.create) NOT_FOUND: KeyRing projects/.../keyRings/Prod-<tenant>-keyring not found.
WARNING: Failed to grant IAM binding to Discovery Engine service account.
  You might need 'roles/cloudkms.admin' on the key project.
Failed to register CMEK key. HTTP Status: 400 (FAILED_PRECONDITION)

Additional Context

Distinct from #106. Same deploy.sh surface as #114.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Level of Effort - LowQuick, well-defined tasks with no unknowns; takes a few hours up to one day to completePriority - MediumStandard features and non-blocking bugs; important for the current milestone but not urgentbugSomething isn't workinggemini for governmentGemini for Government (G4G) related

    Type

    Fields

    No fields configured for Bug.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions