Skip to content

Fix concurrent map iteration and write panic in AppRefTracker#1844

Open
somaz94 wants to merge 1 commit into
carvel-dev:developfrom
somaz94:fix/reftracker-concurrent-map-race
Open

Fix concurrent map iteration and write panic in AppRefTracker#1844
somaz94 wants to merge 1 commit into
carvel-dev:developfrom
somaz94:fix/reftracker-concurrent-map-race

Conversation

@somaz94

@somaz94 somaz94 commented Jul 15, 2026

Copy link
Copy Markdown

What this PR does / why we need it:

AppRefTracker.AppsForRef and RefsForApp returned the internal map by
reference. Callers such as SecretHandler.enqueueAppsForUpdate and
ConfigMapHandler.enqueueAppsForUpdate iterate that map after the tracker lock
is released, so a concurrent ReconcileRefs / RemoveAppFromAllRefs writing the
same map crashes kapp-controller with fatal error: concurrent map iteration and map write. This reproduces on large clusters with thousands of
namespaces/secrets under continuous reconciliation.

Both methods now return a shallow copy of the RefKey set while holding the lock,
so callers iterate a private map and never race the writers. The two callers
only iterate the result (read-only), so returning a copy is behavior-preserving.

Which issue(s) this PR fixes:

Fixes #1843

Does this PR introduce a user-facing change?

Fix panic ("concurrent map iteration and map write") in AppRefTracker that could
crash kapp-controller on large clusters under high reconciliation concurrency.

Additional Notes for your reviewer:

Added two regression tests in pkg/reftracker/ref_tracker_race_test.go
(concurrent read+iterate against a concurrent writer, for both AppsForRef and
RefsForApp). Before the fix they crash with the reported fatal error; after it
they pass, including under -race.

Validation run locally:

  • go test ./pkg/reftracker/... -race passes (existing + new tests)
  • gofmt / go vet clean
Review Checklist:
  • Follows the developer guidelines
  • Relevant tests are added or updated
  • Relevant docs in this repo added or updated (n/a, internal bug fix)
  • Relevant carvel.dev docs added or updated (n/a)
  • Code is at least as readable and maintainable as it was before this change

Signed-off-by: somaz <genius5711@gmail.com>
somaz94 added a commit to somaz94/somaz94 that referenced this pull request Jul 15, 2026
@somaz94
somaz94 marked this pull request as ready for review July 20, 2026 08:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

Follow-up on #1812: Race Condition in AppRefTracker (Concurrent Map Iteration and Write)

2 participants