Skip to content

fix(compactor): await HeartBeat goroutine to fix flaky TestBlocksCleaner on arm64 #7566

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sandy2008 wants to merge 1 commit into from
Closed

fix(compactor): await HeartBeat goroutine to fix flaky TestBlocksCleaner on arm64 #7566

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@
* [BUGFIX] Compactor: Fix stale `cortex_bucket_index_last_successful_update_timestamp_seconds` metric not being cleaned up when tenant ownership changes due to ring rebalancing. This caused false alarms on bucket index update rate when a tenant moved between compactors. #7485
* [BUGFIX] Security: Fix stored XSS vulnerability in Alertmanager and Store Gateway status pages by replacing `text/template` with `html/template`. #7512
* [BUGFIX] Security: Limit decompressed gzip output in `ParseProtoReader` and OTLP ingestion path. The decompressed body is now capped by `-distributor.otlp-max-recv-msg-size`. #7515
* [BUGFIX] Compactor: Wait for the blocks cleaner `VisitMarkerManager.HeartBeat` goroutine to exit before returning from per-user cleanup callbacks, preventing a late `MarkWithStatus(Completed)` and `DeleteVisitMarker` write from racing with test teardown and causing `unlinkat: directory not empty` flakes in `TestBlocksCleaner`. #7564

## 1.21.0 2026-04-24

Expand Down
16 changes: 14 additions & 2 deletions pkg/compactor/blocks_cleaner.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -357,9 +357,15 @@ func (c *BlocksCleaner) cleanUpActiveUsers(ctx context.Context, users []string,
return nil
}
errChan := make(chan error, 1)
go visitMarkerManager.HeartBeat(ctx, errChan, c.cleanerVisitMarkerFileUpdateInterval, true)
var hbWG sync.WaitGroup
hbWG.Add(1)
go func() {
defer hbWG.Done()
visitMarkerManager.HeartBeat(ctx, errChan, c.cleanerVisitMarkerFileUpdateInterval, true)
}()
defer func() {
errChan <- nil
hbWG.Wait()
}()
return errors.Wrapf(c.cleanUser(ctx, userLogger, userBucket, userID, firstRun), "failed to delete blocks for user: %s", userID)
})
Expand Down Expand Up @@ -392,9 +398,15 @@ func (c *BlocksCleaner) cleanDeletedUsers(ctx context.Context, users []string) e
return nil
}
errChan := make(chan error, 1)
go visitMarkerManager.HeartBeat(ctx, errChan, c.cleanerVisitMarkerFileUpdateInterval, true)
var hbWG sync.WaitGroup
hbWG.Add(1)
go func() {
defer hbWG.Done()
visitMarkerManager.HeartBeat(ctx, errChan, c.cleanerVisitMarkerFileUpdateInterval, true)
}()
defer func() {
errChan <- nil
hbWG.Wait()
}()
return errors.Wrapf(c.deleteUserMarkedForDeletion(ctx, userLogger, userBucket, userID), "failed to delete user marked for deletion: %s", userID)
})
Expand Down