fix(stovepipe): mint a distinct message id for each buildsignal re-poll

[behinddwalls]

Summary

Why?

buildsignal schedules its next poll by re-publishing to its own topic, and it reused the build id as the message id — byte-identical to the message build published to start the loop. The MySQL queue dedups on the (topic, partition_key, id) unique key and InsertDelayed swallows the collision with ON DUPLICATE KEY UPDATE topic = topic, returning success. So the reschedule was accepted and silently discarded.

This is deterministic, not a race. The re-publish happens before the delivery is acked, and GC only collects up to the minimum acked offset on idle ticks, so the colliding row is always still present.

The effect: any build that is not terminal on its first poll is never polled again. Build.Status freezes at accepted/running, the request never leaves processing, nothing is published to record, and the queue's in_flight_count slot is never released — so after MaxConcurrent such builds the queue stops admitting work entirely.

Nothing caught it because the fake build runner could not report a non-terminal status until the previous commit, and the unit tests matched the published message with gomock.Any().

What?

publishBuildSignal now mints {buildID}/poll/{generation}, where the generation is read off the id of the delivery being processed and incremented — so a chain runs B -> B/poll/1 -> B/poll/2 -> … The partition key stays the build id, so each build's poll loop keeps its own partition.

The generation advances deterministically rather than randomly, which matters for the case a random suffix handles badly. The next id is a pure function of the delivery, so a redelivery racing the original computes the same id and dedup collapses the two into one message: the build keeps a single poll chain. A random suffix would instead fork a second chain, doubling the poll rate and racing the first chain's status CAS for no benefit.

A stable id is not an option in the other direction either — that is the bug itself. Even a fixed-but-different id like B/poll only survives one extra tick, because the second tick's re-poll then collides with the message being processed.

Test Plan

✅ bazel test //stovepipe/... — new unit test asserts successive re-polls mint distinct ids and never reuse the build id. Verified it genuinely regresses: reverting just the id line fails it with "map[bk-1:{}]" should have 3 item(s), but has 1.

✅ bazel test //test/e2e/stovepipe/... — new e2e ingests a queue carrying the build-slow marker and waits for the build row to reach succeeded, which requires more than one poll tick. The service log shows three distinct ids on the buildsignal topic for one build.

Stack

1. test(stovepipe): add build-slow marker to the fake build runner #464
2. @ fix(stovepipe): mint a distinct message id for each buildsignal re-poll #465
3. refactor(stovepipe): replace recorded greenness states with build outcomes #466
4. feat(stovepipe): record the build outcome on the request and free its slot #467
5. feat(stovepipe)!: key the record stage on the request id #468
6. fix(stovepipe): stop wrapping the buildsignal re-poll publish as retryable #469