Skip to content

feat(runway): dlq reconciler for merge topics - #459

Draft
behinddwalls wants to merge 1 commit into
preetam/runway-terminal-errorsfrom
preetam/runway-dlq-reconciler
Draft

feat(runway): dlq reconciler for merge topics#459
behinddwalls wants to merge 1 commit into
preetam/runway-terminal-errorsfrom
preetam/runway-dlq-reconciler

Conversation

@behinddwalls

@behinddwalls behinddwalls commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Summary

Why?

SubmitQueue records in-flight merge work before publishing and then waits for exactly one MergeResult echoing its correlation id. Runway is stateless and the sole responder on that id, so every request must resolve to a result — or the client waits forever.

The primary controllers resolve what they can name: conflicts and invalid requests become a FAILED result, infrastructure faults are nacked for retry. But a fault that never recovers exhausts the retry budget and dead-letters. Nothing consumed those dead-letter topics, so the request produced no signal at all and the client's correlation id hung indefinitely.

What?

Adds runway/controller/dlq, a reconciler that subscribes to an inbound topic's _dlq queue and, for each dead-lettered MergeRequest, republishes a FAILED MergeResult echoing the correlation id to the corresponding signal topic. dlq.TopicKey derives the DLQ topic key from the primary one so the two stay in lockstep.

Unlike the SubmitQueue and Stovepipe DLQ reconcilers this one writes no entity state — Runway has none, and the signal is the resolution. A payload that cannot be decoded carries no correlation id and is dropped rather than retried forever.

Wires two instances in the server (one per inbound topic) on a dedicated consumer running under errs.AlwaysRetryableProcessor, so a transient publish failure retries indefinitely rather than dead-lettering the dead-letter. The DLQ consumer is started alongside the primary one and stopped with the same 30s drain on shutdown; both stop errors are joined into the exit status.

Test Plan

bazel test //runway/... — 5/5 pass, including new //runway/controller/dlq coverage for republish-on-dead-letter, the drop-undecodable-payload path, and publish-failure propagation
bazel build //service/runway/... — wiring compiles
make gazelle, make fmt

Stack

  1. chore(submitqueue): remove dead pusher extension #457
  2. feat(runway): classify invalid merge requests as terminal #458
  3. @ feat(runway): dlq reconciler for merge topics #459
  4. feat(runway): git-backed merger with REBASE #460
  5. feat(runway): git merger SQUASH_REBASE and MERGE #461
  6. feat(runway): git merger PROMOTE #462
  7. feat(runway): wire the git merger into the server #463

## Summary

### Why?

SubmitQueue records in-flight merge work before publishing and then waits for exactly one `MergeResult` echoing its correlation id. Runway is stateless and the sole responder on that id, so every request must resolve to a result — or the client waits forever.

The primary controllers resolve what they can name: conflicts and invalid requests become a `FAILED` result, infrastructure faults are nacked for retry. But a fault that never recovers exhausts the retry budget and dead-letters. Nothing consumed those dead-letter topics, so the request produced no signal at all and the client's correlation id hung indefinitely.

### What?

Adds `runway/controller/dlq`, a reconciler that subscribes to an inbound topic's `_dlq` queue and, for each dead-lettered `MergeRequest`, republishes a `FAILED` `MergeResult` echoing the correlation id to the corresponding signal topic. `dlq.TopicKey` derives the DLQ topic key from the primary one so the two stay in lockstep.

Unlike the SubmitQueue and Stovepipe DLQ reconcilers this one writes no entity state — Runway has none, and the signal *is* the resolution. A payload that cannot be decoded carries no correlation id and is dropped rather than retried forever.

Wires two instances in the server (one per inbound topic) on a dedicated consumer running under `errs.AlwaysRetryableProcessor`, so a transient publish failure retries indefinitely rather than dead-lettering the dead-letter. The DLQ consumer is started alongside the primary one and stopped with the same 30s drain on shutdown; both stop errors are joined into the exit status.

## Test Plan

✅ `bazel test //runway/...` — 5/5 pass, including new `//runway/controller/dlq` coverage for republish-on-dead-letter, the drop-undecodable-payload path, and publish-failure propagation
✅ `bazel build //service/runway/...` — wiring compiles
✅ `make gazelle`, `make fmt`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant