fix(netwatch): mark the socket broken when a rebind fails, so sends retry it

[jgeluk]

Fixes #200.

Problem

SocketState::rebind() deliberately stays in Closed on a bind failure ("will retry on next attempt"), but nothing arms that retry: UdpSocket::rebind() propagates the error without setting is_broken, and maybe_rebind() — the only recovery path on the send/recv side — is gated on is_broken. A single failed rebind (typically during sleep→wake on macOS, when the network-change handler fires before interfaces are back up) strands the socket in Closed permanently: every subsequent send logs socket closed at WARN and fails with BrokenPipe, at the caller's full send rate, until the process restarts.

Fix

Mark the socket broken when the rebind fails, so the next send/recv retries the bind and the socket heals as soon as the address is bindable again. A deliberate close() still stays closed: it never sets is_broken, pinned by a new test.

Tests

• test_failed_rebind_marks_broken_so_sends_retry — forces a rebind failure, asserts the socket self-heals on the next send once the address is bindable again.
• test_deliberate_close_stays_closed — pins that close() remains terminal.

cargo test -p netwatch --lib: 18 passed.

Field-tested for several days across daemons that previously exhibited the storm: sleep/wake now recovers within a second, zero socket closed events.

🤖 Generated with Claude Code

[jgeluk]

Thanks for taking a close look — the "hammering forever" worry is the right thing to check, so let me be precise about what mark_broken() does and doesn't do here.

It is not a loop. mark_broken() schedules nothing. It only sets the flag that lets the next send/recv re-enter maybe_rebind(). So retry frequency is bounded by caller traffic — a rebind is attempted exactly when the socket is actually needed, and never otherwise. No traffic, no retries. On a quiet post-wake network that's a handful of attempts until the address is bindable again, then it heals and the flag clears.

The status quo is the unbounded one. Today a single failed rebind (bind fails once while interfaces are still coming up) leaves the socket in Closed with is_broken == false, and nothing ever arms a retry. From then on every send hits the Closed arm: WARN + BrokenPipe, at the caller's full send rate, until the process restarts. We measured on the order of ~500 log lines/s per process in that state. So the current code already "hammers" — it just never recovers.

A deliberate close() still stays closed. test_deliberate_close_stays_closed in the PR pins that: close() does not set the flag, and a later send fails with BrokenPipe rather than resurrecting the socket. The change only affects the failed-rebind path.

Same defect class upstream: iroh#4289 ("Failed socket rebind kills noq EndpointDriver") is the same shape one layer up — you noted 7b06ecbe was "in the right direction but maybe not sufficiently careful". I agree iroh is the right place for policy (when to rebind, backoff across interface events); but netwatch is the place that decides whether a failed rebind is recoverable at all, and today it isn't. Fixing it in iroh alone can't reach a socket that netwatch has stranded.

If you still want an explicit bound, I'm happy to make it smaller/safer: a retry counter or backoff on the broken flag (e.g. give up marking broken after N consecutive failed rebinds, or exponential spacing between maybe_rebind attempts). A capped retry is strictly better than a permanent strand — say which shape you'd take and I'll add it.

Logs: the original capture rotated out, but I have a small repro binary (stock netwatch =0.19.1 + noq-udp, calling rebind() from an interface-event loop) that reproduces on roughly every wake here. The interesting window is the first ~2 s after wake. I can attach a netwatch=trace capture of a sleep/wake cycle if that helps — say the word.

[matheus23]

Thank you

[matheus23]

Next time - please use the correct PR template though :/