Fix WSConnection reconnect stall: don't block the event loop during the reconnect delay

[kojira]

Summary

strfry stream and strfry sync — the clients built on WSConnection — cannot reconnect after their first disconnect. The client keeps retrying and failing forever (Attempting to connect → Websocket connection error every 5s) even when the remote is perfectly reachable, and never recovers without a process restart. Since the process never exits, a supervisor (systemd/launchd Restart=) never restarts it — so a brief blip becomes an indefinite stall. (strfry router is not affected: it has its own cron-scheduled reconnect that does not block the loop.)

Root cause

WSConnection::doConnect() implemented the reconnect delay with std::this_thread::sleep_for():

auto doConnect = [&](uint64_t delay = 0){
    if (delay) std::this_thread::sleep_for(std::chrono::milliseconds(delay));
    if (shutdown) return;
    LI << "Attempting to connect to " << url;
    hub.connect(url, nullptr, {}, 5000, hubGroup);
};

That thread also runs the uWS/uv event loop, so the sleep blocks the loop and stalls its cached clock by delay ms (default 5000). The following hub.connect(..., 5000, ...) arms a 5000 ms connection-timeout timer whose deadline is computed against the now-stale clock — already expired in real time — so it fires on the very next loop tick and onEnds the freshly-TCP-connected socket before the WebSocket handshake completes.

The first connection uses doConnect(0) (no sleep → fresh clock) so it works; every reconnect (doConnect(reconnectDelayMilliseconds)) wedges.

Fix

Defer the reconnect with a non-blocking uS::Timer on the hub's event loop instead of sleeping the loop thread, so the loop clock stays accurate and the connection-timeout behaves as intended.

The same principle is why cmd_router.cpp doesn't have this bug: it never sleeps the loop thread — its reconnect delay comes from a separate cron timer that just posts an event to the loop (via hubTrigger), and the actual hub.connect() runs on the loop thread. This patch doesn't copy the router's cron/inbox machinery; a one-shot uS::Timer on the loop is enough for WSConnection.

Reproduce (deterministic)

1. Start a throwaway relay and a stream client pointed at it:

   ./strfry --config relay.conf relay                       # e.g. bind 127.0.0.1:7777
   ./strfry --config client.conf stream ws://127.0.0.1:7777 --dir down
   

   → the client logs Connected.
2. Stop the relay (the client enters the reconnect loop), wait a few seconds, then start the relay again.
3. Before this patch: the client loops Attempting to connect / Websocket connection error every 5s forever and never reconnects, even though the relay is back and a freshly-started client connects to it instantly.
   After this patch: the client reconnects on its next attempt.

Extra confirmation

Lowering reconnectDelayMilliseconds below the 5000 ms connection-timeout (e.g. 250) also makes reconnect succeed with the old code — confirming the delay-vs-timeout clock interaction is the cause.

[kojira]

Background / how this was hit in production (multi-day silent ingest stall): #253