fix: bound the reboot wait so an unkillable thread can't wedge the pool

[dunglas]

Context

Following up on review of #2570 and #2564 for #2553.

@henderkes's comment on #2570 (a reboot-frequency throttle) was right that throttling doesn't help a stall during go_post_read/go_ub_write. Digging into why: frankenphp_force_kill_thread()'s own doc comment says it "interrupts the Zend VM at the next opcode boundary." A thread parked in a blocking write to a stalled client (an unresponsive HTTP/2 peer, for example) has already left PHP execution — there's no opcode boundary for the interrupt to land on, so force-kill can't reach it. On some platforms it can't reach the underlying syscall either: macOS never unblocks a blocking syscall this way at all, Windows only wakes alertable I/O.

rebootAllThreads()'s fallback after a failed force-kill was:

thread.sendKillSignal()
thread.state.WaitFor(state.YieldingForReboot) // no timeout

If force-kill doesn't actually unstick the thread, this blocks forever. Since it runs under scalingMu.Lock() held for the whole reboot, one thread stuck this way wedges the reboot of every other thread permanently — scalingMu never unlocks, isRebooting never clears. The same unbounded wait exists one level up in RequestSafeStateChange(), so Shutdown() then hangs on that same thread too.

This plausibly explains the actual mechanism behind #2553's escalating, non-recovering slowdown, independent of how often or why a reboot is triggered (opcache, the watcher, RestartWorkers()) — and independent of opcache being enabled at all, which matches weinbrenner disabling opcache entirely on that issue without the problem going away.

#2574 now bounds the specific go_ub_write stall with a write deadline, which should make that particular trigger rare. This PR isn't about that one trigger, though — it's a generic backstop for anything force-kill can't reach: a hung stream wrapper, flock(), a slow DNS lookup, or any other extension call blocking outside the Zend VM's opcode loop, on any platform including the ones where force-kill has no syscall-level reach at all.

Fix

• Bound the post-force-kill wait with another rebootGracePeriod.
• If the thread still hasn't yielded, give up on it rather than hang: introduce a terminal Abandoned state. The thread permanently drops out of Ready/Inactive (its OS thread may still be alive, so its slot can't safely be reused), and dispatch/shutdown code skips it cleanly instead of panicking or hanging on it. The rest of the pool reboots and keeps serving normally; only that one thread's capacity is permanently lost.
• Abandoned is also added to RequestSafeStateChange()'s terminal-state bucket, which independently fixes Shutdown() hanging forever on the same thread via that function's unbounded WaitFor(Ready, Inactive, Reserved) fallback.

Test

TestRebootDoesNotHangOnUnkillableThread uses a custom http.ResponseWriter whose Write() blocks forever, deterministically simulating a stalled client (no network timing flakiness needed — this is the same code path a real stalled write would hit). On the unpatched code this hangs indefinitely; with the fix it resolves in ~12s (two grace periods: one waiting for a graceful yield, one after the kill signal), and a fresh request afterwards confirms the pool is still usable.

Full go test -race ./... and the caddy package suite both pass.

[dunglas]

Closing: the underlying issue can't be safely fixed this way

CI caught a real crash (signal: segmentation fault) on every Linux job. Chasing it down, my first theory — that pthread_kill(SIGRTMIN+3) landing on a thread parked inside Go's runtime was corrupting shared state — was wrong: removing that signal call didn't stop the crash, and it broke a real, pre-existing regression test (TestRestartWorkersForceKillsStuckThread, from #2365) that depends on the signal genuinely rescuing a thread stuck in a real blocking syscall like sleep().

The actual cause is more fundamental. php_main() in frankenphp.c runs on every reboot cycle, not just final process shutdown — go_frankenphp_main_thread_is_ready() blocks on WaitFor(state.Done, state.Rebooting) and falls through to the same teardown either way:

/* channel closed, shutdown gracefully. drainPHPThreads has already
 * waited for every PHP thread to exit (state.Done), so SAPI/TSRM
 * teardown here is safe. */
frankenphp_sapi_module.shutdown(&frankenphp_sapi_module);
sapi_shutdown();
tsrm_shutdown();

tsrm_shutdown() tears down the global thread-resource-manager state. Its documented precondition is that every PHP thread has already exited. Abandoned is the first code path that violates that on purpose: rebootAllThreads() calls mainThread.state.Set(state.Rebooting) (which triggers this exact teardown) while the abandoned thread's OS thread is still alive, still registered in TSRM, still holding a live EG() mid-blocking-call. That's a use-after-free by construction, and it reproduces identically whether the thread is stuck in a Go/cgo netpoller wait (this branch's own test) or a real sleep() (the pre-existing one) — same mechanism either way, which is what ruled out the signal-specific theory.

I also checked whether per-thread reboots (thread.reboot(), used for max_requests) could sidestep this: they call frankenphp_new_php_thread(), not frankenphp_new_main_thread(), so they never touch TSRM-global state. But rebootAllThreads() exists specifically to reset global engine state (opcache, compiled bytecode cache) that per-thread reboots can't reach — which is exactly why it needs every thread quiesced first. There's no way to relax that without either actually terminating the stuck OS thread (materially riskier: async cancellation of a thread possibly inside Go's own runtime) or never completing the global reboot while one thread is stuck — which is the original hang, not a fix.

The unbounded wait this PR tried to bound wasn't an oversight — it's the system correctly refusing to do something unsafe. I don't think this is fixable as a bounded-wait/give-up patch. Closing.

#2570 (throttles how often a reboot is triggered) and #2574 (bounds go_ub_write with a deadline so a stalled client fails fast instead of blocking) remain valid and complementary: they reduce how often a thread ever reaches this unrecoverable state in the first place, which is the safe lever actually available here.