Fix runner dispatcher shutdown test isolation

docs/logs/engineering-log.md

@@ -1,5 +1,39 @@
 # Engineering Log
 
+## 2026-07-31 (Runner Dispatcher Shutdown Isolation — Issue #1068)
+
+- Symptom: `go test -race ./internal/harness -count=5` failed four of five
+  repetitions because `TestRunnerWithoutShutdownLeaksDispatcher` found some
+  `poolDispatcher` frame after its target Runner's `Shutdown` returned.
+- Cause: the test scanned all goroutine stacks by shared function name, so the
+  assertion had no target identity. Review then found a second defect: five
+  bounded construction sites in `runner_worker_pool_test.go` create seven
+  Runners per package repetition and omitted `Shutdown`, so their dispatchers
+  survived after their tests completed. The production path already closes a
+  target's `done` channel and waits its `dispatcherWG` before returning.
+- TDD red: a deterministic two-Runner fixture kept a control Runner alive,
+  shut down the target, and failed the old global-absence assertion immediately.
+- Review TDD red: a bounded Runner returned from a subtest without its exact
+  dispatcher-exit hook firing; the parent then shut it down explicitly so the
+  red proof did not itself leak.
+- Fix: replace target lifecycle inference with a narrow per-Runner dispatcher
+  exit hook invoked immediately before the existing `dispatcherWG.Done`.
+  The test blocks that exact target hook, proves `Shutdown` cannot return,
+  releases it, then proves Shutdown returns while the control's global stack
+  frame remains visible.
+- Review fix: a shared worker-pool test constructor now registers cleanup that
+  releases any blocked provider before calling bounded `Runner.Shutdown` with
+  a five-second diagnostic deadline. Every affected worker-pool fixture uses
+  that constructor; the cleanup regression itself blocks inside the provider
+  until cleanup establishes the required release-before-Shutdown ordering.
+- Compatibility: queue draining, inflight accounting, cancellation timeout,
+  idempotency, and production shutdown ordering are unchanged.
+- Verification: the cleanup regression passed normal/race; all worker-pool
+  tests passed normal/race at `-count=100`; complete harness race passed at
+  `-count=5`; harness vet passed; and unchanged foreground non-TTY
+  `./scripts/test-regression.sh` passed at 85.6% total coverage with zero
+  uncovered functions.
+
 ## 2026-07-31 (Provider-Key Matrix Health Wait — Issue #1062)
 
 - Symptom: hosted race run `30583930460` failed

docs/logs/long-term-thinking-log.md

@@ -1,5 +1,34 @@
 # Long-Term Thinking Log
 
+## 2026-07-31 (Runner Dispatcher Shutdown Isolation — Issue #1068)
+
+- Command intent: independently classify and repair the 4/5 aggregate race
+  failure without combining remote cron or terminal-publication work.
+- User intent: make shutdown proof identify the exact Runner so CI neither
+  hides a real goroutine leak nor blocks on a healthy unrelated Runner.
+- Success definition: a deterministic two-Runner control proves target exit
+  while a control dispatcher stays live; focused through hosted gates pass on
+  one unmerged closing PR.
+- Non-goals: PR #1060, PR #1055, issue #1067, worker-pool redesign, or longer sleeps.
+- Guardrails: strict red-green TDD, preserve `done`/`dispatcherWG` ownership,
+  queue drain, active cancellation, idempotency, and exact regression commands.
+- Next verification step: capture aggregate race red evidence, then add the
+  deterministic instance-scoped test before changing production code.
+- Root-cause outcome: the exact aggregate command reproduced 4/5 failures and
+  the two-Runner red classified the target result as a process-global identity
+  false positive. Review found real adjacent test leaks as well: five bounded
+  worker-pool construction sites created seven Runners per repetition without
+  `Shutdown`, leaving dispatchers alive after those tests returned.
+- Implementation outcome: target exit identity now comes from a hook ordered
+  immediately before the existing instance wait-group completion; production
+  shutdown ordering and queue accounting remain unchanged. A shared bounded
+  test constructor releases blocked providers before Shutdown and owns cleanup
+  for every affected worker-pool fixture.
+- Review-fix verification: deterministic cleanup normal/race, all worker-pool
+  normal/race x100, complete harness race x5, harness vet, and the unchanged
+  repository regression gate pass. The local commit is ready for parent
+  promotion; PR #1069 remains open and unmerged pending hosted reruns.
+
 ## 2026-07-31 (Provider-Key Matrix Health Wait — Issue #1062)
 
 - Command intent: isolate and clear the hosted race blocker first observed on

docs/logs/observational-log.md

@@ -2,6 +2,27 @@
 
 Use this file for observations about system behavior without immediately prescribing code changes.
 
+## 2026-07-31 (Runner Dispatcher Identity Under Parallel Load)
+
+- Aggregate observation: the original full-package race command reproduced the
+  reported 4/5 failure rate, while the same Runner's instance-owned wait group
+  completed normally.
+- Repetition observation: `go test -count=5` reuses one test process. Five
+  worker-pool construction sites created seven bounded Runners per repetition
+  without calling `Shutdown`; their dispatchers therefore outlived their tests
+  and contaminated later repetitions. They were real fixture leaks even though
+  they did not prove the target Runner leaked.
+- Identity observation: a function-name match in `runtime.Stack(all=true)` can
+  establish that some dispatcher exists, but cannot establish which Runner
+  owns it. Keeping a second Runner alive makes that ambiguity deterministic.
+- Ordering observation: blocking the target dispatcher's final defer prevents
+  its wait-group completion and therefore prevents target `Shutdown` from
+  returning; releasing that exact hook permits return even while the control
+  dispatcher remains live.
+- Cleanup observation: a failure-safe bounded fixture must unblock provider
+  calls before invoking `Shutdown`; otherwise cleanup can wait on the very run
+  the fixture still holds blocked.
+
 ## 2026-07-31 (Source-Workflow Dual-Error Arbitration)
 
 - Process observation: a child can exit non-zero while closing its stdin also

docs/logs/system-log.md

@@ -1,5 +1,23 @@
 # System Log
 
+## 2026-07-31 (Runner Dispatcher Shutdown Identity)
+
+- System/component: bounded `internal/harness.Runner`, `poolDispatcher`,
+  `done`, and `dispatcherWG`.
+- Ownership/order: each Runner closes only its own `done`; its dispatcher calls
+  an optional narrow test hook and then `dispatcherWG.Done`; Shutdown waits
+  that same instance wait group after inflight work is accounted for.
+- Test boundary: one target Runner's hook supplies lifecycle identity while a
+  second control Runner remains live. Process-global stack inspection is used
+  only to prove the control still exists, never to classify target cleanup.
+- Worker-pool fixture boundary: the shared test constructor owns cleanup for
+  bounded test Runners. Its cleanup first releases provider gates, then calls
+  the public `Shutdown` boundary; this preserves the same production ownership
+  contract and prevents dispatchers surviving across `-count` repetitions.
+- Compatibility/failure modes: no API, config, persistence, client, provider,
+  or tool contract changes; existing queue-drain, timeout, and idempotency
+  behavior remains the rollback boundary.
+
 ## 2026-07-31 (Source-Workflow Terminal Error Arbitration)
 
 - System/component: `internal/workflow.SourceManager.runSourceWorkflow` and its

docs/plans/2026-07-31-issue-1068-dispatcher-shutdown-isolation-impact-map.md

@@ -0,0 +1,93 @@
+# Cross-Surface Impact Map: Issue #1068 Dispatcher Shutdown Isolation
+
+## Task
+
+- Task / issue: GitHub #1068
+- Plan link: `2026-07-31-issue-1068-dispatcher-shutdown-isolation-plan.md`
+- Owner: isolated issue #1068 branch
+- Status: review fix implemented and locally verified; promotion pending
+
+## Current Ownership, Callers, and Data Flow
+
+- Entry points: `NewRunner` starts `poolDispatcher` when `WorkerPoolSize > 0`;
+  callers terminate it through `Runner.Shutdown`.
+- Owning packages/types/functions and source of truth: `Runner.done` signals
+  exit and `Runner.dispatcherWG` is the instance-owned completion contract.
+- Callers, consumers, events, and downstream data: daemon/app shutdown and
+  harness tests; no event or persisted-data flow changes.
+- Similar abstractions searched: `inflight`, `shutdownOnce`, worker-pool tests,
+  and process-wide goroutine helpers in `runner_shutdown_test.go`.
+- Search evidence: `rg -n "poolDispatcher|dispatcherWG|Shutdown|runnerGoroutineStackContains" internal/harness`.
+- Duplication/ownership conclusion: the wait group already owns instance
+  completion; the global stack substring is a parallel and unsafe assertion.
+  Every bounded test fixture must also invoke the public `Shutdown` ownership
+  boundary instead of leaving its dispatcher alive after the test returns.
+
+## Config, API, CLI, and Tools
+
+- User-facing config added or changed: none; `WorkerPoolSize` behavior is retained.
+- Defaults / fallbacks: none changed.
+- Environment variables, config files, or saved settings touched: none.
+- Endpoints, request fields, response fields, or server wiring affected: none.
+- CLI commands, tools, wire formats, or integrations affected: none.
+- Error states / validation changes: none.
+
+## Persistence and Compatibility
+
+- Schemas, migrations, caches, generated data, or ownership changes: none.
+- Backward/forward compatibility and versioning: shutdown remains idempotent
+  and preserves queue-drain and timeout behavior.
+- Partial rollout and mixed-version behavior: not applicable; local lifecycle only.
+
+## Lifecycle, Security, and Reliability
+
+- Concurrency, cancellation, retries, cleanup, and resource ownership: primary
+  surface; target dispatcher completion must be distinguished from unrelated
+  Runner dispatchers while `done`, `inflight`, and `dispatcherWG` ordering stays
+  intact. Worker-pool fixtures must release blocked providers before cleanup
+  calls `Shutdown`, including failure paths.
+- Authentication, authorization, permissions, trust, privacy, and secrets:
+  none; search found no auth or data boundary in this lifecycle path.
+- Failure modes, recovery, idempotency, and data repair: guard against a false
+  leak report, a real lingering dispatcher, deadlock, or early return; no data repair.
+
+## Product and Integration Surfaces
+
+- Server/runtime: more trustworthy bounded Runner shutdown verification.
+- TUI/web/macOS/other clients: no direct contract; app/daemon termination benefits indirectly.
+- Provider/model/tool catalog and routing: none; no provider or catalog calls change.
+- External systems and automation: GitHub race/full regression gates only.
+- UX states, keyboard/focus/accessibility/motion: none; no UI code involved.
+
+## Deployment and Operations
+
+- Deployment/migration order and feature flags: separate PR; no migration or flag.
+- Logs, metrics, traces, alerts, and support diagnostics: test failure identifies
+  target/control identity instead of a shared function name.
+- Rollback triggers and recovery steps: revert if shutdown deadlocks, latency
+  regresses, or queue accounting fails.
+- Runbooks and operator docs: no operator command changes.
+
+## Regression Tests
+
+- Characterization and first expected red test: two live bounded Runners prove
+  target instance exit while the old global scan remains positive.
+- New acceptance tests required: `TestRunnerDispatcherShutdownIsInstanceScoped`
+  and `TestWorkerPoolTestRunnerCleanupStopsDispatcher`.
+- Edge, negative, failure, lifecycle, and security tests: existing queue drain,
+  active cancellation timeout, idempotent shutdown, and unbounded-mode tests.
+- Integration/e2e/real-path proof: full harness package race stress and repository regression.
+- Cross-surface regressions to guard: harness/server normal/race/vet.
+- Exact targeted and full commands: cleanup regression normal/race, all
+  worker-pool tests normal/race at `-count=100`, complete harness race at
+  `-count=5`, harness vet, and the unchanged repository regression gate.
+
+## Documentation and Handoff
+
+- Specs/public docs before code: plan and impact map only; no public docs change.
+- Implementation notes/logs/indexes after code: all four logs plus plans index and active plan.
+- Training/onboarding/release notes: none; internal bug-fix PR evidence is sufficient.
+
+## Warning Check
+
+- Every impact heading is reconciled; `none` entries include the searched lifecycle rationale.

docs/plans/2026-07-31-issue-1068-dispatcher-shutdown-isolation-plan.md

@@ -0,0 +1,106 @@
+# Issue #1068: Instance-Scoped Dispatcher Shutdown
+
+## Context
+
+- Governing GitHub issue: [#1068](https://github.com/dennisonbertram/go-code/issues/1068)
+- Problem: the shutdown regression scans every process goroutine for the shared
+  `poolDispatcher` function name, so an unrelated live Runner can make a target
+  Runner look leaked after its own `Shutdown` has completed. Review also proved
+  that five bounded worker-pool construction sites create seven Runners per
+  repetition and omit `Shutdown`, leaving real dispatchers alive across
+  `go test -count` repetitions.
+- User impact: the race gate can block independent changes and obscure a real
+  lifecycle leak.
+- Constraints: strict red-green TDD; retain queue draining, cancellation,
+  idempotent shutdown, and the existing instance-owned wait-group contract;
+  do not touch PR #1060, PR #1055, or issue #1067.
+
+## Scope
+
+- In scope: reproduce the aggregate failure, add a deterministic two-Runner
+  control, make dispatcher lifecycle assertions instance-safe, and repair the
+  Runner seam only if the instance signal proves a runtime leak; deterministically
+  prove and clean up every bounded worker-pool test fixture, releasing blocked
+  providers before shutdown.
+- Out of scope: worker-pool redesign, remote cron recovery, terminal event
+  publication, or generic goroutine accounting.
+
+## Documentation Contract
+
+- Feature status: `implemented and locally verified; review-fix promotion pending`
+- Public docs affected: none; no user-facing contract changes.
+- Spec docs to update before code: this plan and its impact map.
+- Implementation notes to add after code: engineering, observational, system,
+  and long-term-thinking logs.
+
+## Test Plan (TDD)
+
+- First red: a two-Runner regression keeps a control dispatcher live, shuts
+  down the target, proves the target's instance signal completed, and then
+  demonstrates that the old process-global substring assertion still reports
+  a dispatcher.
+- Review red: a bounded Runner created inside a subtest must fire its exact
+  dispatcher-exit hook when fixture cleanup completes; before the shared
+  cleanup is added, the parent observes the dispatcher still live and then
+  explicitly shuts it down to keep the red itself leak-free.
+- Green contract: target `Shutdown` cannot return until the target dispatcher's
+  instance-owned exit signal completes; the control remains live until its own
+  cleanup.
+- Stress: focused normal/race at `-count=100`, complete harness race at
+  `-count=5`, affected harness/server normal/race/vet, and unchanged foreground
+  non-TTY `./scripts/test-regression.sh`.
+
+## Cross-Surface Impact Map
+
+- See `2026-07-31-issue-1068-dispatcher-shutdown-isolation-impact-map.md`.
+
+## Implementation Checklist
+
+- [x] Define acceptance criteria in tests.
+- [x] Link a contract-complete structured GitHub issue before implementation.
+- [x] Record current architecture, callers, consumers, and source-of-truth search evidence.
+- [x] Document feature status and exact contract before code.
+- [x] Complete and reconcile the cross-surface impact map before implementation.
+- [x] Add characterization coverage before structural refactors (no structural refactor planned).
+- [x] Write failing tests first.
+- [x] Review ownership/copy semantics (no exported or copied mutable type changes).
+- [x] Implement review-requested bounded-fixture cleanup.
+- [x] Refactor while tests remain green.
+- [x] Update docs, status ledgers, and indexes.
+- [x] Update engineering/system/observational logs as needed.
+- [x] Rerun the full test suite for the review fix.
+- [x] Update existing PR #1069 and leave it unmerged.
+
+## Verification Outcome
+
+- Red aggregate: `go test -race ./internal/harness -count=5` failed 4/5
+  repetitions at the process-global post-Shutdown stack assertion.
+- Red deterministic: the two-Runner control failed immediately when the old
+  assertion treated the live control dispatcher as the target's leak.
+- Review finding: five pre-existing bounded construction sites in
+  `runner_worker_pool_test.go` created seven Runners per repetition and omitted
+  `Shutdown`; other bounded harness tests were audited and already own explicit
+  shutdown paths.
+- Review red: `go test ./internal/harness -run
+  '^TestWorkerPoolTestRunnerCleanupStopsDispatcher$' -count=1` failed because
+  the bounded Runner's exact dispatcher-exit hook had not fired after subtest
+  cleanup; the test then explicitly released and shut down that Runner.
+- Review green: the cleanup regression passed normal and race at `-count=1`;
+  all worker-pool tests passed normal and race at `-count=100`.
+- Green focused: normal and race `-count=100` passed.
+- Green aggregate: complete `internal/harness` race `-count=5` passed.
+- Green affected: harness/server normal, race, and vet passed.
+- Green repository: unchanged foreground non-TTY `./scripts/test-regression.sh`
+  passed at 85.6% total coverage with zero uncovered functions.
+- Promotion state: the review fix is committed locally for parent handoff; PR
+  #1069 remains open and unmerged, with no review-fix push performed here.
+
+## Risks and Mitigations
+
+- Risk: a test-only hook could become a second lifecycle source of truth.
+- Mitigation: expose only a close-only signal tied to the same defer that calls
+  `dispatcherWG.Done`; `Shutdown` continues to own waiting through the existing
+  wait group.
+- Risk: shutdown changes could deadlock queue draining or active-run timeout.
+- Mitigation: preserve production ordering unless the deterministic instance
+  test proves it wrong, and rerun all existing shutdown/worker-pool regressions.

docs/plans/INDEX.md

@@ -1,5 +1,7 @@
 # Plans Index
 
+- `2026-07-31-issue-1068-dispatcher-shutdown-isolation-plan.md` — Issue #1068 plan for deterministic instance-scoped Runner dispatcher shutdown verification.
+- `2026-07-31-issue-1068-dispatcher-shutdown-isolation-impact-map.md` — Cross-surface impact map for Issue #1068 lifecycle isolation.
 - `2026-07-31-issue-1062-provider-key-matrix-health-wait-plan.md` — Issue #1062 plan for a contention-tolerant provider API-key matrix startup wait.
 - `2026-07-31-issue-1062-provider-key-matrix-health-wait-impact-map.md` — Cross-surface impact map for Issue #1062.
 - `2026-07-31-issue-1064-workflow-exit-precedence-plan.md` — Issue #1064 deterministic source-workflow process-exit diagnostic precedence repair.

docs/plans/active-plan.md

@@ -1,5 +1,13 @@
 # Active Plan
 
+Current status: Issue #1068 dispatcher shutdown isolation and the review-found
+bounded worker-pool fixture cleanup are implemented and locally verified on a
+dedicated branch. The aggregate 4/5 red, deterministic two-Runner red/green,
+cleanup red/green, worker-pool normal/race x100, complete harness race x5, vet,
+and unchanged regression gate are recorded. PR #1069 remains open and unmerged;
+the local review-fix commit still requires parent promotion and hosted reruns.
+PR #1060, PR #1055, and issue #1067 remain excluded.
+
 Current status: Issue #1023 anytime contextual `/feedback` intake is implemented
 test-first and verified in its isolated worktree; targeted, full normal/race,
 coverage-gate, and real TUI bundle checks pass, with merge pending.
@@ -11,6 +19,7 @@ Remaining work before merge is final verification and any requested
 review/cleanup.
 
 Current active plans:
+- `2026-07-31-issue-1068-dispatcher-shutdown-isolation-plan.md`
 - `2026-07-30-issue-1023-feedback-intake-plan.md`
 - `2026-06-26-adapter-first-eval-harness-plan.md`
 - `2026-04-05-orchestration-program-plan.md`