Continue MPS batches after a failed seed exhausts retries

[charlesmartin14]

Purpose

Prevent one transient Apple-MPS/Metal failure from terminating a multi-seed or multi-optimizer batch after the existing fresh-process retry is exhausted.

Behavior

The default MPS policy is now:

worker fails
→ retry from the last verified finite checkpoint
→ if retry also fails, write failure records
→ skip that optimizer/seed
→ continue to the next requested replicate

The failure is not hidden:

• the incomplete run receives run_failed.json;
• the results root receives _batch_status.json;
• the terminal prints a PARTIAL SUCCESS summary;
• no run_complete.json is written for the failed replicate, so completed-run analysis continues to exclude it.

--fail-fast remains available for debugging and restores immediate termination after retries are exhausted.

If a worker fails before any finite checkpoint is written and another retry remains, the partial run directory is archived and the fresh worker restarts from initialization instead of attempting an impossible resume.

Scientific invariants

This does not change model equations, Muon or MuonClip updates, LR schedules, data sampling, evaluation probes, WeightWatcher settings, checkpoint selection, seeds, or result layout. It only changes batch-level error handling.

Tests

The expanded MPS recovery tests verify:

• fresh-process resume from a verified latest checkpoint;
• a failed seed is recorded and the next seed still runs;
• _batch_status.json accurately reports partial success;
• run_failed.json records the incomplete replicate;
• --fail-fast still stops immediately when requested.