Make strict ODELIA swarm runs reliable and exact-client by default

[Ultimate-Storm]

What changed

• Keep producer-side large-model result references alive for the full long-run budget in all six productive ODELIA jobs.
• Retry only missing/raw-timeout learn-task deliveries, retaining acknowledgements already received.
• Make repeated delivery of the same round idempotent and reject stale rounds.
• Treat missing result-submission permission replies as transient while preserving explicit terminal replies.
• In strict mode, reject bad/rejected contributions, prohibit timeout-driven partial aggregation, and count a contribution only after aggregator.accept() plus contribution hooks complete.
• Make the admin preparation path exact-client safe:
  • bare production preparation defaults to the current named eight ODELIA sites;
  • one --strict-clients value derives mandatory clients, participating clients, and all quorum values;
  • conflicting values fail before job files are written;
  • preparation happens in a temporary sibling and preserves the last valid staged job until validation succeeds;
  • generated admin kits carry the matching patcher with the helper.
• Keep reusable source-job quorum values generic for two-client and other validation workflows; supplying explicit --min-* values opts into that custom/test policy.

Why

Two independent production defects were reproduced.

1. In strict job 6e1d08b9, CAM's result reference had the default 1,800-second lifetime. It expired before MHA received serialized-submission permission at 1,881.55 seconds, so the first fetch returned no ref found and the round desynchronized. The producer TTL is controlled by download_complete_timeout, not tensor inactivity timeout.

2. Canary a8c6a75b reported eight accepted contributions and FINISHED:COMPLETED, but actually logged aggregating 7 update(s). The monitor observed the final trainer's reply_time and began aggregation while aggregator.accept() was still processing UKA. The strict path now publishes that completion marker only after successful acceptance and event handling.

The production policy is exact all-site participation. Retries preserve all eight; terminal site failure fails loudly. No pruning or partial continuation is introduced by this PR.

Impact

Future image builds contain the controller and six corrected job configurations. Future generated admin kits contain the strict-default helper and its matching patcher. Existing client identity/certificate kits do not need to be rebuilt merely to run a new image, but the new admin preparation behavior requires the updated helper/patcher (automatically included in newly generated admin kits).

Validation

• Replacement one-round canary 1e4bface-91f8-460b-bcf0-2e14d60b5d72: exact eight started/finished, exactly eight accepted, aggregating 8 update(s), final broadcast to all peers, server ALL_DONE, no errors.
• Current full job 87c5bbee-6f8b-41cb-bd37-e9625cbecee2: rounds 0–11 each aggregated exactly eight updates; round 12 remains healthy with all eight connected at the latest audit.
• 378 passed, 7 skipped, 1 deselected in the complete non-GPU unit suite; only pre-existing metric warnings.
• Focused deterministic tests cover the final-accept race, rejection/event ordering, strict timeout, retry target retention, explicit-reply preservation, abort handling, and duplicate delivery.
• bash -n for both modified shell scripts.
• Python compilation and git diff --check.
• Exact jefftud/odelia:1.6.0 constructor/config compatibility, including download_complete_timeout.
• End-to-end admin-helper staging smoke against the 1.6.0 image produced exact mandatory/participating client lists and 8/8/8 quorum values.

Closes #513.
Partially addresses #466.