Add opt-in master failover recovery with Ferry-wide source swap (GTID stage 6)

[driv3r]

What

Adds opt-in master failover recovery for Ghostferry's source. When the source MySQL master fails over, Ghostferry reconnects to the promoted writer and the entire run follows it — not just the binlog stream, but also the data iterator, verifiers, and ad-hoc source queries.

Stage 6 of the GTID coordinate stack (builds on #449 / gtid-stage5-replica-progress).

Why

Previously a source failover killed Ghostferry: a non-timeout GetEvent error went straight to ErrorHandler.Fatal, and go-mysql's BinlogSyncer only reconnects to the same configured host. There was no path to follow the writer to a new host.

GTID mode makes safe failover possible: GTID sets are server-independent, so a resume set valid on the old master is meaningful on the new one. File/position coordinates are per-host and cannot be carried across a failover, so the feature refuses to enable in file/position mode.

How

Implemented in three commits:

1. Streamer-level recovery

• MasterWriterResolver (supplied by the embedding application) discovers the new writer. MasterFailoverRecoveryConfig carries it plus retry bounds (MaxAttempts, RetryWait).
• On a non-timeout GetEvent error, the Run loop attempts recovery before fataling. Each attempt: resolve a candidate → validate it → build a fresh syncer against it → StartSyncGTID from the last resumable set (replaying the interrupted transaction).
• validateFailoverTarget fails closed. A candidate is accepted only if it is a writer (not @@read_only), has @@GLOBAL.gtid_mode = ON, and its executed GTID set contains everything already applied downstream — the committed streamed set folded together with the in-flight transaction's GTID (tracked from GTIDEvent/XIDEvent). A candidate missing any applied transaction is rejected to prevent silent target divergence.
• Concurrency: DB/DBConfig swaps happen under a connMu lock; FlushAndStop reads the DB through currentDB() so cutover records the stop coordinate from the new writer. Only failover-owned DB handles are closed; the shared source handle is never closed there.

2. OnFailover notification hook

• MasterFailoverRecoveryConfig.OnFailover is invoked after a successful swap with a MasterFailoverEvent (new writer config + live DB handle, previous config). This is the seam the Ferry uses to repoint the whole run. Callback errors are logged but do not abort the already-recovered stream.

3. Ferry-wide source swap

• swapSource opens a Ferry-owned connection to the promoted master (rather than adopting the streamer's handle, so lifecycles don't entangle) and repoints:
  • Ferry.SourceDB, Config.Source
  • DataIterator.DB + CursorConfig.DB
  • inline verifier SourceDB and its prepared-statement cache (reset, since it's keyed by query text and would otherwise stay bound to the old DB)
  • the configured verifier (iterative: both SourceDB and its CursorConfig; checksum: SourceDB), with typed-nil guards
• Previous source handles are retained, not closed at swap time (in-flight cursors capture their DB by value and cached statements may still reference it); closePreviousSourceDBs releases them at teardown.
• Serialized by Ferry.sourceSwapMu; CurrentSourceDB() is the lock-safe accessor.
• failoverRecoveryConfigForSource shallow-copies the user's config and composes the Ferry swap ahead of any user callback, without mutating the user's config.

Safety notes & scope

• Fully backward compatible and opt-in: nil config keeps the previous behavior (lost connection stays fatal).
• File/position mode is rejected at config validation.
• Cursors and prepared statements created before a swap keep their old DB reference (CursorConfig is copied by value into each Cursor), so a failover mid-row-copy is only fully picked up by work started after the swap. Failover recovery matters most at/after row-copy completion (binlog catchup and cutover), where this is sufficient. Runtime readers that need strict consistency should go through CurrentSourceDB().

Testing

• New unit tests: recovery gating, resolver adapter, GTID union/clone helpers, in-flight GTID tracking, config validation, OnFailover composition, and the Ferry-wide swap (all consumers repointed, stmt-cache reset, previous-handle retention, nil-safety, teardown drain).
• go test ./... unit suite passes with -race.
• Full test/go integration suite passes in both file_position and gtid modes.
• go build + gofmt clean.

Reviewer

Reviewed by automated code review across all three commits; data-safety findings addressed (applied-set + in-flight GTID containment, DB-swap race, shared-handle close semantics, verifier CursorConfig/stmt-cache repointing, DB ownership across failover, nil-safety).