[feature](lance) Durable Lance index job infrastructure with fence, quota and replay

[u70b3]

What problem does this PR solve?

Issue Number: #66497

Related PR: #66637 (merged), #67201 (PR3A, open — independent; this PR shares no files with it), #66671 (open — independent)

Problem Summary:

This is the second sub-PR (PR3B) of delivery slice 3 of the v5.1 design (final 4.2 contract, scope confirmed in this review): the durable Lance index job infrastructure — job records, the compact dual-state lifecycle, the same-name fence, three-level unresolved quotas, typed results, and replay semantics — behind a master-owned manager. This PR has no user-visible entry point: no SQL, no admission, no dispatcher, no worker, no enablement gate. It is the persistence and state-machine foundation that PR3C (admission + job SQL) and PR3D (dispatch + fake-worker fault tests) build on.

What this PR adds:

• The minimal durable job record of Section 7.2 (LanceIndexJob): job identity/creator/revision/bounded timestamps, persisted target identity and same-name fence key material (catalog id, DIRECTORY provider tag, normalized dataset locator, display + normalized index name), mutation intent, admitted dataset version and the ordered schema-contract-v1 representation (Section 4.2), the independent mutation/refresh states, the typed result with bounded sanitized message, dispatch identity (backend id, BE process epoch, immutable invocation id, deadline), possible-live ownership with termination proof, and the FORCE audit fields (populated by PR3E; only replay semantics land here). The record carries no credentials and no unbounded values (Section 4.3/8).
• The Section 6.1/6.2 dual state machines and the Section 6.3 provider-result classification table as data plus one pure classify function (13 typed result codes × CREATE/REPLACE/DROP × IF-flags × external-advancement), including IF_CONDITION_NOOP only for DROP IF EXISTS + LANCE_ERR_NOT_FOUND. Normalization v1 (Section 4.1) for index names (toLowerCase(Locale.ROOT)) and dataset locators (scheme case, trailing slashes, credential-bearing URL rejection); the fence key and the three-level (table-locator/catalog/global) unresolved-quota counters of Section 5.4.
• LanceIndexJobManager (Appendix B seam: a master-owned minimal job/fence manager; it deliberately reuses neither the generic scheduling JobManager nor internal IndexChangeJob, neither of which provides external one-shot CAS, no-redispatch, same-name fence, or possible-live semantics). All durable transitions share one write-path shape — validate under the write lock (state legality, revision CAS, callback identity), append one upsert record to the edit log, then apply the same record locally — so master and followers run identical apply logic. Fence and unresolved quota live and die together exactly as Sections 5.4/6.4 require: held by PENDING/RUNNING, by terminal jobs until their required refresh is DONE, and by UNKNOWN until a durable FORCE_RELEASE; a quota or fence rejection precedes any durable write, leaving no job, no fence, and no record.
• Replay per Section 7.3. replayUpsertJob is a verbatim replace with a monotonic-revision guard and performs no state transformation — a follower tailing a live master must keep a fresh RUNNING record RUNNING. The RUNNING→UNKNOWN transition happens only in the master-election sweep (Env.transferToMaster, after metadata replay and before any master daemon starts, mirroring the insertOverwriteManager.allTaskFail() precedent), which writes UNKNOWN upserts through the same identity-checked channel so followers converge and stale callbacks (revision/invocation/epoch mismatch) are rejected. Refresh RUNNING is downgraded to REQUIRED at the sweep so the idempotent external-table refresh can resume; replay never calls lance-c again.
• Persistence wiring per Doris convention: one new edit-log op (OP_LANCE_INDEX_JOB_UPSERT = 500), JournalEntity/EditLog dispatch, a new lanceIndexJobManager image module appended to PersistMetaModules.MODULE_NAMES (no FeMetaVersion bump; old images simply never invoke the load method, and Env pre-initializes an empty manager).

Explicitly not in this PR: SQL/admission/IF preflight and job SQL (PR3C, including the Section 9.7 gating configuration — quota limits here are parameters, not Config); BE selection, dispatch, possible-live slot reservation on BEs, and worker invocation (PR3D); the FORCE_RELEASE transition and its auth protocol (PR3E — only the durable fields and the replay row exist); Arrow-schema→contract construction (PR3C admission); job-record retention/GC (follows the bounded-retention policy with PR3E). PR1's SHOW INDEX and PR2's inspection surface are untouched; this branch shares no files with PR3A.

Release note

None (internal infrastructure only; no user-visible behavior change).

Check List (For Author)

• Test

  • Regression test
    • None: this PR adds no SQL surface and no external behavior; regression suites for the DDL surface belong to PR3A ([feature](lance) Add statically validated Lance index DDL surface #67201) and the end-to-end lifecycle to later slices.
  • Unit Test
    • New suites under org.apache.doris.datasource.lance.job (105 cases, all green with checkstyle enabled): normalization v1 (incl. the Turkish-İ corner), locator normalization and rejection forms, the full Section 6.3 classification matrix (independently restated per cell), state-machine legality (UNKNOWN has no outgoing transitions; refresh independence; revision CAS), fence/quota co-release timing, quota three-level boundaries, the Section 7.3 replay matrix (PENDING re-dispatchable once; RUNNING swept to UNKNOWN and never redispatchable; terminal jobs resume only refresh; UNKNOWN rebuilds fence/quota/possible-live; force-released UNKNOWN frees the name), stale-callback rejection, replay idempotence and monotonic revision, corrupt-record tolerance, manager image write/read round-trip with derived-state rebuild, and JournalEntity op-500 round-trip.
    • Scoped regression green (469 cases): org.apache.doris.persist.**, org.apache.doris.journal.**, org.apache.doris.dictionary.**, org.apache.doris.indexpolicy.**, org.apache.doris.job.**, org.apache.doris.datasource.lance.** (cd fe && mvn test -pl fe-common,fe-core -am -DfailIfNoTests=false -Dtest='...', no -Dcheckstyle.skip).
  • Manual test
    • mvn compile -pl fe-common,fe-core -am green with checkstyle (validate phase); the new image module binding resolves at PersistMetaModules static init; the edit-log op code 500 verified unique repo-wide.

• Behavior changed:

  • No. New code paths are unreachable from any SQL or RPC surface in this PR; existing edit-log ops, image modules, and manager behaviors are unchanged.

• Does this need documentation?

  • No. User documentation tracks the broader lifecycle work in [Feature](lance) Add vector and scalar index lifecycle management #66497 once mutation is enabled.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?