Add versioned auto-migration for table configs and schemas

[xiangfu0]

Summary

Adds a versioned config-migration framework that transparently upgrades stored TableConfigs and Schemas to the current version on the controller, so cluster upgrades don't surprise users with configs the new controller can no longer parse or validate.

The migration runs as a leader-only controller periodic task. For every table it reads the config exactly as stored, runs an ordered migrator chain, and — only if something changed — persists the result through the standard PinotHelixResourceManager write path (version-checked write plus broker/server cache-refresh messages). It's idempotent: once everything is current, runs are cheap no-ops.

What's included

• pinot-spi — new org.apache.pinot.spi.config.migration package: ConfigMigrator / TableConfigMigrator / SchemaMigrator, ConfigMigrationRegistry (dense ordered chain), MigrationResult, ConfigMigrationUtils. Version markers are rollback-safe: Schema gains a configMigrationVersion field (older readers ignore it via @JsonIgnoreProperties), TableConfig uses a controller-managed config.migration.version custom-config key.
• pinot-segment-local — first concrete migrator (v0 → v1) folds deprecated ingestion fields (tableIndexConfig.streamConfigs, segmentsConfig.segmentPushType/segmentPushFrequency) into ingestionConfig, finally wiring the previously test-only TableConfigUtils.convertFromLegacyTableConfig into a live path. Schema chain ships empty (framework ready; no placeholder transforms).
• pinot-controller — ConfigMigrationManager periodic task + wiring in BaseControllerStarter; config keys in ControllerConf; CONFIG_MIGRATION_SUCCESS/CONFIG_MIGRATION_FAILURE meters.
• pinot-common — fixes toTableConfig to preserve env-var substitution when applyDecorator=false; adds a raw-read getTableConfigWithVersion overload.

Configuration

Key	Default	Notes
controller.config.migration.enabled	true	Set false to opt out
controller.config.migration.frequencyPeriod	1h
controller.config.migration.initialDelaySeconds	randomized
controller.config.migration.cronExpression	(none)

Safety / compatibility

• Optimistic concurrency: version-checked writes never clobber a concurrent operator edit — a lost race is skipped and retried next cycle.
• Validate before persist: a buggy migrator can never write an invalid config; a not-yet-created schema is treated as transient (skip, no failure metric).
• Rollback-safe: an older controller ignores the new marker field/key; JSON with the marker + unknown future fields still deserializes.
• Cache convergence: writes go through the standard RM path so broker/server in-memory caches refresh.

Tests

• Deprecated → new upgrade (pinot-segment-local): stream-only, batch-only, and combined deprecated configs fold into ingestionConfig; deprecated fields cleared; marker stamped; survives a ZK ZNRecord serialization round-trip; already-migrated config is a no-op.
• Registry/markers (pinot-spi): in-order chain, version-compare skip, downgrade left untouched, dense-ordering enforced, TableConfig marker JSON round-trip (preserving user custom configs), Schema marker round-trip + unknown-property tolerance.
• Controller task: persist-through-RM with version check + success metric, already-current not written, missing-schema skip, persist-failure caught + metered, schema migrated once across hybrid halves.

All precommit checks pass (spotless, license, checkstyle: 0 violations) across the four touched modules.

Notes for reviewers

• Enabled by default and rewrites stored configs cluster-wide on first upgrade — please confirm this warrants a release note and docs for the new controller.config.migration.* keys.
• Adds a new public SPI package and a new Schema field — flagging for plugin-maintaining teams (additive, rollback-safe).

[codecov-commenter]

Codecov Report

❌ Patch coverage is 80.76923% with 30 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.10%. Comparing base (f5fee8e) to head (9eb0a5e).

Files with missing lines	Patch %	Lines
...elix/core/periodictask/ConfigMigrationManager.java	74.28%	14 Missing and 4 partials ⚠️
...apache/pinot/controller/BaseControllerStarter.java	0.00%	3 Missing and 1 partial ⚠️
...not/spi/config/migration/ConfigMigrationUtils.java	77.77%	2 Missing and 2 partials ⚠️
...va/org/apache/pinot/controller/ControllerConf.java	75.00%	1 Missing and 1 partial ⚠️
...ache/pinot/common/metadata/ZKMetadataProvider.java	50.00%	1 Missing ⚠️
.../spi/config/migration/ConfigMigrationRegistry.java	96.77%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #19270      +/-   ##
============================================
- Coverage     67.12%   67.10%   -0.03%     
  Complexity     1424     1424              
============================================
  Files          3462     3468       +6     
  Lines        220677   220832     +155     
  Branches      35255    35275      +20     
============================================
+ Hits         148136   148195      +59     
- Misses        60708    60809     +101     
+ Partials      11833    11828       -5     

Flag	Coverage Δ
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (?)
java-25	67.10% <80.76%> (-0.03%)	⬇️
lane-a	100.00% <ø> (ø)
lane-b	0.00% <ø> (ø)
temurin	67.10% <80.76%> (-0.03%)	⬇️
unittests	67.10% <80.76%> (-0.03%)	⬇️
unittests1	57.79% <72.97%> (-0.02%)	⬇️
unittests2	39.14% <77.56%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[xiangfu0]

Self-review pass (multi-domain) + fixes applied

Ran an independent multi-domain review over the branch diff. 0 CRITICAL, 7 MAJOR. Addressed the substantive findings in the latest commit:

Fixed

• Schema re-migration infinite loop (verified bug): marker-only schema migrations were persisted via PinotHelixResourceManager.updateSchema, which short-circuits on schema.equals(oldSchema) — and Schema.equals() deliberately excludes the migration marker, so the marker never persisted and the schema re-migrated every cycle (with false success metrics). Latent today (empty schema chain) but a real trap. Now persisted directly via ZKMetadataProvider.setSchema + best-effort refresh messages. Covered by a new test asserting the stored marker advances and a refresh is sent.
• Default flipped to OFF (opt-in): the task rewrites stored ZK configs cluster-wide, so it now defaults off and is intended to be enabled only after all nodes are upgraded. Resolves the rolling-upgrade/rollback and soak-period concerns.
• Empty-chain short-circuit: no ZK reads/writes when a migration chain is empty (the shipped schema chain is empty), and the schema is fetched once per table.
• Mixed-version field clearing: documented on the migrator; verified current readers resolve via the ingestionConfig-preferred fallback, so same-version reads are unaffected — the opt-in-after-upgrade posture covers the old-binary-rollback edge.
• More tests: deprecated ingestion-config collision folding (existing ingestionConfig wins, deprecated cleared), schema-marker persistence + refresh + once-per-hybrid-run, empty-chain short-circuit, and Schema serialization asserted through the full objectToString path.
• Renamed MIGRATION_VERSION_KEY → CONFIG_MIGRATION_VERSION_KEY; documented the controller-managed reserved custom-config key.

Deferred (tracked, low-risk)

• A ControllerTest-based end-to-end test through the real ZK write path (current controller tests mock the resource manager) — worth adding but the unit coverage now exercises the persistence/skip/validation/metric paths.
• SPI callout: this adds a new org.apache.pinot.spi.config.migration package and a Schema.configMigrationVersion field (both additive, rollback-safe) — flagging for plugin/enterprise maintainers.

All precommit checks (spotless, license, checkstyle: 0 violations) pass on the four touched modules; unit suites green (spi 29, segment-local 6, controller 6).