feat: bind task-owned RSS callbacks to native shuffle plans (4/n)

[pingzh]

Which issue does this PR close?

Part of #5352. This is the fourth foundational PR and does not close the issue.

Previous PRs:

• feat: add RSS partition writer and task-owned JNI callback (1/n) #5473
• feat: add partition writer destinations to native shuffle plans (2/n) #5476
• feat: add destination-aware native shuffle execution (3/n) #5481

Rationale for this change

Previous PRs introduced RSS partition writers, destination-aware shuffle plans, and native RSS execution. However, serialized RSS plans cannot execute without connecting their task-owned JVM callback to the native planner.

This PR completes that connection while preserving existing local shuffle behavior and JNI compatibility.

What changes are included in this PR?

• Add task-scoped RSS callback registration through a separate JNI method, preserving the existing createPlan signature.
• Pass optional shuffle callbacks through CometExecIterator into native execution contexts and the physical planner.
• Bind RSS shuffle plans to their task-owned callback and preserve callback isolation between tasks.
• Reject RSS plans without callbacks, RSS plans containing local output paths, and callbacks supplied to local shuffle plans.
• Apply the JVM’s conservative maximum byte-array size to RSS frame limits.
• Preserve original callback exceptions across JNI boundaries.
• Clean up native resources and all shuffle inputs when callback registration fails.

How are these changes tested?

• Added planner tests covering RSS binding, task isolation, destination validation, and end-to-end native execution.
• Added Spark/JNI integration tests covering complete RSS frames, original callback exception identity, existing JNI signature compatibility, and registration-failure cleanup.
• Passed 105 Rust tests across planner, shuffle, JNI bridge, and protobuf suites.
• Passed all 35 tests in CometNativeShuffleSuite.
• Passed workspace-wide Clippy, formatting checks, and the full Maven reactor build.

[ziting-openai]

The PR currently points at the already-merged 3/n implementation instead of the advertised 4/n task-owned RSS callback integration.

[sunchao]

Reviewed head 6658894632adf114fb86313c282b2ecf95c8453f against base 4897161704b7b8b7dfa909f4bf897c6508b11117 with five independent reviewer agents. Existing validation: 83 targeted Rust tests passed with default features disabled, and a standalone Arrow 18.3 ownership probe confirmed retention until explicit release. Full Spark/JNI integration tests were not rerun.

[sunchao]

Re-reviewed head 69e3cfb8239806648b79b24148607be16504e378 against base 4897161704b7b8b7dfa909f4bf897c6508b11117 with five independent agents; no new actionable findings. The earlier Arrow cleanup P2 is fixed and its Spark regression passes.

Validation: 105 targeted Rust tests and all 35 CometNativeShuffleSuite tests passed on Spark 4.1.3/JDK 17, with native default features disabled. Other Spark versions and live RSS integration were not tested. CI is still running.