table: native scan and Arrow stream integration

[abnobdoss]

Status: active integration branch

Scope:

• Env-gated native Arrow batch-reader path for PyIceberg scans.
• Table-property and env mode selector for PyArrow, Rust read, and Rust plan-and-read.
• Eager to_arrow materializes through the same guarded native batch reader when a native mode is enabled.
• Arrow PyCapsule stream support for append, overwrite, Table, DataScan, to_polars, and to_duckdb.
• Native equality-delete payload conversion at the adapter boundary.
• Opt-in native upsert match-filter route for benchmarking.
• Reproducible benchmark harness that records wall time, CPU time, RSS, rows, checksum, and fallback status.

Current value proven on the rebased branch:

• Local 1M rows across 16 files: Rust plan-and-read is correctness parity but not a speed win. Full, projection, and filter are about 0.85x to 0.93x PyArrow wall time and use more RSS.
• Local planning-heavy 128-file scans: Rust plan-and-read is about 1.32x faster for projection and 1.53x faster for filter, with lower CPU but much higher RSS.
• Local planning-heavy 512-file spot check: Rust plan-and-read is about 1.49x faster for projection and 1.52x faster for filter, again with much higher RSS.
• PyCapsule PyArrow handoff is effectively parity with normal PyArrow in this harness; it improves interop but is not a scan-planning speedup by itself.

Verification:

• Focused pytest suite passed for Arrow capsule, scan backend selection, pyiceberg-core adapters, and native upsert route.
• Commit hooks passed: trailing whitespace, EOF, AST, Ruff, Ruff format, mypy, pydocstyle, codespell.
• Local fork pyiceberg-core release build imports the required scan, expression, schema, and file_io modules.

Known limits:

• Published pyiceberg-core package does not expose the required native scan modules; reviewers need a local fork wheel until upstream packaging catches up.
• Equality-delete support is adapter-level only so far; end-to-end equality-delete read parity still needs a real equality-delete table fixture.
• Remote object-store, latency-injected S3, and concurrent workload behavior are not measured yet.
• Memory amplification is the main production risk in the planning-heavy wins.

No external issue references.