Improve arrow-avro decoding for one-record messages

[jordepic]

Which issue does this PR close?

• Closes Improve arrow-avro decoding of transport-bounded datums and null runs #10712.

Rationale for this change

Some messaging systems deliver records one at a time. For example, Kafka gives a consumer the complete byte array for one message, so the consumer already knows where that message begins and ends. If that message contains one Avro record, the decoder does not need an Avro header to identify the record boundary.

In Avro terminology, one encoded value is called a datum. A datum can be a primitive value or a complete record.

The existing Decoder::decode method expects each record to include an Avro framing prefix. That prefix identifies the Avro format and the writer schema. A caller that already has one complete Kafka message and already knows its schema must therefore add a temporary prefix and copy the payload before decoding it, or separately inspect the schema to determine how many bytes belong to the record.

Nullable nested records have a separate performance cost. Consider an event with three optional nested records where only one is populated on each row. For every absent record, the decoder currently walks through all of its child fields and immediately appends placeholder values. Long sequences of absent records repeat the same work row by row.

What changes are included in this PR?

• Add Decoder::decode_datum, which decodes one complete Avro value directly from the supplied bytes using the writer schema already selected on the decoder. It returns the number of bytes used by that value, leaving any remaining bytes untouched.
• Record consecutive null values immediately in the validity bitmap, but postpone adding their child placeholders. When a non-null value arrives, or the batch is flushed, add the accumulated placeholders as one run.
• Add a benchmark representing three optional nested event records, with the populated record changing every 100 rows.

The second change only affects how the decoder builds arrays internally. The returned Arrow arrays are unchanged, and all child arrays are brought to the correct length before a batch is returned.

Are these changes tested?

Yes.

• cargo test -p arrow-avro --all-features: 475 unit tests passed; 26 documentation tests passed; 1 documentation test ignored.
• cargo clippy -p arrow-avro --all-targets --all-features -- -D warnings
• cargo fmt --all -- --check

The benchmark below decodes all 10,000 rows into one batch. It was run on an Apple M1 Max against a saved main baseline.

Benchmark	main median	this PR median	Change
Three sparse optional nested records	820.16 us	581.44 us	29.2% less time, or 41.3% more throughput

Command:

cargo bench -p arrow-avro --bench decoder -- 'SparseNested\(Struct\)/10000' --baseline main

Are there any user-facing changes?

Yes. Decoder gains a new, non-breaking decode_datum method for callers that already have the complete bytes for one Avro value and have already selected its writer schema. Existing decoding methods and framed input behavior are unchanged.

AI assistance disclosure: Codex was used to help port the implementation, draft tests and the benchmark, and prepare the issue and PR text. The resulting code and all reported outputs were reviewed before submission, and I take responsibility for the contribution.

[alamb]

FYI @jecsand838 -- could you help review this PR?

[jecsand838]

FYI @jecsand838 -- could you help review this PR?

Absolutely! I'll have time tonight / tomorrow morning to review this.

[kinshuk-bb]

@jordepic is there currently a way to convert multiple Avro Datums in a single .avro to a RecordBatches directly? A Raw Binary Encoded Avro