feat(bigquery): add ArrowDeserializer helper utility - #13943
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces the ArrowDeserializer class to convert Apache Arrow schemas and record batches into BigQuery schemas and FieldValueList rows. Feedback on the implementation highlights several critical issues: a potential native memory leak if vector creation fails, a bug in repeated record deserialization where subfields are omitted, missing defensive checks for schema and size mismatches, and an inefficient timestamp conversion that should cast to TimeStampVector and handle different time units.
e149afc to
796b77f
Compare
|
@gemini-code-assist review |
There was a problem hiding this comment.
Code Review
This pull request introduces comprehensive documentation (including developer, user, and storage API guides) and adds support for Apache Arrow serialization and deserialization in the BigQuery client and JDBC driver. It also introduces a centralized type registry and temporal utilities for optimized type coercion. The review feedback highlights several critical improvements: optimizing memory allocation and preventing resource leaks in ArrowDeserializer by passing the BufferAllocator and using proper try-finally blocks; avoiding precision loss in timestamp formatting by using integer division; ensuring that the new Arrow configuration options are actually transmitted to the BigQuery backend by serializing them in QueryJobConfiguration.toPb(); and adhering to timezone handling rules by using legacy Calendar manipulation and LocalTime.ofNanoOfDay for java.sql.Time conversions.
|
@gemini-code-assist review |
There was a problem hiding this comment.
Code Review
This pull request introduces Apache Arrow serialization options and deserialization support to the BigQuery client library, alongside extensive documentation and optimized type registry utilities for the JDBC driver. The review feedback highlights critical issues where the new Arrow configuration fields in QueryJobConfiguration are not serialized or deserialized in toPb() and fromPb(), rendering them ineffective. Additionally, the reviewer recommends reusing a single BufferAllocator in ArrowDeserializer to avoid high allocation overhead, ensuring exception-safe LIFO cleanup of FieldVector instances to prevent memory leaks, and using a timezone-aware manual conversion when converting java.sql.Time to java.time.LocalTime to preserve millisecond precision.
9056c4c to
9ddd9d2
Compare
9ddd9d2 to
712b972
Compare
0a6ecc7 to
b76cd0c
Compare
…nd formatting in ArrowDeserializer
b76cd0c to
6215690
Compare
Stacked PR 2 of 3: Adds the ArrowDeserializer class which handles decoding serialized Arrow schemas and record batches into standard FieldValueList rows.