feat(physical-plan): Make HashTableLookupExpr serializable

[barbarj]

Which issue does this PR close?

• Closes make HashTableLookupExpr serializable #24277

Rationale for this change

In order for DataFusion Distributed (and presumably other distributed DF projects) to make use of HashTableLookupExpr as a dynamic filter across network boundaries, it needs to be serializable. (for instance, see: datafusion-contrib/datafusion-distributed#623)

What changes are included in this PR?

The only sticky part of serializing HashTableLookupExpr is its Map. The inner members of the two variants of Map, HashMap and ArrayMap both support much more functionality than is needed to evaluate this as a dynamic expression. So, in order to simply serialization and minimize the on-wire size, we serialize only the aspects required for expression evaluation (i.e. membership checks)

We replace HashTableLookupExpr's Map with a local version that includes the membership-only variants. These are implemented such that they are only constructible via deserialization.

Are these changes tested?

The roundtrip_hash_table_lookup_expr_to_lit test is replaced with two regular roundtrip tests, one each for the HashMap and ArrayMap versions of HashTableLookupExpr.

This PR also adds a bunch of tests testing the post-deserialization behavior of the new membership-only variants.

Are there any user-facing changes?

• JoinHashMapType has a new required method hashes

[codecov-commenter]

Codecov Report

❌ Patch coverage is 63.46154% with 266 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.24%. Comparing base (e2e161d) to head (1329322).
⚠️ Report is 71 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/proto-models/src/generated/pbjson.rs	0.00%	234 Missing ⚠️
...-plan/src/joins/hash_join/partitioned_hash_eval.rs	93.81%	22 Missing and 6 partials ⚠️
...usion/physical-plan/src/joins/stream_join_utils.rs	0.00%	3 Missing ⚠️
datafusion/proto/src/physical_plan/from_proto.rs	0.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24382      +/-   ##
==========================================
- Coverage   81.27%   81.24%   -0.04%     
==========================================
  Files        1114     1114              
  Lines      393619   394327     +708     
  Branches   393619   394327     +708     
==========================================
+ Hits       319931   320375     +444     
- Misses      54950    55204     +254     
- Partials    18738    18748      +10     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[barbarj]

If this doesn't make it into 55.0, this will need to be moved to 55.1

[stuhood]

@jayshrivastava, @barbarj, @adriangb: The deciding line between List and Hash is already configurable: relatively small sets get List, larger sets get Hash.

Given that, should the presence of a Hash like this already suggest that systems which care about serializing a Hash should probably be using a bloom filter instead? Or is the thinking that there are multiple size thresholds, and Hash needs to be serializable for the middle one?:

• small: List
• medium: Hash
• large: "bloom"

[adriangb]

I think the main thought behind the original design and split was that locally at least there's no point in building a Bloom filter if you already have a hash table in memory. It makes sense to me that if you want to serialize across the wire the tradeoff is very different, a bloom filter would be better. I'm not sure if that means you would want to build a Bloom filter upfront or build it when you serialize.

[barbarj]

My intent here was to keep the List/Hash choice across the network, though I can see a case for bumping the Hash to "bloom" when serializing to keep the wire size down. But I'd expect we'd need another intra-DataFusion reason to add the bloom filter code besides this.

Related: #16435