feat: null aware RightAnti hash join execution + planning support

[saadtajwar]

Which issue does this PR close?

• Closes Support null-aware RightAnti hash join (build on the subquery side) for NOT IN #23931

Rationale for this change (copied from issue):

DataFusion plans NOT IN (subquery) as a null-aware anti join, but HashJoinExec only supports null_aware = true for LeftAnti with a single join key (validated in datafusion-physical-plan/src/joins/hash_join/exec.rs). Since HashJoinExec always builds on the left input, the build side of a null-aware anti join is the outer table, not the subquery.

This has two costs:

Memory scales with the wrong side. For SELECT ... FROM big_fact WHERE key NOT IN (SELECT k FROM small_dim), the hash table is built over the entire fact table. Memory is O(outer) when it could be O(subquery).

The operator cannot be distributed. The null-aware logic coordinates three pieces of global state across probe partitions through in-process shared memory: probe_side_has_null: AtomicBool, probe_side_non_empty: AtomicBool, and the visited-build-row bitmap, with the last probe partition to finish emitting the unmatched build rows (hash_join/stream.rs). This is correct and cheap in one process, but engines that split probe partitions across processes get independent copies of all three and produce duplicated or incorrect rows. Ballista hit exactly this (apache/datafusion-ballista#2187) and currently has to force the join into a single task (apache/datafusion-ballista#2188), losing all parallelism.

What changes are included in this PR?

This PR adds physical execution support for null-aware RightAnti hash joins using the CollectLeft partition mode.
The build side now records whether it contains a NULL join key, and the join outputs no rows when the build side contains NULL, filters probe rows with NULL keys, and outputs all probe rows when the build side is empty.

For planner support, JoinSelection was updated to swap a null-aware LeftAnti join to a null-aware RightAnti join when statistics show that the right side is smaller with the swapped join always using the CollectLeft partition mode

Are these changes tested?

Yes

Are there any user-facing changes?

NOT IN queries may now use a null-aware RightAnti physical plan when the subquery is smaller. Query results remain unchanged, but these queries can use less memory and keep the outer table partitioned.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 95.74468% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.85%. Comparing base (39d5064) to head (93f11b5).

Files with missing lines	Patch %	Lines
...fusion/physical-plan/src/joins/hash_join/stream.rs	95.00%	1 Missing and 1 partial ⚠️
...atafusion/physical-optimizer/src/join_selection.rs	93.75%	0 Missing and 1 partial ⚠️
datafusion/physical-plan/src/joins/mod.rs	85.71%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #23957   +/-   ##
=======================================
  Coverage   80.85%   80.85%           
=======================================
  Files        1099     1099           
  Lines      374304   374352   +48     
  Branches   374304   374352   +48     
=======================================
+ Hits       302642   302681   +39     
- Misses      53607    53613    +6     
- Partials    18055    18058    +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[saadtajwar]

cc @andygrove - looking forward to your feedback! I'll have the second PR with the planner support up shortly :)

[saadtajwar]

Do we need the partition_mode check for CollectLeft here? Or should we just count on it being enforced/never chosen as anything but CollectLeft beforehand? I suppose it's possible to explicitly create a plan with a different partition mode if you bypass the SQL -> plan frontend and create directly? Or do we make the assumption that users will only create logical plans and not physical? 👀

[saadtajwar]

cc @andygrove - just to update, this PR should now fully close out #23931 ! Just this one PR now with all changes instead of two :)