perf(sort-merge): cache current-row bytes in RowValues for SortPreservingMerge - #23990
Open
zhuqi-lucas wants to merge 2 commits into
Open
perf(sort-merge): cache current-row bytes in RowValues for SortPreservingMerge#23990zhuqi-lucas wants to merge 2 commits into
zhuqi-lucas wants to merge 2 commits into
Conversation
…vingMerge Multi-column sorts serialize the sort key into arrow `Rows` and wrap it in `RowValues`. Every loser-tree comparison called `Rows::row(idx)` for each side, and each call walks `Arc<Rows>` -> offsets[idx] / offsets[idx+1] -> buffer slice. The offsets buffer is far too large to stay cache-resident, so those lookups typically cost an L2/L3 hit with a DRAM tail — repeated for the same idx on every compare of a stable loser-tree head. Cache the current row's `(ptr, len)` once per `Cursor::advance` (via the existing `CursorValues::set_offset` hook) and read it in `compare`, so the hot path resolves to two plain field loads plus a memcmp. The pointer is into the Arc-owned buffer heap and stays valid across struct moves (the cursor is written into a `Vec<Option<Cursor<..>>>` slot), so `Send`/`Sync` are implemented by hand with a SAFETY note. `eq` / `eq_to_previous` take arbitrary cross-batch indices and continue to index `Rows` directly. This is the first of a series splitting apart the SPM cursor-cache work for easier review; it carries the largest share of the win (multi-column sort-tpch). Follow-ups add the same pattern to the single-column string cursors and a null-wrapper fast path. sort_tpch10 benchmark on the full series showed the multi-column queries (Q4 +1.23x, Q8 +1.13x, Q9 +1.15x, Q5/Q6/Q11 ~+1.12x) driven by this cache. Tests: `test_row_values_cache_matches_rows_index` checks the cached ordering against per-row `Rows` indexing across every offset of a batch, including the cross-batch `eq` path; `test_row_values_single_row_batch` covers the up-front row-0 cache. Existing `sorts::*` merge tests (83) pass.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #23990 +/- ##
==========================================
+ Coverage 80.75% 80.85% +0.09%
==========================================
Files 1096 1099 +3
Lines 373588 374361 +773
Branches 373588 374361 +773
==========================================
+ Hits 301691 302686 +995
+ Misses 53896 53615 -281
- Partials 18001 18060 +59 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
Author
|
run benchmark sort_tpch10 |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/spm-rowvalues-cache (5fbe9bf) to 2e3626e (merge-base) diff using: sort_tpch10 File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagesort_tpch10 — base (merge-base)
sort_tpch10 — branch
File an issue against this benchmark runner |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Part of the SortPreservingMerge cursor-cache work in #23840, split into smaller PRs for easier review (per @alamb / @rluvaton). This one carries the largest share of the win.
Rationale for this change
SortPreservingMergecompares cursor heads in a loser tree; every emitted row triggerslog2(k)comparecalls. For multi-column sort keys the key is serialized into arrowRowsand wrapped inRowValues, and eachcomparecalledRows::row(idx)for both sides — walkingArc<Rows>→offsets[idx]/offsets[idx+1]→ buffer slice.The offsets buffer (~65 KB per batch per partition, ×16 partitions) is far too large to stay cache-resident, so those lookups typically land in L2/L3 with a DRAM tail — and they are repeated for the same
idxon every compare of a stable loser-tree head, even though the answer never changes until the cursor advances.What changes are included in this PR?
Cache the current row's
(ptr, len)once perCursor::advance(via the existingCursorValues::set_offsethook) and read it incompare, so the hot path resolves to two plain field loads plus amemcmpinstead of two Arc-chased offset walks.Vec<Option<Cursor<..>>>slot), soSend/Syncare implemented by hand with a SAFETY note.eq/eq_to_previoustake arbitrary cross-batch indices and continue to indexRowsdirectly (the cache only holds the current offset).comparekeeps a debug-only assert that the cache invariant holds (indices equal the cursors' current offsets).Only
datafusion/physical-plan/src/sorts/cursor.rschanges. Follow-up PRs will apply the same pattern to the single-column string cursors (ByteArrayValues,StringViewArray) and add a null-wrapper fast path.Are these changes tested?
Yes:
test_row_values_cache_matches_rows_indexdrives the cache across every offset of a multi-row batch and asserts identical ordering to per-rowRowsindexing, plus the cross-batcheqpath.test_row_values_single_row_batchcovers the up-front row-0 cache and the length snapshot.set_offset(skip the refresh) makes the first test fail as expected.sorts::*merge tests (83) pass.Are there any user-facing changes?
No API changes. The
sort_tpch10benchmark from the combined series (#23840) showed the multi-column queries driven by this cache: Q4 +1.23x, Q9 +1.15x, Q8 +1.13x, Q5 / Q6 / Q11 ~+1.12x; no regressions. CI benchmark to confirm on this split.