fix: FusedCodecPipeline must apply outer AA/BB codecs on partial paths - #4202
Merged
Conversation
FusedCodecPipeline.supports_partial_decode/supports_partial_encode passed require_no_aa_bb=False, unlike BatchedCodecPipeline (True). With an outer array-array or bytes-bytes codec around a sharding serializer (e.g. compressors=[GzipCodec()], or filters=[TransposeCodec()]), the fused pipeline's partial read/write branches called ShardingCodec's partial sync methods directly on the raw stored value, skipping those outer codecs entirely. That wrote non-conforming bytes for an outer BB codec (unreadable by BatchedCodecPipeline or any conforming reader) and silently produced wrong data for an outer AA codec. Pass require_no_aa_bb=True in both fused properties so these chains fall through to the full-chunk fused path instead, matching batched behavior. Adds cross-pipeline parity coverage (full and partial read/write) for sharding with an outer compressor and with an outer transpose filter, and removes the "known limitation" exclusion that previously kept the sharding+compressor case out of the nested-sharding parity matrix. Assisted-by: ClaudeCode:claude-sonnet-5
d-v-b
force-pushed
the
fix/fused-partial-aa-bb
branch
from
July 29, 2026 14:32
6f44f3a to
3afccd2
Compare
d-v-b
marked this pull request as ready for review
July 29, 2026 14:37
Contributor
Author
|
self-merging, since this fixes a data corruption issue! |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4202 +/- ##
==========================================
- Coverage 93.90% 93.90% -0.01%
==========================================
Files 91 91
Lines 12672 12670 -2
==========================================
- Hits 11900 11898 -2
Misses 772 772
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the fused codec pipeline to avoid data corruption caused by failing to avoid partial decode path for codec pipelines combining "container" serializers (sharding) flanked by array-array and / or bytes-bytes codecs.
Assisted-by: ClaudeCode:claude-sonnet-5
based on a claude-authored PR here: d-v-b#255
Summary
[Describe what this PR changes and why, in your own words.]
For reviewers
[What would you most value a second look at? What are you already confident in? For a refactor, say whether behavior is meant to be unchanged.]
Author attestation
TODO
docs/user-guide/*.mdchanges/