ENH: Fix multiprocessing pickling issue in NDArrayBacked with datetime64[ns] MultiIndex #63080
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…e64[ns] MultiIndex Signed-off-by: SoulSniper1212 <warush23@gmail.com>
rhshadrach
reviewed
Nov 16, 2025
Member
rhshadrach
left a comment
rhshadrach
left a comment
There was a problem hiding this comment.
This is adding a lot of cases and it is not clear to me why pandas must handle those. In addition, please adhere to the pandas development standards: https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html
Comment on lines
+109
to
+110
| # Handle case where (array, dtype) is passed instead of (data, dtype) | ||
| dtype, data = data, dtype |
Member
There was a problem hiding this comment.
I think it would be good to understand more about what the conditions are and why this case occurs.
Comment on lines
+127
to
+128
| # This can occur when pickle/unpickle happens in multiprocessing contexts like joblib | ||
| # where additional pickling/unpickling steps might create unexpected state formats |
Member
There was a problem hiding this comment.
Same thing here. Why are unexpected state formats occurring?
jbrockmendel
added
the
AI Slop
Member
|
Did a human write this? |
Member
|
Based on the contributor's history, I believe this is an AI generated PR so closing. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
This pull request fixes an issue (#63078) where pandas DataFrames with datetime64[ns] in MultiIndex would fail when processed using joblib or other multiprocessing libraries. The error occurred in the
NDArrayBacked.__setstate__method during unpickling, where unexpected state formats from multiprocessing contexts would trigger aNotImplementedError.Checklist
pandas/_libs/arrays.pyxto handle additional state formats inNDArrayBacked.__setstate__test_fix.pyto verify the fix works for the reported scenarioProof
The fix addresses the issue by adding handling for:
The test script
test_fix.pydemonstrates that the fix resolves the issue by:The changes maintain backward compatibility while adding robustness to handle multiprocessing-related pickling variations.
Closes #63078