py3 compatibility#2
Merged
seperman merged 1 commit intoqlustered:masterfrom Feb 25, 2015
Merged
Conversation
Member
|
Cool. I just added some logic to only need six and future if it is on python 3. |
seperman
added a commit
that referenced
this pull request
May 4, 2026
Code: - deepdiff/_multiprocessing.py (new) — MPConfig, normalize_mp_config, picklability check, _distance_worker (module-level for spawn), compute_distances_parallel with stable job-index ordering. - deepdiff/diff.py — three new opt-in params, normalized into self._mp_config, propagated via _parameters. New _maybe_compute_pair_distances_parallel helper. One extra dict lookup in _get_most_in_common_pairs_in_iterables before the existing serial _get_rough_distance_of_hashed_objs call. Tests: tests/test_multiprocessing.py (23 tests) — config validation, 10× serial-vs-parallel determinism on nested dicts/repeated items/ties/sets/exclude_paths/ignore_string_case/custom hasher, unpickleable-callback fallback, no-nested-pool guarantee. Full suite: 1149 passed, 10 skipped, 0 regressions. Pyright clean. Doc: docs/multi_processing.md now opens with an "Implementation Status" section listing what's in, the code locations, and what's deferred (subtickets #2/#4/#5/#6 extended matrix/#7) with the reasons each is held back. Two notable design points worth flagging: 1. Workers are spawned without _shared_parameters, so they think they're root and would purge _distance_cache/hashes mid-call. Fixed by passing cache_purge_level=0 to the worker DeepDiff (commented in _distance_worker). 2. Sanitization sets both multiprocessing=False and _mp_config=MPConfig(enabled=False, ...) because recursive DeepDiff with _parameters=... skips the constructor's normalization branch.
seperman
added a commit
that referenced
this pull request
May 4, 2026
…eria are met: - ✅ Parallel _create_hashtable lands behind the existing multiprocessing=True opt-in - ✅ Serial and parallel results match for large lists of dicts, lists of lists, sets, repeated items, nested mixed structures - ✅ Both report_repetition=False and report_repetition=True covered - ✅ Result order matches serial output (verified via 10× repeat-comparison) - ✅ Pickling fallback (unpickleable hasher) tested end-to-end - ✅ Full suite green (1160 passed, 10 skipped); pyright clean
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Are you interested in Python 3 compatibility? I tried to do it as clean and concisely as possible, but this requires future and six. I understand future is pretty big so if you're opposed to it then there should be another (albeit messier) way around it...