Determinism implementation in nnunet#2871
Open
Luugaaa wants to merge 3 commits intoMIC-DKFZ:masterfrom
Open
Conversation
This was referenced Jul 18, 2025
Author
|
Note : I was finally able to test the determinism fix on CUDA. Although the reproducibility is greatly improved, the training is not deterministic. I've pinpointed the source of the non determinism to something happening in the backward path from |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi MIC-DKFZ team,
This PR introduces updates to enable fully deterministic training in nnU-Net, which is crucial for reproducibility in research. The changes include adding a
deterministicflag, implementing aseed_everythingfunction, and ensuring the data augmentation pipeline is correctly seeded. I've been working on a similar fix forbatchgeneratorsand believe these changes will work together to make the entire training process reproducible.Problem
Achieving deterministic behavior in a multi-process environment can be tricky. Even with seeding, sources of randomness can persist, especially in the data augmentation pipeline. The original nnU-Net trainer used a non-deterministic data loader by default and didn't have a straightforward way to enforce reproducibility across all components, including PyTorch, NumPy, and the data loading workers. This could lead to slight variations in training results, even with the same initial seed.
Solution
To address this, I've implemented the following changes:
deterministicFlag: Adeterministicboolean flag has been added to thennUNetTrainer's__init__method. When set toTrue, it activates all the changes needed for reproducible training.seed_everythingFunction: A new helper function,seed_everything, is called when thedeterministicflag is active. This function sets the seeds forrandom,numpy, andtorch, and also configurescudnnfor deterministic behavior to eliminate sources of randomness from the GPU.get_dataloadersmethod now checks thedeterministicflag. IfTrue, it usesMultiThreadedAugmenterand passes a unique, generated seed to each worker process. This ensures that the data augmentation pipeline is fully deterministic and produces the same results in the same order for every run. IfFalse, it continues to use the defaultNonDetMultiThreadedAugmenter. Same inget_training_transformsto enable or not the benchmark.How It's Tested
To validate these changes, I've developed a determinism test pipeline that sets up a dummy dataset and runs the entire preprocessing and training pipeline for two epochs with the 2d and 3d configuration. The test works as follows:
nnUNetTraineris instantiated withdeterministic=True, and a short training session is run. The final checkpoint is saved.With these changes, the trainer now passes this test, confirming that the training process is fully reproducible when the
deterministicflag is enabled. This should be a significant help for researchers who need to ensure their results are perfectly reproducible. The changes are self-contained and don't affect the default behavior of the trainer.Notes :
I hope these changes are helpful. Thanks for maintaining this great project, and I look forward to your feedback!
Post Scriptum
Here is a partial output of the determinism test pipeline :