MLflow handler get wrong when running two run without name #6415
Description
Describe the bug
This line in the MLFlowHandler with the or not self.run_name logic will lead to a bug. When running two bundles with run_name=None to the MLFlowHandler, the first one can be run successfully. While the 2nd one comes to this line, since it has no run name, this line will put all previous runs into a list. And the logic below will fetch the latest run and record the 2nd bundle workflow info into it, i.e. the first run.
If simply remove this or not self.run_name logic, it will cause another issue. For example, if there are two MLFlowHandlers in one workflow, like the train workflow, which has two MLFlowHandlers for trainer and validator. Since these two handlers are initialized in different time, they will have two different default run_name, which will make the train information and the validation information of the same bundle workflow into to two different runs.
To Reproduce
Steps to reproduce the behavior:
- Start a MONAI 1.2 docker.
- Download the spleen_ct_segmentation bundle, go into the root folder of the bundle and run the command below twice.
python -m monai.bundle run --config_file configs/train.json --bundle_root ./ --train#trainer#max_epochs 10 --tracking ../tracking.json
The tracking.json file is like:
"handlers_id": {
"trainer": {
"id": "train#trainer",
"handlers": "train#handlers"
},
"validator": {
"id": "evaluate#evaluator",
"handlers": "validate#handlers"
},
"evaluator": {
"id": "evaluator",
"handlers": "handlers"
}
},
"configs": {
"tracking_uri": "http://127.0.0.1:5000",
"experiment_name": "monai_experiment_bundle",
"run_name": null,
"is_not_rank0": "$torch.distributed.is_available() and torch.distributed.is_initialized() and torch.distributed.get_rank() > 0",
"trainer": {
"_target_": "MLFlowHandler",
"_disabled_": "@is_not_rank0",
"tracking_uri": "@tracking_uri",
"experiment_name": "@experiment_name",
"run_name": "@run_name",
"iteration_log": true,
"output_transform": "$monai.handlers.from_engine(['loss'], first=True)",
"close_on_complete": true
},
"validator": {
"_target_": "MLFlowHandler",
"_disabled_": "@is_not_rank0",
"tracking_uri": "@tracking_uri",
"experiment_name": "@experiment_name",
"run_name": "@run_name",
"iteration_log": false
},
"evaluator": {
"_target_": "MLFlowHandler",
"_disabled_": "@is_not_rank0",
"tracking_uri": "@tracking_uri",
"experiment_name": "@experiment_name",
"run_name": "@run_name",
"iteration_log": false,
"close_on_complete": true
}
}
}
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Environment
Ensuring you use the relevant python executable, please paste the output of:
python -c 'import monai; monai.config.print_debug_info()'
Additional context
Add any other context about the problem here.