MLflow handler get wrong when running two run without name

[binliunls]

Describe the bug
This line in the MLFlowHandler with the or not self.run_name logic will lead to a bug. When running two bundles with run_name=None to the MLFlowHandler, the first one can be run successfully. While the 2nd one comes to this line, since it has no run name, this line will put all previous runs into a list. And the logic below will fetch the latest run and record the 2nd bundle workflow info into it, i.e. the first run.

If simply remove this or not self.run_name logic, it will cause another issue. For example, if there are two MLFlowHandlers in one workflow, like the train workflow, which has two MLFlowHandlers for trainer and validator. Since these two handlers are initialized in different time, they will have two different default run_name, which will make the train information and the validation information of the same bundle workflow into to two different runs.

To Reproduce
Steps to reproduce the behavior:

1. Start a MONAI 1.2 docker.
2. Download the spleen_ct_segmentation bundle, go into the root folder of the bundle and run the command below twice.
   python -m monai.bundle run --config_file configs/train.json --bundle_root ./ --train#trainer#max_epochs 10 --tracking ../tracking.json

The tracking.json file is like:

"handlers_id": {
					"trainer": {
						"id": "train#trainer",
						"handlers": "train#handlers"
					},
					"validator": {
						"id": "evaluate#evaluator",
						"handlers": "validate#handlers"
					},
					"evaluator": {
						"id": "evaluator",
						"handlers": "handlers"
					}
				},
				"configs": {
					"tracking_uri": "http://127.0.0.1:5000",
					"experiment_name": "monai_experiment_bundle",
					"run_name": null,
					"is_not_rank0": "$torch.distributed.is_available() and torch.distributed.is_initialized() and torch.distributed.get_rank() > 0",
					"trainer": {
						"_target_": "MLFlowHandler",
						"_disabled_": "@is_not_rank0",
						"tracking_uri": "@tracking_uri",
						"experiment_name": "@experiment_name",
						"run_name": "@run_name",
						"iteration_log": true,
						"output_transform": "$monai.handlers.from_engine(['loss'], first=True)",
						"close_on_complete": true
					},
					"validator": {
						"_target_": "MLFlowHandler",
						"_disabled_": "@is_not_rank0",
						"tracking_uri": "@tracking_uri",
						"experiment_name": "@experiment_name",
						"run_name": "@run_name",
						"iteration_log": false
					},
					"evaluator": {
						"_target_": "MLFlowHandler",
						"_disabled_": "@is_not_rank0",
						"tracking_uri": "@tracking_uri",
						"experiment_name": "@experiment_name",
						"run_name": "@run_name",
						"iteration_log": false,
						"close_on_complete": true
					}
				}
			}

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Environment

Ensuring you use the relevant python executable, please paste the output of:

python -c 'import monai; monai.config.print_debug_info()'

Additional context
Add any other context about the problem here.

[Nic-Ma]

Hi @binliunls ,

What do you mean "running two bundles"? You run 2 bundles in parallel with multi-thread? Or just run 2 bundles one by one? If one by one, I think we should set close_on_complete=True? Does it work?

Thanks.

[binliunls]

Hi @binliunls ,

What do you mean "running two bundles"? You run 2 bundles in parallel with multi-thread? Or just run 2 bundles one by one? If one by one, I think we should set close_on_complete=True? Does it work?

Thanks.

Hi @Nic-Ma ,
I mean run two bundles one by one. And the close_on_complete was already set to true in the tracking.json file. I tried to set the close_on_complete parameter in the validator to true, and it still didn't work.