obtain model performance variability by running multiple seeds

[msrepo]

Tasks

• code to aggregate over multiple seeds

[msrepo]

Finalize what models are we going to put in the final table before running these expensive multiple folds. For example it was found that SwinUNETR does considerably well than UNETR(kind of obvious, given it is essentially an improvement over UNETR using shifted window) i.e. we might keep SwinUNETR instead of UNETR as representative of the best pure transformer model.

[msrepo]

See https://github.com/mle-infrastructure/mle-logging for reference.
there aggregate_over_seed() aggregates dicts.

• configs
  |__ subject_list
  |__ monte_carlo_k_fold_seed_12345
  | |__ paths
  | |__yaml
  |__monte_carlo_k_fold_seed_12346

files

• scripts/generate_train_val_test_paths.py
• scripts/benchmark_table.py
• generate_training_script.py
• generate_evaluation_script.py
• generate_config.py

current workflow

python scripts/generate_train_val_test_paths.py configs/full/Verse2019-DRR-full.yaml

TODO

• save csv into paths dir
• train/test split seed, other randomization seed same? keep it in the yaml file
• structure the scripts dir, create subdirs