We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Describe the bug Training failed: using previously trained model's weight
To Reproduce Steps to reproduce the behavior:
iterative-training_17
Additional info of:iterative-training_17 model:
Environment (please complete the following information):
Additional context Error below - got truncate at the start since it's over character limit - see this file for full console log
INFO: Seed set to 12345 00:42:51 : INFO : Dummy-4 : Seed set to 12345 00:42:51 : ERROR : Dummy-4 : Traceback (most recent call last): File "C:\ProgramData\miniconda3\envs\iterative-training\lib\site-packages\hydra\_internal\instantiate\_instantiate2.py", line 92, in _call_target return _target_(*args, **kwargs) File "C:\ProgramData\miniconda3\envs\iterative-training\lib\site-packages\cyto_dl\datamodules\dataframe\dataframe_datamodule.py", line 113, in __init__ self.datasets = make_multiple_dataframe_splits( File "C:\ProgramData\miniconda3\envs\iterative-training\lib\site-packages\cyto_dl\datamodules\dataframe\utils.py", line 253, in make_multiple_dataframe_splits dataframe = read_dataframe(fpath, required_columns=columns) File "C:\ProgramData\miniconda3\envs\iterative-training\lib\site-packages\cyto_dl\dataframe\readers.py", line 206, in read_dataframe raise ValueError( ValueError: Some or all of the required columns were not found on the given dataframe: {'seg1', 'seg2', 'base_image', 'merge_mask', 'exclude_mask'} The above exception was the direct cause of the following exception: Traceback (most recent call last): File "C:\ProgramData\miniconda3\envs\iterative-training\lib\site-packages\cyto_dl\utils\template_utils.py", line 53, in wrap out = task_func(cfg=cfg, data=data) File "C:\ProgramData\miniconda3\envs\iterative-training\lib\site-packages\cyto_dl\train.py", line 67, in train data = utils.create_dataloader(cfg.data, data) File "C:\ProgramData\miniconda3\envs\iterative-training\lib\site-packages\cyto_dl\utils\array.py", line 19, in create_dataloader dataloader = hydra.utils.instantiate(data_cfg) File "C:\ProgramData\miniconda3\envs\iterative-training\lib\site-packages\hydra\_internal\instantiate\_instantiate2.py", line 226, in instantiate return instantiate_node( File "C:\ProgramData\miniconda3\envs\iterative-training\lib\site-packages\hydra\_internal\instantiate\_instantiate2.py", line 347, in instantiate_node return _call_target(_target_, partial, args, kwargs, full_key) File "C:\ProgramData\miniconda3\envs\iterative-training\lib\site-packages\hydra\_internal\instantiate\_instantiate2.py", line 97, in _call_target raise InstantiationException(msg) from e hydra.errors.InstantiationException: Error in call to target 'cyto_dl.datamodules.dataframe.dataframe_datamodule.DataframeDatamodule': ValueError("Some or all of the required columns were not found on the given dataframe:\n{'seg1', 'seg2', 'base_image', 'merge_mask', 'exclude_mask'}") --------------------------------------------------------------------------- ValueError Traceback (most recent call last) File C:\ProgramData\miniconda3\envs\iterative-training\lib\site-packages\hydra\_internal\instantiate\_instantiate2.py:92, in _call_target(_target_=<class 'cyto_dl.datamodules.dataframe.dataframe_datamodule.DataframeDatamodule'>, _partial_=False, args=(), kwargs={'batch_size': 1, 'cache_dir': r'C:\Users\Administrator\Desktop\segmenter-home/iterative-training_17-1/cache', 'columns': ['raw', 'seg1', 'seg2', 'merge_mask', 'exclude_mask', 'base_image'], 'num_workers': 1, 'path': r'C:\Users\Administrator\Desktop\segmenter-home\iterative-training_17\data', 'pin_memory': True, 'split_column': None, 'transforms': {'train': <monai.transforms.compose.Compose obje...ms.compose.Compose object at 0x00000207C88A6C50>}}, full_key='') 91 try: ---> 92 return _target_(*args, **kwargs) args = () kwargs = {'path': 'C:\\Users\\Administrator\\Desktop\\segmenter-home\\iterative-training_17\\data', 'cache_dir': 'C:\\Users\\Administrator\\Desktop\\segmenter-home/iterative-training_17-1/cache', 'num_workers': 1, 'batch_size': 1, 'pin_memory': True, 'split_column': None, 'columns': ['raw', 'seg1', 'seg2', 'merge_mask', 'exclude_mask', 'base_image'], 'transforms': {'train': <monai.transforms.compose.Compose object at 0x00000207C886A530>, 'test': <monai.transforms.compose.Compose object at 0x00000207C86E57B0>, 'predict': <monai.transforms.compose.Compose object at 0x00000207C88A6890>, 'valid': <monai.transforms.compose.Compose object at 0x00000207C88A6C50>, 'val': <monai.transforms.compose.Compose object at 0x00000207C88A6C50>}} _target_ = <class 'cyto_dl.datamodules.dataframe.dataframe_datamodule.DataframeDatamodule'> 93 except Exception as e: File C:\ProgramData\miniconda3\envs\iterative-training\lib\site-packages\cyto_dl\datamodules\dataframe\dataframe_datamodule.py:113, in DataframeDatamodule.__init__(self=<cyto_dl.datamodules.dataframe.dataframe_datamodule.DataframeDatamodule object>, path=WindowsUPath('C:/Users/Administrator/Desktop/segmenter-home/iterative-training_17/data'), transforms={'train': <monai.transforms.compose.Compose obje...ms.compose.Compose object at 0x00000207C88A6C50>}, split_column=None, columns=['raw', 'seg1', 'seg2', 'merge_mask', 'exclude_mask', 'base_image'], split_map=None, just_inference=False, cache_dir=r'C:\Users\Administrator\Desktop\segmenter-home/iterative-training_17-1/cache', subsample=None, refresh_subsample=False, seed=42, smartcache_args=None, **dataloader_kwargs={'batch_size': 1, 'num_workers': 1, 'pin_memory': True}) 112 if path.is_dir(): --> 113 self.datasets = make_multiple_dataframe_splits( self = <cyto_dl.datamodules.dataframe.dataframe_datamodule.DataframeDatamodule object at 0x00000207C0E4D8D0> path = WindowsUPath('C:/Users/Administrator/Desktop/segmenter-home/iterative-training_17/data') transforms = {'train': <monai.transforms.compose.Compose object at 0x00000207C886A530>, 'test': <monai.transforms.compose.Compose object at 0x00000207C86E57B0>, 'predict': <monai.transforms.compose.Compose object at 0x00000207C88A6890>, 'valid': <monai.transforms.compose.Compose object at 0x00000207C88A6C50>, 'val': <monai.transforms.compose.Compose object at 0x00000207C88A6C50>} columns = ['raw', 'seg1', 'seg2', 'merge_mask', 'exclude_mask', 'base_image'] just_inference = False cache_dir = 'C:\\Users\\Administrator\\Desktop\\segmenter-home/iterative-training_17-1/cache' smartcache_args = None 114 path, 115 transforms, 116 columns, 117 just_inference, 118 cache_dir, 119 smartcache_args=smartcache_args, 120 ) 121 elif path.is_file(): File C:\ProgramData\miniconda3\envs\iterative-training\lib\site-packages\cyto_dl\datamodules\dataframe\utils.py:253, in make_multiple_dataframe_splits(split_path=WindowsUPath('C:/Users/Administrator/Desktop/segmenter-home/iterative-training_17/data'), transforms={'train': <monai.transforms.compose.Compose obje...ms.compose.Compose object at 0x00000207C88A6C50>}, columns=['raw', 'seg1', 'seg2', 'merge_mask', 'exclude_mask', 'base_image'], just_inference=False, cache_dir=r'C:\Users\Administrator\Desktop\segmenter-home/iterative-training_17-1/cache', smartcache_args=None) 252 continue --> 253 dataframe = read_dataframe(fpath, required_columns=columns) fpath = WindowsUPath('C:/Users/Administrator/Desktop/segmenter-home/iterative-training_17/data/test_csv.csv') dataframe = base_image exclude_mask ... seg2 split 0 seg1 C:\Users\Administrator\Desktop\segmenter-home\... ... NaN test 1 seg1 NaN ... NaN test [2 rows x 7 columns] columns = ['raw', 'seg1', 'seg2', 'merge_mask', 'exclude_mask', 'base_image'] 254 dataframe["split"] = split File C:\ProgramData\miniconda3\envs\iterative-training\lib\site-packages\cyto_dl\dataframe\readers.py:206, in read_dataframe(dataframe= ...C:\Users\Administrator\Desktop\SegmenterML-tes..., required_columns=['base_image', 'exclude_mask', 'merge_mask', 'raw', 'seg1', 'seg2'], include_columns=['base_image', 'exclude_mask', 'merge_mask', 'raw', 'seg1', 'seg2']) 205 if missing_columns: --> 206 raise ValueError( missing_columns = {'seg1', 'seg2', 'base_image', 'merge_mask', 'exclude_mask'} 207 f"Some or all of the required columns were not " 208 f"found on the given dataframe:\n{missing_columns}" 209 ) 211 return dataframe ValueError: Some or all of the required columns were not found on the given dataframe: {'seg1', 'seg2', 'base_image', 'merge_mask', 'exclude_mask'} The above exception was the direct cause of the following exception: InstantiationException Traceback (most recent call last) File C:\ProgramData\miniconda3\envs\iterative-training\lib\site-packages\allencell_ml_segmenter\core\view.py:24, in LongTaskThread.run(self=<allencell_ml_segmenter.core.view.LongTaskThread object>) 22 def run(self) -> None: 23 print("running") ---> 24 self._do_work() self._do_work = <bound method TrainingView.doWork of <allencell_ml_segmenter.training.view.TrainingView object at 0x00000207BF77D090>> self = <allencell_ml_segmenter.core.view.LongTaskThread object at 0x00000207BF78CE50> File C:\ProgramData\miniconda3\envs\iterative-training\lib\site-packages\allencell_ml_segmenter\training\view.py:294, in TrainingView.doWork(self=<allencell_ml_segmenter.training.view.TrainingView object>) 290 def doWork(self) -> None: 291 """ 292 Starts training process 293 """ --> 294 self._training_model.dispatch_training() self._training_model = <allencell_ml_segmenter.training.training_model.TrainingModel object at 0x00000207A58E0100> self = <allencell_ml_segmenter.training.view.TrainingView object at 0x00000207BF77D090> File C:\ProgramData\miniconda3\envs\iterative-training\lib\site-packages\allencell_ml_segmenter\training\training_model.py:203, in TrainingModel.dispatch_training(self=<allencell_ml_segmenter.training.training_model.TrainingModel object>) 199 def dispatch_training(self) -> None: 200 """ 201 Dispatches even to start training 202 """ --> 203 self.dispatch(Event.PROCESS_TRAINING) Event.PROCESS_TRAINING = <Event.PROCESS_TRAINING: 'training'> self = <allencell_ml_segmenter.training.training_model.TrainingModel object at 0x00000207A58E0100> File C:\ProgramData\miniconda3\envs\iterative-training\lib\site-packages\allencell_ml_segmenter\core\publisher.py:26, in Publisher.dispatch(self=<allencell_ml_segmenter.training.training_model.TrainingModel object>, event=<Event.PROCESS_TRAINING: 'training'>) 20 """ 21 Dispatches an event to all subscribers 22 """ 23 for _, handler in self._events_to_subscriber_handlers[ 24 event.value 25 ].items(): ---> 26 handler(event) handler = <bound method TrainingService._train_model_handler of <allencell_ml_segmenter.services.training_service.TrainingService object at 0x00000207BF765C90>> event = <Event.PROCESS_TRAINING: 'training'> File C:\ProgramData\miniconda3\envs\iterative-training\lib\site-packages\allencell_ml_segmenter\services\training_service.py:109, in TrainingService._train_model_handler(self=<allencell_ml_segmenter.services.training_service.TrainingService object>, _=<Event.PROCESS_TRAINING: 'training'>) 105 cyto_dl_model.print_config() 106 cyto_dl_model.save_config( 107 self._experiments_model.get_train_config_path() 108 ) --> 109 cyto_dl_model.train() cyto_dl_model = <cyto_dl.api.model.CytoDLModel object at 0x00000207C0D287F0> File C:\ProgramData\miniconda3\envs\iterative-training\lib\site-packages\cyto_dl\api\model.py:108, in CytoDLModel.train(self=<cyto_dl.api.model.CytoDLModel object>, run_async=False, data=None) 106 if run_async: 107 return self._train_async() --> 108 return train_model(self.cfg, data) self.cfg = {'experiment_name': 'experiment_name', 'run_name': 'run_name', 'task_name': 'train', 'tags': ['dev'], 'train': True, 'test': False, 'checkpoint': {'ckpt_path': None, 'weights_only': False, 'strict': True}, 'seed': 12345, 'data': {'_target_': 'cyto_dl.datamodules.dataframe.DataframeDatamodule', 'path': 'C:\\Users\\Administrator\\Desktop\\segmenter-home\\iterative-training_17\\data', 'cache_dir': 'C:\\Users\\Administrator\\Desktop\\segmenter-home/iterative-training_17-1/cache', 'num_workers': 1, 'batch_size': 1, 'pin_memory': True, 'split_column': None, 'columns': ['raw', 'seg1', 'seg2', 'merge_mask', 'exclude_mask', 'base_image'], 'transforms': {'train': {'_target_': 'monai.transforms.Compose', 'transforms': [{'_target_': 'cyto_dl.datamodules.dataframe.utils.RemoveNaNKeysd'}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'raw', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'seg1', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'seg2', 'allow_missing_keys': True, 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.ThresholdIntensityd', 'allow_missing_keys': True, 'keys': ['seg1', 'seg2'], 'threshold': 0.1, 'above': False, 'cval': 1}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['merge_mask'], 'shape_reference_key': 'seg1', 'missing_key_mode': 'ignore'}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['exclude_mask'], 'shape_reference_key': 'seg1', 'missing_key_mode': 'create'}, {'_target_': 'monai.transforms.NormalizeIntensityd', 'keys': 'raw', 'channel_wise': True}, {'_target_': 'cyto_dl.image.transforms.merge.Merged', 'mask_key': 'merge_mask', 'image_keys': ['seg1', 'seg2'], 'base_image_key': 'base_image', 'output_name': 'seg'}, {'_target_': 'monai.transforms.SelectItemsd', 'keys': ['raw', 'seg', 'exclude_mask']}, {'_target_': 'monai.transforms.ToTensord', 'keys': ['raw', 'seg', 'exclude_mask'], 'dtype': 'float16'}, {'_target_': 'cyto_dl.image.transforms.RandomMultiScaleCropd', 'keys': ['raw', 'seg', 'exclude_mask'], 'patch_shape': [32, 64, 64], 'patch_per_image': 1, 'scales_dict': {'seg': [1], 'raw': [1], 'exclude_mask': [1]}}]}, 'test': {'_target_': 'monai.transforms.Compose', 'transforms': [{'_target_': 'cyto_dl.datamodules.dataframe.utils.RemoveNaNKeysd'}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'raw', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'seg1', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'seg2', 'allow_missing_keys': True, 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['merge_mask'], 'shape_reference_key': 'seg1', 'missing_key_mode': 'ignore'}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['exclude_mask'], 'shape_reference_key': 'seg1', 'missing_key_mode': 'create'}, {'_target_': 'monai.transforms.NormalizeIntensityd', 'keys': 'raw', 'channel_wise': True}, {'_target_': 'cyto_dl.image.transforms.merge.Merged', 'mask_key': 'merge_mask', 'image_keys': ['seg1', 'seg2'], 'base_image_key': 'base_image', 'output_name': 'seg'}, {'_target_': 'monai.transforms.ToTensord', 'keys': ['raw', 'seg', 'exclude_mask'], 'dtype': 'float16'}]}, 'predict': {'_target_': 'monai.transforms.Compose', 'transforms': [{'_target_': 'cyto_dl.datamodules.dataframe.utils.RemoveNaNKeysd'}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'raw', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.NormalizeIntensityd', 'keys': 'raw', 'channel_wise': True}, {'_target_': 'monai.transforms.ToTensord', 'keys': ['raw']}]}, 'valid': {'_target_': 'monai.transforms.Compose', 'transforms': [{'_target_': 'cyto_dl.datamodules.dataframe.utils.RemoveNaNKeysd'}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'raw', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'seg1', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'seg2', 'allow_missing_keys': True, 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.ThresholdIntensityd', 'allow_missing_keys': True, 'keys': ['seg1', 'seg2'], 'threshold': 0.1, 'above': False, 'cval': 1}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['merge_mask'], 'shape_reference_key': 'seg1', 'missing_key_mode': 'ignore'}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['exclude_mask'], 'shape_reference_key': 'seg1', 'missing_key_mode': 'create'}, {'_target_': 'monai.transforms.NormalizeIntensityd', 'keys': 'raw', 'channel_wise': True}, {'_target_': 'cyto_dl.image.transforms.merge.Merged', 'mask_key': 'merge_mask', 'image_keys': ['seg1', 'seg2'], 'base_image_key': 'base_image', 'output_name': 'seg'}, {'_target_': 'monai.transforms.SelectItemsd', 'keys': ['raw', 'seg', 'exclude_mask']}, {'_target_': 'monai.transforms.ToTensord', 'keys': ['raw', 'seg', 'exclude_mask'], 'dtype': 'float16'}, {'_target_': 'cyto_dl.image.transforms.RandomMultiScaleCropd', 'keys': ['raw', 'seg', 'exclude_mask'], 'patch_shape': [32, 64, 64], 'patch_per_image': 1, 'scales_dict': {'seg': [1], 'raw': [1], 'exclude_mask': [1]}}]}}}, 'model': {'_target_': 'cyto_dl.models.im2im.MultiTaskIm2Im', 'save_images_every_n_epochs': 1, 'save_dir': 'C:\\Users\\Administrator\\Desktop\\segmenter-home/iterative-training_17-1', 'x_key': 'raw', 'backbone': {'_target_': 'monai.networks.nets.DynUNet', 'spatial_dims': 3, 'in_channels': 1, 'out_channels': 1, 'strides': [1, 2, 2], 'kernel_size': [3, 3, 3], 'upsample_kernel_size': [2, 2], 'dropout': 0.0, 'res_block': True, 'filters': [8, 16, 32]}, 'task_heads': {'seg': {'_target_': 'cyto_dl.nn.head.MaskHead', 'mask_key': 'exclude_mask', 'loss': {'_target_': 'monai.losses.MaskedDiceLoss', 'sigmoid': True}, 'postprocess': {'input': {'_target_': 'cyto_dl.models.im2im.utils.postprocessing.ActThreshLabel', 'rescale_dtype': 'numpy.uint8'}, 'prediction': {'_target_': 'cyto_dl.models.im2im.utils.postprocessing.AutoThreshold', 'method': 'threshold_otsu'}}}}, 'optimizer': {'generator': {'_partial_': True, '_target_': 'torch.optim.Adam', 'lr': 0.0001, 'weight_decay': 0.0001}}, 'lr_scheduler': {'generator': {'_partial_': True, '_target_': 'torch.optim.lr_scheduler.ExponentialLR', 'gamma': 0.995}}, 'inference_args': {'sw_batch_size': 1, 'roi_size': [32, 64, 64], 'overlap': 0.2, 'mode': 'gaussian'}}, 'callbacks': {'model_checkpoint': {'_target_': 'lightning.pytorch.callbacks.ModelCheckpoint', 'dirpath': 'C:\\Users\\Administrator\\Desktop\\segmenter-home/iterative-training_17-1/checkpoints', 'filename': 'epoch_{epoch:03d}', 'monitor': 'val/loss', 'verbose': False, 'save_last': True, 'save_top_k': 1, 'mode': 'min', 'auto_insert_metric_name': False, 'save_weights_only': False, 'every_n_train_steps': None, 'train_time_interval': None, 'every_n_epochs': 1, 'save_on_train_epoch_end': None}, 'early_stopping': {'_target_': 'lightning.pytorch.callbacks.EarlyStopping', 'monitor': 'val/loss', 'min_delta': 0.0, 'patience': 100, 'verbose': False, 'mode': 'min', 'strict': True, 'check_finite': True, 'stopping_threshold': None, 'divergence_threshold': None, 'check_on_train_epoch_end': None}, 'model_summary': {'_target_': 'lightning.pytorch.callbacks.RichModelSummary', 'max_depth': -1}, 'rich_progress_bar': {'_target_': 'lightning.pytorch.callbacks.RichProgressBar'}, 'saving': {'_target_': 'cyto_dl.callbacks.ImageSaver', 'save_dir': 'C:\\Users\\Administrator\\Desktop\\segmenter-home/iterative-training_17-1', 'save_every_n_epochs': 1, 'stages': ['train', 'test', 'val', 'predict'], 'save_input': True}}, 'logger': {'csv': {'_target_': 'lightning.pytorch.loggers.csv_logs.CSVLogger', 'save_dir': 'C:\\Users\\Administrator\\Desktop\\segmenter-home/iterative-training_17-1', 'name': 'csv/', 'prefix': ''}}, 'trainer': {'_target_': 'lightning.Trainer', 'default_root_dir': 'C:\\Users\\Administrator\\Desktop\\segmenter-home/iterative-training_17-1', 'min_epochs': 1, 'max_epochs': 4, 'accelerator': 'gpu', 'devices': 1, 'precision': 16, 'check_val_every_n_epoch': 1, 'deterministic': False, 'detect_anomaly': False, 'max_time': None}, 'paths': {'root_dir': 'C:\\ProgramData\\miniconda3\\envs\\iterative-training\\Lib\\site-packages', 'data_dir': 'C:\\ProgramData\\miniconda3\\envs\\iterative-training\\Lib\\site-packages/data/', 'log_dir': 'C:\\ProgramData\\miniconda3\\envs\\iterative-training\\Lib\\site-packages/logs/', 'output_dir': 'C:\\Users\\Administrator\\Desktop\\segmenter-home/iterative-training_17-1', 'work_dir': 'C:\\Users\\Administrator\\Desktop\\segmenter-home/iterative-training_17-1'}, 'extras': {'ignore_warnings': True, 'enforce_tags': False, 'print_config': False, 'precision': {'_target_': 'torch.set_float32_matmul_precision', 'precision': 'medium'}}, 'persist_cache': False, 'source_col': 'raw', 'target_col1': 'seg1', 'target_col2': 'seg2', 'target_col1_channel': 0, 'target_col2_channel': 0, 'merge_mask_col': 'merge_mask', 'exclude_mask_col': 'exclude_mask', 'base_image_col': 'base_image', 'spatial_dims': 3, 'input_channel': 0, 'raw_im_channels': 1} train_model = <function task_wrapper.<locals>.wrap at 0x00000207BF4213F0> data = None self = <cyto_dl.api.model.CytoDLModel object at 0x00000207C0D287F0> File C:\ProgramData\miniconda3\envs\iterative-training\lib\site-packages\cyto_dl\utils\template_utils.py:56, in task_wrapper.<locals>.wrap(cfg={'experiment_name': 'experiment_name', 'run_name...ms': 3, 'input_channel': 0, 'raw_im_channels': 1}, data=None) 54 except Exception as ex: 55 log.exception("") # save exception to `.log` file ---> 56 raise ex 57 finally: 58 path = Path(cfg.paths.output_dir, "exec_time.log") File C:\ProgramData\miniconda3\envs\iterative-training\lib\site-packages\cyto_dl\utils\template_utils.py:53, in task_wrapper.<locals>.wrap(cfg={'experiment_name': 'experiment_name', 'run_name...ms': 3, 'input_channel': 0, 'raw_im_channels': 1}, data=None) 51 try: 52 start_time = time.time() ---> 53 out = task_func(cfg=cfg, data=data) cfg = {'experiment_name': 'experiment_name', 'run_name': 'run_name', 'task_name': 'train', 'tags': ['dev'], 'train': True, 'test': False, 'checkpoint': {'ckpt_path': None, 'weights_only': False, 'strict': True}, 'seed': 12345, 'data': {'_target_': 'cyto_dl.datamodules.dataframe.DataframeDatamodule', 'path': 'C:\\Users\\Administrator\\Desktop\\segmenter-home\\iterative-training_17\\data', 'cache_dir': 'C:\\Users\\Administrator\\Desktop\\segmenter-home/iterative-training_17-1/cache', 'num_workers': 1, 'batch_size': 1, 'pin_memory': True, 'split_column': None, 'columns': ['raw', 'seg1', 'seg2', 'merge_mask', 'exclude_mask', 'base_image'], 'transforms': {'train': {'_target_': 'monai.transforms.Compose', 'transforms': [{'_target_': 'cyto_dl.datamodules.dataframe.utils.RemoveNaNKeysd'}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'raw', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'seg1', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'seg2', 'allow_missing_keys': True, 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.ThresholdIntensityd', 'allow_missing_keys': True, 'keys': ['seg1', 'seg2'], 'threshold': 0.1, 'above': False, 'cval': 1}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['merge_mask'], 'shape_reference_key': 'seg1', 'missing_key_mode': 'ignore'}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['exclude_mask'], 'shape_reference_key': 'seg1', 'missing_key_mode': 'create'}, {'_target_': 'monai.transforms.NormalizeIntensityd', 'keys': 'raw', 'channel_wise': True}, {'_target_': 'cyto_dl.image.transforms.merge.Merged', 'mask_key': 'merge_mask', 'image_keys': ['seg1', 'seg2'], 'base_image_key': 'base_image', 'output_name': 'seg'}, {'_target_': 'monai.transforms.SelectItemsd', 'keys': ['raw', 'seg', 'exclude_mask']}, {'_target_': 'monai.transforms.ToTensord', 'keys': ['raw', 'seg', 'exclude_mask'], 'dtype': 'float16'}, {'_target_': 'cyto_dl.image.transforms.RandomMultiScaleCropd', 'keys': ['raw', 'seg', 'exclude_mask'], 'patch_shape': [32, 64, 64], 'patch_per_image': 1, 'scales_dict': {'seg': [1], 'raw': [1], 'exclude_mask': [1]}}]}, 'test': {'_target_': 'monai.transforms.Compose', 'transforms': [{'_target_': 'cyto_dl.datamodules.dataframe.utils.RemoveNaNKeysd'}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'raw', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'seg1', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'seg2', 'allow_missing_keys': True, 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['merge_mask'], 'shape_reference_key': 'seg1', 'missing_key_mode': 'ignore'}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['exclude_mask'], 'shape_reference_key': 'seg1', 'missing_key_mode': 'create'}, {'_target_': 'monai.transforms.NormalizeIntensityd', 'keys': 'raw', 'channel_wise': True}, {'_target_': 'cyto_dl.image.transforms.merge.Merged', 'mask_key': 'merge_mask', 'image_keys': ['seg1', 'seg2'], 'base_image_key': 'base_image', 'output_name': 'seg'}, {'_target_': 'monai.transforms.ToTensord', 'keys': ['raw', 'seg', 'exclude_mask'], 'dtype': 'float16'}]}, 'predict': {'_target_': 'monai.transforms.Compose', 'transforms': [{'_target_': 'cyto_dl.datamodules.dataframe.utils.RemoveNaNKeysd'}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'raw', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.NormalizeIntensityd', 'keys': 'raw', 'channel_wise': True}, {'_target_': 'monai.transforms.ToTensord', 'keys': ['raw']}]}, 'valid': {'_target_': 'monai.transforms.Compose', 'transforms': [{'_target_': 'cyto_dl.datamodules.dataframe.utils.RemoveNaNKeysd'}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'raw', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'seg1', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'seg2', 'allow_missing_keys': True, 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.ThresholdIntensityd', 'allow_missing_keys': True, 'keys': ['seg1', 'seg2'], 'threshold': 0.1, 'above': False, 'cval': 1}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['merge_mask'], 'shape_reference_key': 'seg1', 'missing_key_mode': 'ignore'}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['exclude_mask'], 'shape_reference_key': 'seg1', 'missing_key_mode': 'create'}, {'_target_': 'monai.transforms.NormalizeIntensityd', 'keys': 'raw', 'channel_wise': True}, {'_target_': 'cyto_dl.image.transforms.merge.Merged', 'mask_key': 'merge_mask', 'image_keys': ['seg1', 'seg2'], 'base_image_key': 'base_image', 'output_name': 'seg'}, {'_target_': 'monai.transforms.SelectItemsd', 'keys': ['raw', 'seg', 'exclude_mask']}, {'_target_': 'monai.transforms.ToTensord', 'keys': ['raw', 'seg', 'exclude_mask'], 'dtype': 'float16'}, {'_target_': 'cyto_dl.image.transforms.RandomMultiScaleCropd', 'keys': ['raw', 'seg', 'exclude_mask'], 'patch_shape': [32, 64, 64], 'patch_per_image': 1, 'scales_dict': {'seg': [1], 'raw': [1], 'exclude_mask': [1]}}]}}}, 'model': {'_target_': 'cyto_dl.models.im2im.MultiTaskIm2Im', 'save_images_every_n_epochs': 1, 'save_dir': 'C:\\Users\\Administrator\\Desktop\\segmenter-home/iterative-training_17-1', 'x_key': 'raw', 'backbone': {'_target_': 'monai.networks.nets.DynUNet', 'spatial_dims': 3, 'in_channels': 1, 'out_channels': 1, 'strides': [1, 2, 2], 'kernel_size': [3, 3, 3], 'upsample_kernel_size': [2, 2], 'dropout': 0.0, 'res_block': True, 'filters': [8, 16, 32]}, 'task_heads': {'seg': {'_target_': 'cyto_dl.nn.head.MaskHead', 'mask_key': 'exclude_mask', 'loss': {'_target_': 'monai.losses.MaskedDiceLoss', 'sigmoid': True}, 'postprocess': {'input': {'_target_': 'cyto_dl.models.im2im.utils.postprocessing.ActThreshLabel', 'rescale_dtype': 'numpy.uint8'}, 'prediction': {'_target_': 'cyto_dl.models.im2im.utils.postprocessing.AutoThreshold', 'method': 'threshold_otsu'}}}}, 'optimizer': {'generator': {'_partial_': True, '_target_': 'torch.optim.Adam', 'lr': 0.0001, 'weight_decay': 0.0001}}, 'lr_scheduler': {'generator': {'_partial_': True, '_target_': 'torch.optim.lr_scheduler.ExponentialLR', 'gamma': 0.995}}, 'inference_args': {'sw_batch_size': 1, 'roi_size': [32, 64, 64], 'overlap': 0.2, 'mode': 'gaussian'}}, 'callbacks': {'model_checkpoint': {'_target_': 'lightning.pytorch.callbacks.ModelCheckpoint', 'dirpath': 'C:\\Users\\Administrator\\Desktop\\segmenter-home/iterative-training_17-1/checkpoints', 'filename': 'epoch_{epoch:03d}', 'monitor': 'val/loss', 'verbose': False, 'save_last': True, 'save_top_k': 1, 'mode': 'min', 'auto_insert_metric_name': False, 'save_weights_only': False, 'every_n_train_steps': None, 'train_time_interval': None, 'every_n_epochs': 1, 'save_on_train_epoch_end': None}, 'early_stopping': {'_target_': 'lightning.pytorch.callbacks.EarlyStopping', 'monitor': 'val/loss', 'min_delta': 0.0, 'patience': 100, 'verbose': False, 'mode': 'min', 'strict': True, 'check_finite': True, 'stopping_threshold': None, 'divergence_threshold': None, 'check_on_train_epoch_end': None}, 'model_summary': {'_target_': 'lightning.pytorch.callbacks.RichModelSummary', 'max_depth': -1}, 'rich_progress_bar': {'_target_': 'lightning.pytorch.callbacks.RichProgressBar'}, 'saving': {'_target_': 'cyto_dl.callbacks.ImageSaver', 'save_dir': 'C:\\Users\\Administrator\\Desktop\\segmenter-home/iterative-training_17-1', 'save_every_n_epochs': 1, 'stages': ['train', 'test', 'val', 'predict'], 'save_input': True}}, 'logger': {'csv': {'_target_': 'lightning.pytorch.loggers.csv_logs.CSVLogger', 'save_dir': 'C:\\Users\\Administrator\\Desktop\\segmenter-home/iterative-training_17-1', 'name': 'csv/', 'prefix': ''}}, 'trainer': {'_target_': 'lightning.Trainer', 'default_root_dir': 'C:\\Users\\Administrator\\Desktop\\segmenter-home/iterative-training_17-1', 'min_epochs': 1, 'max_epochs': 4, 'accelerator': 'gpu', 'devices': 1, 'precision': 16, 'check_val_every_n_epoch': 1, 'deterministic': False, 'detect_anomaly': False, 'max_time': None}, 'paths': {'root_dir': 'C:\\ProgramData\\miniconda3\\envs\\iterative-training\\Lib\\site-packages', 'data_dir': 'C:\\ProgramData\\miniconda3\\envs\\iterative-training\\Lib\\site-packages/data/', 'log_dir': 'C:\\ProgramData\\miniconda3\\envs\\iterative-training\\Lib\\site-packages/logs/', 'output_dir': 'C:\\Users\\Administrator\\Desktop\\segmenter-home/iterative-training_17-1', 'work_dir': 'C:\\Users\\Administrator\\Desktop\\segmenter-home/iterative-training_17-1'}, 'extras': {'ignore_warnings': True, 'enforce_tags': False, 'print_config': False, 'precision': {'_target_': 'torch.set_float32_matmul_precision', 'precision': 'medium'}}, 'persist_cache': False, 'source_col': 'raw', 'target_col1': 'seg1', 'target_col2': 'seg2', 'target_col1_channel': 0, 'target_col2_channel': 0, 'merge_mask_col': 'merge_mask', 'exclude_mask_col': 'exclude_mask', 'base_image_col': 'base_image', 'spatial_dims': 3, 'input_channel': 0, 'raw_im_channels': 1} task_func = <function train at 0x00000207BF421360> data = None 54 except Exception as ex: 55 log.exception("") # save exception to `.log` file File C:\ProgramData\miniconda3\envs\iterative-training\lib\site-packages\cyto_dl\train.py:67, in train(cfg={'experiment_name': 'experiment_name', 'run_name...ms': 3, 'input_channel': 0, 'raw_im_channels': 1}, data=None) 64 use_batch_tuner = True 65 cfg.data.batch_size = 1 ---> 67 data = utils.create_dataloader(cfg.data, data) data = None utils = <module 'cyto_dl.utils' from 'C:\\ProgramData\\miniconda3\\envs\\iterative-training\\lib\\site-packages\\cyto_dl\\utils\\__init__.py'> cfg = {'experiment_name': 'experiment_name', 'run_name': 'run_name', 'task_name': 'train', 'tags': ['dev'], 'train': True, 'test': False, 'checkpoint': {'ckpt_path': None, 'weights_only': False, 'strict': True}, 'seed': 12345, 'data': {'_target_': 'cyto_dl.datamodules.dataframe.DataframeDatamodule', 'path': 'C:\\Users\\Administrator\\Desktop\\segmenter-home\\iterative-training_17\\data', 'cache_dir': 'C:\\Users\\Administrator\\Desktop\\segmenter-home/iterative-training_17-1/cache', 'num_workers': 1, 'batch_size': 1, 'pin_memory': True, 'split_column': None, 'columns': ['raw', 'seg1', 'seg2', 'merge_mask', 'exclude_mask', 'base_image'], 'transforms': {'train': {'_target_': 'monai.transforms.Compose', 'transforms': [{'_target_': 'cyto_dl.datamodules.dataframe.utils.RemoveNaNKeysd'}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'raw', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'seg1', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'seg2', 'allow_missing_keys': True, 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.ThresholdIntensityd', 'allow_missing_keys': True, 'keys': ['seg1', 'seg2'], 'threshold': 0.1, 'above': False, 'cval': 1}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['merge_mask'], 'shape_reference_key': 'seg1', 'missing_key_mode': 'ignore'}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['exclude_mask'], 'shape_reference_key': 'seg1', 'missing_key_mode': 'create'}, {'_target_': 'monai.transforms.NormalizeIntensityd', 'keys': 'raw', 'channel_wise': True}, {'_target_': 'cyto_dl.image.transforms.merge.Merged', 'mask_key': 'merge_mask', 'image_keys': ['seg1', 'seg2'], 'base_image_key': 'base_image', 'output_name': 'seg'}, {'_target_': 'monai.transforms.SelectItemsd', 'keys': ['raw', 'seg', 'exclude_mask']}, {'_target_': 'monai.transforms.ToTensord', 'keys': ['raw', 'seg', 'exclude_mask'], 'dtype': 'float16'}, {'_target_': 'cyto_dl.image.transforms.RandomMultiScaleCropd', 'keys': ['raw', 'seg', 'exclude_mask'], 'patch_shape': [32, 64, 64], 'patch_per_image': 1, 'scales_dict': {'seg': [1], 'raw': [1], 'exclude_mask': [1]}}]}, 'test': {'_target_': 'monai.transforms.Compose', 'transforms': [{'_target_': 'cyto_dl.datamodules.dataframe.utils.RemoveNaNKeysd'}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'raw', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'seg1', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'seg2', 'allow_missing_keys': True, 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['merge_mask'], 'shape_reference_key': 'seg1', 'missing_key_mode': 'ignore'}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['exclude_mask'], 'shape_reference_key': 'seg1', 'missing_key_mode': 'create'}, {'_target_': 'monai.transforms.NormalizeIntensityd', 'keys': 'raw', 'channel_wise': True}, {'_target_': 'cyto_dl.image.transforms.merge.Merged', 'mask_key': 'merge_mask', 'image_keys': ['seg1', 'seg2'], 'base_image_key': 'base_image', 'output_name': 'seg'}, {'_target_': 'monai.transforms.ToTensord', 'keys': ['raw', 'seg', 'exclude_mask'], 'dtype': 'float16'}]}, 'predict': {'_target_': 'monai.transforms.Compose', 'transforms': [{'_target_': 'cyto_dl.datamodules.dataframe.utils.RemoveNaNKeysd'}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'raw', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.NormalizeIntensityd', 'keys': 'raw', 'channel_wise': True}, {'_target_': 'monai.transforms.ToTensord', 'keys': ['raw']}]}, 'valid': {'_target_': 'monai.transforms.Compose', 'transforms': [{'_target_': 'cyto_dl.datamodules.dataframe.utils.RemoveNaNKeysd'}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'raw', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'seg1', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'seg2', 'allow_missing_keys': True, 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.ThresholdIntensityd', 'allow_missing_keys': True, 'keys': ['seg1', 'seg2'], 'threshold': 0.1, 'above': False, 'cval': 1}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['merge_mask'], 'shape_reference_key': 'seg1', 'missing_key_mode': 'ignore'}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['exclude_mask'], 'shape_reference_key': 'seg1', 'missing_key_mode': 'create'}, {'_target_': 'monai.transforms.NormalizeIntensityd', 'keys': 'raw', 'channel_wise': True}, {'_target_': 'cyto_dl.image.transforms.merge.Merged', 'mask_key': 'merge_mask', 'image_keys': ['seg1', 'seg2'], 'base_image_key': 'base_image', 'output_name': 'seg'}, {'_target_': 'monai.transforms.SelectItemsd', 'keys': ['raw', 'seg', 'exclude_mask']}, {'_target_': 'monai.transforms.ToTensord', 'keys': ['raw', 'seg', 'exclude_mask'], 'dtype': 'float16'}, {'_target_': 'cyto_dl.image.transforms.RandomMultiScaleCropd', 'keys': ['raw', 'seg', 'exclude_mask'], 'patch_shape': [32, 64, 64], 'patch_per_image': 1, 'scales_dict': {'seg': [1], 'raw': [1], 'exclude_mask': [1]}}]}}}, 'model': {'_target_': 'cyto_dl.models.im2im.MultiTaskIm2Im', 'save_images_every_n_epochs': 1, 'save_dir': 'C:\\Users\\Administrator\\Desktop\\segmenter-home/iterative-training_17-1', 'x_key': 'raw', 'backbone': {'_target_': 'monai.networks.nets.DynUNet', 'spatial_dims': 3, 'in_channels': 1, 'out_channels': 1, 'strides': [1, 2, 2], 'kernel_size': [3, 3, 3], 'upsample_kernel_size': [2, 2], 'dropout': 0.0, 'res_block': True, 'filters': [8, 16, 32]}, 'task_heads': {'seg': {'_target_': 'cyto_dl.nn.head.MaskHead', 'mask_key': 'exclude_mask', 'loss': {'_target_': 'monai.losses.MaskedDiceLoss', 'sigmoid': True}, 'postprocess': {'input': {'_target_': 'cyto_dl.models.im2im.utils.postprocessing.ActThreshLabel', 'rescale_dtype': 'numpy.uint8'}, 'prediction': {'_target_': 'cyto_dl.models.im2im.utils.postprocessing.AutoThreshold', 'method': 'threshold_otsu'}}}}, 'optimizer': {'generator': {'_partial_': True, '_target_': 'torch.optim.Adam', 'lr': 0.0001, 'weight_decay': 0.0001}}, 'lr_scheduler': {'generator': {'_partial_': True, '_target_': 'torch.optim.lr_scheduler.ExponentialLR', 'gamma': 0.995}}, 'inference_args': {'sw_batch_size': 1, 'roi_size': [32, 64, 64], 'overlap': 0.2, 'mode': 'gaussian'}}, 'callbacks': {'model_checkpoint': {'_target_': 'lightning.pytorch.callbacks.ModelCheckpoint', 'dirpath': 'C:\\Users\\Administrator\\Desktop\\segmenter-home/iterative-training_17-1/checkpoints', 'filename': 'epoch_{epoch:03d}', 'monitor': 'val/loss', 'verbose': False, 'save_last': True, 'save_top_k': 1, 'mode': 'min', 'auto_insert_metric_name': False, 'save_weights_only': False, 'every_n_train_steps': None, 'train_time_interval': None, 'every_n_epochs': 1, 'save_on_train_epoch_end': None}, 'early_stopping': {'_target_': 'lightning.pytorch.callbacks.EarlyStopping', 'monitor': 'val/loss', 'min_delta': 0.0, 'patience': 100, 'verbose': False, 'mode': 'min', 'strict': True, 'check_finite': True, 'stopping_threshold': None, 'divergence_threshold': None, 'check_on_train_epoch_end': None}, 'model_summary': {'_target_': 'lightning.pytorch.callbacks.RichModelSummary', 'max_depth': -1}, 'rich_progress_bar': {'_target_': 'lightning.pytorch.callbacks.RichProgressBar'}, 'saving': {'_target_': 'cyto_dl.callbacks.ImageSaver', 'save_dir': 'C:\\Users\\Administrator\\Desktop\\segmenter-home/iterative-training_17-1', 'save_every_n_epochs': 1, 'stages': ['train', 'test', 'val', 'predict'], 'save_input': True}}, 'logger': {'csv': {'_target_': 'lightning.pytorch.loggers.csv_logs.CSVLogger', 'save_dir': 'C:\\Users\\Administrator\\Desktop\\segmenter-home/iterative-training_17-1', 'name': 'csv/', 'prefix': ''}}, 'trainer': {'_target_': 'lightning.Trainer', 'default_root_dir': 'C:\\Users\\Administrator\\Desktop\\segmenter-home/iterative-training_17-1', 'min_epochs': 1, 'max_epochs': 4, 'accelerator': 'gpu', 'devices': 1, 'precision': 16, 'check_val_every_n_epoch': 1, 'deterministic': False, 'detect_anomaly': False, 'max_time': None}, 'paths': {'root_dir': 'C:\\ProgramData\\miniconda3\\envs\\iterative-training\\Lib\\site-packages', 'data_dir': 'C:\\ProgramData\\miniconda3\\envs\\iterative-training\\Lib\\site-packages/data/', 'log_dir': 'C:\\ProgramData\\miniconda3\\envs\\iterative-training\\Lib\\site-packages/logs/', 'output_dir': 'C:\\Users\\Administrator\\Desktop\\segmenter-home/iterative-training_17-1', 'work_dir': 'C:\\Users\\Administrator\\Desktop\\segmenter-home/iterative-training_17-1'}, 'extras': {'ignore_warnings': True, 'enforce_tags': False, 'print_config': False, 'precision': {'_target_': 'torch.set_float32_matmul_precision', 'precision': 'medium'}}, 'persist_cache': False, 'source_col': 'raw', 'target_col1': 'seg1', 'target_col2': 'seg2', 'target_col1_channel': 0, 'target_col2_channel': 0, 'merge_mask_col': 'merge_mask', 'exclude_mask_col': 'exclude_mask', 'base_image_col': 'base_image', 'spatial_dims': 3, 'input_channel': 0, 'raw_im_channels': 1} 68 if not isinstance(data, LightningDataModule): 69 if not isinstance(data, MutableMapping) or "train_dataloaders" not in data: File C:\ProgramData\miniconda3\envs\iterative-training\lib\site-packages\cyto_dl\utils\array.py:19, in create_dataloader(data_cfg={'_target_': 'cyto_dl.datamodules.dataframe.DataframeDatamodule', 'batch_size': 1, 'cache_dir': r'C:\Users\Administrator\Desktop\segmenter-home/iterative-training_17-1/cache', 'columns': ['raw', 'seg1', 'seg2', 'merge_mask', 'exclude_mask', 'base_image'], 'num_workers': 1, 'path': r'C:\Users\Administrator\Desktop\segmenter-home\iterative-training_17\data', 'pin_memory': True, 'split_column': None, 'transforms': {'predict': {'_target_': 'monai.transforms.Compose', 'transforms': [{'_target_': 'cyto_dl.datamodules.dataframe.utils.RemoveNaNKeysd'}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'raw', 'reader': [{...}]}, {'_target_': 'monai.transforms.NormalizeIntensityd', 'channel_wise': True, 'keys': 'raw'}, {'_target_': 'monai.transforms.ToTensord', 'keys': ['raw']}]}, 'test': {'_target_': 'monai.transforms.Compose', 'transforms': [{'_target_': 'cyto_dl.datamodules.dataframe.utils.RemoveNaNKeysd'}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'raw', 'reader': [{...}]}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'seg1', 'reader': [{...}]}, {'_target_': 'monai.transforms.LoadImaged', 'allow_missing_keys': True, 'keys': 'seg2', 'reader': [{...}]}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['merge_mask'], 'missing_key_mode': 'ignore', 'shape_reference_key': 'seg1'}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['exclude_mask'], 'missing_key_mode': 'create', 'shape_reference_key': 'seg1'}, {'_target_': 'monai.transforms.NormalizeIntensityd', 'channel_wise': True, 'keys': 'raw'}, {'_target_': 'cyto_dl.image.transforms.merge.Merged', 'base_image_key': 'base_image', 'image_keys': ['seg1', 'seg2'], 'mask_key': 'merge_mask', 'output_name': 'seg'}, {'_target_': 'monai.transforms.ToTensord', 'dtype': 'float16', 'keys': ['raw', 'seg', 'exclude_mask']}]}, 'train': {'_target_': 'monai.transforms.Compose', 'transforms': [{'_target_': 'cyto_dl.datamodules.dataframe.utils.RemoveNaNKeysd'}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'raw', 'reader': [{...}]}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'seg1', 'reader': [{...}]}, {'_target_': 'monai.transforms.LoadImaged', 'allow_missing_keys': True, 'keys': 'seg2', 'reader': [{...}]}, {'_target_': 'monai.transforms.ThresholdIntensityd', 'above': False, 'allow_missing_keys': True, 'cval': 1, 'keys': ['seg1', 'seg2'], 'threshold': 0.1}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['merge_mask'], 'missing_key_mode': 'ignore', 'shape_reference_key': 'seg1'}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['exclude_mask'], 'missing_key_mode': 'create', 'shape_reference_key': 'seg1'}, {'_target_': 'monai.transforms.NormalizeIntensityd', 'channel_wise': True, 'keys': 'raw'}, {'_target_': 'cyto_dl.image.transforms.merge.Merged', 'base_image_key': 'base_image', 'image_keys': ['seg1', 'seg2'], 'mask_key': 'merge_mask', 'output_name': 'seg'}, {'_target_': 'monai.transforms.SelectItemsd', 'keys': ['raw', 'seg', 'exclude_mask']}, {'_target_': 'monai.transforms.ToTensord', 'dtype': 'float16', 'keys': ['raw', 'seg', 'exclude_mask']}, {'_target_': 'cyto_dl.image.transforms.RandomMultiScaleCropd', 'keys': ['raw', 'seg', 'exclude_mask'], 'patch_per_image': 1, 'patch_shape': [32, 64, 64], 'scales_dict': {'exclude_mask': [...], 'raw': [...], 'seg': [...]}}]}, 'valid': {'_target_': 'monai.transforms.Compose', 'transforms': [{'_target_': 'cyto_dl.datamodules.dataframe.utils.RemoveNaNKeysd'}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'raw', 'reader': [{...}]}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'seg1', 'reader': [{...}]}, {'_target_': 'monai.transforms.LoadImaged', 'allow_missing_keys': True, 'keys': 'seg2', 'reader': [{...}]}, {'_target_': 'monai.transforms.ThresholdIntensityd', 'above': False, 'allow_missing_keys': True, 'cval': 1, 'keys': ['seg1', 'seg2'], 'threshold': 0.1}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['merge_mask'], 'missing_key_mode': 'ignore', 'shape_reference_key': 'seg1'}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['exclude_mask'], 'missing_key_mode': 'create', 'shape_reference_key': 'seg1'}, {'_target_': 'monai.transforms.NormalizeIntensityd', 'channel_wise': True, 'keys': 'raw'}, {'_target_': 'cyto_dl.image.transforms.merge.Merged', 'base_image_key': 'base_image', 'image_keys': ['seg1', 'seg2'], 'mask_key': 'merge_mask', 'output_name': 'seg'}, {'_target_': 'monai.transforms.SelectItemsd', 'keys': ['raw', 'seg', 'exclude_mask']}, {'_target_': 'monai.transforms.ToTensord', 'dtype': 'float16', 'keys': ['raw', 'seg', 'exclude_mask']}, {'_target_': 'cyto_dl.image.transforms.RandomMultiScaleCropd', 'keys': ['raw', 'seg', 'exclude_mask'], 'patch_per_image': 1, 'patch_shape': [32, 64, 64], 'scales_dict': {'exclude_mask': [...], 'raw': [...], 'seg': [...]}}]}}}, data=None) 16 data_cfg[f"{split}_dataloaders"]["data"] = data[split] 18 # Instantiate the dataloader with the dataset ---> 19 dataloader = hydra.utils.instantiate(data_cfg) data_cfg = {'_target_': 'cyto_dl.datamodules.dataframe.DataframeDatamodule', 'path': 'C:\\Users\\Administrator\\Desktop\\segmenter-home\\iterative-training_17\\data', 'cache_dir': 'C:\\Users\\Administrator\\Desktop\\segmenter-home/iterative-training_17-1/cache', 'num_workers': 1, 'batch_size': 1, 'pin_memory': True, 'split_column': None, 'columns': ['raw', 'seg1', 'seg2', 'merge_mask', 'exclude_mask', 'base_image'], 'transforms': {'train': {'_target_': 'monai.transforms.Compose', 'transforms': [{'_target_': 'cyto_dl.datamodules.dataframe.utils.RemoveNaNKeysd'}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'raw', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'seg1', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'seg2', 'allow_missing_keys': True, 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.ThresholdIntensityd', 'allow_missing_keys': True, 'keys': ['seg1', 'seg2'], 'threshold': 0.1, 'above': False, 'cval': 1}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['merge_mask'], 'shape_reference_key': 'seg1', 'missing_key_mode': 'ignore'}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['exclude_mask'], 'shape_reference_key': 'seg1', 'missing_key_mode': 'create'}, {'_target_': 'monai.transforms.NormalizeIntensityd', 'keys': 'raw', 'channel_wise': True}, {'_target_': 'cyto_dl.image.transforms.merge.Merged', 'mask_key': 'merge_mask', 'image_keys': ['seg1', 'seg2'], 'base_image_key': 'base_image', 'output_name': 'seg'}, {'_target_': 'monai.transforms.SelectItemsd', 'keys': ['raw', 'seg', 'exclude_mask']}, {'_target_': 'monai.transforms.ToTensord', 'keys': ['raw', 'seg', 'exclude_mask'], 'dtype': 'float16'}, {'_target_': 'cyto_dl.image.transforms.RandomMultiScaleCropd', 'keys': ['raw', 'seg', 'exclude_mask'], 'patch_shape': [32, 64, 64], 'patch_per_image': 1, 'scales_dict': {'seg': [1], 'raw': [1], 'exclude_mask': [1]}}]}, 'test': {'_target_': 'monai.transforms.Compose', 'transforms': [{'_target_': 'cyto_dl.datamodules.dataframe.utils.RemoveNaNKeysd'}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'raw', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'seg1', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'seg2', 'allow_missing_keys': True, 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['merge_mask'], 'shape_reference_key': 'seg1', 'missing_key_mode': 'ignore'}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['exclude_mask'], 'shape_reference_key': 'seg1', 'missing_key_mode': 'create'}, {'_target_': 'monai.transforms.NormalizeIntensityd', 'keys': 'raw', 'channel_wise': True}, {'_target_': 'cyto_dl.image.transforms.merge.Merged', 'mask_key': 'merge_mask', 'image_keys': ['seg1', 'seg2'], 'base_image_key': 'base_image', 'output_name': 'seg'}, {'_target_': 'monai.transforms.ToTensord', 'keys': ['raw', 'seg', 'exclude_mask'], 'dtype': 'float16'}]}, 'predict': {'_target_': 'monai.transforms.Compose', 'transforms': [{'_target_': 'cyto_dl.datamodules.dataframe.utils.RemoveNaNKeysd'}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'raw', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.NormalizeIntensityd', 'keys': 'raw', 'channel_wise': True}, {'_target_': 'monai.transforms.ToTensord', 'keys': ['raw']}]}, 'valid': {'_target_': 'monai.transforms.Compose', 'transforms': [{'_target_': 'cyto_dl.datamodules.dataframe.utils.RemoveNaNKeysd'}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'raw', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'seg1', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'seg2', 'allow_missing_keys': True, 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.ThresholdIntensityd', 'allow_missing_keys': True, 'keys': ['seg1', 'seg2'], 'threshold': 0.1, 'above': False, 'cval': 1}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['merge_mask'], 'shape_reference_key': 'seg1', 'missing_key_mode': 'ignore'}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['exclude_mask'], 'shape_reference_key': 'seg1', 'missing_key_mode': 'create'}, {'_target_': 'monai.transforms.NormalizeIntensityd', 'keys': 'raw', 'channel_wise': True}, {'_target_': 'cyto_dl.image.transforms.merge.Merged', 'mask_key': 'merge_mask', 'image_keys': ['seg1', 'seg2'], 'base_image_key': 'base_image', 'output_name': 'seg'}, {'_target_': 'monai.transforms.SelectItemsd', 'keys': ['raw', 'seg', 'exclude_mask']}, {'_target_': 'monai.transforms.ToTensord', 'keys': ['raw', 'seg', 'exclude_mask'], 'dtype': 'float16'}, {'_target_': 'cyto_dl.image.transforms.RandomMultiScaleCropd', 'keys': ['raw', 'seg', 'exclude_mask'], 'patch_shape': [32, 64, 64], 'patch_per_image': 1, 'scales_dict': {'seg': [1], 'raw': [1], 'exclude_mask': [1]}}]}}} hydra.utils = <module 'hydra.utils' from 'C:\\ProgramData\\miniconda3\\envs\\iterative-training\\lib\\site-packages\\hydra\\utils.py'> 21 return dataloader File C:\ProgramData\miniconda3\envs\iterative-training\lib\site-packages\hydra\_internal\instantiate\_instantiate2.py:226, in instantiate(config={'_target_': 'cyto_dl.datamodules.dataframe.Data...'seg': [1], 'raw': [1], 'exclude_mask': [1]}}]}}}, *args=(), **kwargs={}) 223 _convert_ = config.pop(_Keys.CONVERT, ConvertMode.NONE) 224 _partial_ = config.pop(_Keys.PARTIAL, False) --> 226 return instantiate_node( config = {'_target_': 'cyto_dl.datamodules.dataframe.DataframeDatamodule', 'path': 'C:\\Users\\Administrator\\Desktop\\segmenter-home\\iterative-training_17\\data', 'cache_dir': 'C:\\Users\\Administrator\\Desktop\\segmenter-home/iterative-training_17-1/cache', 'num_workers': 1, 'batch_size': 1, 'pin_memory': True, 'split_column': None, 'columns': ['raw', 'seg1', 'seg2', 'merge_mask', 'exclude_mask', 'base_image'], 'transforms': {'train': {'_target_': 'monai.transforms.Compose', 'transforms': [{'_target_': 'cyto_dl.datamodules.dataframe.utils.RemoveNaNKeysd'}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'raw', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'seg1', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'seg2', 'allow_missing_keys': True, 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.ThresholdIntensityd', 'allow_missing_keys': True, 'keys': ['seg1', 'seg2'], 'threshold': 0.1, 'above': False, 'cval': 1}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['merge_mask'], 'shape_reference_key': 'seg1', 'missing_key_mode': 'ignore'}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['exclude_mask'], 'shape_reference_key': 'seg1', 'missing_key_mode': 'create'}, {'_target_': 'monai.transforms.NormalizeIntensityd', 'keys': 'raw', 'channel_wise': True}, {'_target_': 'cyto_dl.image.transforms.merge.Merged', 'mask_key': 'merge_mask', 'image_keys': ['seg1', 'seg2'], 'base_image_key': 'base_image', 'output_name': 'seg'}, {'_target_': 'monai.transforms.SelectItemsd', 'keys': ['raw', 'seg', 'exclude_mask']}, {'_target_': 'monai.transforms.ToTensord', 'keys': ['raw', 'seg', 'exclude_mask'], 'dtype': 'float16'}, {'_target_': 'cyto_dl.image.transforms.RandomMultiScaleCropd', 'keys': ['raw', 'seg', 'exclude_mask'], 'patch_shape': [32, 64, 64], 'patch_per_image': 1, 'scales_dict': {'seg': [1], 'raw': [1], 'exclude_mask': [1]}}]}, 'test': {'_target_': 'monai.transforms.Compose', 'transforms': [{'_target_': 'cyto_dl.datamodules.dataframe.utils.RemoveNaNKeysd'}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'raw', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'seg1', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'seg2', 'allow_missing_keys': True, 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['merge_mask'], 'shape_reference_key': 'seg1', 'missing_key_mode': 'ignore'}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['exclude_mask'], 'shape_reference_key': 'seg1', 'missing_key_mode': 'create'}, {'_target_': 'monai.transforms.NormalizeIntensityd', 'keys': 'raw', 'channel_wise': True}, {'_target_': 'cyto_dl.image.transforms.merge.Merged', 'mask_key': 'merge_mask', 'image_keys': ['seg1', 'seg2'], 'base_image_key': 'base_image', 'output_name': 'seg'}, {'_target_': 'monai.transforms.ToTensord', 'keys': ['raw', 'seg', 'exclude_mask'], 'dtype': 'float16'}]}, 'predict': {'_target_': 'monai.transforms.Compose', 'transforms': [{'_target_': 'cyto_dl.datamodules.dataframe.utils.RemoveNaNKeysd'}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'raw', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.NormalizeIntensityd', 'keys': 'raw', 'channel_wise': True}, {'_target_': 'monai.transforms.ToTensord', 'keys': ['raw']}]}, 'valid': {'_target_': 'monai.transforms.Compose', 'transforms': [{'_target_': 'cyto_dl.datamodules.dataframe.utils.RemoveNaNKeysd'}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'raw', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'seg1', 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.LoadImaged', 'keys': 'seg2', 'allow_missing_keys': True, 'reader': [{'_target_': 'cyto_dl.image.io.MonaiBioReader', 'dimension_order_out': 'CZYX', 'C': 0}]}, {'_target_': 'monai.transforms.ThresholdIntensityd', 'allow_missing_keys': True, 'keys': ['seg1', 'seg2'], 'threshold': 0.1, 'above': False, 'cval': 1}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['merge_mask'], 'shape_reference_key': 'seg1', 'missing_key_mode': 'ignore'}, {'_target_': 'cyto_dl.image.io.PolygonLoaderd', 'keys': ['exclude_mask'], 'shape_reference_key': 'seg1', 'missing_key_mode': 'create'}, {'_target_': 'monai.transforms.NormalizeIntensityd', 'keys': 'raw', 'channel_wise': True}, {'_target_': 'cyto_dl.image.transforms.merge.Merged', 'mask_key': 'merge_mask', 'image_keys': ['seg1', 'seg2'], 'base_image_key': 'base_image', 'output_name': 'seg'}, {'_target_': 'monai.transforms.SelectItemsd', 'keys': ['raw', 'seg', 'exclude_mask']}, {'_target_': 'monai.transforms.ToTensord', 'keys': ['raw', 'seg', 'exclude_mask'], 'dtype': 'float16'}, {'_target_': 'cyto_dl.image.transforms.RandomMultiScaleCropd', 'keys': ['raw', 'seg', 'exclude_mask'], 'patch_shape': [32, 64, 64], 'patch_per_image': 1, 'scales_dict': {'seg': [1], 'raw': [1], 'exclude_mask': [1]}}]}}} _recursive_ = True _convert_ = <ConvertMode.NONE: 'none'> _partial_ = False args = () 227 config, *args, recursive=_recursive_, convert=_convert_, partial=_partial_ 228 ) 229 elif OmegaConf.is_list(config): 230 # Finalize config (convert targets to strings, merge with kwargs) 231 config_copy = copy.deepcopy(config) File C:\ProgramData\miniconda3\envs\iterative-training\lib\site-packages\hydra\_internal\instantiate\_instantiate2.py:347, in instantiate_node(node={'_target_': 'cyto_dl.datamodules.dataframe.Data...'seg': [1], 'raw': [1], 'exclude_mask': [1]}}]}}}, convert=<ConvertMode.NONE: 'none'>, recursive=True, partial=False, *args=()) 342 value = instantiate_node( 343 value, convert=convert, recursive=recursive 344 ) 345 kwargs[key] = _convert_node(value, convert) --> 347 return _call_target(_target_, partial, args, kwargs, full_key) full_key = '' partial = False _target_ = <class 'cyto_dl.datamodules.dataframe.dataframe_datamodule.DataframeDatamodule'> kwargs = {'path': 'C:\\Users\\Administrator\\Desktop\\segmenter-home\\iterative-training_17\\data', 'cache_dir': 'C:\\Users\\Administrator\\Desktop\\segmenter-home/iterative-training_17-1/cache', 'num_workers': 1, 'batch_size': 1, 'pin_memory': True, 'split_column': None, 'columns': ['raw', 'seg1', 'seg2', 'merge_mask', 'exclude_mask', 'base_image'], 'transforms': {'train': <monai.transforms.compose.Compose object at 0x00000207C886A530>, 'test': <monai.transforms.compose.Compose object at 0x00000207C86E57B0>, 'predict': <monai.transforms.compose.Compose object at 0x00000207C88A6890>, 'valid': <monai.transforms.compose.Compose object at 0x00000207C88A6C50>, 'val': <monai.transforms.compose.Compose object at 0x00000207C88A6C50>}} args = () 348 else: 349 # If ALL or PARTIAL non structured or OBJECT non structured, 350 # instantiate in dict and resolve interpolations eagerly. 351 if convert == ConvertMode.ALL or ( 352 convert in (ConvertMode.PARTIAL, ConvertMode.OBJECT) 353 and node._metadata.object_type in (None, dict) 354 ): File C:\ProgramData\miniconda3\envs\iterative-training\lib\site-packages\hydra\_internal\instantiate\_instantiate2.py:97, in _call_target(_target_=<class 'cyto_dl.datamodules.dataframe.dataframe_datamodule.DataframeDatamodule'>, _partial_=False, args=(), kwargs={'batch_size': 1, 'cache_dir': r'C:\Users\Administrator\Desktop\segmenter-home/iterative-training_17-1/cache', 'columns': ['raw', 'seg1', 'seg2', 'merge_mask', 'exclude_mask', 'base_image'], 'num_workers': 1, 'path': r'C:\Users\Administrator\Desktop\segmenter-home\iterative-training_17\data', 'pin_memory': True, 'split_column': None, 'transforms': {'train': <monai.transforms.compose.Compose obje...ms.compose.Compose object at 0x00000207C88A6C50>}}, full_key='') 95 if full_key: 96 msg += f"\nfull_key: {full_key}" ---> 97 raise InstantiationException(msg) from e msg = 'Error in call to target \'cyto_dl.datamodules.dataframe.dataframe_datamodule.DataframeDatamodule\':\nValueError("Some or all of the required columns were not found on the given dataframe:\\n{\'seg1\', \'seg2\', \'base_image\', \'merge_mask\', \'exclude_mask\'}")' InstantiationException: Error in call to target 'cyto_dl.datamodules.dataframe.dataframe_datamodule.DataframeDatamodule': ValueError("Some or all of the required columns were not found on the given dataframe:\n{'seg1', 'seg2', 'base_image', 'merge_mask', 'exclude_mask'}")
The text was updated successfully, but these errors were encountered:
Successfully merging a pull request may close this issue.
Describe the bug
Training failed: using previously trained model's weight
To Reproduce
Steps to reproduce the behavior:
iterative-training_17, 32x64x64 patch size, small model, 4 epochsAdditional info of:
iterative-training_17model:Expected behavior
Screenshots
Describe your data (image format, 2D /3D etc.)
Environment (please complete the following information):
Additional context
Error below - got truncate at the start since it's over character limit - see this file for full console log
The text was updated successfully, but these errors were encountered: