error when training to 5000 iterations

[Ha0Tang]

2021-09-11 22:59:54,944 - mmedit - INFO - Saving checkpoint at 5000 iterations                                                                                                                                                                                                             
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 4/4, 0.0 task/s, elapsed: 164s, ETA:     0s                                                                                                                                                                                           
                                                                                                                                                                                                                                                                                           
Traceback (most recent call last):                                                                                                                                                                                                                                                         
  File "./tools/train.py", line 145, in <module>                                                                                                                                                                                                                                           
2021-09-11 23:02:40,338 - mmedit - INFO - Exp name: basicvsr_reds4_step20                                                                                                                                                                                                                  
Traceback (most recent call last):                                                                                                                                                                                                                                                         
  File "./tools/train.py", line 145, in <module>                                                                                                                                                                                                                                           
    main()                                                                                                                                                                                                                                                                                 
  File "./tools/train.py", line 141, in main                                                                                                                                                                                                                                               
    main()                                                                                                                                                                                                                                                                                 
  File "./tools/train.py", line 141, in main                                                                                                                                                                                                                                               
    meta=meta)                                                                                                                                                                                                                                                                             
  File "/home/ht1/mmediting/mmedit/apis/train.py", line 71, in train_model                                                                                                                                                                                                               
    meta=meta)                                                                                                                                                                                                                                                                             
  File "/home/ht1/mmediting/mmedit/apis/train.py", line 71, in train_model                                                                                                                                                                                                               
    meta=meta)                                                                                                                                                                                                                                                                             
  File "/home/ht1/mmediting/mmedit/apis/train.py", line 204, in _dist_train                                                                                                                                                                                                              
    meta=meta)                                                                                                                                                                                                                                                                             
  File "/home/ht1/mmediting/mmedit/apis/train.py", line 204, in _dist_train                                                                                                                                                                                                              
    runner.run(data_loaders, cfg.workflow, cfg.total_iters)                                                                                                                                                                                                                                
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/mmcv/runner/iter_based_runner.py", line 133, in run                                                                                                                                                               
    runner.run(data_loaders, cfg.workflow, cfg.total_iters)                                                                                                                                                                                                                                
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/mmcv/runner/iter_based_runner.py", line 133, in run                                                                                                                                                               
    iter_runner(iter_loaders[i], **kwargs)                                                                                                                                                                                                                                                 
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/mmcv/runner/iter_based_runner.py", line 66, in train                                                                                                                                                              
    iter_runner(iter_loaders[i], **kwargs)                                                                                                                                                                                                                                                 
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/mmcv/runner/iter_based_runner.py", line 66, in train                                                                                                                                                              
    self.call_hook('after_train_iter')                                                                                                                                                                                                                                                     
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/mmcv/runner/base_runner.py", line 307, in call_hook                                                                                                                                                               
    self.call_hook('after_train_iter')                                                                                                                                                                                                                                                     
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/mmcv/runner/base_runner.py", line 307, in call_hook                                                                                                                                                               
    getattr(hook, fn_name)(self)                                                                                                                                                                                                                                                           
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/mmcv/runner/hooks/logger/base.py", line 152, in after_train_iter                                                                                                                                                  
    getattr(hook, fn_name)(self)                                                                                                                                                                                                                                                           
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/mmcv/runner/hooks/logger/base.py", line 152, in after_train_iter                                                                                                                                                  
    self.log(runner)                                                                                                                                                                                                                                                                       
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/mmcv/runner/hooks/logger/text.py", line 177, in log                                                                                                                                                               
    self.log(runner)                                                                                                                                                                                                                                                                       
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/mmcv/runner/hooks/logger/text.py", line 177, in log                                                                                                                                                               
    self._log_info(log_dict, runner)                                                                                                                                                                                                                                                       
      File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/mmcv/runner/hooks/logger/text.py", line 96, in _log_info                                                                                                                                                      
self._log_info(log_dict, runner)                                                                                                                                                                                                                                                           
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/mmcv/runner/hooks/logger/text.py", line 96, in _log_info                                                                                                                                                          
    log_str += f'time: {log_dict["time"]:.3f}, ' \                                                                                                                                                                                                                                         
    log_str += f'time: {log_dict["time"]:.3f}, ' \                                                                                                                                                                                                                                         
KeyError: 'data_time'                                                                                                                                                                                                                                                                      
KeyError: 'data_time'                                                                                                                                                                                                                                                                      
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 19377) of binary: /home/ht1/anaconda3/envs/open-mmlab2/bin/python                                                                                                                             
ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed                                                                                                                                                                                             
INFO:torch.distributed.elastic.agent.server.api:[default] Worker group FAILED. 3/3 attempts left; will restart worker group                                                                                                                                                                
INFO:torch.distributed.elastic.agent.server.api:[default] Stopping worker group                                                                                                                                                                                                            
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous'ing worker group                                                                                                                                                                                                      
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result:                                                                                                                                                                                         
  restart_count=1                                                                                                                                                                                                                                                                          
  master_addr=127.0.0.1                                                                                                                                                                                                                                                                    
  master_port=29500                                                                                                                                                                                                                                                                        
  group_rank=0                                                                                                                                                                                                                                                                             
  group_world_size=1                                                                                                                                                                                                                                                                       
  local_ranks=[0, 1]                                                                                                                                                                                                                                                                       
  role_ranks=[0, 1]                                                                                                                                                                                                                                                                        
  global_ranks=[0, 1]                                                                                                                                                                                                                                                                      
  role_world_sizes=[2, 2]                                                                                                                                                                                                                                                                  
  global_world_sizes=[2, 2]        
INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group                                                                                                                                                                                                            
INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_vivn_3xl/none_pjb4mm0f/attempt_1/0/error.json                                                                                                                                              
INFO:torch.distributed.elastic.multiprocessing:Setting worker1 reply file to: /tmp/torchelastic_vivn_3xl/none_pjb4mm0f/attempt_1/1/error.json                                                                                                                                              
Traceback (most recent call last):                                                                                                                                                                                                                                                         
Traceback (most recent call last):                                                                                                                                                                                                                                                         
  File "./tools/train.py", line 145, in <module>
  File "./tools/train.py", line 145, in <module>
    main()
  File "./tools/train.py", line 80, in main
    main()
  File "./tools/train.py", line 80, in main
        init_dist(args.launcher, **cfg.dist_params)init_dist(args.launcher, **cfg.dist_params)

  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/mmcv/runner/dist_utils.py", line 20, in init_dist
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/mmcv/runner/dist_utils.py", line 20, in init_dist
    _init_dist_pytorch(backend, **kwargs)
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/mmcv/runner/dist_utils.py", line 34, in _init_dist_pytorch
    _init_dist_pytorch(backend, **kwargs)
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/mmcv/runner/dist_utils.py", line 34, in _init_dist_pytorch
    dist.init_process_group(backend=backend, **kwargs)    dist.init_process_group(backend=backend, **kwargs)

  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 547, in init_process_group
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 547, in init_process_group
    _store_based_barrier(rank, store, timeout)
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 222, in _store_based_barrier
    _store_based_barrier(rank, store, timeout)
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 222, in _store_based_barrier
    rank, store_key, world_size, worker_count, timeout))    
rank, store_key, world_size, worker_count, timeout))
RuntimeError: RuntimeError: Timed out initializing process group in store based barrier on rank: 0, for key: store_based_barrier_key:1 (world_size=2, worker_count=4, timeout=0:30:00)
Timed out initializing process group in store based barrier on rank: 1, for key: store_based_barrier_key:1 (world_size=2, worker_count=4, timeout=0:30:00)
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 25569) of binary: /home/ht1/anaconda3/envs/open-mmlab2/bin/python
ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed
INFO:torch.distributed.elastic.agent.server.api:[default] Worker group FAILED. 2/3 attempts left; will restart worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Stopping worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous'ing worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result:
  restart_count=2
  master_addr=127.0.0.1
  master_port=29500
  group_rank=0
  group_world_size=1
  local_ranks=[0, 1]
  role_ranks=[0, 1]
  global_ranks=[0, 1]
  role_world_sizes=[2, 2]
  global_world_sizes=[2, 2]

INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group
INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_vivn_3xl/none_pjb4mm0f/attempt_2/0/error.json 
INFO:torch.distributed.elastic.multiprocessing:Setting worker1 reply file to: /tmp/torchelastic_vivn_3xl/none_pjb4mm0f/attempt_2/1/error.json 
Traceback (most recent call last):
  File "./tools/train.py", line 145, in <module>
    main()
  File "./tools/train.py", line 80, in main
    init_dist(args.launcher, **cfg.dist_params)
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/mmcv/runner/dist_utils.py", line 20, in init_dist
Traceback (most recent call last):
  File "./tools/train.py", line 145, in <module>
    main()
  File "./tools/train.py", line 80, in main
    init_dist(args.launcher, **cfg.dist_params)
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/mmcv/runner/dist_utils.py", line 20, in init_dist
    _init_dist_pytorch(backend, **kwargs)
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/mmcv/runner/dist_utils.py", line 34, in _init_dist_pytorch
    _init_dist_pytorch(backend, **kwargs)
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/mmcv/runner/dist_utils.py", line 34, in _init_dist_pytorch
    dist.init_process_group(backend=backend, **kwargs)
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 547, in init_process_group
    dist.init_process_group(backend=backend, **kwargs)
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 547, in init_process_group
    _store_based_barrier(rank, store, timeout)
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 222, in _store_based_barrier
    _store_based_barrier(rank, store, timeout)
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 222, in _store_based_barrier
    rank, store_key, world_size, worker_count, timeout))
    RuntimeError: rank, store_key, world_size, worker_count, timeout)) 
RuntimeError: Timed out initializing process group in store based barrier on rank: 0, for key: store_based_barrier_key:1 (world_size=2, worker_count=6, timeout=0:30:00)
Timed out initializing process group in store based barrier on rank: 1, for key: store_based_barrier_key:1 (world_size=2, worker_count=6, timeout=0:30:00)
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 41557) of binary: /home/ht1/anaconda3/envs/open-mmlab2/bin/python
ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed
INFO:torch.distributed.elastic.agent.server.api:[default] Worker group FAILED. 1/3 attempts left; will restart worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Stopping worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous'ing worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result:
  restart_count=3
  master_addr=127.0.0.1
  master_port=29500
  group_rank=0
  group_world_size=1
  local_ranks=[0, 1]
  role_ranks=[0, 1]
  global_ranks=[0, 1]
  role_world_sizes=[2, 2]
  global_world_sizes=[2, 2]

INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group
INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_vivn_3xl/none_pjb4mm0f/attempt_3/0/error.json 
INFO:torch.distributed.elastic.multiprocessing:Setting worker1 reply file to: /tmp/torchelastic_vivn_3xl/none_pjb4mm0f/attempt_3/1/error.json 
Traceback (most recent call last):
Traceback (most recent call last):
  File "./tools/train.py", line 145, in <module>
  File "./tools/train.py", line 145, in <module>
        main()
main()  File "./tools/train.py", line 80, in main

  File "./tools/train.py", line 80, in main
    init_dist(args.launcher, **cfg.dist_params)
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/mmcv/runner/dist_utils.py", line 20, in init_dist
    init_dist(args.launcher, **cfg.dist_params)
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/mmcv/runner/dist_utils.py", line 20, in init_dist
        _init_dist_pytorch(backend, **kwargs)
_init_dist_pytorch(backend, **kwargs)
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/mmcv/runner/dist_utils.py", line 34, in _init_dist_pytorch
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/mmcv/runner/dist_utils.py", line 34, in _init_dist_pytorch
    dist.init_process_group(backend=backend, **kwargs)    
dist.init_process_group(backend=backend, **kwargs)
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 547, in init_process_group
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 547, in init_process_group
        _store_based_barrier(rank, store, timeout)_store_based_barrier(rank, store, timeout)
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 222, in _store_based_barrier

  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 222, in _store_based_barrier
    rank, store_key, world_size, worker_count, timeout))
RuntimeError: Timed out initializing process group in store based barrier on rank: 0, for key: store_based_barrier_key:1 (world_size=2, worker_count=8, timeout=0:30:00)    rank, store_key, world_size, worker_count, timeout))

RuntimeError: Timed out initializing process group in store based barrier on rank: 1, for key: store_based_barrier_key:1 (world_size=2, worker_count=8, timeout=0:30:00)
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 63726) of binary: /home/ht1/anaconda3/envs/open-mmlab2/bin/python
ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed
INFO:torch.distributed.elastic.agent.server.api:Local worker group finished (FAILED). Waiting 300 seconds for other agents to finish
/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/torch/distributed/elastic/utils/store.py:71: FutureWarning: This is an experimental API and will be changed in future.
  "This is an experimental API and will be changed in future.", FutureWarning
INFO:torch.distributed.elastic.agent.server.api:Done waiting for other agents. Elapsed: 0.0007309913635253906 seconds
{"name": "torchelastic.worker.status.FAILED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": 0, "group_rank": 0, "worker_id": "63726", "role": "default", "hostname": "dgx-r107-3.ling", "state": "FAILED", "total_run_time": 14844, "rdzv_backend": "st
atic", "raw_error": "{\"message\": \"<NONE>\"}", "metadata": "{\"group_world_size\": 1, \"entry_point\": \"python\", \"local_rank\": [0], \"role_rank\": [0], \"role_world_size\": [2]}", "agent_restarts": 3}}
{"name": "torchelastic.worker.status.FAILED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": 1, "group_rank": 0, "worker_id": "63727", "role": "default", "hostname": "dgx-r107-3.ling", "state": "FAILED", "total_run_time": 14844, "rdzv_backend": "st
atic", "raw_error": "{\"message\": \"<NONE>\"}", "metadata": "{\"group_world_size\": 1, \"entry_point\": \"python\", \"local_rank\": [1], \"role_rank\": [1], \"role_world_size\": [2]}", "agent_restarts": 3}}
{"name": "torchelastic.worker.status.SUCCEEDED", "source": "AGENT", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": null, "group_rank": 0, "worker_id": null, "role": "default", "hostname": "dgx-r107-3.ling", "state": "SUCCEEDED", "total_run_time": 14845, "rdzv_backend"
: "static", "raw_error": null, "metadata": "{\"group_world_size\": 1, \"entry_point\": \"python\"}", "agent_restarts": 3}}
/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py:354: UserWarning: 

********************************************************************** 
               CHILD PROCESS FAILED WITH NO ERROR_FILE                 
********************************************************************** 
CHILD PROCESS FAILED WITH NO ERROR_FILE
Child process 63726 (local_rank 0) FAILED (exitcode 1)
Error msg: Process failed with exitcode 1
Without writing an error file to <N/A>.
While this DOES NOT affect the correctness of your application,
no trace information about the error will be available for inspection. 
Consider decorating your top level entrypoint function with
torch.distributed.elastic.multiprocessing.errors.record. Example:

  from torch.distributed.elastic.multiprocessing.errors import record

  @record
  def trainer_main(args):
     # do train
********************************************************************** 
  warnings.warn(_no_error_file_warning_msg(rank, failure))
Traceback (most recent call last):
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/torch/distributed/launch.py", line 173, in <module>
    main()
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/torch/distributed/launch.py", line 169, in main
    run(args)
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/torch/distributed/run.py", line 624, in run
    )(*cmd_args)
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 116, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 348, in wrapper
    return f(*args, **kwargs)
  File "/home/ht1/anaconda3/envs/open-mmlab2/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 247, in launch_agent
    failures=result.failures,
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
***************************************
        ./tools/train.py FAILED        
=======================================
Root Cause:
[0]:
  time: 2021-09-12_00:33:19
  rank: 0 (local_rank: 0)
  exitcode: 1 (pid: 63726)
  error_file: <N/A>
  msg: "Process failed with exitcode 1"
=======================================
Other Failures:
[1]:
  time: 2021-09-12_00:33:19
  rank: 1 (local_rank: 1)
  exitcode: 1 (pid: 63727)
  error_file: <N/A>
  msg: "Process failed with exitcode 1"
***************************************

[ckkelvinchan]

Could you provide us with more details? For example, what command did you run?

[Ha0Tang]

Simply use sh ./tools/dist_train.sh configs/restorers/basicvsr/basicvsr_reds4.py 2

[innerlee]

Related #473

[Ha0Tang]

I tried the method you provided, but it didn't solve my problem

[Ha0Tang]

solved, need to change two places

[innerlee]

need to change two places

Hi, do you mean that the current codebase still have bugs?

[kennymckormick]

solved, need to change two places

Hello, I have met the same problem. Would you please share your fix with me?

[SHHHSA]

I have met this problem ,what is the solution ? Help me please!!!!I have troubles many days.