stage_6.log not found

[doriangrosch]

When training a model in "word" mode, there inevitably comes a error warning that the file stage_6.log was not found:

Traceback (most recent call last):
  File "/opt/pyenv/versions/3.8.2/lib/python3.8/threading.py", line
932, in _bootstrap_inner
    self.run()
  File "/opt/pyenv/versions/3.8.2/lib/python3.8/threading.py", line
870, in run
    self._target(*self._args, **self._kwargs)
  File "/elpis/elpis/engines/kaldi/objects/model.py", line 288, in
background_train_task
    self.results = KaldiModel.get_train_results(self)
  File "/elpis/elpis/engines/kaldi/objects/model.py", line 307, in
get_train_results
    with log_file.open() as fin:
  File "/opt/pyenv/versions/3.8.2/lib/python3.8/pathlib.py", line 1213,
in open
    return io.open(self, mode, buffering, encoding, errors, newline,
  File "/opt/pyenv/versions/3.8.2/lib/python3.8/pathlib.py", line 1069,
in _opener
    return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory:
'/state/models/9b80e210c86a6897e5dbb1e0453f82f4/train-logs/stage_6.log' 

Is there any approach to figure out what could cause this error?

[benfoley]

hi, Can you please share the log output from earlier stages?

[doriangrosch]

Hello, sorry for the very long delay. I hope these logs help with identifying the problem. Is there anything else I could do?

stage_0.log
stage_1.log
stage_2.log
stage_3.log

[benfoley]

Looking at the stage_3.log file, it appears that there's a character present in the data that didn't get added to the config files. Are you able to share the data and I can try and replicate it here to provide more info about how to clean the data to suit?

--> ERROR: phone "(')" is not in {, non}silence.txt (line 117)