Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"rasa test core --evaluate-model-directory" should have directory as default #4896

Closed
wochinge opened this issue Dec 4, 2019 · 0 comments · Fixed by #5122
Closed

"rasa test core --evaluate-model-directory" should have directory as default #4896

wochinge opened this issue Dec 4, 2019 · 0 comments · Fixed by #5122
Assignees
Labels
area:rasa-oss 🎡 Anything related to the open source Rasa framework type:bug 🐛 Inconsistencies or issues which will cause an issue or problem for users or implementors.

Comments

@wochinge
Copy link
Contributor

wochinge commented Dec 4, 2019

Steps to Reproduce:

  1. Hava a rasa project (e.g. by doing rasa init)
  2. Train a single model (rasa train)
  3. Run rasa test core --evaluate-model-directory

Problem:

By default rasa test core has the latest model in the ./models directory as default value for the --model parameter.

This can lead to the following error:
In case there is a regular model in the models directory, this will be assumed as latest model. However, rasa test core --evaluate-model-directory expects a directory and will fail with an error like this:

Traceback (most recent call last):
  File "/Users/tobias/.pyenv/versions/3.7/bin/rasa", line 11, in <module>
    load_entry_point('rasa', 'console_scripts', 'rasa')()
  File "/Users/tobias/Workspace/stack/rasa/__main__.py", line 76, in main
    cmdline_arguments.func(cmdline_arguments)
  File "/Users/tobias/Workspace/stack/rasa/cli/test.py", line 83, in test_core
    test_core_models_in_directory(model_path, stories, output)
  File "/Users/tobias/Workspace/stack/rasa/test.py", line 26, in test_core_models_in_directory
    number_of_stories = io_utils.read_json_file(story_n_path)
  File "/Users/tobias/Workspace/stack/rasa/utils/io.py", line 144, in read_json_file
    content = read_file(filename)
  File "/Users/tobias/Workspace/stack/rasa/utils/io.py", line 136, in read_file
    with open(filename, encoding=encoding) as f:
NotADirectoryError: [Errno 20] Not a directory: 'models/20191204-102617.tar.gz/num_stories.json'

Proposed Solutions

This can by automatically taking the directory instead of the file in case the user used the --evaluate-model-directory flag.

Other things to check and potentially create a new issue for it:
In case the models directory is empty (except the run folders from the comparison training) the default value for --models is None. So we have to actively check if there is a None value in the passed models list.

@wochinge wochinge added type:bug 🐛 Inconsistencies or issues which will cause an issue or problem for users or implementors. area:rasa-oss 🎡 Anything related to the open source Rasa framework labels Dec 4, 2019
@wochinge wochinge assigned wochinge and chkoss and unassigned wochinge Jan 14, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:rasa-oss 🎡 Anything related to the open source Rasa framework type:bug 🐛 Inconsistencies or issues which will cause an issue or problem for users or implementors.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants