Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add error message for rasa validate when text is null in a response #8618

Merged
merged 11 commits into from
May 11, 2021

Conversation

ancalita
Copy link
Member

@ancalita ancalita commented May 6, 2021

Proposed changes:

Status (please check what you already did):

  • added some tests for the functionality
  • updated the documentation
  • updated the changelog (please check changelog for instructions)
  • reformat files using black (please check Readme for instructions)

@ancalita ancalita requested review from joejuzl and alwx May 6, 2021 10:48
@@ -168,12 +168,17 @@ def validate_yaml_schema(yaml_file_content: Text, schema_path: Text) -> None:

try:
c.validate(raise_exception=True)
except SchemaError:
except SchemaError as e:
if e not in c.errors:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these the same kind of errors? Are they even comparable?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't find much about SchemaError in pykwalify documentation, but in their library they're the same kind (c.errors is a list of SchemaErrors)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I double-checked and you're right @joejuzl , there is a nuance, c.errors is a list of SchemaError.SchemaErrorEntry which is an inner class.

I can check e.msg against each message in c.errors to prevent duplication but it doesn't seem to cause any issues in our code if I add the SchemaError in the list of validation_errors alongside c.errors 🤔

I might have to change the type annotation to Union of SchemaError and SchemaErrorEntry in the YamlValidationException. WDYT?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it was fine how it was to begin with tbh. As we know what they are not duplicate now.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've actually kept the check at msg level because in some cases there might be duplicates based on what I could understand from pykwalify codebase - for example when the SchemaError that gets raised during validation includes the c.errors messages rather than it being a custom SchemaError that Rasa introduces through pykwalify extensions. I'm going to have to edit the check (reverse it) based on this insight.

@ancalita ancalita requested review from alwx and joejuzl May 6, 2021 17:05
@joejuzl
Copy link
Contributor

joejuzl commented May 7, 2021

Ok so I looked a bit deeper into the pykwalify code... and I think the problem is with our extension. Because we raise it kicks out of the validation and doesn't add it to the errors. However if we were to return the error message instead, it would be collected as an error and end up in the final message.

@ancalita
Copy link
Member Author

ancalita commented May 7, 2021

Ok so I looked a bit deeper into the pykwalify code... and I think the problem is with our extension. Because we raise it kicks out of the validation and doesn't add it to the errors. However if we were to return the error message instead, it would be collected as an error and end up in the final message.

@joejuzl Yup, that's what I understood as well, but didn't know you can return a Error 😄 so try changing L20 in pykwalify_extensions.py to a return and remove the check I have for msg ?

@joejuzl
Copy link
Contributor

joejuzl commented May 7, 2021

changing L20 in pykwalify_extensions.py to a return

I actually just returned a string value, but does returning the SchemaError also work?

@ancalita
Copy link
Member Author

ancalita commented May 7, 2021

@joejuzl yup it works with both string and SchemaError - the printed error message will differ just slightly:

  • with string only:
Failed to validate YAML. Please make sure the file is correct and all mandatory parameters are specified. Here are the errors found during validation:
  in Line 7:
      Missing 'text' or 'custom' key in response or null 'text' value in response.. Path: /responses/utter_ask_name
  • with SchemaError:
Failed to validate YAML. Please make sure the file is correct and all mandatory parameters are specified. Here are the errors found during validation:
  in Line 7:
      <SchemaError: error code 2: Missing 'text' or 'custom' key in response or null 'text' value in response.: Path: '/'>. Path: /responses/utter_ask_name

Which one do you think it's more informative for the user? I'd vote for 2nd since it makes it clear it's a schema related error, however it's not nicely formatted and that might confuse users?

Copy link
Contributor

@joejuzl joejuzl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💯

Copy link
Contributor

@alwx alwx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀🚀🚀

rasa/validator.py Show resolved Hide resolved
@ancalita ancalita force-pushed the rasa-validate-null-7908 branch from 644d938 to 185f7e7 Compare May 11, 2021 09:01
@ancalita ancalita changed the base branch from main to 2.6.x May 11, 2021 09:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Rasa validator shows empty error message when text: null for a response
3 participants