Data validation with --fail-on-warnings fails but there are no warnings printed

[daanzdev]

Rasa Open Source version

2.8.6

Rasa SDK version

2.8.2

Rasa X version

None

Python version

3.8

What operating system are you using?

OSX

What happened?

Output for rasa data validate:

2021-11-03 16:22:58 INFO     rasa.validator  - Validating intents...
2021-11-03 16:22:58 INFO     rasa.validator  - Validating uniqueness of intents and stories...
2021-11-03 16:22:58 INFO     rasa.validator  - Validating utterances...
2021-11-03 16:22:58 INFO     rasa.validator  - Story structure validation...
Processed story blocks: 100%|█████████████████████████████████████████████████| 38/38 [00:00<00:00, 365.67it/s, # trackers=1]
2021-11-03 16:22:58 INFO     rasa.core.training.story_conflict  - Considering all preceding turns for conflict analysis.
2021-11-03 16:23:12 INFO     rasa.validator  - No story structure conflicts found.

Output for rasa data validate --fail-on-warnings:

2021-11-03 15:35:55 INFO     rasa.validator  - Validating intents...                                                                                                                                                                                        
2021-11-03 15:35:55 INFO     rasa.validator  - Validating uniqueness of intents and stories...                                                                                                                                                              
2021-11-03 15:35:56 INFO     rasa.validator  - Validating utterances...                                                                                                                                                                                     
Project validation completed with errors.

There are no warnings printed for both commands but the one with --fail-on-warnings fails.

When running with -vv option I can see DEBUG level logs (but still no warnings) like:

2021-11-03 17:36:21 DEBUG    rasa.validator  - The intent 'back' is listed in the domain file, but is not found in the NLU training data.

and

2021-11-03 17:36:21 DEBUG    rasa.validator  - The utterance 'utter_default' is not used in any story.

Possibly related source code:

• https://github.com/RasaHQ/rasa/blob/2.8.6/rasa/validator.py#L137
• https://github.com/RasaHQ/rasa/blob/2.8.6/rasa/validator.py#L196

Command / Request

rasa data validate --fail-on-warnings

Relevant log output

2021-11-03 15:35:55 INFO     rasa.validator  - Validating intents...                                                                                                                                                                                        
2021-11-03 15:35:55 INFO     rasa.validator  - Validating uniqueness of intents and stories...                                                                                                                                                              
2021-11-03 15:35:56 INFO     rasa.validator  - Validating utterances...                                                                                                                                                                                     
Project validation completed with errors.

Definition of done

• output for rasa data validate --fail-on-warnings correct

[sara-tagger]

Thanks for the issue, @wochinge will get back to you about it soon!

You may find help in the docs and the forum, too 🤗

[wochinge]

Thanks for bringing this up!