Does Rasa Data Validate not check Rules.yml/Stories.yml?

[koaning]

Rasa version:

Rasa Version     : 2.2.6
Rasa SDK Version : 2.2.0
Rasa X Version   : 0.35.0
Python Version   : 3.7.7
Operating System : Darwin-19.3.0-x86_64-i386-64bit
Python Path      : /Users/vincent/Development/name-form-demo/venv/bin/python3.7

Issue:

It seems that not everything is caught with rasa data validate. In particular, are we sure it is checking rules.yml files? I took a rasa init project and added this rule.

- rule: Ask Name Again
  condition:
    - active_loop: name_form
  steps:
    - slot_was_set:
      - requested_slot: name
    - action: action_check_name_spelling
    - intent: deny
    - action: action_clear_name_slot
    - active_loop: name_form

Running rasa data validate does not give an error if "action_clear_name_slot" and "action_check_name_spelling" are missing from domain.yml. But rasa train does!

> rasa train
/Users/vincent/Development/name-form-demo/venv/lib/python3.7/site-packages/rasa/shared/core/slots.py:268: FutureWarning: UnfeaturizedSlot is deprecated and will be removed in Rasa Open Source 3.0. Please change the type and configure the 'influence_conversation' flag for slot 'requested_slot' instead.
  category=FutureWarning,
2021-01-25 15:27:15 INFO     rasa.shared.utils.validation  - The 'version' key is missing in the training data file /Users/vincent/Development/name-form-demo/data/namelist.yml. Rasa Open Source will read the file as a version '2.0' file. See https://rasa.com/docs/rasa/training-data-format.
2021-01-25 15:27:16 INFO     rasa.model  - Data (domain) for Core model section changed.
2021-01-25 15:27:16 INFO     rasa.model  - Data (messages) for NLU model section changed.
2021-01-25 15:27:16 INFO     rasa.model  - Data (domain) for Core model section changed.
2021-01-25 15:27:16 INFO     rasa.model  - Data (messages) for NLU model section changed.
Training NLU model...
2021-01-25 15:27:17 INFO     rasa.shared.nlu.training_data.training_data  - Training data stats:
2021-01-25 15:27:17 INFO     rasa.shared.nlu.training_data.training_data  - Number of intent examples: 76 (8 distinct intents)

2021-01-25 15:27:17 INFO     rasa.shared.nlu.training_data.training_data  -   Found intents: 'request_name', 'affirm', 'deny', 'mood_great', 'bot_challenge', 'mood_unhappy', 'greet', 'goodbye'
2021-01-25 15:27:17 INFO     rasa.shared.nlu.training_data.training_data  - Number of response examples: 0 (0 distinct responses)
2021-01-25 15:27:17 INFO     rasa.shared.nlu.training_data.training_data  - Number of entity examples: 0 (0 distinct entities)
2021-01-25 15:27:17 INFO     rasa.nlu.model  - Starting to train component WhitespaceTokenizer
2021-01-25 15:27:17 INFO     rasa.nlu.model  - Finished training component.
2021-01-25 15:27:17 INFO     rasa.nlu.model  - Starting to train component RegexFeaturizer
2021-01-25 15:27:18 INFO     rasa.nlu.model  - Finished training component.
2021-01-25 15:27:18 INFO     rasa.nlu.model  - Starting to train component LexicalSyntacticFeaturizer
2021-01-25 15:27:18 INFO     rasa.nlu.model  - Finished training component.
2021-01-25 15:27:18 INFO     rasa.nlu.model  - Starting to train component CountVectorsFeaturizer
2021-01-25 15:27:18 INFO     rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer  - 89 vocabulary slots consumed out of 1089 slots configured for text attribute.
2021-01-25 15:27:18 INFO     rasa.nlu.model  - Finished training component.
2021-01-25 15:27:18 INFO     rasa.nlu.model  - Starting to train component CountVectorsFeaturizer
2021-01-25 15:27:18 INFO     rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer  - 773 vocabulary slots consumed out of 1773 slots configured for text attribute.
2021-01-25 15:27:18 INFO     rasa.nlu.model  - Finished training component.
2021-01-25 15:27:18 INFO     rasa.nlu.model  - Starting to train component DIETClassifier
Epochs:   0%|                                                                  | 0/100 [00:00<?, ?it/s]/Users/vincent/Development/name-form-demo/venv/lib/python3.7/site-packages/rasa/utils/tensorflow/model_data.py:861: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  np.concatenate(np.array(f)),
Epochs: 100%|█████████████████████████████| 100/100 [00:14<00:00,  6.70it/s, t_loss=1.549, i_acc=1.000]
2021-01-25 15:27:42 INFO     rasa.utils.tensorflow.models  - Finished training.
2021-01-25 15:27:42 INFO     rasa.nlu.model  - Finished training component.
2021-01-25 15:27:42 INFO     rasa.nlu.model  - Starting to train component EntitySynonymMapper
2021-01-25 15:27:42 INFO     rasa.nlu.model  - Finished training component.
2021-01-25 15:27:42 INFO     rasa.nlu.model  - Starting to train component ResponseSelector
2021-01-25 15:27:42 INFO     rasa.nlu.selectors.response_selector  - Retrieval intent parameter was left to its default value. This response selector will be trained on training examples combining all retrieval intents.
2021-01-25 15:27:42 INFO     rasa.nlu.model  - Finished training component.
2021-01-25 15:27:43 INFO     rasa.nlu.model  - Successfully saved model into '/var/folders/0v/pj9vtxhd6ml7mb8n94wvcmkr0000gp/T/tmpwkq92n8d/nlu'
NLU model training completed.
Training Core model...
Processed story blocks: 100%|████████████████████████████| 3/3 [00:00<00:00, 1631.18it/s, # trackers=1]
Processed story blocks: 100%|█████████████████████████████| 3/3 [00:00<00:00, 584.44it/s, # trackers=3]
Processed story blocks: 100%|████████████████████████████| 3/3 [00:00<00:00, 156.46it/s, # trackers=12]
Processed story blocks: 100%|█████████████████████████████| 3/3 [00:00<00:00, 34.63it/s, # trackers=39]
Processed rules: 100%|███████████████████████████████████| 7/7 [00:00<00:00, 2455.07it/s, # trackers=1]
Processed trackers: 100%|█████████████████████████████████| 3/3 [00:00<00:00, 802.58it/s, # actions=12]
Processed actions: 12it [00:00, 941.83it/s, # examples=12]
Processed trackers: 100%|█████████████████████████████| 120/120 [00:00<00:00, 228.45it/s, # actions=30]
/Users/vincent/Development/name-form-demo/venv/lib/python3.7/site-packages/rasa/utils/tensorflow/model_data_utils.py:338: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  np.array(values), number_of_dimensions=4
Epochs: 100%|█████████████████| 100/100 [00:06<00:00, 14.34it/s, t_loss=13.794, loss=13.622, acc=0.967]
2021-01-25 15:28:01 INFO     rasa.utils.tensorflow.models  - Finished training.
Processed trackers: 100%|████████████████████████████████| 7/7 [00:00<00:00, 1109.19it/s, # actions=16]
Processed actions: 16it [00:00, 11947.46it/s, # examples=13]
Processed trackers: 100%|█████████████████████████████████| 3/3 [00:00<00:00, 197.35it/s, # actions=12]
Processed trackers:  14%|██████▋                                        | 1/7 [00:00<00:00, 491.65it/s]
ActionNotFoundException: Cannot access action 'action_clear_name_slot', as that name is not a registered action for this domain. Available actions are: 
         - action_listen
         - action_restart
         - action_session_start
         - action_default_fallback
         - action_deactivate_loop
         - action_revert_fallback_events
         - action_default_ask_affirmation
         - action_default_ask_rephrase
         - action_two_stage_fallback
         - action_back
         - ...
         - utter_greet
         - utter_slots_values
         - utter_submit
         - utter_ask_name
         - utter_cheer_up
         - utter_did_that_help
         - utter_goodbye
         - utter_happy
         - utter_iamabot
         - name_form

The annoying thing here is that the time-consuming training procedure needs to run before the error appears on my retina. It'd be nice if rasa data validate can catch this because it might prevent costly training procedures.

[koaning]

I also just confirmed that there is a similar situation when working with custom actions. You can have custom actions in your NLU file with them appearing in the domain.yml file. Running rasa data validate won't trigger an error but running rasa train will prompt one at the end of the training procedure.

[koaning]

It also seems that rasa data validate is completely fine with the following contents in domain.yml.

forms:
  name_form:
    first_name:
      - type: from_text
    last_name:
      - type: from_text

slots:
  first_name:
    type: text
  last_nam:
    type: text

Note that last_nam at the end there is misspelled. So a form can refer to a slot that does not exist.

[kevinthenet]

Just ran into this issue as well -- I've also had to write my own custom validator since it doesn't support the MultiProjectImporter.

Seems strange to me that the rule validation runs only before a core model is trained, and wasn't included in the data validation script.

[koaning]

It also seems like rasa data validate does not check if the actions in the stories appear in the domain.yml file.

> rasa data validate
The configuration for pipeline and policies was chosen automatically. It was written into the config file at 'config.yml'.
2021-04-07 15:54:59 INFO     rasa.validator  - Validating intents...
2021-04-07 15:54:59 INFO     rasa.validator  - Validating uniqueness of intents and stories...
2021-04-07 15:54:59 INFO     rasa.validator  - Validating utterances...
2021-04-07 15:54:59 INFO     rasa.validator  - Story structure validation...
Processed story blocks: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1190.89it/s, # trackers=1]
2021-04-07 15:54:59 INFO     rasa.core.training.story_conflict  - Considering all preceding turns for conflict analysis.
/Users/vincent/Development/how-to-rasa/venv/lib/python3.6/site-packages/rasa/utils/train_utils.py:535: UserWarning: model_confidence is set to `softmax`. It is recommended to try using `model_confidence=linear_norm` to make it easier to tune fallback thresholds.
  category=UserWarning,
2021-04-07 15:54:59 INFO     rasa.validator  - No story structure conflicts found.


> rasa train
The configuration for policies and pipeline was chosen automatically. It was written into the config file at 'config.yml'.
Training NLU model...
/Users/vincent/Development/how-to-rasa/venv/lib/python3.6/site-packages/rasa/utils/train_utils.py:535: UserWarning: model_confidence is set to `softmax`. It is recommended to try using `model_confidence=linear_norm` to make it easier to tune fallback thresholds.
  category=UserWarning,
2021-04-07 15:55:13 INFO     rasa.shared.nlu.training_data.training_data  - Training data stats:
2021-04-07 15:55:13 INFO     rasa.shared.nlu.training_data.training_data  - Number of intent examples: 81 (12 distinct intents)

2021-04-07 15:55:13 INFO     rasa.shared.nlu.training_data.training_data  -   Found intents: 'deny', 'affirm', 'mood_unhappy', 'mood_great', 'bot_challenge', 'request_fancy_form', 'buy_pizza', 'inform', 'request_pizza_form', 'goodbye', 'buy_fancy_pizza', 'greet'
2021-04-07 15:55:13 INFO     rasa.shared.nlu.training_data.training_data  - Number of response examples: 0 (0 distinct responses)
2021-04-07 15:55:13 INFO     rasa.shared.nlu.training_data.training_data  - Number of entity examples: 5 (2 distinct entities)
2021-04-07 15:55:13 INFO     rasa.shared.nlu.training_data.training_data  -   Found entity types: 'pizza_size', 'pizza_type'
/Users/vincent/Development/how-to-rasa/venv/lib/python3.6/site-packages/rasa/shared/utils/io.py:96: UserWarning: Intent 'request_fancy_form' has only 1 training examples! Minimum is 2, training may fail.
/Users/vincent/Development/how-to-rasa/venv/lib/python3.6/site-packages/rasa/shared/utils/io.py:96: UserWarning: Intent 'request_pizza_form' has only 1 training examples! Minimum is 2, training may fail.
/Users/vincent/Development/how-to-rasa/venv/lib/python3.6/site-packages/rasa/shared/utils/io.py:96: UserWarning: Entity entity 'pizza_type' has only 1 training examples! The minimum is 2, because of this the training may fail.
2021-04-07 15:55:13 INFO     rasa.nlu.model  - Starting to train component WhitespaceTokenizer
2021-04-07 15:55:13 INFO     rasa.nlu.model  - Finished training component.
2021-04-07 15:55:13 INFO     rasa.nlu.model  - Starting to train component RegexFeaturizer
2021-04-07 15:55:13 INFO     rasa.nlu.model  - Finished training component.
2021-04-07 15:55:13 INFO     rasa.nlu.model  - Starting to train component LexicalSyntacticFeaturizer
2021-04-07 15:55:13 INFO     rasa.nlu.model  - Finished training component.
2021-04-07 15:55:13 INFO     rasa.nlu.model  - Starting to train component CountVectorsFeaturizer
2021-04-07 15:55:13 INFO     rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer  - 99 vocabulary slots consumed out of 1099 slots configured for text attribute.
2021-04-07 15:55:13 INFO     rasa.nlu.model  - Finished training component.
2021-04-07 15:55:13 INFO     rasa.nlu.model  - Starting to train component CountVectorsFeaturizer
2021-04-07 15:55:13 INFO     rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer  - 862 vocabulary slots consumed out of 1862 slots configured for text attribute.
2021-04-07 15:55:13 INFO     rasa.nlu.model  - Finished training component.
2021-04-07 15:55:13 INFO     rasa.nlu.model  - Starting to train component DIETClassifier
Epochs: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:27<00:00,  3.66it/s, t_loss=1.76, i_acc=1, e_f1=1]
2021-04-07 15:55:41 INFO     rasa.nlu.model  - Finished training component.
2021-04-07 15:55:41 INFO     rasa.nlu.model  - Starting to train component EntitySynonymMapper
2021-04-07 15:55:41 INFO     rasa.nlu.model  - Finished training component.
2021-04-07 15:55:41 INFO     rasa.nlu.model  - Starting to train component ResponseSelector
2021-04-07 15:55:41 INFO     rasa.nlu.selectors.response_selector  - Retrieval intent parameter was left to its default value. This response selector will be trained on training examples combining all retrieval intents.
2021-04-07 15:55:41 INFO     rasa.nlu.model  - Finished training component.
2021-04-07 15:55:41 INFO     rasa.nlu.model  - Starting to train component FallbackClassifier
2021-04-07 15:55:41 INFO     rasa.nlu.model  - Finished training component.
2021-04-07 15:55:41 INFO     rasa.nlu.model  - Successfully saved model into '/var/folders/0v/pj9vtxhd6ml7mb8n94wvcmkr0000gp/T/tmpg1uzv3nl/nlu'
NLU model training completed.
Training Core model...
Processed story blocks: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1224.50it/s, # trackers=1]
Processed story blocks: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 490.35it/s, # trackers=3]
Processed story blocks: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 111.92it/s, # trackers=12]
Processed story blocks: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 28.13it/s, # trackers=39]
Processed rules: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 1742.66it/s, # trackers=1]
Processed trackers: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 622.27it/s, # actions=12]
Processed actions: 12it [00:00, 3372.76it/s, # examples=12]
Processed trackers: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 120/120 [00:00<00:00, 172.97it/s, # actions=30]
Epochs: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:10<00:00,  9.61it/s, t_loss=2.08, loss=1.9, acc=1]
Processed trackers: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 1180.66it/s, # actions=17]
Processed actions: 17it [00:00, 12599.96it/s, # examples=14]
Processed trackers: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 587.88it/s, # actions=12]
Processed trackers:  17%|████████████████████████▊                                                                                                                            | 1/6 [00:00<00:00, 564.66it/s]
ActionNotFoundException: Cannot access action 'pizza_form', as that name is not a registered action for this domain. Available actions are: 
         - action_listen
         - action_restart
         - action_session_start
         - action_default_fallback
         - action_deactivate_loop
         - action_revert_fallback_events
         - action_default_ask_affirmation
         - action_default_ask_rephrase
         - action_two_stage_fallback
         - action_back
         - ...
         - utter_cheer_up
         - utter_did_that_help
         - utter_goodbye
         - utter_greet
         - utter_happy
         - utter_iamabot
         - utter_pizza_slots
         - utter_submit
         - simple_pizza_form
         - fancy_pizza_form

[kevinthenet]

So I guess what's "todo" in this issue is:

• validation of action usage in rules.yml and stories.yml against domain actions during rasa data validate
• support for MultiProjectImporter by using the TrainingDataImporter interface
• validate slot usage in forms registered in domain, should fail if trying to collect a slot that doesn't exist in domain.yml

@koaning does that encompass what's going on here?

[koaning]

There may be more, I'm mostly just making sure that I collected errors here.

[ancalita]

I also just confirmed that there is a similar situation when working with custom actions. You can have custom actions in your NLU file with them appearing in the domain.yml file. Running rasa data validate won't trigger an error but running rasa train will prompt one at the end of the training procedure.

Hi @koaning could you please share the NLU file, domain.yml and the error that gets shown running rasa train - I'm not sure I understand the issue you described in this particular comment.

[koaning]

I'm using the same data that you get when you run rasa init. I make a small change to rules.yml, here's what it looks like.

version: "2.0"

rules:

- rule: Say goodbye anytime the user says goodbye
  steps:
  - intent: goodbye
  - action: utter_goodbye

- rule: Say 'I am a bot' anytime the user challenges
  steps:
  - intent: bot_challenge
  - action: utter_iamabot

- rule: Ask Name Again
  condition:
    - active_loop: name_form
  steps:
    - slot_was_set:
      - requested_slot: name
    - action: action_check_name_spelling
    - intent: deny
    - action: action_clear_name_slot
    - active_loop: name_form

The nlu.yml is kept exactly the same.

nlu.yml.

version: "2.0"

nlu:
- intent: greet
  examples: |
    - hey
    - hello
    - hi
    - hello there
    - good morning
    - good evening
    - moin
    - hey there
    - let's go
    - hey dude
    - goodmorning
    - goodevening
    - good afternoon

- intent: goodbye
  examples: |
    - good afternoon
    - cu
    - good by
    - cee you later
    - good night
    - bye
    - goodbye
    - have a nice day
    - see you around
    - bye bye
    - see you later

- intent: affirm
  examples: |
    - yes
    - y
    - indeed
    - of course
    - that sounds good
    - correct

- intent: deny
  examples: |
    - no
    - n
    - never
    - I don't think so
    - don't like that
    - no way
    - not really

- intent: mood_great
  examples: |
    - perfect
    - great
    - amazing
    - feeling like a king
    - wonderful
    - I am feeling very good
    - I am great
    - I am amazing
    - I am going to save the world
    - super stoked
    - extremely good
    - so so perfect
    - so good
    - so perfect

- intent: mood_unhappy
  examples: |
    - my day was horrible
    - I am sad
    - I don't feel very well
    - I am disappointed
    - super sad
    - I'm so sad
    - sad
    - very sad
    - unhappy
    - not good
    - not very good
    - extremly sad
    - so saad
    - so sad

- intent: bot_challenge
  examples: |
    - are you a bot?
    - are you a human?
    - am I talking to a bot?
    - am I talking to a human?

If I run rasa data validate I get no complaints. However, there are rules referring to an action/form that does not exist. I'd expect the validation to trigger an error here. The unfortunate result is that now I need to wait until after rasa train is done training (which can involve a lot of waiting) that something went wrong.

[alwx]

Speaking about rasa data validate:

1. rasa train doesn't actually do anything, it just fails in the case of issues during training so the this issue might be a bit more complicated than just "add missing checks from rasa train to rasa data validate"
2. Another problem is that rasa train iterates through the policies and calls the train method of a specific policy and only after doing predictions and going though a couple of calls within the policy throws exceptions like ActionNotFoundException: Cannot access action 'action_clear_name_slot', as that name is not a registered action for this domain.
rasa data validate doesn't do this — it doesn't use policies, doesn't iterate through them, doesn't do any action prediction => it's definitely not easy to use the same mechanism to check if actions are missing from the domain. So ideally another mechanism should be used.

However, we plan to change the training process in Rasa 3.0 and separate training and processing. It might be that going through actions and checking if actions are part of the domain would become part of the processing step, and this step might become reusable for data validation. If it's the case (requires some more investigation though) then it's better to skip this issue for now and wait for another implementation of training+processing steps.

[koaning]

It might make more sense to wait for any changes that will happen in 3.0.

That said, if we're designing for 3.0 already ... it might be good to think what we might be able to do for the user in terms of "user experience" here. I've always seen rasa data validate as a linter of sorts. It'd be grand if Rasa offers a command that can help the user with any typos in the configs.