Migration from old response selector format

[dakshvar22]

Description of Problem:
Since, we changed the training data format for response selector, it would be good to help users smoothly transition to the new format through a cli command.

Overview of the Solution:
It can be part of the rasa data convert command that now exists.
Migration changes that need to be covered -

• Change all response templates that are defined for retrieval intents to have the utter_ prefix. For e.g.

## ask_name
* chitchat/ask_name
  - My name is retrieval bot

should get converted to

responses:
   utter_chitchat/ask_name:
     - text: My name is retrieval bot

• Change all occurrences of actions with respond_ prefix to use utter_ prefix instead. These would be in domain and stories.

Definition of Done:

• Tests are added
• Feature described the docs
• Feature mentioned in the changlog

[m-vdb]

Apparently we already have a rasa data convert nlg command that does the conversion from Markdown to Yaml, so I'll modify the command to add the behaviour.

[m-vdb]

@dakshvar22 I am not sure how to implement the updates to the stories and domain files (your second point). My initial thoughts where:

1. use the rasa data convert nlg command
2. rename the response names before the yaml file is dumped, then dump the file
3. for every response renaming that was done, update the stories and domain files

1 and 2 are straightforward. For 3, I'm not sure how people would use it. Should they provide a --domain and --stories argument? Or should we detect where the domain and the stories files reside? For the former, it's straightforward. For the latter, I am not sure how to do it.

cc @degiz do you have any thoughts on this?

[dakshvar22]

I think the default can be that the user provides path to the training data. In that path, we try to search for -

1. NLU files(including responses.yml),
2. Story files
   Additionally, we also search for domain in its default path. Based on what all we get, we should be able to modify.
   The command can also let the user explicitly specify the paths for domain and stories which would override the default paths.

What do you think?

[m-vdb]

The thing is that with this approach, you have to consider the conversion of the whole project: are stories in Markdown or already converted to Yaml? I don't know if we can detect this reasonably well, I'd have to check.

[degiz]

The thing is that with this approach, you have to consider the conversion of the whole project: are stories in Markdown or already converted to Yaml

I believe the only two cases to support here is when users migrate from MD to MD (and we won't do that since MD is deprecated), or from YAML to YAML.

use the rasa data convert nlg command

I honestly think that rasa data convert nlg shouldn't look for domain and stories.
Instead, I'd rather add something like rasa data convert responses, and add descriptive help with an explaination of what it's doing.

So whole workflow for users with existing MD training data will be:

• convert MD to YAML using rasa data convert {nlu|core|nlg}, that only reads MD files
• convert responses to the utter_ prefix with rasa data convert responses, this will modify domain and stories if needed.

[m-vdb]

Thanks @degiz

I honestly think that rasa data convert nlg shouldn't look for domain and stories.

can you briefly explain why?

[degiz]

can you briefly explain why?

rasa data convert {nlu|core|nlg} all work with training data only. Also, we don't require stories to be present when we convert nlu data, we also don't expect the domain to be present when we convert nlg.

Otherwise, we'll end up with a full English dictionary in the args 😄 IMO that's super confusing:
rasa data convert nlg --domain domain.yaml --stories stories.yaml

I realize though, that rasa data convert responses will also require at least --core param for the stories.

[m-vdb]

OK makes sense. I'm for simplifying the implementation of this, because it will result in a simpler usage; having this in a separate command sounds better 👍