New Response Selector training data format

changelog/6591.improvement.md

@@ -0,0 +1,5 @@
+Retrieval actions with `respond_` prefix are now replaced with usual utterance actions with `utter_` prefix.
+
+If you were using retrieval actions before, rename all of them to start with `utter_` prefix. For example, `respond_chitchat` becomes `utter_chitchat`.
+Also, in order to keep the response templates more consistent, you should now add the `utter_` prefix to all response templates defined for retrieval intents. For example, a response template `chitchat/ask_name` becomes `utter_chitchat/ask_name`. Note that the NLU examples for this will still be under `chitchat/ask_name` intent.
+The example `responseselectorbot` should help clarify these changes further. 

data/examples/rasa/demo-rasa-responses.md

@@ -1,7 +1,7 @@
 ##
-* chitchat/ask_weather
+* utter_chitchat/ask_weather
     - It's sunny where I live
 
 ##
-* chitchat/ask_name
+* utter_chitchat/ask_name
     - I am Mr. Bot

data/test_domains/default_retrieval_intents.yml

@@ -7,8 +7,6 @@ intents:
   - mood_unhappy
   - bot_challenge
   - chitchat
-  - chitchat/ask_name
-  - chitchat/ask_weather
 
 responses:
   utter_greet:
@@ -26,7 +24,7 @@ responses:
     - text: I am a bot, powered by Rasa.
 
 actions:
-  - respond_chitchat
+  - utter_chitchat
   - utter_greet
   - utter_cheer_up
   - utter_did_that_help

data/test_responses/default.md

@@ -1,7 +1,7 @@
 ## ask name
-* chitchat/ask_name
+* utter_chitchat/ask_name
     - my name is Sara, Rasa's documentation bot!
 
 ## ask weather
-* chitchat/ask_weather
+* utter_chitchat/ask_weather
     - it's always sunny where I live

docs/docs/components/selectors.mdx

@@ -47,7 +47,8 @@ Selectors predict a bot response from a set of candidate responses.
             {
               "text": "I think it's about to rain."
             }
-          ]
+          ],
+          "template_name": "utter_chitchat/ask_weather"
         },
         "ranking": [
           {
@@ -71,7 +72,7 @@ Selectors predict a bot response from a set of candidate responses.
 * **Description**
 
   Response Selector component can be used to build a response retrieval model to directly predict a bot response from
-  a set of candidate responses. The prediction of this model is used by [Retrieval Actions](../retrieval-actions).
+  a set of candidate responses. The prediction of this model is used by the dialogue manager to utter the predicted responses.
   It embeds user inputs and response labels into the same space and follows the exact same
   neural network architecture and optimization as the [DIETClassifier](../components/intent-classifiers.mdx#dietclassifier).
 
@@ -142,6 +143,10 @@ Selectors predict a bot response from a set of candidate responses.
   template which has a text attribute for training. If none are found, it falls back to using the retrieval intent
   combined with the response key as the label.
 
+  See this [example assistant](https://github.com/RasaHQ/rasa/tree/master/examples/responseselectorbot) to understand
+  how you can use the `ResponseSelector` component in your assistant. Additionally, you will find this tutorial on
+  [handling FAQs](./chitchat-faqs.mdx#handling-faqs-using-a-response-selector) using a `ResponseSelector` useful as well.
+
   <details>
     <summary>
       The above configuration parameters are the ones you should configure to fit your model to your data.
@@ -289,3 +294,94 @@ Selectors predict a bot response from a set of candidate responses.
 
     :::
   </details>
+
+* **Under the hood: Parsing Response Selector Output**
+
+    The parsed output from NLU will have a property named `response_selector`
+    containing the output for each response selector component. Each response selector is
+    identified by `retrieval_intent` parameter of that response selector
+    and stores two properties:
+
+    * `response`: The predicted response key under the corresponding retrieval intent,
+    prediction's confidence and the associated response templates.
+
+    * `ranking`: Ranking with confidences of top 10 candidate response keys.
+
+    Example result:
+
+    ```json
+    {
+        "text": "How's the weather today?",
+        "response_selector": {
+          "faq": {
+            "response": {
+              "id": 1388783286124361986,
+              "confidence": 0.7,
+              "intent_response_key": "chitchat/ask_weather",
+              "response_templates": [
+                {
+                  "text": "It's sunny in Berlin today",
+                  "image": "https://i.imgur.com/nGF1K8f.jpg"
+                },
+                {
+                  "text": "I think it's about to rain."
+                }
+              ],
+              "template_name": "utter_chitchat/ask_weather"
+             },
+            "ranking": [
+              {
+                "id": 1388783286124361986,
+                "confidence": 0.7,
+                "intent_response_key": "chitchat/ask_weather"
+              },
+              {
+                "id": 1388783286124361986,
+                "confidence": 0.3,
+                "intent_response_key": "chitchat/ask_name"
+              }
+            ]
+          }
+        }
+    }
+    ```
+
+    If the `retrieval_intent` parameter of a particular response selector was left to its default value,
+    the corresponding response selector will be identified as `default` in the returned output.
+
+    ```json {4}
+    {
+        "text": "How's the weather today?",
+        "response_selector": {
+          "default": {
+            "response": {
+              "id": 1388783286124361986,
+              "confidence": 0.7,
+              "intent_response_key": "chitchat/ask_weather",
+              "response_templates": [
+                {
+                  "text": "It's sunny in Berlin today",
+                  "image": "https://i.imgur.com/nGF1K8f.jpg"
+                },
+                {
+                  "text": "I think it's about to rain."
+                }
+              ],
+              "template_name": "utter_chitchat/ask_weather"
+             },
+            "ranking": [
+              {
+                "id": 1388783286124361986,
+                "confidence": 0.7,
+                "intent_response_key": "chitchat/ask_weather"
+              },
+              {
+                "id": 1388783286124361986,
+                "confidence": 0.3,
+                "intent_response_key": "chitchat/ask_name"
+              }
+            ]
+          }
+        }
+    }
+    ```

docs/docs/glossary.mdx

@@ -55,6 +55,10 @@ description: Glossary for all Rasa-related terms
 
   Something that a user is trying to convey or accomplish (e,g., greeting, specifying a location).
 
+## Retrieval Intent
+
+  A special instance of an intent which can be divided into smaller sub-intents. Each sub-intent has a fixed response and hence the context of the conversation does not matter when user expresses one of these sub-intents.
+
 ## [Interactive Learning](./writing-stories.mdx#using-interactive-learning)
 
   A mode of training the bot where the user provides feedback to the bot while talking to it.

docs/docs/migration-guide.mdx

@@ -377,13 +377,21 @@ This means that the output in these files should look like -
 }
 ```
 
+* Retrieval actions with `respond_` prefix are now replaced with usual utterance actions with `utter_` prefix.
+If you were using retrieval actions before, rename all of them to start with `utter_` prefix. For example,
+`respond_chitchat` becomes `utter_chitchat`. Also, in order to keep the response templates more consistent,
+you should now add the `utter_` prefix to all response templates defined for retrieval intents. For example,
+a response template `chitchat/ask_name` becomes `utter_chitchat/ask_name`. Note that the NLU examples for this
+will still be under `chitchat/ask_name` intent. The example `responseselectorbot` should help clarify these changes further.
+
 * The output schema of `ResponseSelector` has changed - `full_retrieval_intent` and `name`
 have been deprecated in favour of `intent_response_key` and `response_templates` respectively.
-Additionally a key `all_retrieval_intents` is added to the response selector output which will
-hold a list of all retrieval intents(faq, chitchat, etc.) that are present in the training data.
+Additionally, two keys are added to the output -
+1. `all_retrieval_intents` - Holds a list of all retrieval intents(faq, chitchat, etc.) that are present in the training data.
+2. `template_name` - Holds the name of the response template which is predicted by the response selector(`utter_faq/is_legit`)
 An example output looks like this -
 
-```json {3-4,10,11}
+```json {3-4,10,11,20}
 {
   "response_selector": {
     "all_retrieval_intents": [
@@ -403,6 +411,7 @@ An example output looks like this -
             "text": "I think so."
           }
         ]
+        "template_name": "utter_faq/is_legit"
       },
       "ranking": [
         {

docs/docs/responses.mdx

@@ -258,3 +258,31 @@ responses:
   - text: "Hey, {name}. How are you?"
   - text: "Hey, {name}. How is your day going?"
 ```
+
+## Responses for Retrieval Intents
+
+If you are using retrieval intents in your assistant, you also need to add response templates
+for your assistant's replies to these intents in a separate file:
+
+```yaml-rasa title="responses.yml"
+responses:
+  utter_chitchat/ask_name:
+  - image: "https://i.imgur.com/zTvA58i.jpeg"
+    text: hello, my name is retrieval bot.
+  - text: Oh yeah, I am called the retrieval bot.
+
+  utter_chitchat/ask_weather:
+  - text: Oh, it does look sunny right now in Berlin.
+    image: "https://i.imgur.com/vwv7aHN.png"
+  - text: I am not sure of the whole week but I can see the sun is out today.
+```
+All such response templates start with the `utter_` prefix followed by the retrieval intent name(`chitchat`)
+and the associated response key(`ask_name` or `ask_weather`).
+
+:::info Responses format
+The responses for retrieval intents use the same format as the [responses in the domain](responses.mdx).
+This means, you can also use buttons, images and any other multimedia elements in
+your responses.
+:::
+
+

docs/docs/training-data-format.mdx

@@ -185,6 +185,35 @@ nlu:
 The `metadata` key can contain arbitrary key-value data that stays with an example and is accessible by the components in the NLU pipeline. In the example above, the sentiment of
 the example could be used by a custom component in the pipeline for sentiment analysis.
 
+If you want to specify [retrieval intents](glossary.mdx#retrieval-intent), then your NLU examples will look as follows:
+```yaml-rasa
+nlu:
+- intent: chitchat/ask_name
+  examples: |
+    - What is your name?
+    - May I know your name?
+    - What do people call you?
+    - Do you have a name for yourself?
+
+- intent: chitchat/ask_weather
+  examples: |
+    - What's the weather like today?
+    - Does it look sunny outside today?
+    - Oh, do you mind checking the weather for me please?
+    - I like sunny days in Berlin.
+```
+All retrieval intents have a suffix
+added to them which identifies a particular response key for your assistant, in the
+above example - `ask_name` and `ask_weather`. The suffix is separated from
+the retrieval intent name by a `/` delimiter.
+
+:::note Special meaning of `/`
+As shown in the above examples, the `/` symbol is reserved as a delimiter to separate
+retrieval intents from their associated response keys. Make sure not to use it in the
+name of your intents.
+:::
+
+
 ### Entities
 
 [Entities](glossary.mdx#entity) are structured pieces of information that can be extracted from a user's message. For entity extraction to work, you need to either specify training data to train an ML model or you need to define [regular expressions](#regular-expressions-for-entity-extraction) to extract entities using the [`RegexEntityExtractor`](components/entity-extractors.mdx#regexentityextractor) based on a character pattern.

examples/responseselectorbot/data/responses.yml

@@ -1,12 +1,12 @@
 version: "2.0"
 
 responses:
-  chitchat/ask_name:
+  utter_chitchat/ask_name:
   - image: "https://i.imgur.com/zTvA58i.jpeg"
     text: hello, my name is retrieval bot.
   - text: Oh yeah, I am called the retrieval bot.
 
-  chitchat/ask_weather:
+  utter_chitchat/ask_weather:
   - text: Oh, it does look sunny right now in Berlin.
     image: "https://i.imgur.com/vwv7aHN.png"
   - text: I am not sure of the whole week but I can see the sun is out today.

examples/responseselectorbot/data/rules.yml

@@ -15,4 +15,4 @@ rules:
 - rule: Response with a chitchat utterance whenever user indulges in some chitchat
   steps:
   - intent: chitchat
-  - action: respond_chitchat
+  - action: utter_chitchat

examples/responseselectorbot/domain.yml

@@ -10,9 +10,6 @@ intents:
   - bot_challenge
   - chitchat
 
-actions:
-  - respond_chitchat
-
 responses:
   utter_greet:
   - text: "Hey! How are you?"