Merge 3.0.x into main (#10777)

* Update dependencies in 3.0 to align with rasa-sdk (#10667 * bump rasa-sdk * filtering messages during training/prediction draft * remove unfeaturized messages for some nlu components * update forms docs with dynamic form example for removal of required slot * Use tf.function for model prediction in RasaModel. (#10738) * Use prompt_toolkit ^2.0 (#10761) Co-authored-by: jupyterjazz <[email protected]> Co-authored-by: carlad <[email protected]> Co-authored-by: Anca Lita <[email protected]> Co-authored-by: Matthias Leimeister <[email protected]>
RasaHQ · Feb 1, 2022 · b1f43cc · b1f43cc
1 parent 1e21478
commit b1f43cc
Show file tree

Hide file tree

Showing 22 changed files with 426 additions and 144 deletions.
diff --git a/CHANGELOG.mdx b/CHANGELOG.mdx
@@ -16,6 +16,21 @@ https://github.com/RasaHQ/rasa/tree/main/changelog/ . -->
 
 <!-- TOWNCRIER -->
 
+## [3.0.6] - 2022-01-28
+### Deprecations and Removals
+- [#10590](https://github.com/rasahq/rasa/issues/10590): Removed CompositionView.
+
+### Bugfixes
+- [#10504](https://github.com/rasahq/rasa/issues/10504): Fixes a bug which was caused by `DIETClassifier` (`ResponseSelector`, `SklearnIntentClassifier` and `CRFEntityExtractor` have the same issue) trying to process message which didn't have required features. Implements removing unfeaturized messages for the above-mentioned components before training and prediction.
+- [#10540](https://github.com/rasahq/rasa/issues/10540): Enable slots with `from_entity` mapping that are not part of a form's required slots to be set during active loop.
+- [#10673](https://github.com/rasahq/rasa/issues/10673): Catch `ValueError` for any port values that cannot be cast to integer and re-raise as `RasaException` during the initialisation of `SQLTrackerStore`.
+- [#10728](https://github.com/rasahq/rasa/issues/10728): Use `tf.function` for model prediction to improve inference speed.
+- [#10761](https://github.com/rasahq/rasa/issues/10761): Tie prompt-toolkit to ^2.0 to fix `rasa-shell`.
+
+### Improved Documentation
+- [#10536](https://github.com/rasahq/rasa/issues/10536): Update dynamic form behaviour docs section with an example on how to override `required_slots` in case of removal of a form required slot.
+
+
 ## [3.0.5] - 2022-01-19
 ### Bugfixes
 - [#10519](https://github.com/rasahq/rasa/issues/10519): Corrects `transformer_size` parameter value (`None` by default) with a default size during loading in case `ResponseSelector` contains transformer layers.

diff --git a/changelog/10519.bugfix.md b/changelog/10519.bugfix.md
diff --git a/changelog/10590.removal.md b/changelog/10590.removal.md
diff --git a/changelog/592.misc.md b/changelog/592.misc.md
diff --git a/docs/docs/forms.mdx b/docs/docs/forms.mdx
@@ -350,7 +350,7 @@ By default Rasa Open Source will ask for the next empty slot from the slots
 listed for your form in the domain file. If you use
 [custom slot mappings](forms.mdx#custom-slot-mappings) and the `FormValidationAction`,
 it will ask for the first empty slot returned by the `required_slots` method. If all
-slots in `required_slots` are filled the form will be be deactivated.
+slots in `required_slots` are filled the form will be deactivated.
 
 If needed, you can update the required slots of your form dynamically.
 This is, for example, useful when you need further details based on how
@@ -388,6 +388,35 @@ class ValidateRestaurantForm(FormValidationAction):
         return additional_slots + domain_slots
 ```
 
+If conversely, you want to remove a slot from the form's `required_slots` defined in the domain file under certain conditions,
+you should copy the `domain_slots` over to a new variable and apply changes to that new variable instead of directly modifying
+`domain_slots`. Directly modifying `domain_slots` can cause unexpected behaviour. For example:
+
+```python
+from typing import Text, List, Optional
+
+from rasa_sdk.forms import FormValidationAction
+
+class ValidateBookingForm(FormValidationAction):
+    def name(self) -> Text:
+        return "validate_booking_form"
+
+    async def required_slots(
+        self,
+        domain_slots: List[Text],
+        dispatcher: "CollectingDispatcher",
+        tracker: "Tracker",
+        domain: "DomainDict",
+    ) -> List[Text]:
+        updated_slots = domain_slots.copy()
+        if tracker.slots.get("existing_customer") is True:
+            # If the user is an existing customer,
+            # do not request the `email_address` slot
+            updated_slots.remove("email_address")
+
+        return updated_slots
+```
+
 ### The requested_slot slot
 
 The slot `requested_slot` is automatically added to the domain as a

diff --git a/poetry.lock b/poetry.lock
diff --git a/pyproject.toml b/pyproject.toml
@@ -9,7 +9,7 @@ exclude = "((.eggs | .git | .pytest_cache | build | dist))"
 
 [tool.poetry]
 name = "rasa"
-version = "3.0.5"
+version = "3.0.6"
 description = "Open source machine learning framework to automate text- and voice-based conversations: NLU, dialogue management, connect to Slack, Facebook, and more - Create chatbots and voice assistants"
 authors = [ "Rasa Technologies GmbH <[email protected]>",]
 maintainers = [ "Tom Bocklisch <[email protected]>",]
@@ -90,7 +90,7 @@ colorhash = "~1.0.2"
 jsonschema = "~3.2"
 packaging = ">=20.0,<21.0"
 pytz = ">=2019.1,<2022.0"
-rasa-sdk = ">=3.0.3"
+rasa-sdk = "~3.0.4"
 colorclass = "~2.2"
 terminaltables = "~3.1.0"
 sanic = "^21.6.0"

diff --git a/rasa/core/actions/action.py b/rasa/core/actions/action.py
@@ -1142,6 +1142,9 @@ def _fails_unique_entity_mapping_check(
 
         form = FormAction(form_name, self._action_endpoint)
 
+        if slot_name not in form.required_slots(domain):
+            return False
+
         if form.entity_mapping_is_unique(mapping, domain):
             return False
 

diff --git a/rasa/core/tracker_store.py b/rasa/core/tracker_store.py
@@ -751,6 +751,21 @@ def ensure_schema_exists(session: "Session") -> None:
             raise ValueError(schema_name)
 
 
+def validate_port(port: Any) -> Optional[int]:
+    """Ensure that port can be converted to integer.
+
+    Raises:
+        RasaException if port cannot be cast to integer.
+    """
+    if port is not None and not isinstance(port, int):
+        try:
+            port = int(port)
+        except ValueError as e:
+            raise RasaException(f"The port '{port}' cannot be cast to integer.") from e
+
+    return port
+
+
 class SQLTrackerStore(TrackerStore):
     """Store which can save and retrieve trackers from an SQL database."""
 
@@ -787,6 +802,8 @@ def __init__(
     ) -> None:
         import sqlalchemy.exc
 
+        port = validate_port(port)
+
         engine_url = self.get_db_url(
             dialect, host, port, db, username, password, login_db, query
         )

diff --git a/rasa/nlu/classifiers/diet_classifier.py b/rasa/nlu/classifiers/diet_classifier.py
@@ -706,6 +706,14 @@ def _create_model_data(
                 example for example in training_data if label_attribute in example.data
             ]
 
+        training_data = [
+            message
+            for message in training_data
+            if message.features_present(
+                attribute=TEXT, featurizers=self.component_config.get(FEATURIZERS)
+            )
+        ]
+
         if not training_data:
             # no training data are present to train
             return RasaModelData()
@@ -929,6 +937,8 @@ def _predict(
 
         # create session data from message and convert it into a batch of 1
         model_data = self._create_model_data([message], training=False)
+        if model_data.is_empty():
+            return None
         return self.model.run_inference(model_data)
 
     def _predict_label(

diff --git a/rasa/nlu/classifiers/sklearn_intent_classifier.py b/rasa/nlu/classifiers/sklearn_intent_classifier.py
@@ -19,6 +19,7 @@
 from rasa.nlu.classifiers.classifier import IntentClassifier
 from rasa.shared.nlu.training_data.training_data import TrainingData
 from rasa.shared.nlu.training_data.message import Message
+from rasa.utils.tensorflow.constants import FEATURIZERS
 
 logger = logging.getLogger(__name__)
 
@@ -124,11 +125,15 @@ def train(self, training_data: TrainingData) -> Resource:
             return self._resource
 
         y = self.transform_labels_str2num(labels)
+        training_examples = [
+            message
+            for message in training_data.intent_examples
+            if message.features_present(
+                attribute=TEXT, featurizers=self.component_config.get(FEATURIZERS)
+            )
+        ]
         X = np.stack(
-            [
-                self._get_sentence_features(example)
-                for example in training_data.intent_examples
-            ]
+            [self._get_sentence_features(example) for example in training_examples]
         )
         # reduce dimensionality
         X = np.reshape(X, (len(X), -1))
@@ -190,9 +195,12 @@ def _create_classifier(
     def process(self, messages: List[Message]) -> List[Message]:
         """Return the most likely intent and its probability for a message."""
         for message in messages:
-            if not self.clf:
+            if not self.clf or not message.features_present(
+                attribute=TEXT, featurizers=self.component_config.get(FEATURIZERS)
+            ):
                 # component is either not trained or didn't
-                # receive enough training data
+                # receive enough training data or the input doesn't
+                # have required features.
                 intent = None
                 intent_ranking = []
             else:

diff --git a/rasa/nlu/extractors/crf_entity_extractor.py b/rasa/nlu/extractors/crf_entity_extractor.py
@@ -33,7 +33,7 @@
     SPLIT_ENTITIES_BY_COMMA_DEFAULT_VALUE,
 )
 from rasa.shared.constants import DOCS_URL_COMPONENTS
-from rasa.utils.tensorflow.constants import BILOU_FLAG
+from rasa.utils.tensorflow.constants import BILOU_FLAG, FEATURIZERS
 
 logger = logging.getLogger(__name__)
 
@@ -242,7 +242,13 @@ def train(self, training_data: TrainingData) -> Resource:
 
         # filter out pre-trained entity examples
         entity_examples = self.filter_trainable_entities(training_data.nlu_examples)
-
+        entity_examples = [
+            message
+            for message in entity_examples
+            if message.features_present(
+                attribute=TEXT, featurizers=self.component_config.get(FEATURIZERS)
+            )
+        ]
         dataset = [self._convert_to_crf_tokens(example) for example in entity_examples]
 
         self._train_model(dataset)
@@ -278,7 +284,9 @@ def process(self, messages: List[Message]) -> List[Message]:
 
     def extract_entities(self, message: Message) -> List[Dict[Text, Any]]:
         """Extract entities from the given message using the trained model(s)."""
-        if self.entity_taggers is None:
+        if self.entity_taggers is None or not message.features_present(
+            attribute=TEXT, featurizers=self.component_config.get(FEATURIZERS)
+        ):
             return []
 
         tokens = message.get(TOKENS_NAMES[TEXT])

diff --git a/rasa/utils/tensorflow/models.py b/rasa/utils/tensorflow/models.py
@@ -269,7 +269,7 @@ def _rasa_predict(
 
         # Once we take advantage of TF's distributed training, this is where
         # scheduled functions will be forced to execute and return actual values.
-        outputs = tf_utils.sync_to_numpy_or_python_type(self.predict_step(batch_in))
+        outputs = tf_utils.sync_to_numpy_or_python_type(self._tf_predict_step(batch_in))
         if DIAGNOSTIC_DATA in outputs:
             outputs[DIAGNOSTIC_DATA] = self._empty_lists_to_none_in_dict(
                 outputs[DIAGNOSTIC_DATA]

diff --git a/rasa/version.py b/rasa/version.py
@@ -1,3 +1,3 @@
 # this file will automatically be changed,
 # do not add anything but the version number here!
-__version__ = "3.0.5"
+__version__ = "3.0.6"
diff --git a/scripts/release.py b/scripts/release.py
@@ -160,7 +160,7 @@ def is_valid_version_number(v: Text) -> bool:
                 str(current_version.next_release_candidate("major")),
             ]
         version = questionary.select(
-            f"Which {version} do you want to release?", choices=choices,
+            f"Which {version} do you want to release?", choices=choices
         ).ask()
 
     if version:
@@ -175,7 +175,6 @@ def get_rasa_sdk_version() -> Text:
 
     dependencies_filename = "pyproject.toml"
     toml_data = toml.load(project_root() / dependencies_filename)
-
     try:
         sdk_version = toml_data["tool"]["poetry"]["dependencies"]["rasa-sdk"]
         return sdk_version[1:].strip()

diff --git a/tests/core/test_actions.py b/tests/core/test_actions.py
@@ -2586,3 +2586,77 @@ async def test_action_extract_slots_does_not_raise_disallowed_warning_for_slot_e
             SlotSet("custom_slot_b", "test_B"),
             SlotSet("custom_slot_a", "test_A"),
         ]
+
+
+async def test_action_extract_slots_non_required_form_slot_with_from_entity_mapping():
+    domain_yaml = textwrap.dedent(
+        """
+        version: "3.0"
+
+        intents:
+        - form_start
+        - intent1
+        - intent2
+
+        entities:
+        - form1_info1
+        - form1_slot1
+        - form1_slot2
+
+        slots:
+          form1_info1:
+            type: text
+            mappings:
+            - type: from_entity
+              entity: form1_info1
+
+          form1_slot1:
+            type: text
+            influence_conversation: false
+            mappings:
+            - type: from_intent
+              value: Filled
+              intent: intent1
+              conditions:
+              - active_loop: form1
+                requested_slot: form1_slot1
+
+          form1_slot2:
+            type: text
+            influence_conversation: false
+            mappings:
+            - type: from_intent
+              value: Filled
+              intent: intent2
+              conditions:
+              - active_loop: form1
+                requested_slot: form1_slot2
+        forms:
+          form1:
+            required_slots:
+            - form1_slot1
+            - form1_slot2
+        """
+    )
+    domain = Domain.from_yaml(domain_yaml)
+    initial_events = [
+        UserUttered("Start form."),
+        ActiveLoop("form1"),
+        SlotSet(REQUESTED_SLOT, "form1_slot1"),
+        UserUttered(
+            "Hi",
+            intent={"name": "intent1"},
+            entities=[{"entity": "form1_info1", "value": "info1"}],
+        ),
+    ]
+    tracker = DialogueStateTracker.from_events(sender_id="test_id", evts=initial_events)
+
+    action_extract_slots = ActionExtractSlots(None)
+
+    events = await action_extract_slots.run(
+        CollectingOutputChannel(),
+        TemplatedNaturalLanguageGenerator(domain.responses),
+        tracker,
+        domain,
+    )
+    assert events == [SlotSet("form1_info1", "info1"), SlotSet("form1_slot1", "Filled")]
diff --git a/tests/core/test_tracker_stores.py b/tests/core/test_tracker_stores.py
@@ -35,7 +35,7 @@
     BotUttered,
     Event,
 )
-from rasa.shared.exceptions import ConnectionException
+from rasa.shared.exceptions import ConnectionException, RasaException
 from rasa.core.tracker_store import (
     TrackerStore,
     InMemoryTrackerStore,
@@ -925,3 +925,12 @@ def test_sql_tracker_store_with_token_serialisation(
 ):
     tracker_store = SQLTrackerStore(domain, **{"host": "sqlite:///"})
     prepare_token_serialisation(tracker_store, response_selector_agent, "sql")
+
+
+def test_sql_tracker_store_creation_with_invalid_port(domain: Domain):
+    with pytest.raises(RasaException) as error:
+        TrackerStore.create(
+            EndpointConfig(port="$DB_PORT", type="sql"),
+            domain,
+        )
+    assert "port '$DB_PORT' cannot be cast to integer." in str(error.value)