diff --git a/CHANGELOG.mdx b/CHANGELOG.mdx index 5b228a7fdb98..ab7a35a3c31f 100644 --- a/CHANGELOG.mdx +++ b/CHANGELOG.mdx @@ -16,6 +16,24 @@ https://github.com/RasaHQ/rasa/tree/master/changelog/ . --> +## [2.0.3] - 2020-10-29 + + +### Bugfixes +- [#7089](https://github.com/rasahq/rasa/issues/7089): Fix [ConveRTTokenizer](components.mdx#converttokenizer) failing because of wrong model URL by making the `model_url` parameter of `ConveRTTokenizer` mandatory. + + Since the ConveRT model was taken [offline](https://github.com/RasaHQ/rasa/issues/6806), we can no longer use + the earlier public URL of the model. Additionally, since the licence for the model is unknown, + we cannot host it ourselves. Users can still use the component by setting `model_url` to a community/self-hosted + model URL or path to a local directory containing model files. For example: + ```yaml + pipeline: + - name: ConveRTTokenizer + model_url: + ``` +- [#7108](https://github.com/rasahq/rasa/issues/7108): Update example formbot to use `FormValidationAction` for slot validation + + ## [2.0.2] - 2020-10-22 diff --git a/CODEOWNERS b/CODEOWNERS index d3a3bf89e90c..0ff48ccbfe35 100644 --- a/CODEOWNERS +++ b/CODEOWNERS @@ -1,4 +1,4 @@ # Code in the `rasa.shared` package is potentially re-used by downstream dependencies # such as Rasa X. Hence, changes within this package require double checking. -/rasa/shared/ @backend +/rasa/shared/ @RasaHQ/backend /docs/docs/prototype-an-assistant.mdx @ricwo @alwx diff --git a/docs/docs/components.mdx b/docs/docs/components.mdx index cc1f19564ef0..4bdd3e6cd220 100644 --- a/docs/docs/components.mdx +++ b/docs/docs/components.mdx @@ -453,10 +453,16 @@ word vectors in your pipeline. "intent_split_symbol": "_" # Regular expression to detect tokens "token_pattern": None - # Remote URL of hosted model - "model_url": TF_HUB_MODULE_URL + # Remote URL/Local directory of model files(Required) + "model_url": None ``` + :::note + Since the public URL of the ConveRT model was taken offline recently, it is now mandatory + to set the parameter `model_url` to a community/self-hosted URL or path to a local directory containing model files. + + ::: + ### LanguageModelTokenizer diff --git a/docs/docs/playground.mdx b/docs/docs/playground.mdx index 860388b1bfa5..d22cb5d2a4a0 100644 --- a/docs/docs/playground.mdx +++ b/docs/docs/playground.mdx @@ -100,9 +100,9 @@ responses: What is your email address? utter_subscribed: - text: | - I've subscribed {email} to the newsletter! + Check your inbox at {email} in order to finish subscribing to the newsletter! - text: | - You've been subscribed, the newsletter will be sent to {email}. + You're all set! Check your inbox at {email} to confirm your subscription. ``` diff --git a/docs/docs/setting-up-ci-cd.mdx b/docs/docs/setting-up-ci-cd.mdx index 72b953b9fb64..a37185abf168 100644 --- a/docs/docs/setting-up-ci-cd.mdx +++ b/docs/docs/setting-up-ci-cd.mdx @@ -71,7 +71,7 @@ to your server as part of the continuous deployment process. ### Testing Your Assistant -Testing your trained model on [test stories](./testing-your-assistant.mdx#end-to-end-testing) is the best way to have confidence in how your assistant +Testing your trained model on [test stories](./testing-your-assistant.mdx#writing-test-stories) is the best way to have confidence in how your assistant will act in certain situations. Written in a modified story format, test stories allow you to provide entire conversations and test that, given certain user input, your model will behave in the expected manner. This is especially diff --git a/docs/docs/testing-your-assistant.mdx b/docs/docs/testing-your-assistant.mdx index e08677f9dfc4..9c5e5d39cafe 100644 --- a/docs/docs/testing-your-assistant.mdx +++ b/docs/docs/testing-your-assistant.mdx @@ -3,22 +3,22 @@ id: testing-your-assistant sidebar_label: Testing Your Assistant title: Testing Your Assistant abstract: Rasa Open Source lets you test dialogues end-to-end by running through - test stories. You can also test the dialogue management and the message processing - (NLU) separately. + test stories. In addition, you can + also test the dialogue management and the message processing (NLU) + separately. --- -## End-to-end Testing +## Writing Test Stories -Testing your assistant requires you to write test stories, which include -the user messages and the conversation history. The format is a modified version of [the one -used to specify stories in your training data](./stories.mdx#format). +To test your assistant completely, the best approach is to write test stories. Test stories are like +the stories in your training data, but include the user message as well. Here are some examples: - ```yaml-rasa title="tests/test_stories.yml" + ```yaml-rasa title="tests/test_stories.yml" {5,9,13} stories: - story: A basic end-to-end test steps: @@ -39,7 +39,7 @@ Here are some examples: - ```yaml-rasa title="tests/test_stories.yml" + ```yaml-rasa title="tests/test_stories.yml" {5,12} stories: - story: A test where a custom action returns events steps: @@ -60,7 +60,7 @@ Here are some examples: - ```yaml-rasa title="tests/test_stories.yml" + ```yaml-rasa title="tests/test_stories.yml" {5,9,14,20} stories: - story: A test story with a form steps: @@ -88,7 +88,7 @@ Here are some examples: - ```yaml-rasa title="tests/test_stories.yml" + ```yaml-rasa title="tests/test_stories.yml" {5,9,14,21} stories: - story: A test story with unexpected input during a form steps: @@ -102,7 +102,7 @@ Here are some examples: - action: restaurant_form - active_loop: restaurant_form - user: | - can you share your boss with me? + How's the weather? intent: chitchat - action: utter_chitchat - action: restaurant_form @@ -117,113 +117,145 @@ Here are some examples: -By default Rasa Open Source saves test stories to `tests/test_stories.yml`. +Rasa Open Source looks for test stories in all files with the prefix `test_`, e.g. `tests/test_stories.yml`. You can test your assistant against them by running: ```bash rasa test ``` -The command will always load all stories from any story files with filenames -starting with `test_`, e.g. `test_stories.yml`. Your story test -file names should always start with `test_` for this detection to work. +See the [CLI documentation on `rasa test`](./command-line-interface.mdx#rasa-test) for +more configuration options. :::caution Testing Custom Actions -[Custom Actions](./custom-actions.mdx) are **not executed as part of test stories.** If your custom +[Custom Actions](./custom-actions.mdx) are not executed as part of test stories. If your custom actions append any events to the conversation, this has to be reflected in your test story (e.g. by adding `slot_was_set` events to your test story). To test the code of your custom actions, you should write unit tests -for them and include these tests in your CI/CD pipeline. +for them and include these tests in your [CI/CD pipeline](./setting-up-ci-cd.mdx). ::: -If you have any questions or problems, please share them with us in the dedicated -[testing section on our forum](https://forum.rasa.com/tags/testing). - ## Evaluating an NLU Model -:::tip hyperparameter tuning -If you are looking to tune the hyperparameters of your NLU model, -check out this [tutorial](https://blog.rasa.com/rasa-nlu-in-depth-part-3-hyperparameters/). -::: - - -A standard technique in machine learning is to keep some data separate as a test set. -You can [split your NLU training data](./command-line-interface.mdx#rasa-data-split) +In addition to end-to-end testing, you can also test the natural language understanding (NLU) model separately. +Once your assistant is deployed in the real world, it will be processing messages that it hasn't seen +in the training data. To simulate this, you should always set aside some part of your data for testing. +You can [split your NLU data](./command-line-interface.mdx#rasa-data-split) into train and test sets using: ```bash rasa data split nlu ``` -If you've done this, you can see how well your NLU model predicts the -test cases: +Next, you can see how well your trained NLU model predicts the +data from the test set you generated, using: -```bash -rasa test nlu --nlu train_test_split/test_data.yml +```bash {2} +rasa test nlu + --nlu train_test_split/test_data.yml ``` -If you don't want to create a separate test set, you can -still estimate how well your model generalises using cross-validation. -To do this, add the flag `--cross-validation`: +To test your model more extensively, use cross-validation, which automatically creates +multiple train/test splits: -```bash -rasa test nlu --nlu data/nlu.yml --cross-validation +```bash {3} +rasa test nlu + --nlu data/nlu.yml + --cross-validation ``` You can find the full list of options in the [CLI documentation on `rasa test`](command-line-interface.mdx#rasa-test). +:::tip hyperparameter tuning +To further improve your model check out this +[tutorial on hyperparameter tuning](https://blog.rasa.com/rasa-nlu-in-depth-part-3-hyperparameters/). +::: + ### Comparing NLU Pipelines -By passing multiple pipeline configurations (or a folder containing them) to the CLI, Rasa will run -a comparison between the pipelines: +To get the most out of your training data, you should train and evaluate your model on different pipelines +and different amounts of training data. -```bash -rasa test nlu --config config_1.yml config_2.yml - --nlu data/nlu.yml --runs 3 --percentages 0 25 50 70 90 +To do so, pass multiple configuration files to the `rasa test` command: + +```bash {2} +rasa test nlu --nlu data/nlu.yml + --config config_1.yml config_2.yml ``` -The command in the example above will create a train/test split from your data, -then train each pipeline multiple times with 0, 25, 50, 70 and 90% of your intent data excluded from the training set. -The models are then evaluated on the test set and the f1-score for each exclusion percentage is recorded. This process -runs three times (i.e. with 3 test sets in total) and then a graph is plotted using the means and standard deviations of -the f1-scores. +This performs several steps: +1. Create a global 80% train / 20% test split from `data/nlu.yml`. +2. Exclude a certain percentage of data from the global train split. +3. Train models for each configuration on remaining training data. +4. Evaluate each model on the global test split. + +The above process is repeated with different percentages of training data in step 2 +to give you an idea of how each pipeline will behave if you increase the amount of training data. +Since training is not completely deterministic, the whole process is repeated +three times for each configuration specified. +A graph with the mean and standard deviations of +[f1-scores](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html) +across all runs is plotted. The f1-score graph, along with all train/test sets, the trained models, classification and error reports, will be saved into a folder called `nlu_comparison_results`. -### Intent Classification +Inspecting the f1-score graph can help you understand if you have enough data for your NLU model. +If the graph shows that f1-score is still improving when all of the training data is used, +it may improve further with more data. But if f1-score has plateaued when all training data is used, +adding more data may not help. + +If you want to change the number of runs or exclusion percentages, you can: + +```bash {3} +rasa test nlu --nlu data/nlu.yml + --config config_1.yml config_2.yml + --runs 4 --percentages 0 25 50 70 90 +``` + +### Interpreting the Output -The `rasa test` script will produce a report, confusion matrix, and confidence histogram for your model. +#### Intent Classifiers -The report logs precision, recall and f1 measure for each intent and entity, +The `rasa test` script will produce a report (`intent_report.json`), confusion matrix (`intent_confusion_matrix.png`) +and confidence histogram (`intent_histogram.png`) for your intent classification model. + +The report logs [precision](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html), +[recall](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html) and +[f1-score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html) for each intent, as well as providing an overall average. You can save these reports as JSON files using the `--report` argument. -The confusion matrix shows you which intents are mistaken for others; -any samples which have been incorrectly predicted are logged and saved to a file called `errors.json` for easier debugging. +The confusion matrix shows which intents are mistaken for others. +Any samples which have been incorrectly predicted are logged and saved to a file called `errors.json` for easier debugging. -The histogram that the script produces allows you to visualise the confidence distribution for all predictions, with the volume of correct and incorrect predictions being displayed by blue and red bars respectively. Improving the quality of your training data will move the blue histogram bars to the right and the red histogram bars to the left of the plot. +The histogram allows you to visualize the confidence for all predictions, +with the correct and incorrect predictions being displayed by blue and red bars respectively. +Improving the quality of your training data will move the blue histogram bars to the right and the +red histogram bars to the left of the plot. It should also help in reducing the number red histogram bars itself. -### Response Selection +#### Response Selectors -The evaluation script will produce a combined report for all response selector models in your pipeline. +`rasa test` evaluates response selectors in the same way that it evaluates intent classifiers, producing a +report (`response_selection_report.json`), confusion matrix (`response_selection_confusion_matrix.png`), +confidence histogram (`response_selection_histogram.png`) and errors (`response_selection_errors.json`). +If your pipeline includes multiple response selectors, they are evaluated in a single report. The report logs precision, recall and f1 measure for -each response, as well as providing an overall average. +each sub-intent of a [retrieval intent](./glossary.mdx#retrieval-intent) and provides an overall average. You can save these reports as JSON files using the `--report` argument. -### Entity Extraction - -Only trainable entity extractors, such as the `DIETCLassifier` and `CRFEntityExtractor` will be -evaluated by the `rasa test` script. If you pretrained extractors like the `DucklingHTTPExtractor` -Rasa Open Source will not include these in the evaluation. +#### Entity Extraction -`rasa test` reports recall, precision, and f1 measure for each entity type that +`rasa test` reports recall, precision, and f1-score for each entity type that your trainable entity extractors are trained to recognize. +Only trainable entity extractors, such as the `DIETClassifier` and `CRFEntityExtractor` are +evaluated by `rasa test`. Pretrained extractors like the `DucklingHTTPExtractor` are not evaluated. + :::caution incorrect entity annotations If any of your entities are incorrectly annotated, your evaluation may fail. One common problem is that an entity cannot stop or start inside a token. @@ -235,15 +267,15 @@ multiple tokens. #### Entity Scoring -To evaluate entity extraction we apply a simple tag-based approach. We don't consider BILOU tags, but only the +To evaluate entity extraction we apply a simple tag-based approach. We don't consider +[BILOU tags](nlu-training-data.mdx#bilou-entity-tagging) exactly, but only the entity type tags on a per token basis. For location entity like “near Alexanderplatz” we -expect the labels `LOC LOC` instead of the BILOU-based `B-LOC L-LOC`. +expect the labels `LOC LOC` instead of the BILOU-based `B-LOC L-LOC`. -Our approach is more lenient -when it comes to evaluation, as it rewards partial extraction and does not punish the splitting of entities. +Our approach is more lenient when it comes to evaluation, as it rewards +partial extraction and does not penalize the splitting of entities. For example, given the aforementioned entity “near Alexanderplatz” and a system that extracts -“Alexanderplatz”, our approach rewards the extraction of “Alexanderplatz” and punishes the missed out word “near”. - +“Alexanderplatz”, our approach rewards the extraction of “Alexanderplatz” and penalizes the missed out word “near”. The BILOU-based approach, however, would label this as a complete failure since it expects Alexanderplatz to be labeled as a last token in an entity (`L-LOC`) instead of a single token entity (`U-LOC`). Note also that @@ -271,21 +303,20 @@ rasa test core --stories test_stories.yml --out results ``` This will print any failed stories to `results/failed_test_stories.yml`. -We count a story as failed if at least one of the actions -was predicted incorrectly. +A story fails if at least one of the actions was predicted incorrectly. The test script will also save a confusion matrix to a file called `results/story_confmat.pdf`. For each action in your domain, the confusion matrix shows how often the action was correctly predicted and how often an incorrect action was predicted instead. -## Comparing Policy Configurations +### Comparing Policy Configurations To choose a configuration for your dialogue model, or to choose hyperparameters for a specific policy, you want to measure how well your dialogue model will generalize to conversations it hasn't seen before. Especially in the beginning -of a project, when you don't have a lot of real conversations to use to train -your bot, you may not want to exclude some to use as a test set. +of a project, when you don't have a lot of real conversations to train +your bot on, you may not want to exclude some to use as a test set. Rasa Open Source has some scripts to help you choose and fine-tune your policy configuration. Once you are happy with it, you can then train your final configuration on your @@ -300,9 +331,11 @@ rasa train core -c config_1.yml config_2.yml \ --out comparison_models --runs 3 --percentages 0 5 25 50 70 95 ``` +Similar to how the [NLU model was evaluated](./testing-your-assistant.mdx#comparing-nlu-pipelines), the above +command trains the dialogue model on multiple configurations and different amounts of training data. For each config file provided, Rasa Open Source will train dialogue models with 0, 5, 25, 50, 70 and 95% of your training stories excluded from the training -data. This is done for multiple runs to ensure consistent results. +data. This is repeated three times to ensure consistent results. Once this script has finished, you can pass multiple models to the test script to compare the models you just trained: @@ -312,10 +345,11 @@ rasa test core -m comparison_models --stories stories_folder --out comparison_results --evaluate-model-directory ``` -This will evaluate each of the models on the provided stories +This will evaluate each model on the stories in `stories_folder` (can be either training or test set) and plot some graphs -to show you which policy performs best. By evaluating on the full set of stories, you -can measure how well your model predicts the held-out stories. +to show you which policy performs best. Since the previous train command +excluded some amount of training data to train each model, +the above test command can measure how well your model predicts the held-out stories. To compare single policies, create config files containing only one policy each. :::note @@ -323,3 +357,6 @@ This training process can take a long time, so we'd suggest letting it run somewhere in the background where it can't be interrupted. ::: + +If you have any questions or problems, please share them with us in the dedicated +[testing section on our forum](https://forum.rasa.com/tags/testing)! diff --git a/examples/formbot/actions/actions.py b/examples/formbot/actions/actions.py index 8ab8310083b8..d46f744becd3 100644 --- a/examples/formbot/actions/actions.py +++ b/examples/formbot/actions/actions.py @@ -2,48 +2,14 @@ from rasa_sdk import Tracker from rasa_sdk.executor import CollectingDispatcher -from rasa_sdk.forms import FormAction +from rasa_sdk.forms import FormValidationAction -class RestaurantForm(FormAction): - """Example of a custom form action.""" +class ValidateRestaurantForm(FormValidationAction): + """Example of a form validation action.""" def name(self) -> Text: - """Unique identifier of the form.""" - - return "restaurant_form" - - @staticmethod - def required_slots(tracker: Tracker) -> List[Text]: - """A list of required slots that the form has to fill.""" - - return ["cuisine", "num_people", "outdoor_seating", "preferences", "feedback"] - - def slot_mappings(self) -> Dict[Text, Union[Dict, List[Dict]]]: - """A dictionary to map required slots to - - an extracted entity - - intent: value pairs - - a whole message - or a list of them, where a first match will be picked.""" - - return { - "cuisine": self.from_entity(entity="cuisine", not_intent="chitchat"), - "num_people": [ - self.from_entity( - entity="number", intent=["inform", "request_restaurant"] - ), - ], - "outdoor_seating": [ - self.from_entity(entity="seating"), - self.from_intent(intent="affirm", value=True), - self.from_intent(intent="deny", value=False), - ], - "preferences": [ - self.from_intent(intent="deny", value="no additional preferences"), - self.from_text(not_intent="affirm"), - ], - "feedback": [self.from_entity(entity="feedback"), self.from_text()], - } + return "validate_restaurant_form" @staticmethod def cuisine_db() -> List[Text]: @@ -127,14 +93,3 @@ def validate_outdoor_seating( else: # affirm/deny was picked up as True/False by the from_intent mapping return {"outdoor_seating": value} - - def submit( - self, - dispatcher: CollectingDispatcher, - tracker: Tracker, - domain: Dict[Text, Any], - ) -> List[Dict]: - """Define what the form has to do after all required slots are filled.""" - - dispatcher.utter_message(template="utter_submit") - return [] diff --git a/examples/formbot/domain.yml b/examples/formbot/domain.yml index 34cdc4b7b37e..dc3441310fa0 100644 --- a/examples/formbot/domain.yml +++ b/examples/formbot/domain.yml @@ -113,6 +113,9 @@ forms: entity: feedback - type: from_text +actions: +- validate_restaurant_form + session_config: session_expiration_time: 60 # value in minutes carry_over_slots_to_new_session: true diff --git a/pyproject.toml b/pyproject.toml index 848a837da0e3..3b5043a28fb5 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -9,7 +9,7 @@ exclude = "((.eggs | .git | .pytest_cache | build | dist))" [tool.poetry] name = "rasa" -version = "2.0.2" +version = "2.0.3" description = "Open source machine learning framework to automate text- and voice-based conversations: NLU, dialogue management, connect to Slack, Facebook, and more - Create chatbots and voice assistants" authors = [ "Rasa Technologies GmbH ",] maintainers = [ "Tom Bocklisch ",] diff --git a/rasa/nlu/tokenizers/convert_tokenizer.py b/rasa/nlu/tokenizers/convert_tokenizer.py index f439b0cf0630..a2b4857732f1 100644 --- a/rasa/nlu/tokenizers/convert_tokenizer.py +++ b/rasa/nlu/tokenizers/convert_tokenizer.py @@ -7,17 +7,27 @@ from rasa.nlu.tokenizers.whitespace_tokenizer import WhitespaceTokenizer from rasa.shared.nlu.training_data.message import Message from rasa.utils import common +import rasa.nlu.utils import rasa.utils.train_utils as train_utils +from rasa.exceptions import RasaException import tensorflow as tf +import os -TF_HUB_MODULE_URL = ( +# URL to the old remote location of the model which +# users might use. The model is no longer hosted here. +ORIGINAL_TF_HUB_MODULE_URL = ( "https://github.com/PolyAI-LDN/polyai-models/releases/download/v1.0/model.tar.gz" ) +# Warning: This URL is only intended for running pytests on ConveRT +# related components. This URL should not be allowed to be used by the user. +RESTRICTED_ACCESS_URL = "https://storage.googleapis.com/continuous-integration-model-storage/convert_tf2.tar.gz" + class ConveRTTokenizer(WhitespaceTokenizer): """Tokenizer using ConveRT model. + Loads the ConveRT(https://github.com/PolyAI-LDN/polyai-models#convert) model from TFHub and computes sub-word tokens for dense featurizable attributes of each message object. @@ -30,25 +40,129 @@ class ConveRTTokenizer(WhitespaceTokenizer): "intent_split_symbol": "_", # Regular expression to detect tokens "token_pattern": None, - # Remote URL of hosted model - "model_url": TF_HUB_MODULE_URL, + # Remote URL/Local path to model files + "model_url": None, } def __init__(self, component_config: Dict[Text, Any] = None) -> None: - """Construct a new tokenizer using the WhitespaceTokenizer framework.""" + """Construct a new tokenizer using the WhitespaceTokenizer framework. + Args: + component_config: User configuration for the component + """ super().__init__(component_config) - self.model_url = self.component_config.get("model_url", TF_HUB_MODULE_URL) + self.model_url = self._get_validated_model_url() self.module = train_utils.load_tf_hub_model(self.model_url) self.tokenize_signature = self.module.signatures["tokenize"] + @staticmethod + def _validate_model_files_exist(model_directory: Text) -> None: + """Check if essential model files exist inside the model_directory. + + Args: + model_directory: Directory to investigate + """ + files_to_check = [ + os.path.join(model_directory, "saved_model.pb"), + os.path.join(model_directory, "variables/variables.index"), + os.path.join(model_directory, "variables/variables.data-00001-of-00002"), + os.path.join(model_directory, "variables/variables.data-00000-of-00002"), + ] + + for file_path in files_to_check: + if not os.path.exists(file_path): + raise RasaException( + f"""File {file_path} does not exist. + Re-check the files inside the directory {model_directory}. + It should contain the following model + files - [{", ".join(files_to_check)}]""" + ) + + def _get_validated_model_url(self) -> Text: + """Validates the specified `model_url` parameter. + + The `model_url` parameter cannot be left empty. It can either + be set to a remote URL where the model is hosted or it can be + a path to a local directory. + + Returns: + Validated path to model + """ + model_url = self.component_config.get("model_url", None) + + if not model_url: + raise RasaException( + f"""Parameter "model_url" was not specified in the configuration + of "{ConveRTTokenizer.__name__}". + You can either use a community hosted URL of the model + or if you have a local copy of the model, pass the + path to the directory containing the model files.""" + ) + + if model_url == ORIGINAL_TF_HUB_MODULE_URL: + # Can't use the originally hosted URL + raise RasaException( + f"""Parameter "model_url" of "{ConveRTTokenizer.__name__}" was + set to "{model_url}" which does not contain the model any longer. + You can either use a community hosted URL or if you have a + local copy of the model, pass the path to the directory + containing the model files.""" + ) + + if model_url == RESTRICTED_ACCESS_URL: + # Can't use the URL that is reserved for tests only + raise RasaException( + f"""Parameter "model_url" of "{ConveRTTokenizer.__name__}" was + set to "{model_url}" which is strictly reserved for pytests of Rasa Open Source only. + Due to licensing issues you are not allowed to use the model from this URL. + You can either use a community hosted URL or if you have a + local copy of the model, pass the path to the directory + containing the model files.""" + ) + + if os.path.isfile(model_url): + # Definitely invalid since the specified path should be a directory + raise RasaException( + f"""Parameter "model_url" of "{ConveRTTokenizer.__name__}" was + set to the path of a file which is invalid. You + can either use a community hosted URL or if you have a + local copy of the model, pass the path to the directory + containing the model files.""" + ) + + if rasa.nlu.utils.is_url(model_url): + return model_url + + if os.path.isdir(model_url): + # Looks like a local directory. Inspect the directory + # to see if model files exist. + self._validate_model_files_exist(model_url) + # Convert the path to an absolute one since + # TFHUB doesn't like relative paths + return os.path.abspath(model_url) + + raise RasaException( + f"""{model_url} is neither a valid remote URL nor a local directory. + You can either use a community hosted URL or if you have a + local copy of the model, pass the path to + the directory containing the model files.""" + ) + @classmethod def cache_key( cls, component_meta: Dict[Text, Any], model_metadata: Metadata ) -> Optional[Text]: + """Cache the component for future use. + + Args: + component_meta: configuration for the component. + model_metadata: configuration for the whole pipeline. + + Returns: key of the cache for future retrievals. + """ _config = common.update_existing_keys(cls.defaults, component_meta) return f"{cls.name}-{get_dict_hash(_config)}" diff --git a/rasa/nlu/utils/__init__.py b/rasa/nlu/utils/__init__.py index b1d00249b613..e68a3839c675 100644 --- a/rasa/nlu/utils/__init__.py +++ b/rasa/nlu/utils/__init__.py @@ -47,11 +47,24 @@ def is_model_dir(model_dir: Text) -> bool: def is_url(resource_name: Text) -> bool: - """Return True if string is an http, ftp, or file URL path. - - This implementation is the same as the one used by matplotlib""" - - URL_REGEX = re.compile(r"http://|https://|ftp://|file://|file:\\") + """Check whether the url specified is a well formed one. + + Regex adapted from https://stackoverflow.com/a/7160778/3001665 + + Args: + resource_name: Remote URL to validate + + Returns: `True` if valid, otherwise `False`. + """ + URL_REGEX = re.compile( + r"^(?:http|ftp|file)s?://" # http:// or https:// or file:// + r"(?:(?:[A-Z0-9](?:[A-Z0-9-]{0,61}[A-Z0-9])?\.)+(?:[A-Z]{2,6}\.?|[A-Z0-9-]{2,}\.?)|" # domain + r"localhost|" # localhost + r"\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})" # or ip + r"(?::\d+)?" # optional port + r"(?:/?|[/?]\S+)$", + re.IGNORECASE, + ) return URL_REGEX.match(resource_name) is not None diff --git a/rasa/version.py b/rasa/version.py index 58652243e3f6..60eccf632a5a 100644 --- a/rasa/version.py +++ b/rasa/version.py @@ -1,3 +1,3 @@ # this file will automatically be changed, # do not add anything but the version number here! -__version__ = "2.0.2" +__version__ = "2.0.3" diff --git a/tests/nlu/featurizers/test_convert_featurizer.py b/tests/nlu/featurizers/test_convert_featurizer.py index a2f170b9fe57..e4b90d5d1347 100644 --- a/tests/nlu/featurizers/test_convert_featurizer.py +++ b/tests/nlu/featurizers/test_convert_featurizer.py @@ -1,7 +1,12 @@ import numpy as np import pytest +from typing import Text +from _pytest.monkeypatch import MonkeyPatch -from rasa.nlu.tokenizers.convert_tokenizer import ConveRTTokenizer +from rasa.nlu.tokenizers.convert_tokenizer import ( + ConveRTTokenizer, + RESTRICTED_ACCESS_URL, +) from rasa.shared.nlu.training_data.training_data import TrainingData from rasa.shared.nlu.training_data.message import Message from rasa.nlu.constants import TOKENS_NAMES @@ -10,13 +15,15 @@ from rasa.nlu.featurizers.dense_featurizer.convert_featurizer import ConveRTFeaturizer -# TODO -# skip tests as the ConveRT model is not publicly available anymore (see https://github.com/RasaHQ/rasa/issues/6806) +@pytest.mark.skip_on_windows +def test_convert_featurizer_process(component_builder, monkeypatch: MonkeyPatch): + monkeypatch.setattr( + ConveRTTokenizer, "_get_validated_model_url", lambda x: RESTRICTED_ACCESS_URL + ) -@pytest.mark.skip -def test_convert_featurizer_process(component_builder): - tokenizer = component_builder.create_component_from_class(ConveRTTokenizer) + component_config = {"name": "ConveRTTokenizer", "model_url": RESTRICTED_ACCESS_URL} + tokenizer = ConveRTTokenizer(component_config) featurizer = component_builder.create_component_from_class(ConveRTFeaturizer) sentence = "Hey how are you today ?" @@ -41,9 +48,14 @@ def test_convert_featurizer_process(component_builder): assert np.allclose(sent_vecs[-1][:5], expected_cls, atol=1e-5) -@pytest.mark.skip -def test_convert_featurizer_train(component_builder): - tokenizer = component_builder.create_component_from_class(ConveRTTokenizer) +@pytest.mark.skip_on_windows +def test_convert_featurizer_train(component_builder, monkeypatch: MonkeyPatch): + + monkeypatch.setattr( + ConveRTTokenizer, "_get_validated_model_url", lambda x: RESTRICTED_ACCESS_URL + ) + component_config = {"name": "ConveRTTokenizer", "model_url": RESTRICTED_ACCESS_URL} + tokenizer = ConveRTTokenizer(component_config) featurizer = component_builder.create_component_from_class(ConveRTFeaturizer) sentence = "Hey how are you today ?" @@ -88,6 +100,7 @@ def test_convert_featurizer_train(component_builder): assert sent_vecs is None +@pytest.mark.skip_on_windows @pytest.mark.parametrize( "sentence, expected_text", [ @@ -98,9 +111,15 @@ def test_convert_featurizer_train(component_builder): ("ńöñàśçií", "ńöñàśçií"), ], ) -@pytest.mark.skip -def test_convert_featurizer_tokens_to_text(component_builder, sentence, expected_text): - tokenizer = component_builder.create_component_from_class(ConveRTTokenizer) +def test_convert_featurizer_tokens_to_text( + sentence: Text, expected_text: Text, monkeypatch: MonkeyPatch +): + + monkeypatch.setattr( + ConveRTTokenizer, "_get_validated_model_url", lambda x: RESTRICTED_ACCESS_URL + ) + component_config = {"name": "ConveRTTokenizer", "model_url": RESTRICTED_ACCESS_URL} + tokenizer = ConveRTTokenizer(component_config) tokens = tokenizer.tokenize(Message(data={TEXT: sentence}), attribute=TEXT) actual_text = ConveRTFeaturizer._tokens_to_text([tokens])[0] diff --git a/tests/nlu/test_utils.py b/tests/nlu/test_utils.py index cbb7928ecf09..ec8258146606 100644 --- a/tests/nlu/test_utils.py +++ b/tests/nlu/test_utils.py @@ -4,6 +4,7 @@ import pytest import tempfile import shutil +from typing import Text from rasa.shared.exceptions import RasaException import rasa.shared.nlu.training_data.message @@ -103,6 +104,19 @@ def test_remove_model_invalid(empty_model_dir): os.remove(test_file_path) -def test_is_url(): - assert not utils.is_url("./some/file/path") - assert utils.is_url("https://rasa.com/") +@pytest.mark.parametrize( + "url, result", + [ + ("a/b/c", False), + ("a", False), + ("https://192.168.1.1", True), + ("http://192.168.1.1", True), + ("https://google.com", True), + ("https://www.google.com", True), + ("http://google.com", True), + ("http://www.google.com", True), + ("http://a/b/c", False), + ], +) +def test_is_url(url: Text, result: bool): + assert result == utils.is_url(url) diff --git a/tests/nlu/tokenizers/test_convert_tokenizer.py b/tests/nlu/tokenizers/test_convert_tokenizer.py index e53bb7ecd8e0..ca2770cae6b9 100644 --- a/tests/nlu/tokenizers/test_convert_tokenizer.py +++ b/tests/nlu/tokenizers/test_convert_tokenizer.py @@ -1,15 +1,22 @@ import pytest +from typing import Text, List, Tuple, Optional +from pathlib import Path +import os +from _pytest.monkeypatch import MonkeyPatch from rasa.shared.nlu.training_data.training_data import TrainingData from rasa.shared.nlu.training_data.message import Message from rasa.nlu.constants import TOKENS_NAMES, NUMBER_OF_SUB_TOKENS from rasa.shared.nlu.constants import TEXT, INTENT -from rasa.nlu.tokenizers.convert_tokenizer import ConveRTTokenizer - -# TODO -# skip tests as the ConveRT model is not publicly available anymore (see https://github.com/RasaHQ/rasa/issues/6806) +from rasa.nlu.tokenizers.convert_tokenizer import ( + ConveRTTokenizer, + RESTRICTED_ACCESS_URL, + ORIGINAL_TF_HUB_MODULE_URL, +) +from rasa.exceptions import RasaException +@pytest.mark.skip_on_windows @pytest.mark.parametrize( "text, expected_tokens, expected_indices", [ @@ -25,19 +32,28 @@ ("ńöñàśçií", ["ńöñàśçií"], [(0, 8)]), ], ) -@pytest.mark.skip def test_convert_tokenizer_edge_cases( - component_builder, text, expected_tokens, expected_indices + text: Text, + expected_tokens: List[Text], + expected_indices: List[Tuple[int]], + monkeypatch: MonkeyPatch, ): - tk = component_builder.create_component_from_class(ConveRTTokenizer) - tokens = tk.tokenize(Message(data={TEXT: text}), attribute=TEXT) + monkeypatch.setattr( + ConveRTTokenizer, "_get_validated_model_url", lambda x: RESTRICTED_ACCESS_URL + ) + + component_config = {"name": "ConveRTTokenizer", "model_url": RESTRICTED_ACCESS_URL} + tokenizer = ConveRTTokenizer(component_config) + + tokens = tokenizer.tokenize(Message(data={TEXT: text}), attribute=TEXT) assert [t.text for t in tokens] == expected_tokens assert [t.start for t in tokens] == [i[0] for i in expected_indices] assert [t.end for t in tokens] == [i[1] for i in expected_indices] +@pytest.mark.skip_on_windows @pytest.mark.parametrize( "text, expected_tokens", [ @@ -45,35 +61,109 @@ def test_convert_tokenizer_edge_cases( ("Forecast for LUNCH", ["Forecast for LUNCH"]), ], ) -@pytest.mark.skip -def test_custom_intent_symbol(component_builder, text, expected_tokens): - tk = component_builder.create_component_from_class( - ConveRTTokenizer, intent_tokenization_flag=True, intent_split_symbol="+" +def test_custom_intent_symbol( + text: Text, expected_tokens: List[Text], monkeypatch: MonkeyPatch +): + + monkeypatch.setattr( + ConveRTTokenizer, "_get_validated_model_url", lambda x: RESTRICTED_ACCESS_URL ) + component_config = { + "name": "ConveRTTokenizer", + "model_url": RESTRICTED_ACCESS_URL, + "intent_tokenization": True, + "intent_split_symbol": "+", + } + + tokenizer = ConveRTTokenizer(component_config) + message = Message(data={TEXT: text}) message.set(INTENT, text) - tk.train(TrainingData([message])) + tokenizer.train(TrainingData([message])) assert [t.text for t in message.get(TOKENS_NAMES[INTENT])] == expected_tokens +@pytest.mark.skip_on_windows @pytest.mark.parametrize( "text, expected_number_of_sub_tokens", [("Aarhus is a city", [2, 1, 1, 1]), ("sentence embeddings", [1, 3])], ) -@pytest.mark.skip def test_convert_tokenizer_number_of_sub_tokens( - component_builder, text, expected_number_of_sub_tokens + text: Text, expected_number_of_sub_tokens: List[int], monkeypatch: MonkeyPatch ): - tk = component_builder.create_component_from_class(ConveRTTokenizer) + monkeypatch.setattr( + ConveRTTokenizer, "_get_validated_model_url", lambda x: RESTRICTED_ACCESS_URL + ) + component_config = {"name": "ConveRTTokenizer", "model_url": RESTRICTED_ACCESS_URL} + tokenizer = ConveRTTokenizer(component_config) message = Message(data={TEXT: text}) message.set(INTENT, text) - tk.train(TrainingData([message])) + tokenizer.train(TrainingData([message])) assert [ t.get(NUMBER_OF_SUB_TOKENS) for t in message.get(TOKENS_NAMES[TEXT]) ] == expected_number_of_sub_tokens + + +@pytest.mark.skip_on_windows +@pytest.mark.parametrize( + "model_url, exception_phrase", + [ + (ORIGINAL_TF_HUB_MODULE_URL, "which does not contain the model any longer"), + ( + RESTRICTED_ACCESS_URL, + "which is strictly reserved for pytests of Rasa Open Source only", + ), + (None, """"model_url" was not specified in the configuration"""), + ("", """"model_url" was not specified in the configuration"""), + ], +) +def test_raise_invalid_urls(model_url: Optional[Text], exception_phrase: Text): + + component_config = {"name": "ConveRTTokenizer", "model_url": model_url} + with pytest.raises(RasaException) as excinfo: + _ = ConveRTTokenizer(component_config) + + assert exception_phrase in str(excinfo.value) + + +@pytest.mark.skip_on_windows +def test_raise_wrong_model_directory(tmp_path: Path): + + component_config = {"name": "ConveRTTokenizer", "model_url": str(tmp_path)} + + with pytest.raises(RasaException) as excinfo: + _ = ConveRTTokenizer(component_config) + + assert "Re-check the files inside the directory" in str(excinfo.value) + + +@pytest.mark.skip_on_windows +def test_raise_wrong_model_file(tmp_path: Path): + + # create a dummy file + temp_file = os.path.join(tmp_path, "saved_model.pb") + f = open(temp_file, "wb") + f.close() + component_config = {"name": "ConveRTTokenizer", "model_url": temp_file} + + with pytest.raises(RasaException) as excinfo: + _ = ConveRTTokenizer(component_config) + + assert "set to the path of a file which is invalid" in str(excinfo.value) + + +@pytest.mark.skip_on_windows +def test_raise_invalid_path(): + + component_config = {"name": "ConveRTTokenizer", "model_url": "saved_model.pb"} + + with pytest.raises(RasaException) as excinfo: + _ = ConveRTTokenizer(component_config) + + assert "neither a valid remote URL nor a local directory" in str(excinfo.value)