spaCy 3.0

Makefile

@@ -131,8 +131,6 @@ prepare-spacy:
 	poetry install -E spacy
 	poetry run python -m spacy download en_core_web_md
 	poetry run python -m spacy download de_core_news_sm
-	poetry run python -m spacy link en_core_web_md en --force
-	poetry run python -m spacy link de_core_news_sm de --force
 
 prepare-mitie:
 	wget --progress=dot:giga -N -P data/ https://github.com/mit-nlp/MITIE/releases/download/v0.4/MITIE-models-v0.2.tar.bz2

changelog/7869.feature.md

@@ -0,0 +1,41 @@
+Upgraded Rasa to be compatible with spaCy 3.0. 
+
+This means that we can support more features for more languages but there are also a few changes. 
+
+SpaCy 3.0 deprecated the `spacy link <language model>` command so that means that from now on [the 
+full model name](https://spacy.io/models) needs to be used in the `config.yml` file. 
+
+**Before** 
+
+Before you could run `spacy link en en_core_web_md` and then we would be able
+to pick up the correct model from the `language` parameter. 
+
+```yaml
+language: en
+
+pipeline:
+   - name: SpacyNLP
+```
+
+**Now** 
+
+This behavior will be deprecated and instead you'll want to be explicit in `config.yml`.
+
+```yaml
+language: en
+
+pipeline:
+   - name: SpacyNLP
+     model: en_core_web_md
+```
+
+**Fallback** 
+
+To make the transition easier, Rasa will try to fall back to a medium spaCy model when-ever 
+a compatible language is configured for the entire pipeline in `config.yml` even if you don't
+specify a `model`. This fallback behavior is temporary and will be deprecated in spaCy 3.0.  
+
+We've updated our docs to reflect these changes. All examples now show a direct link to the 
+correct spaCy model. We've also added a warning to the [SpaCyNLP](components.mdx#spacynlp) 
+docs that explains the fallback behavior. 
+

data/test_config/config_pretrained_embeddings_spacy_de.yml

@@ -2,6 +2,7 @@ language: "de"
 
 pipeline:
   - name: SpacyNLP
+    model: "de_core_news_sm"
   - name: SpacyTokenizer
   - name: SpacyFeaturizer
   - name: RegexFeaturizer

data/test_config/config_spacy_entity_extractor.yml

@@ -1,6 +1,7 @@
 language: en
 pipeline:
     - name: "SpacyNLP"
+      model: "en_core_web_md"
     - name: "SpacyTokenizer"
     - name: "SpacyFeaturizer"
     - name: "RegexFeaturizer"

docker/Dockerfile.pretrained_embeddings_spacy_de

@@ -23,9 +23,7 @@ RUN . /opt/venv/bin/activate && poetry build -f wheel -n && \
 # make sure we use the virtualenv
 ENV PATH="/opt/venv/bin:$PATH"
 
-# spacy link
-RUN python -m spacy download de_core_news_sm && \
-    python -m spacy link de_core_news_sm de
+RUN python -m spacy download de_core_news_sm
 
 # start a new build stage
 FROM ${IMAGE_BASE_NAME}:base-${BASE_IMAGE_HASH} as runner

docker/Dockerfile.pretrained_embeddings_spacy_en

@@ -23,9 +23,7 @@ RUN . /opt/venv/bin/activate && poetry build -f wheel -n && \
 # make sure we use the virtualenv
 ENV PATH="/opt/venv/bin:$PATH"
 
-# spacy link
-RUN python -m spacy download en_core_web_md && \
-    python -m spacy link en_core_web_md en
+RUN python -m spacy download en_core_web_md
 
 # start a new build stage
 FROM ${IMAGE_BASE_NAME}:base-${BASE_IMAGE_HASH} as runner

docker/configs/config_pretrained_embeddings_spacy_de.yml

@@ -2,6 +2,7 @@ language: "de"
 
 pipeline:
   - name: SpacyNLP
+    model: "de_core_news_md"
   - name: SpacyTokenizer
   - name: SpacyFeaturizer
   - name: RegexFeaturizer

docker/configs/config_pretrained_embeddings_spacy_en.yml

@@ -2,6 +2,7 @@ language: "en"
 
 pipeline:
   - name: SpacyNLP
+    model: "en_core_web_md"
   - name: SpacyTokenizer
   - name: SpacyFeaturizer
   - name: RegexFeaturizer

docker/configs/config_pretrained_embeddings_spacy_en_duckling.yml

@@ -2,6 +2,7 @@ language: "en"
 
 pipeline:
   - name: SpacyNLP
+    model: "en_core_web_md"
   - name: SpacyTokenizer
   - name: SpacyFeaturizer
   - name: RegexFeaturizer

docs/docs/components.mdx

@@ -106,11 +106,8 @@ word vectors in your pipeline.
 
 * **Configuration**
 
-  You need to specify the language model to use.
-  By default the language configured in the pipeline will be used as the language model name.
-  If the spaCy model to be used has a name that is different from the language tag (`"en"`, `"de"`, etc.),
-  the model name can be specified using the configuration variable `model`.
-  The name will be passed to `spacy.load(name)`.
+  You need to specify the language model to use. The name will be passed to `spacy.load(name)`.
+  You can find more information on the available models on the [spaCy documentation](https://spacy.io/usage/models).
 
   ```yaml-rasa
   pipeline:
@@ -130,12 +127,12 @@ word vectors in your pipeline.
   [installing SpaCy](./installation.mdx#dependencies-for-spacy).
 
   In addition to SpaCy's pretrained language models, you can also use this component to
-  load fastText vectors, which are available for [hundreds of languages](https://github.com/facebookresearch/fastText/blob/master/docs/crawl-vectors.md).
-  If you want to incorporate a custom model you've found into spaCy, check out their page on
-  [adding languages](https://spacy.io/usage/adding-languages/). As described in the documentation, you need to
-  register your language model and link it to the language identifier, which will allow Rasa to load and use your new language
-  by passing in your language identifier as the `language` option.
+  attach spaCy models that you've trained yourself.
 
+:::caution Fallback
+Rasa Open Source will try to fallback to a common model on your behalf if you don't pass a `model` setting. This is a
+temporary feature we've introduced as part of the spaCy 3.0 migration but the fallback will be removed in Rasa Open Source 3.0.0.
+:::
 
 ### HFTransformersNLP
 

docs/docs/installation.mdx

@@ -223,12 +223,11 @@ You can install it with the following commands:
 ```bash
 pip3 install rasa[spacy]
 python3 -m spacy download en_core_web_md
-python3 -m spacy link en_core_web_md en
 ```
 
 This will install Rasa Open Source as well as spaCy and its language model
-for the English language. We recommend using at least the
-“medium” sized models (`_md`) instead of the spaCy's
+for the English language, but there are many other languages availabe too.
+We recommend using at least the "medium" sized models (`_md`) instead of the spaCy's
 default small `en_core_web_sm` model. Small models require less
 memory to run, but will somewhat reduce intent classification performance.
 

docs/docs/migration-guide.mdx

@@ -10,6 +10,51 @@ description: |
 This page contains information about changes between major versions and
 how you can migrate from one version to another.
 
+## Rasa 2.4 to 2.5
+
+### Machine Learning Components
+
+#### SpaCy 3.0
+
+Rasa now supports spaCy 3.0. This means that we can support more features for more
+languages but this also introduced a breaking change. SpaCy 3.0 deprecated the
+`spacy link <language model>` command so that means that from now on
+[the full model name](https://spacy.io/models) needs to be used in the `config.yml` file.
+
+**Before**
+
+Before you could run `spacy link en en_core_web_md` and then we would be able
+to pick up the correct model from the `language` parameter.
+
+```yaml
+language: en
+
+pipeline:
+   - name: SpacyNLP
+```
+
+**Now**
+
+This behavior will be deprecated and instead you'll want to be explicit in `config.yml`.
+
+```yaml
+language: en
+
+pipeline:
+   - name: SpacyNLP
+     model: en_core_web_md
+```
+
+**Fallback**
+
+To make the transition easier, Rasa will try to fall back to a medium spaCy model when-ever
+a compatible language is configured for the entire pipeline in `config.yml` even if you don't
+specify a `model`. This fallback behavior is temporary and will be deprecated in Rasa Open Source 3.0.0.
+
+We've updated our docs to reflect these changes. All examples now show a direct link to the
+correct spaCy model. We've also added a warning to the [SpaCyNLP](components.mdx#spacynlp)
+docs that explains the fallback behavior.
+
 ## Rasa 2.3.3 to Rasa 2.3.4
 
 :::caution

docs/docs/reaching-out-to-user.mdx

@@ -283,6 +283,7 @@ names in your training data, since Spacy has a `PERSON` dimension:
 pipeline:
 # other components
 - name: SpacyNLP
+  model: "en_core_web_md"
 - name: SpacyEntityExtractor
   dimensions: ["PERSON"]
 ```

docs/docs/tuning-your-model.mdx

@@ -41,8 +41,7 @@ we recommend the following pipeline:
 ```
 
 It uses the [SpacyFeaturizer](./components.mdx#spacyfeaturizer), which provides 
-pre-trained word embeddings from either GloVe or fastText in many different languages
-(see [Language Models](./components.mdx#language-models)).
+pre-trained word embeddings (see [Language Models](./components.mdx#language-models)).
 
 If you don't use any pre-trained word embeddings inside your pipeline, you are not bound to a specific language
 and can train your model to be more domain specific.

examples/moodbot/config.yml

@@ -2,6 +2,7 @@ language: en
 
 pipeline:
   - name: "SpacyNLP"
+    model: "en_core_web_md"
   - name: "SpacyTokenizer"
   - name: "SpacyFeaturizer"
   - name: "DIETClassifier"

examples/reminderbot/config.yml

@@ -11,7 +11,7 @@ pipeline:
 - name: "DIETClassifier"
   epochs: 100
 - name: SpacyNLP
-  model: "en"
+  model: "en_core_web_md"
 - name: SpacyEntityExtractor
   dimensions: ["PERSON"]
 - name: "EntitySynonymMapper"