DIET breaks with empty spaCy model

[koaning]

In spaCy it is possible to create a model to detect entities, without the model outputting any vectors. If the model is then included as a featurizer in the pipeline by accident then it currently breaks the DIET classifier.

python 3.7
rasa==1.9.6
rasa-sdk==1.9.0
macos

Issue:

basic-rules.jsonl

{"label":"PYTHON","pattern":[{"LOWER":"python"}]}
{"label":"PYTHON","pattern":[{"LOWER":{"REGEX":"(python\\d+\\.?\\d*.?\\d*)"}}]}
{"label":"PYTHON","pattern":[{"LOWER":"python"}, {"TEXT":{"REGEX":"(\\d+\\.?\\d*.?\\d*)"}}]}
{"label":"JS","pattern":[{"LOWER": {"IN": ["node", "nodejs", "js", "javascript"]}}]}
{"label":"JS","pattern":[{"LOWER": {"IN": ["node", "nodejs", "js", "javascript"]}}, {"TEXT": {"REGEX": "(\\d+\\.?\\d*.?\\d*)"}}]}

mkmodel.py

import argparse 

import spacy
from spacy.lang.en import English
from spacy.pipeline import EntityRuler
from spacy import displacy


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description='Process some integers.')
    parser.add_argument('filepath', type=str, help='Filepath to .jsonl file containing spaCy rules')
    args = parser.parse_args()

    # note that we could have also used `en_core_web_md` as a starting point
    # or another pretrained language model, like Dutch `nl_core_news_sm`
    # we're keeping it minimal for now though
    nlp = English()

    # create a new rule based NER detector loading in settings from disk
    print(f"Will try to open file at {args.filepath}")
    ruler = EntityRuler(nlp).from_disk(args.filepath)
    print(f"File opened. Will now create model.")
    # add the detector to the model
    nlp.add_pipe(ruler, name="proglang-detector")

    # save the model to disk, this is now also the model name
    # you could load it now via `spacy.load("spacy-trained-model")`
    nlp.to_disk("spacy-trained-model")
    print(f"spaCy model saved in `spacy-trained-model` folder")

config.yml that does cause the error

language: en

pipeline:
- name: SpacyNLP
  model: "spacy-trained-model"
- name: SpacyTokenizer
- name: SpacyEntityExtractor
  dimensions: ["PYTHON", "JS"]
- name: SpacyFeaturizer
  pooling: max
- name: CountVectorsFeaturizer
  analyzer: char_wb
  min_ngram: 1
  max_ngram: 2
- name: DIETClassifier
  epochs: 10
  entity_recognition: False

policies:
  - name: MemoizationPolicy
  - name: KerasPolicy
  - name: MappingPolicy

config.yml that does not cause error

language: en

pipeline:
- name: SpacyNLP
  model: "spacy-trained-model"
- name: SpacyTokenizer
- name: SpacyEntityExtractor
  dimensions: ["PYTHON", "JS"]
- name: CountVectorsFeaturizer
  analyzer: char_wb
  min_ngram: 1
  max_ngram: 2
- name: DIETClassifier
  epochs: 10
  entity_recognition: False

policies:
  - name: MemoizationPolicy
  - name: KerasPolicy
  - name: MappingPolicy

Error (including full traceback):

I first run the command to generate the spaCy model.

python mkmodel.py spaCy-rules/more-rules.jsonl;

And then it all breaks after passing a command into rasa.

> rasa train; rasa shell nlu
Nothing changed. You can use the old model stored at '/Users/vincent/Development/rasa-spacy-integration/models/20200415-161323.tar.gz'.
2020-04-15 16:15:26 INFO     rasa.nlu.components  - Added 'SpacyNLP' to component cache. Key 'SpacyNLP-spacy-trained-model'.
NLU model loaded. Type a message and press enter to parse it.
Next message:
yo
Traceback (most recent call last):
  File "/Users/vincent/Development/rasa-spacy-integration/venv/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 2227, in _convert_inputs_to_signature
    expand_composites=True)
  File "/Users/vincent/Development/rasa-spacy-integration/venv/lib/python3.6/site-packages/tensorflow_core/python/util/nest.py", line 883, in flatten_up_to
    expand_composites=expand_composites)
  File "/Users/vincent/Development/rasa-spacy-integration/venv/lib/python3.6/site-packages/tensorflow_core/python/util/nest.py", line 802, in assert_shallow_structure
    expand_composites=expand_composites)
  File "/Users/vincent/Development/rasa-spacy-integration/venv/lib/python3.6/site-packages/tensorflow_core/python/util/nest.py", line 786, in assert_shallow_structure
    input_length=len(input_tree), shallow_length=len(shallow_tree)))
ValueError: The two structures don't have the same sequence length. Input structure has length 4, while shallow structure has length 5.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/vincent/Development/rasa-spacy-integration/venv/bin/rasa", line 11, in <module>
    sys.exit(main())
  File "/Users/vincent/Development/rasa-spacy-integration/venv/lib/python3.6/site-packages/rasa/__main__.py", line 91, in main
    cmdline_arguments.func(cmdline_arguments)
  File "/Users/vincent/Development/rasa-spacy-integration/venv/lib/python3.6/site-packages/rasa/cli/shell.py", line 83, in shell_nlu
    rasa.nlu.run.run_cmdline(nlu_model)
  File "/Users/vincent/Development/rasa-spacy-integration/venv/lib/python3.6/site-packages/rasa/nlu/run.py", line 31, in run_cmdline
    result = interpreter.parse(message)
  File "/Users/vincent/Development/rasa-spacy-integration/venv/lib/python3.6/site-packages/rasa/nlu/model.py", line 376, in parse
    component.process(message, **self.context)
  File "/Users/vincent/Development/rasa-spacy-integration/venv/lib/python3.6/site-packages/rasa/nlu/classifiers/diet_classifier.py", line 783, in process
    out = self._predict(message)
  File "/Users/vincent/Development/rasa-spacy-integration/venv/lib/python3.6/site-packages/rasa/nlu/classifiers/diet_classifier.py", line 671, in _predict
    return self.model.predict(model_data)
  File "/Users/vincent/Development/rasa-spacy-integration/venv/lib/python3.6/site-packages/rasa/utils/tensorflow/models.py", line 213, in predict
    return self._predict_function(batch_in)
  File "/Users/vincent/Development/rasa-spacy-integration/venv/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py", line 568, in __call__
    result = self._call(*args, **kwds)
  File "/Users/vincent/Development/rasa-spacy-integration/venv/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py", line 606, in _call
    results = self._stateful_fn(*args, **kwds)
  File "/Users/vincent/Development/rasa-spacy-integration/venv/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 2362, in __call__
    graph_function, args, kwargs = self._maybe_define_function(args, kwargs)
  File "/Users/vincent/Development/rasa-spacy-integration/venv/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 2661, in _maybe_define_function
    *args, **kwargs)
  File "/Users/vincent/Development/rasa-spacy-integration/venv/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 2185, in canonicalize_function_inputs
    self._flat_input_signature)
  File "/Users/vincent/Development/rasa-spacy-integration/venv/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 2231, in _convert_inputs_to_signature
    format_error_message(inputs, input_signature))
ValueError: Structure of Python function inputs does not match input_signature:
  inputs: (
    (array([[  0,   0,   0],
       [  0,   0,  35],
       [  0,   0, 228],
       [  0,   0, 229],
       [  0,   0, 338],
       [  0,   1,   0],
       [  0,   1,  35],
       [  0,   1, 228],
       [  0,   1, 229],
       [  0,   1, 338]]), array([2., 1., 1., 1., 1., 2., 1., 1., 1., 1.], dtype=float32), array([  1,   2, 346]), array([2.], dtype=float32)))
  input_signature: (
    (TensorSpec(shape=(None, 3), dtype=tf.int64, name=None), TensorSpec(shape=(None,), dtype=tf.float32, name=None), TensorSpec(shape=(3,), dtype=tf.int64, name=None), TensorSpec(shape=(None, None, 300), dtype=tf.float32, name=None), TensorSpec(shape=(None,), dtype=tf.float32, name=None)))

Now the interesting thing here is that this error goes away once we stop using English() and start using nlp.load("en_core_web_md") in mkmodel.py.

@tabergma we've discussed this over a zoom call, this indicates that the DIET classifier is probably picking up an empty element from the spaCyfeaturizer.

[koaning]

It also seems to work fine when using EmbeddingIntentClassifier. So the issue seems to be only happening on the DIETClassifier.