Exception: Not all required packages are installed. To use this pipeline, you need to install the missing dependencies. Please install sklearn_crfsuite, spacy

[Shivbaj]

Rasa Core version: rasa/rasa_core latest (docker image from docker hub)

Python version: Python 2.7.15

Operating system (windows, osx, ...): Linux, openshift docker container

Issue:

I have pulled latest docker image for rasa core from dockerhub. Once the image is pulled successfully , I have created a container. When I am trying to train my nlu model inside container using command :
python bot.py train-nlu , I am getting below exception :

python bot.py train-nlu

2018-06-14 05:33:59 INFO rasa_nlu.training_data.loading - Training data format of data/ChatBotTrainingData.json is rasa_nlu

/usr/local/lib/python2.7/site-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as
np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
Traceback (most recent call last):
File "bot.py", line 260, in
train_nlu()
File "bot.py", line 229, in train_nlu
trainer = Trainer(config.load("CHATBOT_Config.yml"))
File "/usr/local/lib/python2.7/site-packages/rasa_nlu/model.py", line 146, in init
components.validate_requirements(cfg.component_names)
File "/usr/local/lib/python2.7/site-packages/rasa_nlu/components.py", line 63, in validate_requirements
"Please install {}".format(", ".join(failed_imports)))
Exception: Not all required packages are installed. To use this pipeline, you need to install the missing dependencies. Please install sklearn_crfsuite, spacy

I cannot run pip command to install missing dependency inside the docker container, as due to security issues we don't have access to internet inside the container. Please suggest how can I over come this error

pip list contents :

functools32 3.2.3.post2
future 0.16.0
futures 3.2.0
gast 0.2.0
gevent 1.3.1
graphviz 0.8.3
greenlet 0.4.13
grpcio 1.12.0
h5py 2.7.1
html5lib 0.9999999
humanfriendly 4.12.1
hyperlink 18.0.0
idna 2.6
incremental 17.5.0
itsdangerous 0.24
Jinja2 2.10
jmespath 0.9.3
jsonpickle 0.9.6
jsonschema 2.6.0
Keras 2.1.6
kiwisolver 1.0.1
klein 17.10.0
Markdown 2.6.11
MarkupSafe 1.0
matplotlib 2.2.2
mattermostwrapper 2.1
mock 2.0.0
monotonic 1.5
networkx 2.1
numpy 1.14.3
packaging 17.1
pathlib 1.0.1
pbr 4.0.3
pip 10.0.1
protobuf 3.5.2.post1
PyJWT 1.6.3
pykwalify 1.6.0
pyparsing 2.2.0
python-dateutil 2.7.3
python-telegram-bot 10.1.0
pytz 2018.4
PyYAML 3.12
rasa-core 0.9.3 /app
rasa-nlu 0.12.3
redis 2.10.6
requests 2.18.4
ruamel.ordereddict 0.4.13
ruamel.yaml 0.15.37
s3transfer 0.1.13
scikit-learn 0.19.1
scipy 1.1.0
setuptools 39.1.0
simplejson 3.15.0
six 1.11.0
slackclient 1.2.1
subprocess32 3.5.1
tensorboard 1.8.0
tensorflow 1.8.0
termcolor 1.1.0
tqdm 4.23.3
twilio 6.14.0
Twisted 18.4.0
typing 3.6.4
tzlocal 1.5.1
urllib3 1.22
websocket-client 0.47.0
Werkzeug 0.14.1
wheel 0.31.0
zope.interface 4.5.0

configuration YML file contents

language: "en"

pipeline:

• name: "nlp_spacy"
• name: "tokenizer_spacy"
• name: "intent_entity_featurizer_regex"
• name: "intent_featurizer_spacy"
• name: "ner_crf"
• name: "ner_synonyms"
• name: "intent_classifier_sklearn"

[Shivbaj]

@tmbo Could you please share contents of rasa core docker image(Available in dockerhub) with me, so that I can modify it and test it out , if its work well we can use the same. Please suggest if you have some other plan

[akelad]

I think this should be fixed in the newer docker images. Can you check?

[Shivbaj]

@akelad The last updated, rasa core docker image available in docker hub is 2 months old. I will check again and confirm you back.

[Shivbaj]

root@####:/app/examples/restaurantbot# python bot.py train-nlu

2018-07-18 10:18:00 INFO rasa_nlu.training_data.loading - Training data format of data/franken_data.json is rasa_nlu
2018-07-18 10:18:01 INFO rasa_nlu.training_data.training_data - Training data stats:
- intent examples: 1977 (6 distinct intents)
- Found intents: 'deny', 'thankyou', 'request_info', 'affirm', 'greet', 'inform'
- entity examples: 1030 (5 distinct entities)
- found entities: 'info', 'cuisine', 'price', 'location', 'people'

/usr/local/lib/python2.7/site-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
Traceback (most recent call last):
File "bot.py", line 110, in
train_nlu()
File "bot.py", line 79, in train_nlu
trainer = Trainer(config.load("nlu_model_config.yml"))
File "/usr/local/lib/python2.7/site-packages/rasa_nlu/model.py", line 146, in init
components.validate_requirements(cfg.component_names)
File "/usr/local/lib/python2.7/site-packages/rasa_nlu/components.py", line 63, in validate_requirements
"Please install {}".format(", ".join(failed_imports)))
Exception: Not all required packages are installed. To use this pipeline, you need to install the missing dependencies. Please install sklearn_crfsuite, spacy

@akelad The exception still there for the rasa core image in docker hub

Image taken from docker hub : https://hub.docker.com/r/rasa/rasa_core/

[akelad]

Hmm let me look into why the docker image is so old on there. I guess for now just try building the docker image yourself from the repo?

[Shivbaj]

"docker build https://github.com/RasaHQ/rasa_core.git"

Just try building docker image from the repo, but getting some "Unknown instruction SHELL",

@akelad it would be really great if you can update the image in docker hub for rasa core. Thank you for response

I tried with custom docker image taken reference from rasa nlu docker file , just getting exception while downloading debian packages

[akelad]

I just created a PR that should push out a new docker image RasaHQ/rasa_core#778
But that's strange because I just ran docker build . on the repo and it built fine (after the change I made to the Dockerfile)

[akelad]

there should be a new one on here now: https://hub.docker.com/r/rasa/rasa_core/tags/

[Shivbaj]

Hi @akelad

I took latest rasa core image from docker hub and build it in openshift, the image is building up .But I could see the issue is still existing, I am getting same exception . Please check below snapshot . Please suggest .

Took image on saturday , But could see updated 20 hours ago, let me test one more time

[Shivbaj]

Hi @akelad

I checked with latest rasa core image from docker hub , but this time facing some different issue, not able to start rasa core container.

rasa_nlu.model.InvalidProjectError: Failed to load model metadata from '/app/nlu/metadata.json'. [Errno 2] No such file or directory: './nlu/metadata.json'

[akelad]

So actually the first issue you posted - you shouldn't train your rasa_nlu model inside the docker container. That model should be stored locally under ./nlu, that also explains the second error there, you probably don't have your nlu model stored there

[Shivbaj]

Yes you are right ,I will try this approach placing nlu model under ./nlu folder.
But if we train model in rasa nlu container and ship it to ./nlu folder of rasa core container. Will the rasa core container would be able to read the pre-trained model (trained using rasanlu container) present inside ./nlu folder ? I doubt on this, but will try and confirm you , thanks for quick response.

And one quick question, it would be much easier if we can train rasa_nlu model within rasa core container,sometimes the shipped model wont function properly, any thoughts on this?

[Shivbaj]

Hi @akelad
performed below steps :

1. Pretrained rasa_nlu model is placed under ./nlu folder
   2 Started the rasa core container with base image taken from docker hub.
   3 The pod is not getting started checked logs , got above exception (Not all required packages are installed)

[akelad]

This is a good point, all the nlu dependencies need to be installed, with e.g. pip install rasa_nlu[spacy] (see: http://rasa.com/docs/nlu/installation/). You can try doing this locally and building a docker image, and then do you mind creating a PR for this?

[Shivbaj]

@akelad I cant build docker image in my client's machine as we don't have "shell" installed as well as many privacy issues . It would be really great if you can fix it...... otherwise in few days I will try to setup things in my PC and raise PR.