Issues with docker-py and Ubuntu 14.04

[sdemura]

Description of Issue/Question

I have been trying to get docker-py in order to use dockerng modules and states, but I have run into many issues. This is a summary of what I have found.

Setup

1. Salt 2015.8.10, 2016.3.0, 2016.3.1
2. Ubuntu 14.04
3. Docker-py 1.7.2/1.8.1
4. Docker 1.11.2

Steps to Reproduce Issue

The salt minion requires docker-py python module in order to manage docker containers. This should be as simple as running:

python-pip:
  pkg.installed: []

docker-py: 
  pip.installed:    
    - name: docker-py>=1.4.0 # installs 1.8.1 by default
    - reload_modules: True
    - require:
      - pkg: python-pip

This works on CentOS 7 and Debian Jessie without problem. On Ubuntu 14.04, the state is successful, but once I go to spin up a container with dockerng, I get module import errors.

Based on suggetion here: #28036 and #26115 (comment)

I have created a workaround for Ubuntu 14.04 like so:

python-pip:
  pkg.purged: []

python-setuptools:
  pkg.installed: []
  - require:
    - pkg: python-pip

pip-install:
  cmd.run:
    - name: easy_install pip
    - require:
      - pkg: python-setuptools

docker-py:
  cmd.run:
    - name: pip install docker-py==1.7.2
    - reload_modules: True
    - require:
      - cmd: pip-install

It is also worth noting that docker-py 1.8.1 does not work on Ubuntu 14.04 using my workaround:

Using 1.8.1:

----------
          ID: registry
    Function: dockerng.running
      Result: False
     Comment: State 'dockerng.running' was not found in SLS 'docker.registry'
              Reason: 'dockerng' __virtual__ returned False: 'dockerng' __virtual__ returned False: Docker module could not get imported
     Started:
    Duration:
     Changes:

Versions Report

Minions + master:

root@saltminion1:~# salt --versions-report
Salt Version:
           Salt: 2016.3.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.3.0
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.0.1
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.3
            ZMQ: 4.0.4

System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 3.13.0-87-generic
         system: Linux
        version: Ubuntu 14.04 trusty

[Ch3LL]

@sdemura I believe the pip issue is an upstream issue.

For the docker 1.8.1 issue I am able to replicate this issue and I'm seeing this error in the minion debug logs:

AttributeError: 'module' object has no attribute 'connection'

Are you seeing the same error?

[Deshke]

docker-py is more or less always out of date and half the time not working, i switched to docker-compose as a wrapper and that's working

[adilnaimi]

I confirm, having the same issue on ubuntu 14.04

But on debian 8 is working without any problem.

I have two minions, one Debian8 and one Ubntu 14.04, the same sls it's working fine on Debian but it failed on Ubuntu

[0megam]

Updating python-urllib3 package to 1.15.1 on Ubuntu 14.04 helped me to overcome this issue.

[adilnaimi]

I have updated python-urllib3 package to 1.15.1 but unfortunately couldn't fix the issue, I'm still getting the following error:

----------
          ID: redis:3.2-alpine
    Function: dockerng.image_present
      Result: False
     Comment: State 'dockerng.image_present' was not found in SLS 'sentry-v88'
              Reason: 'dockerng' __virtual__ returned False: 'dockerng' __virtual__ returned False: Docker module could not get imported
     Started: 
    Duration: 
     Changes:   

My simple redis image state :

# pull latest image
redis:3.2-alpine:
  dockerng.image_present

pip packages installed on minion:

python-urllib3:
    pip.installed:
      - name: urllib3 == 1.15.1
      - require:
        - pkg: python-pip

docker-py:
    pip.installed:
      - name: docker-py >= 1.4.0
      - require:
        - pkg: python-pip
        - pip: python-urllib3

salt version:

sudo salt-minion --versions-report
Salt Version:
           Salt: 2016.3.3

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.5.3
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 0.9.1
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.2.3
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.0.1
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5

System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 3.13.0-91-generic
         system: Linux
        version: Ubuntu 14.04 trusty

[adilnaimi]

it's working for me after using this formula https://github.com/saltstack-formulas/docker-formula to install docker.

but I'm not sure what is the root cause of the issue

[0megam]

@adilr00t I did deb package update not npm.

[fxdgear]

+1 to this issue.

I have docker-py installed.

ssh-ing into the minion I can run the following:

$ sudo python
Python 2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from docker import Client
>>> cli = Client(base_url='unix://var/run/docker.sock')
>>> cli.images()
[{u'Created': 1474642111, u'Labels': 

versions:

$ pip freeze | grep docker
docker-py==1.10.3
docker-pycreds==0.2.1

versions-report

Salt Version:
           Salt: 2016.3.3

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 1.5
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 0.9.1
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.2.3
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.0.1
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5

System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 3.13.0-96-generic
         system: Linux
        version: Ubuntu 14.04 trusty

and I am using Docker CS engine 1.12

Client:
 Version:      1.12.1-cs1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   e7f4338
 Built:        Wed Sep 21 21:05:28 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.1-cs1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   e7f4338
 Built:        Wed Sep 21 21:05:28 2016
 OS/Arch:      linux/amd64

When I run sudo salt-call state.highstate I get this error:

ID: docker-py
    Function: pip.installed
      Result: False
     Comment: An importable pip module is required but could not be found on your system. This usually means that the system's pip package is not installed properly.
     Started: 16:27:11.926124
    Duration: 0.811 ms
     Changes: 

[fxdgear]

Note: when i manually ran sudo pip install --upgrade pip then ran sudo salt-call state.highstate things ran fine.

SO to clarify using

python-pip:
  pkg.installed

is not sufficent.

it MUST be the pip installed version of pip, not the apt version

Also the error message coming back to the master when running salt '*' state.highstate is different than the message coming from the minion when running salt-call state.highstate

[fxdgear]

Also note that running, from the minion, salt-call state.highstate works.

when running salt '*' state.highstate still does not work.

edit: also running salt '*' cmd.run 'salt-call state.highstate' also works.

[fxdgear]

update: restarting the salt-minion is also required.

After installing pip from pip and installing docker-py the minion needs to be restarted then my salt-states work as expected.

this seems like broken behavior.

[sdemura]

Agree with @fxdgear. Having to restart the salt-minion is required, and that's definitely not desirable.

[alem0lars-yr]

Are there any news on that ?

[Jenpatrick]

Also would like some news. Running Ubuntu 16.04.2 and Salt 2016.11.3 and still getting this error when trying to run:

 sudo salt MYMINION  dockercompose.up /data/dockerfiles/
MUMINION:
    'dockercompose' __virtual__ returned False: The dockercompose execution module not loaded: compose python library not available.

And yet...

root@MYMINION:/data/dockerfiles# pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
backports.ssl-match-hostname (3.5.0.1)
cached-property (1.3.0)
chardet (2.3.0)
colorama (0.3.7)
cryptography (1.2.3)
docker (2.2.0)
docker-compose (1.11.2)
docker-pycreds (0.2.1)
dockerpty (0.4.1)

Docker-compose 1.6.2 is installed and there in /usr/local/bin

[konstest]

+1

knikolaev@kos ~/PT/Docker/postgres $ pip freeze | grep docker
Could not parse requirement: -lxc
docker==2.5.1
docker-compose==1.16.1
docker-pycreds==0.2.1
dockerpty==0.4.1
knikolaev@kos ~/PT/Docker/postgres $ sudo salt-call dockercompose.get /home/knikolaev/PT/Docker/postgres
[ERROR   ] Failed to import module testinframod, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1333, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/usr/lib/python2.7/dist-packages/salt/modules/testinframod.py", line 297, in <module>
    _register_functions()
  File "/usr/lib/python2.7/dist-packages/salt/modules/testinframod.py", line 288, in _register_functions
    for module_ in modules.__all__:
AttributeError: 'module' object has no attribute '__all__'
'dockercompose' __virtual__ returned False: The dockercompose execution module not loaded: compose python library not available.
knikolaev@kos ~/PT/Docker/postgres $ 

Debian jessie

[Ch3LL]

Has anyone tired the most recent salt version?

[jessebye]

@Ch3LL just ran into this. Latest Salt (2017.7.2) on master and minion. Minion is Ubuntu 14.04. Here is the output:

ID: opsgenie_pip_module
    Function: pip.installed
        Name: opsgenie-sdk
      Result: False
     Comment: An importable Python 2 pip module is required but could not be found on your system. This usually means that the system's pip package is not installed properly.
     Started: 22:22:51.609574
    Duration: 0.609 ms
     Changes:

On the minion:

jbye@xxxxxxx:~$ pip --version
pip 9.0.1 from /usr/local/lib/python2.7/dist-packages/pip-9.0.1-py2.7.egg (python 2.7)

[hemebond]

I'm able to reproduce this error reliably with the following:

# salt-call test.versions
local:
    Salt Version:
               Salt: 2017.7.7
     
    Dependency Versions:
               cffi: Not Installed
           cherrypy: Not Installed
           dateutil: 2.2
          docker-py: Not Installed
              gitdb: Not Installed
          gitpython: Not Installed
              ioflo: Not Installed
             Jinja2: 2.9.4
            libgit2: Not Installed
            libnacl: Not Installed
           M2Crypto: Not Installed
               Mako: Not Installed
       msgpack-pure: Not Installed
     msgpack-python: 0.4.2
       mysql-python: 1.2.3
          pycparser: Not Installed
           pycrypto: 2.6.1
       pycryptodome: Not Installed
             pygit2: Not Installed
             Python: 2.7.9 (default, Jun 29 2016, 13:08:31)
       python-gnupg: Not Installed
             PyYAML: 3.11
              PyZMQ: 14.4.0
               RAET: Not Installed
              smmap: Not Installed
            timelib: Not Installed
            Tornado: 4.2.1
                ZMQ: 4.0.5
     
    System Versions:
               dist: debian 8.7 
             locale: UTF-8
            machine: x86_64
            release: 3.16.0-4-amd64
             system: Linux
            version: debian 8.7 

# salt-call pip.freeze
local:
    - backports.ssl-match-hostname==3.5.0.1
    - boto==2.34.0
    - chardet==3.0.4
    - colorama==0.3.2
    - croniter==0.3.4
    - docker-pycreds==0.3.0
    - futures==3.0.3
    - html5lib==0.999
    - idna==2.7
    - ipaddress==1.0.22
    - Jinja2==2.9.4
    - MarkupSafe==0.23
    - msgpack-python==0.4.2
    - MySQL-python==1.2.3
    - pip==18.0
    - psutil==2.1.1
    - pycrypto==2.6.1
    - pycurl==7.19.5
    - python-apt==0.9.3.12
    - python-dateutil==2.2
    - python-systemd==231
    - pytz==2012rc0
    - PyYAML==3.11
    - pyzmq==14.4.0
    - requests==2.19.1
    - salt==2017.7.7
    - setuptools==40.4.3
    - six==1.11.0
    - tornado==4.2.1
    - urllib3==1.23
    - websocket-client==0.53.0
    - wheel==0.24.0

States

docker-repo:
  pkgrepo.managed:
    - name: deb https://apt.dockerproject.org/repo {{ grains["os"]|lower }}-{{ grains["oscodename"] }} main
    - humanname: {{ grains["os"] }} {{ grains["oscodename"]|capitalize }} Docker Package Repository
    - keyid: 58118E89F3A912897C070ADBF76221572C52609D
    - keyserver: hkp://p80.pool.sks-keyservers.net:80
    - refresh_db: True

docker-engine:
  pkg.installed:
    - require:
      - pkgrepo: docker-repo

docker:
  pip.installed:
    - reload_modules: True
    - force_reinstall: True
    - upgrade: True
    - require:
      - pkg: docker-engine

nginx:
  docker_container.running:
    - image: nginx:latest
    - require:
      - pip: docker

Result

# salt-call state.apply testminion
[ERROR   ] Failed to import module dockermod, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1438, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/usr/lib/python2.7/dist-packages/salt/modules/dockermod.py", line 219, in <module>
    import docker
  File "/usr/local/lib/python2.7/dist-packages/docker/__init__.py", line 2, in <module>
    from .api import APIClient
  File "/usr/local/lib/python2.7/dist-packages/docker/api/__init__.py", line 2, in <module>
    from .client import APIClient
  File "/usr/local/lib/python2.7/dist-packages/docker/api/client.py", line 10, in <module>
    from .build import BuildApiMixin
  File "/usr/local/lib/python2.7/dist-packages/docker/api/build.py", line 6, in <module>
    from .. import auth
  File "/usr/local/lib/python2.7/dist-packages/docker/auth.py", line 8, in <module>
    from . import errors
  File "/usr/local/lib/python2.7/dist-packages/docker/errors.py", line 34, in <module>
    class APIError(requests.exceptions.HTTPError, DockerException):
AttributeError: 'module' object has no attribute 'exceptions'
[ERROR   ] State 'docker_container.running' was not found in SLS 'testminion'
Reason: 'docker_container' __virtual__ returned False: 'docker.version' is not available.

local:
----------
          ID: docker-repo
    Function: pkgrepo.managed
        Name: deb https://apt.dockerproject.org/repo debian-jessie main
      Result: True
     Comment: Package repo 'deb https://apt.dockerproject.org/repo debian-jessie main' already configured
     Started: 21:41:48.441692
    Duration: 43.306 ms
     Changes:   
----------
          ID: docker-engine
    Function: pkg.installed
      Result: True
     Comment: The following packages were installed/updated: docker-engine
     Started: 21:41:48.864323
    Duration: 8002.486 ms
     Changes:   
              ----------
              docker-engine:
                  ----------
                  new:
                      17.05.0~ce-0~debian-jessie
                  old:
----------
          ID: docker
    Function: pip.installed
      Result: True
     Comment: All packages were successfully installed
     Started: 21:41:57.524563
    Duration: 2198.257 ms
     Changes:   
              ----------
              docker==3.5.0:
                  Installed
----------
          ID: nginx
    Function: docker_container.running
      Result: False
     Comment: State 'docker_container.running' was not found in SLS 'testminion'
              Reason: 'docker_container' __virtual__ returned False: 'docker.version' is not available.
     Changes:   

Reset With

Removing docker and either certifi or idna will cause the state to fail:

salt-call pip.uninstall certifi,docker && salt-call pkg.purge docker-engine && systemctl restart salt-minion

[Ch3LL]

This error:

class APIError(requests.exceptions.HTTPError, DockerException):
AttributeError: 'module' object has no attribute 'exceptions'

is telling me the version of requests you have installed does not have the exceptions attribute. What happens if you run python -c "import requests.exceptions; requests.exceptions.HTTPError"

[hemebond]

@Ch3LL without idna I get this error:

# python -c "import requests.exceptions; requests.exceptions.HTTPError"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/requests/__init__.py", line 113, in <module>
    from . import packages
  File "/usr/local/lib/python2.7/dist-packages/requests/packages.py", line 7, in <module>
    locals()[package] = __import__(package)
ImportError: No module named idna

I updated the test state:

requests:
  pip.installed:
    - reload_modules: True
    - force_reinstall: True
    - upgrade: True
    - require:
      - pkg: docker-engine
    - require_in:
      - pip: docker

idna:
  pip.installed:
    - reload_modules: True
    - force_reinstall: True
    - upgrade: True
    - require:
      - pkg: docker-engine
    - require_in:
      - pip: docker

certifi:
  pip.installed:
    - reload_modules: True
    - force_reinstall: True
    - upgrade: True
    - require:
      - pkg: docker-engine
    - require_in:
      - pip: docker

and got this (truncated):

----------
          ID: requests
    Function: pip.installed
      Result: True
     Comment: All packages were successfully installed
     Started: 22:49:43.487727
    Duration: 1871.094 ms
     Changes:   
              ----------
              requests==2.19.1:
                  Installed
----------
          ID: idna
    Function: pip.installed
      Result: True
     Comment: All packages were successfully installed
     Started: 22:49:46.307463
    Duration: 1239.396 ms
     Changes:   
              ----------
              idna==2.7:
                  Installed
----------
          ID: certifi
    Function: pip.installed
      Result: True
     Comment: All packages were successfully installed
     Started: 22:49:48.491100
    Duration: 1175.761 ms
     Changes:   
              ----------
              certifi==2018.8.24:
                  Installed
----------
          ID: nginx
    Function: docker_container.running
      Result: False
     Comment: State 'docker_container.running' was not found in SLS 'testminion'
              Reason: 'docker_container' __virtual__ returned False: 'docker.version' is not available.
     Changes:   

[Ch3LL]

is there any other information in the debug log output as to why its not loading?

[hemebond]

Here is the trace log for state.apply. I'm looking through it but not sure what to look for.

testminion.output.zip

[TRACE   ] Error loading module.dockercompose: The dockercompose execution module not loaded: compose python library not available.
[TRACE   ] Error loading module.dockermod: Could not import docker module, is docker-py installed?

I tried adding docker-compose to the state and it failed to install:

Found existing installation: PyYAML 3.11 Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

[Ch3LL]

and if you run this: python -c "import docker" what happens?

[hemebond]

I'm not sure what state you want the machine in. After the failed attempt to install docker-compose:

# salt-call cmd.run 'python -c "import docker"'
[ERROR   ] Command 'python -c "import docker"' failed with return code: 1
[ERROR   ] stdout: Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named docker
[ERROR   ] retcode: 1
local:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ImportError: No module named docker

Which is to be expected since the docker install wasn't attempted (docker-compose state was a pre-requisite). Cleaning that out and trying after the original failing docker install:

# salt-call cmd.run 'python -c "import docker"'
local:

[Ch3LL]

thanks for clarifying. i only focused on the docker error not the yaml one. This error:

Found existing installation: PyYAML 3.11 Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

I am guessing you are using pip 10 and I believe this is a pip and other packaging issue as you can see here: pypa/pip#5247

[hemebond]

Hi @Ch3LL , the minion has this version:

# salt-call pip.version
local:
    18.0

I have a specific state for all my Debian minions to upgrade PIP because of issues.

Edit: I just re-read the issue you linked to. Is it every PIP version over 9 that has the issue? Of course, this error is only when trying to install docker-compose. I don't see the error when trying to install docker; just the original error.

[Ch3LL]

yes any pip versions >9 as they changed the behavior to not handle uninstalling disutils packages since there was risk of not uninstalling everything and would cause issues. Essentially you will need to uninstall the disutils package as pip will not handle that anymore.

[hemebond]

I thought sticking to PIP 9.0.0 had helped (seems to work on current minions) but on a brand new minion it just brought me back to #45292

[Ch3LL]

since this issue is resolved are you okay to close this issue and continue investigating your other issue in #45292 ?

[hemebond]

I'm not sure if this issue is resolved. I seem to get bounced between the two issues. @sdemura was the original reporter.

[Ch3LL]

thanks for clarifying @sdemura are you still seeing this issue?

[hemebond]

Ran into this issue again I believe, with two out of four minions reporting: 'docker.list_containers' is not available. when trying to use docker.list_containers, even after a restart.

I removed the docker PIP module (bad-minion means docker.list_containers returned the error):

bad-minion:
    ----------
    stdout:
        Uninstalling docker-3.5.1:
          Successfully uninstalled docker-3.5.1
working:
    ----------
    stdout:
        Uninstalling docker-3.6.0:
          Successfully uninstalled docker-3.6.0
working:
    ----------
    stdout:
        Uninstalling docker-3.5.0:
          Successfully uninstalled docker-3.5.0
bad-minion:
    ----------
    stdout:
        Uninstalling docker-3.5.1:
          Successfully uninstalled docker-3.5.1

I then did pip.install docker-py on all four minions and they were all successfully executed docker.list_containers without even requiring a restart.

Perhaps the issue is with certain versions (3.5.1?) of the docker PIP module. I replaced docker-py with docker==3.6.0 and it still worked.

New minion with docker==3.6.0 failed with ERROR: Unable to perform containers: cannot import name universaldetector.
New minion with docker-py==1.10.6 failed with 'docker_container' __virtual__ returned False: 'docker.version' is not available. until restarted.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.