[BUG] salt.modules.pip:is_installed doesn't handle locally installed packages

[jholloway7]

Description

I'm setting up a fairly complex virtualenv with a few packages (e.g. cdecimal-2.3, internal packages, etc) that aren't published on pypi for whatever reason. I copy the package into the file system, check the hash and pip install it from a local file.

The normal pip syntax for this is something like this:

 $ /var/venv/base/bin/pip install /var/venv/deps/cdecimal-2.3.tar.gz

But the same effect can be achieved through the pip.installed state:

/var/venv/deps/cdecimal-2.3.tar.gz
  pip.installed:
    - pip_bin: /var/venv/base/bin/pip
    - require:
      - virtualenv: /var/venv/base
      - file: /var/venv/deps/cdecimal-2.3.tar.gz

When you run pip freeze --all on this virtualenv, the output syntax of such packages is slightly different than the normal 'frozen' version number:

$ /var/venv/base/bin/pip freeze --all

    ...
    cdecimal @ file:///var/venv/deps/cdecimal-2.3.tar.gz
    celery==3.1.26.post2
    ...

This syntax isn't handled by the pip.is_installed module:

    $ salt-call pip.is_installed cdecimal /var/venv/base
    [ERROR   ] Can't parse line 'cdecimal @ file:///var/venv/deps/cdecimal-2.3.tar.gz'
    local:
        False

The pip.is_installed module is used by other pip states to determine if a given library is installed. Hence, while it doesn't fail any states, it never sees that the locally installed package has been installed and installs it every time.

Setup
Please note that this setup is contrived for simplicity with a readily accessible example. There are other ways to install this particular package using --find-links that works around the problem. For self-maintained dist packages, I haven't really found a good solution that's as straightforward as simply pip installing the dist tarball directly from the file system.

/var/venv/base:
  virtualenv.managed: []
/var/venv/deps/cdecimal-2.3.tar.gz:
  file.managed:
    - source: https://www.bytereef.org/software/mpdecimal/releases/cdecimal-2.3.tar.gz
    - source_hash: d737cbe43ed1f6ad9874fb86c3db1e9bbe20c0c750868fde5be3f379ade83d8b
    - makedirs: True
    - creates: /var/venv/deps/cdecimal-2.3.tar.gz
  pip.installed:
    - pip_bin: /var/venv/base/bin/pip
    - require:
      - virtualenv: /var/venv/base
      - file: /var/venv/deps/cdecimal-2.3.tar.gz

Steps to Reproduce the behavior

    $ salt-call pip.is_installed cdecimal /var/venv/base
    [ERROR   ] Can't parse line 'cdecimal @ file:///var/venv/deps/cdecimal-2.3.tar.gz'
    local:
        False

Expected behavior
For completeness, pip.is_installed could/should parse the pip freeze syntax of "[pkgname] @ file:///..." to determine that a locally installed package is present.

I think it would be as simple as splitting on the first '@', trimming spaces and comparing the first element to the specified package name.

Screenshots
N/A

Versions Report

salt --versions-report

# salt --versions-report
Salt Version:
           Salt: 3001.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: 4.3.0
          gitdb: 2.0.3
      gitpython: 2.1.8
         Jinja2: 2.10
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: 3.4.7
         pygit2: Not Installed
         Python: 3.6.9 (default, Jul 17 2020, 12:50:27)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 17.1.2
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: ubuntu 18.04 Bionic Beaver
         locale: UTF-8
        machine: x86_64
        release: 5.3.0-1032-aws
         system: Linux
        version: Ubuntu 18.04 Bionic Beaver

Additional context
N/A

[elgalu]

@jholloway7 how did you proceed? I'm currently a bit fed up with thousands of ERROR ] Can't parse line ... :)

[jholloway7]

@jholloway7 how did you proceed? I'm currently a bit fed up with thousands of ERROR ] Can't parse line ... :)

Sorry, this got lost in my github notifications or I would have replied sooner. I ended up using an 'unless' in my pip.installed state:

maybe-install-cdecimal:
  pip.installed:
    - name: /var/venv/deps/cdecimal-2.3.tar.gz
    - pip_bin: /var/venv/base/bin/pip
    - user: <user>
    - unless: /var/venv/base/bin/pip freeze | grep cdecimal-2.3
    - require:
      - virtualenv: /var/venv/base
      - file: /var/venv/deps/cdecimal-2.3.tar.gz

It uses the stdout of pip freeze to determine if cdecimal-2.3 has already been installed from a local tarball and skips the state. This probably isn't very robust in other scenarios, but it was a viable workaround in my case for this particular package.