duplicate-code: similarities is not working in pipenv virtual env

[mcallaghan-bsm]

Steps to reproduce

1. start with two files foo.py and bar.py, have them each define a different function, with 100% duplicated lines (>4)
2. run: pylint --reports=y --disable=all --enable=similarities foo.py bar.py

Here's a same of the two files: (as you can see, nearly 100% of the lines are duplicated)

foo.py

def foo():
    """incomplete docstring"""

    # this list is obtained from GetTimeZones().
    utcOffsetToUnholyTimezoneMapping = {
        # (UTC Offset, ApplyDst), Unholy Mix of Timezone IDs
        (-1, False): "Etc/GMT+1",
        (-1, True): "America/Scoresbysund",
        (-2, False): "Etc/GMT+2",
        (-3, False): "Etc/GMT+3",
        (-3, True): "Etc/GMT+3",
        (-3.5, True): "America/St_Johns",
        (-3.5, False): "America/St_Johns",
        (-4, False): "Etc/GMT+4",
        (-4, True): "America/Halifax",
        (-5, False): "Etc/GMT+5",
        (-5, True): "EST5EDT",
        (-6, False): "Etc/GMT+6",
        (-6, True): "CST6CDT",
        (-7, False): "Etc/GMT+7",
        (-7, True): "MST7MDT",
        (-8, False): "PST8PDT",
        (-8, True): "PST8PDT",
        (-9, False): "America/Anchorage",
        (-9, True): "America/Anchorage",
        (-10, False): "Etc/GMT+10",
        (-10, True): "Pacific/Honolulu"
    }

    print('foo')
    print('bar')
    print('bat')
    print('biz')
    print('egg')
    print('snake')

bar.py

def bar():
    """incomplete docstring"""

    # this list is obtained from GetTimeZones().
    utcOffsetToUnholyTimezoneMapping = {
        # (UTC Offset, ApplyDst), Unholy Mix of Timezone IDs
        (-1, False): "Etc/GMT+1",
        (-1, True): "America/Scoresbysund",
        (-2, False): "Etc/GMT+2",
        (-3, False): "Etc/GMT+3",
        (-3, True): "Etc/GMT+3",
        (-3.5, True): "America/St_Johns",
        (-3.5, False): "America/St_Johns",
        (-4, False): "Etc/GMT+4",
        (-4, True): "America/Halifax",
        (-5, False): "Etc/GMT+5",
        (-5, True): "EST5EDT",
        (-6, False): "Etc/GMT+6",
        (-6, True): "CST6CDT",
        (-7, False): "Etc/GMT+7",
        (-7, True): "MST7MDT",
        (-8, False): "PST8PDT",
        (-8, True): "PST8PDT",
        (-9, False): "America/Anchorage",
        (-9, True): "America/Anchorage",
        (-10, False): "Etc/GMT+10",
        (-10, True): "Pacific/Honolulu"
    }

    print('foo')
    print('bar')
    print('bat')
    print('biz')
    print('egg')
    print('snake')

Current behavior

The output is ALWAYS zero duplicated lines

Duplication
-----------

+-------------------------+------+---------+-----------+
|                         |now   |previous |difference |
+=========================+======+=========+===========+
|nb duplicated lines      |0     |0        |=          |
+-------------------------+------+---------+-----------+
|percent duplicated lines |0.000 |0.000    |=          |
+-------------------------+------+---------+-----------+

(app) root@dbac90fda4f0:/app# pylint --reports=y --enable=similarities  /mnt/root_project/foo.py /mnt/root_project/bar.py  | egrep "duplicated|similar"
|nb duplicated lines      |0     |0        |=          |
|percent duplicated lines |0.000 |0.000    |=          |

It is definitely running on the two files (other warnings etc happen)

+---------+-------+-----------+-----------+------------+---------+
|type     |number |old number |difference |%documented |%badname |
+=========+=======+===========+===========+============+=========+
|module   |2      |2          |=          |0.00        |100.00   |
+---------+-------+-----------+-----------+------------+---------+
|class    |0      |0          |=          |0           |0        |
+---------+-------+-----------+-----------+------------+---------+
|method   |0      |0          |=          |0           |0        |
+---------+-------+-----------+-----------+------------+---------+
|function |2      |2          |=          |100.00      |100.00   |
+---------+-------+-----------+-----------+------------+---------+


+---------------+------+--------+---------+-----------+
|module         |error |warning |refactor |convention |
+===============+======+========+=========+===========+
|foo |0.00  |50.00   |0.00     |50.00      |
+---------------+------+--------+---------+-----------+
|bar |0.00  |50.00   |0.00     |50.00      |
+---------------+------+--------+---------+-----------+

+----------+-------+------+---------+-----------+
|type      |number |%     |previous |difference |
+==========+=======+======+=========+===========+
|code      |64     |86.49 |64       |=          |
+----------+-------+------+---------+-----------+
|docstring |2      |2.70  |2        |=          |
+----------+-------+------+---------+-----------+
|comment   |4      |5.41  |4        |=          |
+----------+-------+------+---------+-----------+
|empty     |4      |5.41  |4        |=          |
+----------+-------+------+---------+-----------+

Expected behavior

We expect it to catch duplicated lines ACROSS files
(note that SAME file is not supported as per #1457)

pylint --version output

# pylint --version
pylint 2.3.1
astroid 2.2.5
Python 3.7.3 (default, May  8 2019, 05:31:59) 
[GCC 6.3.0 20170516]

NOTE we're running inside a docker inside pipenv

(app) root@dbac90fda4f0:/app# pipenv --version
pipenv, version 2018.11.26

(app) root@dbac90fda4f0:/app# which python3
/root/.local/share/virtualenvs/app-4PlAip0Q/bin/python3

[mcallaghan-bsm]

NOTE:

If I run outside the virtual environment, similarities/duplicate line checks work!

$ pylint --reports=y --enable=similarities  foo.py bar.py | egrep -i "duplicat|similar"
bar.py:1:0: R0801: Similar lines in 2 files
    print('snake') (duplicate-code)
Duplication
|nb duplicated lines      |31     |31       |=          |
|percent duplicated lines |44.286 |44.286   |=          |
|duplicate-code    |1           |

$ pylint --version
pylint 2.3.1
astroid 2.2.5
Python 3.6.7 (default, Oct 22 2018, 11:32:17) 
[GCC 8.2.0]

[PCManticore]

This is an interesting behaviour, thanks for the clear examples. Not sure why it's not working in a virtualenv though.

[NicholasCunningham]

Curious if there's any update on this issue? I'm running into this exact problem on my project and I can't find another solution. Thanks!

[Pierre-Sassoulas]

Hello, except if a fix get merged by chance when changing the similar checker (there's two MR for multiprocessing flying #4175 and #4565), everything is happening in this issue.

[Pierre-Sassoulas]

I can't reproduce, so I assume this was fixed in latest version of pylint (2.15.0-dev0 at least).