Pylint 2.10.0 incorrectly assumes that $HOME/.cache/pylint exists

[jfly]

Bug description

I'm trying out Pylint 2.10.0, and seeing a crash (described below).

Configuration

No response

Command used

pylint a.py

Pylint output

PYLINTHOME is now '/home/jeremy/.cache/pylint' but obsolescent '/home/jeremy/.pylint.d' is found; you can safely remove the latter
Traceback (most recent call last):
  File "/home/jeremy/.local/share/virtualenvs/timelines-backend-2BUOoOOv/bin/pylint", line 8, in <module>
    sys.exit(run_pylint())
  File "/home/jeremy/.local/share/virtualenvs/timelines-backend-2BUOoOOv/lib/python3.8/site-packages/pylint/__init__.py", line 21, in run_pylint
    from pylint.lint import Run as PylintRun
  File "/home/jeremy/.local/share/virtualenvs/timelines-backend-2BUOoOOv/lib/python3.8/site-packages/pylint/lint/__init__.py", line 76, in <module>
    from pylint.lint.parallel import check_parallel
  File "/home/jeremy/.local/share/virtualenvs/timelines-backend-2BUOoOOv/lib/python3.8/site-packages/pylint/lint/parallel.py", line 8, in <module>
    from pylint.lint.utils import _patch_sys_path
  File "/home/jeremy/.local/share/virtualenvs/timelines-backend-2BUOoOOv/lib/python3.8/site-packages/pylint/lint/utils.py", line 11, in <module>
    from pylint.config import PYLINT_HOME
  File "/home/jeremy/.local/share/virtualenvs/timelines-backend-2BUOoOOv/lib/python3.8/site-packages/pylint/config/__init__.py", line 89, in <module>
    for filename in os.listdir(PYLINT_HOME):
FileNotFoundError: [Errno 2] No such file or directory: '/home/jeremy/.cache/pylint'

Expected behavior

I would not expect Pylint to crash. I'd expect it to create this ~/.cache/pylint directory as needed.

I believe this was broken in #4661

Pylint version

$ mkdir /home/jeremy/.cache/pylint && pylint --version
PYLINTHOME is now '/home/jeremy/.cache/pylint' but obsolescent '/home/jeremy/.pylint.d' is found; you can safely remove the latter
pylint 2.10.0
astroid 2.7.2
Python 3.8.10 (default, Aug 20 2021, 16:16:35)
[GCC 11.1.0]

OS / Environment

Arch Linux

Additional dependencies

No response

[KanHarI]

Reproduced on windows, python 3.9.4 in a venv:

PYLINTHOME is now 'C:\Users\*****\AppData\Local\pylint\pylint\Cache' but obsolescent 'C:\Users\*****\.pylint.d' is found; you can safely remove the latter
Traceback (most recent call last):
  File "C:\Python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "E:\workspace\Programming\ConnectedPapers\rest-api\venv\lib\site-packages\pylint\__main__.py", line 9, in <module>
    pylint.run_pylint()
  File "E:\workspace\Programming\ConnectedPapers\rest-api\venv\lib\site-packages\pylint\__init__.py", line 21, in run_pylint
    from pylint.lint import Run as PylintRun
  File "E:\workspace\Programming\ConnectedPapers\rest-api\venv\lib\site-packages\pylint\lint\__init__.py", line 76, in <module>
    from pylint.lint.parallel import check_parallel
  File "E:\workspace\Programming\ConnectedPapers\rest-api\venv\lib\site-packages\pylint\lint\parallel.py", line 8, in <module>
    from pylint.lint.utils import _patch_sys_path
  File "E:\workspace\Programming\ConnectedPapers\rest-api\venv\lib\site-packages\pylint\lint\utils.py", line 11, in <module>
    from pylint.config import PYLINT_HOME
  File "E:\workspace\Programming\ConnectedPapers\rest-api\venv\lib\site-packages\pylint\config\__init__.py", line 89, in <module>
    for filename in os.listdir(PYLINT_HOME):
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'C:\\Users\\*****\\AppData\\Local\\pylint\\pylint\\Cache'

[FoamyGuy]

I reproduced this on both Pop_OS linux locally, and inside GitHub actions using ubuntu-latest the latter can be seen in the PR linked above.

In my local environment I found that deleting the old cache directory that is mentioned by the warning will allow the new version of pylint to run successfully.

[dhalbert]

This bug is breaking our CI runs when an earlier version of pylint has already run (in our case, pylint 2.7.1 as pinned by pre-commit.). We eventually need to advance the versions, but we have about 300 repos we have to change. If the crash did not happen, but a warning was simply issued, it wouldn't stop our CI runs.

[CAM-Gerlach]

I'm experiencing this as well on all platforms I tested, Python 3.8 and 3.9, Linux and Windows, and with and without Pre-Commit. The culprit actually appears to be 56fdadb, a commit added directly to master only a few hours before release. The code, a check for a trivial nuisance under a specific run scenario and which did not include a test, assumes that the new home directory already exists, and is executed on file import, before it ever has a chance to be created, and thus crashes pylint. As such, it would appear that it will break running pylint on any machine that has previously run any prior version of Pylint.

To work around this until a non-broken version is out, you can delete the old pylint directory from the path specified, and the problem code is never triggered. Ideally, there really should be at least a minimal smoke test for running Pylint when a previous directory was found and the new directory is not yet present, which would have caught this bug. The simplest fix for the issue itself is guarding the block by an additional check for whether PYLINT_HOME exists; the only downside is a bit of extra spam in pre-commit on the first pylint run (which shouldn't be visible unless other errors occur)

[AWhetter]

The fix for this has been released in v2.10.1. Sorry that took a bit longer than expected. I wish you all a weekend of green CI runs!

[CAM-Gerlach]

Thanks @AWhetter !

[jnewbigin]

This bug still seems to be present in 2.10.2

Unable to create directory /home/docker-user/.cache/pylint
Unable to create file /home/docker-user/.cache/pylint/pagerduty_cli1.stats: [Errno 2] No such file or directory: '/home/docker-user/.cache/pylint/pagerduty_cli1.stats'

The workaround is:
mkdir -p "$HOME/.cache"

But I would expect that if that directory is required then pylint would create it too

[Pierre-Sassoulas]

The issue for that is #4900