Add ability to ignore a file or directory without modifying it

[adamtheturtle]

In a project I maintain, we vendor some third party Python code and the project includes generated files such as files created by python-versioneer.

What I'd like to do is type check all files, including any new files, apart from those that I have explicitly ignored.

Right now I have the following hack workaround:

import subprocess
import sys


def main() -> None:
    args = ['mypy', '.']
    ignore_paths = {
        'src/dcos_e2e/_vendor',
        'src/dcos_e2e/_version.py',
        'versioneer.py',
    }
    result = subprocess.run(args=args, stdout=subprocess.PIPE)
    result_lines = result.stdout.decode().strip().split('\n')
    error_lines = [
        line for line in result_lines
        if not any(line.startswith(path) for path in ignore_paths)
    ]
    print('\n'.join(error_lines))
    sys.exit(int(bool(error_lines)))


if __name__ == '__main__':
    main()

Various tools have mechanisms for doing this, for example:

Adam@MacBook-Pro ~/D/m/d/dcos-e2e> cat .isort.cfg 
[settings]
skip=_vendor,
     src/dcos_e2e/__init__.py,
     versioneer.py,
     setup.py,

[flake8]
exclude=./src/dcos_e2e/_vendor/,
        ./src/dcos_e2e/_version.py,
        ./versioneer.py,

Adam@MacBook-Pro ~/D/m/d/dcos-e2e> cat pylintrc | grep ignore= -A 3
ignore=CVS,
       _vendor,
      versioneer.py,
      _version.py,

Thank you for the software.

[gvanrossum]

If you're just going to ignore the errors from specific paths, a more elegant solution is to use module-specific sections in your mypy.ini with ignore_errors = True -- see config file docs.

[adamtheturtle]

I think that will work and it is something that I was not aware of and I probably should have read the docs closer. Thank you, I will close this.

[danielbraun89]

If you're just going to ignore the errors from specific paths, a more elegant solution is to use module-specific sections in your mypy.ini with ignore_errors = True -- see config file docs.

The proposed solution doesn't work in case the mypy.ini is being controlled by a different entity (DevOps for example).
There is a need for a local solution

[sidmitra]

If you're just going to ignore the errors from specific paths, a more elegant solution is to use module-specific sections in your mypy.ini with ignore_errors = True -- see config file docs.

I have a django codebase, where mypy is catching type errors in auto-generated migration files
These files live in different apps eg. customer/migrations/*.py and myapp/migrations/*.py. In such cases it might be good to add a glob or something to ignore such files instead of adding an individual entry for each app in the codebase(and also each new app we add).
Similarly we might want to ignore unit test case files across the project in different sub directories, but which all named say test_*.py

[gvanrossum]

@danielbraun89

The proposed solution doesn't work in case the mypy.ini is being controlled by a different entity (DevOps for example).

Wait, so you control neither mypy.ini nor the source code in this scenario? (Read the initial description.) That seems a little out of the ordinary.

@sidmitra

In such cases it might be good to add a glob or something to ignore such files

Yes, [mypy-<package>] entries in mypy.ini support globs (but for module names, not filenames, so e.g. customer.migrations.* instead of customer/migrations/*.py).

[Djedouas]

I have a Qt resource file compiled with pyrcc, which I want to ignore. I can't add #type: ignore at the top of the file, which is regenerated by Qt.
My mypy.ini is common to all my python projects and is stored in a location which is not the root of the project.

I would like to add

[mypy-resources]
ignore_errors = True

or

[mypy-**resources]
ignore_errors = True

to ignore all resources.py files recursively... neither works.

[brandtbucher]

@Djedouas I think that..

[mypy-*.resources]
ignore_errors = True

...should work for you. Does it?

[Djedouas]

@brandtbucher thanks, it works ! I was sure I had tested this pattern before, but apparently not... ;-)

[ericfrederich]

I am having same issue with a _version.py file which is generated by versioneer.

Another tool supporting ignore, exclude or skip to add to @adamtheturtle's list is pre-commit. I use it to automatically run black on Python files, also makes sure files end with newline, json and yaml files are valid, etc.

A bit clunky in that their yaml doesn't support an excludes list, just a single regex.

# ...
exclude: "(.idea/)|(src/my_project/__init__.py)|(src/my_project/_version.py)|(versioneer.py)"

[bhrutledge]

Pardon the comment on a closed issue, but this seems to be the most appropriate place.

In pypa/twine#619, I'm ignoring the tests (primarily to avoid noise in my editor) by making tests a package and using:

[mypy-tests.*]
ignore_errors = True

However, there's some concern about the side-effects of making tests a package, and it does feel a little hacky to me. Is there another option?

More detail at https://github.com/pypa/twine/pull/619/files#r423015876.

[Hubro]

I want to ignore errors in unit test files. My unit tests are co-located with the code they test, under <module-name>_test.py, and it doesn't seem to be possible to target those files with a mypy section. This doesn't work:

[mypy-*_test,*.*_test]
ignore_errors = True

[bhrutledge]

@Hubro In order to ignore tests as you're attempting to do, I've resorted to making the tests/ directory a package, by adding a __init__.py. See #4675 (comment) for details.

Of course, that solution depends on your project structure. Also, if you're using setuptools to create a distribution, it could also result in your tests directory being included as a package (but you can work around that).

[waketzheng]

For Django migrations:

[mypy-*.migrations.*]
ignore_errors = True

[micahjsmith]

Another situation where the existing configuration options don't seem to work. Here, the src directory contains data files that are to be used by cookiecutter to render a new project. And unfortunately the project once rendered will be a python package, so mypy detects it. But it complains that the package name is not valid, which is true, but it should be ignored.

$ tree .
.
├── mylib
│   ├── __init__.py
│   ├── bar.py
│   └── template
│       ├── cookiecutter.json
│       └── {{\ cookiecutter.dirname\ }}
│           ├── __init__.py
│           └── foo.py
└── mypy.ini

mypy.ini:

[mypy]

[mypy-mylib.template.*]
ignore_errors = True

Running this:

$ mypy mylib
{{ cookiecutter.dirname }} is not a valid Python package name

Any workarounds?

[hauntsaninja]

I think it's worth adding more support for something like this. We actually now have an example in mypy where ignore_errors = True for modules doesn't feel sufficient. When running mypy on mypy with mypy --namespace-packages mypy, we now pick up all of typeshed.

[michael-k]

It's not only helpful in combination with --namespace-packages but also with #9614. Before v0.800 mypy ignored our tests etc. and now it finds lots of “new” files and complains about Duplicate module named 'conftest'. And it's really hard to find the right bits that tell mypy to ignore that.

I don't have a minimal example atm. that I could share.

[aviau]

Mymy has started linting all of node_modules as of 0.800 for me

[dannykomo]

I have the same problem.

my_project/www/node_modules/gulp-sass/node_modules/node-sass/node_modules/node-gyp/gyp/pylib/gyp/__init__.py:37: error: invalid syntax
Found 1 error in 1 file (errors prevented further checking)

I would appreciate at least something like mypy my_project --exclude my_project/www/ if not adding such a feature into the mypy.ini file

[marcglobality]

I think it's worth adding more support for something like this. We actually now have an example in mypy where ignore_errors = True for modules doesn't feel sufficient. When running mypy on mypy with mypy --namespace-packages mypy, we now pick up all of typeshed.

after the last version of mypy, I started to see all the typehinting for the notebooks folder I have at the root of the project. Using this allowed to reduce the noise. The other option was not possible because notebooks is not a package (or at least not all .py files have __init__ in the path to get to them. The problem now is #8944 ...

I think a --exclude as in flake8 in mypy.ini would solve all issues

[riconnon]

+1
we have a bunch of files not in modules all called helpers.py which get dynamically imported using importlib and mypy now complains about those being duplicate modules all named helpers

[wbekerom]

+1
as of v0.800 mypy started picking up bazel output folders. This folder contains an __init__.py and gets picked up by mypy: bazel-bin/examples/bazel_python/root/test_printer.runfiles resulting in:

test_printer.runfiles is not a valid Python package name

[hauntsaninja]

I opened a PR addressing this at #9992. I'd appreciate if some of you could test it and report back if it doesn't fix your issues.

[Bryant-Yang]

After upgrade to 0.800. When run 'mypy .' in my project root dir as before, I got error "...site-packages is in the PYTHONPATH. Please change directory so it is not". This error is raised in 'modulefinder.py'. I think it may be caused by take my "venv" virtual env dir as a package. May be add 'ignore_path' in config file could fix this?

[wbekerom]

I opened a PR addressing this at #9992. I'd appreciate if some of you could test it and report back if it doesn't fix your issues.

Awesome, for me this solved the issue by ignoring all bazel generated directories:

ignore_path = bazel-bin, bazel-myproject, bazel-out

Mypy still throws many errors that were not there in v0.790... at least those I can start fixing myself 😄

[dmontagu]

I'm running into an issue where I have vendorized git submodules for a C++ extension (which I don't control) in a subfolder of my package, and these vendorized modules include python 2 files. As a result, mypy is stopping due to syntax errors even if I set ignore_errors=true for the parent folder ("module"). I could explicitly list out all files to include when I run mypy, but then I'd have to update it any time I add something at the top level of the package. Having the ability to mark the vendor folder as explicitly ignored, so that this wouldn't cause syntax errors, would be very helpful.

[omry]

Seeing a similar problem with a file in a build directory:

$ mypy --version
mypy 0.800
$ mypy --strict .
omegaconf/__init__.py: error: Duplicate module named 'omegaconf' (also at './build/lib/omegaconf/__init__.py')

It doesn't make sense to me to ignore errors for build/lib but rather to completely exclude it from processing.

[hardikkat24]

Hello, I wanted to ask something. I am using MyPy on a large existing codebase. I am maintaining a list of files that do not have types. How can I run the tests for all files except these files? Thanks!

[micahjsmith]

@hardikkat24 the documentation on the implemented functionality is here. You should provide the list of files to the exclude option explicitly or using regex.

[hardikkat24]

Thanks a lot @micahjsmith

[sonic-chase]

Another situation where the existing configuration options don't seem to work. Here, the src directory contains data files that are to be used by cookiecutter to render a new project. And unfortunately the project once rendered will be a python package, so mypy detects it. But it complains that the package name is not valid, which is true, but it should be ignored.

$ tree .
.
├── mylib
│   ├── __init__.py
│   ├── bar.py
│   └── template
│       ├── cookiecutter.json
│       └── {{\ cookiecutter.dirname\ }}
│           ├── __init__.py
│           └── foo.py
└── mypy.ini

mypy.ini:

[mypy]

[mypy-mylib.template.*]
ignore_errors = True

Running this:

$ mypy mylib
{{ cookiecutter.dirname }} is not a valid Python package name

Any workarounds?

Did you find a solution for this?

[micahjsmith]

@sonic-chase see my above comment linking to the mypy docs. The correct config option is exclude

[sonic-chase]

@sonic-chase see my above comment linking to the mypy docs. The correct config option is exclude

Thanks! It was just a regex issue, ha.