Exclude multiple directories on mypy commandline: improve documentation

[cclauss]

Questions and Help

Please note that this issue tracker is not a help form and this issue will be closed.

Please check here instead:

• Website
• Documentation
• Gitter

This question is about the mypy commandline, not about mypy configuration files.

The --exclude documentation does not explain how to exclude multiple directories on mypy commandline.

Should there be a single --exclude option with comma-separated terms or
should there be multiple --exclude options whose terms would be additive?

mypy --ignore-missing-imports --exclude @mypy_failing_files.txt # does not work as expected.

Given https://github.com/TheAlgorithms/Python we would like to exclude directories until they can be made mypy compatible. We currently have an include list that works but as our mypy compatibility increases, we would like to switch to a shorter exclude list like:

 mypy --ignore-missing-imports \
           --exclude ./arithmetic_analysis/ \
           --exclude ./ciphers/ \
           --exclude ./compression/ \
           --exclude ./conversions/ \
           --exclude ./data_structures/ \
           --exclude ./digital_image_processing/ \
           --exclude ./dynamic_programming/ \
           --exclude ./graphics/ \
           --exclude ./graphs/ \
           --exclude ./hashes/ \
           --exclude ./linear_algebra/ \
           --exclude ./maths/ \
           --exclude ./matrix/ \
           --exclude ./other/ \
           --exclude ./project_euler/ \
           --exclude ./scripts/ \
           --exclude ./searches/ \
           --exclude ./strings/ \
           --exclude ./web_programming/ .

[cclauss]

This works... Closing.

mypy --ignore-missing-imports \
    --exclude '(arithmetic_analysis|ciphers|compression|conversions|data_structures|digital_image_processing|dynamic_programming|graphics|graphs|hashes|linear_algebra|maths|matrix|other|project_euler|scripts|searches|strings|web_programming*)/$' .

[JelleZijlstra]

If the correct solution isn't obvious, the documentation should be improved. We could add an example excluding multiple directories.

[sam-hoffman]

I also found that the correct syntax wasn't obvious :( would really appreciate an example excluding multiple directories and an example excluding from a config file

[JelleZijlstra]

Reopening as a docs issue

[hauntsaninja]

The docs (that you linked to) do technically have an example excluding multiple things: --exclude '/(site-packages|node_modules|__pycache__|\..*)/$'

[cclauss]

That example has \..*)/$ which makes it difficult to understand for people who are not regex gurus.

If I want to land mypy in the continuous integration of a project but there are ten files or directories that are not yet compliant, how can I write a --exclude that spreads out over multiple lines so it more readable than #10250 (comment) ?

[JukkaL]

It could be good to have a few more examples in the docs, each of which only does a fairly specific thing, and clearly explain what each example does exactly. Regular expressions are not very familiar to everybody, and they can be used in subtly incorrect ways that result in false negatives or other unexpected results.

[stevekm]

I consider myself to be pretty decent with regex and even I cannot figure out how to simultaneously exclude

• files in some subdir matching a pattern (e.g. my Django migration files)
• a couple specific files scattered throughout the repo

Being forced to write one massive regex for this is pretty horrible. The command line syntax should be more similar to find and rsync which both allow for multiple inclusion and exclusion criteria in a more sensible way.

[Nickedude]

In the pull request above I added an example on how to combine expressions to exclude multiple file names an directory names. Similar to @cclauss I also tried passing the --exclude flag multiple times. :) I think this should get most of the confusion out of the way!

[JelleZijlstra]

I wonder if we should just make passing multiple --exclude flags work (equivalent to ORing them all together), since that feels like the most intuitive UI.

[emmatyping]

I suppose in the config file exclude would be come a list then? That seems reasonable and more ergonomic.

[hauntsaninja]

That makes sense to me / let's use #10310 to track that. Note the initial non regex implementation of exclude did this, but Jukka requested a single value for the exclude option in #9992 (comment) when we decided to use regexes.

[posita]

Please consider including an example on how to unambiguously identify a single entry at any level without sweeping in others. The current example(s) do not work for that.

As of 0.910, mypy --exclude /build/ will not exclude a directory called build in the top level of the current working directory. mypy --excluded build/ will work, but will also exclude onebuild/, twobuild/, redbuild/, bluebuild/, etc., at any level. Only something like --exclude '(^|/)build/' will exclude a directory called exactly build anywhere in the subtree, including the top level.

As an aside, the ergonomics of that are pretty brutal. I don't know of any other command that requires this much domain knowledge to correctly exclude precisely the desired files from processing.

[trallnag]

Here is the syntax I use:

exclude = """
(?x)(
    file.py$|
   anotherone.py
)
"""

[posita]

Thanks, @trallnag. What does (?x)(…) do?

[trallnag]

@posita, the parenthesis and the question mark are used to set modifier flags and in this case I set the x modifier to enable extended mode. This allows breaking up the regex string with whitespace and newlines.

I copied this from an example in the context of pre-commit some time ago

[posita]

Oh! Duh. It's the inline version of the verbose flag. Thanks! UPDATE: @trallnag, that syntax doesn't work in my .mypy.ini file. I get these errors:

…/.mypy.ini: Source contains parsing errors: '…/.mypy.ini'
        [line 16]: '(?x)(\n'
        [line 18]: ')\n'
        [line 19]: '"""\n'

[asears]

Some further documentation on configuring complex examples of exclusions with pyproject.toml would be helpful. There's some idiosyncrasies between the mypy.ini, the console and pyproject.toml which make it very confusing on how to exclude:

1. multiple folders & files with wildcards
2. files which depend on other modules in the project - those other modules get scanned even when excluded.

[trallnag]

Oh! Duh. It's the inline version of the verbose flag. Thanks! UPDATE: @trallnag, that syntax doesn't work in my .mypy.ini file. I get these errors:

…/.mypy.ini: Source contains parsing errors: '…/.mypy.ini'
        [line 16]: '(?x)(\n'
        [line 18]: ')\n'
        [line 19]: '"""\n'

@posita
Maybe it's because you are using a mypy.ini? For me it definitely works but i use pyproject.toml

[vnguyen2679]

@trallnag Would you mind to give an example of how to use pyproject.toml to exclude multiple directories and files?
I currently need to do this but am not sure how to proceed. Thanks.

[asears]

Here's one example I used.

[tool.mypy]
python_version = 3.8
ignore_missing_imports = true
exclude = "(setup.py|tasks.py)/$"

Would appreciate more.

[vnguyen2679]

Thanks @asears for the example. After a bit more searching, I was also able to find other examples.

[cclauss]

Here's one example I used.

[tool.mypy]
python_version = 3.8
ignore_missing_imports = true
exclude = (data_structures/stacks/next_greater_element.py|graphs/boruvka.py|graphs/breadth_first_search.py|graphs/breadth_first_search_2.py|graphs/check_cycle.py|graphs/finding_bridges.py|graphs/greedy_min_vertex_cover.py|graphs/random_graph_generator.py|maths/average_mode.py|maths/gamma_recursive.py|maths/proth_number.py|maths/series/geometric_series.py|maths/series/p_series.py|matrix_operation.py|other/fischer_yates_shuffle.py|other/least_recently_used.py|other/lfu_cache.py|other/lru_cache.py|searches/simulated_annealing.py|searches/ternary_search.py)

[posita]

Here is the syntax I use:

exclude = """
(?x)(
    file.py$|
   anotherone.py
)
"""

FYI, multiline regexes do not work in pyproject.toml with 0.930. (See #11825.) Proposal for a fix is up: #11828.

[posita]

I think this is fixed by #11329 (and possibly one of #11746 and #11828)? Oops! No. This is a documentation bug now.

[hauntsaninja]

Thanks to everyone who's contributed here. The docs on latest master (and the upcoming 0.931) are substantially different than when this was converted to a docs issue, so I'm closing this issue. If there are specific improvements still to be made, feel free to open a PR

[posita]

FWIW, I think @cclauss's issue remains (even with the doc updates), but perhaps it's better to open a new issue after 0.931 is released? (I'm willing to take a stab at an update once I get some free time. Feel free to @-mention me if a new issue is filed.)