No-untyped-call / no-untyped-def errors reported despite being disabled in mypy.ini

[obi1kenobi]

Bug Report

Mypy 0.800 and 0.812 are reporting no-untyped-def and no-untyped-call errors in modules that are configured to have those checks disabled in the mypy.ini file. This may be a regression introduced between 0.790 and 0.800, though this hypothesis is worth double-checking: on one hand, I'm not seeing those errors in 0.790, but on the other, that may also be because the modules in question aren't scanned in 0.790 and are only getting picked up in 0.800 because of the changes in behavior mentioned in 0.800's release log.

The issue was discovered as part of this PR: arviz-devs/arviz#1528 (comment)

To Reproduce

From the mypy_regression branch of my fork of arviz: https://github.com/obi1kenobi/arviz/tree/mypy_regression

$ virtualenv .v
created virtual environment CPython3.8.5.final.0-64 in 19559ms
    ...
$ source .v/bin/activate
$ pip install mypy==0.800 && pip install -e .
$ mypy .

Observe that mypy reports hundreds of untyped def / untyped call errors in modules like asv_benchmarks.benchmarks. Furthermore, notice that even though the top-level mypy.ini section enables those checks, they are disabled in the mypy.ini config section referring to asv_benchmarks.benchmarks.

The same issue is present for many other files and modules, for example ones in examples.

Expected Behavior

I expected that mypy would not report errors that the mypy.ini file configured to be ignored in the given module. For example, consider this snippet of the mypy.ini file:

[mypy-asv_benchmarks.benchmarks.*]
disallow_untyped_calls = False
disallow_untyped_defs = False

Given this configuration, I expected that mypy would not produce error reports like the following for code in asv_benchmarks.benchmarks:

asv_benchmarks/benchmarks/benchmarks.py:16: error: Function is missing a type annotation  [no-untyped-def]
asv_benchmarks/benchmarks/benchmarks.py:19: error: Call to untyped function "enable_numba" of "Numba" in typed context  [no-untyped-call]
asv_benchmarks/benchmarks/benchmarks.py:21: error: Call to untyped function "disable_numba" of "Numba" in typed context  [no-untyped-call]
asv_benchmarks/benchmarks/benchmarks.py:23: error: Function is missing a type annotation  [no-untyped-def]
asv_benchmarks/benchmarks/benchmarks.py:31: error: Function is missing a type annotation  [no-untyped-def]
asv_benchmarks/benchmarks/benchmarks.py:34: error: Call to untyped function "enable_numba" of "Numba" in typed context  [no-untyped-call]
asv_benchmarks/benchmarks/benchmarks.py:36: error: Call to untyped function "disable_numba" of "Numba" in typed context  [no-untyped-call]

Actual Behavior

mypy produced the above errors and many more, across many modules that all had similar disallow_* = False configuration. Since the full mypy error report and mypy.ini file are considerably large (the latter being autogenerated by typing_copilot), I will not paste them here -- they are available at the linked PR referenced above.

Your Environment

• Mypy version used: 0.800 and 0.812
• Python version used: 3.8.5
• Operating system and version: Ubuntu 20.04.2 LTS inside WSL2
• Mypy configuration options from mypy.ini (and other config files):

# Autogenerated by typing_copilot v0.5.4
[mypy]
no_implicit_optional = True
strict_optional = True
warn_redundant_casts = True
check_untyped_defs = True
disallow_untyped_calls = True
disallow_incomplete_defs = True
disallow_untyped_defs = True
disallow_untyped_decorators = True
ignore_missing_imports = False
warn_unused_ignores = True


# First party per-module rule relaxations
[mypy-arviz.*]
disallow_untyped_calls = False
disallow_untyped_defs = False

[mypy-arviz.data.datasets.*]
check_untyped_defs = False
disallow_incomplete_defs = False

[mypy-arviz.data.inference_data.*]
check_untyped_defs = False
disallow_incomplete_defs = False

[mypy-arviz.data.io_cmdstan.*]
check_untyped_defs = False
disallow_incomplete_defs = False

[mypy-arviz.data.io_cmdstanpy.*]
check_untyped_defs = False
disallow_incomplete_defs = False

[mypy-arviz.data.io_dict.*]
check_untyped_defs = False
disallow_incomplete_defs = False

[mypy-arviz.data.io_emcee.*]
check_untyped_defs = False
disallow_incomplete_defs = False

[mypy-arviz.data.io_numpyro.*]
check_untyped_defs = False
disallow_incomplete_defs = False

[mypy-arviz.data.io_pyjags.*]
check_untyped_defs = False
disallow_incomplete_defs = False

[mypy-arviz.data.io_pymc3.*]
check_untyped_defs = False
disallow_incomplete_defs = False

[mypy-arviz.data.io_pyro.*]
check_untyped_defs = False
disallow_incomplete_defs = False

[mypy-arviz.data.io_pystan.*]
check_untyped_defs = False
disallow_incomplete_defs = False

[mypy-arviz.data.io_tfp.*]
check_untyped_defs = False
disallow_incomplete_defs = False

[mypy-arviz.labels.*]
disallow_incomplete_defs = False

[mypy-arviz.plots.backends.*]
check_untyped_defs = False

[mypy-arviz.plots.backends.bokeh.*]
disallow_incomplete_defs = False

[mypy-arviz.plots.backends.matplotlib.forestplot.*]
disallow_incomplete_defs = False

[mypy-arviz.plots.backends.matplotlib.posteriorplot.*]
disallow_incomplete_defs = False

[mypy-arviz.plots.backends.matplotlib.ppcplot.*]
disallow_incomplete_defs = False

[mypy-arviz.plots.densityplot.*]
check_untyped_defs = False

[mypy-arviz.plots.elpdplot.*]
check_untyped_defs = False

[mypy-arviz.plots.pairplot.*]
disallow_incomplete_defs = False

[mypy-arviz.plots.separationplot.*]
check_untyped_defs = False

[mypy-arviz.plots.traceplot.*]
disallow_incomplete_defs = False

[mypy-arviz.rcparams.*]
check_untyped_defs = False
disallow_incomplete_defs = False

[mypy-arviz.sel_utils.*]
check_untyped_defs = False

[mypy-arviz.stats.density_utils.*]
check_untyped_defs = False

[mypy-arviz.stats.diagnostics.*]
check_untyped_defs = False

[mypy-arviz.stats.stats.*]
check_untyped_defs = False
disallow_incomplete_defs = False

[mypy-arviz.stats.stats_utils.*]
check_untyped_defs = False
disallow_incomplete_defs = False

[mypy-arviz.tests.base_tests.test_data.*]
check_untyped_defs = False
disallow_incomplete_defs = False

[mypy-arviz.tests.base_tests.test_diagnostics.*]
check_untyped_defs = False
disallow_incomplete_defs = False

[mypy-arviz.tests.base_tests.test_diagnostics_numba.*]
disallow_incomplete_defs = False

[mypy-arviz.tests.base_tests.test_plot_utils.*]
disallow_incomplete_defs = False

[mypy-arviz.tests.base_tests.test_plots_bokeh.*]
check_untyped_defs = False
disallow_incomplete_defs = False

[mypy-arviz.tests.base_tests.test_plots_matplotlib.*]
check_untyped_defs = False
disallow_incomplete_defs = False

[mypy-arviz.tests.base_tests.test_rcparams.*]
check_untyped_defs = False
disallow_incomplete_defs = False

[mypy-arviz.tests.base_tests.test_stats.*]
check_untyped_defs = False
disallow_incomplete_defs = False

[mypy-arviz.tests.base_tests.test_stats_numba.*]
disallow_incomplete_defs = False

[mypy-arviz.tests.base_tests.test_stats_utils.*]
check_untyped_defs = False
disallow_incomplete_defs = False

[mypy-arviz.tests.base_tests.test_utils.*]
disallow_incomplete_defs = False

[mypy-arviz.tests.base_tests.test_utils_numba.*]
disallow_incomplete_defs = False

[mypy-arviz.tests.conftest.*]
disallow_incomplete_defs = False

[mypy-arviz.tests.external_tests.test_data_cmdstan.*]
disallow_incomplete_defs = False

[mypy-arviz.tests.external_tests.test_data_cmdstanpy.*]
check_untyped_defs = False
disallow_incomplete_defs = False

[mypy-arviz.tests.external_tests.test_data_emcee.*]
disallow_incomplete_defs = False

[mypy-arviz.tests.external_tests.test_data_numpyro.*]
disallow_incomplete_defs = False

[mypy-arviz.tests.external_tests.test_data_pyjags.*]
disallow_incomplete_defs = False

[mypy-arviz.tests.external_tests.test_data_pymc.*]
check_untyped_defs = False
disallow_incomplete_defs = False

[mypy-arviz.tests.external_tests.test_data_pyro.*]
check_untyped_defs = False
disallow_incomplete_defs = False

[mypy-arviz.tests.external_tests.test_data_pystan.*]
check_untyped_defs = False
disallow_incomplete_defs = False

[mypy-arviz.tests.helpers.*]
check_untyped_defs = False
disallow_incomplete_defs = False

[mypy-arviz.utils.*]
check_untyped_defs = False
disallow_incomplete_defs = False

[mypy-asv_benchmarks.benchmarks.*]
disallow_untyped_calls = False
disallow_untyped_defs = False

[mypy-doc.*]
check_untyped_defs = False
disallow_incomplete_defs = False
disallow_untyped_calls = False
disallow_untyped_defs = False

[mypy-examples.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_autocorr.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_bpv.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_bpv_tstat.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_compare.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_density.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_dist.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_elpd.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_energy.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_ess_evolution.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_ess_local.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_ess_quantile.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_forest.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_forest_ridge.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_hdi.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_joint.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_kde.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_kde_2d.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_kde_2d_bis.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_kde_quantiles.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_khat.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_loo_pit_ecdf.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_loo_pit_overlay.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_mcse.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_mcse_errorbar.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_pair.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_pair_hex.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_pair_kde.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_pair_point_estimate.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_parallel.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_parallel_minmax.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_parallel_normal.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_parallel_rank.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_posterior.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_ppc.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_ppc_cumulative.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_rank.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_separation.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_trace.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_trace_bars.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_trace_circ.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_trace_vlines.*]
disallow_untyped_calls = False

[mypy-examples.matplotlib.mpl_plot_violin.*]
disallow_untyped_calls = False

[mypy-setup.*]
disallow_incomplete_defs = False
disallow_untyped_calls = False
disallow_untyped_defs = False


# Third-party module rule relaxations
[mypy-IPython.*]
ignore_missing_imports = True

[mypy-aesara.*]
ignore_missing_imports = True

[mypy-bokeh.*]
ignore_missing_imports = True

[mypy-cmdstanpy.*]
ignore_missing_imports = True

[mypy-emcee.*]
ignore_missing_imports = True

[mypy-jax.*]
ignore_missing_imports = True

[mypy-matplotlib.*]
ignore_missing_imports = True

[mypy-mpl_toolkits.axes_grid1.*]
ignore_missing_imports = True

[mypy-netCDF4.*]
ignore_missing_imports = True

[mypy-numba.*]
ignore_missing_imports = True

[mypy-numpy.*]
ignore_missing_imports = True

[mypy-numpyro.*]
ignore_missing_imports = True

[mypy-pandas.*]
ignore_missing_imports = True

[mypy-pyjags.*]
ignore_missing_imports = True

[mypy-pymc3.*]
ignore_missing_imports = True

[mypy-pyro.*]
ignore_missing_imports = True

[mypy-pystan.*]
ignore_missing_imports = True

[mypy-scipy.*]
ignore_missing_imports = True

[mypy-setuptools.*]
ignore_missing_imports = True

[mypy-stan.*]
ignore_missing_imports = True

[mypy-tensorflow.*]
ignore_missing_imports = True

[mypy-tensorflow_probability.*]
ignore_missing_imports = True

[mypy-theano.*]
ignore_missing_imports = True

[mypy-zarr.*]
ignore_missing_imports = True

[hauntsaninja]

mypy doesn't treat asv_benchmarks as a package. Use --explicit-package-base or add an __init__.py, see https://mypy.readthedocs.io/en/latest/running_mypy.html#mapping-file-paths-to-modules

[obi1kenobi]

mypy doesn't treat asv_benchmarks as a package. Use --explicit-package-base or add an __init__.py, see https://mypy.readthedocs.io/en/latest/running_mypy.html#mapping-file-paths-to-modules

Thanks for taking a look! Unfortunately, I'm not sure I understand what you mean -- are you saying that while mypy finds type errors in asv_benchmarks, it will not consider mypy.ini configuration that is intended to apply to that directory?

I'm not the maintainer of the arviz repo, but don't think asv_benchmarks or the other affected directories are intended to be packages. While type-checking the code there is desirable, there are obviously some errors that I'd like to suppress for now. Is there a way to tell mypy via mypy.ini to ignore those error codes for the affected directories?

[hauntsaninja]

Haven't checked out the code, but likely:

[mypy-benchmarks.*]
disallow_untyped_calls = False
disallow_untyped_defs = False