CI: Flaky ResourceWarning('unclosed file <_io.BufferedRandom ...

[mroeschke]

Example:
https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=70478&view=logs&jobId=cb04ab33-336b-54ee-68fe-dcd3abde835b&j=cb04ab33-336b-54ee-68fe-dcd3abde835b&t=0558d00d-ff4a-5538-0de9-a0e7f449700f

Previous debugging attempt: #44634

Since it's a io.BufferedRandom, it should be a file handle opened in + mode: #44634 (comment)

This only appears when tm.assert_produces_warning is called since it's checking for extra warnings. It's likely coming from an optional dependency at this point, but unconfirmed.

[twoertwein]

maybe related #44761 (comment)

[mroeschke]

maybe related #44761 (comment)

Thanks, but I believe we've been seeing this error in the CI prior to that PR.

[twoertwein]

might be helpful for the next attempt at ResourceWarnings: my previous attempt at it #39047

[jbrockmendel]

Logging these warnings when they pop up

2021-12-15, #44908, py39GHA, test_dialect_conflict_delimiter, <_io.BufferedRandom name=13>, (pandas/tests/io/parser/conftest.py', 22)

[mroeschke]

Found a suspect from https://github.com/pandas-dev/pandas/runs/4540629821?check_suite_focus=true
PR: #44877
Commit: a59e076
Build: pytest (actions-38-locale.yaml, not slow and not network, language-pack-zh-hans xsel, zh_C

Think it's related to matplotlib/matplotlib#21646 (comment)

TLDR from the log: mpl-data/fonts/ttf/DejaVuSans.ttf was opened and not appeared closed at OS number 14

2021-12-15T06:06:54.3616928Z [gw1] PASSED pandas/tests/libs/test_hashtable.py::TestHashTable::test_lookup[False-Int64HashTable-int640] python      5650 runner   13r      REG                8,1   756072 1827313 /usr/share/miniconda/envs/pandas-dev/lib/python3.8/site-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans.ttf
2021-12-15T06:06:54.3620174Z python      5650 runner   14r      REG                8,1   756072 1827313 /usr/share/miniconda/envs/pandas-dev/lib/python3.8/site-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans.ttf

2021-12-15T06:57:29.3149376Z         if extra_warnings:
2021-12-15T06:57:29.3149999Z >           raise AssertionError(f"Caused unexpected warning(s): {repr(extra_warnings)}")
2021-12-15T06:57:29.3151905Z E           AssertionError: Caused unexpected warning(s): [('ResourceWarning', ResourceWarning('unclosed file <_io.BufferedRandom name=14>'), '/usr/share/miniconda/envs/pandas-dev/lib/python3.8/site-packages/openpyxl/descriptors/serialisable.py', 52)]

[mroeschke]

If matplotlib is related, I am not sure there's any way to avoid this besides blocking the reading of some font files.

After investigating, I think at import time this object is reading certain files and might be leaving them open: https://github.com/matplotlib/matplotlib/blob/f6e0ee49c598f59c6e6cf4eefe473e4dc634a58a/src/ft2font.cpp#L322

EDIT:

Narrowed it down to the matplotlib.ft2font.FT2Font object that appears to keep a file handle open
https://github.com/matplotlib/matplotlib/blob/ad4633fe90668287df3f8cd3317f23e7af97384b/src/ft2font_wrapper.cpp#L363

[mroeschke]

So far the response from matplotlib is "this is the intended behavior". Opened up matplotlib/matplotlib#22017 for a hopeful resolution to this issue.

I don't have many ideas how to reliably handle this case in the meantime

1. Avoid all font files with 'otf', 'ttc', 'ttf' extensions from being read. Matplotlib tests will still run.
2. Have matplotlib tests run last such that the matpliotlib import (and therefore font loading) is delayed until after all the tm.assert_produces_warning checks are done.

Anyone else have suggestions?

[jbrockmendel]

Anyone else have suggestions?

Brainstorming:

3. Ask @pganssle about where to file an issue (BPO?) or open discussion about making these easier to track down
4. Open an issue with psutil about getting proc.open_files to identify these objects
5. IIUC the warning is issued when __del__ is called before close. Could tinker with gc.collect to
   a) raise instead of warn?
   b) disable collection and maybe then proc.open_files will find them if there's a race condition?
6. Run a few hundred builds without matplotlib installed at all and see if we can confirm the hypothesis these files are the source of the problem
   a) An automated way to "run a few hundred builds and collect the results" would be helpful for many of the CI troubles.
7. Throw in the towel and have tm.assert_produces_warning ignore ResourceWarnings
   a) Could have them ignore only if matplotlib is found in sys.modules?

[jbrockmendel]

Another idea:

Add a check in assert_produces_warning along the lines of if "--ignore-resource-warning in sys.argv[1:]": [...] and then use that in some or all of the CI runs.