Skip to content
This repository has been archived by the owner on Jul 31, 2024. It is now read-only.

Fix ResourceWarning from unclosed files in ichnaea tests #1393

Closed
jwhitlock opened this issue Oct 27, 2020 · 2 comments
Closed

Fix ResourceWarning from unclosed files in ichnaea tests #1393

jwhitlock opened this issue Oct 27, 2020 · 2 comments
Labels
dev local dev environment, dev scaffolding/infrastructure, etc

Comments

@jwhitlock
Copy link
Member

With warnings enabled (make testshell, and pytest -Wd ichnaea), there are warnings about unclosed files:

ichnaea/api/submit/tests/test_submit_v1.py::TestView::test_batches
  /app/ichnaea/api/submit/tests/test_submit_v1.py:240: ResourceWarning: unclosed file <_io.BufferedRandom name=14>
    {

ichnaea/api/submit/tests/test_submit_v2.py::TestView::test_batches
  /usr/local/lib/python3.8/site-packages/sqlalchemy/event/attr.py:319: ResourceWarning: unclosed file <_io.BufferedRandom name=14>
    for fn in self.parent_listeners:

ichnaea/content/tests/test_views.py::TestFunctionalContent::test_content[/stats/regions-200-stats.regions-1]
  /usr/local/lib/python3.8/site-packages/chameleon/astutil.py:48: ResourceWarning: unclosed file <_io.BufferedRandom name=14>
    return compile(source, '', mode, ast.PyCF_ONLY_AST)

Further research is needed to determine if this is a issue with test setup and teardown, or if this may be causing problems in production as well. This stack overflow answer has some tips for investigating this issue.

Originally filed as bug 1650870 on 2020-07-06

@jwhitlock jwhitlock added this to the 2020 Holiday Wishlist milestone Oct 29, 2020
@jwhitlock jwhitlock added the dev local dev environment, dev scaffolding/infrastructure, etc label Oct 29, 2020
@jwhitlock
Copy link
Member Author

With platform linux -- Python 3.9.1, pytest-6.2.2, py-1.9.0, pluggy-0.13.1:

ichnaea/api/submit/tests/test_submit_v1.py::TestView::test_batches
  /app/ichnaea/api/submit/tests/test_submit_v1.py:243: ResourceWarning: unclosed file <_io.BufferedRandom name=14>
    "wifiAccessPoints": [{"macAddress": wifi.mac}],

ichnaea/api/submit/tests/test_submit_v2.py::TestView::test_batches
  /app/ichnaea/api/schema.py:106: ResourceWarning: unclosed file <_io.BufferedRandom name=14>
    result = super(RenamingMapping, self)._impl(node, *args, **kw)

ichnaea/webapp/tests.py::TestHeartbeatErrors::test_database
  /usr/local/lib/python3.9/site-packages/pyramid/viewderivers.py:163: ResourceWarning: unclosed file <_io.BufferedRandom name=15>
    wrapper_view = wrapper(view, info)

@jwhitlock
Copy link
Member Author

The Waitress 2.1.0 Release Notes include a note about closing bytestreams:

Fixed an issue whereby BytesIO objects were not properly closed, and thereby would not get cleaned up until garbage collection would get around to it.

This led to potential for random memory spikes/memory issues, see Pylons/waitress#358 and Pylons/waitress#357.

With thanks to Florian Schulze for testing/vaidating this fix!

Running with Waitress 2.1.1 from #1807 did not fix it:

> make testshell
...
$ pytest -Wd ichnaea
...
================================================ warnings summary ================================================
<frozen importlib._bootstrap>:914: 11 warnings
ichnaea/api/locate/tests/test_blue.py: 1 warning
ichnaea/api/locate/tests/test_locate_v1.py: 1 warning
ichnaea/content/tests/test_views.py: 1 warning
ichnaea/data/tests/test_area.py: 1 warning
ichnaea/scripts/tests/test_datamap.py: 1 warning
ichnaea/taskapp/tests.py: 1 warning
ichnaea/webapp/tests.py: 1 warning
  <frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()

ichnaea/api/submit/tests/test_submit_v1.py::TestView::test_batches
  /usr/local/lib/python3.10/json/decoder.py:353: ResourceWarning: unclosed file <_io.BufferedRandom name=14>
    obj, end = self.scan_once(s, idx)

ichnaea/api/submit/tests/test_submit_v2.py::TestView::test_batches
  /usr/local/lib/python3.10/json/decoder.py:353: ResourceWarning: unclosed file <_io.BufferedRandom name=15>
    obj, end = self.scan_once(s, idx)

ichnaea/data/tests/test_station.py::TestBlue::test_block_consistent_obs[ReportSource.query-ReportSource.gnss]
  /usr/local/lib/python3.10/site-packages/sqlalchemy/orm/query.py:4422: ResourceWarning: unclosed file <_io.BufferedRandom name=14>
    _instance = loading._instance_processor(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

I'm pretty sure the needed fix is in our code.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
dev local dev environment, dev scaffolding/infrastructure, etc
Projects
None yet
Development

No branches or pull requests

2 participants