Missing dependencies when inspect parsers using Python 3.10

[shzhou12]

# cat /root/insights-vagrant-host-20180406174409/etc/redhat-release 
Red Hat Enterprise Linux Server release 7.4 (Maipo)

# insights inspect insights.parsers.redhat_release.RedhatRelease /root/insights-vagrant-host-20180406174409/ -D
DEBUG:insights.core.dr:Importing insights.specs.default
DEBUG:insights.core.dr:Importing insights.specs.insights_archive
DEBUG:insights.core.dr:Importing insights.specs.core3_archive
DEBUG:insights.core.dr:Importing insights.specs.sos_archive
DEBUG:insights.core.dr:Importing insights.specs.jdr_archive
DEBUG:insights.core.dr:Importing insights.specs.must_gather_archive
INFO:insights.core.dr:Trying insights.specs.Specs.redhat_release
DEBUG:insights.core.dr:insights.specs.Specs.redhat_release missing requirements All: [] Any: []
INFO:insights.core.dr:Trying insights.parsers.redhat_release.RedhatRelease
DEBUG:insights.core.dr:insights.parsers.redhat_release.RedhatRelease missing requirements All: ['insights.specs.Specs.redhat_release'] Any: 
Missing Dependencies:
    Requires:
        insights.specs.Specs.redhat_release

[xiangce]

It looks more like the next python version that needs to be supported by the framework should be Python 3.9, but not 3.10.

In RHEL 9, Python 3.9 is the default Python implementation. The unversioned python command points to the default Python 3.9 version.

• Chapter 2. Installing and using Python Red Hat Enterprise Linux 9-beta

Anyway, it is a good point to discuss.

[jholecek-rh]

This bug is most probably caused by the MutableMapping class being moved from collections to collections.abc. I've seen some warnings about that in out tests, but don't see them now. Not sure what changed on our side or in Python.

from insights.specs.default import DefaultSpecs works in Python 3.9 and older but fails in Python 3.10 with

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "./.venv-py3.10/lib64/python3.10/site-packages/insights/specs/default.py", line 34, in <module>
    from insights.specs.datasources import (
  File "./.venv-py3.10/lib64/python3.10/site-packages/insights/specs/datasources/ssl_certificate.py", line 6, in <module>
    from insights.combiners.nginx_conf import NginxConfTree
  File "./.venv-py3.10/lib64/python3.10/site-packages/insights/combiners/nginx_conf.py", line 7, in <module>
    from insights.parsers.nginx_conf import NginxConfPEG
  File "./.venv-py3.10/lib64/python3.10/site-packages/insights/parsers/nginx_conf.py", line 10, in <module>
    from insights.contrib.nginxparser import create_parser, UnspacedList
  File "./.venv-py3.10/lib64/python3.10/site-packages/insights/contrib/nginxparser.py", line 25, in <module>
    from insights.contrib.pyparsing import (
  File "./.venv-py3.10/lib64/python3.10/site-packages/insights/contrib/pyparsing.py", line 714, in <module>
    collections.MutableMapping.register(ParseResults)
AttributeError: module 'collections' has no attribute 'MutableMapping'

[jholecek-rh]

I suppose insights.contrib.pyparsing is kept for compatibility with Python 2.7. The module uses collections.MutableMapping and collections.Sequence, both of which have been moved to collections.abc quite some time ago and eventually removed from collections in Python 3.10.

A conditional import at https://github.com/RedHatInsights/insights-core/blob/master/insights/contrib/pyparsing.py#L72 should do the trick

if system_version[:2] >= (3,10):
    import collections.abc as collections
else:
   import collections

This is as far as I can go. I've been unable to run all insights-core tests successfully even with Python 3.6 so far and I have no interest in playing with Python 2.

[ryan-blakley]

I created and linked a PR that should fix this issue, if everyone could review it.

[jholecek-rh]

Thanks for the proper fix, @ryan-blakley. Looks good to me.

I created #3408 to discuss options to improve our ability to troubleshoot issues like this. If I had not tried importing insights.specs.default manually, I would never know what caused this issue.