Netrc file is accessed inside exception handler, trigger exception again

[pletnes]

Describe the bug

Sometimes, the .netrc file is unreadable due to permissions. This is handled in this code:

aiohttp/aiohttp/helpers.py

Line 226 in 6da0469

try:

However, inside the exception handler, this if statement accesses metadata about the path. I assume (but have not yet checked) that netrc_path.is_file() lists metadata of the contents of .netrc's parent directory. This triggers an unchecked exception which, in turn, is not handled by other packages which I'm using.

git blame says the surrounding code is from 5 years ago. I take it this is an edge case, not a new or common issue.

My proposed solution is to warn, but not raise exceptions, if this occurs. This seems to be in spirit with the rest of the code, when the .netrc file is missing.

To Reproduce

I will try to improve on the details here. Preliminary notes:

• Create a system (docker container?) with .netrc in an unreadable directory
• Perform `aiohttp

Expected behavior

I expect aiohttp to warn, but not fail, when .netrc is present. I had no idea aiohttp requires a readable .netrc file and the cloud environment I'm working in does not have one.

Logs/tracebacks

They're unreadable. I will try to make a simple reproduction example.

Python Version

$ python --version
Python 3.8.10

aiohttp Version

$ python -m pip show aiohttp
Name: aiohttp
Version: 3.8.4
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author: None
Author-email: None
License: Apache 2
Location: /local_disk0/.ephemeral_nfs/envs/pythonEnv-bcd1c14b-d199-4b6e-9c19-efcc8df3f6e5/lib/python3.8/site-packages
Requires: aiosignal, async-timeout, attrs, yarl, multidict, frozenlist, charset-normalizer
Required-by: adlfs

multidict Version

$ python -m pip show multidict
Name: multidict
Version: 6.0.4
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: [email protected]
License: Apache 2
Location: /local_disk0/.ephemeral_nfs/envs/pythonEnv-bcd1c14b-d199-4b6e-9c19-efcc8df3f6e5/lib/python3.8/site-packages
Requires: 
Required-by: yarl, aiohttp

yarl Version

$ python -m pip show yarl
Name: yarl
Version: 1.8.2
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl/
Author: Andrew Svetlov
Author-email: [email protected]
License: Apache 2
Location: /local_disk0/.ephemeral_nfs/envs/pythonEnv-bcd1c14b-d199-4b6e-9c19-efcc8df3f6e5/lib/python3.8/site-packages
Requires: multidict, idna
Required-by: aiohttp

OS

I don't quite know, as I am using a hosted cloud environment. Looks Ubuntu linux based:

$ cat /etc/issue
Ubuntu 20.04.5 LTS \n \l
$ uname -a
Linux 0519-075302-a208ww1k-10-177-253-11 5.4.0-1100-azure #106~18.04.1-Ubuntu SMP Mon Dec 12 21:49:35 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Related component

Client

Additional context

Databricks on MS Azure, inside a notebook, using the adlfs module to access files in Azure Blob Storage.

Code of Conduct

• I agree to follow the aio-libs Code of Conduct

[pletnes]

Apologies for not having a simple reproduction. I will try to create one.

[pletnes]

The start of the traceback is from pathlib.py, where it is trying to stat the file.

File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-bcd1c14b-d199-4b6e-9c19-efcc8df3f6e5/lib/python3.8/site-packages/aiohttp/helpers.py", line 233, in netrc_from_env
    client_logger.warning("Could not read .netrc file: %s", e)
  File "/usr/lib/python3.8/pathlib.py", line 1445, in is_file
    return False
  File "/usr/lib/python3.8/pathlib.py", line 1198, in stat
    return self._accessor.stat(self)
PermissionError: [Errno 13] Permission denied: '/root/.netrc'

[pletnes]

If I add this line to my python code, I do not encounter the bug. This is because the left side of the or operator evaluates to True, meaning netrc_path.is_file() never runs.

  os.environ["NETRC"] = 'workaround_for_aiohttp_bug'

As a user, this is a viable workaround if you encounter this issue.

[Dreamsorcerer]

Sounds reasonable, if you have the time to create a test in a PR (and ideally a fix), that'd be great. In the test, you can probably just monkeypatch the open() (or whatever call is involved) and have it raise the exception.

[jgosmann]

We ran into the same problem today. I might look into providing a PR as it should be a fairly trivial fix. Though, I'm not quite sure if something nicer than nesting try/catch blocks can be done there. 🤔

[pletnes]

Thanks for fixing @jgosmann 🥇