--index-url Argument not recognized

[msachsenhauser]

I'm trying to use pip-audit with a corporate package-proxy using the --index-url argument stated in --help .

poetry export -f requirements.txt --output requirements.txt
poetry run pip-audit -r requirements.txt --index-url https://some.internal.url/pypi/pypi-virtual/simple --skip-editable --desc on
# the use of poetry doesn't make any difference

Unfortunatelly, it seems like pip-audit ignores --index-url and tries to access pypi.org directly (what is blocked by the corporate proxy). The provided package-index-address isn't mentioned in the protocol below.

Skipping virtualenv creation, as specified in config file.
Traceback (most recent call last):
File "C:\path\to\venv\lib\site-packages\urllib3\connection.py", line 174, in _new_conn
conn = connection.create_connection(
File "C:\path\to\venv\lib\site-packages\urllib3\util\connection.py", line 95, in create_connection
raise err
File "C:\path\to\venv\lib\site-packages\urllib3\util\connection.py", line 85, in create_connection
sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\path\to\venv\lib\site-packages\urllib3\connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "C:\path\to\venv\lib\site-packages\urllib3\connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "C:\path\to\venv\lib\site-packages\urllib3\connectionpool.py", line 1040, in _validate_conn
conn.connect()
File "C:\path\to\venv\lib\site-packages\urllib3\connection.py", line 358, in connect
conn = self._new_conn()
File "C:\path\to\venv\lib\site-packages\urllib3\connection.py", line 179, in _new_conn
raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPSConnection object at 0x000001DB1C111550>, 'Connection to pypi.org timed out. (connect timeout=15)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\path\to\venv\lib\site-packages\requests\adapters.py", line 440, in send
resp = conn.urlopen(
File "C:\path\to\venv\lib\site-packages\urllib3\connectionpool.py", line 785, in urlopen
retries = retries.increment(
File "C:\path\to\venv\lib\site-packages\urllib3\util\retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/et-xmlfile/1.1.0/json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x000001DB1C11
1550>, 'Connection to pypi.org timed out. (connect timeout=15)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\path\to\venv\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\path\to\venv\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "C:\path\to\venv\Scripts\pip-audit.exe_main.py", line 7, in
File "C:\path\to\venv\lib\site-packages\pip_audit_cli.py", line 403, in audit
for (spec, vulns) in auditor.audit(source):
File "C:\path\to\venv\lib\site-packages\pip_audit_audit.py", line 66, in audit
for dep, vulns in self._service.query_all(specs):
File "C:\path\to\venv\lib\site-packages\pip_audit_service\interface.py", line 149, in query_all
yield self.query(spec)
File "C:\path\to\venv\lib\site-packages\pip_audit_service\pypi.py", line 58, in query
response: requests.Response = self.session.get(url=url, timeout=self.timeout)
File "C:\path\to\venv\lib\site-packages\requests\sessions.py", line 542, in get
return self.request('GET', url, **kwargs)
File "C:\path\to\venv\lib\site-packages\requests\sessions.py", line 529, in request
resp = self.send(prep, **send_kwargs)
File "C:\path\to\venv\lib\site-packages\requests\sessions.py", line 645, in send
r = adapter.send(request, **kwargs)
File "C:\path\to\venv\lib\site-packages\cachecontrol\adapter.py", line 57, in send
resp = super(CacheControlAdapter, self).send(request, **kw)
File "C:\path\to\venv\lib\site-packages\requests\adapters.py", line 507, in send
raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/et-xmlfile/1.1.0/json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x000001DB1C
111550>, 'Connection to pypi.org timed out. (connect timeout=15)'))

Further information:

• The internal package-proxy was (and is) available and accessible (also the referenced package)
• pip and poetry are working flawlessly with the internal package index.

Python 3.9.7
pip-audit 2.3.1
requests 2.27.1
urllib3 1.26.8

[edited for style]

[woodruffw]

Thanks for the report!

What happens if you use --extra-index-url <YOUR-INTERNAL-INDEX>? That's the intended CLI option for configuring additional indices, although I would have expected --index-url to also work in this instance.

[msachsenhauser]

Thanks for the report!

What happens if you use --extra-index-url <YOUR-INTERNAL-INDEX>? That's the intended CLI option for configuring additional indices, although I would have expected --index-url to also work in this instance.

Thank you for the fast reply! This option doesn't work either!

[woodruffw]

Oh, I think I see what's happening here -- we are using your --index-url for the dependency resolution, but we fundamentally rely on PyPI.org itself for the vulnerability feed API (since it's not part of a simple mirror or the simple index format).

Could you try running your same command but with -s osv? That'll try the OSV vulnerability feed instead of PyPI's.

[woodruffw]

(That's definitely something we should explain in the pip-audit documentation -- until PyPI's JSON API is standardized, we're unfortunately dependent on the "canonical" package index for vulnerability information.)

[msachsenhauser]

The "-s osv" argument worked! Is it due to the JSON-API or else?

[msachsenhauser]

The idea behind your package is really great! Is there any downside to using the osv-feed? Can you recommend a comparisson or else?

[woodruffw]

Is it due to the JSON-API or else?

Yep, it's the JSON API. It's not included in mirrors and it's not used by pip (since there's no standard for it), but pip-audit requires it for vulnerability information.

I'll update the documentation to emphasize that --index-url and --extra-index-url only affect dependency resolution, not vulnerability lookup. I'll also tweak the error handling here to check for ConnectTimeout and offer some helpful advice.

Is there any downside to using the osv-feed? Can you recommend a comparisson or else?

PyPI's feed pulls from OSV, so the two should be mostly equivalent.

[msachsenhauser]

Thanks for the incredible fast resolution! Looking forward to checking my dependencies in the future ;-)

[woodruffw]

#287 will improve the error message here, and will include a tip to try a different vulnerability service.