TypeError: 'digest_size' is an invalid keyword argument for openssl_blake2b()

[bittner]

Environment

1. Operating system:

RedHat Linux inside an UBI8 Python 3.9 container image.

2. Version of python you are running:

python --version
Python 3.9.6

3. How did you install twine? Did you use your operating system's package manager or pip or something else?

Using tox, as follows, running tox -e readme:

[testenv:readme]
description = Ensure README renders on PyPI
deps =
    build
    twine
commands =
    {envpython} -m build
    {envpython} -m twine check dist/*

4. Version of twine you have installed (include complete output of):

twine --version
twine version 3.7.1 (importlib_metadata: 4.10.0, pkginfo: 1.8.2, requests:
2.27.1, requests-toolbelt: 0.9.1, tqdm: 4.62.3)

5. Other: (maybe relevant)

My pyproject.toml looks like this:

[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"

The Issue

...
readme run-test: commands[6] | /builds/devsecops/git-malware-scanner/.tox/readme/bin/python -m twine check 'dist/*'
Traceback (most recent call last):
  File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/builds/devsecops/git-malware-scanner/.tox/readme/lib/python3.9/site-packages/twine/__main__.py", line 53, in <module>
    sys.exit(main())
  File "/builds/devsecops/git-malware-scanner/.tox/readme/lib/python3.9/site-packages/twine/__main__.py", line 28, in main
    result = cli.dispatch(sys.argv[1:])
  File "/builds/devsecops/git-malware-scanner/.tox/readme/lib/python3.9/site-packages/twine/cli.py", line 71, in dispatch
    return main(args.args)
  File "/builds/devsecops/git-malware-scanner/.tox/readme/lib/python3.9/site-packages/twine/commands/check.py", line 191, in main
    return check(parsed_args.dists, strict=parsed_args.strict)
  File "/builds/devsecops/git-malware-scanner/.tox/readme/lib/python3.9/site-packages/twine/commands/check.py", line 135, in check
    warnings, is_ok = _check_file(filename, render_warning_stream)
  File "/builds/devsecops/git-malware-scanner/.tox/readme/lib/python3.9/site-packages/twine/commands/check.py", line 78, in _check_file
    package = package_file.PackageFile.from_filename(filename, comment=None)
  File "/builds/devsecops/git-malware-scanner/.tox/readme/lib/python3.9/site-packages/twine/package.py", line 128, in from_filename
    return cls(filename, comment, meta, py_version, dtype)
  File "/builds/devsecops/git-malware-scanner/.tox/readme/lib/python3.9/site-packages/twine/package.py", line 78, in __init__
    hasher = HashManager(filename)
  File "/builds/devsecops/git-malware-scanner/.tox/readme/lib/python3.9/site-packages/twine/package.py", line 261, in __init__
    self._blake_hasher = hashlib.blake2b(digest_size=256 // 8)
TypeError: 'digest_size' is an invalid keyword argument for openssl_blake2b()
Checking dist/git_malware_scanner-1.0.0-py3-none-any.whl: ERROR: InvocationError for command /builds/devsecops/git-malware-scanner/.tox/readme/bin/python -m twine check 'dist/*' (exited with code 1)

I verified that:

• Python uses the digest_size keyword argument (at least) since version 3.7
• The current and all older (tagged) versions of twine use blake2b(digest_size=...). It doesn't seem to be an version issue.

Looking at the error message, it's unclear to me why openssl_blake2b is mentioned (instead of just blake2b):

TypeError: 'digest_size' is an invalid keyword argument for openssl_blake2b()

Does this indicate that there is an incompatible library installed that takes precedence over the C-library that Python is meant to reference? Really just guessing. 🤔

Steps to Reproduce

1. Pull and run the Docker image registry.redhat.io/ubi8/python-39:latest
2. Prepare a tox.ini with the configuration shown above
3. Install tox (pip install tox) and run tox

[bittner]

Maybe related:

• https://github.com/pyca/cryptography/issues?q=digest_size
• https://github.com/pyca/cryptography/issues?q=openssl
• https://github.com/openssl/openssl/issues?q=digest_size

[bittner]

Potentially related code from CPython:

• https://github.com/python/cpython/blob/3.9/Modules/_blake2/blake2b_impl.c
• https://github.com/python/cpython/blob/3.9/Lib/hashlib.py

[sigmavirus24]

I haven't tried to reproduce but this definitely seems to be an issue with whatever distribution of python you're using. You should try to file a bug with whomever build the image you're trying to use

[bittner]

I opened Case #03126473 with RedHat support about the ubi8/python-39 container image.

[bittner]

RedHat has opened Bugzilla ticket #2041710 to track this case internally.

[bhrutledge]

It sounds like this is not an issue with Twine, so I'm going to close this. If that's incorrect, of if there's a related improvement that could be made to Twine, I'm happy to re-open it.

[frenzymadness]

Could we please reopen this issue. It seems that the problem is not in the twine but in our Python interpreter itself but it might be faster to discuss it here where we have all the details.

It seems to me that the problem is caused by using blake2b implementation from openssl. Python should prefer its own implementation of blake2b. The only possible reason I see for this situation is FIPS mode. Is there any chance that you have it enabled? Could you please try this:

(app-root) python
Python 3.9.6 (default, Aug 11 2021, 06:39:25) 
[GCC 8.5.0 20210514 (Red Hat 8.5.0-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import _hashlib
>>> _hashlib.get_fips_mode()
0

[frenzymadness]

I'm able to reproduce the issue. I'm using Python 3.8 because that's the newest I have on an FIPS-enabled machine. Unfortunately, the problem is not just the different API but also the fact, that BLAKE algorithms are not certified for FIPS.

# python3.8
Python 3.8.0 (default, Mar  9 2020, 18:02:46) 
[GCC 8.3.1 20191121 (Red Hat 8.3.1-5)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import _hashlib
>>> _hashlib.get_fips_mode()
1
>>> import hashlib
>>> hashlib.blake2b(digest_size=256 //8)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'digest_size' is an invalid keyword argument for openssl_blake2b()
>>> hashlib.blake2b()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS

[frenzymadness]

Would it make sense to catch also TypeError here?

twine/twine/package.py

Lines 268 to 273 in f69d4b7

	self._blake_hasher = None
	try:
	self._blake_hasher = hashlib.blake2b(digest_size=256 // 8)
	except ValueError:
	# FIPS mode disables blake2
	pass

I know, it would be better to raise the same exception but the check of the arguments happens sooner.

[bhrutledge]

Would it make sense to catch also TypeError here?

@frenzymadness I think that makes sense. Thanks for the digging into this. Would you be game to open a PR?

[frenzymadness]

Sure, on it.

[bittner]

I can confirm that we have FIPS enabled. This causes other apparently related issues, e.g. when using the UBI8 Python 3.9 image mentioned above for Django Web development:

...
  File "/opt/app-root/lib64/python3.9/site-packages/django/db/backends/utils.py", line 218, in names_digest
    h = hashlib.md5()
ValueError: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS

[frenzymadness]

Yeah, a lot of upstream projects are not ready for FIPS and it's hard to implement the support for them because the FIPS mode is not that commonly used.

[sigmavirus24]

And FIPS causes other problems just because updates to the compliant/audited code is near impossible even for security patches

[frenzymadness]

I'm not sure I understand your point and also I'm not an expert in this field but if we (Red Hat) provide a Python package that is FIPS-compliant, we also provide security fixes for that package for the whole support lifespan. Upstream projects like Django are not certified for FIPS so it depends on who provides the software to you.

[sigmavirus24]

There are entire branches of openssl that are frozen in time - vulnerabilities and all - because they were audited for FIPS and the code cannot change without compliance changing (to the best of my understanding). It's an expensive process to go through and it's time-consuming. Things built a-top FIPS certified code can move much faster because that code hasn't been audited for large sums of money. Twine, Python, etc. can be updated just fine.