Twine doesn't fail when server return HTTP code 403 on upload

[usinelogicielle]

Your Environment

Thank you for taking the time to report an issue.

To more efficiently resolve this issue, we'd like to know some basic information about your system and setup.

1. Your operating system:
   Manjaro 21.1.6 Pahvo

2. Version of python you are running:
   Python 3.9.7

3. How did you install twine? Did you use your operating system's package manager or pip or something else?
   I use pip

4. Version of twine you have installed (include complete output of):
   Twine 3.7.1

5. Which package repository are you targeting?
   enterprise repository based on Jfrog Artifactory behind a WAF.

If you're having issues uploading a specific package, you must include a copy of the following:

• The package's PKG-INFO file
• A redacted version of your .pypirc file (REMOVE ALL USERNAMES & PASSWORDS BEFORE UPLOADING)

PKG-INFO

Metadata-Version: 2.1
Name: python-artifactory-internet
Version: 1.0
Summary: Project example for building Python project with JFrog products
Home-page: https://github.com/jfrog/project-examples
Author: Testci
Author-email: [email protected]
License: UNKNOWN
Platform: UNKNOWN

UNKNOWN

no .pypirc file

The Issue

Please describe the issue that you are experiencing.

When we are on internet, our WAF blocks all binaries upload and return a HTTP code 403.

log :

$ twine upload --repository-url https://${REGISTRY_URL}/artifactory/api/pypi/PIP_REPO --username ${USERNAME} --password ${PASSWORD} dist/*
Uploading distributions to https:/URL/artifactory/api/pypi/PIP_REPO
Uploading python_artifactory_internet-1.0-py3-none-any.whl

  0%|          | 0.00/4.68k [00:00<?, ?B/s]
100%|██████████| 4.68k/4.68k [00:00<00:00, 11.4kB/s]
Uploading python_artifactory_internet-1.0.tar.gz

  0%|          | 0.00/3.90k [00:00<?, ?B/s]
100%|██████████| 3.90k/3.90k [00:00<00:00, 40.6kB/s]

The binaries are not uploaded to Artifactory but Twine does not fail.
Even with verbose option, we can't see any problem.

Steps to Reproduce

Difficult to reproduce because it's specific to our environment.
I don't know how to fail a download because it has a code HTTP 403

Is there a solution to make twine failed if the upload fail? Or display the HTTP response to see the 403 code?

[bhrutledge]

From looking at the relevant source, I'd be surprised the HTTP response is actually a 403. It seems pretty clear that Twine will generate an error if it receives an HTTP response that indicates an error:

twine/twine/utils.py

Lines 198 to 210 in ce87465

	try:
	response.raise_for_status()
	except requests.HTTPError as err:
	if not verbose:
	logger.warning(
	"Error during upload. "
	"Retry with the --verbose option for more details."
	)
	if response.text:
	logger.info("Content received from server:\n{}".format(response.text))
	raise err

I wonder if there's something in your WAF config that's mutating the response.

I do think Twine could be more helpful, e.g. by always showing the HTTP response when --verbose is used, instead of just errors. I might be able to tackle that within the next week, but until then, you could try using a debugger (e.g. python3 -m pdb -m twine upload ...) to see the actual response. Or, is there a way you could inspect the network traffic to see the request from Twine and the response?

[sigmavirus24]

When we are on internet, our WAF blocks all binaries upload and return a HTTP code 403.

It's definitely not returning a 403 as Brian already pointed out. There must be something between your WAF and twine that's causing that - perhaps a proxy. That proxy might be mutating the response, but anything that's a 400 or higher would cause us to display an error and halt execution as Brian pointed out.

[bhrutledge]

Closing as an environment-related issue, per #859 (comment)