HTTP 502 errors not causing exception and retry

[gefair]

I am trying to use AWS CLI through a proxy. Things are generally working.

Occasionally, this proxy will return a 502 error, which does not get translated into an exception in the call to http_session.send() from _get_response() in botocore\endpoint.py. Therefore, the retry mechanisms never get attempted.

It appears that a call to http_response.raise_for_status() needs to be made immediately after the send() call. However, even if I add that to the code, the resulting exception doesn't get re-tried for reasons I don't fully understand. The exception ends up getting re-throw in ExceptionRaiser _check_caught_exception() in botocore\retryhandler.py

Am I doing something wrong? Does my raise_for_status() fix make sense, and if so what additional code changes or config edit are needed so that the HTTPStatusCodeChecker causes a retry?

[rayluo]

Issue confirmed. It is because CLI only retries for these HTTP errors: 400, 500, 503, 509.
A quick workaround is to add the 502 error into your botocore/data/_retry.json like this:

$ git diff
diff --git a/botocore/data/_retry.json b/botocore/data/_retry.json
index 90eae9b..d0a94ba 100644
--- a/botocore/data/_retry.json
+++ b/botocore/data/_retry.json
@@ -28,6 +28,13 @@
         }
       }
     },
+    "bad_gateway": {
+      "applies_when": {
+        "response": {
+          "http_status_code": 502
+        }
+      }
+    },
     "service_unavailable": {
       "applies_when": {
         "response": {
@@ -54,6 +61,7 @@
       "policies": {
           "general_socket_errors": {"$ref": "general_socket_errors"},
           "general_server_error": {"$ref": "general_server_error"},
+          "bad_gateway": {"$ref": "bad_gateway"},
           "service_unavailable": {"$ref": "service_unavailable"},
           "limit_exceeded": {"$ref": "limit_exceeded"},
           "throttling_exception": {"$ref": "throttling_exception"},

We may consider to add this into our code base too.

[gefair]

Thank you for fast response.

This fix worked with one code change. I started with a clean enlistment (I reverted my http_response.raise_for_status() edit I mentioned in my original post.) The one code change I did need was to change the "except ResponseParserError as e:" --> "except Exception as e:" on line 687 in parsers.py. The reason for this change was the 502 response was accompanied with an XML-looking HTML response body from our proxy.

I'm including the response body and resulting stack trace it caused. I'm not sure if you'd prefer to fix as I did, or add a separate except block for the catch-all.

502 error from proxy.txt
parser stack trace.txt

Lastly, I'd like to comment that when errors like these occurs, the multi-part-upload gets orphaned. It isn't obvious to the user and the user gets billed for the uploaded, pending parts. It seems AWSCLI should abort() the multi-part-upload on any failures.

Also, some way to view/clear pending multi-part uploads from AWSCLI would be a great feature.

Shall I open a new issue for this?

[jamesls]

I believe the two PRs referenced above should fix these issues. Closing out issue. Let us know if you're still seeing issues.