Creating storage container fails with 500 Operation could not be completed within the specified time

[ondrejhlavacek]

Which service(blob, file, queue, table) does this issue concern?

Blob

Which version of the SDK was used?

microsoft/azure-storage-common 1.5.1
microsoft/azure-storage-blob 1.5.0

What's the PHP/OS version?

7.4.27-apache-buster

What problem was encountered?

Server error: PUT https:\/\/***.blob.core.windows.net\/***?restype=container resulted in a 500 Operation could not be completed within the specified time. response

Steps to reproduce the issue?

Happens randomly when creating containers.

Have you found a mitigation/solution?

No

Is there a failing request ID related to this problem returned by server? What is it?

e16ee727-e01e-0081-7521-12dd10000000

What is the storage account name and time frame of your last reproduce? (UTC YYYY/MM/DD hh:mm:ss)

Jan 25 20:27:38 CET, will disclose the storage account name on request.

(If you think some of the information should not be shared publicly, you can e-mail the main Microsoft contributors of the repository instead.)

{
    "MicrosoftAzure\\Storage\\Common\\Exceptions\\ServiceExceptionresponse": {
        "GuzzleHttp\\Psr7\\ResponsereasonPhrase": "Operation could not be completed within the specified time.",
        "GuzzleHttp\\Psr7\\ResponsestatusCode": 500,
        "GuzzleHttp\\Psr7\\Responseheaders": {
            "Content-Length": [
                "246"
            ],
            "Content-Type": [
                "application\/xml"
            ],
            "Server": [
                "Windows-Azure-Blob\/1.0 Microsoft-HTTPAPI\/2.0"
            ],
            "x-ms-request-id": [
                "***"
            ],
            "Date": [
                "Mon, 24 Jan 2022 19:11:55 GMT"
            ]
        },
        "GuzzleHttp\\Psr7\\ResponseheaderNames": {
            "content-length": "Content-Length",
            "content-type": "Content-Type",
            "server": "Server",
            "x-ms-request-id": "x-ms-request-id",
            "date": "Date"
        },
        "GuzzleHttp\\Psr7\\Responseprotocol": "1.1",
        "GuzzleHttp\\Psr7\\Responsestream": {
            "GuzzleHttp\\Psr7\\Streamstream": "... cannot dump ...",
            "GuzzleHttp\\Psr7\\Streamsize": 246,
            "GuzzleHttp\\Psr7\\Streamseekable": true,
            "GuzzleHttp\\Psr7\\Streamreadable": true,
            "GuzzleHttp\\Psr7\\Streamwritable": true,
            "GuzzleHttp\\Psr7\\Streamuri": "php:\/\/temp",
            "GuzzleHttp\\Psr7\\StreamcustomMetadata": []
        }
    },
    "MicrosoftAzure\\Storage\\Common\\Exceptions\\ServiceExceptionerrorText": "Operation could not be completed within the specified time.",
    "MicrosoftAzure\\Storage\\Common\\Exceptions\\ServiceExceptionerrorMessage": "Operation could not be completed within the specified time.\nRequestId:***\nTime:2022-01-24T19:11:55.9249322Z",
    "*message": "Fail:\nCode: 500\nValue: Operation could not be completed within the specified time.\ndetails (if any): \ufeff\u003C?xml version=\u00221.0\u0022 encoding=\u0022utf-8\u0022?\u003E\n\u003CError\u003E\u003CCode\u003EOperationTimedOut\u003C\/Code\u003E\u003CMessage\u003EOperation could not be completed within the specified time.\nRequestId:9411fe51-e01e-0081-4956-11dd10000000\nTime:2022-01-24T19:11:55.9249322Z\u003C\/Message\u003E\u003C\/Error\u003E.",
    "Exceptionstring": "",
    "*code": 500,
    "*file": "\/var\/www\/html\/legacy-app\/vendor\/microsoft\/azure-storage-common\/src\/Common\/Internal\/ServiceRestProxy.php",
    "*line": 495,
    "Exceptiontrace": [
        {
            "file": "\/var\/www\/html\/legacy-app\/vendor\/microsoft\/azure-storage-common\/src\/Common\/Internal\/ServiceRestProxy.php",
            "line": 411,
            "function": "throwIfError",
            "class": "MicrosoftAzure\\Storage\\Common\\Internal\\ServiceRestProxy",
            "type": "::",
            "args": [
                {
                    "GuzzleHttp\\Psr7\\ResponsereasonPhrase": "Operation could not be completed within the specified time.",
                    "GuzzleHttp\\Psr7\\ResponsestatusCode": 500,
                    "GuzzleHttp\\Psr7\\Responseheaders": {
                        "Content-Length": "...",
                        "Content-Type": "...",
                        "Server": "...",
                        "x-ms-request-id": "...",
                        "Date": "..."
                    },
                    "GuzzleHttp\\Psr7\\ResponseheaderNames": {
                        "content-length": "Content-Length",
                        "content-type": "Content-Type",
                        "server": "Server",
                        "x-ms-request-id": "x-ms-request-id",
                        "date": "Date"
                    },
                    "GuzzleHttp\\Psr7\\Responseprotocol": "1.1",
                    "GuzzleHttp\\Psr7\\Responsestream": {
                        "GuzzleHttp\\Psr7\\Streamstream": "... cannot dump ...",
                        "GuzzleHttp\\Psr7\\Streamsize": 246,
                        "GuzzleHttp\\Psr7\\Streamseekable": true,
                        "GuzzleHttp\\Psr7\\Streamreadable": true,
                        "GuzzleHttp\\Psr7\\Streamwritable": true,
                        "GuzzleHttp\\Psr7\\Streamuri": "php:\/\/temp",
                        "GuzzleHttp\\Psr7\\StreamcustomMetadata": "..."
                    }
                },
                201
            ]
        },
        {
            "file": "\/var\/www\/html\/legacy-app\/vendor\/guzzlehttp\/promises\/src\/Promise.php",
            "line": 204,
            "function": "MicrosoftAzure\\Storage\\Common\\Internal\\{closure}",
            "class": "MicrosoftAzure\\Storage\\Common\\Internal\\ServiceRestProxy",
            "type": "-\u003E",
            "args": [
                {
                    "GuzzleHttp\\Exception\\RequestExceptionrequest": {
                        "GuzzleHttp\\Psr7\\Requestmethod": "PUT",
                        "GuzzleHttp\\Psr7\\RequestrequestTarget": "... cannot dump ...",
                        "GuzzleHttp\\Psr7\\Requesturi": "...",
                        "GuzzleHttp\\Psr7\\Requestheaders": "...",
                        "GuzzleHttp\\Psr7\\RequestheaderNames": "...",
                        "GuzzleHttp\\Psr7\\Requestprotocol": "1.1",
                        "GuzzleHttp\\Psr7\\Requeststream": "..."
                    },
                    "GuzzleHttp\\Exception\\RequestExceptionresponse": {
                        "GuzzleHttp\\Psr7\\ResponsereasonPhrase": "Operation could not be completed within the specified time.",
                        "GuzzleHttp\\Psr7\\ResponsestatusCode": 500,
                        "GuzzleHttp\\Psr7\\Responseheaders": "...",
                        "GuzzleHttp\\Psr7\\ResponseheaderNames": "...",
                        "GuzzleHttp\\Psr7\\Responseprotocol": "1.1",
                        "GuzzleHttp\\Psr7\\Responsestream": "..."
                    },
                    "GuzzleHttp\\Exception\\RequestExceptionhandlerContext": [],
                    "*message": "Server error: `PUT https:\/\/***.blob.core.windows.net\/***?restype=container` resulted in a `500 Operation could not be completed within the specified time.` response",
                    "Exceptionstring": "",
                    "*code": 500,
                    "*file": "\/var\/www\/html\/legacy-app\/vendor\/guzzlehttp\/guzzle\/src\/Exception\/RequestException.php",
                    "*line": 113,
                    "Exceptiontrace": [
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "..."
                    ],
                    "Exceptionprevious": "... cannot dump ..."
                }
            ]
        },
        {
            "file": "\/var\/www\/html\/legacy-app\/vendor\/guzzlehttp\/promises\/src\/Promise.php",
            "line": 169,
            "function": "callHandler",
            "class": "GuzzleHttp\\Promise\\Promise",
            "type": "::",
            "args": [
                2,
                {
                    "GuzzleHttp\\Exception\\RequestExceptionrequest": {
                        "GuzzleHttp\\Psr7\\Requestmethod": "PUT",
                        "GuzzleHttp\\Psr7\\RequestrequestTarget": "... cannot dump ...",
                        "GuzzleHttp\\Psr7\\Requesturi": "...",
                        "GuzzleHttp\\Psr7\\Requestheaders": "...",
                        "GuzzleHttp\\Psr7\\RequestheaderNames": "...",
                        "GuzzleHttp\\Psr7\\Requestprotocol": "1.1",
                        "GuzzleHttp\\Psr7\\Requeststream": "..."
                    },
                    "GuzzleHttp\\Exception\\RequestExceptionresponse": {
                        "GuzzleHttp\\Psr7\\ResponsereasonPhrase": "Operation could not be completed within the specified time.",
                        "GuzzleHttp\\Psr7\\ResponsestatusCode": 500,
                        "GuzzleHttp\\Psr7\\Responseheaders": "...",
                        "GuzzleHttp\\Psr7\\ResponseheaderNames": "...",
                        "GuzzleHttp\\Psr7\\Responseprotocol": "1.1",
                        "GuzzleHttp\\Psr7\\Responsestream": "..."
                    },
                    "GuzzleHttp\\Exception\\RequestExceptionhandlerContext": [],
                    "*message": "Server error: `PUT https:\/\/***.blob.core.windows.net\/***?restype=container` resulted in a `500 Operation could not be completed within the specified time.` response",
                    "Exceptionstring": "",
                    "*code": 500,
                    "*file": "\/var\/www\/html\/legacy-app\/vendor\/guzzlehttp\/guzzle\/src\/Exception\/RequestException.php",
                    "*line": 113,
                    "Exceptiontrace": [
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "..."
                    ],
                    "Exceptionprevious": "... cannot dump ..."
                },
                "... cannot dump ..."
            ]
        },
        {
            "file": "\/var\/www\/html\/legacy-app\/vendor\/guzzlehttp\/promises\/src\/RejectedPromise.php",
            "line": 42,
            "function": "GuzzleHttp\\Promise\\{closure}",
            "class": "GuzzleHttp\\Promise\\Promise",
            "type": "::",
            "args": [
                {
                    "GuzzleHttp\\Exception\\RequestExceptionrequest": {
                        "GuzzleHttp\\Psr7\\Requestmethod": "PUT",
                        "GuzzleHttp\\Psr7\\RequestrequestTarget": "... cannot dump ...",
                        "GuzzleHttp\\Psr7\\Requesturi": "...",
                        "GuzzleHttp\\Psr7\\Requestheaders": "...",
                        "GuzzleHttp\\Psr7\\RequestheaderNames": "...",
                        "GuzzleHttp\\Psr7\\Requestprotocol": "1.1",
                        "GuzzleHttp\\Psr7\\Requeststream": "..."
                    },
                    "GuzzleHttp\\Exception\\RequestExceptionresponse": {
                        "GuzzleHttp\\Psr7\\ResponsereasonPhrase": "Operation could not be completed within the specified time.",
                        "GuzzleHttp\\Psr7\\ResponsestatusCode": 500,
                        "GuzzleHttp\\Psr7\\Responseheaders": "...",
                        "GuzzleHttp\\Psr7\\ResponseheaderNames": "...",
                        "GuzzleHttp\\Psr7\\Responseprotocol": "1.1",
                        "GuzzleHttp\\Psr7\\Responsestream": "..."
                    },
                    "GuzzleHttp\\Exception\\RequestExceptionhandlerContext": [],
                    "*message": "Server error: `PUT https:\/\/***.blob.core.windows.net\/***?restype=container` resulted in a `500 Operation could not be completed within the specified time.` response",
                    "Exceptionstring": "",
                    "*code": 500,
                    "*file": "\/var\/www\/html\/legacy-app\/vendor\/guzzlehttp\/guzzle\/src\/Exception\/RequestException.php",
                    "*line": 113,
                    "Exceptiontrace": [
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "...",
                        "..."
                    ],
                    "Exceptionprevious": "... cannot dump ..."
                }
            ]
        },
        {
            "file": "\/var\/www\/html\/legacy-app\/vendor\/guzzlehttp\/promises\/src\/TaskQueue.php",
            "line": 48,
            "function": "GuzzleHttp\\Promise\\{closure}",
            "class": "GuzzleHttp\\Promise\\RejectedPromise",
            "type": "::",
            "args": []
        },
        {
            "file": "\/var\/www\/html\/legacy-app\/vendor\/guzzlehttp\/guzzle\/src\/Handler\/CurlMultiHandler.php",
            "line": 118,
            "function": "run",
            "class": "GuzzleHttp\\Promise\\TaskQueue",
            "type": "-\u003E",
            "args": []
        },
        {
            "file": "\/var\/www\/html\/legacy-app\/vendor\/guzzlehttp\/guzzle\/src\/Handler\/CurlMultiHandler.php",
            "line": 145,
            "function": "tick",
            "class": "GuzzleHttp\\Handler\\CurlMultiHandler",
            "type": "-\u003E",
            "args": []
        },
        {
            "file": "\/var\/www\/html\/legacy-app\/vendor\/guzzlehttp\/promises\/src\/Promise.php",
            "line": 248,
            "function": "execute",
            "class": "GuzzleHttp\\Handler\\CurlMultiHandler",
            "type": "-\u003E",
            "args": [
                true
            ]
        },
        {
            "file": "\/var\/www\/html\/legacy-app\/vendor\/guzzlehttp\/promises\/src\/Promise.php",
            "line": 224,
            "function": "invokeWaitFn",
            "class": "GuzzleHttp\\Promise\\Promise",
            "type": "-\u003E",
            "args": []
        },
        {
            "file": "\/var\/www\/html\/legacy-app\/vendor\/guzzlehttp\/promises\/src\/Promise.php",
            "line": 269,
            "function": "waitIfPending",
            "class": "GuzzleHttp\\Promise\\Promise",
            "type": "-\u003E",
            "args": []
        },
        {
            "file": "\/var\/www\/html\/legacy-app\/vendor\/guzzlehttp\/promises\/src\/Promise.php",
            "line": 226,
            "function": "invokeWaitList",
            "class": "GuzzleHttp\\Promise\\Promise",
            "type": "-\u003E",
            "args": []
        },
        {
            "file": "\/var\/www\/html\/legacy-app\/vendor\/guzzlehttp\/promises\/src\/Promise.php",
            "line": 62,
            "function": "waitIfPending",
            "class": "GuzzleHttp\\Promise\\Promise",
            "type": "-\u003E",
            "args": []
        },
        {
            "file": "\/var\/www\/html\/legacy-app\/vendor\/microsoft\/azure-storage-blob\/src\/Blob\/BlobRestProxy.php",
            "line": 922,
            "function": "wait",
            "class": "GuzzleHttp\\Promise\\Promise",
            "type": "-\u003E",
            "args": []
        },
        ...
    ],
    "Exceptionprevious": "... cannot dump ...",
    "exceptionClass": "MicrosoftAzure\\Storage\\Common\\Exceptions\\ServiceException",
    "previous": []
}

[meinemitternacht]

We have encountered this issue as well. We just wrap the request in retry logic with a delay and keep going.