Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Blob client crash due to aiohttp Exception (ClientPayloadError: Response payload is not completed) #17974

Closed
gdippolito opened this issue Apr 12, 2021 · 10 comments
Closed

Blob client crash due to aiohttp Exception (ClientPayloadError: Response payload is not completed) #17974

gdippolito opened this issue Apr 12, 2021 · 10 comments
Assignees
@xiafu-msft @tasherif-msft
Labels
bug This issue requires a change to an existing behavior in the product in order to be resolved. Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. Service Attention Workflow: This issue is responsible by Azure service team. Storage Storage Service (Queues, Blobs, Files)

Comments

@gdippolito
Copy link

  • Package Name: azure-storage-blob
  • Package Version: 12.6.0
  • Operating System: Ubuntu 18.04
  • Python Version: 3.7.9

Describe the bug
We are using the azure-blob-storage python client to download blobs using the async client implementation.

From time to time our job will crash with the following exception:

aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed

To Reproduce

The failure does not happen consistently but it is quite sporadic (our job can download data for many hours without facing this problem)

Expected behavior

azure-storage-blob should be able to handle this exception and perform a retry?

Additional context

We are using aiohttp version 3.6.3

Full stack trace:

Traceback (most recent call last):
  File "/app/runfiles/azure_image_storage.py", line 57, in load_image
    loop=self._get_or_create_event_loop(),
  File "/app/python_runtime/python3/lib/python3.7/concurrent/futures/_base.py", line 435, in result
    return self.__get_result()
  File "/app/python_runtime/python3/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/app/runfiles/azure_image_storage.py", line 77, in _load_blob_async_into_byte_stream
    storage_stream_downloader = await blob_client.download_blob()
  File "/app/runfiles/pip-services/pip__azure-identity/azure/core/tracing/decorator_async.py", line 74, in wrapper_use_tracer
    return await func(*args, **kwargs)
  File "/app/runfiles/pip-services/pip__azure-storage-blob/azure/storage/blob/aio/_blob_client_async.py", line 361, in download_blob
    await downloader._setup()  # pylint: disable=protected-access
  File "/app/runfiles/pip-services/pip__azure-storage-blob/azure/storage/blob/aio/_download_async.py", line 225, in _setup
    self._response = await self._initial_request()
  File "/app/runfiles/pip-services/pip__azure-storage-blob/azure/storage/blob/aio/_download_async.py", line 271, in _initial_request
    **self._request_options)
  File "/app/runfiles/pip-services/pip__azure-storage-blob/azure/storage/blob/_generated/aio/operations_async/_blob_operations_async.py", line 174, in download
    pipeline_response = await self._client._pipeline.run(request, stream=True, **kwargs)
  File "/app/runfiles/pip-services/pip__azure-identity/azure/core/pipeline/_base_async.py", line 215, in run
    return await first_node.send(pipeline_request)
  File "/app/runfiles/pip-services/pip__azure-identity/azure/core/pipeline/_base_async.py", line 83, in send
    response = await self.next.send(request)  # type: ignore
  File "/app/runfiles/pip-services/pip__azure-identity/azure/core/pipeline/_base_async.py", line 83, in send
    response = await self.next.send(request)  # type: ignore
  File "/app/runfiles/pip-services/pip__azure-identity/azure/core/pipeline/_base_async.py", line 83, in send
    response = await self.next.send(request)  # type: ignore
  [Previous line repeated 5 more times]
  File "/app/runfiles/pip-services/pip__azure-identity/azure/core/pipeline/policies/_redirect_async.py", line 64, in send
    response = await self.next.send(request)
  File "/app/runfiles/pip-services/pip__azure-identity/azure/core/pipeline/_base_async.py", line 83, in send
    response = await self.next.send(request)  # type: ignore
  File "/app/runfiles/pip-services/pip__azure-storage-blob/azure/storage/blob/_shared/policies_async.py", line 99, in send
    response = await self.next.send(request)
  File "/app/runfiles/pip-services/pip__azure-identity/azure/core/pipeline/_base_async.py", line 83, in send
    response = await self.next.send(request)  # type: ignore
  File "/app/runfiles/pip-services/pip__azure-storage-blob/azure/storage/blob/_shared/policies_async.py", line 57, in send
    await response.http_response.load_body()
  File "/app/runfiles/pip-services/pip__azure-identity/azure/core/pipeline/transport/_aiohttp.py", line 270, in load_body
    self._body = await self.internal_response.read()
  File "/app/runfiles/pip-core/pip__aiohttp/aiohttp/client_reqrep.py", line 973, in read
    self._body = await self.content.read()
  File "/app/runfiles/pip-core/pip__aiohttp/aiohttp/streams.py", line 358, in read
    block = await self.readany()
  File "/app/runfiles/pip-core/pip__aiohttp/aiohttp/streams.py", line 380, in readany
    await self._wait('readany')
  File "/app/runfiles/pip-core/pip__aiohttp/aiohttp/streams.py", line 296, in _wait
    await waiter
aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed
@ghost ghost added needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. customer-reported Issues that are reported by GitHub users external to the Azure organization. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that labels Apr 12, 2021
@xiangyan99 xiangyan99 added bug This issue requires a change to an existing behavior in the product in order to be resolved. Client This issue points to a problem in the data-plane of the library. Service Attention Workflow: This issue is responsible by Azure service team. Storage Storage Service (Queues, Blobs, Files) labels Apr 13, 2021
@ghost ghost removed the needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. label Apr 13, 2021
@ghost
Copy link

ghost commented Apr 13, 2021

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

Issue Details
  • Package Name: azure-storage-blob
  • Package Version: 12.6.0
  • Operating System: Ubuntu 18.04
  • Python Version: 3.7.9

Describe the bug
We are using the azure-blob-storage python client to download blobs using the async client implementation.

From time to time our job will crash with the following exception:

aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed

To Reproduce

The failure does not happen consistently but it is quite sporadic (our job can download data for many hours without facing this problem)

Expected behavior

azure-storage-blob should be able to handle this exception and perform a retry?

Additional context

We are using aiohttp version 3.6.3

Full stack trace:

Traceback (most recent call last):
  File "/app/runfiles/azure_image_storage.py", line 57, in load_image
    loop=self._get_or_create_event_loop(),
  File "/app/python_runtime/python3/lib/python3.7/concurrent/futures/_base.py", line 435, in result
    return self.__get_result()
  File "/app/python_runtime/python3/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/app/runfiles/azure_image_storage.py", line 77, in _load_blob_async_into_byte_stream
    storage_stream_downloader = await blob_client.download_blob()
  File "/app/runfiles/pip-services/pip__azure-identity/azure/core/tracing/decorator_async.py", line 74, in wrapper_use_tracer
    return await func(*args, **kwargs)
  File "/app/runfiles/pip-services/pip__azure-storage-blob/azure/storage/blob/aio/_blob_client_async.py", line 361, in download_blob
    await downloader._setup()  # pylint: disable=protected-access
  File "/app/runfiles/pip-services/pip__azure-storage-blob/azure/storage/blob/aio/_download_async.py", line 225, in _setup
    self._response = await self._initial_request()
  File "/app/runfiles/pip-services/pip__azure-storage-blob/azure/storage/blob/aio/_download_async.py", line 271, in _initial_request
    **self._request_options)
  File "/app/runfiles/pip-services/pip__azure-storage-blob/azure/storage/blob/_generated/aio/operations_async/_blob_operations_async.py", line 174, in download
    pipeline_response = await self._client._pipeline.run(request, stream=True, **kwargs)
  File "/app/runfiles/pip-services/pip__azure-identity/azure/core/pipeline/_base_async.py", line 215, in run
    return await first_node.send(pipeline_request)
  File "/app/runfiles/pip-services/pip__azure-identity/azure/core/pipeline/_base_async.py", line 83, in send
    response = await self.next.send(request)  # type: ignore
  File "/app/runfiles/pip-services/pip__azure-identity/azure/core/pipeline/_base_async.py", line 83, in send
    response = await self.next.send(request)  # type: ignore
  File "/app/runfiles/pip-services/pip__azure-identity/azure/core/pipeline/_base_async.py", line 83, in send
    response = await self.next.send(request)  # type: ignore
  [Previous line repeated 5 more times]
  File "/app/runfiles/pip-services/pip__azure-identity/azure/core/pipeline/policies/_redirect_async.py", line 64, in send
    response = await self.next.send(request)
  File "/app/runfiles/pip-services/pip__azure-identity/azure/core/pipeline/_base_async.py", line 83, in send
    response = await self.next.send(request)  # type: ignore
  File "/app/runfiles/pip-services/pip__azure-storage-blob/azure/storage/blob/_shared/policies_async.py", line 99, in send
    response = await self.next.send(request)
  File "/app/runfiles/pip-services/pip__azure-identity/azure/core/pipeline/_base_async.py", line 83, in send
    response = await self.next.send(request)  # type: ignore
  File "/app/runfiles/pip-services/pip__azure-storage-blob/azure/storage/blob/_shared/policies_async.py", line 57, in send
    await response.http_response.load_body()
  File "/app/runfiles/pip-services/pip__azure-identity/azure/core/pipeline/transport/_aiohttp.py", line 270, in load_body
    self._body = await self.internal_response.read()
  File "/app/runfiles/pip-core/pip__aiohttp/aiohttp/client_reqrep.py", line 973, in read
    self._body = await self.content.read()
  File "/app/runfiles/pip-core/pip__aiohttp/aiohttp/streams.py", line 358, in read
    block = await self.readany()
  File "/app/runfiles/pip-core/pip__aiohttp/aiohttp/streams.py", line 380, in readany
    await self._wait('readany')
  File "/app/runfiles/pip-core/pip__aiohttp/aiohttp/streams.py", line 296, in _wait
    await waiter
aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed
Author: gdippolito
Assignees: xiafu-msft
Labels:

Client, Service Attention, Storage, bug, customer-reported, needs-triage, question
Milestone: -

@xiangyan99 xiangyan99 removed the question The issue doesn't require a change to the product in order to be resolved. Most issues start as that label Apr 13, 2021
@xiangyan99
Copy link
Member

Thanks for the feedback, we’ll investigate asap.

@tasherif-msft tasherif-msft self-assigned this Apr 14, 2021
@tasherif-msft
Copy link
Contributor

Hi @gdippolito, the client does in fact retry with 10 retries by default and you can observe that in the error trace.
Do you happen to have a RequestID of a failing operation? This would help us identify what might be going on here.
Thanks.

@tasherif-msft tasherif-msft added the needs-author-feedback Workflow: More information is needed from author to address the issue. label Apr 14, 2021
@xiafu-msft
Copy link
Contributor

Hi @gdippolito

Thanks for reporting this! This would be fixed by this pr while it still has some error we will try to merge it soon 57443b9

@xiafu-msft xiafu-msft removed the needs-author-feedback Workflow: More information is needed from author to address the issue. label Apr 15, 2021
@gdippolito
Copy link
Author

Hi @xiafu-msft,

Thanks for the update.

Let us know when you have a PR we can subscribe to.

Thanks,

@tasherif-msft
Copy link
Contributor

Here is the PR #17909
This fix will be merged by our next release.
The patch release is expected to be released either tomorrow or early next week. We will keep you updated :)

@gdippolito
Copy link
Author

THanks @tasherif-msft

My error was while fetching data and not uploading them. Also, the size of the data is quite small.

However, I can see that you are introducing the exception ClientPayloadError in the code.

Do you think this PR will also fix my use case?

@tasherif-msft
Copy link
Contributor

Yes it should fix the issue for you! The PR I referenced is solving two issues including yours, the pr naming isn't the best haha
We might split them into two separate PRs.
Regardless, yes, this should fix the download issue :)

This was referenced Apr 16, 2021
Closed
Merged
@tasherif-msft
Copy link
Contributor

Hi @gdippolito the fix has been merged and will be available in our next release which is happening by the end of this week!
I will proceed and close this issue, let me know if you have any further questions.

@gdippolito
Copy link
Author

Hi @gdippolito the fix has been merged and will be available in our next release which is happening by the end of this week!
I will proceed and close this issue, let me know if you have any further questions.

Thanks a lot for your support and quick turnaround!

openapi-sdkautomation bot pushed a commit to AzureSDKAutomation/azure-sdk-for-python that referenced this issue Mar 21, 2022
@bingosummer @zqingqing1 @norshtein
CodeGen from PR 18150 in Azure/azure-rest-api-specs
22c6b4b 
Review request for Microsoft.ContainerService to add version 2022-02-02-preview (Azure#18150)

* Adds base for updating Microsoft.ContainerService from version preview/2022-01-02-preview to version 2022-02-02-preview

* Updates readme

* Updates API version in new specs and examples

* Add key management service profile of a managed cluster for version 2022-02-02-preview (Azure#17955)

* Add key management service profile of a managed cluster for version 2022-02-02-preview

* add kmsProfile

* update after api review

* add more details in description

* update for comments

* add mc snapshot api (Azure#17974)

* add mc snapshot api

* fix format

* add mc snapshot type

* define type for enums

* add properties for example

* Update readmes for the 2022-02-02-preview dev branch of container service (Azure#17889)

* update readme

* remove useless -only tags

* remove nested property in allOf for 2022-02-02-preview managedClusters (Azure#18063)

* Fix credential format typo in 0102 preview (Azure#18117)

* add missing type object (Azure#18116)

* Update several common definitions according to arm feedback for container service 2022-02-02-preview (Azure#18206)

* update definition of serveral parameters and system data

* update definition of Resource and move property location to sub resources

* add missing tags property for prev Resource

* fix for compatibility

* fix definition name

Co-authored-by: Bin Xia <[email protected]>
Co-authored-by: Qingqing <[email protected]>
Co-authored-by: Tongyao Si <[email protected]>
@dtrifiro dtrifiro mentioned this issue Mar 31, 2022
Closed
@github-actions github-actions bot locked and limited conversation to collaborators Apr 12, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@xiafu-msft xiafu-msft

@tasherif-msft tasherif-msft

Labels
bug This issue requires a change to an existing behavior in the product in order to be resolved. Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. Service Attention Workflow: This issue is responsible by Azure service team. Storage Storage Service (Queues, Blobs, Files)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@xiangyan99 @gdippolito @xiafu-msft @tasherif-msft