[BUG] Storage. Multiple large streaming uploads stuck on AKS.

[denis111]

Describe the bug
We have java Spring Boot application with an api where users can upload files of any size and multiple files at a time, the files while uploading are streamed directly to the backend storage (Azure storage in that case). We provide so-called resumable uploads so we upload large files using multiple PUT requests with variable size (depends on user's bandwidth). So when one or more users upload several large files PUT requests get stuck until some timeout. I tried to do that via Block Blob and also Append Blob, the result is the same.
That's not our application issue because when I launch locally or other cloud provider(Google) it just uploads fine to Azure storage, it only happens with AKS. Maybe something weird happens while converting InputStream to Flux but only on AKS instances.
When it's stuck if i just restart kubernetes pod it may continue until stuck another time.

Exception or Stack Trace
Usually it ends with EOF becaouse of client timeout:
Caused by: java.io.EOFException: Unexpected EOF read on the socket

To Reproduce
It might be difficult to explain but any Spring Boot app when some Upload Controller which passes InputStream directly to Azure Storage. Upload about 10 files at a time on AKS instance.

Code Snippet

      blockBlobClient.stageBlockWithResponse(blockId,
          Utility.convertStreamToByteBuffer(eis, partSize, maxBufferSize(), false), partSize,
          StringUtils.isNotEmpty(md5) ? Base64.getDecoder().decode(md5) : null, null)
          .block();

(retryable/seekable can't be true here).

Expected behavior
Uploads finish fine.

Setup (please complete the following information):

• OS: AKS
• IDE : IntelliJ
• Version of the Library used azure-spring-boot-starter-storage v3.4.0, tried azure-storage-blob v12.10, v12.11, v12.12 beta.

Additional context
I tried to use the same code on GKE (Google cloud) connecting to Azure storage and it works fine. Also we support local disk, S3 and GCS as storage provider and upload to GCS from AKS is also working fine. It's only Java -> AKS -> Azure storage problem.

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• Bug Description Added
• Repro Steps Added
• Setup information Added

[joshfree]

@stliu could you please follow up?

/cc @gapra-msft

[stliu]

@denis111 it's difficult to me that to do root cause analysis based on the description , would you provide a sample project to help us reproduce this issue?

[denis111]

@stliu that's not so easy but I guess it's the only way. I'll try to prepare sample project then.

[ghost]

Hi, we're sending this friendly reminder because we haven't heard back from you in a while. We need more information about this issue to help address it. Please be sure to give us your input within the next 7 days. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!