Fix multiple part uploads at once making vector size inconsistent

[gruuya]

Which issue does this PR close?

Closes #2288.

Rationale for this change

I'm working on splitgraph/seafowl#99, and I've been seeing sporadic multipart upload failures with error Missing information for upload part x. After a brief investigation I think the fix is quite simple, since the underlying problem stems from an incorrect assumption about the size of the completed parts vector. In other words, in poll_tasks of CloudMultiPartUpload we should get the size of the completed_parts vector (needed for resizing) for each iteration of the while loop, instead of calculating it prior to entering the loop.

To demonstrate how this issue arises consider the following example:

• there are no parts initially, so completed_parts = []
• tasks for parts 0, 1, and 2 are created and at we enter poll_tasks; total_parts = 0
• imagine parts 1, 0, and 2 finish, in that order
• for part 1, we resize completed_parts to max(1 + 1, 0), and set the element at index 1 to the incoming part: completed_parts = [None, Some(part_1)]
• critically, for part 0, we now resize completed_parts to max(0 + 1, 0), which actually means we truncate the last element (thus losing it), and then set the element at index 0: completed_parts = [Some(part_0)]
• lastly, part 2 comes in; we resize completed_parts to max(2 + 1, 0), and fill in the last element: completed_parts = [Some(part_0), None, Some(part_2)]
• once all other parts are completed, we go to poll_shutdown and the None above leads to Missing information for upload part 1

What changes are included in this PR?

Get the size of the completed parts inside the while loop pooling the individual tasks, since it is changing with each iteration.

Are there any user-facing changes?

No.

[tustvold]

This makes sense to me and I'm happy that the existing integration tests are hitting this, and failing, so we have test coverage - #2288

[gruuya]

Thanks for a speedy turnaround!

Can I also ask what are the chances of this making it into DF 12.0.0 scheduled for tomorrow (apache/datafusion#3097), or at least object-store 0.4.1?

[tustvold]

I'm afraid even if I cut the RC for object-store 5.0.0 today, which I may do, the Apache foundation mandates a 3 day release voting period, so it wouldn't be ready in time.

I suspect patch.crates-io will be your only option. FWIW we're currently using this in IOx

[gruuya]

Understood, thanks.

[ursabot]

Benchmark runs are scheduled for baseline = 0b59f05 and contender = 566ef3d. 566ef3d is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java