Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(storage): return an error on short writes #8562

Merged
merged 1 commit into from
Mar 17, 2022

Conversation

coryan
Copy link
Contributor

@coryan coryan commented Mar 17, 2022

Very rarely the retry policy is exhausted during an upload. If this
happens when the last request was a query to find the status of the
upload the UploadChunk() succeeds with a short write.

We need to determine if that is a good idea (it might be). In the
interim, we need to signal to the application that the stream is no
longer usable. They can use .Suspend() to reset the stream (maybe in
another process) if that can be useful.

Part of the work for #8559


This change is Reviewable

Very rarely the retry policy is exhausted during an upload. If this
happens when the last request was a query to find the status of the
upload the `UploadChunk()` succeeds with a short write.

We need to determine if that is a good idea (it might be). In the
interim, we need to signal to the application that the stream is no
longer usable. They can use `.Suspend()` to reset the stream (maybe in
another process) if that can be useful.
@product-auto-label product-auto-label bot added the api: storage Issues related to the Cloud Storage API. label Mar 17, 2022
@coryan
Copy link
Contributor Author

coryan commented Mar 17, 2022

/fyi: @Jseph

@google-cloud-cpp-bot
Copy link
Collaborator

Google Cloud Build Logs
For commit: 1a5f0cecc7a62cb588ae43886151fe6fbb871618

ℹ️ NOTE: Kokoro logs are linked from "Details" below.

@codecov
Copy link

codecov bot commented Mar 17, 2022

Codecov Report

Merging #8562 (1a5f0ce) into main (3a47220) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main    #8562   +/-   ##
=======================================
  Coverage   93.90%   93.90%           
=======================================
  Files        1459     1459           
  Lines      126428   126447   +19     
=======================================
+ Hits       118720   118744   +24     
+ Misses       7708     7703    -5     
Impacted Files Coverage Δ
...e/cloud/storage/internal/object_write_streambuf.cc 100.00% <100.00%> (ø)
...ud/storage/internal/object_write_streambuf_test.cc 99.78% <100.00%> (+0.01%) ⬆️
google/cloud/internal/async_rpc_details.h 94.44% <0.00%> (-5.56%) ⬇️
...le/cloud/internal/default_completion_queue_impl.cc 97.15% <0.00%> (-0.57%) ⬇️
google/cloud/pubsub/samples/samples.cc 92.33% <0.00%> (+0.23%) ⬆️
google/cloud/pubsub/subscriber_connection_test.cc 97.91% <0.00%> (+0.69%) ⬆️
...le/cloud/storage/internal/curl_download_request.cc 89.75% <0.00%> (+1.06%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3a47220...1a5f0ce. Read the comment docs.

@coryan coryan marked this pull request as ready for review March 17, 2022 21:13
@coryan coryan requested a review from a team as a code owner March 17, 2022 21:13
@coryan coryan merged commit bd611ab into googleapis:main Mar 17, 2022
@coryan coryan deleted the fix-storage-silent-data-loss branch March 17, 2022 22:10
if (actual_next_byte < expected_next_byte) {
// TODO(#8559) - if actual_next_byte < first_buffered_byte and there is
// enough space in the buffer we could save the bytes for a future
// write.
std::ostringstream error_message;
error_message << "Could not continue upload stream. GCS requested byte "
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this error message need to be updated?
If actual next byte is less than expected next byte, that means some bytes have not been committed and therefore might not be uploaded yet?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: storage Issues related to the Cloud Storage API.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants