Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

service/s3/s3manager: Should be able to track download progress #286

Closed
mwhooker opened this issue Jun 17, 2015 · 6 comments
Closed

service/s3/s3manager: Should be able to track download progress #286

mwhooker opened this issue Jun 17, 2015 · 6 comments
Labels
feature-request A feature should be added or improved.

Comments

@mwhooker
Copy link
Contributor

I want to show download progress to my users, which requires getting the total bytes to be downloaded.

I could do a HEAD request first to get this, but it's an additional round trip, and there's no telling if what I download is the same resource that I got the HEAD of.

Having access to downloader.written would also be useful.

If anyone can suggest a workaround, that would also be very helpful.
@lsegal
Copy link
Contributor

lsegal commented Jun 17, 2015

I could do a HEAD request first to get this, but it's an additional round trip, and there's no telling if what I download is the same resource that I got the HEAD of.

This seems like a reasonable workaround to me. If you're worried about the object being concurrently written to while you are downloading it then the initial HEAD request will be the least of your concerns, as the downloader itself is likely making multiple concurrent GET requests-- and the object could theoretically change in between those requests as well.

But I agree that having progress support and totalBytes would be useful. The only problem is that they are not known at the start of the download, so if we exposed this it would likely be via function callback.

@lsegal lsegal added the feature-request A feature should be added or improved. label Jun 17, 2015
@mwhooker
Copy link
Contributor Author

If you're worried about the object being concurrently written to while you are downloading it then the initial HEAD request will be the least of your concerns, as the downloader itself is likely making multiple concurrent GET requests-- and the object could theoretically change in between those requests as well.

you're right. thanks for putting that into perspective.

@awsdave awsdave added blocked Work is blocked on this issue for this codebase. Other labels or comments may indicate why. and removed blocked Work is blocked on this issue for this codebase. Other labels or comments may indicate why. labels Nov 15, 2015
@xunchangguo
Copy link

Feature added?

@jasdel
Copy link
Contributor

jasdel commented Mar 14, 2016

Hi @xunchangguo this feature had not been added yet. It is still an outstanding item in our backlog. We're more than glad to discuss and review PR for feature requests for items such as this too :)

@flavioribeiro flavioribeiro mentioned this issue Jul 12, 2016
Open
@jasdel jasdel changed the title expose downloader.totalBytes when using S3Manager service/s3/s3manager: Should be able to track download progress Apr 12, 2017
@petems
Copy link
Contributor

petems commented Feb 17, 2019

With a little help from @mwhooker, I made an example of how to do this for an S3 download.

Inspired by the PR to show this for upload, I added an example PR here: #2456

@diehlaws
Copy link
Contributor

Example added in 07f5610 - closing issue.

skotambkar added a commit to skotambkar/aws-sdk-go that referenced this issue May 20, 2021
@skotambkar @jasdel
private/model/api: Add support for TimestampFormat in protocols (aws#358
e018a18 
)

Adds support for the timestampForamt API model trait to the V2 SDK. The SDK will now generate API client parameters with the correct time format for APIs modeled with custom time stamp formats specified.

V2 Port of aws#2066
Fix aws#202
Fix aws#286
skotambkar pushed a commit to skotambkar/aws-sdk-go that referenced this issue May 20, 2021
@jasdel
Release v0.11.0 (2019-08-22) (aws#363)
d7de227 
Services
===
* Synced the V2 SDK with latest AWS service API definitions.
  * Fixes [aws#359](aws/aws-sdk-go-v2#359)

SDK Features
===

SDK Enhancements
===
* `private/protocol`: Add support for TimestampFormat in protocols ([aws#358](aws/aws-sdk-go-v2#358))
  * Adds support for the timestampForamt API model trait to the V2 SDK. The SDK will now generate API client parameters with the correct time format for APIs modeled with custom time stamp formats specified.
  * Fixes [aws#202](aws/aws-sdk-go-v2#202)
  * Fixes [aws#286](aws/aws-sdk-go-v2#286)
* `aws`: Add example for custom HTTP client idle connection options ([aws#350](aws/aws-sdk-go-v2#350))
  * Adds example to the SDK for configuring custom HTTP client idle connection keep alive options.

SDK Bugs
===
* `private/model/api`: Fix API doc being generated with wrong value ([aws#359](aws/aws-sdk-go-v2#359))
  * Fixes the SDK's generated API documentation for structure member being generated with the wrong documentation value when the member was included multiple times in the model doc-2.json file, but under different types.
  * V2 port of to v1 [aws#2748](aws#2748)
* `aws/ec2rolecreds`: Fix security creds path to include trailing slash ([aws#356](aws/aws-sdk-go-v2#356))
  * Fixes the iamSecurityCredsPath var to include a trailing slash preventing redirects when making requests to the EC2 Instance Metadata service.
  * Fixes [aws#351](aws/aws-sdk-go-v2#351)
* `service/dynamodb/expression`: Improved reporting of bad key conditions ([aws#360](aws/aws-sdk-go-v2#360))
  * Improved error reporting when invalid key conditions are constructed using KeyConditionBuilder
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature-request A feature should be added or improved.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@lsegal @mwhooker @xunchangguo @jasdel @petems @awsdave @diehlaws