Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use a single IO thread when downloading S3 objects #620

Merged
merged 1 commit into from
Jan 29, 2014

Conversation

jamesls
Copy link
Member

@jamesls jamesls commented Jan 29, 2014

We have seen issues on some OSes that have trouble with
concurrent IO threads to different parts of the same file.
Despite these being non overlapping parts of the file, this
doesn't always work.

The fix is to switch to a single IO thread. All other threads are
now passed an IO queue in which they can submit either IORequests
or IOCompleteRequests that the IO thread understands.

Fixes #619

We have seen issues on some OSes that have trouble with
concurrent IO threads to different parts of the same file.
Despite these being non overlapping parts of the file, this
doesn't always work.

The fix is to switch to a single IO thread. All other threads are
now passed an IO queue in which they can submit either `IORequests`
or `IOCompleteRequests` that the IO thread understands.
@garnaat
Copy link
Contributor

garnaat commented Jan 29, 2014

LGTM. I was wondering how much of an impact this had on download performance?

@toastdriven
Copy link
Contributor

LGTM as well.

@jamesls jamesls merged commit 5204a47 into aws:develop Jan 29, 2014
@jamesls jamesls deleted the single-writer branch June 23, 2014 18:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Switch to single IO thread for S3 downloads
3 participants