Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Only upload large files once. #750

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

hugwijst
Copy link

Currently, upload requests are handled in parallel without knowledge of other ongoing requests. If multiple actions depend on the same set of large locally available artifacts, this results in many upload requests of the same blobs.

To solve this behavior, we store new large blob upload requests in a dashmap. If a request is already ongoing, wait a notification of that upload finishing instead of starting a new upload.

Currently, upload requests are handled in parallel without knowledge of
other ongoing requests. If multiple actions depend on the same set of
large locally available artifacts, this results in many upload requests
of the same blobs.

To solve this behavior, we store new large blob upload requests in a
dashmap. If a request is already ongoing, wait a notification of that
upload finishing instead of starting a new upload.
@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 23, 2024
@hugwijst hugwijst closed this Oct 4, 2024
@hugwijst hugwijst reopened this Oct 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants