Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix S3FileLoader local file clobbering #895

Closed
wants to merge 1 commit into from
Closed

Fix S3FileLoader local file clobbering #895

wants to merge 1 commit into from

Conversation

ringohoffman
Copy link
Contributor

Fixes #735; remove unnecessary call to storeFile(), a remnant from the aws-doc-sdk-examples transferOnStream.cpp example, where this function is used to verify the buffer's contents: https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/cpp/example_code/transfer-manager/transferOnStream.cpp#L131

Changes

  • remove unnecessary call to storeFile()

Testing

Verified files are no longer clobbered through local testing

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 15, 2022
@ejguan ejguan added the ciflow/slow Run tests with decorate of `slowTest` label Nov 15, 2022
NivekT
NivekT reviewed Nov 15, 2022
View reviewed changes
Copy link
Contributor

@NivekT NivekT left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deferring to @ejguan and @ydaiming for review

@ringohoffman
Copy link
Contributor Author

Based on the test failures, it seems like an extra file was added to the s3://ai2-public-datasets/ bucket... shall I go ahead and update 77->78 and 39->40?

ejguan
ejguan approved these changes Nov 15, 2022
View reviewed changes
Copy link
Contributor

@ejguan ejguan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you. LGTM. I am fixing the tests in #896. Will let you know when the fix has been landed and please rebase afterwards.

@ejguan
Copy link
Contributor

ejguan commented Nov 15, 2022

Based on the test failures, it seems like an extra file was added to the s3://ai2-public-datasets/ bucket... shall I go ahead and update 77->78 and 39->40?

Yea. And, I personally think the test failures are unrelated to your PR. To guarantee the changes in this PR, let's wait until my PR is landed and rebase this PR to main branch afterwards. Thank you

@ejguan ejguan added the topic: bug fixes topic category label Nov 15, 2022
@ejguan
Copy link
Contributor

ejguan commented Nov 15, 2022 •
edited
Loading

@ringohoffman Do you mind rebasing your commits onto main branch? #896 has been landed.

@ringohoffman ringohoffman requested review from ejguan and removed request for ydaiming November 15, 2022 20:14
@facebook-github-bot
Copy link
Contributor

@ejguan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

ejguan
ejguan approved these changes Nov 15, 2022
View reviewed changes
Copy link
Contributor

@ejguan ejguan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thank you

@ringohoffman @ejguan
Fix S3FileLoader local file clobbering
ec9c8d9 
Fixes pytorch#735; remove unnecessary call to storeFile, a remnant from the aws-doc-sdk-examples transferOnStream.cpp example, where this function is used to verify the buffer contents: https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/cpp/example_code/transfer-manager/transferOnStream.cpp#L131
@ejguan ejguan force-pushed the bug_735_s3_file_loader_local_file_clobbering branch from 2177fac to ec9c8d9 Compare November 15, 2022 20:37
@facebook-github-bot
Copy link
Contributor

@ejguan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@ringohoffman ringohoffman deleted the bug_735_s3_file_loader_local_file_clobbering branch November 16, 2022 20:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@NivekT NivekT NivekT left review comments

@ejguan ejguan ejguan approved these changes

Assignees
No one assigned
Labels
ciflow/slow Run tests with decorate of `slowTest` CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. topic: bug fixes topic category
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

S3FileLoader clears contents of local file when s3 object name == local file relative path name
4 participants
@ringohoffman @ejguan @facebook-github-bot @NivekT