Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change local file/directory expiry TTL from NEVER_EXPIRE to 100ms #584

Merged
merged 9 commits into from
Nov 14, 2023

Conversation

sauraank
Copy link
Contributor

@sauraank sauraank commented Oct 30, 2023

Description of change

This change allows mountpoint to rewrite to the same file which was deleted remotely using other methods like console/aws cli. Earlier, Due to large validity of files and directories, the inodes corresponding were remembered by kernel forever. So, even deleting remotely did not make kernel call lookup again for those files.

Relevant issues:
#577

Does this change impact existing behavior?

No.

There is no breaking change.


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and I agree to the terms of the Developer Certificate of Origin (DCO).

@sauraank sauraank had a problem deploying to PR integration tests October 30, 2023 03:14 — with GitHub Actions Failure
@sauraank sauraank temporarily deployed to PR integration tests October 30, 2023 03:14 — with GitHub Actions Inactive
@sauraank sauraank temporarily deployed to PR integration tests October 30, 2023 03:14 — with GitHub Actions Inactive
@sauraank sauraank temporarily deployed to PR integration tests October 30, 2023 03:14 — with GitHub Actions Inactive
mountpoint-s3/src/inode.rs Outdated Show resolved Hide resolved
@sauraank sauraank had a problem deploying to PR integration tests October 30, 2023 10:58 — with GitHub Actions Failure
@sauraank sauraank had a problem deploying to PR integration tests October 30, 2023 10:58 — with GitHub Actions Failure
@sauraank sauraank temporarily deployed to PR integration tests October 30, 2023 10:58 — with GitHub Actions Inactive
@sauraank sauraank temporarily deployed to PR integration tests October 30, 2023 10:58 — with GitHub Actions Inactive
@dannycjones dannycjones changed the title Change validity of files and directory from NEVER_EXPIRE to 100 ms Change local file/directory expiry TTL from NEVER_EXPIRE to 100ms Oct 30, 2023
Copy link
Contributor

@dannycjones dannycjones left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One comment to fix, otherwise LGTM.

mountpoint-s3/src/inode.rs Outdated Show resolved Hide resolved
@dannycjones
Copy link
Contributor

The integration tests are failing and I suspect it is related to this change, can you investigate @sauraank? For example, setattr is failing (although not every time!).

@sauraank sauraank had a problem deploying to PR integration tests November 1, 2023 20:48 — with GitHub Actions Failure
@sauraank sauraank had a problem deploying to PR integration tests November 1, 2023 20:48 — with GitHub Actions Failure
@sauraank sauraank temporarily deployed to PR integration tests November 1, 2023 20:48 — with GitHub Actions Inactive
@sauraank sauraank temporarily deployed to PR integration tests November 1, 2023 20:48 — with GitHub Actions Inactive
@sauraank sauraank had a problem deploying to PR integration tests November 2, 2023 14:13 — with GitHub Actions Failure
@sauraank sauraank had a problem deploying to PR integration tests November 2, 2023 14:13 — with GitHub Actions Failure
@sauraank sauraank had a problem deploying to PR integration tests November 2, 2023 14:13 — with GitHub Actions Failure
@sauraank sauraank had a problem deploying to PR integration tests November 2, 2023 14:13 — with GitHub Actions Failure
@sauraank sauraank had a problem deploying to PR integration tests November 7, 2023 14:47 — with GitHub Actions Failure
@sauraank sauraank had a problem deploying to PR integration tests November 7, 2023 14:47 — with GitHub Actions Failure
@sauraank sauraank had a problem deploying to PR integration tests November 7, 2023 14:47 — with GitHub Actions Failure
@sauraank sauraank had a problem deploying to PR integration tests November 7, 2023 14:47 — with GitHub Actions Failure
None,
None,
None,
self.inner.cache_config.file_ttl,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe this will never happen since I'm guessing applications almost always open the file immediately after performing a mknod, but an open 100ms after the mknod will now go to S3 even though we know we'll always choose the local node over a remote one.

Not worried right now, but just noting this observation.

@sauraank sauraank temporarily deployed to PR integration tests November 9, 2023 15:08 — with GitHub Actions Inactive
@sauraank sauraank temporarily deployed to PR integration tests November 9, 2023 15:08 — with GitHub Actions Inactive
@sauraank sauraank temporarily deployed to PR integration tests November 9, 2023 15:08 — with GitHub Actions Inactive
@sauraank sauraank temporarily deployed to PR integration tests November 14, 2023 03:55 — with GitHub Actions Inactive
@sauraank sauraank temporarily deployed to PR integration tests November 14, 2023 03:55 — with GitHub Actions Inactive
@sauraank sauraank temporarily deployed to PR integration tests November 14, 2023 03:55 — with GitHub Actions Inactive
@sauraank sauraank temporarily deployed to PR integration tests November 14, 2023 03:55 — with GitHub Actions Inactive
@sauraank sauraank requested a review from passaro November 14, 2023 10:43
@dannycjones dannycjones linked an issue Nov 14, 2023 that may be closed by this pull request
Copy link
Contributor

@dannycjones dannycjones left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should update the release notes in this PR too as we're fixing #577.

Signed-off-by: Ankit Saurabh <[email protected]>
@sauraank sauraank temporarily deployed to PR integration tests November 14, 2023 11:17 — with GitHub Actions Inactive
@sauraank sauraank temporarily deployed to PR integration tests November 14, 2023 11:18 — with GitHub Actions Inactive
@sauraank sauraank temporarily deployed to PR integration tests November 14, 2023 11:18 — with GitHub Actions Inactive
@sauraank sauraank temporarily deployed to PR integration tests November 14, 2023 11:18 — with GitHub Actions Inactive
Ok(LookedUp {
inode: existing_inode,
inode: existing_inode.clone(),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should not need to clone here:

Suggested change
inode: existing_inode.clone(),
inode: existing_inode,

@sauraank sauraank temporarily deployed to PR integration tests November 14, 2023 11:21 — with GitHub Actions Inactive
@sauraank sauraank temporarily deployed to PR integration tests November 14, 2023 11:21 — with GitHub Actions Inactive
@sauraank sauraank had a problem deploying to PR integration tests November 14, 2023 11:21 — with GitHub Actions Failure
@sauraank sauraank temporarily deployed to PR integration tests November 14, 2023 11:21 — with GitHub Actions Inactive
Copy link
Contributor

@passaro passaro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@sauraank sauraank temporarily deployed to PR integration tests November 14, 2023 11:38 — with GitHub Actions Inactive
@sauraank sauraank temporarily deployed to PR integration tests November 14, 2023 11:38 — with GitHub Actions Inactive
@sauraank sauraank temporarily deployed to PR integration tests November 14, 2023 11:38 — with GitHub Actions Inactive
@sauraank sauraank temporarily deployed to PR integration tests November 14, 2023 11:38 — with GitHub Actions Inactive
@dannycjones dannycjones added this pull request to the merge queue Nov 14, 2023
@passaro passaro mentioned this pull request Nov 14, 2023
Merged via the queue into awslabs:main with commit d38d45f Nov 14, 2023
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Future creates fail for files that are deleted remotely
3 participants