Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mountpoint_s3::fuse: release failed #670

Closed
djhenry2 opened this issue Dec 7, 2023 · 4 comments
Closed

Mountpoint_s3::fuse: release failed #670

djhenry2 opened this issue Dec 7, 2023 · 4 comments
Labels
bug Something isn't working

Comments

@djhenry2
Copy link

djhenry2 commented Dec 7, 2023 •
edited by dannycjones
Loading

Mountpoint for Amazon S3 version

mount-s3 1.1.1

AWS Region

us-east-2

Describe the running environment

Ubuntu 22.04 mounting S3 bucket in the same account.

What happened?

Many errors with "Mountpoint_s3::fuse: release failed"

Mount command:
mount-s3 --allow-root --read-only --log-directory /var/log/mountpoint-s3/ sftpgw-i-***** wn-s3 (bucket name hidden)

Relevant log output

mountpoint-s3-2023-12-07T13-52-09Z.log
@djhenry2 djhenry2 added the bug Something isn't working label Dec 7, 2023
@dannycjones
Copy link
Contributor

Thanks for reporting this. I see its similar to the issue in #582. Thanks for sharing the logs, that should help with a reproduction I hope.

@djhenry2 Can you share any impact to your workload? Is it causing issues or currently just warnings in the logs (which we should fix).

@djhenry2
Copy link
Author

djhenry2 commented Dec 7, 2023

I had an out-of-memory error a few days ago, so I've been digging through the logs trying to find the root cause. So far, it does not appear these errors cause the server to run out of memory, but I'm trying to be thorough.

dannycjones added a commit to dannycjones/mountpoint-s3 that referenced this issue Jan 5, 2024
@dannycjones
Remove fs::read callback to return simple Result instead
0ea0bdc 
We're making this change primarily due to the risk of a race condition introduced.
Before this change, we reply directly to the FUSE driver before exiting the fs module code.
The risk here is that we've already replied to the driver before we drop things like the file handle guard.
Albeit small, this is a race condition and we intend to remove it to avoid any risk from it.

This race condition is suspected to be the root cause for this issue
where FUSE release fails unable to unwrap the file handle reference: awslabs#670

This race condition risk could have a large impact since the file handle holds a reference to prefetched data.

Signed-off-by: Daniel Carl Jones <[email protected]>
@dannycjones dannycjones mentioned this issue Jan 5, 2024
Merged
github-merge-queue bot pushed a commit that referenced this issue Jan 6, 2024
@dannycjones
Replace callback in FS read with simple Result instead (#691)
6e7252d 
* Remove fs::read callback to return simple Result instead

We're making this change primarily due to the risk of a race condition introduced.
Before this change, we reply directly to the FUSE driver before exiting the fs module code.
The risk here is that we've already replied to the driver before we drop things like the file handle guard.
Albeit small, this is a race condition and we intend to remove it to avoid any risk from it.

This race condition is suspected to be the root cause for this issue
where FUSE release fails unable to unwrap the file handle reference: #670

This race condition risk could have a large impact since the file handle holds a reference to prefetched data.

Signed-off-by: Daniel Carl Jones <[email protected]>

* Remove ReadReplier trait

Signed-off-by: Daniel Carl Jones <[email protected]>

---------

Signed-off-by: Daniel Carl Jones <[email protected]>
@dannycjones dannycjones mentioned this issue Jan 19, 2024
Closed
@dannycjones
Copy link
Contributor

We released a possible fix for this in v1.3.2. @djhenry2, can you give it a try and check its solved for your use case?

The fix is to remove a race condition related to file handle de-allocation (6e7252d). There was a chance that before file handles references were released from an operation like read or fsync, the release request would arrive and be unable to de-allocate the file handle. This meant that the file handle would be left in the file handle table, and could lead to increased memory usage since the file handles hold on to prefetched file content.

@sauraank
Copy link
Contributor

sauraank commented Feb 6, 2024

I am closing this issue due to inactivity. If you any further information or question, please re-open this issue or create a new issue. Thanks.

@sauraank sauraank closed this as completed Feb 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dannycjones @sauraank @djhenry2