Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(Draft) Gvfs-fuse implementation and code structure layout #5738

Draft
wants to merge 29 commits into
base: branch-gvfs-fuse-dev
Choose a base branch
from

Conversation

diqiu50
Copy link
Contributor

@diqiu50 diqiu50 commented Dec 3, 2024

What changes were proposed in this pull request?

  1. Implement basic FUSE interfaces.
  2. Implement interfaces for filesystem-related operations.
  3. Implement an in-memory filesystem for testing.
  4. Add a framework for FUSE integration testing.

Why are the changes needed?

Fix: #5734

Does this PR introduce any user-facing change?

No

How was this patch tested?

IT and UT

@diqiu50 diqiu50 self-assigned this Dec 3, 2024
futures-util = "0.3.30"
fuse3 = { version = "0.8.1", "features" = ["tokio-runtime", "unprivileged"] }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you plan to use fuse3?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the thread mode of fuse3 is better than fuser. It uses multiple threads with asynchronous I/O.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does fuse3 have some disadvantages?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FUSE3 is more modern, with fewer legacy burdens, but it has a smaller user base. On the other hand, fuser has been around for a long time, its implementation and technologies are older, but it has a larger user base, making it more stable in practice.

I'm not sure which one would be better to use right now. What do you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My concern about FUSE3 is the stability and smaller user community. It may hard to resolve if encountering some underlying problems.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think we should use fuser?

@diqiu50 diqiu50 force-pushed the gvfs-2 branch 2 times, most recently from a8456cb to cd11e47 Compare December 4, 2024 08:36
/// the `file_id` and `parent_file_id` it is the unique identifier for the file system, it is used to identify the file or directory
/// the `fh` it is the file handle, it is used to identify the opened file, it is used to read or write the file content
pub trait IFileSystem: Send + Sync {
fn get_file_path(&self, file_id: u64) -> String;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add async to all interfaces?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add Result to wrap errors for all interfaces?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll consider adding async later.

Copy link
Contributor Author

@diqiu50 diqiu50 Dec 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, all the interface wrap by Resut<T, Errno>

@FANNG1
Copy link
Contributor

FANNG1 commented Dec 4, 2024

How about could split this PR into sevral parts:

  1. code skeleton framework, include github ci, FuseServer, FileHandleManager, IFilesystem MemoryFilesystem with a dummy implementation
  2. implementation details about different operations
    a. list directory and files
    b. read and write files
    c. get file stats
    d. others
    After merging the code skeleton framework, we could split different operations into different issues.

}

fn add_dir_with_name(&mut self, parent: &str, name: &str) -> FileStat {
let parent_inode = self.dir_name_map.get(parent).unwrap().clone();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's not recommended to unwrap directly for all codes

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I understand. I will only use unwrap when it is guaranteed to succeed.

@diqiu50
Copy link
Contributor Author

diqiu50 commented Dec 4, 2024

How about could split this PR into sevral parts:

  1. code skeleton framework, include github ci, FuseServer, FileHandleManager, IFilesystem MemoryFilesystem with a dummy implementation
  2. implementation details about different operations
    a. list directory and files
    b. read and write files
    c. get file stats
    d. others
    After merging the code skeleton framework, we could split different operations into different issues.

Let's wait until the interfaces stabilize before deciding whether to split the PR. Currently, this is a minimal implementation that I need to use to verify if the structure is reasonable.

Comment on lines +74 to +80
- name: Build and test Gravitino
run: |
./gradlew :clients:filesystem-fuse:build -PenableFuse=true

- name: Package Gravitino
run: |
./gradlew compileDistribution -x test -PjdkVersion=${{ matrix.java-version }} -PenableFuse=true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why the compileDistribution after Build and test Gravitino ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

compileDistribution is use to running the integration test, build and test only run uts

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you mean you mix fuseIT in compileDistribution?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, fuseITs depend on the package of compileDistribution

Copy link
Contributor

@mchades mchades Dec 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's the function of -PenableFuse=true here

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

plz remove this and the below steps since it's unnecessary now

where
T: 'a;

async fn readdirplus<'a>(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What‘s the difference between readdirplus and readdir?
I could have a better name for readdirplus function.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

readdirplus is a new api of fuse. it has better performance

Ok(())
}

async fn rmdir(&self, req: Request, parent: Inode, name: &OsStr) -> fuse3::Result<()> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please change the function name to rm_dir

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function names are the trait of fues3::Filesystem. we can't change them

@diqiu50 diqiu50 changed the title [#5734] feat (gvfs-fuse): Gvfs-fuse basic FUSE-level implementation and code structure layout (Draft) Gvfs-fuse basic FUSE-level implementation and code structure layout Dec 11, 2024
@diqiu50 diqiu50 changed the title (Draft) Gvfs-fuse basic FUSE-level implementation and code structure layout (Draft) Gvfs-fuse implementation and code structure layout Dec 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants