Add FUSE Passthrough Support in Stargz-Snapshotter #1867 #1868

wswsmao · 2024-11-19T04:59:15Z

Here’s a proposed implementation plan for FUSE passthrough:

During the Open phase, attempt to pre-read the entire file instead of reading it in chunks.
Utilize the existing cache's Get method to retrieve the fd of the cached file that has been written to local storage.
Implement the FilePassthroughFder interface in node.file, allowing the fd from step 2 to be registered with the kernel via go-fuse.

By following this approach, subsequent Read operations would not need to return to user space, and go-fuse would release the registered information when necessary.

for details,
#1867

Signed-off-by: abushwang <[email protected]>

ktock

Thanks. Could you fix linter errors? https://github.com/containerd/stargz-snapshotter/actions/runs/11906582478/job/33178914643?pr=1868

ktock · 2024-11-19T07:12:16Z

fs/config/config.go

@@ -148,4 +148,7 @@ type FuseConfig struct {

 	// EntryTimeout defines TTL for directory, name lookup in seconds.
 	EntryTimeout int64 `toml:"entry_timeout"`
+
+	// PassThrough indicates whether to enable FUSE passthrough mode to improve local file read performance. Default is false.
+	PassThrough bool `toml:"passthrough"`


Can we test this in our CI?

ktock · 2024-11-19T07:13:18Z

fs/reader/reader.go

+		b.Reset()
+		b.Grow(int(chunkSize))
+		ip := b.Bytes()[:chunkSize]
+		if _, err := sf.fr.ReadAt(ip, chunkOffset); err != nil && err != io.EOF {


Can we check if the data can be read from the cache?

ktock · 2024-11-19T07:15:08Z

cache/cache.go

@@ -82,6 +82,7 @@ type BlobCache interface {
 type Reader interface {
 	io.ReaderAt
 	Close() error
+	GetReaderAt() io.ReaderAt


Can we add a comment like If a blob is backed by a file, it should return *os.File so that it can be used for FUSE passthrough.?

ktock · 2024-11-19T07:16:03Z

fs/layer/node.go

+				n.fs.s.report(fmt.Errorf("node.Open: %v", err))
+				return nil, 0, syscall.EIO


Can we continue opening this as a non-passthrough file, instead of returning EIO?

ktock · 2024-11-19T07:19:23Z

cache/cache.go

@@ -82,6 +82,7 @@ type BlobCache interface {
 type Reader interface {
 	io.ReaderAt
 	Close() error
+	GetReaderAt() io.ReaderAt


When FUSE passthrough is enabled, we should always set directoryCache.direct to true so that we can ensure that *directoryCache.Get always return *os.File (not a buffer).

wswsmao · 2024-11-20T07:48:15Z

all done @ktock

wswsmao · 2024-11-20T11:14:12Z

This is my benchmark
https://github.com/wswsmao/fuse-performance

I create three sets of files with sizes of 50, 75, and 100MB respectively.
Using these, I've created a test image named abushwang/ocs9:fuseperf-orig.

Subsequently, I used the default value for the --estargz-chunk-size parameter to create an estargz image.

$ nerdctl image convert --estargz --oci abushwang/ocs9:fuseperf-orig abushwang/ocs9:fuseperf-esgz

To ensure a clean environment for each test, I removed the local image and cache, and then restarted the containerd-stargz-grpc service.

$ rm -rf /var/lib/containerd-stargz-grpc/stargz/fscache/*
$ nerdctl images -q | xargs nerdctl rmi -f"

This is the report:

no passthrough

$ nerdctl run --rm -t --snapshotter=stargz abushwang/ocs9:fuseperf-esgz
docker.io/abushwang/ocs9:fuseperf-esgz:                                           resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:13868ec859962d0ae07afe0db9f48208e54f38afbaa337e4237aa1e5831c24c0: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:0abfbeeb608160b63cc9e9e3c351c639b1a53a0bf99a27109d96adc4a75e3c97:   done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 7.1 s                                                                    total:  5.1 Ki (740.0 B/s)                                       
Running file access tests...
Testing file: large_files/large_file_1
Testing file: large_files/large_file_2
Testing file: large_files/large_file_3
Performance Report
==================
File: large_file_1
  Sequential Read Time: 8329.96 ms
--------------------------
File: large_file_2
  Sequential Read Time: 11704.64 ms
--------------------------
File: large_file_3
  Sequential Read Time: 15271.63 ms
--------------------------
Tests completed. See performance_report.txt for details.

with passthrough

$ nerdctl run --rm -t --snapshotter=stargz abushwang/ocs9:fuseperf-esgz
docker.io/abushwang/ocs9:fuseperf-esgz:                                           resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:13868ec859962d0ae07afe0db9f48208e54f38afbaa337e4237aa1e5831c24c0: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:0abfbeeb608160b63cc9e9e3c351c639b1a53a0bf99a27109d96adc4a75e3c97:   done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 7.1 s                                                                    total:  5.1 Ki (740.0 B/s)                                       
Running file access tests...
Testing file: large_files/large_file_1
Testing file: large_files/large_file_2
Testing file: large_files/large_file_3
Performance Report
==================
File: large_file_1
  Sequential Read Time: 344.62 ms
--------------------------
File: large_file_2
  Sequential Read Time: 518.38 ms
--------------------------
File: large_file_3
  Sequential Read Time: 667.25 ms
--------------------------
Tests completed. See performance_report.txt for details.

env:

$ uname -r
6.11.5-200.fc40.x86_64

ktock · 2024-11-20T14:18:29Z

cmd/containerd-stargz-grpc/main.go

+// isFusePthEnable prevents users from enabling passthrough mode on unsupported kernel versions
+func isFusePthEnable() (bool, error) {
+	cmd := exec.Command("sh", "-c", "grep 'CONFIG_FUSE_PASSTHROUGH=y' /boot/config-$(uname -r)")


nit: Let's try not to rely on the shell commands for now. Instead of having this check, let's put a document about how to check if passthrough is a supported on the node (maybe in the following PRs)

Signed-off-by: abushwang <[email protected]>

ktock

Thanks

ujjwal · 2024-12-02T07:23:44Z

Does the kernel version need to be updated to 6.9 to test out passthrough?

wswsmao · 2024-12-02T07:26:09Z

Does the kernel version need to be updated to 6.9 to test out passthrough?

yes，or rebase this feature
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6ce8b2ce0d7e3a621cdc9eb66d74436ca7d0e66e

fuse: support passthrough mode

71930d7

Signed-off-by: abushwang <[email protected]>

wswsmao mentioned this pull request Nov 19, 2024

Add FUSE Passthrough Support in Stargz-Snapshotter #1867

Open

fuse: add kernel fuse passthrough support check

39a2e55

Signed-off-by: abushwang <[email protected]>

wswsmao force-pushed the passthrough branch from 6424a04 to 39a2e55 Compare November 19, 2024 05:02

ktock reviewed Nov 19, 2024

View reviewed changes

wswsmao force-pushed the passthrough branch 9 times, most recently from 3fd3113 to 5dba826 Compare November 20, 2024 06:39

ktock reviewed Nov 20, 2024

View reviewed changes

fuse passthrough: fix some review comments

d16d065

Signed-off-by: abushwang <[email protected]>

wswsmao force-pushed the passthrough branch from 5dba826 to d16d065 Compare November 21, 2024 01:40

ktock approved these changes Nov 21, 2024

View reviewed changes

ktock merged commit 2a280d6 into containerd:main Nov 21, 2024
26 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add FUSE Passthrough Support in Stargz-Snapshotter #1867 #1868

Add FUSE Passthrough Support in Stargz-Snapshotter #1867 #1868

wswsmao commented Nov 19, 2024 •

edited

Loading

ktock left a comment

ktock Nov 19, 2024

ktock Nov 19, 2024

ktock Nov 19, 2024

ktock Nov 19, 2024

ktock Nov 19, 2024

wswsmao commented Nov 20, 2024

wswsmao commented Nov 20, 2024 •

edited

Loading

ktock Nov 20, 2024

ktock left a comment

ujjwal commented Dec 2, 2024

wswsmao commented Dec 2, 2024

		n.fs.s.report(fmt.Errorf("node.Open: %v", err))
		return nil, 0, syscall.EIO

Add FUSE Passthrough Support in Stargz-Snapshotter #1867 #1868

Add FUSE Passthrough Support in Stargz-Snapshotter #1867 #1868

Conversation

wswsmao commented Nov 19, 2024 • edited Loading

ktock left a comment

Choose a reason for hiding this comment

ktock Nov 19, 2024

Choose a reason for hiding this comment

ktock Nov 19, 2024

Choose a reason for hiding this comment

ktock Nov 19, 2024

Choose a reason for hiding this comment

ktock Nov 19, 2024

Choose a reason for hiding this comment

ktock Nov 19, 2024

Choose a reason for hiding this comment

wswsmao commented Nov 20, 2024

wswsmao commented Nov 20, 2024 • edited Loading

ktock Nov 20, 2024

Choose a reason for hiding this comment

ktock left a comment

Choose a reason for hiding this comment

ujjwal commented Dec 2, 2024

wswsmao commented Dec 2, 2024

wswsmao commented Nov 19, 2024 •

edited

Loading

wswsmao commented Nov 20, 2024 •

edited

Loading