Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MemMapFS: missing files after write multi files in concurrency #361

Closed
sko00o opened this issue Jun 16, 2022 · 2 comments · Fixed by #379
Closed

MemMapFS: missing files after write multi files in concurrency #361

sko00o opened this issue Jun 16, 2022 · 2 comments · Fixed by #379
Labels

Comments

@sko00o
Copy link

sko00o commented Jun 16, 2022

I have used afero.MemMapFs for filesystem mock test in my projects.
But I got some unexpected test results.

I want to list all files in target directory that I have wrote multi files in concurrency before.
Occasionally some files are not returned by Readdir function. Also I had the same problem calling Walk function on this directory.

Here is a test to reproduce my problem. https://github.com/sko00o/afero-test/blob/main/memmap/memmap_test.go#L42

failure result:

❯ go test -race -run ^TestListFilesAfterConcurrencyFileWrite$ ./... -test.count 1
--- FAIL: TestListFilesAfterConcurrencyFileWrite (4.39s)
    memmap_test.go:99: 
                Error Trace:    memmap_test.go:99
                Error:          Not equal: 
                                expected: 10000
                                actual  : 9995
                Test:           TestListFilesAfterConcurrencyFileWrite
                Messages:       missing files
FAIL
FAIL    github.com/sko00o/afero-test/memmap     4.677s
FAIL

I notice another issue #298 that found MemMapFs.Mkdir have problem when call in concurrency.
In my opinion, we need a double check here before we create new directory.


So I send a PR for this.

@sko00o
Copy link
Author

sko00o commented Nov 14, 2022

Ping @bep

Please check this issue.

For quick reproduce: https://go.dev/play/p/3xkOSEBVoNS

@bep bep added the bug label Nov 14, 2022
@bep
Copy link
Collaborator

bep commented Nov 14, 2022

Your failing test case looks correct to me (as in, it should not fail), which comes a little as a surprise to mek as I use this pretty concurrently myself ... It's not obvious to me why, though.

I also agree about the Mkdir bug, assuming that that's how most sane real file systems work?

bep added a commit to bep/afero that referenced this issue Nov 14, 2022
* The backing map is protected by a RWMutex
* This commit double checks for the existence of the directory inside the write lock to avoid potential data races when multiple goroutines tries to create
the same directory.

Fixes spf13#361
Fixes spf13#298
bep added a commit to bep/afero that referenced this issue Nov 14, 2022
* The backing map is protected by a RWMutex
* This commit double checks for the existence of the directory inside the write lock to avoid potential data races when multiple goroutines tries to create
the same directory.

Fixes spf13#361
Fixes spf13#298
bep added a commit to bep/afero that referenced this issue Nov 14, 2022
* The backing map is protected by a RWMutex
* This commit double checks for the existence of the directory inside the write lock to avoid potential data races when multiple goroutines tries to create
the same directory.

Fixes spf13#361
Fixes spf13#298
@bep bep closed this as completed in #379 Nov 14, 2022
bep added a commit that referenced this issue Nov 14, 2022
* The backing map is protected by a RWMutex
* This commit double checks for the existence of the directory inside the write lock to avoid potential data races when multiple goroutines tries to create
the same directory.

Fixes #361
Fixes #298
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants