Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework/improve the file scanner / file cache #11753

Closed
PVince81 opened this issue Oct 23, 2014 · 4 comments
Closed

Rework/improve the file scanner / file cache #11753

PVince81 opened this issue Oct 23, 2014 · 4 comments

Comments

@PVince81
Copy link
Contributor

If we had to write it again from scratch, I have the feeling that the file scanner / cache would probably be implemented differently. The current one is mostly evolved from earlier versions.

Things that we need the file cache for:

  1. storing the metadata, mostly of external storage
  2. being able to detect changes on the file system (mostly for external storage)
  3. be robust, support a high level of concurrency
  4. propagate changes in the form of etags
  5. propagate the size to compute folder sizes
  6. somehow support mount points

What kind of different approach could be used to achieve these ?
Would an event system of some sorts help ?
Should the scanner be running in its own process instead of concurrently ?

@icewind1991 did you ever dream of a better file scanner ? How would that one look like ?

@PVince81
Copy link
Contributor Author

Something I noticed is that the scanner still operates and tries to detect changes even for the home storage. It might make sense to disable the scanner completely for the home storage as the use case is that this one is ONLY used through ownCloud.

It would be good to do some performance tests and see whether it helps in any way.

Disabling the scanner would also help with concurrency issues (see #11795) where the file scanner detects FS changes in the home storage, changes done by another process which didn't have a chance yet to update the cache to match the new data.

Or... if the scanner is not disabled, allow it to run only once per user. Which means multiple scanner processes are not allowed to run at the same time for the same user.

@PVince81 PVince81 mentioned this issue Nov 28, 2014
22 tasks
@PVince81
Copy link
Contributor Author

  • prevent scanner concurrency

@PVince81
Copy link
Contributor Author

PVince81 commented Mar 1, 2016

Scanner concurrency issues have been solved and propagation has been reworked too.

I think we can close this.

@PVince81 PVince81 closed this as completed Mar 1, 2016
@lock
Copy link

lock bot commented Aug 6, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked as resolved and limited conversation to collaborators Aug 6, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

1 participant