Discussion about common method of keeping track of (to) scanned files

[bartv2]

From owncloud/music#90 (comment):

I had to find a solution for the 'scan a lot of files' problem when working on search_lucene.

IMHO The files app has the best scanning mechanism because it saves the scan status in the filecache table. Can we find a general solution for scanning mechanisms? How do we mark a file as scanned by search_lucene (I currently join my own status table with the filecache table ... but meh) or by music (or by pictures or by videos)? All of them could extract meta data. I personally would like to explore zend_lucene for searching through that ... (cc @andrewsbrown) ... but I may be biased. Perhaps we can come up with a more general search API that can use either the database or zend lucene ....

Anyway. I propose the following approach: we create a new onscan event/hook that apps that want to extract meta data can register for. Correct me if I am wrong but the files app currently determines the mime type and basic file meta data. music could listen for audio files and then use getid3 to extract more specific meta data. search_lucene likewise. Unfortunately, this naive implementations will cause synchronization to take longer because the file app scans the file on write causing an upload via webdav to wait for all meta extraction to be complete. Therefore we need to separate "file app"-scanning and meta data indexing from each other. This is where background processes come into play. I just don't know yet if adding a separate job for each file makes sense or if just adding a "reindex" job is more elegant. The problem with a "reindex" job is that you don't want to reindex all files but only the ones that are new. In search_lucene I use a status table to keep track of this. Music could use a flag in the songs table but then we are duplicating the mechanism. Single jobs will fill the jobs table quite quickly ...

I also have a table for the antivirus background scanner, so having a common sollution would be nice
@icewind1991 @karlitschek @kabum @DeepDiver1975

[DeepDiver1975]

We recently introduced some hooks to the scanner ... @icewind1991

[DeepDiver1975]

https://github.com/owncloud/core/blob/master/lib/private/files/cache/scanner.php#L159

[MorrisJobke]

@DeepDiver1975 Does this solves the problem for an initial scan?

[DeepDiver1975]

@kabum you are talking about the situation where there are files and an app is enabled and has to perform an initial scan - right?

[MorrisJobke]

@DeepDiver1975 yes

[PVince81]

Is this still valid ?

[MorrisJobke]

@PVince81 Yes. This is about storing a "processed" status flag on a per app basis.

[PVince81]

We already have scan hooks, these should be triggered also only for new files.
Initial scan now happens in a background job.
For the processed flag, every app could, if needed, decide to store such flag in an app-specific table.

@butonic what do you think, in relation to search ? Keep or close ?

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.