Concurrent scanners on the same folder, duplicate entry exception

[PVince81]

Goal is to have two parallel scanners run on the same new folder, indexing it at the same time.

Steps to reproduce

1. Add sleep(5) here: https://github.com/owncloud/core/blob/master/lib/private/files/cache/scanner.php#L319 (in the loop of scanChildren())
2. Close all web UI and sync client to avoid triggering the scanner that way
3. Go to the "data/root/files" folder directly on the disk
4. Create a folder "new" and a few text files inside, for example "new/one.txt", "new/two.txt", etc
5. Change the permissions of "new" to the ones of the web server (to avoid permission issues). Note: such remote change could be simulated on external storage too
6. Prepare two terminals
7. Run curl -X PROPFIND http://root:admin@localhost/owncloud/remote.php/webdav/ in both terminals in parallel

Expected result

No error logged

Actual result

{"reqId":"IqBILakdIGgYzMbq4Yq8","remoteAddr":"127.0.0.1","app":"core","message":"Exception while scanning file \"files\/new\": An exception occurred while executing 'INSERT INTO `oc_filecache` (`mimepart`,`mimetype`,`mtime`,`size`,`etag`,`storage_mtime`,`permissions`,`parent`,`path_hash`,`path`,`name`,`storage`) SELECT ?,?,?,?,?,?,?,?,?,?,?,? FROM `oc_filecache` WHERE `storage` = ? AND `path_hash` = ? HAVING COUNT(*) = 0' with params [\"1\", \"2\", 1432732331, -1, \"5565c2db9ef5c\", 1432732331, 31, \"3\", \"5abe554a39b852348e527570ee931bbb\", \"files\\\/new\", \"new\", \"1\", \"1\", \"5abe554a39b852348e527570ee931bbb\"]:\n\nSQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '1-5abe554a39b852348e527570ee931bbb' for key 'fs_storage_path_hash'","level":0,"time":"2015-05-27T13:13:00+00:00","method":"PROPFIND","url":"\/owncloud\/remote.php\/webdav\/"}

Versions

ownCloud 8.1 git master (3de945d)

The error message itself might be ok, as the scanner will skip the file it couldn't insert.
It still might feel like such cases aren't properly/gracefully handled.

This should improve with high-level file locking #11804 where the scanner should also lock folders to avoid concurrent scanners to enter them.

The reproduction steps will be useful for testing.

@icewind1991 @DeepDiver1975 FYI

[PVince81]

Setting to 8.1, with the hope that the high-level file locking will fix it.

[ghost]

@PVince81 is this ready for testing with the new updates in place?

[PVince81]

Unlikely. There are still issues with scanner locking at the moment which seem quite difficult to solve, see #16963

[PVince81]

Retested with #17017 and #16963 combined and I could not reproduce the issue.