Recognize hard links as new files

[codecivet]

I can see that synoindexwatcher is running:

$ ps aux | grep synoindexwat
root      7003  0.0  0.4  14296  2232 ?        S    Dec15   0:21 /bin/python -m synoindexwatcher --config=/usr/local/etc/synoindexwatcher.conf

And the logfile says it is waiting for updates:

2022-12-15 13:28:44,286 INFO Adding watch for path: /volume1/media/Videos
2022-12-15 13:28:52,937 INFO Waiting for media file changes...

But I can confirm that there is a new file in that hierarchy that has not been added to my media index. I'm assuming it searches recursively, right? Any other info that I can use to debug?

[letorbi]

Hej codecivet,

Synoindex Watcher does not automatically add missing files to the media index. Only new files, which have been moved/copied into one of the watched directories (including sub-directories), will be automatically added to the media index.

However, if you want to re-add all files in the watched folders, you could call

$ python -m synoindexwatcher --rebuild-index --config=/usr/local/etc/synoindexwatcher.conf

from the command-line once. This should re-add all existing files to the media index.

If you want more output for debuuging, you can either set the log-level in your config-file to loglevel=DEBUG or append --loglevel=DEBUG to the command line.

Cheers,
Torben

[codecivet]

Thanks for the reply and I should have been more specific that this was a new file that wasn't being detected. I have enabled debugging and added a bunch more debugging statements. My current suspicion is that NFS appears to be triggering (hundreds of?) thousands of MODIFY and CLOSE_WRITE events for large files and those might be overwhelming synoindexwatcher. For now I've tried disabling detection of the MODIFY event, but I'm not sure what repercussions that will have.

[letorbi]

Thanks for the clarification. I will definitely try to look into the issue, but I am quite busy right now, so it might take some time. However, I am always happy about pull request, so feel free to dig deeper into the issue, if you need a fast solution. Sorry that I cannot offer you more right now.

[codecivet]

I think I figured out the issue, although I am still in the process of confirming:

I was using a program that automatically renames and moves files on the NFS share. By default, it does that by using hard links. I believe something about using hard links caused synoindexwatcher (or perhaps inotify) to miss the files getting moved to their final location, probably since it wasn't really a move operation due to using a hard link.

Fortunately, I was able to disable the hard linking feature of my app and that seems to have resolved my issue. I have now reverted all changes I made to the source to check that everything is still working properly.

I haven't done any research to see if there would be an inotify-compatible method of handling hard links, but I'll close this issue once I have done some more verification.

[letorbi]

Awesome, thanks for your work. I really appreciate it. Please keep the issue open. I'd like to keep it as a reminder that the handling of hard links needs to be fixed.