Only new files or touched files indexed

[sebcapside]

Hi,
I use docker version 2.10
fscrawler just index new files, or touched files
not files already there, or copied from my laptop to server.
it seems to be like that.
very strange.

[dadoonet]

If you ran already FSCrawler once, stopped it and then started again, that's expected.
If you want to start from scratch, you need to use the --restart option or remove the _status.json file in the job dir.

[sebcapside]

That is what i've already try...
--restart doesn't take old files...
new share,
no files
just put some files and folders on (some new files, some old files, copied from my laptop)
mode trace show it will index them : [/Assemblage/exemple.indd] can be indexed: [true]
the files modified today are indexed
the files modified since a while arenot indexed.

folders, or metadatas for all files are indexed...
(just another thing, i can not use excludes: - "/._" as exclusion terme so it index me all metadatas files générates from my mac)
but never old file...
even on fist time scan

Metadata files : (Regexp issue)
15:24:13,497 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] directory = [false], filename = [/Assemblage/TOTO/.SMOOTHI 25.jpg], includes = [[*.pdf, *.psd, *.indd, *.ai, *.tif, *.jpg, *.xls, *.doc, *.psb, *.zip]], excludes = [[.]]
15:24:13,497 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] filename = [/Assemblage/TOTO/.SMOOTHI 25.jpg], excludes = [[.]]
15:24:13,497 TRACE [f.p.e.c.f.f.FsCrawlerUtil] regex is [._.?]
15:24:13,497 TRACE [f.p.e.c.f.f.FsCrawlerUtil] does not match any exclude pattern
15:24:13,497 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] filename = [/Assemblage/TOTO/._SMOOTHI 25.jpg], includes = [[.pdf, *.psd, *.indd, *.ai, .tif, .jpg, .xls, .doc, .psb, .zip]]
15:24:13,497 TRACE [f.p.e.c.f.f.FsCrawlerUtil] regex is [.?.pdf]
15:24:13,497 TRACE [f.p.e.c.f.f.FsCrawlerUtil] regex is [.?.psd]
15:24:13,497 TRACE [f.p.e.c.f.f.FsCrawlerUtil] regex is [.?.indd]
15:24:13,497 TRACE [f.p.e.c.f.f.FsCrawlerUtil] regex is [.?.ai]
15:24:13,497 TRACE [f.p.e.c.f.f.FsCrawlerUtil] regex is [.?.tif]
15:24:13,497 TRACE [f.p.e.c.f.f.FsCrawlerUtil] regex is [.?.jpg]
15:24:13,497 TRACE [f.p.e.c.f.f.FsCrawlerUtil] does match include regex
15:24:13,497 DEBUG [f.p.e.c.f.FsParserAbstract] [/Assemblage/TOTO/._SMOOTHI 25.jpg] can be indexed: [true]
15:24:13,497 DEBUG [f.p.e.c.f.FsParserAbstract] - file: /Assemblage/TOTO/._SMOOTHI 25.jpg
15:24:13,497 DEBUG [f.p.e.c.f.FsParserAbstract] - not modified: creation date 2023-02-09T14:44:57.402865 , file date 2023-02-09T14:44:57.402865, last scan date 2023-02-09T15:21:11.363678

[dadoonet]

The problem is here:

 last scan date 2023-02-09T15:21:11.363678

If restart works, this value should be null.

[sebcapside]

hi,
seems to be good if i remove the folder with status.json
set --rescan is not enought...

but :
if your share drive scan files today at 10h, it add last scan at 10h
so, if you add some files on this share, created several days ago, it seem to compare the date, and then not index them...
that explain my trouble...

it seems i need to remove last scanned date every scan....
(this solution is used for spotlight search)

[dadoonet]

set --rescan is not enought...

restart or rescan?

so, if you add some files on this share, created several days ago, it seem to compare the date, and then not index them...
that explain my trouble...

yes. It depends on what the OS does. If it does not modify any date on move, you need to either touch the file or do a copy instead of a move.

until the new behavior is implemented see #399