Fix: Prevent multiple processes from copying the same file when using…

[dallmann-uniwue]

… the file: protocol

Before submitting

• Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure to update the docs?
• Did you write any new necessary tests?

What does this PR do?

Fixes #351 by guarding copying the index.json to the cache directory with a FileLock when using the file: protocol.

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.

Did you have fun?

Sure! 🙃

[tchaton]

Looks good !

[bhimrazy]

Thank you for the PR, @dallmann-uniwue.

It appears that the errors are primarily due to the change in the total count caused by the addition of lock files.
Should we adjust the total count to account for these changes?

[dallmann-uniwue]

@bhimrazy
Hi!

I have taken a shot at fixing the tests. Rather then just adjusting the expected file counts, I opted to filter out the lock files. What do you think?

[bhimrazy]

@bhimrazy Hi!

I have taken a shot at fixing the tests. Rather then just adjusting the expected file counts, I opted to filter out the lock files. What do you think?

Thank you @dallmann-uniwue , Looks great. 😊

[codecov]

Codecov Report

Attention: Patch coverage is 90.00000% with 1 line in your changes missing coverage. Please review.

Project coverage is 78%. Comparing base (8382067) to head (592cd6b).
Report is 1 commits behind head on main.

Additional details and impacted files

@@         Coverage Diff         @@
##           main   #353   +/-   ##
===================================
  Coverage    78%    78%           
===================================
  Files        34     34           
  Lines      5008   5016    +8     
===================================
+ Hits       3922   3929    +7     
- Misses     1086   1087    +1     

[dallmann-uniwue]

You are welcome @bhimrazy! Thank you for working on this, litdata is making my life a lot easier!