dir.list.filter().toList does not close the directory file descriptor

[bchazalet]

I kept getting a "too many files open" exception in my program which processes a large number of files and scans a large number of directories. After investigating I found out that my number of file descriptors kept forever increasing. The problem is with the way I listed directories, namely:

val jsonOnDisk = jsonDir
  .list
  .filter(f => f.isRegularFile && (f.extension == Some(".json") || f.extension(includeAll = true) == Some(".json.gz")))
  .toList

Doing so, the jsonDir file descriptor is kept open forever. I use lsof -p <pid> | egrep 'DIR|REG' to see the open file descriptors for my process.

I found that doing this solves my problem:

val jsonOnDisk = jsonDir
  .list(f => f.isRegularFile && (f.extension == Some(".json") || f.extension(includeAll = true) == Some(".json.gz")))
  .toList

I am not sure this is a bug, one could say I was not using the library the right way, but I'll let you decide.

[pathikrit]

Yes, the issue is java.nio.file.Files.list which better-file relies on returns a java.util.stream.Stream which needs to be closed.

[bchazalet]

Thank you @pathikrit !