What's the trick to allow datasets to reuse cache when using hydra?

[mariomeissner]

I'm trying to call map in my DataModule in a hydra project but it doesn't load the cache for me. In lightning-transformers this is working properly though, so I was wondering if anyone knows what the trick is to make this work. If anyone wants to venture into checking my code, it's here.

[mariomeissner]

The cache only fails when running inside hydra AND is called within the trainer (in a subprocess I assume), since isolating the script and calling setup with or without hydra loads the cache correctly.

After countless hours of trial and error, it seems the key is the static method. After switching to a static method like is done in this project, I got the cache to work. I'd be happy to learn why this is, if anyone knows!