How to keep only metadata in memory?

[tom-james-watson]

I would have thought that setting temp: true would store only the metadata in-memory, however it seems to store both the metadata AND all the files in memory, which seems to be confirmed by the docs - https://github.com/datproject/dat-node#storage-1

Is there a way to store only the metadata in-memory?

If you don't mind checking out my code, here's what I'm doing - https://github.com/tom-james-watson/dat-cp/blob/master/src/lib/dat-cp.js#L21. See https://github.com/tom-james-watson/dat-cp#dcp---dat-copy for context.

Note that I am manually adding files to the archive with dat.archive.createWriteStream and I never run importFiles. Changing the value of the dir provided (e.g. Dat('literally anything', ...) does not seem to have any affect with this setup, which is a little confusing.

Everything works fine with temp: true, however all files are stored in-memory. Removing temp: true alleviates the memory problem, however this then creates the .dat metadata directories, which I want to avoid.

It seems like it should be possible to keep only the contents of the .dat folder in memory. I get the impression that this could be done by writing my own storage function, but I wouldn't know where to start!

Is there anything I'm misunderstanding? Is there any way to achieve what I want? Thanks!

[tom-james-watson]

OK, so I've managed to get this working how I expected, which is awesome.

Here's the custom storage that I have ended up writing: https://github.com/tom-james-watson/dat-cp/blob/master/src/lib/storage.js, and here's how it's passed into Dat https://github.com/tom-james-watson/dat-cp/blob/master/src/lib/dat-cp.js#L23

As you can see, this is basically a modified clone of dat-storage. I've outlined the differences in the file's docstring.

It looks like I'm basically needing to duplicate a bunch of core dat functionality in order to implement this, which is not ideal. I would have suggest that the createStorage part of dat-storage simply needs to be extracted into its own module, but I also needed to modify the way raf is called here to let me be able to specify a different directory.

It looks like this is because I am breaking dat's assumption that all files contained in an archive will exist at a single root. In the case of dcp that doesn't make sense.

So, two suggestions:

• Modify dat-storage to handle paths that are not under a single root
• Modularize createStorage so that it's easier to write your own custom storage.

Unless, of course, none of what I said made sense and I'm going about this in the wrong way, in which case please let me know!

Thanks!

[arj03]

I ran into the same problem today with sharing single files so I'm adding myself to this issue in the hope that a solution will emerge.

and btw. dcp looks great :)