Needed Filestore Command For Basic Implementation

[kevina]

Relates to #3629

Fundamental:

• ls - list contents of filestore
• verify - verify blocks in filestore
• rm - remove blocks from just the filestore - either than or enhance block rm to support removing blocks from either the filestore or the main store. In either case is important that pinned blocks should be allowed to be removed from one store as long as they are also in the other
• dups - list blocks both in the main store in the filestore.

Nice to have:

• mv move a block into the filestore. That is copy the block into the filesystem and add it to the filestore without having to rehash it.

[dokterbob]

ls, verify and dups have been implemented.

It seems rm is implicitly implemented through garbage collection (is it?) but might be very nice to have, i.e. to remove corrupted/removed files listed by verify.

I'm not sure what mv is supposed to do, would it be equivalent to ipfs get -r and a subsequent ipfs add -r --nocopy? In that case: how much priority has it, what are possible use cases?

[kevina]

Basically but without the -r, it would be equivalent to ipfs get then ipfs add --nocopy. It would move a single file into the filestore without rehashing the file. It is not difficult to implement but I am not sure how useful it will be so it is not a high priority.

[whyrusleeping]

@kevina that could be pretty useful if we could find a way to skip ever adding the item in question to the blockstore. Basically ipfs download straight into the filestore

[dokterbob]

Wouldn’t it be better to focus on getting FUSE stable enough and then symlink’ing there, or perhaps wrapping around that functionality with a cheap shell script?

[whyrusleeping]

@dokterbob Yeah, that would be really nice, but i feel thats likely more work (getting fuse very stable) than the filestore command.

[kevina]

@whyrusleeping bypassing the blockstore could be possible but harder, my current implementation is more like a copy, the node will still be in the blockstore, it is simply faster because it avoids recomputing the hash for the children.

[djdv]

What of the clean command? Would this instead be a parameter to verify i.e. ipfs filestore verify --clean?
https://github.com/ipfs-filestore/go-ipfs/blob/master/filestore/README.md#removing-invalid-blocks

I've run into an issue where I added files via the filestore, changed the names, and then added them again, the daemon was reporting it could not find the files (it was still using the original names) and was not serving data for them from the new name, I added them to the blockstore to deal with it in the meantime since I wasn't sure how to remove them without doing a full gc.

[whyrusleeping]

@kevina thoughts on @djdv's comment?

[kevina]

@whyrusleeping a good first step would be to merge #3815. I can then combine it with the verify command to implement clean which is basically ipfs filestore verify followed by ipfs filestore rm.

[kevina]

related comment (#3629 (comment)) that was never addressed:

@whyrusleeping concerning replacing blocks. How about for now we make it an option? There needs to be a way to fix broken blocks due to files moving. One option would be to remove the block and then readd it, but pinning makes that a more complicated operation.

[yonderblue]

How does this currently work, or is there a doc somewhere about it?
For example if you ipfs add --nocopy somefile.txt and then os.Remove("somefile.txt"), will the file eventually be cleaned out of the filestore? Does that mean that ipfs filestore verify will continue to return no-file until that happens?

[Stebalien]

will the file eventually be cleaned out of the filestore?

Not unless you remove those blocks. The current solution is to run verify and then remove all missing blocks manually.