Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Needed Filestore Command For Basic Implementation #3646

Open
kevina opened this issue Feb 2, 2017 · 12 comments
Open

Needed Filestore Command For Basic Implementation #3646

kevina opened this issue Feb 2, 2017 · 12 comments
Labels
need/community-input Needs input from the wider community

Comments

@kevina
Copy link
Contributor

kevina commented Feb 2, 2017

Relates to #3629

Fundamental:

  • ls - list contents of filestore
  • verify - verify blocks in filestore
  • rm - remove blocks from just the filestore - either than or enhance block rm to support removing blocks from either the filestore or the main store. In either case is important that pinned blocks should be allowed to be removed from one store as long as they are also in the other
  • dups - list blocks both in the main store in the filestore.

Nice to have:

  • mv move a block into the filestore. That is copy the block into the filesystem and add it to the filestore without having to rehash it.
@Kubuxu Kubuxu added the need/community-input Needs input from the wider community label Mar 15, 2017
@dokterbob
Copy link
Contributor

ls, verify and dups have been implemented.

It seems rm is implicitly implemented through garbage collection (is it?) but might be very nice to have, i.e. to remove corrupted/removed files listed by verify.

I'm not sure what mv is supposed to do, would it be equivalent to ipfs get -r and a subsequent ipfs add -r --nocopy? In that case: how much priority has it, what are possible use cases?

@kevina
Copy link
Contributor Author

kevina commented May 13, 2017

Basically but without the -r, it would be equivalent to ipfs get then ipfs add --nocopy. It would move a single file into the filestore without rehashing the file. It is not difficult to implement but I am not sure how useful it will be so it is not a high priority.

@whyrusleeping
Copy link
Member

@kevina that could be pretty useful if we could find a way to skip ever adding the item in question to the blockstore. Basically ipfs download straight into the filestore

@dokterbob
Copy link
Contributor

dokterbob commented May 14, 2017 via email

@whyrusleeping
Copy link
Member

@dokterbob Yeah, that would be really nice, but i feel thats likely more work (getting fuse very stable) than the filestore command.

@kevina
Copy link
Contributor Author

kevina commented May 14, 2017

@whyrusleeping bypassing the blockstore could be possible but harder, my current implementation is more like a copy, the node will still be in the blockstore, it is simply faster because it avoids recomputing the hash for the children.

@djdv
Copy link
Contributor

djdv commented Aug 31, 2017

What of the clean command? Would this instead be a parameter to verify i.e. ipfs filestore verify --clean?
https://github.com/ipfs-filestore/go-ipfs/blob/master/filestore/README.md#removing-invalid-blocks

I've run into an issue where I added files via the filestore, changed the names, and then added them again, the daemon was reporting it could not find the files (it was still using the original names) and was not serving data for them from the new name, I added them to the blockstore to deal with it in the meantime since I wasn't sure how to remove them without doing a full gc.

@whyrusleeping
Copy link
Member

@kevina thoughts on @djdv's comment?

@kevina
Copy link
Contributor Author

kevina commented Sep 2, 2017

@whyrusleeping a good first step would be to merge #3815. I can then combine it with the verify command to implement clean which is basically ipfs filestore verify followed by ipfs filestore rm.

@kevina
Copy link
Contributor Author

kevina commented Sep 2, 2017

related comment (#3629 (comment)) that was never addressed:

@whyrusleeping concerning replacing blocks. How about for now we make it an option? There needs to be a way to fix broken blocks due to files moving. One option would be to remove the block and then readd it, but pinning makes that a more complicated operation.

@yonderblue
Copy link

How does this currently work, or is there a doc somewhere about it?
For example if you ipfs add --nocopy somefile.txt and then os.Remove("somefile.txt"), will the file eventually be cleaned out of the filestore? Does that mean that ipfs filestore verify will continue to return no-file until that happens?

@Stebalien
Copy link
Member

will the file eventually be cleaned out of the filestore?

Not unless you remove those blocks. The current solution is to run verify and then remove all missing blocks manually.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
need/community-input Needs input from the wider community
Projects
None yet
Development

No branches or pull requests

7 participants