'ipfs get' to checkout commits and MerkelDag objects as files

[dborzov]

We get the ipfs get command that allows to retrieve the MerkelDAG objects as files and directories.

Another thing is that I added commits. Now for every run of ipfs add that results in nonzero object addition, there is also a root commit node:

dag.Node{Data: "Commit Message", []*dag.Link:[(link to the root tree node),]}

I had to add it to make ipfs get work: there was this issue with the git add that the filenames weren't reflected in the resultant Node. These names are stored in dag.Link, not dag.Node. It makes perfect sense and is this way in git as well, because this way you can just link to the same node even when the file is renamed, duplicated or moved, but if you want to save filenames you need to have the root node.

A Practical Example.
let's store the IPFS's codebase in the IPFS commits!

     git clone [email protected]:jbenet/go-ipfs
     cd go-ipfs

Now we add all the files (including the .git directory)

     ipfs add -r *

and get a long list of the files added. And the last line contains the commit's hash:

     < ... >
     added commit QmW5SA2TvAB6smGb6dQMn6DME4f9R3N2Ayu7CT3WFzvRNi

Now we erase the whole directory and restore stuff like nothing happened:

     rm -rf *
     ipfs get QmZysrZPbXnmrCL6LeqEfHm5e69aR3hMyAqzTPexJ6k7Ku (commit's hash)

cool beans.

[jbenet]

Another thing is that I added commits. Now for every run of ipfs add that results in nonzero object addition, there is also a root commit node:

Hm, let's not for now. The goal is for the IPFS merkledag is to support applications creating their own data structures, whether they want to use commits or not. at the moment, ipfs add is like the ipfs plumbing part (think git plumbing vs porcelain). We'll want a porcelain commant (maybe it is ipfs add that should be porcelain, and ipfs write-object could be the underlying plumbing).

[jbenet]

but if you want to save filenames you need to have the root node.

I don't understand. ipfs add is not supposed to add a file's filename. filenames are not part of the file, they're part of the directory. e.g.

> mkdir /tmp/foo
> echo "bar" > /tmp/foo/bar
# setup 

> ipfs add /tmp/foo/bar
added /tmp/foo/bar QmaKtFK3wiq9z2LmhMKgoXvuB4XDeY5Xrac8vVBLPjc9CX
# only adds the contents of /tmp/foo/bar

> ipfs add -r /tmp/foo
added /tmp/foo/bar QmaKtFK3wiq9z2LmhMKgoXvuB4XDeY5Xrac8vVBLPjc9CX
added /tmp/foo QmcGSJT7v3sa9RA8pVSSTMrPREMQShj5ioEfU5bog7twqG
# adds the contents of /tmp/foo/bar
# adds the directory entries of /tmp/foo

Make more sense?

[jbenet]

I don't think ipfs add should auto-create commits. Maybe ipfs add should turn into ipfs write-object or ipfs add-object, and ipfs add move to ipfs fs import or something (eeeew, i like the simplicity of ipfs add, though it isn't very clear).

I think your PR is really about allowing creation of commits. I don't think these should be added by default on ipfs add, but instead, we should have the ability to create commits (or other arbitrary data structures) easily from the cli. (basically, we want git-commit-tree, and another generalized way of writing these datastructures, maybe by passing in a (json doc, or being able to specify the format)

[dborzov]

I see your point.
How do you envision these "plumbing commands" allowing you to create arbitrary DAG structures?

Should it be something that uses json serilization to create stuff?
So that you would be able to read it with:

$ ipfs object-cat <object-hash> --json
{"data": "Blah-blah",
 "links": [
     {
         "type": "commit",
         "name": "Such commit message wow",
         "hash": <object's hash>
      }, [...]
]}

and create it with:

$ cat define_object.json
{"data": "Blah-blah",
"links": [
     {
         "type": "commit",
         "name": "Such commit message wow",
         "hash": <object's hash>
      }, [...]
]}
$ cat define_object.json | ipfs object-add
object <new-object's hash> added

or something else?

[dborzov]

Also, how is the object's type (blob, tree, commit, list) to be defined internally? The white paper on page 8 (3.6.2) suggests that the object's data field is reused to define the type of the links from the links slice attribute.

So that we get for tree:

{'data': ["blob", "tree"],
 `links`: [
       {"hash": 1},
       {"hash": 2},
   ]
}

If I am reading this right, what is the motivation to reuse the data field to define the links' types?

How do you see it implemented since data is []byte? Should it be something like:

const (
    blobFlag: byte("b"),
    treeFlag: byte("t"),
   ....
)

and typical data field to be:

node.data = []byte{blobFlag, blobFlag, treeFlag}

or am I missing something?

[dborzov]

Also, a noob question: am I running the tests locally wrong? I ran

 $ go test ./...

from the root directory and it says all tests went ok. And I can't find the one that Travis CI says fails.

¯_(ツ)_/¯

[whyrusleeping]

The failing Travis CI test is because we cant write to the testrunners home directory... Need to find some way to disable/get around that test so we can reliably use Travis CI again.

[jbenet]

Sorry @dborzov -- August was a very, very time consuming month so I didn't get to your questions. Better late than never, i guess.

How do you envision these "plumbing commands" allowing you to create arbitrary DAG structures?

Very much like git's plumbing commands. See:

• https://www.kernel.org/pub/software/scm/git/docs/gitcore-tutorial.html -- walks through various porcelain actions as plumbing.
• http://git-scm.com/book/en/Git-Internals-Git-Objects - one of the best write ups i've seen
• http://rypress.com/tutorials/git/plumbing.html - does too, + visuals

Should it be something that uses json serilization to create stuff?

I really like the idea of making plumbing commands be able to take their input in json. The git plumbing uses their own format all over the place (usually just newline-delimited rows of whitespace delimited tokens), and in some cases that will make sense for us too. BUT! JSON is a great format. We should use it too.

By too, i really mean multi-format. We should write our plumbing command outputs as structs, and connect encoders at the end that can dump out json, newline-delimited stuff (for unix tools and for human readability), etc.

Also, how is the object's type (blob, tree, commit, list) to be defined internally?

The paper will change, btw. it will place much less emphasis on these four types of objects, and just suggest them as the sort of things that are possible.

How do you see it implemented since data is []byte? Should it be something like:

This is something still under discussion. there are multiple options:

1. self-description: the data field itself contains its type, perhaps as a header.
2. a type field added to the object
3. type added as a link with a well-known key name (e.g. @type).
4. using full JSON-LD (IPFS-LD or MDAG-LD), with an @context

So far, I'm leaning: 4 > 1 > 3 > 2

$ go test ./...

Yep, that's right. tests should pass now. We should probably add a make test to the root Makefile, given we have one.

Hope all that helps! Sorry again for delay.

[dborzov]

oh great, tnx for the detailed response!

[whyrusleeping]

What is the status here?

[jbenet]

I think this has several commits already in master-- maybe needs rebasing?

[dborzov]

Hey hey, I will fix this pull request and let you guys know :)