git on IPFS (ipld-git)

[rht]

This could mean either:

1. a git which uses ipfs as transport and host
2. a git which uses ipfs-merkledag as packfile
3. all of the above

The first one has been written https://github.com/cryptix/git-remote-ipfs (and in git-annex https://git-annex.branchable.com/special_remotes/ipfs/).

The second might already have an implementation via git-annex.
I favor this approach (replacing git's underneath to work with distributed large files, i.e. a distributed https://github.com/github/git-lfs) than to rewrite a different dag versioning interface as in #23.

[chriscool]

There is also: https://github.com/larsks/git-remote-ipfs

CC @cryptix @larsks

[cryptix]

I'd like to also push the envelope on this.

During my work on the helper, I wished for a an option in the commit object to just use an IPFS hash to point to the corresponding files instead of their tree objects. I don't think this is possible by just beeing an git-remote-helper.

IIRC git-annex 'just' links additional data on ipfs by adding the hashes into the commit message?

A ipfs-aware git could use multihashes everywhere and wouldn't need to repack files in their format. Having seen the kauderwelsch that is the git codebase and documentation, this idea already hurts... :-/

Maybe it's better if the helper just act as a tranformator - speaking and returning native git while storing commits and trees in a format more suited to what ipfs offers.

[whyrusleeping]

kauderwelsch

Had to google that one

[jbenet]

During my work on the helper, I wished for a an option in the commit object to just use an IPFS hash to point to the corresponding files instead of their tree objects. I don't think this is possible by just beeing an git-remote-helper.

No, i don't think git cares about non git objects

IIRC git-annex 'just' links additional data on ipfs by adding the hashes into the commit message?

yes, git-annex tracks links (with digests too i think) but "off-repo".

A ipfs-aware git could use multihashes everywhere and wouldn't need to repack files in their format. Having seen the kauderwelsch that is the git codebase and documentation, this idea already hurts... :-/

:/ -- but take a look at #23 -- we can make our own VCS structures that can point to anything. And, not sure what's easier, but i think we may be able to reuse lots of the git tooling as it's purely shell scripts on top of others.

Maybe it's better if the helper just act as a tranformator - speaking and returning native git while storing commits and trees in a format more suited to what ipfs offers.

Yeah i think this is fine for the helper.

[cryptix]

:/ -- but take a look at #23 -- we can make our own VCS structures that can point to anything. And, not sure what's easier, but i think we may be able to reuse lots of the git tooling as it's purely shell scripts on top of others.

My statement maybe came of a little too negative. I'm all for #23! I just don't want to fork all of git to be IPFS-aware right now.

[daviddias]

@magik6k mind sharing an update on IGiS and all your work with ipld-git here? :)

[magik6k]

Sure!

• git plugin will be included in the next go-ipfs release by default - plugin: preload git plugin by default kubo#4991
• git-remote-ipld internals are in good shape (it even supports large objects by mapping them to unixfs files), ux still needs to see some terraforming ([WIP] Make ipld remote operate on what ipns helper would point to ipfs-shipyard/git-remote-ipld#13, implementing the ipns helper for real).
• Large repos work quite well, with reasonable performance
  • not as fast as 'native' push/pull (~10 slower with objects in local repo, painfully slow when needing to fetch stuff (though acceptable for go-ipfs sized repos))
  • Managed to push a snapshot of linux-stable with all the tags - https://ipfs.io/ipfs/QmXxHNMkv7itDitAKhUDKMEdKi2aVtf8DYWcxoDqfHCqdc/
  • It's not easily pinnable as some objects can't be fetched due to bitswap object size limit.
    • I plan to implement pinning utility on the remote helper, though if we somehow made object mappings into a standard ipfs pin could possibly work for that.
• IGiS received a LOT of awesome updates, much of that is thanks to @dirkmc
  • Some features aren't exposed yet (e.g. comparing branches)
  • It would be great if js-ipfs got some content discovery mechanism.
    • Even though it's directly connected to the server with the repo pinned, it sometimes fails to fetch some objects (didn't dig deep enough to be sure on which side the problem is (the objects appear in wantlist))
    • You can't push a repo to ipfs and have it 'just work'
  • It has many of the 'basic' gitweb-like features implemented, the fun parts are still to be done
    • Some stuff that could be nice with crdts: issues, commit comments, maybe pull-requests
      • PRs - The 'pull' part will probably need to happen via the terminal, there is some UX to think about
    • Stuff like individual file history, blame, etc. may involve some interesting data structures, can also be a good test-bed for ipld selectors
  • Got a home under https://igis.io
  • Good example repo is https://igis.io/#/repo/QmU1HJJDFSM8JJq4r31wSLfj51oysQCswz7aL78UWZHuMC

[dirkmc]

IGiS is a little slow unless you have the files cached in your local IPFS instance, so here's a quick demo video showing a couple of the features (51s):
https://www.youtube.com/watch?v=FXsezRZXnWA

[whyrusleeping]

@dirkmc any idea what the root of the slowness is? if its anything perf related on go-ipfs i'd love to open an issue to bring things up to speed

[magik6k]

I'm pretty sure it's ipfs/kubo#5247

[Stebalien]

Crap.

[dirkmc]

It occasionally seems to hang when retrieving some objects, to be honest I'm not sure why. Having said that I just tried it now and it seems to be working well today. Note that IGiS logs object fetches at the debug level, in case you want to click around and watch what it's doing:

[dirkmc]

@whyrusleeping I remember you mentioning that bitswap will soon support graph queries rather than just individual blocks. Is that what Bitswap Sessions are about? Will graph queries be exposed to IPFS clients or only internally? I'm thinking for example of the case of asking IPFS for the last 20 commits on a branch.

[Stebalien]

No, they're two separate things. Sessions allow us to remember which peers likely have content so we can ask them instead of repeatedly spamming the entire network. The next step with sessions is to ask for different pieces from different peers.

Graph queries allow us to ask for pieces we don't yet know we want (i.e., children of node x).