[registry-facade] Add IPFS support

[csweichel]

Description

This PR adds IPFS support to registry-facade. This is an optional feature and

• must be explicitly enabled in the config,
• requires IPFS and Redis to run within the cluster,
• has no support for GC, i.e. everything that's uploaded to IPFS remains there,
• has no observability for this feature,
• has unclear performance and stability characteristics: it's unclear how much overhead the Redis GET on each manifest production incurs.

Really, this is something that came out of a four our session from a hackathon.

How it works

Whenever a blob gets pulled through registry-facade, registry-facade will upload that blob to IPFS and record the IPFS CID in Redis with the sha hash of the blob as key. The next time a manifest which contains the blob is produced, we'll add an ipfs:// URL to the OCI descriptor of that blob.

If the snapshotter supports IPFS URLs (like the stargz snapshotter does), image pulls should be accelerated. If the snapshotter does not support IFPS (like most other snapshoters), the image pull will likely fail because the snapshotter cannot handle ipfs URLs.

Related Issue(s)

Fixes #8186

How to test

This PR is best tested locally (i.e. in a Gitpod workspace):

IPFS setup:

# install ipfs
cd /tmp
curl -L https://dist.ipfs.io/go-ipfs/v0.12.0/go-ipfs_v0.12.0_linux-amd64.tar.gz | tar xzv
cd go-ipfs
sudo sh install.sh

# setup and start daemon
ipfs init
jq '.Discovery.MDNS.Enabled=false'  ~/.ipfs/config > ~/.ipfs/config-new && mv ~/.ipfs/config-new ~/.ipfs/config
ipfs daemon

Redis setup:

docker run --name redis -p 6379:6379 -d redis

Tool setup:

go install github.com/csweichel/oci-tool@latest

Start registry-facade

cd components/registry-facade
go run main.go run example-config.json

# MANUAL STEP: please expose port 9090 publicly

Pull an image for the first time (make sure your docker damon cache is empty so that Docker actually pulls the blobs):

ref="$(gp url 9090 | cut -d '/'  -f 3)/fixed/foo:latest"

# inspect the manifest and observe that there are no IPFS URLs
oci-tool fetch manifest $ref

# pull the blobs which triggers the caching
docker pull $ref

# inspect the manifest and observe that now there are IPFS URLs
oci-tool fetch manifest $ref

Release Notes

Experimental IPFS support for cluster-local image caching

[princerachit]

I see an error when trying to test

gitpod /tmp/go-ipfs $ ipfs daemon
Initializing daemon...
go-ipfs version: 0.12.0
Repo version: 12
System version: amd64/linux
Golang version: go1.16.12
2022/03/30 07:41:34 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
2022/03/30 07:41:34 [zeroconf] no suitable IPv4 interface: udp4: failed to join any of these interfaces: []
2022/03/30 07:41:34 [zeroconf] no suitable IPv6 interface: udp6: failed to join any of these interfaces: []
2022-03-30T07:41:34.105Z        ERROR   cmd/ipfs        ipfs/daemon.go:430      error from node construction: no supported interface

Error: no supported interface

also, there might be a mismatch in jq version as it complains that the flag -i is not supported. However, this works:

gitpod /tmp/go-ipfs $ jq '.Discovery.MDNS.Enabled=true'  ~/.ipfs/config > ~/.ipfs/config-new && mv ~/.ipfs/config-new ~/.ipfs/config

Can you share which cluster you tested this in? When I try to increase the buffer size I see an error:

gitpod /sys/fs/cgroup/net_cls $ sysctl -w net.core.rmem_max=2500000
sysctl: cannot stat /proc/sys/net/core/rmem_max: No such file or directory
gitpod /sys/fs/cgroup/net_cls $ 

[csweichel]

Thanks all for the reviews :)
Had another go.

[princerachit]

worth printing the error in the log statement?

my bad. Wrong line comment.

[csweichel]

/hold

[sagor999]

Code looks good, but like Prince I cannot test this for some reason, getting same error:

$ ipfs daemon
Initializing daemon...
go-ipfs version: 0.12.0
Repo version: 12
System version: amd64/linux
Golang version: go1.16.12
2022/03/31 00:11:56 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
2022/03/31 00:11:56 [zeroconf] no suitable IPv4 interface: udp4: failed to join any of these interfaces: []
2022/03/31 00:11:56 [zeroconf] no suitable IPv6 interface: udp6: failed to join any of these interfaces: []
2022-03-31T00:11:56.357Z        ERROR   cmd/ipfs        ipfs/daemon.go:430      error from node construction: no supported interface

Error: no supported interface

@csweichel am I missing a step? I an doing this in gitpod workspace, opening this PR.

[csweichel]

My bad: I had the ipfs settings wrong in the description. The line should read:

jq '.Discovery.MDNS.Enabled=false'  ~/.ipfs/config > ~/.ipfs/config-new && mv ~/.ipfs/config-new ~/.ipfs/config

[sagor999]

Tested. Looks good.

[aledbf]

Why registryFacade is inside workspace?

[csweichel]

here workspace refers to the workspace team

[mrsimonemms]

Looks good. From an Installer point of view, do we want to add an in-cluster Redis, or is that something that we're only explicitly supporting an external resource?

Also, I think this will be a good candidate as the first component that has "graduated" from experimental to the stable config. We're probably not there yet, but this is definitely going to be a useful addition for many of the bigger/enterprise self-hosted users

[csweichel]

/werft run with-clean-slate

👍 started the job as gitpod-build-cw-regfac-ipfs.28

[Furisto]

Rebased this on top of Simon's replicated changes, so that the werft build passes again.

[aledbf]

LGTM

[nandajavarma]

LGTM