Support fetching docker images from V2 registries #32248

ixmatus · 2017-12-02T06:29:48Z

Motivation for this change

The stock dockerTools in nixpkgs support fetching docker images from a V1 docker registry and not a V2 registry; dockerTools pull the whole docker image into the nix store instead of pulling the layers, config, and manifest into the nix store. This is problematic because dockerTools only work with an out-of-date docker registry version and it is storage-inefficient; inefficient because multiple docker images sharing base layers do not take advantage of deduplication in the nix store.

hocker and the fetchdocker utilities fix these issues. hocker implements its own client against the docker distribution REST API and it does not use the docker client or docker daemon to pull an image from a registry. Each utility of hocker is designed to do one thing well, this translates naturally to a set of Nix functions that produce derivations that fetch docker image layers, the docker image configuration json, and produce an image compositor script. The image compositor script assembles the downloaded and the generated pieces of the docker image from the nix store, streaming it to stdout as a tar archive for use with docker load.

Things TODO next, after merging

Write a tutorial
Contribute a manual entry

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

This change adds granular, non-docker daemon docker image fetchers and a docker image layer compositor to be used in conjunction with the `docker2nix` utility provided by the `haskellPackages.hocker` package. This change includes a hackage package version bump and updated sha256 for recent fixes released to `hocker` resulting from formulating this patch.

This change adds a simple integration test exercising the fetchdocker Nix code and hocker utilities for the simple `hello-world` docker container. We exercise: - Fetching the docker image configuration json - Fetching the docker image layers - Building a compositor script - Loading the `hello-world` docker image into docker using the compositor script and `docker load` - Running that loaded container

ixmatus · 2017-12-02T06:30:36Z

CC: @fpletz @grahamc

peti · 2017-12-02T09:07:54Z

pkgs/development/haskell-modules/hackage-packages.nix

@@ -103952,8 +103952,8 @@ self: {
     }:
     mkDerivation {
       pname = "hocker";
-       version = "1.0.0";
-       sha256 = "16indvxpf2zzdkb7hp09zfnn1zkjwc1pcg2560x2vj7x4akh25mv";
+      version = "1.0.2";


The new version fails its test suite on NixOS: https://nix-cache.s3.amazonaws.com/log/fb2a5pp8l6k1ys5gjvj1djh9r8ba27bk-hocker-1.0.2.drv

Thank you for pointing that out, I had a late night oopsie and forgot to include an override that fixes this issue. I also need to take the temporary override I put in place in the test's machine.nix.

I also see the hackage-packages.nix has been updated so I'll drop this hunk.

It's failing its test suite due to a missing data file :-/ I thought I'd fixed that but apparently not. Since we need to override the derivation to wrap the programs with a PATH to nix I added a dontCheck for now; when I fix this fully I'll open another PR to remove the dontCheck.

peti · 2017-12-04T14:44:27Z

pkgs/development/haskell-modules/configuration-common.nix

+      ( dontCheck super.hocker )
+      ( oldDerivation: {
+          testToolDepends =
+            (oldDerivation.testToolDepends or []) ++[ pkgs.nix ];


I guess the package should declare its dependency on Nix, i.e. by depending on http://hackage.haskell.org/package/nix-paths?

The only utility from the hocker package that depends on Nix (other than the tests) is docker2nix; I think that's clear enough for now. I agree that a build-time dependency on Nix is better than a run-time dependency, so thank you for pointing me at nix-paths. Here are my reasons why I don't think we should gate this PR on that, though:

I don't entirely understand how nix-paths works; it looks like it has magic constants in it but I don't understand where those come from; a minimal README.md for the project would help a lot I think (note that if you explain it all to me, I'm also happy to submit a PR with a README.md contribution)

The utility that we need for docker2nix is nix-hash, which is not provided by nix-paths. I can submit a PR to fix that but I don't think we should gate this PR on that and I also want to understand how nix-paths works first before contributing more functionality

http://hackage.haskell.org/package/nix-paths-1.0.1 adds support for nix-hash. The paths are determined are configure-time by Cabal. Look at Setup.lhs for the implementation.

Thanks for adding support for nix-hash @peti. I'll probably get to that soon. In the meantime can I get a review of the other parts of this PR?

I don't think using nix-paths is a blocker for this PR but if it is, could you say so?

I would not call the hocker-related overrides a blocker. That's too strong a term, IMHO. What I can say though is that I won't merge the PR as-is. Other Nixpkgs maintainers may very well feel differently about it.

Okay, thanks for clarifying!

I'd like to clarify my understanding of nix-paths (would you accept a PR adding a README.md with anything I learn? That may help newcomers and provide a reference to point people at).

I see now that Setup.lhs discovers the absolute path to, say, nix-hash using lookupProgram which is great because if nix-hash is not available, it becomes a compile-time error and not a run-time error (I think this is great, btw, I wish this were more easily discovered!) Doing this removes the need for the wrapProgram invocations but it does require that nix-paths is included in the buildInputs of hocker's derivation (as a build-time and run-time dependency). Do I have this right?

Additionally, I think the generated derivation for the nix-paths hackage package only includes the Nix package in the nativeBuildInputs; shouldn't it be in the propagatedBuildInputs so that the same Nix used as a build-time dependency is included in the run-time closure of any program that depends on nix-paths?

Also, the generated derivation for the nix-paths hackage package restricts the hydra build to Linux: hydraPlatforms = [ "i686-linux" "x86_64-linux" ]; what's the reason for this? hocker is usable on Darwin systems, will the hydraPlatforms line prevent hocker from being integrated on a Darwin worker?

ixmatus · 2018-01-10T19:24:59Z

@peti hocker-1.0.4 addresses your concerns (it uses nix-paths). As part of merging master into my branch and resolving the conflict in configuration-common.nix I removed the hocker override that wrapped the executables with the output from the nix package.

ixmatus · 2018-01-30T19:01:18Z

@peti this is ready for re-review. Perhaps @grahamc would like to look at this too since he was interested in this work.

peti · 2018-02-02T10:50:01Z

pkgs/development/haskell-modules/configuration-common.nix

@@ -998,5 +998,4 @@ self: super: {
      cp $data/share/${self.ghc.name}/${pkgs.stdenv.system}-${self.ghc.name}/*/*.info $out/share/info/
    '';
  });
-


That change doesn't seem useful. :-)

ixmatus · 2018-02-11T20:58:59Z

Note that the cabal tests for hocker are failing because I didn't correctly include the tests data dir in hocker's cabal file. I have this fixed and am releasing to Hackage soon.

I discovered this by building the nixos test myself off of this branch which means I think the nixos test on this PR isn't included in the checks run by CI.

@grahamc is there anyway to mark small or light-weight nixos tests on a PR as required for merge?

ixmatus · 2018-02-12T17:18:13Z

The fixes for the tests have been uploaded to hackage so the next time the hackage packages are updated the nixos test in this PR should be working (I will double check it.)

…s into parnell/fetchdocker

ixmatus · 2018-02-13T23:34:53Z

I've run the included nixos test and verified it works.

adelbertc · 2018-02-28T22:43:20Z

Ping 👍 am interested in using this as well

fpletz · 2018-03-04T08:37:24Z

@GrahamcOfBorg test hocker-fetchdocker

GrahamcOfBorg · 2018-03-04T08:37:52Z

Failure on aarch64-linux (full log)

Partial log (click to expand)

Package ‘jailbreak-cabal-1.3.2’ in /var/lib/gc-of-borg/nix-test-rs-4/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-4/pkgs/development/haskell-modules/hackage-packages.nix:120699 is not supported on ‘aarch64-linux’, refusing to evaluate.

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowBroken = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-03-04T08:38:28Z

Failure on x86_64-linux (full log)

Partial log (click to expand)

while evaluating the attribute ‘drvPath’ at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/ogden/lib/customisation.nix:179:13:
while evaluating the attribute ‘drvPath’ at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/ogden/lib/customisation.nix:146:13:
while evaluating the attribute ‘buildCommand’ of the derivation ‘vm-test-run-test-hocker-fetchdocker’ at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/ogden/pkgs/stdenv/generic/make-derivation.nix:148:11:
while evaluating the attribute ‘buildCommand’ of the derivation ‘nixos-test-driver-test-hocker-fetchdocker’ at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/ogden/pkgs/stdenv/generic/make-derivation.nix:148:11:
while evaluating the attribute ‘buildCommand’ of the derivation ‘nixos-vm’ at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/ogden/pkgs/stdenv/generic/make-derivation.nix:148:11:
while evaluating the attribute ‘buildCommand’ of the derivation ‘nixos-system-machine-18.03.git.010a831’ at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/ogden/pkgs/stdenv/generic/make-derivation.nix:148:11:
while evaluating ‘optionalString’ at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/ogden/lib/strings.nix:138:26, called from /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/ogden/nixos/modules/system/activation/top-level.nix:40:9:
while evaluating the attribute ‘closure’ of the derivation ‘initrd’ at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/ogden/pkgs/stdenv/generic/make-derivation.nix:148:11:
while evaluating anonymous function at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/ogden/pkgs/build-support/closure-info.nix:9:1, called from /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/ogden/pkgs/build-support/kernel/make-initrd.nix:33:13:
assertion failed at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/ogden/pkgs/build-support/closure-info.nix:11:1

fpletz · 2018-03-04T17:08:52Z

Not sure why @GrahamcOfBorg is failing but this works for me. 👍

cc @grahamc

ixmatus · 2018-03-04T23:32:31Z

Thank you @fpletz!

grahamc · 2018-03-04T23:41:45Z

Is this possibly a Nix 2 vs. Nix 1 problem?

ixmatus · 2018-03-04T23:49:29Z

@grahamc I'm not sure, I will test it tomorrow morning using Nix 2 to be sure.

ixmatus added 2 commits December 1, 2017 21:00

ixmatus requested a review from peti as a code owner December 2, 2017 06:29

GrahamcOfBorg added 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Dec 2, 2017

peti reviewed Dec 2, 2017

View reviewed changes

ixmatus added 2 commits December 2, 2017 09:20

hocker: Don't check package and wrap the binaries with PATH to nix

6c9de3f

Merge branch 'master' into parnell/fetchdocker

a6cdf0a

peti reviewed Dec 4, 2017

View reviewed changes

Merge remote-tracking branch 'upstream/master' into parnell/fetchdocker

e4ec980

GrahamcOfBorg added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux and removed 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Jan 10, 2018

Merge branch 'master' into parnell/fetchdocker

bd9869d

Gabriella439 approved these changes Jan 30, 2018

View reviewed changes

Merge branch 'master' into parnell/fetchdocker

c2a5f68

fpletz self-requested a review January 30, 2018 20:10

fpletz added this to the 18.03 milestone Jan 30, 2018

peti reviewed Feb 2, 2018

View reviewed changes

peti approved these changes Feb 2, 2018

View reviewed changes

Merge branch 'master' into parnell/fetchdocker

6f95cb1

GrahamcOfBorg added 6.topic: fetch 6.topic: haskell 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: documentation This PR adds or changes documentation labels Feb 11, 2018

ixmatus added 2 commits February 13, 2018 17:28

Merge remote-tracking branch 'upstream/master' into parnell/fetchdocker

0a603ee

Merge branch 'parnell/fetchdocker' of github.com:awakesecurity/nixpkg…

58895aa

…s into parnell/fetchdocker

GrahamcOfBorg removed the 6.topic: haskell label Feb 13, 2018

fpletz approved these changes Mar 4, 2018

View reviewed changes

fpletz merged commit 0f78afd into NixOS:master Mar 4, 2018

ixmatus deleted the parnell/fetchdocker branch March 4, 2018 23:32

badmutex mentioned this pull request Mar 10, 2018

Issue 2018/04 NixOS/nixos-weekly#54

Merged

paulyoung mentioned this pull request Jun 4, 2019

Write a tutorial awakesecurity/hocker#9

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support fetching docker images from V2 registries #32248

Support fetching docker images from V2 registries #32248

ixmatus commented Dec 2, 2017 •

edited

Loading

ixmatus commented Dec 2, 2017

peti Dec 2, 2017

ixmatus Dec 2, 2017

ixmatus Dec 2, 2017

peti Dec 4, 2017

ixmatus Dec 4, 2017

peti Dec 5, 2017

ixmatus Dec 11, 2017

peti Dec 11, 2017

ixmatus Dec 11, 2017 •

edited

Loading

ixmatus commented Jan 10, 2018

ixmatus commented Jan 30, 2018

peti Feb 2, 2018

ixmatus commented Feb 11, 2018

ixmatus commented Feb 12, 2018

ixmatus commented Feb 13, 2018

adelbertc commented Feb 28, 2018

fpletz commented Mar 4, 2018

GrahamcOfBorg commented Mar 4, 2018

GrahamcOfBorg commented Mar 4, 2018

fpletz commented Mar 4, 2018

ixmatus commented Mar 4, 2018

grahamc commented Mar 4, 2018

ixmatus commented Mar 4, 2018

Support fetching docker images from V2 registries #32248

Support fetching docker images from V2 registries #32248

Conversation

ixmatus commented Dec 2, 2017 • edited Loading

Motivation for this change

Things TODO next, after merging

Things done

ixmatus commented Dec 2, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ixmatus Dec 11, 2017 • edited Loading

Choose a reason for hiding this comment

ixmatus commented Jan 10, 2018

ixmatus commented Jan 30, 2018

Choose a reason for hiding this comment

ixmatus commented Feb 11, 2018

ixmatus commented Feb 12, 2018

ixmatus commented Feb 13, 2018

adelbertc commented Feb 28, 2018

fpletz commented Mar 4, 2018

GrahamcOfBorg commented Mar 4, 2018

GrahamcOfBorg commented Mar 4, 2018

fpletz commented Mar 4, 2018

ixmatus commented Mar 4, 2018

grahamc commented Mar 4, 2018

ixmatus commented Mar 4, 2018

ixmatus commented Dec 2, 2017 •

edited

Loading

ixmatus Dec 11, 2017 •

edited

Loading