try to reproduce #245 in CI

[JensGutermuth]

This is an attempt to reproduce #245 in CI.

[priyawadhwa]

So CI did fail but it was because the default test timeout for go is 10 minutes.

You'll probably have to add a -timeout 15m flag here to see if the Dockerfile actually isn't building!

[dlorenc]

Maybe change it to 30m or something? I think they usually take right around 10m anyway, so adding another 10 is still causing a timeout at 15.

[priyawadhwa]

Looks like it passed!

[dlorenc]

Looks like it passed!

That's bad, right :( ?

[JensGutermuth]

yep, this was supposed to fail. I'll write a few further thoughts in #245, as this was just an experiment to reproduce the issue.

[priyawadhwa]

So this Dockerfile did fail, but unfortunately we didn't get the same error. The integration tests built the Dockerfile with kaniko and Docker, and then diffed the images to make sure they are the same. Looks like kaniko was able to build the image, but the two images were pretty different:

    --- FAIL: TestRun/test_Dockerfile_test_reproduce_issue_245 (1212.45s)
    	integration_test.go:172: diff = time="2018-08-06T18:25:25-07:00" level=error msg="Could not obtain size for /tmpfs/tmp/gcr.iokaniko-testdocker-dockerfile_test_reproduce_issue_245554552367/etc/systemd/system/timers.target.wants/apt-daily-upgrade.timer: stat /tmpfs/tmp/gcr.iokaniko-testdocker-dockerfile_test_reproduce_issue_245554552367/etc/systemd/system/timers.target.wants/apt-daily-upgrade.timer: no such file or directory"
    		time="2018-08-06T18:25:25-07:00" level=error msg="Could not obtain size for /tmpfs/tmp/gcr.iokaniko-testdocker-dockerfile_test_reproduce_issue_245554552367/etc/systemd/system/timers.target.wants/apt-daily.timer: stat /tmpfs/tmp/gcr.iokaniko-testdocker-dockerfile_test_reproduce_issue_245554552367/etc/systemd/system/timers.target.wants/apt-daily.timer: no such file or directory"
    		time="2018-08-06T18:25:25-07:00" level=error msg="Could not obtain size for /tmpfs/tmp/gcr.iokaniko-testdocker-dockerfile_test_reproduce_issue_245554552367/lib64/ld-linux-x86-64.so.2: stat /tmpfs/tmp/gcr.iokaniko-testdocker-dockerfile_test_reproduce_issue_245554552367/lib64/ld-linux-x86-64.so.2: no such file or directory"
    		time="2018-08-06T18:25:25-07:00" level=error msg="Error diffing contents of /tmpfs/tmp/gcr.iokaniko-testdocker-dockerfile_test_reproduce_issue_245554552367/var/spool/mail and /tmpfs/tmp/gcr.iokaniko-testkaniko-dockerfile_test_reproduce_issue_245172257808/var/spool/mail: read /tmpfs/tmp/gcr.iokaniko-testdocker-dockerfile_test_reproduce_issue_245554552367/var/spool/mail: is a directory\n"
    		[
    		  {
    		    "Image1": "gcr.io/kaniko-test/docker-dockerfile_test_reproduce_issue_245",
    		    "Image2": "gcr.io/kaniko-test/kaniko-dockerfile_test_reproduce_issue_245",
    		    "DiffType": "File",
    		    "Diff": {
    		      "Adds": [
    		        {
    		          "Name": "/bin/[",
    		          "Size": 1083720
    		        },
    		        {
    		          "Name": "/bin/[[",
    		          "Size": 1083720
    		        },
    		        {
    		          "Name": "/bin/acpid",
    		          "Size": 1083720
    		        },
    		        {
    		          "Name": "/bin/add-shell",
    		          "Size": 1083720
... (many more files)

[JensGutermuth]

Just did a rebase to current master and did the same thing as done in #257 for all the other tests. A different base image would be an obvious cause of very different results.

[priyawadhwa]

Cool so that actually fixed it --

    --- PASS: TestRun/test_Dockerfile_test_reproduce_issue_245 (1220.38s)
    	integration_test.go:172: diff = [
    		  {
    		    "Image1": "gcr.io/kaniko-test/docker-dockerfile_test_reproduce_issue_245",
    		    "Image2": "gcr.io/kaniko-test/kaniko-dockerfile_test_reproduce_issue_245",
    		    "DiffType": "File",
    		    "Diff": {
    		      "Adds": null,
    		      "Dels": null,
    		      "Mods": null
    		    }
    		  },
    		  {
    		    "Image1": "gcr.io/kaniko-test/docker-dockerfile_test_reproduce_issue_245",
    		    "Image2": "gcr.io/kaniko-test/kaniko-dockerfile_test_reproduce_issue_245",
    		    "DiffType": "Metadata",
    		    "Diff": {
    		      "Adds": [],
    		      "Dels": []
    		    }
    		  }
    		]

All of our filesystem diffing tests passed, but when CI tried to compare layers it timed out:

=== RUN   TestLayers
=== RUN   TestLayers/test_layer_Dockerfile_test_reproduce_issue_245
=== RUN   TestLayers/test_layer_Dockerfile_test_run
=== RUN   TestLayers/test_layer_Dockerfile_test_target
=== RUN   TestLayers/test_layer_Dockerfile_test_workdir
=== RUN   TestLayers/test_layer_Dockerfile_test_daemons
=== RUN   TestLayers/test_layer_Dockerfile_test_expose
=== RUN   TestLayers/test_layer_Dockerfile_test_label
=== RUN   TestLayers/test_layer_Dockerfile_test_multistage
=== RUN   TestLayers/test_layer_Dockerfile_test_registry
=== RUN   TestLayers/test_layer_Dockerfile_test_run_2
=== RUN   TestLayers/test_layer_Dockerfile_test_copy
=== RUN   TestLayers/test_layer_Dockerfile_test_extract_fs
=== RUN   TestLayers/test_layer_Dockerfile_test_metadata
=== RUN   TestLayers/test_layer_Dockerfile_test_copy_bucket
=== RUN   TestLayers/test_layer_Dockerfile_test_onbuild
=== RUN   TestLayers/test_layer_Dockerfile_test_scratch
=== RUN   TestLayers/test_layer_Dockerfile_test_mv_add
=== RUN   TestLayers/test_layer_Dockerfile_test_volume
=== RUN   TestLayers/test_layer_Dockerfile_test_add
=== RUN   TestLayers/test_layer_Dockerfile_test_copy_reproducible
=== RUN   TestLayers/test_layer_Dockerfile_test_env
panic: test timed out after 30m0s

[priyawadhwa]

Hey @Delphinator I'm going to close this PR since it seems like #388 fixed the issue. If you're still running into issues please let us know.