TestServiceOptions E2E test is failing with error "/go/bin/helloworld: no such file or directory"

[pratham-m]

Hi Team,
I am working on executing knative-client "kn" CLI E2E test suite.

The issue:
One of the test case TestServiceOptions is consistently failing, with below error message:

    service_options_test.go:121: assertion failed: 
        Actual output: Creating service 'svc5' in namespace 'kne2etests11':
          1.986s The Route is still working to reflect the latest desired specification.
          3.940s Configuration "svc5" is waiting for a Revision to become ready.
         94.791s Revision "svc5-plvxy-1" failed with message: container create failed: time="2020-07-09T14:25:55Z" level=error msg="container_linux.go:349: starting container process caused \"exec: \\\"/go/bin/helloworld\\\": stat /go/bin/helloworld: no such file or directory\""
        container_linux.go:349: starting container process caused "exec: \"/go/bin/helloworld\": stat /go/bin/helloworld: no such file or directory"

Command used to execute the tests:
$ test/local-e2e-tests.sh -timeout=90m -count=1
or
$ test/local-e2e-tests.sh -timeout=15m -count=1 -run ^TestServiceOptions$

Analysis:
The test is executed from ./test/e2e/service_options_test.go file.
(line 97 to 99)

	t.Log("create, update and validate service with cmd and arg options")
	serviceCreateWithOptions(t, it, r, "svc5", "--cmd", "/go/bin/helloworld")
	validateContainerField(t, it, r, "svc5", "command", "[/go/bin/helloworld]")

The serviceCreateWithOptions function uses KnDefaultTestImage as the test image.
This KnDefaultTestImage string is defined in the ./lib/test/integration.go file, which uses gcr.io/knative-samples/helloworld-go
I have replaced the variable to use an image built for my PPC64LE platform, from docker file available @ https://github.com/knative/docs/blob/release-0.13/docs/serving/samples/hello-world/helloworld-go/Dockerfile
The docker file uses /server as the command where as the test used /go/bin/helloworld as the command, causing the test to fail.
I am working on v0.13.2, on OCP 4.3 cluster. I also checked v0.16.0 which has same test code.

Understanding:
With reference to above data, looks like either the test is supposed to use a different test image having /go/bin/helloworld or the test needs modification in the test source code.

Request:
Can you kindly check above data and confirm if the understanding is correct? If not, can you help me understand what the issue is? and what can be done to resolve it?

[navidshaikh]

With reference to above data, looks like either the test is supposed to use a different test image having /go/bin/helloworld or the test needs modification in the test source code.

That's correct. The tests are written based on the image gcr.io/knative-samples/helloworld-go which is built with cmd /go/bin/helloworld. The test image seems to be stale, last updated on Nov 2018 and out of sync from the Dockerfile source linked.

Ideally, the test image should be built from the source present in client repo https://github.com/knative/client/tree/master/test/test_images/helloworld, pushed to a repo and referenced from there. Filed #942 to resolve this. Meanwhile, you can update this line in failing test to refer the cmd as /server to verify e2e.

[navidshaikh]

@RichieEscarez @evankanderson : Do you know if helloworld sample image is built and pushed from https://github.com/knative/docs/blob/master/docs/serving/samples/hello-world/helloworld-go/Dockerfile ?
gcr.io/knative-samples/helloworld-go seems to not receive any updates.

[pratham-m]

@navidshaikh thanks for the reply and for confirming the observations.
By replacing /go/bin/helloworld with /server the test to validate "cmd and arg options" is passing.

[evankanderson]

I'm not sure who owns gcr.io/knative-samples; I don't see it listed in any of the docs or test-infra code.

@chaodaiG may know more or be able to see who owns that repo.

[chaodaiG]

I'm not sure who owns gcr.io/knative-samples; I don't see it listed in any of the docs or test-infra code.

@chaodaiG may know more or be able to see who owns that repo.

I don't have any impression of any flow that automates imaging pushing in docs repo

[pratham-m]

I compared both the docker files:
https://github.com/knative/docs/blob/release-0.13/docs/serving/samples/hello-world/helloworld-go/Dockerfile
https://github.com/knative/client/blob/v0.13.2/test/test_images/helloworld/Dockerfile

The only difference is related to downloading of go mod dependencies in knative/docs docker file.
And as per my understanding, that doesn't have much significance on the functional behavior of helloworld-go image and should not impact the e2e tests. The helloworld.go is same.

[evankanderson]

I'm wondering how/where gcr.io/knative-samples got set up -- if it's something that our test infrastructure has access to, prebuilding the images seems reasonable/handy. If it's not, I worry about linking to an increasingly stale set of images.

[navidshaikh]

Yes, most of the examples at knative.docs and client repo are referring images from gcr.io/knative-samples. I think we should configure dockerfiles in docs repo as source for images at gcr.io/knative-samples?

[RichieEscarez]

@grayside might be able to help? He did a lot of samples work early on.

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[navidshaikh]

/close

We now build the test images part of CI run.

[knative-prow-robot]

@navidshaikh: Closing this issue.

In response to this:

/close

We now build the test images part of CI run.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.