Updating skaffold test command to accept a list of prebuilt images as input

[PriyaModali]

Fixes: #5599

Description
Updating skaffold test command to accept a list of prebuilt images as input for testing

Updated skaffold test cmd to

• accept skaffold build output as input
• accept a list of prebuilt images as input

Use facing changes & usage:

skaffold test --help

Run tests against your built application images

Examples:
  # Build the artifacts and collect the tags into a file
  skaffold build --file-output=tags.json

  # Run test against images previously built by Skaffold into a 'tags.json' file
  skaffold test --build-artifacts=tags.json

  # Build the artifacts and then deploy them
  skaffold build -q | skaffold test --build-artifacts -

  # Test the pre-build image
  skaffold test --image='<image-name>'

Options:
  -a, --build-artifacts=: File containing build result from a previous 'skaffold build --file-output'
  -f, --filename='skaffold.yaml': Path or URL to the Skaffold config file
  -i, --image=: A pre-built image to test
  -m, --module=[]: Filter Skaffold configs to only the provided named modules
      --remote-cache-dir='': Specify the location of the git repositories cache (default $HOME/.skaffold/repos)

Usage:
  skaffold test [flags] [options]

Use "skaffold options" for a list of global command-line options (applies to all commands).

Testing notes

Here are some sample outputs:

skaffold test

You need to:
run [skaffold test] with [--build-artifacts <file-output>] for running tests on artifacts from a given file.
or build the artifacts and then test them with `build -q | skaffold test --build-artifacts -`.
or run [skaffold test] with [--image TAG] for each pre-built artifact.

skaffold test --image='custom-test-example'

Starting test...
Testing images...
Running custom test command: "./test.sh" with timeout 60 s
go custom test 
ok      github.com/GoogleContainerTools/skaffold/integration/examples/custom-tests      (cached)
Command finished successfully.
Running custom test command: "echo Hello world!!"
Hello world!!
Command finished successfully.

`skaffold test -i custom-test-example'

Starting test...
Testing images...
Running custom test command: "./test.sh" with timeout 60 s
go custom test 
ok      github.com/GoogleContainerTools/skaffold/integration/examples/custom-tests      (cached)
Command finished successfully.
Running custom test command: "echo Hello world!!"
Hello world!!
Command finished successfully.

[codecov]

Codecov Report

Merging #5600 (559825b) into master (3a6c3e0) will increase coverage by 0.01%.
The diff coverage is 71.42%.

@@            Coverage Diff             @@
##           master    #5600      +/-   ##
==========================================
+ Coverage   71.07%   71.09%   +0.01%     
==========================================
  Files         402      402              
  Lines       15182    15190       +8     
==========================================
+ Hits        10791    10799       +8     
- Misses       3597     3598       +1     
+ Partials      794      793       -1     

Impacted Files	Coverage Δ
cmd/skaffold/app/tips/tips.go	46.15% <0.00%> (-8.40%)	⬇️
cmd/skaffold/app/cmd/test.go	60.00% <100.00%> (+13.33%)	⬆️
pkg/skaffold/runner/changeset.go	76.47% <0.00%> (-1.31%)	⬇️
pkg/skaffold/runner/dev.go	73.78% <0.00%> (+1.55%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3a6c3e0...559825b. Read the comment docs.

[tejal29]

I am curious on the motivation is behind this. Do you have a usecase in mind?

[gsquared94]

I am curious on the motivation is behind this. Do you have a usecase in mind?

The idea is similar to deploy. Right now it's not possible to run skaffold test without skaffold build. If test is to be its own phase it should support a standalone input of images not necessarily built with skaffold, or without the output of skaffold build being available.

However images flag needs to accept a slice of images and not just a single image.

[nkubala]

closing this while we rethink the approach.