Integration test for cli behavior

[ahmetb]

We've been successfully steering clear from testing the behavior of the CLI, but I think the time is approaching.

Ahead of the major refactoring for allowing multiple indexes (#23) I would love to get some integration test suite going that tests the end-to-end experience of the CLI from a user perspective.

Historically I've used https://github.com/sstephenson/bats for this purpose of running test cases that are mainly command executions in an ordered way, and processing their output/exit code with functionality like teardown/cleanup.

bats is no longer maintained and I think the industry has moved on to doing this with directly writing programs/test suites in Go that exec a bunch of commands with neat helpers.

Having something like this that tests the main user workflow can ensure that we can further safely go into large-scale refactoring, so it would be a great long-term investment.

/kind proposal

[corneliusweig]

I also think the go test suite is the best way to go forward. We can write a fluent wrapper, so that this will also read very nicely.

One concern I have is that the index is an external dependency which is not well-controlled. That's always bad for tests. What do you think about making the (single) index configurable? Maybe behind a hidden CLI flag. That way, we can also use file-based git repos such as file:///tmp/krew/index. WDYT?

[corneliusweig]

On second thoughts.. I think this will work without a configurable repo URL, as long as the test setup initializes the index in KREW_ROOT appropriately.

[ahmetb]

While I agree that we should allow customization of index, and we will in (#23), testing with a remote index and a real plugin would actually help test the real code path executed by users.

[corneliusweig]

Fixed by #203 #208 #242
/close

[k8s-ci-robot]

@corneliusweig: Closing this issue.

In response to this:

Fixed by #203 #208 #242
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.