More integration tests #208

corneliusweig · 2019-06-06T23:03:25Z

These are based on #203 and should be merged after that one. Reviewing before #203 is merged makes also no sense :-/

The tests currently cover the happy path of all subcommands excluding krew upgrade. Upgrade is a special case because it needs two versions of a valid plugin. That will be difficult to set up and may be better in a separate PR.

codecov-io · 2019-06-06T23:22:31Z

Codecov Report

Merging #208 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #208   +/-   ##
=======================================
  Coverage   53.77%   53.77%           
=======================================
  Files          13       13           
  Lines         716      716           
=======================================
  Hits          385      385           
  Misses        279      279           
  Partials       52       52

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 04288cb...287a50b. Read the comment docs.

docs/CONTRIBUTOR_GUIDE.md

hack/run-integration-tests.sh

Missing subcommand: upgrade

0 plugins -> empty string (instead of 2 lines) 1 plugin -> 1 line (instead of 2 lines) 2 plugins -> 2 lines (instead of 3 lines)

…rd build

cmd/krew/cmd/list.go

ahmetb · 2019-06-10T22:28:09Z

test/krew_test.go

+	if string(in) == "" {
+		return nil
+	}
+	return strings.Split(string(in), "\n")


ideally you should do strings.TrimRight(string(in)) so that you avoid taking trailing \n which almost all commands will have.

with this, you don't need the =="" check,

Unfortunately, the =="" is still required. For krew list I need the list of results to grow by one if a plugin is installed. However, when starting from none (resulting in "\n") to one plugin ("whatever\n") would result in 1 entry each.

hmm. I'm not sure if that's accurate.

Since in this case the stdout is not a tty, we don't print the tabular output.

I expect no output or at most \n. Both of which would return []string{} (len=0).

When you install a plugin I expect this to return pluginname\n, if you split it by \n it should return []string{"pluginnname"} (len=1).

Alternatively you can also test that case as output.Contains("pluginname"), which is also just fine.

I expect no output or at most \n. Both of which would return []string{} (len=0).

That is unfortunately not the case. If the input string is empty or "\n", it will result in []string{""}.

ahmetb · 2019-06-11T03:52:45Z

test/krew_test.go

+		"ExecutedVersion",
+		"GitTag",
+		"GitCommit",
+		"IndexURI        https://github.com/kubernetes-sigs/krew-index.git",


I feel like we'll easily break these spaces. :)
I'm not sure if this whole thing is necessary. I currently only care about exitcode=0. :)

I think it's worth a lot to know if something is present in the output. Do you like a regex-based assertion better?

I think anything is fine.

test/krew_test.go

- clarify logic for `krew list` test - check error case for `krew info` - make `krew version` test more resilient against formatting changes

ahmetb · 2019-06-11T17:33:08Z

test/krew_test.go

+		plugin    string
+		shouldErr bool
+	}{
+		{


please don't put negative cases to the same test case.
they are different test cases. and these aren't unit tests. tabular tests like []struct{}{} are good for tabular unit tests. but ideally each integration test case should be a separate method testing a particular scenario workflow.

e.g. try

TestKrewInfo
TestKrewInfoInvalidPlugin

Makes sense. The test cases are not split into separate test functions.

ahmetb · 2019-06-11T17:33:30Z

I recommend removing the WIP label so we can move forward with merging.

corneliusweig · 2019-06-11T17:51:18Z

I recommend removing the WIP label so we can move forward with merging.

The WIP label should already have been gone for a while. At least I don't see it any longer.

ahmetb · 2019-06-11T17:55:16Z

/lgtm
/approve

k8s-ci-robot · 2019-06-11T17:55:24Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahmetb, corneliusweig

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [ahmetb]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Adding corneliusweig to owners. He has been consistently helping both with krew and krew-index repositories in terms of: - developing plugins himself - taking a stab at krew machinery with large scale code refactors - adding integration test suite to the project - adding more validation and test cases - increasing developer documentation Some of his notable work: - kubernetes-sigs#195 - kubernetes-sigs#183 - kubernetes-sigs#191 - kubernetes-sigs#201 - kubernetes-sigs#202 - kubernetes-sigs#203 - kubernetes-sigs#208 He is familiar with the codebase enough to officially review and approve code. Signed-off-by: Ahmet Alp Balkan <[email protected]>

Adding corneliusweig to owners. He has been consistently helping both with krew and krew-index repositories in terms of: - developing plugins himself - taking a stab at krew machinery with large scale code refactors - adding integration test suite to the project - adding more validation and test cases - increasing developer documentation Some of his notable work: - #195 - #183 - #191 - #201 - #202 - #203 - #208 He is familiar with the codebase enough to officially review and approve code. Signed-off-by: Ahmet Alp Balkan <[email protected]>

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 6, 2019

k8s-ci-robot requested review from ahmetb and juanvallejo June 6, 2019 23:03

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jun 6, 2019

corneliusweig force-pushed the more-integration-tests branch from 11d8700 to 31d3f2f Compare June 6, 2019 23:18

corneliusweig marked this pull request as ready for review June 6, 2019 23:19

ahmetb reviewed Jun 6, 2019

View reviewed changes

docs/CONTRIBUTOR_GUIDE.md Outdated Show resolved Hide resolved

ahmetb reviewed Jun 6, 2019

View reviewed changes

hack/run-integration-tests.sh Outdated Show resolved Hide resolved

corneliusweig added 3 commits June 10, 2019 22:56

Add further integration tests

5f9629d

Missing subcommand: upgrade

Add test for krew list

44fdfb4

Be consistent with the number of lines

356a9ee

0 plugins -> empty string (instead of 2 lines) 1 plugin -> 1 line (instead of 2 lines) 2 plugins -> 2 lines (instead of 3 lines)

corneliusweig force-pushed the more-integration-tests branch from 31d3f2f to 356a9ee Compare June 10, 2019 20:57

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jun 10, 2019

corneliusweig added 3 commits June 10, 2019 23:03

Determine the correct krew binary for integration tests

0c30671

Remove instructions where integration tests are run with a non-standa…

9b0d954

…rd build

Check for existence of the krew binary when running integration tests

e73978c

corneliusweig force-pushed the more-integration-tests branch from 8d1a336 to e73978c Compare June 10, 2019 21:17

corneliusweig changed the title ~~[WIP] More integration tests~~ More integration tests Jun 10, 2019

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 10, 2019

ahmetb reviewed Jun 10, 2019

View reviewed changes

cmd/krew/cmd/list.go Outdated Show resolved Hide resolved

ahmetb reviewed Jun 10, 2019

View reviewed changes

Trim krew output in integration tests

48c9855

ahmetb reviewed Jun 11, 2019

View reviewed changes

test/krew_test.go Outdated Show resolved Hide resolved

Test improvements

d9c3689

- clarify logic for `krew list` test - check error case for `krew info` - make `krew version` test more resilient against formatting changes

ahmetb reviewed Jun 11, 2019

View reviewed changes

Split good and bad test cases for krew info

287a50b

k8s-ci-robot assigned ahmetb Jun 11, 2019

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 11, 2019

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 11, 2019

k8s-ci-robot merged commit 95a9ad7 into kubernetes-sigs:master Jun 11, 2019

corneliusweig deleted the more-integration-tests branch June 11, 2019 22:17

ahmetb mentioned this pull request Jun 14, 2019

Make corneliusweig a maintainer #214

Merged

corneliusweig mentioned this pull request Jun 19, 2019

REQUEST: New membership for corneliusweig kubernetes/org#929

Closed

6 tasks

corneliusweig mentioned this pull request Jul 3, 2019

Integration test for cli behavior #198

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

More integration tests #208

More integration tests #208

corneliusweig commented Jun 6, 2019 •

edited

Loading

codecov-io commented Jun 6, 2019 •

edited

Loading

ahmetb Jun 10, 2019

ahmetb Jun 10, 2019

corneliusweig Jun 10, 2019

ahmetb Jun 11, 2019

corneliusweig Jun 11, 2019

ahmetb Jun 11, 2019

corneliusweig Jun 11, 2019

ahmetb Jun 11, 2019

ahmetb Jun 11, 2019

corneliusweig Jun 11, 2019

ahmetb commented Jun 11, 2019

corneliusweig commented Jun 11, 2019

ahmetb commented Jun 11, 2019

k8s-ci-robot commented Jun 11, 2019

More integration tests #208

More integration tests #208

Conversation

corneliusweig commented Jun 6, 2019 • edited Loading

codecov-io commented Jun 6, 2019 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahmetb commented Jun 11, 2019

corneliusweig commented Jun 11, 2019

ahmetb commented Jun 11, 2019

k8s-ci-robot commented Jun 11, 2019

corneliusweig commented Jun 6, 2019 •

edited

Loading

codecov-io commented Jun 6, 2019 •

edited

Loading