cmd/go: add go test -skip to skip specific tests #41583

dprotaso · 2020-09-23T15:32:07Z

t.Skip() is an option but it requires changing the test code.

It's hard to write a correct regex to pass to the -run flag in order to skip specific tests. This is due to the lack of lookaheads in the regexp library. Previous discussion here: https://groups.google.com/g/golang-nuts/c/7qgSDWPIh_E?pli=1

The text was updated successfully, but these errors were encountered:

ianlancetaylor · 2020-09-24T00:13:22Z

Do you have a specific suggestion?

To run specific tests you can already write go test -run="TestOne|TestTwo|TestThree".

dprotaso · 2020-09-24T00:56:15Z

Do you have a specific suggestion?

I didn't want to prescribe a solution but here's some off the top of my head

expand run flag functionality - -run [regex] to allow look aheads - thus negation
- pros
  - most people are familiar with this flag
- cons
  - requires expanded regex implementation
new flag -skip [regex]
- pros
  - mirrors run flag
  - may be easier to use than a negative regex
- cons
  - requires definition of precedence between -skip& -run
  - it'd be nice to see skipped test output - that might require more work to implement

To run specific tests you can already write go test -run="TestOne|TestTwo|TestThree".

This doesn't scale if I have O(10s) of tests in a package and want to skip one or two

dprotaso · 2020-10-06T21:30:47Z

hey @ianlancetaylor - do you have any additional questions or want me to clarify anything above?

ianlancetaylor · 2020-10-06T22:09:37Z

No, I'm good.

This issue is in the proposal process now and will get into the proposal committee in due course. The committee looks at a lot of issues (see minutes at #33502), so it won't be immediately, but it shouldn't take too long.

Thanks.

rsc · 2020-10-07T17:57:50Z

Almost no one on earth understands how to write negation regexps correctly, so I would lean toward -skip if we do this.

bcmills · 2020-10-07T18:09:13Z

requires definition of precedence between -skip & -run

I can see only two reasonable definitions.

Define that the default value of -run matches everything, and therefore -skip takes precedence over -run.
Define that the default value of -run imposes no constraints, and therefore it is an error to specify values of -run and -skip that both match the same test.

I have a slight preference for (1), because it treats the default value of the -run flag the same as other values.

jimmyfrasche · 2020-10-07T18:51:17Z

Does precedence mean the order the filters are applied in or which is used if both are given?

If it's the order, then -run followed by -skip makes the most sense to me and seems more useful than disallowing the combination.

bcmills · 2020-10-07T19:08:28Z

Yeah, it seems especially useful in the context of subtests:

go test foo -run=TestBar -skip=TestBar/flaky_subtest

dprotaso · 2020-10-07T19:17:20Z

Does precedence mean the order the filters are applied in or which is used if both are given?

The latter in my opinion

ChrisHines · 2020-10-08T17:54:25Z

I have almost moved some subtests into their own top level test function on a few occasions exactly because there was not a good way to skip some subset of them. I have also had to run go test -list iterations just to figure out a regex that would run all tests except a select few, and the resulting regex is often overly long given the use case of wanting to skip a small subset when a package has a lot of tests.

rsc · 2020-10-14T17:15:49Z

It sounds like people are generally in favor of adding -skip with a default of "matches nothing at all".
The -skip setting applies after the -run setting (which already defaults to "match everything")
and only applies to tests (not benchmarks).

Do I have that right? Does anyone object to adding -skip as described?

bcmills · 2020-10-14T17:33:09Z

The -skip setting … only applies to tests (not benchmarks).

Hmm. It would be unfortunate to have an easy way to skip specific tests but not specific benchmarks.

I think it would make sense to either have -skip apply to both tests and benchmarks, or to add a separate -benchskip or similar flag for those. (It seems a bit simpler to me to make -skip also apply to benchmarks, but I don't have a strong preference either way.)

dprotaso · 2020-10-14T18:17:44Z

Do I have that right? Does anyone object to adding -skip as described?

Sounds good to me

Hmm. It would be unfortunate to have an easy way to skip specific tests but not specific benchmarks.

Expanding the scope of -skip to cover benchmarks makes sense to me. I don't mind if this comes later if it's not a trivial thing to do.

ChrisHines · 2020-10-14T19:26:23Z

A way to skip benchmarks would be welcome as well.

The -run and -bench flags have different defaults. In my experience go test -run=^$ -bench=. is pretty common today, but would people start trying to use go test -skip=. -bench=. to mean the same thing? It seems to me a separate -benchskip flag would avoid cross pollinating the two sets and reduce the chances for surprise.

magical · 2020-10-18T06:25:23Z

Instead of introducing a new flag, could we add some syntax to the existing flag? Let a pattern starting with ! invert the sense of the match. So go test -run=!Foo|Bar would run all tests that don't match Foo or Bar, same as the proposed -skip flag.

This avoids introducing new flags (-skip and -benchskip) and avoids issues about whether -run or -skip takes precedence. -run already has special handling for / so it is already not a straight regexp.

dprotaso · 2020-10-19T13:24:51Z

-run already has special handling for / so it is already not a straight regexp.

Truthfully I was tripped up by this when I first encountered it

bcmills · 2020-10-19T13:43:36Z

@magical, further complications in the syntax for the -skip flag would be pretty awkward for the “run a specific test but skip a specific subtest” use-case mentioned previously.

bcmills · 2020-10-19T13:45:22Z

@ChrisHines, I think the question is, would someone running go test -bench=BenchmarkFoo -skip=BenchmarkFoo/some_subbench expect it to skip BenchmarkFoo/some_subbench?

I would argue that pretty much everyone would expect that to skip the sub-benchmark, so I suspect that a unified -skip flag would be less confusing than a separate -benchskip flag.

rsc · 2020-10-20T03:27:58Z

The -run and -bench flags have different defaults. In my experience go test -run=^$ -bench=. is pretty common today

FWIW this is usually a bad idea, at least interactively, since you inevitably end up benchmarking broken code. (And boy can you make broken code run fast!)

wojciechka · 2022-03-03T12:56:27Z

Would love to be able to do this to disable flaky tests in CI.

mvdan · 2022-03-03T20:58:28Z

I still intend to get to this for 1.19. There's no more need for more comments to just say it would be nice to have :) See https://github.com/golang/go/wiki/NoPlusOne.

linzhp · 2022-04-02T18:55:24Z

Can you make it support skipping multiple tests from different packages? Like:

go test -skip "pkg1.TestFoo|pkg2.TestBar" ./...

It would be useful to disable flaky tests in different packages on CI

Currently, there's no good way for any other project consuming these tests to selectively skip any of them. Maybe there will be [as of go 1.19](golang/go#41583), but until then, this introduces a new "skipTests" argument that allows users to pass in test names to skip.

mvdan · 2022-05-08T21:35:26Z

@linzhp that sounds like it should be a separate issue or proposal, because it should also affect the -run flag, and this issue does not cover changing the -run flag in any way. The two flags will take the same form of pattern in this proposal, and I don't see a reason why we should make them inconsistent.

linzhp · 2022-08-05T19:26:44Z

Any plan to implement this? It seems not in Go 1.19 yet

mvdan · 2022-08-05T20:36:29Z

I clearly did not get to this for 1.19, so I'm unassigning myself for now. There is no plan or estimate right now; the issue is in the backlog milestone.

gopherbot · 2022-08-05T20:50:58Z

Change https://go.dev/cl/421439 mentions this issue: cmd/go, testing: add go test -skip flag

mvdan · 2022-08-05T21:49:52Z

Not entirely sure whether that was a coincidence :)

06kellyjac · 2022-08-05T21:57:53Z

Woah, next release can I finally stop sed-ing individual tests to skip? 🎉🎊

gopherbot · 2022-11-09T15:54:13Z

Change https://go.dev/cl/449075 mentions this issue: doc/go1.20: add release notes for cmd/go changes

Updates #41696. Updates #50332. Updates #41583. Change-Id: I99e96a2996f14da262570a5cb5273dcdce45df2b Reviewed-on: https://go-review.googlesource.com/c/go/+/449075 Reviewed-by: Russ Cox <[email protected]> Run-TryBot: Bryan Mills <[email protected]> Reviewed-by: Michael Matloob <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Auto-Submit: Bryan Mills <[email protected]>

This implementation is based on a prior discussion with Ozz. I spent some time investigating other patterns for designing integration tests. See below for my notes on that. testing.Short approach: Another possible pattern is to use the "testing.Short" function to check if the user requested only short tests run. The issue with this is integration tests *and* unit tests will be run by default. Also, there is no way to only run integration tests using this pattern. This idea is discussed on stackoverflow. [1] "go test -run|-skip <regex> ..." approach: There is another pattern we could implement when Go 1.20 is released. Currently, it is possible to filter tests by their names using "-run <test-name-regex>". By naming integration tests "Test(...)Integration" (or some other naming pattern), we could filter tests by running: go test -run '.*Integration.*' -v ./... The issue with this approach is there is no way to explicitly skip tests - thus "go test" on its own will run both unit tests and integration tests. It is theoretically possible to invert regex - but it is difficult to read. [2] Rather than create unreadable regex, we could wait for Go 1.20. Luckily for us, Go 1.20 introduces the "-skip" argument for "go test". [3] This new argument would make this approach feasible. References: 1. https://stackoverflow.com/a/41407042 2. https://stackoverflow.com/a/406408 3. golang/go#41583

dprotaso changed the title ~~Make it easier to skip tests with go test CLI~~ Make it easier to skip tests with the go test CLI Sep 23, 2020

dprotaso mentioned this issue Sep 23, 2020

Support skip-tests option to skip some specific tests in alpha or beta features knative/networking#184

Merged

ianlancetaylor changed the title ~~Make it easier to skip tests with the go test CLI~~ cmd/go: make it easier to skip tests with the go test CLI Sep 24, 2020

ianlancetaylor added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Sep 24, 2020

ianlancetaylor changed the title ~~cmd/go: make it easier to skip tests with the go test CLI~~ proposal: cmd/go: make it easier to skip tests with the go test CLI Sep 24, 2020

gopherbot added this to the Proposal milestone Sep 24, 2020

gopherbot added the Proposal label Sep 24, 2020

rsc mentioned this issue Oct 8, 2020

proposal: review meeting minutes #33502

Open

rsc changed the title ~~proposal: cmd/go: make it easier to skip tests with the go test CLI~~ proposal: cmd/go: add go test -skip to skip specific tests Oct 14, 2020

izaaklauer mentioned this issue Apr 21, 2022

Allow some state tests to be skipped hashicorp/waypoint#3265

Merged

srosenberg mentioned this issue May 18, 2022

[wip] skip tests based on central registry cockroachdb/cockroach#81293

Closed

mvdan removed their assignment Aug 5, 2022

dgryski mentioned this issue Aug 10, 2022

go 1.20's "tinygo test -skip" unimplemented tinygo-org/tinygo#3056

Closed

rsc moved this to Accepted in Proposals Aug 10, 2022

rsc added this to Proposals Aug 10, 2022

weikanglim mentioned this issue Sep 7, 2022

Simplify running all cli tests Azure/azure-dev#583

Merged

gopherbot closed this as completed in e4a2c38 Sep 16, 2022

dmitshur modified the milestones: Backlog, Go1.20 Sep 16, 2022

dmitshur added the GoCommand cmd/go label Sep 16, 2022

dprotaso mentioned this issue Dec 5, 2022

Conformance: allow opting out of subtests kubernetes-sigs/gateway-api#1525

Closed

laurentsenta mentioned this issue Aug 18, 2023

Support -skip of multiple tests ipfs/gateway-conformance#141

Open

rsc removed this from Proposals Oct 3, 2023

golang locked and limited conversation to collaborators Nov 9, 2023

gopherbot added the FrozenDueToAge label Nov 9, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/go: add go test -skip to skip specific tests #41583

cmd/go: add go test -skip to skip specific tests #41583

dprotaso commented Sep 23, 2020

ianlancetaylor commented Sep 24, 2020

dprotaso commented Sep 24, 2020

dprotaso commented Oct 6, 2020

ianlancetaylor commented Oct 6, 2020

rsc commented Oct 7, 2020

bcmills commented Oct 7, 2020

jimmyfrasche commented Oct 7, 2020

bcmills commented Oct 7, 2020

dprotaso commented Oct 7, 2020

ChrisHines commented Oct 8, 2020

rsc commented Oct 14, 2020

bcmills commented Oct 14, 2020

dprotaso commented Oct 14, 2020

ChrisHines commented Oct 14, 2020 •

edited

Loading

magical commented Oct 18, 2020

dprotaso commented Oct 19, 2020 •

edited

Loading

bcmills commented Oct 19, 2020

bcmills commented Oct 19, 2020 •

edited

Loading

rsc commented Oct 20, 2020

wojciechka commented Mar 3, 2022

mvdan commented Mar 3, 2022

linzhp commented Apr 2, 2022

mvdan commented May 8, 2022

linzhp commented Aug 5, 2022

mvdan commented Aug 5, 2022

gopherbot commented Aug 5, 2022

mvdan commented Aug 5, 2022

06kellyjac commented Aug 5, 2022

gopherbot commented Nov 9, 2022

cmd/go: add go test -skip to skip specific tests #41583

cmd/go: add go test -skip to skip specific tests #41583

Comments

dprotaso commented Sep 23, 2020

ianlancetaylor commented Sep 24, 2020

dprotaso commented Sep 24, 2020

dprotaso commented Oct 6, 2020

ianlancetaylor commented Oct 6, 2020

rsc commented Oct 7, 2020

bcmills commented Oct 7, 2020

jimmyfrasche commented Oct 7, 2020

bcmills commented Oct 7, 2020

dprotaso commented Oct 7, 2020

ChrisHines commented Oct 8, 2020

rsc commented Oct 14, 2020

bcmills commented Oct 14, 2020

dprotaso commented Oct 14, 2020

ChrisHines commented Oct 14, 2020 • edited Loading

magical commented Oct 18, 2020

dprotaso commented Oct 19, 2020 • edited Loading

bcmills commented Oct 19, 2020

bcmills commented Oct 19, 2020 • edited Loading

rsc commented Oct 20, 2020

wojciechka commented Mar 3, 2022

mvdan commented Mar 3, 2022

linzhp commented Apr 2, 2022

mvdan commented May 8, 2022

linzhp commented Aug 5, 2022

mvdan commented Aug 5, 2022

gopherbot commented Aug 5, 2022

mvdan commented Aug 5, 2022

06kellyjac commented Aug 5, 2022

gopherbot commented Nov 9, 2022

ChrisHines commented Oct 14, 2020 •

edited

Loading

dprotaso commented Oct 19, 2020 •

edited

Loading

bcmills commented Oct 19, 2020 •

edited

Loading