Ability to run Benchmarks from test file

[scriptnull]

First of all, vscode-go is awesome. It is best workflow experience for writing go code.

One of the features, I love the most is ability to run a test with a single click on the "run test" link that appears above the test function

I would like to see the support for running benchmarks just by clicking a link above the benchmark function

Currently, the benchmark functions does not have anything above them. I would like to see the support for "run benchmark" link above the benchmark functions.

[ramya-rao-a]

Thanks for your kind words @scriptnull :)

Currently only the Test* and Example* functions are being identified as "test functions" that can be run.

See https://github.com/Microsoft/vscode-go/blob/0.6.61/src/goTest.ts#L267

Looping in @ironcladlou who wrote most of the initial test related code in the Go extension.

@ironcladlou Looking at history, it seems initially we only identified "Test*" functions. And then somewhere along the line, we added "Example*". Do you know of any reason why "Benchmark*" was excluded?

[ironcladlou]

@ramya-rao-a

Looking at history, it seems initially we only identified "Test*" functions. And then somewhere along the line, we added "Example*". Do you know of any reason why "Benchmark*" was excluded?

At the time, I just didn't think about them because I didn't use benchmarks. So, no special reason. 😁 I absolutely think the plugin should support them!

Thinking about them now:

1. Benchmark functions do seem distinct from tests in practice and I'd expect benchmarks to not get executed automatically when running "test". I'd be interested to discuss this if there's any disagreement.
2. Although I don't have a ton of experience with Go's benchmarking support, I suspect the benchmarks often rely on user input (e.g the -benchtime, and -count flags). I'd be interested to hear anecdotes on this point.
3. I wonder how fancy one could get with the new progress UI support...

If point 2 turns out to be true, is there any good way to collect user input for the benchmark executions within the VScode UI? I clearly haven't had much time to keep up with plugin or VScode development lately, so I'm not sure what capabilities exist.

Thanks for the ping, @ramya-rao-a! Sorry I haven't been able to contribute more than words lately!

[ramya-rao-a]

@ironcladlou Words are as important as code while discussing design and requirements, I am just glad you could respond every time I ping you :)

I'd expect benchmarks to not get executed automatically when running "test".

I agree. Our existing commands to run test in file/pkg/workspace and the test on save feature should not include the benchmark. Benchmarks are about perf and not correctness.

But the codelens for running the test and the command to run test at cursor can be updated to include the benchmark functions.

This way, benchmark functions will be run only if user specifically wants to run them and not in conjunction with the rest of the tests.

benchmarks often rely on user input

If input is indeed needed we can have them either as a setting that gets applied to all benchmarks. Or we can add a prompt to ask user for input everytime any benchmark function needs to be run.

I don't have much experience here either.
cc @mattetti @campoy @rakyll for their input

[buyology]

+1

As a MVP couldn't just all the test buttons be mirrored with corresponding bench buttons (on the package, file and individual bench level)?

[campoy]

Hi all,

First of all I'd like to say that benchmarks rarely rely on user input, so it would be totally ok in my opinion if those flags were configurable only at settings level.
Benchmarks should not be executed when we test a package unless otherwise indicated with the -bench flag.

I am interested on what the benchmarks would look like. Would it be simply something like the "run this test | debug this test" links we currently have? If so it'd be a nice addition

[rakyll]

I'd love to be able to run individual benchmarks the way I run tests individually. It is currently one of the few critical things missing from my daily workflow.

[xmattstrongx]

+1 for this feature. Would love to be able to run individual benchmarks straight from vscode

[ramya-rao-a]

This will be out in the next update

[ramya-rao-a]

This feature is now out in the latest update to the Go extension (0.6.69)
You can now use the codelens above a benchmark function to run it.
Thanks @buyology!