How to build all buildable targets and test all testable targets

[alexeagle]

I want to "build and test all the things" on my CI.

bazel test ... is bad because it doesn't build a target unless some test depends on it. (probably surprising to new users)

bazel build ... && bazel test ... is bad because the first test doesn't start running until the slowest build target is built

At Google we give up on this, and use build_test (alluded to at https://docs.bazel.build/versions/master/be/functions.html#load) which is a special test target that always passes, but consumes the files from some given deps, forcing those deps to build, and thus making the test either fail with a build error, or pass. But I think this is bad because it's easy to forget to create a build_test for some target, and we shouldn't expect everyone to be ever-watchful for that gotcha.

Is there some other way I don't know of?

[iirina]

There's no way to do this (that I'm aware of), since building all buildable targets and testing all testable targets involve 2 different bazel commands with two different sets of targets. Unfortunately it's not how bazel was designed.

bazel build ... && bazel test ... is bad because the first test doesn't start running until the slowest build target is built

On the other hand, when bazel test starts, everything needed would have already been built.

CC: @aehlig @buchgr does BEP help in this case?

[buchgr]

@iirina wouldn't know how.

@alexeagle why don't you do bazel test ... && bazel build ...? The test invocation builds any build targets required by test targets, and the build invocation builds any remaining build targets (as the other ones are in cache).

[laszlocsomor]

bazel test $(bazel query //...) ?

[buchgr]

bazel test $(bazel query //...) ?

Doesn't this fail for non-test targets?

Nevermind. Nice!

[laszlocsomor]

No, you can "bazel test" a non-test target. Bazel will warn you about it, but will build it anyway.
The caveat with my proposal though is that it'll likely exhaust the command length limit.

[alexeagle]

@buchgr bazel test ... && bazel build ... might help in common cases, but you might still have a long-running non-test target like building your production server docker image. I could imagine a critical path becoming longer when it waits for all tests to finish first.

@laszlocsomor that's basically what I wish Bazel could do, without exhausting the command-line limit which seems like an easy pit to fall into.

Consider this a feature request?

[laszlocsomor]

@alexeagle : I'm afraid we don't have free cycles to work on this feature request right now.

[buchgr]

@laszlocsomor that's basically what I wish Bazel could do, without exhausting the command-line limit which seems like an easy pit to fall into.

bazel query //... | xargs bazel test should do the trick.

[alexeagle]

@buchgr not portable, but my main concern for this is CI where I control the environment. I can do this just for Linux/Mac builds to start with, and let Windows testing be slower.
Thanks!

[alexeagle]

It's a bit more tricky. We use tags=["manual"] to exclude tests from the test suite since it's not matched by //....
To restore that behavior, we also need --test_tag_filters=-manual on the bazel test command

[drigz]

bazel test ... is bad because it doesn't build a target unless some test depends on it. (probably surprising to new users)

Can you clarify this? If I have a basic workspace with a cc_binary and a cc_test, bazel test //... builds both and then runs the test. If I pass --build_tests_only then it behaves as you described.

[alexeagle]

If you didn't have a cc_test, it would build nothing. For example, you might build a binary that you distribute, but don't have an integration test for that binary. Then bazel test ... won't even try to build the binary, and you might break it.

…

On Thu, May 17, 2018 at 7:55 AM Rodrigo Queiro ***@***.***> wrote: bazel test ... is bad because it doesn't build a target unless some test depends on it. (probably surprising to new users) Can you clarify this? If I have a basic workspace with a cc_binary and a cc_test, bazel test //... builds both and then runs the test. If I pass --build_tests_only then it behaves as you described. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4257 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAC5I9nqOWzdE_SVEGP4BVhfK8Kj3RpQks5tzY9VgaJpZM4Q6hz5> .

[drigz]

Even without the cc_test, the cc_binary gets built:

> cat BUILD.bazel
cc_binary(
    name = "main",
    srcs = ["main.cc"],
)

#cc_test(
#    name = "test",
#    srcs = ["test.cc"],
#)
> ls bazel-bin
ls: cannot access 'bazel-bin': No such file or directory
> bazel test //...
INFO: Analysed target //:main (6 packages loaded).
INFO: Found 1 target and 0 test targets...
Target //:main up-to-date:
  bazel-bin/main
INFO: Elapsed time: 1.043s, Critical Path: 0.24s
INFO: 3 processes: 2 linux-sandbox, 1 local.
INFO: Build completed successfully, 6 total actions
ERROR: No test targets were found, yet testing was requested
> ls bazel-bin
main  main-2.params  main.runfiles  main.runfiles_manifest  _objs

[alexeagle]

did you clean the output directory first?

…

On Thu, May 17, 2018 at 8:29 AM Rodrigo Queiro ***@***.***> wrote: Even without the cc_test, the cc_binary gets built: > cat BUILD.bazel cc_binary( name = "main", srcs = ["main.cc"], ) #cc_test( # name = "test", # srcs = ["test.cc"], #) > ls bazel-bin ls: cannot access 'bazel-bin': No such file or directory > bazel test //... INFO: Analysed target //:main (6 packages loaded). INFO: Found 1 target and 0 test targets... Target //:main up-to-date: bazel-bin/main INFO: Elapsed time: 1.043s, Critical Path: 0.24s INFO: 3 processes: 2 linux-sandbox, 1 local. INFO: Build completed successfully, 6 total actions ERROR: No test targets were found, yet testing was requested > ls bazel-bin main main-2.params main.runfiles main.runfiles_manifest _objs — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4257 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAC5I8g1ewZffLPhHk_kvF3QlLWK7ABbks5tzZdCgaJpZM4Q6hz5> .

[drigz]

Yep. The bazel output also shows that it has built the binary (6 total actions) even through no test targets were found.

[oferb]

@alexeagle hey :) does this mean we can simplify the instructions at
https://github.com/alexeagle/angular-bazel-example/wiki/Continuous-Integration#build-and-test-all-the-things
to just the bazel test --config=ci //... part?

[alexeagle]

Wow, I'm surprised to learn this. We have an elaborate build_test rule internally that creates a trivial test which depends on building a target, just to work around this. I suppose it was fixed by accident in some Bazel change - @laszlocsomor any idea what changed?

[laszlocsomor]

@alexeagle : sorry, to learn what? I don't know if anything changed in Bazel about how it handles "...".

[alexeagle]

I think target expansion is not what changed - it's that you can use bazel test with some pattern that includes binary targets, and these will be re-built. Meaning the workarounds in this issue are no longer needed, and bazel test //... is exhaustive to build the buildable things and test the testable things in your project.

Confirmed this is the case under Blaze as well.

[jxramos]

Having something close to bazel test --query "kind('py_test', '//...')" would be really ideal since the management would be a lot more tidy. In my case I'm in the process of upgrading python pytest's for our repo and would like to run all the bazel py_test rules.

[ishaangandhi]

bazel test //... is exhaustive to build the buildable things and test the testable things in your project.

Is this still true as of Bazel 4.2.1? It is not building all binaries for me.

[alexeagle]

@ishaangandhi maybe you have the --build_tests_only flag?

[ishaangandhi]

Thanks! Adding --nobuild_tests_only made it work.