Automatically apply buildozer commands from warnings

[ittaiz]

Hi,
I've heard that the internal version of Bazel-watcher (iBazel) knows to watch for log warnings in Bazel outputs which are buildozer commands and to automatically apply them.
The need for this (for me) is to be able to automatically apply strict java/scala deps warnings.
This means a developer can use code from transitive targets, the rule will warn against it with a buildozer command (saying which target needs to be added explicitly where) and ibazel will automatically apply it.
Is there an interest to extend ibazel to that area?

[achew22]

Huh, this is a very interesting idea. I don't think I've yet run into that in iBazel but it doesn't mean that it doesn't do it. Do you have any ideas on how this would work? Are there any rules_ that are emitting buildozer commands to execute in case of failure?

In the specific case of Gazelle it might be reasonable to just blaze run //:gazelle whenever there is an import error but I'm not sure how to actually detect that. WDYT?

[ittaiz]

rules_scala does this
What is Gazelle?
BTW, @cgrushko is the one who told me about this feature in internal ibazel so maybe he has more info

[cgrushko]

IIUC it's nothing more than ibazel looking at the output of Bazel and running some commands when it sees something it recognizes (i.e., a regex --> command map).

[mprobst]

The internal tool parses error messages from the compilation job for well known patterns and then runs the suggested fixer tool (if there is any). For Go code indeed, you'd probably just run gazelle, but for Java code you often need very specific fixes (add a dependency to this target or that).

[ittaiz]

Given this PR <bazelbuild/bazel#3677> which adds printing the buildozer command on strict_java_deps warn/error. Can ibazel parse the message and just run the buildozer command?

…

On Mon, Jul 31, 2017 at 9:24 AM Martin Probst ***@***.***> wrote: The internal tool parses error messages from the compilation job for well known patterns and then runs the suggested fixer tool (if there is any). For Go code indeed, you'd probably just run gazelle, but for Java code you often need very specific fixes (add a dependency to this target or that). — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#18 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUIF039p-uEDPep07g56944IAjCg0h2ks5sTXMGgaJpZM4OmGod> .

[achew22]

That seems like an extremely reasonable thing to do. If you wanted to implement it with a whitelist of regexps (to prevent running something scary) scanning the output from Bazel I think you could implement that without a huge amount of effort.

Why don't you take a whack at it? If you run into problems, feel free to comment on here.

[achew22]

Useful notes to future implementers:

Go will give you access to the commands stdout pipe with cmdReader, err := cmd.StdoutPipe(). Rather than making the command's pipe be stdout you can intercept it line by line and print it and search for matching patterns.

A more complete example can be found at https://nathanleclaire.com/blog/2014/12/29/shelled-out-commands-in-golang/

[jjudd]

I have a rough and rudimentary version of this working. I'm working on this in conjunction with unused_deps for rules_scala. I'm going to keep working on it and polish it up as we get unused_deps working. When I'm confident it works well I'll open a PR for the issue here.

[jjudd]

Quick update on this: we have this working internally and are currently rolling ibazel + this change out to all of engineering. We plan to create a PR soon (1-2 weeks). If this is blocking anyone please let me know and I can try to get the PR ready faster than that.

[ittaiz]

Thanks! How configurable is this? I ask because currently there are two issues with buildozer: 1. Some emitted buildozer commands are illegal and break bazel (for example maven_jar cases) 2. There is discussion of a new tool to replace buildozer which is more semantic

…

On Fri, 27 Apr 2018 at 3:05 James Judd ***@***.***> wrote: Quick update on this: we have this working internally and are currently rolling ibazel + this change out to all of engineering. We plan to create a PR soon (1-2 weeks). If this is blocking anyone please let me know and I can try to get the PR ready faster than that. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#18 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUIFzwz7hRwlt_QO_9dhg-_qI6ZudTQks5tsmDPgaJpZM4OmGod> .

[jjudd]

At the moment, it is hardcoded to buildozer. The goal before merging this is to have there be a config file of some sort that provides regexes with capturing groups indicating the command and arguments.

If we wanted to extend it to do more than that, we could add a blacklist regex.

This should cover 2. As for 1, the blacklist should probably cover that. I'm not sure how it works with maven_jar. We have a fork of bazel-deps, which uses java_import_external and scala_import_external, so we can use strict/unused deps with maven dependencies. We're going to open a PR for that soon as well. It's made dealing with third party dependencies quite nice.

[ittaiz]

scala_import_external will cause the same issues. Essentially if you don’t have ijar currently it’s broken. In addition there are also other use cases where it’s broken

…

On Fri, 27 Apr 2018 at 8:59 James Judd ***@***.***> wrote: At the moment, it is hardcoded to buildozer. The goal before merging this is to have there be a config file of some sort that provides regexes with capturing groups indicating the command and arguments. If we wanted to extend it to do more than that, we could add a blacklist regex. This should cover #2 <#2>. As for #1 <#1>, the blacklist should probably cover that. I'm not sure how it works with maven_jar. We have a fork of bazel-deps, which uses java_import_external and scala_import_external, so we can use strict/unused deps with maven dependencies. We're going to open a PR for that soon as well. It's made dealing with third party dependencies quite nice. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#18 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUIF4xgJPp445YHXS3dmeDeo3vdaYAaks5tsrPQgaJpZM4OmGod> .

[jjudd]

True. We're on a branch which has a jars_to_labels provider, so things are working for scala_import_external internally with buildozer suggestions. We'll work on getting PRs for all these open over the next couple weeks.

Sorry to drag this thread off topic 😄