Output Runner and Multiple Runs #133

borkaehw · 2018-07-31T15:24:35Z

Output Runner

Issue

When we build a target with Bazel, if we got redundant or missing deps, there is a prompt like

You can use the following buildozer command:
buildozer 'add deps @deps_name' //target:target

or

You can use the following buildozer command:
buildozer 'remove deps @deps_name' //target:target

The build fails and hangs.

Fix

We monitor the Bazel logs for these kinds of information. We generate an interactive prompt

Do you want to execute this command?
buildozer 'add deps @deps_name' //target:target
[y/N]

Users can reply y to fix the deps problem and rebuild.

The command matching is defined by regex and it is configurable by JSON. Users are able to decide what to monitor.

To enable output runner, use flag -run_output=config_file_name.json. The config file format is provided in ibazel/output_runner/output_runner_test.json.

Multiple Runs

Issue

iBazel currently only supports run for single target. Running multiple targets requires separate Bazel command executions and busy waiting for build processes.

Fix

Implement a mrun command to run multiple targets with iBazel

ibazel mrun target1 --arg=t1_arg1 --arg=t1_arg2 target2 --arg=t2_arg1 --arg=t2_arg2

Append arbitrary number of targets after mrun with the corresponding arguments, prefixed with --arg=.

borkaehw · 2018-07-31T15:27:06Z

Hey @achew22, sorry for the mess yesterday. I created another PR, hopefully this is clearer. I am happy to take any suggestion for improvement. Thanks in advance.

iBazel now watches the log warning from Bazel outputs. Users can watch for specific commands and apply them automatically. The command matching is implemented by regex and it is configurable. Fixes: bazelbuild#18

alexeagle · 2018-08-01T02:26:43Z

What's the tradeoff of just changing the behavior of run command? Is it that we can't distinguish a second target to run from an argument to the first program? I'm not sure that your scheme for disambiguating args from additional targets-to-run is sufficient either.
I don't have a better proposal after thinking for a couple of minutes - you could read multiple lines from stdin but I think there may be other reasons to pass things to stdin of ibazel? and might be too subtle from the command line, e.g. echo -n "//path/to:target1 --arg1 --arg2\n//path/to:target2 --arg3 --arg4" | ibazel run

Could you clarify who is expected to use -run_output=config_file_name.json - is that any time a user wants to run ibazel with the fixer enabled?

alexeagle

Wow, thanks for all this work!
We will want to use this feature for ts_auto_deps too (Gazelle for TypeScript/Angular)
FYI @mprobst

alexeagle · 2018-08-01T02:27:51Z

bazel/bazel.go

 	b.ctx, b.cancel = context.WithCancel(context.Background())

 	args = append([]string{command}, args...)
+
+	if b.writeToStderr || b.writeToStdout {
+		containsColor := false


how is this color change related to the rest of the PR? could it be separated out?

It isn't. I'm pretty sure this is a leftover remnant from an internal modification to make sure ibazel always used color when running bazel commands. It can be pulled out.

alexeagle · 2018-08-01T02:32:03Z

ibazel/ibazel.go

@@ -37,9 +42,11 @@ var osExit = os.Exit
 var bazelNew = bazel.New
 var commandDefaultCommand = command.DefaultCommand
 var commandNotifyCommand = command.NotifyCommand
+var mrunToFiles = flag.Bool("mrunToFiles", false, "Log mrun to file for simpler log reading")


the need for this feature makes me wonder, if you're going to open two or more log files to see the output, why not just run different ibazel commands in one or more terminals?

I think the answer is that it's hard to script around "start these three programs in three terminals" in any portable way...

I think we are providing this option for those who want to see the outputs, while others who don't need the outputs can use just one terminal for ibazel. The other reason is that if you're running ten targets but only need two log files, you can just have two terminals instead of ten.

alexeagle · 2018-08-01T02:32:53Z

ibazel/ibazel.go

@@ -229,6 +266,13 @@ func (i *IBazel) Run(target string, args []string) error {
 	return i.loop("run", i.run, []string{target})
 }

+// Run the specified target (singular) in the IBazel loop.
+func (i *IBazel) RunMulitple(args, target []string, debugArgs [][]string) error {


spelling: Multiple

alexeagle · 2018-08-01T02:34:14Z

ibazel/ibazel.go

+	}
+}
+
+func (i *IBazel) iterationMultiple(command string, commandToRun runnableCommands, targets []string, debugArgs [][]string, argsLength int) {


wow a ton of code is added to this file - is it possible to simplify?

It's possible. Since test, build, and run share a few functions, mrun has its own functions with different signatures but do similar things, we can try to merge those functions and make mrun a special case. It requires a lot of works and involves changing code for test and build. Should we do it?

jjudd · 2018-08-01T15:35:54Z

Pulling in @BryantN for the mrun vs run change. We were split when we made the change and he was the main proponent for mrun.

iBazel currently only supports "run" for single target. Running multiple targets requires separate Bazel command executions and busy waiting for build processes. Implement "mrun" command which takes in arbitrary number of targets, and run all of them at once. We can pass in corresponding arguments with prefix "--arg=".

borkaehw · 2018-08-01T17:57:17Z

@alexeagle Yes. -run_output=config_file_name.json is a file that users can define what bazel outputs they are watching for. The only use case for us so far is to watch for buildozer command and fix the deps problem, we might encounter other use case.

alexeagle · 2018-10-02T20:49:38Z

Hey @borkaehw - what happened with the multi_binary proposal to capture the mrun use case? Do you have any write-up, or should I start one?

Andrew gathered some notes

[multi_binary(
	name = “all_my_binaries”,
	binaries = {
		“//path/to/binary1”: {
			“args”: mode + [“a”, “b”, “c”],
		},
		“//path/to/binary2”, {
			“args”: mode + [“1”, “2”, “3”],
		},
	},
) for mode in [[], [“--debug”]]]

multi_test(
	name = “tests”,
	binaries = “:all_my_binaries”
	test = “:test”,	
)

borkaehw · 2018-10-02T21:09:24Z

Hey @alexeagle, thanks for asking. We are currently tackling this Bazel issue
bazelbuild/bazel#2832
Once we have the multiplex worker ready, multiple run seems to be a great project.

Feel free to start one though.

borkaehw force-pushed the merge2upstream branch from 9f93d67 to 634c628 Compare July 31, 2018 16:39

Implement output runner from bazel command

e9e3a1b

iBazel now watches the log warning from Bazel outputs. Users can watch for specific commands and apply them automatically. The command matching is implemented by regex and it is configurable. Fixes: bazelbuild#18

borkaehw force-pushed the merge2upstream branch from 634c628 to a8e770e Compare July 31, 2018 16:47

borkaehw mentioned this pull request Jul 31, 2018

Output Runner and Multiple Runs #131

Closed

alexeagle reviewed Aug 1, 2018

View reviewed changes

borkaehw force-pushed the merge2upstream branch from a8e770e to aa39b7e Compare August 1, 2018 17:16

borkaehw closed this Aug 3, 2018

borkaehw deleted the merge2upstream branch August 17, 2018 19:57

Globegitter mentioned this pull request Dec 12, 2018

skylark - add attr.string_dict_dict bazelbuild/bazel#3862

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Output Runner and Multiple Runs #133

Output Runner and Multiple Runs #133

borkaehw commented Jul 31, 2018

borkaehw commented Jul 31, 2018

alexeagle commented Aug 1, 2018

alexeagle left a comment

alexeagle Aug 1, 2018

jjudd Aug 1, 2018

alexeagle Aug 1, 2018

borkaehw Aug 1, 2018

alexeagle Aug 1, 2018

alexeagle Aug 1, 2018

borkaehw Aug 1, 2018

jjudd commented Aug 1, 2018

borkaehw commented Aug 1, 2018

alexeagle commented Oct 2, 2018

borkaehw commented Oct 2, 2018

Output Runner and Multiple Runs #133

Output Runner and Multiple Runs #133

Conversation

borkaehw commented Jul 31, 2018

Output Runner

Issue

Fix

Multiple Runs

Issue

Fix

borkaehw commented Jul 31, 2018

alexeagle commented Aug 1, 2018

alexeagle left a comment

Choose a reason for hiding this comment

alexeagle Aug 1, 2018

Choose a reason for hiding this comment

jjudd Aug 1, 2018

Choose a reason for hiding this comment

alexeagle Aug 1, 2018

Choose a reason for hiding this comment

borkaehw Aug 1, 2018

Choose a reason for hiding this comment

alexeagle Aug 1, 2018

Choose a reason for hiding this comment

alexeagle Aug 1, 2018

Choose a reason for hiding this comment

borkaehw Aug 1, 2018

Choose a reason for hiding this comment

jjudd commented Aug 1, 2018

borkaehw commented Aug 1, 2018

alexeagle commented Oct 2, 2018

borkaehw commented Oct 2, 2018