Allow persistent workers to communicate with Bazel using JSON instead of protos

[Ubehebe]

Description of the problem / feature request:

Currently, persistent workers must use protocol buffers to communicate with Bazel. This limits the languages that workers can be implemented in to those that have first-class protocol buffer support.

Allowing workers to use a JSON serialization of the worker protocol would make it easier to implement workers in languages other than Java/C++. Since languages without first-class proto support often also have performance problems, this could open a number of interesting use cases (for example, a worker written as a shell script).

What underlying problem are you trying to solve with this feature?

I recently implemented a persistent worker in Python. Although it is possible, it is not easy, mostly because of the immaturity of the Python proto toolchain and APIs (no py_proto_library (#2626), no methods for varint-delimited reading/writing (protocolbuffers/protobuf#54), etc.). The repo linked above discusses the workarounds I needed to use to get a Python worker working.

The actual worker protocol is quite simple. I think it would be amenable to JSON serialization, which would make the Python worker much simpler.

[cushon]

The actual worker protocol is quite simple.

Related: #2832

[katre]

Closed for lack of activity. Please ping the reviewers to reopen if this is still a problem.

[susinmotion]

@karljamesgray is looking into this.

[keith]

Looks like this change is related? aa5592e

[susinmotion]

Yes, the infrastructure is there as of September, and is documented in https://docs.bazel.build/versions/master/persistent-workers.html. We're doing a bit more testing; stay tuned for a blog post in the next couple of weeks.

[meisterT]

If you ended up on this issue you might be interested in https://www.youtube.com/watch?v=d7q1cVLuRhY&list=PLxNYxgaZ8RseRybXNbopHRv6-wGmFr04n&index=3

[larsrc-google]

This is now generally available, since 003cfcd also for multiplex workers. It's automatically available for any worker implementation that uses WorkRequestHandler, the execution-requirements must still be updated in order to use it, though.