WIP: Change gRPC to streaming

[pavolloffay]

Opening this to trigger discussion. We should clarify why we need streaming API. I am not fixing tests until it's decided we want to go this way. The binaries should be executable in this PR.

census trace proto https://github.com/census-instrumentation/opencensus-proto/blob/master/src/opencensus/proto/agent/trace/v1/trace_service.proto
census metrics proto https://github.com/census-instrumentation/opencensus-proto/blob/master/src/opencensus/proto/agent/metrics/v1/metrics_service.proto

basic gRPC docs https://grpc.io/docs/tutorials/basic/go.html

[pavolloffay]

One streaming consideration can be performance but I didn't find any good resources. Other use case might be to push configurations to clients. Not sure if we need it for posting batches, maybe if we wanted to get rid of batches and send per span from clients.

[yurishkuro]

Performance is certainly the main consideration, but we need to demonstrate it with perf tests, rather than assuming.

Using streaming for the config endpoints allows switching the agents to the push model. It did come up in our experiments with adaptive sampling where sometimes the 1min polling interval is not fast enough to reduce traffic spikes from certain services (e.g. periodic batch jobs). However, it's not straightforward because we currently share the collector proxy between collector and config APIs, and for collector API we want the agent to load balance across several collectors, where as for push notifications to work they have to be done over a single stable connection, without load balancing (which indirectly implies that we need a stable partitioning scheme in the collectors by the agent id).

[pavolloffay]

Just copying from the bi-weekly meeting.

Before moving to streaming it would be good to clarify:

• performance vs rpc
• load balancing between collectors

for now we have decided to go with rpc and if necessary open another stream port in the future.

[pavolloffay]

Streams seems to be stateful the requests go to the same backed after the connection is established

• https://github.com/grpc/grpc/blob/v1.4.0/src/proto/grpc/reflection/v1alpha/reflection.proto#L39
• https://stackoverflow.com/a/45112952/4158442

In our case we could use between agent and clients where we do not expect load balancing.