OTLP Exporter blocking indefinitely

[jpkrohling]

Describe the bug
When using the collector with a OTLP exporter, the exporter seems to be blocked indefinitely:

^C2020-06-25T16:18:29.765+0200	INFO	service/service.go:277	Received signal from OS	{"signal": "interrupt"}
2020-06-25T16:18:29.765+0200	INFO	service/service.go:448	Starting shutdown...
2020-06-25T16:18:29.766+0200	INFO	service/service.go:377	Stopping receivers...
2020-06-25T16:18:29.767+0200	INFO	service/service.go:383	Stopping processors...
2020-06-25T16:18:29.768+0200	INFO	builder/pipelines_builder.go:70	Pipeline is shutting down...	{"pipeline_name": "traces/custom-2", "pipeline_datatype": "traces"}
2020-06-25T16:18:29.768+0200	INFO	builder/pipelines_builder.go:76	Pipeline is shutdown.	{"pipeline_name": "traces/custom-2", "pipeline_datatype": "traces"}
2020-06-25T16:18:29.768+0200	INFO	builder/pipelines_builder.go:70	Pipeline is shutting down...	{"pipeline_name": "traces/custom-1", "pipeline_datatype": "traces"}
2020-06-25T16:18:29.768+0200	INFO	builder/pipelines_builder.go:76	Pipeline is shutdown.	{"pipeline_name": "traces/custom-1", "pipeline_datatype": "traces"}
2020-06-25T16:18:29.768+0200	INFO	service/service.go:389	Stopping exporters...

And it stays like that until I manually force kill the process (kill -9 PID).

Steps to reproduce

1. make run in the OpenTelemetry Collector side
2. SPAN_STORAGE_TYPE=memory go run ./cmd/collector/main.go --collector.grpc-server.host-port :14251 --log-level=debug on the Jaeger side
3. send a span to the OpenTelemetry collector
4. stop the OpenTelemetry Collector

What did you expect to see?
The process would finish at most after a couple of seconds.

What did you see instead?
You can check out anytime you like, but you can never leave.

What version did you use?
Version: master as of now (4eca960a4eb02104694324cf161ad9ec944c44c9).

What config did you use?
Config:

receivers:
  otlp:
  jaeger:
    protocols:
      grpc:
      thrift_compact:

processors:
  batch:

exporters:
  otlp:
    endpoint: "localhost:55268"
  jaeger:
    endpoint: "localhost:14251"
    insecure: true
  logging:

service:
  pipelines:
    traces/custom-1:
      receivers: [jaeger]
      processors: [batch]
      exporters: [otlp]
    traces/custom-2:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger, logging]

Environment
OS: Fedora 32 with the latest updates
Compiler(if manually compiled): go1.14.3

Additional context
n/a

[jpkrohling]

While this specific instance of the problem might be solved by #1201, the collector needs a way to forcefully shutdown a misbehaving exporter.

[jmacd]

I wonder what we expect to happen when an OTLP request fails. IMO the correct thing to do would be to buffer the data and aggregate it with new data while waiting for a successful export. This would require new processor support, I guess.

[jpkrohling]

What are the current failure conditions? Is it only about networking? Can a request failed due to bad data? We don't want data to live in the pipeline forever because they can't move forward for some reason :-)

[jpkrohling]

I believe this has to be fixed for the next beta, or at the latest for GA.

cc @tigrannajaryan

[jrcamp]

@bogdandrutu to resolve blocked shutdown (and maybe for startup as well) should we set https://golang.org/pkg/context/#WithTimeout on the context before calling Startup/Shutdown? Should this value be configurable globally or per-component? What should the default be?

[bogdandrutu]

@jrcamp @jmacd for normal requests I added timeout/retry and queueing support for all exporters in #1386

@jrcamp for start and stop I would recommend that to be a config on the service and use WithTimeout on the context passed to start and shutdown.

[bogdandrutu]

I would say that this issue should be closed and open a separate issue for start/stop which I think was not the main concern in this initial issue.

[jrcamp]

New issue opened as a feature request for timeouts. ^