[elasticsearch] Jaeger Collector goes down frequently with error "panic: runtime error: invalid memory address or nil pointer dereference"

[yurishkuro]

Reported through the mailing list https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/jaeger-tracing/g9uVCUL28Ls/8SeJy8-ZAQAJ

My Jaeger collector often goes down with below error!

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xd21763]

goroutine 50 [running]:
github.com/jaegertracing/jaeger/pkg/es/config.(*Configuration).NewClient.func2(0x648, 0xc0001d0720, 0x1, 0x1, 0x0, 0x11f5780, 0xc0002ae060)
	/home/travis/gopath/src/github.com/jaegertracing/jaeger/pkg/es/config/config.go:117 +0xe3
github.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic%2ev5.(*bulkWorker).commit(0xc00002fb00, 0x1203860, 0xc0000360b0, 0x1, 0xc00025a001)
	/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic.v5/bulk_processor.go:531 +0x37d
github.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic%2ev5.(*bulkWorker).work(0xc00002fb00, 0x1203860, 0xc0000360b0)
	/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic.v5/bulk_processor.go:468 +0x390
created by github.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic%2ev5.(*BulkProcessor).Start
	/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic.v5/bulk_processor.go:306 +0x1bb
panic: send on closed channel

[yurishkuro]

@pavolloffay this sounds like a potential bug in olivere/elastic, since I don't recall us interacting with any channels.

[pavolloffay]

Yes it looks like from ES bulk processor. I remember there was a fix which added nil checks to prevent similar errors hence we should find out what version of Jaeger they are running.

[pavolloffay]

This is the PR with nill checks #1467, fixed in 1.12.0

[pavolloffay]

Closing done in #1467