Performance tests - Collector

[jpkrohling]

To better understand the performance of the collector, we need a set of performance tests exercising different paths and configuration options.

We can get a good start if we are able to settle on a specific machine-type (bare metal, probably) and change only the options, such as:

• Without any metric backend, with Prometheus as metrics backend and with expvar as metrics backend
• Different --collector.queue-size values. Bonus points if we can come up with a simple formula for calculating the optimal queue size in relation to the available memory. Might be hard/impossible, as it's most likely dependent on the span size. See Memory-sensitive collector queue size #943.
• Different --collector.num-workers values. Same as above.
• Mix of different --collector.num-workers and --collector.queue-size values. Same as above.

This does not include switching the backend storage, as they are part of a different task.

[jpkrohling]

Question: should we create a /dev/null storage backend? This would make exercising the individual components more accurate.

[zhu0516123]

what is the result of jaeger-collector performance test？My test result is that jaeger-collector can only handler about 6000 spans every second when its cpu limit is 6c.

[jpkrohling]

@jkandasa has executed a few performance tests and could probably prepare a report and/or confirm your findings. I'm not sure what you mean by "6c" though.

I'll let @jkandasa expand on that, but from what I remember, the backing storage is decisive in the overall performance. Which storage are you using for your tests?

[zhu0516123]

6c means 6core. The jaeger-collector container could use 6core cpu to run. The performance test result is that jaeger-collector can handler 12000 spans per second after I add its queue-size to 20000. And then it can handler 36000 spans per second after I use protobuf encode. I use kafka as the storage for my test.

[yurishkuro]

old issue