Memory-sensitive collector queue size

[jpkrohling]

Requirement - what kind of business use case are you trying to solve?

Avoiding dropping of spans on the collector side when the backing storage isn't really fast, but enough memory is available for a bigger buffer on the collector size

Problem - what in Jaeger blocks you from solving the requirement?

Under stress conditions, the default queue size of 2000 causes spans to be dropped quite frequently, even though there's enough memory to have a bigger queue size

Proposal - what do you suggest to solve the problem or improve the existing situation?

The first idea is to have a dynamic collector queue size, with the following characteristics

• Heuristics to determine the "average" span size in memory (could also be used for the in-memory storage)
• Flag to specify a memory size for the queue (default: half of the host's memory? 25%?)
• Adjust the queue size based a simple division of the above
• Update a metrics gauge with the current queue size
• Flag to enable usage of this new collector queue mechanism

[yurishkuro]

This does not seem related to #355, correct? Maybe retitle to "memory-sensitive queue size" rather than "dynamic".

[isaachier]

So there are a bunch of algorithms that have been studied and used for network scheduling (https://en.wikipedia.org/wiki/Network_scheduler). My suggestion is either to reimplement one of them directly, or potentially leverage existing OS configurations to drop packets from spans according to one of those existing strategies.

[jpkrohling]

This is not related to #355. This is about automatically adjusting the queue size based on the known span sizes and available memory, to make optimal usage of the assigned resources.