Telemetry Support

[simon-mo]

We should add telemetry for collecting anonymized information to inform project roadmap. Currently, we are unsure about the following:

• The types of hardware, specifically GPU types.
• The relevant performance counters (latency, memory usage) and effectiveness of PagedAttention Kernel.
• The cloud providers vLLM is running on.
• The type of architecture and models users are using.

The lack of such visibility and signals harms our prioritization. We are a small team with very little resource; therefore, we want whatever we focus on to be most impactful.

The telemetry would be on by default for CLI and Docker usage. It can be opt-out easily using environment variable/config switches. For library usage (import vllm), I think it would be trickier but we can experiment with turning it on.

Whatever data we collect and the resulting dashboard will be publicly visible.

[simon-mo]

Current design for a v0:

• Add a hook to LLMEngine class to send information on initialization.
• Gather GPU configuration and topology, model type, and the entrypoint to vLLM (TGI, TensorRT, Ray Serve, vLLM Server).
• Allowing disabling of telemetry via environment variable DO_NOT_TRACK, VLLM_NO_USAGE_STATS or a simple configuration file mkdir -p $HOME/.config/vllm/do_not_track.

Generally following the example from Ray here: https://docs.ray.io/en/latest/cluster/usage-stats.html

[simon-mo]

For the server side, we can host a vector.dev http server for log collection. This gives us flexibility choosing and multiplexing the destinations. It can also be used locally for development.

[hmellor]

@simon-mo is this work still planned?

[gyliu513]

Related with #3789

[simon-mo]

Oh this is mostly usage stats collection and it has been released, therefore closing.

This is not related to OpenTelemetry.