Instrument Tokio Wakers

[seanmonstar]

We want to detect a few things with regards to the Wakers related to a Task:

• Time from wake to poll
• If poll returns Pending but the Waker wasn't cloned.
  • While not fool proof, this could be a good-enough heuristic to detect a "forgot to wake".
• If wake was called from a different thread, or same thread.
• If a wake is called from within poll
  • While not wrong, could be a sign to help someone notice a task that "busy loops".

So, we'll need to instrument Tokio's wakers: emitting events for wake, wake_by_ref, clone, and drop. A couple notes about implementing:

• These are tracing events, not spans.
• We should pick some unique names, so that the subscriber can identify them.
• We need a way to map a waker to the task ID.
  • The current "task ID" used by the subscriber is the span ID of the instrumented spawn.
  • We could possibly have Tokio's runtime record an actual task ID once the task is stored in the scheduler, something that the waker can access too.

[hawkw]

I did a rough prototype pass of this a fairly long while ago (which, I think, is probably outdated and won't work with the latest Tokio). One thing I did is add spans to the various resources' poll methods, as well, so that when a waker is cloned, we can record which resource we are registering interest in, and (in the case of resources that have multiple forms of readiness) which form of readiness we are polling for.

[seanmonstar]

Yes, by instrumenting wakers and also resources (#39), we should be able to correlate these relationships and show them in the Console.

[seanmonstar]

The other way around could also work. Since a "subscriber" already needs to produce a unique span ID, it'd be useful to be able to reuse that than to have a second unique counter. The "other way" would be to save the span ID into Tokio's RawTask struct, so that the waker logs can grab it and emit it.

It does seem like a bit of annoying propagating of an extra argument through all the internal Tokio task creation code, but conditionally (so plenty of cfg_trace). But it seems like the way.

[seanmonstar]

Done in tokio-rs/tokio#3836, opened #42 for next step.