Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tracing: track goroutine ID in Span #59200

Closed
tbg opened this issue Jan 20, 2021 · 1 comment · Fixed by #59717
Closed

tracing: track goroutine ID in Span #59200

tbg opened this issue Jan 20, 2021 · 1 comment · Fixed by #59717
Assignees
Labels
A-tracing Relating to tracing in CockroachDB. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)

Comments

@tbg
Copy link
Member

tbg commented Jan 20, 2021

Is your feature request related to a problem? Please describe.
While Span can cross goroutine boundaries, idiomatically it should not. This gives us a straightforward way to, in many cases, heuristically find the goroutine associated to a long-running trace span. Note that the tracked goroutine may already have terminated, and the goroutine ID may now be reused by a different goroutine. This is why it says "heuristically" - in practice this should work well.

Describe the solution you'd like

We grab the goroutine ID (goid package does it) at span creation time (we could also update it on every access, but not sure this buys us much) and can expose it as part of #55733.

Describe alternatives you've considered

#58609 suggests allowing to hook into the output of a span (making it verbose if need be). However, that gives no clue as to the location of a truly stuck span (say a goroutine sitting on a select). It also involves more complexity.

Additional context

We are at the moment targeting "expert debugging" for this use case only. Chasing down a goroutine in this way will be too complicated for end users, but should be within the realm of engineers, TSEs, CC-SREs, CS, etc with proper playbooks.

@tbg tbg added the A-tracing Relating to tracing in CockroachDB. label Jan 20, 2021
@blathers-crl
Copy link

blathers-crl bot commented Jan 20, 2021

Hi @tbg, I've guessed the C-ategory of your issue and suitably labeled it. Please re-label if inaccurate.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.

@blathers-crl blathers-crl bot added the C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. label Jan 20, 2021
@tbg tbg added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) and removed C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. labels Jan 20, 2021
tbg added a commit to tbg/cockroach that referenced this issue Jan 22, 2021
Once we have cockroachdb#55733 and cockroachdb#59200, it will be very useful to be able to
join together, on each node, the long-running trace *stacks*.

To this end, a `crdb_internal.node_stacks` table is added.

Release note: None
@angelapwen angelapwen self-assigned this Feb 1, 2021
@craig craig bot closed this as completed in a59f0c2 Feb 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-tracing Relating to tracing in CockroachDB. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants