Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reset trace context after fork #1225

Merged
merged 2 commits into from
Nov 3, 2020
Merged

Conversation

delner
Copy link
Contributor

@delner delner commented Oct 29, 2020

When Ruby processes fork, they copy objects from the parent process into the child process: this includes the trace context objects we've assigned to each Thread. As a result, child processes are copying all the spans from the parent process, which could result in redundant transmission of spans.

Instead, we'd like forked processes to act like distributed traces; they should be linked to the parent process (trace ID/parent ID/sampling, etc) but only build and transmit spans generated from its own process.

To do this, we're leveraging the forking helpers implemented in #1221 to make the Datadog::Context fork aware, then lazily clone the Datadog::Context in the Datadog::DefaultContextProvider whenever a context is accessed and observed to have forked.

@delner delner added core Involves Datadog core libraries feature Involves a product feature labels Oct 29, 2020
@delner delner requested review from marcotc, ericmustin and a team October 29, 2020 18:17
@delner delner self-assigned this Oct 29, 2020
marcotc
marcotc previously approved these changes Oct 29, 2020
@delner delner force-pushed the feature/reset_context_after_fork branch from 7966f46 to daae013 Compare October 29, 2020 19:48
marcotc
marcotc previously approved these changes Oct 29, 2020
@delner
Copy link
Contributor Author

delner commented Oct 30, 2020

Note, it appears as though this pull request may address an latent issue where forking mid-trace would cause spans generated in the fork to be lost. e.g.

tracer.trace('parent') do
  fork do
    tracer.trace('fork') do
      # some work
    end
  end
end

Before these changes, the fork would inherit all the spans from the parent process, and when the fork would complete, it would think its "still active" thus skip flushing fork spans when the fork ends. After these changes, the parent spans are absent in the fork, allowing spans generated in the fork to be flushed because the trace is "complete".

@delner delner force-pushed the feature/reset_context_after_fork branch from daae013 to 53f6226 Compare November 3, 2020 19:08
@delner delner merged commit 44e76d3 into master Nov 3, 2020
@delner delner deleted the feature/reset_context_after_fork branch November 3, 2020 21:16
@marcotc marcotc added this to the 0.43.0 milestone Nov 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Involves Datadog core libraries feature Involves a product feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants