Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[resque] Shutdown tracer and flush data on job finish instead of using SyncWriter. #474

Merged
merged 2 commits into from
Jul 23, 2018

Conversation

pawelchcki
Copy link
Contributor

@pawelchcki pawelchcki commented Jun 26, 2018

Using latest improvements to how shutdown is performed - we can safely and quickly flush all data manually when the job has finished running.

Its faster and safer than SyncWriter that needs to spawn two threads and connection to the agent on each trace write.

@pawelchcki pawelchcki changed the title Shutdown tracer and flush data on job finish instead of using SyncWriter. [resque] Shutdown tracer and flush data on job finish instead of using SyncWriter. Jun 26, 2018
@pawelchcki pawelchcki added do-not-merge/WIP Not ready for merge bug Involves a bug core Involves Datadog core libraries labels Jun 26, 2018
@delner
Copy link
Contributor

delner commented Jun 27, 2018

@pawelchcki This is marked as WIP: is it ready for review?

@pawelchcki
Copy link
Contributor Author

@delner for now there is no pressing need to disable SyncWriter. I'll close this PR for now

@pawelchcki pawelchcki closed this Jun 28, 2018
@pawelchcki pawelchcki reopened this Jul 16, 2018
@pawelchcki pawelchcki self-assigned this Jul 16, 2018
@pawelchcki pawelchcki requested a review from delner July 16, 2018 15:40
@pawelchcki pawelchcki removed the do-not-merge/WIP Not ready for merge label Jul 16, 2018
@pawelchcki pawelchcki added this to the 0.14.0 milestone Jul 16, 2018
Copy link
Contributor

@delner delner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@delner delner modified the milestones: 0.14.0, 0.13.1 Jul 16, 2018
@pawelchcki pawelchcki changed the base branch from master to 0.14-dev July 16, 2018 15:54
@delner delner modified the milestones: 0.13.1, 0.14.0 Jul 16, 2018
end
end
end
end
end

Resque.before_first_fork do
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are we sure of that change? I remember we added it to handle some cases. We need to deeply test those changes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We want to ship it first as a beta release and before that do another round of testing.

I've tested this change locally when developing it and coupled with some other improvements in separate PRs, it did flush and shutdown as quickly as SyncWriter.

This change is mostly driven by the fact that SyncWriter performance deteriorates quite a bit under multithreaded setup.

@pawelchcki pawelchcki merged commit 0a5402a into 0.14-dev Jul 23, 2018
@pawelchcki pawelchcki deleted the bugfix/stop_using_sync_writer_for_resque branch July 23, 2018 09:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Involves a bug core Involves Datadog core libraries
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants