Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance improvements #91

Merged
merged 4 commits into from
Sep 17, 2018
Merged

Performance improvements #91

merged 4 commits into from
Sep 17, 2018

Conversation

see https://github.com/frankmcsherry/differential-dataflow/issues/113

- a new Variable implementation by Frank that reduces the cost of
recursive variables by 1/3
- use distinct_total instead of distinct where safe
- don't create recursive variables for non-recursive collections
- use 32-bit timestamp for nested scopes
- differential moved to tag-based release model
- use barrier synchronization to block worker threads instead of idling
when there is no work to do
@ryzhyk ryzhyk requested a review from mihaibudiu September 17, 2018 08:25
@ryzhyk
Copy link
Contributor Author

ryzhyk commented Sep 17, 2018

(merging without review as the current master is broken by differential update)

@ryzhyk ryzhyk merged commit b717dcc into master Sep 17, 2018
@ryzhyk ryzhyk deleted the profiling branch September 17, 2018 08:26
@@ -0,0 +1,109 @@
//! Memory profile of a DDlog program.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't there some overhead for adding all this code, even if profiling is disabled?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be negligible. The runtime callback is invoked once for every batch and basically just updates two keys in two different maps.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants