Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(hotkey): capture data part2 - declare coarse collector #624

Merged
merged 26 commits into from
Oct 27, 2020

Conversation

Smityz
Copy link
Contributor

@Smityz Smityz commented Oct 20, 2020

the implement of coarse collector

Config changes

[pegasus.server]
+ coarse_data_variance_threshold = 3
+ data_capture_hash_bucket_num = 37

@acelyc111 acelyc111 added the type/config-change Added or modified configuration that should be noted on release note of new version. label Oct 20, 2020
src/server/hotkey_collector.h Outdated Show resolved Hide resolved
src/server/hotkey_collector.h Outdated Show resolved Hide resolved
src/server/hotkey_collector.cpp Outdated Show resolved Hide resolved
src/server/hotkey_collector.cpp Outdated Show resolved Hide resolved
@Smityz Smityz closed this Oct 20, 2020
@Smityz Smityz deleted the hotkey-coarse branch October 20, 2020 09:24
@Smityz Smityz restored the hotkey-coarse branch October 20, 2020 09:25
@Smityz Smityz reopened this Oct 20, 2020
acelyc111
acelyc111 previously approved these changes Oct 20, 2020
@acelyc111
Copy link
Member

the implement of coarse collector

Config changes

[pegasus.server]
+ coarse_data_variance_threshold
+ data_capture_hash_bucket_num

Better to add the default value in PR description.

acelyc111
acelyc111 previously approved these changes Oct 21, 2020
src/server/hotkey_collector.cpp Outdated Show resolved Hide resolved
src/server/hotkey_collector.cpp Outdated Show resolved Hide resolved
static int get_bucket_id(dsn::string_view data)
{
size_t hash_value = boost::hash_range(data.begin(), data.end());
return static_cast<int>(hash_value % FLAGS_data_capture_hash_bucket_num);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Calculate hash value of every key may be wastly, is it possible to use https://github.com/XiaoMi/rdsn/blob/master/src/dsn.layer2.thrift#L92 directly?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the current design, it can't get this info

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remember to improve it #628

acelyc111
acelyc111 previously approved these changes Oct 22, 2020
acelyc111
acelyc111 previously approved these changes Oct 26, 2020
src/server/hotkey_collector.cpp Show resolved Hide resolved
src/server/hotkey_collector.cpp Show resolved Hide resolved
src/server/hotkey_collector.cpp Show resolved Hide resolved
src/server/hotkey_collector.cpp Outdated Show resolved Hide resolved
coarse_data_variance_threshold,
3,
"the threshold of variance calculate to find the outliers");

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove the blank line

@acelyc111 acelyc111 merged commit f2d850b into apache:master Oct 27, 2020
@neverchanje neverchanje mentioned this pull request Mar 1, 2021
acelyc111 pushed a commit that referenced this pull request Jun 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/config-change Added or modified configuration that should be noted on release note of new version.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants