Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: introduce data sink to collect snapshot from each metric periodically #1117

Open
wants to merge 24 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
98efadf
feat: take snapshots from metrics and export to data sink
empiredan Aug 1, 2022
73c1796
feat: take snapshots from metrics and export to data sink
empiredan Aug 1, 2022
0a9e3ac
feat: take snapshots from metrics and export to data sink
empiredan Aug 1, 2022
4d1e33b
feat: take snapshots from metrics and export to data sink
empiredan Aug 3, 2022
a8ac596
feat: take snapshots from metrics and export to data sink
empiredan Aug 3, 2022
e1c6bfd
feat: take snapshots from metrics and export to data sink
empiredan Aug 3, 2022
017f945
feat: take snapshots from metrics and export to data sink
empiredan Aug 3, 2022
37fc291
feat: take snapshots from metrics and export to data sink
empiredan Aug 3, 2022
c9629f0
feat: take snapshots from metrics and export to data sink
empiredan Aug 4, 2022
e04d72e
feat: take snapshots from metrics and export to data sink
empiredan Aug 4, 2022
1b9f4af
feat: take snapshots from metrics and export to data sink
empiredan Aug 4, 2022
4768e58
feat: take snapshots from metrics and export to data sink
empiredan Aug 4, 2022
1220915
feat: take snapshots from metrics and export to data sink
empiredan Aug 5, 2022
83200e1
feat: take snapshots from metrics and export to data sink
empiredan Aug 5, 2022
85f3f26
feat: take snapshots from metrics and export to data sink
empiredan Aug 9, 2022
c92ef2d
feat: take snapshots from metrics and export to data sink
empiredan Aug 10, 2022
3c1f503
feat: take snapshots from metrics and export to data sink
empiredan Aug 11, 2022
5be088c
feat: take snapshots from metrics and export to data sink
empiredan Aug 11, 2022
3b3d9dc
feat: take snapshots from metrics and export to data sink
empiredan Aug 11, 2022
f108b67
feat: take snapshots from metrics and export to data sink
empiredan Aug 11, 2022
7e6393a
feat: introduce data sink to collect snapshot from each metric period…
empiredan Aug 12, 2022
5881d9b
feat: introduce data sink to collect snapshot from each metric period…
empiredan Aug 12, 2022
36fac25
feat: introduce data sink to collect snapshot from each metric period…
empiredan Aug 12, 2022
eb116eb
feat: introduce data sink to collect snapshot from each metric period…
empiredan Aug 15, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
feat: take snapshots from metrics and export to data sink
empiredan committed Aug 4, 2022
commit c9629f06d0bacd0462804621348d665e31e47e1b
32 changes: 22 additions & 10 deletions src/rdsn/include/dsn/utility/metrics.h
Original file line number Diff line number Diff line change
@@ -21,6 +21,7 @@
#include <atomic>
#include <bitset>
#include <functional>
#include <map>
#include <memory>
#include <mutex>
#include <set>
@@ -140,7 +141,7 @@ using metric_data_sink_ptr = ref_ptr<metric_data_sink>;
class metric_entity : public ref_counter
{
public:
using attr_map = std::unordered_map<std::string, std::string>;
using attr_map = std::map<std::string, std::string>;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why change to use std:map?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All of the attributes of an entity will be used as the labels of the metric. The labels will later be passed to metric_snapshot, thus each metric snapshot may has multiple labels. To maintain each snapshot in a map, data sink may choose to encode the labels a snapshot has, as what metric_snapshot::encode_attributes has done. In comparison with std::unordered_map, it is easier for std::map to be encoded since it's ordered.

using metric_map = std::unordered_map<const metric_prototype *, metric_ptr>;

const std::string &id() const { return _id; }
@@ -273,6 +274,16 @@ class metric_registry : public utils::singleton<metric_registry>
public:
using entity_map = std::unordered_map<std::string, metric_entity_ptr>;

private:
friend class metric_entity_prototype;
friend class utils::singleton<metric_registry>;
friend class metrics_test;

metric_registry();
~metric_registry();

void on_close();

template <typename MetricDataSink, typename... Args>
void register_data_sink(Args &&... args)
{
@@ -283,15 +294,13 @@ class metric_registry : public utils::singleton<metric_registry>
_sinks.emplace_back(new MetricDataSink(std::forward<Args>(args)...));
}

entity_map entities() const;

private:
friend class metric_entity_prototype;
friend class utils::singleton<metric_registry>;
void unregister_data_sinks()
{
utils::auto_write_lock l(_lock);
std::vector<metric_data_sink_ptr>().swap(_sinks);
}

metric_registry();
~metric_registry();
void on_close();
entity_map entities() const;

metric_entity_ptr find_or_create_entity(const std::string &id, metric_entity::attr_map &&attrs);

@@ -403,7 +412,7 @@ class metric_snapshot
{
public:
using value_type = double;
using attr_map = std::unordered_map<std::string, std::string>;
using attr_map = std::map<std::string, std::string>;

metric_snapshot(const string_view &name, metric_type type, value_type value, attr_map &&attrs);
virtual ~metric_snapshot() = default;
@@ -416,6 +425,9 @@ class metric_snapshot

const attr_map &attrs() const { return _attrs; }

void encode_attrs(std::string &str) const;
static void decode_attrs(const std::string &str, attr_map &attrs);

private:
const string_view _name;
const metric_type _type;
33 changes: 33 additions & 0 deletions src/rdsn/src/utils/metrics.cpp
Original file line number Diff line number Diff line change
@@ -15,11 +15,17 @@
// specific language governing permissions and limitations
// under the License.


#include <dsn/utility/metrics.h>

#include <iterator>

#include <boost/algorithm/string/join.hpp>

#include <dsn/c/api_utilities.h>
#include <dsn/utility/flags.h>
#include <dsn/utility/rand.h>
#include <dsn/utility/strings.h>

#include "shared_io_service.h"

@@ -219,6 +225,33 @@ metric_snapshot::metric_snapshot(const string_view &name,
{
}

void metric_snapshot::encode_attrs(std::string &str) const
{
std::vector<std::string> kvs;
std::transform(attrs.begin(),
attrs.end(),
std::back_inserter(kvs),
[](const attr_map::value_type &kv) {
return fmt::format("{}={}", kv.first, kv.second);
});

str = boost::join(kvs, "|");
}

void metric_snapshot::decode_attrs(const std::string &str, attr_map &attrs)
{
std::vector<std::string> kvs;
utils::split_args(str.c_str(), kvs, '|');

for (const auto &elem : kvs) {
std::vector<std::string> kv;
utils::split_args(elem.c_str(), kv, '=', true);
dcheck_eq(kv.size(), 2);

attrs[kv[0]] = kv[1];
}
}

counter_snapshot::counter_snapshot(const string_view &name,
metric_type type,
value_type value,
12 changes: 10 additions & 2 deletions src/rdsn/src/utils/test/metrics_test.cpp
Original file line number Diff line number Diff line change
@@ -153,9 +153,17 @@ class metrics_test : public testing::Test
metrics_test() = default;
virtual ~metrics_test() = default;

void SetUp() override { metric_registry::instance().register_data_sink<my_data_sink>(); }
void SetUp() override {}

void TearDown() override {}
void TearDown() override
{
metric_registry::instance().unregister_data_sinks();
}

void register_my_data_sink()
{
metric_registry::instance().register_data_sink<my_data_sink>();
}
};

TEST_F(metrics_test, create_entity)