-
Notifications
You must be signed in to change notification settings - Fork 0
/
metrics-and-operational-awareness-at-voxer.html
178 lines (163 loc) · 13.9 KB
/
metrics-and-operational-awareness-at-voxer.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
<!DOCTYPE html>
<html dir="ltr" lang="en-US">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>Metrics and operational awareness at Voxer</title>
<meta itemprop="name" content="Metrics and operational awareness at Voxer">
<meta property="og:title" content="Metrics and operational awareness at Voxer">
<link rel="icon" href="/favicon.png">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:creator" content="@ddtrejo">
<meta name="twitter:site" content="@ddtrejo">
<link rel="me" href="https://infosec.exchange/@dtrejo">
<link rel="canonical" href="https://dtrejo.com/metrics-and-operational-awareness-at-voxer">
<meta property="og:url" content="https://dtrejo.com/metrics-and-operational-awareness-at-voxer">
<meta property="og:image" content="https://dtrejo.com/images/social/metrics-and-operational-awareness-at-voxer.jpg"/>
<meta name="image" content="https://dtrejo.com/images/social/metrics-and-operational-awareness-at-voxer.jpg"/>
<meta itemprop="image" content="https://dtrejo.com/images/social/metrics-and-operational-awareness-at-voxer.jpg"/>
<meta name="twitter:image:src" content="https://dtrejo.com/images/social/metrics-and-operational-awareness-at-voxer.jpg">
<meta name="twitter:card" content="summary_large_image">
<meta property="og:type" content="article">
<link rel="stylesheet" href="/css/tachyons.min.css" />
<style>
* { margin-top: 0 }
* + * { margin-top: 1.5em }
li li { margin-top: 1.5em }
html { scroll-behavior: smooth }
body {
font-family: -apple-system, BlinkMacSystemFont, Segoe UI, Roboto, Oxygen, Ubuntu, Cantarell, Fira Sans, Droid Sans, Helvetica Neue, sans-serif;
font-size: 125%;
background-color: #013250;
color: #dcd9d6; /*rgb(220, 217, 214)*/
-webkit-font-smoothing: antialiased;
-moz-osx-font-smoothing: grayscale;
}
h1, h2, h3, h4, h5, h6 {
line-height: 1.25;
font-weight: 600;
}
a {
color: #f57a59;
font-weight: 500;
text-decoration: none;
}
a:focus, a:active, a:hover {
color: #ff6300;
}
.footer {
border-top: 1px solid rgb(25, 73, 110);
background-color: #021928;
}
.amp {
font-family: Baskerville, "Goudy Old Style", "Palatino", "Palatino Linotype", "Book Antiqua", serif;
font-style: italic;
font-weight: 400;
}
h1, h2, h3, h4, h5, h6, li, p, span, blockquote, pre, code {
text-rendering: optimizeLegibility;
}
hr {
border: none;
border-bottom: 1px solid rgba(0, 0, 0, .1);
border-top: 1px solid rgba(220, 217, 214, .1);
}
/* improve prism styles */
pre, pre[class*=language-] {
line-height: 1.4;
padding: 1.25rem 1.5rem;
margin: .85rem 0;
background-color: #282c34;
background-color: #191919;
border-radius: 6px;
overflow: auto;
}
blockquote {
margin-left: 0;
border-left: .5rem solid rgba(0, 0, 0, .5);
padding-left: 1.5em;
}
.customcolor { color: #013250 }
.b--customcolor { border-color: #013250 }
</style>
<meta name="cf-2fa-verify" content="27f0edaf36e9d24">
</head>
<body class="pt2 lh-copy">
<nav class="mw7 ph2 center flex flex-column flex-row-l"> <a class="fw5 link flex self-center self-start-l" href="/"> <img src="/images/dtrejo.jpg" class="w2 h2 br-100 mr3" alt="David Trejo"> dtrejo </a> <div class="flex self-center ml-auto-l mt0-l"> <a class="f5 fw5 link mt0 mr3 ttu" href="#footer">Articles</a> <a class="f5 fw5 link mt0 mr0 ttu" href="/books">Books</a> <!-- <a class="f5 fw5 link mt0 ttu" href="https://engineeroverflow.com" >Hire Engineers</a > --> </div></nav>
<article class="content mw7 ph2 center">
<h1 id="metrics-and-operational-awareness-at-voxer">Metrics and operational awareness at Voxer</h1>
<p><img src="./images/46879900-Voxer_Logo.png" alt=""></p>
<p><em>Note: written in August 2012 during my summer at Voxer. <code>sky</code> has been released
as <a href="http://voxer.github.io/zag/">zag</a>!</em></p>
<p>Please read the article on Voxer's <a href="/node-single-point-of-failure.html">no single point of failure architecture</a>,
as it explains how process rings work, and how Voxer's load balancing works.</p>
<p><img src="./images/46879901-sky.png" alt=""></p>
<h2 id="what-are-the-salient-implementation-features-of-our-metrics-solution">What are the salient implementation features of our metrics solution?</h2>
<p>Client side (on your app servers, which emit data)</p>
<ul>
<li>metrics are coalesced into 1440 byte udp packets to optimize network
efficiency</li>
<li><a href="/node-single-point-of-failure.html">handy node.js bindings</a> which allow you simply call
<code>sky.counter("http_status|200")</code> whenever a certain call suceeds or fails</li>
<li>keys emitted with the form <code>http_status|200</code> are rolled together with keys
of the form <code>http_status|404</code>, giving you both information on, say, 404s,
200 OKs, AND the high-level view of how many http requests you are doing per
minute.</li>
</ul>
<p>Server side (on the skyservers which aggregate metrics data)</p>
<ul>
<li>skyservers form a ring, and clients load balance themselves so as not to
overwhelm a single server</li>
<li>intra-ring skyserver communication is also coalesced for efficiency.</li>
<li>load balancing allows you to add more skyservers as the number of your
appservers grows</li>
<li>every minute, skyservers talk to a single redis server to roll-up their data</li>
<li>the redis intermediary in front of riak allows for accurate counters,
because redis performs atomic <a href="http://redis.io/commands/incr"><code>INCR</code></a> operations</li>
</ul>
<p>Persisting data</p>
<ul>
<li>a rollup process continuously pulls keys out of redis and puts the latest
time period of data into redis, simultaneously performing 5-minute rollups of
the 1 minute data, if applicable.</li>
<li>riak as our persistence medium allows us to keep historical data forever, as
we can increase the size of our riak cluster by adding a machine.</li>
</ul>
<p>Viewing metrics and adding new ones</p>
<ul>
<li>the redis server keeps a list of keys that have recently been emitted, and exposes
this list to our graphing UI, meaning that to begin recording a new metric,
you need not do any configuration, simply start recording it on your app servers.
The Sky UI will simply pick it up and start showing it to you.</li>
<li>the Sky UI will also hit redis for the latest minute's data, giving
you real-time access to what is going on right now in your infrastructure.</li>
</ul>
<h2 id="why-didnt-we-just-use-etsys-statsd-and-whisper-and-carbon">Why didn't we just use etsy's statsd and whisper and carbon?</h2>
<p>Etsy wrote a library called <a href="https://github.com/etsy/statsd">statsd</a>, which works with <a href="https://github.com/etsy/statsd">whisper</a> (the
backend) to record metrics data. There are a couple reasons we chose not to use
these technologies.</p>
<ul>
<li>rings of servers recieving metrics is the only way we can handle the
volume of data being recorded by our app servers</li>
<li>UDP as the transport mechanism allows our metrics servers to stay up, even
if our app servers send too many events.</li>
<li>riak allows us to store historical data forever (just add another machine)</li>
<li>we think our <a href="http://voxer.github.io/zag/">graphing UI</a> is pretty snazzy, especially since it has
<a href="https://github.com/sentientwaffle/llquantize">heat maps</a></li>
</ul>
<p>If you'd like to read about <a href="http://voxer.github.io/zag/">Zag</a>, our awesome graphing UI, please check out
<a href="http://voxer.github.io/zag/">this article</a>.</p>
<p>Thanks for reading!<br>
<a href="https://dtrejo.com">David Trejo</a><br>
<a href="https://twitter.com/ddtrejo">@ddtrejo</a> & <a href="https://github.com/DTrejo">DTrejo on github</a></p>
</article>
<div class="mw7 ph2 center"> <img src="/images/dtrejo.jpg" title="David Trejo" class="br-100 fl ml2 mr2" height="80" /> <div class="overflow-hidden mr2"> <h4>David Trejo</h4> <p>Engineer at Chime <span class="amp">&</span> consultant. Past clients include Credit Karma, Aconex, Triplebyte, Neo, the Brown Computer Science Department, Voxer, Cloudera, and the Veteran's Benefits Administration.</p> </div></div>
<div id="footer" class="footer mt5 pt5 pb4"><div class="mw7 ph2 center"> <h3>Hiring Engineers? You'll like these articles</h3> <ol><li><a href="/engineer-interview-script">Growth & full stack engineering interview script</a></li><li><a href="/referrals">How to hustle on your referrals so they actually get hired</a></li><li><a href="/moneyball-book-review-and-measuring-revenue-per-engineer">Moneyball book review and measuring revenue per engineer</a></li><li><a href="/questions-that-senior-systems-engineers-want-answered">Questions that Senior Systems Engineers want answered</a></li><li><a href="/tools-for-hiring-engineers">Tools for hiring engineers that save time and increase conversion</a></li><li><a href="/how-to-reliably-get-your-team-to-write-articles-for-your-engineering-blog">How to reliably get your team to write articles for your engineering blog</a></li><li><a href="/why-share-salary">Why share salary ranges with candidates?</a></li><li><a href="/massive-list-of-engineering-recruiting-channels">Massive list of engineering recruiting channels</a></li><li><a href="/how-to-run-an-intern-program-like-cloudera-s">How to run an intern program like Cloudera's</a></li><li><a href="/how-to-replicate-the-warmth-of-referral-hires">How do you replicate the warmth and friendliness of a referral hire, even with a stranger engineer? By treating hiring like dating.</a></li><li><a href="/how-to-run-an-intern-meetup">How to run an intern meetup</a></li></ol>
<h3>Growth Engineering Articles</h3> <ol><li><a href="/engineer-interview-script">Growth & full stack engineering interview script</a></li><li><a href="/engineering-lead">Engineering lead checklists</a></li><li><a href="/days-as-a-growth-engineer">Everything you should do in the first 30 days of a new growth engineering job</a></li><li><a href="/how-do-you-transition-from-rails-lead-engineer-to-growth-engineer">How do you transition from Rails lead engineer to growth engineer?</a></li><li><a href="/chaos-in-your-product-prototyping">Chaos in your product prototyping</a></li></ol>
<h3>How to...</h3> <ol><li><a href="/tailoring-perfect-shirt-pants-jacket">Tailoring the perfect shirt, pants, and jacket</a></li><li><a href="/home-search">Home Searching Advice</a></li><li><a href="/publishing-vue-codesandbox-to-github-pages">How to publish a Vue CodeSandbox.io to Github Pages on a custom domain</a></li><li><a href="/how-to-get-your-business-to-pull-you-forward">How to get your business to pull you forward</a></li><li><a href="/overcome-cold-email-procrastination">Overcome cold-email procrastination</a></li><li><a href="/how-to-write-consistently-painlessly-and-without-writers-block">How to write consistently, painlessly, and without writer's block</a></li><li><a href="/how-to-email">How to email</a></li><li><a href="/how-to-buy-and-cook-steak">How to Buy and Cook Steak</a></li></ol>
<h3>Health</h3> <ol><li><a href="/why-go-carnivore-aka-zero-carb">Why go carnivore? (aka zero carb)</a></li><li><a href="/cancer">Cancer, fasting, carnivore, and chemotherapy side effects</a></li><li><a href="/lifting">Start weightlifting in just 720 days</a></li></ol>
<h3>Node.js</h3> <ol><li><a href="/my-javascript-tooling-wishlist">My Javascript tooling wishlist</a></li><li><a href="/metrics-and-operational-awareness-at-voxer">Metrics and operational awareness at Voxer</a></li><li><a href="/zero-effort-responsive-email-creation">Zero effort responsive email creation</a></li><li><a href="/node-single-point-of-failure">Node single point of failure</a></li></ol>
<h3>Git</h3> <ol><li><a href="/pull-requests">How Open Source People Do Pull Requests and What We Can Learn from Them</a></li><li><a href="/develop-on-a-cloud-machine">Develop on a cloud machine</a></li><li><a href="/faster-git-workflow">A faster Git workflow with Ampline</a></li><li><a href="/my-github-pull-request-workflow">My Github pull request workflow</a></li></ol>
<h3>Top Projects</h3> <ul> <li> <a href="https://duskk.com">Duskk.com</a> Want to unlock the power of your writing? Discover greatness inside yourself with <a href="https://duskk.com">Duskk, a journaling app</a>. </li> <li> <a href="https://EngineerWorth.com">EngineerWorth.com</a> Is that product feature worth coding? Run a quick ROI calculation and find out. </li> <li> <a href="/qlock/qlock.html">jQuery Qlock2</a> Tell time with this $800 clock, for free. </li> <li> <a href="/colorslice/">ColorSlice</a> Wish it were easier to steal and create color schemes? Colorslice helps front-end developers <span class="amp">&</span> designers with color. </li> <li> <a href="/colorpreview/">ColorPreview</a> Vet color combinations and pick only the ones with enough contrast, that your eye finds pleasing. </li> <li> <a href="/lazytemplating/">Lazy Templating</a> What if you could do HTML templating using a spreadsheet? Now you can. </li> <li> <a href="http://github.com/DTrejo/ampline">Ampline</a> Tired of copy-pasting or tab-completing long file paths? Accelerate yourself on the command-line. </li> </ul> <h4><a href="/archived">Archived</a></h4> <h3>Find me</h3> <ul> <li><a href="https://github.com/DTrejo">Github</a></li> <li><a rel="me" href="https://infosec.exchange/@dtrejo">Mastodon (infosec.exchange)</a></li> <li><a href="https://twitter.com/ddtrejo">Twitter</a></li> <li> <a href="#david åt dtrejo.com" title="david åt dtrejo.com" onclick="event.preventDefault(); this.href = 'mailto:' + (this.innerText = 'david' + '@dtrejo.com')"> Email me ✉️</a> </li> </ul> <div> Copyright © 2009-2024 David Trejo. Opinions mine. </div></div></div>
</body>
</html>