This talk was super inspiring. I strongly suggest watching a video when posted.
Link to livestream, starting at this talk: https://www.youtube.com/watch?v=vQBtun-Ein8&index=2&list=PLGJrMkJUF3M3Vi_vgDiyBpWsOP41XzJnM&t=303m5s
- "Wow. that sounds awful"
- "Its fine"
- "alert on the derivative of the message volume per sec"
- now emails are not alerts, they're signal.
- Cognitive leap of monitoring tools as signal for state of the system
- one of 5 lines.
- I was carrying a misapprehension about what monitoring was and whom it was for
- "Monitoring is the thing that makes alerts happen" is a fallacy ###
- Always an ad-hoc, and prescriptive in the wrong direction.
- "Monitoring is not FOR alerting". Not about uptime.
- "Nobody OWNS monitoring"
- "Everyone owns monitoring"
- "Everyone can measure things... Everyone who cares, gets to measure"
- Success because expectations change.
- "What if I told you: You might be fascinated with anomaly detection because your input signal sucks"
- Complexity isolates. Those alerts are the sword ops dies on. So don't.
- Rate limiting - Post: There is no data available at this time interval
- Librato limits metrics ingest to save bill of customer.
- Cynefin Chaotic -> Complex -> Complicated -> obvious --->
- Things you need to move:
- Control
- understanding
- standardization
- Things you need to move:
- "I thought complexity was my lot of life" Me dissecting somebody's JS circa 2003.
- Pain of ops isolates ops from larger world.
- Work to reduce complexity... and monitoring helps simplify.
- helps you find the 1-3 metrics you actually care about.
- Person who curated the view of awesome isn't the person who collected the metrics.
- Other people help you build the bridge to understanding
- Teams need to share metrics. Publish them internally.
- "Everyone gets to measure things, no one owns monitoring"
- "I was alone with my pain, despite being surrounded by engineers"
- "Someone else's problem" does not exist
- "Effective Telemetry + people measuring things = better culture"
- They want to comprehend the Y-Axis of your win.