Heresy and Evangelism: Schism in the Church of Data
A presentation at DevOpsDays Nashville 2019 in in Nashville, TN, USA by fen aldrich
Heresy & Evangelism Schism in the church of monitoring (@elastic) - Aaron Aldrich (@crayzeigh) 1
Hi! ! 4 Community @ 4 Reach me here: 4 aaron.aldrich@elastic.co 4 @CrayZeigh 4 Slides are here: 4 noti.st/crayzeigh 4 This picture is amazing, come @ me. 2
A word from our sponsor (@elastic) - Aaron Aldrich (@crayzeigh) 3
4 We make: 4 Elasticsearch 4 Logstash 4 Kibana 4 Beats 4 Elastic APM (open tracing, ooo) 4 We host: 4 Elastic Search Service 4 Site Search 4 App Search 4 You can run it all where ever 4 Open Source 4 We’re hiring (Fully Distributed, oooh, aaah) 4 Talk to me later (@elastic) - Aaron Aldrich (@crayzeigh) 4
Let’s find out where we’re at. (@elastic) - Aaron Aldrich (@crayzeigh) 5
How many of you deal with monitoring as a job function? (@elastic) - Aaron Aldrich (@crayzeigh) 6
How many of you touch monitoring in some way? (@elastic) - Aaron Aldrich (@crayzeigh) 7
Uptime Performance/Resource Utilization Response time? (@elastic) - Aaron Aldrich (@crayzeigh) 8
Why? (@elastic) - Aaron Aldrich (@crayzeigh) 9
Things Fall Apart * * something about a slouching beast (@elastic) - Aaron Aldrich (@crayzeigh) 10
Incidents Suck (@elastic) - Aaron Aldrich (@crayzeigh) 11
Locus of Control (@elastic) - Aaron Aldrich (@crayzeigh) 12
(@elastic) - Aaron Aldrich (@crayzeigh) 13
(@elastic) - Aaron Aldrich (@crayzeigh) 14
(@elastic) - Aaron Aldrich (@crayzeigh) 15
(@elastic) - Aaron Aldrich (@crayzeigh) 16
100% (@elastic) - Aaron Aldrich (@crayzeigh) 17
99.999% (@elastic) - Aaron Aldrich (@crayzeigh) 18
Just a minute! (@elastic) - Aaron Aldrich (@crayzeigh) 19
Eine Minute, bitte! ! Stolen Joke, if you know where it’s from we’re probably friends (@elastic) - Aaron Aldrich (@crayzeigh) 20
NINES don’t matter… (@elastic) - Aaron Aldrich (@crayzeigh) 21
(@elastic) - Aaron Aldrich (@crayzeigh) 22
NINES don’t matter when USERS aren’t HAPPY ~ Charity Majors (@mipsytipsy) (@elastic) - Aaron Aldrich (@crayzeigh) 23
She doesn’t care whether or not [the datacenter is literally on fire], just as long as the ship’s coming in. ! Cake - Italian Leather Sofa [Lightly Interpreted] (@elastic) - Aaron Aldrich (@crayzeigh) 24
How does your business make money? (@elastic) - Aaron Aldrich (@crayzeigh) 25
How do you help? (@elastic) - Aaron Aldrich (@crayzeigh) 26
DevOps is about delivering Value (@elastic) - Aaron Aldrich (@crayzeigh) 27
(@elastic) - Aaron Aldrich (@crayzeigh) 28
(@elastic) - Aaron Aldrich (@crayzeigh) 29
Observability (@elastic) - Aaron Aldrich (@crayzeigh) 30
Isn’t it just monitoring with better SEO? - You (@elastic) - Aaron Aldrich (@crayzeigh) 31
You’re not wrong… (@elastic) - Aaron Aldrich (@crayzeigh) 32
(@elastic) - Aaron Aldrich (@crayzeigh) 33
Traditional Architecture 4 Predictable 4 Obvious relationships 4 able to be easily modeled 4 System Health is an accurate predictor of user experience 4 Dashboards are useful and valuable (@elastic) - Aaron Aldrich (@crayzeigh) 34
(@elastic) - Aaron Aldrich (@crayzeigh) 35
Complex Systems 4 Always changing 4 Difficult or impossible to model 4 emergent behavior (unknown-unknowns) 4 non-linear relationships 4 feedback loops 4 can adapt and have memory 4 can be nested 4 System health and user experience are no longer directly related (@elastic) - Aaron Aldrich (@crayzeigh) 36
Root Cause is a myth (@elastic) - Aaron Aldrich (@crayzeigh) 37
(@elastic) - Aaron Aldrich (@crayzeigh) 38
One-in-a-million chances crop up nine times out of ten ~ Sir Terry Pratchett ! “Pterry” for short, which gives me joy (@elastic) - Aaron Aldrich (@crayzeigh) 39
SRE (@elastic) - Aaron Aldrich (@crayzeigh) 40
(@elastic) - Aaron Aldrich (@crayzeigh) SLI SLO SLA 41
Services not systems (@elastic) - Aaron Aldrich (@crayzeigh) 42
(@elastic) - Aaron Aldrich (@crayzeigh) 43
Site Reliability Engineering 4 (SLI) What is availability? 4 (SLO) How much do we actually need? 4 (SLA) What happens when we’re not meeting this target? (@elastic) - Aaron Aldrich (@crayzeigh) 44
Site Reliability Engineering 4 (SLI) What is availability? 4 (SLO) How much do we actually need? 4 (SLA) What happens when we’re not meeting this target? (@elastic) - Aaron Aldrich (@crayzeigh) 45
Service Level Indicators 4 Is it up? 4 200OK 4 latency 4 percentiles or medians for meaning (@elastic) - Aaron Aldrich (@crayzeigh) 46
Service Level Indicators 4 Is it up? ! 4 200OK 4 latency 4 percentiles or medians for meaning ! Never trust averages, they hide data (@elastic) - Aaron Aldrich (@crayzeigh) 47
Never trust averages, they hide data (@elastic) - Aaron Aldrich (@crayzeigh) 48
The 99th percentile latency of requests received in <300 ms and responded to with a 200 status (@elastic) - Aaron Aldrich (@crayzeigh) 49
Service Level Objectives How much availability do we need? (@elastic) - Aaron Aldrich (@crayzeigh) 50
99% (@elastic) - Aaron Aldrich (@crayzeigh) 51
99.9% (@elastic) - Aaron Aldrich (@crayzeigh) 52
99.99% (@elastic) - Aaron Aldrich (@crayzeigh) 53
99.999% (@elastic) - Aaron Aldrich (@crayzeigh) 54
Each 9 is exponentially more expensive to provide (@elastic) - Aaron Aldrich (@crayzeigh) 55
availability avg per year avg per day 99% 3.65 days 14.4 minutes 99.9% 8.76 hours 1.44 minutes 99.99% 52.56 minutes 8.64 seconds 99.999% 5.25 minutes 863 ms (@elastic) - Aaron Aldrich (@crayzeigh) 56
A good SLO barely keeps users happy (these should be driving your alerts) (@elastic) - Aaron Aldrich (@crayzeigh) 57
Error Budgets (@elastic) - Aaron Aldrich (@crayzeigh) 58
It’s GOOD to have errors (@elastic) - Aaron Aldrich (@crayzeigh) 59
(@elastic) - Aaron Aldrich (@crayzeigh) 60
Error Budgets Bring Balance to the Force (@elastic) - Aaron Aldrich (@crayzeigh) 61
SLAs = (@elastic) - Aaron Aldrich (@crayzeigh) 62
SLAs = (@elastic) - Aaron Aldrich (@crayzeigh) 63
What about the fire? (@elastic) - Aaron Aldrich (@crayzeigh) 64
(@elastic) - Aaron Aldrich (@crayzeigh) 65
(@elastic) - Aaron Aldrich (@crayzeigh) 66
(@elastic) - Aaron Aldrich (@crayzeigh) 67
(@elastic) - Aaron Aldrich (@crayzeigh) 68
(@elastic) - Aaron Aldrich (@crayzeigh) 69
Observability A system is observable when you can ask arbitrary questions about it and receive meaningful answers without having to resort to writing new code or command line tools. It lets you discover unknown-unknowns and debug in production. (@elastic) - Aaron Aldrich (@crayzeigh) 70
Three Pillars of Observability 4 Metrics 4 Logs 4 APM (@elastic) - Aaron Aldrich (@crayzeigh) 71
These aren’t pillars. (@elastic) - Aaron Aldrich (@crayzeigh) 72
(@elastic) - Aaron Aldrich (@crayzeigh) 73
Three Pillars of Carpentry? stahp. (@elastic) - Aaron Aldrich (@crayzeigh) 74
They’re tools, not pillars You need to know how to use them (@elastic) - Aaron Aldrich (@crayzeigh) 75
Metrics 4 Great, not on their own ! 4 largely contextless 4 need further notation to be valuable (tags) 4 Easy to store lots of them 4 collection can be a pain ! Check out Open Metrics! openmetrics.io (@elastic) - Aaron Aldrich (@crayzeigh) 76
High Cardinality Data 4 UUIDs 4 raw queries 4 comments 4 firstname, lastname 4 PID/PPID 4 app ID 4 device ID 4 build ID 4 IP:port 4 shopping cart ID 4 userid (@elastic) - Aaron Aldrich (@crayzeigh) 77
What’s better at carrying Cardinality? (@elastic) - Aaron Aldrich (@crayzeigh) 78
Events! (@elastic) - Aaron Aldrich (@crayzeigh) 79
(Logs) (@elastic) - Aaron Aldrich (@crayzeigh) 80
But please not these: 64.242.88.10 - - [07/Mar/2004:16:05:49 -0800] “GET /twiki/bin/edit/Main/Double_bounce_sender?topicparent=Main.ConfigurationVariables HTTP/1.1” 401 12846 64.242.88.10 - - [07/Mar/2004:16:06:51 -0800] “GET /twiki/bin/rdiff/TWiki/NewUserTemplate?rev1=1.3&rev2=1.2 HTTP/1.1” 200 4523 64.242.88.10 - - [07/Mar/2004:16:10:02 -0800] “GET /mailman/listinfo/hsdivision HTTP/1.1” 200 6291 64.242.88.10 - - [07/Mar/2004:16:11:58 -0800] “GET /twiki/bin/view/TWiki/WikiSyntax HTTP/1.1” 200 7352 64.242.88.10 - - [07/Mar/2004:16:20:55 -0800] “GET /twiki/bin/view/Main/DCCAndPostFix HTTP/1.1” 200 5253 64.242.88.10 - - [07/Mar/2004:16:23:12 -0800] “GET /twiki/bin/oops/TWiki/AppendixFileSystem?template=oopsmore¶m1=1.12¶m2=1.12 HTTP/1.1” 200 11382 64.242.88.10 - - [07/Mar/2004:16:24:16 -0800] “GET /twiki/bin/view/Main/PeterThoeny HTTP/1.1” 200 4924 64.242.88.10 - - [07/Mar/2004:16:29:16 -0800] “GET /twiki/bin/edit/Main/Header_checks?topicparent=Main.ConfigurationVariables HTTP/1.1” 401 12851 64.242.88.10 - - [07/Mar/2004:16:30:29 -0800] “GET /twiki/bin/attach/Main/OfficeLocations HTTP/1.1” 401 12851 64.242.88.10 - - [07/Mar/2004:16:31:48 -0800] “GET /twiki/bin/view/TWiki/WebTopicEditTemplate HTTP/1.1” 200 3732 64.242.88.10 - - [07/Mar/2004:16:32:50 -0800] “GET /twiki/bin/view/Main/WebChanges HTTP/1.1” 200 40520 64.242.88.10 - - [07/Mar/2004:16:33:53 -0800] “GET /twiki/bin/edit/Main/Smtpd_etrn_restrictions?topicparent=Main.ConfigurationVariables HTTP/1.1” 401 12851 64.242.88.10 - - [07/Mar/2004:16:35:19 -0800] “GET /mailman/listinfo/business HTTP/1.1” 200 6379 64.242.88.10 - - [07/Mar/2004:16:36:22 -0800] “GET /twiki/bin/rdiff/Main/WebIndex?rev1=1.2&rev2=1.1 HTTP/1.1” 200 46373 64.242.88.10 - - [07/Mar/2004:16:37:27 -0800] “GET /twiki/bin/view/TWiki/DontNotify HTTP/1.1” 200 4140 64.242.88.10 - - [07/Mar/2004:16:39:24 -0800] “GET /twiki/bin/view/Main/TokyoOffice HTTP/1.1” 200 3853 64.242.88.10 - - [07/Mar/2004:16:43:54 -0800] “GET /twiki/bin/view/Main/MikeMannix HTTP/1.1” 200 3686 64.242.88.10 - - [07/Mar/2004:16:45:56 -0800] “GET /twiki/bin/attach/Main/PostfixCommands HTTP/1.1” 401 12846 64.242.88.10 - - [07/Mar/2004:16:47:12 -0800] “GET /robots.txt HTTP/1.1” 200 68 64.242.88.10 - - [07/Mar/2004:16:47:46 -0800] “GET /twiki/bin/rdiff/Know/ReadmeFirst?rev1=1.5&rev2=1.4 HTTP/1.1” 200 5724 64.242.88.10 - - [07/Mar/2004:16:49:04 -0800] “GET /twiki/bin/view/Main/TWikiGroups?rev=1.2 HTTP/1.1” 200 5162 64.242.88.10 - - [07/Mar/2004:16:50:54 -0800] “GET /twiki/bin/rdiff/Main/ConfigurationVariables HTTP/1.1” 200 59679 64.242.88.10 - - [07/Mar/2004:16:52:35 -0800] “GET /twiki/bin/edit/Main/Flush_service_name?topicparent=Main.ConfigurationVariables HTTP/1.1” 401 12851 64.242.88.10 - - [07/Mar/2004:16:53:46 -0800] “GET /twiki/bin/rdiff/TWiki/TWikiRegistration HTTP/1.1” 200 34395 64.242.88.10 - - [07/Mar/2004:16:54:55 -0800] “GET /twiki/bin/rdiff/Main/NicholasLee HTTP/1.1” 200 7235 64.242.88.10 - - [07/Mar/2004:16:56:39 -0800] “GET /twiki/bin/view/Sandbox/WebHome?rev=1.6 HTTP/1.1” 200 8545 64.242.88.10 - - [07/Mar/2004:16:58:54 -0800] “GET /mailman/listinfo/administration HTTP/1.1” 200 6459 lordgun.org - - [07/Mar/2004:17:01:53 -0800] “GET /razor.html HTTP/1.1” 200 2869 64.242.88.10 - - [07/Mar/2004:17:09:01 -0800] “GET /twiki/bin/search/Main/SearchResult?scope=text®ex=on&search=Joris%20*Benschop[^A-Za-z] HTTP/1.1” 200 4284 (@elastic) - Aaron Aldrich (@crayzeigh) 81
Structured Data ! { “message”:”user_deleted”, “user”: { “id”:6, “email”:”crayzeigh@example.com”, “created_at”:”2015-12-11T04:31:46.828Z”, “updated_at”:”2015-12-11T04:32:18.340Z”, “name”:”crayzeigh”, “role”:”user”, “invitation_token”:null, “invitation_created_at”:null, “invitation_sent_at”:null, “invitation_accepted_at”:null, “invitation_limit”:null, “invited_by_id”:null, “invited_by_type”:null, “invitations_count”:0 }, “@timestamp”:”2015-12-11T13:35:50.070+00:00”, “@version”:”1”, “severity”:”INFO”, “host”:”app1-web1”, “type”:”apps” } ! from James Turnbull: https://www.kartar.net/2015/12/structured-logging/ (@elastic) - Aaron Aldrich (@crayzeigh) 82
Generate LOTS of events use sampling to store them (@elastic) - Aaron Aldrich (@crayzeigh) 83
OK let’s talk about APM (@elastic) - Aaron Aldrich (@crayzeigh) 84
Distributed Tracing ! Check out Open tracing fron CNCF: opentracing.io (@elastic) - Aaron Aldrich (@crayzeigh) 85
Instrumentation: SLIs are a good place to start (@elastic) - Aaron Aldrich (@crayzeigh) 86
Kill Staging: Test in Production (@elastic) - Aaron Aldrich (@crayzeigh) 87
(@elastic) - Aaron Aldrich (@crayzeigh) 88
This doesn’t eliminate QA or testing (please test before prod) (@elastic) - Aaron Aldrich (@crayzeigh) 89
Kill your staging environment 4 always out of sync 4 can’t replicate prod traffic anyway 4 definitely can’t replicate real users 4 replace with feature flags and canary deploys ! Launch Darkly talks about this a lot. You should listen to what they have to say. (@elastic) - Aaron Aldrich (@crayzeigh) 90
O11y ❤ ‘s QA Start leveraging a common toolset (@elastic) - Aaron Aldrich (@crayzeigh) 91
Every Dashboard sucks (@elastic) - Aaron Aldrich (@crayzeigh) 92
(@elastic) - Aaron Aldrich (@crayzeigh) 93
Not really, some dashboards are pretty good (@elastic) - Aaron Aldrich (@crayzeigh) 94
(@elastic) - Aaron Aldrich (@crayzeigh) 95
It’s about Storytelling know your audience (@elastic) - Aaron Aldrich (@crayzeigh) 96
Ops & Incident Response 4 Interactive 4 Iterative 4 Involve search bars (@elastic) - Aaron Aldrich (@crayzeigh) 97
Vendor Warning: Search & Common Data Schema (@elastic) - Aaron Aldrich (@crayzeigh) 98
Making O11y Evangelists (@elastic) - Aaron Aldrich (@crayzeigh) 99
Don’t just start making changes (@elastic) - Aaron Aldrich (@crayzeigh) 100
(@elastic) - Aaron Aldrich (@crayzeigh) 101
History is important (@elastic) - Aaron Aldrich (@crayzeigh) 102
Change conducted poorly breaks organizations (@elastic) - Aaron Aldrich (@crayzeigh) 103
top-down mandated change never works ☠ Did you know “defenestration” is the act of throwing someone out a window? (@elastic) - Aaron Aldrich (@crayzeigh) 104
Talk to other parts of the business to understand what stories they value (@elastic) - Aaron Aldrich (@crayzeigh) 105
LISTEN It’s all about context (@elastic) - Aaron Aldrich (@crayzeigh) 106
Start measuring business values (@elastic) - Aaron Aldrich (@crayzeigh) 107
Who else might care about dashboards? (@elastic) - Aaron Aldrich (@crayzeigh) 108
What data can we expose to the rest of the business? (@elastic) - Aaron Aldrich (@crayzeigh) 109
110
111
112
Dashboards help tell stories with context (@elastic) - Aaron Aldrich (@crayzeigh) 113
Share results Good and Bad (@elastic) - Aaron Aldrich (@crayzeigh) 114
Are your systems up? Are they responding acceptably? (@elastic) - Aaron Aldrich (@crayzeigh) 115
Who cares? (@elastic) - Aaron Aldrich (@crayzeigh) 116
(@elastic) - Aaron Aldrich (@crayzeigh) 117
Are your services delivering value? (@elastic) - Aaron Aldrich (@crayzeigh) 118
Monitoring and the Church of 9s has a new competing ideology, Observability and the SLO. Without taking a full 99 theses to explore the differences, this talk will explore the differences separating them and build new o11y evangelists.
Resources
The following resources were mentioned during the presentation or are useful additional information.
-
Open Metrics
Landing page for CNCFs Open Metrics project
-
Open Tracing
Landing page for the Open Tracing project.
Buzz and feedback
Here’s what was said about this presentation on social media.
