Heresy and Evangelism: Schism in the Church of Data

A presentation at DevOpsDays Houston in April 2019 in Houston, TX, USA by Aaron Aldrich

Slide 1

Slide 1

Heresy & Evangelism Schism in the church of monitoring (@elastic) - Aaron Aldrich (@crayzeigh) 1

Slide 2

Slide 2

Hi! ! • • Community Advocate • aaron.aldrich@elastic.co • @CrayZeigh • noti.st/crayzeigh (slides will be here) 2

Slide 3

Slide 3

A word from our sponsor (@elastic) - Aaron Aldrich (@crayzeigh) 3

Slide 4

Slide 4

• We make: • Elasticsearch • Logstash • Kibana • Beats • Elastic APM (open tracing, ooo) • We host: • Elastic Search Service • Site Search • App Search • You can run it all where ever • Core is Free and/or free • We’re hiring (Fully Distributed, oooh, aaah) • Talk to us at the booth (@elastic) - Aaron Aldrich (@crayzeigh) 4

Slide 5

Slide 5

Let’s find out where we’re at. (@elastic) - Aaron Aldrich (@crayzeigh) 5

Slide 6

Slide 6

How many of you deal with monitoring as a job function? (@elastic) - Aaron Aldrich (@crayzeigh) 6

Slide 7

Slide 7

How many of you touch monitoring in some way? (@elastic) - Aaron Aldrich (@crayzeigh) 7

Slide 8

Slide 8

Uptime Performance/ Resource Utilization Response time? (@elastic) - Aaron Aldrich (@crayzeigh) 8

Slide 9

Slide 9

Why? (@elastic) - Aaron Aldrich (@crayzeigh) 9

Slide 10

Slide 10

Things Fall Apart * something about a slouching beast (@elastic) - Aaron Aldrich (@crayzeigh) 10

Slide 11

Slide 11

Incidents Suck (@elastic) - Aaron Aldrich (@crayzeigh) 11

Slide 12

Slide 12

(@elastic) - Aaron Aldrich (@crayzeigh) 12

Slide 13

Slide 13

(@elastic) - Aaron Aldrich (@crayzeigh) 13

Slide 14

Slide 14

(@elastic) - Aaron Aldrich (@crayzeigh) 14

Slide 15

Slide 15

(@elastic) - Aaron Aldrich (@crayzeigh) 15

Slide 16

Slide 16

(@elastic) - Aaron Aldrich (@crayzeigh) 16

Slide 17

Slide 17

100% (@elastic) - Aaron Aldrich (@crayzeigh) 17

Slide 18

Slide 18

99.999% (@elastic) - Aaron Aldrich (@crayzeigh) 18

Slide 19

Slide 19

Just a minute! (@elastic) - Aaron Aldrich (@crayzeigh) 19

Slide 20

Slide 20

Eine Minute, bitte! ! Stolen Joke, if you know where it’s from we’re probably friends (@elastic) - Aaron Aldrich (@crayzeigh) 20

Slide 21

Slide 21

NINES don’t matter… ~ Charity Majors (@mipsytipsy) (@elastic) - Aaron Aldrich (@crayzeigh) 21

Slide 22

Slide 22

(@elastic) - Aaron Aldrich (@crayzeigh) 22

Slide 23

Slide 23

NINES don’t matter when USERS aren’t HAPPY ~ Charity Majors (@mipsytipsy) (@elastic) - Aaron Aldrich (@crayzeigh) 23

Slide 24

Slide 24

She doesn’t care whether or not [the datacenter is literally on fire], just as long as the ship’s coming in. (@elastic) - Aaron Aldrich (@crayzeigh) 24

Slide 25

Slide 25

How does your business make money? (@elastic) - Aaron Aldrich (@crayzeigh) 25

Slide 26

Slide 26

How do you help? (@elastic) - Aaron Aldrich (@crayzeigh) 26

Slide 27

Slide 27

It is not enough to do your best; you must know what to do, and then do your best. - W. Edwards Deming (@elastic) - Aaron Aldrich (@crayzeigh) 27

Slide 28

Slide 28

DevOps is about delivering Value (@elastic) - Aaron Aldrich (@crayzeigh) 28

Slide 29

Slide 29

(@elastic) - Aaron Aldrich (@crayzeigh) 29

Slide 30

Slide 30

SRE (@elastic) - Aaron Aldrich (@crayzeigh) 30

Slide 31

Slide 31

(@elastic) - Aaron Aldrich (@crayzeigh) SLI SLO SLA 31

Slide 32

Slide 32

Services not systems (@elastic) - Aaron Aldrich (@crayzeigh) 32

Slide 33

Slide 33

(@elastic) - Aaron Aldrich (@crayzeigh) 33

Slide 34

Slide 34

Site Reliability Engineering • (SLI) What is availability? • (SLO) How much do we actually need? • (SLA) What happens when we’re not meeting this target? (@elastic) - Aaron Aldrich (@crayzeigh) 34

Slide 35

Slide 35

Site Reliability Engineering • (SLI) What is availability? • (SLO) How much do we actually need? • (SLA) What happens when we’re not meeting this target? (@elastic) - Aaron Aldrich (@crayzeigh) 35

Slide 36

Slide 36

Service Level Indicators • Is it up? • 200OK • latency • percentiles or medians for meaning (@elastic) - Aaron Aldrich (@crayzeigh) 36

Slide 37

Slide 37

Service Level Indicators • Is it up? ! • 200OK • latency • percentiles or medians for meaning ! Never trust averages, they hide data (@elastic) - Aaron Aldrich (@crayzeigh) 37

Slide 38

Slide 38

twitter slide: Never trust averages, they hide data (@elastic) - Aaron Aldrich (@crayzeigh) 38

Slide 39

Slide 39

The 99th percentile latency of requests received in the last five minutes <300 ms and responded to with a 200 status (@elastic) - Aaron Aldrich (@crayzeigh) 39

Slide 40

Slide 40

Service Level Objectives How much availability do we need? (@elastic) - Aaron Aldrich (@crayzeigh) 40

Slide 41

Slide 41

99% (@elastic) - Aaron Aldrich (@crayzeigh) 41

Slide 42

Slide 42

99.9% (@elastic) - Aaron Aldrich (@crayzeigh) 42

Slide 43

Slide 43

99.99% (@elastic) - Aaron Aldrich (@crayzeigh) 43

Slide 44

Slide 44

99.999% (@elastic) - Aaron Aldrich (@crayzeigh) 44

Slide 45

Slide 45

Each 9 is exponentially more expensive to provide (@elastic) - Aaron Aldrich (@crayzeigh) 45

Slide 46

Slide 46

availability avg per year avg per day 99% 3.65 days 14.4 minutes 99.9% 8.76 hours 1.44 minutes 99.99% 52.56 minutes 8.64 seconds 99.999% 5.25 minutes 863 ms (@elastic) - Aaron Aldrich (@crayzeigh) 46

Slide 47

Slide 47

A good SLO barely keeps users happy (these should be driving your alerts) (@elastic) - Aaron Aldrich (@crayzeigh) 47

Slide 48

Slide 48

Error Budgets (@elastic) - Aaron Aldrich (@crayzeigh) 48

Slide 49

Slide 49

It’s GOOD to have errors (@elastic) - Aaron Aldrich (@crayzeigh) 49

Slide 50

Slide 50

(@elastic) - Aaron Aldrich (@crayzeigh) 50

Slide 51

Slide 51

(@elastic) - Aaron Aldrich (@crayzeigh) 51

Slide 52

Slide 52

SLAs = (@elastic) - Aaron Aldrich (@crayzeigh) 52

Slide 53

Slide 53

SLAs = (@elastic) - Aaron Aldrich (@crayzeigh) 53

Slide 54

Slide 54

What about the fire? (@elastic) - Aaron Aldrich (@crayzeigh) 54

Slide 55

Slide 55

(@elastic) - Aaron Aldrich (@crayzeigh) 55

Slide 56

Slide 56

(@elastic) - Aaron Aldrich (@crayzeigh) 56

Slide 57

Slide 57

(@elastic) - Aaron Aldrich (@crayzeigh) 57

Slide 58

Slide 58

(@elastic) - Aaron Aldrich (@crayzeigh) 58

Slide 59

Slide 59

(@elastic) - Aaron Aldrich (@crayzeigh) 59

Slide 60

Slide 60

Observability (@elastic) - Aaron Aldrich (@crayzeigh) 60

Slide 61

Slide 61

O11y (@elastic) - Aaron Aldrich (@crayzeigh) 61

Slide 62

Slide 62

Isn’t it just monitoring with another name? (@elastic) - Aaron Aldrich (@crayzeigh) 62

Slide 63

Slide 63

no. (@elastic) - Aaron Aldrich (@crayzeigh) 63

Slide 64

Slide 64

Observability A system is observable when you can ask arbitrary questions about it and receive meaningful answers without having to resort to writing new code or command line tools. It lets you discover unknown-unknowns and debug in production. (@elastic) - Aaron Aldrich (@crayzeigh) 64

Slide 65

Slide 65

our tools must change with our systems. (@elastic) - Aaron Aldrich (@crayzeigh) 65

Slide 66

Slide 66

(@elastic) - Aaron Aldrich (@crayzeigh) 66

Slide 67

Slide 67

Traditional Architecture • Predictable • Obvious relationships • able to be easily modeled • System Health is an accurate predictor of user experience • Dashboards are useful and valuable • known-unknowns cover most issues (@elastic) - Aaron Aldrich (@crayzeigh) 67

Slide 68

Slide 68

(@elastic) - Aaron Aldrich (@crayzeigh) 68

Slide 69

Slide 69

Complex Systems • Always changing • Difficult or impossible to model • emergent behavior (unknown-unknowns) • non-linear relationships • feedback loops • can adapt and have memory • can be nested • System health and user experience are no longer directly related (@elastic) - Aaron Aldrich (@crayzeigh) 69

Slide 70

Slide 70

Root Cause is a myth (@elastic) - Aaron Aldrich (@crayzeigh) 70

Slide 71

Slide 71

(@elastic) - Aaron Aldrich (@crayzeigh) 71

Slide 72

Slide 72

Three Pillars of Observability • Metrics • Logs • APM (@elastic) - Aaron Aldrich (@crayzeigh) 72

Slide 73

Slide 73

These aren’t pillars. (@elastic) - Aaron Aldrich (@crayzeigh) 73

Slide 74

Slide 74

(@elastic) - Aaron Aldrich (@crayzeigh) 74

Slide 75

Slide 75

Three Pillars of Carpentry? stahp. (@elastic) - Aaron Aldrich (@crayzeigh) 75

Slide 76

Slide 76

They’re tools, not pillars You need to know how to use them (@elastic) - Aaron Aldrich (@crayzeigh) 76

Slide 77

Slide 77

Metrics • Great, not on their own ! • largely contextless • need further notation to be valuable (tags) • Easy to store lots of them • collection can be a pain ! Check out Open Metrics! openmetrics.io (@elastic) - Aaron Aldrich (@crayzeigh) 77

Slide 78

Slide 78

High Cardinality Data • UUIDs • raw queries • comments • firstname, lastname • PID/PPID • app ID • device ID • build ID • IP:port • shopping cart ID • userid (@elastic) - Aaron Aldrich (@crayzeigh) 78

Slide 79

Slide 79

One-in-a-million chances crop up nine times out of ten ~ Terry Pratchett (@elastic) - Aaron Aldrich (@crayzeigh) 79

Slide 80

Slide 80

What’s better at carrying Cardinality? (@elastic) - Aaron Aldrich (@crayzeigh) 80

Slide 81

Slide 81

Events! (@elastic) - Aaron Aldrich (@crayzeigh) 81

Slide 82

Slide 82

(Logs) (@elastic) - Aaron Aldrich (@crayzeigh) 82

Slide 83

Slide 83

But please not these: 64.242.88.10 - - [07/Mar/2004:16:05:49 -0800] “GET /twiki/bin/edit/Main/Double_bounce_sender?topicparent=Main.ConfigurationVariables HTTP/1.1” 401 12846 64.242.88.10 - - [07/Mar/2004:16:06:51 -0800] “GET /twiki/bin/rdiff/TWiki/NewUserTemplate?rev1=1.3&rev2=1.2 HTTP/1.1” 200 4523 64.242.88.10 - - [07/Mar/2004:16:10:02 -0800] “GET /mailman/listinfo/hsdivision HTTP/1.1” 200 6291 64.242.88.10 - - [07/Mar/2004:16:11:58 -0800] “GET /twiki/bin/view/TWiki/WikiSyntax HTTP/1.1” 200 7352 64.242.88.10 - - [07/Mar/2004:16:20:55 -0800] “GET /twiki/bin/view/Main/DCCAndPostFix HTTP/1.1” 200 5253 64.242.88.10 - - [07/Mar/2004:16:23:12 -0800] “GET /twiki/bin/oops/TWiki/AppendixFileSystem?template=oopsmore¶m1=1.12¶m2=1.12 HTTP/1.1” 200 11382 64.242.88.10 - - [07/Mar/2004:16:24:16 -0800] “GET /twiki/bin/view/Main/PeterThoeny HTTP/1.1” 200 4924 64.242.88.10 - - [07/Mar/2004:16:29:16 -0800] “GET /twiki/bin/edit/Main/Header_checks?topicparent=Main.ConfigurationVariables HTTP/1.1” 401 12851 64.242.88.10 - - [07/Mar/2004:16:30:29 -0800] “GET /twiki/bin/attach/Main/OfficeLocations HTTP/1.1” 401 12851 64.242.88.10 - - [07/Mar/2004:16:31:48 -0800] “GET /twiki/bin/view/TWiki/WebTopicEditTemplate HTTP/1.1” 200 3732 64.242.88.10 - - [07/Mar/2004:16:32:50 -0800] “GET /twiki/bin/view/Main/WebChanges HTTP/1.1” 200 40520 64.242.88.10 - - [07/Mar/2004:16:33:53 -0800] “GET /twiki/bin/edit/Main/Smtpd_etrn_restrictions?topicparent=Main.ConfigurationVariables HTTP/1.1” 401 12851 64.242.88.10 - - [07/Mar/2004:16:35:19 -0800] “GET /mailman/listinfo/business HTTP/1.1” 200 6379 64.242.88.10 - - [07/Mar/2004:16:36:22 -0800] “GET /twiki/bin/rdiff/Main/WebIndex?rev1=1.2&rev2=1.1 HTTP/1.1” 200 46373 64.242.88.10 - - [07/Mar/2004:16:37:27 -0800] “GET /twiki/bin/view/TWiki/DontNotify HTTP/1.1” 200 4140 64.242.88.10 - - [07/Mar/2004:16:39:24 -0800] “GET /twiki/bin/view/Main/TokyoOffice HTTP/1.1” 200 3853 64.242.88.10 - - [07/Mar/2004:16:43:54 -0800] “GET /twiki/bin/view/Main/MikeMannix HTTP/1.1” 200 3686 64.242.88.10 - - [07/Mar/2004:16:45:56 -0800] “GET /twiki/bin/attach/Main/PostfixCommands HTTP/1.1” 401 12846 64.242.88.10 - - [07/Mar/2004:16:47:12 -0800] “GET /robots.txt HTTP/1.1” 200 68 64.242.88.10 - - [07/Mar/2004:16:47:46 -0800] “GET /twiki/bin/rdiff/Know/ReadmeFirst?rev1=1.5&rev2=1.4 HTTP/1.1” 200 5724 64.242.88.10 - - [07/Mar/2004:16:49:04 -0800] “GET /twiki/bin/view/Main/TWikiGroups?rev=1.2 HTTP/1.1” 200 5162 64.242.88.10 - - [07/Mar/2004:16:50:54 -0800] “GET /twiki/bin/rdiff/Main/ConfigurationVariables HTTP/1.1” 200 59679 64.242.88.10 - - [07/Mar/2004:16:52:35 -0800] “GET /twiki/bin/edit/Main/Flush_service_name?topicparent=Main.ConfigurationVariables HTTP/1.1” 401 12851 64.242.88.10 - - [07/Mar/2004:16:53:46 -0800] “GET /twiki/bin/rdiff/TWiki/TWikiRegistration HTTP/1.1” 200 34395 64.242.88.10 - - [07/Mar/2004:16:54:55 -0800] “GET /twiki/bin/rdiff/Main/NicholasLee HTTP/1.1” 200 7235 64.242.88.10 - - [07/Mar/2004:16:56:39 -0800] “GET /twiki/bin/view/Sandbox/WebHome?rev=1.6 HTTP/1.1” 200 8545 64.242.88.10 - - [07/Mar/2004:16:58:54 -0800] “GET /mailman/listinfo/administration HTTP/1.1” 200 6459 lordgun.org - - [07/Mar/2004:17:01:53 -0800] “GET /razor.html HTTP/1.1” 200 2869 64.242.88.10 - - [07/Mar/2004:17:09:01 -0800] “GET /twiki/bin/search/Main/SearchResult?scope=text®ex=on&search=Joris%20*Benschop[^A-Za-z] HTTP/1.1” 200 4284 (@elastic) - Aaron Aldrich (@crayzeigh) 83

Slide 84

Slide 84

Structured Data ! { “message”:”user_deleted”, “user”: { “id”:6, “email”:”crayzeigh@example.com”, “created_at”:”2015-12-11T04:31:46.828Z”, “updated_at”:”2015-12-11T04:32:18.340Z”, “name”:”crayzeigh”, “role”:”user”, “invitation_token”:null, “invitation_created_at”:null, “invitation_sent_at”:null, “invitation_accepted_at”:null, “invitation_limit”:null, “invited_by_id”:null, “invited_by_type”:null, “invitations_count”:0 }, “@timestamp”:”2015-12-11T13:35:50.070+00:00”, “@version”:”1”, “severity”:”INFO”, “host”:”app1-web1”, “type”:”apps” } ! from James Turnbull: https://www.kartar.net/2015/12/structured-logging/ (@elastic) - Aaron Aldrich (@crayzeigh) 84

Slide 85

Slide 85

Generate LOTS of events use sampling to store them (@elastic) - Aaron Aldrich (@crayzeigh) 85

Slide 86

Slide 86

OK let’s talk about APM (@elastic) - Aaron Aldrich (@crayzeigh) 86

Slide 87

Slide 87

Distributed Tracing ! Check out Open tracing fron CNCF: opentracing.io (@elastic) - Aaron Aldrich (@crayzeigh) 87

Slide 88

Slide 88

Instrumentation: SLIs are a good place to start (@elastic) - Aaron Aldrich (@crayzeigh) 88

Slide 89

Slide 89

Kill Staging: Test in Production (@elastic) - Aaron Aldrich (@crayzeigh) 89

Slide 90

Slide 90

(@elastic) - Aaron Aldrich (@crayzeigh) 90

Slide 91

Slide 91

This doesn’t eliminate QA or testing (please test before prod) (@elastic) - Aaron Aldrich (@crayzeigh) 91

Slide 92

Slide 92

Kill your staging environment • always out of sync • can’t replicate prod traffic anyway • definitely can’t replicate real users • replace with feature flags and canary deploys ! Launch Darkly talks about this a lot. You should listen to what they have to say. (@elastic) - Aaron Aldrich (@crayzeigh) 92

Slide 93

Slide 93

O11y ❤ ‘s QA Start leveraging a common toolset (@elastic) - Aaron Aldrich (@crayzeigh) 93

Slide 94

Slide 94

Every Dashboard sucks (@elastic) - Aaron Aldrich (@crayzeigh) 94

Slide 95

Slide 95

(@elastic) - Aaron Aldrich (@crayzeigh) 95

Slide 96

Slide 96

Not really, some dashboards are pretty good (@elastic) - Aaron Aldrich (@crayzeigh) 96

Slide 97

Slide 97

(@elastic) - Aaron Aldrich (@crayzeigh) 97

Slide 98

Slide 98

It’s about Storytelling know your audience (@elastic) - Aaron Aldrich (@crayzeigh) 98

Slide 99

Slide 99

Ops & Incident Response • Interactive • Iterative • Involve search bars (@elastic) - Aaron Aldrich (@crayzeigh) 99

Slide 100

Slide 100

Vendor Warning: Search & Common Data Schema (@elastic) - Aaron Aldrich (@crayzeigh) 100

Slide 101

Slide 101

Making O11y Evangelists (@elastic) - Aaron Aldrich (@crayzeigh) 101

Slide 102

Slide 102

Don’t just start making changes (@elastic) - Aaron Aldrich (@crayzeigh) 102

Slide 103

Slide 103

(@elastic) - Aaron Aldrich (@crayzeigh) 103

Slide 104

Slide 104

History is important (@elastic) - Aaron Aldrich (@crayzeigh) 104

Slide 105

Slide 105

Change conducted poorly breaks organizations (@elastic) - Aaron Aldrich (@crayzeigh) 105

Slide 106

Slide 106

top-down mandated change never works ☠ Did you know “defenestration” is the act of throwing someone out a window? (@elastic) - Aaron Aldrich (@crayzeigh) 106

Slide 107

Slide 107

Talk to other parts of the business to understand what stories they value (@elastic) - Aaron Aldrich (@crayzeigh) 107

Slide 108

Slide 108

LISTEN It’s all about context (@elastic) - Aaron Aldrich (@crayzeigh) 108

Slide 109

Slide 109

Start measuring business values (@elastic) - Aaron Aldrich (@crayzeigh) 109

Slide 110

Slide 110

Who else might care about dashboards? (@elastic) - Aaron Aldrich (@crayzeigh) 110

Slide 111

Slide 111

What data can we expose to the rest of the business? (@elastic) - Aaron Aldrich (@crayzeigh) 111

Slide 112

Slide 112

112

Slide 113

Slide 113

113

Slide 114

Slide 114

114

Slide 115

Slide 115

Dashboards help tell stories with context (@elastic) - Aaron Aldrich (@crayzeigh) 115

Slide 116

Slide 116

Share results Good and Bad (@elastic) - Aaron Aldrich (@crayzeigh) 116

Slide 117

Slide 117

Are your systems up? Are they responding acceptably? (@elastic) - Aaron Aldrich (@crayzeigh) 117

Slide 118

Slide 118

Who cares? (@elastic) - Aaron Aldrich (@crayzeigh) 118

Slide 119

Slide 119

(@elastic) - Aaron Aldrich (@crayzeigh) 119

Slide 120

Slide 120

Are your services delivering value? (@elastic) - Aaron Aldrich (@crayzeigh) 120