BY OUR POWERS COMBINED: OBSERVABILITY FOR DEVELOPERS @CrayZeigh — #abstractionscon
Slide 1
Slide 2
HI, ABSTRACTIONS!
! $ ” : theelasticast.com : @CrayZeigh : aaron.aldrich@elastic.co : noti.st/crayzeigh
!
Slide 3
OBSERVABILITY @CrayZeigh — #abstractionscon
Slide 4
DEVOPS @CrayZeigh — #abstractionscon
Slide 5
DEVOPS @CrayZeigh — #abstractionscon
Slide 6
@CrayZeigh — #abstractionscon
Slide 7
@CrayZeigh — #abstractionscon
Slide 8
DEVOPS > Wave 1: Ops learns code & automation 1 > Wave 2: Dev owns code through production 1 https://vimeo.com/341142053 @CrayZeigh — #abstractionscon
Slide 9
*Simon Wardley: https://twitter.com/swardley/status/1014883354481741825?lang=en
Slide 10
https://dev.to/molly_struve/making-on-call-not-suck-490
Slide 11
SHARED LANGUAGE @CrayZeigh — #abstractionscon
Slide 12
SHARED TOOLS @CrayZeigh — #abstractionscon
Slide 13
SHARED SOURCE OF TRUTH @CrayZeigh — #abstractionscon
Slide 14
OBSERVABILITY A system is observable when you can ask arbitrary questions about it and receive meaningful answers without having to resort to writing new code or command line tools. It lets you discover unknown-unknowns and debug in production. @CrayZeigh — #abstractionscon
Slide 15
Isn’t it just Monitoring with better SEO? — You @CrayZeigh — #abstractionscon
Slide 16
YOU’RE NOT WRONG.. @CrayZeigh — #abstractionscon
Slide 17
@CrayZeigh — #abstractionscon
Slide 18
@CrayZeigh — #abstractionscon
Slide 19
Software is inherently opaque, we have to instrument it to output meaningful information @CrayZeigh — #abstractionscon
Slide 20
THE THREE PILLARS OF OBSERVABILITY 1. Logs 2. Metrics 3. APM @CrayZeigh — #abstractionscon
Slide 21
THE THREE PILLARS OF OBSERVABILITY 1. Logs Events 2. Metrics 3. APM @CrayZeigh — #abstractionscon
Slide 22
THE THREE PILLARS OF OBSERVABILITY 1. Logs Events 2. Metrics 3. APM 4. Distributed Tracing @CrayZeigh — #abstractionscon
Slide 23
THE THREE PILLARS OF OBSERVABILITY 1. Logs Events 2. Metrics 3. APM & Distributed Tracing 4. Distributed Tracing @CrayZeigh — #abstractionscon
Slide 24
@CrayZeigh — #abstractionscon
Slide 25
LOGS/METRICS/APM ARE THE MEDIA WE WORK IN @CrayZeigh — #abstractionscon
Slide 26
METRICS @CrayZeigh — #abstractionscon
Slide 27
@CrayZeigh — #abstractionscon
Slide 28
EVENTS @CrayZeigh — #abstractionscon
Slide 29
CARDINALITY & YOU @CrayZeigh — #abstractionscon
Slide 30
EXISTING LOGS 66.249.65.159 - - [06/Nov/2014:19:10:38 +0600] “GET /news/53f8d72920ba2744fe873ebc.html HTTP/1.1” 404 177 “-” “Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version /6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)” 66.249.65.3 - - [06/Nov/2014:19:11:24 +0600] “GET /?q=%E0%A6%AB%E0%A6%BE%E0%A7%9F%E0%A6%BE%E0%A6%B0 HTTP/ 1.1” 200 4223 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)” 66.249.65.62 - - [06/Nov/2014:19:12:14 +0600] “GET /?q=%E0%A6%A6%E0%A7%8B%E0%A7%9F%E0%A6%BE HTTP/1.1” 200 4356 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)” @CrayZeigh — #abstractionscon
Slide 31
STRUCTURED LOGGING { “@timestamp”: “2019-08-04T12:30:04.000Z”, … “container”: { “image.id”: “48f5af6667f3457be0a2c7814caefe21ed3c94fb94bd6243096b3a61ea502b1d”, “version”: “version”, }, “build.id”: “efdd0b5e69b0742fa5e5bad0771df4d1df2459d1” … “transaction”: “transaction_ID”, “user”: “importantPerson”, “account”: “0129”, “os”: “osx”, … “api_endpoint”: “endpoint”, … “response”: 400, … “message”: “Some informative thing, probably more human readable and friendly, but difficult to parse” } @CrayZeigh — #abstractionscon
Slide 32
APM & TRACING @CrayZeigh — #abstractionscon
Slide 33
@CrayZeigh — #abstractionscon
Slide 34
@CrayZeigh — #abstractionscon
Slide 35
@CrayZeigh — #abstractionscon
Slide 36
SHARED LANGUAGE > Learning to speak Prod > Teaching Prod to speak Dev (Structured Logs; Traces) @CrayZeigh — #abstractionscon
Slide 37
SHARED TOOLS > Debugging in Prod > Ops skills transferrable and replicable > New knowledge and methods shareable @CrayZeigh — #abstractionscon
Slide 38
SHARED SOURCE OF TRUTH > Real production data > Draw better lines from code to prod > Write better, production ready code. @CrayZeigh — #abstractionscon
Slide 39
WHERE DO WE GO FROM HERE? @CrayZeigh — #abstractionscon
Slide 40
Testing & Experimentation @CrayZeigh — #abstractionscon
Slide 41
TEST IN PRODUCTION @CrayZeigh — #abstractionscon
Slide 42
@CrayZeigh — #abstractionscon
Slide 43
FEATURE FLAGS ! https://martinfowler.com/articles/feature-toggles.html @CrayZeigh — #abstractionscon
Slide 44
config.json { “featureFlags”: { “newThing”: true, } } app.js import { featureFlags } from “./config.json” if (featureFlags.newThing) { // do the new thing! } else { // do the old thing! } @CrayZeigh — #abstractionscon
Slide 45
DON’T DEBATE EXPERIMENT @CrayZeigh — #abstractionscon
Slide 46
2 Speaking of Testing : 2 QA https://theelasticast.com/episodes/0017-qa/ @CrayZeigh — #abstractionscon
Slide 47
Slide 48
Chaos Engineering is about refining our mental models @CrayZeigh — #abstractionscon
Slide 49
DEVOPS > Wave 1: Ops learns code & automation 1 > Wave 2: Dev owns code through production 1 https://vimeo.com/341142053 @CrayZeigh — #abstractionscon
Slide 50
DEV OWNS CODE THROUGH PRODUCTION > Better, more production-ready code > Real World experimentation > Improved operational resiliency @CrayZeigh — #abstractionscon
Slide 51
THE POWER IS YOURS
Slide 52
THANKS! > Slides & References: noti.st/crayzeigh > Trial: ela.st/aaron-aldrich-trial > Come say, “Hi!” at the Elastic booth (31) @CrayZeigh — #abstractionscon
