By Our Powers Combined: Observability for Developers
A presentation at Abstractions II in in Pittsburgh, PA, USA by fen aldrich
BY OUR POWERS COMBINED: OBSERVABILITY FOR DEVELOPERS @CrayZeigh — #Abstractions
HI, ABSTRACTIONS!
! $ ” : theelasticast.com : @CrayZeigh : aaron.aldrich@elastic.co : noti.st/crayzeigh
!
OBSERVABILITY @CrayZeigh — #Abstractions
DEVOPS @CrayZeigh — #Abstractions
DEVOPS @CrayZeigh — #Abstractions
@CrayZeigh — #Abstractions
@CrayZeigh — #Abstractions
DEVOPS > Wave 1: Ops learns code & automation 1 > Wave 2: Dev owns code through production 1 https://vimeo.com/341142053 @CrayZeigh — #Abstractions
*Simon Wardley: https://twitter.com/swardley/status/1014883354481741825?lang=en
https://dev.to/molly_struve/making-on-call-not-suck-490
SHARED LANGUAGE @CrayZeigh — #Abstractions
SHARED TOOLS @CrayZeigh — #Abstractions
SHARED SOURCE OF TRUTH @CrayZeigh — #Abstractions
OBSERVABILITY A system is observable when you can ask arbitrary questions about it and receive meaningful answers without having to resort to writing new code or command line tools. It lets you discover unknown-unknowns and debug in production. @CrayZeigh — #Abstractions
Isn’t it just Monitoring with better SEO? — You @CrayZeigh — #Abstractions
YOU’RE NOT WRONG.. @CrayZeigh — #Abstractions
@CrayZeigh — #Abstractions
@CrayZeigh — #Abstractions
Software is inherently opaque, we have to instrument it to output meaningful information @CrayZeigh — #Abstractions
THE THREE PILLARS OF OBSERVABILITY 1. Logs 2. Metrics 3. APM @CrayZeigh — #Abstractions
THE THREE PILLARS OF OBSERVABILITY 1. Logs Events 2. Metrics 3. APM @CrayZeigh — #Abstractions
THE THREE PILLARS OF OBSERVABILITY 1. Logs Events 2. Metrics 3. APM 4. Distributed Tracing @CrayZeigh — #Abstractions
THE THREE PILLARS OF OBSERVABILITY 1. Logs Events 2. Metrics 3. APM & Distributed Tracing 4. Distributed Tracing @CrayZeigh — #Abstractions
@CrayZeigh — #Abstractions
LOGS/METRICS/APM ARE THE MEDIA WE WORK IN @CrayZeigh — #Abstractions
METRICS @CrayZeigh — #Abstractions
@CrayZeigh — #Abstractions
EVENTS @CrayZeigh — #Abstractions
CARDINALITY & YOU @CrayZeigh — #Abstractions
EXISTING LOGS 66.249.65.159 - - [06/Nov/2014:19:10:38 +0600] “GET /news/53f8d72920ba2744fe873ebc.html HTTP/1.1” 404 177 “-” “Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version /6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)” 66.249.65.3 - - [06/Nov/2014:19:11:24 +0600] “GET /?q=%E0%A6%AB%E0%A6%BE%E0%A7%9F%E0%A6%BE%E0%A6%B0 HTTP/ 1.1” 200 4223 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)” 66.249.65.62 - - [06/Nov/2014:19:12:14 +0600] “GET /?q=%E0%A6%A6%E0%A7%8B%E0%A7%9F%E0%A6%BE HTTP/1.1” 200 4356 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)” @CrayZeigh — #Abstractions
STRUCTURED LOGGING { “@timestamp”: “2019-08-04T12:30:04.000Z”, … “container”: { “image.id”: “48f5af6667f3457be0a2c7814caefe21ed3c94fb94bd6243096b3a61ea502b1d”, “version”: “version”, }, “build.id”: “efdd0b5e69b0742fa5e5bad0771df4d1df2459d1” … “transaction”: “transaction_ID”, “user”: “importantPerson”, “account”: “0129”, “os”: “osx”, … “api_endpoint”: “endpoint”, … “response”: 400, … “message”: “Some informative thing, probably more human readable and friendly, but difficult to parse” } @CrayZeigh — #Abstractions
APM & TRACING @CrayZeigh — #Abstractions
@CrayZeigh — #Abstractions
@CrayZeigh — #Abstractions
@CrayZeigh — #Abstractions
SHARED LANGUAGE > Learning to speak Prod > Teaching Prod to speak Dev (Structured Logs; Traces) @CrayZeigh — #Abstractions
SHARED TOOLS > Debugging in Prod > Ops skills transferrable and replicable > New knowledge and methods shareable @CrayZeigh — #Abstractions
SHARED SOURCE OF TRUTH > Real production data > Draw better lines from code to prod > Write better, production ready code. @CrayZeigh — #Abstractions
WHERE DO WE GO FROM HERE? @CrayZeigh — #Abstractions
Testing & Experimentation @CrayZeigh — #Abstractions
TEST IN PRODUCTION @CrayZeigh — #Abstractions
@CrayZeigh — #Abstractions
DON’T DEBATE EXPERIMENT @CrayZeigh — #Abstractions
2 Speaking of Testing : 2 QA https://theelasticast.com/episodes/0017-qa/ @CrayZeigh — #Abstractions
DEVOPS > Wave 1: Ops learns code & automation 1 > Wave 2: Dev owns code through production 1 https://vimeo.com/341142053 @CrayZeigh — #Abstractions
DEV OWNS CODE THROUGH PRODUCTION > Better, more production-ready code > Real World experimentation > Improved operational resiliency @CrayZeigh — #Abstractions
THE POWER IS YOURS
THANKS! > Slides & References: noti.st/crayzeigh > Trial: ela.st/aaron-aldrich-trial @CrayZeigh — #Abstractions
The world of maintaining applications is shifting. Our Systems Administrators aren’t monitoring our application servers, it’s the SREs determining SLIs and SLOs for our Micro-Services and making sure our Cloud-Distributed systems are Observable in near-real time. It’s enough for buzzword burnout. Furthermore, the DevOps movement asking developers to work closer with our operations teams and handle on-call shifts! Does that mean you have to learn everything about DevOps and SRE just to keep doing what you truly love, building things out of code?
Not entirely, but just as SysOps folk have started to learn valuable lessons about codifying their work, so too can developers learn how to build more easily maintainable software from those that have to maintain it every day.
Without diving through the whole world of Software Reliability Engineering and systems monitoring, this talk will level set and give a crash course on Observability. Let’s cross-class some Ops with our Dev and build reliable, maintainable software together!
Attendees will walk away with an understanding of:
- A level-set understanding of Observability so they can collaborate with their colleagues meaningfully.
- How cross-collaboration and a shared language provide for better quality software development
- A handful of ideas to keep in mind during development, to make operating software easier
- Why Developers should be On-Call, too, and the value that provides for both the Ops teams AND for building better software.
Resources
The following resources were mentioned during the presentation or are useful additional information.
-
Molly Struve - Making On-Call Not Suck
-
Monitorama PDX 2019 - Christine Yen - Observability: Superpowers for Developers
(Wave 2 of DevOps: Devs own code through production)
-
Wardley Maps
-
Introduction to Chaos Engineering from Gremlin
-
The Netflix Simian Army
(Chaos Engineering)
-
Testing in Production: Yes you can (and should)
Buzz and feedback
Here’s what was said about this presentation on social media.
