Sticking Together while Staying Apart: Resilience in the time of global pandemic
A presentation at Seattle DevOps Meetup by fen aldrich
Sticking Together while Staying Apart Resilience in the time of global pandemic Aaron Aldrich @CrayZeigh
Million-to-one chances… crop up nine times out of ten. —Sir Terry Pratchet GNU @CrayZeigh
Everything’s a little bit broken all of the time… but it keeps working anyway @CrayZeigh
Resilience @CrayZeigh
Resilience Rebound Graceful Extensibility Robustness Sustained Adaptability @CrayZeigh
Rebound Return to “normal” after a surprise or traumatic incident. Work done ahead of time @CrayZeigh
Robustness The ability to withstand and absorb well-modeled disturbances. Known-knowns @CrayZeigh
Graceful Extensibility The ability to stretch with challenges to operational boundaries. Opposed to brittleness. @CrayZeigh
Sustained Adaptability Recognizing and managing adaptive capabilities over long timescales. Requires people @CrayZeigh
@CrayZeigh
Bone • Continuously created and destroyed • Reconstruction directed by mechanical strain • Process directed by signals through layered networks at cell-level @CrayZeigh
https://youtu.be/8LbePBiOvZ4 @CrayZeigh
Rebound Graceful Extensibility Robustness @CrayZeigh
Socio-Technical Systems @CrayZeigh
Conway’s Law Designed systems represent an organization’s communication structure @CrayZeigh
@CrayZeigh
@CrayZeigh
@CrayZeigh
Blunt End Removed from experience, upstream decision makers Sharp End Closest to the work, practitioners @CrayZeigh
• • • • Constantly building and destroying systems Strong signaling Improve systems based on strain Will do so naturally given ownership Sharp End Closest to the work, practitioners @CrayZeigh
Teams that do well dealing with impact [surprises/incidents] are those that have a strong common ground —J. Paul Reed (@jpaulreed), Sr Applied Resilience Engineer, Net ix Failover Conf
fl @CrayZeigh
If we want to improve a team’s resilience, we must build a strong common ground —Me, Just now. @CrayZeigh
Common Ground • • • • Basic Compact Goal Alignment/ Commitment Inter-predictability Sustain & Repair @CrayZeigh
Building Common Ground • • • • Blameless Postmortems Chaos Engineering Game Days Modeling Vulnerability @CrayZeigh
@CrayZeigh
Our analysis found that this culture of psychological safety is predictive of software delivery performance, organizational performance and productivity. — Accelerate State of DevOps 2019 @CrayZeigh
https://youtu.be/SgCGD7rutSw @CrayZeigh
Resilience is about creating the conditions that maximize everyone’s potential —Rein Henrichs, >Code Podcast, 174: Resilience @CrayZeigh
@CrayZeigh
@CrayZeigh
What happens when governments fail? @CrayZeigh
It’s left to us @CrayZeigh
Community Building is Resilience Engineering —Me again, just now again. @CrayZeigh
Strong Communities • • • • • Diverse High Trust & Safety Sustain & Repair Inter-predictability Loosely Coupled, layered networks @CrayZeigh
@CrayZeigh
@CrayZeigh
@CrayZeigh
@CrayZeigh
@CrayZeigh
@CrayZeigh
@CrayZeigh
https://bit.ly/2Ym7Tp9 @CrayZeigh
@CrayZeigh
@CrayZeigh
@CrayZeigh
https://desertedislanddevops.com @CrayZeigh
https://youtu.be/L9A6ZauhOhg @CrayZeigh
Enable potential and get out of the way @CrayZeigh
Slides & Resources speaking.crayzeigh.com OSMIhelp.org Aaron Aldrich Managed OpenShift Black Belt EmotionalAPI.com @CrayZeigh devopsdays.org
twitch.tv/desertedislandtv discord.gg/CPM5Jcg
I love you Do Good out there We’re all in this together @CrayZeigh
Watching/Listening Four concepts for resilience and the implications for the future of resilience engineering - David Wood https://bit.ly/3bITTdc The Marvelous Resilience of Bone - Dr. Richard Cook, REdeploy 201 https://www.youtube.com/watch?v=8LbePBiOvZ4 Greater Than Code, 174: Resilienc https://www.greaterthancode.com/resilience The Worst Year Ever, How to Save your Community When The Government Fail https://ihr.fm/3eVNFbI s 9 e s @CrayZeigh
Watching/Listening Behind Human Error(2nd Edition) - Woods, Dekker, Cook, Johannessen, Carter The Woolworths Experimen https://safetydifferently.com/the-woolworths-experiment/ The Field Guide to Understanding Human Error - Sydney Dekker Literally every video from REdeploy https://www.youtube.com/channel/UCHbJcI6Kfyx Rqdv26b3Qw On Borrowing From Yourself - Aaron Aldric https://dev.to/crayzeigh/a-re ection-on-borrowing-from-yourself-3jhf fl h : t fl @CrayZeigh
Watching/Listening Kick ‘Em or Keep ‘Em - Collaborating on our own Deserted Islands - Matt Stratto https://youtu.be/SgCGD7rutSw ACCELERATE State of DevOps 2019 https://services.google.com/fh/ les/misc/state-of-devops-2019.pdf n fi @CrayZeigh
Everything is a little bit broken all of the time. Sometimes, like recently, as in right now, the core truths and assumptions about the universe seem to be shifted out from underneath us without warning. And yet, Netflix streams movies, GitHub serves code, we all keep going and working from home and surviving. These systems, both technical and social, are resilient. And through their resilience they are able to handle all those one-in-a-million occurrences that crop up nine-times-in-ten. But what does it mean to be resilient? And further, how can we recognize and grow this resilience to better deal with the systemic surprises we’re dealing with right now and in the future? What does a resilient team look like, and how do you foster that?
This talk will answer:
- What is Resilience?
- How can we create resilience in ourselves, teams and systems?
- How do our networks of systems (people, teams, organizations, applications) create resilience for each other?
- What happens when major systems and networks fail?
- How can we discover and foster resilience in our people, teams and systems?
Video
Resources
The following resources were mentioned during the presentation or are useful additional information.
-
Four concepts for resilience and the implications for the future of resilience engineering - David Woods
white paper from David D. Woods
-
Uprising: A Guide from Portland
Uprising is an excellent overview of the resilience of a social uprising, especially episode two: We Do This Every Night which examines how the protests have evolved since they first started after George Floyd’s murder by police in Minneapolis.
-
The Marvelous Resilience of Bone - Dr. Richard Cook
Video from Dr. Cook at REdeploy 2019
-
Get Started with OpenShift
“I don’t care about all this human interaction bit, I just want my applications to be more robust!” Ok, fine, then there are a million and one options out there. Check out OpenShift as one of them and see what it fills in around your container/Kubernetes ecosystem.
-
Google SRE books
“I care about team practices and behavior, but I want practical technical steps to take” Ok, sure. Check out how SRE practice approaches the concepts of reliable software. If you’re not thinking about SLOs and SLIs yet, this will make some technical progress for you.
-
Implementing SLOs
Further SRE-style reading.
-
Behind Human Error (2nd Edition) - Woods, Dekker, Cook, Johannessen, Carter
-
The Field Guide to Understanding Human Error - Sydney Dekker
-
The Woolworths Experiment
Background on Safety Differently and how empowering employees at the sharp end and removing beurocratic top-down policies improved safety overall.
-
All the REdeploy videos
Seriously every talk from this conference was on point one way or another. Some are more practical, some are more philosophical, all of them are about resilience and worth taking the time to watch and think about. Some more than once.
-
The Halo of Resilience Engineering
J. Paul Reed’s talk from Failover conf earlier this year, 2020
-
Accelerate State of DevOps 2019
The State of DevOps 2019 Report
-
Kick ‘Em or Keep ‘Em - Collaborating on our own Deserted Islands
Matt Stratton’s talk on collaborating in teams with emphasis on Psychological Safety and practical ways to enable that in our teams.
-
Greater Than Code, 174: Resilience
Greater Than Code Podcast episode regarding resilience
-
The Worst Year Ever, How to Save your Community When the Governement Fails
The Worst Year Ever podcast episode specifically around community action in the absence of government resources. You should listen to a number of episodes on this one, though.
-
On Borrowing from Yourself
My blog post reflecting on the costs and alternatives to borrowing work time from your home time in this moment of crisis. Some shameless self promotion, but words I truly want to share and couldn’t fit in the talk.
Buzz and feedback
Here’s what was said about this presentation on social media.
