“iPhone Ringtone Trap – Remix” by Jasser Labidi… 🎵
This week I was wrong.
I said on a call “Managing addons are not the hard part of what we do” thinking there are some I personally prefer devops teams to manage over third party providers wrapping anything and everything and serving it.
Not even two days later I ate those words as my excellent team presented on “Why Managing Addons is a Nightmare.”
Specifically, when comparing the dependency chain of which addons should be installed first with the words that the dependency chain “…is kind of like that conspiracy meme.” It is also not great to have to uninstall ones teams don’t plan to use when a managed product already has them baked in.
I don’t think I’ve laughed so hard in a while.
Both at the idea that dependency chains are like this, which is an excellent metaphor – thank you, team:
But also at the universe showing me quite quickly at just how wrong I was with regards to where I thought the pain lived.
It isn’t who and which team manages the dependencies we all use for Kubernetes as much as it is the order of which they are installed that can also be a pain. More importantly, many managers and higher are often wrong in our understanding of what the actual pain points are for teams on a day to day basis.
For example, it may be true that managing a specific addon itself is a pain in the butt or it isn’t.
Or it may be that an addon isn’t painful but that no one is handling a dependency chain problem it’s had for years. Well-meaning leaders and those listening keep thinking that there is money to be made by “wrapping the problem” and passing off who manages that addon instead of fixing a bug that’s been sitting in the repo their community needed addressed because all they heard was “it’s hard to manage them” not “it is specifically this bug and this dependency that makes it hard.”
As an example pulling from some “bad code I wrote” archives – Have you ever learned Python from scratch and tried to also make it work in AWS? Somehow learning anything these days starts with installing brew and a bunch of other tools to get setup. But who actually tells you that? To learn Python you first have to learn all the stuff to setup your local environment in order to learn. And then you realize to even make a simple .py script run now the top of your “This was supposed to be a… simple script” looks like this.
import boto3 # In shell run "pip3 install boto3" if you run into errors
import numpy # In shell run "pip3 install numpy" if you run into errors
import math
import random
import time
import sys
import json
import csv
from uuid import uuid4
from datetime import datetime
# See this error? Check your python version in shell with "python --version".
# Run this script using Python 3 by typing "python3 SpawnDragonsToFirehose.py"
if sys.version_info[0] < 3:
raise Exception("Python 3 or a more recent version is required.")This probably would have had even more dependencies if I didn’t keep the purpose of that script “as small as humanly possible.” As you get further in your career this happens to apply to everything and largely the dependency chaining gets even more complicated and absurd.
You Can Be the Person who Is Okay With Being Wrong Or You Can be the Person Who Isn’t
This reminds me that at pivotal moments like this we have two choices: We can say “I was wrong and I welcome this information.”
Or we can dig a hole in the ground and put our faces in it and yell that we weren’t included or some other irresponsible way of responding to new information that shows how wrong we all can be at different points in time.
These are two different ways of leading.
The only way to know pain is to live it directly – which takes time.
Re-Evaluating with Updated Information & Letting the People with the Pain Solve the Problem
How much pain does each of us want to take on?
It is often we are wrong about what we think needs our approval versus what may only need us to be informed or where our review and input is valued, but parties are willing to risk if we don’t get it in time.
It’s important to note that if teams already have forums for discussion of change – weekly standup, code review sessions, ops reviews – then if other systems and tools add more people, people who are not living in the pain, what those tools are doing is adding bottlenecks that prevent learning in production.
Often when faced against the image one may no longer have (or should no longer have) authority a leader can lose a sense of identity and purpose as so much of it is tied to ownership of being able to prevent a problem from happening – which is complicated if one believes it will make a team safer to prevent failure.
More importantly, leaders feel left out when their approval no longer matters in an issue that might need to evolve or to solve the pain, moving faster is required. And that feeling sucks – at first.
In my past, I have both been told “I am intentionally leaving you out because I want to quickly get this done and move on” with regards to something I had already started working on months before with another group by a leader who wasn’t invested until that moment, didn’t believe it in before, and now felt like they had to move as fast as possible. They now had their own vision of how to do that, because the problem had reached escalated levels of pain even though I tried to include them before, and they disagreed it was a problem worth solving. Which meant now that it was back I had to convince again, why we needed to be invested in a solution where we would take some time to really understand the problem and not a quick fix.
I have also seen a proposal with many leaders involved representing 4 different teams and an email thread one person was BCCed on. I personally got reprimanded that “I had left the person I BCCed out” (highest level leader) despite working my butt off, having them BCCed at the earliest emails, and inviting them live on a call to make sure they saw the proposal when it was finally ready. No amount of bringing them in on the issue was early enough and they had wanted to be on every call around it which surprised me. I perceived that I was including them and trying to bring them in on the important parts thinking we were wasting their time if we (the rest of the people with the problem) had not brought a verified solution now that the pain was back again or possibly, the person in front of me, wasn’t actually ready to hear it having been against this solution a year ago. The only thing I could do is say, “I’m sorry. Can you please make time to read this proposal?” and pray they believed that I was sincerely trying to balance inclusion with making sure we had thoroughly investigated everything and wanted to be respectful of their time. I had made a judgement call on when to include them and how, and it was the wrong one.
Inclusion isn’t easy.
It’s quite easy to be seen as the jerk based on timing, the seat you are in, the people and problems you represent, and the level of pain for those people versus the pains the person’s support you need has on their plate. The hardest parties to win may keep you trying for years to where the investment in the problem becomes personal and you have to decide if your time is worth it and if they’ll ever believe in you.
In both cases, I wish I had reminded in all of that mess: It’s important that the people with the pain own the problem regardless of how that problem evolves. We can’t make everyone happy – we can only show those who expect more of us, we are trying, say “I’m sorry” (Because in two stories above about leaders and when they want to be included, I did mess up in their eyes). We can only make sure others understand who a problem impacts the most and are truly invested if they need to be. That is my job.
These days I’m thinking that no matter how many people do or do not want to be involved in any issue and who they report to…It’s more often appropriate to step aside and trust the people with the most pain to solve the problem. Bring me in on the parts of the story that matter if we need help. I do not want to be angry at when I was or was not brought in only fascinated at where we are now and excited by the possibility that there is more information we can use to make a better decision than last time. I’d much rather say ‘no,’ let a problem get more interesting, and then say ‘yes’ with a full understanding of what teams are really solving.
Because it’s far more important for the people with the pain to own the problem in the way they need even when we “wish none of it had happened”[8], than it is for anyone else to try to rescue them, thinking they might know better.
–
Header Image Credit by NASA from the James Webb Space Telescope in July 2023. More images can be found on Flickr. This is an image of a Closeup of the Rho Ophiuchi, the closest-star-forming region to Earth. Specifically “Webb Celebrates First Year of Science With Close-up on Birth of Sun-like Stars: Webb spotted around 50 young stars, many close in mass to our star, giving us a glimpse into the early life of the Sun. “
–
[8] This is the eighth clue to the puzzle.
