SEV 1 Party

“The Battle from ‘The Chronicles of Narnia‘” re-arranged and programmed by Ashton Gleckman. Original composition by Harry Gregson-Williams 🎵

One of the best parts of Accelerate by Nicole Forsgren PhD, Jen Humble, and Gene Kim is an absolutely juicy chapter on Measuring and Changing Culture to adopt continuous delivery (and the same can be applied to continuous deployment).

In it, they cover “Westrum’s Typology of Organizational Culture” from 2014. This theory describes a belief that at any point in time, a company’s culture can exist as Pathological (Power-Oriented), Bureaucratic (Rule-Oriented), or Generative (Performance-Oriented) (Forsgren, 30).^[1] Westrum’s research showed that the best cultures cared the most about the missions they were on and sharing information with “Generative organisations get the needed information to the right person in the right form and in the right time frame” as opposed to any other cultural aspects that would deviate from that mission (Westrum, ii22).^[2]

One of the reason Accelerate resonates is that the research in it clearly shows that culture, not only technological choices, are the reason lower performing teams move slower and make larger mistakes when instead they can move faster with accurately delivered, focused and accessible information, and less overall risk impact – by being willing to break things fast and in smaller increments each time (ideally not on the weekend though 😉 ).

For this post I want to talk about the 3 different types of cultures and ways I’ve seen those manifest – as well as concrete outcomes in Generative teams that you can do to change culture by changing yourself to model a different one.

It’s not enough to simply measure PRs, merge time, and severity. You and everyone around you has to personally model a new personality (or as some refer to it – “Become a clown.” and laugh a whole lot more when shit goes wrong).

Pathological (Power-Oriented)

A pathological culture is defined as “characterized by large amounts of fear and threat where people often horde information or withhold it for political reasons, or distort it to make themselves look better” (Forsgren, 30). In this culture there is “low cooperation, messengers are ‘shot’, responsibilities are shirked, bridging is discouraged, failure [becomes] scapegoating” and more (Westrum, ii22).

In DevOps, FAANG, and in general this industry I’ve seen this manifest in a few ways over 25+ games and a few multi-billion dollar public companies that were my customers – I’m going to list real situations I’ve seen and then later list what good looks like as a target.

In Pathological cultures, you may or may not have seen the following:

1. Shooting the Change-maker: An engineer pushes to production a change that had been reviewed by many parties, but because it was “their change” they are PIPed, fired, or grilled by executive leadership. Their own engineering managers or internal customer who asked for the change to be made are the ones who escalated up and threw them under the bus. Some who were part of the overall change actively work to hide, not speak up, and do little to shield the final change maker or support that engineer emotionally beyond reprimanding them.
2. Re-Orgs are the Only Solve: An account team fails to close a deal and in an effort to shield the quota owners (executive account managers) individual sales engineers are blamed instead and moved off the account or the account manager is moved off and the sales engineers stays. Neither are given the opportunity together to try again as a team with a new account. It had to be “someone’s fault.”
3. Death by Perfectionism: Cosmetic bugs found in a Beta client-side app demo for an executive partner, client, or investor as it was delivered result in a Release Manager getting grilled for those bugs. “Why didn’t YOU find them!” and “Why didn’t YOU fix them IN TIME?”
4. Silo-Culture & Racing: Game design teams are “not allowed” to speak to engineering teams for fear they may incorrectly influence each other or cause scope creep. Anyone who crosses those boundaries via email, sitting next to those employees, will have it come up in their performance review. A SDM actively discourages his PM from speaking to other service teams that he does not like. A PM doesn’t include other parties for fear they may “share their work to evil people and others may race to submit their proposals first so it won’t get approved.” And those things actually happen.
5. Pretending to Work Together: Two acquisitions forced to merge will pretend they are doing work together. That work is small and not strategically impactful because they do not really want to align fundamentals of their missions as they have different beliefs about how to do that and when – often taking years to address. Nothing is requiring them to have the same fundamentals beyond “please merge eventually.” An unwillingness to let go of technologies is a vibrant and visible through inaction not words.
6. Incident Resolution is More Important than Hands On Learning: Often solving an incident is more important than other people learning to solve them. This means heroism is rewarded over learning. Heroes, those with tenure and subject matter expertise, are called upon too often – bypassing any policies and opportunities for learning – and tanking team health overall as tenured people become too much in demand and newcomers never get to learn in real experiences.
7. Image is More Important than the Issue and Learning: A department head yells at another department head or their SVP that there was an incident without understanding or even acknowledging the depth and breadth of the impact – or appreciating that it happened to reveal so much more. A internal or external customer demands that they are faultless and doesn’t want to work on partnering or report their own issues because “they are the customer” so they shouldn’t have to.
8. Unreporting: Incidents go unreported in some teams because those team have been yelled at so much that they fear any incident they report will result in more yelling, more pressure, more punishment – so they only report ones which can be quantified as high severity, high visibility.

It’s easy to look at these things and think “What could I otherwise be doing if the stakes are high? Shouldn’t people be punished for their mistakes?” The problem with this mindset is it leads to:

The team does not feel safe to work for you – that’s your fault.
Products get worse. Everyone gets more angry over time.

At scale? That leads towards a higher attrition rate, longer times to hire (due to shared knowledge of company culture), and work productivity decreasing as people over-analyze each choice and change they make for fear they will have a bad day.

It also in my experience leads to a ton of “footballing” – which is the term I use when people don’t take responsibility for their own mistakes (as accountability and ownership is directly associated with punishment) and instead just stop working entirely on impactful work – They make other teams do that work – whilst still having the perceived authority over the product with regards to whether it is meeting the use case. This lands at good internal teams who are performance-oriented becoming overworked with the hardest problems.

Bureaucratic (Rule-Oriented)

A bureaucratic culture is defined as one that “protects departments. Those in the department want to maintain their ‘turf,’ insist on their own rules, and generally do things by the book – their book” (Forsgren, 31). In this culture there is “modest cooperation, messengers are neglected” and departments insist on having “narrow responsibilities” where “bridging is tolerated and failure [becomes] justice” (Westrum, ii22). Teams care more about protecting and shielding their departments than they do about the mission itself and following the rules above all else.

1. Piling on Approvers: An individual engineer tries to make a change and a system enforces that that change require a board of approvers to get out the door in addition to their own team as reviewers.
2. Pointing to the Rules Instead of Understanding the Root Cause: A team will point that the rules were not followed in an incident in another team for example, “The wrong person opened up the incident” instead of “How did this incident happen?”
3. Solving for Process Instead of Speed: After an incident a team will spend their action items (time) on adding more process and documenting more rules instead of better architecture or automation. They will insist that more reporting and more communication, even if there already was a lot of communication, will mean there is less likely to be an outage next time.
4. Meetings that Stall: Teams have meetings with other departments, but often the updates are the same. Items do not move forward. When asked to create due dates, teams will push back on creating due dates.
5. There’s More Rules than Can be Reasonably Followed: All rules for how to solve an issue are documented in such detail that when it comes to actually solving an issue there is no way to actively follow the rules in real time or do them correctly. There may be 7 versions of rules across different teams because the solution every time there was a problem was to “write down the rules” instead of understanding that maybe – less is more. “Writing them down in painstaking detail and getting everyone to agree” was more important than the mission of people knowing at bare minimum what to do through practice and collaboration when they needed help. The rules were often pointed to (Perfectionism) instead of people, when they got the main idea correct, being championed.
6. It’s Hard to Get Games Launched: As teams protect their ‘turf’ they make it so their work has to be implemented first. Organizations put in a lot of release reviews for pilots. Many pitches. They make it harder to get player feedback early for fear of leaks.
7. Shrinking in Scope: And finally, individual teams that feel this, with the “least amount of power” and constantly yelled at by internal and external customers stop making many changes for which have actual production or player impact – focusing instead of less business impacting work where they are less likely to make mistakes as shown through lowered merges, lowered PRs, and less code in general overall being contributed to. They feel safe “In the rules” and are no longer bold.

Generative (Performance-Oriented)

Pardon the correlation with “Generative AI” as it is not intentional, but a generative culture is defined as one that cares about how teams work to achieve the mission above all else. They care deeply about getting better and see getting better not as getting rid of other people nor as defining more rules. They care about sharing the right information to the right people as needed – it’s not locked away, but it’s also not “everyone” all at once and people aren’t punished for missing.

1. Department-Crossing Collaboration: Roundtables are productive – teams contribute ideas together working towards a common goal they believe in instead of talking about only why the goal is not achievable. Even small meetings of 2-4 people still result in information being shared to those in the larger group as minor progress is made.
2. People Want to Change: Individual engineers, game designers, product managers, and producers are actively rewarded for wanting to change the way things are done and encouraged to continue to ask questions until they find wins that work for the company in a way that can move the fleet with the right timing.
3. People Credit Each Other – Boldly for all Work: Those in leadership spend a lot of time and energy in making sure even those several levels below them are recognized and credited for their work. CEOs actively apologize when they leave key parties out and credit the wrong parties multiple times – those given too much credit, call out their peers – in their team, in other orgs, in other studios. In private DMs and public calls team members thank each other. In support chats they call out people who made good things happen. They are grateful, and vocal, and kind. They do so not because of reward but because it feels good, is seen as a valued part of their time, and know their leaders will take note they did it.
4. Everyone Feels Safe to Open an Incident – And are REWARDED For It: In healthy devops cultures where teams know there is a production outage or a possible one occurring, no one waits for someone else to open one first. They do it – and if they don’t have time or can’t because they are looking into it, they bring in a friend to help – they don’t point blame at who should be the one opening up comms and ask for help. They are REWARDED for their transparency – not for responding to alerts (that’s good too), but for transparency and frequency.
5. Healthy Boundaries are Respected: Escalation paths are used instead of DMing whomever you want when team members are needed. Trust is about respecting who is available and the opportunities for learning for engineers who may not have as much experience even at the risk of slowing down while an issue is escalated appropriately (Remember the Mission: Long term speed through knowledge sharing). Collaboration and the systems that support it are respected so that those who haven’t had exposure are involved and not left out regardless of severity and subject matter expertise giving them the opportunity to say “I Don’t know” try and bring in their own friends.
6. No Place for Regret^[7]: Teams in performance-oriented cultures don’t regret that an issue, a bug, a demo happened, they only decide what to do with their time that is given to them – both during and after – to find another opportunity to do a better job. They move on quickly and boldly champion lessons learned with pride, proud of even their greatest mistakes.
7. They laugh a whole lot and bury the fear underneath the floor when it’s there through laughter.

In Summary

I’ve worked on over 25 games and stopped counting years ago when my clients became much much bigger. I saw a repeat of these issues across customers very much interwoven directly with architecture. We couldn’t solve for an industry wide architecture problem without also knowing it’s cultural ones – I have seen the spectrum that can happen and no one is alone in this.

I would never pretend these types of transitions are easy or any one team or one culture exists exclusively in a single state.

It’s okay to look at any culture one is in and say “We have stuff we’ve got to fix.” – then be an active participant in fixing it.

Don’t only complain. Model what you want to see slowly – regardless of being an executive, and employee, a engineer, a designer, or a producer.

Become the person who credits other people and points to the work they are crafting that they are thankful for instead of the person who only wants to complain when things go wrong. Don’t only recognize those who are doing the easy work in a culture change – recognize those who are doing the hard, painstaking work that cause conflict and lands them in conflict resolution over and over. Protect those people fiercely and join them.

Don’t avoid what’s hard.

Don’t erase a year….2 years…3 years…5 years of work with people when the stakes get more challenging.

Don’t forget what entire teams have done because of an incident not realizing the big picture of what has gotten better over years – not taking accountability and responsibility within your own organizational silos and reflecting that perhaps, your attitude is setting the wrong model. It goes both ways – it is a two way street. Always. It is easy to yell and be angry at those around you.

It’s much harder to be calm and laugh knowing that with kindness comes speed when something goes wrong and absolutely zero of us are perfect at this.

Admitting that we might live in this spectrum, all of us at any company, is valuable. It’s important to realize that a culture can be generative and other parts of it can be rules-oriented. The worst parts, for which cultures are ashamed of, are still power-oriented and still happen because sometimes you invite in the wrong people into a culture and have to teach them to not revert.

Admitting any of that exists is the first step – because the second is deciding not to model it yourself.

Eventually after several small attempts to make changes, to document where we’ve been and where we’re going, we will notice looking back that our culture, through the lens of our people and their happiness, has changed. People will try to pull you off mission with new shiny ideas that are easier – saying that this is a new problem that is “bigger” and you have to remind yourself that it is not.

Or that “this one time we did this and it was okay. It was okay we were assholes this time because it was a SEV-1” or “We thought it was a high severity” and realized it wasn’t – Nothing is more important than culture. Even I sometimes don’t show up correctly with that in mind.

I don’t say this as someone who “thinks it will happen.”

I’ve lived good and bad culture already over 25+ games. My blog is me constantly reminding myself that I don’t want to be that person ever again and re-center on practicing what it means to laugh when it’s hardest.

–

[1] Forsgren N, Humble J, and Kim G (2018). “Chapter 3: Measuring and Changing Culture.” Accelerate: Building and Scaling High Performing Technology Organizations. Portland. It Revolution. pp. 30-33.

[2] Westrum R. “A Typology of Organisational Cultures.” BMJ Quality & Safety. 2004;13:ii22-ii27.

Header Image Credit by NASA from the James Webb Space Telescope in January 2024. More images can be found on Flickr. This is an image of 19 Nearby Spiral Galaxies. Specifically “NASA’s Webb Depicts Staggering Structure in 19 Nearby Spiral Galaxies: Nineteen Webb images of face-on spiral galaxies are combined in a mosaic. Some appear within squares, and others horizontal or vertical rectangles. Many galaxies have blue hazes toward the centers…Some of the galaxies’ arms form clear spiral shapes, while others are more irregular. Some of the galaxies’ arms appear to rotate clockwise and others counterclockwise. Most galaxy cores are centered, but a few appear toward an image’s edge. Most galaxies appear to extend beyond the captured observations.“

–

[7] This is the seventh clue to the puzzle.