548 Days to Fix 1 Line of Code

People say that retrospectives are the most important meeting. The idea that if the team is able to meet and improve, then everything else will work out in time. How many retrospectives have you been in where improvement didn't happen? The team meets, writes stickies, forms action items, and not much changes.

This is a story about a team, retrospectives, and the year and a half it took to fix one line of code that was causing problems.

This is the situation of the team that I joined as a senior engineer. We were going through the motions of retrospectives, but nothing was changing.

From the day I joined there was an item that would appear in every retrospective: Flaky automated tests.

This issue was at the heart of our year and a half journey.

In early retrospectives the hope was that I would resolve this issue. I brought a new set of skills and deep experience with automated tests. In the early retrospectives, I was the manifestation of their solution.

After three months, the issue still made appearances during retrospectives. In the meantime, I had built out new automated testing tools and began to teach others how to use it. The tests were flaky because people were getting used to these new tools and writing tests. The solution was to give it some time while everyone came up to speed.

After six months, the issue continued to appear. Everyone was comfortable with the new tools and skills around tests, but they were still flaky. The retrospectives failed to give improvement so far. No progress on this issue made the team tired of seeing it. The team could not fix this flaky automated test issue, and gave up. The flaky test issue was played out, and made did not appearances in future retrospectives.

After a year, a conference I attended introduced me to several ideas, and I began to push for changes to how we treated our retrospectives.

1. We were assuming we knew how to fix this issue

Every time tried to make an improvement during retrospectives, we made assumptions. Those assumptions informed our solution. What I realized that if our assumptions were wrong, our solution wouldn't work. I also began to see a pattern in our solutions. If someone cared about databases, their improvement ideas would involve databases. I wanted to make the assumptions visible to to the team and focus on testing these assumptions.

2. We were attempting too many improvements

Our retrospective format encouraged the team to come up with 3-5 actions every time. We had these actions to work on every sprint, but we often didn't work on any. I wanted us to focus on completing just one improvement before we attempted more.

3. We were not following through on any improvement

Every retrospective we had was a blank slate. We didn't discuss what happened with our previous improvements. We didn't talk about if we improved the problem with our attempted fix. We populated a board from scratch every time. I wanted us to see if we had any impact with our improvements before moving on.

So I went to my Scrum Master. I made my requests to change our retrospectives.

I asked that we move to using a hypothesis format instead of an action item one. This would help us identify the assumptions we were making. I asked that we only make one such hypothesis in a retrospective, so that we would not lose focus. Last, I asked that we only work on one problem until fixed. I didn't want us to reset our board. I wanted us to work on the issue bothering us until it didn't.

Three months passed after I made my requests. The truth is, that I didn't empathize with my Scrum Master at all during this time. While he was very much a role-model to me, during these conversations I never tried to understand him. I didn't care about the work he had, goals he was working on, concerns he had, or how I could help. I asked that they make all these changes. I still think about what could have been different if I wasn’t a jerk.

After those three months we came into a retrospective, and the scrum master drew a sailboat on water, and explained the Anchors and Engines exercise. He asked us to put the things anchoring us down below the water, and the things pushing our boat forward above the water like the wind. He then asked us to prioritize the things holding us back, and the thing holding us back the most would be the anchor. Flaky automated tests emerged as the anchor.

This was an interesting moment.The team had stopped talking about this issue during previous retrospectives. We were burnt out on trying to fix it and not getting anywhere. Our previous retrospectives allowed us to censor the very thing that was holding us back the most. I saw that different activities produce different results.

Our scrum master then asked us to make 3-5 action items. My heart sank, and I became a jerk. By now two tribes had formed around why our tests were so bad. I began a line of questions to each about what assumptions they were making. This line of questioning did not earn me any friends. After tearing their ideas apart, they turned and asked what I thought we should do to fix the issue. I replied that I have no idea how to fix the issue, because I don't know what the actual problem is. My suggestion was that we form a hypothesis, or a question, and our goal will be to answer it. We decided to track failures in our tests, and see if there were any trends in our failing tests.

Three months passed. A colleague pulled me over after an change to our data collection. Almost every single failure happened within one single test. That test was one line of code. That line of code had one mistake in it. It was obvious. Around 99% of our problems were coming from this one mistake in this one line in this one test.

He fixed the issue, our tests began passing. Instead of a celebration there was panic.

Our tests only passed if something seriously broken and reported false positives. After a few minutes the team realized this wasn't the case. There wasn’t a catastrophic failure, but something we had never seen: Passing tests.

We stopped work, we went to the break room, and celebrated. We had fixed the problem that had held us back for a year and a half.

This was a meaningful improvement worth celebration.

In the years since then I've thought about this time, and tried to distill the lessons I learned. These are what I've taken with me:

Different retrospective activities give different results

We did retrospectives the same way every time, and the group used that comfort to hide from the issue that held them back. It wasn't until the Scrum Master brought Anchors and Engines did we see the real issue we needed to work on.

Single problem focus gave us feedback

In our earlier retrospectives, we never stopped to ask if we were improving. We focused on creating new actions to problems. This left us without the critical feedback we needed to hone in on what it took for us to actually improve. When we shifted to keeping the same problem on the board across retrospectives, we were forced acknowledge that we weren’t improving. After the change to keep the problem across retrospectives, we would admit that we had done nothing during our sprint to make an improvement. We had the feedback, and over time we got better.

Move from action items to questions/hypothesis

We made a lot of assumptions about the problems we tried to tackle during retrospectives. We needed information about our problems. The more information we had, the fewer assumptions we would make. Living in the problem is worthwhile. Instead of forming solutions, we would ask questions and try to answer them.

Empathy matters

I was a jerk during a lot of this time. I was unkind and unhelpful. I didn't care to understand the team I was on or my Scrum Master. This kept everyone out of participating in these ideas. It kept us limited to the best that I brought to the table. It slowed us all down. Learn about your colleagues.

This time during my career taught me a lot of lessons about retrospectives, change, and teams. I’ve brought the lessons with me as I’ve moved on from being an engineer to someone who is passionate about building great teams. I hope that there is something useful in my story, and I am more than happy to talk about this experience or anything else. Find me on twitter with @recursivefaults, read my ramblings at https://ryanlatta.com/, or stalk me on LinkedIn.