Failing to measure is the first step towards failure. But, knowing what to measure, why, and how to adjust along the way is just as critical if you want your engineers to thrive.
Most of us have at some point finished a project or changed a process only to end up unsure whether we actually improved anything. This is why measuring progress is so critical. That said, there are also plenty of pitfalls in the process.
The only way to know for certain that we’re making actual progress is to put a plan in place and include how we can measure its value. In other words,
“That which gets measured gets managed” - Peter Drucker
Or more simply,
“If you're not assessin’ you're guessin’” - Unknown
When we measure, we uncover the current state of things as well as gain a better understanding of how improvements play out throughout our new plan or project. Often, these take the form of KPIs (key performance indicators).
The first step of effective measurement is deciding what to actually measure. At the most basic level, this comes down to outputs vs outcomes. These can range from the number of widgets produced (output) to the count of errors or flaws per a set number of items produced (quality).
We can get a general idea of whether we’re improving by how we feel (qualified measurement), but if we want specific results then we need a more specific metric to track. In each instance, there are output and outcome metrics that help us to understand the measurement of success.
Output is the measurement of quantity. This could be the number of tasks completed in a project or the number of projects a business completes in a year. Often these are the easiest metrics to capture. However, they are also often the most misunderstood. Outputs are a good thing to know most of the time. But, they need other metrics to be useful.
Going back to the count of items produced. If we have a project that increases product output, then having a count is going to be great. However, it does not demonstrate a complete picture of events. For example, an increase in output can be accompanied by a decrease in quality. This is where outcome or results-based metrics come into play.
Determining which metrics to measure in a process might take some experimentation.
The Project Management Institute (PMI), for example, uses what they called the triple constraint (also known as the iron triangle) for management and a basic understanding of measuring for projects. The three constraints are the scope, schedule, and budget of the project.
They put boundaries on those three to understand when a project is going to exceed or fall short. However, since these are often only output metrics, there was no consistent eye on quality, outcomes, or results.
In other words, this approach is helpful in some instances but not an ideal mechanism for establishing the right outcome. The underlying philosophy is that projects usually go off the rails due to a lack of constraints and discipline allowing for little flexibility or adaptability which is required in some project environments.
This is where the Agile Manifesto comes in (note this is the manifesto for software development, but you can also find versions for marketing, HR, and more). Agile practitioners now had a set of values and principles to inform how projects should be managed. However, some people mistook this as a call for the complete abandonment of discipline.
Agile focuses on measuring value instead of outputs by placing a strong focus on stakeholders. In other words, it presses you to think about who you’re doing the work for and how you can best deliver something valuable to them. In practice, this translates into a strong emphasis on outcomes over outputs. However, this can be misinterpreted.
One team I worked with briefly, assumed this meant they could ignore documentation and planning entirely: to the point of not even developing a long-term goal outside of the immediate requests.
Without a clear plan, some level of constraint, and discipline, the work produced by this team was haphazard, and consistently had issues delivering a goal that was not quite in focus. In the end, the loss of one key team member resulted in months of rework and the loss of critical institutional knowledge.
Too much structure or not enough? That is the question.
To find the balance, we must be prepared to experiment with structure and adaptability. A clear understanding of the constraints, as well as the points of adaptability, are crucial if we want to be successful in an endeavor.
Some questions to consider when we are ready to begin measuring outcomes.
The answer to maintaining balance in measurement is to make sure it is done proactively. This is where the importance of the last Principle in the Agile Manifesto (Responding to change over following a plan) comes into play. We need to have a plan in place to regularly review and adjust our work as well as our measurements of that work. In addition to the need to adjust until our current plan works well. What works now, might not work forever.
Measurement is good, necessary, and not always easy. Because it is challenging there will be people who don’t want to do it. Ultimately, it can be hard to do but that should not stop us from doing so if we want to be successful.
“Success is not achieved by accident.” – John Maxwell
What can you actually do to make effective retro a reality? The good news is, it’s not as difficult as you might think. We’ve broken it down into 5 elements to make it as easy as possible to massively improve your sprint retrospectives.
Most engineering managers instinctively think of productivity as complex processes like workflows, solutions, and deliverables expressed in purely quantitative terms. The result is a dynamic that’s usually counterproductive. But there is another way.
Often that dynamic plays out in metrics and misguided attempts at fostering engineer productivity, usually by measuring the wrong things and offering the wrong incentives. Fortunately, there are better ways to use metrics to genuinely empower engineers to accomplish more.