Agile Metrics – Feeding the Beast

On Thursday, September 8th, 2016 federal regulators announced Wells Fargo employees created millions of fake customer accounts using real customer money.  As a result, Wells Fargo terminated 5,300 employees.  Wells Fargo was fined $185M and is facing a global loss of trust by customers – the impact of which is still not understood.

What’s more interesting than the impact of the scandal is the root cause: metrics.

Wells Fargo offered big bonuses to employees that cross-sold financial products to customers.  From a corporate perspective, this bonus program was a measure of employee performance, product value and customer satisfaction – when in reality the program had an inverse effect on all three.

In the world of agile software delivery, some organizations treat teams in a similar fashion.  Too often the measurement de jour is around story points and velocity.  Over on the Agile Uprising Coalition, we have an interesting discussion going about if story points are valuable.

There are a number of ideas about how to estimate using something other than time. Points, Gummi Bears, Fibonacci numbers, T-shirt sizes. These were originally invented to obscure the time aspect, so that management wouldn’t be tempted to misuse the estimates. (I know: I was there when they were invented. I may actually have invented Points. If I did, I’m sorry now.)
— Ron Jeffries

Why measuring points is bad

Organizations, managers and stakeholders are preconditioned to embrace metrics as a measure of safety.  In traditional project delivery, the PMOs made a strong value proposition by taking the mystifying and highly complex craft of software delivery and distilling the status into dashboards and metrics.  Red/yellow/green, percentage complete and my least favorite, earned value – all have been used to show the external members tracking of how the team is performing with the investment they have been granted.  It is rather easy to see how this quickly translates to the obvious team metric of story points.

But here’s the rub: story points are meaningless to anyone outside of the agile team.  At the core, a story point is a measure of relative sizing of an unknown atomic unit of work given the team context and collective knowledge of the problem they are looking to solve.  It is a means for a team to build a language that is internal to them to gain consensus, and also highlight internal misalignments to drive conversations and interactions.  Taking this internal measure and aggregating into forecasts is very dangerous if the report reader is not aware of the context of the measurement.  Whats worse, since points are internal to teams. comparing a team that put out 20 points to a team that put out 70 points in the same sprint is a very toxic approach.

So, if not points, what should we measure?

The best way to figure out what to measure is to understand what behavior you are looking to address.  Critical to measurement is the core understanding that humans are competitive and will always game any measurement.  It is not a bad thing to address this fact.  Even managers, directors, executives and thought leaders game measurements.  Its part of our animal makeup. There are lots and lots and lots and lots of agile assessments out there to measure people and teams on agile practices.  But blindly applying a measurement to your teams has very low value.

An example of understanding patterns and behaviors you want to change can be seen in a measurement change I took in the past.  I had a team issue with partially done work in sprints.  Lots of work started, some completed.  Looking at the work items, the team was taking on very large items and assigning them to one or two members to work for the duration of the sprint.  Knowing this, we shifted measurements to focus on the number of stories completed.  Recognizing that people want to be seen as successful, we started seeing the desired shift.  More stories were getting done sooner, and (here is the cool part) the teams started slicing the stories more.  The atomic work units got smaller, and smaller again.  This allowed for the teams to deliver more stories sooner and the organization to realize value faster.

Another great measurement is team engagement and satisfaction.  Happier teams write better code.  Better code makes for happier clients.  Happier clients promote your product in industry.  The inverse is also true.  My teams regularly use OfficeVibe to passively and anonymously allow for engagement measurements.  There are other similar tools on the market, but when entering a selection criteria phase be aware of what behaviors you are looking to correct or craft and ensure the tool drives those behaviors or outcomes.

In agile development, just like in banking, metrics that are initially designed to be a positive rating can be misappropriated if there is not regular inspection and awareness of the corporate culture.  Leaders that read metrics need to understand the pulse and spirit of the organization being measured.  Assuming the metrics are an actual measurement or predictability and behavior is short sighted in every industry.