This is part I of Allstacks’ KPI Best Practices Series aimed at helping engineering measure what matters, stay aligned with business objectives, and continuously improve their engineering operations.
Most people are familiar with the concept of KPIs (Key Performance Indicators), but a number of my colleagues struggle to nail down exactly what KPIs matter to their engineering team and their organization as a whole.
There are many moving parts, and it feels like any single indicator is either very measurable and not truly indicative of performance, or very well aligned with business goals, but not actually measurable. As engineering leaders, we have the responsibility to translate our engineering activities into business outcomes; this is a challenge for many, and I hope to offer some practical advice for those who need to overcome it.
The trick is to determine what KPIs that matter to you, your teams, and your organization. This process starts with understanding why your company even has an engineering team to begin with.
The Role of the Engineering Team
Company leaders determine core objectives that all departments work towards. As an engineering leader, it’s important to figure out how your KPIs map directly back to those greater goals.
At a high-level, company goals often seek to:
- Gain profitability
- Drive growth
- Deliver value
- Offer great customer experience
Determining KPIs is about taking those big, vague measures of success and pinpointing the incremental steps needed for your department to reach them. These are the outputs that need to be stated clearly, mapped out, and measured in order to gauge success.
Types of KPIs
Let’s discuss what KPIs should measure and how they relate to overarching organizational goals and outcomes.
Continuous attainment target [Activities]
Activity KPIs measure things like SLAs, uptime, compliance, headcount, bug rates, and more. Target KPIs should be continuously checked and monitored. They are most often entirely within engineering’s control, but can sometimes slip through the cracks. A great example would be your escaped defect rate. Most people are aware of bugs, and their impact on customers, but don’t dig deeper to track where those bugs originate. It’s a common practice to differentiate between bugs that are found in live code and bugs that get caught during internal QA. When you start tracking this type of target, you’re establishing a new KPI. Your internal QA process may be built around reducing or eliminating the rate of escaped defects.
Externally tracked target [Outcomes]
Examples of externally tracked target KPIs are things like your NPS score and feature utilization. These items are intended to measure the perception of your application. It is recommended to use externally tracked targets as guiding metrics to validate when the application is running properly and/or help you do a better job when it’s not. External targets are typically items that are beyond the direct control of your team. Short of asking your engineers to fill out NPS surveys, it’s impossible to ship a feature that will increase NPS by 1 point directly. However, these types of targets should drive other KPIs or goals in your team. If the overwhelming response from the survey is that the app is hard to use, you may set a target to spend 10% of your engineering time on UX improvements. The latter would be an activity that supports the outcome.
Destination target [Goals]
These are longer-term goals used to meet business objectives. The objectives can either be internal or external. Marketing could be building out a campaign in support of a release, and sales may need a new feature added to break into a new industry. Goals are typically viewed as the breakpoints on a roadmap. Regardless of the time or energy required, your team is working to build or deliver something. The source of the work may differ depending on the type of goal, but the general tracking process is the same. Feature delivery is the goal that most of us will recognize, but this also includes important pure-engineering efforts as well. Tech debt payoffs, upgrading to the next major version of a library, and migrating a production database would all be types of goals that may not show up on the normal product roadmap.
Customer success happens when all departments are tracking outputs toward those larger company goals. But how should engineering set up KPIs to make sure customer success happens? Keep reading…
Steps for Building Your KPI Tree and Focusing on Outcomes
If KPIs are defined as “what you need to do and in what timeframe,” meeting KPIs can easily become a swirl of urgent but ultimately unimportant tasks. That’s why your KPIs should be set up with purpose. Here are the steps you need to take to match outputs with the overall desired outcomes of your organization.
1. Know the lineage back to the original company outcome or goal.
Whatever you’re measuring, from running costs to average downtime to the number of comments per pull request, it needs to track back to the company goal. How are costs affecting profitability? Is downtime getting shorter? Is customer experience getting better all the time? KPIs need to be specific and linked to overall goals.
2. Be clear about how each engineering KPI impacts and supports the overarching, company KPIs
Each engineering KPI should be developed with the same level of rigor that was used to develop the organizational and company KPIs. For instance, it should be clear that measuring “mean time to recovery (MTTR)” helps meet your SLA target. Also, each KPI should be able to be fully explained by its derivative KPIs. If you’re only tracking MTTR, but not SLAs for the discovered defects, your target outcome will be lost.
3. Each KPI should be measurable or fed by measurable KPIs.
While your KPIs may not have well-defined numerical values or concrete actions, they should be at the next level down. For example, if a goal is to communicate information about new features and their delivery date for marketing to plan promotions, the KPIs below that should have measurable metrics. The associated trackable activities for engineering would be to document releases in Jira, outline what was included in the Epic, and create a Confluence page with feature screenshots and full descriptions. Your marketing team can validate that it meets their requirements.
4. Targets for individual teams should be within their control.
A KPI is not a KPI if it isn’t a target your team can control. Specificity is key here. Broad goals like reducing hosting costs can’t be pinned to a single task owner. Keeping things broad means no one takes ownership.
Instead, set narrow, achievable goals. Let’s say your data pipeline team thinks they can make things 30% more efficient, and your database team thinks that caching can save you 20% in disk IOP (Input/Output Operations Per Second) spend, translating to a 10% reduction with 80% confidence. For a solid KPI, set your commitment to a 5% reduction in 3 months, and start delegating the lower targets to the teams and individuals that will implement the changes. This broad goal can now be acted on by specific teams and has defined outcomes. Those teams will more than likely set internal targets to reach, and will understand why they’re doing it and what the outcome should be.
FINAL THOUGHTS
KPIs are less about perfection and more about continuous improvement. Implementing an outcomes-based KPI strategy can help provide the gravitational field that keeps engineering aligned with the rest of the company instead of being caught up in the vague pressures of daily operations. Stay tuned for more information on how to develop and track engineering KPIs in this ongoing series. Part II will focus on common pitfalls of developing and tracking KPIs and how to overcome them.
What are your thoughts on building outcome-based KPIs? How are you tracking against your desired outcomes? If you need a helping hand in improving your engineering operations, reach out to us to see how we can help or schedule a demo to see our solution in action.