Metrics can be a murky area for many developer programs. There is no shortage of things to measure: API calls, SDK downloads, revenue generated by developers, signups, web visitors, event attendees, and many more. However, a disconnect between a business’s goals/objectives and the numbers indicated by metrics, can occur when programs either aren’t measuring the right things, or aren’t making effective use of the metrics they do collect.
The key to developer program metrics is alignment. Overarching the developer program are the company’s business goals and objectives that the program should contribute to. Program metrics should be aligned with these objectives. In short, it’s how the developer program will contribute to the success of the overall business.
When properly aligned with business goals, program metrics allow you to:
Communicate priorities to and motivate developer program teams, and show them how they contribute to the overall business.
Measure progress towards your goals and objectives.
Forecast if those goals and objectives can be reached.
Make improvements to your strategy, tactics, and resources.
Results Versus Activities
These program metrics are then aligned with activity metrics and community metrics. A common misstep is to focus program metrics solely on activities: things like number of events participated in, blog posts written, and so on.
Program metrics should focus on results and outcomes that contribute to the overall business, such as:
Number of API calls or SDK downloads.
Number of active developers.
Number of apps/services successfully integrating or using an API.
Developer satisfaction/net promoter score.
Program metrics must reflect your developer program strategy: is it sound, and delivering value to your business.
Underneath program metrics lie activity and community metrics. Activity metrics allow you to track the things you are doing to achieve your program goals and metrics, while community metrics track your progress in building a developer community. When combined, they show the effectiveness of your tactics, and how they are contributing to the program metrics.
Activity metrics take on several different forms, all focused on the things your program and its team does:
Functional metrics around certain job roles, such as promotion or evangelism.
Deliverable metrics, such as number of support articles written or hackathons attended.
Developer cycle stage metrics, such as the time to the first “Hello World” or the percentage of apps that pass a certification process.
Similarly, there are many potential community metrics, which illustrate the size and depth of your developer community and how your program interacts with and supports it. These can include:
Metrics that show developer adoption, such as the number of mentions on Stack Overflow, the number of GitHub project starts, or the number of stars/forks for company-hosted GitHub projects.
Metrics that show community support, such as the percentage of answered/unanswered questions on Stack Overflow.
When applied correctly and taken together, your metrics should help you see how your program strategy is working and contributing to the overall business goals (in the program metrics), and whether your tactics are contributing towards your program’s success (in the activity and community metrics).
Keep in mind there is no one specific correct set of metrics for every program; the best fit differs from program to program based on the overall business, its strategy, program resources, and more. But every metric should pass the “So what?” test. Ask “So what?” after every metric. What does the number tell you? What does it mean? Does it tell you something you can act on, or show how what’s being measured contributes to your overall success? If you can’t answer these questions, chances are the metric isn’t relevant.
Metrics in Action
Beyond simply showing that your developer program is doing 'something', metrics can be extremely useful in helping to make adjustments to your strategy and tactics. Take for example the chart below, showing the number of developers that progress through the various stages of a developer program:
The chart shows the number of developers that enter this process (1000), and some examples of the metrics that could be used to measure success at each stage. Notice that as the developer travels further through the process, the metrics evolve from activity or community-driven ones into higher-level program metrics: this is that alignment we’ve discussed, in action.
By having the right metrics in place at each stage, we can also see where we’re losing developers in their journey, or where our program and its tactics need enhancement. The chart below shows some general problems in each stage, and the metrics that can help illustrate them.
Using this approach can help you to get a better grasp on exactly where and how your program needs to improve.