Xing WangWe interviewed the product managers at a number of the larger API-first companies that are based in San Francisco. The companies are all publicly traded, have TTM revenue of more than $100M and are in the fields of billing, security, communications and workflow automation.
The PMs were asked what were their favorite tools and what API metrics they cared most about. Where possible we identified tools and metrics that were common across all market segments, excluding the (many) edge cases that you’d expect when your customer base numbered in the 1,000s.
Not surprisingly, our answers nicely segmented down into three classic areas: adoption, engagement and retention. Before we dive into those areas, we need to get the data into our analytics ecosystem.
Getting Started is Tough — Huge Amounts of Data
One of the most consistent take-aways is that the storage and data processing required to analyze billions of APIs calls is huge. Data lakes often grow so large that retroactive analyses have to be limited to just a few days, or even the last few hours.
In many cases, the first step companies take is to dump the unstructured API data, or the entire raw dump of their syslog, into a data lake in Amazon Redshift or Splunk. From there, the data infrastructure team pulls out the syslog events the PM is interested in and passes them to the data warehouse, often in Snowflake, where it’s more easily queryable. Here, the actual processing and aggregating of metrics takes place, often under the auspices of the Business Intelligence team, the PMs and perhaps even engineers.
Adoption: Tools and Metrics
For the majority of api-first companies we talked to, one of the first, and arguably most important metrics that PMs track, is developer activation. In general, the steps in product adoption are straightforward:
- Sign up for an account
- First API Call
- Deploy a Working API
Our cohort of established API-first companies use a Tableau or Looker dashboard that displays how many people are signing up, of those sign ups how many are logging in, of those logins how many are creating an app, and of those apps how many mint API tokens.
Predominantly PMs’ OKRs are devoted towards increasing the developer activation rate and making sure the time to activation is decreased. Since devs could stay in a single funnel stage for days or longer, it’s important to track both the conversion rate for each step, and also the time it takes to reach the next step.
If the normal sales cycle is 90 days, the PMs like to look at the quartiles: what’s the fiftieth quartile doing, what’s the seventy-fifth quartile doing, and then they use that as a proxy to determine how useful are their SDKs and documentation.
Once the API is adopted, PMs want to see usage increase leading to a paid plan, a highlighting of popular endpoints and an ability to identify missing features. At this stage the buying motion of the customer bifurcates depending on their company size: large enterprises or SMBs/startups.
Engagement: Enterprise Customer Tools and Metrics
For the most part, the majority of developers are asked by their leadership to evaluate the API product as a possibility. Somethings they create a developer org and try out all of the features. And then when their company decides to sign on the dotted line, they actually end up provisioning a separate account. Mapping of the developer org to the paid, and tying the revenue accounts in Salesforce, is not always super clear. So instead of trying to solve that mapping problem, PMs sometimes just focus on more adoption, since adoption is a really good proxy for whether or not customers are going to use the product.
API Tokens By Acquisition Channel
Most companies see utility in tracking activity in the user facing console to help grow usage and engagement. When customers are signing up, configuring their account, managing which APIs are available, or switching on and off features, they go through the management web interface. If your API monitoring tool is not user-centric (i.e. it doesn’t have the ability to drill down into the API call and identify which user and company it’s attributed to), then PMs have to deploy analytics tools like Heap or Google Analytics 360. These tools are then configured to associate users on the web interface with the API calls others in their organizations might be making.
PMs can then track marketing channel attribution to the respective Google or Facebook ad. They can track from account creation, through when the customer converted into a paid plan, and onto when they first started making API calls.
In a user-centric tool, UTM parameters are monitored in effectively the same way as HTTP status response codes. This enables grouping of API tokens by UTM source or UTM campaign, to better understand which marketing channels contribute to engagement.
Weekly Active API Tokens
The number of distinct tokens accessing the API in a given week, the Weekly Active Tokens (WAT), is one of the best north star metrics that PMs use to track their products. Unlike infrastructure metrics like Uptime, SLOs or Requests per Minute that are aligned with engineering goals, WAT is directly aligned with the business goal of driving adoption and increasing engagement. To calculate WAT the data infrastructure team needs to pull out the relevant syslog event from Redshift and pass it into Snowflake. Once there, the BI team writes the SQL query and visualizes it in Tableau.
Since a single developer account can create multiple API tokens, such as for sandbox and production environments, a more accurate measurement would be Weekly Active Users or Weekly Active Companies. However, this requires analytics infrastructure that can link API tokens to a respective user or company account.
Number of Users
”Make it easy to invite others”
Some PMs found that there’s a direct correlation between accounts converting and number of users. More users often meant that the customer was more serious about the project. So PMs made a push to invite others to join in the sign-up flow, by saying things like “invite someone else to this project to help you do the work.” Often, an added bonus was that that’s another chance to get a corporate email from a user, since the inviter might not know the invitee’s Gmail, but would know their work email.
Engagement: SMB/Startup Self-Service Customers
In a self-service buying motion the customer’s an independent developer, at a five or 10 person startup or at an SMB, where he can just put in his CTOs credit card and start using the paid service right away.
It’s difficult to get additional insights from this group over and above what PMs do for enterprise accounts, since most devs vastly prefer the self-service route.
“It’s not an absolute statement but for the most part developers don’t want to talk to you, they’re averse to talk to sales and they don’t want to respond to emails. In fact, they often sign up with personal emails to try and hide who they’re working for,” PM in San Francisco.
However, a proxy for developer sentiment can somewhat be gleaned by looking at what the devs are using in the product, what they click on, what API calls they make and what their usage stats are from the API’s SDKs in GitHub.
Retention
Once the PMs had a good understanding of adoption and engagement, they looked to API product retention to find areas that required improvement. Product retention is a concept born out of revenue retention and requires segmenting the user base into cohorts such as via sign-up date. The PM tracks the percentage of each cohort returning to engage with your platform. In the example below API retention is grouped by user’s SDK. You can see that PHP has a far lower retention percentage than the other SDKs, implying that PHP is buggy, or it has a performance issue that requires fixing.
Another way to determine which product features to add or deprecate is to watch billing SKUs. Many APIs are divided into a set of SKUs, with each different activity type assigned its own single SKU. By looking at who is paying for what, it’s possible to identify which ones are being used and which are not.
Setting Up Metric Tracking is a Bear
The velocity for monitoring business intelligence is definitely a problem from the PMs’ point of view.
”It just takes way too long after putting in a request for a new metric to you get the statistics out the other end,” said a disgruntled PM.
It’s a five-step process to set up tracking of a metric. It involves making a request to the separate BI team who then have to triage the request, then pull it in and there’s often negotiations and politics involved. Representative steps include: 1) Does the data in question have an event? 2) If the answer is yes, then is it in the data warehouse? If the answer is no, then someone in the data infrastructure team needs to create a new syslog event and then pull it in. 3) Create the requirements for how the metric is to visualized in Tableau or change the reports. 4) The BI data team has to execute the request. 5) If BI can’t visualize it because they’re too busy or its outside their capabilities, then the PM will have to ask engineering to do a custom SQL query on the database itself.
User-Centric Agile Reporting
”PMs are never going to turn down a more actionable, or sort of more agile reporting toolset,” PM at a leader in security.
In many of the companies we interviewed they formed the DevEx group from scratch, just adopting whatever tools the BI team was using. Custom queries into data warehouses were built because there weren’t any off the shelf options. But since then tooling’s come a long way.
Today, API analytics tools, help everyone at API-driven organizations learn from their API data and make smarter decisions that drive growth.
We’re at the stage where product managers at API-first companies recognize that good tools can give them unique insights into developer success, and can be just as important to corporate success as reliable SDKs or complete documentation.
