KDeciding whether to hire and build an API analytics platform vs purchasing from a third party vendor can be a daunting task. Not only do you need to investigate ROI, you also have to navigate politics and may run into Not Invented Here syndrome, among other things. In the long run, by purchasing a ready made solution like Moesif, your product and engineering teams will be able to focus on what they do best: building products that customers love.
Initial cost of building an API analytics platform
The cost of building an API analytics system depends on data volume and feature complexity but can be broken down into three areas:
- Data processing infrastructure
- Visualization, reporting, and integrations
- Security and compliance features
Data processing infrastructure
Designing and building a system capable of tens or hundreds of millions of API calls is not easy. It requires investment in good architecture for high-availability data collection, data pipelines and aggregation, and storing data securely. Care needs be given to performance to ensure your analytics system does not slow down your APIs or cause an outage, leading to lost revenue.
From what we've seen, the most common requirements for API analytics systems are handling 25 million API calls a month, 100 million API calls a month, and over 1 billion API calls a month. As the analytics system scales to higher volume, it can easily cost over a million dollars.
| Monthly Volume | Team Required | Build Time | Build Cost for Infrastructure |
|---|---|---|---|
| 25 million API calls | 6 people | 13 weeks | $154,296 |
| 100 million API calls | 13 People | 26 weeks | $668,616 |
| 1 billion API calls | 19 people | 33 weeks | $1,240,302 |
The cost of each team member is based on the national average salary for a data engineer which was $102,864 / year in February 2020 on Glassdoor.
Visualization, reporting, and integrations
Besides data volume, initial cost is also dependent on who is leveraging the reporting and will the data be truly actionable. Will only a few engineers be using the analytics system and are happy querying via raw SQL, or do you expect other users? Many engineering leaders looking to implement an API analytics system would like the data accessible by decision-makers across the company including product, marketing, support, and customer success. Unless the analytics system has infrastructure required to enable self-service access even by non-technical users, the data team will still become the bottleneck, slowing decision making and experimentation which can cause lost market share. This can include both a visualization tool that enables teams to create and experiment with their own dashboards and metrics, but also connectors with tools like Salesforce, HubSpot, and Segment, to make the data actionable.
There are multiple options when choosing a business intelligence tool which sit on top of your processed data in your warehouse.
| Number of Users | Person-weeks for visualization | Connectors Required | Person-weeks for connectors | Build Cost for Visualization and Integrations |
|---|---|---|---|---|
| 10 Team Members | 5 weeks | 2 | 6 weeks | $21,758 |
| 50 Team Members | 5 weeks | 7 | 14 weeks | $51,431 |
| 150 Team Members | 5 weeks | 21 | 42 weeks | $134,513 |
A BI visualization tool like Tableau or Looker requires an estimated 5 person-weeks to implement with your data infrastructure and set up the reporting. Each additional connector to a tool like Salesforce or Hubspot requires an estimated 3 person-weeks. We've usually seen each team of 7 requires a new connector for their tool of choice.
If the analytics system requires real-time alerting and anomaly detection, these should also be factors. If it takes 24 hours for a product owner to realize something is wrong with a customer flow, then revenue can be lost.
Security and compliance
If your business has security and compliance requirements, these costs should be accounted for also. This can include legal requirements such as having security audit logs and data breach detection along with mechanisms in place to comply with regulatory requirements such as General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA) and ways to scrub Personally identifiable information like credit card numbers. This means you may need to involve legal and security teams to review your analytics system to ensure you're not exposing too much risk to the business.
Your company may also have other requirements for internal systems such as enterprise single-sign on and breached password detection to aid account management. Because your analytics system contains a large amount of customer data, good practices include leveraging role based access control and custom permissions to prevent data leakage along with systems in place to monitor if a breach does occur. Many visualization tools have RBAC in place, but can be easily circumvented by anyone who has direct access to the data infrastructure.
Below are typical costs for security related features:
| Feature | Person-weeks | Build Cost for Security and Compliance |
|---|---|---|
| Security Audit Logs | 3 weeks | $5,934 |
| Data Scrubbing/GDPR & CCPA Support | 4 weeks | $7,912 |
| Data Breach Detection | 4 weeks | $7,912 |
Ongoing cost of maintaining an API analytics solution
Once the system is live, there are still high ongoing costs for a system processing terabytes of data and usually come from three areas:
- Compute and storage cost
- License fees for visualization tools
- Ongoing maintenance and fixes
Contrary to popular assumption, maintenance cost can grow significantly over time due to engineers moving on to other projects requiring ramp up time. Many companies implement an API analytics system to drive adoption and growth of the API business, yet this growth can directly correlate to additional compute cost. If technical debt grows, the system will soon require a follow up investment to handle the additional load by adding compute resources and leveraging newer methodologies in data processing.
Compute cost
From our own measurement, 100 Million API Calls requires on average 1 Terabyte in storage. If one year's worth of data history is required, 100 Million API Calls / month will require 12 Terabytes annually.
As of February 2020, AWS Redshift on-demand pricing is $0.85 per Hour for a 2TB HDD, 4 core VM and $6.80 per Hour for a 16TB HDD, 36 core VM.
Below cost assumes data is kept for one year and is then deleted.
| Monthly Volume | Storage Required | Annual Cost for Compute and Storage |
|---|---|---|
| 25 million API calls | 4 Terabytes | $14,902 |
| 100 million API calls | 12 Terabytes | $44,706 |
| 1 billion API calls | 120 Terabytes | $418,308 |
Visualization cost
Majority of cost for visualization and reporting is in license fees for a Business Intelligence tool such as Tableau or Looker.
As of February 2020, each Tableau Creator seat costs $70 per month, billed annually.
| Number of Users | Annual Fee for Visualization Tooling |
|---|---|
| 10 Team Members | $8,400 |
| 50 Team Members | $42,000 |
| 150 Team Members | $126,000 |
Maintenance cost
Ongoing annual maintenance is typically 25% of initial build time for a complex system such as an analytics service, but can go higher. This maintenance includes bug fixes (including critical security fixes), feature requests from various business units, upgrading outdated software, performance optimization as API traffic increases, and fixing data quality issues.
Support is not included, but this should be accounted for especially if the analytics system will be used company-wide by business users.
| Monthly Volume | Team Required | Person Days | Annual Cost for Maintenance |
|---|---|---|---|
| 25 million API calls | 6 people | 16 days | $37,980 |
| 100 million API calls | 13 People | 32 days | $164,582 |
| 1 billion API calls | 19 people | 41 days | $308,196 |
Total Cost
The all in first year cost includes the cost to build the data infrastructure, cost to build the necessary integrations to connect with existing tools,
compute and storage costs, and any annual license fees for visualization tools.
First year cost
| Monthly Volume | Number of Users | Build Cost for Infrastructure | Build Cost for Visualization and Integrations | Cost for Compute and Storage | Fees for Visualization Tooling | Total First Year Cost |
|---|---|---|---|---|---|---|
| 25 million API calls | 10 Team Members | $154,296 | $21,758 | $14,902 | $8,400 | $199,356 |
| 100 million API calls | 50 Team Members | $668,616 | $51,431 | $44,706 | $42,000 | $806,753 |
| 1 billion API calls | 150 Team Members | $1,240,302 | $134,513 | $418,308 | $126,000 | $1,919,123 |
Ongoing yearly cost
Ongoing yearly cost include maintenance cost to maintain the internal system includes bug fixes and security fixes, along with annual fees in compute, storage, and visualization tools.
| Monthly Volume | Number of Users | Annual Cost for Maintenance | Cost for Compute and Storage | Fees for Visualization Tooling | Total Ongoing Yearly Cost |
|---|---|---|---|---|---|
| 25 million API calls | 10 Team Members | $37,980 | $14,902 | $8,400 | $61,282 |
| 100 million API calls | 50 Team Members | $164,582 | $44,706 | $42,000 | $251,288 |
| 1 billion API calls | 150 Team Members | $308,196 | $418,308 | $126,000 | $852,504 |
Why purchase Moesif?
Companies purchasing Moesif over a homegrown build can realize 5x to 10x in cost savings in both the initial setup along with ongoing annual cost. As a best-in-class API analytics service, Moesif is constantly adding advanced features like conversion funnels and retention analysis along with leveraging machine learning to empower teams to make good decisions quickly without
getting buried in metrics and reporting.
By purchasing Moesif, Engineering teams have extra bandwidth to focus on what they do best: building a great product customers love, instead of getting buried supporting legacy homegrown analytics systems.
Moesif customers also gain our expertise in growing API platforms. Enterprise customers have access to our API experts to aid your internal teams in defining their success criteria while supporting them when they have questions on how to best use the platform.
Looking for a modern API analytics solution?
This article was written by Moesif founder and CEO Derric Gilling for the Moesif blog.
