What Happened to the API Economy?.

TL;DR: We have an API Economy today, but it’s not exactly what people were expecting 10 years ago when it started. The good news is we’re starting a new phase of innovation that will change things up. Let’s talk about why and how.

At the start of the 2010s, the Cloud was transforming everything. SaaS was everywhere. CRMs like Salesforce, Ticketing Systems like Zendesk, and e-commerce solutions like Shopify were all transforming every aspect of the economy.

Every business process was turning into a digital experience, delivered through the browser. All you needed was an internet connection and a desktop or mobile device.

But what was next? How would this innovation continue or even accelerate when every business process and consumer experience has turned into a SaaS product?

Enter the API economy, where every application turns into a platform!

The Siren Song of the Platform

You build a number of tools, you set some rules and then you allow third-party vendors to come and build new experiences on top of your business, including access to the customer base.

Think of what Apple did with the iPhone. First you sell the product (iPhone.) Then, you create an environment where businesses and individuals can build additional functionality that you can sell to your users (the app store.) The iPhone turns into a platform and the relationship between consumer, Apple, and app store vendor, into a marketplace.

Everyone saw Apple and wanted to repeat the success of the appstore.

But how do you do that with a SaaS product delivered over the Internet?

That’s where the API Economy and APIs in general became really important.

Turning the Product Inside Out

Most commonly, an API refers to a mechanism for one piece of software to communicate with another. For over 30 years, companies have been innovating ways to interoperate with each other. This includes technologies and specifications like RPC, RMI, CORBA, SOAP, REST and GraphQL.

APIs have been used for years to build applications. But they were mainly used internally. Engineers were architecting large systems by decomposing them into smaller parts and using APIs, to create contracts on how the systems would communicate and finally compose into the overall system that they had designed.

The main idea behind the API economy was to take this esoteric piece of technology and expose it to outside stakeholders.

What if the API that our UI is using to fetch and present the tickets to our user, is also exposed for arbitrary software services to consume?

If we do that, then we allow anyone to build on top of our infrastructure and by exposing the guts of our systems, we turn our product into a platform for others to build on top.

Going Through the Hype Cycle

By now you should understand why we wanted to build platforms and that we had the technical means to do it, what’s next? Time to create demand and supply.

Find the people who are willing to buy (build on top of the platform) and the people who are willing to sell (offer the platform) and connect them together.

This is where things start getting interesting and less dry. We start communicating colorful visions of the future, of how the world will be different and better by adopting new paradigms.

How we can be more creative than ever before and how many opportunities are created and waiting for smart people to seize them.

Now, also imagine that there are as many platforms as SaaS applications out there. Developers can build on top of each one but also by combining the platforms together.

That’s how the API Economy was branded and marketed.

Educating people on the things that can be achieved by engaging in a market where there’s a technology provider and developers who are using this technology to build new products for consumers.

As always, the vision describes reality in a very asymptotic way. It never matches and this is a good thing. It means that there’s still space for improvement and progress.

The asymptotic nature of vision vs reality is not marketing lying, it’s just how the world works and how humanity is able to create progress.

How is reality different compared to that vision?

Going Through the Trough of Disillusionment

Someone would expect that the API Economy became a perfect world where every technology is exposed as a service (API) and developers are free to do whatever they want with them and build amazing things.

Maybe someone builds on top of Salesforce the next generation of CRMs or someone else delivers a completely different paradigm of managing customer tickets on top of Zendesk.

Reality is a bit different than that though. What happened is that SaaS products turned into platforms by exposing APIs.

But these APIs and platforms were designed in a way to protect the SaaS product and imposed a very specific worldview to the developer.

Yes, you can interact with the Zendesk and Stripe data models through the API but the world you’ll live in is a very strict and well defined world by these data models and APIs.

But, If you want to change the data model, you can’t.

Also, if you want to change the way your app interacts with the platform, you are on your own.

To be honest, what I described is not necessarily a bad thing. Companies wanted to create gated market places, to replicate the success of Apple.

But how do you do that when there’s no hardware and consumer branding moat like the iPhone? Software can easily be copied, how do we expose it without destroying our position in the market?

Because of all the above, the API Economy turned into an engine for enhancing existing products. If you are using a CRM, there’s probably an app that allows you to record and track phone calls within the CRM.

If you are using Shopify, there are plenty of options for apps that allow you to automate marketing outside Shopify.

The platforms benefited greatly from this. They increased the value of the platform with new features that wouldn’t be possible to build and can profit from revenue sharing, similar to Apple.

Developers figured out more places to build and sell apps and make some money.

Third party vendors found new ways to generate leads and increase their market share. Getting on one of these platforms exposed their products to the platform’s entire customer base and that fueled growth.

But, no matter how good the current paradigm is, it’s not the world changing vision that the API Economy was promising ten years ago.

And just like that, we reach the Plateau of Productivity. The current paradigm has reached its peak and its limitations are more than obvious.

Time to iterate, innovate and build new things!

How the World Looks Today

As we said, building on these platforms is a great opportunity as long as you stay confined within the limits of that platform.

Building experiences that combine more than one of these platforms becomes hard. It’s also impossibly hard for not very technical people.

In the best case scenario you end up with something like Zapier. Create some canned and limited experience, combining different platforms with an awful developer experience.

Have you ever tried to debug a Zap that is broken?

Today though, we have platforms like Retool that are promising to offer to a broader audience the ability to build applications. But how can this happen without fixing the API Economy?

The experience will still suck as the developer will still have to deal with all the issues around working with the different platform APIs that exist today.

The reason that tools like Retool are focused mainly on internal applications, is because of the current state of the API Economy.

It’s unbelievably difficult to build a scalable experience while relying completely and only on the APIs of different platforms.

Different latencies, different rate limits, different ways of handling changes, different ways of error handling and all while the developer has zero control over the data model of each of these platforms.

The control over the data model is often overlooked, but in my opinion, this issue is critical.

Building a user facing app is primarily an exercise in modeling a problem appropriately and managing the state of that model.

A CRM cannot operate on top of a Ticketing System data model efficiently over the long term.

Even if we fix all the technical limitations of REST APIs, we still have the major issue of data modeling control and handling state.

How can we deal with that?

The Future

In 2015, something happened in the analytics industry that laid the foundation for the changes we will see in building apps and using SaaS platforms.

The emergence of the Cloud Data Pipeline changed everything by exposing the idea that you can take your data out of any SaaS app, push it into a data warehouse, add your production data there and do whatever you want with the data.

The problem of how to perform proper analytics when your sales data is on Salesforce and your billing data is on Stripe was solved.

But most importantly, people realized that when your data is freed from the confines of a platform, opportunities exist.

This emergence led to the 2020 hype around reETL. The allure of taking the data out of the data warehouse, pushing it back into SaaS applications, and enriching the experience that SaaS platforms can deliver reached peak hype.

The taboo of the siloed data on the different SaaS vendors was broken. However, we are still not able to take this data and manage its state in whatever way we want, while delivering a new experience to thousands of users.

There are a couple of reasons for that.

First, the liberation of data from the SaaS vendor happened inside the context of analytical workloads.

These workloads have very different requirements compared to their transactional counterparts.

A data warehouse is a database that is optimized for low concurrency, stable workloads and a few but very long running queries.

This is not the environment where you want to build transactional systems used by thousands of users.

Second, rETL was built with some very specific use cases in mind, mainly to support revops and marketing-ops needs.

Here you mainly want to take the results of the analytics run on a warehouse and mainly share them with the users of the SaaS platforms.

For example, create an audience and upload it to Marketo or calculate a lead score and load it to Salesforce.

These use cases are important but they are very batch in nature and do not require from the database system to interact with many concurrent users as it happens in a typical SaaS application.

What we need and haven’t achieved yet, is a way to abstract all these APIs behind a state management system that behaves like a transactional database.

Just like the MySQL and Postgres instances that run the millions of web apps out there.

This is a completely different technical challenge compared to what the ELT and rELT products deliver today.

The opportunity

The market is ready for a new version of the API Economy vision.

One that will be much much closer to what was promised 10+ years ago.

The technology for this vision also already exists today.

It’s a matter of visionary people building the right products, and there’s no better time to do that.

This market of App building - automation, API Economy or whatever you want to call it is ripe for disruption and I have a feeling we will see some amazing technology emerging soon.

All this is why I’m so excited about the innovation that companies like Sequin and Convex are building.

They both deliver products that can disrupt the current market state and we should all be watching.

Acknowledgements

I’d like to thank Anthony Accomazzo and Eric Goldman for their feedback and help in writing this post and for being so inspirational with the great work they are doing at Sequin.

Convex has a very interesting interpretation of Ingress/Egress. If you check their pricing page, you will see that they build features for both importing and exporting data from/to the platform, based on Airbyte.