The Fastly API is huge. We have lots of customers who want to interact with it using their chosen programming language but our small set of manually maintained clients was not sufficient to handle the job of our ever-evolving API. We needed a way to scale up our API client support, and OpenAPI was the answer.

We describe our API using the OpenAPI specification, which defines a standard, language-agnostic interface. The use of OpenAPI enables us to programmatically interact with the API metadata to produce content, code examples, tools and other valuable resources.

Here is an example of one of our API endpoints. It highlights how we define a GET request and what a successful (200 OK) response looks like:

info:
  title: Backend
  description: A backend is a server identified by IP address or hostname.
  version: 1.0.0

externalDocs:
  url: https://developer.fastly.com/reference/api/services/backend

servers:
  - url: https://api.fastly.com

paths:
  /service/{service_id}/version/{version_id}/backend:
    parameters:
      - $ref: "service.yaml#/components/parameters/service_id"
      - $ref: "version.yaml#/components/parameters/version_id"
    get:
      summary: List backends
      description: List all backends for a particular service and version.
      operationId: list-backends
      responses:
        "200":
          description: OK
          content:
            application/json:
              schema:
                type: array
                items:
                  $ref: "#/components/schemas/backend_response"
              examples:
                body:
                  value:
                    - $ref: "#/components/examples/backend_response"

HINT: To reduce duplication we often put common parameters and values into external files that can be referenced separately, we also do this for large nested sections that can otherwise make it difficult to focus on the critical points of an API’s specification (see $ref).

The problem: 700 endpoints, 7 languages

In order to enable customers to access our API within a programmatic environment, we provide API clients in multiple languages. Over time we produced multiple API clients, all varying in their level of API coverage, code quality and consistency.

There were multiple factors that meant our API clients didn’t all receive the care and attention they deserved, and so would suffer from code bugs and generally be out of date with our API. For example, we needed:

Engineers who knew multiple languages and had enough experience with them to not only be able to write code in these languages but to write idiomatic code.
The resources to continually revisit clients when the API changes, as well as the sheer amount of work to support such a large number of endpoints.

The plan

In 2020 the Developer Relations team took ownership of the API clients and we began devising a plan to make them more complete, consistent, and where possible, idiomatic. This wouldn’t be achievable by maintaining all the clients as separate projects, so the primary focus was to create and maintain common tooling to enable a broad stable of API clients to keep up with our API as it evolves.

We also aimed to allow our Fastly co-workers to easily contribute to the clients when an API changes, avoiding (a) DevRel having to try and implement all API changes into the clients, and (b) other teams having to learn different standards and structures for a bunch of different API clients in order to ship a feature - neither of which would be scalable.

The solution: OpenAPI Generator

To achieve these goals we investigated various options and ultimately decided on using a community-driven fork of the official Swagger CodeGen project: OpenAPI Generator (OAG).

OAG is a template-driven engine that can generate documentation, API clients and server stubs in different languages by parsing your OpenAPI specification.

How OAG works

OAG is a Java project that uses Mustache templates to configure each supported programming language. It provides a CLI openapi-generator-cli that will download the appropriate JAR file and invoke the java executable to run OAG.

To produce a new API client, we pass openapi-generator-cli our OpenAPI specification, tell it what language we want to use, and where to output the API client code. Here is a snippet from our Makefile that uses Docker to help get the Java CLI tool configured correctly:

LOCAL_ROOT:=/local
OPENAPI_GENERATOR_CLI_TAG=v5.4.0
OAPI_GENERATOR_CLI:=docker run --rm -v $(PWD):$(LOCAL_ROOT) openapitools/openapi-generator-cli:$(OPENAPI_GENERATOR_CLI_TAG)

$(OAPI_GENERATOR_CLI) generate \
    --global-property apiTests=$(GENERATE_TESTS_API),modelTests=$(GENERATE_TESTS_MODEL),apiDocs=$(GENERATE_DOCS_API),modelDocs=$(GENERATE_DOCS_MODEL) \
    -i $(LOCAL_ROOT)/$(BUILT_SCHEMAS_DIR)/$(COLLECTION) \
    -g $(GENERATOR) \
    -t $(LOCAL_ROOT)/$(TEMPLATES_DIR)/$(TEMPLATE) \
    -c $(LOCAL_ROOT)/$(TEMPLATES_DIR)/$(TEMPLATE).yaml \
    -o $(LOCAL_ROOT)/$(TEMPLATE)$(if [$(COLLECTION_NAME) == "fastly"],,/$(COLLECTION_NAME)) \
    >$(TEMPLATE)/.tmp.$(COLLECTION_NAME).log \
    2>$(TEMPLATE)/.tmp.$(COLLECTION_NAME).err.log

Why it’s a good choice

Unlike most other tools we investigated, OAG had good support for multiple languages while also supporting the OpenAPI 3.x specification, which is the major version our own API specification was written in (most other open-source tools at the time only supported 2.x).

OAG is also a very flexible tool allowing us to control various aspects of the generated code, as well as execute post-processors and the ability to define custom generator templates if none of the built-in templates fit our requirements.

What languages we chose and why

Prior to this initiative we had five API clients, so it was important that whatever process we came up with, would support at least Ruby, Python, PHP, Perl and Go*.

* This is our hand-coded Go client - the OAG version is still in the works.

On top of that list, we also wanted to produce greenfield JavaScript and Rust clients as these are popular languages our customers are using.

Although OAG was doing the hard work of generating API clients for lots of different languages and keeping them up-to-date with our API specifications, as you’ll see, this wasn’t a panacea.

Difficulties and solutions

There were a few issues we encountered. The first was we needed a custom build process to manage a large number of OpenAPI specification files (at the time we had approximately 120 and it was growing).

The openapi-generator-cli tool could only process one specification file at a time, which meant we needed to first aggregate our specifications into a single file (with the help of redocly).

To ensure we would be able to run the code-generation process within our Continuous Integration (CI) pipeline, we needed to use a Dockerised version of openapi-generator-cli (the coordination of which was a complex process). See the next section on how we auto-regenerate clients on spec change.

Another challenge we encountered was the quality of the community-driven language templates. For the most part, they worked but they didn’t appear to be well maintained over time, nor did every feature in our OpenAPI specification have a corresponding implementation in the language templates. In some cases, the language templates didn’t support the latest programming language version, and the generated documentation wasn’t as good as we needed it to be.

Additionally, the quality of our OpenAPI specification files would cause problems in the generator, by either triggering runtime errors in the underlying Java parser or in the generated code itself due to incompatibility with the language templates.

Each of these issues would result in hard-to-debug compilation errors within the code-generation pipeline, and so we would have to figure out which of the three potential layers the issue was stemming from:

The Fastly OpenAPI specification documents.
The openapi-generator mustache templates.
The openapi-generator language parsers (written in Java).

Sometimes we would tweak the OpenAPI specification such that the generator’s language parser would take a code path that didn’t trigger a Java error, while solving issues with the language templates would mean having to manually copy the source templates and modify them to suit our needs (this was made possible thanks to OAG supporting custom template files).

Essentially we had to iterate on both ends of the integration, a bit like trying to fit a square peg into a round hole, then gradually changing the shape of both the peg and the hole until they fit.

Whenever possible we would push fixes upstream to the open-source openapi-generator project so the wider community could benefit. This was our small way to say thank you to those who came before us and who provided us with these great tools to begin with.

We also did a lot of work to restructure the generated documentation (which to be fair was already thousands of times more detailed than what we had in our original hand coded API clients) to make it much easier for users to consume and understand the API client interface.

In some cases, we were unable to resolve the incompatibility so to help unblock ourselves we implemented a custom OpenAPI extension that integrates with our CI pipeline and enables us to exclude API endpoints at a very granular level by specifying if the exclusion should be for all endpoints within a specification file or a specific sub-section. We can also exclude all clients or just those for specific languages.

Examples of this look like

x-fastly-preprocess-exclude:
  - api-client  # exclude all clients

x-fastly-preprocess-exclude:
  - fastly-rust # exclude the rust client
  - fastly-js   # exclude the javascript client

Alpha releases

The next step after successfully generating an API client was to push the code to the relevant public GitHub repository and then publish the code to its module registry (e.g. crates.io for Rust, npmjs.com for JavaScript, pypi.org for Python etc).

We selected a target group of customers and internal stakeholders to help test out the new API clients and report any bugs in the code, invalid documentation, or missing functionality. There were a few minor issues reported but otherwise, the generated API clients were working pretty well.

Auto-regenerating the clients on spec change

The last piece of the puzzle was the ability to automatically regenerate an API client when any OpenAPI specification files were modified. We achieved this by adding two new CI pipeline flows across two separate repositories that would coordinate with each other.

At a high level, this looked like:

HINT: We use GitHub Actions for our CI/CD. You can learn more about it by reading our blog post “How Fastly deploys Gatsby CMS websites to GCS using GitHub Actions”.

We wrote a CI job within the GitHub repository containing our OpenAPI specification files. That job runs whenever changes are merged into the mainline branch and starts by validating and combining the specs.

Of course, when the API definition changes, it makes sense to regenerate all the API clients; not just because we want to keep our public clients up to date, but also because as part of reviewing open PRs, it's great to be able to grab a client and use it to actually test out your new API before we integrate its definition into our official OpenAPI spec.

GitHub Actions turns out to have an incredibly useful hook called workflow_dispatch, which allows a change in one repo to trigger an action in a different repo! We define a step that uses a conditional check to prevent it from being executed if there were no specification changes, otherwise, it triggers a workflow file within the GitHub repository that contains our API client generator logic/build process.

The CI job defined in the API client generator repo is configured with the workflow_dispatch event. It receives the branch and commit SHA from the OpenAPI specification repo, and it posts status updates back to the OpenAPI repo informing it of the build progress.

The next step in the API client generator CI job checks out the appropriate branch of the OpenAPI specification repo and uses it to build all the API clients.

Finally, the CI job sends a status update to the OpenAPI specification repo letting it know where the generated API clients were uploaded to.

Conclusion

Auto-generating our API clients has been a long project. The work started with the initial move to document our API using OpenAPI specifications. This enabled us to make a much better API reference on our Developer Hub that consumes those specification files and produces content from them.

The code generation itself hasn’t been easy: we’ve had to balance the complexity of coordinating changes across multiple repositories while handling errors over multiple layers, and producing API clients that are not only idiomatic to the language they’re built in but also correct and accessible.

That said, for us, this was a task that we needed to take to ensure we were providing our customers with the best possible tools and experience, and enabling them to interact with the Fastly platform more efficiently.

We’re now able to display code examples for each supported language in our API documentation, making it easy for customers to get started using our APIs.

Any time the API definition is improved, we end up improving the quality of the API documentation. Lastly, this work has encouraged our own developers to engage with the API definition.

Take a look at the clients and let us know what you think. We love hearing how customers are using our tools and our platform.

Better Fastly API clients with OpenAPI Generator