How do you set up a CI/CD pipeline for serverless applications?

This blog post will explain how to use Travis CI, The Serverless Framework and the AVA testing framework to set up a fully-automated build, deploy and test pipeline for a serverless application. It will use a real example of a production serverless application, built using Apache OpenWhisk and running on IBM Cloud Functions. The CI/CD pipeline will execute the following tasks...

• Run project unit tests.
• Deploy application to test environment.
• Run acceptance tests against test environment.
• Deploy application to production environment.
• Run smoke tests against production environment.

Before diving into the details of the CI/CD pipeline setup, let's start by showing the example serverless application being used for this project...

Serverless Project - http://apache.jamesthom.as/

The "Apache OpenWhisk Release Verification" project is a serverless web application to help committers verify release candidates for the open-source project. It automates running the verification steps from the ASF release checklist using serverless functions. Automating release candidate validation makes it easier for committers to participate in release voting.

The project consists of a static web assets (HTML, JS, CSS files) and HTTP APIs. Static web assets are hosted by Github Pages from the project repository. HTTP APIs are implemented as Apache OpenWhisk actions and exposed using the API Gateway service. IBM Cloud Functions is used to host the Apache OpenWhisk application.

No other cloud services, like databases, are needed by the backend. Release candidate information is retrieved in real-time by parsing the HTML page from the ASF website.

Configuration

The Serverless Framework (with the Apache OpenWhisk provider plugin) is used to define the serverless functions used in the application. HTTP endpoints are also defined in the YAML configuration file.

service: release-verfication

provider:
  name: openwhisk
  runtime: nodejs:10

functions:
  versions:
    handler: index.versions
    events:
      - http: GET /api/versions
  version_files:
    handler: index.version_files
    events:
      - http:
          method: GET
          path: /api/versions/{version}
          resp: http
...

plugins:
  - serverless-openwhisk

The framework handles all deployment and configuration tasks for the application. Setting up the application in a new environment is as simple as running the serverless deploy command.

Environments

Apache OpenWhisk uses namespaces to group individual packages, actions, triggers and rules. Different namespaces can be used to provide isolated environments for applications.

IBM Cloud Functions automatically creates user-based namespaces in platform instances. These auto-generated namespaces mirror the IBM Cloud organisation and space used to access the instance. Creating new spaces within an organisation will provision extra namespaces.

I'm using a custom organisation for the application with three different spaces: dev, test and prod.

dev is used as a test environment to deploy functions during development. test is used by the CI/CD pipeline to deploy a temporary instance of the application during acceptance tests. prod is the production environment hosting the external application actions.

Credentials

The IBM Cloud CLI is used to handle IBM Cloud Functions credentials. Platform API keys will be used to log in the CLI from the CI/CD system.

When Cloud Functions CLI commands are issued (after targeting a new region, organisation or space), API keys for that Cloud Functions instance are automatically retrieved and stored locally. The Serverless Framework knows how to use these local credentials when interacting with the platform.

High Availability?

The Apache OpenWhisk Release Verifier is not a critical cloud application which needs "five nines" of availability. The application is idle most of the time. It does not need a highly available serverless architecture. This means the build pipeline does not have to...

• Deploy application instances in multiple cloud regions.
• Set up a global load balancer between regional instances.
• Support "zero downtime deploys" to minimise downtime during deployments.
• Automatic roll-back to previous versions on production issues.

New deployments will simply overwrite resources in the production namespace in a single region. If the production site is broken after a deployment, the smoke tests should catch this and email me to fix it!

Testing

Given this tool will be used to check release candidates for the open-source project, I wanted to ensure it worked properly! Incorrect validation results could lead to invalid source archives being published.

I've chosen to rely heavily on unit tests to check the core business logic. These tests ensure all validation tasks work correctly, including PGP signature verification, cryptographic hash matching, LICENSE file contents and other ASF requirements for project releases.

Additionally, I've used end-to-end acceptance tests to validate the HTTP APIs work as expected. HTTP requests are sent to the API GW endpoints, with responses compared against expected values. All available release candidates are run through the validation process to check no errors are returned.

Unit Tests

Unit tests are implemented with the AVA testing framework. Unit tests live in the unit/test/ folder.

The npm test command alias runs the ava test/unit/ command to execute all unit tests. This command can be executed locally, during development, or from the CI/CD pipeline.

$ npm test

> release-verification@1.0.0 test ~/code/release-verification
> ava test/unit/

 27 tests passed

Acceptance Tests

Acceptance tests check API endpoints return the expected responses for valid (and invalid) requests. Acceptance tests are executed against the API Gateway endpoints for an application instance.

The hostname used for HTTP requests is controlled using an environment variable (HOST). Since the same test suite test is used for acceptance and smoke tests, setting this environment variable is the only configuration needed to run tests against different environments.

API endpoints in the test and production environments are exposed using different custom sub-domains (apache-api.jamesthom.as and apache-api-test.jamesthom.as). NPM scripts are used to provide commands (acceptance-test & acceptance-prod) which set the environment hostname before running the test suite.

"scripts": {
    "acceptance-test": "HOST=apache-api-test.jamesthom.as ava -v --fail-fast test/acceptance/",
    "acceptance-prod": "HOST=apache-api.jamesthom.as ava -v --fail-fast test/acceptance/"
  },

$ npm run acceptance-prod

> release-verification@1.0.0 acceptance-prod ~/code/release-verification
> HOST=apache-api.jamesthom.as ava -v --fail-fast  test/acceptance/

  ✔ should return list of release candidates (3.7s)
    ℹ running api testing against https://apache-api.jamesthom.as/api/versions
  ✔ should return 404 for file list when release candidate is invalid (2.1s)
    ℹ running api testing against https://apache-api.jamesthom.as/api/versions/unknown
  ...

  6 tests passed

Acceptance tests are also implemented with the AVA testing framework. All acceptance tests live in a single test file (unit/acceptance/api.js).

CI/CD Pipeline

When new commits are pushed to the master branch on the project repository, the following steps needed to be kicked off by the build pipeline…

• Run project unit tests.
• Deploy application to test environment.
• Run acceptance tests against test environment.
• Deploy application to production environment.
• Run smoke tests against production environment.

If any of the steps fail, the build pipeline should stop and send me a notification email.

Travis

Travis CI is used to implement the CI/CD build pipeline. Travis CI uses a custom file (.travis.yml) in the project repository to configure the build pipeline. This YAML file defines commands to execute during each phase of build pipeline. If any of the commands fail, the build will stop at that phase without proceeding.

Here is the completed .travis.yml file for this project: https://github.com/jthomas/openwhisk-release-verification/blob/master/.travis.yml

I'm using the following Travis CI build phases to implement the pipeline: install, before_script, script, before_deploy and deploy. Commands will run in the Node.js 10 build environment, which pre-installs the language runtime and package manager.

language: node_js
node_js:
  - "10"

install

In the install phase, I need to set up the build environment to deploy the application and run tests.

This means installing the IBM Cloud CLI, Cloud Functions CLI plugin, The Serverless Framework (with Apache OpenWhisk plugin), application test framework (AvaJS) and other project dependencies.

The IBM Cloud CLI is installed using a shell script. Running a CLI sub-command installs the Cloud Functions plugin.

The Serverless Framework is installed as global NPM package (using npm -g install). The Apache OpenWhisk provider plugin is handled as normal project dependency, along with the test framework. Both those dependencies are installed using NPM.

install:
  - curl -fsSL https://clis.cloud.ibm.com/install/linux | sh
  - ibmcloud plugin install cloud-functions
  - npm install serverless -g
  - npm install

before_script

This phase is used to run unit tests, catching errors in core business logic, before setting up credentials (used in the script phase) for the acceptance test environment. Unit test failures will halt the build immediately, skipping test and production deployments.

Custom variables provide the API key, platform endpoint, organisation and space identifiers which are used for the test environment. The CLI is authenticated using these values, before running the ibmcloud fn api list command. This ensures Cloud Functions credentials are available locally, as used by The Serverless Framework.

before_script:
  - npm test
  - ibmcloud login --apikey $IBMCLOUD_API_KEY -a $IBMCLOUD_API_ENDPOINT
  - ibmcloud target -o $IBMCLOUD_ORG -s $IBMCLOUD_TEST_SPACE
  - ibmcloud fn api list > /dev/null
  - ibmcloud target

script

With the build system configured, the application can be deployed to test environment, followed by running acceptance tests. If either deployment or acceptance tests fail, the build will stop, skipping the production deployment.

Acceptance tests use an environment variable to configure the hostname test cases are executed against. The npm run acceptance-test alias command sets this value to the test environment hostname (apache-api-test.jamesthom.as) before running the test suite.

script:
  - sls deploy
  - npm run acceptance-test

before_deploy

Before deploying to production, Cloud Functions credentials need to be updated. The IBM Cloud CLI is used to target the production environment, before running a Cloud Functions CLI command. This updates local credentials with the production environment credentials.

before_deploy:
  - ibmcloud target -s $IBMCLOUD_PROD_SPACE
  - ibmcloud fn api list > /dev/null
  - ibmcloud target

deploy

If all the proceeding stages have successfully finished, the application can be deployed to the production. Following this final deployment, smoke tests are used to check production APIs still work as expected.

Smoke tests are just the same acceptance tests executed against the production environment. The npm run acceptance-prod alias command sets the hostname configuration value to the production environment (apache-api.jamesthom.as) before running the test suite.

deploy:
  provider: script
  script: sls deploy && npm run acceptance-prod
  skip_cleanup: true

Using the skip_cleanup parameter leaves installed artifacts from previous phases in the build environment. This means we don't have to re-install the IBM Cloud CLI, The Serverless Framework or NPM dependencies needed to run the production deployment and smoke tests.

success?

If all of the build phases are successful, the latest project code should have been deployed to the production environment. 💯💯💯

If the build failed due to unit test failures, the test suite can be ran locally to fix any errors. Deployment failures can be investigated using the console output logs from Travis CI. Acceptance test issues, against test or production environments, can be debugged by logging into those environments locally and running the test suite from my development machine.

Conclusion

Using Travis CI with The Serverless Framework and a JavaScript testing framework, I was able to set up a fully-automated CI/CD deployment pipeline for the Apache OpenWhisk release candidate verification tool.

Using a CI/CD pipeline, rather than a manual approach, for deployments has the following advantages...

• No more manual and error-prone deploys relying on a human 👨‍💻 :)
• Automatic unit & acceptance test execution catch errors before deployments.
• Production environment only accessed by CI/CD system, reducing accidental breakages.
• All cloud resources must be configured in code. No "snowflake" environments allowed.

Having finished code for new project features or bug fixes, all I have to do is push changes to the GitHub repository. This fires the Travis CI build pipeline which will automatically deploy the updated application to the production environment. If there are any issues, due to failed tests or deployments, I'll be notified by email.

This allows me to get back to adding new features to the tool (and fixing bugs) rather than wrestling with deployments, managing credentials for multiple environments and then trying to remember to run tests against the correct instances!