Taavi RehemägiIf you're like me, a sucker for event-driven programming, you'll want to continue reading. Today we'll take a look at triggering AWS Lambda functions from AWS SNS messages. I've covered a few interesting topics regarding serverless architectures and AWS already, but nothing like this yet. Dig down, and get ready. Let's go.
TL;DR
- What are we building?
- What is AWS SNS?
- Build the API with the Serverless Framework
- Deploy the API to AWS Lambda
- Test the API with Dashbird
**Note: All the code from this tutorial is already on GitHub if you want to check out the end result right away.
What are we building?
Our focus will solely be on the steps to create the infrastructure components our app will need. The code itself will just mimic the behavior of a random complex computation. I've chosen a recursive function that calculates the factorial of the number passed to it. Here's a nice diagram, because diagrams are awesome of course!
The init function is the only exposed function, which is hooked up to API Gateway. It takes a single number parameter which it validates, upon success, it publishes an SNS topic and sends along the number value.
The SNS topic will trigger a second function called calculate. This function will perform the calculation and log out the result to the console. This mimics a heavy computational background task such as data processing, image manipulation or machine learning calculations.
If the calculate function fails, the Dead Letter Queue SNS topic will receive a message and trigger the error function.
Every function invoked asynchronously will twice retry its execution upon failure. Using the Dead Letter Queue as a pool for your error logs is a smart use-case.
Now you're wondering, why all the complication with SNS instead of just invoking the second lambda function from the first one with Lambda's invoke API?
First of all, it's a huge anti-pattern for asynchronous workflows, which is our case. Otherwise, it's fine if you need the response from the second lambda function right away. Another issue is hitting Lambda's concurrency limits pretty fast. It can result in losing invocations and dropping data. Sending your data through a pub/sub service like SNS or a queue like SQS will make sure you have data integrity.
Makes sense now? Sweet, let's talk a bit more about SNS.
What is AWS SNS?
Before we start coding we need to cover the basics. We know what AWS Lambda is, but what about SNS? The AWS docs are pretty straightforward.
Amazon Simple Notification Service (SNS) is a flexible, fully managed pub/sub messaging and mobile notifications service for coordinating the delivery of messages to subscribing endpoints and clients.
-- AWS Docs
In English, this means it's a way of sending notifications between services on a publisher/subscriber basis. One service publishes some data about a topic and sends it along its way. SNS will then funnel it to all the subscribers of that particular topic. The key focus is on topic here, you'll see why a bit further down.
Build the API with the Serverless Framework
The first thing to do, as always, is to set up the project and install dependencies.
1. Install the Serverless Framework
My go-to development and deployment tool for serverless apps is the Serverless Framework. Let's go ahead and install it.
$ npm i -g serverless
Note: If you're using Linux, you may need to run the command as sudo.
Once installed globally on your machine, the commands will be available to you from wherever in the terminal. But for it to communicate with your AWS account you need to configure an IAM User. Jump over here for the explanation, then come back and run the command below, with the provided keys.
$ serverless config credentials\
--provider aws\
--key xxxxxxxxxxxxxx\
--secret xxxxxxxxxxxxxx
Now your Serverless installation knows what account to connect to when you run any terminal command. Let's jump in and see it in action.
2. Create a service
Create a new directory to house your Serverless application services. Fire up a terminal in there. Now you're ready to create a new service.
What's a service? It's like a project. It's where you define AWS Lambda functions, the events that trigger them and any AWS infrastructure resources they require, including SNS which we'll add today, all in a file called serverless.yml.
Back in your terminal type:
$ serverless create --template aws-nodejs\
--path lambda-sns-dlq-error-handling
The create command will create a new service. What a surprise! We also pick a runtime for the function. This is called the template. Passing in aws-nodejs will set the runtime to Node.js. Just what we want. The path will create a folder for the service.
3. Explore the service directory with a code editor
Open up the lambda-sns-dlq-error-handling folder with your favorite code editor. There should be three files in there, but for now, we'll only focus on the serverless.yml. It contains all the configuration settings for this service. Here you specify both general configuration settings and per function settings. Your serverless.yml will be full of boilerplate code and comments. Feel free to delete it all and paste this in.
service: lambda-sns-dlq-error-handling
plugins:
- serverless-pseudo-parameters
provider:
name: aws
runtime: nodejs8.10
stage: dev
region: eu-central-1
memorySize: 128
environment:
accountId: '#{AWS::AccountId}'
region: '#{AWS::Region}'
iamRoleStatements:
- Effect: "Allow"
Resource: "*"
Action:
- "sns:*"
functions:
init:
handler: init.handler
events:
- http:
path: init
method: post
cors: true
calculate:
handler: calculate.handler
events:
- sns: calculate-topic # created immediately
onError: arn:aws:sns:#{AWS::Region}:#{AWS::AccountId}:dlq-topic
error:
handler: error.handler
events:
- sns: dlq-topic # created immediately
Let's break down what's going one here. Check out the functions section. There are three functions here. From top to bottom they're init, calculate, and error. The init function will get triggered by a simple HTTP request, which we invoke through API Gateway. Familiar territory for us.
However, the calculate and error functions are triggered by SNS topics. Meaning we will have logic in the init function that will publish messages to a topic named calculate-topic while the calculate function is subscribed to the same topic.
Moving on, the error function is subscribed to the dlq-topic while the calculate function will publish messages to this topic if it fails, as you can see with the onError property. Now stuff makes sense, right?
Make a mental note to yourself, once you add the SNS topics as events for your functions, the resources will automatically be created once you deploy the service.
What else, take a look at the iamRoleStatements, they specify that our functions have permission to trigger, and get invoked by SNS topics. While the serverless-pseudo-parameters plugin lets us reference our AccountId and Region with the CloudFormation syntax, making it much easier to keep our SNS ARNs consistent across all resources.
4. Install dependencies
Luckily, this part will be short. Just one package to install. First, initialize npm and then you can install serverless-pseudo-parameters.
$ npm init -y && npm i serverless-pseudo-parameters
That'll do.
5. Write business logic
With all things considered, the configuration process was rather simple. The code we'll write now is just as straightforward. Nothing extraordinary to see, I'm sorry to disappoint.
Let's keep all three functions in separate files, to keep it simple. First of all create an init.js file and paste this snippet in.
// init.js
const aws = require('aws-sdk')
const sns = new aws.SNS({ region: 'eu-central-1' })
function generateResponse (code, payload) {
console.log(payload)
return {
statusCode: code,
body: JSON.stringify(payload)
}
}
function generateError (code, err) {
console.error(err)
return generateResponse(code, {
message: err.message
})
}
async function publishSnsTopic (data) {
const params = {
Message: JSON.stringify(data),
TopicArn: `arn:aws:sns:${process.env.region}:${process.env.accountId}:calculate-topic`
}
return sns.publish(params).promise()
}
module.exports.handler = async (event) => {
const data = JSON.parse(event.body)
if (typeof data.number !== 'number') {
return generateError(400, new Error('Invalid number.'))
}
try {
const metadata = await publishSnsTopic(data)
return generateResponse(200, {
message: 'Successfully added the calculation.',
data: metadata
})
} catch (err) {
return generateError(500, new Error('Couldn\'t add the calculation due to an internal error.'))
}
}
We have a few helper functions and the exported lambda function at the bottom. What's going on here? The lambda validates input and publishes some data to the calculate-topic SNS topic. That's everything this function is doing. The calculate-topic SNS topic will trigger the calculate lambda function. Let's add that now.
Create a file and name it calculate.js. Paste this snippet in.
// calculate.js
module.exports.handler = async (event) => {
const { number } = JSON.parse(event.Records[0].Sns.Message)
const factorial = (x) => x === 0 ? 1 : x * factorial(x - 1)
const result = factorial(number)
console.log(`The factorial of ${number} is ${result}.`)
return result
}
As you see this is just a simple factorial calculation implemented with a recursive function. It'll calculate the factorial of the number we published to the SNS topic from the init function.
An important note here is that if the calculate function fails a total of three times, it'll publish messages to the Dead Letter Queue SNS topic we specified with the onError property in the serverless.yml file. The Dead Letter Queue will then trigger the error function. Let's create it now so it can log out errors to CloudWatch. Create an error.js file, and paste these lines in.
// error.js
module.exports.handler = async (event) => {
console.error(event)
}
For now, this will do. However, ideally, you would have structured logging with detailed info about everything going on. That's a topic for another article.
Deploy the API to AWS Lambda
Here comes the easy part. Deploying the API is as simple as running one command.
$ serverless deploy\
You can see the endpoint get logged to the console. That's where you will be sending your requests.
Test the API with Dashbird
The simplest way of testing an API is with CURL. Let's create a simple CURL command and send a JSON payload to our endpoint.
$ curl -H "Content-Type: application/json"\
-d '{"number":1000}'\
https://<id>.execute-api.eu-central-1.amazonaws.com/dev/init
If everything works as it should, the result of the calculation will be logged to CloudWatch. If not, well then you're out of luck. In cases like these, I default to using Dashbird to debug what's going on. It's free and doesn't require a credit card to set up.
After hitting the endpoint a few times with a couple of different values here's the result. The init function works as expected.
But, what really interests us is the calculate function. Here's what that looks like when it's successful.
When it fails it'll specify a crash and show the error logs.
After two retries it'll send a message to the Dead Letter Queue and trigger the error function.
Sweet! We've tested all the different scenarios. Hope this clears things up a bit.
Wrapping up
That's all folks. We've covered creating SNS triggers for Lambda while also implementing a Dead Letter Queue to catch errors from failed invocations. Using serverless for various intermittent calculations is a valid use-case that will only grow in popularity in the future.
There are no servers you need to worry about, and you only pay for the time it runs. Just deploy the code and rest assured it'll work. If something breaks, you have Dashbird watching your back, alerting you in Slack or E-mail if something is wrong. You just have to love Slack integration!
Again, here's the GitHub repo, if you want to take a look at the code. It can act as a starter for your own use-cases where you need SNS messages triggering Lambda functions. Give it a star if you like it and want more people to see it on GitHub.
