Integrating API Gateway with other AWS services offers several benefits, including improved performance and reduced costs. Let's explore some practices I've adopted when developing application backends, with a focus on DynamoDB.
1. Background
I've been involved in the development of multiple web applications using both AWS CDK and other infrastructure frameworks. Additionally, I've created several applications on my own, ranging from small to large projects, where I had the freedom to choose the tech stack (typically TypeScript) and architecture (serverless when possible).
In this post, I'll share seven practices I learned while developing application backends.
2. Disclaimer
Your situation might differ from mine, and you might disagree with the practices I'm about to share, which is perfectly fine.
I prefer applying the single-table design principles in DynamoDB. You might not agree with this approach, and that's okay too.
The practices listed below are based on my experience. They have worked well for me, but that doesn't guarantee they will work for you.
3. Some practices for DynamoDB integration
You might already be familiar with many of the strategies below. Some I learned early on, while others I discovered more recently.
The examples provided illustrate direct integrations between API Gateway and DynamoDB. This means using VTL in API Gateway's request and/or response templates to define DynamoDB operations, and request and response payloads. The index keys (table and any local/global secondary index partition keys and sort keys) presented here were designed for a single table, which explains their unusual structure.
All examples are written in TypeScript, the language originally used to create CDK.
3.1. Use JSON.stringify() in CDK
Instead of writing template strings in the CDK code, we can use JSON.stringify()
to create mapping templates.
For instance, if we want to request a specific user item and assume the username is johndoe
, we might have a partition key PK
and sort key SK
like this:
{
"PK": "USER#johndoe",
"SK": "USER#johndoe"
}
The CDK integration code can be written as follows:
const integration = new apigateway.AwsIntegration({
service: 'dynamodb',
action: 'GetItem',
// the GetItem action is actually a POST request to the DynamoDB API
integrationHttpMethod: 'POST',
options: {
// API Gateway needs GetItem permission to the table
credentialsRole: myRole,
passthroughBehavior: apigateway.PassthroughBehavior.WHEN_NO_TEMPLATES,
requestTemplates: {
'application/json': JSON.stringify({
TableName: tableName,
Key: {
PK: { S: `USER#$input.params('userName')` },
SK: { S: `USER#$input.params('userName')` },
},
}),
},
// ...other properties
}
})
The requestTemplates
value is the same GetItemCommandInput
object we would define in a Lambda function if we used one.
The $input.params('userName')
expression extracts the userName
query (or path) parameter's value, enabling us to generate dynamic user requests. For example, the request template above will map the /user?userName=johndoe
query string to the item with PK
and SK
values of USER#johndoe
in the DynamoDB table.
3.2. Use if/else to send a response if the requested item doesn't exist
Even if the requested item doesn't exist in DynamoDB, I want to ensure a response is sent back to API Gateway and the client.
For example, if the item with PK = USER#johndoe
and SK = USER#johndoe
doesn't exist in the database, the responseTemplates part of AwsIntegration
might look like this:
const integration = new apigateway.AwsIntegration({
action: 'GetItem',
options: {
integrationResponses: [
{
statusCode: '200',
responseTemplates: {
'application/json': `
#set($inputRoot = $input.path('$'))
#if($inputRoot.Item && !$inputRoot.Item.isEmpty())
#set($user = $inputRoot.Item)
{
"status": "success",
"data": {
"userName": "$user.UserName.S",
}
}
#else
#set($context.responseOverride.status = 404)
{
"status": "error",
"error": {
"message": "Item not found"
}
}
#end
`
}
}
]
// ...other properties
},
// ...other properties
})
The #if/#else/#end
block works similarly to other programming languages.
If the item with the required PK
and SK
exists, we'll transform the DynamoDB response by removing the S
type descriptor and adding the UserName
to the userName
property.
If it doesn't exist, DynamoDB will still return 200 OK to API Gateway because the request was valid and well-formatted. It's not DynamoDB's fault that a non-existing item was requested. Thus, we override the 200
status code from DynamoDB to 404
and add a descriptive message. (Some properties are omitted for brevity.)
Previously, I used Lambda functions to create these responses and error messages. Now, I try to avoid Lambdas whenever possible, opting for direct integrations with mapping templates. This approach eliminates cold starts and usually keeps response times under 100 ms.
3.3. Don't generalize templates - most are unique anyway
I used to spend a lot of time studying VTL syntax and trying to create a generic mapping template that would work for most requests and responses. However, I realized that API Gateway doesn't support the entire VTL ecosystem, and the supported VTL syntax is not well documented. This led to a lot of trial and error, consuming hours without significant progress.
Instead of creating a single, all-encompassing mapping template, I now create separate templates for each access pattern.
While this approach might seem contrary to clean code principles, it has proven more practical for me. If all table items only have string attributes, creating a generic template is feasible. I even have an example of such a template here.
However, as the application's access patterns grow more complex, most mapping templates end up being different. Handling multiple data types, lists (arrays), maps, and nested attributes in a single template proved slow and unproductive. Instead, I now do something like this:
#set($user = $inputRoot.Item)
{
"userName": "$user.UserName.S",
"address": "$user.Address.L",
"age": "$user.Age.N",
}
I can also create nested response objects by iterating over lists and maps, and change key names to be different from the table attribute names (more on that below). This approach is quick, flexible, and easier to debug in case of errors.
I'm not against extracting repeated code into its own function. If I have a specific response or error format used in multiple templates, I write a simple function with a config argument to return customized messages while maintaining the same format. This is one reason why CDK is great!
3.4. Use specific data attributes
When designing an application with a single table, it's common to have multiple different entities, such as users, messages, events, and tasks. To avoid confusion, especially as the business grows and wants to identify individual entities with unique IDs, it's important to use specific data attributes.
I avoid using generic attributes like Id
or Name
because, with multiple entities in the table, it becomes difficult to remember which entity an Id
refers to – is it a user or a task ID?
Instead, I use specific attributes, like UserId
, UserName
, TaskId
and MessageId
. This approach is straightforward and makes the code easier to read and work with.
3.5. Use ExpressionAttributeNames
DynamoDB has a list of reserved words such as Date
(not case sensitive), which commonly appears as a database attribute.
Using reserved words directly in request input expressions will result in errors. For example, the following GetItem
request template will not work:
requestTemplates: {
'application/json': JSON.stringify({
TableName: 'MyTable',
Key: {
PK: { S: `TASK#$input.params('taskId')` },
SK: { S: `TASK#$input.params('taskId')` },
},
ProjectionExpression: 'TaskId, StartTime, EndTime, Date',
}),
},
Here, ProjectionExpression
includes TaskId
, StartTime
, EndTime
and Date
, which are attributes on the Task
item we want to return to the client. We don't need the entire large item for this specific access pattern, only these properties.
To make the code work, we can use ExpressionAttributeNames
:
requestTemplates: {
'application/json': JSON.stringify({
TableName: 'MyTable',
Key: {
PK: { S: `TASK#$input.params('taskId')` },
SK: { S: `TASK#$input.params('taskId')` },
},
ProjectionExpression: '#taskid, #starttime, #endtime, #date',
ExpressionAttributeNames: {
'#taskid': 'TaskId',
'#starttime': 'StartTime',
'#endtime': 'EndTime',
'#date': 'Date',
},
}),
},
Now, the template validator will accept it as valid code, and our stack will deploy. In this specific case, it's enough to create an expression attribute name for Date
since it's the only reserved word among the projected attributes.
3.6. Payload and database attributes are the same
Keeping the payload and database attributes the same simplifies development.
Let's say we want to create a new Task
with the following payload:
{
"taskId": "3ecd0d50-6ece-468e-a6ed-66c58c9c6525",
"startTime": "10:00",
"endTime": "14:00",
"date": "2024-07-08"
}
In this case, I want the Task
item data attributes in the DynamoDB table to be taskId
, startTime
, endTime
and date
. Alternatively, I might make minor conversions, like taskId
to TaskId
, which is straightforward to implement.
When using Lambda functions for additional logic or input validation during create and update operations, I write a small function to convert payload keys to attributes. This conversion logic is always straightforward, and a generic function covers all use cases without exceptions.
3.7. Enable Amazon Q Developer
I found Amazon Q Developer (formerly known as CodeWhisperer) to be an excellent tool that helps me write code faster. It effectively predicts the code I'm about to write based on my previous code in the project folder. Q Developer can also recommend VTL mapping template code, and I'm satisfied with its accuracy.
I use VS Code as my code editor, and Q Developer seamlessly integrates with it. It also supports multiple programming languages, including TypeScript. Amazon Q works well in the command line, though this feature is currently available only for macOS.
4. Summary
Direct integrations between API Gateway and various AWS services can often eliminate the need for Lambda functions. Writing VTL request and response mapping templates can be daunting, especially for those with limited experience.
However, by applying some tricks and techniques, we can become more productive and create fast web applications more efficiently.
5. Further reading
Initialize REST API setup in API Gateway - API Gateway setup guide
Setting up REST API integrations - Relevant documentation section
Getting started with DynamoDB - DynamoDB basics and table creation
Integrations for REST APIs in API Gateway - The different integration types