AWS is a zero-trust platform¹. That is, every call to AWS must provide credentials so that the caller can be validated and her authorization checked. How one manages these credentials will vary depending on the execution environment. A developer, who gets his workstation set up with his own AWS credentials, will often find that the application he is building cannot (and should not) consume credentials in the same way. Why do these differences exist? And what should he do to manage credentials?

For credentials on your workstation, AWS recommends using IAM Identity Center SSO. This lets you verify your identity (often with a standard, non-AWS identity provider like Google or Okta), and then that user can assume an IAM role to provide a set of temporary credentials. This works well and is fairly secure, especially if your identity provider is set up with multi-factor authentication (MFA). Because the AWS credentials are short-lived, even if they leak out, they expire quickly thus limiting exposure. Why can't we take this approach with the applications we are building?

We want to have the application assume a role and pick up short-term credentials. However, we can't use the workstation approach because we need user authentication (SSO/MFA) to establish the user, and that's not possible at the application's runtime. To get out of this jam, we can rely on the fact that all our application runtimes are serverless and will happen within an AWS service (in our case Lambda or Fargate). This establishes sufficient trust such that we can assign the execution environment a role and let it obtain short-term credentials.

In this article, I want to examine how your application running in either Lambda or Fargate ought to get its AWS credentials. We'll discuss how the AWS SDKs use credential provider chains to establish precedence and one corner case I found myself in. Let's dig in.

Credential Sources

As mentioned earlier, you can provide credentials to your application in several ways including (but not limited to) environment variables, config/credentials files, SSO through IAM Identity Center, instance metadata, and (don't do this) directly from code. Irrespective of which method you choose, you can allow the AWS SDK to automatically grab credentials via its built-in "credentials provider."

The mechanism for selecting the credential type is called the "credential provider" and the built-in precedence (i.e., the order in which it checks credential sources) is called the "credential provider chain." This is language agnostic. Per AWS documentation, "All the AWS SDKs have a series of places (or sources) that they check to get valid credentials to use to make a request to an AWS service." And, once they locate any credentials, the chain stops and the credentials are used.

For the NodeJS SDK, that precedence is generally:

Explicit credentials in code (again, please don't do this)
Environment Variables
Shared config/credentials file
Task IAM Role for ECS/Fargate
Instance Metadata Service (IMDS) for EC2

So, we can pass in credentials in many ways. Why should we choose one over another? Each approach varies in terms of security and ease of use. Fortunately, AWS allows us to easily set up our credentials without compromising security. We are going to focus on two recommended approaches: environment variables (for Lambda) and the task IAM role (for Fargate)

Environment Variable Credentials

Credentials in environment variables are, perhaps, the easiest way to configure your AWS SDK. They are near the top of the precedence list and will get scooped up automatically when you instantiate your SDK. For example, if you set the following environment variables in your runtime:

export AWS_ACCESS_KEY_ID="AKIA1234567890"
export AWS_SECRET_ACCESS_KEY="ABCDEFGH12345678"

Then when you instantiate your AWS SDK, these credentials will get loaded automatically, like so:

import { DynamoDBClient } from '@aws-sdk/client-dynamodb'
const client = new DynamoDBClient({ region: 'us-east-1' }); // Loads env credentials

Note that the AWS_ACCESS_KEY_ID begins with "AKIA". This signifies that this is a long-term access key with no expiration. These types of keys are attached to an IAM user or, if you are reckless, the AWS account root user².

Alternatively, you may run across AWS credentials that look like the following:

AWS_ACCESS_KEY_ID=ASIA1234567890
AWS_SECRET_ACCESS_KEY=ABCDEFGH12345678
AWS_SESSION_TOKEN=XYZ+ReallyLongString==

These credentials are short-lived. You can tell this both by the presence of the AWS_SESSION_TOKEN and that the AWS_ACCESS_KEY_ID begins with "ASIA" instead of "AKIA".

When you use a credential provider, it consumes the Access Key, Secret, and Session Token. These tokens can be set to expire anywhere from 15 minutes to 12 hours from issuance. This would be a drag if you had to repeatedly go fetch these short-lived tokens and save them so your application can use them. Fortunately, you don't have to. Both Lambda and ECS offer built-in mechanics to provide your application with short-term credentials. Let's start with Lambda.

Using Credentials in Lambda

Add the following line to one of your Lambdas:

console.log(process.env);

And you'll see AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN. How did they get there? AWS added them for you using the (required) IAM role attached to your Lambda. During initialization, AWS calls out to their Secure Token Service (STS), obtains short-term credentials, and then conveniently injects those credentials into your Lambda's environment variables.

Lambdas are special in this regard. You (or the credentials provider) don't have to do anything extra to fetch credentials, they are just there for you to use. Why?

Lambdas are short-lived. Even under constant use, every hour or so they are automatically recycled. This means that a single short-lived token can serve the Lambda that is using it; no re-fetching of tokens is necessary. For example, if AWS sets Lambdas to last no more than an hour before being decommissioned, it can set the expiration for the access token to just over 60 minutes and the application using the token will never need to fetch another.

Having your credentials provider automatically find and use the credentials in Lambda's environment variables is both the recommended and easiest approach. This is a true win/win.

Using Credentials in Fargate

ECS Fargate shares many traits with Lambda: they're both managed by AWS (in neither case are we taking care of the underlying servers), they scale up and down automatically, and each can have an IAM role that provides permissions for the application's runtime.

However, Fargate containers don't automatically recycle. They are relatively long-lived when compared to Lambda and can easily live longer than the maximum STS token expiration. This means the method used by Lambda to inject the STS tokens into the runtime environment won't work.

Instead, you can use the--optional but recommended--Task Role ARN property of your ECS task definition to specify the permissions you would like your task to have. Then your credentials provider can assume this role to obtain short-term credentials it can use. It manages this for you and you don't have to do anything but set the TaskRoleArn in your task definition.

Why You Should Know This

The AWS SDK's credentials provider doesn't know "I'm in a Lambda" or "I'm in Fargate." When invoked, the SDK will use the default credentials provider to step through a chain of locations to look for credentials and it will stop as soon as it finds one. This means things often "just work." But, it also means you can short-circuit the precedence chain if you are not careful (or you can do it purposefully; I'll give an example later).

If you are using Lambda, and you new up an SDK client like this:

const client = new DynamoDBClient({
  region: 'us-east-1',
  credentials: {
    accessKeyId: 'ABC', // Don't do this
    seretAccessKey: '123', // Don't do this, either
  },
});

your credentials provider will never check the environment variables for credentials and will run with what you gave it.

Likewise, in Fargate, if you either pass in direct credentials or set environment variables of AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, your credentials provider will never use your TaskRoleArn. This can be confusing if you are not used to it.

Breaking the Chain on Purpose

I was working with a client on a container migration project, where they needed to move their container workloads from Kubernetes (K8) on EC2 over to Fargate on ECS. At one point during the transition, the same container needed to be simultaneously running in both places. I knew I wanted to use the TaskRoleArn in Fargate, but that would not fly in the K8 deployment as it would grab the credentials from the EC2 instance on which it ran. And, since that EC2 instance served many disparate K8 pods, it was a poor³ place to manage the runtime permissions of the containers underneath it.

The K8 configuration had environment variables set to long-term credentials for an IAM user. At first, the ECS task definition just used the same credentials (from env vars). Then, we created a dedicated IAM role for the task and attached it to the definition as a TaskRoleArn. OK, time for a quick quiz:

What happens now? The ECS container will:
A) Use the IAM role from TaskRoleArn.
B) Use the environment variable credentials.
C) Throw a ConflictingCredentialsError.

The correct answer is B. As long as those environment variable credentials are present, the credentials provider will stop looking after discovering them. During the migration, we used this to our advantage as we kept the code the same and just modified the configuration based on the destination (environment variable credentials in K8, none in Fargate). Eventually, we were only using the TaskRoleArn and we could retire those long-term credentials and the environment variables that surfaced them.

What Can Go Wrong?

Long-term credentials pose a real risk of leaking. AWS advises its users to take advantage of SSO and IAM roles for their user and application runtimes, respectively. I know an engineer who inadvertently checked in a hard-coded AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY into a public GitHub repository. Within minutes, they had been scraped, and expensive BitCoin miners were deployed in far-away regions of his company's AWS account (kudos to AWS for expunging those actions from their AWS bill the following month).

The engineer had thought the repository was private. However, the fundamental issue was using hard-coded, long-lived credentials in the first place. Using fully managed, serverless architectures like Lambda and Fargate along with role-based, short-term credentials, you can avoid this kind of headache.

AWS Credentials for Serverless