The classic twelve-factor app has config in environment variables as a core component:

Apps sometimes store config as constants in the code. This is a violation of twelve-factor, which requires strict separation of config from code. Config varies substantially across deploys, code does not. A litmus test for whether an app has all config correctly factored out of the code is whether the codebase could be made open source at any moment, without compromising any credentials.

This approach has been widely adopted. Although configuration as code allows for versioning and consistency across environments, there are always configurations that differ based on where the code is running. The most common being secrets.

The problem arises when a secret such as a database connection URL or API key is stored in the environment variable. This practice should be avoided because it’s an easy way for an attacker to gain access to other systems once they have access to the environment.

Environment variables are stored as plain text so if you manage to gain access to the runtime environment - perhaps through a remote code execution vulnerability or by overly verbose logging - you can then access all the environment variables. This is documented as a specific common weakness (CWE-526) and has been actively exploited by hacking groups like TeamTNT.

In this post we’ll discuss the problems with storing secrets in environment variables and what you should do instead.

Pitfall #1: Lateral movement - secrets are a stepping stone

Think of your environment variables like exposed footholds on a wall. If an attacker gains a toehold in your system (maybe through a remote code execution bug), they can quickly scale up to grab the secrets.

Try it locally - open a terminal and run env

david@mbp:$ env
COLORTERM=truecolor
COMMAND_MODE=unix2003
HOME=/Users/david
LANG=en_US.UTF-8
SHELL=/bin/zsh
...

This assumes you have access to a shell, but every programming language has a way to access environment variables as well. Try it with Python:

import os
for key, value in os.environ.items():
    print(f"{key}: {value}")

Once they have those database credentials or API keys, they're not just messing around in your app anymore (which might be locked down) – they're hopping to other parts of your infrastructure.

Pitfall #2: Accidental exposure

Logging and process dumps are another way your secrets can be exposed. That innocent console.log(process.env) you added during debugging could end up sending your API keys off to some third-party logging tool. This is where you need to be super careful about redacting sensitive data before it ever leaves your environment.

This is where you would redact sensitive fields from your logs, not just for secrets but also for personal information you need to sanitize.

Pitfall #3: Management challenges

Environment variables feel deceptively easy for secrets, but quickly become a management nightmare:

• Where's the audit trail? Accidentally deleted a critical API key? Need to know whether an attacker accessed a particular system? Good luck figuring out who touched it and when. No version history means no easy way back.
• Rollbacks are a pain. Need to rotate a compromised secret? Prepare for some downtime and juggling multiple values across environments while your services restart.
• Which secret is where? Did that database password change for production, staging, or just your dev machine? Environment variables on different systems easily get out of sync.

Ever spent hours untangling a mess caused by one of these issues? That's the real-world cost of treating environment variables as a secret vault. Secrets managers exist for a reason – they save you these headaches.

Solution: Runtime secrets injection

The solution is to query the secret value at runtime. There are a couple of approaches to this:

1 - Set the environment variable to the secret ID

This approach is flexible if you work across multiple cloud platforms or want to build your own secrets management abstraction. Instead of storing the secret values themselves in environment variables, you store references or IDs.

In AWS the value can be a Secrets Manager ARN. In GCP the value could be the Secret Manager ID. At startup, your application uses these IDs to fetch the real secrets from a central secrets store.

You could build a custom wrapper for this by pattern matching the ID format. For example, AWS Secrets Manager IDs have the prefix “arn:aws:secretsmanager” so you know this ID is referencing a secret stored in AWS. If there are no matches it can fall back to reading the value directly from the environment variable which would be useful if you were running in a local dev environment or CI/CD.

Alternatively, there are libraries that abstract the underlying provider into a generic interface. This avoids writing your own and they support multiple providers:

• Go: Konf, Koanf.
• JS: Dotenv.
• Ruby: Config.

2 - Use the platform secrets manager directly

If you're fully committed to a single cloud provider (AWS, GCP, Azure, etc.), leveraging their native secrets manager is often the most streamlined way to go. Each provider offers an SDK that you can integrate directly into your application code. Here's how this typically works:

• Permissions: Ensure your application has the necessary IAM (or equivalent) roles to access secrets. This allows more fine grained access controls and auditing of the access.
• SDK Integration: Use the provider's SDK to fetch secrets at startup or when needed.
• Secrets as Objects: Many SDKs provide convenient objects to represent secrets, allowing you to access them by name within your code. You also get type safety as a bonus.

In Go, accessing the latest value of a secret from AWS Secrets Manager is straightforward:

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/aws/aws-sdk-go-v2/aws"
    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/secretsmanager"
)

func getSecret(secretName string) (string, error) {
    // Create a Secrets Manager client
    awsCfg, err := config.LoadDefaultConfig(
        context.Background(), config.WithRegion("us-east-1"))

    client := secretsmanager.NewFromConfig(awsCfg)

    // Get the secret value
    input := &secretsmanager.GetSecretValueInput{
        SecretId: aws.String(secretName),
    }

    result, err := client.GetSecretValue(context.Background(), input)
    if err != nil {
        return "", fmt.Errorf("error fetching secret: %w", err)
    }

    // Decryption is handled automatically if it's a string secret
    return *result.SecretString, nil
}

func main() {
    secretName := "my/database/password"
    secretValue, err := getSecret(secretName)
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println("Database Password:", secretValue)
}

3 - Use a secrets manager product

Specialized secrets management products offer a centralized and streamlined experience. They are provider agnostic so you only need to define the secrets once and they will be synced to different providers (AWS, GCP, Vercel, etc.).

They also offer advanced features like granular access controls, versioning, and more robust auditing. Managing rotation is often easier as well.

Popular options include Doppler, HashiCorp Vault Secrets, 1Password, and Infisical.

Conclusion

Don't make it easy for attackers! Environment variables are convenient but leave your secrets vulnerable. Here's the takeaway:

• Fetch secrets at runtime, not before.
• Logs are an easy way to leak sensitive values. Make sure they’re appropriately sanitized.
• Embrace secrets managers. They offer security, auditing, and streamlined management that environment variables simply can't match.