Proactively Securing AWS: Automated Alarms for Pending Deletion KMS Keys and Secrets Using Terraform

Alexey Vidanov - Jul 26 - - Dev Community

In cloud computing, security is a critical concern. While AWS provides backup solutions for many resources, custom configurations often require additional protection. Two key services, AWS Key Management Service (KMS) and AWS Secrets Manager, don't offer direct backup options. However, they implement a deletion grace period— by default 30 days and this is the maximum — allowing for potential restoration.

In cloud computing, security is a critical concern. While AWS provides backup solutions for many resources, custom configurations often require additional protection. Two key services, AWS Key Management Service (KMS) and AWS Secrets Manager, don't offer direct backup options. However, they implement a deletion grace period— by default 30 days and this is the maximum — allowing for potential restoration.

A crucial question arises: How can you be notified of a pending deletion? This is where alarms play a vital role. Implementing alarms for AWS KMS keys and AWS Secrets Manager secrets that are pending deletion can significantly enhance your security posture.

This blog post demonstrates how to create these essential alarms using Terraform, an infrastructure-as-code tool. By leveraging Terraform, you can automate the process of setting up notifications for potential security risks, ensuring that your sensitive information and encryption keys remain protected.

We'll walk through the process of configuring alarms that alert you when KMS keys or Secrets Manager secrets are scheduled for deletion, giving you time to review and potentially prevent unintended data loss or security breaches.

Prerequisites

Before we begin, ensure you have:

  • An AWS account with appropriate permissions
  • Terraform installed and configured
  • Basic knowledge of AWS services, particularly CloudTrail, KMS, Secrets Manager, and CloudWatch
  • Familiarity with Terraform syntax and AWS provider

Step-by-Step Guide

1. Setting Up Terraform Variables

First, define the necessary variables in your terraform.tfvars or within your Terraform configuration. These variables include the notification email, the name of the CloudWatch log group attached to CloudTrail logs, and the AWS region.

variable "notification_email"
{
    description = "The email address to notify in case of an alarm"
  type        = string
}

variable "cloud_trail_cloudwatch_log_group" {
  description = "The name of the CloudWatch log group attached to CloudTrail logs"
  type        = string
}

variable "region" {
  description = "The AWS region"
  default     = "eu-central-1"
  type        = string
}
Enter fullscreen mode Exit fullscreen mode

2. Configuring the AWS Provider and Resources

Ensure your AWS provider is configured with the specified region. We also need the information for the AWS CloudWatch log group to create the resources.

provider "aws" {
  region = var.region
}

data "aws_cloudwatch_log_group" "this" {
  name = var.cloud_trail_cloudwatch_log_group
}
Enter fullscreen mode Exit fullscreen mode

Create an IAM role and policy that allows CloudTrail to interact with CloudWatch logs. Additionally, set up an SNS topic for sending notifications and subscribe it to the provided email.

resource "aws_iam_role" "this" {
  name = "CloudTrailCloudWatchLogsRole"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "cloudtrail.amazonaws.com"
        }
      },
    ]
  })
}

resource "aws_iam_role_policy_attachment" "cloudtrail_cloudwatch_logs" {
  role       = aws_iam_role.this.name
  policy_arn = aws_iam_policy.this.arn
}

# SNS Topic for Notifications

resource "aws_sns_topic" "pending_deletion_topic" {
  name = "kms-key-pending-deletion-topic"
}

# Subscription to send notifications to an email

resource "aws_sns_topic_subscription" "email_subscription" {
  topic_arn = aws_sns_topic.pending_deletion_topic.arn
  protocol  = "email"
  endpoint  = var.notification_email
}
Enter fullscreen mode Exit fullscreen mode

3. Detecting Deletion Attempts with CloudWatch Log Metric Filters

Use CloudWatch log metric filters to detect deletion attempts or unauthorized access. For AWS Secrets Manager, filter events where a pending deletion secret is accessed. For AWS KMS, filter events where a pending deletion key is used.

# Metric filter to detect the attempt to access secrets marked for deletion

resource "aws_cloudwatch_log_metric_filter" "secrets_deletion_attempts_filter" {
  name           = "SecretsDeletionAttempts"
  pattern        = "{ $.eventName = \"GetSecretValue\" && $.errorMessage = \"*secret because it was marked for deletion*\" }"
  log_group_name = data.aws_cloudwatch_log_group.this.name
  metric_transformation {
    name      = "SecretsDeletionAttempts"
    namespace = "SecretsManagerMetrics"
    value     = "1"
  }
}
Enter fullscreen mode Exit fullscreen mode

If you want to receive notifications not only for attempts to access secret values but also immediately after an attempt to delete secrets, you can change the pattern to $.eventName = \"DeleteSecret\" instead of $.eventName = \"GetSecretValue\"

# Metric filter to detect the attempt to access MS keys marked for deletion

resource "aws_cloudwatch_log_metric_filter" "kms_key_pending_deletion_filter" {
  name           = "KMSKeyPendingDeletion"
  pattern        = "{ $.eventSource = kms* && $.errorMessage = \"* is pending deletion.\"}"
  log_group_name = data.aws_cloudwatch_log_group.this.name
  metric_transformation {
    name      = "KMSKeyPendingDeletionEvents"
    namespace = "CloudTrailMetrics"
    value     = "1"
  }
}
Enter fullscreen mode Exit fullscreen mode

If you want to receive notifications not only for attempts to access keys but also immediately after an attempt to delete keys, you can change the pattern to $.eventSource = kms* instead of $.eventSource = ScheduleKeyDeletion*

4. Creating CloudWatch Alarms

Finally, create CloudWatch alarms based on the metric filters. These alarms will trigger when an attempt is made to access a secret or use a pending deletion KMS key.

# CloudWatch Alarm for KMS Key Pending Deletion

resource "aws_cloudwatch_metric_alarm" "kms_key_pending_deletion_alarm" {
  alarm_name          = "KMSKeyPendingDeletionAlarm"
  comparison_operator = "GreaterThanOrEqualToThreshold"
  evaluation_periods  = "1"
  metric_name         = aws_cloudwatch_log_metric_filter.kms_key_pending_deletion_filter.metric_transformation[0].name
  namespace           = aws_cloudwatch_log_metric_filter.kms_key_pending_deletion_filter.metric_transformation[0].namespace
  period              = 300
  statistic           = "Sum"
  threshold           = 1
  alarm_description   = "Alarm when KMS key pending deletion is used."
  alarm_actions       = [aws_sns_topic.pending_deletion_topic.arn]
  treat_missing_data  = "notBreaching"
}

# CloudWatch Alarm for Secrets Manager Secrets Deletion Attempts

resource "aws_cloudwatch_metric_alarm" "secrets_deletion_attempts_alarm" {
  alarm_name          = "SecretsDeletionAttemptsAlarm"
  comparison_operator = "GreaterThanOrEqualToThreshold"
  evaluation_periods  = "1"
  metric_name         = aws_cloudwatch_log_metric_filter.secrets_deletion_attempts_filter.metric_transformation[0].name
  namespace           = aws_cloudwatch_log_metric_filter.secrets_deletion_attempts_filter.metric_transformation[0].namespace
  period              = 300
  statistic           = "Sum"
  threshold           = 1
  alarm_description   = "Alarm when an attempt to delete an AWS Secrets Manager secret is made."
  alarm_actions       = [aws_sns_topic.pending_deletion_topic.arn]
  treat_missing_data  = "notBreaching"
}
Enter fullscreen mode Exit fullscreen mode

Operation

  1. After running Terraform, a confirmation message will be sent to the email address registered in Terraform. You must confirm the subscription in this message.

    1. In case of an alarm, you will receive a warning message to the registered email address.
- **[KMS Keys](https://eu-central-1.console.aws.amazon.com/kms/home?region=eu-central-1#/kms/keys): Check the KMS page for Customer Managed Keys with the status "Pending deletion."**
  ![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/h7emdddanpjo5c885dbu.jpg)

- **[Secrets Manager](https://eu-central-1.console.aws.amazon.com/secretsmanager/listsecrets?region=eu-central-1): Adjust the settings to view deleted secrets as shown in the provided image.**
  ![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7eiu7dpwz853t1juz9jk.jpg)
Enter fullscreen mode Exit fullscreen mode

Important Considerations

Choosing the Right Alarms Filter

In a production environment, you might want to be instantly notified of any deletion attempts. For staging or development, it may make more sense to only receive notifications when someone tries to access secrets or KMS keys marked for deletion. Consider the application lifecycle, especially if secrets or encryption keys are rarely used.

Deletion Grace Period

When setting up alarms for pending deletion of KMS keys and AWS Secrets, it's crucial to consider the deletion grace period. This period should be long enough to allow responsible parties to react to any unauthorized deletion attempts.

For KMS keys, AWS enforces a minimum waiting period of 7 days, with a default and maximum of 30 days. For Secrets Manager secrets, the minimum waiting period is also 7 days, with a default and maximum of 30 days.

When configuring your deletion grace periods, consider the following:

  • Team Responsiveness: How quickly can your team typically respond to alerts?
  • Criticality of the Resource: More critical resources might warrant longer grace periods.

To set an appropriate grace period:

# Creating an AWS kms key with 14 day for the deletion grace period

resource "aws_kms_key" "example" {
  description             = "Example KMS key"
  deletion_window_in_days = 14  # Adjust this value as needed (7-30 days)
}

# Creating a secret with 14 day for the deletion grace period

resource "aws_secretsmanager_secret" "example" {
  name                    = "example-secret"
  recovery_window_in_days = 14  # Adjust this value as needed (7-30 days)
}
Enter fullscreen mode Exit fullscreen mode

By setting an appropriate grace period, you give your team ample time to respond to alerts and prevent accidental or malicious deletions of critical resources.

Proper Access Rights

Ensure that access rights are configured properly to prevent unauthorized access and deletion of secrets and KMS keys. This includes implementing the principle of least privilege and regularly reviewing access policies to limit exposure.

Use SCP on the Organization Level

To further limit the risk of deletion, use Service Control Policies (SCPs) at the organization level. SCPs allow you to set permission guardrails that apply to all accounts within an AWS Organization. By using SCPs, you can prevent the deletion of critical resources such as secrets and KMS keys across all accounts, ensuring a higher level of security and compliance.

Cross-Account Protection

For enhanced security, consider placing production-critical KMS keys and secrets in a separate account. This isolation adds an additional layer of protection against unauthorized access and deletion. For more details on setting up cross-account and cross-region backups, refer to another tecRacer blog post on Automating Cross-Account/Cross-Region Backups with AWS Backup in AWS Organizations.

Troubleshooting

  1. If you're not receiving alarm notifications:
    • Check if you've confirmed the SNS topic subscription.
    • Verify that the CloudWatch log group exists and contains the expected log events.
  2. If alarms are triggering too frequently:
    • Review your metric filter patterns and adjust if necessary.
  3. If deletion of KMS keys or secrets are not being captured:
    • Ensure that CloudTrail is properly configured to log these events.
    • Verify that the IAM role has the necessary permissions to write to CloudWatch Logs.

Conclusion

By implementing these alarms, you significantly enhance your AWS environment's security posture. You'll be promptly informed about potentially malicious activities or misconfigurations related to KMS keys and Secrets Manager secrets. Remember, proactive security measures are crucial in cloud computing.

References

— Alexey

Image
Langkah Naikkan Pemasukan Taruhan Sabung Ayam Online
Image
Zamknij Hałas: Sztuka Wyciszania Ścian w Twoim Domu
Image
How Heavy-Duty Cooling Solutions Can Improve Vehicle Performance
Image
Complete Guide to Garage Door Repair in Seattle: Expert Solutions for Your Home
Image
L Shape Bunks Explained In Fewer Than 140 Characters
Image
Building a Creative Career: My Ongoing Interior Design Diploma

design, architecture, news, learning
Image
How At-Home Care Can Support Independence for Seniors
Image
Cypress Alternatives: Exploring Top Testing Tools for Web Applications

webdev, javascript, programming, tutorial
Image
Replace Bifold Doors: The Good, The Bad, And The Ugly
Image
How to add a card to Apple Wallet and Google Wallet 📱 in React Native?
Image
The Adventures of Blink S2e9: Gathering Metrics with Prometheus and Grafana

monitoring, flask, python, devops
Image
C debugger Not able to Print or take Input when in debugger mode is On

vscode, c, debugging
Image
How Coffee Pot Drip Became The Hottest Trend Of 2024
Image
Drive Business Success with Custom iOS App Development – Partner with Webronex Today

ios, iosapp, iosappdevelopmentcompany, iosappdeveloper
Image
How Xbanking is Simplifying Crypto for Everyone

web3, cryptocurrency, blockchain
Image
This Is The Drip Pot Coffee Maker Case Study You'll Never Forget
Image
Realitas Sabung Ayam Online yang Popular di Internet
Image
Spokój w sercu miasta: Najskuteczniejsze sposoby na wyciszenie mieszkań
Image
Understanding Stored XSS Attacks and How to Mitigate Them with Hono

security, xss, hono
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .