In cloud computing, security is a critical concern. While AWS provides backup solutions for many resources, custom configurations often require additional protection. Two key services, AWS Key Management Service (KMS) and AWS Secrets Manager, don't offer direct backup options. However, they implement a deletion grace period— by default 30 days and this is the maximum — allowing for potential restoration.

A crucial question arises: How can you be notified of a pending deletion? This is where alarms play a vital role. Implementing alarms for AWS KMS keys and AWS Secrets Manager secrets that are pending deletion can significantly enhance your security posture.

This blog post demonstrates how to create these essential alarms using Terraform, an infrastructure-as-code tool. By leveraging Terraform, you can automate the process of setting up notifications for potential security risks, ensuring that your sensitive information and encryption keys remain protected.

We'll walk through the process of configuring alarms that alert you when KMS keys or Secrets Manager secrets are scheduled for deletion, giving you time to review and potentially prevent unintended data loss or security breaches.

Prerequisites

Before we begin, ensure you have:

An AWS account with appropriate permissions
Terraform installed and configured
Basic knowledge of AWS services, particularly CloudTrail, KMS, Secrets Manager, and CloudWatch
Familiarity with Terraform syntax and AWS provider

Step-by-Step Guide

1. Setting Up Terraform Variables

First, define the necessary variables in your terraform.tfvars or within your Terraform configuration. These variables include the notification email, the name of the CloudWatch log group attached to CloudTrail logs, and the AWS region.

variable "notification_email"
{
    description = "The email address to notify in case of an alarm"
  type        = string
}

variable "cloud_trail_cloudwatch_log_group" {
  description = "The name of the CloudWatch log group attached to CloudTrail logs"
  type        = string
}

variable "region" {
  description = "The AWS region"
  default     = "eu-central-1"
  type        = string
}

2. Configuring the AWS Provider and Resources

Ensure your AWS provider is configured with the specified region. We also need the information for the AWS CloudWatch log group to create the resources.

provider "aws" {
  region = var.region
}

data "aws_cloudwatch_log_group" "this" {
  name = var.cloud_trail_cloudwatch_log_group
}

Create an IAM role and policy that allows CloudTrail to interact with CloudWatch logs. Additionally, set up an SNS topic for sending notifications and subscribe it to the provided email.

resource "aws_iam_role" "this" {
  name = "CloudTrailCloudWatchLogsRole"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "cloudtrail.amazonaws.com"
        }
      },
    ]
  })
}

resource "aws_iam_role_policy_attachment" "cloudtrail_cloudwatch_logs" {
  role       = aws_iam_role.this.name
  policy_arn = aws_iam_policy.this.arn
}

# SNS Topic for Notifications

resource "aws_sns_topic" "pending_deletion_topic" {
  name = "kms-key-pending-deletion-topic"
}

# Subscription to send notifications to an email

resource "aws_sns_topic_subscription" "email_subscription" {
  topic_arn = aws_sns_topic.pending_deletion_topic.arn
  protocol  = "email"
  endpoint  = var.notification_email
}

3. Detecting Deletion Attempts with CloudWatch Log Metric Filters

Use CloudWatch log metric filters to detect deletion attempts or unauthorized access. For AWS Secrets Manager, filter events where a pending deletion secret is accessed. For AWS KMS, filter events where a pending deletion key is used.

# Metric filter to detect the attempt to access secrets marked for deletion

resource "aws_cloudwatch_log_metric_filter" "secrets_deletion_attempts_filter" {
  name           = "SecretsDeletionAttempts"
  pattern        = "{ $.eventName = \"GetSecretValue\" && $.errorMessage = \"*secret because it was marked for deletion*\" }"
  log_group_name = data.aws_cloudwatch_log_group.this.name
  metric_transformation {
    name      = "SecretsDeletionAttempts"
    namespace = "SecretsManagerMetrics"
    value     = "1"
  }
}

If you want to receive notifications not only for attempts to access secret values but also immediately after an attempt to delete secrets, you can change the pattern to $.eventName = \"DeleteSecret\" instead of $.eventName = \"GetSecretValue\"

# Metric filter to detect the attempt to access MS keys marked for deletion

resource "aws_cloudwatch_log_metric_filter" "kms_key_pending_deletion_filter" {
  name           = "KMSKeyPendingDeletion"
  pattern        = "{ $.eventSource = kms* && $.errorMessage = \"* is pending deletion.\"}"
  log_group_name = data.aws_cloudwatch_log_group.this.name
  metric_transformation {
    name      = "KMSKeyPendingDeletionEvents"
    namespace = "CloudTrailMetrics"
    value     = "1"
  }
}

If you want to receive notifications not only for attempts to access keys but also immediately after an attempt to delete keys, you can change the pattern to $.eventSource = kms* instead of $.eventSource = ScheduleKeyDeletion*

4. Creating CloudWatch Alarms

Finally, create CloudWatch alarms based on the metric filters. These alarms will trigger when an attempt is made to access a secret or use a pending deletion KMS key.

# CloudWatch Alarm for KMS Key Pending Deletion

resource "aws_cloudwatch_metric_alarm" "kms_key_pending_deletion_alarm" {
  alarm_name          = "KMSKeyPendingDeletionAlarm"
  comparison_operator = "GreaterThanOrEqualToThreshold"
  evaluation_periods  = "1"
  metric_name         = aws_cloudwatch_log_metric_filter.kms_key_pending_deletion_filter.metric_transformation[0].name
  namespace           = aws_cloudwatch_log_metric_filter.kms_key_pending_deletion_filter.metric_transformation[0].namespace
  period              = 300
  statistic           = "Sum"
  threshold           = 1
  alarm_description   = "Alarm when KMS key pending deletion is used."
  alarm_actions       = [aws_sns_topic.pending_deletion_topic.arn]
  treat_missing_data  = "notBreaching"
}

# CloudWatch Alarm for Secrets Manager Secrets Deletion Attempts

resource "aws_cloudwatch_metric_alarm" "secrets_deletion_attempts_alarm" {
  alarm_name          = "SecretsDeletionAttemptsAlarm"
  comparison_operator = "GreaterThanOrEqualToThreshold"
  evaluation_periods  = "1"
  metric_name         = aws_cloudwatch_log_metric_filter.secrets_deletion_attempts_filter.metric_transformation[0].name
  namespace           = aws_cloudwatch_log_metric_filter.secrets_deletion_attempts_filter.metric_transformation[0].namespace
  period              = 300
  statistic           = "Sum"
  threshold           = 1
  alarm_description   = "Alarm when an attempt to delete an AWS Secrets Manager secret is made."
  alarm_actions       = [aws_sns_topic.pending_deletion_topic.arn]
  treat_missing_data  = "notBreaching"
}

Operation

After running Terraform, a confirmation message will be sent to the email address registered in Terraform. You must confirm the subscription in this message.
1. In case of an alarm, you will receive a warning message to the registered email address.

- **[KMS Keys](https://eu-central-1.console.aws.amazon.com/kms/home?region=eu-central-1#/kms/keys): Check the KMS page for Customer Managed Keys with the status "Pending deletion."**
  ![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/h7emdddanpjo5c885dbu.jpg)

- **[Secrets Manager](https://eu-central-1.console.aws.amazon.com/secretsmanager/listsecrets?region=eu-central-1): Adjust the settings to view deleted secrets as shown in the provided image.**
  ![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7eiu7dpwz853t1juz9jk.jpg)

Important Considerations

Choosing the Right Alarms Filter

In a production environment, you might want to be instantly notified of any deletion attempts. For staging or development, it may make more sense to only receive notifications when someone tries to access secrets or KMS keys marked for deletion. Consider the application lifecycle, especially if secrets or encryption keys are rarely used.

Deletion Grace Period

When setting up alarms for pending deletion of KMS keys and AWS Secrets, it's crucial to consider the deletion grace period. This period should be long enough to allow responsible parties to react to any unauthorized deletion attempts.

For KMS keys, AWS enforces a minimum waiting period of 7 days, with a default and maximum of 30 days. For Secrets Manager secrets, the minimum waiting period is also 7 days, with a default and maximum of 30 days.

When configuring your deletion grace periods, consider the following:

Team Responsiveness: How quickly can your team typically respond to alerts?
Criticality of the Resource: More critical resources might warrant longer grace periods.

To set an appropriate grace period:

# Creating an AWS kms key with 14 day for the deletion grace period

resource "aws_kms_key" "example" {
  description             = "Example KMS key"
  deletion_window_in_days = 14  # Adjust this value as needed (7-30 days)
}

# Creating a secret with 14 day for the deletion grace period

resource "aws_secretsmanager_secret" "example" {
  name                    = "example-secret"
  recovery_window_in_days = 14  # Adjust this value as needed (7-30 days)
}

By setting an appropriate grace period, you give your team ample time to respond to alerts and prevent accidental or malicious deletions of critical resources.

Proper Access Rights

Ensure that access rights are configured properly to prevent unauthorized access and deletion of secrets and KMS keys. This includes implementing the principle of least privilege and regularly reviewing access policies to limit exposure.

Use SCP on the Organization Level

To further limit the risk of deletion, use Service Control Policies (SCPs) at the organization level. SCPs allow you to set permission guardrails that apply to all accounts within an AWS Organization. By using SCPs, you can prevent the deletion of critical resources such as secrets and KMS keys across all accounts, ensuring a higher level of security and compliance.

Cross-Account Protection

For enhanced security, consider placing production-critical KMS keys and secrets in a separate account. This isolation adds an additional layer of protection against unauthorized access and deletion. For more details on setting up cross-account and cross-region backups, refer to another tecRacer blog post on Automating Cross-Account/Cross-Region Backups with AWS Backup in AWS Organizations.

Troubleshooting

If you're not receiving alarm notifications:
- Check if you've confirmed the SNS topic subscription.
- Verify that the CloudWatch log group exists and contains the expected log events.
If alarms are triggering too frequently:
- Review your metric filter patterns and adjust if necessary.
If deletion of KMS keys or secrets are not being captured:
- Ensure that CloudTrail is properly configured to log these events.
- Verify that the IAM role has the necessary permissions to write to CloudWatch Logs.

Conclusion

By implementing these alarms, you significantly enhance your AWS environment's security posture. You'll be promptly informed about potentially malicious activities or misconfigurations related to KMS keys and Secrets Manager secrets. Remember, proactive security measures are crucial in cloud computing.

References

— Alexey

Proactively Securing AWS: Automated Alarms for Pending Deletion KMS Keys and Secrets Using Terraform