AWS Secrets Manager for ECS: Rotation, IAM, and Audit Trails

A database password baked into a task definition is the same password sitting in your CloudWatch Logs the first time someone runs env inside the container. It's in your Terraform state file, your CI build logs, and the laptop of every engineer who has ever pulled the repo. Secrets Manager fixes that—but only if you wire it up correctly. This guide covers the production setup: ECS integration, least-privilege IAM, automatic rotation, cross-account access, and the audit trail your SOC2 auditor will ask for.

Secrets Manager vs. SSM Parameter Store

Both can store secrets. They are not interchangeable.

Rule of thumb: Use Secrets Manager for anything that rotates (database credentials, API keys with expiry, signing keys). Use SSM Parameter Store for config that happens to be sensitive but doesn't rotate (third-party API tokens you can't rotate without manual intervention, license keys). Don't pay $0.40/month for a value that never changes.

Storing a Secret

resource "aws_secretsmanager_secret" "api_db_password" {
  name                    = "production/api/db-password"
  description             = "Aurora password for the api service"
  kms_key_id              = aws_kms_key.secrets.arn
  recovery_window_in_days = 30

  tags = {
    Environment = "production"
    Service     = "api"
    Rotation    = "30d"
  }
}

resource "aws_secretsmanager_secret_version" "api_db_password" {
  secret_id     = aws_secretsmanager_secret.api_db_password.id
  secret_string = jsonencode({
    username = "api_user"
    password = random_password.api_db.result
    host     = aws_rds_cluster.api.endpoint
    port     = 5432
    dbname   = "api"
  })

  lifecycle {
    ignore_changes = [secret_string]  # Managed by rotation Lambda
  }
}

resource "random_password" "api_db" {
  length  = 32
  special = true
}

Naming convention matters. production/api/db-password is greppable, scopes IAM cleanly with wildcards (arn:aws:secretsmanager:*:*:secret:production/api/*), and prevents the db-password-prod-v2-FINAL graveyard. Pick a <env>/<service>/<purpose> pattern and enforce it with an SCP if you have AWS Organizations (see SCP guide).

recovery_window_in_days = 30 gives you 30 days to undelete a secret. The default is 30; setting it to 0 allows immediate deletion. Never use 0 in production—a fat-finger terraform destroy should be recoverable.

Wiring Into an ECS Task Definition

Two paths: inject as environment variable, or mount as a file. Environment variables are simpler; files are better for multi-line secrets (TLS keys) and avoid leaking into docker inspect.

Environment variable injection

resource "aws_ecs_task_definition" "api" {
  family                   = "api"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = "1024"
  memory                   = "2048"
  execution_role_arn       = aws_iam_role.api_execution.arn
  task_role_arn            = aws_iam_role.api_task.arn

  container_definitions = jsonencode([{
    name      = "api"
    image     = "${aws_ecr_repository.api.repository_url}:${var.image_tag}"
    essential = true

    portMappings = [{ containerPort = 8080, protocol = "tcp" }]

    secrets = [
      {
        name      = "DB_PASSWORD"
        valueFrom = "${aws_secretsmanager_secret.api_db_password.arn}:password::"
      },
      {
        name      = "DB_HOST"
        valueFrom = "${aws_secretsmanager_secret.api_db_password.arn}:host::"
      },
      {
        name      = "STRIPE_API_KEY"
        valueFrom = aws_secretsmanager_secret.stripe_key.arn
      }
    ]

    environment = [
      { name = "NODE_ENV", value = "production" },
      { name = "AWS_REGION", value = "us-east-1" }
    ]

    logConfiguration = {
      logDriver = "awslogs"
      options = {
        awslogs-group         = aws_cloudwatch_log_group.api.name
        awslogs-region        = "us-east-1"
        awslogs-stream-prefix = "api"
      }
    }
  }])
}

The valueFrom JSON-key syntax: arn:...:secret:name:json-key:version-stage:version-id. Trailing colons are required even when empty.

• arn:...:secret-name → entire JSON blob as a string
• arn:...:secret-name:password:: → just the password field, current version
• arn:...:secret-name:password:AWSPREVIOUS: → previous version (useful during rotation cutover)

ECS resolves these at task launch using the execution role, not the task role. This is the most common wiring mistake.

IAM: Execution Role vs. Task Role

These are different roles with different jobs. Confusing them is the second most common wiring mistake.

Execution role (for the secrets block)

resource "aws_iam_role" "api_execution" {
  name = "ecs-execution-api"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = { Service = "ecs-tasks.amazonaws.com" }
      Action = "sts:AssumeRole"
    }]
  })
}

resource "aws_iam_role_policy_attachment" "api_execution_managed" {
  role       = aws_iam_role.api_execution.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}

# Least-privilege secrets access: scoped to specific secret ARNs only
resource "aws_iam_role_policy" "api_execution_secrets" {
  name = "api-execution-secrets"
  role = aws_iam_role.api_execution.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = ["secretsmanager:GetSecretValue"]
        Resource = [
          aws_secretsmanager_secret.api_db_password.arn,
          aws_secretsmanager_secret.stripe_key.arn,
        ]
      },
      {
        Effect = "Allow"
        Action = ["kms:Decrypt"]
        Resource = [aws_kms_key.secrets.arn]
        Condition = {
          StringEquals = {
            "kms:ViaService" = "secretsmanager.us-east-1.amazonaws.com"
          }
        }
      }
    ]
  })
}

Three things every secrets-policy review should check:

1. Resource is a specific ARN, never *. A wildcard means "every secret in the account," including secrets from other services and other environments.
2. kms:Decrypt is conditioned on kms:ViaService. Without this, the role can decrypt anything KMS-managed (S3, EBS, RDS snapshots) using your secrets key. The condition limits decryption to calls that flow through Secrets Manager.
3. No secretsmanager:DescribeSecret for the execution role. It doesn't need to list secrets—only fetch the ones it's told about.

Task role (for app-initiated calls)

If your app fetches a secret at runtime (rotation, on-demand access to a different secret), use the task role:

resource "aws_iam_role_policy" "api_task_secrets" {
  name = "api-task-secrets"
  role = aws_iam_role.api_task.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"]
      Resource = ["arn:aws:secretsmanager:us-east-1:*:secret:production/api/*"]
    }]
  })
}

Wildcard in the resource is acceptable here because it's scoped to the production/api/ prefix—the naming convention pays off.

Automatic Rotation

Rotation is the entire reason to pay $0.40/month per secret. A static password that has never rotated is a credential that has been leaked at least once—you just don't know it yet.

RDS/Aurora rotation (AWS-managed)

For RDS, Aurora, Redshift, and DocumentDB, AWS provides ready-made rotation Lambdas. You don't write code:

resource "aws_secretsmanager_secret_rotation" "api_db_password" {
  secret_id           = aws_secretsmanager_secret.api_db_password.id
  rotation_lambda_arn = aws_serverlessapplicationrepository_cloudformation_stack.rotator.outputs.RotationLambdaARN

  rotation_rules {
    automatically_after_days = 30
  }
}

resource "aws_serverlessapplicationrepository_cloudformation_stack" "rotator" {
  name           = "SecretsManagerRDSPostgreSQLRotation"
  application_id = "arn:aws:serverlessrepo:us-east-1:297356227824:applications/SecretsManagerRDSPostgreSQLRotationSingleUser"
  capabilities   = ["CAPABILITY_IAM", "CAPABILITY_RESOURCE_POLICY"]

  parameters = {
    endpoint            = "https://secretsmanager.us-east-1.amazonaws.com"
    functionName        = "rotator-api-db-password"
    vpcSubnetIds        = join(",", var.private_subnet_ids)
    vpcSecurityGroupIds = aws_security_group.rotator.id
  }
}

Single-user vs. multi-user (alternating) rotation: Single-user is simpler—one DB user, password rotates in place. Multi-user creates two DB users and alternates which one your app uses. Multi-user is zero-downtime; single-user has a sub-second window where the old password is invalid but task definitions still reference it.

For services running more than one task, single-user rotation can cause brief connection errors during the swap. Use multi-user for anything customer-facing.

Rotation staging labels

During rotation, Secrets Manager goes through four steps via staging labels:

createSecret  → generates new password, stores as AWSPENDING
setSecret     → updates the database with the new password
testSecret    → verifies the new credentials work
finishSecret  → promotes AWSPENDING to AWSCURRENT, demotes old to AWSPREVIOUS

AWSCURRENT and AWSPREVIOUS exist simultaneously for a window. This is how multi-user rotation achieves zero downtime—your app can fetch AWSPREVIOUS for grace-period connections during the swap.

Forcing ECS to pick up rotated secrets

Here's the catch: ECS injects secrets at task launch. A running task does not see rotated values. After rotation, you need new tasks to pick up the new password.

Options:

1. Trigger a service update when rotation completes. EventBridge rule on aws.secretsmanager events → Lambda → aws ecs update-service --force-new-deployment.
2. App-side fetch with a cache TTL. The app reads via Secrets Manager SDK on a 5-minute TTL using the task role. Slightly more code; survives rotation without redeploy.

resource "aws_cloudwatch_event_rule" "secret_rotated" {
  name = "secrets-manager-rotation-trigger-redeploy"

  event_pattern = jsonencode({
    source      = ["aws.secretsmanager"]
    detail-type = ["AWS API Call via CloudTrail"]
    detail = {
      eventName = ["RotationSucceeded"]
      requestParameters = {
        secretId = [aws_secretsmanager_secret.api_db_password.arn]
      }
    }
  })
}

resource "aws_cloudwatch_event_target" "trigger_redeploy" {
  rule      = aws_cloudwatch_event_rule.secret_rotated.name
  target_id = "trigger-ecs-redeploy"
  arn       = aws_lambda_function.force_new_deployment.arn
}

The Lambda is a 10-line boto3 call: ecs.update_service(cluster=..., service=..., forceNewDeployment=True). Pair this with the ECS deployment circuit breaker—if the rotated secret is malformed, the breaker rolls back automatically.

KMS: AWS-Managed Key vs. Customer-Managed Key

Every secret is encrypted at rest with KMS. By default, Secrets Manager uses the AWS-managed key aws/secretsmanager. Free, no setup. Sufficient for many workloads.

You need a customer-managed key (CMK) when:

• You need cross-account access (resource policy on the CMK)
• You need to deny KMS access via SCP without breaking Secrets Manager elsewhere
• Your auditor wants per-application key isolation
• You need to revoke access fast (disable the CMK, all secrets become unreadable)

resource "aws_kms_key" "secrets" {
  description             = "Production secrets encryption key"
  deletion_window_in_days = 30
  enable_key_rotation     = true

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid       = "AllowRootAccount"
        Effect    = "Allow"
        Principal = { AWS = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:root" }
        Action    = "kms:*"
        Resource  = "*"
      },
      {
        Sid       = "AllowSecretsManagerService"
        Effect    = "Allow"
        Principal = { Service = "secretsmanager.amazonaws.com" }
        Action = [
          "kms:Encrypt",
          "kms:Decrypt",
          "kms:ReEncrypt*",
          "kms:GenerateDataKey*",
          "kms:DescribeKey"
        ]
        Resource = "*"
      }
    ]
  })
}

resource "aws_kms_alias" "secrets" {
  name          = "alias/production-secrets"
  target_key_id = aws_kms_key.secrets.id
}

enable_key_rotation = true is non-negotiable. AWS rotates the underlying key material annually with zero application impact. There is no reason to leave it off.

Cross-Account Access

Multi-account organizations frequently need a sandbox or CI account to read a secret from a production secret store (e.g., a CI pipeline pulling a deploy key). You need both an IAM policy in the source account and a resource policy on the secret and a key policy on the KMS CMK.

resource "aws_secretsmanager_secret_policy" "deploy_key" {
  secret_arn = aws_secretsmanager_secret.deploy_key.arn

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { AWS = "arn:aws:iam::${var.ci_account_id}:role/github-actions-deploy" }
      Action    = "secretsmanager:GetSecretValue"
      Resource  = aws_secretsmanager_secret.deploy_key.arn
    }]
  })
}

Then update the KMS key policy in the production account to allow kms:Decrypt for the CI role. The CI role's IAM policy needs secretsmanager:GetSecretValue on the cross-account ARN. All three layers must align—missing any one fails with a generic AccessDenied.

Audit Trail: Detecting Leaked Secrets

Every GetSecretValue is logged to CloudTrail. That's the easy part. Turning that into actionable detection is the work.

The threat: a credential is leaked. An attacker calls GetSecretValue from an EC2 instance in a region you don't operate in, or from an IP outside your VPC. You want to know within minutes.

resource "aws_cloudwatch_event_rule" "anomalous_secret_access" {
  name = "secrets-manager-anomalous-access"

  event_pattern = jsonencode({
    source      = ["aws.secretsmanager"]
    detail-type = ["AWS API Call via CloudTrail"]
    detail = {
      eventName = ["GetSecretValue"]
      # Alert on access from outside expected regions
      awsRegion = [{ "anything-but" = ["us-east-1", "us-east-2"] }]
    }
  })
}

resource "aws_cloudwatch_event_target" "page_oncall" {
  rule      = aws_cloudwatch_event_rule.anomalous_secret_access.name
  target_id = "page-oncall"
  arn       = aws_sns_topic.security_alerts.arn
}

This is the same EventBridge → SNS pattern used by the ECS deployment circuit breaker post. Once you have an SNS topic for security events, attach more rules to it:

• GetSecretValue calls from a role that hasn't called it in 30 days (use GuardDuty's anomaly detection or a custom Athena query against CloudTrail)
• Bulk reads: more than N GetSecretValue calls per minute from a single principal
• Reads of production secrets from non-production accounts
• Any DeleteSecret or UpdateSecret outside business hours

Pipe the CloudTrail log group to an Athena table for query-on-demand investigations:

SELECT useridentity.arn, sourceipaddress, awsregion, eventtime, requestparameters
FROM cloudtrail_logs
WHERE eventsource = 'secretsmanager.amazonaws.com'
  AND eventname = 'GetSecretValue'
  AND eventtime > current_timestamp - interval '24' hour
ORDER BY eventtime DESC;

SOC2 Evidence

What your auditor will ask for, and where it lives:

For Vanta/Drata integration, point the SOC2 controls at the CloudTrail S3 bucket and IAM. The evidence collection is automatic once the controls are correctly configured—see the Vanta SOC2 implementation guide.

Common Wiring Mistakes

Production Checklist

• Naming convention enforced: <env>/<service>/<purpose>
• Customer-managed KMS key with rotation enabled
• IAM execution role policy scoped to specific secret ARNs (no *)
• kms:Decrypt conditioned on kms:ViaService for Secrets Manager
• Automatic rotation enabled with ≤ 30 day cadence on database credentials
• Multi-user rotation for customer-facing services
• EventBridge rule triggering force-new-deployment on rotation success
• CloudTrail data events enabled for Secrets Manager
• EventBridge rules alerting on cross-region or anomalous GetSecretValue
• Recovery window ≥ 30 days on every production secret
• No secrets in Terraform state under version control (use remote state with encryption)
• No secrets in CI build logs (mask them in the CI provider)

Conclusion

Secrets Manager is cheap, but the wiring is where engineering teams get hurt. The defaults are not safe—wildcards in IAM policies, no rotation, no alerting on reads, and ECS tasks that quietly cache stale credentials past a rotation. Get the execution role scoped, turn on rotation, wire the EventBridge trigger so deploys pick up new values, and pipe CloudTrail into alerts.

Key principles:

• Use Secrets Manager for anything that rotates; SSM Parameter Store for static config
• Scope IAM policies to specific secret ARNs—never Resource = "*"
• Use the execution role for ECS secrets injection, not the task role
• Enable rotation on every database credential, and wire ECS to pick up new values automatically
• Customer-managed KMS keys for cross-account and revoke-fast scenarios
• Every GetSecretValue is auditable—turn the logs into alerts before you need them

---

Want help auditing your secrets posture or wiring rotation across a multi-account org? Let's talk—we'll find the wildcards before your auditor does.