Home › AWS › AWS data services › KMS

AWS data services · Security, identity and network · about 10 min read

AWS Key Management Service (KMS)

Managed encryption keys that never leave AWS; one key per catalog lets you see, limit and revoke every use of that data.

RoleSecurity, identity and network
StatusGenerally available
Verdict on this sitePursue
AzureAzure Key Vault (keys) and Managed HSM
Google CloudCloud KMS and Cloud HSM
DatabricksCustomer-managed keys for managed services and workspace storage, which point at KMS keys
Open sourceHashiCorp Vault transit engine
Verdict: Pursue. One customer managed key per catalog, with its own bucket, so revoking all access to the recipes is one action and every use of the key is logged in CloudTrail. Turn on S3 Bucket Keys so the request bill and throttling stay small.

In plain words

What it is

AWS Key Management Service (KMS) creates and holds encryption keys and does the encrypting and decrypting for you. The key material never leaves KMS in readable form. Services such as S3, EBS, Kinesis and Secrets Manager call KMS when they write or read encrypted data, and every call is checked against a key policy and recorded in CloudTrail.

Most data is not encrypted with the KMS key directly. KMS uses envelope encryption: it generates a short-lived data key, the service encrypts the file with that data key, and the data key itself is stored next to the file, encrypted under the KMS key. To read the file, the service asks KMS to decrypt the small data key. So whoever is not allowed to use the KMS key cannot read the file, even if they can download it.

There are three kinds of key. AWS owned keys are used by AWS inside its services and you cannot see them. AWS managed keys, with names like aws/s3, are created for you and you cannot control their policy. Customer managed keys are yours: you write the key policy, turn on rotation, disable or delete them, and see every use. A data platform that wants to answer the question who could read this needs customer managed keys.

Take a tyre manufacturer whose rubber compound recipes are its most valuable data. The recipes catalog lives in its own S3 bucket, encrypted with its own KMS key. Only the recipes storage role and two named break-glass roles appear in the key policy. If the company suspects a leak, disabling that one key stops every read of every recipe file within seconds, in every tool, and CloudTrail shows exactly which role decrypted which data key and when.

A bank vault that keeps the master keys, cuts a fresh small key for every box, and writes the name of everyone who asks to open a box in a ledger.

Big picture

Where it fits

SourcesMove inStreamStoreProcessServeUseAcross everything: catalog · security · orchestration · observability and cost · infrastructure as codegets data fromhands data toIAM and STSTerraform key definitionsKMSSecurity, identity and networkS3Kinesis Data StreamsSecrets Manager

The lit lane is where KMS works. Boxes on the left feed it; boxes on the right use what it produces.

KMS sits beside storage in the reference platform rather than in a lane of its own. The design rule is one bucket and one key per catalog: landing, bronze, silver, gold, the recipes catalog and the token vault each have their own key. The landing bucket per environment has its own key too. Kinesis streams, Secrets Manager secrets and the CloudTrail log bucket are encrypted with customer managed keys as well.

Databricks uses the keys in two places. Unity Catalog storage roles get kms:Decrypt, kms:Encrypt and kms:GenerateDataKey on their catalog's key, and the key policy names that role, so Databricks can read and write the bucket. Separately, Databricks customer-managed keys, an Enterprise tier feature, point the workspace at KMS keys for its own managed services data, such as notebooks, queries and secrets in the control plane, and for workspace storage. Losing access to those keys makes the workspace unusable, so treat them as carefully as the data keys.

For recovery in Hyderabad, data replicated from Mumbai must be encrypted again under a key in the second region. Either replication names a separate key in Hyderabad, or the platform uses multi-Region keys, which share key material across regions so the replica can be read with the same key identity. KMS key use logs from CloudTrail are joined with Databricks audit tables in the ops catalog to build the quarterly access report.

How it works

The ideas you need

IdeaWhat it means
Customer managed keyA KMS key you create and control: its policy, rotation, enable or disable, and deletion.
Key policyThe resource policy on a key. Every key has exactly one, and IAM policies only work on the key if the key policy allows them to.
Envelope encryptionData is encrypted with a short-lived data key; the data key is encrypted with the KMS key and stored next to the data.
S3 Bucket KeyA bucket-level data key that S3 reuses for many objects, cutting KMS requests by up to 99 percent for SSE-KMS buckets.
SSE-KMSS3 server-side encryption with a KMS key. Reads need permission on the bucket and on the key.
GrantA temporary, narrow permission on a key that AWS services create for themselves, for example when EBS attaches a volume.
Key rotationKMS makes new key material on a schedule, by default every year, and keeps the old material to decrypt old data. Key ID and policy stay the same.
Multi-Region keyRelated keys in several regions that share key material, so data encrypted in Mumbai can be decrypted in Hyderabad.
Encryption contextExtra non-secret values bound to an encryption call, logged in CloudTrail and usable in policy conditions.

Choosing

When to use it, and when not

Use it when

  • Use a customer managed key per catalog bucket so access can be revoked per data domain and audited per key.
  • Use KMS for every place sensitive data rests: S3, Kinesis, EBS volumes of classic clusters, Secrets Manager and log buckets.
  • Use Databricks customer-managed keys when policy requires that the company controls the keys for notebooks, query history and workspace storage.
  • Use multi-Region keys or a planned key per region when data is replicated for disaster recovery.
  • Use encryption context and key policy conditions when you need to prove which service and which bucket used a key.

Avoid it when

  • Avoid one shared key for every bucket. Revoking it stops the whole platform, and its logs cannot tell catalogs apart.
  • Avoid customer managed keys on buckets with very high request rates and no Bucket Keys; the request bill and throttling will surprise you.
  • Avoid external key stores or CloudHSM-backed keys unless a regulator requires keys outside AWS-managed hardware; they add latency, cost and a new way to fail.
  • Avoid using KMS directly to encrypt large files; it accepts at most 4 KB of plaintext per call and is built for keys, not data.

Production grade

What goes wrong in production

ProblemWhat breaksWhat to do
Access denied on readA role can read the bucket but gets AccessDenied because it cannot decrypt the data key.Grant kms:Decrypt to the role and allow it in the key policy; check CloudTrail for the failing Decrypt call and its key ARN.
KMS throttlingSpark jobs reading millions of small files make one Decrypt call per object and hit the shared request quota.Turn on S3 Bucket Keys, compact small files, and watch the request rate against the quota in Service Quotas.
Cross-account confusionAnother account's role uses your key and the throttling counts against the caller, while the key policy must allow the other account.Name exact principals in the key policy; avoid granting the whole account root unless IAM policies there are tightly controlled.
Replication and keysS3 replication to Hyderabad fails or skips objects encrypted with SSE-KMS.Enable replication of SSE-KMS objects, name the destination key, and give the replication role Decrypt on the source key and Encrypt on the destination key.
Accidental key deletionDeleting a key makes everything encrypted under it unreadable forever.Deletion waits 7 to 30 days; use 30. Alarm on ScheduleKeyDeletion and DisableKey events, and block them with an SCP outside the break-glass role.
Databricks workspace key lostRevoking a Databricks customer-managed key for managed services stops the workspace from working.Keep these keys in the security account with change control, and test the rotation runbook in dev first.
Macie and other scanners cannot readSecurity tools fail on objects encrypted with a customer managed key.Add the scanner's service-linked role to the key policy with Decrypt only, as for Macie on landing buckets.

Alternatives

What to compare it with

AlternativePick the alternative whenPick KMS when
S3 managed keys (SSE-S3)The data is low sensitivity and you want encryption with no key cost and no extra policy.You need to revoke access per catalog or prove who decrypted what; SSE-S3 has no key policy or per-key log.
AWS managed keys (aws/s3)You want KMS logging with no setup.You need to control the key policy or share data across accounts; you cannot change the policy of an AWS managed key.
CloudHSM and external key storesA regulator requires keys in single-tenant hardware or held outside AWS.Standard KMS hardware meets your rules; custom key stores cost more and have lower request quotas.
Azure Key VaultThe same design runs on Azure Databricks, with a key per storage account.You are on AWS.
Google Cloud KMSYou compare clouds; the key-per-dataset pattern is the same.You only run AWS.
Databricks default encryptionControl plane data is encrypted by Databricks with its own keys, enough for many teams.Policy requires company-controlled keys for notebooks, query history and workspace storage; then add customer-managed keys.

Cost

How you pay

Each customer managed key costs a flat monthly fee, prorated by the hour, and requests are billed per 10,000 calls. The AWS pricing page gives 1 USD per key per month and 0.03 USD per 10,000 symmetric requests (list price, check your region). Rotation adds the same monthly fee for the first and second rotations only; later rotations are free. AWS managed keys have no monthly fee but their requests are billed. There is a small free tier of requests per month.

What drives the bill

  • Number of S3 objects read and written under SSE-KMS without Bucket Keys, one request each.
  • Number of customer managed keys and their first two rotations.
  • Cross-account use, where the calling account pays for requests.
  • CloudTrail logging volume, since every KMS call is a management event.

Cost traps

  • SSE-KMS on a bronze bucket full of small files, without Bucket Keys, can produce a large request bill and throttling at the same time.
  • A key per table instead of per catalog multiplies monthly fees and policies with no real gain in control.
  • Disabled keys still cost the monthly fee; only keys scheduled for deletion stop billing.

Prices change; check the AWS pricing page for your region before deciding.

Security and governance

Keeping it safe

  • Key policies name exact principals: the catalog storage role, the replication role, scanners with Decrypt only, and a break-glass role. Key administrators cannot use the key for data.
  • Separate who manages keys from who uses them; key administration sits in the security account team, not the data platform team.
  • SCPs deny ScheduleKeyDeletion, DisableKey and PutKeyPolicy outside the security role, and CloudWatch alarms fire on any attempt.
  • Reach KMS through an interface VPC endpoint from private subnets and use the aws:SourceVpce condition on sensitive keys.
  • Automatic yearly rotation is on for every key; the key ID stays the same, so nothing in Terraform or Databricks changes.
  • CloudTrail logs every Decrypt with the caller and encryption context; these logs are evidence for who read personal data and recipes.

Operating it

What to watch, and the limits

WatchWhy
CloudTrail Decrypt and GenerateDataKey eventsWho used each key, how often, and from which role; spikes on the recipes key deserve a look.
ThrottlingException on KMS callsRequests over the shared symmetric quota; usually small files without Bucket Keys.
Service Quotas: cryptographic operations (symmetric) usageRequest rate against the regional quota, with a CloudWatch alarm at 80 percent.
ScheduleKeyDeletion and DisableKey eventsAny attempt to remove or disable a key; page the security team.
SecondsUntilKeyMaterialExpirationFor keys with imported material, how long until the material expires and data becomes unreadable.
LimitValue and notes
Customer managed keys100,000 per account per region, adjustable
Symmetric request rate10,000 per second shared across Decrypt, Encrypt and GenerateDataKey in most regions, including Mumbai; higher in some large regions
Grants per key50,000
Aliases per key50
On-demand rotations25 per key, not adjustable
Deletion waiting period7 to 30 days

Limits and quotas checked 2026-11-11; many can be raised through Service Quotas.

Decisions

Situations and verdicts

SituationChoiceVerdict
The recipes catalog must be revocable in one actionIts own bucket and its own customer managed key, with a narrow key policyPursue: disabling the key stops every read everywhere, and CloudTrail shows every past use.
Bronze holds millions of small sensor files under SSE-KMSTurn on S3 Bucket Keys and compact filesPursue with care: Bucket Keys cut requests sharply, but CloudTrail then shows the bucket, not each object, as the context.
Security asks for company-controlled keys for notebooks and query historyDatabricks customer-managed keys for managed services and workspace storagePursue with care: needs the Enterprise tier, and losing the key stops the workspace.
A regulator is said to want keys held outside AWSExternal key storeDrop: confirm the rule first; for most manufacturers standard KMS with per-catalog keys and logs is enough.

FAQ

Questions people ask

Should I use SSE-S3 or SSE-KMS for a data lake?

Use SSE-KMS with a customer managed key for any catalog with sensitive or valuable data, because it gives a key policy and a log of every use. Turn on Bucket Keys to keep request costs low. SSE-S3 is fine for low-risk scratch data.

Does key rotation break my data or Databricks?

No. Rotation creates new key material but keeps the same key ID and keeps the old material for decrypting old data. Nothing in Unity Catalog, Terraform or bucket settings needs to change.

What happens if a KMS key is disabled?

Every read of data encrypted under it fails with an access error until the key is enabled again. Data is not lost. Deleting a key, after the waiting period, makes the data unreadable for good.

Do I need one key per table?

No. One key per catalog, matching one bucket per catalog, gives revocation and audit at the level people actually manage access. More keys add policy work without more control.

Does Databricks need special KMS setup?

The Unity Catalog storage role needs Decrypt, Encrypt and GenerateDataKey on the bucket key, and the key policy must allow it. Customer-managed keys for the workspace itself are a separate Enterprise tier setting with their own key policies.

Sources

Checked against

Checked 2026-11-11.

Related

Where it connects

AWS and Databricks reference platform · AWS IAM and STS · Amazon S3 · Amazon Macie · S3 Versioning, Replication and Object Lock · All AWS data services

← IAM and STS · Macie →