Home › AWS › AWS data services › KMS
AWS data services · Security, identity and network · about 10 min read
AWS Key Management Service (KMS)
Managed encryption keys that never leave AWS; one key per catalog lets you see, limit and revoke every use of that data.
In plain words
What it is
AWS Key Management Service (KMS) creates and holds encryption keys and does the encrypting and decrypting for you. The key material never leaves KMS in readable form. Services such as S3, EBS, Kinesis and Secrets Manager call KMS when they write or read encrypted data, and every call is checked against a key policy and recorded in CloudTrail.
Most data is not encrypted with the KMS key directly. KMS uses envelope encryption: it generates a short-lived data key, the service encrypts the file with that data key, and the data key itself is stored next to the file, encrypted under the KMS key. To read the file, the service asks KMS to decrypt the small data key. So whoever is not allowed to use the KMS key cannot read the file, even if they can download it.
There are three kinds of key. AWS owned keys are used by AWS inside its services and you cannot see them. AWS managed keys, with names like aws/s3, are created for you and you cannot control their policy. Customer managed keys are yours: you write the key policy, turn on rotation, disable or delete them, and see every use. A data platform that wants to answer the question who could read this needs customer managed keys.
Take a tyre manufacturer whose rubber compound recipes are its most valuable data. The recipes catalog lives in its own S3 bucket, encrypted with its own KMS key. Only the recipes storage role and two named break-glass roles appear in the key policy. If the company suspects a leak, disabling that one key stops every read of every recipe file within seconds, in every tool, and CloudTrail shows exactly which role decrypted which data key and when.
A bank vault that keeps the master keys, cuts a fresh small key for every box, and writes the name of everyone who asks to open a box in a ledger.
Big picture
Where it fits
The lit lane is where KMS works. Boxes on the left feed it; boxes on the right use what it produces.
KMS sits beside storage in the reference platform rather than in a lane of its own. The design rule is one bucket and one key per catalog: landing, bronze, silver, gold, the recipes catalog and the token vault each have their own key. The landing bucket per environment has its own key too. Kinesis streams, Secrets Manager secrets and the CloudTrail log bucket are encrypted with customer managed keys as well.
Databricks uses the keys in two places. Unity Catalog storage roles get kms:Decrypt, kms:Encrypt and kms:GenerateDataKey on their catalog's key, and the key policy names that role, so Databricks can read and write the bucket. Separately, Databricks customer-managed keys, an Enterprise tier feature, point the workspace at KMS keys for its own managed services data, such as notebooks, queries and secrets in the control plane, and for workspace storage. Losing access to those keys makes the workspace unusable, so treat them as carefully as the data keys.
For recovery in Hyderabad, data replicated from Mumbai must be encrypted again under a key in the second region. Either replication names a separate key in Hyderabad, or the platform uses multi-Region keys, which share key material across regions so the replica can be read with the same key identity. KMS key use logs from CloudTrail are joined with Databricks audit tables in the ops catalog to build the quarterly access report.
How it works
The ideas you need
| Idea | What it means |
|---|---|
| Customer managed key | A KMS key you create and control: its policy, rotation, enable or disable, and deletion. |
| Key policy | The resource policy on a key. Every key has exactly one, and IAM policies only work on the key if the key policy allows them to. |
| Envelope encryption | Data is encrypted with a short-lived data key; the data key is encrypted with the KMS key and stored next to the data. |
| S3 Bucket Key | A bucket-level data key that S3 reuses for many objects, cutting KMS requests by up to 99 percent for SSE-KMS buckets. |
| SSE-KMS | S3 server-side encryption with a KMS key. Reads need permission on the bucket and on the key. |
| Grant | A temporary, narrow permission on a key that AWS services create for themselves, for example when EBS attaches a volume. |
| Key rotation | KMS makes new key material on a schedule, by default every year, and keeps the old material to decrypt old data. Key ID and policy stay the same. |
| Multi-Region key | Related keys in several regions that share key material, so data encrypted in Mumbai can be decrypted in Hyderabad. |
| Encryption context | Extra non-secret values bound to an encryption call, logged in CloudTrail and usable in policy conditions. |
Choosing
When to use it, and when not
Use it when
- Use a customer managed key per catalog bucket so access can be revoked per data domain and audited per key.
- Use KMS for every place sensitive data rests: S3, Kinesis, EBS volumes of classic clusters, Secrets Manager and log buckets.
- Use Databricks customer-managed keys when policy requires that the company controls the keys for notebooks, query history and workspace storage.
- Use multi-Region keys or a planned key per region when data is replicated for disaster recovery.
- Use encryption context and key policy conditions when you need to prove which service and which bucket used a key.
Avoid it when
- Avoid one shared key for every bucket. Revoking it stops the whole platform, and its logs cannot tell catalogs apart.
- Avoid customer managed keys on buckets with very high request rates and no Bucket Keys; the request bill and throttling will surprise you.
- Avoid external key stores or CloudHSM-backed keys unless a regulator requires keys outside AWS-managed hardware; they add latency, cost and a new way to fail.
- Avoid using KMS directly to encrypt large files; it accepts at most 4 KB of plaintext per call and is built for keys, not data.
Production grade
What goes wrong in production
| Problem | What breaks | What to do |
|---|---|---|
| Access denied on read | A role can read the bucket but gets AccessDenied because it cannot decrypt the data key. | Grant kms:Decrypt to the role and allow it in the key policy; check CloudTrail for the failing Decrypt call and its key ARN. |
| KMS throttling | Spark jobs reading millions of small files make one Decrypt call per object and hit the shared request quota. | Turn on S3 Bucket Keys, compact small files, and watch the request rate against the quota in Service Quotas. |
| Cross-account confusion | Another account's role uses your key and the throttling counts against the caller, while the key policy must allow the other account. | Name exact principals in the key policy; avoid granting the whole account root unless IAM policies there are tightly controlled. |
| Replication and keys | S3 replication to Hyderabad fails or skips objects encrypted with SSE-KMS. | Enable replication of SSE-KMS objects, name the destination key, and give the replication role Decrypt on the source key and Encrypt on the destination key. |
| Accidental key deletion | Deleting a key makes everything encrypted under it unreadable forever. | Deletion waits 7 to 30 days; use 30. Alarm on ScheduleKeyDeletion and DisableKey events, and block them with an SCP outside the break-glass role. |
| Databricks workspace key lost | Revoking a Databricks customer-managed key for managed services stops the workspace from working. | Keep these keys in the security account with change control, and test the rotation runbook in dev first. |
| Macie and other scanners cannot read | Security tools fail on objects encrypted with a customer managed key. | Add the scanner's service-linked role to the key policy with Decrypt only, as for Macie on landing buckets. |
Alternatives
What to compare it with
| Alternative | Pick the alternative when | Pick KMS when |
|---|---|---|
| S3 managed keys (SSE-S3) | The data is low sensitivity and you want encryption with no key cost and no extra policy. | You need to revoke access per catalog or prove who decrypted what; SSE-S3 has no key policy or per-key log. |
| AWS managed keys (aws/s3) | You want KMS logging with no setup. | You need to control the key policy or share data across accounts; you cannot change the policy of an AWS managed key. |
| CloudHSM and external key stores | A regulator requires keys in single-tenant hardware or held outside AWS. | Standard KMS hardware meets your rules; custom key stores cost more and have lower request quotas. |
| Azure Key Vault | The same design runs on Azure Databricks, with a key per storage account. | You are on AWS. |
| Google Cloud KMS | You compare clouds; the key-per-dataset pattern is the same. | You only run AWS. |
| Databricks default encryption | Control plane data is encrypted by Databricks with its own keys, enough for many teams. | Policy requires company-controlled keys for notebooks, query history and workspace storage; then add customer-managed keys. |
Cost
How you pay
Each customer managed key costs a flat monthly fee, prorated by the hour, and requests are billed per 10,000 calls. The AWS pricing page gives 1 USD per key per month and 0.03 USD per 10,000 symmetric requests (list price, check your region). Rotation adds the same monthly fee for the first and second rotations only; later rotations are free. AWS managed keys have no monthly fee but their requests are billed. There is a small free tier of requests per month.
What drives the bill
- Number of S3 objects read and written under SSE-KMS without Bucket Keys, one request each.
- Number of customer managed keys and their first two rotations.
- Cross-account use, where the calling account pays for requests.
- CloudTrail logging volume, since every KMS call is a management event.
Cost traps
- SSE-KMS on a bronze bucket full of small files, without Bucket Keys, can produce a large request bill and throttling at the same time.
- A key per table instead of per catalog multiplies monthly fees and policies with no real gain in control.
- Disabled keys still cost the monthly fee; only keys scheduled for deletion stop billing.
Prices change; check the AWS pricing page for your region before deciding.
Security and governance
Keeping it safe
- Key policies name exact principals: the catalog storage role, the replication role, scanners with Decrypt only, and a break-glass role. Key administrators cannot use the key for data.
- Separate who manages keys from who uses them; key administration sits in the security account team, not the data platform team.
- SCPs deny ScheduleKeyDeletion, DisableKey and PutKeyPolicy outside the security role, and CloudWatch alarms fire on any attempt.
- Reach KMS through an interface VPC endpoint from private subnets and use the aws:SourceVpce condition on sensitive keys.
- Automatic yearly rotation is on for every key; the key ID stays the same, so nothing in Terraform or Databricks changes.
- CloudTrail logs every Decrypt with the caller and encryption context; these logs are evidence for who read personal data and recipes.
Operating it
What to watch, and the limits
| Watch | Why |
|---|---|
| CloudTrail Decrypt and GenerateDataKey events | Who used each key, how often, and from which role; spikes on the recipes key deserve a look. |
| ThrottlingException on KMS calls | Requests over the shared symmetric quota; usually small files without Bucket Keys. |
| Service Quotas: cryptographic operations (symmetric) usage | Request rate against the regional quota, with a CloudWatch alarm at 80 percent. |
| ScheduleKeyDeletion and DisableKey events | Any attempt to remove or disable a key; page the security team. |
| SecondsUntilKeyMaterialExpiration | For keys with imported material, how long until the material expires and data becomes unreadable. |
| Limit | Value and notes |
|---|---|
| Customer managed keys | 100,000 per account per region, adjustable |
| Symmetric request rate | 10,000 per second shared across Decrypt, Encrypt and GenerateDataKey in most regions, including Mumbai; higher in some large regions |
| Grants per key | 50,000 |
| Aliases per key | 50 |
| On-demand rotations | 25 per key, not adjustable |
| Deletion waiting period | 7 to 30 days |
Limits and quotas checked 2026-11-11; many can be raised through Service Quotas.
Decisions
Situations and verdicts
| Situation | Choice | Verdict |
|---|---|---|
| The recipes catalog must be revocable in one action | Its own bucket and its own customer managed key, with a narrow key policy | Pursue: disabling the key stops every read everywhere, and CloudTrail shows every past use. |
| Bronze holds millions of small sensor files under SSE-KMS | Turn on S3 Bucket Keys and compact files | Pursue with care: Bucket Keys cut requests sharply, but CloudTrail then shows the bucket, not each object, as the context. |
| Security asks for company-controlled keys for notebooks and query history | Databricks customer-managed keys for managed services and workspace storage | Pursue with care: needs the Enterprise tier, and losing the key stops the workspace. |
| A regulator is said to want keys held outside AWS | External key store | Drop: confirm the rule first; for most manufacturers standard KMS with per-catalog keys and logs is enough. |
FAQ
Questions people ask
Should I use SSE-S3 or SSE-KMS for a data lake?
Use SSE-KMS with a customer managed key for any catalog with sensitive or valuable data, because it gives a key policy and a log of every use. Turn on Bucket Keys to keep request costs low. SSE-S3 is fine for low-risk scratch data.
Does key rotation break my data or Databricks?
No. Rotation creates new key material but keeps the same key ID and keeps the old material for decrypting old data. Nothing in Unity Catalog, Terraform or bucket settings needs to change.
What happens if a KMS key is disabled?
Every read of data encrypted under it fails with an access error until the key is enabled again. Data is not lost. Deleting a key, after the waiting period, makes the data unreadable for good.
Do I need one key per table?
No. One key per catalog, matching one bucket per catalog, gives revocation and audit at the level people actually manage access. More keys add policy work without more control.
Does Databricks need special KMS setup?
The Unity Catalog storage role needs Decrypt, Encrypt and GenerateDataKey on the bucket key, and the key policy must allow it. Customer-managed keys for the workspace itself are a separate Enterprise tier setting with their own key policies.
Sources
Checked against
- AWS: KMS pricing
- AWS docs: KMS resource quotas
- AWS docs: KMS request quotas
- Databricks docs: Customer-managed keys for encryption
- Databricks docs: S3 external location and KMS permissions
Checked 2026-11-11.
Related
Where it connects
AWS and Databricks reference platform · AWS IAM and STS · Amazon S3 · Amazon Macie · S3 Versioning, Replication and Object Lock · All AWS data services
← IAM and STS · Macie →