Home › AWS › AWS data services › Secrets Manager

AWS data services · Security, identity and network · about 11 min read

AWS Secrets Manager

One encrypted home for passwords, API keys and tokens, with automatic rotation, audit and copies in other regions.

RoleSecurity, identity and network
StatusGenerally available; managed external secrets (AWS rotates some SaaS credentials for you) added since Nov 2025
Verdict on this sitePursue
AzureAzure Key Vault (secrets)
Google CloudSecret Manager
DatabricksWorkspace secret scopes, and Unity Catalog secrets, which can be backed by Secrets Manager (Beta in Nov 2026)
Open sourceHashiCorp Vault or its open fork OpenBao
Verdict: Pursue. The one home for every database password, SaaS token and key reference on the platform. Databricks reads it through a Unity Catalog service credential, so no secret is ever copied into a notebook, a job or Git.

In plain words

What it is

A secret is any small piece of text that grants access: a database password, an API key for a SaaS system, an SFTP private key, the salt used to fingerprint customer IDs. Every data platform has dozens of them, and the classic failure is that they end up pasted into notebooks, job settings, config files or Git, where anyone with read access can copy them and nobody can say who used them.

AWS Secrets Manager is a managed store for those secrets. Each secret is stored encrypted with an AWS KMS key, has a name such as prod/sap/odata-user, and can only be read by an IAM identity that a policy allows. Every read is an API call, GetSecretValue, which is logged in CloudTrail. Secrets Manager can also rotate a secret on a schedule: it generates a new password, sets it in the database, tests it, and only then makes it the current version, so nobody ever has to know or type the password.

A concrete example: a manufacturer copies change data from its plant MES database, a SQL Server instance, into S3 with AWS DMS. The DMS endpoint does not hold the password itself; it points to a secret. A rotation function changes that password every 90 days. The night it rotates, DMS picks up the new version on its next connection, the old password stops working, and no person has seen either one.

A key cabinet in the security office that hands out keys only to people on its list, writes every handout in a ledger, and quietly changes the locks every quarter.

Big picture

Where it fits

SourcesMove inStreamStoreProcessServeUseAcross everything: catalog · security · orchestration · observability and cost · infrastructure as codegets data fromhands data toKMSIAM and STSRotation functions on LambdaSecrets ManagerSecurity, identity and networkAWS DMSAppFlowDatabricks jobs through a Unity Catalog service credential

The lit lane is where Secrets Manager works. Boxes on the left feed it; boxes on the right use what it produces.

On the reference platform, Secrets Manager is the single home for credentials used by the movers and by Databricks. DMS endpoints, AppFlow connections and Transfer Family connectors point to secrets for SAP, MES, quality and dealer systems. The metadata store records only the secret's name per source, never the value, so the configuration in Git is safe to read.

Databricks reaches the same secrets through a Unity Catalog service credential. A service credential is a Unity Catalog object that wraps an IAM role Databricks may assume; an admin grants a group or service principal the right to use it, and code in a job asks Unity Catalog for short-lived AWS credentials and then calls Secrets Manager. Access is governed and audited in Unity Catalog like a table, instead of being tied to a cluster's instance profile.

Secrets are replicated to the recovery region, Hyderabad, so that after a disaster the movers and the lander find their credentials in place. Database passwords rotate every 90 days without people. The shift-left checks block any pull request that contains a secret, and CI signs in with OIDC, so there are no stored keys in GitHub either.

How it works

The ideas you need

IdeaWhat it means
SecretA named, encrypted value, usually a small JSON document holding a username, password, host and port, up to 64 KB.
Version and staging labelEach change creates a version. Labels mark which version is AWSCURRENT, AWSPREVIOUS or AWSPENDING during a rotation, so readers always get a working one.
RotationReplacing the secret value on a schedule. Managed rotation does it for supported AWS databases; for anything else a Lambda function you own does the four steps: create, set, test, finish.
Managed external secretsA feature, launched in November 2025, where AWS rotates credentials for supported SaaS partners, such as Snowflake or Salesforce, using partner-provided logic with no Lambda in your account.
Resource policyA policy attached to the secret itself, saying which roles or accounts may read it. Used for cross-account access.
Replica secretA read-only copy of a secret in another region, kept in sync, which can be promoted to a standalone secret during a disaster.
Unity Catalog service credentialA Databricks object that wraps an IAM role for calling AWS services other than storage, such as Secrets Manager or Glue. Granted to users, groups or service principals, not to clusters.
Databricks secret scope and Unity Catalog secretA workspace secret scope is Databricks' own per-workspace store, read with dbutils.secrets.get. A newer Unity Catalog secret is a governed object named catalog.schema.secret; it can hold the value itself or, in Beta, point to a Secrets Manager secret so only the name lives in Databricks.

Choosing

When to use it, and when not

Use it when

  • Use it for every password and API key used by movers, pipelines and applications, so there is one place to rotate, audit and revoke.
  • Use it when database passwords must rotate on a schedule without anyone logging in to change them.
  • Use it when the same credential is needed by AWS services such as DMS or Glue and by Databricks; both can read the same secret.
  • Use it when an auditor asks who read the SAP password last month; CloudTrail answers from the GetSecretValue events.
  • Use replication when a recovery region must start with working credentials and no manual copy.

Avoid it when

  • Avoid it for plain configuration such as hostnames, schedules or feature flags; those belong in the metadata store or in AWS Systems Manager Parameter Store, which is cheaper.
  • Avoid calling GetSecretValue for every record or every task in a Spark job; read once per job or cache, or you pay for millions of calls and risk throttling.
  • Avoid using it as a store for encryption keys themselves; keys belong in KMS, which never lets the key material out.
  • Avoid keeping a second copy of the same secret in a Databricks secret scope; two homes means one of them is always stale after rotation.

Production grade

What goes wrong in production

ProblemWhat breaksWhat to do
Rotation breaks a running jobThe password rotates while a long DMS task or Spark job holds the old one, and its next reconnect fails.Use the alternating-users rotation strategy for databases, so the previous user still works until the next rotation; make clients re-read the secret on authentication errors.
Rotation function cannot reach the databaseThe Lambda that rotates runs in the wrong subnet, or cannot reach Secrets Manager, so rotation fails and the secret is stuck with an AWSPENDING version.Put the function in a subnet that reaches the database, give it a Secrets Manager VPC endpoint, and alarm on failed rotation events.
Secrets copied into notebooksEngineers print a secret once to debug and it ends up in notebook output, logs or Git history.Read secrets only in job code, never print them; Databricks redacts secret-scope values in output but not values you fetch yourself, so wrap reads in a small helper that never returns the value to the notebook.
Throttled readsA job fans out to thousands of tasks that each call GetSecretValue and hits the regional rate limit.Fetch once on the driver and pass connection details, or use the AWS caching client; GetSecretValue allows 10,000 calls a second per region.
Cross-account read failsA Databricks role in the workspace account cannot read a secret in the shared-services account.Allow the role in the secret's resource policy, encrypt the secret with a customer-managed KMS key (not the AWS managed key), and allow the role kms:Decrypt on that key.
Recovery region has old valuesSecrets were copied to the recovery region by hand once and have drifted since.Use built-in replication; test a promote-to-standalone in the yearly disaster test.
Secret sprawlHundreds of secrets, unclear owners, some unused for a year, all billed monthly.Tag owner and source on every secret through Terraform; review LastAccessedDate quarterly and delete what is unused.

Alternatives

What to compare it with

AlternativePick the alternative whenPick Secrets Manager when
Unity Catalog secrets backed by Secrets Manager (Beta)You want dbutils.secrets and per-secret Unity Catalog grants while the value stays in Secrets Manager, and you accept a Beta feature.You need it in production today on older runtimes, Spark configuration or environment variables, which it does not support.
Databricks secret scopesOnly Databricks code needs the value and you want the simplest dbutils.secrets.get call with output redaction.AWS services such as DMS also need the secret, or you want rotation and CloudTrail audit in one place.
AWS Systems Manager Parameter StoreThe value is plain configuration or a low-value secret, and you want no per-item fee.You need built-in rotation, cross-region replication or larger values.
HashiCorp Vault or OpenBaoYou run several clouds and on-premises systems and want dynamic, short-lived database credentials from one tool.You do not want to run and secure a highly available Vault cluster yourself.
Azure Key VaultThe platform runs on Azure; Databricks on Azure can back a secret scope directly with Key Vault.The platform runs on AWS.
Google Secret ManagerThe platform runs on Google Cloud.The platform runs on AWS.

Cost

How you pay

Secrets Manager charges a monthly fee per secret, prorated by the hour, and a fee per 10,000 API calls. As a guide, the list price is about 0.40 US dollars per secret per month and 0.05 dollars per 10,000 calls (list price, check your region). Each replica in another region is billed as its own secret. Rotation itself has no extra fee, but rotation functions run on Lambda and the KMS calls behind each read are billed by KMS.

What drives the bill

  • The number of secrets, multiplied by the number of regions they are replicated to.
  • How often code calls GetSecretValue, which is driven by caching or the lack of it.
  • Lambda runs and VPC endpoint hours for rotation and private access.
  • KMS requests when secrets use a customer-managed key.

Cost traps

  • Reading a secret inside a per-row or per-task function multiplies API calls by millions.
  • One secret per environment per source per region grows quietly into hundreds of billed secrets.
  • An interface VPC endpoint for Secrets Manager costs per hour per availability zone in every account; share endpoints where your network design allows.
  • Secrets scheduled for deletion keep their state for 7 to 30 days; restoring a wrongly deleted one is possible, but forgotten restores keep billing.

Prices change; check the AWS pricing page for your region before deciding.

Security and governance

Keeping it safe

  • Grant read access per secret, by name prefix or tag, to the exact role that needs it; never secretsmanager:GetSecretValue on all resources.
  • Encrypt secrets with a customer-managed KMS key per environment, so that revoking the key revokes the whole set and every decrypt is logged.
  • Reach Secrets Manager through an interface VPC endpoint so calls never leave the AWS network, and restrict the secret's resource policy to that endpoint with aws:SourceVpce.
  • Databricks reads secrets through a Unity Catalog service credential granted to service principals, so access is listed and audited in Unity Catalog, and jobs never run as a person.
  • CloudTrail records every GetSecretValue with the caller's role; join it with Databricks audit tables for the quarterly access review.
  • Turn on Security Hub checks that flag secrets without rotation or unused for 90 days.

Operating it

What to watch, and the limits

WatchWhy
RotationFailed and RotationSucceeded eventsSent to EventBridge; alarm on any failure, because a stuck rotation becomes an outage at the next one.
LastRotatedDate and NextRotationDateShown by DescribeSecret; check that every database secret rotated within its 90-day window.
LastAccessedDateThe last day a secret was read; secrets unused for 90 days are candidates for deletion.
CloudTrail GetSecretValue events by roleWho read which secret, and spikes that show a job reading in a loop.
ThrottlingException in client logsToo many reads; add caching before the limit hurts.
Replication statusShows whether each replica in the recovery region is in sync or failed.
LimitValue and notes
Secret value size65,536 bytes per secret
Secrets per region500,000 per account per region
GetSecretValue rate10,000 requests per second per region, not adjustable
Versions per secret100; avoid updating a secret more than once every 10 minutes
Resource policy size20,480 characters per secret

Limits and quotas checked 2026-11-11; many can be raised through Service Quotas.

Decisions

Situations and verdicts

SituationChoiceVerdict
SAP, MES and dealer system passwords used by DMS and AppFlowSecrets Manager with 90-day rotationPursue: one home, rotation without people, every read in CloudTrail.
Databricks jobs need the same credentialsUnity Catalog service credential that reads Secrets ManagerPursue: governed access by service principal, no second copy in a secret scope.
Recovery region must start with working credentialsReplicate secrets to HyderabadPursue: replicas stay in sync; promote them during the disaster test.
Notebook-only token for a team's own experimentDatabricks secret scopeKeep if you have it: fine for small, Databricks-only use; move to Secrets Manager once a pipeline depends on it.
Hostnames, schedules and flagsMetadata store in Git, not Secrets ManagerDrop: configuration is not secret; paying per item and hiding it from review is the wrong trade.

FAQ

Questions people ask

How does Databricks on AWS read AWS Secrets Manager?

Create a Unity Catalog service credential that wraps an IAM role allowed to call GetSecretValue on the secrets, grant the credential to the job's service principal, and in code ask Unity Catalog for a credentials provider to pass to boto3. Older setups use an instance profile on the cluster, which ties access to compute instead of to identity.

Secrets Manager or Databricks secret scopes?

On AWS, use Secrets Manager as the one source of truth when any AWS service or more than one tool needs the value. Workspace secret scopes hold their own copy, which goes stale after rotation. Unity Catalog secrets backed by Secrets Manager (Beta in Nov 2026) store only the name in Databricks and need Runtime 17.3 LTS or serverless environment 4; once they are generally available they are the simplest way to read the same secret.

How much does AWS Secrets Manager cost?

A monthly fee per secret and a small fee per 10,000 API calls, with replicas billed as secrets. For a data platform the bill is usually small; it grows only with secret sprawl or code that reads secrets in a loop.

Secrets Manager or Parameter Store?

Use Secrets Manager for real secrets that need rotation, replication and resource policies. Use Parameter Store for plain configuration and cheap values.

Does rotation break DMS or JDBC connections?

Open connections keep working; the next reconnect needs the new value. Use the alternating-users strategy and make clients re-read the secret on an authentication error.

Sources

Checked against

Checked 2026-11-11.

Related

Where it connects

AWS and Databricks reference platform · Metadata store · AWS KMS · AWS IAM and STS · AWS DMS · All AWS data services

← PrivateLink · VPC, endpoints and NAT →