Home › AWS › AWS data services › IAM and STS

AWS data services · Security, identity and network · about 12 min read

AWS IAM and STS (roles Databricks assumes, instance profiles, Unity Catalog storage credentials)

Who may do what in AWS: roles, policies and short-lived credentials, including the roles Databricks assumes to reach your buckets.

RoleSecurity, identity and network
StatusGenerally available. Databricks has required storage-credential roles to be self-assuming since 20 Jan 2025; non-self-assuming roles are blocked.
Verdict on this sitePursue
AzureMicrosoft Entra ID with managed identities and Azure RBAC; Databricks access connectors
Google CloudCloud IAM with service accounts and workload identity federation
DatabricksUnity Catalog grants for data; storage and service credentials wrap the IAM roles underneath
Open sourceKeycloak for people; HashiCorp Vault for short-lived cloud credentials
Verdict: Pursue. IAM is not optional, so the question is how to use it. On this platform people never get AWS keys: they sign in through the company identity provider, Databricks reaches each bucket through one narrow role per catalog held in a Unity Catalog storage credential, and CI signs in with OIDC. Instance profiles are kept only for the rare classic job that needs them.

In plain words

What it is

AWS Identity and Access Management (IAM) is the part of AWS that decides whether a request is allowed. Every call to AWS, such as reading a file from S3 or writing to a Kinesis stream, carries an identity, and IAM checks that identity against written rules called policies. A policy is a small JSON document that says which actions are allowed or denied on which resources, under which conditions. If no policy allows a call, the answer is no.

The identity that matters most on a data platform is the role. A role is an identity with no password and no permanent keys. Someone or something that is trusted may assume it, and in return gets temporary credentials that expire, usually after an hour. The AWS Security Token Service (STS) is the part that hands out those temporary credentials. Each role has two halves: a trust policy that says who may assume it, and permission policies that say what it can do once assumed.

Take a tyre and rubber manufacturer whose plants send press readings to an S3 landing bucket and whose SAP orders land next to them. The data engineers never hold AWS keys. They sign in to Databricks through the company login, and Unity Catalog decides which tables they may read. When a query touches the silver orders table, Databricks assumes an IAM role that may read only the silver bucket, gets an hour of credentials from STS, and reads the files. If that role is removed, Databricks can no longer reach the bucket, whatever grants exist in the catalog.

This page covers the IAM and STS pieces a data platform needs: the cross-account role that lets Databricks start machines in your account, the per-catalog roles behind Unity Catalog storage credentials, instance profiles for classic clusters, and OIDC sign-in for CI pipelines. It does not cover application sign-in for end users, which is a job for an identity provider such as Microsoft Entra ID or Amazon Cognito.

A hotel key card system: no one gets a master key, each card opens only the rooms written on it, and every card stops working at checkout.

Big picture

Where it fits

SourcesMove inStreamStoreProcessServeUseAcross everything: catalog · security · orchestration · observability and cost · infrastructure as codegets data fromhands data toCompany identity provider (Entra ID, Okta)GitHub Actions OIDC tokensOrganizations and Control TowerIAM and STSSecurity, identity and networkS3KMSSecrets Manager

The lit lane is where IAM and STS works. Boxes on the left feed it; boxes on the right use what it produces.

IAM sits underneath every lane of the reference platform rather than in one of them. Movers such as DMS, AppFlow and Transfer Family each run as a service role that may write only to its own landing folder. Kinesis producers at the plants use roles that may put records on one stream. Databricks uses three kinds of role: the cross-account role that lets the Databricks control plane create clusters in your VPC, the storage roles that Unity Catalog uses to read and write each catalog's bucket, and service roles behind Unity Catalog service credentials for calls to other AWS services such as Secrets Manager.

The design is one bucket, one KMS key and one IAM role per catalog. The prod silver role can read and write the prod silver bucket and use its key, and nothing else. Unity Catalog stores that role as a storage credential and ties it to an external location, the S3 path it covers. People are granted access to tables in Unity Catalog; they never get the role directly. This keeps two layers apart: Unity Catalog decides which person may see which table, and IAM decides which buckets Databricks may touch at all.

People sign in through the company identity provider, and groups sync to Databricks with SCIM, the standard for copying users and groups between systems. Engineers who need the AWS console use IAM Identity Center with short sessions. GitHub Actions signs in with OIDC (OpenID Connect, a standard for one system to prove its identity to another), so no long-lived access key exists anywhere. CloudTrail records every AssumeRole call, and those records are joined with Databricks audit tables for the quarterly access review.

How it works

The ideas you need

IdeaWhat it means
RoleAn identity with no password or permanent keys. Trusted callers assume it and receive temporary credentials from STS.
Trust policyThe part of a role that says who may assume it, for example the Databricks Unity Catalog role together with an external ID.
Permission policyA JSON document listing allowed or denied actions on resources. Identity policies attach to roles; resource policies, such as bucket policies, attach to the resource.
External IDA shared value that must be present when a third party assumes your role. It stops another Databricks customer from tricking Databricks into using your role, the confused deputy problem.
Cross-account roleThe role in your account that the Databricks control plane assumes to create and manage clusters, network interfaces and volumes for classic compute.
Storage credentialA Unity Catalog object that wraps an IAM role so Databricks can reach an S3 path. External locations point a path at a storage credential.
Self-assuming roleA role whose trust policy also names itself. Databricks requires this for storage credential roles.
Instance profileA container for a role that is attached to an EC2 machine, so code on that machine gets credentials automatically. The older way Databricks clusters reached S3.
Service credentialA Unity Catalog object that wraps an IAM role for calls to AWS services other than storage, such as Secrets Manager, governed with catalog grants.
Permissions boundary and SCPCeilings on what a role can ever do. A boundary limits one role; a service control policy limits a whole account from the Organization.

Choosing

When to use it, and when not

Use it when

  • Use one IAM role per catalog bucket, wrapped in a Unity Catalog storage credential, so a mistake in one catalog cannot expose another.
  • Use roles with OIDC federation for CI pipelines and external tools, so no access key is ever created, stored or rotated.
  • Use service roles scoped to one landing prefix for each mover, so a DMS task cannot overwrite what AppFlow wrote.
  • Use Unity Catalog service credentials when notebooks or jobs need to call AWS services such as Secrets Manager, so access is granted in the catalog and audited there.
  • Use IAM Access Analyzer to find roles and permissions that have not been used for months, and remove them before the access review.

Avoid it when

  • Avoid IAM as the way to decide which person sees which table. That belongs in Unity Catalog grants, masks and row filters; IAM cannot see rows or columns.
  • Avoid IAM users with access keys for anything automated. Keys leak into notebooks, Git and laptops, and they never expire on their own.
  • Avoid one broad role for every bucket shared by all clusters through an instance profile. Anyone on that cluster can bypass Unity Catalog and read any path directly.
  • Avoid writing policies by hand in the console. They drift from Git and nobody can review them; build them in Terraform instead.

Production grade

What goes wrong in production

ProblemWhat breaksWhat to do
Storage credential fails validationThe role works in the AWS console but Databricks reports it cannot assume it, often after a new credential is created.Check the trust policy names the Databricks Unity Catalog role, the external ID from the credential, and the role itself (self-assuming). Run the credential's validate check after every change.
Access denied with no obvious causeA job fails reading a bucket even though the role policy allows it.Look for the other layers: bucket policy, KMS key policy, SCP, permissions boundary and the VPC endpoint policy. Any explicit deny wins. CloudTrail errorCode and the IAM policy simulator show which one.
KMS denies even though S3 allowsFiles encrypted with a customer managed key need permission on the key as well as the bucket.Give the role kms:Decrypt, kms:Encrypt and kms:GenerateDataKey on the catalog key, and add the role to the key policy.
Instance profiles bypass the catalogA cluster with an instance profile can read S3 paths directly, outside Unity Catalog grants and audit.Use Unity Catalog storage credentials and standard access mode; keep instance profiles only for named legacy jobs and review them quarterly.
Policy size limitsOne role covering many buckets grows past the managed policy size and cannot be saved.One role per catalog keeps policies small. Use prefixes and tags in conditions rather than listing every path.
STS throttlingMany parallel tasks assuming roles at the same moment hit the STS request quota and fail with throttling errors.Reuse credentials until they near expiry, use regional STS endpoints through the VPC endpoint, and spread start times of big job fans.
Role chaining cuts sessions shortA role assumed from another role gets at most a one-hour session, so long transfers fail midway.Avoid chains for long jobs; let the service assume the final role directly, or refresh credentials inside the job.
Orphaned accessRoles for retired movers and old projects keep their permissions for years.Tag every role with an owner and a data product, use Access Analyzer unused-access findings, and delete what Terraform no longer declares.

Alternatives

What to compare it with

AlternativePick the alternative whenPick IAM and STS when
Unity Catalog grantsYou decide which person or group may read a table, column or row; this is the right layer for people.You need to control which buckets Databricks itself may reach. That stays in IAM, under the catalog.
Instance profilesA legacy job on classic compute must reach S3 or another service and cannot yet use Unity Catalog.You can use storage or service credentials; instance profiles give the whole cluster the same broad access.
Microsoft Entra ID and managed identities (Azure)The same platform also runs on Azure Databricks; access connectors play the storage credential role there.You are on AWS; the concepts map one to one but the setup is different.
Google Cloud IAM and service accountsYou compare clouds; GCP uses service accounts and workload identity federation in the same way.You only run AWS.
IAM Identity CenterPeople need AWS console or CLI access; it gives short sessions mapped from your identity provider groups.Machines need access; use roles with OIDC or service trust instead.

Cost

How you pay

IAM and STS themselves are free: there is no charge for users, roles, policies or AssumeRole calls. The costs come from what sits around them. IAM Access Analyzer external access findings are free, while unused access findings and custom policy checks are billed per role analysed or per check (list price, check your region). CloudTrail data events, if you log S3 object reads to see which role read what, are billed per event. The biggest hidden cost is people's time spent untangling denied requests across many policy layers.

What drives the bill

  • Number of roles and principals analysed by Access Analyzer unused access findings.
  • CloudTrail data event volume if every S3 read by every role is logged.
  • Engineer time debugging denies, which grows with the number of overlapping policy layers.

Cost traps

  • Logging S3 data events on busy bronze buckets can cost more than the storage; log reads on sensitive catalogs only.
  • A broad role saves a day of setup and costs weeks later in an audit, when nobody can say who could read the recipes.

Prices change; check the AWS pricing page for your region before deciding.

Security and governance

Keeping it safe

  • No IAM users with access keys: people use the identity provider and Identity Center, machines use roles, and CI uses OIDC with trust conditions limited to the repository and branch.
  • Every trust policy for a third party carries an external ID and names exact principals; the Databricks storage roles are also self-assuming, as Databricks requires.
  • Least privilege per catalog: one role per bucket and key, with explicit denies in bucket policies for any principal that is not the catalog role or the break-glass role.
  • SCPs from the Organization block disabling CloudTrail, leaving the region and creating IAM users with keys, so a mistake in one account cannot undo the guardrails.
  • CloudTrail records every AssumeRole and every denied call; these are joined with Databricks audit system tables to answer who read what, through which role.
  • Personal data access is decided in Unity Catalog with masks and row filters; IAM only ensures nothing outside Databricks can read the vault bucket at all.

Operating it

What to watch, and the limits

WatchWhy
CloudTrail AssumeRole eventsWho assumed each Databricks and mover role, from where, and how often; alert on unknown source accounts.
CloudTrail AccessDenied errorCodeDenied calls by role and action; a sudden rise after a deploy usually means a broken policy.
Access Analyzer findingsExternal access to buckets or keys, and unused roles or permissions to remove.
Storage credential validationDatabricks check that each role can be assumed and can list, read and write its path.
Service Quotas: roles per accountHow close the account is to the role limit as catalogs and movers grow.
STS throttling errorsThrottlingException on AssumeRole in CloudTrail, a sign of job fans starting at once.
LimitValue and notes
Roles per account1,000 by default, raised automatically up to 10,000
Managed policies per role20 by default, up to 25
Managed policy size6,144 characters, not counting white space; cannot be raised
Role trust policy length2,048 characters by default, up to 8,192
Role session duration1 to 12 hours as set on the role; one hour when role chaining
STS request rate600 requests per second per account per region for calls made with AWS credentials

Limits and quotas checked 2026-11-11; many can be raised through Service Quotas.

Decisions

Situations and verdicts

SituationChoiceVerdict
Databricks needs to read and write the silver catalog bucketOne IAM role for that bucket and its KMS key, as a Unity Catalog storage credentialPursue: access is decided once in the catalog, the role cannot reach other catalogs, and every use is audited on both sides.
A legacy classic job reads a partner bucket with an instance profileKeep the instance profile on that one job cluster for nowKeep if you have it: move it to an external location with a storage credential when the job is next touched.
GitHub Actions deploys Terraform and bundlesOIDC trust from the repository to a deploy role per environmentPursue: no stored keys, and the trust condition limits which branch can deploy to prod.
A team asks for an IAM user with keys for a desktop toolIdentity Center sign-in with a short session insteadDrop: keys never expire on their own and end up in config files.

FAQ

Questions people ask

What is the difference between a storage credential and an instance profile in Databricks?

Both wrap an IAM role. A storage credential is used by Unity Catalog, which checks table and volume grants before using the role, and records the access. An instance profile gives every user on a cluster the role directly, outside the catalog. New work should use storage credentials.

Why does Databricks require a self-assuming role?

AWS changed how roles may assume themselves in 2023, and Databricks uses self-assumption to issue narrower credentials. Since January 2025 Databricks blocks storage credentials whose roles do not name themselves in the trust policy. Add the role's own ARN as a principal and validate again.

What does the Databricks cross-account role do?

It lets the Databricks control plane create and manage classic compute in your VPC: instances, network interfaces, volumes and security groups. It does not read your data. Serverless compute runs in the Databricks serverless plane and does not use it.

Can IAM replace Unity Catalog permissions?

No. IAM works at the level of buckets, prefixes and API calls; it knows nothing about tables, columns or rows. Use IAM to fence what Databricks may touch and Unity Catalog to decide what each person sees.

How should CI sign in to AWS without keys?

Register GitHub as an OIDC identity provider in IAM and create a deploy role that trusts it, with conditions on the repository and branch. Each run gets credentials that expire in about an hour.

Sources

Checked against

Checked 2026-11-11.

Related

Where it connects

AWS and Databricks reference platform · AWS KMS · AWS Secrets Manager · AWS Organizations, Control Tower and SCPs · AWS on this site · All AWS data services

← Direct Connect, VPN, Transit Gateway · KMS →