Home › AWS › AWS data services › CDK and CloudFormation

AWS data services · Infrastructure as code and CI/CD · about 11 min read

AWS CloudFormation and AWS Cloud Development Kit (CDK)

AWS's own way to describe infrastructure as code: CloudFormation templates deployed as stacks, and CDK to write them in Python or TypeScript.

RoleInfrastructure as code and CI/CD
StatusGenerally available. Express mode (faster deploys that skip waiting for resources to settle) launched 30 Jun 2026; stack refactoring launched Feb 2025.
Verdict on this siteKeep if you have it
AzureAzure Resource Manager templates and Bicep
Google CloudInfrastructure Manager (managed Terraform)
DatabricksDatabricks Terraform provider for infrastructure, Declarative Automation Bundles (formerly Asset Bundles) for jobs and pipelines
Open sourceTerraform or OpenTofu, Pulumi
Verdict: Keep if you have it. Our foundation is Terraform because one tool must build both the AWS side and the Databricks side, and Databricks itself recommends Terraform for its resources. If your company already runs its landing zone and network in CloudFormation or CDK, keep them; let Terraform own the data platform layer on top.

In plain words

What it is

Infrastructure as code means you describe your cloud resources, such as buckets, networks, roles and encryption keys, in text files kept in Git, and a tool creates or changes the real resources to match. Nobody clicks around the console, every change is reviewed in a pull request, and you can build the same platform again in another account or region.

AWS CloudFormation is the AWS service that does this. You write a template, a YAML or JSON file that lists resources and their settings, and hand it to CloudFormation, which creates them as one unit called a stack. When you change the template, CloudFormation works out the difference and updates the stack; if an update fails, it rolls the stack back to the last good state. The state of what exists is kept by AWS, not in a file you must look after.

The AWS Cloud Development Kit, CDK, lets you write the same thing in a programming language such as Python or TypeScript. You build classes called constructs, for example a 'landing bucket' construct that always comes with encryption, versioning and a lifecycle rule, and CDK turns your code into a CloudFormation template. CDK is a code generator in front of CloudFormation, not a separate engine.

Take a retailer that wants a landing bucket for order files from each region. In CloudFormation they write one template with the bucket, its KMS key and its bucket policy, and deploy it once per region with different parameters. In CDK they write a small Python class and loop over the regions. In both cases the result is a set of stacks that CloudFormation tracks.

CloudFormation is a building contractor that builds exactly what is on the plan and keeps the site log; CDK is the drafting software that draws that plan for you.

Big picture

Where it fits

SourcesMove inStreamStoreProcessServeUseAcross everything: catalog · security · orchestration · observability and cost · infrastructure as codegets data fromhands data toGit repository with templates or CDK codeCodePipeline and CodeBuildGitHub Actions with OIDCCDK and CloudFormationInfrastructure as code and CI/CDVPC, endpoints and NATIAM and STSKMS

The lit lane is where CDK and CloudFormation works. Boxes on the left feed it; boxes on the right use what it produces.

CloudFormation and CDK sit in the build and deploy lane, outside the data path. They create the foundation that every data hop runs on: accounts and guardrails, the VPC with its endpoints, the S3 buckets and KMS keys per catalog, and the IAM roles Databricks may assume. No order, sensor reading or SAP record passes through them.

The reference platform builds that foundation with Terraform, in layered folders applied in order: organisation, network, storage and keys, workspaces, catalogs. Terraform is used because one tool, with the AWS and Databricks providers side by side, can create a bucket and its IAM role and then register that role in Unity Catalog as a storage credential in the same plan. CloudFormation cannot create Databricks objects natively, and Databricks' own deployment guide says to use Terraform for Databricks resources and that CloudFormation is no longer the preferred way to launch a workspace.

CloudFormation still shows up in most AWS estates. Control Tower uses it under the hood, AWS Quick Starts and many vendor integrations ship as CloudFormation templates, and plenty of central cloud teams have years of CDK code for networks and accounts. The realistic pattern is a split: the landing zone and shared network may stay in CloudFormation or CDK, owned by the cloud team, and the data platform layer is Terraform, reading the outputs it needs such as subnet and VPC IDs.

How it works

The ideas you need

IdeaWhat it means
TemplateA YAML or JSON file listing resources, parameters and outputs that CloudFormation should create.
StackOne deployed copy of a template. CloudFormation creates, updates and deletes all its resources together.
Change setA preview of what an update will add, change or replace, reviewed before it runs, like a Terraform plan.
StackSetA way to deploy one template into many accounts and regions at once, used for guardrails and baseline roles across an organisation.
ConstructA CDK building block written in code. Level 1 constructs mirror raw resources; higher levels add safe defaults.
SynthesisThe CDK step that runs your code and produces the CloudFormation template that is actually deployed.
DriftA difference between what the template says and what really exists, usually from someone changing a resource by hand.
Express modeA deployment option, launched June 2026, that reports success once settings are applied instead of waiting for every resource to become ready.
Stack refactoringA feature that moves resources between stacks or renames them without deleting and recreating them.

Choosing

When to use it, and when not

Use it when

  • Your company's landing zone, accounts and shared network are already built with CloudFormation, CDK or Control Tower customisations and owned by a central cloud team.
  • You want AWS to keep the deployment state for you, with automatic rollback, and no state file or lock table to protect.
  • You need to push the same baseline, such as an audit role or a config rule, into dozens of accounts with StackSets.
  • Your engineers prefer real programming languages, tests and reusable classes, and CDK constructs fit how they already work.
  • A vendor or AWS Quick Start ships only as a CloudFormation template and you would rather deploy it as published.

Avoid it when

  • You want one tool to build both AWS resources and Databricks workspaces, storage credentials, catalogs and grants; CloudFormation has no first-class Databricks resources, so you would end up with custom resources calling APIs.
  • Your platform spans more than one cloud, or may move to Azure or Google Cloud later; a template written for CloudFormation does not travel.
  • You are starting fresh with a small team that already knows Terraform; running two infrastructure tools doubles the review rules, pipelines and knowledge needed.
  • You plan to use Express mode for production releases; it reports success before resources are ready and turns rollback off by default.

Production grade

What goes wrong in production

ProblemWhat breaksWhat to do
Stack stuck in a failed stateAn update fails and the rollback also fails, leaving the stack in UPDATE_ROLLBACK_FAILED; no further changes can be applied until someone intervenes.Keep stacks small and per layer; fix the blocking resource and continue the rollback; read the stack events before retrying; never edit resources by hand to unblock.
Accidental replacementChanging certain properties, such as a bucket name or a key policy shape, replaces the resource; for a bucket or KMS key that can mean lost data or unreadable files.Review every change set for Replacement equal to True; set DeletionPolicy and UpdateReplacePolicy to Retain on buckets, keys and tables; add stack policies that block replacement.
Drift from console changesSomeone widens a bucket policy or security group by hand during an incident and the template no longer tells the truth.Run drift detection on a schedule; deny console write access in prod with SCPs; put the fix back into code and redeploy.
Databricks objects outside the stackWorkspaces, storage credentials, external locations and catalogs are not native CloudFormation resources, so teams bolt on Lambda-backed custom resources that are hard to update and debug.Let Terraform with the Databricks provider own every Databricks object; pass AWS outputs such as role ARNs to it through SSM parameters or stack outputs.
CDK hides the templateEngineers review Python, not the generated template, so a library upgrade or a changed default can alter real resources without anyone noticing.Run cdk diff in every pull request and post the output; pin CDK library versions; add snapshot tests of synthesised templates.
Two tools, one platformNetwork in CDK and data platform in Terraform means two pipelines, two sets of reviewers and confusion about who owns a resource.Draw a hard line by layer, publish the handover values (VPC, subnet, endpoint IDs) as SSM parameters, and never let both tools manage the same resource.
Slow deploysSome resources take many minutes to stabilise, so a full deploy of network and endpoints can take a long time.Split stacks so only changed layers deploy; use Express mode in dev for fast feedback, standard mode in pre-prod and prod.
Quota surprisesLarge estates hit limits on stacks per account or resources per template, and nested stacks become hard to reason about.Split by layer and domain; use stack refactoring to move resources without recreating them; request quota increases early.

Alternatives

What to compare it with

AlternativePick the alternative whenPick CDK and CloudFormation when
Terraform or OpenTofuYou need AWS and Databricks in one plan, work across clouds, or the team already knows HCL. This is the site's choice.You do not want to look after a state file, its lock and its encryption, or you rely on StackSets across hundreds of accounts.
PulumiYour engineers want real programming languages and you also need Databricks resources; Pulumi has a Databricks provider.You want a tool your auditors and platform teams already know, with the widest set of examples for Databricks on AWS.
Databricks Declarative Automation BundlesYou are deploying jobs, pipelines, dashboards and other workspace content; renamed from Asset Bundles in March 2026.You need to build AWS resources or workspaces themselves; bundles deploy content into a workspace that already exists.
Azure Resource Manager and BicepYou are building the same platform on Azure; Bicep is the Azure counterpart of CloudFormation.You are on AWS; nothing carries over except the ideas.
Google Infrastructure ManagerYou are on Google Cloud and want Terraform run as a managed service.You are on AWS.
AWS Control Tower Account Factory for TerraformYou want new accounts vended by Control Tower but defined in Terraform.You only have a few accounts and a simple Organizations setup is enough.

Cost

How you pay

CloudFormation itself has no charge for resources from AWS's own namespaces; you pay for the resources it creates. Handler operations for third-party and private registry resource types are billed per operation and per second of run time beyond a free monthly allowance (list price, check your region). CDK is a free open-source library. The real cost is engineering time: writing, reviewing and debugging templates.

What drives the bill

  • The resources the templates create, such as NAT gateways, endpoints and KMS keys, which dominate any bill.
  • Use of third-party or private resource types and Hooks, which are billed per handler operation.
  • Engineer time spent on failed rollbacks, drift clean-up and keeping two infrastructure tools in step.
  • CI build minutes for synthesis, diff and deploy runs.

Cost traps

  • A template that forgets to retain a bucket on delete can destroy data when a stack is torn down, which is the most expensive mistake of all.
  • Dev stacks left running create NAT gateways and interface endpoints that charge by the hour whether used or not.
  • Custom resources backed by Lambda to reach Databricks add their own run cost and a lot of maintenance time.
  • Express mode in production can report success while resources are still not usable, so a dependent job fails and reruns cost compute.

Prices change; check the AWS pricing page for your region before deciding.

Security and governance

Keeping it safe

  • Deploy with a dedicated service role passed to CloudFormation, so the person or pipeline only needs permission to call CloudFormation and pass that role, not broad rights on every resource.
  • Let CI sign in with short-lived OIDC tokens from GitHub Actions to an IAM role scoped to one account and one repository, never with stored access keys.
  • Use CloudFormation Guard rules or Hooks to block templates that create public buckets, unencrypted volumes or wildcard IAM policies before they deploy.
  • Set termination protection on prod stacks and stack policies that forbid replacing buckets and KMS keys.
  • Never put secrets in templates or parameters in plain text; reference Secrets Manager or SSM SecureString values, which CloudFormation resolves at deploy time.
  • CloudTrail records every stack operation and every API call CloudFormation makes for you, so changes are traceable to the role and pipeline run.

Operating it

What to watch, and the limits

WatchWhy
Stack statusAny stack in a *_FAILED or UPDATE_ROLLBACK_FAILED state needs a person; alert on it through EventBridge stack status events.
Drift statusResults of scheduled drift detection; a DRIFTED stack means the code no longer describes reality.
Change set replacementsCount of resources marked for replacement in each pull request's change set or cdk diff output.
Stack eventsThe ordered list of create, update and failure reasons, the first place to look when a deploy fails.
CloudTrail events by CloudFormationAPI calls made with the CloudFormation service role, to show who changed what and when.
Guard and Hook findingsTemplates rejected by policy checks, a sign of risky patterns being attempted.
LimitValue and notes
Resources per template500 resources per stack template
Template body size51,200 bytes passed directly; 1 MB when uploaded to S3
Stacks per account2,000 stacks per region by default, adjustable
Parameters and outputs200 parameters and 200 outputs per template

Limits and quotas checked 2026-11-11; many can be raised through Service Quotas.

Decisions

Situations and verdicts

SituationChoiceVerdict
Greenfield data platform on AWS with DatabricksTerraform with AWS and Databricks providersPursue: Terraform. One plan covers buckets, keys, roles and the Unity Catalog storage credentials that point at them.
Central cloud team already runs network and accounts in CDKKeep CDK for those layers, Terraform for the data platformKeep if you have it: agree a hard boundary and share VPC, subnet and endpoint IDs through SSM parameters.
Need Databricks workspaces and catalogs in CloudFormationCustom resources calling the Databricks APIDrop: brittle and unsupported as a main path; Databricks recommends Terraform for its resources.
Fast feedback for platform engineers in devCloudFormation Express modePursue with care: fine in dev; keep standard mode with rollback for pre-prod and prod.
Baseline audit role in every accountStackSets from the management account or Control TowerPursue: StackSets are the simplest way to put the same small baseline everywhere.

FAQ

Questions people ask

Should I use CloudFormation or Terraform for Databricks on AWS?

For the Databricks side, Terraform. Databricks' deployment guide says to use Terraform for Databricks resources and that CloudFormation is no longer the preferred way to launch a workspace. CloudFormation or CDK can still build the AWS side if that is what your company uses.

Is CDK a different engine from CloudFormation?

No. CDK runs your code and produces a CloudFormation template, which CloudFormation then deploys as a stack. Every CDK deploy is a CloudFormation deploy underneath.

What is CloudFormation Express mode?

A deployment option launched in June 2026 that marks a stack operation complete once settings are applied, without waiting for resources to stabilise. It speeds up development loops, and in the CDK you use cdk deploy with the express flag. Rollback is off by default, so keep it out of production releases.

Can I bring existing resources under CloudFormation?

Yes. Resource import adopts existing resources into a stack, and the IaC generator scans an account and writes a template from what it finds. Expect to clean up the generated template, since it uses literal values instead of parameters.

Do I need to manage state like in Terraform?

No. CloudFormation keeps the state of each stack inside AWS. That is one less thing to protect, but it also means the state is tied to one account and region.

Sources

Checked against

Checked 2026-11-11.

Related

Where it connects

Data platform on AWS and Databricks · AWS CodePipeline and CodeBuild · AWS Organizations and Control Tower · AWS IAM and STS · Shift-left checks · AWS platform overview · All AWS data services

← Cost Explorer, CUR, Budgets · CodePipeline and CodeBuild →