Home › AWS › AWS data services › CodePipeline and CodeBuild

AWS data services · Infrastructure as code and CI/CD · about 11 min read

AWS CodePipeline and AWS CodeBuild

AWS's managed release pipeline (CodePipeline) and build runner (CodeBuild); an alternative to GitHub Actions for testing and deploying platform code.

RoleInfrastructure as code and CI/CD
StatusGenerally available. CodePipeline V2 pipelines are billed per action minute. CodeCommit, AWS's Git host, reopened to new customers on 24 Nov 2025 after being closed since July 2024.
Verdict on this siteKeep if you have it
AzureAzure Pipelines (Azure DevOps)
Google CloudCloud Build and Cloud Deploy
Databricksnone: Databricks is deployed to by CI using the CLI and Declarative Automation Bundles
Open sourceGitHub Actions, GitLab CI, Jenkins
Verdict: Keep if you have it. The reference platform uses GitHub Actions, signing in to AWS and Databricks with short-lived OIDC tokens, because the code already lives in GitHub. If your company standardises on CodePipeline and CodeBuild, they do the same job; CodeBuild is also a good home for GitHub Actions runners that must run inside your VPC.

In plain words

What it is

Continuous integration and delivery, CI/CD, is the habit of testing every change automatically and releasing it the same way every time. When an engineer opens a pull request, a machine runs the linting, unit tests and checks; when the change is approved and merged, the same machine deploys it to dev, then pre-prod, then prod, with an approval step before prod.

AWS CodeBuild is the machine that runs those steps. You give it a buildspec, a YAML file listing commands such as install the Python packages, run pytest, run terraform plan, and it starts a fresh container, runs them, and saves the logs and outputs. You pay only while it runs. AWS CodePipeline is the conductor. It watches a source such as a GitHub repository through an AWS CodeConnections link (the service formerly called CodeStar Connections), then runs stages in order: build, test, approve, deploy, passing files from one stage to the next.

Picture a manufacturer's data team that changes a silver-layer pipeline for press sensor readings. They open a pull request. CodeBuild runs the unit tests and the Databricks bundle validation and posts the result. After merge, CodePipeline runs a CodeBuild step that deploys the bundle to dev, runs an integration test on sample readings, waits for a lead to click approve, then deploys to prod. Every step's log is kept in CloudWatch.

CodeBuild is a clean workbench you rent by the minute, and CodePipeline is the clipboard that says which bench job comes next and who must sign it off.

Big picture

Where it fits

SourcesMove inStreamStoreProcessServeUseAcross everything: catalog · security · orchestration · observability and cost · infrastructure as codegets data fromhands data toGitHub repositoryAWS CodeCommitCDK and CloudFormationCodePipeline and CodeBuildInfrastructure as code and CI/CDTerraform apply for the foundationDatabricks bundle deployCloudWatch

The lit lane is where CodePipeline and CodeBuild works. Boxes on the left feed it; boxes on the right use what it produces.

These services sit in the build and release lane, beside the platform rather than in the data path. They deploy two kinds of code. The foundation (accounts, network, buckets, keys, IAM roles, workspaces, catalogs) is Terraform with the AWS and Databricks providers. The data products (jobs, pipelines, dashboards) are Declarative Automation Bundles, the Databricks feature renamed from Asset Bundles in March 2026.

The reference platform runs CI in GitHub Actions. A pull request runs lint, shift-left checks, unit tests, contract checks and a Terraform plan without touching Databricks; after approval, the apply and the bundle deploy run. GitHub Actions signs in to AWS through an IAM OIDC identity provider and to Databricks through workload identity federation, so no long-lived key is stored anywhere. That choice follows the code: the repositories are in GitHub, and pull request checks feel native there.

CodePipeline and CodeBuild can do every one of those steps. They make most sense where a company already runs its releases on them, where security insists that build machines run inside AWS accounts it controls, or where a build must reach something private, such as a Terraform state bucket only reachable through a VPC endpoint. A useful middle path is CodeBuild-hosted GitHub Actions runners: workflows stay in GitHub, but the jobs run on CodeBuild machines inside your VPC.

How it works

The ideas you need

IdeaWhat it means
BuildspecThe YAML file that tells CodeBuild which commands to run in each phase and which files to keep.
Build projectA CodeBuild definition: source, machine image and size, IAM service role, VPC settings and buildspec.
PipelineA CodePipeline definition made of stages, each with one or more actions such as build, approve or deploy.
ArtifactFiles passed between pipeline stages, stored in an S3 bucket that CodePipeline uses.
V2 pipeline typeThe current pipeline type, with triggers on branches, tags and file paths, variables, and per action minute billing.
CodeConnectionsThe service, formerly CodeStar Connections, that links CodePipeline and CodeBuild to GitHub, GitLab or Bitbucket.
Reserved capacity fleetA pool of CodeBuild machines kept warm for you, billed by the minute whether busy or not, to avoid queueing.
OIDC federationSigning in with a short-lived token issued by the CI system and trusted by AWS or Databricks, instead of a stored secret key.

Choosing

When to use it, and when not

Use it when

  • Your company already releases its applications with CodePipeline and you want the data platform to follow the same approval and audit process.
  • Security requires that build machines run inside your own AWS accounts and VPC, with no third-party hosted runners touching production.
  • A build step must reach private resources, such as a Terraform state bucket or an internal package mirror, only reachable through VPC endpoints.
  • You want GitHub Actions workflows but run the jobs on CodeBuild-hosted runners, so they get IAM roles and private network access.
  • You need cross-account deploys from a tooling account into dev, pre-prod and prod accounts using IAM roles in each.

Avoid it when

  • Your code is in GitHub and the team already uses GitHub Actions; adding CodePipeline creates a second place to look for results and approvals.
  • You want pull request checks with rich feedback on the pull request page; GitHub Actions gives that with less wiring.
  • Your platform must stay portable across clouds; a pipeline built on CodePipeline does not move to Azure or Google Cloud.
  • You only need a few runs a day of simple tests; a hosted CI service with a free allowance is cheaper to set up and run.

Production grade

What goes wrong in production

ProblemWhat breaksWhat to do
Long-lived keys in CITeams paste an AWS access key or a Databricks personal access token into CI settings; it leaks or outlives its owner.Use IAM roles for CodeBuild, OIDC from GitHub Actions to AWS, and Databricks workload identity federation for a service principal; turn off personal access tokens.
Over-powerful build roleOne CodeBuild service role with administrator rights deploys every environment, so a bad pull request can change prod.Separate roles per environment in separate accounts; pull request builds get read-only plan rights; only the approved pipeline stage may assume the prod deploy role.
Plan and apply driftThe Terraform plan reviewed in the pull request is not what is applied later because main moved in between.Save the plan file as an artifact and apply exactly that file; fail the apply if the plan is stale.
Queueing and slow startsOn-demand builds wait for capacity or spend minutes downloading images and packages on every run.Use local or S3 caching in the buildspec, a slim custom image in ECR, or a reserved capacity fleet for busy hours.
Private network reachBuilds in a VPC lose internet access and fail to download packages, or reach the internet through a costly NAT gateway.Add VPC endpoints for S3, ECR, STS and CloudWatch Logs; use CodeArtifact or an S3 mirror for packages; keep NAT only where needed.
Secrets in logsCommands echo tokens or connection strings into build logs, which many people can read.Read secrets from Secrets Manager in the buildspec's secrets-manager section, which masks them; restrict log group access.
Pull requests from forksA webhook builds untrusted code with the project's IAM role and can steal credentials.Filter webhooks to trusted branches and actors; never give pull request builds deploy roles; require approval before running outside contributions.
Unclear ownership of failuresResults are split between GitHub and the AWS console, so nobody notices a failed prod deploy.Send pipeline state change events from EventBridge to the team channel and post build status back to the pull request.

Alternatives

What to compare it with

AlternativePick the alternative whenPick CodePipeline and CodeBuild when
GitHub ActionsCode is in GitHub and you want checks on the pull request page with OIDC to AWS and Databricks. This is the site's choice.Policy forbids hosted runners near production, or jobs must reach private networks; then run them on CodeBuild-hosted or self-hosted runners.
GitLab CIYour company already uses GitLab; it supports OIDC to AWS and Databricks the same way.Your code is in GitHub.
Azure PipelinesYou are in an Azure DevOps shop or building the same platform on Azure.You are AWS-only with code in GitHub.
Google Cloud Build and Cloud DeployYou are building on Google Cloud.You are on AWS.
JenkinsYou have a large existing Jenkins estate with shared libraries the team relies on.You are starting fresh; patching and scaling Jenkins controllers is work you no longer need.
Databricks bundle deploy from any CIDeploying jobs and pipelines; it works the same from GitHub Actions, CodeBuild or GitLab.Building AWS resources or workspaces; that is Terraform's job.

Cost

How you pay

CodeBuild is billed for build time: per minute on on-demand EC2 machines by size, per second on Lambda compute, or per instance minute for reserved capacity fleets whether busy or not. CodePipeline V2 pipelines are billed per action execution minute, rounded up per action, with a small free monthly allowance; older V1 pipelines are billed per active pipeline per month. Manual approvals are not charged. All rates are list prices, check your region.

What drives the bill

  • Number and length of builds: every pull request, push and scheduled run uses build minutes.
  • Machine size: larger compute types cost more per minute, so heavy test suites on large machines add up.
  • Reserved capacity fleets, which charge for every minute they are kept, even idle.
  • Network around private builds: NAT gateway data charges and interface endpoints in the build VPC.
  • Artifact and cache storage in S3, and logs in CloudWatch Logs.

Cost traps

  • A V2 pipeline action that waits, such as a deploy action polling for a long Databricks job, is billed for every minute it waits.
  • A reserved fleet sized for peak hours runs, and bills, all night and at weekends.
  • Builds in a VPC that pull packages through a NAT gateway pay per GB for every dependency download.
  • Verbose build logs kept forever in CloudWatch Logs; set a retention period on the log groups.

Prices change; check the AWS pricing page for your region before deciding.

Security and governance

Keeping it safe

  • Give each CodeBuild project its own IAM service role with only the actions it needs; keep prod deploy roles in the prod account and allow only the approved pipeline to assume them.
  • For GitHub Actions, create an IAM OIDC identity provider for token.actions.githubusercontent.com and a role whose trust policy pins the repository and environment in the token's subject.
  • For Databricks, use workload identity federation with a federation policy on a service principal, so CI receives a short-lived Databricks token; the subject must match exactly.
  • Run builds that touch private data or state inside a VPC with VPC endpoints, and encrypt artifacts and caches with a KMS key you own.
  • Pull secrets from Secrets Manager at build time, never from environment variables typed into the console; the shift-left checks block secrets in Git.
  • CloudTrail records pipeline and build API calls, and approvals record who approved which release, which is the audit trail for production changes.

Operating it

What to watch, and the limits

WatchWhy
FailedBuilds and SucceededBuildsCodeBuild metrics per project; a rise in failures on main means broken releases.
Duration and QueuedDurationHow long builds take and how long they wait for a machine; rising queue time suggests a fleet or quota problem.
Pipeline execution state change eventsEventBridge events for FAILED or SUPERSEDED executions, routed to the team channel.
Manual approval ageHow long releases wait for approval; long waits mean changes pile up and get riskier.
Build logsCloudWatch Logs per build, the first place to look for a failed Terraform plan or bundle deploy.
AssumeRoleWithWebIdentity in CloudTrailEach OIDC sign-in from CI, with the repository and branch, to spot unexpected callers.
LimitValue and notes
Build timeoutUp to 36 hours per build (default 60 minutes)
Pipeline stagesBetween 2 and 50 stages per pipeline
Pipelines per region1,000 per account by default, adjustable
Concurrent buildsSet per compute type by account quota, adjustable on request

Limits and quotas checked 2026-11-11; many can be raised through Service Quotas.

Decisions

Situations and verdicts

SituationChoiceVerdict
Code in GitHub, greenfield data platformGitHub Actions with OIDC to AWS and DatabricksPursue: GitHub Actions. Checks sit on the pull request and no long-lived keys are stored.
Company standard is CodePipeline for all releasesCodePipeline with CodeBuild running Terraform and bundle deploysKeep if you have it: same steps with IAM roles instead of stored keys, one release process across teams.
CI jobs must reach a private state bucket and internal mirrorCodeBuild-hosted GitHub Actions runners in the VPCPursue with care: keeps workflows in GitHub while jobs run on AWS; add endpoints so you avoid NAT data charges.
Hosting new repositories on CodeCommitAWS CodeCommit, reopened to new customers Nov 2025Pursue with care: available again, but most data teams and their tooling already live on GitHub or GitLab.

FAQ

Questions people ask

Should I use CodePipeline or GitHub Actions for Databricks on AWS?

If your code is in GitHub, GitHub Actions is simpler: checks appear on the pull request and both AWS and Databricks accept its OIDC tokens. Use CodePipeline if your company already standardises on it or requires builds to run inside AWS.

How does GitHub Actions sign in to Databricks without a token?

Through Databricks workload identity federation. You create a federation policy on a service principal that trusts GitHub's token issuer and the exact subject of your repository or environment, give the workflow id-token write permission and set the auth type to github-oidc. Databricks then issues a short-lived token.

Is CodeCommit still being shut down?

No. AWS stopped onboarding new customers in July 2024 but returned CodeCommit to general availability on 24 November 2025, with new features promised for 2026. Existing repositories continue as normal.

What are CodeBuild-hosted GitHub Actions runners?

A CodeBuild feature where a GitHub workflow job runs on a CodeBuild machine instead of a GitHub-hosted one. You keep writing workflows in GitHub, but the job gets an IAM role, VPC access and AWS billing.

How is CodePipeline billed now?

V2 pipelines are billed per action execution minute, rounded up per action, with a free monthly allowance. Older V1 pipelines are billed per active pipeline per month. Build time in CodeBuild is billed separately.

Sources

Checked against

Checked 2026-11-11.

Related

Where it connects

Data platform on AWS and Databricks · AWS CDK and CloudFormation · AWS IAM and STS · AWS Secrets Manager · Shift-left checks · AWS platform overview · All AWS data services

← CDK and CloudFormation · AWS DMS →