Home › Learn › Data systems › Data platform › Environments and CI/CD

Data platform · concern X16 · Platform-wide

Environments and CI/CD

Every change reaches production the same way: from git, tested where it looks like production, promoted by a pipeline, and easy to undo.

How dev, test and production are kept apart, what data each one holds, how a change travels to production, and the problems special to data: test data, source systems' own test environments, and changing tables that already hold years of history.

For leaders

In plain words

Rehearse, dress rehearsal, then opening night

A theatre rehearses in a practice room, then holds a dress rehearsal on the real stage with real lights, then opens to the public. Nobody rewrites the script on opening night. If something goes wrong, they know exactly which version of the script was played.

A data platform works the same way. Dev is the practice room. Test is the dress rehearsal: real size, real shapes, masked people. Prod is opening night. Every change travels from git through test to production by the same pipeline, so it's always known what is running and how to go back.

The picture

Code moves up, masked data moves down

git: code · manifests · table changes · grants · jobs · infrastructure · semantic definitions · dashboardsdevbuild and trytestprove it, at real sizeprodthe real thingmerge, then deployapprove, then deploymasked sample of datasmaller masked samplecode moves up, only through the pipeline · data never moves up · only masked data moves down
Code moves up through the pipeline. Data never moves up; only masked samples move down.

The environments

What each environment is for

EnvironmentWho works thereWhat dataHow changes arriveWhat is checked
devEngineers, each in their own spaceSmall masked samples and synthetic edge casesEngineers deploy their own branchStatic checks, unit tests
testThe CI pipeline; testersA masked copy of production, at real volume where it mattersOnly the pipeline, after mergeIntegration, contract and data tests, reconciliation, performance
prodService accounts only; people readReal dataOnly the pipeline, after approvalSmoke tests after deploy, then monitoring
UAT or pre-prod (optional)Business users signing offMasked production dataOnly the pipelineUser acceptance; needed in some regulated settings

The layout page shows how these become catalogs: dev_*, test_* and prod_*.

Everything as code

What lives in git

If it isn't in git, it can't be reviewed, tested, promoted or rebuilt.

WhatExamples
Transformation codeSQL, PySpark, dbt models
Manifests and metadataTable entries, quality rules, schedules, retention
Table changesMigrations: add column, change type, backfill steps
PermissionsGrants, masks and row filters, as code
Jobs and orchestrationJob and pipeline definitions per environment target
InfrastructureWorkspaces, storage, networks, catalogs (Terraform and similar)
Semantic definitionsMetric views and semantic models
DashboardsExported definitions, deployed like code
TestsUnit, contract and data tests, and their test data

CI/CD

Six stages from pull request to production

StageWhat happensWhat it catches
1 · Pull requestStatic checks, manifest validation, unit tests, reviewSyntax errors, bad names, broken rules, risky code patterns
2 · Merge and deploy to testThe pipeline deploys everything from git to testAnything that only works on the developer's machine
3 · TestIntegration, contract and data tests on masked data; reconciliation; performance at volumeWrong joins, broken contracts, slow merges, numbers that don't add up
4 · ApproveA named person approves, seeing test results and the change listChanges nobody expected
5 · Deploy to prodThe same artefact, with prod settingsDrift between environments
6 · Smoke test and watchA small run and comparison; freshness and quality alertsProblems only real data shows; roll back if needed

Stage 1 is where shift-left checks run; stage 3 is the test architecture for data.

Concept vs product

The same idea on each platform

PlatformKeeping environments apartDeploying
DatabricksA workspace per environment; catalogs bound to their workspaceAsset Bundles with a target per environment; Terraform for workspaces; service principals deploy
Microsoft FabricA workspace per stageDeployment pipelines plus git integration; deployment rules switch connections per stage
AWSAn account per environmentCDK, Terraform or CloudFormation; CodePipeline or GitHub Actions
Google CloudA project per environmentTerraform; Cloud Build; Dataform release and workflow configurations
SnowflakeA database or account per environmentTerraform or schemachange; zero-copy clones to fill test
Open sourceSeparate clusters or namespacesTerraform, GitHub Actions, dbt targets, Airflow deployments

Product features change often; check the current docs before you build.

The question list

Every question, why we ask, and what breaks if nobody does

#Question to chaseWhy we askWhat breaks in production if nobody asksDo this (not this)
A · Environmentshow many, and kept apart how
A1How many environments are there, and what is each one for?An environment with three jobs does none of them well.'Test' is used for demos, user sign-off and development at once, and is never stable.Dev, test and prod, each with one purpose; add UAT only when sign-off needs it. Not: one shared non-prod.
A2Are environments separate accounts or workspaces, with separate storage?Shared storage means a test job can delete production files.A clean-up script in test empties a production folder.Separate workspaces or accounts and storage per environment. Not: folders named /dev and /prod in one bucket.
A3Are environments built from code, and rebuilt regularly?Hand-built environments drift apart.It works in test and fails in production because a setting differs.Infrastructure as code for every environment; rebuild test from it on a schedule. Not: clicking in the console.
B · Data in lower environmentswhat test data looks like
B1What data does test hold, and at what size?Logic and performance problems only show on realistic data.Tests pass on 1,000 synthetic rows; production has 2 billion, skewed to one customer, and the merge runs for hours.A masked copy of production, plus volume tests on the biggest tables. Not: hand-made rows only.
B2Is personal data masked before it leaves production?Lower environments have more people and weaker controls.Real customer data ends up on a contractor's laptop.Mask inside production, consistently, before copying down. Not: copy first, mask later.
B3How is test data refreshed?Stale test data misses new codes and new shapes.A new status code appears in production that test has never seen; the release breaks.A scheduled refresh of the masked copy. Not: a copy taken once at the start of the project.
C · Source systems' own environmentsthe part teams forget
C1Does each source have a test environment, and does it hold realistic data?Source test systems often hold a dozen made-up records.Integration tests pass against the source's test system; production breaks on the first real file.Use the source's test system to prove connectivity; test logic on masked production copies. Not: trusting source test data.
C2Can test connect to sources without touching production sources?A test pipeline pointed at a production source can load it, or write back to it.A test job sends reverse-ETL updates into the production CRM.Separate connections per environment, set by the deployment target; write-backs go to a sandbox outside prod. Not: shared connection strings.
D · Promotionhow a change reaches production
D1Who can deploy to production, and how?Most incidents follow a change.A hotfix made in the production UI is overwritten by the next deploy.Only the pipeline's service account deploys; break-glass access is logged and reviewed. Not: personal admin rights in prod.
D2What must pass before a change reaches production?Gates turn 'we think it works' into evidence.A change that broke reconciliation in test is approved because nobody looked.Required checks in the pipeline, with results shown at approval. Not: an approval button with no evidence.
D3How do environment settings and secrets differ, and where do they live?Code is promoted unchanged; only settings change.A production password is found in dev settings, or code still points at dev after promotion.Settings per deployment target; secrets in a vault per environment. Not: if env == 'prod' in code.
E · Changing tables that hold datathe hard part of data CI/CD
E1How do we change the schema of a table with years of data?A data table can't be thrown away and recreated like a web page.A column-type change fails halfway, or a rebuild takes three days.Expand and contract: add the new column, backfill, move readers, then drop the old one, as migrations in git. Not: ALTER by hand in production.
E2When logic changes, how is history rebuilt, and who is told?A deploy can quietly change last year's numbers.Finance finds that last year's revenue moved after a release.Backfills are planned jobs: compare before and after, then announce. Not: letting the next run rewrite history.
E3What happens to streaming jobs and their checkpoints on deploy?A streaming job's saved position decides what it reads next.A redeploy re-reads a week of events, or skips an hour.Compatible changes keep the checkpoint; incompatible ones start a new one with a planned backfill. Not: deleting checkpoints to make it start.
F · Rollbackthe way back
F1Can we undo a bad release: both the code and the data it wrote?Rolling back code doesn't fix data already written.The code is rolled back, but a day of wrong rows stays in gold.Keep table history (time travel) and restore; for big changes, build the new table alongside and switch a view. Not: code-only rollback.
G · Costwhat non-production costs
G1What do dev and test cost, and when are they switched off?Non-production can quietly cost as much as production.A full-size test environment runs all weekend, every weekend.Smaller compute, auto-stop, sampled data, scheduled shutdowns; scale up only for volume tests. Not: prod-sized and always on.

Design decisions

The decisions to take

How many environments?

OptionVerdict
Dev, test, prodPursue · Pursue. Each has one purpose.
Add UAT or pre-prodPursue when · Pursue only when business sign-off or regulation needs a separate stage.
Dev and prod onlyDrop · Drop. Changes are first tested on real users.

What data goes in test?

OptionVerdict
A masked copy of production, plus synthetic edge casesPursue · Pursue. Real shapes and volumes, no real people.
Synthetic data onlyPursue when · Pursue only before production exists (a new system).
An unmasked copy of productionDrop · Drop. It's a data breach waiting to happen.

How do changes deploy?

OptionVerdict
A pipeline from git, with infrastructure as codePursue · Pursue. Every environment is built the same way.
By hand in each environment's UIDrop · Drop. Environments drift and nobody knows what changed.

How do environments differ in code?

OptionVerdict
Settings per deployment targetPursue · Pursue. The same artefact runs everywhere.
if env == 'prod' branches in codeDrop · Drop. The production path is never tested.

How do big tables change shape?

OptionVerdict
Expand and contract, as migrations in gitPursue · Pursue. Readers never see a broken table.
Drop and rebuildPursue when · Pursue only for small tables that rebuild in minutes.
Build the new version alongside and switch a viewPursue when · Pursue for large or risky logic changes; switching back is instant.

FAQ

Questions people ask

How many environments does a data platform need?

Usually three: dev for building, test for proving on realistic masked data, and prod. Add a user-acceptance stage only when business sign-off or regulation requires one.

Should test use production data?

A masked copy of production, yes: realistic shapes and volumes are what find problems. An unmasked copy, no. Mask inside production, consistently, before anything is copied down.

What is different about CI/CD for data compared with apps?

Tables hold history, so you can't simply replace them; changes need migrations and planned backfills. Tests need realistic data. And a rollback must undo the data a bad release wrote, not just the code.

Next

Where to go next

Related concerns: Layout: catalogs, schemas and naming; Testing (coming; until then, test architecture for data).

Read more: CI/CD for Databricks with Asset Bundles (blog), CI/CD and infrastructure as code.

Back up: The data platform, whole, with all 24 concerns.