Home › Learn › Data systems › Data platform › Environments and CI/CD
Data platform · concern X16 · Platform-wide
Environments and CI/CD
Every change reaches production the same way: from git, tested where it looks like production, promoted by a pipeline, and easy to undo.
How dev, test and production are kept apart, what data each one holds, how a change travels to production, and the problems special to data: test data, source systems' own test environments, and changing tables that already hold years of history.
For leaders
- Code moves up from dev to test to production through a pipeline. Data never moves up; only masked samples move down.
- Nobody changes production by hand. A pipeline deploys from git after tests pass and a named person approves.
- The hard parts of data CI/CD are not the code. They are realistic test data, source systems whose own test environments hold little real data, and changing tables that already hold years of history.
- Everything lives in git: code, table definitions, permissions, schedules, semantic definitions, dashboards and infrastructure, so any environment can be rebuilt.
- A realistic test environment costs money. Finding the same problem in production costs more.
In plain words
Rehearse, dress rehearsal, then opening night
A theatre rehearses in a practice room, then holds a dress rehearsal on the real stage with real lights, then opens to the public. Nobody rewrites the script on opening night. If something goes wrong, they know exactly which version of the script was played.
A data platform works the same way. Dev is the practice room. Test is the dress rehearsal: real size, real shapes, masked people. Prod is opening night. Every change travels from git through test to production by the same pipeline, so it's always known what is running and how to go back.
The picture
Code moves up, masked data moves down
The environments
What each environment is for
| Environment | Who works there | What data | How changes arrive | What is checked |
|---|---|---|---|---|
| dev | Engineers, each in their own space | Small masked samples and synthetic edge cases | Engineers deploy their own branch | Static checks, unit tests |
| test | The CI pipeline; testers | A masked copy of production, at real volume where it matters | Only the pipeline, after merge | Integration, contract and data tests, reconciliation, performance |
| prod | Service accounts only; people read | Real data | Only the pipeline, after approval | Smoke tests after deploy, then monitoring |
| UAT or pre-prod (optional) | Business users signing off | Masked production data | Only the pipeline | User acceptance; needed in some regulated settings |
The layout page shows how these become catalogs: dev_*, test_* and prod_*.
Everything as code
What lives in git
If it isn't in git, it can't be reviewed, tested, promoted or rebuilt.
| What | Examples |
|---|---|
| Transformation code | SQL, PySpark, dbt models |
| Manifests and metadata | Table entries, quality rules, schedules, retention |
| Table changes | Migrations: add column, change type, backfill steps |
| Permissions | Grants, masks and row filters, as code |
| Jobs and orchestration | Job and pipeline definitions per environment target |
| Infrastructure | Workspaces, storage, networks, catalogs (Terraform and similar) |
| Semantic definitions | Metric views and semantic models |
| Dashboards | Exported definitions, deployed like code |
| Tests | Unit, contract and data tests, and their test data |
CI/CD
Six stages from pull request to production
| Stage | What happens | What it catches |
|---|---|---|
| 1 · Pull request | Static checks, manifest validation, unit tests, review | Syntax errors, bad names, broken rules, risky code patterns |
| 2 · Merge and deploy to test | The pipeline deploys everything from git to test | Anything that only works on the developer's machine |
| 3 · Test | Integration, contract and data tests on masked data; reconciliation; performance at volume | Wrong joins, broken contracts, slow merges, numbers that don't add up |
| 4 · Approve | A named person approves, seeing test results and the change list | Changes nobody expected |
| 5 · Deploy to prod | The same artefact, with prod settings | Drift between environments |
| 6 · Smoke test and watch | A small run and comparison; freshness and quality alerts | Problems only real data shows; roll back if needed |
Stage 1 is where shift-left checks run; stage 3 is the test architecture for data.
Concept vs product
The same idea on each platform
| Platform | Keeping environments apart | Deploying |
|---|---|---|
| Databricks | A workspace per environment; catalogs bound to their workspace | Asset Bundles with a target per environment; Terraform for workspaces; service principals deploy |
| Microsoft Fabric | A workspace per stage | Deployment pipelines plus git integration; deployment rules switch connections per stage |
| AWS | An account per environment | CDK, Terraform or CloudFormation; CodePipeline or GitHub Actions |
| Google Cloud | A project per environment | Terraform; Cloud Build; Dataform release and workflow configurations |
| Snowflake | A database or account per environment | Terraform or schemachange; zero-copy clones to fill test |
| Open source | Separate clusters or namespaces | Terraform, GitHub Actions, dbt targets, Airflow deployments |
Product features change often; check the current docs before you build.
The question list
Every question, why we ask, and what breaks if nobody does
| # | Question to chase | Why we ask | What breaks in production if nobody asks | Do this (not this) |
|---|---|---|---|---|
| A · Environmentshow many, and kept apart how | ||||
| A1 | How many environments are there, and what is each one for? | An environment with three jobs does none of them well. | 'Test' is used for demos, user sign-off and development at once, and is never stable. | Dev, test and prod, each with one purpose; add UAT only when sign-off needs it. Not: one shared non-prod. |
| A2 | Are environments separate accounts or workspaces, with separate storage? | Shared storage means a test job can delete production files. | A clean-up script in test empties a production folder. | Separate workspaces or accounts and storage per environment. Not: folders named /dev and /prod in one bucket. |
| A3 | Are environments built from code, and rebuilt regularly? | Hand-built environments drift apart. | It works in test and fails in production because a setting differs. | Infrastructure as code for every environment; rebuild test from it on a schedule. Not: clicking in the console. |
| B · Data in lower environmentswhat test data looks like | ||||
| B1 | What data does test hold, and at what size? | Logic and performance problems only show on realistic data. | Tests pass on 1,000 synthetic rows; production has 2 billion, skewed to one customer, and the merge runs for hours. | A masked copy of production, plus volume tests on the biggest tables. Not: hand-made rows only. |
| B2 | Is personal data masked before it leaves production? | Lower environments have more people and weaker controls. | Real customer data ends up on a contractor's laptop. | Mask inside production, consistently, before copying down. Not: copy first, mask later. |
| B3 | How is test data refreshed? | Stale test data misses new codes and new shapes. | A new status code appears in production that test has never seen; the release breaks. | A scheduled refresh of the masked copy. Not: a copy taken once at the start of the project. |
| C · Source systems' own environmentsthe part teams forget | ||||
| C1 | Does each source have a test environment, and does it hold realistic data? | Source test systems often hold a dozen made-up records. | Integration tests pass against the source's test system; production breaks on the first real file. | Use the source's test system to prove connectivity; test logic on masked production copies. Not: trusting source test data. |
| C2 | Can test connect to sources without touching production sources? | A test pipeline pointed at a production source can load it, or write back to it. | A test job sends reverse-ETL updates into the production CRM. | Separate connections per environment, set by the deployment target; write-backs go to a sandbox outside prod. Not: shared connection strings. |
| D · Promotionhow a change reaches production | ||||
| D1 | Who can deploy to production, and how? | Most incidents follow a change. | A hotfix made in the production UI is overwritten by the next deploy. | Only the pipeline's service account deploys; break-glass access is logged and reviewed. Not: personal admin rights in prod. |
| D2 | What must pass before a change reaches production? | Gates turn 'we think it works' into evidence. | A change that broke reconciliation in test is approved because nobody looked. | Required checks in the pipeline, with results shown at approval. Not: an approval button with no evidence. |
| D3 | How do environment settings and secrets differ, and where do they live? | Code is promoted unchanged; only settings change. | A production password is found in dev settings, or code still points at dev after promotion. | Settings per deployment target; secrets in a vault per environment. Not: if env == 'prod' in code. |
| E · Changing tables that hold datathe hard part of data CI/CD | ||||
| E1 | How do we change the schema of a table with years of data? | A data table can't be thrown away and recreated like a web page. | A column-type change fails halfway, or a rebuild takes three days. | Expand and contract: add the new column, backfill, move readers, then drop the old one, as migrations in git. Not: ALTER by hand in production. |
| E2 | When logic changes, how is history rebuilt, and who is told? | A deploy can quietly change last year's numbers. | Finance finds that last year's revenue moved after a release. | Backfills are planned jobs: compare before and after, then announce. Not: letting the next run rewrite history. |
| E3 | What happens to streaming jobs and their checkpoints on deploy? | A streaming job's saved position decides what it reads next. | A redeploy re-reads a week of events, or skips an hour. | Compatible changes keep the checkpoint; incompatible ones start a new one with a planned backfill. Not: deleting checkpoints to make it start. |
| F · Rollbackthe way back | ||||
| F1 | Can we undo a bad release: both the code and the data it wrote? | Rolling back code doesn't fix data already written. | The code is rolled back, but a day of wrong rows stays in gold. | Keep table history (time travel) and restore; for big changes, build the new table alongside and switch a view. Not: code-only rollback. |
| G · Costwhat non-production costs | ||||
| G1 | What do dev and test cost, and when are they switched off? | Non-production can quietly cost as much as production. | A full-size test environment runs all weekend, every weekend. | Smaller compute, auto-stop, sampled data, scheduled shutdowns; scale up only for volume tests. Not: prod-sized and always on. |
Design decisions
The decisions to take
How many environments?
| Option | Verdict |
|---|---|
| Dev, test, prod | Pursue · Pursue. Each has one purpose. |
| Add UAT or pre-prod | Pursue when · Pursue only when business sign-off or regulation needs a separate stage. |
| Dev and prod only | Drop · Drop. Changes are first tested on real users. |
What data goes in test?
| Option | Verdict |
|---|---|
| A masked copy of production, plus synthetic edge cases | Pursue · Pursue. Real shapes and volumes, no real people. |
| Synthetic data only | Pursue when · Pursue only before production exists (a new system). |
| An unmasked copy of production | Drop · Drop. It's a data breach waiting to happen. |
How do changes deploy?
| Option | Verdict |
|---|---|
| A pipeline from git, with infrastructure as code | Pursue · Pursue. Every environment is built the same way. |
| By hand in each environment's UI | Drop · Drop. Environments drift and nobody knows what changed. |
How do environments differ in code?
| Option | Verdict |
|---|---|
| Settings per deployment target | Pursue · Pursue. The same artefact runs everywhere. |
| if env == 'prod' branches in code | Drop · Drop. The production path is never tested. |
How do big tables change shape?
| Option | Verdict |
|---|---|
| Expand and contract, as migrations in git | Pursue · Pursue. Readers never see a broken table. |
| Drop and rebuild | Pursue when · Pursue only for small tables that rebuild in minutes. |
| Build the new version alongside and switch a view | Pursue when · Pursue for large or risky logic changes; switching back is instant. |
FAQ
Questions people ask
How many environments does a data platform need?
Usually three: dev for building, test for proving on realistic masked data, and prod. Add a user-acceptance stage only when business sign-off or regulation requires one.
Should test use production data?
A masked copy of production, yes: realistic shapes and volumes are what find problems. An unmasked copy, no. Mask inside production, consistently, before anything is copied down.
What is different about CI/CD for data compared with apps?
Tables hold history, so you can't simply replace them; changes need migrations and planned backfills. Tests need realistic data. And a rollback must undo the data a bad release wrote, not just the code.
Next
Where to go next
Related concerns: Layout: catalogs, schemas and naming; Testing (coming; until then, test architecture for data).
Read more: CI/CD for Databricks with Asset Bundles (blog), CI/CD and infrastructure as code.
Back up: The data platform, whole, with all 24 concerns.