Home › Learn › Data platform architecture › On AWS + Databricks
Data platform architecture · binding · AWS + Databricks
The data platform on AWS and Databricks
The same architecture as the abstract picture, with a product in every box, for a tyre manufacturer that already runs on AWS and wants Databricks. The design stays the same; only the names change.
The one-minute version
Your account, your data, one catalog
Databricks runs inside your own AWS account. Data stays in your S3 buckets, in your VPC, encrypted with KMS keys you own. Databricks brings the engine, and Unity Catalog (its one catalog for tables, files and models) governs who sees what, tracks lineage and records every access.
The design does not depend on Databricks. It follows the same specs and the same 22 cross-cutting concerns as the abstract architecture. This page is the binding: the one place where product names appear. That is what keeps you free to change a product later.
Most of the value is in the shared parts. Ingestion, metadata, quality, cost, testing and the conversation layer are built once and reused, so each new data product (OEE, warranty claim rate, dealer sell-out, demand forecast) costs less than the one before.
The big picture
One platform, from plant and business systems to answers
Read it left to right along the middle row: data moves one way, from the plants' and the business's systems to dashboards, Genie and AI. Above it, the control plane decides what runs, and agents may only suggest. Below it, the horizontal assets serve every layer, on an AWS and Databricks foundation you own. Governance runs down the right side.
Horizontal assets
Built once, used by every data product
A horizontal asset is something every pipeline needs, whatever the subject. Databricks gives the plumbing; these assets are the parts that make the platform trustworthy and cheap to grow. The status column is honest: some are built, some are designed, some are still to build.
| Asset | What every team gets | Built on | Status |
|---|---|---|---|
| CostSherlock | Cost per data product, per pipeline run and per team, from Databricks billing system tables and the AWS Cost and Usage Report, with findings and fixes. | Ours, on Databricks + AWS | Separate briefing |
| Common Ingestion Framework | A new source is a YAML entry, not a new notebook. One lander from landing to bronze, with ingest metadata, replay and watermarks. | Ours, on Databricks; movers behind a port (DMS, AppFlow, Transfer Family, Kinesis, our puller) | Built; first end-to-end Databricks run in progress |
| Metadata store | One place where every source, table, rule, owner and budget is declared. Pipelines read it; nobody configures by hand. | Ours; Lakebase (Databricks managed PostgreSQL) | Built in the framework (design published); the Lakebase deployment is to do |
| Data quality framework | Rules declared per table, run at every hop, bad rows parked in quarantine, all results in one table. | Databricks expectations and DQX behind one port; rules from the metadata store | Partly built: rules from metadata run in silver; 66 assurance rules written |
| Medallion template | Bronze, silver and gold built the same way for every source, with the rules that keep each layer honest. | Lakeflow Declarative Pipelines | Built for the first slice |
| Semantic model template | A starter set of metric views (OEE, scrap rate, first-pass yield, warranty claim rate, sell-out) with owners and definitions. | Unity Catalog Metric Views | To build |
| Conversation UI | Staff ask questions in plain words and get answers that cite the metric used. The model picks metrics; it never writes SQL. | Ours, on Databricks Apps, reading Metric Views | Designed (spec written) |
| Shift-left checks bundle | Checks that run on the code before it deploys: PySpark, SQL and Databricks mistakes caught in CI, not in production. | Open source, ours; runs in CI | Built: catches 88 of 101 planted problems, no false alarms |
| DataTwin | A pretend tyre plant and dealer network that sends data on its own clock and breaks things on purpose, so the platform proves it catches each fault. | Ours, open source | Studio built; the full tool designed |
| Observability and run log | Every run gets one id that joins its quality results, lineage, cost and audit, so any number can be traced to evidence. | System tables + CloudWatch + our run log | Run log built; joined evidence view to build |
The foundation
Accounts, network, storage and keys, before any pipeline
Before a single pipeline runs, the platform needs somewhere safe to live: AWS accounts to hold it, a private network, buckets for the data with keys you own, Databricks workspaces to work in, and one catalog that decides who sees what. Think of it as the land, the roads and the locks. Done right once, nobody thinks about it again; done wrong, every later layer inherits the problem.
The nine choices
| Decision | Our pick | Why |
|---|---|---|
| AWS accounts | One AWS account per environment: dev, pre-prod, prod. All three sit under your AWS Organization and send logs to your central security and log accounts. | A mistake in dev cannot touch prod, permissions stay simple, and each environment's bill is separate. |
| Region | Mumbai (ap-south-1) is home. Hyderabad (ap-south-2) holds the recovery copy. | Both are in India, so recovery does not move data out of the country. The full recovery plan is in the running-it section. |
| EU plants | Plant and production data from EU plants comes to Mumbai too. Personal data about EU people (staff, dealers, consumers) moves only under the legal basis your legal team picks, and is tokenised on arrival. | Press readings and scrap counts are not personal data. One platform costs half of two. A separate EU platform (Frankfurt) only if your legal team requires it. |
| Databricks workspaces | Three workspaces (dev, pre-prod, prod) on one Unity Catalog. Each catalog is bound to its workspace, so prod tables are visible only in prod. | Databricks allows one Unity Catalog per region, so the three share it; binding keeps the environments apart. |
| Network | No public IP addresses anywhere. Classic compute runs in your own VPC, across two availability zones; serverless compute runs in Databricks' serverless plane in the same region and reaches your data over private connections. Databricks is reached through PrivateLink, never the open internet. | This is the first thing a manufacturer's security team asks. Two zones keep the platform up if one data centre fails. |
| Storage and keys | One S3 bucket and one KMS key per catalog. A landing bucket per environment. Recipes get their own bucket and key with the tightest access. | Recipes are the crown jewels: with their own key, revoking all access is one action, and every use of the key is logged. |
| Compute | Serverless by default for SQL, jobs and pipelines. Classic compute (clusters inside your VPC) only where a job must reach a private system that serverless cannot yet reach. | Starts in seconds, nothing to patch or size. Serverless private reach on AWS: check current status. |
| Sign-in and identity | People sign in through your existing identity provider; groups sync to Databricks automatically (SCIM). Jobs run as service principals, never as a person. | When someone leaves, one change removes all their access, and no job breaks because its owner left. |
| How it is built | Terraform builds all of the above from Git (AWS and Databricks providers). Pipelines are deployed separately with Databricks Asset Bundles. | The whole foundation can be rebuilt the same way in every environment, and in Hyderabad after a disaster. |
What we assumed
This design is written for a typical Indian manufacturer already on AWS. Where your answer differs, the right column says what changes. None of them changes the layers above.
| Question | We assume | If it is different |
|---|---|---|
| Is there already an AWS landing zone (Control Tower or similar)? | Yes, with central security and log-archive accounts | If not, we add a minimal one first: an Organization, CloudTrail to a log account, guardrails (SCPs). About a week. |
| Which identity provider? | Microsoft Entra ID | Okta or another SAML / OIDC provider works the same way; only the Terraform settings change. |
| How do plants reach AWS? | Direct Connect from the head office, Indian plants on the company network (MPLS or SD-WAN); EU plants through a site-to-site VPN to the Mumbai Transit Gateway | With only VPN, sensor streaming is sized smaller and the plant historian is pulled in batches. |
| Where does SAP run? | SAP S/4HANA on AWS (RISE) or in a data centre linked to AWS | The mover changes (see the ingestion table); the foundation does not. |
| Must EU data stay in the EU? | Not for plant data; personal data under Standard Contractual Clauses | If yes for everything, add a fourth workspace in Frankfurt on its own Unity Catalog, and share gold tables back with Delta Sharing. |
Which workspace sees which catalog
Catalogs are named by environment and layer (prod_silver, dev_gold). Binding each catalog to workspaces is what keeps a test in dev from ever writing to prod, even by mistake.
| Catalog | Dev workspace | Pre-prod workspace | Prod workspace |
|---|---|---|---|
| dev_* catalogs | Read and write | Not visible | Not visible |
| preprod_* catalogs | Not visible | CI writes; engineers read the results schemas only, not the clones | Not visible |
| prod_bronze, prod_silver, prod_gold, prod_semantic | Not visible | Read only, by the CI service principal (for real-data tests and shallow clones); people in pre-prod see the results, not the prod rows | Jobs write; people read through grants |
| prod_restricted | Not visible | Not visible | Recipe group only |
| prod_pii_vault | Not visible | Not visible | Privacy case handlers only |
| prod_ops | Not visible | Read only | Pipelines write; everyone on the platform reads, except the ai_log schema |
Buckets and keys
Every bucket blocks public access, is encrypted with a KMS key you own, and is reached only through the VPC endpoint or by Databricks through an IAM role it is allowed to assume.
| Bucket | Holds | Key | Rules |
|---|---|---|---|
| landing-<env> | Raw files from the movers, as they arrived | Environment key | Files deleted 30 days after they reach bronze; versioning on, old versions expire with them |
| landing-pii-<env> | Raw files from sources with personal data | Its own key | Deleted within a day of being loaded by an hourly purge; held files kept up to 7 days; old versions expire with them |
| <env>-pii-vault | The token vault and the suppression list | Its own key; only privacy case handlers | Every lookup logged |
| <env>-bronze, -silver, -gold, -semantic, -ops | The managed tables of each catalog | One key per catalog | Delta handles history; old versions cleaned by VACUUM after 30 days |
| <env>-restricted | Recipes and compound formulations | Its own key; only the recipe group can use it | Access logged and reviewed monthly |
| audit-<env> | Copies of audit and config change logs | Environment key | Object Lock: cannot be changed or deleted for 7 years |
| dr-* (Hyderabad) | Recovery copies: bronze, silver, gold, restricted and the vault | Keys in Hyderabad, same groups | See the recovery plan; erasures are applied here too; restore tested twice a year |
Built from code
The foundation is Terraform in Git, applied in five steps. The environments differ only in one settings file each, so dev, pre-prod and prod are the same platform at different sizes.
foundation/ # Terraform, one folder per layer, applied in this order 1-org/ accounts, guardrails (SCPs), CloudTrail to the log account 2-network/ VPC per env, private subnets in 2 zones, endpoints (S3, STS, Kinesis), PrivateLink 3-storage/ buckets and KMS keys per catalog, IAM roles Databricks may assume 4-databricks/ workspaces, metastore, catalogs, workspace bindings, external locations 5-identity/ service principals, catalog-level grants (groups come from your identity provider) envs/ dev.tfvars, preprod.tfvars, prod.tfvars, dr.tfvars # the only differences
Folder layout of the foundation code. Pipelines are not here: they are deployed with Asset Bundles.
Recovery: if Mumbai is lost, the same Terraform with dr.tfvars builds the workspace in Hyderabad on the copied data, and the data-platform deploy puts back every grant, filter and mask before anyone can read. The full plan, with targets per layer, is in running it.
Metadata and ingestion on AWS
One metadata database, one ingestion framework
The Common Ingestion Framework is metadata-driven: adding a source is a few lines of YAML, not a new pipeline. It follows the Metadata DB Design for Data Engineering Platforms, already built in our framework. Full table definitions, the view and the reader code are on the metadata store page.
The tables
| Table | What it holds | Written by |
|---|---|---|
| platform_config | Settings for the whole platform: default storage paths, default quality action, budget limits | Platform team, through YAML |
| source_master | One row per source system: SAP S/4HANA, MES, historian, quality LIMS, warranty, dealer DMS | Through YAML |
| source_config | Key-value settings per source, grouped (ingestion, quality, cost): connection secret reference, mover, schedule | Through YAML |
| dataset_master | One row per dataset (table) of a source, with its target schema and table | Through YAML |
| dataset_config | Key-value settings per dataset: load mode, keys, history (SCD) type, profiling settings, PII purpose and retention, owner and deputy, freshness target and tier. Quality rules have their own table, dq_rule | Through YAML |
| dataset_view | The one view every pipeline reads: platform, then source, then dataset settings merged into JSON, the most specific winning | Generated |
| run_log | One row per run: dataset, status, row counts, cost, and a fingerprint of the exact settings used | Pipelines |
| watermark | How far each incremental load has got | Pipelines |
How it all runs, step by step from a source to gold, with every job, trigger, id and log row: The ETL, end to end.
The framework also has a key registry (every allowed setting and its type), per-column settings, quality results and a PII policy table on the same database.
YAML in GitHub, and a SQL database
We need both, for different jobs. People write and approve settings as YAML in GitHub; a loader validates every key and writes them to the database, one database per environment (dev, pre-prod, prod) with the same schema. Pipelines only ever read the database.
| YAML in GitHub | SQL metadata database | |
|---|---|---|
| What it holds | Configuration: sources, datasets, load modes, keys, rules, owners, PII tags, budgets | The same configuration loaded from Git, plus runtime state: watermarks, run log, quality results |
| Who writes it | People, and agents, by pull request | The loader writes configuration; pipelines write runtime state |
| Why we need it | Review and approval, full history, easy rollback, the same file promoted dev → pre-prod → prod | Fast lookups, keys between tables, the merged view, many jobs writing at once |
| Do pipelines read it? | Never | Always |
Two rules. Nobody edits configuration in the database by hand: the next load would overwrite it and Git would have no record. Runtime state (watermarks, run log) never goes into Git.
Which database on AWS
The metadata store needs keys between tables, fast small reads and writes, safe parallel writes from many jobs, and good JSON for the merged view. That is a relational database, and PostgreSQL fits the design as written.
| Database | Fit | Verdict | Why |
|---|---|---|---|
| Databricks Lakebase (managed PostgreSQL inside Databricks) | PostgreSQL run by Databricks, governed by Unity Catalog, reachable from serverless compute with no network setup, with point-in-time restore. Strong JSON (jsonb) for the merged view, as in any PostgreSQL. Generally available on AWS since early 2026 in selected regions. | Pursue: chosen (4 Nov) | Chosen over Aurora PostgreSQL to drop one AWS service and the network hop; stored procedures stay the only way in. Confirm Mumbai availability before go-live (check current status). |
| Amazon Aurora PostgreSQL Serverless v2 | Managed PostgreSQL with automatic failover, IAM sign-in, backups, and compute that scales down to almost nothing in dev. Strong JSON (jsonb) for the merged view. | Considered, not chosen | The default until 4 Nov and a good fit for the design, but one more AWS service to run, connect privately and back up; Lakebase gives the same PostgreSQL inside the platform. |
| Amazon RDS for PostgreSQL | The same PostgreSQL on a small fixed-size server. Cheaper and simpler; slower failover and no automatic scaling. | Drop | Same drawback as Aurora: one more AWS service to run, with no gain over Lakebase. |
| Aurora MySQL or RDS MySQL | Works, but JSON handling is weaker than PostgreSQL's and the view would need rewriting. | Drop | No gain over PostgreSQL for this design. |
| Amazon DynamoDB | Key-value store with no joins, so no merged view and no keys between tables. | Drop | Wrong shape for relational metadata. |
| Delta tables in Unity Catalog only | No keys between tables, slow small writes, write conflicts when many jobs log at once. | Drop as the store | Keep a copy of run_log in Delta for dashboards and cost reports. |
Where it sits: on Lakebase, inside the Databricks platform, in the same region as the workspace (Mumbai; Frankfurt only if EU data must have its own platform). It was chosen over Aurora PostgreSQL on 4 Nov so that there is one AWS service fewer to run, connect and back up. Serverless and classic compute reach it with no extra network setup, and it is registered in Unity Catalog as a catalog. Stored procedures remain the only way in. The runtime tables are copied into Delta every 15 minutes so dashboards and cost reports can join them without touching the database.
Three schemas in one database
One PostgreSQL database per environment, split into three schemas so that permissions are simple: the loader may write only config and audit, pipelines may write only runtime, and dashboards read everything.
| Schema | Holds | Who writes | Size and retention |
|---|---|---|---|
| config | What to do: platform_config, source_master, source_config, dataset_master, dataset_config, the key registry, quality rules, PII policy, and dataset_view | Only the loader writes; pipelines read | Small; changes only through Git |
| runtime | What happened: run_log, step_log, error_log, dq_result, watermark, config_snapshot | Pipelines write; nobody else | Grows every run; monthly partitions, 13 months kept, then summarised into prod_ops |
| audit | Who changed what: config_load_log (every loader run, Git commit, who approved, what changed) | Only the loader appends; never updated or deleted | Small; kept 7 years, like the audit bucket |
Logging: every run can be followed and replayed
Every run writes to the runtime schema. One run id ties the rows together, and it is also stored on the Databricks job run, so the logs join with Databricks system tables (cost, lineage, audit) and with the quality results.
| Table | One row per | Main columns |
|---|---|---|
| run_log | One row per run of one dataset | run_id, dataset_id, Databricks job and run ids, status, started and ended, rows read / written / rejected, config fingerprint, Git commit, estimated cost |
| step_log | One row per step inside a run: land, bronze, silver, gold | run_id, step, status, timings, row counts in and out |
| error_log | One row per failure | run_id, step, error class (source down, schema change, bad data, bug), message, retry number, whether it was retried |
| dq_result | One row per quality rule checked | run_id, rule id, passed or failed, failing row count, action taken (warn, quarantine, stop) |
| config_snapshot | The exact merged settings a run used, stored once per fingerprint | fingerprint, merged JSON; any replay can prove what it ran with |
The watermark: load only what is new, safely
A watermark is a bookmark: the last value loaded from a source (a timestamp, an id, a change position). The next run starts after it. The danger is moving the bookmark before the data is safely written, or never moving it after a crash. These rules prevent both.
-- schema runtime: written only by pipelines CREATE TABLE runtime.run_log ( run_id uuid PRIMARY KEY, dataset_id uuid NOT NULL REFERENCES config.dataset_master(dataset_id), job_run_id text, -- Databricks job run id, to join system tables status text NOT NULL, -- started, succeeded, failed, skipped, blocked started_at timestamptz NOT NULL, ended_at timestamptz, window_low text, -- watermark this run started from window_high text, -- watermark this run loaded up to rows_read bigint, rows_written bigint, rows_rejected bigint, config_fp text NOT NULL, -- fingerprint of the merged settings git_commit text, est_cost_usd numeric(12,4) ); CREATE TABLE runtime.watermark ( dataset_id uuid PRIMARY KEY REFERENCES config.dataset_master(dataset_id), wm_column text NOT NULL, -- e.g. updated_at, change sequence number wm_type text NOT NULL, -- timestamp, number, cdc_position, delta_token wm_value text NOT NULL, -- last value loaded successfully last_run_id uuid REFERENCES runtime.run_log(run_id), updated_at timestamptz NOT NULL DEFAULT now() );
The two runtime tables most people ask about first. The full config schema is on the metadata store page.
Ingestion: movers bring data in, our lander loads it
The full list of sources, the five ingestion technologies and the landing contract are on The ETL, end to end. Where this page and that one differ on how the runtime works, that page is the one to follow.
Ingestion has two steps. A mover brings raw data from the source into an S3 landing folder; which mover is a setting in source_config, so it can change without code. Then one lander, our PySpark job on Databricks that reads exactly the new files listed in its file log, reads its settings from dataset_view, writes bronze with arrival details, records a run_log row and is safe to run twice. Lakeflow Jobs are generated from the metadata and deployed with Databricks Asset Bundles.
| Source | Mover (first hop to S3) | Note |
|---|---|---|
| SAP S/4HANA (sales, stock, procurement, finance) | Amazon AppFlow, SAP OData connector, on released CDS views and extractors with delta | Never SAP tables directly; Datasphere or Business Data Cloud only if already licensed |
| MES and quality / LIMS databases | AWS DMS with change capture | Change capture keeps loads small and brings deletes |
| Small and reference tables, databases without CDC | Our puller (the framework's built connectors) as a Databricks job | Full copy or watermark, straight to bronze through the same lander steps (no landing) |
| Plant historian and press / mixer sensors | AWS IoT SiteWise → Kinesis or S3 | The streaming lane: same layers, trigger-once every 15 minutes |
| Dealer management system, CRM (SaaS) | Amazon AppFlow; our puller's REST connector where AppFlow has none | Lakeflow Connect is not used in v1 |
| Supplier files, raw-material prices | AWS Transfer Family (SFTP) → S3 | The lander reads the new files listed in its file log |
Orchestration: which service runs what, and when
One rule: one orchestrator owns each pipeline. Lakeflow Jobs owns everything from the landing folder onwards; AWS services only start the movers on the AWS side and hand over. Two schedulers for the same pipeline is the most common cause of runs that overlap or never start.
| Service | What it does here | Verdict | Why |
|---|---|---|---|
| Lakeflow Jobs (Databricks) | Runs every Databricks task: lander, silver, gold, quality, refresh of metric views. Schedules, file-arrival and table-update triggers, retries, alerts, run history in system tables. Generated from metadata, deployed with Asset Bundles. | Pursue: the main orchestrator | Everything after the landing folder runs in Databricks, so the orchestrator should too; no second tool to run or pay for. |
| Amazon EventBridge (Scheduler and rules) | Schedules and events on the AWS side: an S3 file arrived, a DMS task finished, a nightly start time. | Pursue: for AWS-side triggers only | Small, serverless, cheap; it starts a mover or calls the Databricks Jobs API. |
| AWS Step Functions | A visual state machine over AWS services: start an AppFlow flow, wait, run a Lambda, then call a Databricks job. | Use only if a mover needs several AWS steps | Good glue for AWS services; avoid putting Databricks steps inside it. |
| Amazon MWAA (managed Airflow) | Airflow as a service, for many systems beyond Databricks. | Only if the company already runs Airflow | A strong tool, but a second orchestrator to operate; Lakeflow Jobs covers our needs. |
| AWS Glue Workflows | Orchestrates Glue jobs and crawlers. | Drop | We do not use Glue jobs for transformation. |
Is there an Azure Data Factory on AWS? Not as one tool; its parts are spread across services. We use them only as movers. The framework itself is Python, so it is tested, reviewed and portable, instead of hundreds of hand-built visual pipelines.
| Azure Data Factory part | On AWS | On Databricks |
|---|---|---|
| Copy activity (databases, CDC) | AWS DMS | Lakeflow Connect |
| Copy activity (SaaS, SAP OData) | Amazon AppFlow | Lakeflow Connect |
| Files and SFTP | AWS Transfer Family, DataSync | Auto Loader |
| Data flows (visual ETL) | Glue Studio | Notebooks, Declarative Pipelines |
| Pipelines and triggers | Step Functions, MWAA (Airflow), Glue Workflows | Lakeflow Jobs |
| Lookup + ForEach over a control table | (build it) | Our framework reading dataset_view |
Data quality and profiling
Rules written once, checked at every hop
Two different jobs. Data quality rules are checks we write down: "every reading belongs to a known curing press", "scrap quantity is never negative". Profiling measures what the data actually looks like (how many blanks, how many different values, the smallest and largest) so we notice when it changes, and so we can find rules nobody thought to write. Both are horizontal assets: switched on for every dataset from the metadata, not coded per pipeline.
Which engine checks the rules
Rules are stored as data in the metadata store, and the engine sits behind our quality port (a fixed interface), so the engine can change without touching a single rule.
| Engine | Used for | Verdict | Why |
|---|---|---|---|
| DQX (Databricks Labs) | Every PySpark step: the lander, the gold window-recompute jobs, and any silver table not built as a Declarative Pipeline | Pursue | Rules as data (YAML or a table), row-level and table-level checks, splits good and bad rows, has a profiler that suggests rules. Open source, so no lock-in. Labs project: check current status for support terms. |
| Lakeflow expectations | Inside Declarative Pipelines (silver and gold tables built that way) | Pursue | The default for silver and gold built as Declarative Pipelines. Native to the pipeline, results land in the pipeline event log, no extra code. We generate them from the same rules, so a rule is written once. |
| Plain Spark SQL | Fallback: any rule the two above cannot express, such as a reconcile across two tables | Pursue | A predicate in SQL: rows where it is true are the bad rows. This is how the framework's rules already work today; DQX is not yet wired into the framework. |
| Great Expectations, Soda | Not used | Drop | Good tools, but a second rule language and a second results store. DQX plus expectations already cover the need inside Databricks. |
| AWS Glue Data Quality | Only if a team already uses it on Glue jobs | Drop | Runs outside Databricks; results would not join our run id without extra work. |
What is checked at each hop
Each layer checks what it promises, and nothing more. Bronze keeps data as it arrived, so the real cleaning checks start at silver.
| Hop | What is checked | What happens to bad data |
|---|---|---|
| Landing to bronze | File checks only: readable, not empty, row count against the source's count. New or unexpected columns are kept (rescued data) | Stop on an unreadable or empty file; bronze keeps everything else as it arrived, so it can be replayed |
| Bronze to silver | First the contract check (promised columns and types), then row rules: not null, ranges, allowed values, keys unique, references exist (press, plant, material) | Bad rows go to the silver table's quarantine table; good rows carry on. Stop only if the bad share passes a limit |
| Silver to gold | Table rules: totals reconcile with silver, share of 'unknown' members under a limit, and every expected press-hour present | Stop if totals do not reconcile: a wrong gold number is worse than a late one. A missing press-hour does not stop the load: it is written as an incomplete row, which OEE leaves out of both the top and the bottom of every part, so the number is on time, not skewed, and says how complete it is |
| Gold to semantic | Metric checks: OEE between 0 and 100%, scrap rate against last month's band, completeness of the period shown | Warn, and flag the metric on the dashboard; every dashboard and Genie answer shows how complete and how fresh the data is |
A rule file, for one table
Stewards write rules as YAML next to the dataset's settings. The same file switches profiling on.
# config/dq/silver_mes_press_reading.yaml (one file per dataset and layer)
dataset: mes.press_reading
layer: silver
rules:
- name: press_known
check: press_id IN (SELECT press_id FROM ${silver}.mes.press) # ${silver} = dev_silver, preprod_silver or prod_silver
severity: error
action: quarantine
- name: cure_temp_in_range
check: cure_temp_c BETWEEN 120 AND 220 # example range; the plant engineer owns it
severity: error
action: quarantine
- name: not_in_future
check: reading_ts <= current_timestamp() + INTERVAL 5 MINUTES
severity: warning
action: warn
- name: bad_share_limit # table-level: stop if too much is bad
check: quarantined_pct <= 2
severity: error
action: stop
profile:
enabled: true
when: daily
scope: new_rows
sample_pct: 10
columns: all
suggest_rules: trueRules and profiling settings for curing-press readings in silver. Ranges are examples; the plant's process engineer owns the real ones.
Where the rules and results live
In the same metadata database, split by the three schemas: rules and settings in config (changed only through Git), results in runtime. These tables already exist in our framework; on AWS and Databricks they move to Lakebase (Databricks managed PostgreSQL).
| Table | Holds | Written by |
|---|---|---|
| config.dq_rule | One row per rule: dataset, layer, the check, when it runs, severity, action (warn, quarantine, stop), who wrote it, whether it was suggested by the profiler | The loader, from YAML in Git |
| runtime.dq_result | One row per rule per run: passed or failed, failing row count, action taken, time taken | Pipelines |
| runtime.dq_violation | Keys of a sample of failing rows (capped, 100 by default), for the steward to look at | Pipelines |
| runtime.dq_profile | One row per column per profiled run: null share, distinct count, min, max, mean, top values, and the change from the last profile | Pipelines |
| <table>_quarantine (Delta) | The failing rows themselves, next to each silver table, with the run id and the rule id | Pipelines; stewards review weekly |
Profiling: three tools, each where it pays
Profiling every table on every run costs compute for little gain. So the default profiler runs on new data, the managed monitor runs only on the tables the business watches most, and personal-data detection runs on the catalogs.
| Profiler | Where | What it does |
|---|---|---|
| DQX profiler | Every profiled dataset | Measures columns and proposes candidate rules. Runs inside our step, so its output carries our run id. |
| Lakehouse Monitoring (data profiling) | Tier-1 gold tables only (OEE, scrap, warranty claims) | Managed profiles and drift over time, with its own dashboard. Costs compute on every refresh, so not everywhere. Name and features: check current status. |
| Unity Catalog data classification | Every catalog except restricted | Finds columns that look like personal data (names, phone numbers, emails of dealers and fleet drivers) and suggests PII tags. Check current status. |
Profiling is set per dataset with six settings in dataset_config:
| Setting | Values | Meaning |
|---|---|---|
| profile.enabled | true | Turn profiling on or off for this dataset |
| profile.when | every_run, daily, weekly, on_change | How often; daily is the default for silver, every run for tier-1 gold |
| profile.scope | new_rows, full_table | Profile only what this run loaded, or the whole table |
| profile.sample_pct | 10 | Share of rows to read; large sensor tables use a sample |
| profile.columns | all, or a list | Which columns; recipe columns are always left out |
| profile.suggest_rules | true | Let the profiler propose new rules (they are never switched on by themselves) |
From profile to rule: a person always decides
Later, an agent: it may read the profiles and quarantine tables, explain what changed in plain words and draft the rule change. It still arrives as a pull request for the steward.
Pipeline observability
The ops dashboard and alerts
Observability means that when a number looks wrong or a dashboard is late, anyone can see within a minute what ran, what failed, why, and what it cost, without opening a notebook. It is built once as a horizontal asset and switched on for every pipeline from the metadata.
Where the signals come from
| Signal source | What it tells us | How it reaches the ops layer |
|---|---|---|
| Our runtime schema (Lakebase) | Every run, step, error, quality result and watermark, with the run id; Lakebase's own health (load, connections, storage) comes from Databricks' Lakebase monitoring (check current status for what system tables expose) | Copied into prod_ops every 15 minutes (see below) |
| Databricks system tables | Job and task runs, billing (DBUs and cost), lineage, audit of who read what | Already in Unity Catalog; joined on the job run id |
| Lakehouse Monitoring / data profiling | Profiles and drift of key tables: nulls, distinct counts, distributions | Check current status and name; behind our profiling port |
| Amazon CloudWatch | Health of the AWS movers: DMS lag, AppFlow failures, SiteWise ingestion, Kinesis delay | CloudWatch alarms for the AWS side; a metric stream lands in prod_ops every 5 minutes |
| Data contracts and freshness targets | What each data product promises: by when, how complete, owner | From dataset_config; the dashboard checks against them |
How each signal lands
Everything lands in one catalog, prod_ops, so one SQL query can join it. The rule: read in place where Databricks already keeps it, copy only what lives outside.
| Signal | How it reaches prod_ops | Delay | Note |
|---|---|---|---|
| Our runtime schema (Lakebase) | Lakebase is registered in Unity Catalog as a catalog, so a small job reads it directly every 15 minutes and MERGEs new rows into prod_ops | 15 minutes | Dashboards never query Lakebase directly, so a busy dashboard can never slow the pipelines that write to it |
| Databricks system tables | Read in place; nothing to copy | Minutes to an hour, table by table (check current status) | Kept by Databricks for a limited time; ops gold keeps 13 months of the summaries we need |
| Amazon CloudWatch metrics | A metric stream sends DMS, AppFlow, Kinesis and SiteWise metrics through Firehose to S3; Auto Loader reads them into prod_ops | About 5 minutes | Only the metrics we name are streamed, so the cost stays small |
| CloudWatch alarms | Alarm state changes go through EventBridge to the same S3 path | Seconds to minutes | So AWS-side alarms join the same incident list as Databricks alerts |
| Pipeline event logs (Declarative Pipelines) | Read from each pipeline's event log: expectation results, flow progress | Per update | Expectation results merge into the same quality fact as DQX results |
| Spark driver and executor logs | Not loaded. Kept in S3 for 30 days for debugging | - | Our framework writes structured events to the runtime tables; free-text logs are for engineers only |
How they join: four keys
Signals from three systems only help if they line up. Our framework stamps the same keys everywhere, so a failed run can be followed to its cost, its quality results and the AWS mover that fed it.
The ops tables
A small star schema of its own, in prod_ops, built by an ordinary pipeline with the same tests as any other.
| Table | One row per | Holds |
|---|---|---|
| ops.dim_dataset | One row per dataset | Owner, tier, freshness target, volume band, budget, mover id; from the config schema |
| ops.fct_run | One row per dataset run | Our run_log joined to the Databricks job run and its cost: status, timings, rows, cause of failure, cost, config fingerprint |
| ops.fct_step | One row per step of a run | Land, bronze, silver, gold: timings and rows in and out |
| ops.fct_quality | One row per rule per run | DQX results and pipeline expectation results together |
| ops.fct_freshness | One row per data product per day | Promised time, actual time, met or missed, minutes late |
| ops.fct_mover_health | One row per mover per 5 minutes | DMS lag, AppFlow failures, stream delay, from CloudWatch; Lakebase load from Databricks |
| ops.fct_incident | One row per incident | Opened, owner, severity, cause class, acknowledged, resolved, linked runs and alerts, agent diagnosis |
| ops.fct_access | One row per read of sensitive data | From audit system tables and CloudTrail; feeds the quarterly access review |
The ops dashboard
One AI/BI dashboard on the ops gold tables, for everyone from the on-call engineer to the plant's data product owner. Numbers in the sketch are made up.
Start with what Databricks already gives. The first five panels come straight from Databricks system tables (read-only tables in Unity Catalog that record jobs, billing, compute, queries, lineage and audit), with no code of ours. Databricks also ships ready-made usage and job-monitoring dashboards to import and adapt (check current status). We add only what system tables cannot know: the business meaning (which data product, which plant, which target), quality results, row counts, watermarks and the cause of a failure. The job run id stored in our run_log is the join key.
| Panel | What it shows | Built from | Who looks |
|---|---|---|---|
| Runs today | Succeeded, failed, running, skipped; drill to the failing task | System tables: job and task run timelines | Ops / on-call |
| Run time | Duration against the usual 95th percentile; slow runs before they become late runs | System tables: job run timeline | Platform engineers |
| Cost | Cost per job, warehouse and cluster against budget | System tables: billing usage × list prices (CostSherlock adds cost per data product) | Platform lead, finance |
| Compute and queries | Busy or idle clusters, slow SQL queries, who ran what | System tables: compute, query history, audit | Platform engineers |
| Lineage | What feeds each table and what breaks if it fails | System tables: table and column lineage | Data engineers, stewards |
| Failures by cause | Source down, schema changed, bad data, our bug; trend over 7 and 30 days | Mixed: failure state from system tables, the cause class from our error_log | Platform lead |
| Freshness against target | Each data product green, amber or red against its promised time | Ours: targets in dataset_config, judged against each input's bookmark (input_watermark), not the last run | Data product owners, business |
| Quality | Failing rules, rows parked in quarantine, by dataset and plant | Ours: dq_result (plus pipeline expectation events) | Data quality lead, stewards |
| Volumes | Rows in and out against the usual range; sudden drops are the early warning | Ours: run_log | Ops |
| Watermark lag | How far behind each incremental source is | Ours: watermark | Ops |
| AWS movers | DMS replication lag, AppFlow flow failures, sensor stream delay | Amazon CloudWatch | Ops |
Targets first, then alerts
An alert only makes sense against a promise. Each data product states its targets in the metadata; the dashboard shows them and the alerts fire when they are missed.
| Target (SLO) | Measures | Example |
|---|---|---|
| Freshness | Data product ready by its promised time | OEE per plant by 06:30 for the morning meeting, on 99% of days |
| Success | Runs that succeed without a person stepping in | 98% of scheduled runs per week |
| Quality | Rows passing every rule | 99.5% of rows in silver; zero failures on stop rules |
| Recovery from a failed run | Time from failure to data back on time | Under 2 hours for tier-1 data products (losing a region is covered by the recovery plan, not this target) |
Alerts
Databricks SQL alerts and job notifications cover what happens inside Databricks; CloudWatch alarms with Amazon SNS cover the AWS movers. The metadata database runs on Lakebase, so its health is watched from Databricks, not CloudWatch. All of them post to the same Teams or Slack channel and, for tier-1 data products, to the on-call pager.
| Alert | Fires when | Severity | Goes to |
|---|---|---|---|
| Pipeline failed after retries | run_log status failed and retries used up | High | Teams/Slack + email; pager at night for tier-1 data products |
| Data product late | Freshness past the promised time in dataset_config | High for tier 1, Medium otherwise | Owner of the data product, on-call |
| Quality rule set to stop failed | dq_result failed on a rule whose action is stop | High | Data quality lead and steward; the load is held |
| Volume out of range | Rows written outside the usual band for that dataset and weekday; for streams, per 15-minute window | Medium | On-call; often the first sign of a source problem |
| A press went silent | A press in the press master sent no reading for 30 minutes during a planned shift | High (OEE is tier 1, so it pages) | Plant maintenance and on-call; one silent press is too small to move a volume band, so it gets its own check |
| Schema changed at source | Contract check finds a new, missing or retyped column | Medium | The source owner named in the contract; silver is protected by the contract |
| Run much slower than usual | Duration above 2 × the usual p95 | Low | Platform engineers, next working day |
| Cost over budget | Daily or monthly cost per pipeline, per SQL warehouse or for the AWS side above its budget | Medium | Platform lead and finance (CostSherlock) |
| Cost jump | Yesterday's total cost (Databricks and AWS) more than 1.5 × its 7-day average | High | Platform lead; the runbook can pause tier-3 schedules and shrink warehouses within the hour |
| Source mover unhealthy | CloudWatch alarm: DMS lag, AppFlow failure, Kinesis delay | High | On-call through Amazon SNS |
| Metadata store unhealthy | Lakebase load, connections or storage past their limits, from Databricks' Lakebase monitoring (check current status); recorder timeouts in error_log | High | Platform on-call; the heartbeat and job alerts do not need Lakebase |
From alert to incident
Later, self-healing: for known causes (a source briefly down, a late file) an agent may propose the fix and prove it on a pre-prod clone; a person approves before anything touches production. See agents.
Testing and data contracts
Catch it early, prove it before release
Most data platforms are tested by looking at the dashboard after go-live. We test in six gates, from the developer's laptop to production, and each gate catches what the earlier ones cannot. All of it is part of the platform from day one, not a project for later.
| Layer | What it proves | Built with | Runs |
|---|---|---|---|
| Shift-left checks | Problems found by reading the code, before it runs: Python and PySpark anti-patterns, SQL injection, secrets, collect() on big data, cartesian joins, missing keys in MERGE, costly job settings, metric view mistakes | Our shift-left bundle (built: 131 checks specified; caught 88 of 101 planted problems on a sample project with no false alarms) on Ruff, sqlfluff and dbt-bouncer | Laptop and pull request |
| Unit tests | Each transformation as a pure function, tested table-driven: a table of input rows and expected output rows per case (SCD2 new, changed, unchanged, deleted; duplicates; late rows; time zones) | pytest + local Spark and Delta (no cluster); the shift-left test kit for SCD2 and idempotency | Laptop and pull request |
| Contract tests | Each contract is valid, and a change does not break consumers (a removed column, a narrowed type, a renamed key) | ODCS schema validation + a breaking-change diff against the last released version | Pull request |
| Integration tests | One slice end to end on real Databricks: lander, silver with SCD2, gold, metric view; run twice to prove re-runs change nothing; reconciliation counts and sums agree | Databricks Asset Bundles deploy to the dev workspace on a fresh schema per run; small fixed sample files in S3 | After merge, in dev |
| DataTwin | A pretend plant and business that produce data for 30 simulated days with faults planted on purpose; checks that the platform caught each one, held bad data back, and every number is accounted for | DataTwin (designed; the DataTwin Studio and Problem Lab are built and run on real Spark and Delta) | Nightly in dev, and before each release in pre-prod |
| Pre-production run | The release on real data: shallow clones of prod, a parallel run against prod with every difference explained, profiles compared, Genie benchmarks per role | The pre-prod workspace and the CI service principal | Every release |
| Production checks | Contract check at silver, quality rules with warn / quarantine / stop, freshness and volume alerts | Lakeflow expectations or DQX behind our quality port; rules from config.dq_rule | Every run |
The full test strategy, cloud-neutral, is on Test architecture for a data platform.
How this uses what we are building. The shift-left bundle and its test kit run at the first two gates; the Problem Lab scenarios (duplicates, late data, failure mid-job, schema drift, SCD2 pitfalls) become the unit and integration cases; DataTwin is the release gate. Because pipelines are generated from metadata, a new source gets these tests without anyone writing them again.
DataTwin: a pretend plant with faults planted on purpose
Real data rarely shows the bad days on schedule. DataTwin produces data the way the plants and the business would, then plants the faults below. The release passes only if the platform catches every one and the dashboard still matches the pretend world's true numbers.
| Fault planted in the pretend plant | The platform passes if |
|---|---|
| SAP sends the same delivery twice | No duplicate in silver; sales not doubled in gold |
| MES adds a column and renames another | Contract check stops it at silver; alert to the MES owner; gold untouched |
| A press sensor goes silent for 3 hours | The silent-press alert fires and pages; OEE is on time, marked incomplete, and neither too high nor too low |
| The warranty file arrives a day late | Freshness alert; the late rows land in the right day when they arrive |
| A dealer's master record changes twice in a day | SCD2 keeps both versions with correct dates |
| Daylight saving at the plant in Europe | No lost or doubled hour in hourly scrap |
| A job dies halfway | Watermark not moved; the re-run loads the same window once |
Data contracts: do we need them?
A data contract is a written agreement between the team that produces data and the teams that use it: which columns, what they mean, how fresh, how good, and who owns it. It is written as a YAML file in Git using the open standard ODCS (Open Data Contract Standard, version 3), so it is both readable and checkable by machine. The short answer: yes, at the two boundaries, and no, not everywhere.
| Where | Contract? | Why |
|---|---|---|
| At sources we do not control (SAP, MES, quality LIMS, dealer DMS) | Yes | Their teams change systems on their own timetable; a contract turns a silent break into a clear alert with an owner. |
| At data products other teams use (OEE, scrap rate, warranty claims) | Yes | Consumers need to know the columns, the meaning, how fresh and who to call; it is also what Genie and dashboards rely on. |
| Between our own internal layers (bronze to silver to gold) | No | Our tests and quality rules already cover them; a contract there is paperwork. |
| A full contract platform or marketplace tool | Not now | YAML in Git plus the checks we already run is enough to start; revisit when there are dozens of producing teams. |
| Part of the contract | Example | Enforced by |
|---|---|---|
| Schema | Columns, types, keys, which may be empty | Contract check at silver; breaking-change diff in pull requests |
| Quality | Rules such as 'scrap quantity is never negative' | The same rules run as quality checks |
| Freshness and volume | Ready by 06:00, between 40k and 60k rows a day | Freshness and volume alerts |
| Meaning and owner | What a column means; the producing and consuming owners | Unity Catalog comments and tags; alert routing |
| Privacy | Which columns hold personal data or recipe IP | Unity Catalog tags, column masks and restricted catalogs |
| Version | Major version for breaking changes, with notice to consumers | Pull request review; consumers named in the contract approve |
Nothing new has to be built to enforce them: the contract check at silver, the quality rules, the freshness and volume alerts and Unity Catalog tags already do it. The contract simply makes the promise explicit and gives every alert an owner. The open-source datacontract command-line tool lints the ODCS files and tests them in the pull-request gate.
The right medallion
One medallion, two speeds
The medallion is the row of layers in the middle: bronze (raw), silver (clean) and gold (shaped for the business). Streaming does not need a second architecture. Nightly batches and sensor events flow through the same tables, with the same contracts and the same checks; only the speed differs. Streams run trigger-once on a schedule, every 15 minutes, never continuously.
What each layer promises
| Layer | Promise | Rules | Databricks features |
|---|---|---|---|
| Landing (S3) | Files exactly as received. | Never changed. Short retention. Encrypted with your KMS key. | S3 lifecycle rules; Unity Catalog volumes or external locations |
| Bronze | Everything that arrived, replayable. | Append-only. Adds ingest metadata (file, time, batch id). Keeps the source schema. Safe to run twice. | Auto Loader, streaming tables, rescued-data column |
| Silver | One clean, trusted row per real thing. | Typed and deduplicated. Keys conformed (material, plant, press, batch, dealer). History kept as SCD Type 2. Bad rows go to quarantine, never silently dropped. Personal data and recipe columns tagged. | AUTO CDC (formerly APPLY CHANGES) for SCD1/SCD2; expectations; column tags |
| Gold | Shaped for a business question. | Reads only silver. Star schemas and data products with a stated grain. Rebuilt safely by partition. | Materialized views; liquid clustering; predictive optimization |
| Semantic | Each metric defined once. | No tables of its own. Definitions, joins and names that every tool uses. | Unity Catalog Metric Views (check current status) |
Bronze to silver
Thirteen generic stages run for every dataset, set per column in the metadata: contract check, select and rename, hygiene, parse and type, time, standardise values, personal data, row filter, quality rules, duplicates, keys, history (SCD type 1 or 2 by kind of data) and provenance. No business logic and no free-form SQL in this step. The stages, the column settings, the schema change rules and the history choices are on The ETL, end to end.
Silver keeps one table per source entity, with conformed keys. Joining sources together (SAP orders with MES production, for example) is a separate, named step, not hidden inside a silver table. Each silver table has one quarantine table, which its steward reviews weekly.
Common mistakes, and the fix
More detail, with one record followed through every layer: Medallion architecture, drawn right.
Gold, the semantic layer and AI/BI
Shape it once, define each metric once, then ask in plain words
Ask three tools for "scrap rate" and you can get three numbers. That happens when every report shapes the data and writes the formula itself. The fix has two halves: gold shapes the data once, as star schemas the business recognises, and the semantic layer defines each metric once, as a metric view in Unity Catalog. Every front end, from dashboards to Genie, reads those definitions. Then AI on top is safe: the model only chooses which metric, which breakdown and which filter.
The choices
| Choice | Our pick | Verdict | Why |
|---|---|---|---|
| How gold is modelled | Star schemas (Kimball): facts at a stated grain, shared dimensions | Pursue | Business people read them, BI tools and Genie understand them, and shared dimensions make numbers line up across subjects. Data Vault adds a layer nobody here needs; one big flat table repeats data and breaks on history. |
| How gold is built | SQL files in Git, run as materialized views in Declarative Pipelines | Pursue | Incremental refresh and lineage come with the platform. One model per file, so moving to dbt Core later is a copy, not a rewrite. |
| dbt | Not now | Drop for now | Adopt dbt Core if the team already knows it, if gold passes about 50 models across teams, or if gold must run outside Databricks. Never dbt Cloud or a second metric layer. |
| Where metrics are defined | Unity Catalog metric views, one per business process | Pursue | Defined once, governed by the catalog, read by dashboards, Genie and our Conversation UI alike. A separate semantic tool (Cube, AtScale) only if many tools outside Databricks need the same metrics. |
| Asking in plain words | Genie for analysts and managers; our Conversation UI inside staff apps and wherever a wrong number is costly | Pursue | Genie is already in the platform. The Conversation UI never lets the model write SQL. |
| Existing BI tools | Read through a serverless SQL warehouse, from metric views (check current status) or gold | Pursue | Nobody has to throw away reports on day one. |
| Compute for questions | Separate serverless SQL warehouses for dashboards, Genie and ad hoc work | Pursue | Each workload gets the right size, and CostSherlock can show what each one costs. |
Gold: star schemas
A star schema puts the numbers (facts) in the middle and the words people filter by (dimensions) around it. Each fact table states its grain: what one row means. Getting the grain right is most of the work.
The bus matrix shows which facts share which dimensions. A shared ("conformed") dimension is what lets you put scrap next to warranty claims for the same material and get a sensible answer.
| Fact | Grain | Date | Plant | Press | Shift | Material | Dealer | Customer / fleet | Defect |
|---|---|---|---|---|---|---|---|---|---|
| Production | one row per press per hour, split by batch type (production or trial) | ✓ | ✓ | ✓ | ✓ | ✓ | |||
| Downtime | one row per stop | ✓ | ✓ | ✓ | ✓ | ||||
| Scrap | one row per scrapped tyre batch | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ||
| Quality tests | one row per test result | ✓ | ✓ | ✓ | ✓ | ✓ | |||
| SAP order lines | one row per order line | ✓ | ✓ | ✓ | ✓ | ✓ | |||
| Dealer sell-out | one row per dealer per material per day | ✓ | ✓ | ✓ | ✓ | ||||
| Warranty claims | one row per claim | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Where gold lives
| Schema | Holds | Owner |
|---|---|---|
| prod_gold.shared | The shared dimensions: date (with the April-to-March financial year), plant, press, shift, material, dealer, customer, defect code | Platform team |
| prod_gold.manufacturing | Production, downtime, scrap | Manufacturing data owner |
| prod_gold.quality | Quality tests, warranty claims | Quality data owner |
| prod_gold.sales | SAP order lines, dealer sell-out, dealer stock | Sales data owner |
| prod_semantic.* | Metric views only, one schema per business area; no tables | Each metric's owner |
Facts built from several inputs
Some facts need more than one source: production needs MES counts, the press master, the material master and the shift calendar, which arrive at different times. Five rules keep them right.
The semantic layer: metric views
A metric view is a definition, not a table: which gold fact, which joins, which dimensions, and the formula for each measure. It is stored in Unity Catalog with an owner, so it is governed like a table.
# prod_semantic.manufacturing.press_performance (syntax: check current status)
version: 0.1
source: prod_gold.manufacturing.fct_production # one row per press per hour
joins:
- name: press
source: prod_gold.shared.dim_press
on: source.press_key = press.press_key
- name: material # SCD2: the version valid at cure time
source: prod_gold.shared.dim_material
on: source.material_key = material.material_key
dimensions:
- name: Plant
expr: press.plant_name
- name: Batch type
expr: source.batch_type
- name: Press
expr: press.press_code
- name: Compound family
expr: material.compound_family
- name: Production date (plant time)
expr: source.plant_local_date
measures:
- name: Tyres cured
expr: SUM(source.tyres_cured)
- name: Scrap rate # production batches only
expr: SUM(source.tyres_scrapped) FILTER (WHERE source.batch_type = 'production') / NULLIF(SUM(source.tyres_cured) FILTER (WHERE source.batch_type = 'production'), 0)
- name: Completeness # share of expected press-hours that arrived
expr: SUM(source.hours_received) / NULLIF(SUM(source.hours_expected), 0)
# OEE parts count complete press-hours only (top and bottom alike), so a silent press
# makes OEE incomplete, never wrongly high or low; Completeness shows how much is missing.
- name: Availability
expr: SUM(source.run_minutes) FILTER (WHERE source.is_complete) / NULLIF(SUM(source.planned_minutes) FILTER (WHERE source.is_complete), 0)
- name: Performance
expr: SUM(source.ideal_cycle_seconds * source.tyres_cured) FILTER (WHERE source.is_complete) / NULLIF(SUM(source.run_minutes * 60) FILTER (WHERE source.is_complete), 0)
- name: Quality
expr: SUM(source.tyres_good) FILTER (WHERE source.is_complete) / NULLIF(SUM(source.tyres_cured) FILTER (WHERE source.is_complete), 0)
- name: OEE # good tyres x ideal cycle time over planned time
expr: SUM(source.ideal_cycle_seconds * source.tyres_good) FILTER (WHERE source.is_complete) / NULLIF(SUM(source.planned_minutes * 60) FILTER (WHERE source.is_complete), 0)One metric view for press performance. Formulas are sums divided by sums, computed for whatever slice is asked.
The starter metrics
| Metric | Definition | Broken down by | Owner |
|---|---|---|---|
| OEE | Availability × performance × quality, each computed from sums (run time over planned time; ideal cycle time × count over run time; good tyres over all tyres) | Press, shift, plant, day | Plant operations head |
| Scrap rate | Scrapped tyres over tyres cured, production batches only. Both counts come from the production fact; the breakdown by defect comes from the scrap fact, whose totals reconcile with it | Press, material, day; defect through the scrap fact | Quality head |
| First-pass yield | Tyres passing every test the first time, over tyres tested | Plant, material, day | Quality head |
| Downtime hours | Sum of stop durations, by cause | Press, cause, shift | Maintenance head |
| On-time in-full (OTIF) | Order lines delivered complete and on the promised date, over all order lines | Plant, customer, week | Supply chain head |
| Sell-out units | Tyres sold by dealers to end customers | Dealer, material, region, week | Sales head |
| Warranty claim rate | Claims per 10,000 tyres, counted against the month the tyres were made, not the month of the claim | Plant, material, production month | Quality head |
Six rules for the semantic layer
To confirm when building: whether reading a metric view also needs a grant on the gold tables under it, and whether outside BI tools can query metric views directly (check current status).
From a question to an answer
Genie: one space per audience
| Space | For | Reads | Benchmark |
|---|---|---|---|
| Plant performance | Plant managers, shift leads | Press performance, downtime and scrap metric views | 40 questions |
| Quality and warranty | Quality engineers | Scrap, quality tests and warranty metric views | 40 questions |
| Sales and dealers | Sales and aftermarket managers | Order lines, sell-out and dealer stock metric views | 30 questions |
| Platform ops | The data team | Ops gold from the observability section | 20 questions |
Genie or our Conversation UI?
| Need | Genie | Our Conversation UI | Pick | Verdict |
|---|---|---|---|---|
| Analysts and managers asking new questions on curated tables | Strong fit: fast to set up, curated by instructions and example questions | Possible, but more work | Genie | Pursue Genie first; it is in the platform already. |
| Answers inside a staff app or chat channel, next to an agent | Not designed for embedding in our own apps (check current status) | Built for this: an API and a chat window | Conversation UI | Pursue for embedded use. |
| A hard rule that the model never writes SQL | Genie writes SQL, guided by trusted assets and instructions | The model only picks metrics; the engine writes SQL | Conversation UI | Pursue where a wrong number is costly (finance, plant KPIs, warranty). |
| Proving answers are right before go-live | Benchmark questions built in | Our eval harness with the same benchmark set | Both | Use the same benchmark questions for both. |
| Consumers or the public | Not the right tool | Only behind the agent harness, with its guardrails | Neither directly | Drop for now; route through the agent harness. |
Security, privacy and governance
Who may see what, and how we prove it
Three kinds of data need protecting here: personal data (dealers, fleet drivers, consumers who register a warranty, operators on shift logs), commercial data (prices, discounts, margins) and the recipes that make the tyres. The design protects each one where the data lives, in the catalog, so the protection holds in every tool: a dashboard, a notebook, Genie or an agent.
Four levels of sensitivity
| Level | Examples | Where it lives | Who sees it | Protection |
|---|---|---|---|---|
| Internal | Production counts, OEE, scrap, downtime, stock | prod_gold, prod_semantic | Every employee group that needs it; plant staff see their own plant's rows | Row filter by plant |
| Confidential | Prices, dealer discounts, margins, cost per tyre | prod_gold.sales, columns tagged confidential | Finance and sales groups | Column mask for everyone else |
| Personal | Dealer contacts, fleet drivers, consumers who register a warranty, operator ids on shift logs, names inside free-text warranty notes | Tokenised at the lander; real values only in the token vault (prod_pii_vault, its own bucket and key) | Almost nobody needs the real value; the warranty team for a live claim | Tokens everywhere, mask on the vault, purpose and retention per category |
| Restricted | Compound recipes, cure recipes, process settings | prod_restricted only, its own bucket and key | The named recipe keepers | No Genie space, no copies except the encrypted recovery copy, every read alerted |
Roles and what they get
People read; only service principals write to prod. That one rule removes most accidents.
| Role (a group) | Gets | Limits |
|---|---|---|
| Platform engineer | Read and write in dev; read prod_ops | Deploys only through CI/CD |
| Data engineer | Read and write in dev; read the results schemas in pre-prod | Never writes to prod by hand; prod clones in pre-prod are readable only by CI |
| Analyst, data scientist | Read prod_gold and prod_semantic | Personal data only as tokens; confidential columns masked |
| Plant manager, shift lead | Read prod_semantic; dashboards and the Plant performance Genie space | Own plant's rows only |
| Quality engineer | Read quality and manufacturing gold and metric views | Personal data only as tokens |
| Sales and finance | Read sales gold and metric views, confidential columns visible | Own region's rows for regional managers |
| Recipe keeper | Read prod_restricted | Named people, reviewed monthly |
| Privacy case handler | Read and erase in prod_pii_vault | The privacy team and, for a live claim, the warranty team; every lookup logged |
| Auditor | Read audit system tables and the audit schema | No business data |
| Service principals | One per pipeline area; write only their own target schemas in prod | The only identities that write to prod |
How access is enforced
Personal data: India's DPDP Act and the EU's GDPR
Both laws ask the same things in practice: keep personal data only for a stated purpose, only as long as needed, protect it, and erase it when asked. The hard part in a lakehouse is erasure, because bronze keeps everything for replay. The design solves it at the door.
When someone asks to be erased
Tokenising uses the framework's existing pii_policy table (actions: pass through, hash, mask, tokenise, redact, per environment). The vault is its own catalog, prod_pii_vault, with its own bucket and key, readable only by privacy case handlers; recipe keepers cannot read it, and the privacy team cannot read recipes. The fingerprint key and the vault key live in AWS Secrets Manager and KMS.
Recipes
Recipes and process settings sit in prod_restricted, in their own bucket with their own key. Gold carries only the compound family, never the recipe. Only the named recipe keepers can read it, every read raises a notice to the recipe owner (the head of compound development), access is reviewed every month, and revoking the key cuts all access at once. It is never added to Genie or to an agent.
AI on governed data
Who decides
| Role | Who | Decides |
|---|---|---|
| Data owner | A business head (plant operations, quality, sales) | Decides who may see the data and signs off metric definitions |
| Data steward | A named person per source | Writes quality rules, reviews quarantine weekly, approves suggested rules |
| Custodian | The platform team | Runs the platform, applies the owner's decisions, never decides access |
| Privacy lead | From the legal or privacy team | Owns purposes, retention, erasure requests and breach response |
Proof, not promises. Databricks audit system tables, AWS CloudTrail and KMS key-use logs are joined in prod_ops into an access report: who read what, from which tool. The data owners review it every quarter, and it is the evidence pack for an auditor.
CI/CD and environments
Every change takes the same path to production
Code, configuration and definitions all change through Git and reach production the same way, through the six test gates. Nobody clicks a change into prod, and every release can be undone.
Two repositories
| Repository | Holds | Owner | Checks |
|---|---|---|---|
| platform-foundation | Terraform for accounts, network, storage, keys, workspaces, catalogs | Platform team | Terraform plan on every pull request; apply after approval |
| data-platform | The framework code, all YAML config (sources, datasets, rules, contracts, grants), gold SQL, metric views, dashboards, Genie space definitions, Asset Bundles | Shared, with CODEOWNERS per folder | The release path below |
Who approves which folder
GitHub CODEOWNERS sends each change to the people who own it: a quality rule to its steward, a metric to its owner.
| Folder | Approved by |
|---|---|
| framework/ | Platform team |
| config/sources/, config/datasets/ | Data engineers for that source |
| config/dq/ | The source's data steward |
| contracts/ | The producing team and the platform team |
| gold/, semantic/ | The business area's data owner and an analytics engineer |
| config/grants/, config/access/ | The data owner, plus the privacy lead for personal data |
What deploys what
| What | Deployed by | From |
|---|---|---|
| Accounts, network, buckets, keys, workspaces, catalogs | Terraform | platform-foundation |
| Jobs, pipelines, dashboards | Databricks Asset Bundles | data-platform |
| Sources, datasets, rules, profiling settings | Our config loader, into the metadata database | data-platform |
| Metric views, row filters, masks, schema and table grants | SQL run by the deploy, generated from YAML | data-platform |
| Genie spaces | Their definitions in Git, applied through the Genie API (check current status) | data-platform |
The path, step by step
| Step | What runs | Where | Takes |
|---|---|---|---|
| Pull request | Lint, shift-left checks, unit tests, contract checks, config dry-run against the key registry, bundle validate, Terraform plan | Laptop and CI runner, no Databricks needed | minutes |
| Merge to main | Deploy to dev; per-hop integration tests on fixed fixtures | Dev workspace, fresh schema per run | about 20 minutes |
| Release tag | Deploy to pre-prod; end-to-end on synthetic data, DataTwin faults, real-data run on shallow clones, parallel run against prod, Genie benchmarks | Pre-prod workspace | an hour or two |
| Approval | The release report goes to the data owner of each changed area; one approval per area | GitHub environment approval | same day |
| Prod deploy | Deployed by the prod service principal; risky changes to one plant first for one full day, promoted to all only if the smoke checks and reconciliations stay green | Prod workspace | minutes, then one day |
| After deploy | Smoke checks: first runs green, row counts within band, metrics within band | Prod | first run |
Three environments
| Environment | Data | Who | Size and cost guard |
|---|---|---|---|
| Dev | Synthetic data and small samples; no personal data | Data and platform engineers | Small serverless; jobs stop after 2 hours; budget alert |
| Pre-prod | Shallow clones of prod, personal data already tokens, recipes and the vault excluded | CI writes and reads prod; engineers see results | Like prod, for an hour or two per release; DataTwin's nightly run happens in dev |
| Prod | Real data | Read through groups; only service principals write; personal access tokens off for people | Sized for the plants' shifts (see sizing); budget policies per data product |
Five rules
Agents
Agents propose, people decide
Agents help most with work that is slow and repetitive for people: reading logs after a failure, drafting a quality rule, writing YAML for a new source. In this design an agent never changes production. It reads, explains, and proposes a change as a pull request, which then takes the same release path as any other change. The agents run on the agent harness on AWS.
Seven agents, one job each
| Agent | Starts when | Does | May |
|---|---|---|---|
| Triage | An alert opens an incident | Collects the evidence: the failed run, the error, recent config and code changes, upstream lineage, CloudWatch for the mover. Writes a diagnosis and the likely cause class into the incident. | Read only |
| Fix proposer | A diagnosed incident with a known cause | Starts only after a person has confirmed the triage diagnosis. Its clone runs go through the CI pipeline, so it never reads prod itself. Drafts the fix as a pull request (a widened type, a new rule limit, a watermark reset plan), and proves it by re-running the failed window on a shallow clone in pre-prod. | Pull request + pre-prod clone |
| Rule suggester | A new profile or a full quarantine table | Explains what changed in plain words and drafts quality rule changes for the steward. | Pull request |
| Source onboarder | A sample file or a database table from a new source | Drafts the source and dataset YAML, the contract and the first quality rules. | Pull request |
| Genie curator | Thumbs-down answers and failed benchmark questions | Drafts better instructions, example questions and new benchmark questions for the space owner. | Pull request |
| Cost finder | CostSherlock findings | Drafts the change behind each finding (a smaller warehouse, a job schedule, a clustering key) with the expected saving. | Pull request |
| Documenter | A table or metric without a description | Drafts table and column comments and glossary entries from the code, the contract and lineage. | Pull request |
How far each may go
| Level | Means | Who |
|---|---|---|
| Read | ops tables, metadata, lineage, code, profiles | All agents |
| Propose | Open a pull request; comment on an incident | All agents |
| Act on a test copy | Deploy a branch and re-run a window on a pre-prod shallow clone | Fix proposer only |
| Act in prod | Change data, config or code in prod | None. Prod changes only through the normal release path, approved by a person |
Five rules
Running it
Sizing, recovery, retention and the people
A design is only finished when it says how big it is, how it comes back after a disaster, how long it keeps things, and who runs it. The numbers below are our assumptions for a typical tyre maker; each one is a setting, so the design holds if they change.
Sizing and targets
| What | We assume | What it means for the design |
|---|---|---|
| Plants | 7: five in India, two in Europe; three shifts | Plant is the first key for clustering, row filters and canary releases |
| Machines streaming | About 2,000 curing presses and mixers, 20 signals each, one reading every 5 seconds: about 8,000 readings a second, 700 million a day | Streaming tables read Kinesis directly, trigger-once every 15 minutes; about 10 GB a day once compressed in Delta |
| Business data | SAP: about 200,000 order and delivery lines a day; 3,000 dealers sending daily sell-out; about 2,000 warranty claims a day | Batch every 30 minutes in plant hours, full nightly |
| Size after five years | About 25 TB, mostly sensor history | Bronze sensor data older than 3 years moves to cheaper storage |
| Freshness | A reading reaches silver within one hour (the manifest's target; in practice one 15-minute run plus the run itself); press-hour gold within 15 minutes after the hour; OEE ready by 06:30 | These are the tier-1 targets the alerts watch |
| People asking | About 400 dashboard users, 50 queries at once at the peak (06:00 to 07:30, morning meetings) | Dashboards answer in under 5 seconds for 95% of views; Genie in under 30 seconds |
| What the platform is not | Machine alarms and safety interlocks stay on the plant floor (SiteWise Edge, the machines' own controls) | The platform is for analysis and decisions, not for stopping a press |
Streaming sensor data
Recovery: what comes back, how fast, and what can be lost
Two words first. The recovery point (RPO) is how much recent data can be lost; the recovery time (RTO) is how long until people can work again. Different layers get different targets, because they cost different amounts to copy.
| Layer | How it is copied | Most data lost (RPO) | How it comes back |
|---|---|---|---|
| Landing | S3 cross-region replication, same expiry rules (personal-data files still deleted after a day) | 15 minutes | Lets the lander replay the last minutes that had not reached bronze |
| Bronze | S3 cross-region replication to Hyderabad (tokens only) | 15 minutes | Registered at the last version whose files have all arrived (replication does not keep order), then silver and gold catch up from bronze by run id |
| Silver, gold, recipes, token vault | Incremental Delta deep clone every 4 hours | 4 hours | Registered on the clones; erasures since the last clone are replayed |
| Metadata database (Lakebase) | Lakebase's own point-in-time restore in Mumbai; for tier-1, a read replica in Hyderabad (check current status) | Seconds with the replica; without it, configuration is reloaded from Git and bookmarks are rebuilt from bronze | Promote the replica, or reload from Git; watermarks and run log carry on |
| Unity Catalog | Not copied: Hyderabad has its own catalog, built from Git | Nothing lost | Terraform builds catalogs and catalog grants; the data-platform deploy adds schema grants, row filters, masks, metric views and Genie spaces |
| Streams | Checkpoints are not copied; a Kinesis stream in Hyderabad is built in advance | Gap filled from the edge buffer | The edges switch to the Hyderabad stream; streams start from the last bronze row |
| Secrets | Secrets Manager replicates every secret to Hyderabad | Nothing lost | Movers and the lander find their credentials in place |
| Code and config | In GitHub | Nothing lost | Deployed as in any release |
How long each layer is kept
Assumed periods; your legal and finance teams confirm them. Retention jobs run nightly and log what they delete.
| Layer | Kept for |
|---|---|
| Landing | 30 days; files with personal data 1 day |
| Bronze | 3 years in standard storage, then an archive class Spark can still read (S3 Glacier Instant Retrieval) until year 8, then deleted. Tokens only |
| Silver | 8 years for anything that supports the accounts (as Indian company law asks for books of account); 5 years for the rest |
| Gold | 10 years, for trends. Rows older than silver's retention are kept only as summaries without tokens, so an erasure never needs to rebuild them |
| Quarantine tables | 90 days after the steward's decision |
| Runtime and ops tables | 13 months live, then summarised |
| AI question log | 1 year |
| Audit bucket | 7 years, locked |
| Token vault | Per purpose: for warranty, the warranty period plus 3 years; then the entry is deleted and the tokens point to nobody |
Master data: who owns the facts everything joins on
OEE is only right if each press's ideal cycle time and each plant's shift calendar are right. These are small tables with big effects, so each has a named owner.
| Master data | Comes from | Owner |
|---|---|---|
| Plant: country, time zone, financial calendar | SAP plant master, plus a small YAML file for time zone | Plant operations head |
| Press: line, ideal cycle time per material | MES press master; ideal cycle times from process engineering (YAML if not in MES) | Process engineering head |
| Shift calendar and planned stops | MES, or a YAML file per plant | Each plant manager |
| Material and compound family | SAP material master | SAP master data team |
| Key mappings (SAP and MES material, dealer ids across systems) | Mapping tables in silver, kept from SAP or YAML | The steward of that domain; an unmapped key becomes 'unknown' and raises an alert |
| Defect codes | The quality system | Quality head |
Adding a source: the checklist
"A new source is a YAML entry" is true for the pipeline. The checklist is everything around it; the source onboarder agent drafts most of it as one pull request.
Backfills
Secrets
The team
| Who | Size | Does |
|---|---|---|
| Platform team | About 6: a lead, three data engineers, a cloud engineer, an analytics engineer | Runs the platform and the horizontal assets; on-call in a weekly rota of four |
| Security and privacy | Part-time, from your existing teams | Reviews access quarterly, owns erasure and breach response |
| Data owners | Business heads (plant operations, quality, sales, finance) | Decide access, sign off metrics and releases in their area |
| Stewards | One per source, about two hours a week | Rules, quarantine reviews, suggested rules |
| Data forum | Owners, stewards and the platform lead, monthly | Priorities, metric changes, the incident and cost trends. This is the 'centre of excellence' the testing page refers to |
Migration, sharing and ML
What you already have, who outside needs data, and models
Moving from Glue, Redshift and QuickSight
| What you have | How it moves | Done when |
|---|---|---|
| Glue jobs | Rebuilt as metadata-driven pipelines, one subject at a time, each run in parallel with the old job for two weeks | Counts and sums reconcile for two weeks; then the Glue job is switched off |
| Redshift | Kept for existing reports; new gold on Databricks; Redshift reads our tables as Iceberg (UniForm) or through Delta Sharing while reports move | The last report has moved; then the cluster is shut down |
| Glue Catalog and Lake Formation | Unity Catalog is the one catalog for everything new; old Glue tables are read through catalog federation during the move (check current status) | No new grants are made in Lake Formation from day one |
| QuickSight | Existing reports repointed to the SQL warehouse; new ones on AI/BI | Each report reads gold or metric views |
| Athena | Kept for small one-off queries on S3 | Nothing to migrate |
Sharing with partners
Every share is a data product: it has a contract, an owner and a filter to the partner's own rows, and its use is audited and reviewed every quarter.
| Partner | What they get | How | Protection |
|---|---|---|---|
| Car makers (OEMs) | Quality and traceability of the tyres fitted to their cars, by batch and DOT code | Delta Sharing (an open protocol, readable without Databricks) of a gold view, filtered to that car maker | No personal data; contract and owner per share |
| Dealers | Their own sell-out, stock and claim status | The existing dealer portal reads an API over gold, or a Databricks App | Each dealer sees only their own rows |
| Suppliers | Quality results for the lots they delivered | Delta Sharing | Filtered to the supplier |
| Auditors and regulators | Evidence packs: access reviews, erasure log, release reports | Exported on request | Approved by the data owner |
Machine learning
Models use the same platform, catalog and release path as everything else; nothing is copied out to a separate ML system.
| Part | How |
|---|---|
| Features | Feature tables in gold (prod_gold.features), built and tested like any gold table |
| Models | Tracked with MLflow and registered in Unity Catalog, with the same owners and grants |
| Serving | Batch scoring for forecasts; Model Serving where an app needs an answer in seconds |
| First use | Weekly demand forecast per material and region; later, predicting press breakdowns from sensor history |
| Checks | Each model is evaluated against a held-out period before release and watched for drift after (check current status for inference tables) |
22 concerns, bound
What Databricks gives, what AWS gives, what we add
For each of the 22 cross-cutting concerns: reuse what Databricks does well, use AWS where the job sits outside Databricks, and fill only the gap. Every concern is switched on and configured from the metadata store, and every product sits behind a port (a fixed interface), so a product can be swapped without touching the pipelines.
Our main value is joining the dots: one run id links the quality results, lineage, run log, cost and audit for every number, even where every piece underneath is Databricks or AWS.
| Concern | Databricks covers | Databricks feature | AWS piece | The gap, and who fills it |
|---|---|---|---|---|
| X1 Metadata & control | partial | Asset Bundles, Unity Catalog tags, Lakebase (Databricks managed PostgreSQL) for the metadata store | - | YAML in Git → loader → metadata store → generated jobs. Ours (Common Ingestion Framework). |
| X2 Orchestration | full | Lakeflow Jobs: schedules, file-arrival triggers, retries | EventBridge for events from outside | Jobs generated from metadata, not clicked together. Ours. |
| X3 Contracts & schema | partial | Delta schema enforcement and evolution; Auto Loader rescued data | Glue Schema Registry for streams, if used | Contracts in the open ODCS standard, checked in CI and at run time. Ours. |
| X4 Data quality | partial | Declarative Pipelines expectations; DQX (Databricks Labs) | - | Rules from metadata, quarantine, one results table. Ours, with DQX behind a port. |
| X5 Profiling | partial | DQX profiler; Lakehouse Monitoring on tier-1 gold; Unity Catalog classification for PII (check current status) | - | Profiling settings per dataset, profile compared with the last one, suggested rules as pull requests, all tied to the run id. Ours. |
| X6 Reconciliation | none | - | - | Counts and sums from source to gold, per run. Ours. |
| X7 Lineage | full | Unity Catalog table and column lineage | - | Lineage before Databricks (movers, source systems) from the run log and mover ids; joined to the run id. Ours. |
| X8 Logs, metrics, traces | partial | System tables for jobs, queries, pipeline event logs | CloudWatch for DMS, Kinesis, MSK | One run log with one run id across both. Ours. |
| X9 Freshness & SLOs | partial | Job duration alerts; freshness checks in monitoring (check current status) | CloudWatch alarms | A freshness promise per table, declared in metadata and alerted. Ours. |
| X10 Cost & FinOps | partial | Billing system tables, budget policies, tags | Cost and Usage Report, Cost Explorer, Budgets | Cost per data product and per run, with findings. CostSherlock. |
| X11 Security & access | full | Unity Catalog grants, row filters, column masks, ABAC (check current status) | IAM roles, KMS, VPC, PrivateLink | Grants declared in metadata and applied by the pipeline. Ours. |
| X12 Privacy & PII | partial | Column masks, tags, data classification (check current status) | Macie scans landing buckets for personal data that slipped through; Comprehend finds names in free text such as warranty notes | Tokenise at the door, the token vault, erasure, suppression and retention jobs. Ours. |
| X13 Governance & semantics | partial | Unity Catalog owners, comments, tags; Metric Views | - | Glossary and metric ownership workflow. Ours (OpenMetadata optional). |
| X14 Audit & run log | partial | Audit system tables | CloudTrail | Run log with a fingerprint of the config each run used. Ours. |
| X15 Replay & backfill | partial | Delta time travel, MERGE, full refresh of pipelines | S3 versioning | Backfill plans and watermark resets from metadata, never doubling a number. Ours. |
| X16 Environments & CI/CD | full | Asset Bundles, Git folders, Terraform provider | GitHub Actions with short-lived OIDC sign-in; separate accounts per environment | Templates only. Ours. |
| X17 Testing | none | - | - | Unit, contract and data tests (pytest), shift-left checks, DataTwin worlds. Ours, open source. |
| X18 Resilience & DR | partial | Time travel, deep clone, job retries | S3 replication and deep clone to Hyderabad (no AWS Backup on personal data) | A written, tested restore drill. Ours. |
| X19 Performance | full | Photon, liquid clustering, predictive optimization, serverless | - | Checks in CI and cost findings. Shift-left bundle + CostSherlock. |
| X20 Self-healing | none | - | - | Detect → fix proved on a pre-prod clone → pull request → verify → roll back. Ours. |
| X21 Agent assistance | partial | Databricks Assistant for code | Bedrock, if preferred for models | Agents that draft manifest entries, rules and docs as pull requests. Ours. |
| X22 Simulation | none | - | - | A pretend plant with planted faults (a press goes silent, a batch sent twice). DataTwin, ours. |
Coverage is our judgement for a typical setup. Product names and preview status change often; items marked "check current status" should be confirmed before the deck goes out.
AWS native or Databricks
Which product for which job
You do not have to replace what already works. This is our recommendation per job, with the reason.
| Job | AWS native | Databricks | Verdict |
|---|---|---|---|
| Plant-floor data collection | AWS IoT SiteWise, IoT Core, Greengrass | Auto Loader / streaming tables read what SiteWise lands | Keep SiteWise at the plant edge: it talks to the machines. Databricks takes the data from S3 or Kinesis for analytics. |
| Ingest from SAP | AppFlow or a partner extractor | SAP Business Data Cloud connector (check current status) | Use an SAP-certified route (Datasphere or a certified extractor) behind one port; never read SAP tables directly. |
| Ingest from databases (CDC) | AWS DMS | Lakeflow Connect | DMS for v1, into S3 landing, so every source passes the lander's tokenising and run log. Lakeflow Connect writes straight into tables; revisit per source later. |
| Batch and streaming pipelines | Glue jobs, EMR | Lakeflow Declarative Pipelines and Jobs | Pursue Databricks: one engine for batch and streaming, built-in quality rules and lineage. |
| Warehouse for BI | Redshift | Databricks SQL warehouse | If Redshift is heavily used, keep it and let it read our tables as Iceberg (UniForm) or through Delta Sharing (check current status); new work on Databricks SQL. See the migration plan. |
| Ad hoc queries on S3 | Athena | SQL warehouse | Keep Athena for small one-off queries; it can read the tables as Iceberg (UniForm). |
| Catalog and access | Glue Catalog, Lake Formation | Unity Catalog | Pursue Unity Catalog as the one catalog: lineage, masks and audit in one place. Avoid running two access models. |
| ML and AI | SageMaker, Bedrock | Mosaic AI: Vector Search, Model Serving | Use Databricks for ML on lakehouse data; call Bedrock models from Databricks where the company already uses Bedrock. |
| Dashboards | QuickSight (now part of Amazon Quick Suite, check current status) | AI/BI dashboards and Genie | Keep existing QuickSight reports; build new ones on AI/BI over the same metric views. |
| Cost view | Cost Explorer, CUR | Billing system tables | Use both; CostSherlock joins them per data product. |
How we would start
One thin slice first
Crawl · 4 to 6 weeks
One slice end to end in your AWS account: curing-press sensor data and MES production records through silver to one gold fact, one metric view (OEE or scrap rate), one Genie space for plant managers with benchmark questions.
You see: the whole path working, with quality, lineage and cost on from day one.
Walk · next quarter
Three to five more sources (SAP sales and stock, quality lab, warranty claims, dealer sell-out). The horizontal assets on by default. CostSherlock reporting cost per data product.
You see: each new source added as a YAML entry, not a project.
Run · after that
A small platform team owns the assets; domain teams build data products on them. The Conversation UI in staff apps. Agents propose fixes as pull requests.
You see: the tenth data product faster and safer than the first.
Open decisions
What must be settled before or during the first slice
Most items marked "check current status" on this page are product details that do not change the design. These ten do: each has a fallback, an owner and a date.
| Open question | Why it matters | Who decides | By when |
|---|---|---|---|
| Can serverless compute reach the plant network privately? | Decides whether the puller runs serverless or on classic compute (the metadata store, on Lakebase, needs no network setup) | Platform lead | Before the crawl starts |
| Do row filters hold through metric views, and can outside BI tools query them? | The whole Genie and dashboard security rests on it | Platform lead and analytics engineer, by test | Crawl, week 2 |
| Are tag-based access policies (ABAC) available in Mumbai? | If not, the deploy generates a mask per table from the same tags | Platform lead | Crawl, week 1 |
| Which route out of SAP? | Licence and support terms decide it | SAP team and procurement | Before the walk phase |
| What is the legal basis for EU personal data in India? | Until decided, EU sources load without personal columns | Legal team | Before the first EU source |
| Does Databricks run in Hyderabad (ap-south-2)? | The recovery plan builds the workspace there. If not, the data copies stay in Hyderabad and compute is rebuilt in the nearest supported region, with legal sign-off for data leaving India | Platform lead and legal team | Before go-live |
| Which personal-data detector reads German free text? | Until decided, EU warranty notes are held out of bronze | Privacy lead and platform lead | Before the first EU source |
| Is Lakebase available in Mumbai, with a read replica in Hyderabad? | Sets where the metadata store runs and its recovery point: seconds with the replica, or a reload from Git and bronze without it | Platform lead | Before go-live |
| What are the DPDP Rules timelines for breach notice? | Fills in the breach runbook | Legal team | Before go-live |
| Can Genie be embedded and driven by API? | Decides how much of the Conversation UI we build | Product owner | Walk phase |
How we tested this design
Twelve situations, walked through before anything was built
We tested the design on paper first. Two reviewers who had not written it worked from the page alone. One walked twelve real situations through it step by step; the other checked every section against every other and against 25 topics a complete design must cover. They found 3 situations that failed outright, about 30 places where two sections disagreed, and several topics that were missing (sizing, recovery, migration, sharing). Every finding was fixed in the page above. A third reviewer then re-tested the three failures and looked for new contradictions; that round found 27 more, mostly in recovery and privacy details, and those are fixed too.
| Situation | First test | After fixes | What changed |
|---|---|---|---|
| A new EU plant's MES must reach OEE within a sprint | Passes with gaps | Passes | EU network path, master data owners, onboarding checklist, EU legal basis |
| SAP renames a column without warning | Passes with gaps | Passes | Contract check fixed at silver; replay from landing, never re-extract from SAP; freshness shown on every metric |
| A press goes silent for 3 hours on the night shift | Fails | Passes (round 2 fixed how OEE treats incomplete hours) | Gold writes incomplete rows instead of stopping; a completeness measure; a silent-press alert; paging by tier |
| A warranty file arrives two days late | Passes with gaps | Passes | Restatements logged and shown; freshness alerts grouped by date; warranty tier and target stated |
| A consumer in Germany asks to be erased | Fails | Passes once two open decisions are settled: the legal basis and a German-capable detector | Vault in its own catalog; landing purged in a day; recovery copies erased; suppression list; free text scanned; no restore across an erasure |
| Scrap rate is redefined to exclude trial batches | Passes with gaps | Passes | One source for scrap rate; batch type in gold; definition changes dated; owner updates benchmarks |
| A release is wrong for one plant | Passes with gaps | Passes | Recompute that plant only; canary for a full day; business banner; platform lead decides |
| The Mumbai region is down for a day | Fails | Passes once Databricks in Hyderabad is confirmed | Recovery per layer with targets; security deployed before readers; metadata database copied; network ready; who declares |
| A plant manager asks Genie for all plants | Passes with gaps | Passes, once the metric-view test is green | Row security as a release test; 'for your plant' wording; AI log restricted |
| An engineer joins, a recipe keeper leaves | Passes with gaps | Passes | Owners checked against the identity provider nightly; deputies; personal access tokens off; pre-prod sees results only |
| The bill doubles overnight | Passes with gaps | Passes | Cost-jump alert on the whole account; runbook to pause and shrink within the hour; cost as a cause |
| An agent's diagnosis is wrong | Passes with gaps | Passes | A person confirms the diagnosis; proof on a pre-prod clone; full gates for limits and watermarks; hit rate tracked |
Four situations pass only once an open decision is settled: the EU legal basis, a German-capable detector, Databricks in Hyderabad, and the metric-view security test. All are in the open decisions above.
Questions
Questions leaders ask
Does our data leave our AWS account?
No. Classic compute runs in your account and your VPC; serverless compute runs in Databricks' serverless plane in the same region. All tables live in your S3 buckets, encrypted with KMS keys you own. The Databricks control plane manages the workspace; for serverless compute, check the current data-residency terms for your region.
Why Databricks if we already have Glue, EMR and Redshift?
One engine for batch and streaming, one catalog (Unity Catalog) for access, lineage and audit, and AI/BI with Genie on the same governed data. You do not have to switch everything at once: keep what works and put new work on the lakehouse.
What is a metric view?
A definition of a business metric (for example, scrap rate) stored once in Unity Catalog, with its formula, dimensions and filters. Dashboards, Genie and the Conversation UI all use it, so the number is the same everywhere.
What are horizontal assets?
Parts built once and reused by every pipeline and data product: ingestion, metadata, quality, cost, testing and the conversation layer. They are what makes the tenth data product much cheaper than the first.
How are compound recipes and personal data protected?
Recipes and formulation data sit in a restricted catalog with its own storage and encryption key, readable only by a named group. Personal data of dealers, fleet drivers and consumers is swapped for tokens as it enters the platform; the real values live only in a separate token vault, and erasing a person means deleting their vault entry, blocking them from being loaded again, and erasing the recovery copies too. Commercial columns such as prices are masked by Unity Catalog for everyone outside finance and sales, in every tool, not only in dashboards. All data stays in India (Mumbai, with recovery in Hyderabad); personal data from the EU travels only as the law allows (see the privacy section), and a separate EU platform is added only if your legal team requires it. Access is audited in system tables and CloudTrail.
Next
Where to go next
↑ The abstract architecture · Spec-driven development for data platforms · Medallion layers · Shift-left checks · DataTwin