Home › Learn › Data platform architecture › On AWS + Databricks

Data platform architecture · binding · AWS + Databricks

The data platform on AWS and Databricks

The same architecture as the abstract picture, with a product in every box, for a tyre manufacturer that already runs on AWS and wants Databricks. The design stays the same; only the names change.

↑ The abstract pictureThe ETL, end to endSpec-driven development for dataMedallion, drawn right

The one-minute version

Your account, your data, one catalog

Databricks runs inside your own AWS account. Data stays in your S3 buckets, in your VPC, encrypted with KMS keys you own. Databricks brings the engine, and Unity Catalog (its one catalog for tables, files and models) governs who sees what, tracks lineage and records every access.

The design does not depend on Databricks. It follows the same specs and the same 22 cross-cutting concerns as the abstract architecture. This page is the binding: the one place where product names appear. That is what keeps you free to change a product later.

Most of the value is in the shared parts. Ingestion, metadata, quality, cost, testing and the conversation layer are built once and reused, so each new data product (OEE, warranty claim rate, dealer sell-out, demand forecast) costs less than the one before.

The big picture

One platform, from plant and business systems to answers

Read it left to right along the middle row: data moves one way, from the plants' and the business's systems to dashboards, Genie and AI. Above it, the control plane decides what runs, and agents may only suggest. Below it, the horizontal assets serve every layer, on an AWS and Databricks foundation you own. Governance runs down the right side.

Agent planeself-healing · onboarding drafts · rule suggester · cost advisor · docs. Agents open pull requests against the manifest; they never touch production.Control planeYAML in Git → loader → metadata store · Lakeflow Jobs run the pipelines · Databricks Asset Bundles + CI/CD: dev → pre-prod → prodDelta tables on S3 · Lakeflow Declarative Pipelines · governed by Unity CatalogOpen: SourcesSourcesSAP S/4HANA (ERP)MESPlant historian,press + mixer sensorsQuality / LIMSWarranty claimsDealer DMS, CRMRubber prices, fleetsOpen: IngestionIngestionCommon IngestionFramework,metadata-drivenSAP: Datasphere /certified extractorDBs: DMS, LakeflowFiles: Auto LoaderSensors: SiteWise,Kinesis or MSKOpen: S3 landingS3 landingfiles as receivednever changedshort retentionKMS-encryptedin your bucketOpen: BronzeBronzeraw, append-only+ ingest metadatasource schemareplayablestreaming tablesOpen: SilverSilvertyped, dedupedconformed keysSCD2 history(AUTO CDC)expectationsbad rows parkedpersonal data taggedOpen: GoldGoldbusiness shapestar schemadata productsreads only silverliquid clusteringOpen: SemanticSemanticMetric Viewsin Unity Catalogeach metric once:OEE, scrap rate,first-pass yield,warranty rateno own tablesOpen: ExperiencesExperiencesAI/BI dashboardsGenie spacesConversation UI(Databricks Apps)SQL warehousefor BI toolsVector Search +Model ServingDelta SharingStreaming lane: press and mixer sensor data through the same layers, run trigger-once every 15 minutes. SiteWise, Kinesis or MSK → bronze streaming tables → silver → gold incremental. Same contracts, same checks.Horizontal assets: built once, reused by every pipeline and every data productCostSherlockCostSherlockcost per data product and runCommon Ingestion FrameworkCommon Ingestion Frameworka new source is a YAML entryMetadata storeMetadata storeone config every pipeline readsData quality frameworkData quality frameworkrules from metadata, behind a portSemantic model templateSemantic model templateeach metric defined onceConversation UIConversation UIpicks metrics, never writes SQLShift-left checksShift-left checkscatch problems before deployDataTwinDataTwina pretend plant, faults plantedObservability and run logObservability and run logone run id joins it allall 22 concerns on by defaultThe foundationAWS foundation: your accountCustomer-managed VPC · PrivateLink to DatabricksS3 holds all data · KMS keys you own · IAM roles · CloudTrailMumbai, with Hyderabad for recovery · EU only if legal requiresThe foundationDatabricks on AWS: runs in your accountWorkspace with serverless and classic compute · PhotonUnity Catalog: one catalog for every table, file and modelSystem tables: billing, audit, lineage, job runsGovernanceLineagetable and columnlevel, automaticAccessgrants, tags,ABAC (check status)Masks and filtersrows and columnshidden by rolePersonal datadealers, drivers:tagged, maskedRecipe IPcompound recipes ina restricted catalogAuditaudit system tables+ AWS CloudTrailResidencyIndia: Mumbai + Hyd.EU only if legal asksDPDP + GDPRIndia and EU rules:consent, erasureOwnersevery table hasa named owner
The data platform on AWS + Databricks. Click a layer or an asset to open its page.

Horizontal assets

Built once, used by every data product

A horizontal asset is something every pipeline needs, whatever the subject. Databricks gives the plumbing; these assets are the parts that make the platform trustworthy and cheap to grow. The status column is honest: some are built, some are designed, some are still to build.

AssetWhat every team getsBuilt onStatus
CostSherlockCost per data product, per pipeline run and per team, from Databricks billing system tables and the AWS Cost and Usage Report, with findings and fixes.Ours, on Databricks + AWSSeparate briefing
Common Ingestion FrameworkA new source is a YAML entry, not a new notebook. One lander from landing to bronze, with ingest metadata, replay and watermarks.Ours, on Databricks; movers behind a port (DMS, AppFlow, Transfer Family, Kinesis, our puller)Built; first end-to-end Databricks run in progress
Metadata storeOne place where every source, table, rule, owner and budget is declared. Pipelines read it; nobody configures by hand.Ours; Lakebase (Databricks managed PostgreSQL)Built in the framework (design published); the Lakebase deployment is to do
Data quality frameworkRules declared per table, run at every hop, bad rows parked in quarantine, all results in one table.Databricks expectations and DQX behind one port; rules from the metadata storePartly built: rules from metadata run in silver; 66 assurance rules written
Medallion templateBronze, silver and gold built the same way for every source, with the rules that keep each layer honest.Lakeflow Declarative PipelinesBuilt for the first slice
Semantic model templateA starter set of metric views (OEE, scrap rate, first-pass yield, warranty claim rate, sell-out) with owners and definitions.Unity Catalog Metric ViewsTo build
Conversation UIStaff ask questions in plain words and get answers that cite the metric used. The model picks metrics; it never writes SQL.Ours, on Databricks Apps, reading Metric ViewsDesigned (spec written)
Shift-left checks bundleChecks that run on the code before it deploys: PySpark, SQL and Databricks mistakes caught in CI, not in production.Open source, ours; runs in CIBuilt: catches 88 of 101 planted problems, no false alarms
DataTwinA pretend tyre plant and dealer network that sends data on its own clock and breaks things on purpose, so the platform proves it catches each fault.Ours, open sourceStudio built; the full tool designed
Observability and run logEvery run gets one id that joins its quality results, lineage, cost and audit, so any number can be traced to evidence.System tables + CloudWatch + our run logRun log built; joined evidence view to build

The foundation

Accounts, network, storage and keys, before any pipeline

Before a single pipeline runs, the platform needs somewhere safe to live: AWS accounts to hold it, a private network, buckets for the data with keys you own, Databricks workspaces to work in, and one catalog that decides who sees what. Think of it as the land, the roads and the locks. Done right once, nobody thinks about it again; done wrong, every later layer inherits the problem.

Your login systemEntra ID · groups syncedDatabricks account · Unity Catalog in Mumbaiusers and groups · system tables for audit, billing, lineagePrivateLinkAWS account: devVPC, 2 zones, no public IPscompute (classic, if needed)endpoints: S3, STS, KinesisWorkspace + Lakebase: devsees dev_* catalogs onlyS3: landing + a bucketper catalog, each withits own KMS keyPrivateLinkAWS account: pre-prodVPC, 2 zones, no public IPscompute (classic, if needed)endpoints: S3, STS, KinesisWorkspace + Lakebase: pre-prodsees preprod_* catalogs onlyS3: landing + a bucketper catalog, each withits own KMS keyPrivateLinkAWS account: prodVPC, 2 zones, no public IPscompute (classic, if needed)endpoints: S3, STS, KinesisWorkspace + Lakebase: prodsees prod_* catalogs onlyS3: landing + a bucketper catalog, each withits own KMS keypre-prod reads prod, read onlyPlants + SAPDirect Connectto every VPC througha Transit GatewayHyderabad (recovery)bronze to gold, recipes, vaultreplicated and clonedworkspace rebuiltfrom Terraform
Three environments, each in its own AWS account and private network, on one Unity Catalog. Nothing is reachable from the internet.

The nine choices

DecisionOur pickWhy
AWS accountsOne AWS account per environment: dev, pre-prod, prod. All three sit under your AWS Organization and send logs to your central security and log accounts.A mistake in dev cannot touch prod, permissions stay simple, and each environment's bill is separate.
RegionMumbai (ap-south-1) is home. Hyderabad (ap-south-2) holds the recovery copy.Both are in India, so recovery does not move data out of the country. The full recovery plan is in the running-it section.
EU plantsPlant and production data from EU plants comes to Mumbai too. Personal data about EU people (staff, dealers, consumers) moves only under the legal basis your legal team picks, and is tokenised on arrival.Press readings and scrap counts are not personal data. One platform costs half of two. A separate EU platform (Frankfurt) only if your legal team requires it.
Databricks workspacesThree workspaces (dev, pre-prod, prod) on one Unity Catalog. Each catalog is bound to its workspace, so prod tables are visible only in prod.Databricks allows one Unity Catalog per region, so the three share it; binding keeps the environments apart.
NetworkNo public IP addresses anywhere. Classic compute runs in your own VPC, across two availability zones; serverless compute runs in Databricks' serverless plane in the same region and reaches your data over private connections. Databricks is reached through PrivateLink, never the open internet.This is the first thing a manufacturer's security team asks. Two zones keep the platform up if one data centre fails.
Storage and keysOne S3 bucket and one KMS key per catalog. A landing bucket per environment. Recipes get their own bucket and key with the tightest access.Recipes are the crown jewels: with their own key, revoking all access is one action, and every use of the key is logged.
ComputeServerless by default for SQL, jobs and pipelines. Classic compute (clusters inside your VPC) only where a job must reach a private system that serverless cannot yet reach.Starts in seconds, nothing to patch or size. Serverless private reach on AWS: check current status.
Sign-in and identityPeople sign in through your existing identity provider; groups sync to Databricks automatically (SCIM). Jobs run as service principals, never as a person.When someone leaves, one change removes all their access, and no job breaks because its owner left.
How it is builtTerraform builds all of the above from Git (AWS and Databricks providers). Pipelines are deployed separately with Databricks Asset Bundles.The whole foundation can be rebuilt the same way in every environment, and in Hyderabad after a disaster.

What we assumed

This design is written for a typical Indian manufacturer already on AWS. Where your answer differs, the right column says what changes. None of them changes the layers above.

QuestionWe assumeIf it is different
Is there already an AWS landing zone (Control Tower or similar)?Yes, with central security and log-archive accountsIf not, we add a minimal one first: an Organization, CloudTrail to a log account, guardrails (SCPs). About a week.
Which identity provider?Microsoft Entra IDOkta or another SAML / OIDC provider works the same way; only the Terraform settings change.
How do plants reach AWS?Direct Connect from the head office, Indian plants on the company network (MPLS or SD-WAN); EU plants through a site-to-site VPN to the Mumbai Transit GatewayWith only VPN, sensor streaming is sized smaller and the plant historian is pulled in batches.
Where does SAP run?SAP S/4HANA on AWS (RISE) or in a data centre linked to AWSThe mover changes (see the ingestion table); the foundation does not.
Must EU data stay in the EU?Not for plant data; personal data under Standard Contractual ClausesIf yes for everything, add a fourth workspace in Frankfurt on its own Unity Catalog, and share gold tables back with Delta Sharing.

Which workspace sees which catalog

Catalogs are named by environment and layer (prod_silver, dev_gold). Binding each catalog to workspaces is what keeps a test in dev from ever writing to prod, even by mistake.

CatalogDev workspacePre-prod workspaceProd workspace
dev_* catalogsRead and writeNot visibleNot visible
preprod_* catalogsNot visibleCI writes; engineers read the results schemas only, not the clonesNot visible
prod_bronze, prod_silver, prod_gold, prod_semanticNot visibleRead only, by the CI service principal (for real-data tests and shallow clones); people in pre-prod see the results, not the prod rowsJobs write; people read through grants
prod_restrictedNot visibleNot visibleRecipe group only
prod_pii_vaultNot visibleNot visiblePrivacy case handlers only
prod_opsNot visibleRead onlyPipelines write; everyone on the platform reads, except the ai_log schema

Buckets and keys

Every bucket blocks public access, is encrypted with a KMS key you own, and is reached only through the VPC endpoint or by Databricks through an IAM role it is allowed to assume.

BucketHoldsKeyRules
landing-<env>Raw files from the movers, as they arrivedEnvironment keyFiles deleted 30 days after they reach bronze; versioning on, old versions expire with them
landing-pii-<env>Raw files from sources with personal dataIts own keyDeleted within a day of being loaded by an hourly purge; held files kept up to 7 days; old versions expire with them
<env>-pii-vaultThe token vault and the suppression listIts own key; only privacy case handlersEvery lookup logged
<env>-bronze, -silver, -gold, -semantic, -opsThe managed tables of each catalogOne key per catalogDelta handles history; old versions cleaned by VACUUM after 30 days
<env>-restrictedRecipes and compound formulationsIts own key; only the recipe group can use itAccess logged and reviewed monthly
audit-<env>Copies of audit and config change logsEnvironment keyObject Lock: cannot be changed or deleted for 7 years
dr-* (Hyderabad)Recovery copies: bronze, silver, gold, restricted and the vaultKeys in Hyderabad, same groupsSee the recovery plan; erasures are applied here too; restore tested twice a year

Built from code

The foundation is Terraform in Git, applied in five steps. The environments differ only in one settings file each, so dev, pre-prod and prod are the same platform at different sizes.

foundation/                     # Terraform, one folder per layer, applied in this order
  1-org/          accounts, guardrails (SCPs), CloudTrail to the log account
  2-network/      VPC per env, private subnets in 2 zones, endpoints (S3, STS, Kinesis), PrivateLink
  3-storage/      buckets and KMS keys per catalog, IAM roles Databricks may assume
  4-databricks/   workspaces, metastore, catalogs, workspace bindings, external locations
  5-identity/     service principals, catalog-level grants (groups come from your identity provider)
  envs/           dev.tfvars, preprod.tfvars, prod.tfvars, dr.tfvars   # the only differences

Folder layout of the foundation code. Pipelines are not here: they are deployed with Asset Bundles.

Recovery: if Mumbai is lost, the same Terraform with dr.tfvars builds the workspace in Hyderabad on the copied data, and the data-platform deploy puts back every grant, filter and mask before anyone can read. The full plan, with targets per layer, is in running it.

Metadata and ingestion on AWS

One metadata database, one ingestion framework

The Common Ingestion Framework is metadata-driven: adding a source is a few lines of YAML, not a new pipeline. It follows the Metadata DB Design for Data Engineering Platforms, already built in our framework. Full table definitions, the view and the reader code are on the metadata store page.

The tables

TableWhat it holdsWritten by
platform_configSettings for the whole platform: default storage paths, default quality action, budget limitsPlatform team, through YAML
source_masterOne row per source system: SAP S/4HANA, MES, historian, quality LIMS, warranty, dealer DMSThrough YAML
source_configKey-value settings per source, grouped (ingestion, quality, cost): connection secret reference, mover, scheduleThrough YAML
dataset_masterOne row per dataset (table) of a source, with its target schema and tableThrough YAML
dataset_configKey-value settings per dataset: load mode, keys, history (SCD) type, profiling settings, PII purpose and retention, owner and deputy, freshness target and tier. Quality rules have their own table, dq_ruleThrough YAML
dataset_viewThe one view every pipeline reads: platform, then source, then dataset settings merged into JSON, the most specific winningGenerated
run_logOne row per run: dataset, status, row counts, cost, and a fingerprint of the exact settings usedPipelines
watermarkHow far each incremental load has gotPipelines

How it all runs, step by step from a source to gold, with every job, trigger, id and log row: The ETL, end to end.

The framework also has a key registry (every allowed setting and its type), per-column settings, quality results and a PII policy table on the same database.

YAML in GitHub, and a SQL database

We need both, for different jobs. People write and approve settings as YAML in GitHub; a loader validates every key and writes them to the database, one database per environment (dev, pre-prod, prod) with the same schema. Pipelines only ever read the database.

YAML in GitHubSQL metadata database
What it holdsConfiguration: sources, datasets, load modes, keys, rules, owners, PII tags, budgetsThe same configuration loaded from Git, plus runtime state: watermarks, run log, quality results
Who writes itPeople, and agents, by pull requestThe loader writes configuration; pipelines write runtime state
Why we need itReview and approval, full history, easy rollback, the same file promoted dev → pre-prod → prodFast lookups, keys between tables, the merged view, many jobs writing at once
Do pipelines read it?NeverAlways

Two rules. Nobody edits configuration in the database by hand: the next load would overwrite it and Git would have no record. Runtime state (watermarks, run log) never goes into Git.

Which database on AWS

The metadata store needs keys between tables, fast small reads and writes, safe parallel writes from many jobs, and good JSON for the merged view. That is a relational database, and PostgreSQL fits the design as written.

DatabaseFitVerdictWhy
Databricks Lakebase (managed PostgreSQL inside Databricks)PostgreSQL run by Databricks, governed by Unity Catalog, reachable from serverless compute with no network setup, with point-in-time restore. Strong JSON (jsonb) for the merged view, as in any PostgreSQL. Generally available on AWS since early 2026 in selected regions.Pursue: chosen (4 Nov)Chosen over Aurora PostgreSQL to drop one AWS service and the network hop; stored procedures stay the only way in. Confirm Mumbai availability before go-live (check current status).
Amazon Aurora PostgreSQL Serverless v2Managed PostgreSQL with automatic failover, IAM sign-in, backups, and compute that scales down to almost nothing in dev. Strong JSON (jsonb) for the merged view.Considered, not chosenThe default until 4 Nov and a good fit for the design, but one more AWS service to run, connect privately and back up; Lakebase gives the same PostgreSQL inside the platform.
Amazon RDS for PostgreSQLThe same PostgreSQL on a small fixed-size server. Cheaper and simpler; slower failover and no automatic scaling.DropSame drawback as Aurora: one more AWS service to run, with no gain over Lakebase.
Aurora MySQL or RDS MySQLWorks, but JSON handling is weaker than PostgreSQL's and the view would need rewriting.DropNo gain over PostgreSQL for this design.
Amazon DynamoDBKey-value store with no joins, so no merged view and no keys between tables.DropWrong shape for relational metadata.
Delta tables in Unity Catalog onlyNo keys between tables, slow small writes, write conflicts when many jobs log at once.Drop as the storeKeep a copy of run_log in Delta for dashboards and cost reports.

Where it sits: on Lakebase, inside the Databricks platform, in the same region as the workspace (Mumbai; Frankfurt only if EU data must have its own platform). It was chosen over Aurora PostgreSQL on 4 Nov so that there is one AWS service fewer to run, connect and back up. Serverless and classic compute reach it with no extra network setup, and it is registered in Unity Catalog as a catalog. Stored procedures remain the only way in. The runtime tables are copied into Delta every 15 minutes so dashboards and cost reports can join them without touching the database.

Three schemas in one database

One PostgreSQL database per environment, split into three schemas so that permissions are simple: the loader may write only config and audit, pipelines may write only runtime, and dashboards read everything.

SchemaHoldsWho writesSize and retention
configWhat to do: platform_config, source_master, source_config, dataset_master, dataset_config, the key registry, quality rules, PII policy, and dataset_viewOnly the loader writes; pipelines readSmall; changes only through Git
runtimeWhat happened: run_log, step_log, error_log, dq_result, watermark, config_snapshotPipelines write; nobody elseGrows every run; monthly partitions, 13 months kept, then summarised into prod_ops
auditWho changed what: config_load_log (every loader run, Git commit, who approved, what changed)Only the loader appends; never updated or deletedSmall; kept 7 years, like the audit bucket

Logging: every run can be followed and replayed

Every run writes to the runtime schema. One run id ties the rows together, and it is also stored on the Databricks job run, so the logs join with Databricks system tables (cost, lineage, audit) and with the quality results.

TableOne row perMain columns
run_logOne row per run of one datasetrun_id, dataset_id, Databricks job and run ids, status, started and ended, rows read / written / rejected, config fingerprint, Git commit, estimated cost
step_logOne row per step inside a run: land, bronze, silver, goldrun_id, step, status, timings, row counts in and out
error_logOne row per failurerun_id, step, error class (source down, schema change, bad data, bug), message, retry number, whether it was retried
dq_resultOne row per quality rule checkedrun_id, rule id, passed or failed, failing row count, action taken (warn, quarantine, stop)
config_snapshotThe exact merged settings a run used, stored once per fingerprintfingerprint, merged JSON; any replay can prove what it ran with

The watermark: load only what is new, safely

A watermark is a bookmark: the last value loaded from a source (a timestamp, an id, a change position). The next run starts after it. The danger is moving the bookmark before the data is safely written, or never moving it after a crash. These rules prevent both.

1
Read Read the last good watermark (low) from runtime.watermark.
2
Fix the window Fix the upper end (high) at the start, e.g. the source's current maximum, minus a small safety gap. The run loads only rows above low and up to high, so it never chases a moving target.
3
Load safely Append to bronze, every row stamped with the run id. Bronze is never updated, so replay always works. Each batch is appended with a transaction id from the file log, so a batch committed before a crash is never appended twice.
4
Commit once Only after the data is committed: mark the run succeeded and move the watermark to high, in one database transaction.
5
Crash in between? The watermark did not move, so the next run reloads the same window, and silver's duplicate step makes that harmless.
6
Late rows A look-back overlap (say 2 hours, set in dataset_config) re-reads a little of the last window; silver drops the repeats.
7
Files and streams Auto Loader and streaming keep their own checkpoints; we record the checkpoint location, not a second watermark. CDC sources use the change position (LSN or SCN). SAP moves its own delta token when it hands over an extract, so after a failure we never ask SAP again: we replay from landing or bronze.
-- schema runtime: written only by pipelines
CREATE TABLE runtime.run_log (
  run_id          uuid PRIMARY KEY,
  dataset_id      uuid NOT NULL REFERENCES config.dataset_master(dataset_id),
  job_run_id      text,                    -- Databricks job run id, to join system tables
  status          text NOT NULL,           -- started, succeeded, failed, skipped, blocked
  started_at      timestamptz NOT NULL,
  ended_at        timestamptz,
  window_low      text,                    -- watermark this run started from
  window_high     text,                    -- watermark this run loaded up to
  rows_read       bigint, rows_written bigint, rows_rejected bigint,
  config_fp       text NOT NULL,           -- fingerprint of the merged settings
  git_commit      text,
  est_cost_usd    numeric(12,4)
);
CREATE TABLE runtime.watermark (
  dataset_id      uuid PRIMARY KEY REFERENCES config.dataset_master(dataset_id),
  wm_column       text NOT NULL,           -- e.g. updated_at, change sequence number
  wm_type         text NOT NULL,           -- timestamp, number, cdc_position, delta_token
  wm_value        text NOT NULL,           -- last value loaded successfully
  last_run_id     uuid REFERENCES runtime.run_log(run_id),
  updated_at      timestamptz NOT NULL DEFAULT now()
);

The two runtime tables most people ask about first. The full config schema is on the metadata store page.

Ingestion: movers bring data in, our lander loads it

The full list of sources, the five ingestion technologies and the landing contract are on The ETL, end to end. Where this page and that one differ on how the runtime works, that page is the one to follow.

Ingestion has two steps. A mover brings raw data from the source into an S3 landing folder; which mover is a setting in source_config, so it can change without code. Then one lander, our PySpark job on Databricks that reads exactly the new files listed in its file log, reads its settings from dataset_view, writes bronze with arrival details, records a run_log row and is safe to run twice. Lakeflow Jobs are generated from the metadata and deployed with Databricks Asset Bundles.

SourceMover (first hop to S3)Note
SAP S/4HANA (sales, stock, procurement, finance)Amazon AppFlow, SAP OData connector, on released CDS views and extractors with deltaNever SAP tables directly; Datasphere or Business Data Cloud only if already licensed
MES and quality / LIMS databasesAWS DMS with change captureChange capture keeps loads small and brings deletes
Small and reference tables, databases without CDCOur puller (the framework's built connectors) as a Databricks jobFull copy or watermark, straight to bronze through the same lander steps (no landing)
Plant historian and press / mixer sensorsAWS IoT SiteWise → Kinesis or S3The streaming lane: same layers, trigger-once every 15 minutes
Dealer management system, CRM (SaaS)Amazon AppFlow; our puller's REST connector where AppFlow has noneLakeflow Connect is not used in v1
Supplier files, raw-material pricesAWS Transfer Family (SFTP) → S3The lander reads the new files listed in its file log

Orchestration: which service runs what, and when

One rule: one orchestrator owns each pipeline. Lakeflow Jobs owns everything from the landing folder onwards; AWS services only start the movers on the AWS side and hand over. Two schedulers for the same pipeline is the most common cause of runs that overlap or never start.

ServiceWhat it does hereVerdictWhy
Lakeflow Jobs (Databricks)Runs every Databricks task: lander, silver, gold, quality, refresh of metric views. Schedules, file-arrival and table-update triggers, retries, alerts, run history in system tables. Generated from metadata, deployed with Asset Bundles.Pursue: the main orchestratorEverything after the landing folder runs in Databricks, so the orchestrator should too; no second tool to run or pay for.
Amazon EventBridge (Scheduler and rules)Schedules and events on the AWS side: an S3 file arrived, a DMS task finished, a nightly start time.Pursue: for AWS-side triggers onlySmall, serverless, cheap; it starts a mover or calls the Databricks Jobs API.
AWS Step FunctionsA visual state machine over AWS services: start an AppFlow flow, wait, run a Lambda, then call a Databricks job.Use only if a mover needs several AWS stepsGood glue for AWS services; avoid putting Databricks steps inside it.
Amazon MWAA (managed Airflow)Airflow as a service, for many systems beyond Databricks.Only if the company already runs AirflowA strong tool, but a second orchestrator to operate; Lakeflow Jobs covers our needs.
AWS Glue WorkflowsOrchestrates Glue jobs and crawlers.DropWe do not use Glue jobs for transformation.

Is there an Azure Data Factory on AWS? Not as one tool; its parts are spread across services. We use them only as movers. The framework itself is Python, so it is tested, reviewed and portable, instead of hundreds of hand-built visual pipelines.

Azure Data Factory partOn AWSOn Databricks
Copy activity (databases, CDC)AWS DMSLakeflow Connect
Copy activity (SaaS, SAP OData)Amazon AppFlowLakeflow Connect
Files and SFTPAWS Transfer Family, DataSyncAuto Loader
Data flows (visual ETL)Glue StudioNotebooks, Declarative Pipelines
Pipelines and triggersStep Functions, MWAA (Airflow), Glue WorkflowsLakeflow Jobs
Lookup + ForEach over a control table(build it)Our framework reading dataset_view

Data quality and profiling

Rules written once, checked at every hop

Two different jobs. Data quality rules are checks we write down: "every reading belongs to a known curing press", "scrap quantity is never negative". Profiling measures what the data actually looks like (how many blanks, how many different values, the smallest and largest) so we notice when it changes, and so we can find rules nobody thought to write. Both are horizontal assets: switched on for every dataset from the metadata, not coded per pipeline.

Rules YAML in Gitwritten by stewardsconfig.dq_ruleloaded and validatedbronzesilvergoldrules run at every hop (DQX, expectations, SQL)Passrow carries onWarnlogged, shownQuarantinerow parkedStopload held, owner toldruntime.dq_result+ run id, to the ops dashboardProfilerDQX, Lakehouse Monitoringruntime.dq_profilechange vs last profileSuggested rulespull requeststeward approves
Rules come from Git and run at every hop; every check ends in one of four actions and one result row. The profiler watches the data beside it and proposes new rules for a person to approve.

Which engine checks the rules

Rules are stored as data in the metadata store, and the engine sits behind our quality port (a fixed interface), so the engine can change without touching a single rule.

EngineUsed forVerdictWhy
DQX (Databricks Labs)Every PySpark step: the lander, the gold window-recompute jobs, and any silver table not built as a Declarative PipelinePursueRules as data (YAML or a table), row-level and table-level checks, splits good and bad rows, has a profiler that suggests rules. Open source, so no lock-in. Labs project: check current status for support terms.
Lakeflow expectationsInside Declarative Pipelines (silver and gold tables built that way)PursueThe default for silver and gold built as Declarative Pipelines. Native to the pipeline, results land in the pipeline event log, no extra code. We generate them from the same rules, so a rule is written once.
Plain Spark SQLFallback: any rule the two above cannot express, such as a reconcile across two tablesPursueA predicate in SQL: rows where it is true are the bad rows. This is how the framework's rules already work today; DQX is not yet wired into the framework.
Great Expectations, SodaNot usedDropGood tools, but a second rule language and a second results store. DQX plus expectations already cover the need inside Databricks.
AWS Glue Data QualityOnly if a team already uses it on Glue jobsDropRuns outside Databricks; results would not join our run id without extra work.

What is checked at each hop

Each layer checks what it promises, and nothing more. Bronze keeps data as it arrived, so the real cleaning checks start at silver.

HopWhat is checkedWhat happens to bad data
Landing to bronzeFile checks only: readable, not empty, row count against the source's count. New or unexpected columns are kept (rescued data)Stop on an unreadable or empty file; bronze keeps everything else as it arrived, so it can be replayed
Bronze to silverFirst the contract check (promised columns and types), then row rules: not null, ranges, allowed values, keys unique, references exist (press, plant, material)Bad rows go to the silver table's quarantine table; good rows carry on. Stop only if the bad share passes a limit
Silver to goldTable rules: totals reconcile with silver, share of 'unknown' members under a limit, and every expected press-hour presentStop if totals do not reconcile: a wrong gold number is worse than a late one. A missing press-hour does not stop the load: it is written as an incomplete row, which OEE leaves out of both the top and the bottom of every part, so the number is on time, not skewed, and says how complete it is
Gold to semanticMetric checks: OEE between 0 and 100%, scrap rate against last month's band, completeness of the period shownWarn, and flag the metric on the dashboard; every dashboard and Genie answer shows how complete and how fresh the data is

A rule file, for one table

Stewards write rules as YAML next to the dataset's settings. The same file switches profiling on.

# config/dq/silver_mes_press_reading.yaml  (one file per dataset and layer)
dataset: mes.press_reading
layer: silver
rules:
  - name: press_known
    check: press_id IN (SELECT press_id FROM ${silver}.mes.press)   # ${silver} = dev_silver, preprod_silver or prod_silver
    severity: error
    action: quarantine
  - name: cure_temp_in_range
    check: cure_temp_c BETWEEN 120 AND 220        # example range; the plant engineer owns it
    severity: error
    action: quarantine
  - name: not_in_future
    check: reading_ts <= current_timestamp() + INTERVAL 5 MINUTES
    severity: warning
    action: warn
  - name: bad_share_limit                         # table-level: stop if too much is bad
    check: quarantined_pct <= 2
    severity: error
    action: stop
profile:
  enabled: true
  when: daily
  scope: new_rows
  sample_pct: 10
  columns: all
  suggest_rules: true

Rules and profiling settings for curing-press readings in silver. Ranges are examples; the plant's process engineer owns the real ones.

Where the rules and results live

In the same metadata database, split by the three schemas: rules and settings in config (changed only through Git), results in runtime. These tables already exist in our framework; on AWS and Databricks they move to Lakebase (Databricks managed PostgreSQL).

TableHoldsWritten by
config.dq_ruleOne row per rule: dataset, layer, the check, when it runs, severity, action (warn, quarantine, stop), who wrote it, whether it was suggested by the profilerThe loader, from YAML in Git
runtime.dq_resultOne row per rule per run: passed or failed, failing row count, action taken, time takenPipelines
runtime.dq_violationKeys of a sample of failing rows (capped, 100 by default), for the steward to look atPipelines
runtime.dq_profileOne row per column per profiled run: null share, distinct count, min, max, mean, top values, and the change from the last profilePipelines
<table>_quarantine (Delta)The failing rows themselves, next to each silver table, with the run id and the rule idPipelines; stewards review weekly

Profiling: three tools, each where it pays

Profiling every table on every run costs compute for little gain. So the default profiler runs on new data, the managed monitor runs only on the tables the business watches most, and personal-data detection runs on the catalogs.

ProfilerWhereWhat it does
DQX profilerEvery profiled datasetMeasures columns and proposes candidate rules. Runs inside our step, so its output carries our run id.
Lakehouse Monitoring (data profiling)Tier-1 gold tables only (OEE, scrap, warranty claims)Managed profiles and drift over time, with its own dashboard. Costs compute on every refresh, so not everywhere. Name and features: check current status.
Unity Catalog data classificationEvery catalog except restrictedFinds columns that look like personal data (names, phone numbers, emails of dealers and fleet drivers) and suggests PII tags. Check current status.

Profiling is set per dataset with six settings in dataset_config:

SettingValuesMeaning
profile.enabledtrueTurn profiling on or off for this dataset
profile.whenevery_run, daily, weekly, on_changeHow often; daily is the default for silver, every run for tier-1 gold
profile.scopenew_rows, full_tableProfile only what this run loaded, or the whole table
profile.sample_pct10Share of rows to read; large sensor tables use a sample
profile.columnsall, or a listWhich columns; recipe columns are always left out
profile.suggest_rulestrueLet the profiler propose new rules (they are never switched on by themselves)

From profile to rule: a person always decides

1
Profile The profiler measures each column of new data and writes runtime.dq_profile with the run id.
2
Compare Each profile is compared with the last one. A big change (nulls jump from 0.1% to 8%, a new value in plant_code) raises a warning on the ops dashboard.
3
Suggest If suggest_rules is on, candidate rules are written as a pull request to the rules YAML, marked as suggested. They are never switched on automatically.
4
Approve The data steward for that source reviews the pull request, adjusts the limits and merges. The loader puts the rule in config.dq_rule.
5
Enforce From the next run the rule is checked at its hop; the result lands in runtime.dq_result, failing rows in quarantine.
6
Review Every week the steward looks at the quarantine tables, fixes the source or the rule, and releases or drops the parked rows.

Later, an agent: it may read the profiles and quarantine tables, explain what changed in plain words and draft the rule change. It still arrives as a pull request for the steward.

Pipeline observability

The ops dashboard and alerts

Observability means that when a number looks wrong or a dashboard is late, anyone can see within a minute what ran, what failed, why, and what it cost, without opening a notebook. It is built once as a horizontal asset and switched on for every pipeline from the metadata.

Our runtime schemarun, step, error, quality, watermarkDatabricks system tablesjob runs, billing, lineage, auditLakehouse Monitoringdata profiles, drift (check status)Amazon CloudWatchDMS, AppFlow, SiteWise, KinesisOps goldone run id joins it allLakebase read through Unity CatalogOps dashboardAI/BI, one screenAlertsSQL alerts, job alerts, CloudWatchAsk about opsGenie: what failed last night, why?On-call personowner from config+ runbook link
Four sources of signals plus the targets from the metadata, one ops layer joined on the run id, three ways out: a dashboard, alerts and questions in plain words.

Where the signals come from

Signal sourceWhat it tells usHow it reaches the ops layer
Our runtime schema (Lakebase)Every run, step, error, quality result and watermark, with the run id; Lakebase's own health (load, connections, storage) comes from Databricks' Lakebase monitoring (check current status for what system tables expose)Copied into prod_ops every 15 minutes (see below)
Databricks system tablesJob and task runs, billing (DBUs and cost), lineage, audit of who read whatAlready in Unity Catalog; joined on the job run id
Lakehouse Monitoring / data profilingProfiles and drift of key tables: nulls, distinct counts, distributionsCheck current status and name; behind our profiling port
Amazon CloudWatchHealth of the AWS movers: DMS lag, AppFlow failures, SiteWise ingestion, Kinesis delayCloudWatch alarms for the AWS side; a metric stream lands in prod_ops every 5 minutes
Data contracts and freshness targetsWhat each data product promises: by when, how complete, ownerFrom dataset_config; the dashboard checks against them

How each signal lands

Everything lands in one catalog, prod_ops, so one SQL query can join it. The rule: read in place where Databricks already keeps it, copy only what lives outside.

SignalHow it reaches prod_opsDelayNote
Our runtime schema (Lakebase)Lakebase is registered in Unity Catalog as a catalog, so a small job reads it directly every 15 minutes and MERGEs new rows into prod_ops15 minutesDashboards never query Lakebase directly, so a busy dashboard can never slow the pipelines that write to it
Databricks system tablesRead in place; nothing to copyMinutes to an hour, table by table (check current status)Kept by Databricks for a limited time; ops gold keeps 13 months of the summaries we need
Amazon CloudWatch metricsA metric stream sends DMS, AppFlow, Kinesis and SiteWise metrics through Firehose to S3; Auto Loader reads them into prod_opsAbout 5 minutesOnly the metrics we name are streamed, so the cost stays small
CloudWatch alarmsAlarm state changes go through EventBridge to the same S3 pathSeconds to minutesSo AWS-side alarms join the same incident list as Databricks alerts
Pipeline event logs (Declarative Pipelines)Read from each pipeline's event log: expectation results, flow progressPer updateExpectation results merge into the same quality fact as DQX results
Spark driver and executor logsNot loaded. Kept in S3 for 30 days for debugging-Our framework writes structured events to the runtime tables; free-text logs are for engineers only

How they join: four keys

Signals from three systems only help if they line up. Our framework stamps the same keys everywhere, so a failed run can be followed to its cost, its quality results and the AWS mover that fed it.

runtime.run_log (ours)run_id, dataset_idjob_run_idsystem.lakeflow job runsjob_id, run_idstatus, timingssystem.billing.usageusage_metadata.job_run_idDBUs, costconfig (dataset + source)dataset_id, owner, tiermover_id, set per sourceCloudWatch (via S3)dimension: task or flow idlag, failuresruntime.dq_result + eventsrun_id, rule_idpassed, rows failedjob_run_idjob_run_iddataset_idmover_idrun_idFour keys join everything: run_id, job_run_id, dataset_id and the mover id. Our framework writes all four into its own tables.
The join map. No guessing by timestamps: every link is a key our framework writes.

The ops tables

A small star schema of its own, in prod_ops, built by an ordinary pipeline with the same tests as any other.

TableOne row perHolds
ops.dim_datasetOne row per datasetOwner, tier, freshness target, volume band, budget, mover id; from the config schema
ops.fct_runOne row per dataset runOur run_log joined to the Databricks job run and its cost: status, timings, rows, cause of failure, cost, config fingerprint
ops.fct_stepOne row per step of a runLand, bronze, silver, gold: timings and rows in and out
ops.fct_qualityOne row per rule per runDQX results and pipeline expectation results together
ops.fct_freshnessOne row per data product per dayPromised time, actual time, met or missed, minutes late
ops.fct_mover_healthOne row per mover per 5 minutesDMS lag, AppFlow failures, stream delay, from CloudWatch; Lakebase load from Databricks
ops.fct_incidentOne row per incidentOpened, owner, severity, cause class, acknowledged, resolved, linked runs and alerts, agent diagnosis
ops.fct_accessOne row per read of sensitive dataFrom audit system tables and CloudTrail; feeds the quarterly access review

The ops dashboard

One AI/BI dashboard on the ops gold tables, for everyone from the on-call engineer to the plant's data product owner. Numbers in the sketch are made up.

Platform ops · todayall plants · all data productsRuns412398 ok · 9 failed · 5 runningFresh on time46 / 482 data products lateQuality3 rulesfailing; 1,240 rows parkedCost today$18692% of daily budgetFreshness against target (SLO)OEE per planton timeScrap rateon timeWarranty claimslate 40 minDealer sell-outon timeDemand planlate 3 hRaw-material coston timeFailures by cause (7 days)source down11schema changed6bad data4our bug2Run time against its usual (p95)usual p95
Sketch of the ops dashboard. Every tile drills down to the run, the step and the error.

Start with what Databricks already gives. The first five panels come straight from Databricks system tables (read-only tables in Unity Catalog that record jobs, billing, compute, queries, lineage and audit), with no code of ours. Databricks also ships ready-made usage and job-monitoring dashboards to import and adapt (check current status). We add only what system tables cannot know: the business meaning (which data product, which plant, which target), quality results, row counts, watermarks and the cause of a failure. The job run id stored in our run_log is the join key.

PanelWhat it showsBuilt fromWho looks
Runs todaySucceeded, failed, running, skipped; drill to the failing taskSystem tables: job and task run timelinesOps / on-call
Run timeDuration against the usual 95th percentile; slow runs before they become late runsSystem tables: job run timelinePlatform engineers
CostCost per job, warehouse and cluster against budgetSystem tables: billing usage × list prices (CostSherlock adds cost per data product)Platform lead, finance
Compute and queriesBusy or idle clusters, slow SQL queries, who ran whatSystem tables: compute, query history, auditPlatform engineers
LineageWhat feeds each table and what breaks if it failsSystem tables: table and column lineageData engineers, stewards
Failures by causeSource down, schema changed, bad data, our bug; trend over 7 and 30 daysMixed: failure state from system tables, the cause class from our error_logPlatform lead
Freshness against targetEach data product green, amber or red against its promised timeOurs: targets in dataset_config, judged against each input's bookmark (input_watermark), not the last runData product owners, business
QualityFailing rules, rows parked in quarantine, by dataset and plantOurs: dq_result (plus pipeline expectation events)Data quality lead, stewards
VolumesRows in and out against the usual range; sudden drops are the early warningOurs: run_logOps
Watermark lagHow far behind each incremental source isOurs: watermarkOps
AWS moversDMS replication lag, AppFlow flow failures, sensor stream delayAmazon CloudWatchOps

Targets first, then alerts

An alert only makes sense against a promise. Each data product states its targets in the metadata; the dashboard shows them and the alerts fire when they are missed.

Target (SLO)MeasuresExample
FreshnessData product ready by its promised timeOEE per plant by 06:30 for the morning meeting, on 99% of days
SuccessRuns that succeed without a person stepping in98% of scheduled runs per week
QualityRows passing every rule99.5% of rows in silver; zero failures on stop rules
Recovery from a failed runTime from failure to data back on timeUnder 2 hours for tier-1 data products (losing a region is covered by the recovery plan, not this target)

Alerts

Databricks SQL alerts and job notifications cover what happens inside Databricks; CloudWatch alarms with Amazon SNS cover the AWS movers. The metadata database runs on Lakebase, so its health is watched from Databricks, not CloudWatch. All of them post to the same Teams or Slack channel and, for tier-1 data products, to the on-call pager.

AlertFires whenSeverityGoes to
Pipeline failed after retriesrun_log status failed and retries used upHighTeams/Slack + email; pager at night for tier-1 data products
Data product lateFreshness past the promised time in dataset_configHigh for tier 1, Medium otherwiseOwner of the data product, on-call
Quality rule set to stop faileddq_result failed on a rule whose action is stopHighData quality lead and steward; the load is held
Volume out of rangeRows written outside the usual band for that dataset and weekday; for streams, per 15-minute windowMediumOn-call; often the first sign of a source problem
A press went silentA press in the press master sent no reading for 30 minutes during a planned shiftHigh (OEE is tier 1, so it pages)Plant maintenance and on-call; one silent press is too small to move a volume band, so it gets its own check
Schema changed at sourceContract check finds a new, missing or retyped columnMediumThe source owner named in the contract; silver is protected by the contract
Run much slower than usualDuration above 2 × the usual p95LowPlatform engineers, next working day
Cost over budgetDaily or monthly cost per pipeline, per SQL warehouse or for the AWS side above its budgetMediumPlatform lead and finance (CostSherlock)
Cost jumpYesterday's total cost (Databricks and AWS) more than 1.5 × its 7-day averageHighPlatform lead; the runbook can pause tier-3 schedules and shrink warehouses within the hour
Source mover unhealthyCloudWatch alarm: DMS lag, AppFlow failure, Kinesis delayHighOn-call through Amazon SNS
Metadata store unhealthyLakebase load, connections or storage past their limits, from Databricks' Lakebase monitoring (check current status); recorder timeouts in error_logHighPlatform on-call; the heartbeat and job alerts do not need Lakebase
1
Every alert has an owner and a deputy Both come from dataset_config. A nightly check compares every named owner with your identity provider; a leaver is flagged to the data owner the next morning.
2
Every alert has a runbook A link in the message to what to check first and how to re-run safely.
3
One alert per incident Repeats are grouped by run id and dataset, so a source outage pages once, not 40 times.
4
Severity follows the data product tier Paging: a High alert on a tier-1 product pages at any hour; on tier 2 it pages during plant hours; tier 3 and every Medium or Low alert wait for working hours. OEE for the morning meeting is tier 1; warranty claims are tier 2, due 09:00 daily.
5
Every alert is reviewed A noisy alert is tuned or removed within a week; an alert nobody acts on is deleted.

From alert to incident

1
Alerts open incidents; incidents are what people work. Several alerts about one cause (a source down, forty datasets late) group into one incident by source and time window. Freshness alerts, which have no run, group by dataset and expected date.
2
Every incident has an owner from the metadata. The data product's owner and the source's engineer, never a shared mailbox.
3
If business users saw a wrong or late number, tell them. A banner on the affected dashboards and a note to the metric owner, from the incident, until it is closed. Recomputed history is logged as a restatement in ops and shown the same way.
4
Close with a cause class. Source down, schema change, bad data, our bug, platform, capacity, cost. The trend of causes tells us what to fix next.
5
Tier-1 incidents get a short review. What happened, why the tests did not catch it, and the new test or rule that will. Each review adds a DataTwin fault.

Later, self-healing: for known causes (a source briefly down, a late file) an agent may propose the fix and prove it on a pre-prod clone; a person approves before anything touches production. See agents.

Testing and data contracts

Catch it early, prove it before release

Most data platforms are tested by looking at the dashboard after go-live. We test in six gates, from the developer's laptop to production, and each gate catches what the earlier ones cannot. All of it is part of the platform from day one, not a project for later.

Laptoppre-commit· shift-left checks· purity check· unit testssecondsPull requestCI· full shift-left scan· unit + coverage· contract checksminutesDevafter merge· one test per hop· end to end, synthetic· fresh schema15–30 minDataTwinnightly· pretend plant, 30 days· planted faults· caught? held?an hourPre-productionbefore release· real data (clones)· parallel run vs prod· profile comparean hour or twoProductionevery run· contract check· quality rules· freshness alertsalways oncheapest and fastest on the left · a problem caught one gate earlier costs a fraction to fix
Six gates. The same tests run for every pipeline, because the pipelines are generated from the same metadata.
LayerWhat it provesBuilt withRuns
Shift-left checksProblems found by reading the code, before it runs: Python and PySpark anti-patterns, SQL injection, secrets, collect() on big data, cartesian joins, missing keys in MERGE, costly job settings, metric view mistakesOur shift-left bundle (built: 131 checks specified; caught 88 of 101 planted problems on a sample project with no false alarms) on Ruff, sqlfluff and dbt-bouncerLaptop and pull request
Unit testsEach transformation as a pure function, tested table-driven: a table of input rows and expected output rows per case (SCD2 new, changed, unchanged, deleted; duplicates; late rows; time zones)pytest + local Spark and Delta (no cluster); the shift-left test kit for SCD2 and idempotencyLaptop and pull request
Contract testsEach contract is valid, and a change does not break consumers (a removed column, a narrowed type, a renamed key)ODCS schema validation + a breaking-change diff against the last released versionPull request
Integration testsOne slice end to end on real Databricks: lander, silver with SCD2, gold, metric view; run twice to prove re-runs change nothing; reconciliation counts and sums agreeDatabricks Asset Bundles deploy to the dev workspace on a fresh schema per run; small fixed sample files in S3After merge, in dev
DataTwinA pretend plant and business that produce data for 30 simulated days with faults planted on purpose; checks that the platform caught each one, held bad data back, and every number is accounted forDataTwin (designed; the DataTwin Studio and Problem Lab are built and run on real Spark and Delta)Nightly in dev, and before each release in pre-prod
Pre-production runThe release on real data: shallow clones of prod, a parallel run against prod with every difference explained, profiles compared, Genie benchmarks per roleThe pre-prod workspace and the CI service principalEvery release
Production checksContract check at silver, quality rules with warn / quarantine / stop, freshness and volume alertsLakeflow expectations or DQX behind our quality port; rules from config.dq_ruleEvery run

The full test strategy, cloud-neutral, is on Test architecture for a data platform.

How this uses what we are building. The shift-left bundle and its test kit run at the first two gates; the Problem Lab scenarios (duplicates, late data, failure mid-job, schema drift, SCD2 pitfalls) become the unit and integration cases; DataTwin is the release gate. Because pipelines are generated from metadata, a new source gets these tests without anyone writing them again.

DataTwin: a pretend plant with faults planted on purpose

Real data rarely shows the bad days on schedule. DataTwin produces data the way the plants and the business would, then plants the faults below. The release passes only if the platform catches every one and the dashboard still matches the pretend world's true numbers.

Fault planted in the pretend plantThe platform passes if
SAP sends the same delivery twiceNo duplicate in silver; sales not doubled in gold
MES adds a column and renames anotherContract check stops it at silver; alert to the MES owner; gold untouched
A press sensor goes silent for 3 hoursThe silent-press alert fires and pages; OEE is on time, marked incomplete, and neither too high nor too low
The warranty file arrives a day lateFreshness alert; the late rows land in the right day when they arrive
A dealer's master record changes twice in a daySCD2 keeps both versions with correct dates
Daylight saving at the plant in EuropeNo lost or doubled hour in hourly scrap
A job dies halfwayWatermark not moved; the re-run loads the same window once

Data contracts: do we need them?

A data contract is a written agreement between the team that produces data and the teams that use it: which columns, what they mean, how fresh, how good, and who owns it. It is written as a YAML file in Git using the open standard ODCS (Open Data Contract Standard, version 3), so it is both readable and checkable by machine. The short answer: yes, at the two boundaries, and no, not everywhere.

WhereContract?Why
At sources we do not control (SAP, MES, quality LIMS, dealer DMS)YesTheir teams change systems on their own timetable; a contract turns a silent break into a clear alert with an owner.
At data products other teams use (OEE, scrap rate, warranty claims)YesConsumers need to know the columns, the meaning, how fresh and who to call; it is also what Genie and dashboards rely on.
Between our own internal layers (bronze to silver to gold)NoOur tests and quality rules already cover them; a contract there is paperwork.
A full contract platform or marketplace toolNot nowYAML in Git plus the checks we already run is enough to start; revisit when there are dozens of producing teams.
Part of the contractExampleEnforced by
SchemaColumns, types, keys, which may be emptyContract check at silver; breaking-change diff in pull requests
QualityRules such as 'scrap quantity is never negative'The same rules run as quality checks
Freshness and volumeReady by 06:00, between 40k and 60k rows a dayFreshness and volume alerts
Meaning and ownerWhat a column means; the producing and consuming ownersUnity Catalog comments and tags; alert routing
PrivacyWhich columns hold personal data or recipe IPUnity Catalog tags, column masks and restricted catalogs
VersionMajor version for breaking changes, with notice to consumersPull request review; consumers named in the contract approve

Nothing new has to be built to enforce them: the contract check at silver, the quality rules, the freshness and volume alerts and Unity Catalog tags already do it. The contract simply makes the promise explicit and gives every alert an owner. The open-source datacontract command-line tool lints the ODCS files and tests them in the pull-request gate.

The right medallion

One medallion, two speeds

The medallion is the row of layers in the middle: bronze (raw), silver (clean) and gold (shaped for the business). Streaming does not need a second architecture. Nightly batches and sensor events flow through the same tables, with the same contracts and the same checks; only the speed differs. Streams run trigger-once on a schedule, every 15 minutes, never continuously.

Batch sourcesSAP, MES: hourlyEvent sourcespress sensors, IoTBronzeAuto Loaderone set of tablesSilverAUTO CDC + rulesone set of tablesGoldmarts, productsone set of tablesSemanticMetric Viewssolid line: batch on a schedule · dashed line: stream, trigger-once every 15 minutes · same contracts and checks on both
Batch and streaming share one set of layers.

What each layer promises

LayerPromiseRulesDatabricks features
Landing (S3)Files exactly as received.Never changed. Short retention. Encrypted with your KMS key.S3 lifecycle rules; Unity Catalog volumes or external locations
BronzeEverything that arrived, replayable.Append-only. Adds ingest metadata (file, time, batch id). Keeps the source schema. Safe to run twice.Auto Loader, streaming tables, rescued-data column
SilverOne clean, trusted row per real thing.Typed and deduplicated. Keys conformed (material, plant, press, batch, dealer). History kept as SCD Type 2. Bad rows go to quarantine, never silently dropped. Personal data and recipe columns tagged.AUTO CDC (formerly APPLY CHANGES) for SCD1/SCD2; expectations; column tags
GoldShaped for a business question.Reads only silver. Star schemas and data products with a stated grain. Rebuilt safely by partition.Materialized views; liquid clustering; predictive optimization
SemanticEach metric defined once.No tables of its own. Definitions, joins and names that every tool uses.Unity Catalog Metric Views (check current status)

Bronze to silver

Thirteen generic stages run for every dataset, set per column in the metadata: contract check, select and rename, hygiene, parse and type, time, standardise values, personal data, row filter, quality rules, duplicates, keys, history (SCD type 1 or 2 by kind of data) and provenance. No business logic and no free-form SQL in this step. The stages, the column settings, the schema change rules and the history choices are on The ETL, end to end.

Silver keeps one table per source entity, with conformed keys. Joining sources together (SAP orders with MES production, for example) is a separate, named step, not hidden inside a silver table. Each silver table has one quarantine table, which its steward reviews weekly.

Common mistakes, and the fix

1
Dashboards reading silver or bronze. Two numbers for the same metric. Fix: experiences read only gold through metric views; lineage is checked against the allowed paths.
2
A separate architecture for streaming. Twice the code and two answers. Fix: one medallion, two speeds: streaming tables in the same layers.
3
Dropping bad rows silently. Counts drift and nobody knows why. Fix: quarantine with a reason, and reconcile counts from source to gold.
4
Overwriting history in silver. Yesterday's report can no longer be explained. Fix: SCD Type 2 where the business needs history.
5
One notebook per source. A hundred sources become a hundred snowflakes. Fix: metadata-driven ingestion; a new source is a YAML entry.
6
Protecting recipes and personal data only in the dashboard. Anyone with SQL access sees them. Fix: tag in silver, use a restricted catalog and Unity Catalog column masks everywhere.

More detail, with one record followed through every layer: Medallion architecture, drawn right.

Gold, the semantic layer and AI/BI

Shape it once, define each metric once, then ask in plain words

Ask three tools for "scrap rate" and you can get three numbers. That happens when every report shapes the data and writes the formula itself. The fix has two halves: gold shapes the data once, as star schemas the business recognises, and the semantic layer defines each metric once, as a metric view in Unity Catalog. Every front end, from dashboards to Genie, reads those definitions. Then AI on top is safe: the model only chooses which metric, which breakdown and which filter.

The choices

ChoiceOur pickVerdictWhy
How gold is modelledStar schemas (Kimball): facts at a stated grain, shared dimensionsPursueBusiness people read them, BI tools and Genie understand them, and shared dimensions make numbers line up across subjects. Data Vault adds a layer nobody here needs; one big flat table repeats data and breaks on history.
How gold is builtSQL files in Git, run as materialized views in Declarative PipelinesPursueIncremental refresh and lineage come with the platform. One model per file, so moving to dbt Core later is a copy, not a rewrite.
dbtNot nowDrop for nowAdopt dbt Core if the team already knows it, if gold passes about 50 models across teams, or if gold must run outside Databricks. Never dbt Cloud or a second metric layer.
Where metrics are definedUnity Catalog metric views, one per business processPursueDefined once, governed by the catalog, read by dashboards, Genie and our Conversation UI alike. A separate semantic tool (Cube, AtScale) only if many tools outside Databricks need the same metrics.
Asking in plain wordsGenie for analysts and managers; our Conversation UI inside staff apps and wherever a wrong number is costlyPursueGenie is already in the platform. The Conversation UI never lets the model write SQL.
Existing BI toolsRead through a serverless SQL warehouse, from metric views (check current status) or goldPursueNobody has to throw away reports on day one.
Compute for questionsSeparate serverless SQL warehouses for dashboards, Genie and ad hoc workPursueEach workload gets the right size, and CostSherlock can show what each one costs.

Gold: star schemas

A star schema puts the numbers (facts) in the middle and the words people filter by (dimensions) around it. Each fact table states its grain: what one row means. Getting the grain right is most of the work.

fct_productionone row per press per hourtyres cured, scrapped,run and planned minutesdim_datefinancial year Apr-Mardim_plantIndia and Europedim_presscuring press, linedim_shiftplant local timedim_materialSCD2: compound as of cureFacts hold the numbers; dimensions hold the words you filter and group by.
The production fact and its dimensions. Other facts share the same dimensions.

The bus matrix shows which facts share which dimensions. A shared ("conformed") dimension is what lets you put scrap next to warranty claims for the same material and get a sensible answer.

FactGrainDatePlantPressShiftMaterialDealerCustomer / fleetDefect
Productionone row per press per hour, split by batch type (production or trial)✓✓✓✓✓
Downtimeone row per stop✓✓✓✓
Scrapone row per scrapped tyre batch✓✓✓✓✓✓
Quality testsone row per test result✓✓✓✓✓
SAP order linesone row per order line✓✓✓✓✓
Dealer sell-outone row per dealer per material per day✓✓✓✓
Warranty claimsone row per claim✓✓✓✓✓✓

Where gold lives

SchemaHoldsOwner
prod_gold.sharedThe shared dimensions: date (with the April-to-March financial year), plant, press, shift, material, dealer, customer, defect codePlatform team
prod_gold.manufacturingProduction, downtime, scrapManufacturing data owner
prod_gold.qualityQuality tests, warranty claimsQuality data owner
prod_gold.salesSAP order lines, dealer sell-out, dealer stockSales data owner
prod_semantic.*Metric views only, one schema per business area; no tablesEach metric's owner

Facts built from several inputs

Some facts need more than one source: production needs MES counts, the press master, the material master and the shift calendar, which arrive at different times. Five rules keep them right.

1
Give every input a role The driver (the events the fact counts), dimensions looked up as of the event time, and contributor facts that add measures.
2
Never lose a row to a missing dimension A press that is not in the master yet gets an 'unknown' member, flagged, and fixed when the master catches up. No inner joins to dimensions.
3
Join history as of the event A material's compound at the moment the tyre was cured, not today's.
4
Recompute changed windows When any input changes, recompute the affected hours or days in full and replace them. Simpler and safer than patching single rows.
5
Check after every run Totals reconcile with silver, and the share of 'unknown' members stays under its limit.

The semantic layer: metric views

A metric view is a definition, not a table: which gold fact, which joins, which dimensions, and the formula for each measure. It is stored in Unity Catalog with an owner, so it is governed like a table.

# prod_semantic.manufacturing.press_performance   (syntax: check current status)
version: 0.1
source: prod_gold.manufacturing.fct_production        # one row per press per hour
joins:
  - name: press
    source: prod_gold.shared.dim_press
    on: source.press_key = press.press_key
  - name: material                                    # SCD2: the version valid at cure time
    source: prod_gold.shared.dim_material
    on: source.material_key = material.material_key
dimensions:
  - name: Plant
    expr: press.plant_name
  - name: Batch type
    expr: source.batch_type
  - name: Press
    expr: press.press_code
  - name: Compound family
    expr: material.compound_family
  - name: Production date (plant time)
    expr: source.plant_local_date
measures:
  - name: Tyres cured
    expr: SUM(source.tyres_cured)
  - name: Scrap rate                                  # production batches only
    expr: SUM(source.tyres_scrapped) FILTER (WHERE source.batch_type = 'production') / NULLIF(SUM(source.tyres_cured) FILTER (WHERE source.batch_type = 'production'), 0)
  - name: Completeness                                # share of expected press-hours that arrived
    expr: SUM(source.hours_received) / NULLIF(SUM(source.hours_expected), 0)
  # OEE parts count complete press-hours only (top and bottom alike), so a silent press
  # makes OEE incomplete, never wrongly high or low; Completeness shows how much is missing.
  - name: Availability
    expr: SUM(source.run_minutes) FILTER (WHERE source.is_complete) / NULLIF(SUM(source.planned_minutes) FILTER (WHERE source.is_complete), 0)
  - name: Performance
    expr: SUM(source.ideal_cycle_seconds * source.tyres_cured) FILTER (WHERE source.is_complete) / NULLIF(SUM(source.run_minutes * 60) FILTER (WHERE source.is_complete), 0)
  - name: Quality
    expr: SUM(source.tyres_good) FILTER (WHERE source.is_complete) / NULLIF(SUM(source.tyres_cured) FILTER (WHERE source.is_complete), 0)
  - name: OEE                                         # good tyres x ideal cycle time over planned time
    expr: SUM(source.ideal_cycle_seconds * source.tyres_good) FILTER (WHERE source.is_complete) / NULLIF(SUM(source.planned_minutes * 60) FILTER (WHERE source.is_complete), 0)

One metric view for press performance. Formulas are sums divided by sums, computed for whatever slice is asked.

The starter metrics

MetricDefinitionBroken down byOwner
OEEAvailability × performance × quality, each computed from sums (run time over planned time; ideal cycle time × count over run time; good tyres over all tyres)Press, shift, plant, dayPlant operations head
Scrap rateScrapped tyres over tyres cured, production batches only. Both counts come from the production fact; the breakdown by defect comes from the scrap fact, whose totals reconcile with itPress, material, day; defect through the scrap factQuality head
First-pass yieldTyres passing every test the first time, over tyres testedPlant, material, dayQuality head
Downtime hoursSum of stop durations, by causePress, cause, shiftMaintenance head
On-time in-full (OTIF)Order lines delivered complete and on the promised date, over all order linesPlant, customer, weekSupply chain head
Sell-out unitsTyres sold by dealers to end customersDealer, material, region, weekSales head
Warranty claim rateClaims per 10,000 tyres, counted against the month the tyres were made, not the month of the claimPlant, material, production monthQuality head

Six rules for the semantic layer

1
Metrics read gold only. Never silver, never another metric view's output.
2
Ratios are computed at query time, from sums. Never store a percentage and never average percentages: the average of three plants' OEE is not the company's OEE.
3
Every metric has an owner, a definition in plain words and a certified tag. Changing a definition is a pull request approved by the owner, who also updates the expected benchmark answers in the same pull request. Metric views recompute history with the new definition, so the change is logged with its date and shown on dashboards; where last year must stay comparable, the old definition is kept as a second measure for a year. Metrics that share a part (OEE's quality part and scrap rate) share one definition of it.
4
Shift metrics use plant local time. Stored in UTC, reported in the plant's time, so the night shift is not split across two days.
5
Joins live in the metric view. Pre-joined or pre-summed tables only when a dashboard is measured slow or costly, and then as materialized views in gold.
6
Front ends read metric views. Direct access to gold is for analysts and data scientists only.

To confirm when building: whether reading a metric view also needs a grant on the gold tables under it, and whether outside BI tools can query metric views directly (check current status).

From a question to an answer

Question“Scrap rate lastweek at Plant A,by press?”Intentmodel picks:metric, dimensions,filters, periodMetric viewscrap_rate definedonce: scrapped tyresover tyres curedGoverned SQLthe engine writes it;Unity Catalog rowfilters applyAnswernumbers + chart+ the definitionit usedThe model never writes SQL. It only chooses from metrics and dimensions that exist.Three front ends, one set of metric definitionsAI/BI dashboards · Genie spaces · our Conversation UI · BI tools through the SQL warehouse
From a question to an answer that shows its own definition.

Genie: one space per audience

SpaceForReadsBenchmark
Plant performancePlant managers, shift leadsPress performance, downtime and scrap metric views40 questions
Quality and warrantyQuality engineersScrap, quality tests and warranty metric views40 questions
Sales and dealersSales and aftermarket managersOrder lines, sell-out and dealer stock metric views30 questions
Platform opsThe data teamOps gold from the observability section20 questions
1
One space per audience, not one for everything. A small, curated space answers better than a large one; keep each to the views its audience needs.
2
Metric views first. They carry the definitions, so Genie picks a defined metric instead of inventing a formula.
3
Write instructions and example questions. Plant names and their short forms, which date to use, what 'last week' means, with worked examples.
4
Benchmark before go-live and after every change. A set of questions with known right answers, run once per role with test identities, so filtered answers are checked too; the space goes live at 90% right or better, and a change that lowers the score does not merge.
5
Say when a filter applied. The space instructions make Genie say 'for your plant' when the person's row filter narrowed the answer, so one plant's figure is never read as the company's.
6
Read the thumbs-down every week. Each wrong answer becomes a fix to the instructions or a new benchmark question.
7
Security comes from the catalog. Genie runs as the person asking, so a plant manager sees only that plant's rows.

Genie or our Conversation UI?

NeedGenieOur Conversation UIPickVerdict
Analysts and managers asking new questions on curated tablesStrong fit: fast to set up, curated by instructions and example questionsPossible, but more workGeniePursue Genie first; it is in the platform already.
Answers inside a staff app or chat channel, next to an agentNot designed for embedding in our own apps (check current status)Built for this: an API and a chat windowConversation UIPursue for embedded use.
A hard rule that the model never writes SQLGenie writes SQL, guided by trusted assets and instructionsThe model only picks metrics; the engine writes SQLConversation UIPursue where a wrong number is costly (finance, plant KPIs, warranty).
Proving answers are right before go-liveBenchmark questions built inOur eval harness with the same benchmark setBothUse the same benchmark questions for both.
Consumers or the publicNot the right toolOnly behind the agent harness, with its guardrailsNeither directlyDrop for now; route through the agent harness.

Security, privacy and governance

Who may see what, and how we prove it

Three kinds of data need protecting here: personal data (dealers, fleet drivers, consumers who register a warranty, operators on shift logs), commercial data (prices, discounts, margins) and the recipes that make the tyres. The design protects each one where the data lives, in the catalog, so the protection holds in every tool: a dashboard, a notebook, Genie or an agent.

Four levels of sensitivity

LevelExamplesWhere it livesWho sees itProtection
InternalProduction counts, OEE, scrap, downtime, stockprod_gold, prod_semanticEvery employee group that needs it; plant staff see their own plant's rowsRow filter by plant
ConfidentialPrices, dealer discounts, margins, cost per tyreprod_gold.sales, columns tagged confidentialFinance and sales groupsColumn mask for everyone else
PersonalDealer contacts, fleet drivers, consumers who register a warranty, operator ids on shift logs, names inside free-text warranty notesTokenised at the lander; real values only in the token vault (prod_pii_vault, its own bucket and key)Almost nobody needs the real value; the warranty team for a live claimTokens everywhere, mask on the vault, purpose and retention per category
RestrictedCompound recipes, cure recipes, process settingsprod_restricted only, its own bucket and keyThe named recipe keepersNo Genie space, no copies except the encrypted recovery copy, every read alerted

Roles and what they get

People read; only service principals write to prod. That one rule removes most accidents.

Role (a group)GetsLimits
Platform engineerRead and write in dev; read prod_opsDeploys only through CI/CD
Data engineerRead and write in dev; read the results schemas in pre-prodNever writes to prod by hand; prod clones in pre-prod are readable only by CI
Analyst, data scientistRead prod_gold and prod_semanticPersonal data only as tokens; confidential columns masked
Plant manager, shift leadRead prod_semantic; dashboards and the Plant performance Genie spaceOwn plant's rows only
Quality engineerRead quality and manufacturing gold and metric viewsPersonal data only as tokens
Sales and financeRead sales gold and metric views, confidential columns visibleOwn region's rows for regional managers
Recipe keeperRead prod_restrictedNamed people, reviewed monthly
Privacy case handlerRead and erase in prod_pii_vaultThe privacy team and, for a live claim, the warranty team; every lookup logged
AuditorRead audit system tables and the audit schemaNo business data
Service principalsOne per pipeline area; write only their own target schemas in prodThe only identities that write to prod

How access is enforced

1
Groups from your identity provider Every grant goes to a group, never to a person. Joining or leaving a group is the only way access changes.
2
Tags set from metadata In silver, the pipeline tags each column as personal, confidential or recipe, from dataset_config. Nobody tags by hand.
3
Policies on tags One policy per tag masks the column for every table that carries it, through attribute-based access control (check current status). Where that is not yet available, the pipeline generates a column mask per table from the same tags.
4
Row filters by plant and region A small mapping table (group to plants, group to regions), loaded from Git like all config, drives one row-filter function used on every fact table.
5
Grants from metadata Catalog-level grants live in Terraform; schema and table grants are declared in YAML next to the datasets and applied by the deploy, so the catalog always matches Git and drift is reported.

Personal data: India's DPDP Act and the EU's GDPR

Both laws ask the same things in practice: keep personal data only for a stated purpose, only as long as needed, protect it, and erase it when asked. The hard part in a lakehouse is erasure, because bronze keeps everything for replay. The design solves it at the door.

1
Tokenise at the door. The lander swaps names, phone numbers and emails for tokens before bronze, and checks the suppression list. Free-text fields such as warranty notes go through a personal-data detector behind a port, which must handle English and German (Amazon Comprehend's detection covers English and Spanish, check current status; an open-source detector such as Presidio with German models otherwise); until that is confirmed, EU free text is held out of bronze. For sources with personal data, a file with a column nobody declared is held and the dataset blocked until the contract says what it is (only the column's name is logged), so a new personal column cannot slip in as rescued data. Bronze stays append-only and still holds nothing personal, so erasure never fights the replay design.
2
Real values never wait in landing. Files from sources with personal data use their own landing folder, deleted one day after the lander has read them, with old versions expiring the same day. Macie scans landing weekly for personal data in the wrong place.
3
EU personal data leaves the EU only on a legal basis. India has no EU adequacy decision, so personal data from EU plants and dealers travels under the company's Standard Contractual Clauses with a transfer impact assessment, or is tokenised in the EU before it leaves. Your legal team picks; until then, EU sources are loaded without personal columns.
4
A purpose for every personal column. dataset_config records why it is kept (warranty claim handling, dealer payments) and for how long; columns without a purpose are not loaded.
5
Retention runs itself. A nightly job deletes personal data past its retention period and logs what it removed.
6
Logs never hold values. The lander, which sees real values before tokenising, never logs row values; its error messages are scrubbed before they reach error_log.
7
Tokens are random. A token is a random value, mapped to the person only in the vault, so deleting the vault entry really does make the data anonymous. The suppression fingerprint uses a separate secret key.
8
Consent stays at the source. Warranty and dealer systems collect consent; we load the consent flag and filter on it, we never decide it.
9
Breach response is written down. Who decides, who tells the regulator and the people affected, inside the deadlines of the DPDP Act and GDPR (72 hours for GDPR; DPDP Rules timelines: check current status).

When someone asks to be erased

1
Request arrives From the privacy team's case system, with the person's identifier.
2
Find the token Look up the person's token in the vault (prod_pii_vault).
3
Find every place Lineage and tags list every table holding that token: silver, gold, quarantine tables.
4
Erase or restrict? The privacy lead decides. An open claim or a product recall may need the identity kept for now: processing is then restricted instead, and the decision is recorded in the case.
5
Erase Delete the vault row first: from that moment the token points to nobody. Add a one-way fingerprint of the identifier to the suppression list, so a re-sent extract cannot bring the person back. Then blank the token's rows in silver (including old SCD2 versions) and in the quarantine tables, and rebuild the affected gold windows. Where the law requires a claim to be kept (product liability), the claim stays and only the identity goes.
6
Erase the copies Run the same erasure on the recovery copies in Hyderabad, and delete the person's files from landing, including old versions.
7
Clean old files Delta keeps old file versions for 30 days; after VACUUM, the data is gone from storage. Bronze only ever held the token. VACUUM runs in Hyderabad too, every bucket expires old object versions within 30 days, and AWS Backup is not used for the vault or personal-data tables. No table may be restored to a version from before an erasure; if one is, the erasure log is replayed straight after.
8
Record evidence Write what was erased, where and when to the audit schema, and close the case.

Tokenising uses the framework's existing pii_policy table (actions: pass through, hash, mask, tokenise, redact, per environment). The vault is its own catalog, prod_pii_vault, with its own bucket and key, readable only by privacy case handlers; recipe keepers cannot read it, and the privacy team cannot read recipes. The fingerprint key and the vault key live in AWS Secrets Manager and KMS.

Recipes

Recipes and process settings sit in prod_restricted, in their own bucket with their own key. Gold carries only the compound family, never the recipe. Only the named recipe keepers can read it, every read raises a notice to the recipe owner (the head of compound development), access is reviewed every month, and revoking the key cuts all access at once. It is never added to Genie or to an agent.

AI on governed data

1
Genie and the Conversation UI run as the person asking. Row filters and masks apply exactly as in a dashboard.
2
prod_restricted and prod_pii_vault are never added to a Genie space or an agent's tools. Recipes and real identities are not answerable in plain words, by design.
3
Models run inside the platform or your AWS account. Databricks-hosted models or Amazon Bedrock in your account; no data is sent to a public chat service.
4
Every question and answer is logged. With the person, the metric used and the SQL the engine ran, in a separate schema (prod_ops.ai_log) that only the space owners and the platform lead can read, because answers carry other plants' figures and prices.
5
Row security is tested, not assumed. The gold-to-metrics test runs as a plant-manager test identity and must see one plant only, through every metric view and every Genie space. It is a release gate.

Who decides

RoleWhoDecides
Data ownerA business head (plant operations, quality, sales)Decides who may see the data and signs off metric definitions
Data stewardA named person per sourceWrites quality rules, reviews quarantine weekly, approves suggested rules
CustodianThe platform teamRuns the platform, applies the owner's decisions, never decides access
Privacy leadFrom the legal or privacy teamOwns purposes, retention, erasure requests and breach response

Proof, not promises. Databricks audit system tables, AWS CloudTrail and KMS key-use logs are joined in prod_ops into an access report: who read what, from which tool. The data owners review it every quarter, and it is the evidence pack for an auditor.

CI/CD and environments

Every change takes the same path to production

Code, configuration and definitions all change through Git and reach production the same way, through the six test gates. Nobody clicks a change into prod, and every release can be undone.

Pull requestchecks on the codeDevper-hop testsPre-prodreal data, DataTwin,parallel run, benchmarksApprovaldata owners signProdone plant firstdeveloper opens itmerge to mainrelease tagrelease reportprod service principalOne repo, three things deployed together: code (Asset Bundles), config (loader into the metadata database), definitions (metric views, grants, Genie)If a release is wrong: redeploy the last tag, and restore each table to the version before the run
From a pull request to production, with the checks at each step and the way back.

Two repositories

RepositoryHoldsOwnerChecks
platform-foundationTerraform for accounts, network, storage, keys, workspaces, catalogsPlatform teamTerraform plan on every pull request; apply after approval
data-platformThe framework code, all YAML config (sources, datasets, rules, contracts, grants), gold SQL, metric views, dashboards, Genie space definitions, Asset BundlesShared, with CODEOWNERS per folderThe release path below

Who approves which folder

GitHub CODEOWNERS sends each change to the people who own it: a quality rule to its steward, a metric to its owner.

FolderApproved by
framework/Platform team
config/sources/, config/datasets/Data engineers for that source
config/dq/The source's data steward
contracts/The producing team and the platform team
gold/, semantic/The business area's data owner and an analytics engineer
config/grants/, config/access/The data owner, plus the privacy lead for personal data

What deploys what

WhatDeployed byFrom
Accounts, network, buckets, keys, workspaces, catalogsTerraformplatform-foundation
Jobs, pipelines, dashboardsDatabricks Asset Bundlesdata-platform
Sources, datasets, rules, profiling settingsOur config loader, into the metadata databasedata-platform
Metric views, row filters, masks, schema and table grantsSQL run by the deploy, generated from YAMLdata-platform
Genie spacesTheir definitions in Git, applied through the Genie API (check current status)data-platform

The path, step by step

StepWhat runsWhereTakes
Pull requestLint, shift-left checks, unit tests, contract checks, config dry-run against the key registry, bundle validate, Terraform planLaptop and CI runner, no Databricks neededminutes
Merge to mainDeploy to dev; per-hop integration tests on fixed fixturesDev workspace, fresh schema per runabout 20 minutes
Release tagDeploy to pre-prod; end-to-end on synthetic data, DataTwin faults, real-data run on shallow clones, parallel run against prod, Genie benchmarksPre-prod workspacean hour or two
ApprovalThe release report goes to the data owner of each changed area; one approval per areaGitHub environment approvalsame day
Prod deployDeployed by the prod service principal; risky changes to one plant first for one full day, promoted to all only if the smoke checks and reconciliations stay greenProd workspaceminutes, then one day
After deploySmoke checks: first runs green, row counts within band, metrics within bandProdfirst run

Three environments

EnvironmentDataWhoSize and cost guard
DevSynthetic data and small samples; no personal dataData and platform engineersSmall serverless; jobs stop after 2 hours; budget alert
Pre-prodShallow clones of prod, personal data already tokens, recipes and the vault excludedCI writes and reads prod; engineers see resultsLike prod, for an hour or two per release; DataTwin's nightly run happens in dev
ProdReal dataRead through groups; only service principals write; personal access tokens off for peopleSized for the plants' shifts (see sizing); budget policies per data product

Five rules

1
No secrets in CI. GitHub Actions signs in to AWS and Databricks with short-lived OIDC tokens (check current status for Databricks), not stored keys.
2
A config change is a deploy. A new rule or source goes through the same pipeline, with lighter gates when no code changed, except changes to quality limits, watermarks, replacement rules and grants, which always take the full gates.
3
Breaking schema changes are versioned. A new column passes: it lands in bronze and is ignored until someone maps it. A removed, renamed or narrowed column stops at silver's contract check and needs a new contract version, a migration step and a notice to consumers.
4
Rollback is redeploying the last tag. If wrong data was written for one plant, recompute only that plant's windows with the previous tag, so good runs for other plants are kept; watermarks are reset by a backfill plan, not by hand. A whole-table RESTORE only when every later run is also wrong, and never to a version before an erasure. The platform lead decides rollback or fix.
5
Small, frequent releases. Trunk-based: short branches, merged within a day or two, released weekly or on demand.

Agents

Agents propose, people decide

Agents help most with work that is slow and repetitive for people: reading logs after a failure, drafting a quality rule, writing YAML for a new source. In this design an agent never changes production. It reads, explains, and proposes a change as a pull request, which then takes the same release path as any other change. The agents run on the agent harness on AWS.

Seven agents, one job each

AgentStarts whenDoesMay
TriageAn alert opens an incidentCollects the evidence: the failed run, the error, recent config and code changes, upstream lineage, CloudWatch for the mover. Writes a diagnosis and the likely cause class into the incident.Read only
Fix proposerA diagnosed incident with a known causeStarts only after a person has confirmed the triage diagnosis. Its clone runs go through the CI pipeline, so it never reads prod itself. Drafts the fix as a pull request (a widened type, a new rule limit, a watermark reset plan), and proves it by re-running the failed window on a shallow clone in pre-prod.Pull request + pre-prod clone
Rule suggesterA new profile or a full quarantine tableExplains what changed in plain words and drafts quality rule changes for the steward.Pull request
Source onboarderA sample file or a database table from a new sourceDrafts the source and dataset YAML, the contract and the first quality rules.Pull request
Genie curatorThumbs-down answers and failed benchmark questionsDrafts better instructions, example questions and new benchmark questions for the space owner.Pull request
Cost finderCostSherlock findingsDrafts the change behind each finding (a smaller warehouse, a job schedule, a clustering key) with the expected saving.Pull request
DocumenterA table or metric without a descriptionDrafts table and column comments and glossary entries from the code, the contract and lineage.Pull request

How far each may go

LevelMeansWho
Readops tables, metadata, lineage, code, profilesAll agents
ProposeOpen a pull request; comment on an incidentAll agents
Act on a test copyDeploy a branch and re-run a window on a pre-prod shallow cloneFix proposer only
Act in prodChange data, config or code in prodNone. Prod changes only through the normal release path, approved by a person

Five rules

1
Each agent is its own service principal. With read access to prod_ops and the metadata, and no access to prod_restricted or prod_pii_vault: agents see tokens, never personal data.
2
Tools are listed, not open. Each agent gets a short list of tools (read a run, read lineage, open a pull request); no general SQL on prod.
3
Every step is logged. Prompts, tool calls and outputs go to prod_ops.ai_log with the incident id (restricted like the Genie log), so a reviewer can see why the agent said what it said.
4
Tested before trusted, measured after. The DataTwin faults are the agents' exam: the triage agent must name the planted cause for each fault before it is switched on, and again after every change. In production, its cause is compared with the cause the person closes the incident with, and the hit rate is on the ops dashboard.
5
Models stay in the account. Databricks-hosted models or Amazon Bedrock in your AWS account.

Running it

Sizing, recovery, retention and the people

A design is only finished when it says how big it is, how it comes back after a disaster, how long it keeps things, and who runs it. The numbers below are our assumptions for a typical tyre maker; each one is a setting, so the design holds if they change.

Sizing and targets

WhatWe assumeWhat it means for the design
Plants7: five in India, two in Europe; three shiftsPlant is the first key for clustering, row filters and canary releases
Machines streamingAbout 2,000 curing presses and mixers, 20 signals each, one reading every 5 seconds: about 8,000 readings a second, 700 million a dayStreaming tables read Kinesis directly, trigger-once every 15 minutes; about 10 GB a day once compressed in Delta
Business dataSAP: about 200,000 order and delivery lines a day; 3,000 dealers sending daily sell-out; about 2,000 warranty claims a dayBatch every 30 minutes in plant hours, full nightly
Size after five yearsAbout 25 TB, mostly sensor historyBronze sensor data older than 3 years moves to cheaper storage
FreshnessA reading reaches silver within one hour (the manifest's target; in practice one 15-minute run plus the run itself); press-hour gold within 15 minutes after the hour; OEE ready by 06:30These are the tier-1 targets the alerts watch
People askingAbout 400 dashboard users, 50 queries at once at the peak (06:00 to 07:30, morning meetings)Dashboards answer in under 5 seconds for 95% of views; Genie in under 30 seconds
What the platform is notMachine alarms and safety interlocks stay on the plant floor (SiteWise Edge, the machines' own controls)The platform is for analysis and decisions, not for stopping a press

Streaming sensor data

1
One path for sensors. SiteWise or IoT Core at the plant edge, Amazon Kinesis Data Streams in Mumbai, a streaming table into bronze and on into silver, run trigger-once every 15 minutes (each run takes what has arrived since its checkpoint and stops; nothing runs continuously); each closed hour's sensor aggregates feed the window-recompute job, the only writer of the production fact.
2
Late readings have a limit. Streams wait 2 hours for late readings (an event-time watermark). Anything later lands in bronze and is picked up by the gold window recompute, so it is never lost.
3
Replay has a window. Kinesis keeps 7 days; bronze keeps everything after that. Checkpoints live in S3 per environment.
4
If the link to AWS drops, the edge buffers readings and sends them when the link is back (check current status for buffer size); OEE for those hours shows as incomplete until they arrive.

Recovery: what comes back, how fast, and what can be lost

Two words first. The recovery point (RPO) is how much recent data can be lost; the recovery time (RTO) is how long until people can work again. Different layers get different targets, because they cost different amounts to copy.

LayerHow it is copiedMost data lost (RPO)How it comes back
LandingS3 cross-region replication, same expiry rules (personal-data files still deleted after a day)15 minutesLets the lander replay the last minutes that had not reached bronze
BronzeS3 cross-region replication to Hyderabad (tokens only)15 minutesRegistered at the last version whose files have all arrived (replication does not keep order), then silver and gold catch up from bronze by run id
Silver, gold, recipes, token vaultIncremental Delta deep clone every 4 hours4 hoursRegistered on the clones; erasures since the last clone are replayed
Metadata database (Lakebase)Lakebase's own point-in-time restore in Mumbai; for tier-1, a read replica in Hyderabad (check current status)Seconds with the replica; without it, configuration is reloaded from Git and bookmarks are rebuilt from bronzePromote the replica, or reload from Git; watermarks and run log carry on
Unity CatalogNot copied: Hyderabad has its own catalog, built from GitNothing lostTerraform builds catalogs and catalog grants; the data-platform deploy adds schema grants, row filters, masks, metric views and Genie spaces
StreamsCheckpoints are not copied; a Kinesis stream in Hyderabad is built in advanceGap filled from the edge bufferThe edges switch to the Hyderabad stream; streams start from the last bronze row
SecretsSecrets Manager replicates every secret to HyderabadNothing lostMovers and the lander find their credentials in place
Code and configIn GitHubNothing lostDeployed as in any release
1
Declare. The head of IT, or their deputy, declares the disaster. Nobody else can.
2
Build. Terraform with dr.tfvars builds the workspace, catalogs, Kinesis stream and mover definitions in Hyderabad. The network path is already in place: the Transit Gateway in Hyderabad is peered, the Direct Connect gateway reaches both regions, and the EU plants' VPN tunnels to Hyderabad are configured but idle.
3
Reset the bookmarks. The Lakebase replica is seconds behind, but bronze can be up to 15 minutes behind, so each watermark is set back to the highest window actually present in Hyderabad bronze, minus the usual overlap. The lander then replays the gap from the copied landing files; nothing is marked loaded that is not there.
4
Lock, then secure. Readers are locked out. The data-platform deploy applies column tags from dataset_config first, then every grant, row filter and mask; the security test then checks one row filter and one masked column as plant-manager and analyst test identities. Nobody reads until it passes.
5
Open by tier. Tier-1 data products open first (target: 24 hours), the rest within 72 hours. Dashboards show a banner saying the platform is running from recovery.
6
Fail back. When Mumbai returns, clone the changed tables back, reconcile counts, switch the movers, and close the incident with a review.
7
Rehearse. Twice a year, against the clock, with the results in the ops dashboard.

How long each layer is kept

Assumed periods; your legal and finance teams confirm them. Retention jobs run nightly and log what they delete.

LayerKept for
Landing30 days; files with personal data 1 day
Bronze3 years in standard storage, then an archive class Spark can still read (S3 Glacier Instant Retrieval) until year 8, then deleted. Tokens only
Silver8 years for anything that supports the accounts (as Indian company law asks for books of account); 5 years for the rest
Gold10 years, for trends. Rows older than silver's retention are kept only as summaries without tokens, so an erasure never needs to rebuild them
Quarantine tables90 days after the steward's decision
Runtime and ops tables13 months live, then summarised
AI question log1 year
Audit bucket7 years, locked
Token vaultPer purpose: for warranty, the warranty period plus 3 years; then the entry is deleted and the tokens point to nobody

Master data: who owns the facts everything joins on

OEE is only right if each press's ideal cycle time and each plant's shift calendar are right. These are small tables with big effects, so each has a named owner.

Master dataComes fromOwner
Plant: country, time zone, financial calendarSAP plant master, plus a small YAML file for time zonePlant operations head
Press: line, ideal cycle time per materialMES press master; ideal cycle times from process engineering (YAML if not in MES)Process engineering head
Shift calendar and planned stopsMES, or a YAML file per plantEach plant manager
Material and compound familySAP material masterSAP master data team
Key mappings (SAP and MES material, dealer ids across systems)Mapping tables in silver, kept from SAP or YAMLThe steward of that domain; an unmapped key becomes 'unknown' and raises an alert
Defect codesThe quality systemQuality head

Adding a source: the checklist

"A new source is a YAML entry" is true for the pipeline. The checklist is everything around it; the source onboarder agent drafts most of it as one pull request.

1
Network path How the source reaches AWS (head office link, plant VPN, SaaS over a private endpoint).
2
Mover and secret Which mover, and its credentials in Secrets Manager.
3
Source and dataset YAML Load mode, keys, history type, schedule, freshness target and tier.
4
Contract The ODCS file, agreed with the source's owner.
5
People Data owner, steward and their deputies, and the CODEOWNERS entry.
6
Personal data Which columns, the purpose and retention of each, or a statement that there is none.
7
Quality rules The first rules, then profiling switched on to suggest more.
8
Master data Key mappings, and for a new plant its time zone, shift calendar and ideal cycle times.
9
Access For a new plant, the row-filter mapping (which groups see it).
10
Tests A DataTwin profile for the source, so its faults are tested from the first release.

Backfills

1
A backfill is a plan in Git. Dataset, window, reason, expected cost (from CostSherlock), reviewed like any change.
2
It runs on a clone first. In pre-prod, compared with prod, before it touches prod.
3
It never moves a watermark. A backfill is its own run type; the normal schedule carries on untouched.
4
It runs when it is cheap. Outside the 06:00 to 07:30 peak, with its own budget.
5
It is announced. The changed gold windows are logged as a restatement and shown on the dashboards.

Secrets

1
One home. AWS Secrets Manager holds every password and key reference; Databricks reads them through a Unity Catalog service credential or an IAM role (check current status), never a copy.
2
Rotated without people. Database passwords rotate every 90 days automatically.
3
Never in Git. The shift-left checks block a pull request that contains a secret.
4
No long-lived keys. CI signs in with OIDC; people sign in through your identity provider; personal access tokens are off.

The team

WhoSizeDoes
Platform teamAbout 6: a lead, three data engineers, a cloud engineer, an analytics engineerRuns the platform and the horizontal assets; on-call in a weekly rota of four
Security and privacyPart-time, from your existing teamsReviews access quarterly, owns erasure and breach response
Data ownersBusiness heads (plant operations, quality, sales, finance)Decide access, sign off metrics and releases in their area
StewardsOne per source, about two hours a weekRules, quarantine reviews, suggested rules
Data forumOwners, stewards and the platform lead, monthlyPriorities, metric changes, the incident and cost trends. This is the 'centre of excellence' the testing page refers to

Migration, sharing and ML

What you already have, who outside needs data, and models

Moving from Glue, Redshift and QuickSight

What you haveHow it movesDone when
Glue jobsRebuilt as metadata-driven pipelines, one subject at a time, each run in parallel with the old job for two weeksCounts and sums reconcile for two weeks; then the Glue job is switched off
RedshiftKept for existing reports; new gold on Databricks; Redshift reads our tables as Iceberg (UniForm) or through Delta Sharing while reports moveThe last report has moved; then the cluster is shut down
Glue Catalog and Lake FormationUnity Catalog is the one catalog for everything new; old Glue tables are read through catalog federation during the move (check current status)No new grants are made in Lake Formation from day one
QuickSightExisting reports repointed to the SQL warehouse; new ones on AI/BIEach report reads gold or metric views
AthenaKept for small one-off queries on S3Nothing to migrate
1
Inventory first. Every job, table, report and its users, with an owner and a move-or-retire decision.
2
One subject at a time. Production first, then quality, then sales, so each move is small and can be undone.
3
Never write twice. During a move the old system reads from the new one, not both from the source.
4
Dates for switching off. Every old system has an agreed shut-down date, or it never goes.

Sharing with partners

Every share is a data product: it has a contract, an owner and a filter to the partner's own rows, and its use is audited and reviewed every quarter.

PartnerWhat they getHowProtection
Car makers (OEMs)Quality and traceability of the tyres fitted to their cars, by batch and DOT codeDelta Sharing (an open protocol, readable without Databricks) of a gold view, filtered to that car makerNo personal data; contract and owner per share
DealersTheir own sell-out, stock and claim statusThe existing dealer portal reads an API over gold, or a Databricks AppEach dealer sees only their own rows
SuppliersQuality results for the lots they deliveredDelta SharingFiltered to the supplier
Auditors and regulatorsEvidence packs: access reviews, erasure log, release reportsExported on requestApproved by the data owner

Machine learning

Models use the same platform, catalog and release path as everything else; nothing is copied out to a separate ML system.

PartHow
FeaturesFeature tables in gold (prod_gold.features), built and tested like any gold table
ModelsTracked with MLflow and registered in Unity Catalog, with the same owners and grants
ServingBatch scoring for forecasts; Model Serving where an app needs an answer in seconds
First useWeekly demand forecast per material and region; later, predicting press breakdowns from sensor history
ChecksEach model is evaluated against a held-out period before release and watched for drift after (check current status for inference tables)

22 concerns, bound

What Databricks gives, what AWS gives, what we add

For each of the 22 cross-cutting concerns: reuse what Databricks does well, use AWS where the job sits outside Databricks, and fill only the gap. Every concern is switched on and configured from the metadata store, and every product sits behind a port (a fixed interface), so a product can be swapped without touching the pipelines.

Our main value is joining the dots: one run id links the quality results, lineage, run log, cost and audit for every number, even where every piece underneath is Databricks or AWS.

ConcernDatabricks coversDatabricks featureAWS pieceThe gap, and who fills it
X1 Metadata & controlpartialAsset Bundles, Unity Catalog tags, Lakebase (Databricks managed PostgreSQL) for the metadata store-YAML in Git → loader → metadata store → generated jobs. Ours (Common Ingestion Framework).
X2 OrchestrationfullLakeflow Jobs: schedules, file-arrival triggers, retriesEventBridge for events from outsideJobs generated from metadata, not clicked together. Ours.
X3 Contracts & schemapartialDelta schema enforcement and evolution; Auto Loader rescued dataGlue Schema Registry for streams, if usedContracts in the open ODCS standard, checked in CI and at run time. Ours.
X4 Data qualitypartialDeclarative Pipelines expectations; DQX (Databricks Labs)-Rules from metadata, quarantine, one results table. Ours, with DQX behind a port.
X5 ProfilingpartialDQX profiler; Lakehouse Monitoring on tier-1 gold; Unity Catalog classification for PII (check current status)-Profiling settings per dataset, profile compared with the last one, suggested rules as pull requests, all tied to the run id. Ours.
X6 Reconciliationnone--Counts and sums from source to gold, per run. Ours.
X7 LineagefullUnity Catalog table and column lineage-Lineage before Databricks (movers, source systems) from the run log and mover ids; joined to the run id. Ours.
X8 Logs, metrics, tracespartialSystem tables for jobs, queries, pipeline event logsCloudWatch for DMS, Kinesis, MSKOne run log with one run id across both. Ours.
X9 Freshness & SLOspartialJob duration alerts; freshness checks in monitoring (check current status)CloudWatch alarmsA freshness promise per table, declared in metadata and alerted. Ours.
X10 Cost & FinOpspartialBilling system tables, budget policies, tagsCost and Usage Report, Cost Explorer, BudgetsCost per data product and per run, with findings. CostSherlock.
X11 Security & accessfullUnity Catalog grants, row filters, column masks, ABAC (check current status)IAM roles, KMS, VPC, PrivateLinkGrants declared in metadata and applied by the pipeline. Ours.
X12 Privacy & PIIpartialColumn masks, tags, data classification (check current status)Macie scans landing buckets for personal data that slipped through; Comprehend finds names in free text such as warranty notesTokenise at the door, the token vault, erasure, suppression and retention jobs. Ours.
X13 Governance & semanticspartialUnity Catalog owners, comments, tags; Metric Views-Glossary and metric ownership workflow. Ours (OpenMetadata optional).
X14 Audit & run logpartialAudit system tablesCloudTrailRun log with a fingerprint of the config each run used. Ours.
X15 Replay & backfillpartialDelta time travel, MERGE, full refresh of pipelinesS3 versioningBackfill plans and watermark resets from metadata, never doubling a number. Ours.
X16 Environments & CI/CDfullAsset Bundles, Git folders, Terraform providerGitHub Actions with short-lived OIDC sign-in; separate accounts per environmentTemplates only. Ours.
X17 Testingnone--Unit, contract and data tests (pytest), shift-left checks, DataTwin worlds. Ours, open source.
X18 Resilience & DRpartialTime travel, deep clone, job retriesS3 replication and deep clone to Hyderabad (no AWS Backup on personal data)A written, tested restore drill. Ours.
X19 PerformancefullPhoton, liquid clustering, predictive optimization, serverless-Checks in CI and cost findings. Shift-left bundle + CostSherlock.
X20 Self-healingnone--Detect → fix proved on a pre-prod clone → pull request → verify → roll back. Ours.
X21 Agent assistancepartialDatabricks Assistant for codeBedrock, if preferred for modelsAgents that draft manifest entries, rules and docs as pull requests. Ours.
X22 Simulationnone--A pretend plant with planted faults (a press goes silent, a batch sent twice). DataTwin, ours.

Coverage is our judgement for a typical setup. Product names and preview status change often; items marked "check current status" should be confirmed before the deck goes out.

AWS native or Databricks

Which product for which job

You do not have to replace what already works. This is our recommendation per job, with the reason.

JobAWS nativeDatabricksVerdict
Plant-floor data collectionAWS IoT SiteWise, IoT Core, GreengrassAuto Loader / streaming tables read what SiteWise landsKeep SiteWise at the plant edge: it talks to the machines. Databricks takes the data from S3 or Kinesis for analytics.
Ingest from SAPAppFlow or a partner extractorSAP Business Data Cloud connector (check current status)Use an SAP-certified route (Datasphere or a certified extractor) behind one port; never read SAP tables directly.
Ingest from databases (CDC)AWS DMSLakeflow ConnectDMS for v1, into S3 landing, so every source passes the lander's tokenising and run log. Lakeflow Connect writes straight into tables; revisit per source later.
Batch and streaming pipelinesGlue jobs, EMRLakeflow Declarative Pipelines and JobsPursue Databricks: one engine for batch and streaming, built-in quality rules and lineage.
Warehouse for BIRedshiftDatabricks SQL warehouseIf Redshift is heavily used, keep it and let it read our tables as Iceberg (UniForm) or through Delta Sharing (check current status); new work on Databricks SQL. See the migration plan.
Ad hoc queries on S3AthenaSQL warehouseKeep Athena for small one-off queries; it can read the tables as Iceberg (UniForm).
Catalog and accessGlue Catalog, Lake FormationUnity CatalogPursue Unity Catalog as the one catalog: lineage, masks and audit in one place. Avoid running two access models.
ML and AISageMaker, BedrockMosaic AI: Vector Search, Model ServingUse Databricks for ML on lakehouse data; call Bedrock models from Databricks where the company already uses Bedrock.
DashboardsQuickSight (now part of Amazon Quick Suite, check current status)AI/BI dashboards and GenieKeep existing QuickSight reports; build new ones on AI/BI over the same metric views.
Cost viewCost Explorer, CURBilling system tablesUse both; CostSherlock joins them per data product.

How we would start

One thin slice first

Crawl · 4 to 6 weeks

One slice end to end in your AWS account: curing-press sensor data and MES production records through silver to one gold fact, one metric view (OEE or scrap rate), one Genie space for plant managers with benchmark questions.

You see: the whole path working, with quality, lineage and cost on from day one.

Walk · next quarter

Three to five more sources (SAP sales and stock, quality lab, warranty claims, dealer sell-out). The horizontal assets on by default. CostSherlock reporting cost per data product.

You see: each new source added as a YAML entry, not a project.

Run · after that

A small platform team owns the assets; domain teams build data products on them. The Conversation UI in staff apps. Agents propose fixes as pull requests.

You see: the tenth data product faster and safer than the first.

Open decisions

What must be settled before or during the first slice

Most items marked "check current status" on this page are product details that do not change the design. These ten do: each has a fallback, an owner and a date.

Open questionWhy it mattersWho decidesBy when
Can serverless compute reach the plant network privately?Decides whether the puller runs serverless or on classic compute (the metadata store, on Lakebase, needs no network setup)Platform leadBefore the crawl starts
Do row filters hold through metric views, and can outside BI tools query them?The whole Genie and dashboard security rests on itPlatform lead and analytics engineer, by testCrawl, week 2
Are tag-based access policies (ABAC) available in Mumbai?If not, the deploy generates a mask per table from the same tagsPlatform leadCrawl, week 1
Which route out of SAP?Licence and support terms decide itSAP team and procurementBefore the walk phase
What is the legal basis for EU personal data in India?Until decided, EU sources load without personal columnsLegal teamBefore the first EU source
Does Databricks run in Hyderabad (ap-south-2)?The recovery plan builds the workspace there. If not, the data copies stay in Hyderabad and compute is rebuilt in the nearest supported region, with legal sign-off for data leaving IndiaPlatform lead and legal teamBefore go-live
Which personal-data detector reads German free text?Until decided, EU warranty notes are held out of bronzePrivacy lead and platform leadBefore the first EU source
Is Lakebase available in Mumbai, with a read replica in Hyderabad?Sets where the metadata store runs and its recovery point: seconds with the replica, or a reload from Git and bronze without itPlatform leadBefore go-live
What are the DPDP Rules timelines for breach notice?Fills in the breach runbookLegal teamBefore go-live
Can Genie be embedded and driven by API?Decides how much of the Conversation UI we buildProduct ownerWalk phase

How we tested this design

Twelve situations, walked through before anything was built

We tested the design on paper first. Two reviewers who had not written it worked from the page alone. One walked twelve real situations through it step by step; the other checked every section against every other and against 25 topics a complete design must cover. They found 3 situations that failed outright, about 30 places where two sections disagreed, and several topics that were missing (sizing, recovery, migration, sharing). Every finding was fixed in the page above. A third reviewer then re-tested the three failures and looked for new contradictions; that round found 27 more, mostly in recovery and privacy details, and those are fixed too.

SituationFirst testAfter fixesWhat changed
A new EU plant's MES must reach OEE within a sprintPasses with gapsPassesEU network path, master data owners, onboarding checklist, EU legal basis
SAP renames a column without warningPasses with gapsPassesContract check fixed at silver; replay from landing, never re-extract from SAP; freshness shown on every metric
A press goes silent for 3 hours on the night shiftFailsPasses (round 2 fixed how OEE treats incomplete hours)Gold writes incomplete rows instead of stopping; a completeness measure; a silent-press alert; paging by tier
A warranty file arrives two days latePasses with gapsPassesRestatements logged and shown; freshness alerts grouped by date; warranty tier and target stated
A consumer in Germany asks to be erasedFailsPasses once two open decisions are settled: the legal basis and a German-capable detectorVault in its own catalog; landing purged in a day; recovery copies erased; suppression list; free text scanned; no restore across an erasure
Scrap rate is redefined to exclude trial batchesPasses with gapsPassesOne source for scrap rate; batch type in gold; definition changes dated; owner updates benchmarks
A release is wrong for one plantPasses with gapsPassesRecompute that plant only; canary for a full day; business banner; platform lead decides
The Mumbai region is down for a dayFailsPasses once Databricks in Hyderabad is confirmedRecovery per layer with targets; security deployed before readers; metadata database copied; network ready; who declares
A plant manager asks Genie for all plantsPasses with gapsPasses, once the metric-view test is greenRow security as a release test; 'for your plant' wording; AI log restricted
An engineer joins, a recipe keeper leavesPasses with gapsPassesOwners checked against the identity provider nightly; deputies; personal access tokens off; pre-prod sees results only
The bill doubles overnightPasses with gapsPassesCost-jump alert on the whole account; runbook to pause and shrink within the hour; cost as a cause
An agent's diagnosis is wrongPasses with gapsPassesA person confirms the diagnosis; proof on a pre-prod clone; full gates for limits and watermarks; hit rate tracked

Four situations pass only once an open decision is settled: the EU legal basis, a German-capable detector, Databricks in Hyderabad, and the metric-view security test. All are in the open decisions above.

Questions

Questions leaders ask

Does our data leave our AWS account?

No. Classic compute runs in your account and your VPC; serverless compute runs in Databricks' serverless plane in the same region. All tables live in your S3 buckets, encrypted with KMS keys you own. The Databricks control plane manages the workspace; for serverless compute, check the current data-residency terms for your region.

Why Databricks if we already have Glue, EMR and Redshift?

One engine for batch and streaming, one catalog (Unity Catalog) for access, lineage and audit, and AI/BI with Genie on the same governed data. You do not have to switch everything at once: keep what works and put new work on the lakehouse.

What is a metric view?

A definition of a business metric (for example, scrap rate) stored once in Unity Catalog, with its formula, dimensions and filters. Dashboards, Genie and the Conversation UI all use it, so the number is the same everywhere.

What are horizontal assets?

Parts built once and reused by every pipeline and data product: ingestion, metadata, quality, cost, testing and the conversation layer. They are what makes the tenth data product much cheaper than the first.

How are compound recipes and personal data protected?

Recipes and formulation data sit in a restricted catalog with its own storage and encryption key, readable only by a named group. Personal data of dealers, fleet drivers and consumers is swapped for tokens as it enters the platform; the real values live only in a separate token vault, and erasing a person means deleting their vault entry, blocking them from being loaded again, and erasing the recovery copies too. Commercial columns such as prices are masked by Unity Catalog for everyone outside finance and sales, in every tool, not only in dashboards. All data stays in India (Mumbai, with recovery in Hyderabad); personal data from the EU travels only as the law allows (see the privacy section), and a separate EU platform is added only if your legal team requires it. Access is audited in system tables and CloudTrail.

Next

Where to go next

↑ The abstract architecture · Spec-driven development for data platforms · Medallion layers · Shift-left checks · DataTwin