Home › Data platform architecture › Mind map

Data engineering · Data platform

Data platform architecture mind map: every concern, on every platform

A data platform is the same set of concerns whatever you build it on: it must run, be trusted, be seen, be guarded and, one day, help fix itself. What changes between Azure, Databricks, AWS, Google Cloud and an open-source stack is the product that fills each concern. This page puts all 24 concerns in one table, then cuts each one three ways: what the platform gives you, what we write once for all of them, and what each platform still has to implement.

Written 20 November 2026. The Azure and Databricks entries in column Z come from real builds: a 3-day run passed on Azure Databricks that day. AWS entries come from the repository's files, not from a fresh run. GCP has no binding at all, so everything in its column is a plan. Product names for concerns the rest of this site already maps are taken from there; the others were written from general knowledge and are not checked against each vendor's current documentation.

The map

Data platform24 concerns, 5 familiesRunDoes it run safely, every day?X1 Metadata and control planeX2 OrchestrationX15 Replay, backfill and idempotencyX16 Environments, CI/CD and releaseX17 TestingX18 Resilience and recoveryX23 Layout: catalogs and schemasX24 StreamingTrustCan I trust the number?X3 Contracts and schema evolutionX4 Data qualityX5 ProfilingX6 ReconciliationX9 Freshness and SLOsSeeCan I see what happened and what it cost?X7 LineageX8 ObservabilityX10 Cost and FinOpsX19 PerformanceGuardWho may see and change what?X11 Security and accessX12 Privacy and personal dataX13 Governance, catalog and semanticsX14 Audit and run logSelfCan it help fix itself, safely?X20 Self-healingX21 Agent assistanceX22 Simulation

The table

X · The platformWhat the cloud or data platform gives out of the box. Different on every stack, and often just a building block.
Y · The horizontal assetWhat we write once and reuse unchanged on every platform: the manifest, the rules, the stages, the checks, the simulated world. It is the same code on Azure as on AWS.
Z · Still to implementThe thin layer that each platform still needs before the asset can run there: an adapter, a job definition, a renderer. This is the real work of moving platform.
The concernWhich product fills it, on each platformThe three-way cut
Concern familyConcernAzureDatabricksAWSGCPOpen stackX · What the platform providesY · Horizontal asset (written once)Z · What each platform still has to implement
RunDoes it run safely, every day?X1 · Metadata and control planeEverything declared in one manifest, validated, stored.Azure Database for PostgreSQL / Azure SQLLakebase (managed Postgres)RDS for PostgreSQLCloud SQL for PostgreSQLPostgreSQLA managed database to hold the manifest and the run records.The manifest format, the validator and loader, and the stored procedures that read and write it.A connection adapter per database. Azure: Lakebase project found and used (19 Nov). GCP: none. Open: Postgres adapter exists.
X2 · OrchestrationWhat runs when, in what order, with retries.Data Factory / Fabric pipelinesLakeflow JobsMWAA (Airflow) / Step FunctionsCloud Composer (Airflow)Airflow or DagsterA scheduler that starts tasks, retries them and shows their state.The day loop, leases and bookmarks that keep order safe; one job entry per hop.A job or pipeline definition per platform. Azure Databricks: generated bundle, ran 3 days. AWS: a realm with a deployment record exists. GCP: none.
X15 · Replay, backfill and idempotencyRe-running never doubles a number.Delta MERGE; Data Factory re-runDelta MERGE; Auto Loader checkpoints; repair runIceberg MERGE; Glue job bookmarksBigQuery MERGEDelta or Iceberg MERGE; Airflow backfillA table format that can merge, and a way to re-run a failed task.Window ledgers, withdraw-and-rebuild, housekeeping, one writer per table.A table-writer adapter per table format. Delta: done. Iceberg on AWS: not checked here. BigQuery: none.
X16 · Environments, CI/CD and releaseDev to test to production, infrastructure as code.Bicep or Terraform; GitHub Actions or Azure PipelinesDeclarative Automation Bundles; GitHub ActionsTerraform or CDK; GitHub Actions + CodeDeployTerraform; GitHub Actions + Cloud BuildDocker Compose; GitHub ActionsRepositories, build runners and a way to describe infrastructure.The generated bundle from one engine file; the release gate's logic; naming rules per environment.One infrastructure module set per cloud. Azure: dev environment applied (45 resources). AWS: a realm exists. GCP: none. Build workflows are not in the public snapshot.
X17 · TestingUnit, contract, data and simulation tests.Azure Pipelines test tasksDatabricks Connect + bundles with pytestCodeBuild with pytestCloud Build with pytestpytest, chispa, dbt testsA runner that executes tests on every change. No platform tests your data rules for you.The table-driven tests, the simulated world, the daily end-to-end test (3 days, scale 0.1).A contract-test run per adapter against the real service. Azure storage: passed on a real account. Others: fakes only or not run.
X18 · Resilience and recoveryRetries, retention, time travel.ADLS soft delete and geo-redundancy; Delta time travelDelta time travel and RESTORE; job retriesS3 versioning; Iceberg snapshotsCloud Storage versioning; BigQuery time travelMinIO versioning; Delta or Iceberg snapshotsObject versioning, table history and automatic retries.The source guard, landing retention and the recovery routines.Turn on and prove versioning and retention per account. Azure: versioning not yet checked. GCP: none.
X23 · Layout: catalogs and schemasFrom a table's name you know its environment, layer and domain.Storage account per environment, container per layerUnity Catalog: <env>_<layer>.<domain>.<table>Glue databases with S3 prefixesBigQuery datasetsFolder paths and Postgres schemasA namespace for tables: catalog, schema or dataset.One naming rule (ADR-012) and the checker that tests table names.Map the rule onto each platform's namespace. Azure: written into Terraform. GCP: none.
X24 · StreamingA second speed on the same line, only where a decision needs it.Event Hubs / Fabric EventstreamStructured StreamingKinesis / Amazon MSKPub/Sub + DataflowKafka or RedpandaA durable event log and a stream engine.Trigger-once stream pipeline, the same checks and ledgers as batch.A stream adapter per log. AWS: Kinesis exists. Azure: Event Hubs not built (AZ-4). GCP: none. Open: local stream, laptop only.
TrustCan I trust the number?X3 · Contracts and schema evolutionA table says what it promises; a source changing shape is caught.Delta schema enforcement in FabricDelta schema enforcement and constraintsGlue Schema Registry; Iceberg schema evolutionBigQuery schema checksdbt model contracts; Iceberg schema evolutionThe table format refuses a write with the wrong columns.The contract check and column rules read from the manifest.Nothing per cloud beyond the table writer. The registry formats the lander accepts are not all readable (AZ-9).
X4 · Data qualityRules on rows and tables; bad rows set aside with a reason.Purview data qualityPipeline expectationsGlue Data QualityDataplex data qualityGreat Expectations or SodaA rule engine that can pass, warn or fail a table.The quality stage, the quarantine layer and the rule catalog (DAC) with tests.A quarantine write per format. The results table is not written in a normal run yet.
X5 · ProfilingKnowing what the data looks like, so change is visible.Purview profilingLakehouse MonitoringGlue DataBrewDataplex profilingydata-profiling or Great ExpectationsColumn statistics on demand.Profile and drift functions that compare today with history.None per cloud: the code runs on rows. Wire the platform profile as a second opinion where wanted.
X6 · ReconciliationCounts and sums agree from source to gold.No native featureNo native featureNo native featureNo native featureSoda or datacompy checksNothing. Platforms offer SQL, not the idea of a reconciled hop.The compare function and live-source reconciliation, per dataset per day.A read adapter for each live source type. Azure: change capture not validated (AZ-6).
X9 · Freshness and SLOsIs the data as new as promised.Azure Monitor alertsData quality monitoring and job alertsCloudWatch alarms; Glue DQ freshnessDataplex freshness ruledbt source freshness or SodaAlerts on time and late jobs.Freshness checks and the success scorecard (20 rows).Route the alert to the platform's alert service. Not built on any cloud yet.
SeeCan I see what happened and what it cost?X7 · LineageWhere each table came from and what it feeds.Purview lineageUnity Catalog lineageAmazon DataZone lineageDataplex lineageOpenLineage + MarquezAutomatic lineage inside the platform's own engines.A reader of the platform's lineage tables, and OpenLineage events built in tests.A reader per lineage store. Databricks: done. Azure, AWS, GCP: none.
X8 · ObservabilityOne run followable end to end.Azure Monitor + App InsightsSystem tables + MLflow tracingCloudWatch + X-RayCloud Logging + TraceOpenTelemetry + GrafanaLogs, metrics and traces collected and searchable.The run recorder, trace helper and operations page, written once.An OpenTelemetry exporter pointed at each cloud's backend. Exists as an adapter; run on none of the new clouds.
X10 · Cost and FinOpsWho spent what, against which budget.Cost ManagementBilling system tablesCost Explorer + CURBilling export to BigQueryQuery plans and run timesBilling data and budgets with alerts.Cost tags on every job. A separate tool reads the cost; the platform only tags.A budget per environment. Azure: $20 a month budget applied. Others: not set. Reading the bill is not built.
X19 · PerformanceFile sizes, layout, compaction, right-sized compute.Fabric Spark autoscale; Delta OPTIMIZEPredictive optimization; liquid clusteringIceberg compaction (S3 Tables); Glue autoscalingBigQuery partitioning and clusteringOPTIMIZE jobs; Spark adaptive executionAutoscaling compute and some automatic file maintenance.Hop timing, the daily time budget and the table store layout.A maintenance job per format. Measured on Azure Databricks: 5.7 minutes a day at scale 0.1 (target 6).
GuardWho may see and change what?X11 · Security and accessWho may read or change each layer, granted on purpose.Entra ID + RBAC; Purview policiesUnity Catalog grants, row filters, column masksIAM + Lake FormationIAM + BigQuery row and column securityKeycloak + Postgres roles, Ranger or OPAIdentity and permission enforcement on the data.The access model: who may do what, planned once and rendered into each platform's grants.A renderer per platform. Databricks: generated and tested locally, not applied. Azure, AWS, GCP: none applied.
X12 · Privacy and personal dataFind, mask, keep only as long as allowed.Purview sensitivity labelsUnity Catalog tags + column masksMacie + Lake Formation tagsSensitive Data ProtectionPresidioA scanner that finds personal data, and masks.Keyed tokens at landing, a separate store with a one-day purge for personal data, restricted columns declared.A separate personal-data account and purge rule per cloud. Azure: written and applied. Others: none.
X13 · Governance, catalog and semanticsOwners, glossary, metric definitions.Microsoft Purview; Power BI semantic modelsUnity Catalog; metric viewsGlue Data Catalog; dbt semantic layerDataplex; LookerOpenMetadata; Cube CoreA catalog of tables and a place for metric definitions.Metric definitions as data, rendered into each engine; the question-and-answer checks.A metric renderer per semantic layer. Databricks: done. Others: not built.
X14 · Audit and run logA record of every run and every change.Azure Activity logSystem tables (audit)CloudTrailCloud Audit LogsPostgres run log + OpenTelemetryImmutable logs of who did what in the cloud.The run log schema and recorder: what ran, with which config fingerprint.None beyond the metadata database. Cloud audit logs are read separately.
SelfCan it help fix itself, safely?X20 · Self-healingDetect, fix in development, deploy, verify, roll back.Retries and alerts onlyRepair run, retriesRetries and alerts onlyRetries and alerts onlyRetries and alerts onlyRetries and notifications. No platform repairs a data fault by itself.Not built. Only the release gate's logic is written.Everything. No code exists on any platform.
X21 · Agent assistanceAgents draft manifests, rules and docs.Copilot in FabricDatabricks AssistantAmazon QGemini in BigQueryYour own agent (see Agent Harness)A coding assistant inside the vendor's own tools.Not built. The specs, rules and manifests are written so an agent can read them.Everything. No agent code exists here.
X22 · SimulationA fake world that breaks the platform on purpose.NonedbldatagenNoneNoneFaker or SDVRandom-data generators. No platform simulates a company with planted faults.The world engine: a simulated company, 12 planted faults and the true totals; same on every platform.A mover per platform to land the world's files. Azure Databricks: ran. AWS: a realm exists. GCP: none.

How much of each concern comes from where

For each concern, how much of the work does the platform give you (X), how much do we write once for every platform (Y), and how much does each platform still have to implement (Z)? On average, by our judgement, about 34% comes from the platform, 44% is the horizontal asset we write once, and 22% is what each platform still has to implement.

X · the platform givesY · we write once (horizontal)Z · each platform still implementsRunX1 · Metadata and control plane305515X2 · Orchestration503020X15 · Replay, backfill and idempotency305515X16 · Environments, CI/CD and release502525X17 · Testing156520X18 · Resilience and recovery503515X23 · Layout: catalogs and schemas256015X24 · Streaming503020TrustX3 · Contracts and schema evolution306010X4 · Data quality256015X5 · Profiling355510X6 · Reconciliation8515X9 · Freshness and SLOs305515SeeX7 · Lineage602515X8 · Observability454015X10 · Cost and FinOps502030X19 · Performance503020GuardX11 · Security and access453520X12 · Privacy and personal data305020X13 · Governance, catalog and semantics404020X14 · Audit and run log503515SelfX20 · Self-healing101575X21 · Agent assistance201070X22 · Simulation8015

Our judgement, not a measurement: a typical platform of the five, the work of having the concern running, each bar summing to 100. The platforms differ (Databricks hands over more lineage and access than an open stack does), so read the bars as the usual case, not as one platform's score.

What this shows. Read down column Y: most of the platform is horizontal, written once. Read column Z: the work per platform is small and concrete (adapters and definitions), not a rewrite. The expensive rows are the ones where X says "nothing": reconciliation (X6), self-healing (X20) and simulation (X22). No cloud sells them, so the horizontal asset is the whole answer, or, for X20 and X21, there is no answer yet.

What to check before relying on this

Where to go next