Home › Data platform architecture › Mind map
Data engineering · Data platform
Data platform architecture mind map: every concern, on every platform
A data platform is the same set of concerns whatever you build it on: it must run, be trusted, be seen, be guarded and, one day, help fix itself. What changes between Azure, Databricks, AWS, Google Cloud and an open-source stack is the product that fills each concern. This page puts all 24 concerns in one table, then cuts each one three ways: what the platform gives you, what we write once for all of them, and what each platform still has to implement.
Written 20 November 2026. The Azure and Databricks entries in column Z come from real builds: a 3-day run passed on Azure Databricks that day. AWS entries come from the repository's files, not from a fresh run. GCP has no binding at all, so everything in its column is a plan. Product names for concerns the rest of this site already maps are taken from there; the others were written from general knowledge and are not checked against each vendor's current documentation.
The map
The table
| The concern | Which product fills it, on each platform | The three-way cut | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Concern family | Concern | Azure | Databricks | AWS | GCP | Open stack | X · What the platform provides | Y · Horizontal asset (written once) | Z · What each platform still has to implement |
| RunDoes it run safely, every day? | X1 · Metadata and control planeEverything declared in one manifest, validated, stored. | Azure Database for PostgreSQL / Azure SQL | Lakebase (managed Postgres) | RDS for PostgreSQL | Cloud SQL for PostgreSQL | PostgreSQL | A managed database to hold the manifest and the run records. | The manifest format, the validator and loader, and the stored procedures that read and write it. | A connection adapter per database. Azure: Lakebase project found and used (19 Nov). GCP: none. Open: Postgres adapter exists. |
| X2 · OrchestrationWhat runs when, in what order, with retries. | Data Factory / Fabric pipelines | Lakeflow Jobs | MWAA (Airflow) / Step Functions | Cloud Composer (Airflow) | Airflow or Dagster | A scheduler that starts tasks, retries them and shows their state. | The day loop, leases and bookmarks that keep order safe; one job entry per hop. | A job or pipeline definition per platform. Azure Databricks: generated bundle, ran 3 days. AWS: a realm with a deployment record exists. GCP: none. | |
| X15 · Replay, backfill and idempotencyRe-running never doubles a number. | Delta MERGE; Data Factory re-run | Delta MERGE; Auto Loader checkpoints; repair run | Iceberg MERGE; Glue job bookmarks | BigQuery MERGE | Delta or Iceberg MERGE; Airflow backfill | A table format that can merge, and a way to re-run a failed task. | Window ledgers, withdraw-and-rebuild, housekeeping, one writer per table. | A table-writer adapter per table format. Delta: done. Iceberg on AWS: not checked here. BigQuery: none. | |
| X16 · Environments, CI/CD and releaseDev to test to production, infrastructure as code. | Bicep or Terraform; GitHub Actions or Azure Pipelines | Declarative Automation Bundles; GitHub Actions | Terraform or CDK; GitHub Actions + CodeDeploy | Terraform; GitHub Actions + Cloud Build | Docker Compose; GitHub Actions | Repositories, build runners and a way to describe infrastructure. | The generated bundle from one engine file; the release gate's logic; naming rules per environment. | One infrastructure module set per cloud. Azure: dev environment applied (45 resources). AWS: a realm exists. GCP: none. Build workflows are not in the public snapshot. | |
| X17 · TestingUnit, contract, data and simulation tests. | Azure Pipelines test tasks | Databricks Connect + bundles with pytest | CodeBuild with pytest | Cloud Build with pytest | pytest, chispa, dbt tests | A runner that executes tests on every change. No platform tests your data rules for you. | The table-driven tests, the simulated world, the daily end-to-end test (3 days, scale 0.1). | A contract-test run per adapter against the real service. Azure storage: passed on a real account. Others: fakes only or not run. | |
| X18 · Resilience and recoveryRetries, retention, time travel. | ADLS soft delete and geo-redundancy; Delta time travel | Delta time travel and RESTORE; job retries | S3 versioning; Iceberg snapshots | Cloud Storage versioning; BigQuery time travel | MinIO versioning; Delta or Iceberg snapshots | Object versioning, table history and automatic retries. | The source guard, landing retention and the recovery routines. | Turn on and prove versioning and retention per account. Azure: versioning not yet checked. GCP: none. | |
| X23 · Layout: catalogs and schemasFrom a table's name you know its environment, layer and domain. | Storage account per environment, container per layer | Unity Catalog: <env>_<layer>.<domain>.<table> | Glue databases with S3 prefixes | BigQuery datasets | Folder paths and Postgres schemas | A namespace for tables: catalog, schema or dataset. | One naming rule (ADR-012) and the checker that tests table names. | Map the rule onto each platform's namespace. Azure: written into Terraform. GCP: none. | |
| X24 · StreamingA second speed on the same line, only where a decision needs it. | Event Hubs / Fabric Eventstream | Structured Streaming | Kinesis / Amazon MSK | Pub/Sub + Dataflow | Kafka or Redpanda | A durable event log and a stream engine. | Trigger-once stream pipeline, the same checks and ledgers as batch. | A stream adapter per log. AWS: Kinesis exists. Azure: Event Hubs not built (AZ-4). GCP: none. Open: local stream, laptop only. | |
| TrustCan I trust the number? | X3 · Contracts and schema evolutionA table says what it promises; a source changing shape is caught. | Delta schema enforcement in Fabric | Delta schema enforcement and constraints | Glue Schema Registry; Iceberg schema evolution | BigQuery schema checks | dbt model contracts; Iceberg schema evolution | The table format refuses a write with the wrong columns. | The contract check and column rules read from the manifest. | Nothing per cloud beyond the table writer. The registry formats the lander accepts are not all readable (AZ-9). |
| X4 · Data qualityRules on rows and tables; bad rows set aside with a reason. | Purview data quality | Pipeline expectations | Glue Data Quality | Dataplex data quality | Great Expectations or Soda | A rule engine that can pass, warn or fail a table. | The quality stage, the quarantine layer and the rule catalog (DAC) with tests. | A quarantine write per format. The results table is not written in a normal run yet. | |
| X5 · ProfilingKnowing what the data looks like, so change is visible. | Purview profiling | Lakehouse Monitoring | Glue DataBrew | Dataplex profiling | ydata-profiling or Great Expectations | Column statistics on demand. | Profile and drift functions that compare today with history. | None per cloud: the code runs on rows. Wire the platform profile as a second opinion where wanted. | |
| X6 · ReconciliationCounts and sums agree from source to gold. | No native feature | No native feature | No native feature | No native feature | Soda or datacompy checks | Nothing. Platforms offer SQL, not the idea of a reconciled hop. | The compare function and live-source reconciliation, per dataset per day. | A read adapter for each live source type. Azure: change capture not validated (AZ-6). | |
| X9 · Freshness and SLOsIs the data as new as promised. | Azure Monitor alerts | Data quality monitoring and job alerts | CloudWatch alarms; Glue DQ freshness | Dataplex freshness rule | dbt source freshness or Soda | Alerts on time and late jobs. | Freshness checks and the success scorecard (20 rows). | Route the alert to the platform's alert service. Not built on any cloud yet. | |
| SeeCan I see what happened and what it cost? | X7 · LineageWhere each table came from and what it feeds. | Purview lineage | Unity Catalog lineage | Amazon DataZone lineage | Dataplex lineage | OpenLineage + Marquez | Automatic lineage inside the platform's own engines. | A reader of the platform's lineage tables, and OpenLineage events built in tests. | A reader per lineage store. Databricks: done. Azure, AWS, GCP: none. |
| X8 · ObservabilityOne run followable end to end. | Azure Monitor + App Insights | System tables + MLflow tracing | CloudWatch + X-Ray | Cloud Logging + Trace | OpenTelemetry + Grafana | Logs, metrics and traces collected and searchable. | The run recorder, trace helper and operations page, written once. | An OpenTelemetry exporter pointed at each cloud's backend. Exists as an adapter; run on none of the new clouds. | |
| X10 · Cost and FinOpsWho spent what, against which budget. | Cost Management | Billing system tables | Cost Explorer + CUR | Billing export to BigQuery | Query plans and run times | Billing data and budgets with alerts. | Cost tags on every job. A separate tool reads the cost; the platform only tags. | A budget per environment. Azure: $20 a month budget applied. Others: not set. Reading the bill is not built. | |
| X19 · PerformanceFile sizes, layout, compaction, right-sized compute. | Fabric Spark autoscale; Delta OPTIMIZE | Predictive optimization; liquid clustering | Iceberg compaction (S3 Tables); Glue autoscaling | BigQuery partitioning and clustering | OPTIMIZE jobs; Spark adaptive execution | Autoscaling compute and some automatic file maintenance. | Hop timing, the daily time budget and the table store layout. | A maintenance job per format. Measured on Azure Databricks: 5.7 minutes a day at scale 0.1 (target 6). | |
| GuardWho may see and change what? | X11 · Security and accessWho may read or change each layer, granted on purpose. | Entra ID + RBAC; Purview policies | Unity Catalog grants, row filters, column masks | IAM + Lake Formation | IAM + BigQuery row and column security | Keycloak + Postgres roles, Ranger or OPA | Identity and permission enforcement on the data. | The access model: who may do what, planned once and rendered into each platform's grants. | A renderer per platform. Databricks: generated and tested locally, not applied. Azure, AWS, GCP: none applied. |
| X12 · Privacy and personal dataFind, mask, keep only as long as allowed. | Purview sensitivity labels | Unity Catalog tags + column masks | Macie + Lake Formation tags | Sensitive Data Protection | Presidio | A scanner that finds personal data, and masks. | Keyed tokens at landing, a separate store with a one-day purge for personal data, restricted columns declared. | A separate personal-data account and purge rule per cloud. Azure: written and applied. Others: none. | |
| X13 · Governance, catalog and semanticsOwners, glossary, metric definitions. | Microsoft Purview; Power BI semantic models | Unity Catalog; metric views | Glue Data Catalog; dbt semantic layer | Dataplex; Looker | OpenMetadata; Cube Core | A catalog of tables and a place for metric definitions. | Metric definitions as data, rendered into each engine; the question-and-answer checks. | A metric renderer per semantic layer. Databricks: done. Others: not built. | |
| X14 · Audit and run logA record of every run and every change. | Azure Activity log | System tables (audit) | CloudTrail | Cloud Audit Logs | Postgres run log + OpenTelemetry | Immutable logs of who did what in the cloud. | The run log schema and recorder: what ran, with which config fingerprint. | None beyond the metadata database. Cloud audit logs are read separately. | |
| SelfCan it help fix itself, safely? | X20 · Self-healingDetect, fix in development, deploy, verify, roll back. | Retries and alerts only | Repair run, retries | Retries and alerts only | Retries and alerts only | Retries and alerts only | Retries and notifications. No platform repairs a data fault by itself. | Not built. Only the release gate's logic is written. | Everything. No code exists on any platform. |
| X21 · Agent assistanceAgents draft manifests, rules and docs. | Copilot in Fabric | Databricks Assistant | Amazon Q | Gemini in BigQuery | Your own agent (see Agent Harness) | A coding assistant inside the vendor's own tools. | Not built. The specs, rules and manifests are written so an agent can read them. | Everything. No agent code exists here. | |
| X22 · SimulationA fake world that breaks the platform on purpose. | None | dbldatagen | None | None | Faker or SDV | Random-data generators. No platform simulates a company with planted faults. | The world engine: a simulated company, 12 planted faults and the true totals; same on every platform. | A mover per platform to land the world's files. Azure Databricks: ran. AWS: a realm exists. GCP: none. | |
How much of each concern comes from where
For each concern, how much of the work does the platform give you (X), how much do we write once for every platform (Y), and how much does each platform still have to implement (Z)? On average, by our judgement, about 34% comes from the platform, 44% is the horizontal asset we write once, and 22% is what each platform still has to implement.
Our judgement, not a measurement: a typical platform of the five, the work of having the concern running, each bar summing to 100. The platforms differ (Databricks hands over more lineage and access than an open stack does), so read the bars as the usual case, not as one platform's score.
What to check before relying on this
- GCP. No cloud-layer file exists for it. Treat the whole column as a proposal and confirm each service against Google's current documentation and region availability.
- Product names. Cells for concerns that the site already maps agree with the architecture page. The rest were written without re-reading each vendor's page, and product names change often.
- Column Z status words such as "done" or "none" describe our own builds as of 20 November 2026. They are not a statement about the products.
- Prices and tiers. Not shown on purpose. Get them from each vendor for your volume.
Where to go next
- All 24 concerns, grouped by the question they answer.
- Data platform architecture, with the diagrams.
- Our platform, picture to code, the same concerns down to the code.
- The same kind of map for the agent harness.