AI Universe · Data

Data zoom 2

What does the AI know? 5 concept buckets. Each one is a job you can learn once; underneath it, who does that job on each platform.

Concept

Ingestion

Bringing data in from source systems, reliably and only once.

batch and streamingchange data captureschema drift

Physical: who does this job, where

Open sourceAirbyte, Debezium, Kafka
AzureData Factory, Fabric pipelines, Event Hubs
AWSGlue, DMS, Kinesis
GCPDataflow, Datastream, Pub/Sub
VendorsFivetran, Databricks Lakeflow

Concept

Lakehouse layers

Raw → clean → business-shaped data (bronze, silver, gold).

table formatstransformationsdimensional models

Physical: who does this job, where

Open sourceDelta Lake, Iceberg, Spark, dbt
AzureFabric OneLake, Azure Databricks
AWSS3 + Iceberg, EMR, Redshift
GCPBigQuery, BigLake
VendorsDatabricks, Snowflake

Concept

Semantic layer

One agreed definition of every business number, so every tool gives the same answer.

metrics and measurescertified definitionsaccess rules

Physical: who does this job, where

Open sourceCube, dbt MetricFlow
AzurePower BI semantic models
AWSQuickSight topics
GCPLooker modelling layer
VendorsDatabricks metric views, AtScale

Concept

Knowledge and retrieval

Finding the right passages for a question (RAG): chunk, embed, search, filter.

chunkingembeddingsvector and hybrid searchpermission filters

Physical: who does this job, where

Open sourcepgvector, Qdrant, OpenSearch
AzureAzure AI Search
AWSBedrock Knowledge Bases, OpenSearch
GCPVertex AI Search
VendorsPinecone, Weaviate

Concept

Quality and governance

Is the data right, who may see it, and where did it come from?

data quality checkscatalog and lineageaccess control

Physical: who does this job, where

Open sourceGreat Expectations, OpenLineage, ODCS
AzurePurview, Unity Catalog on Azure
AWSLake Formation, DataZone
GCPDataplex
VendorsUnity Catalog, Monte Carlo

Zoom in further

← Compute & infrastructure · Model →

Compute & infrastructureDataModelModel accessThe AI application (agent harness)ExperienceAssurance & operations