AI Universe · Data
Data zoom 2
What does the AI know? 5 concept buckets. Each one is a job you can learn once; underneath it, who does that job on each platform.
Concept
Ingestion
Bringing data in from source systems, reliably and only once.
batch and streamingchange data captureschema drift
Physical: who does this job, where
| Open source | Airbyte, Debezium, Kafka |
|---|---|
| Azure | Data Factory, Fabric pipelines, Event Hubs |
| AWS | Glue, DMS, Kinesis |
| GCP | Dataflow, Datastream, Pub/Sub |
| Vendors | Fivetran, Databricks Lakeflow |
Concept
Lakehouse layers
Raw → clean → business-shaped data (bronze, silver, gold).
table formatstransformationsdimensional models
Physical: who does this job, where
| Open source | Delta Lake, Iceberg, Spark, dbt |
|---|---|
| Azure | Fabric OneLake, Azure Databricks |
| AWS | S3 + Iceberg, EMR, Redshift |
| GCP | BigQuery, BigLake |
| Vendors | Databricks, Snowflake |
Concept
Semantic layer
One agreed definition of every business number, so every tool gives the same answer.
metrics and measurescertified definitionsaccess rules
Physical: who does this job, where
| Open source | Cube, dbt MetricFlow |
|---|---|
| Azure | Power BI semantic models |
| AWS | QuickSight topics |
| GCP | Looker modelling layer |
| Vendors | Databricks metric views, AtScale |
Concept
Knowledge and retrieval
Finding the right passages for a question (RAG): chunk, embed, search, filter.
chunkingembeddingsvector and hybrid searchpermission filters
Physical: who does this job, where
| Open source | pgvector, Qdrant, OpenSearch |
|---|---|
| Azure | Azure AI Search |
| AWS | Bedrock Knowledge Bases, OpenSearch |
| GCP | Vertex AI Search |
| Vendors | Pinecone, Weaviate |
Concept
Quality and governance
Is the data right, who may see it, and where did it come from?
data quality checkscatalog and lineageaccess control
Physical: who does this job, where
| Open source | Great Expectations, OpenLineage, ODCS |
|---|---|
| Azure | Purview, Unity Catalog on Azure |
| AWS | Lake Formation, DataZone |
| GCP | Dataplex |
| Vendors | Unity Catalog, Monte Carlo |