cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Code map

Data PlatformZoom level 1

Code map: the whole platform on one page, and the way into every line of code

This is the page to start from when you want to find code. The picture shows the whole system. Every box opens the page that explains it, and below the picture every starting point and every concern is linked to a line of code. From any function you can see who calls it.

UPSTREAMDATA PLATFORMDOWNSTREAMOpen: Source-side extractionSource-side extractionchange capture, exports,file drops (push)zoom in ›Open: Source systemsSource systemsdatabases, files, APIs,streamszoom in ›Open: Simulated companySimulated companyplays a source on every dayshown here, knows the truthzoom in ›Open: LandingLandingfiles as deliveredzoom in ›Open: LanderLanderlanding to bronze,entry steps, ingest()zoom in ›Open: PullerPullersource to bronze: full copy,pages, stream, ingest()zoom in ›Open: BronzeBronzeevery batch, as receivednothing lostzoom in ›pushpullOpen: SilverSilverbuild_silverclean, keyed, historyzoom in ›Open: GoldGoldrun_golddimensions, factszoom in ›Open: SemanticSemanticmetric_viewone meaningzoom in ›Open: Metadata store (Lakebase, a managed PostgreSQL)Metadata store (Lakebase, a managed PostgreSQL)settings, columns, keys, file log, bookmarks, run logevery step reads its settings here and writes what it did herezoom in ›Across every layer: the concerns, numbered in the table belowOpen: Databricks JobsDatabricks JobsX2 schedules, tasks, retrieszoom in ›Open: Day loop, leases, bookmarksDay loop, leases, bookmarksX2 X15, ourszoom in ›Open: Data expectation servicesData expectation servicesX4 X5 X6, oursquality, profiling, reconciliationzoom in ›Open: Lakeflow expectationsLakeflow expectationsX4 X5, on thedeclarative enginezoom in ›Open: Run log, tracing, alertsRun log, tracing, alertsX8 X9 X14, ourszoom in ›Open: Monitor, lineage, auditMonitor, lineage, auditX7 X8 X14, adoptedfrom the platformzoom in ›Open: Metric viewsMetric viewscertified definitionszoom in ›Open: DashboardsDashboardsoperations (ours), jobhealth (Databricks)zoom in ›Open: Conversation spaceConversation spacegenerated from the metricszoom in ›Open: Access and governanceAccess and governanceX11 X13, grants and masks(Unity Catalog)zoom in ›Not in this repositoryML, reverse ETL,BI toolsbuilt hereprovided by Databricks, adoptedoutside this repository
Read left to right. Data comes in by two ways, pushed to landing or pulled by the platform, and both end in bronze. The metadata store drives every step. Blue boxes are built here, yellow boxes are provided by Databricks.

How to find your way

  1. Know the part. Click its box above, or find its row in one of the two tables below.
  2. Go to the starting function. Each row names the function where that part begins, with its file and line. The link opens that line on GitHub.
  3. See who starts it. The same row lists the callers. Who calls what lists callers for every function and method in the code.
  4. Find any file. Code index lists every module with what it is for and which page explains it.

Where each part starts, and who starts it

PartStarting pointCalled by (found by reading the code)What it doesExplained on
The whole day loopmain scripts/run_platform.py:315module level · scripts/run_platform.py:745Plays the world, moves files, then runs ingestion, silver, gold and the checks for each dayOrchestration, end to end
One hop as its own processmain scripts/run_hop.py:84module level · scripts/run_hop.py:131Runs one of ingestion, silver or gold for one business date and saves its reportOrchestration, end to end
Upstream: the simulated companyWorld world/engine.py:72main · scripts/run_platform.py:444
main · scripts/run_world.py:175
Harness.build · twin/runner.py:95
Writes each day into the mock sources and records the true totalsThe simulated company
Push to landingChangeFeedMover adapters/local_movers.py:23mover · scripts/run_platform.py:232Takes change-capture output and file drops to the landing areaSources and movers
Ingestion: landing or source to bronzeingest ingest/runner.py:112run_ingest · scripts/run_hop.py:60
run_days · ingest/daybyday.py:129
Picks one of five pipelines per datasetRun loop
Bronze to silverbuild_silver silver/runner.py:69run_silver · scripts/run_hop.py:71
prove_rebuild · silver/baseline_writer.py:112
silver_day · silver/daybyday.py:177
Reader, plan, thirteen stages, one writeBronze to silver
Silver to goldrun_gold gold/harness.py:127run_gold · scripts/run_hop.py:78
gold_day · gold/daybyday.py:57
Dimensions first, then facts, window by windowSilver to gold
Semanticanswers semantic/answers.py:36gold_metrics · scripts/run_platform.py:293Asks each metric its known questionsSemantic layer
Operations pagewrite_ops_page scripts/run_platform.py:123main · scripts/run_platform.py:632Writes the page and the ops tables from the run logOperations page and dashboards
Load the metadata storeload meta/loader.py:154main · realms/aws-databricks/meta_setup.py:79Validates manifests, then writes themLoader

The callers are found by reading the code, not by running it. A call written as object.method() is matched by the method's name, so a method of the same name in another class can appear. Those are marked by name on the caller pages.

How a day is orchestrated, in one paragraph

One script, run_platform.py, owns the day loop. For each day it asks the simulated company to act, moves the day's deliveries to landing, runs ingestion, then silver, then gold, and compares each layer with the true answers. The same script is what the deployed Databricks job runs, as a single task. Underneath, three mechanisms keep the order safe: a lease per dataset so two runs never write one table, bookmarks that move only when a dataset run succeeds, and the run log. The detail, with the job definitions, is on Orchestration, end to end.

Every concern, numbered, and who provides it

The platform's cross-cutting concerns are numbered X1 to X22 in the catalog. X23 and X24 were added on the site's platform page and are not in the catalog's list yet. Who provides it says whether we built the code, Databricks provides the capability and we read or configure it, or both. It is read from the code and from the adopt-first table in the observability notes, not from a claim. The rule counts are computed on 2026-11-19 from the catalog's obligations and the tests of this repository: a rule is tested when a test names it, built only when only code names it, and none when nothing does.

Run

#ConcernWho provides itWhere it starts in the codeExplained onRules (DAC)
X1Metadata and control plane
Everything is declared in one manifest and validated.
Ours, stored on Lakebaseload meta/loader.py:154
MetadataStore meta/port.py:181
Metadata store · Loader7 rules
3 tested, 2 built only, 2 none
X2Orchestration
What runs when, and in what order. Databricks Jobs start the work. Our day loop, leases and bookmarks keep the order safe.
No schedule or file-arrival trigger is defined in this snapshot.
Bothmain scripts/run_platform.py:315
run_days ingest/daybyday.py:102
busy leases.py:20
Orchestration, end to end3 rules
3 tested, 0 built only, 0 none
X15Replay, backfill and idempotency
Re-running never doubles a number.
Ourswithdraw recovery.py:18
run_housekeeping housekeeping.py:121
silver/baseline_writer.py
Recovery and housekeeping · Writer15 rules
13 tested, 0 built only, 2 none
X16Environments, CI/CD and release
Development to test to production, infrastructure as code.
The build workflows are not in the public snapshot.
Bothrealms/databricks/engine/databricks.yml
realms/aws-databricks/generate_bundle.py
Orchestration, end to end6 rules
0 tested, 1 built only, 5 none
X17Testing
Unit, contract, data and simulation tests.
Ourstests/dataplatform/conftest.py
twin/runner.py
Tests and proof5 rules
5 tested, 0 built only, 0 none
X18Resilience and recovery
Retries, retention, time travel, recovery.
BothSourceGuard resilience.py:98
apply_landing_retention retention.py:87
Resilience · Recovery and housekeeping4 rules
2 tested, 0 built only, 2 none
X23Layout: catalogs and schemas
From a table's name you can tell its environment, layer and domain.
Added on the site's platform page, not yet in the catalog's list of 22.
Bothadapters/uc_catalog.py
tests/dataplatform/test_uc_table_names.py
no page yetno rules in the catalog
X24Streaming
A second speed on the same line, used only where a decision needs it.
Added on the site's platform page. Trigger-once only, never continuous.
Ours, laptop onlystream_events ingest/pipelines.py:319
adapters/local_stream.py
adapters/kinesis.py
Five pipelinesno rules in the catalog

Trust

#ConcernWho provides itWhere it starts in the codeExplained onRules (DAC)
X3Contracts and schema evolution
What a table promises, and what happens when a source changes shape.
Ourscontract_check stages/clean.py:42
check_columns contract.py:43
1 Contract check · Entry steps10 rules
4 tested, 1 built only, 5 none
X4Data quality
Rules on rows and tables; bad rows set aside with a reason.
The results table is never written in a normal run.
Bothquality stages/clean.py:216
contract_rules silver/contract_rules.py:58
9 Quality12 rules
8 tested, 1 built only, 3 none
X5Profiling
Knowing what the data looks like, so change is visible.
Bothprofile_rows profiling.py:25
drift profiling.py:63
Reconciliation and profiling1 rules
0 tested, 0 built only, 1 none
X6Reconciliation
Counts and sums agree from source to gold.
Ourscompare reconcile.py:7
silver/live_recon.py
Reconciliation and profiling7 rules
6 tested, 0 built only, 1 none

See

#ConcernWho provides itWhere it starts in the codeExplained onRules (DAC)
X7Lineage
Where each table came from and what it feeds.
Our own lineage events are built only in a test.
Databricks, read by usobs/lineage.py
adapters/databricks_system.py
Lineage2 rules
1 tested, 1 built only, 0 none
X8Observability
One run is followable end to end.
BothRunRecorder obs/recorder.py:179
obs/trace.py
adapters/otel_tracer.py
Run log2 rules
2 tested, 0 built only, 0 none
X9Freshness and SLOs
Is the data as new as promised.
Ourscheck obs/freshness.py:127
obs/success.py
Freshness5 rules
2 tested, 1 built only, 2 none
X10Cost and FinOps
Who spent what, against which budget.
The platform only tags its jobs. Cost is read by a separate tool.
Not herecompute_tags governance/tables.py:104no page yet3 rules
0 tested, 1 built only, 2 none
X19Performance
File sizes, layout, compaction, right-sized compute.
Bothhop_delay timing.py:38
adapters/table_store.py
Operations page and dashboards2 rules
1 tested, 0 built only, 1 none

Guard

#ConcernWho provides itWhere it starts in the codeExplained onRules (DAC)
X11Security and access
Who may read or change what. We plan it, Unity Catalog enforces it.
Generated and tested locally. Applying it to a workspace is not shown in this snapshot.
Bothaccess/model.py
realms/databricks/semantic_governance/render.py
Semantic layer4 rules
3 tested, 0 built only, 1 none
X12Privacy and personal data
Find, mask, keep only as long as allowed.
Ourspersonal_data stages/clean.py:187
adapters/keyed_tokens.py
7 Personal data3 rules
2 tested, 1 built only, 0 none
X13Governance, catalog and semantics
Owners, glossary, metric definitions.
Bothmetric_view semantic/metrics.py:151
governance/catalog.py
Semantic layer6 rules
3 tested, 1 built only, 2 none
X14Audit and run log
A record of every run and every change.
Bothobs/runlog.py
adapters/postgres_runlog.py
Run log · Runtime tables5 rules
3 tested, 1 built only, 1 none

Self

#ConcernWho provides itWhere it starts in the codeExplained onRules (DAC)
X20Self-healing
Detect, fix in development, deploy, verify, roll back.
No code for it exists in this repository. Only the release gate's logic is written (twin/gate.py).
Not builtno codeno page yet1 rules
0 tested, 0 built only, 1 none
X21Agent assistance
Agents draft manifests, rules and docs.
No agent code exists in this repository.
Not builtno codeno page yet1 rules
0 tested, 0 built only, 1 none
X22Simulation
A fake world that breaks the platform on purpose.
OursWorld world/engine.py:72
twin/runner.py
The simulated company · Planted incidents1 rules
1 tested, 0 built only, 0 none
Not built yet, or known gaps
  • X20 self-healing and X21 agent assistance have no code in this repository.
  • X10 cost is not in the platform. Jobs are tagged so a separate tool can attribute spend.
  • The three-task platform job that the day loop's --hops job option expects is not defined in this snapshot.
  • The picture is hand-drawn from the code. The tables are checked against it at build time: a function that is renamed or removed breaks the build.

Look inside