Data PlatformZoom level 1
Code map: the whole platform on one page, and the way into every line of code
This is the page to start from when you want to find code. The picture shows the whole system. Every box opens the page that explains it, and below the picture every starting point and every concern is linked to a line of code. From any function you can see who calls it.
How to find your way
- Know the part. Click its box above, or find its row in one of the two tables below.
- Go to the starting function. Each row names the function where that part begins, with its file and line. The link opens that line on GitHub.
- See who starts it. The same row lists the callers. Who calls what lists callers for every function and method in the code.
- Find any file. Code index lists every module with what it is for and which page explains it.
Where each part starts, and who starts it
| Part | Starting point | Called by (found by reading the code) | What it does | Explained on |
|---|---|---|---|---|
| The whole day loop | main scripts/run_platform.py:315 | module level · scripts/run_platform.py:745 | Plays the world, moves files, then runs ingestion, silver, gold and the checks for each day | Orchestration, end to end |
| One hop as its own process | main scripts/run_hop.py:84 | module level · scripts/run_hop.py:131 | Runs one of ingestion, silver or gold for one business date and saves its report | Orchestration, end to end |
| Upstream: the simulated company | World world/engine.py:72 | main · scripts/run_platform.py:444main · scripts/run_world.py:175Harness.build · twin/runner.py:95 | Writes each day into the mock sources and records the true totals | The simulated company |
| Push to landing | ChangeFeedMover adapters/local_movers.py:23 | mover · scripts/run_platform.py:232 | Takes change-capture output and file drops to the landing area | Sources and movers |
| Ingestion: landing or source to bronze | ingest ingest/runner.py:112 | run_ingest · scripts/run_hop.py:60run_days · ingest/daybyday.py:129 | Picks one of five pipelines per dataset | Run loop |
| Bronze to silver | build_silver silver/runner.py:69 | run_silver · scripts/run_hop.py:71prove_rebuild · silver/baseline_writer.py:112silver_day · silver/daybyday.py:177 | Reader, plan, thirteen stages, one write | Bronze to silver |
| Silver to gold | run_gold gold/harness.py:127 | run_gold · scripts/run_hop.py:78gold_day · gold/daybyday.py:57 | Dimensions first, then facts, window by window | Silver to gold |
| Semantic | answers semantic/answers.py:36 | gold_metrics · scripts/run_platform.py:293 | Asks each metric its known questions | Semantic layer |
| Operations page | write_ops_page scripts/run_platform.py:123 | main · scripts/run_platform.py:632 | Writes the page and the ops tables from the run log | Operations page and dashboards |
| Load the metadata store | load meta/loader.py:154 | main · realms/aws-databricks/meta_setup.py:79 | Validates manifests, then writes them | Loader |
The callers are found by reading the code, not by running it. A call written as object.method() is matched by the method's name, so a method of the same name in another class can appear. Those are marked by name on the caller pages.
How a day is orchestrated, in one paragraph
One script, run_platform.py, owns the day loop. For each day it asks the simulated company to act, moves the day's deliveries to landing, runs ingestion, then silver, then gold, and compares each layer with the true answers. The same script is what the deployed Databricks job runs, as a single task. Underneath, three mechanisms keep the order safe: a lease per dataset so two runs never write one table, bookmarks that move only when a dataset run succeeds, and the run log. The detail, with the job definitions, is on Orchestration, end to end.
Every concern, numbered, and who provides it
The platform's cross-cutting concerns are numbered X1 to X22 in the catalog. X23 and X24 were added on the site's platform page and are not in the catalog's list yet. Who provides it says whether we built the code, Databricks provides the capability and we read or configure it, or both. It is read from the code and from the adopt-first table in the observability notes, not from a claim. The rule counts are computed on 2026-11-19 from the catalog's obligations and the tests of this repository: a rule is tested when a test names it, built only when only code names it, and none when nothing does.
Run
| # | Concern | Who provides it | Where it starts in the code | Explained on | Rules (DAC) |
|---|---|---|---|---|---|
| X1 | Metadata and control plane Everything is declared in one manifest and validated. | Ours, stored on Lakebase | load meta/loader.py:154MetadataStore meta/port.py:181 | Metadata store · Loader | 7 rules 3 tested, 2 built only, 2 none |
| X2 | Orchestration What runs when, and in what order. Databricks Jobs start the work. Our day loop, leases and bookmarks keep the order safe. No schedule or file-arrival trigger is defined in this snapshot. | Both | main scripts/run_platform.py:315run_days ingest/daybyday.py:102busy leases.py:20 | Orchestration, end to end | 3 rules 3 tested, 0 built only, 0 none |
| X15 | Replay, backfill and idempotency Re-running never doubles a number. | Ours | withdraw recovery.py:18run_housekeeping housekeeping.py:121silver/baseline_writer.py | Recovery and housekeeping · Writer | 15 rules 13 tested, 0 built only, 2 none |
| X16 | Environments, CI/CD and release Development to test to production, infrastructure as code. The build workflows are not in the public snapshot. | Both | realms/databricks/engine/databricks.ymlrealms/aws-databricks/generate_bundle.py | Orchestration, end to end | 6 rules 0 tested, 1 built only, 5 none |
| X17 | Testing Unit, contract, data and simulation tests. | Ours | tests/dataplatform/conftest.pytwin/runner.py | Tests and proof | 5 rules 5 tested, 0 built only, 0 none |
| X18 | Resilience and recovery Retries, retention, time travel, recovery. | Both | SourceGuard resilience.py:98apply_landing_retention retention.py:87 | Resilience · Recovery and housekeeping | 4 rules 2 tested, 0 built only, 2 none |
| X23 | Layout: catalogs and schemas From a table's name you can tell its environment, layer and domain. Added on the site's platform page, not yet in the catalog's list of 22. | Both | adapters/uc_catalog.pytests/dataplatform/test_uc_table_names.py | no page yet | no rules in the catalog |
| X24 | Streaming A second speed on the same line, used only where a decision needs it. Added on the site's platform page. Trigger-once only, never continuous. | Ours, laptop only | stream_events ingest/pipelines.py:319adapters/local_stream.pyadapters/kinesis.py | Five pipelines | no rules in the catalog |
Trust
| # | Concern | Who provides it | Where it starts in the code | Explained on | Rules (DAC) |
|---|---|---|---|---|---|
| X3 | Contracts and schema evolution What a table promises, and what happens when a source changes shape. | Ours | contract_check stages/clean.py:42check_columns contract.py:43 | 1 Contract check · Entry steps | 10 rules 4 tested, 1 built only, 5 none |
| X4 | Data quality Rules on rows and tables; bad rows set aside with a reason. The results table is never written in a normal run. | Both | quality stages/clean.py:216contract_rules silver/contract_rules.py:58 | 9 Quality | 12 rules 8 tested, 1 built only, 3 none |
| X5 | Profiling Knowing what the data looks like, so change is visible. | Both | profile_rows profiling.py:25drift profiling.py:63 | Reconciliation and profiling | 1 rules 0 tested, 0 built only, 1 none |
| X6 | Reconciliation Counts and sums agree from source to gold. | Ours | compare reconcile.py:7silver/live_recon.py | Reconciliation and profiling | 7 rules 6 tested, 0 built only, 1 none |
See
| # | Concern | Who provides it | Where it starts in the code | Explained on | Rules (DAC) |
|---|---|---|---|---|---|
| X7 | Lineage Where each table came from and what it feeds. Our own lineage events are built only in a test. | Databricks, read by us | obs/lineage.pyadapters/databricks_system.py | Lineage | 2 rules 1 tested, 1 built only, 0 none |
| X8 | Observability One run is followable end to end. | Both | RunRecorder obs/recorder.py:179obs/trace.pyadapters/otel_tracer.py | Run log | 2 rules 2 tested, 0 built only, 0 none |
| X9 | Freshness and SLOs Is the data as new as promised. | Ours | check obs/freshness.py:127obs/success.py | Freshness | 5 rules 2 tested, 1 built only, 2 none |
| X10 | Cost and FinOps Who spent what, against which budget. The platform only tags its jobs. Cost is read by a separate tool. | Not here | compute_tags governance/tables.py:104 | no page yet | 3 rules 0 tested, 1 built only, 2 none |
| X19 | Performance File sizes, layout, compaction, right-sized compute. | Both | hop_delay timing.py:38adapters/table_store.py | Operations page and dashboards | 2 rules 1 tested, 0 built only, 1 none |
Guard
| # | Concern | Who provides it | Where it starts in the code | Explained on | Rules (DAC) |
|---|---|---|---|---|---|
| X11 | Security and access Who may read or change what. We plan it, Unity Catalog enforces it. Generated and tested locally. Applying it to a workspace is not shown in this snapshot. | Both | access/model.pyrealms/databricks/semantic_governance/render.py | Semantic layer | 4 rules 3 tested, 0 built only, 1 none |
| X12 | Privacy and personal data Find, mask, keep only as long as allowed. | Ours | personal_data stages/clean.py:187adapters/keyed_tokens.py | 7 Personal data | 3 rules 2 tested, 1 built only, 0 none |
| X13 | Governance, catalog and semantics Owners, glossary, metric definitions. | Both | metric_view semantic/metrics.py:151governance/catalog.py | Semantic layer | 6 rules 3 tested, 1 built only, 2 none |
| X14 | Audit and run log A record of every run and every change. | Both | obs/runlog.pyadapters/postgres_runlog.py | Run log · Runtime tables | 5 rules 3 tested, 1 built only, 1 none |
Self
| # | Concern | Who provides it | Where it starts in the code | Explained on | Rules (DAC) |
|---|---|---|---|---|---|
| X20 | Self-healing Detect, fix in development, deploy, verify, roll back. No code for it exists in this repository. Only the release gate's logic is written (twin/gate.py). | Not built | no code | no page yet | 1 rules 0 tested, 0 built only, 1 none |
| X21 | Agent assistance Agents draft manifests, rules and docs. No agent code exists in this repository. | Not built | no code | no page yet | 1 rules 0 tested, 0 built only, 1 none |
| X22 | Simulation A fake world that breaks the platform on purpose. | Ours | World world/engine.py:72twin/runner.py | The simulated company · Planted incidents | 1 rules 1 tested, 0 built only, 0 none |
- X20 self-healing and X21 agent assistance have no code in this repository.
- X10 cost is not in the platform. Jobs are tagged so a separate tool can attribute spend.
- The three-task platform job that the day loop's
--hops joboption expects is not defined in this snapshot. - The picture is hand-drawn from the code. The tables are checked against it at build time: a function that is renamed or removed breaks the build.