Data PlatformZoom level 3
Who calls what: Scripts
Every class, function and method in Scripts, with who calls it. Found by reading the code. A call written as object.method() is matched by the method's name, so a same-named method elsewhere can appear: those rows say by name.
94 definitions in 10 modules. Jump to a module: dac_coverage.py · demo_end_to_end.py · dp_compile_domain.py · explain.py · export_settings_registry.py · platform_binding.py · run_hop.py · run_platform.py · run_world.py · show_postgres_delta.py
scripts/dac_coverage.py
DAC coverage report (B20 C20): which platform rules have a passing test, a waiver, or neither. none yet
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
textsfunction, line 24 | main · scripts/dac_coverage.py:35 | no test names it | |
mainfunction, line 28 | module level · scripts/dac_coverage.py:55 | no test names it |
scripts/demo_end_to_end.py
End-to-end local demo — Source → Bronze → Silver → Gold + lineage. none yet
scripts/dp_compile_domain.py
Compile what the job generator needs to know about a domain from its DPM manifest folder. none yet
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
ManifestErrorclass, line 22 | summarise · scripts/dp_compile_domain.py:30 | test_bundle.py, test_meta_loader.py and 1 more | |
summarisefunction, line 26 | Pure over the files it reads: the same manifest folder gives the same summary, keys and lists sorted. | main · scripts/dp_compile_domain.py:52 | test_bundle.py |
mainfunction, line 49 | module level · scripts/dp_compile_domain.py:62 | no test names it |
scripts/explain.py
The platform explains itself in plain words. No model and no network: the same inputs always give the same text. none yet
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
catalog_offunction, line 30 | cmd_run · scripts/explain.py:54cmd_metadata · scripts/explain.py:65 | no test names it | |
manifestsfunction, line 36 | cmd_run · scripts/explain.py:54cmd_metadata · scripts/explain.py:65 | test_gold_plan.py, test_meta_loader.py and 3 more | |
cmd_runfunction, line 47 | main · scripts/explain.py:147 (value) | no test names it | |
cmd_metadatafunction, line 62 | main · scripts/explain.py:147 (value) | no test names it | |
cmd_cicdfunction, line 79 | main · scripts/explain.py:147 (value) | no test names it | |
cmd_codefunction, line 97 | main · scripts/explain.py:147 (value) | no test names it | |
mainfunction, line 134 | module level · scripts/explain.py:157 | no test names it |
scripts/export_settings_registry.py
Export the settings registry for the metadata store (DPC-0034) from LakeIngestionFramework's key registry. none yet
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
buildfunction, line 24 | main · scripts/export_settings_registry.py:52 | test_semantic_governance.py, conftest.py and 17 more | |
mainfunction, line 48 | module level · scripts/export_settings_registry.py:64 | no test names it |
scripts/platform_binding.py
The adapters a run picks, shared by the day loop (run_platform.py) and by one hop running as its own process (run_hop.py), so both bind the same engine, tables and metadata store from the same options. none yet
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
Engineclass, line 17 | What silver and gold run on. Every field is None when the run stops before that layer. | build_engine · scripts/platform_binding.py:48_duckdb · scripts/platform_binding.py:68_table_port · scripts/platform_binding.py:97_legacy · scripts/platform_binding.py:110 | conftest.py, test_processors_pii.py and 1 more |
attach_metafunction, line 29 | The metadata store a run already filled, opened by another process. Nothing is created or emptied. | none found in the code | no test names it |
build_enginefunction, line 42 | Silver and gold adapters for the options `a` (silver, gold, engine, table_format, spark, out, env). The domain's gold models folder and gold judge come from the world `dwd` (never from a fixed domain). `on_store(table_store)` runs once the | none found in the code | no test names it |
_duckdbfunction, line 56 | build_engine · scripts/platform_binding.py:50 | no test names it | |
_duckdb.local_sessionfunction, line 64 | none found in the code | conftest.py, test_obs_steps.py | |
_table_portfunction, line 76 | build_engine · scripts/platform_binding.py:52 | no test names it | |
_table_port.local_sessionfunction, line 85 | none found in the code | conftest.py, test_obs_steps.py | |
_legacyfunction, line 106 | build_engine · scripts/platform_binding.py:53 | no test names it |
scripts/run_hop.py
One hop of the platform as its own process: ingestion, silver or gold for one business date. Orchestration, end to end
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
run_ingestfunction, line 34 | module level · scripts/run_hop.py:81 (value) | no test names it | |
run_silverfunction, line 67 | module level · scripts/run_hop.py:81 (value) | test_explainer.py | |
run_goldfunction, line 75 | module level · scripts/run_hop.py:81 (value) | test_explainer.py, test_gold_harness.py and 1 more | |
mainfunction, line 84 | module level · scripts/run_hop.py:131 | no test names it |
scripts/run_platform.py
Run the tyre world and the platform together, day by day, locally: world -> sources -> landing -> bronze -> silver. (--slice 03-clinical runs the clinical trial instead: the world in the slice picks the domain, models and gold judge.) Orchestration, end to end
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
meta_storefunction, line 63 | main · scripts/run_platform.py:396 | no test names it | |
lakebase_fresh_storefunction, line 79 | DP_META_INSTANCE (and DP_META_DATABASE, default dp_meta_proof): create the database if missing, empty it, install the schema. The manifest's own store (dp_meta, loaded by meta_setup.py) is never touched by a proof run. | meta_store · scripts/run_platform.py:67 | no test names it |
WallClockclass, line 99 | Real time. A run whose hops are separate processes cannot share a simulated clock with them. | main · scripts/run_platform.py:395 | no test names it |
WallClock.setmethod, line 102 | Nothing to set: the hops stamp their own real times. | none found in the code | no test names it |
WallClock.nowmethod, line 105 | step_prepare · realms/databricks/tasks/main.py:71 (by name)module level · realms/databricks/v1/tasks/hop.py:28 (by name)main · scripts/dac_coverage.py:40 (by name)write_ops_page · scripts/run_platform.py:130 (by name)main.pushed · scripts/run_platform.py:408 (by name)DuckStageRunner.run · adapters/duckdb_silver.py:67 (by name)and 19 more | no test names it | |
SimClockclass, line 109 | The clock port for a local run: the simulated day's time, plus the real seconds spent since it was set, so the run log has true step durations inside a simulated day (ManualClock alone stood still: every duration 0). | main · scripts/run_platform.py:395 | no test names it |
SimClock.__init__method, line 113 | none found in the code | no test names it | |
SimClock.setmethod, line 116 | none found in the code | no test names it | |
SimClock.nowmethod, line 119 | SimClock.set · scripts/run_platform.py:117step_prepare · realms/databricks/tasks/main.py:71 (by name)module level · realms/databricks/v1/tasks/hop.py:28 (by name)main · scripts/dac_coverage.py:40 (by name)WallClock.now · scripts/run_platform.py:106 (by name)write_ops_page · scripts/run_platform.py:130 (by name)and 21 more | no test names it | |
write_ops_pagefunction, line 123 | The local ops page (B10): the ops tables the dashboards read, plus every layer against the oracle, the planted incidents' verdicts and the delay per hop, in one HTML file beside the run. | main · scripts/run_platform.py:632 | no test names it |
hop_delaysfunction, line 154 | Delay per hop per dataset from silver's stamps (Spark engine only). | write_ops_page · scripts/run_platform.py:140main · scripts/run_platform.py:634 | no test names it |
success_scorecardfunction, line 167 | B20: the success criteria this run can measure (obs/success.py), from its own evidence. | main · scripts/run_platform.py:633 | no test names it |
write_calendarfunction, line 195 | The shared, generated date dimension (B15): five years either side of the world's start, fiscal year from April (the Indian financial year), written whole each run (idempotent). | main · scripts/run_platform.py:462 | no test names it |
upkeep_allfunction, line 212 | Upkeep on every table of every layer: on Iceberg, compaction plus snapshot expiry (a 14-day run without it reached 1,483 snapshots and 4,191 data files and ran a 1 GB driver out of heap on day 14, 13 Nov). | main.on_day · scripts/run_platform.py:607 | no test names it |
change_feedsfunction, line 225 | The datasets a change-capture mover delivers (database sources read as a change feed). | main · scripts/run_platform.py:415 | no test names it |
moverfunction, line 231 | main · scripts/run_platform.py:622 | test_azure_adapters.py, test_cloud_adapters.py and 2 more | |
mover.movefunction, line 235 | none found in the code | no test names it | |
show_silverfunction, line 249 | main.on_day · scripts/run_platform.py:588 | no test names it | |
show_goldfunction, line 266 | main.on_day · scripts/run_platform.py:601 | no test names it | |
gold_metricsfunction, line 282 | Every certified known answer asked of gold the way the metric view defines it (W5's DP_GOLD_METRICS file). | main · scripts/run_platform.py:631 | no test names it |
showfunction, line 298 | main.on_day · scripts/run_platform.py:575 | no test names it | |
mainfunction, line 315 | module level · scripts/run_platform.py:745 | no test names it | |
main.pushedfunction, line 406 | main.run_hop · scripts/run_platform.py:559main · scripts/run_platform.py:410 (value) | test_obs_dashboards.py, test_obs_traced_store.py | |
main.reset_tablesfunction, line 449 | main · scripts/run_platform.py:458 (value) | no test names it | |
main.fresh_sessionfunction, line 464 | A new session for the next day: every table is on disk, so nothing is lost, and a 1 GB driver never carries fourteen days of plans, caches and Delta log snapshots (Spark ran out of heap on day 12); on DuckDB a new in-memory connection drops | main.on_day · scripts/run_platform.py:611 | no test names it |
main.live_rowsfunction, line 478 | The live table of a database source now (REC-05: live source against silver); None for other sources. | main.on_day · scripts/run_platform.py:582 (value) | no test names it |
main.housekeepingfunction, line 494 | Bronze housekeeping (B8) with the run's correction window: old batches go once a checked baseline holds. | main.on_day · scripts/run_platform.py:603 | test_housekeeping.py |
main.prove_rebuild_afterfunction, line 518 | main · scripts/run_platform.py:630 | no test names it | |
main.run_platform_jobfunction, line 534 | --hops job: the day's mock sources copied where the job can open them, then the deployed platform job (ingestion, silver, gold as three tasks) triggered for the window and waited for. | main.run_hop · scripts/run_platform.py:557 | no test names it |
main.run_hopfunction, line 548 | One hop outside this process, then its saved report read back and the monitor told. process: the hop as its own process (scripts/run_hop.py). job: the deployed platform job runs all three hops once per day; the first hop asked for triggers | main.on_day · scripts/run_platform.py:582main · scripts/run_platform.py:628 | test_explainer.py |
main.on_dayfunction, line 574 | main · scripts/run_platform.py:615 (value) | test_silver_world.py, test_twin_engine.py |
scripts/run_world.py
Run the tyre world locally: N simulated days into the four mock sources, with the oracle's totals per day. none yet
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
catalog_forfunction, line 44 | The slice folder a run uses: `--slice 03-clinical` picks slices/03-clinical of the catalog (DP_CATALOG, default ~/data-platform-catalog); the domain, its gold judge and its gold models all follow from the world in it. | none found in the code | no test names it |
aws_sinksfunction, line 50 | The aws binding: file drop and stream on the buckets and streams Terraform made (realms/aws-databricks/infra). The database and the dealer interface stay local until their cloud adapters exist. | sinks · scripts/run_world.py:84 | no test names it |
sinksfunction, line 62 | The world's sources. `attach` opens the sources another process already holds (a hop running as its own process reads them); without it every source starts empty. | main · scripts/run_world.py:173 | test_semantic_metrics.py, test_world_clinical.py and 1 more |
corrections_storefunction, line 89 | A metadata store loaded with the slice's manifest, fresh each run, for the corrections the world records. | main · scripts/run_world.py:174 | no test names it |
read_backfunction, line 111 | Read every source through its port, the way an ingestion job would, and count what is there. | main · scripts/run_world.py:179 | no test names it |
mainfunction, line 145 | module level · scripts/run_world.py:214 | no test names it |
scripts/show_postgres_delta.py
Demo inspection script: postgres → ADF Mimic → local Delta, with row dump. none yet
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
_build_artifact_treefunction, line 50 | Hand-roll a minimal LinkedService + 2 Datasets + Pipeline tree. | main · scripts/show_postgres_delta.py:287 | no test names it |
_seed_metadata_dbfunction, line 183 | Create the run_log table the mimic emulator writes to. | main · scripts/show_postgres_delta.py:289 | no test names it |
_wait_for_postgresfunction, line 201 | main · scripts/show_postgres_delta.py:256 | no test names it | |
mainfunction, line 216 | module level · scripts/show_postgres_delta.py:350 | no test names it |