cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Code map › Who calls what › Scripts

Data PlatformZoom level 3

Who calls what: Scripts

Every class, function and method in Scripts, with who calls it. Found by reading the code. A call written as object.method() is matched by the method's name, so a same-named method elsewhere can appear: those rows say by name.

94 definitions in 10 modules. Jump to a module: dac_coverage.py · demo_end_to_end.py · dp_compile_domain.py · explain.py · export_settings_registry.py · platform_binding.py · run_hop.py · run_platform.py · run_world.py · show_postgres_delta.py

scripts/dac_coverage.py

DAC coverage report (B20 C20): which platform rules have a passing test, a waiver, or neither. none yet

DefinitionWhat it does (its own first sentence)Called byNamed in tests
texts
function, line 24
main · scripts/dac_coverage.py:35no test names it
main
function, line 28
module level · scripts/dac_coverage.py:55no test names it

scripts/demo_end_to_end.py

End-to-end local demo — Source → Bronze → Silver → Gold + lineage. none yet

DefinitionWhat it does (its own first sentence)Called byNamed in tests
_heading
function, line 57
_preflight · scripts/demo_end_to_end.py:139
main · scripts/demo_end_to_end.py:885
no test names it
_step
function, line 64
_boot_postgres · scripts/demo_end_to_end.py:194
_teardown_postgres · scripts/demo_end_to_end.py:241
_seed_postgres · scripts/demo_end_to_end.py:258
main · scripts/demo_end_to_end.py:887
no test names it
_ok
function, line 68
_preflight · scripts/demo_end_to_end.py:142
_boot_postgres · scripts/demo_end_to_end.py:227
_teardown_postgres · scripts/demo_end_to_end.py:248
_seed_postgres · scripts/demo_end_to_end.py:327
main · scripts/demo_end_to_end.py:898
no test names it
_warn
function, line 72
_preflight · scripts/demo_end_to_end.py:158
_print_delta_summary · scripts/demo_end_to_end.py:811
_print_event_summary · scripts/demo_end_to_end.py:821
main · scripts/demo_end_to_end.py:1003
no test names it
_fail
function, line 76
_preflight · scripts/demo_end_to_end.py:144
main · scripts/demo_end_to_end.py:859
no test names it
_dim
function, line 80
main · scripts/demo_end_to_end.py:1068no test names it
PreflightResult
class, line 90
_preflight · scripts/demo_end_to_end.py:163no test names it
_docker_available
function, line 97
_preflight · scripts/demo_end_to_end.py:140no test names it
_extras_available
function, line 112
_preflight · scripts/demo_end_to_end.py:146no test names it
_dbt_available
function, line 127
_preflight · scripts/demo_end_to_end.py:153no test names it
_preflight
function, line 138
main · scripts/demo_end_to_end.py:857no test names it
PostgresHandle
class, line 183
_boot_postgres · scripts/demo_end_to_end.py:228no test names it
_boot_postgres
function, line 192
main · scripts/demo_end_to_end.py:880no test names it
_teardown_postgres
function, line 240
main · scripts/demo_end_to_end.py:1078no test names it
_seed_postgres
function, line 251
main · scripts/demo_end_to_end.py:882no test names it
_ingest_table_to_bronze
function, line 335
Ingest one postgres table to a Bronze Delta path; emit Source lineage.main · scripts/demo_end_to_end.py:888no test names it
_silver_run
function, line 412
main · scripts/demo_end_to_end.py:914no test names it
_setup_processor_metadata
function, line 461
Seed an in-memory metadata DB with column transforms + per-env PII policy + processor_groups, then return per-table contexts.main · scripts/demo_end_to_end.py:906no test names it
_generate_gold
function, line 671
main · scripts/demo_end_to_end.py:968no test names it
_dbt_invocation
function, line 726
_dbt_run · scripts/demo_end_to_end.py:740no test names it
_dbt_run
function, line 730
main · scripts/demo_end_to_end.py:972no test names it
_emit_gold_lineage
function, line 758
main · scripts/demo_end_to_end.py:993no test names it
_print_query
function, line 779
main · scripts/demo_end_to_end.py:1022no test names it
_print_path
function, line 792
main · scripts/demo_end_to_end.py:1036no test names it
_print_delta_summary
function, line 807
main · scripts/demo_end_to_end.py:1049no test names it
_print_event_summary
function, line 819
main · scripts/demo_end_to_end.py:1054no test names it
main
function, line 837
module level · scripts/demo_end_to_end.py:1084no test names it

scripts/dp_compile_domain.py

Compile what the job generator needs to know about a domain from its DPM manifest folder. none yet

DefinitionWhat it does (its own first sentence)Called byNamed in tests
ManifestError
class, line 22
summarise · scripts/dp_compile_domain.py:30test_bundle.py, test_meta_loader.py and 1 more
summarise
function, line 26
Pure over the files it reads: the same manifest folder gives the same summary, keys and lists sorted.main · scripts/dp_compile_domain.py:52test_bundle.py
main
function, line 49
module level · scripts/dp_compile_domain.py:62no test names it

scripts/explain.py

The platform explains itself in plain words. No model and no network: the same inputs always give the same text. none yet

DefinitionWhat it does (its own first sentence)Called byNamed in tests
catalog_of
function, line 30
cmd_run · scripts/explain.py:54
cmd_metadata · scripts/explain.py:65
no test names it
manifests
function, line 36
cmd_run · scripts/explain.py:54
cmd_metadata · scripts/explain.py:65
test_gold_plan.py, test_meta_loader.py and 3 more
cmd_run
function, line 47
main · scripts/explain.py:147 (value)no test names it
cmd_metadata
function, line 62
main · scripts/explain.py:147 (value)no test names it
cmd_cicd
function, line 79
main · scripts/explain.py:147 (value)no test names it
cmd_code
function, line 97
main · scripts/explain.py:147 (value)no test names it
main
function, line 134
module level · scripts/explain.py:157no test names it

scripts/export_settings_registry.py

Export the settings registry for the metadata store (DPC-0034) from LakeIngestionFramework's key registry. none yet

DefinitionWhat it does (its own first sentence)Called byNamed in tests
build
function, line 24
main · scripts/export_settings_registry.py:52test_semantic_governance.py, conftest.py and 17 more
main
function, line 48
module level · scripts/export_settings_registry.py:64no test names it

scripts/platform_binding.py

The adapters a run picks, shared by the day loop (run_platform.py) and by one hop running as its own process (run_hop.py), so both bind the same engine, tables and metadata store from the same options. none yet

DefinitionWhat it does (its own first sentence)Called byNamed in tests
Engine
class, line 17
What silver and gold run on. Every field is None when the run stops before that layer.build_engine · scripts/platform_binding.py:48
_duckdb · scripts/platform_binding.py:68
_table_port · scripts/platform_binding.py:97
_legacy · scripts/platform_binding.py:110
conftest.py, test_processors_pii.py and 1 more
attach_meta
function, line 29
The metadata store a run already filled, opened by another process. Nothing is created or emptied.none found in the codeno test names it
build_engine
function, line 42
Silver and gold adapters for the options `a` (silver, gold, engine, table_format, spark, out, env). The domain's gold models folder and gold judge come from the world `dwd` (never from a fixed domain). `on_store(table_store)` runs once the none found in the codeno test names it
_duckdb
function, line 56
build_engine · scripts/platform_binding.py:50no test names it
_duckdb.local_session
function, line 64
none found in the codeconftest.py, test_obs_steps.py
_table_port
function, line 76
build_engine · scripts/platform_binding.py:52no test names it
_table_port.local_session
function, line 85
none found in the codeconftest.py, test_obs_steps.py
_legacy
function, line 106
build_engine · scripts/platform_binding.py:53no test names it

scripts/run_hop.py

One hop of the platform as its own process: ingestion, silver or gold for one business date. Orchestration, end to end

DefinitionWhat it does (its own first sentence)Called byNamed in tests
run_ingest
function, line 34
module level · scripts/run_hop.py:81 (value)no test names it
run_silver
function, line 67
module level · scripts/run_hop.py:81 (value)test_explainer.py
run_gold
function, line 75
module level · scripts/run_hop.py:81 (value)test_explainer.py, test_gold_harness.py and 1 more
main
function, line 84
module level · scripts/run_hop.py:131no test names it

scripts/run_platform.py

Run the tyre world and the platform together, day by day, locally: world -> sources -> landing -> bronze -> silver. (--slice 03-clinical runs the clinical trial instead: the world in the slice picks the domain, models and gold judge.) Orchestration, end to end

DefinitionWhat it does (its own first sentence)Called byNamed in tests
meta_store
function, line 63
main · scripts/run_platform.py:396no test names it
lakebase_fresh_store
function, line 79
DP_META_INSTANCE (and DP_META_DATABASE, default dp_meta_proof): create the database if missing, empty it, install the schema. The manifest's own store (dp_meta, loaded by meta_setup.py) is never touched by a proof run.meta_store · scripts/run_platform.py:67no test names it
WallClock
class, line 99
Real time. A run whose hops are separate processes cannot share a simulated clock with them.main · scripts/run_platform.py:395no test names it
WallClock.set
method, line 102
Nothing to set: the hops stamp their own real times.none found in the codeno test names it
WallClock.now
method, line 105
step_prepare · realms/databricks/tasks/main.py:71 (by name)
module level · realms/databricks/v1/tasks/hop.py:28 (by name)
main · scripts/dac_coverage.py:40 (by name)
write_ops_page · scripts/run_platform.py:130 (by name)
main.pushed · scripts/run_platform.py:408 (by name)
DuckStageRunner.run · adapters/duckdb_silver.py:67 (by name)
and 19 more
no test names it
SimClock
class, line 109
The clock port for a local run: the simulated day's time, plus the real seconds spent since it was set, so the run log has true step durations inside a simulated day (ManualClock alone stood still: every duration 0).main · scripts/run_platform.py:395no test names it
SimClock.__init__
method, line 113
none found in the codeno test names it
SimClock.set
method, line 116
none found in the codeno test names it
SimClock.now
method, line 119
SimClock.set · scripts/run_platform.py:117
step_prepare · realms/databricks/tasks/main.py:71 (by name)
module level · realms/databricks/v1/tasks/hop.py:28 (by name)
main · scripts/dac_coverage.py:40 (by name)
WallClock.now · scripts/run_platform.py:106 (by name)
write_ops_page · scripts/run_platform.py:130 (by name)
and 21 more
no test names it
write_ops_page
function, line 123
The local ops page (B10): the ops tables the dashboards read, plus every layer against the oracle, the planted incidents' verdicts and the delay per hop, in one HTML file beside the run.main · scripts/run_platform.py:632no test names it
hop_delays
function, line 154
Delay per hop per dataset from silver's stamps (Spark engine only).write_ops_page · scripts/run_platform.py:140
main · scripts/run_platform.py:634
no test names it
success_scorecard
function, line 167
B20: the success criteria this run can measure (obs/success.py), from its own evidence.main · scripts/run_platform.py:633no test names it
write_calendar
function, line 195
The shared, generated date dimension (B15): five years either side of the world's start, fiscal year from April (the Indian financial year), written whole each run (idempotent).main · scripts/run_platform.py:462no test names it
upkeep_all
function, line 212
Upkeep on every table of every layer: on Iceberg, compaction plus snapshot expiry (a 14-day run without it reached 1,483 snapshots and 4,191 data files and ran a 1 GB driver out of heap on day 14, 13 Nov).main.on_day · scripts/run_platform.py:607no test names it
change_feeds
function, line 225
The datasets a change-capture mover delivers (database sources read as a change feed).main · scripts/run_platform.py:415no test names it
mover
function, line 231
main · scripts/run_platform.py:622test_azure_adapters.py, test_cloud_adapters.py and 2 more
mover.move
function, line 235
none found in the codeno test names it
show_silver
function, line 249
main.on_day · scripts/run_platform.py:588no test names it
show_gold
function, line 266
main.on_day · scripts/run_platform.py:601no test names it
gold_metrics
function, line 282
Every certified known answer asked of gold the way the metric view defines it (W5's DP_GOLD_METRICS file).main · scripts/run_platform.py:631no test names it
show
function, line 298
main.on_day · scripts/run_platform.py:575no test names it
main
function, line 315
module level · scripts/run_platform.py:745no test names it
main.pushed
function, line 406
main.run_hop · scripts/run_platform.py:559
main · scripts/run_platform.py:410 (value)
test_obs_dashboards.py, test_obs_traced_store.py
main.reset_tables
function, line 449
main · scripts/run_platform.py:458 (value)no test names it
main.fresh_session
function, line 464
A new session for the next day: every table is on disk, so nothing is lost, and a 1 GB driver never carries fourteen days of plans, caches and Delta log snapshots (Spark ran out of heap on day 12); on DuckDB a new in-memory connection dropsmain.on_day · scripts/run_platform.py:611no test names it
main.live_rows
function, line 478
The live table of a database source now (REC-05: live source against silver); None for other sources.main.on_day · scripts/run_platform.py:582 (value)no test names it
main.housekeeping
function, line 494
Bronze housekeeping (B8) with the run's correction window: old batches go once a checked baseline holds.main.on_day · scripts/run_platform.py:603test_housekeeping.py
main.prove_rebuild_after
function, line 518
main · scripts/run_platform.py:630no test names it
main.run_platform_job
function, line 534
--hops job: the day's mock sources copied where the job can open them, then the deployed platform job (ingestion, silver, gold as three tasks) triggered for the window and waited for.main.run_hop · scripts/run_platform.py:557no test names it
main.run_hop
function, line 548
One hop outside this process, then its saved report read back and the monitor told. process: the hop as its own process (scripts/run_hop.py). job: the deployed platform job runs all three hops once per day; the first hop asked for triggers main.on_day · scripts/run_platform.py:582
main · scripts/run_platform.py:628
test_explainer.py
main.on_day
function, line 574
main · scripts/run_platform.py:615 (value)test_silver_world.py, test_twin_engine.py

scripts/run_world.py

Run the tyre world locally: N simulated days into the four mock sources, with the oracle's totals per day. none yet

DefinitionWhat it does (its own first sentence)Called byNamed in tests
catalog_for
function, line 44
The slice folder a run uses: `--slice 03-clinical` picks slices/03-clinical of the catalog (DP_CATALOG, default ~/data-platform-catalog); the domain, its gold judge and its gold models all follow from the world in it.none found in the codeno test names it
aws_sinks
function, line 50
The aws binding: file drop and stream on the buckets and streams Terraform made (realms/aws-databricks/infra). The database and the dealer interface stay local until their cloud adapters exist.sinks · scripts/run_world.py:84no test names it
sinks
function, line 62
The world's sources. `attach` opens the sources another process already holds (a hop running as its own process reads them); without it every source starts empty.main · scripts/run_world.py:173test_semantic_metrics.py, test_world_clinical.py and 1 more
corrections_store
function, line 89
A metadata store loaded with the slice's manifest, fresh each run, for the corrections the world records.main · scripts/run_world.py:174no test names it
read_back
function, line 111
Read every source through its port, the way an ingestion job would, and count what is there.main · scripts/run_world.py:179no test names it
main
function, line 145
module level · scripts/run_world.py:214no test names it

scripts/show_postgres_delta.py

Demo inspection script: postgres → ADF Mimic → local Delta, with row dump. none yet

DefinitionWhat it does (its own first sentence)Called byNamed in tests
_build_artifact_tree
function, line 50
Hand-roll a minimal LinkedService + 2 Datasets + Pipeline tree.main · scripts/show_postgres_delta.py:287no test names it
_seed_metadata_db
function, line 183
Create the run_log table the mimic emulator writes to.main · scripts/show_postgres_delta.py:289no test names it
_wait_for_postgres
function, line 201
main · scripts/show_postgres_delta.py:256no test names it
main
function, line 216
module level · scripts/show_postgres_delta.py:350no test names it