cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Code map › Who calls what › twin

Data PlatformZoom level 3

Who calls what: twin

Every class, function and method in the twin package, with who calls it. Found by reading the code. A call written as object.method() is matched by the method's name, so a same-named method elsewhere can appear: those rows say by name.

56 definitions in 5 modules. Jump to a module: engine.py · gate.py · learning.py · runner.py · scenario.py

twin/engine.py

An engine layer's run of the world, attached to the DataTwin harness so the scenarios' `engine` checks run. Tests and proof

DefinitionWhat it does (its own first sentence)Called byNamed in tests
LayerRun
class, line 29
from_ingestion · twin/engine.py:44
from_silver · twin/engine.py:49
from_gold · twin/engine.py:53
module level · twin/runner.py:64 (value)
no test names it
_by_incident
function, line 36
from_ingestion · twin/engine.py:44
from_silver · twin/engine.py:49
from_gold · twin/engine.py:53
no test names it
from_ingestion
function, line 43
none found in the codetest_twin_engine.py
from_silver
function, line 48
none found in the codetest_twin_engine.py
from_gold
function, line 52
none found in the codetest_twin_engine.py
check
function, line 56
One `engine` check on a layer's run: the incident's verdict, a day's totals, or a bookmark's movement.check_engine · twin/runner.py:162test_azure_adapters.py, test_contract_rules.py and 30 more

twin/gate.py

The release gate (G10, TEST-03): run a change on clones of prod, compare against the oracle and prod, pass or block. Tests and proof

DefinitionWhat it does (its own first sentence)Called byNamed in tests
GateError
class, line 29
MemoryTables.clone · adapters/memory_tables.py:18
MemoryTables.measure · adapters/memory_tables.py:27
check_name · twin/gate.py:106
test_twin_gate.py
TableMeasures
class, line 34
DatabricksTables.measure.one · adapters/databricks_clone.py:64
DatabricksTables.measure · adapters/databricks_clone.py:70
MemoryTables.measure · adapters/memory_tables.py:35
MemoryTables._measure · adapters/memory_tables.py:44
_compare_windows · twin/gate.py:146
module level · twin/gate.py:39 (value)
test_twin_gate.py
TableSpec
class, line 43
A table the change touches: its prod name, business keys and the measure columns to sum.spec_from_gold · twin/gate.py:100test_lakeformation.py, test_access_model.py and 4 more
Finding
class, line 54
_explain · twin/gate.py:138
_compare_windows · twin/gate.py:152
_compare_table · twin/gate.py:168
module level · twin/gate.py:67 (value)
no test names it
GateResult
class, line 65
run_gate · twin/gate.py:210no test names it
GateResult.blocking
method, line 72
none found in the codetest_twin_gate.py
TableCloner
class, line 76
none found in the codeno test names it
TableCloner.clone
method, line 77
A shallow clone of `source` named `target` (replacing any older clone of that name).run_gate · twin/gate.py:199 (by name)test_twin_gate.py
TableCloner.drop
method, line 81
mover · scripts/run_platform.py:233 (value)
DropMover.__init__ · adapters/local_movers.py:53 (value)
main.reset_tables · scripts/run_platform.py:454 (by name)
SparkGoldEngine.overlay · adapters/spark_gold.py:154 (by name)
SparkGoldEngine.add_keys · adapters/spark_gold.py:202 (by name)
DeltaSilverTables.write · adapters/spark_silver.py:182 (by name)
and 14 more
no test names it
TableMeasurer
class, line 84
none found in the codeno test names it
TableMeasurer.measure
method, line 85
Rows, measure totals and a key digest; with `window`, the same per window value in `windows`.known_answers_sql · realms/databricks/semantic_governance/render.py:144 (value)
_tile_query · realms/databricks/semantic_governance/render.py:167 (value)
_widget · realms/databricks/semantic_governance/render.py:180 (value)
dashboard_json · realms/databricks/semantic_governance/render.py:215 (value)
oracle_value · semantic/metrics.py:280 (value)
_explain · twin/gate.py:133 (value)
and 2 more
test_semantic_governance.py, test_end_to_end.py and 7 more
spec_from_gold
function, line 91
A gold fact's gate spec from its manifest (W1d): the grain as keys, the reconcile measures' columns as the measures to sum, and their tolerance (W3's note: the gate's TableSpec comes from the manifest).none found in the codetest_twin_gate.py
check_name
function, line 104
clone_sql · adapters/databricks_clone.py:33
drop_sql · adapters/databricks_clone.py:37
measure_sql · adapters/databricks_clone.py:47
MemoryTables.clone · adapters/memory_tables.py:17
MemoryTables.drop · adapters/memory_tables.py:22
MemoryTables.measure · adapters/memory_tables.py:26
and 2 more
no test names it
clone_name
function, line 110
The clone sits beside the prod table, named for the change, so two gates never share a clone.run_gate · twin/gate.py:195test_twin_gate.py
_close
function, line 117
_compare_windows · twin/gate.py:151
_compare_table · twin/gate.py:170
no test names it
compare
function, line 121
Findings for one table. `declared` names the measures (and "keys") the change says it changes; `explained` maps a measure, "keys" or "window:<w>" to the reason a difference is expected (DIFF-01).run_gate · twin/gate.py:204test_profiling.py, test_silver_stages.py and 4 more
_explain
function, line 130
compare · twin/gate.py:127no test names it
_compare_windows
function, line 141
compare · twin/gate.py:126no test names it
_compare_table
function, line 160
compare · twin/gate.py:125no test names it
run_gate
function, line 191
none found in the codetest_twin_gate.py

twin/learning.py

The learning loop (G10): every production failure becomes one more scenario. Tests and proof

DefinitionWhat it does (its own first sentence)Called byNamed in tests
_slug
function, line 23
learned_scenario · twin/learning.py:51no test names it
layer_for
function, line 27
learned_scenario · twin/learning.py:61no test names it
failures
function, line 35
What failed in one run: failed or blocked steps, then errors, then failed dataset runs with no step to show.write_learned · twin/learning.py:76test_obs_freshness.py, test_resilience.py and 3 more
learned_scenario
function, line 48
write_learned · twin/learning.py:77test_twin_scenarios.py
write_learned
function, line 70
One scenario file per failure in the run; an existing file is left as it is (someone may have edited it).none found in the codetest_twin_scenarios.py

twin/runner.py

The DataTwin scenario runner and its oracle checks (G10, TEST-05, SIM-01). Tests and proof

DefinitionWhat it does (its own first sentence)Called byNamed in tests
CheckResult
class, line 40
run · twin/runner.py:176
module level · twin/runner.py:52 (value)
no test names it
ScenarioResult
class, line 49
run · twin/runner.py:191no test names it
Harness
class, line 57
none found in the codetest_obs_dashboards.py, test_obs_steps.py and 2 more
Harness.available
method, line 67
none found in the codetest_twin_scenarios.py, test_postgres_to_delta.py
Harness.attach
method, line 70
none found in the codetest_twin_engine.py, test_dq_repository.py
Harness.events
method, line 75
Monitor._evaluate · obs/monitor.py:82 (value)
Monitor.tables · obs/monitor.py:122 (value)
test_baseline_stage.py, test_explain.py and 13 more
Harness.date
method, line 78
The business date of a world day.main.housekeeping · scripts/run_platform.py:512 (value)
main.on_day · scripts/run_platform.py:576 (value)
main · scripts/run_platform.py:651 (value)
module level · realms/databricks/v1/tasks/hop.py:28 (by name)
main · scripts/dac_coverage.py:40 (by name)
write_calendar · scripts/run_platform.py:202 (by name)
and 28 more
no test names it
Harness.build
method, line 83
cmd_code · scripts/explain.py:101 (by name)
_fact · gold/harness.py:291 (by name)
reproduce · gold/inspect.py:78 (by name)
Monitor.tables · obs/monitor.py:122 (by name)
test_semantic_governance.py, conftest.py and 17 more
_dig
function, line 109
check_oracle · twin/runner.py:127no test names it
check_planted
function, line 115
module level · twin/runner.py:165 (value)no test names it
check_oracle
function, line 125
module level · twin/runner.py:165 (value)no test names it
check_on_dashboard
function, line 137
module level · twin/runner.py:165 (value)no test names it
check_one_alert
function, line 151
module level · twin/runner.py:166 (value)no test names it
check_engine
function, line 161
module level · twin/runner.py:166 (value)no test names it
run
function, line 169
Run every check whose layer is available (default: what the harness has); the rest are pending.Harness.attach · twin/runner.py:71 (value)no test names it
unexplained_alerts
function, line 194
SIM-01 for alerts: every alert that fired belongs to a scenario's expected alert (none on a clean day).none found in the codetest_twin_scenarios.py

twin/scenario.py

A DataTwin scenario as a file: `apiVersion: dwd-scenario/v0` (goal G10, TEST-05). Tests and proof

DefinitionWhat it does (its own first sentence)Called byNamed in tests
InvalidScenario
class, line 44
Said at load, where it is cheap, rather than mid-run.parse · twin/scenario.py:80test_twin_scenarios.py
Check
class, line 49
parse · twin/scenario.py:92
module level · twin/scenario.py:66 (value)
no test names it
Check.waits_for
method, line 56
none found in the codetest_twin_scenarios.py
Scenario
class, line 61
parse · twin/scenario.py:99test_silver_scd1.py
Scenario.layers
method, line 71
none found in the codetest_lakeformation.py, test_access_model.py and 11 more
Scenario.pending_layers
method, line 74
none found in the codeno test names it
parse
function, line 78
learned_scenario · twin/learning.py:66
load · twin/scenario.py:106
test_ingest_formats.py, test_ingest_steps.py and 19 more
load
function, line 103
load_all · twin/scenario.py:110no test names it
load_all
function, line 109
none found in the codetest_twin_engine.py, test_twin_scenarios.py