cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Code map › Who calls what › adapters

Data PlatformZoom level 3

Who calls what: adapters

Every class, function and method in the adapters package, with who calls it. Found by reading the code. A call written as object.method() is matched by the method's name, so a same-named method elsewhere can appear: those rows say by name.

663 definitions in 46 modules. Jump to a module: azure_blob.py · customer_runlog.py · databricks_clone.py · databricks_native.py · databricks_notify.py · databricks_ops_views.py · databricks_pipeline_events.py · databricks_system.py · delta_bronze.py · duckdb_frames.py · duckdb_gold.py · duckdb_silver.py · file_drop.py · files.py · http_api.py · keyed_tokens.py · kinesis.py · lakebase.py · lakeview.py · local_movers.py · local_stream.py · memory.py · memory_bronze.py · memory_catalog.py · memory_meta.py · memory_pipeline_events.py · memory_runlog.py · memory_sources.py · memory_system.py · memory_tables.py · null_runlog.py · otel_tracer.py · postgres_meta.py · postgres_runlog.py · postgres_source.py · s3.py · secrets.py · spark_delta.py · spark_gold.py · spark_ops_tables.py · spark_silver.py · sql_source.py · sqlite_source.py · store_registry.py · table_store.py · uc_catalog.py

adapters/azure_blob.py

Object storage on Azure Blob Storage (cloud layer): landing_store, the partner file drop and object_store. Needs azure-storage-blob (the `azure` extra) only when a real client is built; nothing here imports an Azure SDK. The container is pa none yet

DefinitionWhat it does (its own first sentence)Called byNamed in tests
account_name
function, line 35
The storage account for an environment: <prefix><env>. Azure allows lowercase letters and digits only, 3 to 24 characters, and the name is global across Azure; a name that does not fit is refused, never changed.test_prefix_rule_matches_what_azure_allows · realms/azure-databricks/tests/test_realm.py:98
test_example_names_are_legal · realms/azure-databricks/tests/test_realm.py:107
AzureObjectStore.location · adapters/azure_blob.py:162
AzureObjectStore.versioned · adapters/azure_blob.py:167
test_azure_adapters.py
_is_exists
function, line 44
AzureLanding.put · adapters/azure_blob.py:83no test names it
_names
function, line 48
AzureLanding.put · adapters/azure_blob.py:73
AzureLanding.deliveries · adapters/azure_blob.py:90
AzureFileDrop.list · adapters/azure_blob.py:136
no test names it
AzureLanding
class, line 52
landing_store on a blob container (the `landing` container of the environment's account), the same layout and semantics as S3Landing.none found in the codetest_azure_adapters.py, test_cloud_adapters.py
AzureLanding.__init__
method, line 56
none found in the codeno test names it
AzureLanding._dir
method, line 59
AzureLanding.put · adapters/azure_blob.py:71
AzureLanding.deliveries · adapters/azure_blob.py:90
AzureLanding.read_bytes · adapters/azure_blob.py:102
AzureLanding.manifest · adapters/azure_blob.py:105
LandingArchive.archive · adapters/files.py:84 (by name)
LandingArchive.delete · adapters/files.py:90 (by name)
and 1 more
no test names it
AzureLanding.write
method, line 62
none found in the codegold_helpers.py, test_cloud_adapters.py and 31 more
AzureLanding.put
method, line 68
Written once: a delivery whose manifest exists is refused (DUP-02).AzureLanding.write · adapters/azure_blob.py:63
ChangeFeedMover.move · adapters/local_movers.py:38 (by name)
DropMover.move · adapters/local_movers.py:65 (by name)
DmsMover.move · adapters/local_movers.py:105 (by name)
World.write · world/engine.py:133 (by name)
no test names it
AzureLanding.deliveries
method, line 88
step_prepare · realms/databricks/tasks/main.py:81 (by name)
ChangeFeedMover.checkpoint · adapters/local_movers.py:28 (by name)
DropMover.move · adapters/local_movers.py:56 (by name)
push · ingest/pipelines.py:135 (by name)
land · lander.py:31 (by name)
gold_helpers.py, test_cloud_adapters.py and 5 more
AzureLanding.read
method, line 98
none found in the codeno test names it
AzureLanding.read_bytes
method, line 101
AzureLanding.read · adapters/azure_blob.py:99
LocalFileDrop.list · adapters/file_drop.py:38 (by name)
LocalFileDrop.get · adapters/file_drop.py:41 (by name)
FileLanding.deliveries · adapters/files.py:56 (by name)
FileLanding.read_bytes · adapters/files.py:64 (by name)
push · ingest/pipelines.py:144 (by name)
test_ingest_formats.py, test_ingest_ports.py and 3 more
AzureLanding.manifest
method, line 104
ChangeFeedMover.checkpoint · adapters/local_movers.py:28 (by name)test_bundle.py, gold_helpers.py and 34 more
AzureLanding._get
method, line 109
AzureLanding.deliveries · adapters/azure_blob.py:93
AzureLanding.read_bytes · adapters/azure_blob.py:102
AzureLanding.manifest · adapters/azure_blob.py:106
no test names it
AzureFileDrop
class, line 114
FileDrop (source_ports) on a blob container: <prefix><system>/<dataset>/<name>. The service stamps its own upload time, so the arrival time the world declares travels as blob metadata `arrived_at` (with the checksum as `checksum`, so listinnone found in the codetest_azure_adapters.py, test_source_ports.py
AzureFileDrop.__init__
method, line 119
none found in the codeno test names it
AzureFileDrop._dir
method, line 122
AzureFileDrop.put · adapters/azure_blob.py:130
AzureFileDrop.list · adapters/azure_blob.py:136
AzureFileDrop.get · adapters/azure_blob.py:144
LandingArchive.archive · adapters/files.py:84 (by name)
LandingArchive.delete · adapters/files.py:90 (by name)
LandingArchive.listing · adapters/files.py:97 (by name)
no test names it
AzureFileDrop.put
method, line 125
ChangeFeedMover.move · adapters/local_movers.py:38 (by name)
DropMover.move · adapters/local_movers.py:65 (by name)
DmsMover.move · adapters/local_movers.py:105 (by name)
World.write · world/engine.py:133 (by name)
no test names it
AzureFileDrop.list
method, line 134
read_back · scripts/run_world.py:122 (by name)
DropMover.move · adapters/local_movers.py:58 (by name)
no test names it
AzureFileDrop.get
method, line 143
none found in the codeno test names it
AzureObjectStore
class, line 148
object_store: one storage account per environment (account_name) and one container per layer, named exactly by the layer. The location is the ADLS Gen2 (abfss) URL.none found in the codetest_azure_adapters.py
AzureObjectStore.__init__
method, line 156
none found in the codeno test names it
AzureObjectStore.location
method, line 159
none found in the codetest_azure_adapters.py, test_cloud_adapters.py and 2 more
AzureObjectStore.versioned
method, line 164
none found in the codetest_azure_adapters.py, test_cloud_adapters.py and 2 more
_iso
function, line 170
AzureFileDrop.put · adapters/azure_blob.py:128no test names it

adapters/customer_runlog.py

EXAMPLE SKELETON: a customer's own run log on the run-log reader port (level 2 of ADR-024 rule 6). none yet

DefinitionWhat it does (its own first sentence)Called byNamed in tests
CustomerRunLog
class, line 32
none found in the codeno test names it
CustomerRunLog.__init__
method, line 33
none found in the codeno test names it
CustomerRunLog.snapshot
method, line 36
Monitor.run_finished · obs/monitor.py:99 (by name)
Monitor.check_lateness · obs/monitor.py:108 (by name)
Monitor.tables · obs/monitor.py:120 (by name)
test_meta_loader.py, test_obs_recorder.py and 10 more

adapters/databricks_clone.py

The release gate's ports (dataplatform.twin.gate) on Databricks: Delta SHALLOW CLONE and one measuring query. none yet

DefinitionWhat it does (its own first sentence)Called byNamed in tests
_q
function, line 23
Quote a dotted table name or a column, part by part.clone_sql · adapters/databricks_clone.py:33
drop_sql · adapters/databricks_clone.py:37
measure_sql · adapters/databricks_clone.py:42
no test names it
clone_sql
function, line 32
DatabricksTables.clone · adapters/databricks_clone.py:55test_twin_gate.py
drop_sql
function, line 36
DatabricksTables.drop · adapters/databricks_clone.py:58test_twin_gate.py
measure_sql
function, line 40
One row for the table, or one row per window value (`_w`) when `window` names the column (DIFF-01).DatabricksTables.measure · adapters/databricks_clone.py:66test_twin_gate.py
DatabricksTables
class, line 50
none found in the codetest_twin_gate.py
DatabricksTables.__init__
method, line 51
none found in the codeno test names it
DatabricksTables.clone
method, line 54
run_gate · twin/gate.py:199 (by name)test_twin_gate.py
DatabricksTables.drop
method, line 57
mover · scripts/run_platform.py:233 (value)
DropMover.__init__ · adapters/local_movers.py:53 (value)
main.reset_tables · scripts/run_platform.py:454 (by name)
SparkGoldEngine.overlay · adapters/spark_gold.py:154 (by name)
SparkGoldEngine.add_keys · adapters/spark_gold.py:202 (by name)
DeltaSilverTables.write · adapters/spark_silver.py:182 (by name)
and 14 more
no test names it
DatabricksTables.measure
method, line 60
known_answers_sql · realms/databricks/semantic_governance/render.py:144 (value)
_tile_query · realms/databricks/semantic_governance/render.py:167 (value)
_widget · realms/databricks/semantic_governance/render.py:180 (value)
dashboard_json · realms/databricks/semantic_governance/render.py:215 (value)
oracle_value · semantic/metrics.py:280 (value)
_explain · twin/gate.py:133 (value)
and 2 more
test_semantic_governance.py, test_end_to_end.py and 7 more
DatabricksTables.measure.one
function, line 62
DatabricksTables.measure · adapters/databricks_clone.py:66no test names it

adapters/databricks_native.py

Databricks bindings for the adopted platform-native screens and notifications (dataplatform.obs.native). none yet

DefinitionWhat it does (its own first sentence)Called byNamed in tests
render_native_yaml
function, line 53
The adopted set as a deterministic YAML file (no YAML library: plain, sorted, quoted).files · realms/databricks/dashboards/generate.py:35no test names it

adapters/databricks_notify.py

The notifier port (dataplatform.obs.monitor.Notifier) through the workspace's notification destinations. none yet

DefinitionWhat it does (its own first sentence)Called byNamed in tests
UnroutedChannel
class, line 22
An alert's channel has no destination bound: refused at the first send, not dropped silently.DestinationNotifier.__init__ · adapters/databricks_notify.py:39
DestinationNotifier.send · adapters/databricks_notify.py:45
test_obs_notify.py
payload
function, line 26
DestinationNotifier.send · adapters/databricks_notify.py:46test_genie_deploy.py, test_governance_catalog.py and 6 more
DestinationNotifier
class, line 35
none found in the codetest_obs_notify.py
DestinationNotifier.__init__
method, line 36
none found in the codeno test names it
DestinationNotifier.send
method, line 42
Monitor._evaluate · obs/monitor.py:81 (by name)no test names it

adapters/databricks_ops_views.py

The ops metric views (dataplatform.obs.views) as Databricks metric views: one `CREATE ... WITH METRICS` per view. none yet

adapters/databricks_pipeline_events.py

The pipeline event-log port (dataplatform.obs.pipeline_events) on Lakeflow Declarative Pipelines' event log. none yet

DefinitionWhat it does (its own first sentence)Called byNamed in tests
_ts
function, line 47
DatabricksPipelineEvents.table_updates · adapters/databricks_pipeline_events.py:64no test names it
DatabricksPipelineEvents
class, line 51
none found in the codetest_obs_pipeline_events.py
DatabricksPipelineEvents.__init__
method, line 52
`pipelines`: pipeline name -> pipeline id, from the bundle (silver and gold pipelines of a domain).none found in the codeno test names it
DatabricksPipelineEvents.table_updates
method, line 56
Monitor.tables · obs/monitor.py:121 (by name)test_obs_pipeline_events.py

adapters/databricks_system.py

The system-records port (dataplatform.obs.system) on Databricks system tables. none yet

DefinitionWhat it does (its own first sentence)Called byNamed in tests
_ts
function, line 65
_opt_ts · adapters/databricks_system.py:70
DatabricksSystemTables.jobs · adapters/databricks_system.py:82
DatabricksSystemTables.lineage · adapters/databricks_system.py:95
DatabricksSystemTables.access · adapters/databricks_system.py:102
no test names it
_opt_ts
function, line 69
DatabricksSystemTables.jobs · adapters/databricks_system.py:82no test names it
DatabricksSystemTables
class, line 73
none found in the codetest_obs_system.py
DatabricksSystemTables.__init__
method, line 74
none found in the codeno test names it
DatabricksSystemTables._params
method, line 77
DatabricksSystemTables.jobs · adapters/databricks_system.py:85
DatabricksSystemTables.lineage · adapters/databricks_system.py:98
DatabricksSystemTables.access · adapters/databricks_system.py:104
no test names it
DatabricksSystemTables.jobs
method, line 80
DatabricksSystemTables._run_ids · adapters/databricks_system.py:91
read_all · obs/system.py:69 (by name)
no test names it
DatabricksSystemTables._run_ids
method, line 88
Job run id -> platform run id, over a window a day wider than asked (a run may start the day before).DatabricksSystemTables.lineage · adapters/databricks_system.py:94no test names it
DatabricksSystemTables.lineage
method, line 93
read_all · obs/system.py:69 (by name)test_obs_dashboards.py, test_obs_lineage.py and 10 more
DatabricksSystemTables.access
method, line 101
generate · realms/databricks/semantic_governance/generate.py:173 (value)
read_all · obs/system.py:69 (by name)
test_lakeformation.py, test_semantic_governance.py and 5 more

adapters/delta_bronze.py

The bronze writer on Delta tables in a local folder, through delta-rs (`deltalake`, Apache 2.0) and pyarrow: no JVM, no Spark, the same approach as LakeIngestionFramework's LocalDeltaWriter (core/writers/local_delta.py), which only appends; Landing to bronze

DefinitionWhat it does (its own first sentence)Called byNamed in tests
_arrow_type
function, line 21
DeltaBronze.write · adapters/delta_bronze.py:61
DeltaBronze.write_many · adapters/delta_bronze.py:81
no test names it
DeltaBronze
class, line 34
run_ingest · scripts/run_hop.py:58
main · scripts/run_platform.py:492
test_ingest_ports.py, test_ingest_world.py and 2 more
DeltaBronze.__init__
method, line 35
none found in the codeno test names it
DeltaBronze._handle
method, line 39
DeltaBronze.write · adapters/delta_bronze.py:63
DeltaBronze.write_many · adapters/delta_bronze.py:84
DeltaBronze.remove · adapters/delta_bronze.py:95
DeltaBronze._dataset · adapters/delta_bronze.py:103
no test names it
DeltaBronze.path
method, line 51
DeltaBronze._handle · adapters/delta_bronze.py:43
DeltaBronze.write · adapters/delta_bronze.py:65
DeltaBronze.write_many · adapters/delta_bronze.py:87
no test names it
DeltaBronze.write
method, line 56
none found in the codegold_helpers.py, test_cloud_adapters.py and 31 more
DeltaBronze.write_many
method, line 72
New batches in one Delta commit; the lander logs them only after it returns.flush · ingest/steps.py:365 (by name)no test names it
DeltaBronze.remove
method, line 91
Housekeeping (B8): delete the rows of these batches in one Delta commit; returns the rows removed.main.housekeeping · scripts/run_platform.py:506 (value)
_sweep_leased · housekeeping.py:170 (value)
_sweep_leased · housekeeping.py:170 (by name)
RunHandle.dataset · obs/recorder.py:160 (by name)
SilverBaseliner.write · silver/baseline_writer.py:70 (by name)
test_lakeformation.py, test_housekeeping.py and 2 more
DeltaBronze._dataset
method, line 102
DeltaBronze.fingerprint · adapters/delta_bronze.py:121
DeltaBronze.keys · adapters/delta_bronze.py:132
no test names it
DeltaBronze._filter
method, line 107
DeltaBronze.fingerprint · adapters/delta_bronze.py:124
DeltaBronze.keys · adapters/delta_bronze.py:135
no test names it
DeltaBronze.fingerprint
method, line 116
_one · ingest/runner.py:149 (value)
FixtureRunner.run_day · obs/fixture.py:117 (value)
FixtureRunner._rerun · obs/fixture.py:134 (value)
run_days · ingest/daybyday.py:133 (by name)
gold_helpers.py, silver_helpers.py and 14 more
DeltaBronze.keys
method, line 131
none found in the codeno test names it

adapters/duckdb_frames.py

DuckDB as a frame engine for the pure stage and build code, through SQLFrame (the PySpark DataFrame API compiled to SQL by sqlglot): one logic, several engines (docs/DUCKDB-SPIKE.md). For laptop and CI runs only; nothing here ever writes a none yet

DefinitionWhat it does (its own first sentence)Called byNamed in tests
connection
function, line 20
activate · adapters/duckdb_frames.py:36test_lakebase.py, test_resilience.py and 9 more
activate
function, line 26
Point every later `import pyspark...` at SQLFrame on DuckDB (process-wide). Returns the connection.none found in the codeconftest.py
local_session
function, line 41
A SQLFrame DuckDB session (the SparkSession stand-in) on the activated connection, or a new one.none found in the codeconftest.py, test_obs_steps.py
checkpoint
function, line 48
localCheckpoint for a SQLFrame frame: run its query once into a DuckDB table and read that table (a plain table in the session's in-memory database: SQLFrame finds a temporary table's columns in no catalog it reads). Without it a frame builDuckGoldEngine.hold · adapters/duckdb_gold.py:68
DuckStageRunner.run · adapters/duckdb_silver.py:65
silver_helpers.py

adapters/duckdb_gold.py

Gold's engine operations on DuckDB through SQLFrame (docs/DUCKDB-SPIKE.md): the Spark adapter's operations, with the five engine-specific pieces replaced. For laptop and CI runs only; Databricks keeps adapters/spark_gold.py. The harness

DefinitionWhat it does (its own first sentence)Called byNamed in tests
surrogate
function, line 32
keys.surrogate_key: the first 15 hex digits of SHA-256(key text) as a whole number, digit by digit (a hex digit's value from its character code: 0-9 are 48-57, a-f are 97-102).module level · adapters/duckdb_gold.py:58 (value)test_semantic_governance.py, test_gold_plan.py
bind
function, line 42
The query with each :name parameter replaced by its value as a SQL string literal (sqlglot quotes it).DuckGoldEngine.sql · adapters/duckdb_gold.py:65
DuckGoldReader.rows · adapters/duckdb_gold.py:157
no test names it
bind.literal
function, line 49
bind · adapters/duckdb_gold.py:54 (value)test_mimic_expressions.py, test_runtime_sql_views.py
DuckGoldEngine
class, line 57
_duckdb · scripts/platform_binding.py:72test_gold_engine.py
DuckGoldEngine.row_hash
method, line 61
none found in the codeno test names it
DuckGoldEngine.sql
method, line 64
CustomerRunLog.__init__ · adapters/customer_runlog.py:34 (value)
DatabricksPipelineEvents.__init__ · adapters/databricks_pipeline_events.py:54 (value)
module level · realms/databricks/engine/tasks/probe.py:40 (by name)
create · realms/databricks/engine/tasks/probe.py:49 (by name)
module level · realms/databricks/engine/tasks/run_days.py:44 (by name)
create_views · realms/databricks/engine/tasks/semantic_layer.py:25 (by name)
and 41 more
no test names it
DuckGoldEngine.hold
method, line 67
_fact · gold/harness.py:296 (by name)
_key_lookup · gold/harness.py:377 (by name)
no test names it
DuckGoldEngine.release
method, line 70
none found in the codetest_meta_store.py, test_silver_ports.py and 2 more
DeltaRsGoldTables
class, line 74
The read and write edges of gold on Delta folders, by delta-rs; the frames are DuckDB's (SQLFrame)._duckdb · scripts/platform_binding.py:71no test names it
DeltaRsGoldTables._read
method, line 77
DeltaRsGoldTables.silver · adapters/duckdb_gold.py:83no test names it
DeltaRsGoldTables.silver
method, line 81
_dimension · gold/harness.py:179 (by name)
_inputs · gold/harness.py:233 (by name)
reproduce · gold/inspect.py:76 (by name)
test_bundle.py, test_semantic_governance.py and 62 more
DeltaRsGoldTables.silver_version
method, line 92
_inputs · gold/harness.py:232 (by name)no test names it
DeltaRsGoldTables.silver_at
method, line 101
reproduce · gold/inspect.py:76 (by name)no test names it
DeltaRsGoldTables.write_dimension
method, line 108
_dimension · gold/harness.py:186 (by name)test_gold_ports.py
DeltaRsGoldTables.replace_windows
method, line 116
_fact · gold/harness.py:318 (by name)test_gold_ports.py
DuckGoldReader
class, line 138
GoldReader on the gold folders through DuckDB: each table a DuckDB table under its own name, the question (Spark SQL, as the metric views and the judge write it) bound and transpiled to DuckDB by sqlglot._duckdb · scripts/platform_binding.py:72no test names it
DuckGoldReader.__init__
method, line 142
none found in the codeno test names it
DuckGoldReader.rows
method, line 146
none found in the codeno test names it

adapters/duckdb_silver.py

Silver on DuckDB (through SQLFrame) and local Delta tables (through delta-rs): the same stages (dataplatform.stages) as the Spark adapter, other IO (docs/DUCKDB-SPIKE.md). For laptop and CI runs only: nothing here writes a governed table on Writer

DefinitionWhat it does (its own first sentence)Called byNamed in tests
to_arrow
function, line 31
A SQLFrame frame's rows as an Arrow table (the frame's query run on its DuckDB connection).DeltaRsGoldTables.write_dimension · adapters/duckdb_gold.py:111
DeltaRsGoldTables.replace_windows · adapters/duckdb_gold.py:123
DeltaRsSilverTables._append · adapters/duckdb_silver.py:106
DeltaRsSilverTables._merge · adapters/duckdb_silver.py:113
no test names it
read_delta
function, line 37
A Delta folder as a frame (at `version` when given: time travel, INSP-01): delta-rs reads it, DuckDB holds the rows in a table of the session.DeltaRsGoldTables._read · adapters/duckdb_gold.py:78
DeltaRsGoldTables.silver_at · adapters/duckdb_gold.py:102
DeltaRsSilverTables._read · adapters/duckdb_silver.py:97
no test names it
DuckStageRunner
class, line 55
stage_runner on DuckDB: the same run_stages, the rows run once into a table before they are counted and written (the counts and the write read the same rows)._duckdb · scripts/platform_binding.py:69silver_helpers.py
DuckStageRunner.__init__
method, line 59
none found in the codeno test names it
DuckStageRunner.run
method, line 62
none found in the codeno test names it
DuckStageRunner.release
method, line 89
none found in the codetest_meta_store.py, test_silver_ports.py and 2 more
DeltaRsSilverTables
class, line 93
The read and write edges on Delta folders, by delta-rs; the frames are DuckDB's (SQLFrame)._duckdb · scripts/platform_binding.py:68no test names it
DeltaRsSilverTables._read
method, line 96
none found in the codeno test names it
DeltaRsSilverTables._append
method, line 99
step_prepare · realms/databricks/tasks/main.py:70 (by name)no test names it
DeltaRsSilverTables._merge
method, line 110
none found in the codeno test names it
DeltaRsSilverTables.count_by_hour
method, line 126
hourly · silver/daybyday.py:73 (by name)test_silver_ports.py

adapters/file_drop.py

FileDrop on a local folder: the partner and lab drop zones. Object storage is a later adapter with the same port. Sources and movers

adapters/files.py

landing_store on a plain folder. On Databricks the folder is a Unity Catalog volume path, which serverless Python reads and writes as ordinary files, so the same adapter serves both. none yet

DefinitionWhat it does (its own first sentence)Called byNamed in tests
to_bytes
function, line 21
Stable serialisation: the same rows always give the same bytes (so a resend has the same checksum).AzureLanding.write · adapters/azure_blob.py:63
FileLanding.write · adapters/files.py:35
ChangeFeedMover.move · adapters/local_movers.py:43
S3Landing.write · adapters/s3.py:50
no test names it
FileLanding
class, line 26
step_prepare · realms/databricks/tasks/main.py:80
step_land · realms/databricks/tasks/main.py:113
run_ingest · scripts/run_hop.py:56
main · scripts/run_platform.py:431
test_cloud_adapters.py, test_ingest_formats.py and 7 more
FileLanding.__init__
method, line 27
none found in the codeno test names it
FileLanding._dir
method, line 30
FileLanding.put · adapters/files.py:41
FileLanding.deliveries · adapters/files.py:54
FileLanding.read_bytes · adapters/files.py:64
FileLanding.manifest · adapters/files.py:67
LandingArchive.archive · adapters/files.py:84 (by name)
LandingArchive.delete · adapters/files.py:90 (by name)
and 1 more
no test names it
FileLanding.write
method, line 33
Slice 1's movers hand over rows; they land as JSON lines.none found in the codegold_helpers.py, test_cloud_adapters.py and 31 more
FileLanding.put
method, line 37
FileLanding.write · adapters/files.py:35
ChangeFeedMover.move · adapters/local_movers.py:38 (by name)
DropMover.move · adapters/local_movers.py:65 (by name)
DmsMover.move · adapters/local_movers.py:105 (by name)
World.write · world/engine.py:133 (by name)
no test names it
FileLanding.deliveries
method, line 52
step_prepare · realms/databricks/tasks/main.py:81 (by name)
ChangeFeedMover.checkpoint · adapters/local_movers.py:28 (by name)
DropMover.move · adapters/local_movers.py:56 (by name)
push · ingest/pipelines.py:135 (by name)
land · lander.py:31 (by name)
gold_helpers.py, test_cloud_adapters.py and 5 more
FileLanding.read
method, line 60
none found in the codeno test names it
FileLanding.read_bytes
method, line 63
FileLanding.read · adapters/files.py:61
LocalFileDrop.list · adapters/file_drop.py:38 (by name)
LocalFileDrop.get · adapters/file_drop.py:41 (by name)
FileLanding.deliveries · adapters/files.py:56 (by name)
push · ingest/pipelines.py:144 (by name)
test_ingest_formats.py, test_ingest_ports.py and 3 more
FileLanding.manifest
method, line 66
ChangeFeedMover.checkpoint · adapters/local_movers.py:28 (by name)test_bundle.py, gold_helpers.py and 34 more
LandingArchive
class, line 72
The landing archive on a local folder (ADR-029): <root>/_archive/<source>/<dataset>/<window>/. Moving a delivery keeps its file and manifest together; on a cloud this is a cheaper storage class set by lifecycle rules.none found in the codetest_retention.py
LandingArchive.__init__
method, line 76
none found in the codeno test names it
LandingArchive._dir
method, line 80
LandingArchive.archive · adapters/files.py:84
LandingArchive.delete · adapters/files.py:90
LandingArchive.listing · adapters/files.py:97
LandingArchive.archive · adapters/files.py:84 (by name)
LandingArchive.delete · adapters/files.py:90 (by name)
LandingArchive.listing · adapters/files.py:97 (by name)
no test names it
LandingArchive.archive
method, line 83
apply_landing_retention · retention.py:100 (by name)test_retention.py, test_sftp_connector.py
LandingArchive.delete
method, line 89
DeltaBronze.remove · adapters/delta_bronze.py:99 (by name)
apply_landing_retention · retention.py:102 (by name)
test_baseline_stage.py, test_dac_coverage.py and 20 more
LandingArchive.listing
method, line 94
(window, name, place) for every delivery, hot and archived.apply_landing_retention · retention.py:94 (by name)azure_fakes.py, test_obs_dashboards.py and 2 more

adapters/http_api.py

RecordInterface as a small local HTTP service: the dealer management system. Sources and movers

DefinitionWhat it does (its own first sentence)Called byNamed in tests
LocalRecordStore
class, line 26
sinks · scripts/run_world.py:70
main · scripts/run_world.py:164
test_source_ports.py
LocalRecordStore.__init__
method, line 27
none found in the codeno test names it
LocalRecordStore.reset
method, line 37
none found in the codetest_governance_catalog.py, test_meta_store.py and 7 more
LocalRecordStore.upsert
method, line 42
HttpRecordClient.upsert · adapters/http_api.py:116 (by name)
World.write · world/engine.py:137 (by name)
test_baseline_stage.py, test_ingest_pipelines.py and 5 more
LocalRecordStore.page
method, line 51
read_back · scripts/run_world.py:129 (by name)
_handler.Handler.do_GET · adapters/http_api.py:79 (by name)
pull_pages · ingest/pipelines.py:288 (by name)
no test names it
_handler
function, line 65
serve · adapters/http_api.py:98no test names it
_handler.Handler
class, line 66
none found in the codeno test names it
_handler.Handler.do_GET
method, line 67
none found in the codeno test names it
_handler.Handler._send
method, line 82
_handler.Handler.do_GET · adapters/http_api.py:71 (by name)no test names it
_handler.Handler.log_message
method, line 90
none found in the codeno test names it
serve
function, line 96
Start the interface in a background thread; port 0 picks a free port (server.server_address[1]).run_ingest · scripts/run_hop.py:53
main · scripts/run_platform.py:436
read_back · scripts/run_world.py:125
main · scripts/run_world.py:164
test_source_ports.py
HttpRecordClient
class, line 106
Reads the interface over HTTP. Writes go to the store: a real dealer system is not written by its readers.run_ingest · scripts/run_hop.py:54
main · scripts/run_platform.py:437
read_back · scripts/run_world.py:126
test_source_ports.py
HttpRecordClient.__init__
method, line 109
none found in the codeno test names it
HttpRecordClient.upsert
method, line 113
World.write · world/engine.py:137 (by name)test_baseline_stage.py, test_ingest_pipelines.py and 5 more
HttpRecordClient.page
method, line 118
read_back · scripts/run_world.py:129 (by name)
_handler.Handler.do_GET · adapters/http_api.py:79 (by name)
pull_pages · ingest/pipelines.py:288 (by name)
no test names it

adapters/keyed_tokens.py

The tokenizer as a keyed hash (HMAC-SHA256, standard library): the same value always gives the same token, so joins and duplicate checks still work, and nobody without the key can reverse or recompute it. The key is the binding's secret (DP none yet

DefinitionWhat it does (its own first sentence)Called byNamed in tests
KeyedTokenizer
class, line 16
none found in the codetest_ingest_pipelines.py, test_ingest_ports.py and 4 more
KeyedTokenizer.__init__
method, line 19
none found in the codeno test names it
KeyedTokenizer.token
method, line 24
main.on_day · scripts/run_platform.py:582 (value)
instance_token · adapters/lakebase.py:65 (value)
main.live_rows · scripts/run_platform.py:489 (by name)
mass_delete_problem.stored · ingest/steps.py:256 (by name)
protect · ingest/steps.py:291 (by name)
test_bundle.py, test_semantic_governance.py and 18 more
KeyedTokenizer.from_env_or_file
method, line 29
The key from the environment, else from a local file made once with a random key (local runs only).run_ingest · scripts/run_hop.py:60 (by name)
main · scripts/run_platform.py:443 (by name)
test_ingest_ports.py

adapters/kinesis.py

stream_bus on Amazon Kinesis Data Streams (cloud layer, dbs/cloud/aws.yaml). Needs boto3; the client is passed in, so tests use a local stand-in (moto) and the platform passes one built from the job's role. Sources and movers

DefinitionWhat it does (its own first sentence)Called byNamed in tests
KinesisStream
class, line 20
aws_sinks · scripts/run_world.py:59test_cloud_adapters.py, test_source_ports.py
KinesisStream.__init__
method, line 21
none found in the codeno test names it
KinesisStream._name
method, line 24
KinesisStream.append · adapters/kinesis.py:35
KinesisStream.shards · adapters/kinesis.py:46
KinesisStream.read · adapters/kinesis.py:58
no test names it
KinesisStream.append
method, line 27
none found in the codeazure_fakes.py, gold_helpers.py and 45 more
KinesisStream.shards
method, line 44
read_back · scripts/run_world.py:137 (by name)
stream_events · ingest/pipelines.py:342 (by name)
test_cloud_adapters.py, test_ingest_pipelines.py and 2 more
KinesisStream.read
method, line 54
none found in the codeno test names it

adapters/lakebase.py

The metadata store on Lakebase, Databricks' managed PostgreSQL (ADR-010, binding dbs/engine/databricks.yaml). Metadata store

DefinitionWhat it does (its own first sentence)Called byNamed in tests
_project_endpoint
function, line 22
(endpoint resource name, host) of a Lakebase project's read-write endpoint on its default branch, or None when the workspace has no such project. Lakebase on Azure Databricks is the project (autoscaling) kind, found over REST so an older SDinstance_dsn · adapters/lakebase.py:46
instance_token · adapters/lakebase.py:59
no test names it
instance_dsn
function, line 44
host, port, database and user for the instance; no password (the token is passed separately).connect_lakebase · adapters/lakebase.py:81test_lakebase.py
instance_token
function, line 58
connect_lakebase · adapters/lakebase.py:81test_lakebase.py
connect_lakebase
function, line 70
A metadata store on a Lakebase instance, connected as the caller's own identity.attach_meta · scripts/platform_binding.py:36
lakebase_fresh_store · scripts/run_platform.py:87
meta_store_from_env · adapters/lakebase.py:91
test_lakebase.py
meta_store_from_env
function, line 86
The binding's choice, from the environment: DP_META_INSTANCE (+ DP_META_DATABASE) means Lakebase as this identity; otherwise DP_META_DSN (plain PostgreSQL, the laptop and other clouds); one adapter either way.module level · realms/databricks/declarative/silver.py:71
lakebase · realms/databricks/engine/tasks/probe.py:79
test_lakebase.py

adapters/lakeview.py

The ops dashboards (dataplatform.obs.dashboards) as Databricks AI/BI (Lakeview) dashboard files. none yet

DefinitionWhat it does (its own first sentence)Called byNamed in tests
_ident
function, line 29
_measure_sql · adapters/lakeview.py:47
query_sql · adapters/lakeview.py:55
_field · adapters/lakeview.py:85
no test names it
_literal
function, line 35
query_sql · adapters/lakeview.py:69no test names it
_measure_sql
function, line 43
query_sql · adapters/lakeview.py:61no test names it
query_sql
function, line 51
The query as SQL lines (Lakeview keeps a dataset's query as a list of lines).compile_dashboard · adapters/lakeview.py:135test_obs_dashboards.py
_field
function, line 84
_widget · adapters/lakeview.py:107no test names it
_encodings
function, line 88
_widget · adapters/lakeview.py:109no test names it
_widget
function, line 102
_layout · adapters/lakeview.py:126no test names it
_layout
function, line 114
compile_dashboard · adapters/lakeview.py:136no test names it
compile_dashboard
function, line 131
render_file · adapters/lakeview.py:141test_obs_dashboards.py
render_file
function, line 140
files · realms/databricks/dashboards/generate.py:31no test names it
bundle_resources
function, line 144
The bundle resource file: one dashboard resource per file, published to the bundle's SQL warehouse. `native` names the platform's own dashboards kept unchanged under native/<id>.lvdash.json (adopted, not rebuilt).files · realms/databricks/dashboards/generate.py:34no test names it

adapters/local_movers.py

The push movers for the local stack: they stand in for what a cloud supplies (a change-capture service, a file transfer service) and meet the landing contract every push mover meets (DPC-0038): each delivery written once, under a unique nam Sources and movers

DefinitionWhat it does (its own first sentence)Called byNamed in tests
ChangeFeedMover
class, line 23
mover · scripts/run_platform.py:232test_ingest_pipelines.py, test_ingest_world.py and 2 more
ChangeFeedMover.__init__
method, line 24
none found in the codeno test names it
ChangeFeedMover.checkpoint
method, line 27
ChangeFeedMover.move · adapters/local_movers.py:32
DmsMover.move · adapters/local_movers.py:87 (by name)
silver_helpers.py
ChangeFeedMover.move
method, line 31
mover.move · scripts/run_platform.py:242 (by name)no test names it
DropMover
class, line 51
mover · scripts/run_platform.py:233test_ingest_pipelines.py, test_ingest_world.py and 2 more
DropMover.__init__
method, line 52
none found in the codeno test names it
DropMover.move
method, line 55
mover.move · scripts/run_platform.py:242 (by name)no test names it
DmsMover
class, line 71
A change-capture mover shaped like a managed database migration service writing to object storage (B4, ADR-031): the same changes as ChangeFeedMover, delivered the way such a service writes them. The first delivery is a full-load file (LOADmover · scripts/run_platform.py:232test_ingest_formats.py
DmsMover.move
method, line 83
mover.move · scripts/run_platform.py:242 (by name)no test names it

adapters/local_stream.py

EventStream on local files: a log of shards, each an append-only JSON-lines file. A managed stream is a later adapter with the same port; the shard count and the partition-key hashing mirror how such streams spread records. Sources and movers

adapters/memory.py

In-memory adapters for tests. Same ports as the Spark/Delta ones. none yet

DefinitionWhat it does (its own first sentence)Called byNamed in tests
MemoryMeta
class, line 9
none found in the codetest_cloud_adapters.py, test_slice1_local.py
MemoryMeta.__init__
method, line 10
none found in the codeno test names it
MemoryMeta.dataset_config
method, line 14
step_land · realms/databricks/tasks/main.py:112 (by name)
step_finish · realms/databricks/tasks/main.py:123 (by name)
test_exceptions.py, test_runtime_sql_views.py
MemoryMeta.batch_status
method, line 17
StampingRunLog.batch_status · adapters/memory_runlog.py:96 (by name)
StoreBatchRegistry.batch_status · adapters/store_registry.py:24 (by name)
enter · ingest/steps.py:318 (by name)
land · lander.py:34 (by name)
test_meta_store.py, test_slice1_local.py
MemoryMeta.record_batch
method, line 20
land · lander.py:38 (by name)no test names it
MemoryTables
class, line 24
none found in the codetest_cloud_adapters.py, test_slice1_local.py and 1 more
MemoryTables.__init__
method, line 25
none found in the codeno test names it
MemoryTables.append
method, line 28
none found in the codeazure_fakes.py, gold_helpers.py and 45 more
MemoryLog
class, line 33
none found in the codetest_cloud_adapters.py, test_slice1_local.py
MemoryLog.__init__
method, line 34
none found in the codeno test names it
MemoryLog.write
method, line 37
none found in the codegold_helpers.py, test_cloud_adapters.py and 31 more

adapters/memory_bronze.py

The bronze writer in memory, for tests: the reference behaviour the table-format adapters are tested against (M5). none yet

adapters/memory_catalog.py

Catalog state in memory: the reference adapter for the CatalogState port (tests, dry runs, the first deploy). none yet

adapters/memory_meta.py

The metadata store in memory, for unit tests. Same port and the same refusals as the PostgreSQL adapter; the contract test table (tests/dataplatform/test_meta_store.py) runs every row against both (M5). Metadata store

DefinitionWhat it does (its own first sentence)Called byNamed in tests
_plain
function, line 45
MemoryMetaStore.load_domain · adapters/memory_meta.py:86
MemoryMetaStore.load_dataset · adapters/memory_meta.py:100
MemoryMetaStore.start_dataset_run · adapters/memory_meta.py:187
MemoryMetaStore.run_report · adapters/memory_meta.py:294
no test names it
MemoryMetaStore
class, line 49
meta_store · scripts/run_platform.py:65
corrections_store · scripts/run_world.py:106
Harness.build · twin/runner.py:90
test_gold_harness.py, test_housekeeping.py and 20 more
MemoryMetaStore.__init__
method, line 50
none found in the codeno test names it
MemoryMetaStore.register_settings
method, line 73
StampingRunLog.register_settings · adapters/memory_runlog.py:48 (by name)
load · meta/loader.py:174 (by name)
test_housekeeping.py, test_ingest_pipelines.py and 7 more
MemoryMetaStore._known
method, line 79
MemoryMetaStore.load_domain · adapters/memory_meta.py:84
MemoryMetaStore.load_dataset · adapters/memory_meta.py:92
no test names it
MemoryMetaStore.load_domain
method, line 82
StampingRunLog.load_domain · adapters/memory_runlog.py:51 (by name)
load · meta/loader.py:175 (by name)
gold_helpers.py, test_housekeeping.py and 10 more
MemoryMetaStore.load_dataset
method, line 88
StampingRunLog.load_dataset · adapters/memory_runlog.py:54 (by name)
load · meta/loader.py:179 (by name)
load_gold · meta/loader.py:235 (by name)
gold_helpers.py, test_housekeeping.py and 10 more
MemoryMetaStore._get
method, line 102
MemoryMetaStore.settings · adapters/memory_meta.py:117
MemoryMetaStore.columns · adapters/memory_meta.py:134
MemoryMetaStore.business_keys · adapters/memory_meta.py:139
MemoryMetaStore.dataset_view · adapters/memory_meta.py:142
MemoryMetaStore.start_dataset_run · adapters/memory_meta.py:184
MemoryMetaStore.void_batch · adapters/memory_meta.py:230
and 9 more
no test names it
MemoryMetaStore.settings_registry
method, line 108
StampingRunLog.settings_registry · adapters/memory_runlog.py:57 (by name)test_meta_store.py
MemoryMetaStore.datasets
method, line 111
Monitor._lateness · obs/monitor.py:88 (value)
main · realms/aws-databricks/meta_setup.py:85 (by name)
module level · realms/databricks/declarative/silver.py:73 (by name)
lakebase · realms/databricks/engine/tasks/probe.py:80 (by name)
hop_delays · scripts/run_platform.py:160 (by name)
StampingRunLog.datasets · adapters/memory_runlog.py:60 (by name)
and 10 more
test_bundle.py, test_semantic_governance.py and 29 more
MemoryMetaStore.settings
method, line 116
MemoryMetaStore.dataset_view · adapters/memory_meta.py:143
_read · ingest/pipelines.py:67 (value)
push · ingest/pipelines.py:147 (value)
enter · ingest/steps.py:328 (value)
main · realms/aws-databricks/meta_setup.py:88 (by name)
StampingRunLog.settings · adapters/memory_runlog.py:63 (by name)
test_bundle.py, gold_helpers.py and 20 more
MemoryMetaStore.columns
method, line 131
MemoryMetaStore.dataset_view · adapters/memory_meta.py:152
MemoryMetaStore.set_dq_rules · adapters/memory_meta.py:348
_ingest_table_to_bronze · scripts/demo_end_to_end.py:389 (value)
CustomerRunLog.__init__ · adapters/customer_runlog.py:34 (value)
_widget · adapters/lakeview.py:103 (value)
UcIcebergStore._create · adapters/table_store.py:403 (value)
and 10 more
test_lakeformation.py, test_semantic_governance.py and 67 more
MemoryMetaStore.business_keys
method, line 136
MemoryMetaStore.dataset_view · adapters/memory_meta.py:153
StampingRunLog.business_keys · adapters/memory_runlog.py:69 (by name)
plans · gold/harness.py:96 (by name)
_inputs · gold/harness.py:245 (by name)
gold_helpers.py, silver_helpers.py and 26 more
MemoryMetaStore.dataset_view
method, line 141
module level · realms/databricks/declarative/silver.py:77 (by name)
StampingRunLog.dataset_view · adapters/memory_runlog.py:72 (by name)
StoreBatchRegistry.dataset_config · adapters/store_registry.py:21 (by name)
plans · gold/harness.py:94 (by name)
run_gold · gold/harness.py:138 (by name)
_sweep · housekeeping.py:142 (by name)
and 7 more
test_ingest_pipelines.py, test_meta_loader.py and 4 more
MemoryMetaStore._run
method, line 158
MemoryMetaStore._ds_run · adapters/memory_meta.py:164
MemoryMetaStore.start_dataset_run · adapters/memory_meta.py:182
MemoryMetaStore.log_error · adapters/memory_meta.py:243
MemoryMetaStore.finish_run · adapters/memory_meta.py:263
MemoryMetaStore.run_report · adapters/memory_meta.py:291
MemoryMetaStore.dq_results · adapters/memory_meta.py:379
no test names it
MemoryMetaStore._ds_run
method, line 163
MemoryMetaStore.log_step · adapters/memory_meta.py:197
MemoryMetaStore.log_file · adapters/memory_meta.py:206
MemoryMetaStore.finish_dataset_run · adapters/memory_meta.py:252
MemoryMetaStore.log_dq_results · adapters/memory_meta.py:368
MemoryMetaStore.save_profile · adapters/memory_meta.py:385
no test names it
MemoryMetaStore.start_run
method, line 170
StampingRunLog.start_run · adapters/memory_runlog.py:77 (by name)
run_gold · gold/harness.py:132 (by name)
run_housekeeping · housekeeping.py:127 (by name)
ingest · ingest/runner.py:126 (by name)
RunRecorder.run · obs/recorder.py:191 (by name)
TracedStore.start_run · obs/traced_store.py:33 (by name)
and 1 more
gold_helpers.py, test_housekeeping.py and 7 more
MemoryMetaStore.start_dataset_run
method, line 180
StampingRunLog.start_dataset_run · adapters/memory_runlog.py:83 (by name)
run_gold · gold/harness.py:139 (by name)
_sweep · housekeeping.py:146 (by name)
_one · ingest/runner.py:149 (by name)
RunHandle.dataset · obs/recorder.py:148 (by name)
TracedStore.start_dataset_run · obs/traced_store.py:39 (by name)
and 2 more
gold_helpers.py, test_housekeeping.py and 7 more
MemoryMetaStore.log_step
method, line 194
StampingRunLog.log_step · adapters/memory_runlog.py:88 (by name)
_step · gold/harness.py:168 (by name)
_sweep_leased · housekeeping.py:167 (by name)
_profile_step · ingest/runner.py:72 (by name)
_detect_steps · ingest/runner.py:109 (by name)
_one_leased · ingest/runner.py:196 (by name)
and 5 more
test_meta_store.py, test_obs_traced_store.py
MemoryMetaStore.log_file
method, line 201
StampingRunLog.log_file · adapters/memory_runlog.py:92 (by name)
StoreBatchRegistry.record_batch · adapters/store_registry.py:28 (by name)
_record · ingest/steps.py:381 (by name)
DatasetHandle.file · obs/recorder.py:112 (by name)
SilverBaseliner.write · silver/baseline_writer.py:75 (by name)
gold_helpers.py, test_housekeeping.py and 4 more
MemoryMetaStore.batch_status
method, line 209
MemoryMetaStore.void_batch · adapters/memory_meta.py:234
StampingRunLog.batch_status · adapters/memory_runlog.py:96 (by name)
StoreBatchRegistry.batch_status · adapters/store_registry.py:24 (by name)
enter · ingest/steps.py:318 (by name)
land · lander.py:34 (by name)
test_meta_store.py, test_slice1_local.py
MemoryMetaStore.acquire_lease
method, line 215
StampingRunLog.acquire_lease · adapters/memory_runlog.py:143 (by name)
run_gold · gold/harness.py:141 (by name)
_sweep · housekeeping.py:148 (by name)
_one · ingest/runner.py:151 (by name)
_one · silver/runner.py:121 (by name)
test_housekeeping.py, test_meta_store.py and 1 more
MemoryMetaStore.release_lease
method, line 225
StampingRunLog.release_lease · adapters/memory_runlog.py:146 (by name)
run_gold · gold/harness.py:153 (by name)
_sweep · housekeeping.py:155 (by name)
_one · ingest/runner.py:158 (by name)
_one · silver/runner.py:128 (by name)
test_meta_store.py
MemoryMetaStore.void_batch
method, line 229
StampingRunLog.void_batch · adapters/memory_runlog.py:149 (by name)
_sweep_leased · housekeeping.py:173 (by name)
withdraw · recovery.py:20 (by name)
test_meta_store.py, test_silver_runner.py
MemoryMetaStore.log_error
method, line 241
StampingRunLog.log_error · adapters/memory_runlog.py:100 (by name)
_fail · gold/harness.py:173 (by name)
_one_leased · ingest/runner.py:175 (by name)
DatasetHandle.error · obs/recorder.py:107 (by name)
RunHandle.dataset · obs/recorder.py:154 (by name)
RunRecorder.run · obs/recorder.py:197 (by name)
and 1 more
test_meta_store.py
MemoryMetaStore.finish_dataset_run
method, line 247
StampingRunLog.finish_dataset_run · adapters/memory_runlog.py:106 (by name)
_fail · gold/harness.py:174 (by name)
_dimension · gold/harness.py:189 (by name)
_fact · gold/harness.py:342 (by name)
_sweep_leased · housekeeping.py:177 (by name)
_one_leased · ingest/runner.py:176 (by name)
and 5 more
gold_helpers.py, test_housekeeping.py and 5 more
MemoryMetaStore.finish_run
method, line 262
StampingRunLog.finish_run · adapters/memory_runlog.py:110 (by name)
run_gold · gold/harness.py:156 (by name)
run_housekeeping · housekeeping.py:134 (by name)
ingest · ingest/runner.py:135 (by name)
RunRecorder.run · obs/recorder.py:201 (by name)
TracedStore.finish_run · obs/traced_store.py:60 (by name)
and 1 more
gold_helpers.py, test_housekeeping.py and 4 more
MemoryMetaStore.watermark
method, line 272
push · ingest/pipelines.py:161 (value)
pull_pages · ingest/pipelines.py:302 (value)
stream_events · ingest/pipelines.py:357 (value)
DailyCopy.__init__ · obs/system.py:84 (value)
DailyCopy.copy · obs/system.py:92 (value)
DailyCopy.on_demand · obs/system.py:100 (value)
and 2 more
gold_helpers.py, test_ingest_pipelines.py and 13 more
MemoryMetaStore.bookmarks
method, line 276
StampingRunLog.bookmarks · adapters/memory_runlog.py:118 (by name)
silver_watermark · gold/harness.py:106 (by name)
feed_marks · gold/harness.py:122 (by name)
_fact · gold/harness.py:289 (by name)
push · ingest/pipelines.py:156 (by name)
_feed_bookmarks · ingest/pipelines.py:196 (by name)
and 6 more
gold_helpers.py, test_gold_harness.py and 6 more
MemoryMetaStore.reset_watermark
method, line 280
StampingRunLog.reset_watermark · adapters/memory_runlog.py:122 (by name)test_meta_store.py
MemoryMetaStore.run_report
method, line 290
success_scorecard · scripts/run_platform.py:184 (by name)
StampingRunLog.run_report · adapters/memory_runlog.py:125 (by name)
StampingRunLog.snapshot · adapters/memory_runlog.py:175 (by name)
_steps · gold/inspect.py:55 (by name)
test_ingest_pipelines.py, test_meta_store.py and 5 more
MemoryMetaStore.dataset_files
method, line 303
StampingRunLog.dataset_files · adapters/memory_runlog.py:128 (by name)
silver_watermark · gold/harness.py:107 (by name)
_sweep_leased · housekeeping.py:162 (by name)
push · ingest/pipelines.py:132 (by name)
pull_copy · ingest/pipelines.py:250 (by name)
pull_pages · ingest/pipelines.py:278 (by name)
and 4 more
gold_helpers.py, test_ingest_pipelines.py and 2 more
MemoryMetaStore.record_corrections
method, line 310
StampingRunLog.record_corrections · adapters/memory_runlog.py:132 (by name)
withdraw · recovery.py:23 (by name)
World.write · world/engine.py:145 (by name)
test_gold_harness.py, test_meta_store.py
MemoryMetaStore.corrections
method, line 330
_inputs · gold/harness.py:243 (value)
_fact · gold/harness.py:333 (value)
_record · gold/harness.py:456 (value)
main · scripts/run_platform.py:669 (by name)
main · scripts/run_world.py:189 (by name)
StampingRunLog.corrections · adapters/memory_runlog.py:135 (by name)
and 1 more
gold_helpers.py, test_gold_engine.py and 7 more
MemoryMetaStore.backfill_plans
method, line 342
main · scripts/run_world.py:188 (by name)
StampingRunLog.backfill_plans · adapters/memory_runlog.py:138 (by name)
_active_plans · gold/harness.py:208 (by name)
test_meta_store.py, test_world_tyre.py
MemoryMetaStore.set_dq_rules
method, line 347
StampingRunLog.set_dq_rules · adapters/memory_runlog.py:152 (by name)
load · meta/loader.py:180 (by name)
test_meta_store.py, test_silver_runner.py
MemoryMetaStore.dq_rules
method, line 362
module level · realms/databricks/declarative/silver.py:81 (by name)
lakebase · realms/databricks/engine/tasks/probe.py:83 (by name)
StampingRunLog.dq_rules · adapters/memory_runlog.py:155 (by name)
SilverBaseliner.write · silver/baseline_writer.py:56 (by name)
_one_leased · silver/runner.py:148 (by name)
test_meta_loader.py, test_meta_store.py and 2 more
MemoryMetaStore.log_dq_results
method, line 366
StampingRunLog.log_dq_results · adapters/memory_runlog.py:158 (by name)test_meta_store.py
MemoryMetaStore.dq_results
method, line 378
StampingRunLog.dq_results · adapters/memory_runlog.py:161 (by name)test_meta_store.py, test_cli_dq.py and 1 more
MemoryMetaStore.save_profile
method, line 384
StampingRunLog.save_profile · adapters/memory_runlog.py:164 (by name)
_profile_step · ingest/runner.py:75 (by name)
test_meta_store.py
MemoryMetaStore.last_profile
method, line 388
StampingRunLog.last_profile · adapters/memory_runlog.py:167 (by name)
_profile_step · ingest/runner.py:69 (by name)
test_meta_store.py

adapters/memory_pipeline_events.py

The pipeline event-log port (dataplatform.obs.pipeline_events) in memory: rows added by tests. none yet

DefinitionWhat it does (its own first sentence)Called byNamed in tests
MemoryPipelineEvents
class, line 10
none found in the codetest_obs_pipeline_events.py
MemoryPipelineEvents.__init__
method, line 11
none found in the codeno test names it
MemoryPipelineEvents.table_updates
method, line 14
Monitor.tables · obs/monitor.py:121 (by name)test_obs_pipeline_events.py

adapters/memory_runlog.py

StampingRunLog: any metadata store, plus the run-log reader port (dataplatform.obs.runlog) in process. none yet

DefinitionWhat it does (its own first sentence)Called byNamed in tests
StampingRunLog
class, line 36
main · scripts/run_platform.py:397
Harness.build · twin/runner.py:90
test_obs_recorder.py, test_obs_runlog.py and 1 more
StampingRunLog.__init__
method, line 37
none found in the codeno test names it
StampingRunLog.register_settings
method, line 47
load · meta/loader.py:174 (by name)test_housekeeping.py, test_ingest_pipelines.py and 7 more
StampingRunLog.load_domain
method, line 50
load · meta/loader.py:175 (by name)gold_helpers.py, test_housekeeping.py and 10 more
StampingRunLog.load_dataset
method, line 53
load · meta/loader.py:179 (by name)
load_gold · meta/loader.py:235 (by name)
gold_helpers.py, test_housekeeping.py and 10 more
StampingRunLog.settings_registry
method, line 56
none found in the codetest_meta_store.py
StampingRunLog.datasets
method, line 59
Monitor._lateness · obs/monitor.py:88 (value)
main · realms/aws-databricks/meta_setup.py:85 (by name)
module level · realms/databricks/declarative/silver.py:73 (by name)
lakebase · realms/databricks/engine/tasks/probe.py:80 (by name)
hop_delays · scripts/run_platform.py:160 (by name)
plans · gold/harness.py:91 (by name)
and 9 more
test_bundle.py, test_semantic_governance.py and 29 more
StampingRunLog.settings
method, line 62
_read · ingest/pipelines.py:67 (value)
push · ingest/pipelines.py:147 (value)
enter · ingest/steps.py:328 (value)
main · realms/aws-databricks/meta_setup.py:88 (by name)
test_bundle.py, gold_helpers.py and 20 more
StampingRunLog.columns
method, line 65
_ingest_table_to_bronze · scripts/demo_end_to_end.py:389 (value)
CustomerRunLog.__init__ · adapters/customer_runlog.py:34 (value)
_widget · adapters/lakeview.py:103 (value)
UcIcebergStore._create · adapters/table_store.py:403 (value)
validate · obs/dashboards.py:105 (value)
contract_check · stages/clean.py:44 (value)
and 7 more
test_lakeformation.py, test_semantic_governance.py and 67 more
StampingRunLog.business_keys
method, line 68
plans · gold/harness.py:96 (by name)
_inputs · gold/harness.py:245 (by name)
gold_helpers.py, silver_helpers.py and 26 more
StampingRunLog.dataset_view
method, line 71
module level · realms/databricks/declarative/silver.py:77 (by name)
StoreBatchRegistry.dataset_config · adapters/store_registry.py:21 (by name)
plans · gold/harness.py:94 (by name)
run_gold · gold/harness.py:138 (by name)
_sweep · housekeeping.py:142 (by name)
ingest · ingest/runner.py:133 (by name)
and 6 more
test_ingest_pipelines.py, test_meta_loader.py and 4 more
StampingRunLog.start_run
method, line 75
run_gold · gold/harness.py:132 (by name)
run_housekeeping · housekeeping.py:127 (by name)
ingest · ingest/runner.py:126 (by name)
RunRecorder.run · obs/recorder.py:191 (by name)
TracedStore.start_run · obs/traced_store.py:33 (by name)
build_silver · silver/runner.py:85 (by name)
gold_helpers.py, test_housekeeping.py and 7 more
StampingRunLog.start_dataset_run
method, line 81
run_gold · gold/harness.py:139 (by name)
_sweep · housekeeping.py:146 (by name)
_one · ingest/runner.py:149 (by name)
RunHandle.dataset · obs/recorder.py:148 (by name)
TracedStore.start_dataset_run · obs/traced_store.py:39 (by name)
_one · silver/runner.py:123 (by name)
and 1 more
gold_helpers.py, test_housekeeping.py and 7 more
StampingRunLog.log_step
method, line 87
_step · gold/harness.py:168 (by name)
_sweep_leased · housekeeping.py:167 (by name)
_profile_step · ingest/runner.py:72 (by name)
_detect_steps · ingest/runner.py:109 (by name)
_one_leased · ingest/runner.py:196 (by name)
busy · leases.py:23 (by name)
and 4 more
test_meta_store.py, test_obs_traced_store.py
StampingRunLog.log_file
method, line 91
StoreBatchRegistry.record_batch · adapters/store_registry.py:28 (by name)
_record · ingest/steps.py:381 (by name)
DatasetHandle.file · obs/recorder.py:112 (by name)
SilverBaseliner.write · silver/baseline_writer.py:75 (by name)
gold_helpers.py, test_housekeeping.py and 4 more
StampingRunLog.batch_status
method, line 95
StoreBatchRegistry.batch_status · adapters/store_registry.py:24 (by name)
enter · ingest/steps.py:318 (by name)
land · lander.py:34 (by name)
test_meta_store.py, test_slice1_local.py
StampingRunLog.log_error
method, line 98
_fail · gold/harness.py:173 (by name)
_one_leased · ingest/runner.py:175 (by name)
DatasetHandle.error · obs/recorder.py:107 (by name)
RunHandle.dataset · obs/recorder.py:154 (by name)
RunRecorder.run · obs/recorder.py:197 (by name)
_fail · silver/runner.py:102 (by name)
test_meta_store.py
StampingRunLog.finish_dataset_run
method, line 103
_fail · gold/harness.py:174 (by name)
_dimension · gold/harness.py:189 (by name)
_fact · gold/harness.py:342 (by name)
_sweep_leased · housekeeping.py:177 (by name)
_one_leased · ingest/runner.py:176 (by name)
busy · leases.py:24 (by name)
and 4 more
gold_helpers.py, test_housekeeping.py and 5 more
StampingRunLog.finish_run
method, line 109
run_gold · gold/harness.py:156 (by name)
run_housekeeping · housekeeping.py:134 (by name)
ingest · ingest/runner.py:135 (by name)
RunRecorder.run · obs/recorder.py:201 (by name)
TracedStore.finish_run · obs/traced_store.py:60 (by name)
build_silver · silver/runner.py:97 (by name)
gold_helpers.py, test_housekeeping.py and 4 more
StampingRunLog.watermark
method, line 114
push · ingest/pipelines.py:161 (value)
pull_pages · ingest/pipelines.py:302 (value)
stream_events · ingest/pipelines.py:357 (value)
DailyCopy.__init__ · obs/system.py:84 (value)
DailyCopy.copy · obs/system.py:92 (value)
DailyCopy.on_demand · obs/system.py:100 (value)
and 1 more
gold_helpers.py, test_ingest_pipelines.py and 13 more
StampingRunLog.bookmarks
method, line 117
silver_watermark · gold/harness.py:106 (by name)
feed_marks · gold/harness.py:122 (by name)
_fact · gold/harness.py:289 (by name)
push · ingest/pipelines.py:156 (by name)
_feed_bookmarks · ingest/pipelines.py:196 (by name)
pull_copy · ingest/pipelines.py:249 (by name)
and 5 more
gold_helpers.py, test_gold_harness.py and 6 more
StampingRunLog.reset_watermark
method, line 120
none found in the codetest_meta_store.py
StampingRunLog.run_report
method, line 124
success_scorecard · scripts/run_platform.py:184 (by name)
StampingRunLog.snapshot · adapters/memory_runlog.py:175 (by name)
_steps · gold/inspect.py:55 (by name)
test_ingest_pipelines.py, test_meta_store.py and 5 more
StampingRunLog.dataset_files
method, line 127
silver_watermark · gold/harness.py:107 (by name)
_sweep_leased · housekeeping.py:162 (by name)
push · ingest/pipelines.py:132 (by name)
pull_copy · ingest/pipelines.py:250 (by name)
pull_pages · ingest/pipelines.py:278 (by name)
stream_events · ingest/pipelines.py:324 (by name)
and 3 more
gold_helpers.py, test_ingest_pipelines.py and 2 more
StampingRunLog.record_corrections
method, line 131
withdraw · recovery.py:23 (by name)
World.write · world/engine.py:145 (by name)
test_gold_harness.py, test_meta_store.py
StampingRunLog.corrections
method, line 134
_inputs · gold/harness.py:243 (value)
_fact · gold/harness.py:333 (value)
_record · gold/harness.py:456 (value)
main · scripts/run_platform.py:669 (by name)
main · scripts/run_world.py:189 (by name)
_corrections · gold/harness.py:218 (by name)
gold_helpers.py, test_gold_engine.py and 7 more
StampingRunLog.backfill_plans
method, line 137
main · scripts/run_world.py:188 (by name)
_active_plans · gold/harness.py:208 (by name)
test_meta_store.py, test_world_tyre.py
StampingRunLog.acquire_lease
method, line 142
run_gold · gold/harness.py:141 (by name)
_sweep · housekeeping.py:148 (by name)
_one · ingest/runner.py:151 (by name)
_one · silver/runner.py:121 (by name)
test_housekeeping.py, test_meta_store.py and 1 more
StampingRunLog.release_lease
method, line 145
run_gold · gold/harness.py:153 (by name)
_sweep · housekeeping.py:155 (by name)
_one · ingest/runner.py:158 (by name)
_one · silver/runner.py:128 (by name)
test_meta_store.py
StampingRunLog.void_batch
method, line 148
_sweep_leased · housekeeping.py:173 (by name)
withdraw · recovery.py:20 (by name)
test_meta_store.py, test_silver_runner.py
StampingRunLog.set_dq_rules
method, line 151
load · meta/loader.py:180 (by name)test_meta_store.py, test_silver_runner.py
StampingRunLog.dq_rules
method, line 154
module level · realms/databricks/declarative/silver.py:81 (by name)
lakebase · realms/databricks/engine/tasks/probe.py:83 (by name)
SilverBaseliner.write · silver/baseline_writer.py:56 (by name)
_one_leased · silver/runner.py:148 (by name)
test_meta_loader.py, test_meta_store.py and 2 more
StampingRunLog.log_dq_results
method, line 157
none found in the codetest_meta_store.py
StampingRunLog.dq_results
method, line 160
none found in the codetest_meta_store.py, test_cli_dq.py and 1 more
StampingRunLog.save_profile
method, line 163
_profile_step · ingest/runner.py:75 (by name)test_meta_store.py
StampingRunLog.last_profile
method, line 166
_profile_step · ingest/runner.py:69 (by name)test_meta_store.py
StampingRunLog.snapshot
method, line 169
Monitor.run_finished · obs/monitor.py:99 (by name)
Monitor.check_lateness · obs/monitor.py:108 (by name)
Monitor.tables · obs/monitor.py:120 (by name)
test_meta_loader.py, test_obs_recorder.py and 10 more
StampingRunLog._add
method, line 181
StampingRunLog.snapshot · adapters/memory_runlog.py:178no test names it

adapters/memory_sources.py

The four source ports in memory: the reference behaviour every real adapter is tested against (M5). none yet

DefinitionWhat it does (its own first sentence)Called byNamed in tests
MemorySourceDb
class, line 11
Harness.build · twin/runner.py:93test_ingest_pipelines.py, test_source_ports.py and 2 more
MemorySourceDb.__init__
method, line 12
none found in the codeno test names it
MemorySourceDb.ensure
method, line 17
World.prepare · world/engine.py:82 (by name)test_ingest_pipelines.py, test_source_ports.py and 2 more
MemorySourceDb.apply
method, line 23
World.write · world/engine.py:125 (by name)silver_helpers.py, test_baseline_stage.py and 15 more
MemorySourceDb.publish_copy
method, line 42
World.write · world/engine.py:129 (by name)test_ingest_pipelines.py, test_source_ports.py
MemorySourceDb.rows
method, line 48
none found in the codeno test names it
MemorySourceDb.feed
method, line 51
read_back · scripts/run_world.py:120 (by name)
ChangeFeedMover.move · adapters/local_movers.py:32 (by name)
DmsMover.move · adapters/local_movers.py:88 (by name)
no test names it
MemorySourceDb.copy
method, line 54
none found in the codeno test names it
MemorySourceDb.copies
method, line 58
pull_copy · ingest/pipelines.py:255 (by name)test_housekeeping.py, test_ingest_pipelines.py and 8 more
MemoryFileDrop
class, line 62
Harness.build · twin/runner.py:93test_ingest_pipelines.py, test_source_ports.py and 1 more
MemoryFileDrop.__init__
method, line 63
none found in the codeno test names it
MemoryFileDrop.put
method, line 66
ChangeFeedMover.move · adapters/local_movers.py:38 (by name)
DropMover.move · adapters/local_movers.py:65 (by name)
DmsMover.move · adapters/local_movers.py:105 (by name)
World.write · world/engine.py:133 (by name)
no test names it
MemoryFileDrop.list
method, line 70
read_back · scripts/run_world.py:122 (by name)
DropMover.move · adapters/local_movers.py:58 (by name)
no test names it
MemoryFileDrop.get
method, line 74
none found in the codeno test names it
MemoryRecords
class, line 78
Harness.build · twin/runner.py:93test_ingest_pipelines.py, test_ingest_world.py and 7 more
MemoryRecords.__init__
method, line 79
none found in the codeno test names it
MemoryRecords.upsert
method, line 82
HttpRecordClient.upsert · adapters/http_api.py:116 (by name)
World.write · world/engine.py:137 (by name)
test_baseline_stage.py, test_ingest_pipelines.py and 5 more
MemoryRecords.page
method, line 85
read_back · scripts/run_world.py:129 (by name)
_handler.Handler.do_GET · adapters/http_api.py:79 (by name)
pull_pages · ingest/pipelines.py:288 (by name)
no test names it
MemoryStream
class, line 92
Harness.build · twin/runner.py:93test_ingest_pipelines.py, test_source_ports.py and 1 more
MemoryStream.__init__
method, line 93
none found in the codeno test names it
MemoryStream.append
method, line 99
none found in the codeazure_fakes.py, gold_helpers.py and 45 more
MemoryStream.shards
method, line 105
read_back · scripts/run_world.py:137 (by name)
stream_events · ingest/pipelines.py:342 (by name)
test_cloud_adapters.py, test_ingest_pipelines.py and 2 more
MemoryStream.read
method, line 108
none found in the codeno test names it

adapters/memory_system.py

The system-records port (dataplatform.obs.system) in memory: rows added by tests or the fixture runner. none yet

adapters/memory_tables.py

The release gate's ports (dataplatform.twin.gate) in memory: tables are lists of rows; a clone is a copy of the list (a shallow clone shares data until written; in memory a copy behaves the same for the gate). none yet

adapters/null_runlog.py

The run-log reader port (dataplatform.obs.runlog) with no run log connected: level 1 of ADR-024 rule 6. none yet

DefinitionWhat it does (its own first sentence)Called byNamed in tests
NullRunLog
class, line 14
none found in the codetest_obs_dashboards.py
NullRunLog.snapshot
method, line 15
Monitor.run_finished · obs/monitor.py:99 (by name)
Monitor.check_lateness · obs/monitor.py:108 (by name)
Monitor.tables · obs/monitor.py:120 (by name)
test_meta_loader.py, test_obs_recorder.py and 10 more

adapters/otel_tracer.py

The telemetry port (dataplatform.obs.trace) on the OpenTelemetry API. none yet

DefinitionWhat it does (its own first sentence)Called byNamed in tests
_api
function, line 17
OtelTracer.__init__ · adapters/otel_tracer.py:68no test names it
_attr
function, line 21
_attrs · adapters/otel_tracer.py:63no test names it
OtelSpan
class, line 25
OtelSpan.child · adapters/otel_tracer.py:42
OtelTracer.start · adapters/otel_tracer.py:77
no test names it
OtelSpan.__init__
method, line 26
none found in the codeno test names it
OtelSpan.trace_id
method, line 31
OtelSpan.child · adapters/otel_tracer.py:42 (value)
RunHandle.__init__ · obs/recorder.py:138 (value)
RunRecorder.run · obs/recorder.py:191 (value)
LocalSpan.child · obs/trace.py:82 (value)
test_obs_recorder.py, test_obs_trace.py and 1 more
OtelSpan.span_id
method, line 35
RunHandle.log_fields · obs/recorder.py:176 (value)
LocalSpan.child · obs/trace.py:82 (value)
test_obs_trace.py
OtelSpan.child
method, line 38
DatasetHandle.step · obs/recorder.py:78 (by name)
RunHandle.dataset · obs/recorder.py:146 (by name)
TracedStore.start_dataset_run · obs/traced_store.py:42 (by name)
TracedStore.log_step · obs/traced_store.py:49 (by name)
test_obs_trace.py, test_mimic_runner.py and 1 more
OtelSpan.set
method, line 47
none found in the codeno test names it
OtelSpan.end
method, line 50
_complete · gold/harness.py:279 (by name)
_fact · gold/harness.py:329 (by name)
DatasetHandle.step · obs/recorder.py:83 (by name)
DatasetHandle._finish · obs/recorder.py:122 (by name)
RunRecorder.run · obs/recorder.py:203 (by name)
TracedStore.log_step · obs/traced_store.py:51 (by name)
and 2 more
no test names it
_attrs
function, line 62
OtelSpan.child · adapters/otel_tracer.py:45
OtelSpan.set · adapters/otel_tracer.py:48
OtelTracer.start · adapters/otel_tracer.py:73
no test names it
OtelTracer
class, line 66
none found in the codetest_obs_trace.py
OtelTracer.__init__
method, line 67
none found in the codeno test names it
OtelTracer.start
method, line 71
none found in the codesilver_helpers.py, test_gold_calendar.py and 15 more

adapters/postgres_meta.py

The metadata store on PostgreSQL (ADR-010). Every read and write is one call to a function in adapters/sql/postgres_meta.sql; this file holds no table names (DPC-0033, checked by tests/dataplatform/test_meta_store.py). Metadata store

DefinitionWhat it does (its own first sentence)Called byNamed in tests
_ident
function, line 45
lakebase_fresh_store · scripts/run_platform.py:89
create_database · adapters/postgres_meta.py:62
install · adapters/postgres_meta.py:70
no test names it
_dbname
function, line 51
create_database · adapters/postgres_meta.py:59no test names it
create_database
function, line 55
Create the database named in a key=value DSN if it is missing (local and tests; a cloud binding makes it).meta_store · scripts/run_platform.py:70
corrections_store · scripts/run_world.py:97
PostgresMetaStore.connect · adapters/postgres_meta.py:88
test_meta_store.py, test_obs_runlog.py
install
function, line 65
Create or update the schemas, tables and functions (idempotent). With a service role, grant it the functions and nothing else: the role can call every procedure and read no table (DPC-0033).meta_store · scripts/run_platform.py:74
lakebase_fresh_store · scripts/run_platform.py:94
corrections_store · scripts/run_world.py:101
PostgresMetaStore.connect · adapters/postgres_meta.py:91
test_meta_store.py, test_obs_runlog.py and 5 more
PostgresMetaStore
class, line 78
connect_lakebase · adapters/lakebase.py:83test_meta_store.py, test_obs_runlog.py
PostgresMetaStore.__init__
method, line 79
none found in the codeno test names it
PostgresMetaStore.connect
method, line 83
connect_lakebase · adapters/lakebase.py:80 (value)
main · realms/aws-databricks/meta_setup.py:78 (by name)
_seed_postgres · scripts/demo_end_to_end.py:266 (by name)
_ingest_table_to_bronze · scripts/demo_end_to_end.py:372 (by name)
attach_meta · scripts/platform_binding.py:39 (by name)
meta_store · scripts/run_platform.py:71 (by name)
and 12 more
test_lakebase.py, test_meta_store.py and 21 more
PostgresMetaStore._call
method, line 95
PostgresMetaStore._one · adapters/postgres_meta.py:107
PostgresMetaStore.register_settings · adapters/postgres_meta.py:116
PostgresMetaStore.load_domain · adapters/postgres_meta.py:120
PostgresMetaStore.load_dataset · adapters/postgres_meta.py:123
PostgresMetaStore.settings_registry · adapters/postgres_meta.py:128
PostgresMetaStore.datasets · adapters/postgres_meta.py:131
and 22 more
no test names it
PostgresMetaStore._one
method, line 106
PostgresMetaStore.dataset_view · adapters/postgres_meta.py:143
PostgresMetaStore.start_dataset_run · adapters/postgres_meta.py:152
PostgresMetaStore.batch_status · adapters/postgres_meta.py:167
PostgresMetaStore.finish_run · adapters/postgres_meta.py:183
PostgresMetaStore.watermark · adapters/postgres_meta.py:186
PostgresMetaStore.run_report · adapters/postgres_meta.py:196
and 1 more
no test names it
PostgresMetaStore._json
method, line 111
PostgresMetaStore.register_settings · adapters/postgres_meta.py:117
PostgresMetaStore.load_domain · adapters/postgres_meta.py:120
PostgresMetaStore.load_dataset · adapters/postgres_meta.py:123
PostgresMetaStore.start_dataset_run · adapters/postgres_meta.py:153
PostgresMetaStore.log_step · adapters/postgres_meta.py:158
PostgresMetaStore.finish_dataset_run · adapters/postgres_meta.py:179
and 4 more
no test names it
PostgresMetaStore.register_settings
method, line 115
StampingRunLog.register_settings · adapters/memory_runlog.py:48 (by name)
load · meta/loader.py:174 (by name)
test_housekeeping.py, test_ingest_pipelines.py and 7 more
PostgresMetaStore.load_domain
method, line 119
StampingRunLog.load_domain · adapters/memory_runlog.py:51 (by name)
load · meta/loader.py:175 (by name)
gold_helpers.py, test_housekeeping.py and 10 more
PostgresMetaStore.load_dataset
method, line 122
StampingRunLog.load_dataset · adapters/memory_runlog.py:54 (by name)
load · meta/loader.py:179 (by name)
load_gold · meta/loader.py:235 (by name)
gold_helpers.py, test_housekeeping.py and 10 more
PostgresMetaStore.settings_registry
method, line 125
StampingRunLog.settings_registry · adapters/memory_runlog.py:57 (by name)test_meta_store.py
PostgresMetaStore.datasets
method, line 130
Monitor._lateness · obs/monitor.py:88 (value)
main · realms/aws-databricks/meta_setup.py:85 (by name)
module level · realms/databricks/declarative/silver.py:73 (by name)
lakebase · realms/databricks/engine/tasks/probe.py:80 (by name)
hop_delays · scripts/run_platform.py:160 (by name)
StampingRunLog.datasets · adapters/memory_runlog.py:60 (by name)
and 10 more
test_bundle.py, test_semantic_governance.py and 29 more
PostgresMetaStore.settings
method, line 133
_read · ingest/pipelines.py:67 (value)
push · ingest/pipelines.py:147 (value)
enter · ingest/steps.py:328 (value)
main · realms/aws-databricks/meta_setup.py:88 (by name)
StampingRunLog.settings · adapters/memory_runlog.py:63 (by name)
test_bundle.py, gold_helpers.py and 20 more
PostgresMetaStore.columns
method, line 136
_ingest_table_to_bronze · scripts/demo_end_to_end.py:389 (value)
CustomerRunLog.__init__ · adapters/customer_runlog.py:34 (value)
_widget · adapters/lakeview.py:103 (value)
UcIcebergStore._create · adapters/table_store.py:403 (value)
validate · obs/dashboards.py:105 (value)
contract_check · stages/clean.py:44 (value)
and 8 more
test_lakeformation.py, test_semantic_governance.py and 67 more
PostgresMetaStore.business_keys
method, line 139
StampingRunLog.business_keys · adapters/memory_runlog.py:69 (by name)
plans · gold/harness.py:96 (by name)
_inputs · gold/harness.py:245 (by name)
gold_helpers.py, silver_helpers.py and 26 more
PostgresMetaStore.dataset_view
method, line 142
module level · realms/databricks/declarative/silver.py:77 (by name)
StampingRunLog.dataset_view · adapters/memory_runlog.py:72 (by name)
StoreBatchRegistry.dataset_config · adapters/store_registry.py:21 (by name)
plans · gold/harness.py:94 (by name)
run_gold · gold/harness.py:138 (by name)
_sweep · housekeeping.py:142 (by name)
and 7 more
test_ingest_pipelines.py, test_meta_loader.py and 4 more
PostgresMetaStore.start_run
method, line 147
StampingRunLog.start_run · adapters/memory_runlog.py:77 (by name)
run_gold · gold/harness.py:132 (by name)
run_housekeeping · housekeeping.py:127 (by name)
ingest · ingest/runner.py:126 (by name)
RunRecorder.run · obs/recorder.py:191 (by name)
TracedStore.start_run · obs/traced_store.py:33 (by name)
and 1 more
gold_helpers.py, test_housekeeping.py and 7 more
PostgresMetaStore.start_dataset_run
method, line 150
StampingRunLog.start_dataset_run · adapters/memory_runlog.py:83 (by name)
run_gold · gold/harness.py:139 (by name)
_sweep · housekeeping.py:146 (by name)
_one · ingest/runner.py:149 (by name)
RunHandle.dataset · obs/recorder.py:148 (by name)
TracedStore.start_dataset_run · obs/traced_store.py:39 (by name)
and 2 more
gold_helpers.py, test_housekeeping.py and 7 more
PostgresMetaStore.log_step
method, line 156
StampingRunLog.log_step · adapters/memory_runlog.py:88 (by name)
_step · gold/harness.py:168 (by name)
_sweep_leased · housekeeping.py:167 (by name)
_profile_step · ingest/runner.py:72 (by name)
_detect_steps · ingest/runner.py:109 (by name)
_one_leased · ingest/runner.py:196 (by name)
and 5 more
test_meta_store.py, test_obs_traced_store.py
PostgresMetaStore.log_file
method, line 161
StampingRunLog.log_file · adapters/memory_runlog.py:92 (by name)
StoreBatchRegistry.record_batch · adapters/store_registry.py:28 (by name)
_record · ingest/steps.py:381 (by name)
DatasetHandle.file · obs/recorder.py:112 (by name)
SilverBaseliner.write · silver/baseline_writer.py:75 (by name)
gold_helpers.py, test_housekeeping.py and 4 more
PostgresMetaStore.batch_status
method, line 166
StampingRunLog.batch_status · adapters/memory_runlog.py:96 (by name)
StoreBatchRegistry.batch_status · adapters/store_registry.py:24 (by name)
enter · ingest/steps.py:318 (by name)
land · lander.py:34 (by name)
test_meta_store.py, test_slice1_local.py
PostgresMetaStore.log_error
method, line 170
StampingRunLog.log_error · adapters/memory_runlog.py:100 (by name)
_fail · gold/harness.py:173 (by name)
_one_leased · ingest/runner.py:175 (by name)
DatasetHandle.error · obs/recorder.py:107 (by name)
RunHandle.dataset · obs/recorder.py:154 (by name)
RunRecorder.run · obs/recorder.py:197 (by name)
and 1 more
test_meta_store.py
PostgresMetaStore.finish_dataset_run
method, line 175
StampingRunLog.finish_dataset_run · adapters/memory_runlog.py:106 (by name)
_fail · gold/harness.py:174 (by name)
_dimension · gold/harness.py:189 (by name)
_fact · gold/harness.py:342 (by name)
_sweep_leased · housekeeping.py:177 (by name)
_one_leased · ingest/runner.py:176 (by name)
and 5 more
gold_helpers.py, test_housekeeping.py and 5 more
PostgresMetaStore.finish_run
method, line 182
StampingRunLog.finish_run · adapters/memory_runlog.py:110 (by name)
run_gold · gold/harness.py:156 (by name)
run_housekeeping · housekeeping.py:134 (by name)
ingest · ingest/runner.py:135 (by name)
RunRecorder.run · obs/recorder.py:201 (by name)
TracedStore.finish_run · obs/traced_store.py:60 (by name)
and 1 more
gold_helpers.py, test_housekeeping.py and 4 more
PostgresMetaStore.watermark
method, line 185
push · ingest/pipelines.py:161 (value)
pull_pages · ingest/pipelines.py:302 (value)
stream_events · ingest/pipelines.py:357 (value)
DailyCopy.__init__ · obs/system.py:84 (value)
DailyCopy.copy · obs/system.py:92 (value)
DailyCopy.on_demand · obs/system.py:100 (value)
and 2 more
gold_helpers.py, test_ingest_pipelines.py and 13 more
PostgresMetaStore.bookmarks
method, line 189
StampingRunLog.bookmarks · adapters/memory_runlog.py:118 (by name)
silver_watermark · gold/harness.py:106 (by name)
feed_marks · gold/harness.py:122 (by name)
_fact · gold/harness.py:289 (by name)
push · ingest/pipelines.py:156 (by name)
_feed_bookmarks · ingest/pipelines.py:196 (by name)
and 6 more
gold_helpers.py, test_gold_harness.py and 6 more
PostgresMetaStore.reset_watermark
method, line 192
StampingRunLog.reset_watermark · adapters/memory_runlog.py:122 (by name)test_meta_store.py
PostgresMetaStore.run_report
method, line 195
success_scorecard · scripts/run_platform.py:184 (by name)
StampingRunLog.run_report · adapters/memory_runlog.py:125 (by name)
StampingRunLog.snapshot · adapters/memory_runlog.py:175 (by name)
_steps · gold/inspect.py:55 (by name)
test_ingest_pipelines.py, test_meta_store.py and 5 more
PostgresMetaStore.dataset_files
method, line 199
StampingRunLog.dataset_files · adapters/memory_runlog.py:128 (by name)
silver_watermark · gold/harness.py:107 (by name)
_sweep_leased · housekeeping.py:162 (by name)
push · ingest/pipelines.py:132 (by name)
pull_copy · ingest/pipelines.py:250 (by name)
pull_pages · ingest/pipelines.py:278 (by name)
and 4 more
gold_helpers.py, test_ingest_pipelines.py and 2 more
PostgresMetaStore.record_corrections
method, line 206
StampingRunLog.record_corrections · adapters/memory_runlog.py:132 (by name)
withdraw · recovery.py:23 (by name)
World.write · world/engine.py:145 (by name)
test_gold_harness.py, test_meta_store.py
PostgresMetaStore.corrections
method, line 212
_inputs · gold/harness.py:243 (value)
_fact · gold/harness.py:333 (value)
_record · gold/harness.py:456 (value)
main · scripts/run_platform.py:669 (by name)
main · scripts/run_world.py:189 (by name)
StampingRunLog.corrections · adapters/memory_runlog.py:135 (by name)
and 1 more
gold_helpers.py, test_gold_engine.py and 7 more
PostgresMetaStore.backfill_plans
method, line 217
main · scripts/run_world.py:188 (by name)
StampingRunLog.backfill_plans · adapters/memory_runlog.py:138 (by name)
_active_plans · gold/harness.py:208 (by name)
test_meta_store.py, test_world_tyre.py
PostgresMetaStore.acquire_lease
method, line 222
StampingRunLog.acquire_lease · adapters/memory_runlog.py:143 (by name)
run_gold · gold/harness.py:141 (by name)
_sweep · housekeeping.py:148 (by name)
_one · ingest/runner.py:151 (by name)
_one · silver/runner.py:121 (by name)
test_housekeeping.py, test_meta_store.py and 1 more
PostgresMetaStore.release_lease
method, line 226
StampingRunLog.release_lease · adapters/memory_runlog.py:146 (by name)
run_gold · gold/harness.py:153 (by name)
_sweep · housekeeping.py:155 (by name)
_one · ingest/runner.py:158 (by name)
_one · silver/runner.py:128 (by name)
test_meta_store.py
PostgresMetaStore.void_batch
method, line 229
StampingRunLog.void_batch · adapters/memory_runlog.py:149 (by name)
_sweep_leased · housekeeping.py:173 (by name)
withdraw · recovery.py:20 (by name)
test_meta_store.py, test_silver_runner.py
PostgresMetaStore.save_profile
method, line 232
StampingRunLog.save_profile · adapters/memory_runlog.py:164 (by name)
_profile_step · ingest/runner.py:75 (by name)
test_meta_store.py
PostgresMetaStore.last_profile
method, line 235
StampingRunLog.last_profile · adapters/memory_runlog.py:167 (by name)
_profile_step · ingest/runner.py:69 (by name)
test_meta_store.py
PostgresMetaStore.set_dq_rules
method, line 240
StampingRunLog.set_dq_rules · adapters/memory_runlog.py:152 (by name)
load · meta/loader.py:180 (by name)
test_meta_store.py, test_silver_runner.py
PostgresMetaStore.dq_rules
method, line 245
module level · realms/databricks/declarative/silver.py:81 (by name)
lakebase · realms/databricks/engine/tasks/probe.py:83 (by name)
StampingRunLog.dq_rules · adapters/memory_runlog.py:155 (by name)
SilverBaseliner.write · silver/baseline_writer.py:56 (by name)
_one_leased · silver/runner.py:148 (by name)
test_meta_loader.py, test_meta_store.py and 2 more
PostgresMetaStore.log_dq_results
method, line 249
StampingRunLog.log_dq_results · adapters/memory_runlog.py:158 (by name)test_meta_store.py
PostgresMetaStore.dq_results
method, line 254
StampingRunLog.dq_results · adapters/memory_runlog.py:161 (by name)test_meta_store.py, test_cli_dq.py and 1 more

adapters/postgres_runlog.py

The run-log reader port (dataplatform.obs.runlog) on PostgreSQL: one call to runtime.ops_run_log (sql/postgres_obs.sql), installed beside the metadata store's own schema (postgres_meta.sql). No table names here (DPC-0033). Timestamps are th none yet

DefinitionWhat it does (its own first sentence)Called byNamed in tests
install
function, line 24
Install (or update) the read procedure. Needs the store's schema first (postgres_meta.install).none found in the codetest_meta_store.py, test_obs_runlog.py and 5 more
_ts
function, line 30
parse · adapters/postgres_runlog.py:42no test names it
_at
function, line 34
parse · adapters/postgres_runlog.py:42no test names it
parse
function, line 38
The procedure's document as port rows (pure; tested on its own).PostgresRunLogReader.snapshot · adapters/postgres_runlog.py:61test_ingest_formats.py, test_ingest_steps.py and 19 more
PostgresRunLogReader
class, line 55
none found in the codetest_obs_runlog.py
PostgresRunLogReader.__init__
method, line 56
none found in the codeno test names it
PostgresRunLogReader.snapshot
method, line 59
Monitor.run_finished · obs/monitor.py:99 (by name)
Monitor.check_lateness · obs/monitor.py:108 (by name)
Monitor.tables · obs/monitor.py:120 (by name)
test_meta_loader.py, test_obs_recorder.py and 10 more

adapters/postgres_source.py

SourceDatabase on PostgreSQL: the mock ERP and MES databases, one schema per system. Sources and movers

adapters/s3.py

Object storage on Amazon S3 (cloud layer, dbs/cloud/aws.yaml): landing_store, the partner file drop and object_store. Needs boto3 (the `s3` extra). The client is passed in, so tests use a local stand-in (moto) and the platform passes one bu Sources and movers

DefinitionWhat it does (its own first sentence)Called byNamed in tests
bucket_name
function, line 26
The one naming rule shared with Terraform (modules/storage): <prefix>-<env>-<layer>.aws_sinks · scripts/run_world.py:58
S3ObjectStore.location · adapters/s3.py:134
S3ObjectStore.versioned · adapters/s3.py:137
test_cloud_adapters.py
_keys
function, line 33
S3Landing.put · adapters/s3.py:58
S3Landing.deliveries · adapters/s3.py:69
S3FileDrop.list · adapters/s3.py:115
no test names it
S3Landing
class, line 40
landing_store on an S3 bucket (the <env>-landing bucket), the same layout as FileLanding.none found in the codetest_cloud_adapters.py, test_ingest_ports.py
S3Landing.__init__
method, line 43
none found in the codeno test names it
S3Landing._dir
method, line 46
S3Landing.put · adapters/s3.py:57
S3Landing.deliveries · adapters/s3.py:69
S3Landing.read_bytes · adapters/s3.py:81
S3Landing.manifest · adapters/s3.py:84
LandingArchive.archive · adapters/files.py:84 (by name)
LandingArchive.delete · adapters/files.py:90 (by name)
and 1 more
no test names it
S3Landing.write
method, line 49
none found in the codegold_helpers.py, test_cloud_adapters.py and 31 more
S3Landing.put
method, line 54
Written once: a delivery whose manifest exists is refused (DUP-02).S3Landing.write · adapters/s3.py:50
ChangeFeedMover.move · adapters/local_movers.py:38 (by name)
DropMover.move · adapters/local_movers.py:65 (by name)
DmsMover.move · adapters/local_movers.py:105 (by name)
World.write · world/engine.py:133 (by name)
no test names it
S3Landing.deliveries
method, line 67
step_prepare · realms/databricks/tasks/main.py:81 (by name)
ChangeFeedMover.checkpoint · adapters/local_movers.py:28 (by name)
DropMover.move · adapters/local_movers.py:56 (by name)
push · ingest/pipelines.py:135 (by name)
land · lander.py:31 (by name)
gold_helpers.py, test_cloud_adapters.py and 5 more
S3Landing.read
method, line 77
none found in the codeno test names it
S3Landing.read_bytes
method, line 80
S3Landing.read · adapters/s3.py:78
LocalFileDrop.list · adapters/file_drop.py:38 (by name)
LocalFileDrop.get · adapters/file_drop.py:41 (by name)
FileLanding.deliveries · adapters/files.py:56 (by name)
FileLanding.read_bytes · adapters/files.py:64 (by name)
push · ingest/pipelines.py:144 (by name)
test_ingest_formats.py, test_ingest_ports.py and 3 more
S3Landing.manifest
method, line 83
ChangeFeedMover.checkpoint · adapters/local_movers.py:28 (by name)test_bundle.py, gold_helpers.py and 34 more
S3Landing._get
method, line 88
S3Landing.deliveries · adapters/s3.py:72
S3Landing.read_bytes · adapters/s3.py:81
S3Landing.manifest · adapters/s3.py:85
no test names it
S3FileDrop
class, line 93
FileDrop (source_ports) on S3: <prefix><system>/<dataset>/<name>, listed with ListObjectsV2. S3 stamps its own upload time, so the arrival time the world declares travels as object metadata x-amz-meta-arrived-at (with the checksum as x-amz-aws_sinks · scripts/run_world.py:58test_source_ports.py
S3FileDrop.__init__
method, line 98
none found in the codeno test names it
S3FileDrop._dir
method, line 101
S3FileDrop.put · adapters/s3.py:109
S3FileDrop.list · adapters/s3.py:115
S3FileDrop.get · adapters/s3.py:123
LandingArchive.archive · adapters/files.py:84 (by name)
LandingArchive.delete · adapters/files.py:90 (by name)
LandingArchive.listing · adapters/files.py:97 (by name)
no test names it
S3FileDrop.put
method, line 104
ChangeFeedMover.move · adapters/local_movers.py:38 (by name)
DropMover.move · adapters/local_movers.py:65 (by name)
DmsMover.move · adapters/local_movers.py:105 (by name)
World.write · world/engine.py:133 (by name)
no test names it
S3FileDrop.list
method, line 113
read_back · scripts/run_world.py:122 (by name)
DropMover.move · adapters/local_movers.py:58 (by name)
no test names it
S3FileDrop.get
method, line 122
none found in the codeno test names it
S3ObjectStore
class, line 127
object_store: one bucket per environment and layer, named by bucket_name().none found in the codetest_cloud_adapters.py
S3ObjectStore.__init__
method, line 130
none found in the codeno test names it
S3ObjectStore.location
method, line 133
none found in the codetest_azure_adapters.py, test_cloud_adapters.py and 2 more
S3ObjectStore.versioned
method, line 136
none found in the codetest_azure_adapters.py, test_cloud_adapters.py and 2 more
_iso
function, line 141
S3FileDrop.put · adapters/s3.py:107no test names it

adapters/secrets.py

secret_store adapters (cloud_ports.SecretStore, DPC-0029). The value is returned to the caller only; nothing here logs it, and an error names the reference, never the value. none yet

DefinitionWhat it does (its own first sentence)Called byNamed in tests
EnvSecrets
class, line 21
none found in the codetest_secrets.py
EnvSecrets.__init__
method, line 22
none found in the codeno test names it
EnvSecrets.get
method, line 25
none found in the codeno test names it
AwsSecretsManager
class, line 35
none found in the codetest_secrets.py
AwsSecretsManager.__init__
method, line 36
none found in the codeno test names it
AwsSecretsManager.get
method, line 40
none found in the codeno test names it
_is_not_found
function, line 51
Key Vault's "no such secret" error. The SDK is imported only here and only when an error has happened, and a missing SDK falls back to the class name, so this module never needs the `azure` extra at import.AzureKeyVault.get · adapters/secrets.py:84no test names it
key_vault_name
function, line 61
The Key Vault secret name for a port name. Key Vault allows letters, digits and hyphens only (1 to 127), so the "/" separator becomes "--". That is reversible only if no name already holds "--" (the port refuses it) and no hyphen sits next test_secret_references_map_to_key_vault_names · realms/azure-databricks/tests/test_realm.py:119
AzureKeyVault.get · adapters/secrets.py:82
no test names it
AzureKeyVault
class, line 73
none found in the codetest_secrets.py, test_adf_generator.py
AzureKeyVault.__init__
method, line 74
none found in the codeno test names it
AzureKeyVault.get
method, line 78
none found in the codeno test names it

adapters/spark_delta.py

The slice 1 ports on Spark + Delta tables: Databricks (serverless, Spark Connect) or open-source Spark locally. none yet

DefinitionWhat it does (its own first sentence)Called byNamed in tests
ident
function, line 23
Validate a (catalog.)(schema.)table name before it goes into SQL (the repo's no-SQL-by-string rule).create_meta · adapters/spark_delta.py:49
DeltaWriter.append · adapters/spark_delta.py:106
SparkBronzeWriter._name · adapters/spark_delta.py:146
test_semantic_governance.py, test_value_objects.py
now
function, line 43
SparkMeta.record_batch · adapters/spark_delta.py:93
SparkRunLog.write · adapters/spark_delta.py:118
SparkChecks.write · adapters/spark_delta.py:129
no test names it
create_meta
function, line 47
none found in the codeno test names it
_append
function, line 62
SparkMeta.record_batch · adapters/spark_delta.py:91
DeltaWriter.append · adapters/spark_delta.py:109
SparkRunLog.write · adapters/spark_delta.py:117
SparkChecks.write · adapters/spark_delta.py:127
no test names it
SparkMeta
class, line 67
metadata_service on Delta tables. Reads all batch statuses once per run (no per-file queries).none found in the codeno test names it
SparkMeta.__init__
method, line 70
none found in the codeno test names it
SparkMeta.dataset_config
method, line 75
step_land · realms/databricks/tasks/main.py:112 (by name)
step_finish · realms/databricks/tasks/main.py:123 (by name)
test_exceptions.py, test_runtime_sql_views.py
SparkMeta.batch_status
method, line 86
StampingRunLog.batch_status · adapters/memory_runlog.py:96 (by name)
StoreBatchRegistry.batch_status · adapters/store_registry.py:24 (by name)
enter · ingest/steps.py:318 (by name)
land · lander.py:34 (by name)
test_meta_store.py, test_slice1_local.py
SparkMeta.record_batch
method, line 89
land · lander.py:38 (by name)no test names it
DeltaWriter
class, line 96
table_writer for bronze: typed append to a Delta table.none found in the codeno test names it
DeltaWriter.__init__
method, line 102
none found in the codeno test names it
DeltaWriter.append
method, line 105
none found in the codeazure_fakes.py, gold_helpers.py and 45 more
SparkRunLog
class, line 112
none found in the codeno test names it
SparkRunLog.__init__
method, line 113
none found in the codeno test names it
SparkRunLog.write
method, line 116
none found in the codegold_helpers.py, test_cloud_adapters.py and 31 more
SparkChecks
class, line 122
none found in the codeno test names it
SparkChecks.__init__
method, line 123
none found in the codeno test names it
SparkChecks.write
method, line 126
none found in the codegold_helpers.py, test_cloud_adapters.py and 31 more
SparkBronzeWriter
class, line 133
The bronze writer (dataplatform.ingest.ports.BronzeWriter) on Delta tables through Spark: Databricks, or open-source Spark with delta-spark locally. One write replaces the rows of its batch key (replaceWhere), so a batch written twice leavenone found in the codetest_ingest_ports.py
SparkBronzeWriter.__init__
method, line 142
none found in the codeno test names it
SparkBronzeWriter._name
method, line 145
SparkBronzeWriter.write · adapters/spark_delta.py:153
SparkBronzeWriter.write_many · adapters/spark_delta.py:175
SparkBronzeWriter._rows · adapters/spark_delta.py:181
no test names it
SparkBronzeWriter.write
method, line 148
none found in the codegold_helpers.py, test_cloud_adapters.py and 31 more
SparkBronzeWriter.write_many
method, line 164
New batches in one append (one commit); the lander logs them only after it returns.flush · ingest/steps.py:365 (by name)no test names it
SparkBronzeWriter._rows
method, line 178
SparkBronzeWriter.fingerprint · adapters/spark_delta.py:193
SparkBronzeWriter.keys · adapters/spark_delta.py:200
no test names it
SparkBronzeWriter.fingerprint
method, line 189
_one · ingest/runner.py:149 (value)
FixtureRunner.run_day · obs/fixture.py:117 (value)
FixtureRunner._rerun · obs/fixture.py:134 (value)
run_days · ingest/daybyday.py:133 (by name)
gold_helpers.py, silver_helpers.py and 14 more
SparkBronzeWriter.keys
method, line 199
none found in the codeno test names it

adapters/spark_gold.py

Gold on Spark and Delta tables: the harness's engine operations (PySpark) and the gold read and write edges. The harness

DefinitionWhat it does (its own first sentence)Called byNamed in tests
_safe
function, line 48
DeltaRsGoldTables.silver · adapters/duckdb_gold.py:83
DeltaRsGoldTables.replace_windows · adapters/duckdb_gold.py:127
DuckGoldReader.rows · adapters/duckdb_gold.py:156
SparkGoldEngine.build · adapters/spark_gold.py:170
DeltaGoldTables.path · adapters/spark_gold.py:313
DeltaGoldTables.silver · adapters/spark_gold.py:321
and 9 more
no test names it
as_text
function, line 54
A timestamp as text: date_format patterns at even places, literal text at odd places.DeltaRsSilverTables.count_by_hour · adapters/duckdb_silver.py:135
key_text · adapters/spark_gold.py:63
SparkGoldEngine.window_text · adapters/spark_gold.py:95
SparkGoldEngine.first_seen · adapters/spark_gold.py:295
no test names it
key_text
function, line 59
keys.key_text in the engine: the same text the Python recipe hashes.surrogate · adapters/duckdb_gold.py:35
surrogate · adapters/spark_gold.py:69
test_gold_plan.py
surrogate
function, line 67
keys.surrogate_key in Spark: the same text, the same SHA-256, the first 15 hex digits.module level · adapters/spark_gold.py:85 (value)test_semantic_governance.py, test_gold_plan.py
_free
function, line 73
Release a cached or checkpointed frame where the engine allows it (serverless refuses UNPERSIST).SparkGoldEngine.release · adapters/spark_gold.py:250
DeltaGoldTables.write_dimension · adapters/spark_gold.py:356
StoreGoldTables.write_dimension · adapters/spark_gold.py:449
no test names it
SparkGoldEngine
class, line 79
The harness's frame operations on PySpark. Pure: no reads or writes (ADR-013)._table_port · scripts/platform_binding.py:101
_legacy · scripts/platform_binding.py:116
test_gold_engine.py, test_gold_harness.py and 2 more
SparkGoldEngine.__init__
method, line 87
none found in the codeno test names it
SparkGoldEngine.window_text
method, line 92
SparkGoldEngine.finish · adapters/spark_gold.py:211
SparkGoldEngine.only_windows · adapters/spark_gold.py:218
SparkGoldEngine.window_numbers · adapters/spark_gold.py:232
SparkGoldEngine.windows_with_unknown · adapters/spark_gold.py:285
SparkGoldEngine.rows_incomplete · adapters/spark_gold.py:290
no test names it
SparkGoldEngine.dimension
method, line 98
_where · realms/databricks/semantic_governance/render.py:131 (value)
_tile_query · realms/databricks/semantic_governance/render.py:169 (value)
_widget · realms/databricks/semantic_governance/render.py:191 (value)
scorecard · obs/success.py:157 (value)
_where · semantic/answers.py:21 (value)
_dimension · gold/harness.py:180 (by name)
and 1 more
test_semantic_governance.py, gold_helpers.py and 10 more
SparkGoldEngine._reserved
method, line 114
SparkGoldEngine.dimension · adapters/spark_gold.py:112no test names it
SparkGoldEngine.overlay
method, line 145
_inputs · gold/harness.py:245 (by name)test_gold_engine.py
SparkGoldEngine.query_text
method, line 159
_trace · gold/harness.py:360 (by name)test_lineage_end_to_end.py, test_adf_generator.py and 1 more
SparkGoldEngine.build
method, line 166
cmd_code · scripts/explain.py:101 (by name)
_fact · gold/harness.py:291 (by name)
reproduce · gold/inspect.py:78 (by name)
Monitor.tables · obs/monitor.py:122 (by name)
test_semantic_governance.py, conftest.py and 17 more
SparkGoldEngine.sql
method, line 175
One SQL query with named parameters bound by the engine (never pasted into the text).SparkGoldEngine.build · adapters/spark_gold.py:171
CustomerRunLog.__init__ · adapters/customer_runlog.py:34 (value)
DatabricksPipelineEvents.__init__ · adapters/databricks_pipeline_events.py:54 (value)
module level · realms/databricks/engine/tasks/probe.py:40 (by name)
create · realms/databricks/engine/tasks/probe.py:49 (by name)
module level · realms/databricks/engine/tasks/run_days.py:44 (by name)
and 42 more
no test names it
SparkGoldEngine._function
method, line 179
SparkGoldEngine.build · adapters/spark_gold.py:172no test names it
SparkGoldEngine.add_keys
method, line 191
_key_lookup · gold/harness.py:377 (by name)
_which_join · gold/harness.py:392 (by name)
reproduce · gold/inspect.py:80 (by name)
test_gold_engine.py
SparkGoldEngine.finish
method, line 208
none found in the codetest_gold_engine.py, test_meta_store.py and 2 more
SparkGoldEngine.only_windows
method, line 217
SparkGoldEngine.window_numbers · adapters/spark_gold.py:232
_fact · gold/harness.py:307 (by name)
_checks · gold/harness.py:443 (by name)
reproduce · gold/inspect.py:80 (by name)
no test names it
SparkGoldEngine.window_numbers
method, line 224
_fact · gold/harness.py:300 (by name)
reproduce · gold/inspect.py:81 (by name)
test_gold_engine.py
SparkGoldEngine.row_hash
method, line 240
A 64-bit hash of a row's values (window digests compare one engine's runs with each other only).SparkGoldEngine.window_numbers · adapters/spark_gold.py:230no test names it
SparkGoldEngine.hold
method, line 245
_fact · gold/harness.py:296 (by name)
_key_lookup · gold/harness.py:377 (by name)
no test names it
SparkGoldEngine.release
method, line 248
none found in the codetest_meta_store.py, test_silver_ports.py and 2 more
SparkGoldEngine.count
method, line 252
none found in the codetest_contract_rules.py, test_dac_coverage.py and 29 more
SparkGoldEngine.duplicates
method, line 255
_dimension · gold/harness.py:181 (by name)
_checks · gold/harness.py:424 (by name)
test_contract_rules.py, test_gold_engine.py and 6 more
SparkGoldEngine.unknown_rows
method, line 260
_checks · gold/harness.py:429 (by name)test_gold_engine.py
SparkGoldEngine.key_tally
method, line 267
_ledger · gold/harness.py:406 (by name)no test names it
SparkGoldEngine.windows_with_unknown
method, line 281
_range · gold/harness.py:372 (by name)no test names it
SparkGoldEngine.rows_incomplete
method, line 287
_fact · gold/harness.py:301 (by name)no test names it
SparkGoldEngine.first_seen
method, line 293
_expected_feeds · gold/harness.py:264 (by name)no test names it
SparkGoldEngine.totals
method, line 299
_checks · gold/harness.py:443 (by name)test_ingest_world.py, test_reconcile_engine.py and 3 more
DeltaGoldTables
class, line 306
The read and write edges of gold on Delta folders (silver read as W1c wrote it)._legacy · scripts/platform_binding.py:116test_gold_harness.py, test_gold_ports.py and 2 more
DeltaGoldTables.__init__
method, line 309
none found in the codeno test names it
DeltaGoldTables.path
method, line 312
DeltaGoldTables.gold · adapters/spark_gold.py:331
DeltaGoldTables.write_dimension · adapters/spark_gold.py:354
DeltaGoldTables.replace_windows · adapters/spark_gold.py:365
no test names it
DeltaGoldTables._read
method, line 315
DeltaGoldTables.silver · adapters/spark_gold.py:321
DeltaGoldTables.gold · adapters/spark_gold.py:331
no test names it
DeltaGoldTables.silver
method, line 319
_dimension · gold/harness.py:179 (by name)
_inputs · gold/harness.py:233 (by name)
reproduce · gold/inspect.py:76 (by name)
test_bundle.py, test_semantic_governance.py and 62 more
DeltaGoldTables.gold
method, line 330
build · realms/databricks/semantic_governance/generate.py:110 (value)
main · scripts/run_platform.py:459 (value)
_expected_feeds · gold/harness.py:261 (by name)
_fact · gold/harness.py:292 (by name)
reproduce · gold/inspect.py:79 (by name)
no test names it
DeltaGoldTables._silver_path
method, line 333
DeltaGoldTables.silver_version · adapters/spark_gold.py:339
DeltaGoldTables.silver_at · adapters/spark_gold.py:349
DeltaRsGoldTables.silver_version · adapters/duckdb_gold.py:93 (by name)
DeltaRsGoldTables.silver_at · adapters/duckdb_gold.py:102 (by name)
no test names it
DeltaGoldTables.silver_version
method, line 338
_inputs · gold/harness.py:232 (by name)no test names it
DeltaGoldTables.silver_at
method, line 348
reproduce · gold/inspect.py:76 (by name)no test names it
DeltaGoldTables.write_dimension
method, line 352
_dimension · gold/harness.py:186 (by name)test_gold_ports.py
DeltaGoldTables.replace_windows
method, line 360
_fact · gold/harness.py:318 (by name)test_gold_ports.py
SparkGoldReader
class, line 379
GoldReader on the gold folders: each table visible as a view by its own name._legacy · scripts/platform_binding.py:117test_gold_harness.py, test_gold_ports.py and 1 more
SparkGoldReader.__init__
method, line 382
none found in the codeno test names it
SparkGoldReader.rows
method, line 387
none found in the codeno test names it
StoreGoldTables
class, line 407
The read and write edges of gold through the table port (ADR-027): silver read from the silver layer, gold tables <schema>__<name> in the gold layer, on any format the store speaks._table_port · scripts/platform_binding.py:101no test names it
StoreGoldTables.__init__
method, line 411
none found in the codeno test names it
StoreGoldTables.table
method, line 415
StoreGoldTables.gold · adapters/spark_gold.py:442
StoreGoldTables.write_dimension · adapters/spark_gold.py:447
StoreGoldTables.replace_windows · adapters/spark_gold.py:458
test_lakeformation.py, test_semantic_governance.py and 84 more
StoreGoldTables.silver
method, line 419
_dimension · gold/harness.py:179 (by name)
_inputs · gold/harness.py:233 (by name)
reproduce · gold/inspect.py:76 (by name)
test_bundle.py, test_semantic_governance.py and 62 more
StoreGoldTables.silver_version
method, line 429
_inputs · gold/harness.py:232 (by name)no test names it
StoreGoldTables.silver_at
method, line 433
reproduce · gold/inspect.py:76 (by name)no test names it
StoreGoldTables.gold
method, line 441
build · realms/databricks/semantic_governance/generate.py:110 (value)
main · scripts/run_platform.py:459 (value)
_expected_feeds · gold/harness.py:261 (by name)
_fact · gold/harness.py:292 (by name)
reproduce · gold/inspect.py:79 (by name)
no test names it
StoreGoldTables.write_dimension
method, line 445
_dimension · gold/harness.py:186 (by name)test_gold_ports.py
StoreGoldTables.replace_windows
method, line 453
_fact · gold/harness.py:318 (by name)test_gold_ports.py
StoreGoldReader
class, line 471
GoldReader through the table port: each gold table visible as a view by its own name._table_port · scripts/platform_binding.py:102no test names it
StoreGoldReader.__init__
method, line 474
none found in the codeno test names it
StoreGoldReader.rows
method, line 479
none found in the codeno test names it

adapters/spark_ops_tables.py

Publish the ops tables (obs/ops_export.py file) as catalog tables, then the ops metric views over them (B13). none yet

DefinitionWhat it does (its own first sentence)Called byNamed in tests
publish
function, line 19
module level · realms/databricks/engine/tasks/run_days.py:58test_gold_harness.py, test_source_ports.py

adapters/spark_silver.py

Silver on Spark and Delta tables: the stage runner (dataplatform.stages on PySpark) and the read and write edges. Writer

DefinitionWhat it does (its own first sentence)Called byNamed in tests
local_session
function, line 36
A small local Spark with Delta. Uses the Spark inside the installed `pyspark` package (a SPARK_HOME pointing at another Spark install would mismatch it) and this Python for the workers. With `iceberg_warehouse`, an Iceberg folder catalog na_duckdb · scripts/platform_binding.py:67
_table_port.local_session · scripts/platform_binding.py:90
_table_port · scripts/platform_binding.py:92
_legacy · scripts/platform_binding.py:109
_duckdb · scripts/platform_binding.py:68 (value)
_table_port · scripts/platform_binding.py:97 (value)
and 1 more
conftest.py, test_obs_steps.py
_quoted
function, line 85
A Spark SQL identifier in backticks, any backtick inside doubled (Spark's escape), so a column name that came from a source cannot end the identifier and add SQL of its own.DeltaSilverTables._merge · adapters/spark_silver.py:200no test names it
SparkStageRunner
class, line 92
stage_runner on PySpark: the stages' frames are checkpointed once so the counts and the write read the same rows (and a merge into the table the stages read from sees a fixed input)._table_port · scripts/platform_binding.py:98
_legacy · scripts/platform_binding.py:111
silver_helpers.py, test_silver_ports.py and 2 more
SparkStageRunner.__init__
method, line 96
none found in the codeno test names it
SparkStageRunner.run
method, line 99
none found in the codeno test names it
SparkStageRunner.release
method, line 126
none found in the codetest_meta_store.py, test_silver_ports.py and 2 more
DeltaSilverTables
class, line 131
The read and write edges on Delta folders._legacy · scripts/platform_binding.py:110test_silver_ports.py, test_silver_world.py and 1 more
DeltaSilverTables.__init__
method, line 134
none found in the codeno test names it
DeltaSilverTables._folder
method, line 138
DeltaSilverTables.silver_path · adapters/spark_silver.py:144
DeltaSilverTables.bronze · adapters/spark_silver.py:161
StoreSilverTables._table · adapters/spark_silver.py:290 (by name)
no test names it
DeltaSilverTables.silver_path
method, line 143
DeltaSilverTables.reset · adapters/spark_silver.py:157
DeltaSilverTables.current · adapters/spark_silver.py:168
DeltaSilverTables.write · adapters/spark_silver.py:172
DeltaSilverTables.current_keys · adapters/spark_silver.py:245
DeltaSilverTables.frame · adapters/spark_silver.py:277
test_bronze_to_silver.py, test_lineage_end_to_end.py and 7 more
DeltaSilverTables._exists
method, line 147
DeltaSilverTables._read · adapters/spark_silver.py:151
DeltaSilverTables.write · adapters/spark_silver.py:181
no test names it
DeltaSilverTables._read
method, line 150
DeltaSilverTables.bronze · adapters/spark_silver.py:162
DeltaSilverTables.current · adapters/spark_silver.py:168
DeltaSilverTables.current_keys · adapters/spark_silver.py:245
DeltaSilverTables.frame · adapters/spark_silver.py:277
no test names it
DeltaSilverTables.reset
method, line 153
none found in the codetest_governance_catalog.py, test_meta_store.py and 7 more
DeltaSilverTables.bronze
method, line 160
SilverBaseliner.write · silver/baseline_writer.py:63 (by name)
SilverBaseliner._check · silver/baseline_writer.py:81 (by name)
scan · silver/privacy_scan.py:32 (by name)
_one_leased · silver/runner.py:171 (by name)
test_access_model.py, test_baseline_stage.py and 45 more
DeltaSilverTables.current
method, line 167
DeltaSilverTables.measures · adapters/spark_silver.py:220
DeltaSilverTables.bucket_totals · adapters/spark_silver.py:237
DeltaSilverTables.keys_in_buckets · adapters/spark_silver.py:241
DeltaSilverTables.versions · adapters/spark_silver.py:253
DeltaSilverTables.count_by_hour · adapters/spark_silver.py:263
history · stages/history.py:314 (value)
and 5 more
gold_helpers.py, silver_helpers.py and 14 more
DeltaSilverTables.write
method, line 171
none found in the codegold_helpers.py, test_cloud_adapters.py and 31 more
DeltaSilverTables._append
method, line 188
DeltaSilverTables.write · adapters/spark_silver.py:176
step_prepare · realms/databricks/tasks/main.py:70 (by name)
no test names it
DeltaSilverTables._merge
method, line 193
DeltaSilverTables.write · adapters/spark_silver.py:185no test names it
DeltaSilverTables.measures
method, line 219
lint · obs/views.py:121 (value)
prove_rebuild.measures · silver/baseline_writer.py:108 (value)
compare · silver/daybyday.py:85 (value)
run_gate · twin/gate.py:202 (value)
prove_rebuild.measures · silver/baseline_writer.py:108 (by name)
compare · silver/daybyday.py:85 (by name)
test_semantic_governance.py, gold_helpers.py and 17 more
DeltaSilverTables.bucket_totals
method, line 235
reconcile_live · silver/live_recon.py:60 (by name)test_live_recon.py, test_reconcile_buckets.py and 1 more
DeltaSilverTables.keys_in_buckets
method, line 240
reconcile_live · silver/live_recon.py:68 (by name)test_live_recon.py
DeltaSilverTables.current_keys
method, line 244
_guard_keys · ingest/pipelines.py:239 (by name)test_ingest_pipelines.py, test_silver_ports.py
DeltaSilverTables.versions
method, line 252
UcIcebergStore._create · adapters/table_store.py:399 (value)
judge · silver/daybyday.py:117 (by name)
silver_helpers.py, test_gold_harness.py and 5 more
DeltaSilverTables.count_by_hour
method, line 262
hourly · silver/daybyday.py:73 (by name)test_silver_ports.py
DeltaSilverTables.frame
method, line 275
The silver (or quarantine) table as a frame, for checks and tests.none found in the codeconftest.py, gold_helpers.py and 5 more
StoreSilverTables
class, line 280
The read and write edges of silver through the table port (ADR-027): the same behaviour as DeltaSilverTables on any format the store speaks. Bronze is <source>__<dataset> in the bronze layer; silver and its quarantine (<source>__<dataset>___table_port · scripts/platform_binding.py:98no test names it
StoreSilverTables.__init__
method, line 285
none found in the codeno test names it
StoreSilverTables._table
method, line 289
StoreSilverTables.bronze · adapters/spark_silver.py:293
StoreSilverTables.current · adapters/spark_silver.py:299
StoreSilverTables.reset · adapters/spark_silver.py:303
StoreSilverTables.write · adapters/spark_silver.py:307
StoreSilverTables.current_keys · adapters/spark_silver.py:324
StoreSilverTables.frame · adapters/spark_silver.py:331
no test names it
StoreSilverTables.bronze
method, line 292
SilverBaseliner.write · silver/baseline_writer.py:63 (by name)
SilverBaseliner._check · silver/baseline_writer.py:81 (by name)
scan · silver/privacy_scan.py:32 (by name)
_one_leased · silver/runner.py:171 (by name)
test_access_model.py, test_baseline_stage.py and 45 more
StoreSilverTables.current
method, line 298
history · stages/history.py:314 (value)
duplicates · stages/rows.py:56 (value)
DeltaRsSilverTables.count_by_hour · adapters/duckdb_silver.py:127 (by name)
SilverBaseliner._check · silver/baseline_writer.py:86 (by name)
scan · silver/privacy_scan.py:26 (by name)
_one_leased · silver/runner.py:171 (by name)
gold_helpers.py, silver_helpers.py and 14 more
StoreSilverTables.reset
method, line 301
none found in the codetest_governance_catalog.py, test_meta_store.py and 7 more
StoreSilverTables.write
method, line 306
none found in the codegold_helpers.py, test_cloud_adapters.py and 31 more
StoreSilverTables.current_keys
method, line 323
_guard_keys · ingest/pipelines.py:239 (by name)test_ingest_pipelines.py, test_silver_ports.py
StoreSilverTables.frame
method, line 330
none found in the codeconftest.py, gold_helpers.py and 5 more
_bucket_col
function, line 334
The bucket of each row's business key, as dataplatform.reconcile.bucket_of computes it: sha256 of the key parts as text, first 8 hex digits, mod buckets.bucket_totals_frame · adapters/spark_silver.py:352
bucket_keys_frame · adapters/spark_silver.py:362
no test names it
_bucket_col.digit
function, line 340
_bucket_col · adapters/spark_silver.py:344test_value_objects.py
bucket_totals_frame
function, line 347
dataplatform.reconcile.bucket_totals computed in the engine over silver's current, not deleted rows: the same bucket function (sha256 of the key parts as text, first 8 hex digits, mod buckets), only the totals come back.DeltaSilverTables.bucket_totals · adapters/spark_silver.py:238test_reconcile_engine.py
bucket_keys_frame
function, line 359
The business keys (as text) of silver's current, not deleted rows in the wanted buckets only: the drill-down after a bucket differs, so only a few buckets' keys ever reach the driver.DeltaSilverTables.keys_in_buckets · adapters/spark_silver.py:242no test names it
hop_delays_frame
function, line 367
dataplatform.timing.BATCH_HOPS on a silver frame, computed in the engine: seconds between the stamps of each hop, then the percentiles; only hops whose stamps the frame has, and only rows that carry both stamps.hop_delays · scripts/run_platform.py:163test_timing.py

adapters/sql_source.py

SourceDatabase over any DB-API connection. Two thin adapters use it: postgres_source.py and sqlite_source.py. none yet

DefinitionWhat it does (its own first sentence)Called byNamed in tests
ident
function, line 26
module level · adapters/postgres_source.py:28
_create_database · adapters/postgres_source.py:39
SqlSource.ensure · adapters/sql_source.py:72
SqlSource._upsert · adapters/sql_source.py:105
SqlSource._delete · adapters/sql_source.py:115
SqlSource._to_feed · adapters/sql_source.py:126
and 4 more
test_semantic_governance.py, test_value_objects.py
Dialect
class, line 33
module level · adapters/postgres_source.py:17
module level · adapters/sqlite_source.py:12
no test names it
_plain
function, line 41
Values read back as the world wrote them: ISO strings for time, floats for money.SqlSource._select · adapters/sql_source.py:145no test names it
SqlSource
class, line 52
none found in the codeno test names it
SqlSource.__init__
method, line 53
none found in the codeno test names it
SqlSource.ensure
method, line 58
World.prepare · world/engine.py:82 (by name)test_ingest_pipelines.py, test_source_ports.py and 2 more
SqlSource.apply
method, line 90
World.write · world/engine.py:125 (by name)silver_helpers.py, test_baseline_stage.py and 15 more
SqlSource._upsert
method, line 102
SqlSource.apply · adapters/sql_source.py:96no test names it
SqlSource._delete
method, line 114
SqlSource.apply · adapters/sql_source.py:94no test names it
SqlSource._to_feed
method, line 121
SqlSource.apply · adapters/sql_source.py:99no test names it
SqlSource.publish_copy
method, line 130
World.write · world/engine.py:129 (by name)test_ingest_pipelines.py, test_source_ports.py
SqlSource._select
method, line 141
SqlSource.rows · adapters/sql_source.py:149
SqlSource.feed · adapters/sql_source.py:152
SqlSource.copy · adapters/sql_source.py:160
SqlSource.copies · adapters/sql_source.py:167
no test names it
SqlSource.rows
method, line 147
none found in the codeno test names it
SqlSource.feed
method, line 151
read_back · scripts/run_world.py:120 (by name)
ChangeFeedMover.move · adapters/local_movers.py:32 (by name)
DmsMover.move · adapters/local_movers.py:88 (by name)
no test names it
SqlSource.copy
method, line 158
none found in the codeno test names it
SqlSource.copies
method, line 164
pull_copy · ingest/pipelines.py:255 (by name)test_housekeeping.py, test_ingest_pipelines.py and 8 more
_same_instant
function, line 171
SqlSource.copy · adapters/sql_source.py:161no test names it
_runs
function, line 175
Consecutive changes grouped by kind (delete or not), keeping their order.SqlSource.apply · adapters/sql_source.py:92no test names it

adapters/sqlite_source.py

SourceDatabase on SQLite (standard library): for tests and for a laptop with no Postgres. SQLite has no schemas, so a system's tables are prefixed with its name: erp_db__sales_order_line. Sources and movers

DefinitionWhat it does (its own first sentence)Called byNamed in tests
SqliteSource
class, line 28
sinks · scripts/run_world.py:82test_ingest_world.py, test_semantic_metrics.py and 5 more
SqliteSource.__init__
method, line 29
none found in the codeno test names it

adapters/store_registry.py

Slice 1's batch registry (dataplatform.ports.MetadataStore: batch_status(key), record_batch) on the W1a metadata store, so slice 1's lander (dataplatform.lander) records into the same file log as W1b's ingestion, through the store's procedu none yet

adapters/table_store.py

The table port's engine side (data-platform-catalog dpc/ports.yaml table_writer, ADR-027): one small adapter per open table format, so silver, gold and bronze code never names a format. none yet

DefinitionWhat it does (its own first sentence)Called byNamed in tests
_check
function, line 33
DeltaPathStore.path · adapters/table_store.py:158
IcebergStore.name · adapters/table_store.py:236
IcebergStore.exists · adapters/table_store.py:258
UcIcebergStore.name · adapters/table_store.py:431
no test names it
quoted
function, line 38
A Spark SQL identifier in backticks, any backtick inside doubled, so a column name from a source cannot end the identifier and add SQL of its own._SqlStore.add_columns · adapters/table_store.py:93
_SqlStore.merge · adapters/table_store.py:104
DeltaPathStore.name · adapters/table_store.py:162
IcebergStore.name · adapters/table_store.py:237
IcebergStore._namespace · adapters/table_store.py:254
IcebergStore.tables · adapters/table_store.py:267
and 8 more
test_semantic_governance.py, test_mssql_connector.py and 3 more
iceberg_jar
function, line 46
The Iceberg Spark runtime for local runs: DP_ICEBERG_JAR, else ~/.dataplatform/jars/<jar>; None if absent.local_session · adapters/spark_silver.py:52conftest.py, test_table_store.py
TableStore
class, line 52
none found in the codeno test names it
TableStore.name
method, line 56
none found in the codeno test names it
TableStore.exists
method, line 57
main · realms/aws-databricks/generate_bundle.py:196 (by name)
main · realms/azure-databricks/generate_bundle.py:85 (by name)
test_bundle_has_the_engine_job_and_its_files_exist · realms/azure-databricks/tests/test_realm.py:285 (by name)
main · realms/databricks/dashboards/generate.py:55 (by name)
module level · realms/databricks/declarative/silver.py:19 (by name)
module level · realms/databricks/engine/tasks/run_days.py:26 (by name)
and 36 more
test_bundle.py, test_semantic_governance.py and 32 more
TableStore.read
method, line 58
none found in the codeno test names it
TableStore.tables
method, line 59
none found in the codetest_genie_deploy.py, test_lakeformation.py and 39 more
TableStore.append
method, line 60
none found in the codeazure_fakes.py, gold_helpers.py and 45 more
TableStore.merge
method, line 61
DeltaRsSilverTables._merge · adapters/duckdb_silver.py:117 (by name)
DeltaSilverTables._merge · adapters/spark_silver.py:209 (by name)
StoreSilverTables.write · adapters/spark_silver.py:319 (by name)
_profile_step · ingest/runner.py:68 (by name)
test_baseline_stage.py, test_meta_store.py and 7 more
TableStore.replace_where
method, line 62
StoreGoldTables.replace_windows · adapters/spark_gold.py:467 (by name)
UcIcebergStore.replace_where · adapters/table_store.py:413 (by name)
StoreBronzeWriter.write · adapters/table_store.py:539 (by name)
StoreBronzeWriter.write_many · adapters/table_store.py:559 (by name)
test_table_store.py
TableStore.overwrite
method, line 63
module level · realms/databricks/engine/tasks/probe.py:67 (by name)
write_calendar · scripts/run_platform.py:208 (by name)
StoreGoldTables.write_dimension · adapters/spark_gold.py:447 (by name)
StoreGoldTables.replace_windows · adapters/spark_gold.py:460 (by name)
azure_fakes.py, test_azure_adapters.py and 12 more
TableStore.add_columns
method, line 64
IcebergStore.append · adapters/table_store.py:285 (by name)
IcebergStore.replace_where · adapters/table_store.py:298 (by name)
UcIcebergStore.append · adapters/table_store.py:479 (by name)
no test names it
TableStore.upkeep
method, line 65
upkeep_all · scripts/run_platform.py:218 (by name)test_table_store.py
TableStore.delete_where
method, line 66
StoreBronzeWriter.remove · adapters/table_store.py:598 (by name)test_table_store.py
TableStore.version
method, line 67
DeltaRsGoldTables.silver_version · adapters/duckdb_gold.py:98 (by name)
StoreGoldTables.silver_version · adapters/spark_gold.py:431 (by name)
UcIcebergStore.version · adapters/table_store.py:384 (by name)
test_semantic_governance.py, gold_helpers.py and 23 more
TableStore.read_at
method, line 68
StoreGoldTables.silver_at · adapters/spark_gold.py:435 (by name)test_table_store.py
TableStore.drop
method, line 69
mover · scripts/run_platform.py:233 (value)
DropMover.__init__ · adapters/local_movers.py:53 (value)
main.reset_tables · scripts/run_platform.py:454 (by name)
SparkGoldEngine.overlay · adapters/spark_gold.py:154 (by name)
SparkGoldEngine.add_keys · adapters/spark_gold.py:202 (by name)
DeltaSilverTables.write · adapters/spark_silver.py:182 (by name)
and 14 more
no test names it
_SqlStore
class, line 72
What both formats share: MERGE and ADD COLUMNS in Spark SQL against the table's SQL name.none found in the codeno test names it
_SqlStore.name
method, line 77
_SqlStore.add_columns · adapters/table_store.py:96
_SqlStore.merge · adapters/table_store.py:110
_SqlStore.version · adapters/table_store.py:124
_SqlStore.read_at · adapters/table_store.py:133
_SqlStore.delete_where · adapters/table_store.py:145
no test names it
_SqlStore.exists
method, line 80
_SqlStore.version · adapters/table_store.py:121
_SqlStore.read_at · adapters/table_store.py:130
main · realms/aws-databricks/generate_bundle.py:196 (by name)
main · realms/azure-databricks/generate_bundle.py:85 (by name)
test_bundle_has_the_engine_job_and_its_files_exist · realms/azure-databricks/tests/test_realm.py:285 (by name)
main · realms/databricks/dashboards/generate.py:55 (by name)
and 38 more
test_bundle.py, test_semantic_governance.py and 32 more
_SqlStore.read
method, line 83
_SqlStore.add_columns · adapters/table_store.py:87
_SqlStore.delete_where · adapters/table_store.py:139
no test names it
_SqlStore.add_columns
method, line 86
_SqlStore.merge · adapters/table_store.py:100
IcebergStore.append · adapters/table_store.py:285 (by name)
IcebergStore.replace_where · adapters/table_store.py:298 (by name)
UcIcebergStore.append · adapters/table_store.py:479 (by name)
no test names it
_SqlStore.merge
method, line 99
DeltaRsSilverTables._merge · adapters/duckdb_silver.py:117 (by name)
DeltaSilverTables._merge · adapters/spark_silver.py:209 (by name)
StoreSilverTables.write · adapters/spark_silver.py:319 (by name)
_profile_step · ingest/runner.py:68 (by name)
test_baseline_stage.py, test_meta_store.py and 7 more
_SqlStore.version
method, line 119
The table's current version (Delta commit, Iceberg snapshot id), or None (INSP-01).DeltaRsGoldTables.silver_version · adapters/duckdb_gold.py:98 (by name)
StoreGoldTables.silver_version · adapters/spark_gold.py:431 (by name)
UcIcebergStore.version · adapters/table_store.py:384 (by name)
test_semantic_governance.py, gold_helpers.py and 23 more
_SqlStore.read_at
method, line 128
The table as it was at `version` (time travel).StoreGoldTables.silver_at · adapters/spark_gold.py:435 (by name)test_table_store.py
_SqlStore.delete_where
method, line 137
Delete the rows matching `predicate` (built by the caller from checked values only); returns how many.StoreBronzeWriter.remove · adapters/table_store.py:598 (by name)test_table_store.py
DeltaPathStore
class, line 149
Delta tables in folders: <roots[layer]>/<table>.make_store · adapters/table_store.py:499test_table_store.py
DeltaPathStore.__init__
method, line 154
none found in the codeno test names it
DeltaPathStore.path
method, line 157
DeltaPathStore.name · adapters/table_store.py:162
DeltaPathStore.exists · adapters/table_store.py:165
DeltaPathStore.read · adapters/table_store.py:170
DeltaPathStore.append · adapters/table_store.py:179
DeltaPathStore._version · adapters/table_store.py:187
DeltaPathStore.replace_where · adapters/table_store.py:196
and 2 more
no test names it
DeltaPathStore.name
method, line 161
DeltaPathStore.upkeep · adapters/table_store.py:205no test names it
DeltaPathStore.exists
method, line 164
DeltaPathStore.read · adapters/table_store.py:168
DeltaPathStore._version · adapters/table_store.py:183
DeltaPathStore.replace_where · adapters/table_store.py:192
DeltaPathStore.upkeep · adapters/table_store.py:202
main · realms/aws-databricks/generate_bundle.py:196 (by name)
main · realms/azure-databricks/generate_bundle.py:85 (by name)
and 39 more
test_bundle.py, test_semantic_governance.py and 32 more
DeltaPathStore.read
method, line 167
none found in the codeno test names it
DeltaPathStore.tables
method, line 172
none found in the codetest_genie_deploy.py, test_lakeformation.py and 39 more
DeltaPathStore.append
method, line 176
none found in the codeazure_fakes.py, gold_helpers.py and 45 more
DeltaPathStore._version
method, line 182
DeltaPathStore.append · adapters/table_store.py:177no test names it
DeltaPathStore.replace_where
method, line 190
StoreGoldTables.replace_windows · adapters/spark_gold.py:467 (by name)
UcIcebergStore.replace_where · adapters/table_store.py:413 (by name)
StoreBronzeWriter.write · adapters/table_store.py:539 (by name)
StoreBronzeWriter.write_many · adapters/table_store.py:559 (by name)
test_table_store.py
DeltaPathStore.overwrite
method, line 198
module level · realms/databricks/engine/tasks/probe.py:67 (by name)
write_calendar · scripts/run_platform.py:208 (by name)
StoreGoldTables.write_dimension · adapters/spark_gold.py:447 (by name)
StoreGoldTables.replace_windows · adapters/spark_gold.py:460 (by name)
azure_fakes.py, test_azure_adapters.py and 12 more
DeltaPathStore.upkeep
method, line 201
upkeep_all · scripts/run_platform.py:218 (by name)test_table_store.py
DeltaPathStore.drop
method, line 208
mover · scripts/run_platform.py:233 (value)
DropMover.__init__ · adapters/local_movers.py:53 (value)
main.reset_tables · scripts/run_platform.py:454 (by name)
SparkGoldEngine.overlay · adapters/spark_gold.py:154 (by name)
SparkGoldEngine.add_keys · adapters/spark_gold.py:202 (by name)
DeltaSilverTables.write · adapters/spark_silver.py:182 (by name)
and 14 more
no test names it
IcebergStore
class, line 214
Iceberg tables in a Spark catalog: <catalog>.<layer>.<table>. Locally a folder ("hadoop") catalog configured by local_session(iceberg_warehouse=...); on Databricks the workspace's Unity Catalog with managed Iceberg tables.make_store · adapters/table_store.py:501test_table_store.py
IcebergStore.__init__
method, line 226
none found in the codeno test names it
IcebergStore.name
method, line 235
IcebergStore.read · adapters/table_store.py:262
IcebergStore._committed · adapters/table_store.py:271
IcebergStore.append · adapters/table_store.py:278
IcebergStore.replace_where · adapters/table_store.py:296
IcebergStore.overwrite · adapters/table_store.py:314
IcebergStore.version · adapters/table_store.py:337
and 1 more
no test names it
IcebergStore._with_properties
method, line 239
IcebergStore.append · adapters/table_store.py:281
IcebergStore.replace_where · adapters/table_store.py:296
IcebergStore.overwrite · adapters/table_store.py:314
no test names it
IcebergStore._plain
method, line 244
IcebergStore.exists · adapters/table_store.py:259no test names it
IcebergStore._prepare
method, line 247
Whatever must exist before a table is written (here: the layer's namespace).IcebergStore.append · adapters/table_store.py:277
IcebergStore.replace_where · adapters/table_store.py:294
IcebergStore.overwrite · adapters/table_store.py:313
no test names it
IcebergStore._namespace
method, line 251
IcebergStore._prepare · adapters/table_store.py:249
IcebergStore.tables · adapters/table_store.py:265
no test names it
IcebergStore.exists
method, line 257
IcebergStore.read · adapters/table_store.py:262
IcebergStore.append · adapters/table_store.py:280
IcebergStore.replace_where · adapters/table_store.py:295
IcebergStore.upkeep · adapters/table_store.py:317
IcebergStore.version · adapters/table_store.py:334
IcebergStore.drop · adapters/table_store.py:342
and 42 more
test_bundle.py, test_semantic_governance.py and 32 more
IcebergStore.read
method, line 261
none found in the codeno test names it
IcebergStore.tables
method, line 264
none found in the codetest_genie_deploy.py, test_lakeformation.py and 39 more
IcebergStore._committed
method, line 270
IcebergStore.append · adapters/table_store.py:283no test names it
IcebergStore.append
method, line 276
none found in the codeazure_fakes.py, gold_helpers.py and 45 more
IcebergStore.replace_where
method, line 290
One commit that deletes the rows matching `predicate` and inserts `df`. Iceberg's overwrite-by-filter only drops whole files and refuses a file that holds rows on both sides of the filter, so this is a MERGE that matches nothing: every new StoreGoldTables.replace_windows · adapters/spark_gold.py:467 (by name)
UcIcebergStore.replace_where · adapters/table_store.py:413 (by name)
StoreBronzeWriter.write · adapters/table_store.py:539 (by name)
StoreBronzeWriter.write_many · adapters/table_store.py:559 (by name)
test_table_store.py
IcebergStore.overwrite
method, line 312
module level · realms/databricks/engine/tasks/probe.py:67 (by name)
write_calendar · scripts/run_platform.py:208 (by name)
StoreGoldTables.write_dimension · adapters/spark_gold.py:447 (by name)
StoreGoldTables.replace_windows · adapters/spark_gold.py:460 (by name)
azure_fakes.py, test_azure_adapters.py and 12 more
IcebergStore.upkeep
method, line 316
upkeep_all · scripts/run_platform.py:218 (by name)test_table_store.py
IcebergStore.version
method, line 332
Open-source Iceberg has no DESCRIBE HISTORY: the newest id in the snapshots metadata table.DeltaRsGoldTables.silver_version · adapters/duckdb_gold.py:98 (by name)
StoreGoldTables.silver_version · adapters/spark_gold.py:431 (by name)
UcIcebergStore.version · adapters/table_store.py:384 (by name)
test_semantic_governance.py, gold_helpers.py and 23 more
IcebergStore.drop
method, line 341
mover · scripts/run_platform.py:233 (value)
DropMover.__init__ · adapters/local_movers.py:53 (value)
main.reset_tables · scripts/run_platform.py:454 (by name)
SparkGoldEngine.overlay · adapters/spark_gold.py:154 (by name)
SparkGoldEngine.add_keys · adapters/spark_gold.py:202 (by name)
DeltaSilverTables.write · adapters/spark_silver.py:182 (by name)
and 14 more
no test names it
layer_properties
function, line 356
The create-time table properties of a managed Iceberg table in this layer.UcIcebergStore._create · adapters/table_store.py:399test_uc_table_names.py
cluster_clause
function, line 364
CLUSTER BY clause for a new table: the layer's keys that the table has, else AUTO; None when the layer is not clustered (landing, ops).UcIcebergStore._create · adapters/table_store.py:403test_uc_table_names.py
UcIcebergStore
class, line 374
Unity Catalog managed Iceberg tables on Databricks (probed 13 Nov on Free Edition serverless): <env>_<layer>.<source or schema>.<dataset or name> (ADR-012: one catalog per environment and layer). Managed Iceberg has no readable snapshot metmake_store · adapters/table_store.py:503test_uc_table_names.py
UcIcebergStore.version
method, line 382
Unity Catalog managed Iceberg answers DESCRIBE HISTORY like Delta (its snapshots table is not readable).DeltaRsGoldTables.silver_version · adapters/duckdb_gold.py:98 (by name)
StoreGoldTables.silver_version · adapters/spark_gold.py:431 (by name)
test_semantic_governance.py, gold_helpers.py and 23 more
UcIcebergStore.__init__
method, line 386
none found in the codeno test names it
UcIcebergStore._create
method, line 396
Create with the layer's properties (format version), then set its clustering (B17).UcIcebergStore.replace_where · adapters/table_store.py:411
UcIcebergStore.overwrite · adapters/table_store.py:417
UcIcebergStore.append · adapters/table_store.py:470
no test names it
UcIcebergStore.replace_where
method, line 408
UcIcebergStore.append · adapters/table_store.py:477
StoreGoldTables.replace_windows · adapters/spark_gold.py:467 (by name)
StoreBronzeWriter.write · adapters/table_store.py:539 (by name)
StoreBronzeWriter.write_many · adapters/table_store.py:559 (by name)
test_table_store.py
UcIcebergStore.overwrite
method, line 415
module level · realms/databricks/engine/tasks/probe.py:67 (by name)
write_calendar · scripts/run_platform.py:208 (by name)
StoreGoldTables.write_dimension · adapters/spark_gold.py:447 (by name)
StoreGoldTables.replace_windows · adapters/spark_gold.py:460 (by name)
azure_fakes.py, test_azure_adapters.py and 12 more
UcIcebergStore._split
method, line 420
schema__table. The catalog already names the layer, so a schema that repeats it loses the prefix (gold's gold_sales__fact_otif -> <env>_gold.sales.fact_otif, as the contracts' physicalName says, ADR-012).UcIcebergStore.name · adapters/table_store.py:432
UcIcebergStore._plain · adapters/table_store.py:436
UcIcebergStore._prepare · adapters/table_store.py:440
no test names it
UcIcebergStore.name
method, line 430
UcIcebergStore._create · adapters/table_store.py:398
UcIcebergStore.append · adapters/table_store.py:480
UcIcebergStore.upkeep · adapters/table_store.py:487
UcIcebergStore.drop · adapters/table_store.py:492
no test names it
UcIcebergStore._plain
method, line 435
none found in the codeno test names it
UcIcebergStore._prepare
method, line 439
UcIcebergStore.replace_where · adapters/table_store.py:409
UcIcebergStore.overwrite · adapters/table_store.py:416
UcIcebergStore.append · adapters/table_store.py:468
no test names it
UcIcebergStore.tables
method, line 447
Every table of the layer's catalog as schema__table. SHOW reads the catalog itself (information_schema can lag a table created moments before, 13 Nov).none found in the codetest_genie_deploy.py, test_lakeformation.py and 39 more
UcIcebergStore.append
method, line 467
none found in the codeazure_fakes.py, gold_helpers.py and 45 more
UcIcebergStore.upkeep
method, line 483
upkeep_all · scripts/run_platform.py:218 (by name)test_table_store.py
UcIcebergStore.drop
method, line 490
mover · scripts/run_platform.py:233 (value)
DropMover.__init__ · adapters/local_movers.py:53 (value)
main.reset_tables · scripts/run_platform.py:454 (by name)
SparkGoldEngine.overlay · adapters/spark_gold.py:154 (by name)
SparkGoldEngine.add_keys · adapters/spark_gold.py:202 (by name)
DeltaSilverTables.write · adapters/spark_silver.py:182 (by name)
and 14 more
no test names it
make_store
function, line 495
The store for a local run under `out`: delta folders out/<layer>/, or the Iceberg catalog set up by local_session(iceberg_warehouse=out/iceberg)._table_port · scripts/platform_binding.py:93test_table_store.py
StoreBronzeWriter
class, line 507
BronzeWriter (dataplatform.ingest.ports) through the table port: bronze table <source>__<dataset>. One write replaces the rows of its batch key, so a batch written twice leaves the table unchanged (DUP-01) and a failed write leaves nothing _table_port · scripts/platform_binding.py:97test_table_store.py
StoreBronzeWriter.__init__
method, line 516
none found in the codeno test names it
StoreBronzeWriter._table
method, line 519
StoreBronzeWriter.write · adapters/table_store.py:532
StoreBronzeWriter.write_many · adapters/table_store.py:557
StoreBronzeWriter._rows · adapters/table_store.py:566
StoreBronzeWriter.remove · adapters/table_store.py:598
no test names it
StoreBronzeWriter.write
method, line 525
none found in the codegold_helpers.py, test_cloud_adapters.py and 31 more
StoreBronzeWriter.write_many
method, line 543
New batches in one append: one commit. The mark is the batch keys' digest, so a retry of the same set after a crash between the commit and the file log writes nothing twice.flush · ingest/steps.py:365 (by name)no test names it
StoreBronzeWriter._rows
method, line 565
StoreBronzeWriter.fingerprint · adapters/table_store.py:576
StoreBronzeWriter.keys · adapters/table_store.py:584
no test names it
StoreBronzeWriter.fingerprint
method, line 574
_one · ingest/runner.py:149 (value)
FixtureRunner.run_day · obs/fixture.py:117 (value)
FixtureRunner._rerun · obs/fixture.py:134 (value)
run_days · ingest/daybyday.py:133 (by name)
gold_helpers.py, silver_helpers.py and 14 more
StoreBronzeWriter.keys
method, line 583
none found in the codeno test names it
StoreBronzeWriter.remove
method, line 589
Housekeeping (B8): delete the rows of these batches in one commit; returns the rows removed.main.housekeeping · scripts/run_platform.py:506 (value)
_sweep_leased · housekeeping.py:170 (value)
_sweep_leased · housekeeping.py:170 (by name)
RunHandle.dataset · obs/recorder.py:160 (by name)
SilverBaseliner.write · silver/baseline_writer.py:70 (by name)
test_lakeformation.py, test_housekeeping.py and 2 more

adapters/uc_catalog.py

Catalog state read from Unity Catalog's information schema (Databricks; dbs/engine/databricks.yaml data_catalog). none yet

DefinitionWhat it does (its own first sentence)Called byNamed in tests
UcCatalogState
class, line 40
none found in the codetest_governance_catalog.py
UcCatalogState.__init__
method, line 41
none found in the codeno test names it
UcCatalogState.entries
method, line 44
catalog_actions · realms/databricks/semantic_governance/generate.py:129 (by name)test_governance_catalog.py, test_silver_plan.py