cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Code map › Who calls what › obs

Data PlatformZoom level 3

Who calls what: obs

Every class, function and method in the obs package, with who calls it. Found by reading the code. A call written as object.method() is matched by the method's name, so a same-named method elsewhere can appear: those rows say by name.

262 definitions in 22 modules. Jump to a module: alerts.py · clock.py · dashboards.py · explain.py · explain_meta.py · fixture.py · freshness.py · lineage.py · monitor.py · native.py · ops.py · ops_export.py · page.py · pipeline_events.py · recorder.py · runlog.py · steps.py · success.py · system.py · trace.py · traced_store.py · views.py

obs/alerts.py

Alerts that people still read (OPS-01, OPS-06, DPC-0019, DPC-0056). Rules are configuration; evaluation is pure. Operations

DefinitionWhat it does (its own first sentence)Called byNamed in tests
AlertConfigError
class, line 37
An alert rule, route or snooze the config may not hold.AlertConfig.rule · obs/alerts.py:77
_need · obs/alerts.py:140
_aware · obs/alerts.py:147
parse_snooze · obs/alerts.py:155
parse_config · obs/alerts.py:169
test_obs_alerts.py
Rule
class, line 42
parse_config · obs/alerts.py:166
module level · obs/alerts.py:66 (value)
no test names it
Route
class, line 50
parse_config · obs/alerts.py:178
module level · obs/alerts.py:67 (value)
no test names it
Snooze
class, line 56
parse_snooze · obs/alerts.py:159
module level · obs/alerts.py:69 (value)
no test names it
AlertConfig
class, line 65
parse_config · obs/alerts.py:188no test names it
AlertConfig.rule
method, line 73
payload · adapters/databricks_notify.py:30 (value)
counts_per_rule_per_day · obs/alerts.py:295 (value)
alert_tables · obs/ops.py:287 (value)
unexplained_alerts · twin/runner.py:197 (value)
no test names it
Signal
class, line 81
One subject's verdict for one rule's signal at one check._maker.make · obs/ops.py:328
evaluate · obs/alerts.py:238 (value)
test_obs_alerts.py
Alert
class, line 94
One message to one route: a rule's alert for a business day, with every subject that joined it._fail · obs/alerts.py:276
_recover · obs/alerts.py:286
test_obs_alerts.py, test_obs_notify.py
Event
class, line 106
_fail · obs/alerts.py:266
_recover · obs/alerts.py:285
test_resilience.py, test_lineage_dbt_manifest.py
Scope
class, line 117
What one evaluation speaks for: these signal kinds, and (if given) only subjects of these datasets. A subject outside the scope keeps its state; inside it, a subject absent from the signals has recovered.evaluate · obs/alerts.py:234
Monitor._lateness · obs/monitor.py:92
Monitor.run_finished · obs/monitor.py:103
test_obs_alerts.py
Scope.covers
method, line 124
evaluate · obs/alerts.py:252 (by name)test_access_model.py, test_obs_dashboards.py and 1 more
Evaluation
class, line 129
evaluate · obs/alerts.py:254no test names it
_need
function, line 138
parse_snooze · obs/alerts.py:153
parse_config · obs/alerts.py:165
no test names it
_aware
function, line 144
parse_snooze · obs/alerts.py:156no test names it
parse_snooze
function, line 151
parse_config · obs/alerts.py:184no test names it
parse_config
function, line 163
load_config · obs/alerts.py:195test_obs_alerts.py
load_config
function, line 192
main · scripts/run_platform.py:424
Harness.build · twin/runner.py:96
test_obs_alerts.py, test_obs_dashboards.py
_zone
function, line 203
business_day · obs/alerts.py:208no test names it
business_day
function, line 207
evaluate · obs/alerts.py:236
counts_per_rule_per_day · obs/alerts.py:295
runs_table · obs/ops.py:115
dataset_runs_table · obs/ops.py:146
steps_table · obs/ops.py:191
restatements_table · obs/ops.py:214
and 9 more
no test names it
route_for
function, line 211
_fail · obs/alerts.py:263no test names it
snoozed
function, line 219
_fail · obs/alerts.py:265test_obs_alerts.py
_key
function, line 226
_fail · obs/alerts.py:259no test names it
evaluate
function, line 230
One check. Subjects a rule knew as failing that are absent from this check's signals have recovered (within the scope: by default every signal and every subject).Monitor._evaluate · obs/monitor.py:78test_obs_alerts.py, test_config_sync.py and 3 more
_fail
function, line 257
evaluate · obs/alerts.py:249no test names it
_recover
function, line 280
evaluate · obs/alerts.py:253no test names it
counts_per_rule_per_day
function, line 290
OPS-06 evidence: alerts sent per rule per business day (fired only; grouped and snoozed are not sent).alert_tables · obs/ops.py:289test_obs_alerts.py

obs/clock.py

The clock port (DPC-0032): observability reads time only through this, so a simulated week runs in a second. Operations

DefinitionWhat it does (its own first sentence)Called byNamed in tests
Clock
class, line 13
none found in the codetest_resilience.py
Clock.now
method, line 14
An aware datetime.step_prepare · realms/databricks/tasks/main.py:71 (by name)
module level · realms/databricks/v1/tasks/hop.py:28 (by name)
main · scripts/dac_coverage.py:40 (by name)
WallClock.now · scripts/run_platform.py:106 (by name)
write_ops_page · scripts/run_platform.py:130 (by name)
main.pushed · scripts/run_platform.py:408 (by name)
and 20 more
no test names it
SystemClock
class, line 19
none found in the codeno test names it
SystemClock.now
method, line 20
step_prepare · realms/databricks/tasks/main.py:71 (by name)
module level · realms/databricks/v1/tasks/hop.py:28 (by name)
main · scripts/dac_coverage.py:40 (by name)
WallClock.now · scripts/run_platform.py:106 (by name)
write_ops_page · scripts/run_platform.py:130 (by name)
main.pushed · scripts/run_platform.py:408 (by name)
and 19 more
no test names it
ManualClock
class, line 24
Harness.build · twin/runner.py:89test_obs_recorder.py, test_obs_runlog.py and 1 more
ManualClock.__init__
method, line 25
none found in the codeno test names it
ManualClock.now
method, line 30
step_prepare · realms/databricks/tasks/main.py:71 (by name)
module level · realms/databricks/v1/tasks/hop.py:28 (by name)
main · scripts/dac_coverage.py:40 (by name)
WallClock.now · scripts/run_platform.py:106 (by name)
write_ops_page · scripts/run_platform.py:130 (by name)
main.pushed · scripts/run_platform.py:408 (by name)
and 20 more
no test names it
ManualClock.set
method, line 33
none found in the codeno test names it
ManualClock.advance
method, line 38
FixtureRunner._rerun · obs/fixture.py:138 (by name)
FixtureRunner._dataset · obs/fixture.py:149 (by name)
test_obs_recorder.py, test_obs_runlog.py and 3 more

obs/dashboards.py

The ops dashboards as definitions (G8, DPC-0057, ADR-024). Product-free: a realm compiles them to its BI tool's file (a dashboard-file adapter writes them into the realm's dashboards folder), and `evaluate` runs any of them on the ops table Operations

DefinitionWhat it does (its own first sentence)Called byNamed in tests
Measure
class, line 35
module level · des/dashboards.py:21
_m · obs/dashboards.py:173
module level · obs/dashboards.py:46 (value)
test_obs_dashboards.py
Query
class, line 42
module level · des/dashboards.py:21
module level · obs/dashboards.py:183
module level · obs/dashboards.py:75 (value)
test_obs_dashboards.py, test_mssql_connector.py
Query.fields
method, line 54
none found in the codetest_ingest_ports.py, test_meta_store.py and 16 more
Widget
class, line 59
module level · des/dashboards.py:34
module level · obs/dashboards.py:197
module level · obs/dashboards.py:76 (value)
no test names it
Dashboard
class, line 71
module level · des/dashboards.py:15
module level · obs/dashboards.py:176
bundle_resources · adapters/lakeview.py:144 (value)
every · obs/dashboards.py:297 (value)
test_obs_dashboards.py
Dashboard.query
method, line 80
_handler.Handler.do_GET · adapters/http_api.py:73 (value)
validate · obs/dashboards.py:116 (value)
render · obs/dashboards.py:313 (value)
check_on_dashboard · twin/runner.py:143 (value)
validate · obs/dashboards.py:116 (by name)
render · obs/dashboards.py:313 (by name)
and 1 more
silver_helpers.py, test_contract_rules.py and 12 more
DefinitionError
class, line 87
validate · obs/dashboards.py:94test_obs_dashboards.py
validate
function, line 91
compile_dashboard · adapters/lakeview.py:132test_obs_dashboards.py, test_slice1_local.py and 3 more
_match
function, line 124
evaluate · obs/dashboards.py:153no test names it
_agg
function, line 132
evaluate · obs/dashboards.py:163no test names it
_sort_key
function, line 145
evaluate · obs/dashboards.py:155no test names it
_sort_key.key
function, line 146
none found in the codeno test names it
evaluate
function, line 152
render · obs/dashboards.py:313test_obs_alerts.py, test_config_sync.py and 3 more
_m
function, line 172
module level · obs/dashboards.py:183no test names it
every
function, line 297
Observability's dashboards and the DES-owned ones (generated side by side, owned apart).files · realms/databricks/dashboards/generate.py:30
by_id · obs/dashboards.py:305
test_bundle.py, test_lakeformation.py and 56 more
by_id
function, line 304
none found in the codeno test names it
render
function, line 311
Every widget's rows, by widget id.none found in the codetest_semantic_governance.py, test_obs_dashboards.py and 3 more

obs/explain.py

A run explained in plain words, deterministically: the run's report and ops tables in, a Markdown narrative out. Operations

obs/explain_meta.py

A dataset's metadata and the repository's checks explained in plain words, deterministically: no model, no network. Operations

DefinitionWhat it does (its own first sentence)Called byNamed in tests
listed
function, line 59
_promise · obs/explain_meta.py:95
_history · obs/explain_meta.py:110
_checked_and_owned · obs/explain_meta.py:167
_workflow · obs/explain_meta.py:209
test_azure_adapters.py, test_cloud_adapters.py and 7 more
_arrival
function, line 64
explain_dataset · obs/explain_meta.py:185no test names it
_promise
function, line 91
explain_dataset · obs/explain_meta.py:185no test names it
_history
function, line 99
explain_dataset · obs/explain_meta.py:185no test names it
_loading
function, line 126
explain_dataset · obs/explain_meta.py:185no test names it
_bad_rows
function, line 146
explain_dataset · obs/explain_meta.py:186no test names it
_checked_and_owned
function, line 160
explain_dataset · obs/explain_meta.py:186no test names it
explain_dataset
function, line 178
The Markdown narrative of one dataset manifest (and its contract, when given).cmd_metadata · scripts/explain.py:76test_explain.py
_triggers
function, line 197
_workflow · obs/explain_meta.py:209no test names it
_step_name
function, line 204
_workflow · obs/explain_meta.py:211no test names it
_workflow
function, line 208
explain_cicd · obs/explain_meta.py:226no test names it
explain_cicd
function, line 216
The checks a change passes. help_lines: the Makefile's `make <target> <what it does>` lines; hooks: (repo, hook id) pairs; workflows: file name -> parsed workflow.cmd_cicd · scripts/explain.py:94test_explain.py

obs/fixture.py

FIXTURE - a tiny fake runner that fills the run log until W1b-W1e produce real runs. Operations

DefinitionWhat it does (its own first sentence)Called byNamed in tests
Effect
class, line 40
What the run log shows when a detector meets its incident (the engine's expected behaviour, per DAC).module level · obs/fixture.py:52
module level · obs/fixture.py:51 (value)
no test names it
FixtureRun
class, line 73
FixtureRunner.__init__ · obs/fixture.py:100no test names it
_counts
function, line 78
FixtureRunner._effect · obs/fixture.py:179no test names it
FixtureRunner
class, line 90
Harness.build · twin/runner.py:97no test names it
FixtureRunner.__init__
method, line 91
none found in the codeno test names it
FixtureRunner.day_start
method, line 102
FixtureRunner.run_day · obs/fixture.py:109
FixtureRunner._readers · obs/fixture.py:217
Harness.build · twin/runner.py:88 (by name)
ClinicalTrial._sites · world/domains/clinical.py:752 (by name)
ClinicalTrial._readings · world/domains/clinical.py:788 (by name)
ClinicalTrial._reading_truth · world/domains/clinical.py:853 (by name)
and 5 more
no test names it
FixtureRunner._pushed
method, line 105
FixtureRunner.__init__ · obs/fixture.py:97 (value)no test names it
FixtureRunner.run_day
method, line 108
Harness.build · twin/runner.py:101 (by name)no test names it
FixtureRunner._rerun
method, line 130
FixtureRunner.run_day · obs/fixture.py:121no test names it
FixtureRunner._dataset
method, line 143
FixtureRunner.run_day · obs/fixture.py:118no test names it
FixtureRunner._effect
method, line 170
FixtureRunner._dataset · obs/fixture.py:155no test names it
FixtureRunner._bookmarks
method, line 189
FixtureRunner._dataset · obs/fixture.py:167no test names it
FixtureRunner._system
method, line 204
FixtureRunner._rerun · obs/fixture.py:140
FixtureRunner._dataset · obs/fixture.py:168
no test names it
FixtureRunner._readers
method, line 216
FixtureRunner.run_day · obs/fixture.py:122no test names it

obs/freshness.py

Freshness against the manifest's targets (FRESH-01, OPS-01, G10). Pure functions, plus one reader of the store. Operations

DefinitionWhat it does (its own first sentence)Called byNamed in tests
FreshnessTarget
class, line 31
target_from_view · obs/freshness.py:82test_obs_freshness.py
FreshnessTarget.name
method, line 41
none found in the codeno test names it
FreshnessResult
class, line 46
check · obs/freshness.py:135no test names it
FreshnessResult.subject
method, line 59
snoozed · obs/alerts.py:221 (value)
_fail · obs/alerts.py:259 (value)
freshness_tables · obs/ops.py:257 (value)
alert_tables · obs/ops.py:287 (value)
freshness_signals · obs/ops.py:356 (value)
unexplained_alerts · twin/runner.py:197 (value)
test_ingest_steps.py, test_obs_alerts.py and 3 more
tier_for
function, line 64
target_from_view · obs/freshness.py:82test_obs_freshness.py
target_from_view
function, line 73
A dataset's targets from the store's merged view; None when the dataset declares no freshness target.targets_from_store · obs/freshness.py:92test_obs_freshness.py
targets_from_store
function, line 87
none found in the codetest_obs_freshness.py
state_of
function, line 98
check · obs/freshness.py:136test_obs_freshness.py
_hours
function, line 106
check · obs/freshness.py:134no test names it
last_success
function, line 110
check · obs/freshness.py:130
next_due · obs/freshness.py:158
test_obs_freshness.py
_as_time
function, line 119
check · obs/freshness.py:139no test names it
check
function, line 127
Every target's dataset freshness, then each time-valued bookmark's lag behind its newest sibling.none found in the codetest_azure_adapters.py, test_contract_rules.py and 30 more
read_bookmarks
function, line 150
none found in the codeno test names it
next_due
function, line 154
When the next dataset would pass its error target if nothing loads before then: the one moment a lateness check is worth running (ADR-024 rule 4, no fixed interval). None when nothing can become late (never loaded datasets are already repornone found in the codetest_obs_freshness.py

obs/lineage.py

Lineage before the engine's catalog (X7, LIN-01, B9): the engine's catalog records lineage from bronze on; the hops before it (source system -> mover -> landing -> bronze) are only in our file log. This turns file-log records into run event Operations

DefinitionWhat it does (its own first sentence)Called byNamed in tests
events_from_files
function, line 24
Run events for the batches in `files`. `system_of` maps a source name to its system (erp -> erp_db), which names the input dataset's namespace; `at` is the event time (ISO, from the clock port). Duplicates skipped by the lander produce no enone found in the codetest_obs_lineage.py

obs/monitor.py

Monitoring, pushed by runs (ADR-024 rule 4): nothing here runs on a fixed interval. Operations

DefinitionWhat it does (its own first sentence)Called byNamed in tests
Notifier
class, line 34
Where an alert goes: a pager, a ticket queue, a digest. The route says which; the adapter delivers.none found in the codeno test names it
Notifier.send
method, line 37
Monitor._evaluate · obs/monitor.py:81 (by name)no test names it
MemoryNotifier
class, line 40
main · scripts/run_platform.py:424
FixtureRunner.__init__ · obs/fixture.py:95
test_obs_dashboards.py, test_obs_notify.py
MemoryNotifier.__init__
method, line 41
none found in the codeno test names it
MemoryNotifier.send
method, line 44
Monitor._evaluate · obs/monitor.py:81 (by name)no test names it
CheckOutput
class, line 49
Monitor._evaluate · obs/monitor.py:84
Monitor.run_finished · obs/monitor.py:105
no test names it
_only
function, line 55
Monitor.run_finished · obs/monitor.py:100no test names it
Monitor
class, line 63
main · scripts/run_platform.py:423
FixtureRunner.__init__ · obs/fixture.py:96
test_obs_dashboards.py
Monitor.__init__
method, line 64
none found in the codeno test names it
Monitor._evaluate
method, line 77
Monitor._lateness · obs/monitor.py:92
Monitor.run_finished · obs/monitor.py:103
no test names it
Monitor._lateness
method, line 86
Monitor.run_finished · obs/monitor.py:104
Monitor.check_lateness · obs/monitor.py:108
no test names it
Monitor.run_finished
method, line 96
Pushed by the recorder as a run ends: the run's own datasets, detectors and checks, then lateness (a run that loads can end a lateness).main.pushed · scripts/run_platform.py:408 (by name)
FixtureRunner._pushed · obs/fixture.py:106 (by name)
no test names it
Monitor.check_lateness
method, line 107
Monitor.wake · obs/monitor.py:114no test names it
Monitor.wake
method, line 110
The scheduler's one wake-up: a lateness check only when one is due (`self.due`), else nothing runs.write_ops_page · scripts/run_platform.py:131 (by name)
FixtureRunner.run_day · obs/fixture.py:124 (by name)
no test names it
Monitor.copy_system
method, line 116
FixtureRunner.run_day · obs/fixture.py:127 (by name)test_obs_dashboards.py
Monitor.tables
method, line 119
none found in the codetest_genie_deploy.py, test_lakeformation.py and 39 more

obs/native.py

Platform-native screens and notifications, adopted, not rebuilt (ADR-024 rule 1, set a). Operations

DefinitionWhat it does (its own first sentence)Called byNamed in tests
Native
class, line 18
module level · obs/native.py:26no test names it

obs/ops.py

The ops builder (G8, DPC-0057): run log + platform telemetry + freshness + alerts -> the compact ops tables. Pure. Operations

DefinitionWhat it does (its own first sentence)Called byNamed in tests
_secs
function, line 93
runs_table · obs/ops.py:117
_dataset_row · obs/ops.py:140
step_durations_table · obs/ops.py:230
no test names it
check_kind
function, line 97
none found in the codeno test names it
_telemetry_jobs
function, line 101
Platform job runs no run-log run accounts for (all of them when no run log is connected).runs_table · obs/ops.py:121
dataset_runs_table · obs/ops.py:155
no test names it
runs_table
function, line 107
build · obs/ops.py:301no test names it
_dataset_row
function, line 136
dataset_runs_table · obs/ops.py:146no test names it
dataset_runs_table
function, line 144
build · obs/ops.py:295no test names it
_performance
function, line 159
Attempts per dataset per day (a retry is any attempt after the first) and the time against the dataset's own median so far (a slow-down is SLOW_RATIO or more). Only what needs the run log: per dataset, not per job.dataset_runs_table · obs/ops.py:156no test names it
_count_by
function, line 176
build · obs/ops.py:303no test names it
steps_table
function, line 184
Every owner's step row (the one step format), with the owner and the owner's screen to go to.build · obs/ops.py:305no test names it
_fields
function, line 201
restatements_table · obs/ops.py:211no test names it
restatements_table
function, line 205
One row per restated window and measure (rows included): what finance asks when last month's number moved.build · obs/ops.py:309test_ops_restatements.py
step_durations_table
function, line 220
Time per step: from the dataset run's start (or the step before) to the step's record.build · obs/ops.py:306no test names it
ingestion_tables
function, line 234
build · obs/ops.py:296no test names it
freshness_tables
function, line 255
build · obs/ops.py:297no test names it
usage_daily
function, line 269
build · obs/ops.py:312no test names it
lineage_table
function, line 281
build · obs/ops.py:313no test names it
alert_tables
function, line 286
build · obs/ops.py:298no test names it
build
function, line 293
none found in the codetest_semantic_governance.py, conftest.py and 17 more
_maker
function, line 322
run_signals · obs/ops.py:337
freshness_signals · obs/ops.py:355
no test names it
_maker.make
function, line 325
run_signals · obs/ops.py:341
freshness_signals · obs/ops.py:356
no test names it
run_signals
function, line 333
A finished run's signals (pushed as it ends): every dataset it ran, failing or not, so recoveries resolve. Detectors and checks come only from the step rows (the one step format); details stay with the owner, and the alert carries the ownernone found in the codeno test names it
freshness_signals
function, line 354
none found in the codeno test names it

obs/ops_export.py

The ops tables as a typed, portable file, so a realm can publish them where its dashboards read (B13). Operations

DefinitionWhat it does (its own first sentence)Called byNamed in tests
type_of
function, line 20
column_types · obs/ops_export.py:42no test names it
column_types
function, line 34
Columns in first-seen order, each typed by its first non-empty value; integer widens to decimal if mixed.export · obs/ops_export.py:66test_obs_dashboards.py, test_ops_export.py
_carry
function, line 47
export · obs/ops_export.py:68no test names it
export
function, line 59
Declared columns win (so an empty table keeps its shape); a column the rows have beyond them is added, typed by its values.module level · realms/databricks/engine/tasks/run_days.py:91
write_ops_page · scripts/run_platform.py:150
main · scripts/run_platform.py:737
test_meta_loader.py, test_obs_trace.py and 1 more
_back
function, line 72
restore · obs/ops_export.py:84no test names it
restore
function, line 82
publish · adapters/spark_ops_tables.py:33test_ops_export.py

obs/page.py

The local ops page (B10): one self-contained HTML file written after a local run, so a person can see what happened without a workspace. It reads only the ops tables (obs/ops.py, the same rows the ops metric views and dashboards read) plus Operations

DefinitionWhat it does (its own first sentence)Called byNamed in tests
_cls
function, line 31
_cell · obs/page.py:49no test names it
_cell
function, line 43
table · obs/page.py:60no test names it
table
function, line 53
Rows as an HTML table; columns default to every key of the first row. More than `limit` rows are cut, and the cut is said.render · obs/page.py:103test_lakeformation.py, test_semantic_governance.py and 84 more
_tile
function, line 65
render · obs/page.py:88no test names it
_sum
function, line 69
render · obs/page.py:91no test names it
render
function, line 73
The page. `layer_checks`: {day, layer, equals_oracle}; `incidents`: {day, layer, incident, detector, dataset, verdict, evidence}; `hops`: {dataset, hop, n, p50, p95, p99, basis}.write_ops_page · scripts/run_platform.py:141test_semantic_governance.py, test_obs_dashboards.py and 3 more

obs/pipeline_events.py

Declarative pipelines report themselves (ADR-024 rule 5): the pipeline event log, behind one read-only port. Operations

DefinitionWhat it does (its own first sentence)Called byNamed in tests
TableUpdateRow
class, line 24
DatabricksPipelineEvents.table_updates · adapters/databricks_pipeline_events.py:62test_obs_pipeline_events.py
PipelineEvents
class, line 35
none found in the codeno test names it
PipelineEvents.table_updates
method, line 36
Every table a pipeline update refreshed that started in [since, until), oldest first.Monitor.tables · obs/monitor.py:121 (by name)test_obs_pipeline_events.py

obs/recorder.py

The run recorder: one id from start to finish (G8, OPS-03, DPC-0057). Operations

DefinitionWhat it does (its own first sentence)Called byNamed in tests
RecorderError
class, line 40
The recorder was used out of order (a dataset finished twice, a step after the dataset closed).DatasetHandle._open · obs/recorder.py:69test_obs_recorder.py
StepHandle
class, line 45
DatasetHandle.step · obs/recorder.py:78no test names it
StepHandle.warn
method, line 50
FixtureRunner._effect · obs/fixture.py:186 (by name)no test names it
StepHandle.skip
method, line 53
FixtureRunner._dataset · obs/fixture.py:165 (by name)no test names it
DatasetHandle
class, line 57
RunHandle.dataset · obs/recorder.py:149no test names it
DatasetHandle.__init__
method, line 58
none found in the codeno test names it
DatasetHandle.store
method, line 64
StoreGoldTables.__init__ · adapters/spark_gold.py:412 (value)
StoreGoldReader.__init__ · adapters/spark_gold.py:475 (value)
StoreSilverTables.__init__ · adapters/spark_silver.py:286 (value)
StoreBatchRegistry.__init__ · adapters/store_registry.py:18 (value)
_inputs · gold/harness.py:247 (value)
FixtureRunner.__init__ · obs/fixture.py:93 (value)
and 4 more
gold_helpers.py, silver_helpers.py and 33 more
DatasetHandle._open
method, line 67
DatasetHandle._log · obs/recorder.py:72
DatasetHandle.error · obs/recorder.py:106
DatasetHandle.file · obs/recorder.py:111
DatasetHandle._finish · obs/recorder.py:117
no test names it
DatasetHandle._log
method, line 71
DatasetHandle.step · obs/recorder.py:82
DatasetHandle.check · obs/recorder.py:95
DatasetHandle.detect · obs/recorder.py:100
no test names it
DatasetHandle.step
method, line 77
MemoryMetaStore.log_step · adapters/memory_meta.py:198 (value)
PostgresMetaStore.log_step · adapters/postgres_meta.py:158 (value)
SparkRunLog.write · adapters/spark_delta.py:117 (value)
steps_table · obs/ops.py:188 (value)
step_durations_table · obs/ops.py:230 (value)
run_signals · obs/ops.py:343 (value)
and 4 more
no test names it
DatasetHandle.check
method, line 89
A quality rule, a reconciliation entry or a profile: its verdict as a step of this dataset run.MemoryMetaStore.set_dq_rules · adapters/memory_meta.py:359 (value)
PostgresMetaStore.set_dq_rules · adapters/postgres_meta.py:241 (value)
SparkChecks.write · adapters/spark_delta.py:128 (value)
FixtureRunner._dataset · obs/fixture.py:159 (by name)
Monitor._lateness · obs/monitor.py:88 (by name)
test_azure_adapters.py, test_contract_rules.py and 30 more
DatasetHandle.detect
method, line 97
A detector fired (DAC id, e.g. DEL-01). Handled is warned; blocked (the load stopped) is failed.FixtureRunner._effect · obs/fixture.py:179 (by name)silver_helpers.py, test_baseline_stage.py and 10 more
DatasetHandle.error
method, line 104
An error the step handled (a contract mismatch held back, a retry that later passed): logged, not raised.main · scripts/run_hop.py:108 (by name)
main · scripts/run_platform.py:372 (by name)
main · scripts/run_world.py:161 (by name)
FixtureRunner._effect · obs/fixture.py:177 (by name)
test_bundle.py, test_azure_adapters.py and 34 more
DatasetHandle.file
method, line 109
cmd_code · scripts/explain.py:107 (value)
FixtureRunner._dataset · obs/fixture.py:152 (by name)
FixtureRunner._effect · obs/fixture.py:174 (by name)
no test names it
DatasetHandle._finish
method, line 115
DatasetHandle.succeed · obs/recorder.py:126
DatasetHandle.fail · obs/recorder.py:129
DatasetHandle.skip · obs/recorder.py:132
no test names it
DatasetHandle.succeed
method, line 124
FixtureRunner._rerun · obs/fixture.py:139 (by name)
FixtureRunner._dataset · obs/fixture.py:167 (by name)
RunHandle.dataset · obs/recorder.py:162 (by name)
test_end_to_end.py, test_ingest_world.py and 2 more
DatasetHandle.fail
method, line 128
RunHandle.dataset · obs/recorder.py:157 (by name)
RunRecorder.run · obs/recorder.py:200 (by name)
no test names it
DatasetHandle.skip
method, line 131
FixtureRunner._dataset · obs/fixture.py:165 (by name)no test names it
RunHandle
class, line 135
RunRecorder.run · obs/recorder.py:192no test names it
RunHandle.__init__
method, line 136
none found in the codeno test names it
RunHandle.dataset
method, line 144
main · realms/aws-databricks/meta_setup.py:88 (value)
module level · realms/databricks/declarative/silver.py:77 (value)
lakebase · realms/databricks/engine/tasks/probe.py:83 (value)
hop_delays · scripts/run_platform.py:161 (value)
main.housekeeping · scripts/run_platform.py:513 (value)
AzureLanding.read_bytes · adapters/azure_blob.py:102 (value)
and 70 more
test_bundle.py, gold_helpers.py and 65 more
RunHandle.job_tags
method, line 164
Tags for the job and compute this run uses: domain, data product and run id. Spend itself is the separate cost product's (ADR-024 rule 8); these tags are what lets it attribute spend to a run and a product.FixtureRunner._system · obs/fixture.py:208 (by name)test_obs_recorder.py
RunHandle.lineage_run_id
method, line 170
The run id as lineage events carry it (UUID form of the same 128 bits).none found in the codetest_obs_recorder.py
RunHandle.log_fields
method, line 174
Fields for every structured log line written during the run.none found in the codetest_obs_recorder.py
RunRecorder
class, line 179
FixtureRunner.__init__ · obs/fixture.py:97test_obs_recorder.py, test_obs_runlog.py and 1 more
RunRecorder.__init__
method, line 180
none found in the codeno test names it
RunRecorder._finished
method, line 183
RunRecorder.run · obs/recorder.py:204no test names it
RunRecorder.run
method, line 188
none found in the codeno test names it

obs/runlog.py

The run log, read back for operations (DPC-0057): every run in a time window with its timestamps. Operations

DefinitionWhat it does (its own first sentence)Called byNamed in tests
RunRow
class, line 21
StampingRunLog._add · adapters/memory_runlog.py:184
parse · adapters/postgres_runlog.py:41
no test names it
DatasetRunRow
class, line 33
StampingRunLog._add · adapters/memory_runlog.py:190
parse · adapters/postgres_runlog.py:43
test_obs_freshness.py
StepRow
class, line 48
StampingRunLog._add · adapters/memory_runlog.py:194
parse · adapters/postgres_runlog.py:46
test_ops_restatements.py
FileRow
class, line 60
StampingRunLog._add · adapters/memory_runlog.py:197
parse · adapters/postgres_runlog.py:48
no test names it
ErrorRow
class, line 74
StampingRunLog._add · adapters/memory_runlog.py:200
parse · adapters/postgres_runlog.py:50
no test names it
RunLogSnapshot
class, line 85
StampingRunLog.snapshot · adapters/memory_runlog.py:170
NullRunLog.snapshot · adapters/null_runlog.py:16
parse · adapters/postgres_runlog.py:40
_only · obs/monitor.py:56
merge · obs/runlog.py:100
test_obs_pipeline_events.py, test_ops_restatements.py
RunLogReader
class, line 93
none found in the codeno test names it
RunLogReader.snapshot
method, line 94
Runs of the domain that started in [since, until), with everything recorded for them, in record order.Monitor.run_finished · obs/monitor.py:99 (by name)
Monitor.check_lateness · obs/monitor.py:108 (by name)
Monitor.tables · obs/monitor.py:120 (by name)
test_meta_loader.py, test_obs_recorder.py and 10 more
merge
function, line 99
none found in the codetest_baseline_stage.py, test_meta_store.py and 7 more

obs/steps.py

The one step format (ADR-024 rule 2): every owner writes one result row per check to the run log. Operations

DefinitionWhat it does (its own first sentence)Called byNamed in tests
kind_of
function, line 33
The owner kind of a step name, or None for a pipeline step (land, bronze, silver.<stage>, ingest.*).check_kind · obs/ops.py:98
steps_table · obs/ops.py:188
step_durations_table · obs/ops.py:230
run_signals · obs/ops.py:343
owner_of · obs/steps.py:40
test_gold_harness.py, test_hop_reports.py and 4 more
owner_of
function, line 39
steps_table · obs/ops.py:190
run_signals · obs/ops.py:344
test_obs_steps.py
severity
function, line 44
steps_table · obs/ops.py:193test_contract_rules.py, test_explain.py and 17 more
violations
function, line 48
What breaks the one step format in this row (empty when it keeps it).none found in the codetest_gold_harness.py, test_obs_steps.py and 3 more

obs/success.py

The platform's success criteria and the scorecard a run computes against them (B20; CCA-P D4 4.1: define what good means first, one metric per dimension with a target, no blended score). Pure. Operations

DefinitionWhat it does (its own first sentence)Called byNamed in tests
Criterion
class, line 28
module level · obs/success.py:36no test names it
Evidence
class, line 60
What one run saw. Empty fields mean the run did not look, never that all was well.success_scorecard · scripts/run_platform.py:173
module level · obs/success.py:148 (value)
test_obs_success.py
Score
class, line 74
scorecard · obs/success.py:157no test names it
_pct
function, line 84
_all · obs/success.py:91no test names it
_all
function, line 88
_c1 · obs/success.py:97
_c2 · obs/success.py:101
_c3 · obs/success.py:105
_c4 · obs/success.py:109
_c5 · obs/success.py:118
no test names it
_c1
function, line 94
module level · obs/success.py:149 (value)no test names it
_c2
function, line 100
module level · obs/success.py:149 (value)no test names it
_c3
function, line 104
module level · obs/success.py:149 (value)no test names it
_c4
function, line 108
module level · obs/success.py:149 (value)no test names it
_c5
function, line 112
module level · obs/success.py:149 (value)no test names it
_c8
function, line 122
module level · obs/success.py:149 (value)no test names it
_c9
function, line 133
module level · obs/success.py:149 (value)no test names it
_c15
function, line 140
module level · obs/success.py:149 (value)no test names it
scorecard
function, line 152
success_scorecard · scripts/run_platform.py:192test_obs_success.py
rows
function, line 165
The scorecard as ops table rows (ops_success), for the dashboards.none found in the codeno test names it

obs/system.py

The platform's own records, behind one port: job runs, lineage, and access (ADR-024: platform telemetry adopted). Operations

DefinitionWhat it does (its own first sentence)Called byNamed in tests
JobRunRow
class, line 26
DatabricksSystemTables.jobs · adapters/databricks_system.py:81
FixtureRunner._system · obs/fixture.py:209
test_obs_dashboards.py, test_obs_system.py
LineageRow
class, line 37
DatabricksSystemTables.lineage · adapters/databricks_system.py:95
FixtureRunner._system · obs/fixture.py:213
test_obs_dashboards.py, test_obs_system.py
AccessRow
class, line 46
DatabricksSystemTables.access · adapters/databricks_system.py:102
FixtureRunner._readers · obs/fixture.py:220
test_obs_dashboards.py, test_obs_system.py
SystemSnapshot
class, line 54
read_all · obs/system.py:69
DailyCopy.__init__ · obs/system.py:86
DailyCopy.copy · obs/system.py:93
DailyCopy.on_demand · obs/system.py:100
test_obs_pipeline_events.py
SystemTables
class, line 60
none found in the codeno test names it
SystemTables.jobs
method, line 61
read_all · obs/system.py:69 (by name)no test names it
SystemTables.lineage
method, line 63
read_all · obs/system.py:69 (by name)test_obs_dashboards.py, test_obs_lineage.py and 10 more
SystemTables.access
method, line 65
generate · realms/databricks/semantic_governance/generate.py:173 (value)
read_all · obs/system.py:69 (by name)
test_lakeformation.py, test_semantic_governance.py and 5 more
read_all
function, line 68
DailyCopy.copy · obs/system.py:92
DailyCopy.on_demand · obs/system.py:100
test_obs_system.py
_business_day
function, line 72
DailyCopy.copy · obs/system.py:89no test names it
DailyCopy
class, line 76
The telemetry, copied once per business day from a watermark (ADR-024 rule 4).Monitor.__init__ · obs/monitor.py:69test_obs_system.py
DailyCopy.__init__
method, line 83
none found in the codeno test names it
DailyCopy.copy
method, line 88
none found in the codeno test names it
DailyCopy.on_demand
method, line 98
The copy plus what is newer than the watermark, read now for the one who asked (costs a query).none found in the codetest_obs_system.py

obs/trace.py

The telemetry port: one id for a run, everywhere (G8). The run id IS the trace id. Operations

DefinitionWhat it does (its own first sentence)Called byNamed in tests
Span
class, line 25
TracedStore.__init__ · obs/traced_store.py:26 (value)no test names it
Span.trace_id
method, line 27
OtelSpan.child · adapters/otel_tracer.py:42 (value)
RunHandle.__init__ · obs/recorder.py:138 (value)
RunRecorder.run · obs/recorder.py:191 (value)
LocalSpan.child · obs/trace.py:82 (value)
test_obs_recorder.py, test_obs_trace.py and 1 more
Span.span_id
method, line 30
RunHandle.log_fields · obs/recorder.py:176 (value)
LocalSpan.child · obs/trace.py:82 (value)
test_obs_trace.py
Span.child
method, line 32
DatasetHandle.step · obs/recorder.py:78 (by name)
RunHandle.dataset · obs/recorder.py:146 (by name)
TracedStore.start_dataset_run · obs/traced_store.py:42 (by name)
TracedStore.log_step · obs/traced_store.py:49 (by name)
test_obs_trace.py, test_mimic_runner.py and 1 more
Span.set
method, line 34
none found in the codeno test names it
Span.end
method, line 36
OtelSpan.end · adapters/otel_tracer.py:58 (by name)
_complete · gold/harness.py:279 (by name)
_fact · gold/harness.py:329 (by name)
DatasetHandle.step · obs/recorder.py:83 (by name)
DatasetHandle._finish · obs/recorder.py:122 (by name)
RunRecorder.run · obs/recorder.py:203 (by name)
and 3 more
no test names it
Tracer
class, line 39
none found in the codeno test names it
Tracer.start
method, line 40
A new root span: a new trace, so a new run id.none found in the codesilver_helpers.py, test_gold_calendar.py and 15 more
valid_trace_id
function, line 45
new_trace_id · obs/trace.py:52
as_uuid · obs/trace.py:65
test_obs_trace.py
new_trace_id
function, line 49
OtelTracer.start · adapters/otel_tracer.py:75
LocalTracer.start · obs/trace.py:101
test_obs_trace.py
new_span_id
function, line 56
OtelSpan.child · adapters/otel_tracer.py:43
OtelTracer.start · adapters/otel_tracer.py:75
LocalSpan.child · obs/trace.py:82
LocalTracer.start · obs/trace.py:101
no test names it
as_uuid
function, line 63
The run id in the UUID form lineage events want for their run id (same 128 bits).RunHandle.lineage_run_id · obs/recorder.py:172test_obs_recorder.py, test_obs_trace.py
LocalSpan
class, line 71
LocalSpan.child · obs/trace.py:82
LocalTracer.start · obs/trace.py:101
no test names it
LocalSpan.child
method, line 81
DatasetHandle.step · obs/recorder.py:78 (by name)
RunHandle.dataset · obs/recorder.py:146 (by name)
TracedStore.start_dataset_run · obs/traced_store.py:42 (by name)
TracedStore.log_step · obs/traced_store.py:49 (by name)
test_obs_trace.py, test_mimic_runner.py and 1 more
LocalSpan.set
method, line 84
none found in the codeno test names it
LocalSpan.end
method, line 87
OtelSpan.end · adapters/otel_tracer.py:58 (by name)
_complete · gold/harness.py:279 (by name)
_fact · gold/harness.py:329 (by name)
DatasetHandle.step · obs/recorder.py:83 (by name)
DatasetHandle._finish · obs/recorder.py:122 (by name)
RunRecorder.run · obs/recorder.py:203 (by name)
and 3 more
no test names it
LocalTracer
class, line 96
main · scripts/run_platform.py:393
Harness.build · twin/runner.py:97
test_obs_recorder.py, test_obs_runlog.py and 3 more
LocalTracer.__init__
method, line 97
none found in the codeno test names it
LocalTracer.start
method, line 100
none found in the codesilver_helpers.py, test_gold_calendar.py and 15 more

obs/traced_store.py

Live wiring of observability (B10, ADR-024): a metadata store that traces every run as it is recorded and pushes "run finished" to its hooks. The engine runners already record through the store port; wrapping the store wires them to the tra Operations

DefinitionWhat it does (its own first sentence)Called byNamed in tests
TracedStore
class, line 23
main · scripts/run_platform.py:410test_obs_traced_store.py
TracedStore.__init__
method, line 24
none found in the codeno test names it
TracedStore.__getattr__
method, line 29
none found in the codeno test names it
TracedStore.start_run
method, line 32
StampingRunLog.start_run · adapters/memory_runlog.py:77 (by name)
run_gold · gold/harness.py:132 (by name)
run_housekeeping · housekeeping.py:127 (by name)
ingest · ingest/runner.py:126 (by name)
RunRecorder.run · obs/recorder.py:191 (by name)
build_silver · silver/runner.py:85 (by name)
gold_helpers.py, test_housekeeping.py and 7 more
TracedStore.start_dataset_run
method, line 37
StampingRunLog.start_dataset_run · adapters/memory_runlog.py:83 (by name)
run_gold · gold/harness.py:139 (by name)
_sweep · housekeeping.py:146 (by name)
_one · ingest/runner.py:149 (by name)
RunHandle.dataset · obs/recorder.py:148 (by name)
_one · silver/runner.py:123 (by name)
and 1 more
gold_helpers.py, test_housekeeping.py and 7 more
TracedStore.log_step
method, line 45
StampingRunLog.log_step · adapters/memory_runlog.py:88 (by name)
_step · gold/harness.py:168 (by name)
_sweep_leased · housekeeping.py:167 (by name)
_profile_step · ingest/runner.py:72 (by name)
_detect_steps · ingest/runner.py:109 (by name)
_one_leased · ingest/runner.py:196 (by name)
and 4 more
test_meta_store.py, test_obs_traced_store.py
TracedStore.finish_dataset_run
method, line 53
StampingRunLog.finish_dataset_run · adapters/memory_runlog.py:106 (by name)
_fail · gold/harness.py:174 (by name)
_dimension · gold/harness.py:189 (by name)
_fact · gold/harness.py:342 (by name)
_sweep_leased · housekeeping.py:177 (by name)
_one_leased · ingest/runner.py:176 (by name)
and 4 more
gold_helpers.py, test_housekeeping.py and 5 more
TracedStore.finish_run
method, line 59
StampingRunLog.finish_run · adapters/memory_runlog.py:110 (by name)
run_gold · gold/harness.py:156 (by name)
run_housekeeping · housekeeping.py:134 (by name)
ingest · ingest/runner.py:135 (by name)
RunRecorder.run · obs/recorder.py:201 (by name)
build_silver · silver/runner.py:97 (by name)
gold_helpers.py, test_housekeeping.py and 4 more

obs/views.py

Certified ops definitions (ADR-024 rule 7): ops metric views over the compact ops tables. Product-free. Operations

DefinitionWhat it does (its own first sentence)Called byNamed in tests
ViewMeasure
class, line 18
_vm · obs/views.py:38
module level · obs/views.py:31 (value)
no test names it
OpsView
class, line 26
module level · obs/views.py:42no test names it
OpsView.measure
method, line 33
known_answers_sql · realms/databricks/semantic_governance/render.py:144 (value)
_tile_query · realms/databricks/semantic_governance/render.py:167 (value)
_widget · realms/databricks/semantic_governance/render.py:180 (value)
dashboard_json · realms/databricks/semantic_governance/render.py:215 (value)
oracle_value · semantic/metrics.py:280 (value)
_explain · twin/gate.py:133 (value)
and 2 more
test_semantic_governance.py, test_end_to_end.py and 7 more
_vm
function, line 37
module level · obs/views.py:44no test names it
view_for
function, line 87
query_sql · adapters/lakeview.py:53
problem · obs/views.py:94
no test names it
problem
function, line 91
Why an aggregate query cannot be answered by the certified view over its table (None when it can).none found in the codetest_gold_plan.py, test_meta_loader.py and 1 more
lint
function, line 107
Definition errors in VIEWS (an unknown table, a repeated name, a view per table at most).none found in the codeno test names it