| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|
MetaError class, line 34 | Base for every refusal by the metadata store. | none found in the code | no test names it |
UnknownSetting class, line 38 | A setting key not in the registry for its scope (DPC-0034). | MemoryMetaStore.load_domain · adapters/memory_meta.py:85
MemoryMetaStore.load_dataset · adapters/memory_meta.py:93
module level · adapters/postgres_meta.py:40 (value) | test_meta_store.py |
NotFound class, line 42 | A dataset, run or dataset run the store does not have. | MemoryMetaStore.load_dataset · adapters/memory_meta.py:90
MemoryMetaStore._get · adapters/memory_meta.py:106
MemoryMetaStore._run · adapters/memory_meta.py:160
MemoryMetaStore.start_run · adapters/memory_meta.py:174
module level · adapters/postgres_meta.py:40 (value) | test_lakebase.py, test_meta_store.py |
InvalidArgument class, line 46 | A value outside its allowed set (status, run type, reason too short, ...). | MemoryMetaStore.register_settings · adapters/memory_meta.py:76
MemoryMetaStore.load_dataset · adapters/memory_meta.py:97
MemoryMetaStore.business_keys · adapters/memory_meta.py:138
MemoryMetaStore.start_run · adapters/memory_meta.py:172
MemoryMetaStore.log_step · adapters/memory_meta.py:196
MemoryMetaStore.log_file · adapters/memory_meta.py:203 and 8 more | test_meta_store.py |
RunStateError class, line 50 | A run step out of order: finishing a run never started, starting one twice, finishing with open datasets. | MemoryMetaStore._ds_run · adapters/memory_meta.py:168
MemoryMetaStore.start_run · adapters/memory_meta.py:176
MemoryMetaStore.start_dataset_run · adapters/memory_meta.py:183
MemoryMetaStore.void_batch · adapters/memory_meta.py:236
MemoryMetaStore.finish_dataset_run · adapters/memory_meta.py:254
MemoryMetaStore.finish_run · adapters/memory_meta.py:265 and 2 more | test_meta_store.py |
SettingDef class, line 55 | | PostgresMetaStore.settings_registry · adapters/postgres_meta.py:126
load · meta/registry.py:19
MemoryMetaStore.__init__ · adapters/memory_meta.py:51 (value)
by_key · meta/registry.py:24 (value) | test_ingest_steps.py, test_meta_store.py and 2 more |
Setting class, line 66 | | MemoryMetaStore.settings · adapters/memory_meta.py:124
PostgresMetaStore.settings · adapters/postgres_meta.py:134 | no test names it |
Column class, line 73 | | MemoryMetaStore.columns · adapters/memory_meta.py:132
PostgresMetaStore.columns · adapters/postgres_meta.py:137
Spec.from_view · ingest/steps.py:91
module level · ingest/steps.py:67 (value) | test_ingest_steps.py |
DatasetRef class, line 83 | | MemoryMetaStore.datasets · adapters/memory_meta.py:112
PostgresMetaStore.datasets · adapters/postgres_meta.py:131 | no test names it |
StepRecord class, line 94 | | _step · gold/harness.py:168
_sweep_leased · housekeeping.py:167
_profile_step · ingest/runner.py:72
_detect_steps · ingest/runner.py:109
_one_leased · ingest/runner.py:197
busy · leases.py:23 and 3 more | test_meta_store.py, test_obs_steps.py and 1 more |
FileRecord class, line 105 | | MemoryMetaStore.dataset_files · adapters/memory_meta.py:305
PostgresMetaStore.dataset_files · adapters/postgres_meta.py:200
StoreBatchRegistry.record_batch · adapters/store_registry.py:29
_record · ingest/steps.py:382
DatasetHandle.file · obs/recorder.py:112
SilverBaseliner.write · silver/baseline_writer.py:75 and 1 more | gold_helpers.py, test_housekeeping.py and 4 more |
BackfillPlan class, line 121 | A plan that activates corrections on a date and names the gold windows to rebuild (ADR-017, DPC-0082). | PostgresMetaStore.backfill_plans · adapters/postgres_meta.py:218
withdraw · recovery.py:21
ClinicalTrial._correct · world/domains/clinical.py:526
TyreCompany._correct · world/domains/tyre.py:516
MemoryMetaStore.__init__ · adapters/memory_meta.py:63 (value)
ClinicalTrial.__init__ · world/domains/clinical.py:147 (value) and 2 more | test_gold_harness.py, test_meta_store.py |
Correction class, line 134 | One corrected value (DPC-0082: entity, field, value, applies from and to, who, why, status). The entity is the plan's dataset; who and why are the plan's. `key` holds the business key columns and their values. | MemoryMetaStore.record_corrections · adapters/memory_meta.py:327
MemoryMetaStore.corrections · adapters/memory_meta.py:338
PostgresMetaStore.corrections · adapters/postgres_meta.py:213
ClinicalTrial._correct · world/domains/clinical.py:522
TyreCompany._correct · world/domains/tyre.py:507 | test_gold_engine.py, test_gold_harness.py and 1 more |
DqRule class, line 154 | One quality rule of a dataset (DPC-0010, ADR-009: severity sets the action). `column` is empty for a rule on the whole row or batch; `check` names the check (not_null, range, allowed, unique, freshness, ...). | MemoryMetaStore.set_dq_rules · adapters/memory_meta.py:359
PostgresMetaStore.dq_rules · adapters/postgres_meta.py:246
dq_rules_of · meta/loader.py:77 | test_contract_rules.py, test_meta_store.py and 8 more |
DqResult class, line 168 | What one rule found in one dataset run: rows checked, rows failed, the action taken (DPC-0042). | MemoryMetaStore.log_dq_results · adapters/memory_meta.py:375
PostgresMetaStore.dq_results · adapters/postgres_meta.py:255 | test_meta_store.py |
MetadataStore class, line 181 | | none found in the code | no test names it |
MetadataStore.register_settings method, line 183 | | StampingRunLog.register_settings · adapters/memory_runlog.py:48 (by name)
load · meta/loader.py:174 (by name) | test_housekeeping.py, test_ingest_pipelines.py and 7 more |
MetadataStore.load_domain method, line 185 | doc: domain, business_timezone, environments, settings {key: value}, manifest, fingerprint. Domain settings are defaults for dataset keys (or platform keys); unknown keys are refused. | StampingRunLog.load_domain · adapters/memory_runlog.py:51 (by name)
load · meta/loader.py:175 (by name) | gold_helpers.py, test_housekeeping.py and 10 more |
MetadataStore.load_dataset method, line 189 | doc: domain, source_name, source_env, dataset_name, is_enabled, kind, archetypes, ingest_mode, ingest_read, history_kind, load_pattern, contract_id, contract_version, fingerprint, settings {key: value}, columns [Column fields], business_key | StampingRunLog.load_dataset · adapters/memory_runlog.py:54 (by name)
load · meta/loader.py:179 (by name)
load_gold · meta/loader.py:235 (by name) | gold_helpers.py, test_housekeeping.py and 10 more |
MetadataStore.settings_registry method, line 195 | | StampingRunLog.settings_registry · adapters/memory_runlog.py:57 (by name) | test_meta_store.py |
MetadataStore.datasets method, line 197 | | Monitor._lateness · obs/monitor.py:88 (value)
main · realms/aws-databricks/meta_setup.py:85 (by name)
module level · realms/databricks/declarative/silver.py:73 (by name)
lakebase · realms/databricks/engine/tasks/probe.py:80 (by name)
hop_delays · scripts/run_platform.py:160 (by name)
StampingRunLog.datasets · adapters/memory_runlog.py:60 (by name) and 10 more | test_bundle.py, test_semantic_governance.py and 29 more |
MetadataStore.settings method, line 199 | Every registered dataset key, merged: dataset over domain over the registry default. (DPC-0002 also names a source level; DPM has no source manifest yet, so nothing feeds it.) | _read · ingest/pipelines.py:67 (value)
push · ingest/pipelines.py:147 (value)
enter · ingest/steps.py:328 (value)
main · realms/aws-databricks/meta_setup.py:88 (by name)
StampingRunLog.settings · adapters/memory_runlog.py:63 (by name) | test_bundle.py, gold_helpers.py and 20 more |
MetadataStore.columns method, line 203 | | _ingest_table_to_bronze · scripts/demo_end_to_end.py:389 (value)
CustomerRunLog.__init__ · adapters/customer_runlog.py:34 (value)
_widget · adapters/lakeview.py:103 (value)
UcIcebergStore._create · adapters/table_store.py:403 (value)
validate · obs/dashboards.py:105 (value)
contract_check · stages/clean.py:44 (value) and 8 more | test_lakeformation.py, test_semantic_governance.py and 67 more |
MetadataStore.business_keys method, line 205 | | StampingRunLog.business_keys · adapters/memory_runlog.py:69 (by name)
plans · gold/harness.py:96 (by name)
_inputs · gold/harness.py:245 (by name) | gold_helpers.py, silver_helpers.py and 26 more |
MetadataStore.dataset_view method, line 207 | The merged configuration for one dataset with its fingerprint (port operation dataset_view). | module level · realms/databricks/declarative/silver.py:77 (by name)
StampingRunLog.dataset_view · adapters/memory_runlog.py:72 (by name)
StoreBatchRegistry.dataset_config · adapters/store_registry.py:21 (by name)
plans · gold/harness.py:94 (by name)
run_gold · gold/harness.py:138 (by name)
_sweep · housekeeping.py:142 (by name) and 7 more | test_ingest_pipelines.py, test_meta_loader.py and 4 more |
MetadataStore.start_run method, line 211 | | StampingRunLog.start_run · adapters/memory_runlog.py:77 (by name)
run_gold · gold/harness.py:132 (by name)
run_housekeeping · housekeeping.py:127 (by name)
ingest · ingest/runner.py:126 (by name)
RunRecorder.run · obs/recorder.py:191 (by name)
TracedStore.start_run · obs/traced_store.py:33 (by name) and 1 more | gold_helpers.py, test_housekeeping.py and 7 more |
MetadataStore.start_dataset_run method, line 214 | Open the dataset's run record with its config snapshot; returns the watermark before the run. | StampingRunLog.start_dataset_run · adapters/memory_runlog.py:83 (by name)
run_gold · gold/harness.py:139 (by name)
_sweep · housekeeping.py:146 (by name)
_one · ingest/runner.py:149 (by name)
RunHandle.dataset · obs/recorder.py:148 (by name)
TracedStore.start_dataset_run · obs/traced_store.py:39 (by name) and 2 more | gold_helpers.py, test_housekeeping.py and 7 more |
MetadataStore.log_step method, line 218 | | StampingRunLog.log_step · adapters/memory_runlog.py:88 (by name)
_step · gold/harness.py:168 (by name)
_sweep_leased · housekeeping.py:167 (by name)
_profile_step · ingest/runner.py:72 (by name)
_detect_steps · ingest/runner.py:109 (by name)
_one_leased · ingest/runner.py:196 (by name) and 5 more | test_meta_store.py, test_obs_traced_store.py |
MetadataStore.log_file method, line 220 | | StampingRunLog.log_file · adapters/memory_runlog.py:92 (by name)
StoreBatchRegistry.record_batch · adapters/store_registry.py:28 (by name)
_record · ingest/steps.py:381 (by name)
DatasetHandle.file · obs/recorder.py:112 (by name)
SilverBaseliner.write · silver/baseline_writer.py:75 (by name) | gold_helpers.py, test_housekeeping.py and 4 more |
MetadataStore.batch_status method, line 222 | The latest status recorded for a batch key of this dataset, or None if never seen. | StampingRunLog.batch_status · adapters/memory_runlog.py:96 (by name)
StoreBatchRegistry.batch_status · adapters/store_registry.py:24 (by name)
enter · ingest/steps.py:318 (by name)
land · lander.py:34 (by name) | test_meta_store.py, test_slice1_local.py |
MetadataStore.log_error method, line 225 | Message is redacted and capped by the store, whoever calls. | StampingRunLog.log_error · adapters/memory_runlog.py:100 (by name)
_fail · gold/harness.py:173 (by name)
_one_leased · ingest/runner.py:175 (by name)
DatasetHandle.error · obs/recorder.py:107 (by name)
RunHandle.dataset · obs/recorder.py:154 (by name)
RunRecorder.run · obs/recorder.py:197 (by name) and 1 more | test_meta_store.py |
MetadataStore.finish_dataset_run method, line 229 | Close the dataset run. Watermark and bookmarks move only when status is succeeded, in one transaction. | StampingRunLog.finish_dataset_run · adapters/memory_runlog.py:106 (by name)
_fail · gold/harness.py:174 (by name)
_dimension · gold/harness.py:189 (by name)
_fact · gold/harness.py:342 (by name)
_sweep_leased · housekeeping.py:177 (by name)
_one_leased · ingest/runner.py:176 (by name) and 5 more | gold_helpers.py, test_housekeeping.py and 5 more |
MetadataStore.finish_run method, line 234 | Close the run; its status follows its dataset runs. Returns the status. | StampingRunLog.finish_run · adapters/memory_runlog.py:110 (by name)
run_gold · gold/harness.py:156 (by name)
run_housekeeping · housekeeping.py:134 (by name)
ingest · ingest/runner.py:135 (by name)
RunRecorder.run · obs/recorder.py:201 (by name)
TracedStore.finish_run · obs/traced_store.py:60 (by name) and 1 more | gold_helpers.py, test_housekeeping.py and 4 more |
MetadataStore.watermark method, line 237 | | push · ingest/pipelines.py:161 (value)
pull_pages · ingest/pipelines.py:302 (value)
stream_events · ingest/pipelines.py:357 (value)
DailyCopy.__init__ · obs/system.py:84 (value)
DailyCopy.copy · obs/system.py:92 (value)
DailyCopy.on_demand · obs/system.py:100 (value) and 2 more | gold_helpers.py, test_ingest_pipelines.py and 13 more |
MetadataStore.bookmarks method, line 239 | | StampingRunLog.bookmarks · adapters/memory_runlog.py:118 (by name)
silver_watermark · gold/harness.py:106 (by name)
feed_marks · gold/harness.py:122 (by name)
_fact · gold/harness.py:289 (by name)
push · ingest/pipelines.py:156 (by name)
_feed_bookmarks · ingest/pipelines.py:196 (by name) and 6 more | gold_helpers.py, test_gold_harness.py and 6 more |
MetadataStore.reset_watermark method, line 241 | An operator reset (backfill, correction): audited, with a reason. | StampingRunLog.reset_watermark · adapters/memory_runlog.py:122 (by name) | test_meta_store.py |
MetadataStore.run_report method, line 244 | Everything recorded for one run: run, datasets, steps, files, errors. | success_scorecard · scripts/run_platform.py:184 (by name)
StampingRunLog.run_report · adapters/memory_runlog.py:125 (by name)
StampingRunLog.snapshot · adapters/memory_runlog.py:175 (by name)
_steps · gold/inspect.py:55 (by name) | test_ingest_pipelines.py, test_meta_store.py and 5 more |
MetadataStore.dataset_files method, line 247 | Every batch recorded for a dataset (optionally one window), oldest first: what the lander has handled, and where a reload finds the batches of a window again. | StampingRunLog.dataset_files · adapters/memory_runlog.py:128 (by name)
silver_watermark · gold/harness.py:107 (by name)
_sweep_leased · housekeeping.py:162 (by name)
push · ingest/pipelines.py:132 (by name)
pull_copy · ingest/pipelines.py:250 (by name)
pull_pages · ingest/pipelines.py:278 (by name) and 4 more | gold_helpers.py, test_ingest_pipelines.py and 2 more |
MetadataStore.record_corrections method, line 252 | Record a plan and its corrections in one transaction. A plan id is recorded once; the key columns must be the dataset's primary key and the field one of its columns. | StampingRunLog.record_corrections · adapters/memory_runlog.py:132 (by name)
withdraw · recovery.py:23 (by name)
World.write · world/engine.py:145 (by name) | test_gold_harness.py, test_meta_store.py |
MetadataStore.corrections method, line 256 | The dataset's corrections with their status on a business date (active once the plan's date is reached); as_of None lists every correction as recorded. | _inputs · gold/harness.py:243 (value)
_fact · gold/harness.py:333 (value)
_record · gold/harness.py:456 (value)
main · scripts/run_platform.py:669 (by name)
main · scripts/run_world.py:189 (by name)
StampingRunLog.corrections · adapters/memory_runlog.py:135 (by name) and 1 more | gold_helpers.py, test_gold_engine.py and 7 more |
MetadataStore.backfill_plans method, line 260 | | main · scripts/run_world.py:188 (by name)
StampingRunLog.backfill_plans · adapters/memory_runlog.py:138 (by name)
_active_plans · gold/harness.py:208 (by name) | test_meta_store.py, test_world_tyre.py |
MetadataStore.acquire_lease method, line 262 | ORCH-04: take the writer lease on a table. None when this run holds it (new, re-entered, or taken over from a run whose lease expired); otherwise the run id holding it, and this run must not write. | StampingRunLog.acquire_lease · adapters/memory_runlog.py:143 (by name)
run_gold · gold/harness.py:141 (by name)
_sweep · housekeeping.py:148 (by name)
_one · ingest/runner.py:151 (by name)
_one · silver/runner.py:121 (by name) | test_housekeeping.py, test_meta_store.py and 1 more |
MetadataStore.release_lease method, line 266 | Give the lease back (only the holder's release counts). | StampingRunLog.release_lease · adapters/memory_runlog.py:146 (by name)
run_gold · gold/harness.py:153 (by name)
_sweep · housekeeping.py:155 (by name)
_one · ingest/runner.py:158 (by name)
_one · silver/runner.py:128 (by name) | test_meta_store.py |
MetadataStore.void_batch method, line 269 | Void a loaded batch (ADR-025): from now on batch_status and dataset_files report it as `void`, so silver skips it and a resend of the same bytes is not loaded again. Refused: a batch that never loaded, one already void, no approver or a rea | StampingRunLog.void_batch · adapters/memory_runlog.py:149 (by name)
_sweep_leased · housekeeping.py:173 (by name)
withdraw · recovery.py:20 (by name) | test_meta_store.py, test_silver_runner.py |
MetadataStore.save_profile method, line 274 | Keep a dataset run's profile (B9, DPC-0051). Refused: a dataset run that was never started. | StampingRunLog.save_profile · adapters/memory_runlog.py:164 (by name)
_profile_step · ingest/runner.py:75 (by name) | test_meta_store.py |
MetadataStore.last_profile method, line 277 | The most recent profile kept for the dataset, or None. | StampingRunLog.last_profile · adapters/memory_runlog.py:167 (by name)
_profile_step · ingest/runner.py:69 (by name) | test_meta_store.py |
MetadataStore.set_dq_rules method, line 280 | Replace a dataset's quality rules (the loader writes them from the contract). Refused: a rule id twice, a column the dataset does not have, a severity or action outside DQ_SEVERITIES / DQ_ACTIONS. | StampingRunLog.set_dq_rules · adapters/memory_runlog.py:152 (by name)
load · meta/loader.py:180 (by name) | test_meta_store.py, test_silver_runner.py |
MetadataStore.dq_rules method, line 284 | The dataset's rules, by rule id. | module level · realms/databricks/declarative/silver.py:81 (by name)
lakebase · realms/databricks/engine/tasks/probe.py:83 (by name)
StampingRunLog.dq_rules · adapters/memory_runlog.py:155 (by name)
SilverBaseliner.write · silver/baseline_writer.py:56 (by name)
_one_leased · silver/runner.py:148 (by name) | test_meta_loader.py, test_meta_store.py and 2 more |
MetadataStore.log_dq_results method, line 287 | Record what each rule found in a dataset run. Refused: a run or dataset run that was never started, a rule the dataset does not have, failed above checked or below zero. | StampingRunLog.log_dq_results · adapters/memory_runlog.py:158 (by name) | test_meta_store.py |
MetadataStore.dq_results method, line 291 | The results of a run (optionally of one dataset), in the order they were logged. | StampingRunLog.dq_results · adapters/memory_runlog.py:161 (by name) | test_meta_store.py, test_cli_dq.py and 1 more |
redact function, line 303 | Secrets out, then cap (the repo's rule: logs never contain secrets; error messages at most 4 KB). | MemoryMetaStore.log_step · adapters/memory_meta.py:199
MemoryMetaStore.log_file · adapters/memory_meta.py:207
MemoryMetaStore.log_error · adapters/memory_meta.py:245
MemoryMetaStore.finish_dataset_run · adapters/memory_meta.py:259
MemoryMetaStore.log_dq_results · adapters/memory_meta.py:376 | test_processors_pii.py, test_silver_pipeline_processors.py |
run_status function, line 310 | A run's status from its dataset runs (kept from LakeIngestionFramework's sp_master_run_log_finished). | MemoryMetaStore.finish_run · adapters/memory_meta.py:269 | no test names it |