cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Code map › Who calls what › meta

Data PlatformZoom level 3

Who calls what: meta

Every class, function and method in the meta package, with who calls it. Found by reading the code. A call written as object.method() is matched by the method's name, so a same-named method elsewhere can appear: those rows say by name.

66 definitions in 3 modules. Jump to a module: loader.py · port.py · registry.py

meta/loader.py

The validating loader: a domain's manifest folder -> the metadata store (M3: domains are data; REL-01: only the loader writes configuration, and only from manifest files in version control). Metadata store

DefinitionWhat it does (its own first sentence)Called byNamed in tests
LoadReport
class, line 32
load · meta/loader.py:176no test names it
_text
function, line 39
settings_of · meta/loader.py:49
compile_domain · meta/loader.py:131
no test names it
settings_of
function, line 47
The manifest's config block plus its dedup block as registry keys (dedup.mode, ...), all as text.dataset_doc · meta/loader.py:100no test names it
columns_of
function, line 56
dataset_doc · meta/loader.py:101
gold_doc · meta/loader.py:210
test_slice1_local.py
dq_rules_of
function, line 63
The contract's quality rules (ODCS `quality` on a property or on the table) as the store's rules (B5): the rule id is <column or table>-<metric or type>-<n>; ODCS severity critical, major or minor sets the action (ADR-009); an unknown or midataset_doc · meta/loader.py:102no test names it
dataset_doc
function, line 82
compile_domain · meta/loader.py:140no test names it
setting_problems
function, line 109
(errors: unregistered keys, warnings: values that do not fit their type).compile_domain · meta/loader.py:141no test names it
compile_domain
function, line 122
Read and validate a domain's manifest folder. Returns (domain doc, dataset docs, warnings).load · meta/loader.py:162test_silver_engines.py
load
function, line 154
Register the settings, then write the domain and every dataset. Validation happens before any write.main · realms/aws-databricks/meta_setup.py:79no test names it
gold_doc
function, line 189
A gold manifest as a dataset of the store: source `gold_<manifest domain>` (its gold schema, ADR-012), the contract's columns, the grain as its primary key. Settings stay empty: gold has no registered keys yet.load_gold · meta/loader.py:231gold_helpers.py
load_gold
function, line 216
Write every gold manifest (manifest/gold/*.yaml) into the store beside the source datasets, after `load`. A gold manifest's kind or contract version that does not match is refused before anything is written.none found in the codetest_twin_engine.py

meta/port.py

The metadata store port (ADR-010; dpc/ports.yaml metadata_service and run_log). The kernel imports only these shapes. Metadata store

DefinitionWhat it does (its own first sentence)Called byNamed in tests
MetaError
class, line 34
Base for every refusal by the metadata store.none found in the codeno test names it
UnknownSetting
class, line 38
A setting key not in the registry for its scope (DPC-0034).MemoryMetaStore.load_domain · adapters/memory_meta.py:85
MemoryMetaStore.load_dataset · adapters/memory_meta.py:93
module level · adapters/postgres_meta.py:40 (value)
test_meta_store.py
NotFound
class, line 42
A dataset, run or dataset run the store does not have.MemoryMetaStore.load_dataset · adapters/memory_meta.py:90
MemoryMetaStore._get · adapters/memory_meta.py:106
MemoryMetaStore._run · adapters/memory_meta.py:160
MemoryMetaStore.start_run · adapters/memory_meta.py:174
module level · adapters/postgres_meta.py:40 (value)
test_lakebase.py, test_meta_store.py
InvalidArgument
class, line 46
A value outside its allowed set (status, run type, reason too short, ...).MemoryMetaStore.register_settings · adapters/memory_meta.py:76
MemoryMetaStore.load_dataset · adapters/memory_meta.py:97
MemoryMetaStore.business_keys · adapters/memory_meta.py:138
MemoryMetaStore.start_run · adapters/memory_meta.py:172
MemoryMetaStore.log_step · adapters/memory_meta.py:196
MemoryMetaStore.log_file · adapters/memory_meta.py:203
and 8 more
test_meta_store.py
RunStateError
class, line 50
A run step out of order: finishing a run never started, starting one twice, finishing with open datasets.MemoryMetaStore._ds_run · adapters/memory_meta.py:168
MemoryMetaStore.start_run · adapters/memory_meta.py:176
MemoryMetaStore.start_dataset_run · adapters/memory_meta.py:183
MemoryMetaStore.void_batch · adapters/memory_meta.py:236
MemoryMetaStore.finish_dataset_run · adapters/memory_meta.py:254
MemoryMetaStore.finish_run · adapters/memory_meta.py:265
and 2 more
test_meta_store.py
SettingDef
class, line 55
PostgresMetaStore.settings_registry · adapters/postgres_meta.py:126
load · meta/registry.py:19
MemoryMetaStore.__init__ · adapters/memory_meta.py:51 (value)
by_key · meta/registry.py:24 (value)
test_ingest_steps.py, test_meta_store.py and 2 more
Setting
class, line 66
MemoryMetaStore.settings · adapters/memory_meta.py:124
PostgresMetaStore.settings · adapters/postgres_meta.py:134
no test names it
Column
class, line 73
MemoryMetaStore.columns · adapters/memory_meta.py:132
PostgresMetaStore.columns · adapters/postgres_meta.py:137
Spec.from_view · ingest/steps.py:91
module level · ingest/steps.py:67 (value)
test_ingest_steps.py
DatasetRef
class, line 83
MemoryMetaStore.datasets · adapters/memory_meta.py:112
PostgresMetaStore.datasets · adapters/postgres_meta.py:131
no test names it
StepRecord
class, line 94
_step · gold/harness.py:168
_sweep_leased · housekeeping.py:167
_profile_step · ingest/runner.py:72
_detect_steps · ingest/runner.py:109
_one_leased · ingest/runner.py:197
busy · leases.py:23
and 3 more
test_meta_store.py, test_obs_steps.py and 1 more
FileRecord
class, line 105
MemoryMetaStore.dataset_files · adapters/memory_meta.py:305
PostgresMetaStore.dataset_files · adapters/postgres_meta.py:200
StoreBatchRegistry.record_batch · adapters/store_registry.py:29
_record · ingest/steps.py:382
DatasetHandle.file · obs/recorder.py:112
SilverBaseliner.write · silver/baseline_writer.py:75
and 1 more
gold_helpers.py, test_housekeeping.py and 4 more
BackfillPlan
class, line 121
A plan that activates corrections on a date and names the gold windows to rebuild (ADR-017, DPC-0082).PostgresMetaStore.backfill_plans · adapters/postgres_meta.py:218
withdraw · recovery.py:21
ClinicalTrial._correct · world/domains/clinical.py:526
TyreCompany._correct · world/domains/tyre.py:516
MemoryMetaStore.__init__ · adapters/memory_meta.py:63 (value)
ClinicalTrial.__init__ · world/domains/clinical.py:147 (value)
and 2 more
test_gold_harness.py, test_meta_store.py
Correction
class, line 134
One corrected value (DPC-0082: entity, field, value, applies from and to, who, why, status). The entity is the plan's dataset; who and why are the plan's. `key` holds the business key columns and their values.MemoryMetaStore.record_corrections · adapters/memory_meta.py:327
MemoryMetaStore.corrections · adapters/memory_meta.py:338
PostgresMetaStore.corrections · adapters/postgres_meta.py:213
ClinicalTrial._correct · world/domains/clinical.py:522
TyreCompany._correct · world/domains/tyre.py:507
test_gold_engine.py, test_gold_harness.py and 1 more
DqRule
class, line 154
One quality rule of a dataset (DPC-0010, ADR-009: severity sets the action). `column` is empty for a rule on the whole row or batch; `check` names the check (not_null, range, allowed, unique, freshness, ...).MemoryMetaStore.set_dq_rules · adapters/memory_meta.py:359
PostgresMetaStore.dq_rules · adapters/postgres_meta.py:246
dq_rules_of · meta/loader.py:77
test_contract_rules.py, test_meta_store.py and 8 more
DqResult
class, line 168
What one rule found in one dataset run: rows checked, rows failed, the action taken (DPC-0042).MemoryMetaStore.log_dq_results · adapters/memory_meta.py:375
PostgresMetaStore.dq_results · adapters/postgres_meta.py:255
test_meta_store.py
MetadataStore
class, line 181
none found in the codeno test names it
MetadataStore.register_settings
method, line 183
StampingRunLog.register_settings · adapters/memory_runlog.py:48 (by name)
load · meta/loader.py:174 (by name)
test_housekeeping.py, test_ingest_pipelines.py and 7 more
MetadataStore.load_domain
method, line 185
doc: domain, business_timezone, environments, settings {key: value}, manifest, fingerprint. Domain settings are defaults for dataset keys (or platform keys); unknown keys are refused.StampingRunLog.load_domain · adapters/memory_runlog.py:51 (by name)
load · meta/loader.py:175 (by name)
gold_helpers.py, test_housekeeping.py and 10 more
MetadataStore.load_dataset
method, line 189
doc: domain, source_name, source_env, dataset_name, is_enabled, kind, archetypes, ingest_mode, ingest_read, history_kind, load_pattern, contract_id, contract_version, fingerprint, settings {key: value}, columns [Column fields], business_keyStampingRunLog.load_dataset · adapters/memory_runlog.py:54 (by name)
load · meta/loader.py:179 (by name)
load_gold · meta/loader.py:235 (by name)
gold_helpers.py, test_housekeeping.py and 10 more
MetadataStore.settings_registry
method, line 195
StampingRunLog.settings_registry · adapters/memory_runlog.py:57 (by name)test_meta_store.py
MetadataStore.datasets
method, line 197
Monitor._lateness · obs/monitor.py:88 (value)
main · realms/aws-databricks/meta_setup.py:85 (by name)
module level · realms/databricks/declarative/silver.py:73 (by name)
lakebase · realms/databricks/engine/tasks/probe.py:80 (by name)
hop_delays · scripts/run_platform.py:160 (by name)
StampingRunLog.datasets · adapters/memory_runlog.py:60 (by name)
and 10 more
test_bundle.py, test_semantic_governance.py and 29 more
MetadataStore.settings
method, line 199
Every registered dataset key, merged: dataset over domain over the registry default. (DPC-0002 also names a source level; DPM has no source manifest yet, so nothing feeds it.)_read · ingest/pipelines.py:67 (value)
push · ingest/pipelines.py:147 (value)
enter · ingest/steps.py:328 (value)
main · realms/aws-databricks/meta_setup.py:88 (by name)
StampingRunLog.settings · adapters/memory_runlog.py:63 (by name)
test_bundle.py, gold_helpers.py and 20 more
MetadataStore.columns
method, line 203
_ingest_table_to_bronze · scripts/demo_end_to_end.py:389 (value)
CustomerRunLog.__init__ · adapters/customer_runlog.py:34 (value)
_widget · adapters/lakeview.py:103 (value)
UcIcebergStore._create · adapters/table_store.py:403 (value)
validate · obs/dashboards.py:105 (value)
contract_check · stages/clean.py:44 (value)
and 8 more
test_lakeformation.py, test_semantic_governance.py and 67 more
MetadataStore.business_keys
method, line 205
StampingRunLog.business_keys · adapters/memory_runlog.py:69 (by name)
plans · gold/harness.py:96 (by name)
_inputs · gold/harness.py:245 (by name)
gold_helpers.py, silver_helpers.py and 26 more
MetadataStore.dataset_view
method, line 207
The merged configuration for one dataset with its fingerprint (port operation dataset_view).module level · realms/databricks/declarative/silver.py:77 (by name)
StampingRunLog.dataset_view · adapters/memory_runlog.py:72 (by name)
StoreBatchRegistry.dataset_config · adapters/store_registry.py:21 (by name)
plans · gold/harness.py:94 (by name)
run_gold · gold/harness.py:138 (by name)
_sweep · housekeeping.py:142 (by name)
and 7 more
test_ingest_pipelines.py, test_meta_loader.py and 4 more
MetadataStore.start_run
method, line 211
StampingRunLog.start_run · adapters/memory_runlog.py:77 (by name)
run_gold · gold/harness.py:132 (by name)
run_housekeeping · housekeeping.py:127 (by name)
ingest · ingest/runner.py:126 (by name)
RunRecorder.run · obs/recorder.py:191 (by name)
TracedStore.start_run · obs/traced_store.py:33 (by name)
and 1 more
gold_helpers.py, test_housekeeping.py and 7 more
MetadataStore.start_dataset_run
method, line 214
Open the dataset's run record with its config snapshot; returns the watermark before the run.StampingRunLog.start_dataset_run · adapters/memory_runlog.py:83 (by name)
run_gold · gold/harness.py:139 (by name)
_sweep · housekeeping.py:146 (by name)
_one · ingest/runner.py:149 (by name)
RunHandle.dataset · obs/recorder.py:148 (by name)
TracedStore.start_dataset_run · obs/traced_store.py:39 (by name)
and 2 more
gold_helpers.py, test_housekeeping.py and 7 more
MetadataStore.log_step
method, line 218
StampingRunLog.log_step · adapters/memory_runlog.py:88 (by name)
_step · gold/harness.py:168 (by name)
_sweep_leased · housekeeping.py:167 (by name)
_profile_step · ingest/runner.py:72 (by name)
_detect_steps · ingest/runner.py:109 (by name)
_one_leased · ingest/runner.py:196 (by name)
and 5 more
test_meta_store.py, test_obs_traced_store.py
MetadataStore.log_file
method, line 220
StampingRunLog.log_file · adapters/memory_runlog.py:92 (by name)
StoreBatchRegistry.record_batch · adapters/store_registry.py:28 (by name)
_record · ingest/steps.py:381 (by name)
DatasetHandle.file · obs/recorder.py:112 (by name)
SilverBaseliner.write · silver/baseline_writer.py:75 (by name)
gold_helpers.py, test_housekeeping.py and 4 more
MetadataStore.batch_status
method, line 222
The latest status recorded for a batch key of this dataset, or None if never seen.StampingRunLog.batch_status · adapters/memory_runlog.py:96 (by name)
StoreBatchRegistry.batch_status · adapters/store_registry.py:24 (by name)
enter · ingest/steps.py:318 (by name)
land · lander.py:34 (by name)
test_meta_store.py, test_slice1_local.py
MetadataStore.log_error
method, line 225
Message is redacted and capped by the store, whoever calls.StampingRunLog.log_error · adapters/memory_runlog.py:100 (by name)
_fail · gold/harness.py:173 (by name)
_one_leased · ingest/runner.py:175 (by name)
DatasetHandle.error · obs/recorder.py:107 (by name)
RunHandle.dataset · obs/recorder.py:154 (by name)
RunRecorder.run · obs/recorder.py:197 (by name)
and 1 more
test_meta_store.py
MetadataStore.finish_dataset_run
method, line 229
Close the dataset run. Watermark and bookmarks move only when status is succeeded, in one transaction.StampingRunLog.finish_dataset_run · adapters/memory_runlog.py:106 (by name)
_fail · gold/harness.py:174 (by name)
_dimension · gold/harness.py:189 (by name)
_fact · gold/harness.py:342 (by name)
_sweep_leased · housekeeping.py:177 (by name)
_one_leased · ingest/runner.py:176 (by name)
and 5 more
gold_helpers.py, test_housekeeping.py and 5 more
MetadataStore.finish_run
method, line 234
Close the run; its status follows its dataset runs. Returns the status.StampingRunLog.finish_run · adapters/memory_runlog.py:110 (by name)
run_gold · gold/harness.py:156 (by name)
run_housekeeping · housekeeping.py:134 (by name)
ingest · ingest/runner.py:135 (by name)
RunRecorder.run · obs/recorder.py:201 (by name)
TracedStore.finish_run · obs/traced_store.py:60 (by name)
and 1 more
gold_helpers.py, test_housekeeping.py and 4 more
MetadataStore.watermark
method, line 237
push · ingest/pipelines.py:161 (value)
pull_pages · ingest/pipelines.py:302 (value)
stream_events · ingest/pipelines.py:357 (value)
DailyCopy.__init__ · obs/system.py:84 (value)
DailyCopy.copy · obs/system.py:92 (value)
DailyCopy.on_demand · obs/system.py:100 (value)
and 2 more
gold_helpers.py, test_ingest_pipelines.py and 13 more
MetadataStore.bookmarks
method, line 239
StampingRunLog.bookmarks · adapters/memory_runlog.py:118 (by name)
silver_watermark · gold/harness.py:106 (by name)
feed_marks · gold/harness.py:122 (by name)
_fact · gold/harness.py:289 (by name)
push · ingest/pipelines.py:156 (by name)
_feed_bookmarks · ingest/pipelines.py:196 (by name)
and 6 more
gold_helpers.py, test_gold_harness.py and 6 more
MetadataStore.reset_watermark
method, line 241
An operator reset (backfill, correction): audited, with a reason.StampingRunLog.reset_watermark · adapters/memory_runlog.py:122 (by name)test_meta_store.py
MetadataStore.run_report
method, line 244
Everything recorded for one run: run, datasets, steps, files, errors.success_scorecard · scripts/run_platform.py:184 (by name)
StampingRunLog.run_report · adapters/memory_runlog.py:125 (by name)
StampingRunLog.snapshot · adapters/memory_runlog.py:175 (by name)
_steps · gold/inspect.py:55 (by name)
test_ingest_pipelines.py, test_meta_store.py and 5 more
MetadataStore.dataset_files
method, line 247
Every batch recorded for a dataset (optionally one window), oldest first: what the lander has handled, and where a reload finds the batches of a window again.StampingRunLog.dataset_files · adapters/memory_runlog.py:128 (by name)
silver_watermark · gold/harness.py:107 (by name)
_sweep_leased · housekeeping.py:162 (by name)
push · ingest/pipelines.py:132 (by name)
pull_copy · ingest/pipelines.py:250 (by name)
pull_pages · ingest/pipelines.py:278 (by name)
and 4 more
gold_helpers.py, test_ingest_pipelines.py and 2 more
MetadataStore.record_corrections
method, line 252
Record a plan and its corrections in one transaction. A plan id is recorded once; the key columns must be the dataset's primary key and the field one of its columns.StampingRunLog.record_corrections · adapters/memory_runlog.py:132 (by name)
withdraw · recovery.py:23 (by name)
World.write · world/engine.py:145 (by name)
test_gold_harness.py, test_meta_store.py
MetadataStore.corrections
method, line 256
The dataset's corrections with their status on a business date (active once the plan's date is reached); as_of None lists every correction as recorded._inputs · gold/harness.py:243 (value)
_fact · gold/harness.py:333 (value)
_record · gold/harness.py:456 (value)
main · scripts/run_platform.py:669 (by name)
main · scripts/run_world.py:189 (by name)
StampingRunLog.corrections · adapters/memory_runlog.py:135 (by name)
and 1 more
gold_helpers.py, test_gold_engine.py and 7 more
MetadataStore.backfill_plans
method, line 260
main · scripts/run_world.py:188 (by name)
StampingRunLog.backfill_plans · adapters/memory_runlog.py:138 (by name)
_active_plans · gold/harness.py:208 (by name)
test_meta_store.py, test_world_tyre.py
MetadataStore.acquire_lease
method, line 262
ORCH-04: take the writer lease on a table. None when this run holds it (new, re-entered, or taken over from a run whose lease expired); otherwise the run id holding it, and this run must not write.StampingRunLog.acquire_lease · adapters/memory_runlog.py:143 (by name)
run_gold · gold/harness.py:141 (by name)
_sweep · housekeeping.py:148 (by name)
_one · ingest/runner.py:151 (by name)
_one · silver/runner.py:121 (by name)
test_housekeeping.py, test_meta_store.py and 1 more
MetadataStore.release_lease
method, line 266
Give the lease back (only the holder's release counts).StampingRunLog.release_lease · adapters/memory_runlog.py:146 (by name)
run_gold · gold/harness.py:153 (by name)
_sweep · housekeeping.py:155 (by name)
_one · ingest/runner.py:158 (by name)
_one · silver/runner.py:128 (by name)
test_meta_store.py
MetadataStore.void_batch
method, line 269
Void a loaded batch (ADR-025): from now on batch_status and dataset_files report it as `void`, so silver skips it and a resend of the same bytes is not loaded again. Refused: a batch that never loaded, one already void, no approver or a reaStampingRunLog.void_batch · adapters/memory_runlog.py:149 (by name)
_sweep_leased · housekeeping.py:173 (by name)
withdraw · recovery.py:20 (by name)
test_meta_store.py, test_silver_runner.py
MetadataStore.save_profile
method, line 274
Keep a dataset run's profile (B9, DPC-0051). Refused: a dataset run that was never started.StampingRunLog.save_profile · adapters/memory_runlog.py:164 (by name)
_profile_step · ingest/runner.py:75 (by name)
test_meta_store.py
MetadataStore.last_profile
method, line 277
The most recent profile kept for the dataset, or None.StampingRunLog.last_profile · adapters/memory_runlog.py:167 (by name)
_profile_step · ingest/runner.py:69 (by name)
test_meta_store.py
MetadataStore.set_dq_rules
method, line 280
Replace a dataset's quality rules (the loader writes them from the contract). Refused: a rule id twice, a column the dataset does not have, a severity or action outside DQ_SEVERITIES / DQ_ACTIONS.StampingRunLog.set_dq_rules · adapters/memory_runlog.py:152 (by name)
load · meta/loader.py:180 (by name)
test_meta_store.py, test_silver_runner.py
MetadataStore.dq_rules
method, line 284
The dataset's rules, by rule id.module level · realms/databricks/declarative/silver.py:81 (by name)
lakebase · realms/databricks/engine/tasks/probe.py:83 (by name)
StampingRunLog.dq_rules · adapters/memory_runlog.py:155 (by name)
SilverBaseliner.write · silver/baseline_writer.py:56 (by name)
_one_leased · silver/runner.py:148 (by name)
test_meta_loader.py, test_meta_store.py and 2 more
MetadataStore.log_dq_results
method, line 287
Record what each rule found in a dataset run. Refused: a run or dataset run that was never started, a rule the dataset does not have, failed above checked or below zero.StampingRunLog.log_dq_results · adapters/memory_runlog.py:158 (by name)test_meta_store.py
MetadataStore.dq_results
method, line 291
The results of a run (optionally of one dataset), in the order they were logged.StampingRunLog.dq_results · adapters/memory_runlog.py:161 (by name)test_meta_store.py, test_cli_dq.py and 1 more
redact
function, line 303
Secrets out, then cap (the repo's rule: logs never contain secrets; error messages at most 4 KB).MemoryMetaStore.log_step · adapters/memory_meta.py:199
MemoryMetaStore.log_file · adapters/memory_meta.py:207
MemoryMetaStore.log_error · adapters/memory_meta.py:245
MemoryMetaStore.finish_dataset_run · adapters/memory_meta.py:259
MemoryMetaStore.log_dq_results · adapters/memory_meta.py:376
test_processors_pii.py, test_silver_pipeline_processors.py
run_status
function, line 310
A run's status from its dataset runs (kept from LakeIngestionFramework's sp_master_run_log_finished).MemoryMetaStore.finish_run · adapters/memory_meta.py:269no test names it

meta/registry.py

The settings registry (DPC-0034): every key a manifest may set, with its scope, type and default. Metadata store

DefinitionWhat it does (its own first sentence)Called byNamed in tests
load
function, line 17
none found in the codeno test names it
by_key
function, line 24
none found in the codetest_meta_loader.py, test_silver_runner.py and 1 more
type_problem
function, line 28
Why a value does not fit its registered type, or None. The loader reports these; unknown keys it refuses.none found in the codetest_meta_loader.py