Data PlatformZoom level 2
The loader: from YAML manifests in git to rows in the store
People describe a domain in YAML files kept in git. The loader is the one piece of code that reads those files and writes the configuration into the metadata store.
It exists so that the running platform never depends on a file. Pipelines read the store. The loader runs on a laptop or in a build, validates the whole folder, and refuses to write anything if one file is wrong.
The manifest folder
| Path under the domain's folder | What it is | Read by this loader |
|---|---|---|
manifest/platform.yaml | What applies to the whole domain: name, time zone, environments, optional settings. | Yes, by compile_domain. |
manifest/datasets/*.yaml | One file per source dataset, named source, dot, dataset. | Yes, by compile_domain. |
contracts/*.odcs.yaml | The data contract of each dataset: columns, types, keys, quality rules. Each manifest points at its contract with a relative path. | Yes, through the manifest's contract field. |
manifest/gold/*.yaml | One file per gold table. | Yes, by load_gold. |
manifest/metrics/, manifest/models/ | Metric and model definitions. | No. This loader does not open them. |
data-platform-catalog/slices/02-tyre/manifest/platform.yaml:3The tyre company's platform manifest, down to its quality defaults. It names no cloud and no product.kind: platform
domain: tyre
business_timezone: Asia/Kolkata
environments: [dev, prod]
quality_defaults: # ADR-009: severity sets the action
critical: {rows: quarantine, dataset: block, page: true}
major: {rows: quarantine, dataset: alert_owner, page: false}
minor: {rows: flag, dataset: digest, page: false}data-platform-catalog/slices/02-tyre/manifest/datasets/erp.material.yaml:3The material master's dataset manifest, without its last blocks on volume, reconciliation and owners.source_name: erp
source_env: prod
dataset_name: material
is_enabled: true
kind: dataset
config:
load_type: FULL
sla_freshness_hours: "24"
sla_freshness_error_hours: "48"
schedule_trigger: cron
schedule_cron: "0 1 * * *"
schedule_timezone: Asia/Kolkata
cost_center: manufacturing-analytics
dedup: {mode: business_key_latest, business_keys: [material_code], order_by: extracted_at}
contract: ../../contracts/material.odcs.yaml
contract_version: 0.1.0
archetypes: [PA2]
ingest: {mode: pull, read: full_copy}
history: {kind: master, track_columns: [description, compound_code, rim_size_in, tyre_class]}
load:
pattern: history
history_order_by: extracted_at
batch_key: [source_batch_id]
new_columns: hold_in_bronze
deletes: {detect: key_compare, apply: mark_deleted}
quarantine_stop_share: 0.0
mass_delete_max_share: 0.05The steps, in order
- Read the dictionary. The list of allowed settings is read from the registry file.
- Compile the domain. The platform file becomes one domain document with a fingerprint.
- Compile each dataset. Each dataset manifest is read with its contract, validated, and turned into one dataset document.
- Collect every problem. Errors from all files are gathered. If there is at least one, the loader raises one error that lists them all and stops. Nothing has been written.
- Add binding settings. Optional extra settings from the deployment are laid over a dataset's own, and its fingerprint is recomputed to cover them.
- Write. The dictionary is registered, then the domain, then each dataset followed by its quality rules.
- Report. The loader returns what it loaded, any warnings, and any dataset the store has that the folder no longer declares.
Building a dataset document
src/dataplatform/meta/loader.py:82One dataset as the store's load function wants it.def dataset_doc(domain: str, manifest: dict[str, Any], contract: dict[str, Any]) -> dict[str, Any]:
plain = json.loads(json.dumps(manifest, default=str))
return {
"domain": domain,
"source_name": manifest["source_name"],
"source_env": manifest.get("source_env", ""),
"dataset_name": manifest["dataset_name"],
"is_enabled": bool(manifest["is_enabled"]),
"kind": manifest.get("kind", "dataset"),
"archetypes": list(manifest["archetypes"]),
"ingest_mode": (manifest.get("ingest") or {}).get("mode", ""),
"ingest_read": (manifest.get("ingest") or {}).get("read", ""),
"history_kind": (manifest.get("history") or {}).get("kind", ""),
"load_pattern": manifest["load"].get("pattern", ""),
"contract_id": contract["id"],
"contract_version": str(contract["version"]),
"fingerprint": config_fingerprint({"manifest": plain,
"contract": json.loads(json.dumps(contract, default=str))}),
"settings": settings_of(manifest),
"columns": columns_of(contract),
"dq_rules": dq_rules_of(contract),
"business_keys": {"primary": odcs.primary_key(contract),
"dedup": list((manifest.get("dedup") or {}).get("business_keys") or [])},
"manifest": plain,
}The document mixes three origins. The identity and the typed fields come from the manifest. The columns and the primary key come from the contract. The fingerprint is a hash of the manifest and the contract together, so a change in either gives a new fingerprint. The manifest is also stored whole, because the pipelines read its load and volume blocks straight from it.
src/dataplatform/meta/loader.py:47Which parts of a manifest become settings: the config block as it is, and three keys of the dedup block with a dedup prefix.def settings_of(manifest: dict[str, Any]) -> dict[str, str]:
"""The manifest's config block plus its dedup block as registry keys (dedup.mode, ...), all as text."""
out = {k: _text(v) for k, v in (manifest.get("config") or {}).items()}
for k in DEDUP_KEYS:
if k in (manifest.get("dedup") or {}):
out[f"dedup.{k}"] = _text(manifest["dedup"][k])
return outEvery value is turned into text. A list becomes its items joined by commas, which is why the sales order line's dedup keys are stored as order_id,line_no.
src/dataplatform/keys.py:18The fingerprint: the first 12 hex characters of a hash of the document with its keys sorted.def config_fingerprint(config: dict[str, Any]) -> str:
"""A short, stable hash of the merged config a run used. Key order does not matter."""
return hashlib.sha256(json.dumps(config, sort_keys=True, default=str).encode()).hexdigest()[:12]Validation before any write
src/dataplatform/meta/loader.py:122Reading and checking the whole folder. Note the single raise at the end.def compile_domain(manifest_dir: Path, defs: list[SettingDef]) -> Docs:
"""Read and validate a domain's manifest folder. Returns (domain doc, dataset docs, warnings)."""
import yaml # local and CI only
platform = yaml.safe_load((manifest_dir / "platform.yaml").read_text())
domain = platform["domain"]
plain = json.loads(json.dumps(platform, default=str))
domain_doc = {"domain": domain, "business_timezone": platform.get("business_timezone", "UTC"),
"environments": list(platform.get("environments") or []),
"settings": {k: _text(v) for k, v in (platform.get("settings") or {}).items()},
"manifest": plain, "fingerprint": config_fingerprint(plain)}
errors: list[str] = []
warnings: list[str] = []
docs: list[dict[str, Any]] = []
for path in sorted((manifest_dir / "datasets").glob("*.yaml")):
manifest = yaml.safe_load(path.read_text())
contract = yaml.safe_load((path.parent / manifest["contract"]).read_text())
errors += [f"{path.name}: {e}" for e in validate(manifest, contract)]
doc = dataset_doc(domain, manifest, contract)
e, w = setting_problems(doc, defs)
errors, warnings = errors + e, warnings + w
errors += [f"{path.name}: {p}" for p in engine_problems(
f"{doc['source_name']}.{doc['dataset_name']}", manifest, doc["settings"], domain_doc["settings"])]
docs.append(doc)
domain_keys = registry.by_key(defs, "platform") | registry.by_key(defs, "dataset")
errors += [f"platform.yaml: setting '{k}' is not registered" for k in domain_doc["settings"]
if k not in domain_keys]
if errors:
raise ManifestError("; ".join(errors))
return domain_doc, docs, warningsWalk through the loop. For each dataset file, the manifest and its contract are parsed. validate checks the required fields, that the contract version the manifest pins is the contract's version, and that the history order column is a contract column. setting_problems checks the settings against the dictionary. engine_problems checks that the chosen silver engine can handle the dataset. After the loop the platform file's own settings are checked. All errors are joined into one message, so a person fixing a folder sees every problem at once.
src/dataplatform/meta/loader.py:109An unknown key is an error. A value of the wrong type is a warning.def setting_problems(doc: dict[str, Any], defs: list[SettingDef]) -> tuple[list[str], list[str]]:
"""(errors: unregistered keys, warnings: values that do not fit their type)."""
known = registry.by_key(defs, "dataset")
name = f"{doc['source_name']}.{doc['dataset_name']}"
errors = [f"{name}: setting '{k}' is not registered" for k in doc["settings"] if k not in known]
warnings = [f"{name}: {p}" for k, v in doc["settings"].items() if k in known
for p in [registry.type_problem(known[k], v)] if p]
return errors, warningsWriting
src/dataplatform/meta/loader.py:154The only place that writes. Every store call comes after compile_domain has returned.def load(store: MetadataStore, manifest_dir: Path, defs: list[SettingDef] | None = None,
extra_settings: dict[str, dict[str, str]] | None = None) -> LoadReport:
"""Register the settings, then write the domain and every dataset. Validation happens before any write.
`extra_settings` ({"source.dataset": {key: value}}) adds binding-level settings on top of the manifest, e.g. the
delivery shape of a change-capture service (B4); every key must be registered, and the dataset's fingerprint
covers them, so the run record shows which shape a run read."""
defs = defs if defs is not None else registry.load()
domain_doc, docs, warnings = compile_domain(manifest_dir, defs)
known = registry.by_key(defs, "dataset")
bad = [f"{name}: setting '{k}' is not registered" for name, extra in (extra_settings or {}).items()
for k in extra if k not in known]
if bad:
raise ManifestError("; ".join(bad))
for doc in docs:
extra = (extra_settings or {}).get(f"{doc['source_name']}.{doc['dataset_name']}")
if extra:
doc["settings"] = {**doc["settings"], **extra}
binding = dict(sorted(extra.items()))
doc["fingerprint"] = config_fingerprint({"manifest": doc["fingerprint"], "binding": binding})
store.register_settings(defs)
store.load_domain(domain_doc)
report = LoadReport(domain_doc["domain"], warnings=warnings)
for doc in docs:
rules = doc.pop("dq_rules", [])
store.load_dataset(doc)
store.set_dq_rules(doc["source_name"], doc["dataset_name"], rules)
report.loaded.append(f"{doc['source_name']}.{doc['dataset_name']}")
declared = set(report.loaded)
report.not_in_manifest = sorted(f"{d.source}.{d.dataset}" for d in store.datasets(domain_doc["domain"])
if f"{d.source}.{d.dataset}" not in declared)
return reportThe order of the writes follows the foreign keys: the dictionary first, because settings point at it, then the domain, then each dataset. A dataset's quality rules are written right after the dataset. The last lines compare what the store lists for the domain with what the folder declared. A dataset found only in the store is reported and left alone.
src/dataplatform/meta/loader.py:216Gold tables go into the same store as datasets, after the source datasets.def load_gold(store: MetadataStore, manifest_dir: Path) -> list[str]:
"""Write every gold manifest (manifest/gold/*.yaml) into the store beside the source datasets, after `load`.
A gold manifest's kind or contract version that does not match is refused before anything is written."""
import yaml # local and CI only
domain = yaml.safe_load((manifest_dir / "platform.yaml").read_text())["domain"]
docs, errors = [], []
for path in sorted((manifest_dir / "gold").glob("*.yaml")):
manifest = yaml.safe_load(path.read_text())
contract = yaml.safe_load((path.parent / manifest["contract"]).read_text())
if manifest.get("kind") not in GOLD_KINDS:
errors.append(f"{path.name}: kind {manifest.get('kind')!r} is not one of {GOLD_KINDS}")
if str(manifest.get("contract_version")) != str(contract["version"]):
errors.append(f"{path.name}: contract_version {manifest.get('contract_version')} but the contract is "
f"{contract['version']}")
docs.append(gold_doc(domain, manifest, contract))
if errors:
raise ManifestError("; ".join(errors))
for doc in docs:
store.load_dataset(doc)
return [f"{d['source_name']}.{d['dataset_name']}" for d in docs]A worked example: one mistake, nothing written
The table shows what happens when one file of the tyre folder is broken. The left column is the change, the middle column is the message, the right column is the state of the store afterwards.
| Change to the folder | Message | Store afterwards |
|---|---|---|
retry_cuont: 3 added to the delivery manifest | setting 'retry_cuont' is not registered | Empty. No domain, no dataset, no dictionary. |
The press manifest pins contract version 9.9.9 | manifest pins contract 9.9.9 but the contract is another version | Empty. |
owners removed from the inspection manifest | missing field: owners | Empty. |
| The press manifest file deleted after an earlier good load | No error. The report lists mes.press as not in the manifest. | The press dataset is still there. |
The settings the loader reads
| Manifest field | What the loader does with it | Required |
|---|---|---|
source_name, dataset_name, is_enabled | Identity and the enabled flag. | Yes |
config | Each key becomes a dataset setting. Each key must be in the dictionary at dataset scope. | Yes |
dedup | mode, business_keys and order_by become settings. The business keys also become the dedup key. | No |
contract, contract_version | The path to the contract and the version it must have. | Yes |
ingest.mode, ingest.read, history.kind, load.pattern | Copied into typed columns of the dataset row. | load is required |
archetypes, reconciliation, owners | Required to be present. Kept inside the manifest JSON. | Yes |
settings in platform.yaml | Each key becomes a domain setting. Each key must be in the dictionary at dataset or platform scope. | No |
How it is tested
tests/dataplatform/test_meta_loader.py loads the real tyre folder into the in-memory store. It checks the nine datasets one by one, checks that loading twice gives the same fingerprints, and breaks one file at a time to prove that a bad folder leaves the store empty. A last test loads the folder into PostgreSQL as well and compares every dataset view between the two adapters.
tests/dataplatform/test_meta_loader.py:113The broken-folder cases: which file, what is changed, and the message that must appear. ("unregistered config key", "datasets/erp.delivery.yaml", lambda d: d["config"].update(retry_cuont="3"),
"setting 'retry_cuont' is not registered"),
("contract version not the contract's", "datasets/mes.press.yaml", lambda d: d.update(contract_version="9.9.9"),
"manifest pins contract 9.9.9"),
("required field missing", "datasets/lims.inspection_result.yaml", lambda d: d.pop("owners"),
"missing field: owners"),
("history ordered by a column the contract lacks", "datasets/erp.sales_order_line.yaml",
lambda d: d["load"].update(history_order_by="change_no"), "load.history_order_by 'change_no'"),
("unregistered domain setting", "platform.yaml", lambda d: d.update(settings={"timezone": "UTC"}),
"platform.yaml: setting 'timezone' is not registered"),tests/dataplatform/test_meta_loader.py:125The test body. The last line is the point: the store is still empty.def test_bad_manifest_refused_before_any_write(tmp_path, case, file, change, message):
shutil.copytree(TYRE.parent, tmp_path / "slice")
_break(tmp_path / "slice/manifest" / file, change)
store = MemoryMetaStore()
with pytest.raises(ManifestError, match=message.replace(".", r"\.")):
loader.load(store, tmp_path / "slice/manifest")
assert (store.domains, store.ds, store.registry) == ({}, {}, {})- A wrong value type is only a warning. The value is written as it is.
- A dataset removed from the folder stays in the store and stays enabled. The loader reports it and a person has to act.
- The write phase is several function calls, each its own transaction. If the connection failed halfway, some datasets would be on the new configuration and some on the old. Running the loader again repairs it, because each write replaces a dataset whole.
load_goldchecks less than the dataset path. It checks the kind and the contract version. Gold tables are stored with no settings, because no gold setting is in the dictionary yet.- There is no source-level manifest file, so nothing can declare a source's own settings.
- The loader needs a YAML library, which is installed only for local runs and the build. The platform itself never parses YAML.