cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Metadata store › Loader

Data PlatformZoom level 2

The loader: from YAML manifests in git to rows in the store

People describe a domain in YAML files kept in git. The loader is the one piece of code that reads those files and writes the configuration into the metadata store.

It exists so that the running platform never depends on a file. Pipelines read the store. The loader runs on a laptop or in a build, validates the whole folder, and refuses to write anything if one file is wrong.

platform.yamlone per domaindatasets/*.yamlone per datasetcontracts/*.yamlcolumns and rulescompile_domainread, validate,build documentsErrorsstop, nothing writtenloadwrites in orderOpen: The storeThe storeconfig schemazoom in ›any errornone
Everything on the left is read and checked first. The store is touched only on the lower path.

The manifest folder

Path under the domain's folderWhat it isRead by this loader
manifest/platform.yamlWhat applies to the whole domain: name, time zone, environments, optional settings.Yes, by compile_domain.
manifest/datasets/*.yamlOne file per source dataset, named source, dot, dataset.Yes, by compile_domain.
contracts/*.odcs.yamlThe data contract of each dataset: columns, types, keys, quality rules. Each manifest points at its contract with a relative path.Yes, through the manifest's contract field.
manifest/gold/*.yamlOne file per gold table.Yes, by load_gold.
manifest/metrics/, manifest/models/Metric and model definitions.No. This loader does not open them.
data-platform-catalog/slices/02-tyre/manifest/platform.yaml:3The tyre company's platform manifest, down to its quality defaults. It names no cloud and no product.
kind: platform
domain: tyre
business_timezone: Asia/Kolkata
environments: [dev, prod]
quality_defaults:                                     # ADR-009: severity sets the action
  critical: {rows: quarantine, dataset: block, page: true}
  major: {rows: quarantine, dataset: alert_owner, page: false}
  minor: {rows: flag, dataset: digest, page: false}
data-platform-catalog/slices/02-tyre/manifest/datasets/erp.material.yaml:3The material master's dataset manifest, without its last blocks on volume, reconciliation and owners.
source_name: erp
source_env: prod
dataset_name: material
is_enabled: true
kind: dataset
config:
  load_type: FULL
  sla_freshness_hours: "24"
  sla_freshness_error_hours: "48"
  schedule_trigger: cron
  schedule_cron: "0 1 * * *"
  schedule_timezone: Asia/Kolkata
  cost_center: manufacturing-analytics
dedup: {mode: business_key_latest, business_keys: [material_code], order_by: extracted_at}
contract: ../../contracts/material.odcs.yaml
contract_version: 0.1.0
archetypes: [PA2]
ingest: {mode: pull, read: full_copy}
history: {kind: master, track_columns: [description, compound_code, rim_size_in, tyre_class]}
load:
  pattern: history
  history_order_by: extracted_at
  batch_key: [source_batch_id]
  new_columns: hold_in_bronze
  deletes: {detect: key_compare, apply: mark_deleted}
  quarantine_stop_share: 0.0
  mass_delete_max_share: 0.05

The steps, in order

  1. Read the dictionary. The list of allowed settings is read from the registry file.
  2. Compile the domain. The platform file becomes one domain document with a fingerprint.
  3. Compile each dataset. Each dataset manifest is read with its contract, validated, and turned into one dataset document.
  4. Collect every problem. Errors from all files are gathered. If there is at least one, the loader raises one error that lists them all and stops. Nothing has been written.
  5. Add binding settings. Optional extra settings from the deployment are laid over a dataset's own, and its fingerprint is recomputed to cover them.
  6. Write. The dictionary is registered, then the domain, then each dataset followed by its quality rules.
  7. Report. The loader returns what it loaded, any warnings, and any dataset the store has that the folder no longer declares.

Building a dataset document

src/dataplatform/meta/loader.py:82One dataset as the store's load function wants it.
def dataset_doc(domain: str, manifest: dict[str, Any], contract: dict[str, Any]) -> dict[str, Any]:
    plain = json.loads(json.dumps(manifest, default=str))
    return {
        "domain": domain,
        "source_name": manifest["source_name"],
        "source_env": manifest.get("source_env", ""),
        "dataset_name": manifest["dataset_name"],
        "is_enabled": bool(manifest["is_enabled"]),
        "kind": manifest.get("kind", "dataset"),
        "archetypes": list(manifest["archetypes"]),
        "ingest_mode": (manifest.get("ingest") or {}).get("mode", ""),
        "ingest_read": (manifest.get("ingest") or {}).get("read", ""),
        "history_kind": (manifest.get("history") or {}).get("kind", ""),
        "load_pattern": manifest["load"].get("pattern", ""),
        "contract_id": contract["id"],
        "contract_version": str(contract["version"]),
        "fingerprint": config_fingerprint({"manifest": plain,
                                           "contract": json.loads(json.dumps(contract, default=str))}),
        "settings": settings_of(manifest),
        "columns": columns_of(contract),
        "dq_rules": dq_rules_of(contract),
        "business_keys": {"primary": odcs.primary_key(contract),
                          "dedup": list((manifest.get("dedup") or {}).get("business_keys") or [])},
        "manifest": plain,
    }

The document mixes three origins. The identity and the typed fields come from the manifest. The columns and the primary key come from the contract. The fingerprint is a hash of the manifest and the contract together, so a change in either gives a new fingerprint. The manifest is also stored whole, because the pipelines read its load and volume blocks straight from it.

src/dataplatform/meta/loader.py:47Which parts of a manifest become settings: the config block as it is, and three keys of the dedup block with a dedup prefix.
def settings_of(manifest: dict[str, Any]) -> dict[str, str]:
    """The manifest's config block plus its dedup block as registry keys (dedup.mode, ...), all as text."""
    out = {k: _text(v) for k, v in (manifest.get("config") or {}).items()}
    for k in DEDUP_KEYS:
        if k in (manifest.get("dedup") or {}):
            out[f"dedup.{k}"] = _text(manifest["dedup"][k])
    return out

Every value is turned into text. A list becomes its items joined by commas, which is why the sales order line's dedup keys are stored as order_id,line_no.

src/dataplatform/keys.py:18The fingerprint: the first 12 hex characters of a hash of the document with its keys sorted.
def config_fingerprint(config: dict[str, Any]) -> str:
    """A short, stable hash of the merged config a run used. Key order does not matter."""
    return hashlib.sha256(json.dumps(config, sort_keys=True, default=str).encode()).hexdigest()[:12]

Validation before any write

src/dataplatform/meta/loader.py:122Reading and checking the whole folder. Note the single raise at the end.
def compile_domain(manifest_dir: Path, defs: list[SettingDef]) -> Docs:
    """Read and validate a domain's manifest folder. Returns (domain doc, dataset docs, warnings)."""
    import yaml  # local and CI only

    platform = yaml.safe_load((manifest_dir / "platform.yaml").read_text())
    domain = platform["domain"]
    plain = json.loads(json.dumps(platform, default=str))
    domain_doc = {"domain": domain, "business_timezone": platform.get("business_timezone", "UTC"),
                  "environments": list(platform.get("environments") or []),
                  "settings": {k: _text(v) for k, v in (platform.get("settings") or {}).items()},
                  "manifest": plain, "fingerprint": config_fingerprint(plain)}
    errors: list[str] = []
    warnings: list[str] = []
    docs: list[dict[str, Any]] = []
    for path in sorted((manifest_dir / "datasets").glob("*.yaml")):
        manifest = yaml.safe_load(path.read_text())
        contract = yaml.safe_load((path.parent / manifest["contract"]).read_text())
        errors += [f"{path.name}: {e}" for e in validate(manifest, contract)]
        doc = dataset_doc(domain, manifest, contract)
        e, w = setting_problems(doc, defs)
        errors, warnings = errors + e, warnings + w
        errors += [f"{path.name}: {p}" for p in engine_problems(
            f"{doc['source_name']}.{doc['dataset_name']}", manifest, doc["settings"], domain_doc["settings"])]
        docs.append(doc)
    domain_keys = registry.by_key(defs, "platform") | registry.by_key(defs, "dataset")
    errors += [f"platform.yaml: setting '{k}' is not registered" for k in domain_doc["settings"]
               if k not in domain_keys]
    if errors:
        raise ManifestError("; ".join(errors))
    return domain_doc, docs, warnings

Walk through the loop. For each dataset file, the manifest and its contract are parsed. validate checks the required fields, that the contract version the manifest pins is the contract's version, and that the history order column is a contract column. setting_problems checks the settings against the dictionary. engine_problems checks that the chosen silver engine can handle the dataset. After the loop the platform file's own settings are checked. All errors are joined into one message, so a person fixing a folder sees every problem at once.

src/dataplatform/meta/loader.py:109An unknown key is an error. A value of the wrong type is a warning.
def setting_problems(doc: dict[str, Any], defs: list[SettingDef]) -> tuple[list[str], list[str]]:
    """(errors: unregistered keys, warnings: values that do not fit their type)."""
    known = registry.by_key(defs, "dataset")
    name = f"{doc['source_name']}.{doc['dataset_name']}"
    errors = [f"{name}: setting '{k}' is not registered" for k in doc["settings"] if k not in known]
    warnings = [f"{name}: {p}" for k, v in doc["settings"].items() if k in known
                for p in [registry.type_problem(known[k], v)] if p]
    return errors, warnings

Writing

src/dataplatform/meta/loader.py:154The only place that writes. Every store call comes after compile_domain has returned.
def load(store: MetadataStore, manifest_dir: Path, defs: list[SettingDef] | None = None,
         extra_settings: dict[str, dict[str, str]] | None = None) -> LoadReport:
    """Register the settings, then write the domain and every dataset. Validation happens before any write.

    `extra_settings` ({"source.dataset": {key: value}}) adds binding-level settings on top of the manifest, e.g. the
    delivery shape of a change-capture service (B4); every key must be registered, and the dataset's fingerprint
    covers them, so the run record shows which shape a run read."""
    defs = defs if defs is not None else registry.load()
    domain_doc, docs, warnings = compile_domain(manifest_dir, defs)
    known = registry.by_key(defs, "dataset")
    bad = [f"{name}: setting '{k}' is not registered" for name, extra in (extra_settings or {}).items()
           for k in extra if k not in known]
    if bad:
        raise ManifestError("; ".join(bad))
    for doc in docs:
        extra = (extra_settings or {}).get(f"{doc['source_name']}.{doc['dataset_name']}")
        if extra:
            doc["settings"] = {**doc["settings"], **extra}
            binding = dict(sorted(extra.items()))
            doc["fingerprint"] = config_fingerprint({"manifest": doc["fingerprint"], "binding": binding})
    store.register_settings(defs)
    store.load_domain(domain_doc)
    report = LoadReport(domain_doc["domain"], warnings=warnings)
    for doc in docs:
        rules = doc.pop("dq_rules", [])
        store.load_dataset(doc)
        store.set_dq_rules(doc["source_name"], doc["dataset_name"], rules)
        report.loaded.append(f"{doc['source_name']}.{doc['dataset_name']}")
    declared = set(report.loaded)
    report.not_in_manifest = sorted(f"{d.source}.{d.dataset}" for d in store.datasets(domain_doc["domain"])
                                    if f"{d.source}.{d.dataset}" not in declared)
    return report

The order of the writes follows the foreign keys: the dictionary first, because settings point at it, then the domain, then each dataset. A dataset's quality rules are written right after the dataset. The last lines compare what the store lists for the domain with what the folder declared. A dataset found only in the store is reported and left alone.

src/dataplatform/meta/loader.py:216Gold tables go into the same store as datasets, after the source datasets.
def load_gold(store: MetadataStore, manifest_dir: Path) -> list[str]:
    """Write every gold manifest (manifest/gold/*.yaml) into the store beside the source datasets, after `load`.
    A gold manifest's kind or contract version that does not match is refused before anything is written."""
    import yaml  # local and CI only

    domain = yaml.safe_load((manifest_dir / "platform.yaml").read_text())["domain"]
    docs, errors = [], []
    for path in sorted((manifest_dir / "gold").glob("*.yaml")):
        manifest = yaml.safe_load(path.read_text())
        contract = yaml.safe_load((path.parent / manifest["contract"]).read_text())
        if manifest.get("kind") not in GOLD_KINDS:
            errors.append(f"{path.name}: kind {manifest.get('kind')!r} is not one of {GOLD_KINDS}")
        if str(manifest.get("contract_version")) != str(contract["version"]):
            errors.append(f"{path.name}: contract_version {manifest.get('contract_version')} but the contract is "
                          f"{contract['version']}")
        docs.append(gold_doc(domain, manifest, contract))
    if errors:
        raise ManifestError("; ".join(errors))
    for doc in docs:
        store.load_dataset(doc)
    return [f"{d['source_name']}.{d['dataset_name']}" for d in docs]

A worked example: one mistake, nothing written

The table shows what happens when one file of the tyre folder is broken. The left column is the change, the middle column is the message, the right column is the state of the store afterwards.

Change to the folderMessageStore afterwards
retry_cuont: 3 added to the delivery manifestsetting 'retry_cuont' is not registeredEmpty. No domain, no dataset, no dictionary.
The press manifest pins contract version 9.9.9manifest pins contract 9.9.9 but the contract is another versionEmpty.
owners removed from the inspection manifestmissing field: ownersEmpty.
The press manifest file deleted after an earlier good loadNo error. The report lists mes.press as not in the manifest.The press dataset is still there.

The settings the loader reads

Manifest fieldWhat the loader does with itRequired
source_name, dataset_name, is_enabledIdentity and the enabled flag.Yes
configEach key becomes a dataset setting. Each key must be in the dictionary at dataset scope.Yes
dedupmode, business_keys and order_by become settings. The business keys also become the dedup key.No
contract, contract_versionThe path to the contract and the version it must have.Yes
ingest.mode, ingest.read, history.kind, load.patternCopied into typed columns of the dataset row.load is required
archetypes, reconciliation, ownersRequired to be present. Kept inside the manifest JSON.Yes
settings in platform.yamlEach key becomes a domain setting. Each key must be in the dictionary at dataset or platform scope.No

How it is tested

tests/dataplatform/test_meta_loader.py loads the real tyre folder into the in-memory store. It checks the nine datasets one by one, checks that loading twice gives the same fingerprints, and breaks one file at a time to prove that a bad folder leaves the store empty. A last test loads the folder into PostgreSQL as well and compares every dataset view between the two adapters.

tests/dataplatform/test_meta_loader.py:113The broken-folder cases: which file, what is changed, and the message that must appear.
        ("unregistered config key", "datasets/erp.delivery.yaml", lambda d: d["config"].update(retry_cuont="3"),
         "setting 'retry_cuont' is not registered"),
        ("contract version not the contract's", "datasets/mes.press.yaml", lambda d: d.update(contract_version="9.9.9"),
         "manifest pins contract 9.9.9"),
        ("required field missing", "datasets/lims.inspection_result.yaml", lambda d: d.pop("owners"),
         "missing field: owners"),
        ("history ordered by a column the contract lacks", "datasets/erp.sales_order_line.yaml",
         lambda d: d["load"].update(history_order_by="change_no"), "load.history_order_by 'change_no'"),
        ("unregistered domain setting", "platform.yaml", lambda d: d.update(settings={"timezone": "UTC"}),
         "platform.yaml: setting 'timezone' is not registered"),
tests/dataplatform/test_meta_loader.py:125The test body. The last line is the point: the store is still empty.
def test_bad_manifest_refused_before_any_write(tmp_path, case, file, change, message):
    shutil.copytree(TYRE.parent, tmp_path / "slice")
    _break(tmp_path / "slice/manifest" / file, change)
    store = MemoryMetaStore()
    with pytest.raises(ManifestError, match=message.replace(".", r"\.")):
        loader.load(store, tmp_path / "slice/manifest")
    assert (store.domains, store.ds, store.registry) == ({}, {}, {})
Not built yet, or known gaps
  • A wrong value type is only a warning. The value is written as it is.
  • A dataset removed from the folder stays in the store and stays enabled. The loader reports it and a person has to act.
  • The write phase is several function calls, each its own transaction. If the connection failed halfway, some datasets would be on the new configuration and some on the old. Running the loader again repairs it, because each write replaces a dataset whole.
  • load_gold checks less than the dataset path. It checks the kind and the contract version. Gold tables are stored with no settings, because no gold setting is in the dictionary yet.
  • There is no source-level manifest file, so nothing can declare a source's own settings.
  • The loader needs a YAML library, which is installed only for local runs and the build. The platform itself never parses YAML.