cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Metadata store

Data PlatformZoom level 1

The metadata store: configuration and run state in one database

Where this starts in the code
PartStarting pointCalled by (found by reading the code)
Load a domain's manifests into the storeload meta/loader.py:154main · realms/aws-databricks/meta_setup.py:79
The store's contract (every call the runners make)MetadataStore meta/port.py:181started from outside the code (a job or a command)

A metadata-driven platform is one where the code is the same for every dataset and the differences between datasets are data. The metadata store is the database that holds that data: which datasets exist, how each one is loaded, and what happened in every run.

Without it, each difference between datasets ends up as an if in a job, and the record of what ran ends up in log files that nobody can query. With it, adding a dataset is adding rows, and the question of what the platform did last night is one query.

The highlighted box is the gate. The loader and every pipeline pass through it. Click a box to zoom in.

The pages under this one

PageWhat it answers
Config data modelWhich configuration tables exist, every column in them, and how they point at each other.
Settings and the mergeWhich settings are allowed, where a value can be set, and how the final value is chosen.
FunctionsEvery function in the store, what it does, what it refuses and who calls it.
LoaderHow YAML files in git become rows, and why nothing is written until everything is checked.
Runtime tablesThe tables that record runs, files, errors, watermarks, bookmarks and leases, and the rules that keep them honest.

The design decisions

DecisionWhat it meansWhy
Configuration lives in a database, not in codeA dataset's settings, columns and keys are rows. The pipelines read them at run time through one merged view. They never open a YAML file.A run can record exactly which configuration it used. A setting can be queried, compared and audited. The same pipeline code serves every dataset.
Git is still the sourcePeople edit YAML manifests in git. A loader validates them and writes the rows. Nothing else writes configuration.A change to configuration gets a review and a history, like code.
Two schemasconfig holds what to load and how. runtime holds what happened.The two change at different speeds and are written by different actors. The loader writes config. Running pipelines write runtime.
Functions are the only way inEvery table is revoked from public access. A service role is granted the right to execute the functions and nothing else.The rules live in one place. A pipeline with a bug cannot move a watermark for a failed run, because the only function that moves a watermark checks the status first.
One adapter for any PostgreSQLThe same SQL file and the same Python adapter serve a local server, a managed server and the engine's managed PostgreSQL. Only the connection string changes.There is no second code path to test per product.
src/dataplatform/adapters/sql/postgres_meta.sql:10The header of the SQL file states the rule and the four refusal codes.
-- The functions below are the only way in (DPC-0033): they are SECURITY DEFINER, tables are revoked from PUBLIC,
-- and a service role gets USAGE on the schemas and EXECUTE on the functions only (see the end of this file).
-- Refusals use our own SQLSTATEs, which the adapter maps to the kernel's exceptions:
--   DPM01 unknown setting   DPM02 not found   DPM03 invalid argument   DPM04 run out of order

The four refusal codes

When a function refuses a call it raises an error with one of four codes of our own. The Python adapter turns each code into one exception class, so the calling code can tell a typing mistake from a run that is out of order.

CodeMeaningPython exceptionA real example from the SQL
DPM01Unknown settingUnknownSettingA manifest sets a key that is not in the dictionary of allowed settings.
DPM02Not foundNotFoundA dataset that is not loaded, a domain that is not loaded, a run that was never started.
DPM03InvalidInvalidArgumentA run type that is not normal, reload or backfill. A watermark reset whose reason is shorter than 10 characters.
DPM04Out of orderRunStateErrorStarting the same run twice. Finishing a run that still has a dataset running. Logging a step for a dataset that was not started in that run.
src/dataplatform/adapters/postgres_meta.py:40The adapter's map from refusal code to exception.
ERRORS: dict[str, type[MetaError]] = {"DPM01": UnknownSetting, "DPM02": NotFound, "DPM03": InvalidArgument,
                                      "DPM04": RunStateError}
src/dataplatform/adapters/postgres_meta.py:95Every call goes through this method. A refusal code becomes its exception. Any other database error is raised as it is.
    def _call(self, sql: str, args: tuple[Any, ...] = ()) -> list[tuple[Any, ...]]:
        import psycopg

        try:
            return list(self.conn.execute(sql, args).fetchall())
        except psycopg.Error as exc:
            kind = ERRORS.get(getattr(exc, "sqlstate", "") or "")
            if kind is None:
                raise
            raise kind(str(getattr(exc.diag, "message_primary", "") or exc)) from exc

A worked example: one call, one refusal

The tyre company's ingestion run starts the dataset run for the sales order lines and later closes it. Each line below is one function call the pipeline makes and what the store answers.

CallAnswer
runtime.start_run('ingest-2026-11-04', 'tyre', 'dev', 'schedule', 'normal')A new row in runtime.run with status running.
The same call againRefused with DPM04: the run was already started.
runtime.start_dataset_run(run, 'erp', 'sales_order_lines', ...) with the dataset name misspeltRefused with DPM02: the dataset is not loaded.
runtime.finish_dataset_run(run, 'erp', 'sales_order_line', 'done', ...)Refused with DPM03: the status is not succeeded, failed or skipped.

How the gate is granted

src/dataplatform/adapters/postgres_meta.py:65Installing the store. With a service role, the role gets the functions and no table.
def install(conn: Any, service_role: str | None = None) -> None:
    """Create or update the schemas, tables and functions (idempotent). With a service role, grant it the functions
    and nothing else: the role can call every procedure and read no table (DPC-0033)."""
    conn.execute(DDL.read_text())
    if service_role:
        r = _ident(service_role)
        for stmt in (f"GRANT USAGE ON SCHEMA config, runtime TO {r}",
                     f"REVOKE ALL ON ALL TABLES IN SCHEMA config, runtime FROM {r}",
                     f"GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA config, runtime TO {r}"):
            conn.execute(stmt)
    conn.commit()

The SQL file itself revokes every table from public access. The three grant statements for a service role are in this Python function, not in the SQL file. The header comment of the SQL file says to look at the end of that file for them, which is out of date.

The settings the store itself reads

NameWhat it meansDefault
DP_META_DSNEnvironment variable with the connection string of a plain PostgreSQL.dbname=dp_meta
DP_META_INSTANCEEnvironment variable naming a managed PostgreSQL instance of the engine. When set, the job connects as its own identity with a short-lived token.Not set
DP_META_DATABASEThe database name on that instance.dp_meta
src/dataplatform/adapters/lakebase.py:86The binding's choice comes from the environment. Either way the same adapter class is returned.
def meta_store_from_env(client: Any = None, connect: Any = None) -> Any:
    """The binding's choice, from the environment: DP_META_INSTANCE (+ DP_META_DATABASE) means Lakebase as this
    identity; otherwise DP_META_DSN (plain PostgreSQL, the laptop and other clouds); one adapter either way."""
    instance = os.environ.get("DP_META_INSTANCE")
    if instance:
        return connect_lakebase(instance, os.environ.get("DP_META_DATABASE", "dp_meta"), client, connect)
    return PostgresMetaStore.connect()

How it is tested

tests/dataplatform/test_meta_store.py holds one table of 112 cases. Every case runs against two adapters: the store in memory and the store on PostgreSQL. The PostgreSQL half is skipped when no local server answers. Two further tests prove the gate. The first reads the adapter's source and checks that every statement it sends is a call to a function. The second connects as a service role and checks that calling a function works and reading or writing a table is denied.

tests/dataplatform/test_meta_store.py:469The static half of the proof: no statement in the adapter names a table.
def test_adapter_only_calls_procedures():
    """DPC-0033, static: every statement the service adapter sends is a call to a config./runtime. function."""
    service = ADAPTER.read_text().split("class PostgresMetaStore")[1]  # install() and create_database() are setup
    calls = re.findall(r'"(SELECT[^"]*)"', service)
    assert calls, "found no statements"
    for sql in calls:
        assert re.fullmatch(r"SELECT (\* FROM )?(config|runtime)\.[a-z_]+\((%s(::jsonb)?(, )?)*\)", sql), sql
tests/dataplatform/test_meta_store.py:491The dynamic half: the five things a service role tries, and whether each is allowed.
        ("call a reader procedure", "SELECT count(*) FROM config.list_datasets('shop')", True),
        ("call a writer procedure", "SELECT runtime.start_run('r1', 'shop', 'dev', 'x', 'normal')", True),
        ("read a config table", "SELECT count(*) FROM config.dataset", False),
        ("read a runtime table", "SELECT count(*) FROM runtime.watermark", False),
        ("write a watermark directly", "INSERT INTO runtime.watermark (dataset_id, watermark_value) VALUES (1, 'x')",
         False),
Not built yet, or known gaps
  • The store has no function that returns the domain row. The domain's time zone, its environments and the whole platform manifest are stored, but a pipeline cannot read them back through the gate. The bindings pass the time zone in from the manifest files instead.
  • There is one real adapter, for PostgreSQL, and one in memory for tests. No adapter exists for another database family.
  • Schema changes are made by running the SQL file again. Two columns were added in place with ALTER TABLE lines in the file. There is no versioned migration tool. See Runtime tables.
  • I could not verify a run against the engine's managed PostgreSQL from the code alone. tests/dataplatform/test_lakebase.py tests the connection logic with fakes.

Look inside