cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Metadata store › Functions

Data PlatformZoom level 2

Every function in the store, grouped by purpose

The functions are the store's whole public surface. This page lists every one. The main file src/dataplatform/adapters/sql/postgres_meta.sql defines 38. The operations file src/dataplatform/adapters/sql/postgres_obs.sql adds one more.

Reading them as a list is the fastest way to learn what the platform can and cannot do with its metadata. If an action has no function, no pipeline can perform it.

Every caller holds the same Python object. Each of its methods is one SELECT of one function.

What every function has in common

src/dataplatform/adapters/sql/postgres_meta.sql:300The helper nearly every function starts with. It turns a source and dataset name into an id, or refuses with DPM02.
CREATE OR REPLACE FUNCTION config._dataset_id(p_source TEXT, p_dataset TEXT) RETURNS BIGINT
LANGUAGE plpgsql STABLE SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
DECLARE v BIGINT;
BEGIN
    SELECT d.dataset_id INTO v FROM config.dataset d JOIN config.source s USING (source_id)
    WHERE s.source_name = p_source AND d.dataset_name = p_dataset;
    IF v IS NULL THEN
        RAISE EXCEPTION 'dataset %.% is not loaded', p_source, p_dataset USING ERRCODE = 'DPM02';
    END IF;
    RETURN v;
END $$;

Three things in the second line apply to all of them. SECURITY DEFINER makes the function run with its owner's rights, which is how a caller with no right on any table can still read and write through it. SET search_path pins the schema search to the system catalogue, so a caller cannot make the function pick up a table of the caller's own. Functions that only read are marked STABLE.

Group 1: writing configuration

FunctionWhat it doesWhat it refusesWho calls it
config.register_settings(defs)Writes the dictionary of allowed settings. An entry that exists is updated.A scope that is not platform, source or dataset.The loader.
config.load_domain(doc)Writes one domain row and replaces its domain settings.A setting key that is not in the dictionary at dataset or platform scope.The loader.
config.load_dataset(doc)Writes one dataset: creates the source row if needed, then replaces the dataset's settings, columns and keys whole.A domain that is not loaded. A key that is not a dataset-scope setting. A business key kind other than primary or dedup. A business key that names a column the dataset does not have.The loader, for source datasets and for gold tables.
config.set_dq_rules(source, dataset, rules)Replaces a dataset's quality rules. A rule that already has logged results is kept and updated, never dropped.A rule id twice. An empty rule id. A column the dataset does not have. An unknown severity or action.The loader.

Group 2: reading configuration

FunctionWhat it doesWhat it refusesWho calls it
config.dataset_view(source, dataset)Returns one JSON document with everything about a dataset: identity, merged settings and their origins, columns, keys, manifest, fingerprint.A dataset that is not loaded.Ingestion, silver, gold, housekeeping, retention and freshness. This is the function pipelines live on.
config.list_datasets(domain)Lists a domain's datasets with enabled flag, kind, ingest mode and fingerprint, in name order.Nothing. An unknown domain gives an empty list.Every runner, to decide what to run.
config.dataset_settings(source, dataset)Returns every dataset-scope key with its final value and origin.A dataset that is not loaded.config.dataset_view internally. No direct caller in the platform code that I found.
config.dataset_columns(source, dataset)Returns the contract's columns in order.A dataset that is not loaded.Gold.
config.business_keys(source, dataset, kind)Returns the columns of the primary or the dedup key, in position order.A kind other than primary or dedup.Gold.
config.dq_rules(source, dataset)Returns the dataset's quality rules by rule id.A dataset that is not loaded.Silver.
config.settings_registry()Returns the whole dictionary.Nothing.Only the tests, as far as I found.
config._dataset_id(source, dataset)Helper. Finds the dataset's id.A dataset that is not loaded.Other functions.

Group 3: opening and closing runs

FunctionWhat it doesWhat it refusesWho calls it
runtime.start_run(run_id, domain, env, trigger, run_type)Opens a run with status running.An unknown run type. A domain that is not loaded. A run id that was already used.Every runner, first thing.
runtime.start_dataset_run(run_id, source, dataset, fingerprint, config)Opens a dataset's record inside a run, saves the configuration snapshot, and returns the watermark as it stands.A run that is finished. A dataset started twice in the same run.Every runner, once per dataset.
runtime.finish_dataset_run(run_id, source, dataset, status, counts, watermark, bookmarks, message)Closes the dataset's record. Moves the watermark and bookmarks only when the status is succeeded.A status other than succeeded, failed or skipped. A dataset not started in this run. A dataset run already closed.Every runner, and the lease helper when a table is busy.
runtime.finish_run(run_id)Closes the run and works out its status from its dataset runs.A run already closed. A run that still has a dataset running.Every runner, last thing.
runtime._check_run, runtime._dataset_runHelpers. The first checks the run exists. The second checks the dataset was started in the run.DPM02 and DPM04.Other functions.
src/dataplatform/adapters/sql/postgres_meta.sql:694How a run's status follows its datasets: none failed is succeeded, all failed is failed, anything between is partial, and a run with no dataset is failed.
CREATE OR REPLACE FUNCTION runtime.finish_run(p_run_id TEXT) RETURNS TEXT
LANGUAGE plpgsql SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
DECLARE st TEXT; n INT; failed INT; running INT; final TEXT;
BEGIN
    PERFORM runtime._check_run(p_run_id);
    SELECT status INTO st FROM runtime.run WHERE run_id = p_run_id FOR UPDATE;
    IF st <> 'running' THEN
        RAISE EXCEPTION 'run % is already %', p_run_id, st USING ERRCODE = 'DPM04';
    END IF;
    SELECT count(*), count(*) FILTER (WHERE status = 'failed'), count(*) FILTER (WHERE status = 'running')
    INTO n, failed, running FROM runtime.dataset_run WHERE run_id = p_run_id;
    IF running > 0 THEN
        RAISE EXCEPTION 'run % still has datasets running', p_run_id USING ERRCODE = 'DPM04';
    END IF;
    final := CASE WHEN n = 0 THEN 'failed' WHEN failed = 0 THEN 'succeeded' WHEN failed = n THEN 'failed'
                  ELSE 'partial' END;
    UPDATE runtime.run SET status = final, ended_at = now() WHERE run_id = p_run_id;
    RETURN final;
END $$;

Group 4: logging what happened

FunctionWhat it doesWhat it refusesWho calls it
runtime.log_step(...)Appends one step row with a status, counts and a message.An unknown step status. A dataset not started in this run.Every runner and the lease helper.
runtime.log_file(...)Appends one file log row for a batch, whatever its outcome.A status that is not one of the five lander outcomes. An unknown load type. A dataset not started in this run.The ingestion steps, and silver's baseline writer.
runtime.log_error(...)Appends one error row. The dataset may be empty for an error that belongs to the whole run.A run that was never started.Ingestion, silver and gold.
runtime.log_dq_results(rows)Appends what each quality rule found, all rows or none.A rule the dataset does not have. More failed than checked. An unknown action.No caller in the platform code that I found. Only the tests.
runtime.save_profile(run_id, source, dataset, profile)Keeps a dataset run's profile, which is a summary of its rows and columns.A dataset not started in this run.The ingestion runner.
runtime._redact(message)Helper. Replaces secrets in a message and cuts it to 4000 characters.Nothing.Every function that stores a message.

Group 5: watermarks, bookmarks and batches

FunctionWhat it doesWhat it refusesWho calls it
runtime.get_watermark(source, dataset)Returns the dataset's watermark, which is the highest change value already taken.A dataset that is not loaded.The ingestion pipelines.
runtime.get_bookmarks(source, dataset)Returns the dataset's named positions, such as a page cursor or a stream position.A dataset that is not loaded.Ingestion, silver, gold and freshness.
runtime.reset_watermark(source, dataset, value, reason, changed_by)Sets the watermark by hand and writes an audit row with the old value, the new value, who and why.A reason shorter than 10 characters. No name of who made it.No caller in the platform code that I found. It is there for an operator.
runtime.batch_status(source, dataset, batch_key)Returns the latest status of a batch, or void if it was voided, or nothing if it was never seen.A dataset that is not loaded.The ingestion steps, to skip a batch already loaded.
runtime.dataset_files(source, dataset, window)Returns every file log row of a dataset, oldest first, with voided batches shown as void.A dataset that is not loaded.Ingestion, silver, gold, housekeeping and retention.
runtime.void_batch(source, dataset, batch_key, reason, approved_by)Marks a loaded batch as void, so every later read skips it.A batch that never loaded. A batch already void. No approver. A reason shorter than 10 characters.The recovery module and housekeeping.
src/dataplatform/adapters/sql/postgres_meta.sql:615A void wins over whatever the file log says. Otherwise the newest file log row of the batch key decides.
CREATE OR REPLACE FUNCTION runtime.batch_status(p_source TEXT, p_dataset TEXT, p_batch_key TEXT) RETURNS TEXT
LANGUAGE plpgsql STABLE SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
DECLARE ds BIGINT := config._dataset_id(p_source, p_dataset);
BEGIN
    IF EXISTS (SELECT 1 FROM runtime.batch_void v WHERE v.dataset_id = ds AND v.batch_key = p_batch_key) THEN
        RETURN 'void';
    END IF;
    RETURN (SELECT f.status FROM runtime.file_log f WHERE f.dataset_id = ds AND f.batch_key = p_batch_key
            ORDER BY f.file_id DESC LIMIT 1);
END $$;

Group 6: corrections and backfill plans

FunctionWhat it doesWhat it refusesWho calls it
runtime.record_corrections(plan, rows)Records a backfill plan and its corrections together. A correction says an old value was wrong. It is a row here and never an edit of silver.A plan id seen before. No reason or no author. No windows. A key that is not the dataset's primary key. A field that is not a column.The recovery module, and the simulated company when its sales team corrects a date.
runtime.corrections(source, dataset, as_of)Lists a dataset's corrections, each marked active once its plan's date is reached.A dataset that is not loaded.Gold.
runtime.backfill_plans(source, dataset)Lists a dataset's plans with the windows each one rebuilds.A dataset that is not loaded.Gold.

Group 7: leases

FunctionWhat it doesWhat it refusesWho calls it
runtime.acquire_lease(table, run_id, ttl_seconds)Takes the writer lease on a table. Returns nothing when this run holds it, otherwise the run that does.A time limit that is zero or negative.Ingestion, silver, gold and housekeeping, before they write a table.
runtime.release_lease(table, run_id)Gives the lease back. Only the holder's call has any effect.Nothing.The same callers, when they are done.

Group 8: reading the record back

FunctionWhat it doesWhat it refusesWho calls it
runtime.run_report(run_id)Returns everything recorded for one run as one JSON document: the run, its datasets, steps, files and errors.A run that was never started.Gold's inspection tool and the platform script.
runtime.dq_results(run_id, source, dataset)Returns the quality results of a run.A run that was never started.Only the tests, as far as I found.
runtime.last_profile(source, dataset)Returns the newest profile kept for the dataset.A dataset that is not loaded.The ingestion runner, to compare with the new profile.
runtime.ops_run_log(domain, since, until)Returns every run of a domain in a time span with its datasets, steps, files and errors. It is in the operations file.Nothing.The run log reader behind the operations page. See Run log.

The adapter: one method per function

src/dataplatform/adapters/postgres_meta.py:175A typical method. It holds no table name and no rule. It passes its arguments to the function.
    def finish_dataset_run(self, run_id: str, source: str, dataset: str, status: str,
                           counts: dict[str, int] | None = None, watermark: str | None = None,
                           bookmarks: dict[str, str] | None = None, message: str = "") -> None:
        self._call("SELECT runtime.finish_dataset_run(%s, %s, %s, %s, %s::jsonb, %s, %s::jsonb, %s)",
                   (run_id, source, dataset, status, self._json(counts or {}), watermark,
                    self._json(bookmarks or {}), message))

The connection is set to commit each statement on its own, so one function call is one transaction. That is what makes a function such as finish_dataset_run all or nothing.

A worked example: the calls for one dataset in one ingestion run

The table shows the order in which the ingestion run for the tyre company's material master calls the store. The left column is the function. The right column is what it is for at that moment.

FunctionPurpose at that moment
runtime.start_runOpen the run.
config.list_datasetsFind the enabled datasets of the domain.
config.dataset_viewRead the material dataset's merged configuration.
runtime.start_dataset_runOpen the dataset's record and save the configuration snapshot.
runtime.acquire_leaseMake sure no other run is writing the bronze table.
runtime.get_bookmarks, runtime.dataset_filesFind which full copies were already taken.
runtime.batch_statusCheck the new copy was not loaded before.
runtime.log_fileRecord the batch and its outcome.
runtime.log_step, runtime.save_profileRecord the counts and the profile.
runtime.finish_dataset_runClose the record and move the bookmarks if it succeeded.
runtime.release_lease, runtime.finish_runGive the lease back and close the run.

Run loop walks through the same sequence in the Python code.

The settings these functions read

The functions read no setting. Their limits are constants in the SQL: a reason must have at least 10 characters, a stored message is cut at 4000 characters, and a lease needs a positive time limit. The same constants are in src/dataplatform/meta/port.py so the in-memory adapter behaves the same way.

How it is tested

Each function has cases in the contract table of tests/dataplatform/test_meta_store.py, both for what it returns and for each refusal. The same cases run against the in-memory adapter, which is how the two are kept in step.

tests/dataplatform/test_meta_store.py:198Two refusal cases as the table writes them: an action and the exception expected.
    ("run: same run id twice refused", seq(started, lambda s: s.start_run("r1", "shop", "dev")), RunStateError),
    ("run: run type must be known", lambda s: s.start_run("r1", "shop", "dev", "schedule", "replay"),
     InvalidArgument),
Not built yet, or known gaps
  • Three functions have no caller in the platform code that I could find: runtime.reset_watermark, runtime.log_dq_results and runtime.dq_results. They are tested but not yet used by a pipeline or a tool. The table runtime.dq_result therefore stays empty in a normal run.
  • There is no function to reset or delete a bookmark, to delete a dataset, or to read the domain row.
  • There is no function that lists runs for a pipeline to inspect. The only reader across runs is runtime.ops_run_log in the operations file.
  • My list of callers comes from searching the source for each method name. A caller that reaches the store through a wrapper with a different name could be missing from it.