Data PlatformZoom level 2
Every function in the store, grouped by purpose
The functions are the store's whole public surface. This page lists every one. The main file src/dataplatform/adapters/sql/postgres_meta.sql defines 38. The operations file src/dataplatform/adapters/sql/postgres_obs.sql adds one more.
Reading them as a list is the fastest way to learn what the platform can and cannot do with its metadata. If an action has no function, no pipeline can perform it.
What every function has in common
src/dataplatform/adapters/sql/postgres_meta.sql:300The helper nearly every function starts with. It turns a source and dataset name into an id, or refuses with DPM02.CREATE OR REPLACE FUNCTION config._dataset_id(p_source TEXT, p_dataset TEXT) RETURNS BIGINT
LANGUAGE plpgsql STABLE SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
DECLARE v BIGINT;
BEGIN
SELECT d.dataset_id INTO v FROM config.dataset d JOIN config.source s USING (source_id)
WHERE s.source_name = p_source AND d.dataset_name = p_dataset;
IF v IS NULL THEN
RAISE EXCEPTION 'dataset %.% is not loaded', p_source, p_dataset USING ERRCODE = 'DPM02';
END IF;
RETURN v;
END $$;Three things in the second line apply to all of them. SECURITY DEFINER makes the function run with its owner's rights, which is how a caller with no right on any table can still read and write through it. SET search_path pins the schema search to the system catalogue, so a caller cannot make the function pick up a table of the caller's own. Functions that only read are marked STABLE.
Group 1: writing configuration
| Function | What it does | What it refuses | Who calls it |
|---|---|---|---|
config.register_settings(defs) | Writes the dictionary of allowed settings. An entry that exists is updated. | A scope that is not platform, source or dataset. | The loader. |
config.load_domain(doc) | Writes one domain row and replaces its domain settings. | A setting key that is not in the dictionary at dataset or platform scope. | The loader. |
config.load_dataset(doc) | Writes one dataset: creates the source row if needed, then replaces the dataset's settings, columns and keys whole. | A domain that is not loaded. A key that is not a dataset-scope setting. A business key kind other than primary or dedup. A business key that names a column the dataset does not have. | The loader, for source datasets and for gold tables. |
config.set_dq_rules(source, dataset, rules) | Replaces a dataset's quality rules. A rule that already has logged results is kept and updated, never dropped. | A rule id twice. An empty rule id. A column the dataset does not have. An unknown severity or action. | The loader. |
Group 2: reading configuration
| Function | What it does | What it refuses | Who calls it |
|---|---|---|---|
config.dataset_view(source, dataset) | Returns one JSON document with everything about a dataset: identity, merged settings and their origins, columns, keys, manifest, fingerprint. | A dataset that is not loaded. | Ingestion, silver, gold, housekeeping, retention and freshness. This is the function pipelines live on. |
config.list_datasets(domain) | Lists a domain's datasets with enabled flag, kind, ingest mode and fingerprint, in name order. | Nothing. An unknown domain gives an empty list. | Every runner, to decide what to run. |
config.dataset_settings(source, dataset) | Returns every dataset-scope key with its final value and origin. | A dataset that is not loaded. | config.dataset_view internally. No direct caller in the platform code that I found. |
config.dataset_columns(source, dataset) | Returns the contract's columns in order. | A dataset that is not loaded. | Gold. |
config.business_keys(source, dataset, kind) | Returns the columns of the primary or the dedup key, in position order. | A kind other than primary or dedup. | Gold. |
config.dq_rules(source, dataset) | Returns the dataset's quality rules by rule id. | A dataset that is not loaded. | Silver. |
config.settings_registry() | Returns the whole dictionary. | Nothing. | Only the tests, as far as I found. |
config._dataset_id(source, dataset) | Helper. Finds the dataset's id. | A dataset that is not loaded. | Other functions. |
Group 3: opening and closing runs
| Function | What it does | What it refuses | Who calls it |
|---|---|---|---|
runtime.start_run(run_id, domain, env, trigger, run_type) | Opens a run with status running. | An unknown run type. A domain that is not loaded. A run id that was already used. | Every runner, first thing. |
runtime.start_dataset_run(run_id, source, dataset, fingerprint, config) | Opens a dataset's record inside a run, saves the configuration snapshot, and returns the watermark as it stands. | A run that is finished. A dataset started twice in the same run. | Every runner, once per dataset. |
runtime.finish_dataset_run(run_id, source, dataset, status, counts, watermark, bookmarks, message) | Closes the dataset's record. Moves the watermark and bookmarks only when the status is succeeded. | A status other than succeeded, failed or skipped. A dataset not started in this run. A dataset run already closed. | Every runner, and the lease helper when a table is busy. |
runtime.finish_run(run_id) | Closes the run and works out its status from its dataset runs. | A run already closed. A run that still has a dataset running. | Every runner, last thing. |
runtime._check_run, runtime._dataset_run | Helpers. The first checks the run exists. The second checks the dataset was started in the run. | DPM02 and DPM04. | Other functions. |
src/dataplatform/adapters/sql/postgres_meta.sql:694How a run's status follows its datasets: none failed is succeeded, all failed is failed, anything between is partial, and a run with no dataset is failed.CREATE OR REPLACE FUNCTION runtime.finish_run(p_run_id TEXT) RETURNS TEXT
LANGUAGE plpgsql SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
DECLARE st TEXT; n INT; failed INT; running INT; final TEXT;
BEGIN
PERFORM runtime._check_run(p_run_id);
SELECT status INTO st FROM runtime.run WHERE run_id = p_run_id FOR UPDATE;
IF st <> 'running' THEN
RAISE EXCEPTION 'run % is already %', p_run_id, st USING ERRCODE = 'DPM04';
END IF;
SELECT count(*), count(*) FILTER (WHERE status = 'failed'), count(*) FILTER (WHERE status = 'running')
INTO n, failed, running FROM runtime.dataset_run WHERE run_id = p_run_id;
IF running > 0 THEN
RAISE EXCEPTION 'run % still has datasets running', p_run_id USING ERRCODE = 'DPM04';
END IF;
final := CASE WHEN n = 0 THEN 'failed' WHEN failed = 0 THEN 'succeeded' WHEN failed = n THEN 'failed'
ELSE 'partial' END;
UPDATE runtime.run SET status = final, ended_at = now() WHERE run_id = p_run_id;
RETURN final;
END $$;Group 4: logging what happened
| Function | What it does | What it refuses | Who calls it |
|---|---|---|---|
runtime.log_step(...) | Appends one step row with a status, counts and a message. | An unknown step status. A dataset not started in this run. | Every runner and the lease helper. |
runtime.log_file(...) | Appends one file log row for a batch, whatever its outcome. | A status that is not one of the five lander outcomes. An unknown load type. A dataset not started in this run. | The ingestion steps, and silver's baseline writer. |
runtime.log_error(...) | Appends one error row. The dataset may be empty for an error that belongs to the whole run. | A run that was never started. | Ingestion, silver and gold. |
runtime.log_dq_results(rows) | Appends what each quality rule found, all rows or none. | A rule the dataset does not have. More failed than checked. An unknown action. | No caller in the platform code that I found. Only the tests. |
runtime.save_profile(run_id, source, dataset, profile) | Keeps a dataset run's profile, which is a summary of its rows and columns. | A dataset not started in this run. | The ingestion runner. |
runtime._redact(message) | Helper. Replaces secrets in a message and cuts it to 4000 characters. | Nothing. | Every function that stores a message. |
Group 5: watermarks, bookmarks and batches
| Function | What it does | What it refuses | Who calls it |
|---|---|---|---|
runtime.get_watermark(source, dataset) | Returns the dataset's watermark, which is the highest change value already taken. | A dataset that is not loaded. | The ingestion pipelines. |
runtime.get_bookmarks(source, dataset) | Returns the dataset's named positions, such as a page cursor or a stream position. | A dataset that is not loaded. | Ingestion, silver, gold and freshness. |
runtime.reset_watermark(source, dataset, value, reason, changed_by) | Sets the watermark by hand and writes an audit row with the old value, the new value, who and why. | A reason shorter than 10 characters. No name of who made it. | No caller in the platform code that I found. It is there for an operator. |
runtime.batch_status(source, dataset, batch_key) | Returns the latest status of a batch, or void if it was voided, or nothing if it was never seen. | A dataset that is not loaded. | The ingestion steps, to skip a batch already loaded. |
runtime.dataset_files(source, dataset, window) | Returns every file log row of a dataset, oldest first, with voided batches shown as void. | A dataset that is not loaded. | Ingestion, silver, gold, housekeeping and retention. |
runtime.void_batch(source, dataset, batch_key, reason, approved_by) | Marks a loaded batch as void, so every later read skips it. | A batch that never loaded. A batch already void. No approver. A reason shorter than 10 characters. | The recovery module and housekeeping. |
src/dataplatform/adapters/sql/postgres_meta.sql:615A void wins over whatever the file log says. Otherwise the newest file log row of the batch key decides.CREATE OR REPLACE FUNCTION runtime.batch_status(p_source TEXT, p_dataset TEXT, p_batch_key TEXT) RETURNS TEXT
LANGUAGE plpgsql STABLE SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
DECLARE ds BIGINT := config._dataset_id(p_source, p_dataset);
BEGIN
IF EXISTS (SELECT 1 FROM runtime.batch_void v WHERE v.dataset_id = ds AND v.batch_key = p_batch_key) THEN
RETURN 'void';
END IF;
RETURN (SELECT f.status FROM runtime.file_log f WHERE f.dataset_id = ds AND f.batch_key = p_batch_key
ORDER BY f.file_id DESC LIMIT 1);
END $$;Group 6: corrections and backfill plans
| Function | What it does | What it refuses | Who calls it |
|---|---|---|---|
runtime.record_corrections(plan, rows) | Records a backfill plan and its corrections together. A correction says an old value was wrong. It is a row here and never an edit of silver. | A plan id seen before. No reason or no author. No windows. A key that is not the dataset's primary key. A field that is not a column. | The recovery module, and the simulated company when its sales team corrects a date. |
runtime.corrections(source, dataset, as_of) | Lists a dataset's corrections, each marked active once its plan's date is reached. | A dataset that is not loaded. | Gold. |
runtime.backfill_plans(source, dataset) | Lists a dataset's plans with the windows each one rebuilds. | A dataset that is not loaded. | Gold. |
Group 7: leases
| Function | What it does | What it refuses | Who calls it |
|---|---|---|---|
runtime.acquire_lease(table, run_id, ttl_seconds) | Takes the writer lease on a table. Returns nothing when this run holds it, otherwise the run that does. | A time limit that is zero or negative. | Ingestion, silver, gold and housekeeping, before they write a table. |
runtime.release_lease(table, run_id) | Gives the lease back. Only the holder's call has any effect. | Nothing. | The same callers, when they are done. |
Group 8: reading the record back
| Function | What it does | What it refuses | Who calls it |
|---|---|---|---|
runtime.run_report(run_id) | Returns everything recorded for one run as one JSON document: the run, its datasets, steps, files and errors. | A run that was never started. | Gold's inspection tool and the platform script. |
runtime.dq_results(run_id, source, dataset) | Returns the quality results of a run. | A run that was never started. | Only the tests, as far as I found. |
runtime.last_profile(source, dataset) | Returns the newest profile kept for the dataset. | A dataset that is not loaded. | The ingestion runner, to compare with the new profile. |
runtime.ops_run_log(domain, since, until) | Returns every run of a domain in a time span with its datasets, steps, files and errors. It is in the operations file. | Nothing. | The run log reader behind the operations page. See Run log. |
The adapter: one method per function
src/dataplatform/adapters/postgres_meta.py:175A typical method. It holds no table name and no rule. It passes its arguments to the function. def finish_dataset_run(self, run_id: str, source: str, dataset: str, status: str,
counts: dict[str, int] | None = None, watermark: str | None = None,
bookmarks: dict[str, str] | None = None, message: str = "") -> None:
self._call("SELECT runtime.finish_dataset_run(%s, %s, %s, %s, %s::jsonb, %s, %s::jsonb, %s)",
(run_id, source, dataset, status, self._json(counts or {}), watermark,
self._json(bookmarks or {}), message))The connection is set to commit each statement on its own, so one function call is one transaction. That is what makes a function such as finish_dataset_run all or nothing.
A worked example: the calls for one dataset in one ingestion run
The table shows the order in which the ingestion run for the tyre company's material master calls the store. The left column is the function. The right column is what it is for at that moment.
| Function | Purpose at that moment |
|---|---|
runtime.start_run | Open the run. |
config.list_datasets | Find the enabled datasets of the domain. |
config.dataset_view | Read the material dataset's merged configuration. |
runtime.start_dataset_run | Open the dataset's record and save the configuration snapshot. |
runtime.acquire_lease | Make sure no other run is writing the bronze table. |
runtime.get_bookmarks, runtime.dataset_files | Find which full copies were already taken. |
runtime.batch_status | Check the new copy was not loaded before. |
runtime.log_file | Record the batch and its outcome. |
runtime.log_step, runtime.save_profile | Record the counts and the profile. |
runtime.finish_dataset_run | Close the record and move the bookmarks if it succeeded. |
runtime.release_lease, runtime.finish_run | Give the lease back and close the run. |
Run loop walks through the same sequence in the Python code.
The settings these functions read
The functions read no setting. Their limits are constants in the SQL: a reason must have at least 10 characters, a stored message is cut at 4000 characters, and a lease needs a positive time limit. The same constants are in src/dataplatform/meta/port.py so the in-memory adapter behaves the same way.
How it is tested
Each function has cases in the contract table of tests/dataplatform/test_meta_store.py, both for what it returns and for each refusal. The same cases run against the in-memory adapter, which is how the two are kept in step.
tests/dataplatform/test_meta_store.py:198Two refusal cases as the table writes them: an action and the exception expected. ("run: same run id twice refused", seq(started, lambda s: s.start_run("r1", "shop", "dev")), RunStateError),
("run: run type must be known", lambda s: s.start_run("r1", "shop", "dev", "schedule", "replay"),
InvalidArgument),- Three functions have no caller in the platform code that I could find:
runtime.reset_watermark,runtime.log_dq_resultsandruntime.dq_results. They are tested but not yet used by a pipeline or a tool. The tableruntime.dq_resulttherefore stays empty in a normal run. - There is no function to reset or delete a bookmark, to delete a dataset, or to read the domain row.
- There is no function that lists runs for a pipeline to inspect. The only reader across runs is
runtime.ops_run_login the operations file. - My list of callers comes from searching the source for each method name. A caller that reaches the store through a wrapper with a different name could be missing from it.