Data PlatformZoom level 2
Settings: the dictionary, the three levels, and how a value is chosen
A setting is one named value that changes how a dataset is handled, such as retry_count or watermark_column. The store keeps a dictionary of every setting that is allowed, and it works out the final value of each one for each dataset.
The dictionary exists because of typing mistakes. A manifest that says retyr_count: 5 would otherwise be accepted, ignored, and the dataset would quietly run with the default. Here it is refused before anything is written.
The dictionary, counted
The dictionary is the file src/dataplatform/meta/settings_registry.json. It is generated by a script from an older ingestion framework's key list plus the keys the specifications propose, and it must not be edited by hand. I counted its entries for this page.
| Scope | Entries | Of which proposed | What the scope is meant for |
|---|---|---|---|
| dataset | 135 | 21 | A value that can differ per dataset. |
| source | 43 | 0 | A value that belongs to a source system, such as a server name or an authentication method. |
| platform | 35 | 0 | A value for the whole platform, such as a default schedule. |
| Total | 213 | 21 |
Three details in the count matter. The source-scope key client_id appears twice in the file, so the table config.setting_def, whose primary key is scope and key, ends with 212 rows. Ten keys are registered at both platform scope and dataset scope: the eight processor_groups switches and the two lifecycle day counts. Of the 213 entries, 60 carry a default and 46 carry a list of allowed values.
src/dataplatform/meta/settings_registry.json:1201One entry as the file holds it. The lines just above this cut list the two allowed values, job and declarative."default": "job", "description": "Which engine builds silver for this dataset (ADR-035): job = our stages on job compute (default); declarative = the same stages inside a declarative pipeline, history by its managed change capture. The loader refuses a combination the engine cannot honour.", "key": "silver_engine", "scope": "dataset", "status": "proposed", "value_type": "string"
| Field | What it holds |
|---|---|
scope | platform, source or dataset. |
value_type | The file uses ten types. The most common are string with 98 entries, enum with 45, int with 34 and bool with 12. |
default | The value when nothing sets the key, or nothing at all. |
allowed | The permitted values of an enum. |
status | registered for a settled key, proposed for one the specifications ask for. Both are accepted by the loader. The status changes nothing at run time. |
src/dataplatform/adapters/sql/postgres_meta.sql:26The dictionary as a table.CREATE TABLE IF NOT EXISTS config.setting_def (
scope TEXT NOT NULL CHECK (scope IN ('platform', 'source', 'dataset')),
key TEXT NOT NULL,
value_type TEXT NOT NULL,
default_value TEXT,
allowed TEXT [],
description TEXT NOT NULL DEFAULT '',
status TEXT NOT NULL DEFAULT 'registered' CHECK (status IN ('registered', 'proposed')),
PRIMARY KEY (scope, key)
);Where a value can be set
| Level | Written in | Stored in | Wins over |
|---|---|---|---|
| Default | The dictionary file | config.setting_def.default_value | Nothing |
| Domain | The settings block of platform.yaml | config.domain_setting | The default |
| Dataset | The config and dedup blocks of a dataset manifest | config.dataset_setting | The domain value and the default |
How the final value is chosen
src/dataplatform/adapters/sql/postgres_meta.sql:438The whole merge is one query.-- The merged view (DPC-0002): every registered dataset key, the narrowest value winning, with where it came from.
CREATE OR REPLACE FUNCTION config.dataset_settings(p_source TEXT, p_dataset TEXT)
RETURNS TABLE (key TEXT, value TEXT, origin TEXT)
LANGUAGE plpgsql STABLE SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
DECLARE ds BIGINT := config._dataset_id(p_source, p_dataset);
BEGIN
RETURN QUERY
SELECT r.key, COALESCE(dv.value, gv.value, r.default_value),
CASE WHEN dv.value IS NOT NULL THEN 'dataset' WHEN gv.value IS NOT NULL THEN 'domain' ELSE 'default' END
FROM config.setting_def r
LEFT JOIN config.dataset_setting dv ON dv.dataset_id = ds AND dv.key = r.key
LEFT JOIN config.domain_setting gv ON gv.key = r.key AND gv.domain = (
SELECT s.domain FROM config.dataset d JOIN config.source s USING (source_id) WHERE d.dataset_id = ds)
WHERE r.scope = 'dataset'
ORDER BY r.key COLLATE "C";
END $$;Walk through it. The query starts from the dictionary and keeps only the rows whose scope is dataset. For each such key it looks for a row in config.dataset_setting for this dataset, and for a row in config.domain_setting for the dataset's domain. COALESCE returns the first of the three values that exists: dataset, domain, default. The CASE beside it names which one that was. The result has one row for every dataset key in the dictionary, including keys nobody set and that have no default. Their value is empty and their origin is default.
A worked example
These are four keys of the tyre company's sales order lines and its dealers, as the merge returns them. The columns are the dataset, the key, the final value and where the value came from.
| Dataset | Key | Value | Origin |
|---|---|---|---|
erp.sales_order_line | watermark_column | change_seq | dataset |
erp.sales_order_line | retry_count | 2 | default |
dealer_api.dealer | retry_count | 3 | dataset |
erp.delivery | csv_has_header | true | default |
No row in this example has the origin domain. The tyre company's data-platform-catalog/slices/02-tyre/manifest/platform.yaml has no settings block, so the domain level is empty for it. The domain level is exercised by the contract test, which loads a small made-up domain with one domain setting.
The view the pipelines read
No pipeline calls the merge directly. Pipelines call config.dataset_view, which returns one JSON document with everything about a dataset: its identity, the merged settings, the origin of each, the columns, the keys and the manifest.
src/dataplatform/adapters/sql/postgres_meta.sql:479The view is built in one statement.CREATE OR REPLACE FUNCTION config.dataset_view(p_source TEXT, p_dataset TEXT) RETURNS JSONB
LANGUAGE plpgsql STABLE SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
DECLARE ds BIGINT := config._dataset_id(p_source, p_dataset); out JSONB;
BEGIN
SELECT jsonb_build_object(
'domain', s.domain, 'source', s.source_name, 'dataset', d.dataset_name, 'source_env', s.source_env,
'is_enabled', d.is_enabled, 'kind', d.kind, 'archetypes', to_jsonb(d.archetypes),
'ingest_mode', d.ingest_mode, 'ingest_read', d.ingest_read, 'history_kind', d.history_kind,
'load_pattern', d.load_pattern, 'contract_id', d.contract_id, 'contract_version', d.contract_version,
'fingerprint', d.fingerprint,
'settings', (SELECT COALESCE(jsonb_object_agg(m.key, m.value), '{}')
FROM config.dataset_settings(p_source, p_dataset) m WHERE m.value IS NOT NULL),
'origins', (SELECT COALESCE(jsonb_object_agg(m.key, m.origin), '{}')
FROM config.dataset_settings(p_source, p_dataset) m WHERE m.value IS NOT NULL),
'columns', (SELECT COALESCE(jsonb_agg(jsonb_build_object(
'name', c.column_name, 'logical_type', c.logical_type, 'required', c.is_required,
'primary_key_position', c.primary_key_position, 'description', c.description,
'classification', c.classification)
ORDER BY c.ordinal), '[]')
FROM config.dataset_column c WHERE c.dataset_id = ds),
'business_keys', jsonb_build_object(
'primary', (SELECT COALESCE(jsonb_agg(b.column_name ORDER BY b.ordinal), '[]') FROM config.business_key b
WHERE b.dataset_id = ds AND b.key_kind = 'primary'),
'dedup', (SELECT COALESCE(jsonb_agg(b.column_name ORDER BY b.ordinal), '[]') FROM config.business_key b
WHERE b.dataset_id = ds AND b.key_kind = 'dedup')),
'manifest', d.manifest)
INTO out
FROM config.dataset d JOIN config.source s USING (source_id) WHERE d.dataset_id = ds;
RETURN out;
END $$;Two lines are worth a second look. The settings object keeps only the keys that have a value, so a key with no default that nobody set is absent from the view. The origins object has the same keys and says where each value came from. The document also carries the dataset's fingerprint, which the run record stores.
Refusing an unknown key
src/dataplatform/adapters/sql/postgres_meta.sql:362config.load_dataset: a key that is not a dataset-scope key in the dictionary stops the load with DPM01. FOR k IN SELECT jsonb_object_keys(COALESCE(p_doc -> 'settings', '{}')) LOOP
IF NOT EXISTS (SELECT 1 FROM config.setting_def WHERE scope = 'dataset' AND key = k) THEN
RAISE EXCEPTION 'setting ''%'' is not registered', k USING ERRCODE = 'DPM01';
END IF;
END LOOP;src/dataplatform/meta/registry.py:28The loader's check of a value against its type. A problem here is a warning, not a refusal.def type_problem(d: SettingDef, value: str) -> str | None:
"""Why a value does not fit its registered type, or None. The loader reports these; unknown keys it refuses."""
if d.allowed and value not in d.allowed:
return f"{d.key}={value!r} is not one of {', '.join(d.allowed)}"
if d.value_type == "int":
try:
int(value)
except ValueError:
return f"{d.key}={value!r} is not a whole number"
if d.value_type == "bool" and value.lower() not in ("true", "false"):
return f"{d.key}={value!r} is not true or false"
return NoneTwo gaps to know about
The dictionary registers 43 source-scope settings. The table config.source has four columns: an id, the domain, the name and the environment. There is no source_setting table, no function that stores a value for a source-scope key, and no manifest file for a source. A source-scope key written into a dataset manifest is refused, because the load accepts only dataset-scope keys. The merge has three levels in the specification and two in the code. The Python port says so itself.
src/dataplatform/meta/port.py:199The port's own note on the missing level. def settings(self, source: str, dataset: str) -> list[Setting]:
"""Every registered dataset key, merged: dataset over domain over the registry default. (DPC-0002 also names a
source level; DPM has no source manifest yet, so nothing feeds it.)"""config.load_domain accepts a key in platform.yaml when the dictionary has it at dataset scope or at platform scope, and stores it in config.domain_setting with the scope it found. The merge keeps only dictionary rows whose scope is dataset. A platform-scope value therefore sits in the table and no function returns it. For the ten keys that exist at both scopes the load picks the dataset scope, so those do take part in the merge.
src/dataplatform/adapters/sql/postgres_meta.sql:345config.load_domain: the scope is looked up among dataset and platform, and dataset sorts first. FOR k, v IN SELECT * FROM jsonb_each_text(COALESCE(p_doc -> 'settings', '{}')) LOOP
SELECT scope INTO sc FROM config.setting_def WHERE key = k AND scope IN ('dataset', 'platform')
ORDER BY scope LIMIT 1;
IF sc IS NULL THEN
RAISE EXCEPTION 'setting ''%'' is not registered', k USING ERRCODE = 'DPM01';
END IF;
INSERT INTO config.domain_setting (domain, scope, key, value) VALUES (p_doc ->> 'domain', sc, k, v);
END LOOP;Which settings the pipelines really read
The dictionary is much larger than what the code uses, because it was carried over from an older framework for a different cloud. Many of its keys describe connectors this platform does not have. The pages for each layer list the keys their code reads. For ingestion they are the file format keys, the three change feed column keys, file_pattern, watermark_column and the two retry keys. See Entry steps and Five pipelines.
How it is tested
tests/dataplatform/test_meta_store.py:150Four real cases from the contract table: each level winning in turn, and platform keys left out of the merge. ("merge: the dataset value wins", lambda s: setting(s, "retry_count"), ("5", "dataset")),
("merge: the domain value beats the default", lambda s: setting(s, "schedule_timezone"),
("Asia/Kolkata", "domain")),
("merge: the registry default fills the rest", lambda s: setting(s, "cost_center"), (None, "default")),
("merge: every registered dataset key, platform keys left out",
lambda s: [x.key for x in s.settings("erp", "orders")],
["cost_center", "load_type", "retry_count", "schedule_timezone"]),tests/dataplatform/test_meta_loader.py adds two things. One test checks each kind of type problem. Another rebuilds the dictionary with the export script and checks that the file in the repository is exactly what the script produces.
tests/dataplatform/test_meta_loader.py:137The type check's cases: a value, and the problem reported or none. ("load_type", "INCREMENTAL", None),
("load_type", "incremental", "load_type='incremental' is not one of FULL, INCREMENTAL"),
("retry_count", "3", None),
("retry_count", "three", "retry_count='three' is not a whole number"),
("csv_has_header", "TRUE", None),
("csv_has_header", "yes", "csv_has_header='yes' is not true or false"),
("cost_center", "anything", None),- Source-scope settings cannot be stored or merged. See gap 1 above.
- Platform-scope values stored at domain level are not returned by any function. See gap 2 above.
- A value that does not fit its type or its allowed list is reported as a warning by the loader and is still written. The store keeps every value as text and does not check it.
- The functions
config.dataset_settingsandconfig.settings_registryhave no caller in the platform code that I could find. Pipelines useconfig.dataset_view. Only the tests call the other two. - The duplicate
client_identry in the dictionary file is silently merged into one row. - The
quality_defaultsblock ofplatform.yamlis not a setting. It is kept inside the domain's manifest JSON and no function reads it back.