cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Metadata store › Settings and the merge

Data PlatformZoom level 2

Settings: the dictionary, the three levels, and how a value is chosen

A setting is one named value that changes how a dataset is handled, such as retry_count or watermark_column. The store keeps a dictionary of every setting that is allowed, and it works out the final value of each one for each dataset.

The dictionary exists because of typing mistakes. A manifest that says retyr_count: 5 would otherwise be accepted, ignored, and the dataset would quietly run with the default. Here it is refused before anything is written.

Registry defaultconfig.setting_defDomain valueconfig.domain_settingDataset valueconfig.dataset_settingconfig.dataset_settingsthe narrowest valuethat exists winskey, value, originone row per dataset key3rd2nd1st
The function looks at the dataset's own value first, then the domain's, then the default. It reports which of the three it used.

The dictionary, counted

The dictionary is the file src/dataplatform/meta/settings_registry.json. It is generated by a script from an older ingestion framework's key list plus the keys the specifications propose, and it must not be edited by hand. I counted its entries for this page.

ScopeEntriesOf which proposedWhat the scope is meant for
dataset13521A value that can differ per dataset.
source430A value that belongs to a source system, such as a server name or an authentication method.
platform350A value for the whole platform, such as a default schedule.
Total21321

Three details in the count matter. The source-scope key client_id appears twice in the file, so the table config.setting_def, whose primary key is scope and key, ends with 212 rows. Ten keys are registered at both platform scope and dataset scope: the eight processor_groups switches and the two lifecycle day counts. Of the 213 entries, 60 carry a default and 46 carry a list of allowed values.

src/dataplatform/meta/settings_registry.json:1201One entry as the file holds it. The lines just above this cut list the two allowed values, job and declarative.
  "default": "job",
  "description": "Which engine builds silver for this dataset (ADR-035): job = our stages on job compute (default); declarative = the same stages inside a declarative pipeline, history by its managed change capture. The loader refuses a combination the engine cannot honour.",
  "key": "silver_engine",
  "scope": "dataset",
  "status": "proposed",
  "value_type": "string"
FieldWhat it holds
scopeplatform, source or dataset.
value_typeThe file uses ten types. The most common are string with 98 entries, enum with 45, int with 34 and bool with 12.
defaultThe value when nothing sets the key, or nothing at all.
allowedThe permitted values of an enum.
statusregistered for a settled key, proposed for one the specifications ask for. Both are accepted by the loader. The status changes nothing at run time.
CREATE TABLE IF NOT EXISTS config.setting_def (
    scope TEXT NOT NULL CHECK (scope IN ('platform', 'source', 'dataset')),
    key TEXT NOT NULL,
    value_type TEXT NOT NULL,
    default_value TEXT,
    allowed TEXT [],
    description TEXT NOT NULL DEFAULT '',
    status TEXT NOT NULL DEFAULT 'registered' CHECK (status IN ('registered', 'proposed')),
    PRIMARY KEY (scope, key)
);

Where a value can be set

LevelWritten inStored inWins over
DefaultThe dictionary fileconfig.setting_def.default_valueNothing
DomainThe settings block of platform.yamlconfig.domain_settingThe default
DatasetThe config and dedup blocks of a dataset manifestconfig.dataset_settingThe domain value and the default

How the final value is chosen

-- The merged view (DPC-0002): every registered dataset key, the narrowest value winning, with where it came from.
CREATE OR REPLACE FUNCTION config.dataset_settings(p_source TEXT, p_dataset TEXT)
RETURNS TABLE (key TEXT, value TEXT, origin TEXT)
LANGUAGE plpgsql STABLE SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
DECLARE ds BIGINT := config._dataset_id(p_source, p_dataset);
BEGIN
    RETURN QUERY
    SELECT r.key, COALESCE(dv.value, gv.value, r.default_value),
           CASE WHEN dv.value IS NOT NULL THEN 'dataset' WHEN gv.value IS NOT NULL THEN 'domain' ELSE 'default' END
    FROM config.setting_def r
    LEFT JOIN config.dataset_setting dv ON dv.dataset_id = ds AND dv.key = r.key
    LEFT JOIN config.domain_setting gv ON gv.key = r.key AND gv.domain = (
        SELECT s.domain FROM config.dataset d JOIN config.source s USING (source_id) WHERE d.dataset_id = ds)
    WHERE r.scope = 'dataset'
    ORDER BY r.key COLLATE "C";
END $$;

Walk through it. The query starts from the dictionary and keeps only the rows whose scope is dataset. For each such key it looks for a row in config.dataset_setting for this dataset, and for a row in config.domain_setting for the dataset's domain. COALESCE returns the first of the three values that exists: dataset, domain, default. The CASE beside it names which one that was. The result has one row for every dataset key in the dictionary, including keys nobody set and that have no default. Their value is empty and their origin is default.

A worked example

These are four keys of the tyre company's sales order lines and its dealers, as the merge returns them. The columns are the dataset, the key, the final value and where the value came from.

DatasetKeyValueOrigin
erp.sales_order_linewatermark_columnchange_seqdataset
erp.sales_order_lineretry_count2default
dealer_api.dealerretry_count3dataset
erp.deliverycsv_has_headertruedefault

No row in this example has the origin domain. The tyre company's data-platform-catalog/slices/02-tyre/manifest/platform.yaml has no settings block, so the domain level is empty for it. The domain level is exercised by the contract test, which loads a small made-up domain with one domain setting.

The view the pipelines read

No pipeline calls the merge directly. Pipelines call config.dataset_view, which returns one JSON document with everything about a dataset: its identity, the merged settings, the origin of each, the columns, the keys and the manifest.

src/dataplatform/adapters/sql/postgres_meta.sql:479The view is built in one statement.
CREATE OR REPLACE FUNCTION config.dataset_view(p_source TEXT, p_dataset TEXT) RETURNS JSONB
LANGUAGE plpgsql STABLE SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
DECLARE ds BIGINT := config._dataset_id(p_source, p_dataset); out JSONB;
BEGIN
    SELECT jsonb_build_object(
        'domain', s.domain, 'source', s.source_name, 'dataset', d.dataset_name, 'source_env', s.source_env,
        'is_enabled', d.is_enabled, 'kind', d.kind, 'archetypes', to_jsonb(d.archetypes),
        'ingest_mode', d.ingest_mode, 'ingest_read', d.ingest_read, 'history_kind', d.history_kind,
        'load_pattern', d.load_pattern, 'contract_id', d.contract_id, 'contract_version', d.contract_version,
        'fingerprint', d.fingerprint,
        'settings', (SELECT COALESCE(jsonb_object_agg(m.key, m.value), '{}')
                     FROM config.dataset_settings(p_source, p_dataset) m WHERE m.value IS NOT NULL),
        'origins', (SELECT COALESCE(jsonb_object_agg(m.key, m.origin), '{}')
                    FROM config.dataset_settings(p_source, p_dataset) m WHERE m.value IS NOT NULL),
        'columns', (SELECT COALESCE(jsonb_agg(jsonb_build_object(
                        'name', c.column_name, 'logical_type', c.logical_type, 'required', c.is_required,
                        'primary_key_position', c.primary_key_position, 'description', c.description,
                        'classification', c.classification)
                        ORDER BY c.ordinal), '[]')
                    FROM config.dataset_column c WHERE c.dataset_id = ds),
        'business_keys', jsonb_build_object(
            'primary', (SELECT COALESCE(jsonb_agg(b.column_name ORDER BY b.ordinal), '[]') FROM config.business_key b
                        WHERE b.dataset_id = ds AND b.key_kind = 'primary'),
            'dedup', (SELECT COALESCE(jsonb_agg(b.column_name ORDER BY b.ordinal), '[]') FROM config.business_key b
                      WHERE b.dataset_id = ds AND b.key_kind = 'dedup')),
        'manifest', d.manifest)
    INTO out
    FROM config.dataset d JOIN config.source s USING (source_id) WHERE d.dataset_id = ds;
    RETURN out;
END $$;

Two lines are worth a second look. The settings object keeps only the keys that have a value, so a key with no default that nobody set is absent from the view. The origins object has the same keys and says where each value came from. The document also carries the dataset's fingerprint, which the run record stores.

Refusing an unknown key

src/dataplatform/adapters/sql/postgres_meta.sql:362config.load_dataset: a key that is not a dataset-scope key in the dictionary stops the load with DPM01.
    FOR k IN SELECT jsonb_object_keys(COALESCE(p_doc -> 'settings', '{}')) LOOP
        IF NOT EXISTS (SELECT 1 FROM config.setting_def WHERE scope = 'dataset' AND key = k) THEN
            RAISE EXCEPTION 'setting ''%'' is not registered', k USING ERRCODE = 'DPM01';
        END IF;
    END LOOP;
src/dataplatform/meta/registry.py:28The loader's check of a value against its type. A problem here is a warning, not a refusal.
def type_problem(d: SettingDef, value: str) -> str | None:
    """Why a value does not fit its registered type, or None. The loader reports these; unknown keys it refuses."""
    if d.allowed and value not in d.allowed:
        return f"{d.key}={value!r} is not one of {', '.join(d.allowed)}"
    if d.value_type == "int":
        try:
            int(value)
        except ValueError:
            return f"{d.key}={value!r} is not a whole number"
    if d.value_type == "bool" and value.lower() not in ("true", "false"):
        return f"{d.key}={value!r} is not true or false"
    return None

Two gaps to know about

Gap 1: source-scope settings have nowhere to live

The dictionary registers 43 source-scope settings. The table config.source has four columns: an id, the domain, the name and the environment. There is no source_setting table, no function that stores a value for a source-scope key, and no manifest file for a source. A source-scope key written into a dataset manifest is refused, because the load accepts only dataset-scope keys. The merge has three levels in the specification and two in the code. The Python port says so itself.

src/dataplatform/meta/port.py:199The port's own note on the missing level.
    def settings(self, source: str, dataset: str) -> list[Setting]:
        """Every registered dataset key, merged: dataset over domain over the registry default. (DPC-0002 also names a
        source level; DPM has no source manifest yet, so nothing feeds it.)"""
Gap 2: platform-scope values can be stored but are never returned

config.load_domain accepts a key in platform.yaml when the dictionary has it at dataset scope or at platform scope, and stores it in config.domain_setting with the scope it found. The merge keeps only dictionary rows whose scope is dataset. A platform-scope value therefore sits in the table and no function returns it. For the ten keys that exist at both scopes the load picks the dataset scope, so those do take part in the merge.

src/dataplatform/adapters/sql/postgres_meta.sql:345config.load_domain: the scope is looked up among dataset and platform, and dataset sorts first.
    FOR k, v IN SELECT * FROM jsonb_each_text(COALESCE(p_doc -> 'settings', '{}')) LOOP
        SELECT scope INTO sc FROM config.setting_def WHERE key = k AND scope IN ('dataset', 'platform')
        ORDER BY scope LIMIT 1;
        IF sc IS NULL THEN
            RAISE EXCEPTION 'setting ''%'' is not registered', k USING ERRCODE = 'DPM01';
        END IF;
        INSERT INTO config.domain_setting (domain, scope, key, value) VALUES (p_doc ->> 'domain', sc, k, v);
    END LOOP;

Which settings the pipelines really read

The dictionary is much larger than what the code uses, because it was carried over from an older framework for a different cloud. Many of its keys describe connectors this platform does not have. The pages for each layer list the keys their code reads. For ingestion they are the file format keys, the three change feed column keys, file_pattern, watermark_column and the two retry keys. See Entry steps and Five pipelines.

How it is tested

tests/dataplatform/test_meta_store.py:150Four real cases from the contract table: each level winning in turn, and platform keys left out of the merge.
    ("merge: the dataset value wins", lambda s: setting(s, "retry_count"), ("5", "dataset")),
    ("merge: the domain value beats the default", lambda s: setting(s, "schedule_timezone"),
     ("Asia/Kolkata", "domain")),
    ("merge: the registry default fills the rest", lambda s: setting(s, "cost_center"), (None, "default")),
    ("merge: every registered dataset key, platform keys left out",
     lambda s: [x.key for x in s.settings("erp", "orders")],
     ["cost_center", "load_type", "retry_count", "schedule_timezone"]),

tests/dataplatform/test_meta_loader.py adds two things. One test checks each kind of type problem. Another rebuilds the dictionary with the export script and checks that the file in the repository is exactly what the script produces.

tests/dataplatform/test_meta_loader.py:137The type check's cases: a value, and the problem reported or none.
        ("load_type", "INCREMENTAL", None),
        ("load_type", "incremental", "load_type='incremental' is not one of FULL, INCREMENTAL"),
        ("retry_count", "3", None),
        ("retry_count", "three", "retry_count='three' is not a whole number"),
        ("csv_has_header", "TRUE", None),
        ("csv_has_header", "yes", "csv_has_header='yes' is not true or false"),
        ("cost_center", "anything", None),
Not built yet, or known gaps
  • Source-scope settings cannot be stored or merged. See gap 1 above.
  • Platform-scope values stored at domain level are not returned by any function. See gap 2 above.
  • A value that does not fit its type or its allowed list is reported as a warning by the loader and is still written. The store keeps every value as text and does not check it.
  • The functions config.dataset_settings and config.settings_registry have no caller in the platform code that I could find. Pipelines use config.dataset_view. Only the tests call the other two.
  • The duplicate client_id entry in the dictionary file is silently merged into one row.
  • The quality_defaults block of platform.yaml is not a setting. It is kept inside the domain's manifest JSON and no function reads it back.