cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Semantic layer

Data PlatformZoom level 1

The semantic layer: a metric defined once and checked against a known answer

Where this starts in the code
PartStarting pointCalled by (found by reading the code)
Read a metric definitionmetric_view semantic/metrics.py:151build · realms/databricks/semantic_governance/generate.py:110
gold_metrics · scripts/run_platform.py:292
Ask the known questions of built goldanswers semantic/answers.py:36gold_metrics · scripts/run_platform.py:293

Gold holds the rows. The semantic layer holds the meaning: what exactly is the on-time-in-full percentage, which columns it is computed from, and by which dimensions it may be cut. A metric is one such named calculation. It is defined once, in a metric file, and everything that shows it is generated from that file.

Without this, the same word ends up with two formulas. One dashboard counts cancelled orders and another does not, and nobody can say which number is right. Here a metric cannot be certified until it comes with a known answer: a number, at a stated place and time, that the simulated tyre company knows to be true.

Metric fileone permetric viewRead and lintrefuses abad definitionCertified viewcertifiedmeasures onlyPreview viewdraft and reviewedmeasures tooKnown answersasked of goldDashboardsone per domainQuestion spacecertified onlyOpen: GoldGoldfacts anddimensionszoom in ›Everything to the right of the highlighted box is generated. Nothing there is written by hand.
Read it left to right. The highlighted box is the gate. The dashed box holds measures that are not certified yet, and nothing on the right reads it.

The steps

  1. Read the metric file. Each file names one gold fact as its source, the dimensions it joins, the dimensions a reader may cut by, and the measures. A source that is not a gold fact, or a join the fact does not have, is refused.
  2. Lint. Linting is checking a definition against rules before using it. The rules cover the view's name, the lifecycle state of each measure, a known answer on every certified measure, and one meaning per word. Any problem stops generation.
  3. Split by certification. Certified measures go into the view that carries the file's name. Measures that are still draft or reviewed go into a second view with the suffix _preview. Deprecated measures go nowhere.
  4. Ask the known answers. Each certified measure's known answer is turned into a query and asked of gold. The result must equal the oracle of the same run.
  5. Generate what people use. One dashboard per domain and one question space are derived from the certified views only.
  6. Render for the product. A separate folder turns all of the above into files for Databricks: view definitions, the known-answer query, dashboard files, the question space, catalog tags, grants and masks.

The first five steps live in src/dataplatform/semantic and name no product. Only the last step knows about Databricks. This is the same split as everywhere else in the platform: a kernel that holds the rules, and a realm that renders them for one product.

A metric file

data-platform-catalog/slices/02-tyre/manifest/metrics/mv_otif.yaml:3The metric file for on time in full: one source fact, two joins, three dimensions, two measures.
kind: metric_view
name: mv_otif
domain: sales
source: fact_otif
joins: [dim_dealer, dim_material]
dimensions:
  - {name: Promised month, expr: "DATE_TRUNC('MONTH', promised_date)"}
  - {name: Region, expr: dim_dealer.region}
  - {name: Tyre class, expr: dim_material.tyre_class}
measures:
  - name: OTIF %
    expr: SUM(on_time_in_full) FILTER (WHERE _is_complete) / COUNT(1) FILTER (WHERE _is_complete)
    additive: false
    time_dimension: Promised month
    description: Share of open order lines (not cancelled or rejected) delivered by the date first promised and in full, over complete windows only (a line not yet due is neither on time nor missed; W1d finding, 5 Nov).
    certification: certified
    synonyms: [on-time delivery, on time in full, OTIF, delivery performance]
    known_answer: {where: "region = 'south' AND promised month = 2026-11", value: 0.8961, tolerance: 0.0005}
  - name: Order lines
    expr: COUNT(1)
    additive: true
    time_dimension: Promised month
    certification: certified
    synonyms: [lines, order lines]
    known_answer: {where: "promised month = 2026-11", value: 107188}

Read the first measure. Its expression counts lines that were on time and in full, over the rows whose window is complete, and divides by the number of such rows. It is not additive, which means it cannot be summed across months. Its time dimension is the promised month. Its synonyms are the other words people use for it. Its known answer says that for the south region in November 2026 the value is 0.8961, give or take 0.0005.

The keys of a metric file

The table lists each key, what it means, and its allowed values and default, as metric_view and lint read them.

KeyWhat it meansAllowed values and default
kindMarks the file as a metric view.metric_view. Anything else is refused.
nameThe view's name.Must start with mv_ and be lower snake case.
domainThe domain that owns the view. It decides the gold schema.Required.
sourceThe gold fact the measures are computed over.Must be a gold manifest whose name starts with fact_.
joinsThe gold dimensions joined to the fact.Each must be a dimension of the source fact. The join key comes from the fact's manifest, not from this file.
dimensions[].name, .exprA name a reader cuts by, and the expression behind it.An expression that reads dim_x.column is marked as coming from that dimension.
measures[].name, .exprThe metric's name and its formula.A name may be defined in one view only.
measures[].additiveWhether the measure may be summed across its dimensions.True or false. Default false.
measures[].time_dimensionThe dimension that is the measure's time axis.Must be one of the view's dimensions.
measures[].certificationWhere the measure is in its life.draft, reviewed, certified or deprecated. Default draft.
measures[].synonymsOther words for the measure.Among certified measures, one word may mean one measure only.
measures[].known_answer.whereThe place and time of the known answer.Dimension equals value, joined by AND. A value is quoted text, a month as 2026-11, or a day as 2026-11-12.
measures[].known_answer.value, .toleranceThe number, and how far a result may be from it.Tolerance defaults to 0. Required on a certified measure.
owners.datasetThe team that owns the view.Required.

Reading and linting

src/dataplatform/semantic/metrics.py:151One metric file becomes one record. The join keys are looked up in the gold manifests.
def metric_view(doc: Mapping[str, Any], gold: Mapping[str, Mapping[str, Any]]) -> MetricView:
    """One metric file -> a metric view. `gold` maps gold manifest names (facts and dimensions) to their documents;
    each join's key and schema come from there, so the metric file names a dimension and nothing more."""
    if doc.get("kind") != "metric_view":
        raise MetricError(f"{doc.get('name')}: kind is not metric_view")
    source = str(doc["source"])
    fact = gold.get(source)
    if fact is None:
        raise MetricError(f"{doc['name']}: source '{source}' is not a gold manifest")
    keys = {str(d["name"]): str(d["key"]) for d in fact.get("dimensions") or ()}
    joins = []
    for j in doc.get("joins") or ():
        if j not in keys or j not in gold:
            raise MetricError(f"{doc['name']}: join '{j}' is not a dimension of {source}")
        joins.append(Join(str(j), keys[j], str(gold[j]["domain"])))
    time_dims = {str(m.get("time_dimension")) for m in doc["measures"] if m.get("time_dimension")}
    dims = tuple(_dimension(d, time_dims) for d in doc["dimensions"])
    return MetricView(
        name=str(doc["name"]),
        domain=str(doc["domain"]),
        source=source,
        source_domain=str(fact["domain"]),
        joins=tuple(joins),
        dimensions=dims,
        measures=tuple(_measure(m, dims) for m in doc["measures"]),
        owner=str((doc.get("owners") or {}).get("dataset", "")),
    )

The function takes the file and the gold manifests. It looks the source fact up and refuses a source that gold does not have. For each join it takes the key column from the fact's own manifest, so the metric file only names a dimension and can never state a wrong key. It then builds the dimensions and the measures.

src/dataplatform/semantic/metrics.py:183Every rule a set of metric views must pass before anything is generated.
def lint(views: Iterable[MetricView]) -> list[str]:
    """Problems that stop generation. Empty means every view may be published."""
    views = list(views)
    errors: list[str] = []
    seen_measure: dict[str, str] = {}
    seen_word: dict[str, str] = {}
    for v in views:
        if not VIEW_NAME.match(v.name):
            errors.append(f"{v.name}: metric view names are mv_ + lower snake case (ADR-012, NAME-01)")
        if not FACT_NAME.match(v.source):
            errors.append(f"{v.name}: source '{v.source}' is not a fact_ table")
        errors += [f"{v.name}: join '{j.name}' is not a dim_ table" for j in v.joins if not DIM_NAME.match(j.name)]
        if not v.owner:
            errors.append(f"{v.name}: no owner")
        dim_names = {d.name for d in v.dimensions}
        for m in v.measures:
            where = f"{v.name}.{m.name}"
            if m.certification not in LIFECYCLE:
                errors.append(f"{where}: certification '{m.certification}' is not one of {', '.join(LIFECYCLE)}")
            if m.time_dimension not in dim_names:
                errors.append(f"{where}: time dimension '{m.time_dimension}' is not a dimension of the view")
            if m.certification == "certified" and m.known_answer is None:
                errors.append(f"{where}: certified without a known-answer test (ADR-022, SL-SEM-07)")
            if m.name in seen_measure:
                errors.append(f"{where}: also defined in {seen_measure[m.name]} (SERV-04: defined once)")
                continue  # its words would only repeat the same finding
            seen_measure[m.name] = v.name
            if m.certification != "certified":
                continue
            for word in (m.name, *m.synonyms):
                key = word.lower()
                if key in seen_word and seen_word[key] != where:
                    errors.append(f"{where}: '{word}' already means {seen_word[key]} (one word, one metric)")
                seen_word.setdefault(key, where)
    return errors

The rules are in order: the view's name, the source's name, each join's name, an owner, then for each measure a valid lifecycle state, a time dimension the view has, and a known answer if it is certified. The last two rules look across all views. A measure name that appears in two views is an error, because a metric is defined once. Among certified measures, a name or synonym that already means another measure is an error, because one word must mean one metric.

Certified view and preview

src/dataplatform/semantic/metrics.py:223Which views are created from one metric file.
def published(view: MetricView) -> list[MetricView]:
    """The views to create: the certified view under the manifest's name (only certified measures; what dashboards
    and the conversation interface read, SERV-06), and, if any measure is draft or reviewed, a _preview view with
    every measure that is not deprecated, tagged with its lowest state. A view with no certified measure publishes
    only its preview."""
    live = tuple(m for m in view.measures if m.certification != "deprecated")
    certified = tuple(m for m in live if m.certification == "certified")
    out = []
    if certified:
        out.append(replace(view, measures=certified, certified_only=True))
    if len(certified) < len(live):
        out.append(replace(view, name=view.name + PREVIEW_SUFFIX, measures=live, certified_only=False))
    return out

The table shows what this does to the three tyre metric files. Each row is one file. The middle column is the view that dashboards and the question space read. The right column is the preview, which holds every measure that is not deprecated.

Metric fileCertified view and its measuresPreview view and its measures
mv_otifmv_otif: OTIF %, Order linesnone, because every measure is certified
mv_salesmv_sales: Sell-in units, Sell-out unitsmv_sales_preview: the two above and Net sales value, which is a draft
mv_scrapmv_scrap: Scrap rate %mv_scrap_preview: Scrap rate % and Tyres cured, which is reviewed

Known answers: one number everywhere

A known answer is a small test that travels with the metric. It says where to look and what number must be there. The same known answer is used three ways: it is looked up in the oracle, which is the simulated company's own record of what is true, it is asked of gold, and it is asked of the view on the engine. All three must agree.

src/dataplatform/semantic/metrics.py:101The text of a known answer's place and time becomes a list of filters on the view's dimensions.
def parse_where(where: str, dimensions: Iterable[Dimension]) -> tuple[Filter, ...]:
    """'region = 'south' AND promised month = 2026-11' -> filters on the view's dimensions (names ignore case)."""
    by_lower = {d.name.lower(): d.name for d in dimensions}
    filters = []
    for part in re.split(r"\s+AND\s+", where.strip(), flags=re.IGNORECASE):
        lhs, sep, rhs = part.partition("=")
        if not sep:
            raise MetricError(f"known answer filter '{part}' is not 'dimension = value'")
        dim = by_lower.get(lhs.strip().lower())
        if dim is None:
            raise MetricError(f"known answer filter names unknown dimension '{lhs.strip()}'")
        raw = rhs.strip()
        quoted = _QUOTED.match(raw)
        if quoted:
            filters.append(Filter(dim, quoted.group(1), "equals"))
        elif _DAY.match(raw):
            filters.append(Filter(dim, raw, "day"))
        elif _MONTH.match(raw):
            filters.append(Filter(dim, raw, "month"))
        else:
            raise MetricError(f"known answer value '{raw}' is neither 'text', YYYY-MM nor YYYY-MM-DD")
    return tuple(filters)
src/dataplatform/semantic/metrics.py:241One known answer as a test case, and the address under which the oracle keeps the same number.
@dataclass(frozen=True)
class KnownAnswerCase:
    view: str
    measure: str
    time_dimension: str
    answer: KnownAnswer

    @property
    def label(self) -> str:
        return f"{self.view} {self.measure} " + " ".join(f.value for f in self.answer.filters)

    @property
    def oracle_key(self) -> str:
        """The world oracle's address for this answer: the time value first, then the other filter values in order
        ('2026-11/south', '2026-11-12/P-CHN', '2026-11')."""
        time = [f.value for f in self.answer.filters if f.dimension == self.time_dimension]
        rest = [f.value for f in self.answer.filters if f.dimension != self.time_dimension]
        return "/".join(time + rest)

A worked example

The known answer of OTIF % reads region = 'south' AND promised month = 2026-11. Each row below is one form the same question takes.

Where it is askedWhat the question looks like
As filtersRegion equals south. Promised month is the month 2026-11.
In the oracleThe measure OTIF % at the address 2026-11/south. The time value comes first, then the other values.
Of goldThe measure's expression over fact_otif joined to dim_dealer and dim_material, filtered on the two dimension expressions, in the business time zone.
Of the view on the engineMEASURE(`OTIF %`) from mv_otif with the same two filters.
src/dataplatform/semantic/answers.py:28The question written the way the metric view defines it, to be asked of gold directly.
def question_sql(view: MetricView, case: KnownAnswerCase) -> str:
    measure = next(m for m in view.measures if m.name == case.measure)
    joins = " ".join(f"LEFT JOIN {j.name} AS {j.name} ON source.{j.key} = {j.name}.{j.key}" for j in view.joins)
    where = " AND ".join(_where(view, f) for f in case.answer.filters)
    # shiftleft: allow SL-SEC-01 -- the text comes from the reviewed metric file (as known_answers.sql), values are quoted
    return f"SELECT CAST({measure.expr} AS DOUBLE) AS got FROM {view.source} AS source {joins} WHERE {where}"

This is what lets a laptop check the metric before any workspace exists. The query is built from the metric file alone: the measure's expression, a left join to each dimension on its key, and one condition per filter. It runs through the gold reader, on Spark or on DuckDB, with the session set to the business time zone so that a day means a business day.

src/dataplatform/semantic/metrics.py:286Each case is asked of whatever can answer it. A missing value fails.
def check_known_answers(
    cases: Iterable[KnownAnswerCase], read: Callable[[KnownAnswerCase], float | None]
) -> list[Verdict]:
    """Ask `read` (a metric view, the oracle, a dashboard's dataset) each case; a missing value fails."""
    out = []
    for c in cases:
        got = read(c)
        ok = got is not None and abs(got - c.answer.value) <= c.answer.tolerance + 1e-9
        out.append(Verdict(c, got, ok))
    return out

What the recorded runs say

Each row is one known answer. The second column is the value in the metric file. The third is what gold gave in the recorded 14-day run of 5 November 2026 at one tenth of full scale, which equalled the oracle of that run in every row. The fourth is what gold gave on 4 November 2026 when it was built once over the full-scale silver of day 14.

Known answerIn the metric fileGold at one tenth scaleGold at full scale, built once
OTIF %, south, November0.89610.88510.8961
Order lines, November107,18810,216107,188
Sell-in units, north, November1,279,642115,6371,279,642
Sell-out units, north, October150,803150,803150,803
Scrap rate %, plant P-CHN, 12 November0.01800.0179810.017981

The values in the metric files are true at full scale after 14 days. At one tenth scale three of the five depend on the number of orders and so differ from the file, while still equalling the oracle of their own run. Dealer files and presses do not scale, so the last two match at any scale. In that run the known-answer tests gave 12 passed and 4 failed, and the 4 were exactly the scale-dependent ones. A separate test checks that the full-scale oracle equals every value in the metric files, and its note says it passed on 4 November 2026. The same five numbers were reproduced on DuckDB. The sources are docs/W1D-GOLD.md and docs/DUCKDB-SPIKE.md.

A known answer found a wrong definition

The first version of OTIF % divided by every order line of the month. Asked of gold it gave 0.6706 at full scale where the oracle said 0.8961, because lines whose promised date had not come yet were counted as missed. The definition was changed to count only rows whose window is complete, which is the filter you see in the metric file above. This is the kind of mistake the known answer exists to catch.

Dashboards and the question space

Nothing that people look at is drawn by hand. A dashboard is derived per domain: one page per certified view, and for each certified measure a headline number, a trend over its time dimension, and a breakdown by each other dimension. A question space is a place where a person asks in plain words and a model answers from the data. The platform generates its definition too.

src/dataplatform/semantic/experiences.py:105The question space: certified views only, instructions, sample questions, and the measures it must decline.
def conversation_space(
    name: str, title: str, views: Iterable[MetricView], business_timezone: str, all_views: Iterable[MetricView]
) -> ConversationSpace:
    """`views` are the published views; only the certified ones are attached. `all_views` (as read from the manifest,
    before publishing) supply the measures to decline."""
    certified = sorted((v for v in views if v.certified_only), key=lambda v: v.name)
    allowed = {m.name for v in certified for m in v.measures}
    declined = sorted({m.name for v in all_views for m in v.measures if m.name not in allowed})
    words = [f"{m.name}: {', '.join(m.synonyms)}" for v in certified for m in v.measures if m.synonyms]
    instructions = [
        "Answer only from the attached metric views. Every measure in them is certified (ADR-022); never write your "
        "own formula for a metric and never read any other table.",
        "Show the metric view, the measure and the query with every answer.",
        f"Dates and months are in the business time zone {business_timezone}.",
        "Business words and the measure they mean: " + "; ".join(words) + ".",
    ]
    if declined:
        instructions.append(
            "These measures exist but are not certified; if a question needs one, say it is not "
            "certified yet and do not estimate it: " + ", ".join(declined) + "."
        )
    return ConversationSpace(
        name,
        title,
        tuple(v.name for v in certified),
        tuple(instructions),
        tuple(sample_questions(certified)),
        tuple(declined),
    )

Three things make this safe. Only certified views are attached, so the space cannot read a draft measure even if asked. The instructions list each business word beside the measure it means. And the measures that exist but are not certified are listed by name, with the instruction to say so and not to estimate. For the tyre company those are Net sales value and Tyres cured.

What is generated for Databricks

The folder realms/databricks/semantic_governance/generate.py reads the tyre manifests, contracts and an access file, and writes one set of files per environment. The files are committed. A test fails if a committed file differs from what the generator would write now, so a hand edit cannot survive.

Generated fileWhat it isMade from
semantic/mv_*.sqlOne statement per published view that creates a metric view.The metric files and the gold manifests
semantic/known_answers.sqlOne query that asks every known answer of the views and says whether each is within tolerance.The known answers
bi/*.lvdash.jsonOne dashboard file per domain.The certified views
bi/genie_space.jsonThe question space.The certified views, synonyms and known answers
governance/catalog_sync.sqlComments and tags for every table, view and column.The contracts, the manifests and the metric views
governance/audit.sqlA query over the audit log for reads of tables with protected columns and changes to grants and tags.The contracts
access/access.sqlGrants to groups and a mask function per protected column.The access file and the contracts' classifications
findings.mdDifferences between a contract and its manifest that were reported instead of being quietly resolved.Contracts and manifests
realms/databricks/semantic_governance/generated/tyre/dev/semantic/mv_otif.sql:3The generated metric view for on time in full, for the dev environment.
CREATE OR REPLACE VIEW `dev_gold`.`sales`.`mv_otif`
WITH METRICS
LANGUAGE YAML
AS $$
version: 1.1
comment: "Metric view over fact_otif. Certified measures only (ADR-022). OTIF %: Share of open order lines (not cancelled or rejected) delivered by the date first promised and in full, over complete windows only (a line not yet due is neither on time nor missed; W1d finding, 5 Nov)."
source: dev_gold.sales.fact_otif
joins:
  - name: dim_dealer
    source: dev_gold.sales.dim_dealer
    "on": source.dealer_sk = dim_dealer.dealer_sk
    cardinality: many_to_one
  - name: dim_material
    source: dev_gold.manufacturing.dim_material
    "on": source.material_sk = dim_material.material_sk
    cardinality: many_to_one
dimensions:
  - name: "Promised month"
    expr: "DATE_TRUNC('MONTH', promised_date)"
    comment: "Time dimension of OTIF %, Order lines."
  - name: "Region"
    expr: "dim_dealer.region"
    comment: "From the shared dimension dim_dealer, as of the event (ADR-005)."
  - name: "Tyre class"
    expr: "dim_material.tyre_class"
    comment: "From the shared dimension dim_material, as of the event (ADR-005)."
measures:
  - name: "OTIF %"
    expr: "SUM(on_time_in_full) FILTER (WHERE _is_complete) / COUNT(1) FILTER (WHERE _is_complete)"
    comment: "Share of open order lines (not cancelled or rejected) delivered by the date first promised and in full, over complete windows only (a line not yet due is neither on time nor missed; W1d finding, 5 Nov). Certification: certified. Additive: no. Time dimension: Promised month."
    synonyms: ["on-time delivery", "on time in full", "OTIF", "delivery performance"]
  - name: "Order lines"
    expr: "COUNT(1)"
    comment: "Certification: certified. Additive: yes. Time dimension: Promised month."
    synonyms: ["lines", "order lines"]
$$;

Compare it with the metric file. The source is now the full three-part name of the gold fact. Each join has the key from the fact's manifest and is declared many to one, which is true because a dimension has one row per key, so no measure can be counted twice. The expressions are copied unchanged. The comments are composed from the description, the certification and the time dimension.

realms/databricks/semantic_governance/render.py:116The renderer: a record in, text out, the same text every time for the same input.
def metric_view_sql(view: MetricView, env: str, comment: str) -> str:
    name = table(env, "gold", view.domain, view.name)
    return (
        HEADER + f"-- {view.name}: {'certified measures only' if view.certified_only else 'preview'} "
        f"(owner {view.owner})\nCREATE OR REPLACE VIEW {name}\nWITH METRICS\nLANGUAGE YAML\nAS $$\n"
        + metric_view_yaml(view, env, comment)
        + "$$;\n"
    )
realms/databricks/semantic_governance/generated/tyre/dev/semantic/known_answers.sql:3The start of the generated known-answer query: expected, tolerance, and what the view gives.
SELECT *, got IS NOT NULL AND abs(got - expected) <= tolerance AS ok FROM (
  SELECT 'mv_otif' AS view_name, 'OTIF %' AS measure, 'mv_otif OTIF % south 2026-11' AS address,
         CAST(0.8961 AS DOUBLE) AS expected,
         CAST(0.0005 AS DOUBLE) AS tolerance,
         (SELECT CAST(MEASURE(`OTIF %`) AS DOUBLE) FROM `dev_gold`.`sales`.`mv_otif` WHERE `Region` = 'south' AND date_format(`Promised month`, 'yyyy-MM') = '2026-11') AS got
realms/databricks/engine/tasks/semantic_layer.py:34After a run on the engine: each view's number beside gold's and the world's.
def check(spark: Any, generated: Path, report: dict[str, Any]) -> list[dict[str, Any]]:
    """One row per certified known answer: the deployed view's number, gold's (Python) and the world's."""
    rows = spark.sql(_statements((generated / "semantic" / "known_answers.sql").read_text())).collect()
    gold = report.get("gold_known_answers") or {}
    world = report.get("known_answers") or {}
    out = []
    for r in rows:
        view, measure = str(r["view_name"]), str(r["measure"])
        got = None if r["got"] is None else float(r["got"])
        g = next((v for k, v in gold.items() if k.startswith(f"{view}|{measure}|")), None)
        base = view.removesuffix("_preview")  # a preview view answers the certified view's question
        w = next((v.get("world") for k, v in world.items() if k.startswith(f"{base} {measure} ")), None)
        ok = _same(got, g) and _same(got, None if w is None else float(w))
        out.append({"view": view, "measure": measure, "address": str(r["address"]), "view_value": got,
                    "gold_value": g, "world_value": w, "spec_value": float(r["expected"]), "ok": ok})
    return out

This last function is the three-way comparison. It runs the generated query against the views that were just created, and for each row takes the number gold gave in Python and the number the simulated world holds. A row is in order only if the view equals both. The value from the metric file is carried along and reported, not judged, because it only holds at full scale.

Governance: what the catalog must say

A data catalog is the place where tables are listed with their owners, descriptions and labels. The platform does not ask anyone to fill it in. It works out what the catalog must say from the contracts and manifests, compares that with what the catalog holds, and produces the smallest list of changes.

src/dataplatform/governance/tables.py:29The tag standard: the only tag keys the platform ever sets or removes.
TABLE_TAGS = ("domain", "layer", "data_product", "used_by", "owner", "contract", "certification", "glossary_terms")
COLUMN_TAGS = ("classification", "protection")
MANAGED_TAGS = tuple(sorted(set(TABLE_TAGS) | set(COLUMN_TAGS)))
COMPUTE_TAGS = ("env", "domain", "data_product", "source", "dataset", "cost_center")
src/dataplatform/governance/catalog.py:64The difference for one table or column.
def _diff(ref: ObjectRef, want: Entry, have: Entry | None) -> list[Action]:
    actions = []
    if want.comment is not None and (have is None or have.comment != want.comment):
        actions.append(Action("comment", ref, want.kind, comment=want.comment))
    current = have.tag_map() if have else {}
    to_set = tuple((k, v) for k, v in want.tags if current.get(k) != v)
    if to_set:
        actions.append(Action("set_tags", ref, want.kind, tags=to_set))
    wanted = want.tag_map()
    stale = tuple(sorted(k for k in current if k in MANAGED_TAGS and k not in wanted))
    if stale:
        actions.append(Action("unset_tags", ref, want.kind, keys=stale))
    return actions
src/dataplatform/governance/catalog.py:79The plan for the whole catalog, and also the drift report: an empty plan means the catalog is in step.
def plan_sync(desired: Mapping[ObjectRef, Entry], current: Mapping[ObjectRef, Entry]) -> list[Action]:
    """Actions that make the catalog match the metadata, in a stable order (tables before their columns). Objects the
    metadata does not declare lose only our tags (a dropped contract must not leave its owner tag behind)."""
    actions: list[Action] = []
    for ref in sorted(set(desired) | set(current), key=lambda r: (r.layer, r.schema, r.name, r.column or "")):
        if ref.layer not in SYNCED_LAYERS:
            continue
        want, have = desired.get(ref), current.get(ref)
        if want is not None:
            actions += _diff(ref, want, have)
        elif have is not None:
            stale = tuple(sorted(k for k, _ in have.tags if k in MANAGED_TAGS))
            if stale:
                actions.append(Action("unset_tags", ref, have.kind, keys=stale))
    return actions
SituationWhat the plan does
The catalog is emptyWrites every comment and every tag.
Someone changed an owner tag by handSets it back.
Someone added a tag of their ownLeaves it alone. Only keys in the tag standard are ever touched.
A tag of ours is no longer wantedRemoves it.
A table is in the catalog but no contract declares itRemoves only our tags from it.
A table in bronzeNever touched. Only silver and gold have contracts.
A contract has no descriptionThe existing comment is not cleared.

Each column gets a classification tag. A contract that says nothing means internal. A column classified restricted is personal data that was replaced by a token before it reached bronze, and is tagged as tokenised. A column classified confidential is tagged as masked. A data product is a gold fact with its metric views, and each silver table and dimension is tagged with the products that read it.

Access: groups, masks and row filters

Access is given to groups, never to a named person. An access file lists the groups. For each it says which layers the group reads, which domains, and whether it sees protected columns or sees them masked.

groups:
  - name: data-platform-jobs          # service principals that run jobs and pipelines (ENV-02); must read what they load
    layers: [landing, bronze, silver, gold]
    domains: all
    protected: see
  - name: platform-engineers          # people who build and debug the platform: everything, protected columns masked
    layers: [bronze, silver, gold]
    domains: all
    protected: masked
  - name: sales-data                  # contract team: owns the sales domain
    layers: [silver, gold]
    domains: [sales]
    protected: see
  - name: manufacturing-data
    layers: [silver, gold]
    domains: [manufacturing]
    protected: see
  - name: quality-data
    layers: [silver, gold]
    domains: [quality]
    protected: see
  - name: analysts                    # ADR-012: analysts see gold only
    layers: [gold]
    domains: all
    protected: masked
row_filters: []                       # none declared by the specs yet; the mechanism is tested (tests/dataplatform/test_access_model.py)
src/dataplatform/access/model.py:130From groups and classified columns to grants, masks and row filters.
def plan_access(model: AccessModel, tables: Iterable[TableSpec]) -> AccessPlan:
    tables = list(tables)
    grants: set[Grant] = set()
    for g in model.groups:
        for layer in g.layers:
            if g.every_domain:
                grants.add(Grant(g.name, "read", "catalog", layer))
                continue
            for t in tables:
                if _reads(g, t) and t.layer == layer:
                    grants.add(Grant(g.name, "use", "catalog", layer))
                    grants.add(Grant(g.name, "use", "schema", layer, t.schema))
                    grants.add(Grant(g.name, "read", "table", layer, t.schema, t.name))
    masks = []
    for t in tables:
        allowed = tuple(g.name for g in model.groups if g.protected == "see" and _reads(g, t))
        masks += [
            ColumnMask(t.layer, t.schema, t.name, c.name, c.logical_type, c.classification, c.tokenised, allowed)
            for c in t.columns
            if c.protected
        ]
    known = {(t.layer, t.schema, t.name) for t in tables}
    for f in model.row_filters:
        if (f.layer, f.schema, f.table) not in known:
            raise AccessError(f"row filter on {f.layer}.{f.schema}.{f.table}: no such published table")
    order = ("catalog", "schema", "table")
    return AccessPlan(
        tuple(sorted(grants, key=lambda x: (x.layer, order.index(x.scope), x.schema, x.name, x.group, x.privilege))),
        tuple(sorted(masks, key=lambda m: (m.layer, m.schema, m.table, m.column))),
        model.row_filters,
        model,
    )
  1. Grants. A group that reads every domain gets read on the whole layer. A group limited to some domains gets read on each of its tables, and permission to enter the layer and the schema.
  2. Masks. Every column classified confidential or restricted gets a mask. The groups allowed past the mask are those declared see that can read the table.
  3. Row filters. A filter names a table, a column, the groups that see every row, and the values each limited group may see. A reader not named on the filter sees no rows. A filter on a table that is not published is refused.
src/dataplatform/access/model.py:176The check: what one group sees in each column of one table.
def visible_columns(plan: AccessPlan, group: str, table: TableSpec) -> dict[str, str] | None:
    """What `group` sees in each column (clear, token or masked); None if it cannot read the table at all."""
    if not can_read(plan, group, table):
        return None
    masks = {m.column: m for m in plan.masks if (m.layer, m.schema, m.table) == (table.layer, table.schema, table.name)}
    out = {}
    for c in table.columns:
        m = masks.get(c.name)
        if m is None:
            out[c.name] = CLEAR
        elif group not in m.allowed:
            out[c.name] = MASKED
        else:
            out[c.name] = TOKEN if m.tokenised else CLEAR
    return out

This function is how the plan is tested without any database. For a group and a table it returns, per column, one of three words. clear is the real value. token is the stored token of a restricted column, which is all even an allowed group ever sees, because nothing in this code turns a token back into the value. masked is the mask. If the group cannot read the table at all it returns nothing.

A worked example

The silver dealer table has a dealer code that is internal and a contact e-mail column that is restricted. Each row is one group from the access file and what it sees.

GroupCan read silver dealers?dealer codecontact e-mail
sales-datayes, its own domainclearthe token
platform-engineersyes, every domainclearmasked
data-platform-jobsyes, every domainclearthe token
manufacturing-datano, another domain
analystsno, gold only
realms/databricks/semantic_governance/generated/tyre/dev/access/access.sql:50The generated mask for that column: a function that returns the stored value to two groups and three stars to everyone else.
CREATE OR REPLACE FUNCTION `dev_ops`.`access`.`mask_silver_dealer_api_dealer_contact_email`(v STRING)
RETURNS STRING
COMMENT 'restricted: allowed groups see the token (never detokenised); everyone else sees it masked'
RETURN CASE WHEN is_account_group_member('data-platform-jobs') OR is_account_group_member('sales-data') THEN v ELSE '***' END;
ALTER TABLE `dev_silver`.`dealer_api`.`dealer` ALTER COLUMN `contact_email` SET MASK `dev_ops`.`access`.`mask_silver_dealer_api_dealer_contact_email`;

What was run where

This matters, so it is stated plainly. Each row is one part of this page. The second column says what the public code and its recorded notes show was actually run.

PartWhat the evidence shows
Reading, linting, splitting, known-answer parsingTested locally with no engine, as tables of cases.
Known answers asked of goldRun locally on Spark and on DuckDB over a 14-day simulated run at one tenth of full scale, recorded on 5 November 2026. Gold built once over full-scale silver on 4 November 2026 gave the metric files' values. The full-scale 14-day run is owed.
Generated view, dashboard and question space filesGenerated and committed. Tested locally for content and for being up to date. These tests read the files. They do not run them.
Creating the metric views on a workspace and checking them against gold and the worldThe code that does this after a run on the engine is in the repository. The public snapshot holds no saved result of such a run, so this page cannot say what the views returned on a workspace.
The question space on a workspaceA deploy script exists. Comments in the code record a first deploy on 15 November that found the product refuses lists that are not sorted, which the renderer now sorts. No transcript of questions and answers is in the snapshot.
Dashboards on a workspaceA bundle file that deploys the two dashboards exists. Whether it was deployed cannot be told from the snapshot.
Catalog sync, grants, masks and the audit queryGenerated and tested locally only. The adapter that reads a live catalog is tested against made-up rows. Nothing in the snapshot applies these statements to a workspace.
A masked column read by a user without privilegeNot run. The design note lists it as waiting for a workspace.

How it is tested

Test fileWhat its tables cover
tests/dataplatform/test_semantic_metrics.pyParsing a known answer's filters, refusals when reading, every lint rule, the split into certified and preview, looking a case up in the oracle, the verdict with a tolerance, and the tyre files themselves.
tests/dataplatform/test_semantic_experiences.pyOne dashboard per domain from certified views only, the tiles per measure, and what the question space attaches, declines and asks.
tests/databricks_realm/test_semantic_governance.pyThe committed files are what the generator writes, generation is repeatable, identifiers and values are quoted safely, and the tyre output has the properties it must have.
tests/dataplatform/test_access_model.pyWhat each group sees per column, row filters, grants, masks, and refusals of a bad access file.
tests/dataplatform/test_governance_catalog.pyTables from contracts, who reads what, the sync plan for each kind of drift, and that applying a plan leaves nothing to do.
tests/dataplatform/test_semantic_metrics.py:126The lint rules as a table: a set of views, and a piece of each error it must produce. An empty list means clean.
LINT = {
    "clean": ([view()], []),
    "name without mv_": ([view(name="metrics_x")], ["mv_ + lower snake"]),
    "name not snake case": ([view(name="mv_X")], ["mv_ + lower snake"]),
    "source not a fact_": (
        [metric_view(doc(source="dim_a", joins=[]), {**GOLD, "dim_a": {**GOLD["dim_a"], "dimensions": []}})],
        ["is not a fact_ table"],
    ),
    "certified without known answer": ([view(measures=[measure(ka=False)])], ["without a known-answer test"]),
    "draft without known answer is fine": ([view(measures=[measure(cert="draft", ka=False)])], []),
    "unknown lifecycle state": ([view(measures=[measure(cert="approved")])], ["not one of"]),
    "time dimension not in view": ([view(measures=[measure(time_dimension="Week")])], ["time dimension 'Week'"]),
    "measure defined twice": ([view(), view(name="mv_y")], ["also defined in mv_x"]),
    "synonym means two metrics": (
        [view(measures=[measure("A", synonyms=["sales"]), measure("B", synonyms=["sales"])])],
        ["already means"],
    ),
    "draft synonym may repeat a word": (
        [view(measures=[measure("A", synonyms=["s"]), measure("B", cert="draft", synonyms=["s"])])],
        [],
    ),
    "no owner": ([view(owners={})], ["no owner"]),
}
tests/dataplatform/test_semantic_metrics.py:348One number everywhere: for every known answer, the view's number and the oracle's number of the same run must be within the tolerance.
@needs_slice
@pytest.mark.parametrize("case", _cases(), ids=lambda c: c.label)
def test_metric_view_matches_world_oracle_pending_gold(case: KnownAnswerCase):
    """Pending W1d: the metric view and the oracle of the same world run give the same number (one number
    everywhere). DP_GOLD_ORACLE names the oracle day file of the run that fed gold."""
    answers = _gold_answers()
    oracle_path = os.environ.get("DP_GOLD_ORACLE")
    if not oracle_path:
        pytest.skip("pending W1d (gold): set DP_GOLD_ORACLE to the oracle file of the run that fed gold")
    oracle = json.loads(Path(oracle_path).read_text())["metrics"]
    want = oracle_value(oracle, case)
    got = answers.get(f"{case.view}|{case.measure}|{case.oracle_key}")
    assert want is not None
    assert got is not None
    assert abs(got - want) <= case.answer.tolerance + 1e-9

The word pending in the test's name is historical. It was written before gold existed and skips unless two files from a run are named. The build command for the known answers names them.

tests/dataplatform/test_access_model.py:70What each group must see: a group, a table, and the expected word per column, or nothing when it cannot read the table.
SEES = {
    "lowest privilege cannot read silver at all": ("analysts", DEALER, None),
    "analysts read gold in the clear": ("analysts", FACT, {"order_id": CLEAR, "otif": CLEAR}),
    "domain team sees its restricted column as a token": (
        "sales-data",
        DEALER,
        {"code": CLEAR, "email": TOKEN, "margin": CLEAR},
    ),
    "engineers read everything but protected columns are masked": (
        "engineers",
        DEALER,
        {"code": CLEAR, "email": MASKED, "margin": MASKED},
    ),
    "jobs see tokens, never clear personal data": ("jobs", DEALER, {"code": CLEAR, "email": TOKEN, "margin": CLEAR}),
    "another domain's team cannot read the table": ("quality-data", DEALER, None),
    "quality team sees its inspector token": ("quality-data", INSPECT, {"id": CLEAR, "inspector": TOKEN}),
    "sales team cannot read quality's table": ("sales-data", INSPECT, None),
    "quality team has no gold": ("quality-data", FACT, None),
    "a group nobody declared sees nothing": ("interns", FACT, None),
}
tests/databricks_realm/test_semantic_governance.py:171The generated question space attaches the certified views and names the two measures it must decline.
@needs_slice
@pytest.mark.parametrize("env", ["dev", "prod"])
def test_genie_reads_certified_views_only(files, built, env):
    space = json.loads(files[f"tyre/{env}/bi/genie_space.json"])["space"]
    tables = {t["identifier"] for t in space["data_sources"]["tables"]}
    certified = {f"{env}_gold.{v.domain}.{v.name}" for v in built.published if v.certified_only}
    assert tables == certified
    assert not any(t.endswith("_preview") for t in tables)
    text = " ".join(c for i in space["instructions"]["text_instructions"] for c in i["content"])
    assert "Net sales value" in text
    assert "Tyres cured" in text
tests/dataplatform/test_governance_catalog.py:238For every kind of drift, applying the plan and planning again gives an empty plan.
@pytest.mark.parametrize("case", PLAN)
def test_plan_converges(case):
    """Applying the plan leaves nothing to do: the realm's statements, run once, close every gap."""
    current, _ = PLAN[case]
    assert plan_sync(WANT, apply(current, plan_sync(WANT, current))) == []
Not built yet, or known gaps
  • No result of the generated views, dashboards or question space running on a real workspace is in the public snapshot. Treat everything in the Databricks folder as generated and locally tested unless the table above says more.
  • The catalog sync, the grants, the masks and the audit query have not been applied to a live catalog as far as the snapshot shows.
  • Grants are planned, not compared with what exists. The generated file grants what the model says and does not revoke a grant someone made by hand.
  • No row filter is declared for the tyre company. The mechanism is built and tested with a made-up filter.
  • The access file lives in the Databricks folder of the platform repository, not with the tyre manifests.
  • The catalog has no glossary object. Glossary terms are the certified measure names, written as a tag and in the view's comment.
  • The contract and the manifest of the inspection results name different owners. The generator reports the difference and takes the contract's owner. Nobody has decided which is right.
  • The business day of a view's time dimension depends on the session's time zone. Locally the reader sets it. On a workspace the warehouse must be set to the business time zone, and nothing in the generated files does that.
  • Sample questions are derived from the known answers and the first synonym. A metric file cannot yet carry its own wording for them.
  • Net sales value and Tyres cured are not certified, so they appear on no dashboard and the question space declines them.
  • The metric files' known answers hold only at full scale after 14 days. At any other scale the test against the file's value fails for the measures that depend on order volume, and only the test against the oracle of the same run is meaningful.