Home › Learn › Data systems › Upstreams › Where the answers go
Data systems · upstreams, in depth · for data leaders, then builders
Where the answers go: contract, manifest, catalog, code
Every answer is written once, in one home, and everything else is generated from it.
Where each discovery answer lives: the ODCS data contract, the pipeline manifest, Unity Catalog, quality tests, the ontology, the engine's code, decision records and the source agreement. With a worked example and how to choose liquid clustering columns.
For leaders
- Each answer gets one home. Everything else (Unity Catalog metadata, pipeline settings, tests, the glossary) is generated from it, never typed by hand.
- The data contract (ODCS) says what the data promises. The manifest says how we load it. Unity Catalog is where both are enforced at run time.
- The ontology connects the same business thing across systems, which no single contract can do.
- The code is one generic engine that reads these files; a new dataset is configuration, not a release.
In plain words
One recipe card, many uses
A restaurant writes each recipe once, on one card. The shopping list, the prep sheet, the allergy labels on the menu and the oven timer all come from that card. If the chef changes the recipe, everything else changes with it. Nobody keeps a second copy of the recipe in their head.
The answers from discovery work the same way. Each one is written once, in its home. The pipeline settings, the Unity Catalog metadata, the quality tests and the business glossary are generated from it, and the code reads them instead of hard-coding them.
The homes
Eight homes, each with one job
| Home | What it holds | Format and place | Owner | Changed by |
|---|---|---|---|---|
| 1 · Data contract (ODCS) | What the dataset promises: schema, keys, descriptions, relationships, classification, quality rules, freshness, owners | YAML in git, one file per dataset | The source owner and the data product owner | Pull request, reviewed and versioned |
| 2 · Pipeline manifest | How we load it: load pattern, watermark, CDC settings, merge keys, history type per column, clustering, schedule, throttling | YAML in git, loaded into the metadata store | The platform team | Pull request; validated against the contract |
| 3 · Unity Catalog | The enforced copy at run time: comments, tags, key constraints, masks, row filters, grants, clustering, lineage | Generated by the deployment pipeline | Nobody edits it by hand | Every deploy |
| 4 · Quality tests | Checks from the contract's rules and the profile's thresholds; reconciliation against control totals | Generated into the quality framework | The platform team | Every deploy |
| 5 · Ontology and glossary | Business entities and terms, and which datasets in which systems represent them | A small model in git (YAML or a graph), linked from contracts | Domain owners, data governance | Pull request |
| 6 · The engine | Generic bronze to silver to gold code that reads 2 and 1 | Code in git | The platform team | Normal releases; never edited per dataset |
| 7 · Decision records | Why: full or incremental, why no CDC, why Type 2 on these columns | Short documents in git | The technical anchor | Written when a decision is made |
| 8 · Source agreement and runbook | Extract window, load budget, contacts, kill switch, escalation | A document linked from the contract | The source owner and the platform team | Reviewed at each release of the source |
The map
Each group of answers, where it lives and who uses it
Letters are the question groups on the Upstreams page. The last column is the part of the platform each answer shapes; it will link to the hop pages as they are written.
| Answers | Home (written once) | Generated into | Used by |
|---|---|---|---|
| A · Ownership and change | Contract: team, roles, support | UC owner and tags; alert routing | Every hop: who to call |
| B · Access route | Contract: servers. Secrets go in a vault, never in a contract | Manifest connection reference | Hop 1 |
| C · Protecting the source | Source agreement (8) | Manifest throttle and window; capture-lag alerts; runbook | Hop 1, operations |
| D, E, F · Push, pull, full or incremental | Manifest (2), with the reason in a decision record (7) | Hop 1 job settings | Hop 1 |
| G · How data changes, history type | Manifest: delete handling, look-back, Type 1 or 2 per column | Merge settings | Hop 2 |
| H · Keys, dates, ordering, duplicates | Contract: primary key, types, descriptions. Manifest: merge key, sequence, tie-break | UC primary key and comments; uniqueness tests | Hop 2 |
| I · Relationships | Contract: relationships. Ontology for links across systems | UC foreign keys; orphan and cardinality tests | Hop 3 joins |
| J · Schema and meaning | Contract: descriptions; reference tables; ontology terms | UC comments; semantic layer definitions | Hops 3 and 4, Genie, agents |
| K · Volume and timing | Contract: freshness promise. Manifest: schedule or trigger, clustering | Freshness alerts; table layout; cost estimate | Every hop |
| L · Profiling | Profile results in the ops catalog | Quality thresholds in the contract | Hops 1 to 3 |
| M · Sensitivity and rules | Contract: classification, retention | UC tags, column masks, row filters; storage lifecycle | Every layer, hop 5 access |
| N · Trust and reconciliation | Contract: quality rules (counts, sums) | Reconciliation jobs; tests against the trusted report | Hops 1 to 3 |
The split
Contract or manifest? The data's promise versus how we load it
ODCS describes what a dataset promises. How we load it (watermarks, merge keys, history type, clustering, throttling) is our choice and changes more often, so it lives in the manifest, checked against the contract. Run-time state and secrets live in neither.
| Answer | Home | Why |
|---|---|---|
| order_no is the business key | Contract | It's a promise about the data |
| Merge on order_no; latest by updated_at, ties by ingest time | Manifest | It's how we load |
| email is personal data | Contract | A fact about the data, used for masking everywhere |
| Load incrementally with a 2-hour overlap; full compare weekly | Manifest | A loading choice; the reason goes in a decision record |
| Data is no older than 24 hours by 07:00 | Contract | A promise to consumers |
| Cluster silver by order_no | Manifest | Physical layout, our choice |
| The last watermark value | Metadata store state | Run-time state, not configuration |
| The database password | Vault | Never in git |
Physical layout
Partitioning, Z-order or liquid clustering
All three decide which files a merge or a query must read and rewrite. On Databricks, liquid clustering is the default for new tables: it replaces both partitioning and Z-order, works incrementally, and its keys can be changed later without rewriting the table. Partitioning is kept for very large tables that need whole-partition operations. Z-order is for older tables until they move to clustering. The inputs come from upstream answers: the merge key (H), the date people filter by (H), the table's size and growth (K), retention rules (M) and how the data is read downstream.
| Option | What it does | Use it for | Watch for |
|---|---|---|---|
| Liquid clustering | Rows with similar values in the same files; reorganised incrementally; keys can be changed without rewriting the table | Default for new Delta tables on Databricks: silver and gold, merge-heavy tables, tables whose query patterns change | Can't be combined with partitioning or Z-order on the same table; pick up to four keys |
| Partitioning | One folder per value of a column (usually a date) | Only very large tables (well over a terabyte) on a low-cardinality column, when you need whole-partition operations: replace a day, drop old months, or other engines must read the layout | High-cardinality columns create millions of tiny files; changing the scheme means rewriting the table |
| Z-ordering | OPTIMIZE ... ZORDER BY sorts data across files by a few columns | Existing tables not yet moved to liquid clustering, or runtimes and engines without it | Not incremental: each run rewrites data, and it must be re-run as data arrives |
| Nothing | Let file statistics and compaction do the work | Small tables (a few GB), reference data, most bronze tables | Fine until the table grows; review at a size threshold |
Liquid clustering: which columns, by layer
Up to four columns; fewer is usually better. The choice goes in the manifest and is generated into the table.
| Layer | Cluster by | Why |
|---|---|---|
| Bronze | Usually nothing, or the ingest date | Bronze is read in batches by ingest time for replay; keep it cheap to write |
| Silver | The merge key, plus the most common date filter if there is one | Merges then rewrite few files; this is the merge-cost lever in the cost calculator |
| Gold facts | The date plus the most filtered dimension key (customer, store) | Dashboard queries skip most files |
| Gold dimensions | The business key, or nothing for small ones | Small tables don't need clustering |
| Not sure yet | CLUSTER BY AUTO with predictive optimization | Databricks picks keys from real query history; review the choice later |
On other platforms the same decision has different names: Iceberg has hidden partitioning and sort orders, BigQuery has partitioning and clustering, Snowflake has clustering keys. Pick by the same inputs.
Worked example
One dataset, from answers to generated metadata
An ERP's orders table. The comments point to the questions each line answers.
The contract
# contracts/erp-orders.yaml (abridged and illustrative; check field names against the ODCS 3.2 schema)
apiVersion: v3.1.0
kind: DataContract
id: erp-orders
version: 1.2.0
domain: sales
team:
- username: priya.s # source owner (A1)
role: owner
schema:
- name: orders
physicalName: SALES.ORDERS
description: One row per customer order.
properties:
- name: order_no
logicalType: string
primaryKey: true # H1: business key, never reused
required: true
- name: customer_id
logicalType: string # I1: joins to customers.customer_id
- name: order_date
logicalType: date
description: When the customer ordered (transaction date, H6).
- name: status
logicalType: string
description: See reference table order_status (J1).
- name: email
logicalType: string
classification: restricted # M1: personal data
- name: updated_at
logicalType: timestamp
description: Set by the database on every change (E1, E2).
quality:
- metric: rowCount # N2: compared with the source's daily control total
slaProperties:
- property: freshness # K3
value: 24
unit: h
The manifest
# manifest/erp/orders.yaml (how we load it)
dataset: erp.orders
contract: contracts/erp-orders.yaml
source: {connection: erp_prod_replica, object: SALES.ORDERS} # B2: read the replica
load:
pattern: incremental_watermark # F: decision record DR-014
watermark_column: updated_at
overlap: 2h
full_compare: weekly # F5: catch deletes and drift
window: "01:00-05:00" # C1: agreed with the source
max_rows_per_second: 20000
silver:
merge_keys: [order_no]
sequence_by: updated_at # H7
tie_break: _ingested_at # H4
history: {type2: [status], type1: all_other} # G2
deletes: {soft_flag: is_deleted} # G1
lookback_days: 35 # G: late corrections
cluster_by: [order_no, order_date] # merge key + date filter
What the deployment generates in Unity Catalog
-- generated by the deployment pipeline from the two files above
COMMENT ON TABLE prod_silver.sales.orders IS 'One row per customer order.';
ALTER TABLE prod_silver.sales.orders ALTER COLUMN order_date COMMENT 'When the customer ordered (transaction date).';
ALTER TABLE prod_silver.sales.orders SET TAGS ('domain' = 'sales', 'owner' = 'priya.s', 'contract' = 'erp-orders@1.2.0');
ALTER TABLE prod_silver.sales.orders ALTER COLUMN email SET TAGS ('classification' = 'restricted');
ALTER TABLE prod_silver.sales.orders ALTER COLUMN email SET MASK governance.masks.email_mask;
ALTER TABLE prod_silver.sales.orders ADD CONSTRAINT orders_pk PRIMARY KEY (order_no); -- informational
ALTER TABLE prod_silver.sales.orders ADD CONSTRAINT orders_customer_fk
FOREIGN KEY (customer_id) REFERENCES prod_silver.sales.customers (customer_id); -- informational
ALTER TABLE prod_silver.sales.orders CLUSTER BY (order_no, order_date);
The engine then reads merge_keys, sequence_by, history and deletes to build silver. The quality framework turns the contract's rules into a uniqueness test on order_no, an orphan test on customer_id and a daily row-count reconciliation. Unity Catalog's key constraints are informational, so those tests are what actually catch duplicates and orphans.
The question list
Every question, why we ask, and what breaks if nobody does
| # | Question to chase | Why we ask | What breaks in production if nobody asks | Do this (not this) |
|---|---|---|---|---|
| WA · Where the answers goone home per answer | ||||
| WA1 | Does every answer have exactly one home, and is everything else generated from it? | Answers typed into several places drift apart. | The key in the code, the catalog and the contract differ, and nobody knows which is right. | One home per answer (the table above); the deployment pipeline generates the rest. Not: the same fact typed in three places. |
| WA2 | Is anything edited by hand in Unity Catalog? | Hand edits are overwritten on the next deploy, or block it. | A mask added by hand disappears after a release; personal data is visible for a week. | Unity Catalog metadata is generated from git. Not: console edits. |
| WA3 | Does the code hold dataset-specific keys, columns or rules? | Then every new dataset is a code change. | A key change needs a release of the engine. | The engine reads the manifest and the contract. Not: if table == 'orders' in code. |
| WA4 | Who approves a contract change, and is the contract version checked at deploy time? | A contract change is a change to a promise. | A column is dropped from the contract and three consumers break. | The owner approves in a pull request; breaking changes need a new major version. Not: editing the contract quietly. |
| WA5 | How is each big table laid out (liquid clustering, partitioning or Z-order), on which columns, and why? | Clustering decides how many files merges and queries touch. | Silver merges rewrite a fifth of a 50 TB table every run. | Choose from the merge key, the date filter and real query patterns, or CLUSTER BY AUTO; record it in the manifest. Not: clustering on whatever column comes first. |
Design decisions
The decisions to take
Where do loading settings live?
| Option | Verdict |
|---|---|
| Contract for promises, manifest for loading | Pursue · Pursue. Each file has one job and one owner. |
| Everything in the ODCS contract, using customProperties | Pursue when · Pursue only for very small platforms; the contract turns into a config file. |
What role does Unity Catalog play?
| Option | Verdict |
|---|---|
| Generated target, enforced at run time | Pursue · Pursue. Git stays the truth; the catalog can be rebuilt. |
| The place people edit metadata | Drop · Drop. Edits are lost or conflict with deploys. |
How do we lay out a big table?
| Option | Verdict |
|---|---|
| Liquid clustering | Pursue · Pursue for new silver and gold tables on Databricks. |
| Partition by date | Pursue when · Pursue only for very large tables that need whole-day or whole-month operations, or that other engines read by folder. |
| Partition by a high-cardinality column (customer, order) | Drop · Drop. Millions of tiny files. |
| Z-order | Pursue when · Pursue only on existing tables until they move to liquid clustering. |
How do we choose clustering columns?
| Option | Verdict |
|---|---|
| From the merge key, date filters and query history, recorded in the manifest | Pursue · Pursue for the big tables. |
| CLUSTER BY AUTO | Pursue when · Pursue where predictive optimization is on and query patterns aren't known yet. |
FAQ
Questions people ask
Where should data contract information live on Databricks?
In ODCS files in git, one per dataset, as the source of truth. The deployment pipeline generates Unity Catalog comments, tags, key constraints and masks from them, and the pipeline manifest holds how each dataset is loaded.
Which columns should a Delta table be liquid clustered on?
For silver tables with merges, the merge key, plus the most common date filter if there is one. For gold facts, the date and the most filtered dimension key. Small tables need none. If you don't know the query patterns yet, CLUSTER BY AUTO lets Databricks choose from query history.
Should I partition, Z-order or liquid cluster a Delta table?
On Databricks, liquid cluster new tables: it replaces partitioning and Z-order and adapts as query patterns change. Partition only very large tables that need whole-partition operations, on a low-cardinality column such as date. Use Z-order only on existing tables until they are moved to liquid clustering.
Are Unity Catalog primary and foreign keys enforced?
No. They are informational: they document the key and help some query plans, but they don't stop duplicates or orphans. Uniqueness and orphan tests are still needed.
Next
Where to go next
Related: the metadata store (where the manifest is loaded), Layout: catalogs, schemas and naming, and data contracts.
Back up: Upstreams: know your sources, and the printable questionnaire.