Home › Learn › Data systems › Upstreams › Where the answers go

Data systems · upstreams, in depth · for data leaders, then builders

Where the answers go: contract, manifest, catalog, code

Every answer is written once, in one home, and everything else is generated from it.

Where each discovery answer lives: the ODCS data contract, the pipeline manifest, Unity Catalog, quality tests, the ontology, the engine's code, decision records and the source agreement. With a worked example and how to choose liquid clustering columns.

For leaders

In plain words

One recipe card, many uses

A restaurant writes each recipe once, on one card. The shopping list, the prep sheet, the allergy labels on the menu and the oven timer all come from that card. If the chef changes the recipe, everything else changes with it. Nobody keeps a second copy of the recipe in their head.

The answers from discovery work the same way. Each one is written once, in its home. The pipeline settings, the Unity Catalog metadata, the quality tests and the business glossary are generated from it, and the code reads them instead of hard-coding them.

Questionnairediscovery, profiling,owner interviews1 · Data contractODCS YAML in git, one per dataset:what the data promisesValidate + generatein the deploymentpipeline2 · Manifesthow we load it3 · Unity Catalogcomments, tags, keys,masks, clustering4 · Quality testsrules, thresholds,reconciliation5 · Ontologybusiness meaningacross systems6 · One generic engine: bronze → silver → goldreads the manifest and contract; no keys, columns or rules written in code7, 8 · Decisions, agreementsdecision records, sourceagreement, runbook
Written once in the contract, generated everywhere else. Reasons and agreements live beside it.

The homes

Eight homes, each with one job

HomeWhat it holdsFormat and placeOwnerChanged by
1 · Data contract (ODCS)What the dataset promises: schema, keys, descriptions, relationships, classification, quality rules, freshness, ownersYAML in git, one file per datasetThe source owner and the data product ownerPull request, reviewed and versioned
2 · Pipeline manifestHow we load it: load pattern, watermark, CDC settings, merge keys, history type per column, clustering, schedule, throttlingYAML in git, loaded into the metadata storeThe platform teamPull request; validated against the contract
3 · Unity CatalogThe enforced copy at run time: comments, tags, key constraints, masks, row filters, grants, clustering, lineageGenerated by the deployment pipelineNobody edits it by handEvery deploy
4 · Quality testsChecks from the contract's rules and the profile's thresholds; reconciliation against control totalsGenerated into the quality frameworkThe platform teamEvery deploy
5 · Ontology and glossaryBusiness entities and terms, and which datasets in which systems represent themA small model in git (YAML or a graph), linked from contractsDomain owners, data governancePull request
6 · The engineGeneric bronze to silver to gold code that reads 2 and 1Code in gitThe platform teamNormal releases; never edited per dataset
7 · Decision recordsWhy: full or incremental, why no CDC, why Type 2 on these columnsShort documents in gitThe technical anchorWritten when a decision is made
8 · Source agreement and runbookExtract window, load budget, contacts, kill switch, escalationA document linked from the contractThe source owner and the platform teamReviewed at each release of the source

The map

Each group of answers, where it lives and who uses it

Letters are the question groups on the Upstreams page. The last column is the part of the platform each answer shapes; it will link to the hop pages as they are written.

AnswersHome (written once)Generated intoUsed by
A · Ownership and changeContract: team, roles, supportUC owner and tags; alert routingEvery hop: who to call
B · Access routeContract: servers. Secrets go in a vault, never in a contractManifest connection referenceHop 1
C · Protecting the sourceSource agreement (8)Manifest throttle and window; capture-lag alerts; runbookHop 1, operations
D, E, F · Push, pull, full or incrementalManifest (2), with the reason in a decision record (7)Hop 1 job settingsHop 1
G · How data changes, history typeManifest: delete handling, look-back, Type 1 or 2 per columnMerge settingsHop 2
H · Keys, dates, ordering, duplicatesContract: primary key, types, descriptions. Manifest: merge key, sequence, tie-breakUC primary key and comments; uniqueness testsHop 2
I · RelationshipsContract: relationships. Ontology for links across systemsUC foreign keys; orphan and cardinality testsHop 3 joins
J · Schema and meaningContract: descriptions; reference tables; ontology termsUC comments; semantic layer definitionsHops 3 and 4, Genie, agents
K · Volume and timingContract: freshness promise. Manifest: schedule or trigger, clusteringFreshness alerts; table layout; cost estimateEvery hop
L · ProfilingProfile results in the ops catalogQuality thresholds in the contractHops 1 to 3
M · Sensitivity and rulesContract: classification, retentionUC tags, column masks, row filters; storage lifecycleEvery layer, hop 5 access
N · Trust and reconciliationContract: quality rules (counts, sums)Reconciliation jobs; tests against the trusted reportHops 1 to 3

The split

Contract or manifest? The data's promise versus how we load it

ODCS describes what a dataset promises. How we load it (watermarks, merge keys, history type, clustering, throttling) is our choice and changes more often, so it lives in the manifest, checked against the contract. Run-time state and secrets live in neither.

AnswerHomeWhy
order_no is the business keyContractIt's a promise about the data
Merge on order_no; latest by updated_at, ties by ingest timeManifestIt's how we load
email is personal dataContractA fact about the data, used for masking everywhere
Load incrementally with a 2-hour overlap; full compare weeklyManifestA loading choice; the reason goes in a decision record
Data is no older than 24 hours by 07:00ContractA promise to consumers
Cluster silver by order_noManifestPhysical layout, our choice
The last watermark valueMetadata store stateRun-time state, not configuration
The database passwordVaultNever in git

Physical layout

Partitioning, Z-order or liquid clustering

All three decide which files a merge or a query must read and rewrite. On Databricks, liquid clustering is the default for new tables: it replaces both partitioning and Z-order, works incrementally, and its keys can be changed later without rewriting the table. Partitioning is kept for very large tables that need whole-partition operations. Z-order is for older tables until they move to clustering. The inputs come from upstream answers: the merge key (H), the date people filter by (H), the table's size and growth (K), retention rules (M) and how the data is read downstream.

OptionWhat it doesUse it forWatch for
Liquid clusteringRows with similar values in the same files; reorganised incrementally; keys can be changed without rewriting the tableDefault for new Delta tables on Databricks: silver and gold, merge-heavy tables, tables whose query patterns changeCan't be combined with partitioning or Z-order on the same table; pick up to four keys
PartitioningOne folder per value of a column (usually a date)Only very large tables (well over a terabyte) on a low-cardinality column, when you need whole-partition operations: replace a day, drop old months, or other engines must read the layoutHigh-cardinality columns create millions of tiny files; changing the scheme means rewriting the table
Z-orderingOPTIMIZE ... ZORDER BY sorts data across files by a few columnsExisting tables not yet moved to liquid clustering, or runtimes and engines without itNot incremental: each run rewrites data, and it must be re-run as data arrives
NothingLet file statistics and compaction do the workSmall tables (a few GB), reference data, most bronze tablesFine until the table grows; review at a size threshold

Liquid clustering: which columns, by layer

Up to four columns; fewer is usually better. The choice goes in the manifest and is generated into the table.

LayerCluster byWhy
BronzeUsually nothing, or the ingest dateBronze is read in batches by ingest time for replay; keep it cheap to write
SilverThe merge key, plus the most common date filter if there is oneMerges then rewrite few files; this is the merge-cost lever in the cost calculator
Gold factsThe date plus the most filtered dimension key (customer, store)Dashboard queries skip most files
Gold dimensionsThe business key, or nothing for small onesSmall tables don't need clustering
Not sure yetCLUSTER BY AUTO with predictive optimizationDatabricks picks keys from real query history; review the choice later

On other platforms the same decision has different names: Iceberg has hidden partitioning and sort orders, BigQuery has partitioning and clustering, Snowflake has clustering keys. Pick by the same inputs.

Worked example

One dataset, from answers to generated metadata

An ERP's orders table. The comments point to the questions each line answers.

The contract

# contracts/erp-orders.yaml  (abridged and illustrative; check field names against the ODCS 3.2 schema)
apiVersion: v3.1.0
kind: DataContract
id: erp-orders
version: 1.2.0
domain: sales
team:
  - username: priya.s        # source owner (A1)
    role: owner
schema:
  - name: orders
    physicalName: SALES.ORDERS
    description: One row per customer order.
    properties:
      - name: order_no
        logicalType: string
        primaryKey: true      # H1: business key, never reused
        required: true
      - name: customer_id
        logicalType: string   # I1: joins to customers.customer_id
      - name: order_date
        logicalType: date
        description: When the customer ordered (transaction date, H6).
      - name: status
        logicalType: string
        description: See reference table order_status (J1).
      - name: email
        logicalType: string
        classification: restricted   # M1: personal data
      - name: updated_at
        logicalType: timestamp
        description: Set by the database on every change (E1, E2).
    quality:
      - metric: rowCount      # N2: compared with the source's daily control total
slaProperties:
  - property: freshness       # K3
    value: 24
    unit: h

The manifest

# manifest/erp/orders.yaml  (how we load it)
dataset: erp.orders
contract: contracts/erp-orders.yaml
source: {connection: erp_prod_replica, object: SALES.ORDERS}   # B2: read the replica
load:
  pattern: incremental_watermark        # F: decision record DR-014
  watermark_column: updated_at
  overlap: 2h
  full_compare: weekly                  # F5: catch deletes and drift
  window: "01:00-05:00"                 # C1: agreed with the source
  max_rows_per_second: 20000
silver:
  merge_keys: [order_no]
  sequence_by: updated_at               # H7
  tie_break: _ingested_at               # H4
  history: {type2: [status], type1: all_other}   # G2
  deletes: {soft_flag: is_deleted}      # G1
  lookback_days: 35                     # G: late corrections
  cluster_by: [order_no, order_date]    # merge key + date filter

What the deployment generates in Unity Catalog

-- generated by the deployment pipeline from the two files above
COMMENT ON TABLE prod_silver.sales.orders IS 'One row per customer order.';
ALTER TABLE prod_silver.sales.orders ALTER COLUMN order_date COMMENT 'When the customer ordered (transaction date).';
ALTER TABLE prod_silver.sales.orders SET TAGS ('domain' = 'sales', 'owner' = 'priya.s', 'contract' = 'erp-orders@1.2.0');
ALTER TABLE prod_silver.sales.orders ALTER COLUMN email SET TAGS ('classification' = 'restricted');
ALTER TABLE prod_silver.sales.orders ALTER COLUMN email SET MASK governance.masks.email_mask;
ALTER TABLE prod_silver.sales.orders ADD CONSTRAINT orders_pk PRIMARY KEY (order_no);           -- informational
ALTER TABLE prod_silver.sales.orders ADD CONSTRAINT orders_customer_fk
  FOREIGN KEY (customer_id) REFERENCES prod_silver.sales.customers (customer_id);                 -- informational
ALTER TABLE prod_silver.sales.orders CLUSTER BY (order_no, order_date);

The engine then reads merge_keys, sequence_by, history and deletes to build silver. The quality framework turns the contract's rules into a uniqueness test on order_no, an orphan test on customer_id and a daily row-count reconciliation. Unity Catalog's key constraints are informational, so those tests are what actually catch duplicates and orphans.

The question list

Every question, why we ask, and what breaks if nobody does

#Question to chaseWhy we askWhat breaks in production if nobody asksDo this (not this)
WA · Where the answers goone home per answer
WA1Does every answer have exactly one home, and is everything else generated from it?Answers typed into several places drift apart.The key in the code, the catalog and the contract differ, and nobody knows which is right.One home per answer (the table above); the deployment pipeline generates the rest. Not: the same fact typed in three places.
WA2Is anything edited by hand in Unity Catalog?Hand edits are overwritten on the next deploy, or block it.A mask added by hand disappears after a release; personal data is visible for a week.Unity Catalog metadata is generated from git. Not: console edits.
WA3Does the code hold dataset-specific keys, columns or rules?Then every new dataset is a code change.A key change needs a release of the engine.The engine reads the manifest and the contract. Not: if table == 'orders' in code.
WA4Who approves a contract change, and is the contract version checked at deploy time?A contract change is a change to a promise.A column is dropped from the contract and three consumers break.The owner approves in a pull request; breaking changes need a new major version. Not: editing the contract quietly.
WA5How is each big table laid out (liquid clustering, partitioning or Z-order), on which columns, and why?Clustering decides how many files merges and queries touch.Silver merges rewrite a fifth of a 50 TB table every run.Choose from the merge key, the date filter and real query patterns, or CLUSTER BY AUTO; record it in the manifest. Not: clustering on whatever column comes first.

Design decisions

The decisions to take

Where do loading settings live?

OptionVerdict
Contract for promises, manifest for loadingPursue · Pursue. Each file has one job and one owner.
Everything in the ODCS contract, using customPropertiesPursue when · Pursue only for very small platforms; the contract turns into a config file.

What role does Unity Catalog play?

OptionVerdict
Generated target, enforced at run timePursue · Pursue. Git stays the truth; the catalog can be rebuilt.
The place people edit metadataDrop · Drop. Edits are lost or conflict with deploys.

How do we lay out a big table?

OptionVerdict
Liquid clusteringPursue · Pursue for new silver and gold tables on Databricks.
Partition by datePursue when · Pursue only for very large tables that need whole-day or whole-month operations, or that other engines read by folder.
Partition by a high-cardinality column (customer, order)Drop · Drop. Millions of tiny files.
Z-orderPursue when · Pursue only on existing tables until they move to liquid clustering.

How do we choose clustering columns?

OptionVerdict
From the merge key, date filters and query history, recorded in the manifestPursue · Pursue for the big tables.
CLUSTER BY AUTOPursue when · Pursue where predictive optimization is on and query patterns aren't known yet.

FAQ

Questions people ask

Where should data contract information live on Databricks?

In ODCS files in git, one per dataset, as the source of truth. The deployment pipeline generates Unity Catalog comments, tags, key constraints and masks from them, and the pipeline manifest holds how each dataset is loaded.

Which columns should a Delta table be liquid clustered on?

For silver tables with merges, the merge key, plus the most common date filter if there is one. For gold facts, the date and the most filtered dimension key. Small tables need none. If you don't know the query patterns yet, CLUSTER BY AUTO lets Databricks choose from query history.

Should I partition, Z-order or liquid cluster a Delta table?

On Databricks, liquid cluster new tables: it replaces partitioning and Z-order and adapts as query patterns change. Partition only very large tables that need whole-partition operations, on a low-cardinality column such as date. Use Z-order only on existing tables until they are moved to liquid clustering.

Are Unity Catalog primary and foreign keys enforced?

No. They are informational: they document the key and help some query plans, but they don't stop duplicates or orphans. Uniqueness and orphan tests are still needed.

Next

Where to go next

Related: the metadata store (where the manifest is loaded), Layout: catalogs, schemas and naming, and data contracts.

Back up: Upstreams: know your sources, and the printable questionnaire.