Home › Learn › Data systems › Downstreams

Data systems · block 3 of 3 · for data leaders, then builders

Downstreams: know your consumers

Every consumer is named, gets the data it needs through the right door, and hears about changes first.

Who uses the data after the platform, what each of them needs, the doors they come in by, and the questions to ask every consumer, each with the reason and what breaks in production when nobody asks.

For leaders

In plain words

Different taps need different water

A water company serves a hospital, a factory, homes and gardens. The hospital needs it clean and never interrupted. The factory needs pressure at 6 a.m. The garden doesn't mind a cut. A good company knows each customer, what they need, which pipe they're on, and calls them before maintenance. A data platform serves its consumers the same way.

Data platformgold tables and thesemantic layerSemantic layerone meaningper metricSQL and APIsqueries, lookups,servingPush outreverse ETL, sharing,events, filesBI dashboardsreports, KPIsAnalysts, Geniead-hoc questionsAI agentstools over certified dataMLfeatures, trainingAppslookups, recommendationsSaaS toolsCRM, marketingPartners, regulatorsshares, filingsevery consumer is named, with an owner, a need and a way in
Three doors out of the platform, and the consumers behind them.

Kinds of consumers

Eight kinds of consumers

KindExamplesWhat they need mostHow they usually readWhat goes wrong
BI dashboards and reportsSales, finance, operations dashboardsThe same number every time; clear freshnessSemantic layer or metric views; scheduled refreshA metric defined inside a dashboard disagrees with another dashboard
Ad-hoc analysts and GenieAnalysts in SQL; business users asking in plain wordsFindable, documented, certified tables and metricsSQL warehouse; Genie over metric viewsQuestions answered from the wrong table, with confidence
AI agents and assistantsSupport agents, copilots, internal assistantsCertified data through typed tools; documents for retrievalTools over metric views and ops tables; vector searchAn agent writes its own SQL and returns a plausible wrong answer
ML training and featuresChurn, fraud, demand modelsHistory as it was at the time; the same features in training and servingFeature tables; point-in-time joins; online tablesA model trained on values that weren't known yet works in testing and fails live
Operational appsCustomer 360 lookups, recommendations in an appMillisecond reads, always onAn API over a serving database or online tableA dashboard warehouse kept on 24x7 to answer app lookups
SaaS tools (reverse ETL)Scores into the CRM, audiences into marketing toolsFresh, correct records written to another systemReverse ETLA bad load overwrites thousands of CRM records
Partners and other teamsSuppliers, group companies, data products for other domainsGoverned access without copiesDelta Sharing or a marketplace listingEmailed extracts that nobody can revoke
Regulators and finance closeStatutory filings, month-end closeNumbers that reconcile and can be reproduced laterReconciled gold, snapshots, sign-offA filed number can't be reproduced a year later

Integration patterns, outbound

The doors out of the platform

Upstreams can push, be pulled or share; downstreams are the same in reverse. Most consumers should come in through the semantic layer; every other door is chosen on purpose and recorded.

DoorPush or pullGood forWatch for
Semantic layer (metric views)PullEvery metric defined once, used by BI, Genie and agents alikeNeeds owners for definitions; not for row-level exports
SQL warehouse, directPullAnalysts and tools that need tablesWithout the semantic layer, metric logic spreads into every query
BI import or extractPull, copiedFast dashboards on modest dataA second copy with its own freshness; refresh schedules wake warehouses
API or serving tablePull, low latencyApps and agents needing single records fastAlways-on cost; design for the night-time load
Reverse ETLPushGetting data into the tools people work inWrites into someone else's system: test, limit and log every write
Sharing (Delta Sharing, marketplace)Push, no copyPartners and other platformsWho pays for the reads; revoking access
Events outPushOther systems reacting to changes in secondsOrdering, duplicates, schema changes in messages
File exportsPush, copiedLegacy and regulatory recipientsCopies that live forever; hard to revoke

Per consumer

The consumer register: what we record for each

The downstream twin of the upstream source questionnaire. It lives in the platform's metadata, kept current from query history and lineage, and it is how a change upstream is traced to the people it affects.

FieldWhat it isExample
ConsumerThe dashboard, app, model, agent or partnerFinance revenue dashboard
OwnerThe person who answers for itHead of FP&A
Decision it supportsWhat someone does with itWeekly forecast review
Data it usesGold tables and metrics, by namesales_metrics.net_revenue
Freshness neededHow old the data may be, and by whenBy 07:00, yesterday complete
CriticalityWhat happens if it is wrong or lateTier 1: board reporting
Access rulesWho may see which rows and columnsRegion managers see their region
Way inWhich door it usesMetric views via BI
Change noticeHow much warning it needs before a changeTwo weeks
Contact for incidentsWho we tell when something is wrongfpa-oncall

The question list

Every question, why we ask, and what breaks if nobody does

Ask it for each consumer, or each group of similar consumers. Answers go in the consumer register, the semantic layer and the outbound contracts.

#Question to chaseWhy we askWhat breaks in production if nobody asksDo this (not this)
A · Who consumesevery consumer named, with an owner
A1Which dashboards, reports, apps, models, agents, tools and partners use the data, and who owns each?We can't protect consumers we don't know about.A column change breaks a finance report nobody knew existed, on the day of the close.A consumer register, kept current from query history and lineage. Not: finding consumers through complaints.
A2What decision does each consumer support, and what happens if it is wrong or late?Criticality decides service levels, testing and who gets woken up.Every dashboard is treated the same; the board pack fails while effort goes to a dashboard nobody reads.Give each consumer a tier (for example 1 to 3) with matching freshness, testing and support. Not: one service level for everything.
B · What they needfreshness, grain, history, completeness
B1How fresh must the data be for each consumer, and by what time?Freshness is a promise to a person, and it sets the whole chain upstream.Streaming built for a report read once a week, or a daily load for a decision made hourly.Write the freshness need per consumer, then design the hops backwards from it. Not: one schedule for all.
B2At what level of detail do they need it: per transaction, per day, per customer?Grain decides gold table design and cost.A dashboard scans billions of rows to show a daily total.Agree the grain per use; build aggregates for the common ones. Not: raw facts for every dashboard.
B3Do they need history as it was (last year's region), or only as it is now?This decides Type 2 history and as-of joins upstream.Last year's numbers move whenever reference data changes.Name the reports that need as-of history and keep it for the columns they group by. Not: deciding after go-live.
B4Must they see complete periods only, or partial data as it arrives?Partial data looks like a drop in the business.This morning's half-loaded day shows sales down 60% and a manager escalates.Publish complete periods to most consumers; label partial data clearly where it is shown. Not: showing whatever has landed.
C · Meaning and definitionsone meaning per number
C1Which metrics do they use, who owns each definition, and are there competing versions today?Two definitions of revenue in two tools is the most common data argument.Two dashboards show two revenues in one meeting and both teams stop trusting the data.Each metric defined once in the semantic layer, with a named owner. Not: logic in each dashboard.
C2Is business logic hidden in today's reports, spreadsheets or BI models?Migrations break when that logic is left behind.The new dashboard is 'correct' but doesn't match the old one, because the old one excluded test orders.Harvest the logic from current reports and move it into the semantic layer or gold. Not: rebuilding visuals only.
D · Access and securitywho may see what, inside and outside
D1Who may see which rows and columns: by region, by role, with or without personal data?Access rules belong in the platform, not in each tool.A filter in a dashboard is the only thing stopping a region manager seeing every region.Row filters and column masks in Unity Catalog, applied to every door. Not: security inside the BI tool only.
D2Does any data leave the organisation: partners, regulators, vendors, AI services?External use has legal and contractual limits.Customer data is sent to an outside AI service without anyone checking the contract.Approve every external flow, prefer sharing without copies, log what leaves. Not: ad-hoc exports.
E · How they readthe door each consumer uses
E1Through which door does each consumer read: semantic layer, SQL, BI extract, API, reverse ETL, sharing, events, files?The door decides cost, freshness and where logic lives.Every team picks its own way in, and metric logic is copied into each.Default to the semantic layer; other doors on purpose, recorded in the register. Not: direct table access by default.
E2How many people query at once, when, and how heavy are the queries?This sizes warehouses and decides how long they stay on.A warehouse sized for Monday 9 a.m. runs all week, or Monday's queries queue for minutes.Size from peak concurrency and use auto-scaling; schedule refreshes for when people read. Not: one large warehouse always on.
E3Does anything need answers in milliseconds, or around the clock?Low-latency, always-on serving is a different design from dashboards.An app's lookups keep the BI warehouse awake all night.A serving table or database for lookups, sized for night-time load. Not: the dashboard warehouse.
F · AI and agentscertified data through tools
F1Which questions should agents and Genie answer, and from which certified tables and metrics?AI answers are only as right as the data it is allowed to use.Genie answers from a staging table and gives a confident wrong number to an executive.Agents call typed tools over metric views and certified tables; Genie spaces over metric views only. Not: agents writing raw SQL on everything.
F2Which documents do assistants need, who owns them, and how are they kept current?Retrieval is only as current as its documents.An assistant quotes last year's policy.A document register with owners and refresh rules, like the consumer register. Not: a folder uploaded once.
F3How will we test AI answers, and what does each question cost?AI consumers need evaluation and a cost budget like any other.Quality drifts unnoticed and token costs double in a month.A set of known questions with expected answers, run on every change; cost per question tracked. Not: trying a few questions by hand.
G · MLfeatures and training data
G1Do models need features exactly as they were at prediction time, and the same in training and serving?Leakage and skew make models look good and fail live.A churn model uses a field filled in after the customer left; it is perfect in testing and useless in production.Point-in-time joins for training; one feature definition shared by training and serving. Not: features rebuilt separately for each.
G2How often do models retrain, and on how much history?Retraining cadence drives history retention and compute.History needed for retraining was cleaned up to save storage.Agree retention with the model owners. Not: retention set only by storage cost.
H · Change and contractsconsumers are told before things change
H1How do consumers hear about changes, and how much notice do they need?Most breakages are changes nobody announced.A renamed column breaks three dashboards and an agent tool overnight.Outbound contracts for published tables and metrics; change notice from the register; lineage to find who is affected. Not: changes discovered in production.
H2How are tables and metrics retired?Unused things cost money and confuse people; used things can't simply vanish.A table 'nobody uses' is dropped and the finance close fails.Check usage in query history, mark as deprecated, keep a view for a period, then remove. Not: deleting on a hunch.
I · Trust and supportconsumers know when to trust a number
I1How does a consumer know whether today's data is complete and checked?Trust depends on seeing freshness and quality, not on promises.People keep their own spreadsheets because they can't tell when the dashboard is right.Show freshness and a quality status beside each dashboard and metric; mark certified data. Not: silent dashboards.
I2Who do consumers call when a number looks wrong, and how fast is the answer?Unanswered questions turn into lost trust.A wrong number circulates for a week before anyone on the data team hears.A named channel, an incident process and a note to affected consumers from the register. Not: personal messages to whoever built it.
J · Usage and costwho uses what, and what it costs
J1Which dashboards, tables and metrics are actually used, by whom and how often?Usage tells you what to protect and what to retire.Hundreds of dashboards are refreshed every hour; a dozen are read.Usage from query and BI audit logs, reviewed quarterly. Not: keeping everything forever.
J2What does each consumer cost, and who pays?Warehouses, refreshes, serving and AI tokens are consumer-driven costs.One team's 15-minute refreshes double the warehouse bill.Tag consumer workloads and show cost per consumer. Not: one shared bill. See the cost calculator's hop 5.

Design decisions

The decisions this block forces

What is the default way in for consumers?

OptionVerdict
The semantic layer (metric views), with certified gold behind itPursue · Pursue. One meaning per number for BI, Genie and agents.
Direct access to gold tablesPursue when · Pursue for analysts and ML who need rows, with the register recording it.
Each tool defines its own metricsDrop · Drop. Guaranteed disagreement.

How do AI agents reach data?

OptionVerdict
Typed tools over metric views and certified tablesPursue · Pursue. Answers are checkable and permissions are enforced.
The model writes SQL against any tableDrop · Drop. Plausible wrong answers, and access you can't reason about.

How does data reach partners?

OptionVerdict
Sharing without copies (Delta Sharing, marketplace)Pursue · Pursue. Governed, revocable, always current.
File exportsPursue when · Pursue only when the recipient can't read a share; log and expire them.
Emailed extractsDrop · Drop. Uncontrolled copies.

How do apps get fast lookups?

OptionVerdict
A serving table or database sized for the lookupsPursue · Pursue. Milliseconds, without waking the dashboard warehouse.
The BI warehouse kept on 24x7Drop · Drop. Expensive, and still slow for single-record reads.

FAQ

Questions people ask

What are downstream consumers in a data platform?

Everything that uses the data after the platform: dashboards and reports, analysts and Genie, AI agents, ML models, operational apps, SaaS tools fed by reverse ETL, partners through data sharing, and regulators.

Why keep a register of data consumers?

So every consumer has an owner, a freshness need, a criticality tier and a contact. Then a change upstream can be traced to who it affects, service levels match importance, and unused things can be retired safely.

How should AI agents use enterprise data?

Through typed tools over certified tables and metric definitions, not by writing their own SQL. Genie spaces should sit over metric views. Answers are then consistent with dashboards and permissions are enforced.

Next

Where to go next

The other edge: Upstreams: know your sources.

Inside: The data platform, whole; hop 4 (gold to semantic) and hop 5 (semantic to experiences) build what these consumers read.

Cost of serving consumers: hop 5 in the Databricks cost calculator.

Back up: Data systems, end to end.