Home › Learn › The technical anchor › Project review › The checks

Project review · the catalog

The checks: all 210, by category

These are the questions the project review skills ask of a data project's documents, in three packs, one per discipline: 43 business analysis, 64 project management, 91 technical, plus 12 that only apply on Databricks. Each pack is also a skill on its own, and the full review checks the seams between them. Open a check to see why it matters, when it applies, what good evidence looks like, by which stage it is due, and the cheapest test that could prove a team wrong.

To review them: mark each check Keep, Change or Drop and add a note. Your marks stay in this browser. When you're done, Copy my review puts them on your clipboard as plain text to send back. The checks come from the repository, so this page always matches the skill.

Business analysis 43
Project management 64
Technical 91
Databricks only 12

Business analysis (43)

What the business needs, in terms that can be tested against data: requirements, definitions and metrics, data requirements, scope, acceptance.

Read first: Source semantic understanding: a key to data program success. What to ask about the meaning of each source (grain, statuses, corrections, dates, amounts, the number the business trusts) before anything is built.

Requirements, definitions, acceptance criteria (10)

REQ-01Vague words in the requirementshighBusiness analyst
Question
Has every vague word in the requirements, such as "real-time", "fast", "all data", "user-friendly", "accurate" or "complete", been replaced with a number or a test? If the users reject the system at the end, will it be because we each read those words differently?
Why it matters
"Real-time" meant within a second to the sponsor and by the next morning to the team. The design was built for daily loads and had to be rebuilt late in the project.
Applies to
any · components: any · patterns: any
Owner
Business analyst · also Technical anchor, Test lead
Evidence
in: requirements
looks like: Every quality word paired with a measure, for example "available by 7 a.m. on working days" instead of "timely", or "the last three years" instead of "all history".
How the gap shows
  • Words such as real-time, near real-time, fast, quickly, timely, all data, complete, accurate, user-friendly, intuitive, seamless, scalable or robust appear without a number next to them.
  • Lists of required data or reports end with "etc." or "and so on".
Compare
Compare the requirements with the architecture: if the requirement says "real-time" and the architecture schedules a daily batch, someone's reading of the word is wrong.
Due by
planstage 1: Requirement received
designedstage 4: Discovery sign-off
Fail fast
Ask the requester to put a number on each vague word in a fifteen-minute call, and write the answers down.
Script signal
vague-words (found without AI)
Related
REQ-02, REQ-06
REQ-02No acceptance criteriahighBusiness analyst
Question
Does every requirement have acceptance criteria that the data can be tested against? At user acceptance, how will we and the business agree that a requirement is met?
Why it matters
Requirements said "show monthly sales by region". At user acceptance the business compared it with their own report, got a different number, and nobody had agreed in advance which one was right.
Applies to
any · components: any · patterns: any
Owner
Business analyst · also Test lead, Technical anchor
Evidence
in: requirements, test-strategy
looks like: For each requirement, one or more criteria in the form "given this data, the result is this", ideally with a known figure the business already trusts.
How the gap shows
  • Requirements or user stories have no "acceptance criteria" or "done when" section.
  • Acceptance criteria repeat the requirement in other words rather than describing a test.
  • Backlog items are marked ready for build without criteria.
Compare
Compare the requirements with the test strategy: requirements that have no matching test, or tests that trace to no requirement, show the two were written apart.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
Fail fast
Take the three most important requirements and write a test for each with the business owner, using a figure from a report they already trust.
Script signal
no-acceptance-criteria (found without AI)
Related
REQ-01, REQ-03
REQ-03Business definitions not agreed and signedhighBusiness analyst
Question
Are the key business definitions (for example revenue, active customer, order date) written down and signed by the person who owns each one? If two departments see different numbers after go-live, which signed definition settles it?
Why it matters
Sales counted revenue when the order was booked; finance counted it when it was paid. Both signed the requirements, which said "revenue", and the dashboard was disowned by one of them in the first week.
Applies to
any · components: modelling, serving · patterns: any
Owner
Business analyst · also Business owner and sponsor, Data consumers, Technical anchor
Evidence
in: data-model, decisions
looks like: A definitions list with the exact rule for each term, the grain, the owner who signed it and the date.
How the gap shows
  • The data dictionary has names but no rules, owners or sign-off.
  • The same term is used in requirements from different departments with no definition.
  • Definitions are marked "to be confirmed" in a document that has passed discovery sign-off.
Compare
Compare the requirements with the data model: if a term in the requirements has a different rule, or no rule, in the data model, the definition was never agreed.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
Fail fast
Ask two of the main consumers to write their own definition of the top five terms separately, then compare the answers.
Related
REQ-02, STK-02
REQ-04Requirements written without looking at the datahighBusiness analyst
Question
Were the requirements checked against a profile of the real source data? If a requirement turns out to be impossible, will we find out in discovery or in testing?
Why it matters
The requirement asked for sales by product category. The source had no category for a third of products, which only came to light when the report was tested.
Applies to
any · components: any · patterns: any
Owner
Business analyst · also Technical anchor, Source system owners
Evidence
in: discovery, requirements
looks like: A data profile per source (volumes, keys, nulls, odd values) and a note against each requirement saying which source fields support it and any gaps found.
How the gap shows
  • No data profiling section or results in the discovery document.
  • Requirements name fields or attributes that do not appear in any source description.
  • Phrases such as "data to be confirmed" or "assumed available" against required fields.
Compare
Compare the requirements with the data model or source mapping: a required field with no source mapped to it is a requirement nobody checked against the data.
Due by
planstage 3: Discovery
evidencedstage 4: Discovery sign-off
Fail fast
Pull one real extract from each source and map the top ten requirements to actual columns in an afternoon.
Related
EST-07, REQ-03
REQ-05The users are not the requestersmediumBusiness analyst
Question
Have the people who will use the output every day been asked what they need, or only the people who requested it? If nobody uses the system three months after go-live, will it be because it answered the requester's question and not the users'?
Why it matters
A director requested a dashboard and approved the design. The analysts who were meant to use it needed row-level detail to act on, kept their old reports, and the dashboard went unused.
Applies to
any · components: serving, modelling · patterns: any
Owner
Business analyst · also Data consumers, Business owner and sponsor
Evidence
in: requirements, stakeholders
looks like: Named users or user groups, how they will use the output, which decisions it supports, and evidence that some of them took part in workshops and will take part in user acceptance testing.
How the gap shows
  • Requirements name only a sponsor or requester, not users.
  • No user workshops, interviews or user personas are mentioned.
  • User acceptance testers are not named, or are the same person as the requester.
Compare
Compare the stakeholder map with the requirements sources: user groups on the map who are not listed as contributors to the requirements were never asked.
Due by
planstage 1: Requirement received
designedstage 4: Discovery sign-off
evidencedstage 7: Test
Fail fast
Show a rough mock-up or a real sample of the output to three actual users and ask what decision they would make with it.
Related
STK-01, STK-03
REQ-06Non-functional needs missinghighBusiness analyst
Question
Are freshness, volume and growth, recovery time and acceptable data loss, retention, and who may see what written as requirements with numbers? If the system works but is rejected, will it be because it is too slow, too stale, or unrecoverable?
Why it matters
Nobody asked how fresh the data had to be or how quickly it had to recover. After the first overnight failure the business found out it would take a day to recover, and the system was not accepted.
Applies to
any · components: any · patterns: any
Owner
Business analyst · also Technical anchor, Operations and support, Security and governance
Evidence
in: requirements, architecture
looks like: A short non-functional list: freshness deadline, expected volume and growth, how long it may be down, how much data may be lost, how long data is kept, and access rules.
How the gap shows
  • No "non-functional" heading in the requirements.
  • Freshness, recovery or retention are mentioned only as "to be agreed".
  • Volume is described in words ("large") rather than numbers.
Compare
Compare the requirements with the runbook or support model: if the runbook promises recovery within hours but no requirement says what is needed, or the reverse, the two disagree on what was promised.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 7: Test
Fail fast
Ask the business owner: "If this is down for a day, what happens?" and write down the answer as a number.
Related
REQ-01, HAND-04
REQ-07Goals with no measure of successmediumBusiness analyst
Question
What business result is this project for, how will it be measured, and what is the measure today? A year after go-live, how will anyone know whether it was worth it?
Why it matters
The goal was "better insight into customers". No measure was agreed, so at closure the project could not show any benefit, and the next request for funding was refused.
Applies to
any · components: any · patterns: any
Owner
Business analyst · also Business owner and sponsor
Evidence
in: discovery, requirements
looks like: Two or three goals, each with a measure, today's value (the baseline), a target, and where the data for measuring it will come from.
How the gap shows
  • Goals are written as activities ("build a data platform") rather than results.
  • No baseline figures are given for any goal.
  • Success measures need data the project does not collect.
Compare
Compare the goals in the discovery document with the scope: a goal whose measure depends on data or reports that are out of scope cannot be measured.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
provenstage 10: Project sign-off
Fail fast
Ask the sponsor to state the one number they hope will change, and check today whether that number can be produced.
Related
HAND-05
REQ-08Everything is a "must"highBusiness analyst
Question
Are the requirements prioritised (for example must, should, could and won't), with real items in the "won't" list and fewer musts than the first release can deliver? If time runs out, has the business already said which requirements go first?
Why it matters
Every one of sixty requirements was marked essential. When the build ran late the team cut by effort instead of value, dropped the one report the finance director actually used, and the release was rejected.
Applies to
any · components: any · patterns: any
Owner
Business analyst · also Business owner and sponsor, Project manager
Evidence
in: requirements, plan
looks like: Each requirement with a priority agreed by the business owner, a "won't this time" list with real items in it, and the musts sized to fit the first release with room to spare.
How the gap shows
  • No priority column in the requirements, or every requirement has the same priority.
  • The "won't" or "not this release" list is empty.
  • Musts make up most of the requirements.
Compare
Compare the must-have requirements with the first release in the plan: musts that are not in the first release, or a first release full of shoulds and coulds, mean priority and plan disagree.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
Fail fast
Ask the business owner: "If we could deliver only five of these, which five?" and record the answer as the priority.
Related
SCOPE-05, SCOPE-01
REQ-09No trace from goal to requirement to testmediumBusiness analyst
Question
Can every requirement be traced back to a business goal or decision and forward to the data it needs, the thing that builds it and the test that proves it? When someone asks why a table exists, or whether a goal has been tested, can the team show it in minutes?
Why it matters
A requirement for weekly churn by segment was dropped when the backlog was re-planned. Nothing linked it to the retention goal it served, so nobody noticed until the sponsor asked for the churn figure at closure.
Applies to
any · components: any · patterns: any
Owner
Business analyst · also Test lead, Technical anchor
Evidence
in: requirements, test-strategy
looks like: A simple trace table: goal, requirement, sources and fields, backlog item, test case, and status, kept up to date as items are added, changed or dropped.
How the gap shows
  • Requirements have no ids, or ids are not used in the backlog or test cases.
  • Goals and requirements live in separate documents with no link between them.
  • Test cases do not say which requirement they prove.
Compare
Compare the requirement ids in the requirements with those referenced in the test strategy: requirements with no test, and tests pointing at no requirement, show where the trace is broken.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
Fail fast
Pick the main goal and try to list, in ten minutes, every requirement and test that serves it.
Related
REQ-02, REQ-07
REQ-10Requirements change without a version or a trailmediumBusiness analyst
Question
When a requirement, rule or definition changes, is the change versioned, dated, approved by its owner, and carried through to the data model, the backlog and the tests? If a tester tests against last month's version of a rule, will anyone know?
Why it matters
The definition of an active customer changed twice in workshops. The data model held the first version, the tests the second, and the report the original, and user testing failed on all three.
Applies to
any · components: any · patterns: any
Owner
Business analyst · also Test lead, Developers, Project manager
Evidence
in: requirements, decisions
looks like: Requirements with a version and date, a change history per requirement or rule, the approver of each change, and a note of which designs and tests were updated.
How the gap shows
  • The requirements document has no version, date or change history.
  • Definitions or rules differ between the requirements, the data model and the test cases.
  • Changes are agreed in meetings or chat and not reflected in the requirements.
Compare
Compare the rule or definition text in the requirements with the same rule in the data model and test cases: any difference means a change reached some documents and not others.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
Fail fast
Take the three most-discussed definitions and check that the requirements, data model and tests say the same thing today.
Related
SCOPE-03, REQ-09, OWN-02

In scope, out of scope, change control (4)

SCOPE-01Scope written only as "in"highBusiness analyst
Question
Is there a written list of what is out of scope, as well as what is in? When someone says "I assumed that was included", which document will we point to?
Why it matters
The scope listed the tables to build. The business assumed the reports on top were included too, and the argument cost a month and the sponsor's trust.
Applies to
any · components: any · patterns: any
Owner
Business analyst · also Project manager, Technical anchor, Business owner and sponsor
Evidence
in: discovery, requirements
looks like: Three lists, in scope, out of scope and later, each item specific enough that a stranger could say which list a new request belongs to.
How the gap shows
  • No "out of scope" heading or list in the discovery, scope or requirements document.
  • The out-of-scope list is generic ("anything not listed above").
Compare
Compare the scope with the requirements: a requirement that is neither in the in-scope list nor explicitly out is a future argument.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
Fail fast
Ask the business owner to read only the out-of-scope list and say which item surprises them.
Script signal
no-out-of-scope (found without AI)
Related
SCOPE-02, SCOPE-06
SCOPE-02The data scope traps left silenthighTechnical anchor
Question
Has each of the ten data scope traps been written down as in, out or later: history and backfill, deletes at the source, late and corrected data, changes over time, bad rows, source data quality, personal data, reports and dashboards, access and onboarding, and recovery? Which of them will someone assume was included?
Why it matters
Nobody wrote down whether deleted customers should disappear from the reports. The business assumed yes, the team assumed no, and the gap was found by an auditor.
Applies to
any · components: any · patterns: any
Owner
Technical anchor · also Business analyst, Business owner and sponsor
Evidence
in: discovery, requirements
looks like: A table with one row per trap, the decision (in, out or later), who decided and why.
How the gap shows
  • Several of the words history, backfill, delete, late, correction, bad rows, quality, personal data, reports, access and recovery do not appear in the scope section.
  • Traps are mentioned but marked "to be confirmed" after discovery sign-off.
Compare
Compare the scope with the architecture: a trap the architecture handles (for example history kept) that the scope does not mention, or the reverse, shows the two were never reconciled.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
Fail fast
Walk the ten traps with the business owner in thirty minutes and record each answer.
Related
SCOPE-01, EST-07
SCOPE-04"Phase 2" used as a hiding placemediumProject manager
Question
Is everything labelled "phase 2" or "later" actually planned, funded and owned, and does the business know that phase 2 may never happen? If phase 2 is cancelled, will the first phase still be useful on its own?
Why it matters
Hard items such as history and data quality were moved to phase 2 to make the first phase look affordable. Phase 2 was never funded, and phase 1 was unusable without them.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Business owner and sponsor, Business analyst
Evidence
in: plan, discovery
looks like: A "later" list in which each item states whether phase 1 depends on it, an owner, and whether funding exists, plus a statement that phase 1 is usable without them.
How the gap shows
  • Many items marked "phase 2", "future phase" or "later" with no dates or owners.
  • Phase 2 has no estimate or budget.
  • Items that the main goal depends on are listed under phase 2.
Compare
Compare the goals or success measures with the phase-2 list: a goal that can only be met by phase-2 items means phase 1 alone does not deliver the value promised.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
Fail fast
Ask the sponsor: "If there is no phase 2, would you still pay for phase 1 as written?"
Related
SCOPE-05, REQ-07
SCOPE-06Scope not signed by the business ownermediumProject manager
Question
Has the person who will judge success signed the in, out and later lists, and is that signature recorded with a date? If scope is disputed later, whose signature ends the argument?
Why it matters
The scope was agreed with a delegate. The real business owner saw it for the first time at user acceptance and rejected half of the out-of-scope list.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Business owner and sponsor, Business analyst
Evidence
in: decisions, discovery
looks like: A dated sign-off record naming the business owner, the version of the scope signed, and any conditions.
How the gap shows
  • The scope document has no approver, approval date or version.
  • Approval is recorded as "agreed in meeting" without a name.
  • The approver is not the business owner named in the stakeholder map.
Compare
Compare the scope sign-off with the stakeholder map: if the person who signed is not the one the map names as the decision owner, the sign-off may not hold.
Due by
planstage 4: Discovery sign-off
evidencedstage 4: Discovery sign-off
Fail fast
Send the out-of-scope list to the business owner today and ask for a written yes.
Related
SCOPE-01, STK-05

How the business gets the numbers today - reports replaced, trusted figures, manual adjustments, decisions served (4)

CUR-01The current process and reports not written downhighBusiness analyst
Question
Is the way the business produces this information today written down: which reports, spreadsheets, extracts and manual steps, who runs them, how often, and who reads them? If an old report keeps being used after go-live, will it be one we never knew existed?
Why it matters
The project replaced the monthly sales pack. Nobody had mapped the twelve spreadsheets that fed it, and one of them carried a manual rebate adjustment, so the new numbers were wrong by exactly that amount and the old pack stayed in use.
Applies to
any · components: modelling, serving · patterns: any
Owner
Business analyst · also Data consumers, Business owner and sponsor
Evidence
in: discovery, requirements
looks like: A current-state inventory: one row per report, spreadsheet or extract being replaced, with its producer, frequency, readers, the sources it pulls from and the manual steps in between.
How the gap shows
  • The discovery document describes the future solution but has no "current state" or "as is" section.
  • Reports to be replaced are mentioned by name only, with no producer, frequency or readers.
  • Phrases such as "replaces existing reporting" with no list of what that reporting is.
Compare
Compare the current-state inventory in the discovery document with the in-scope list: a report or spreadsheet in use today that is neither replaced nor declared out of scope will keep running beside the new system.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
Fail fast
Sit for one hour with the person who produces today's main report and write down every step, file and manual change they make.
Related
CUR-02, CUR-04, RLC-03
CUR-02The number the business trusts today not identifiedhighBusiness analyst
Question
For each key figure, which report or person does the business trust today, and has the business agreed that the new system will be judged against it? When the new number differs from the old one, who will decide which is wrong?
Why it matters
The new revenue figure was two per cent lower than finance's ledger. Nobody had agreed in advance that the ledger was the reference, so three weeks were lost arguing about whose number was right before anyone looked for the cause.
Applies to
any · components: modelling, serving · patterns: any
Owner
Business analyst · also Business owner and sponsor, Data consumers, Test lead
Evidence
in: requirements, test-strategy
looks like: For each key figure, the trusted reference (report, ledger, application screen), who owns it, a sample value for a known period, and whether the business believes that reference is itself correct.
How the gap shows
  • No reference report or trusted figure is named anywhere in the requirements or test strategy.
  • The requirements say the new figures must be "accurate" or "correct" without saying compared with what.
  • The trusted figure is named but nobody has captured a value for a known period.
Compare
Compare the requirements with the test strategy: if the requirements name a trusted report but the reconciliation tests compare against something else, or nothing, acceptance will be argued, not tested.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 7: Test
Fail fast
Ask the business owner for last month's value of the top three figures and where they got it, and store the screenshots with the requirements.
Related
REC-02, TEST-04, UAT-01
CUR-03Decisions the data will drive not namedhighBusiness analyst
Question
Which decisions will people make with this data, how often, and what do they do today instead? If the project delivers every table and report on time but nothing changes in the business, will it be because nobody said which decision it was for?
Why it matters
The team built a full customer model because "the business needs a single view of the customer". The one decision it was actually needed for, which customers to call each Monday, required a field that was never captured.
Applies to
any · components: any · patterns: any
Owner
Business analyst · also Business owner and sponsor, Data consumers
Evidence
in: discovery, requirements
looks like: A short list of decisions, each with who makes it, how often, what information they need to make it, what they use today, and what is painful about that.
How the gap shows
  • Requirements list data and reports but never the decision or action they support.
  • The problem statement is about technology ("legacy reporting", "move to the cloud") rather than a business pain.
  • Words such as "insight", "visibility" or "single view" appear without a named decision.
Compare
Compare the decisions in the discovery document with the freshness and grain in the requirements: a daily decision served by monthly data, or a decision about individual customers served by totals, means the two were not connected.
Due by
planstage 1: Requirement received
designedstage 3: Discovery
Fail fast
Ask three intended users: "What will you do differently on the day this is live?" and write the answers against the requirements.
Related
REQ-07, REQ-05, DATA-04
CUR-04Manual adjustments hidden in today's numbershighBusiness analyst
Question
Do today's trusted figures include manual adjustments, hand-kept mappings, overrides or exclusions that live in someone's spreadsheet or head, and has each one been listed with a decision to automate it, keep it manual, or drop it? If the new numbers never match the old ones, will it be because of a correction nobody told us about?
Why it matters
Each month an analyst removed two intercompany customers and reclassified a product line by hand before the board pack went out. The new system did neither, and it took a month of reconciliation to find out why.
Applies to
any · components: transformation, modelling · patterns: any
Owner
Business analyst · also Data consumers, Technical anchor
Evidence
in: discovery, requirements
looks like: A list of every manual adjustment, override, exclusion and hand-kept mapping in today's process, with its size, who makes it, why, and whether the new system will automate it, keep it as a governed manual input, or stop it.
How the gap shows
  • The current state is described as "extracted from the source system" with no manual steps.
  • Spreadsheets are mentioned as inputs to today's report with no description of what they change.
  • No requirement covers manual inputs, overrides or exclusions.
Compare
Compare the current-state description with the business rules or data model: an adjustment people make today that has no rule, no manual input and no "dropped" decision will show up as an unexplained difference.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
Fail fast
Take last month's trusted figure and the raw source total for the same period; ask the report owner to explain the difference line by line.
Related
CUR-01, CUR-02, RULE-01

The business side of the data needed - sources and owners, grain, history, freshness, late data, source of truth (7)

DATA-01Sources not named, each with an ownerhighBusiness analyst
Question
Is every source the requirements depend on named, with the system, the data needed from it, and a named person who owns it and can answer questions about it? If a source turns out to be unavailable in the build, will we find out then, or now?
Why it matters
The requirements said "customer and order data". In the build the team learned that returns lived in a separate warehouse system nobody had listed, owned by a team that had no time for the project that quarter.
Applies to
any · components: ingestion · patterns: any
Owner
Business analyst · also Source system owners, Technical anchor
Evidence
in: requirements, discovery
looks like: A source list: one row per source with the system, the business data it provides, the requirements that need it, the source owner's name, and whether access has been requested.
How the gap shows
  • Data needs are described as subjects ("customer data", "sales data") with no system named.
  • Sources are listed without an owner, or with a team name instead of a person.
  • A source appears in the architecture that is not in the requirements, or the reverse.
Compare
Compare the source list in the requirements with the sources drawn in the architecture: a source in one and not the other is either unneeded work or a missing dependency.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
Fail fast
For each requirement, write the source system it comes from; any requirement you cannot fill in is a gap to raise this week.
Related
REQ-04, OWN-01
DATA-02Grain the users need not compared with the grain the sources holdhighBusiness analyst
Question
For each requirement, what level of detail do the users need (each transaction, each customer per day, each store per month), and does the source hold data at that level? If users ask to drill into a total after go-live, will the detail exist?
Why it matters
Users needed sales by individual product. The source only kept daily totals by product group, which nobody checked until the first drill-down in user testing.
Applies to
any · components: modelling, serving · patterns: any
Owner
Business analyst · also Data consumers, Technical anchor
Evidence
in: requirements, data-model
looks like: For each key requirement, the grain needed in plain words (for example "one row per order line per day"), the grain available at the source, and how any gap was resolved.
How the gap shows
  • Requirements say "by product", "by customer" or "detail" without saying the lowest level needed.
  • Drill-down or "detail" is requested with no statement of how far down.
  • The data model's grain is finer or coarser than anything stated in the requirements.
Compare
Compare the grain in the requirements with the grain of the tables in the data model: a requirement needing more detail than its table holds cannot be met.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 7: Test
Fail fast
Show users one real row from the source and ask: "Is this the smallest thing you will ever need to see?"
Related
DES-04, REQ-04
DATA-03History depth asked for without a reasonmediumBusiness analyst
Question
How many years of history does the business need, for which data, and what use requires that depth (year-on-year comparison, trend models, regulation)? If the history load doubles the estimate, will we know which part of it the business could live without?
Why it matters
The business asked for "all history". Ten years were loaded at great cost, the oldest seven had different product codes nobody could map, and the only use was a two-year comparison.
Applies to
any · components: ingestion, modelling · patterns: any
Owner
Business analyst · also Business owner and sponsor, Technical anchor
Evidence
in: requirements, discovery
looks like: For each data area, the years of history needed, the use that needs them, whether old data must match today's definitions, and who will check the old numbers.
How the gap shows
  • History is requested as "all", "full" or "as much as possible".
  • A number of years is given with no reason.
  • Nobody is named to check that the loaded history matches what the business remembers.
Compare
Compare the history depth in the requirements with the history load in the estimate or plan: a different number of years, or none in the estimate, means the cost of history was never seen by the business.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
Fail fast
Ask: "Which report or decision needs data older than two years?" and price each extra year before agreeing.
Related
HIST-01, EST-07, SCD-01
DATA-04Freshness not tied to a decisionhighBusiness analyst
Question
How fresh must the data be, as a time of day or a delay in minutes or hours, and which decision or action needs it that fresh? If the business later asks for faster data, will we know whether the decision really needs it?
Why it matters
The sponsor asked for hourly data. The only decision it served was a weekly stock order, so the team built and paid for hourly loads that nobody looked at between Mondays.
Applies to
any · components: any · patterns: any
Owner
Business analyst · also Business owner and sponsor, Data consumers, Technical anchor
Evidence
in: requirements
looks like: For each consumer or report, a freshness target such as "ready by 7 a.m. local time on working days" or "within fifteen minutes of the sale", paired with the decision that needs it and what happens if it is late.
How the gap shows
  • Freshness is stated as "real-time", "up to date" or "latest" with no number.
  • One freshness target is stated for everything, with no difference between consumers.
  • No reason or decision is given for the freshness asked for.
Compare
Compare the freshness in the requirements with the decisions in the discovery document: data needed faster than the decision it serves is cost with no benefit, and slower is a decision made on stale data.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 7: Test
Fail fast
Ask the user: "If this arrived at noon instead of 7 a.m., what would you do differently?" and price the two options side by side.
Related
REQ-01, REQ-06, PERF-02, CUR-03
DATA-05No business rule for late and corrected datahighBusiness analyst
Question
When data arrives late or a past figure is corrected, what should the business see: should last month change, which date decides the period (order, shipment, invoice or posting date), and when is a period closed and never restated? If a published figure changes after the board has seen it, has the business agreed that it may?
Why it matters
Returns posted in March for February sales changed February's revenue after it had gone to the board. The finance director had assumed closed months never change, the team had assumed they always do, and neither was written down.
Applies to
any · components: modelling, serving · patterns: any
Owner
Business analyst · also Business owner and sponsor, Data consumers, Technical anchor
Evidence
in: requirements, decisions
looks like: A short restatement policy: which date assigns a record to a period, how far back corrections may change published figures, when a period closes, and how users are told that a past figure changed.
How the gap shows
  • The requirements never mention late data, corrections, restatement, closing or period end.
  • The date that decides the reporting period is not stated for any measure.
  • Late and corrected data is marked as a technical matter only.
Compare
Compare the restatement policy in the requirements with the late-data handling in the architecture: if the architecture rebuilds past periods but the business expects closed periods to be frozen, or the reverse, one of them is wrong.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 7: Test
Fail fast
Show the business owner one real correction from last month and ask which month it should change and whether anyone must be told.
Related
LATE-01, LATE-04, SCOPE-02
DATA-06No source of truth when systems disagreehighBusiness analyst
Question
When two sources hold the same thing differently (a customer's address in the sales system and the billing system, a product's price in two catalogues), which one wins, for which fields, and who decided? When a user finds the two disagree after go-live, which written decision answers them?
Why it matters
The sales and billing systems had different customer segments. The team picked one, the business assumed the other, and every segment report was disputed until the decision was made at steering level.
Applies to
any · components: transformation, modelling · patterns: any
Owner
Business analyst · also Business owner and sponsor, Source system owners, Technical anchor
Evidence
in: decisions, data-model
looks like: For each shared business entity, a recorded decision on which source wins for which fields, what happens when a record exists in only one, and who owns the decision.
How the gap shows
  • The same entity (customer, product, employee, site) comes from more than one source with no rule for which wins.
  • Phrases such as "merge customer data" or "single customer view" with no matching rule.
  • The decision log has no entry about sources of truth.
Compare
Compare the source list with the decision log: every entity supplied by two or more sources needs a decision recorded; any without one will be decided silently in code.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
Fail fast
Pull twenty customers that exist in both systems, show the differences to the business owner, and ask which column wins for each field.
Related
DATA-01, OWN-01, DUP-03
DATA-07As it was, or as it is now, not decidedmediumBusiness analyst
Question
When something changes over time (a customer moves region, a product changes category, a salesperson changes team), should past figures be reported under the old value or the new one, and for which reports? When last year's numbers move because of a reorganisation, will the business call it a bug?
Why it matters
After a sales reorganisation, last year's revenue by region changed overnight because history was shown under the new regions. Finance needed it as it was originally reported, and the model had to be rebuilt.
Applies to
any · components: modelling, serving · patterns: any
Owner
Business analyst · also Data consumers, Technical anchor
Evidence
in: requirements, data-model
looks like: For each attribute that can change (region, category, team, status), whether reports should show the value at the time, the current value, or both, with a worked example the business has agreed.
How the gap shows
  • The requirements never say whether history is shown "as it was" or "as it is now".
  • Hierarchies or organisation structures are in scope with no mention of reorganisations.
  • The data model keeps history of changes for attributes the requirements never ask about, or the reverse.
Compare
Compare the requirements with the data model: an attribute the business wants "as it was" that the model stores only as the latest value has lost its history the day it changes.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 7: Test
Fail fast
Take one real customer who changed region last year and ask the users which region last year's sales should appear under.
Related
SCD-01, SCD-04

Each metric fully defined, departmental conflicts resolved, a business glossary, roll-up rules (4)

MET-01Metric defined by name, not by formulahighBusiness analyst
Question
Does every key metric have a complete definition: the formula, what is included and excluded, the grain, which date puts it in a period, the calendar used, the currency or unit and how amounts are converted? If two developers built it separately from the definition, would they get the same number?
Why it matters
"Net sales" was defined as "sales minus returns". It did not say whether tax, discounts or shipping were included, or which exchange rate applied, so the number came out three different ways in three reports.
Applies to
any · components: modelling, serving · patterns: any
Owner
Business analyst · also Data consumers, Technical anchor, Test lead
Evidence
in: data-model, requirements
looks like: A definition card per metric: formula, inclusions and exclusions, grain, the date that decides the period, calendar (fiscal or calendar), currency or unit and conversion rule, a worked example with real numbers, an owner and a sign-off date.
How the gap shows
  • Metrics are listed by name with a one-line description and no formula.
  • Definitions never mention tax, discounts, returns, cancellations, test accounts or currency.
  • No metric has a worked example with numbers.
Compare
Compare the metric definitions in the data model with the calculations described in report specifications or test cases: a calculation that adds a filter or rule the definition does not have means the definition is incomplete.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 7: Test
Fail fast
Give the definition of one key metric to two people separately, ask each to calculate it from the same sample, and compare.
Related
REQ-03, SERV-04, MET-02
MET-02Conflicting definitions across departments left hiddenhighBusiness analyst
Question
Have the departments that use the same term (customer, revenue, headcount, active) been asked for their definition, and are the differences written down and resolved, either into one definition or into clearly different names? When sales and finance show different numbers for the same word, has that argument already happened?
Why it matters
Marketing counted an active customer as anyone who opened an email in ninety days; finance as anyone who paid in twelve months. The dashboard used finance's rule, marketing stopped trusting it, and built their own.
Applies to
any · components: modelling, serving · patterns: any
Owner
Business analyst · also Business owner and sponsor, Data consumers
Evidence
in: data-model, decisions
looks like: For each shared term, the definition each department uses today, the differences, and the decision: one shared definition, or separate named measures (for example "booked revenue" and "invoiced revenue").
How the gap shows
  • Only one department was consulted on definitions that several departments use.
  • The same word appears in requirements from different teams with no note on whether it means the same.
  • No decision log entry records a resolved definition conflict.
Compare
Compare the stakeholder map with the definition sign-offs: a department that uses a term but did not sign its definition has not agreed to it.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
Fail fast
Email the top five terms to one person in each department that uses them, ask for their definition in one sentence, and put the answers side by side.
Related
REQ-03, STK-04, OWN-02
MET-03No business glossarymediumBusiness analyst
Question
Is there one glossary of the business terms used in the requirements, reports and data model, each with a plain definition and an owner, kept where users and the team both look? When a new analyst asks what a term means, is there one place to point them?
Why it matters
Three documents used "order", "booking" and "sale" for overlapping things. The developers treated them as the same, the users did not, and the confusion only came out in user testing.
Applies to
any · components: modelling, serving, governance · patterns: any
Owner
Business analyst · also Data consumers, Technical anchor
Evidence
in: data-model, requirements
looks like: A glossary listing each business term, its plain definition, synonyms people use, the owner, and the fields or metrics it maps to.
How the gap shows
  • Requirements use several words for what may be the same thing (order, booking, sale).
  • Abbreviations and internal jargon appear with no definition.
  • The data dictionary describes columns but not business terms.
Compare
Compare the words used in the requirements with the names in the data model: a business term with no match in the model, or a model name no business person would recognise, shows the two speak different languages.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
Fail fast
List every noun in the top ten requirements and ask the business owner which are the same thing.
Related
MET-01, HAND-02
MET-04Totals that cannot simply be added upmediumBusiness analyst
Question
For each metric that is a count of distinct things, a ratio, an average, a percentage or a balance at a point in time, has the business said how it rolls up across days, regions and products? When a user adds up the regional figures and gets a different total, will the definition explain why?
Why it matters
Daily unique customers were summed to give a monthly figure, counting a customer who bought on ten days ten times. Margin percentages were averaged instead of recalculated. Both errors reached the board pack.
Applies to
any · components: modelling, serving · patterns: any
Owner
Business analyst · also Data consumers, Technical anchor, Test lead
Evidence
in: data-model, requirements
looks like: Each metric marked as additive, partly additive (for example a stock balance that adds across stores but not across days) or not additive, with the rule for totals and a worked example.
How the gap shows
  • Ratios, averages, distinct counts or balances are defined with no rule for totals.
  • Requirements ask for the same metric "by day, week and month" with no word on how it rolls up.
  • Test cases check only the lowest level, not totals.
Compare
Compare the metric definitions with the report designs: a report that shows totals or subtotals for a metric with no roll-up rule will show a number nobody defined.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 7: Test
Fail fast
For each ratio or distinct count, calculate the monthly total two ways on a real sample and show the business owner the difference.
Related
MET-01

Business rules as testable rules with examples, exceptions, and what users see when data is bad or missing (3)

RULE-01Business rules written as prose, not as testable ruleshighBusiness analyst
Question
Is every business rule (how a customer is classified, when an order counts, how a discount is applied) written as a precise rule with at least one worked example of input and expected result? If a tester and a developer read the rule separately, would they agree on the answer for a tricky case?
Why it matters
The rule said "VIP customers get priority". Nobody wrote down that VIP meant spend over a threshold in the last twelve rolling months, so developers used the calendar year and a fifth of VIPs were missing.
Applies to
any · components: transformation, modelling · patterns: any
Owner
Business analyst · also Test lead, Developers, Data consumers
Evidence
in: requirements, test-strategy
looks like: Each rule numbered, stated as a condition and a result, with two or three worked examples using real values, including at least one edge case, and the person who confirmed it.
How the gap shows
  • Rules are embedded in long paragraphs rather than listed and numbered.
  • Rules have no examples, or only the obvious case.
  • Words such as "usually", "normally", "where appropriate" or "as needed" appear in rules.
Compare
Compare the business rules in the requirements with the test cases: a rule with no test, or a test that checks something no rule states, shows the rules were never made testable.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
Fail fast
Pick the three most complex rules and ask the business owner to work one real record through each by hand.
Related
REQ-02, RULE-03
RULE-02What the user sees when data is bad or missing not decided by the businesshighBusiness analyst
Question
When a value is missing, a record fails a check, or a source is late, what should the user see: a blank, a placeholder such as "unknown", yesterday's figure, a warning, or nothing at all? Has the business chosen, for each key report? If a manager makes a decision on incomplete data, will the report have told them?
Why it matters
A store's sales file failed to load. The dashboard showed the store at zero without any warning, and a regional manager phoned the store manager to ask why nothing had been sold.
Applies to
any · components: modelling, serving · patterns: any
Owner
Business analyst · also Data consumers, Technical anchor
Evidence
in: requirements
looks like: For each key report or consumer, what is shown for missing values, rejected rows, late sources and partial periods, agreed with the users, with how they are told the data is incomplete.
How the gap shows
  • The requirements describe only the happy path.
  • Rules for bad data exist in the technical design but not in the requirements.
  • Words such as "unknown", "missing", "incomplete", "partial" or "late" never appear in the requirements.
Compare
Compare the requirements with the data quality design: if the design quarantines or drops rows but the requirements never say what users should see when that happens, users will see silent gaps.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 7: Test
Fail fast
Show users a mock-up of a key report with one region missing and ask what they expect it to look like.
Related
DQ-02, SERV-02, DQ-04
RULE-03The exceptions nobody mentionedmediumBusiness analyst
Question
Have the unusual cases been asked about and given rules: returns, cancellations, refunds, credit notes, intercompany sales, test and internal accounts, free items, split shipments, merged customers? Which exception will the users spot in the first week of testing?
Why it matters
Nobody asked about staff purchases. They were twelve per cent of transactions at one site, were excluded from every report the business trusted, and were included in every number the new system produced.
Applies to
any · components: transformation, modelling · patterns: any
Owner
Business analyst · also Data consumers, Source system owners
Evidence
in: requirements, discovery
looks like: A list of exception types found in the data profile and in interviews, each with how often it occurs, and a rule for whether and how it counts in each metric.
How the gap shows
  • No requirement mentions returns, cancellations, internal accounts, test records or adjustments.
  • The data profile shows unusual record types or status codes that no requirement explains.
  • Rules cover only the standard case.
Compare
Compare the status codes and record types in the data profile with the business rules: every code with no rule will be treated by default, and the default is rarely what the business wants.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
Fail fast
List every distinct status or type code in the main source and ask the business owner what each one means and whether it counts.
Related
RULE-01, DQ-05, CUR-04

Named business owners for data domains, definitions and mappings, and who approves changes to meaning (3)

OWN-01No named business owner for each data domainhighBusiness analyst
Question
For each data domain the project uses (customer, product, finance, employee), is there a named business person who decides what the data means, what good quality is, and who may use it? When the team has a question about a field, who answers, and how long does it take?
Why it matters
Product data had no business owner. Every question about categories went to the source system's technical team, who did not know, and the product hierarchy was argued about until the week before go-live.
Applies to
any · components: any · patterns: any
Owner
Business analyst · also Business owner and sponsor, Security and governance
Evidence
in: stakeholders, data-model
looks like: A table of data domains, each with a named business owner, a day-to-day steward if different, what they decide, and their agreement to the role.
How the gap shows
  • Data owners are listed as teams or systems rather than people.
  • The technical owner of a source system is named as the owner of the business meaning of its data.
  • Some domains in scope have no owner, or the owner is "to be confirmed".
Compare
Compare the data domains in the data model with the stakeholder map: a domain whose owner is not on the stakeholder map will not be consulted when its meaning is decided.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
Fail fast
List the domains and ask the sponsor to put one name against each in a ten-minute call, then ask each named person to confirm.
Related
STK-02, DATA-01, OWN-02
OWN-02Nobody approves changes to a definitionmediumBusiness analyst
Question
After go-live, who may change a business definition or rule, how is the change requested, who approves it, and how are users told that past figures will move? When a manager asks the team to "just exclude" something, who has to say yes first?
Why it matters
A regional director asked a developer to exclude one large customer from the margin report. The change went in quietly, the group margin moved, and finance found out when the board asked why.
Applies to
any · components: modelling, serving, governance · patterns: any
Owner
Business analyst · also Business owner and sponsor, Data consumers, Operations and support
Evidence
in: decisions, runbook
looks like: A definition change process: who can request, who approves (the definition's owner), how other users are consulted, how the change is versioned and dated, and how users are told.
How the gap shows
  • Definitions have owners but no process for changing them.
  • The support model handles technical changes but says nothing about changes to meaning.
  • Change requests to metrics go straight to developers.
Compare
Compare the definition owners in the data model with the support model's change process: if the support model lets the team change logic without the definition owner's approval, ownership exists only on paper.
Due by
planstage 4: Discovery sign-off
designedstage 6: Build
Fail fast
Ask the support lead: "If a director asks you tomorrow to change how revenue is counted, what do you do?"
Related
REQ-03, SERV-03, MET-02, REQ-10
OWN-03Mappings and hierarchies with no business maintainermediumBusiness analyst
Question
Who in the business maintains the mappings and hierarchies the reports depend on (product to category, store to region, account to reporting line, cost centre to department), how do they change them, and how quickly? When a new product or store appears, who notices it is unmapped?
Why it matters
New products launched every month. The mapping to categories lived in a spreadsheet the analyst who left had maintained, so new products fell into "unknown" and the category report drifted away from reality.
Applies to
any · components: transformation, modelling, governance · patterns: any
Owner
Business analyst · also Data consumers, Operations and support
Evidence
in: requirements, runbook
looks like: A list of business-maintained mappings and hierarchies, each with a named maintainer, where it is edited, how often it changes, and what happens to records that are not yet mapped.
How the gap shows
  • Mappings or hierarchies are described as "reference data" with no owner.
  • Mappings are kept in personal spreadsheets or e-mailed files.
  • Nothing says what happens to a new code that is not in the mapping.
Compare
Compare the reference data in the data model with the support model: a mapping the reports depend on that no one in the support model maintains will go stale.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
Fail fast
Count how many codes in last month's source data are missing from today's mapping, and ask who would fix them.
Related
OWN-01, RULE-02, CUR-04

Every consumer and channel named with what it needs, and self-service expectations (2)

USE-01Not every consumer and channel namedhighBusiness analyst
Question
Is every consumer of the output named, with the channel they use (dashboard, scheduled extract, file to a partner, application interface, machine learning model, AI assistant answering questions) and what each needs from it? Who will turn up at go-live asking where their feed is?
Why it matters
The project built dashboards for the sales team. Two weeks before go-live the pricing team said their model read a nightly extract from the old system, which nobody had listed, and the cut-over was postponed.
Applies to
any · components: serving · patterns: any
Owner
Business analyst · also Data consumers, Technical anchor
Evidence
in: requirements, stakeholders
looks like: A consumer list: one row per consumer or channel with who they are, how they receive the data, what data and grain they need, how fresh, and any needs such as languages, accessibility or mobile use.
How the gap shows
  • Requirements describe only dashboards or reports, with no extracts, interfaces, models or assistants.
  • Downstream systems that read today's data are not listed.
  • Consumers are named as "the business" or "users" without groups.
Compare
Compare the consumer list in the requirements with the outputs drawn in the architecture: an output with no named consumer is unowned work, and a consumer with no output will be missed at go-live.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
Fail fast
Ask the source and current-report owners for the list of everyone who receives or queries today's data, and check each one against the requirements.
Related
SERV-01, REQ-05, CUR-01, RLC-05
USE-02Self-service expectations not statedmediumBusiness analyst
Question
Does the business expect users to build their own reports and answer their own questions, and if so, which users, with what skills, on which data, with what training and support? If users say at go-live "we thought we could build our own", will that be in scope?
Why it matters
The sponsor expected analysts to explore the data freely. The team built fixed reports only, with no documented model for users to build on, and the analysts went back to exporting to spreadsheets.
Applies to
any · components: serving, modelling · patterns: any
Owner
Business analyst · also Business owner and sponsor, Data consumers
Evidence
in: requirements, discovery
looks like: A clear statement of who will only read reports and who will build their own, what data the builders may use, what training and documentation they will get, and who supports them.
How the gap shows
  • Words such as "self-service", "explore" or "ad hoc" appear with no named users or scope.
  • No training, documentation or support for business users is in the plan.
  • The out-of-scope list says nothing about self-service.
Compare
Compare the self-service promises in the discovery document with the plan and handover: self-service promised but no user documentation, training or support planned will not happen.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
Fail fast
Ask the sponsor to name three people who will build their own reports, and ask those three what they plan to build.
Related
USE-01, SCOPE-01

Privacy purpose, retention and audit needs stated by the business (the tech pack covers implementation) (3)

POL-01Use of personal data not confirmed as allowedhighBusiness analyst
Question
For each use of personal data in the requirements, has the business confirmed with its privacy or legal team that the use is allowed: the purpose, any consent it depends on, and who may see it? If the privacy team stops the project a month before go-live, will it be over a use nobody asked them about?
Why it matters
The marketing dashboard joined customer purchases to support-call notes. Customers had never been told their call notes would be used for marketing, and the privacy officer stopped the release in user testing.
Applies to
any · components: any · patterns: any
Owner
Business analyst · also Security and governance, Business owner and sponsor
Evidence
in: requirements, decisions
looks like: A list of the personal data each requirement needs, the business purpose, the basis that allows it (consent, contract, legal duty), who may see it, and the privacy team's recorded answer.
How the gap shows
  • Requirements need customer, employee or patient data with no mention of purpose or permission.
  • The privacy or legal team is not on the stakeholder map.
  • Personal data is treated only as a security or masking question.
Compare
Compare the personal data named in the requirements with the stakeholder map and decision log: personal data in use with no privacy sign-off recorded is a go-live blocker waiting to be found.
Due by
planstage 3: Discovery
evidencedstage 4: Discovery sign-off
Fail fast
Send the privacy team a one-page list of the personal data and its purposes this week and ask for a yes, a no or a condition.
Related
SEC-01, SEC-05, STK-01
POL-02Retention needs not stated by the businessmediumBusiness analyst
Question
For each kind of data, how long does the business need to keep it, how long does law or policy require it to be kept, and when must it be deleted? When a user asks for seven-year trends and policy says delete after three, who decides?
Why it matters
The business asked for ten years of customer history. Company policy required customer data to be deleted three years after the relationship ended, and the history load had to be cut back after it was done.
Applies to
any · components: governance, modelling · patterns: any
Owner
Business analyst · also Security and governance, Business owner and sponsor
Evidence
in: requirements, decisions
looks like: A retention table per data kind: the business need, the legal or policy minimum and maximum, the agreed period, and who signed it, including what happens to aggregated figures after detail is deleted.
How the gap shows
  • Retention is not mentioned in the requirements, or is "to be agreed".
  • The history requested is longer than any retention period stated.
  • No one from records management, legal or compliance is named.
Compare
Compare the history depth requested in the requirements with the retention periods: history the business wants beyond the period policy allows must be resolved before it is loaded.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
Fail fast
Ask the records or compliance owner for the retention rule for the two most sensitive data kinds, and compare it with the history asked for.
Related
SEC-03, DATA-03
POL-03What must be proven to an auditor not statedmediumBusiness analyst
Question
Will auditors, regulators or internal control need to see how a reported figure was produced, reproduce a past figure exactly as it was reported, or see who changed what and when? Has the business written down those needs, so the design can meet them?
Why it matters
A regulator asked the bank to reproduce a capital figure reported six months earlier. The data had been corrected since, nobody had kept what was reported, and the bank could not show where the number came from.
Applies to
any · components: modelling, serving, governance · patterns: any
Owner
Business analyst · also Security and governance, Business owner and sponsor, Technical anchor
Evidence
in: requirements
looks like: A list of audit and control needs in business terms: which figures must be reproducible as reported and for how long, which changes need a record of who and when, and which reports need sign-off before release.
How the gap shows
  • Figures go to regulators, auditors or external parties but no audit need is stated.
  • The requirements never mention reproducing past figures, sign-off of published numbers, or change history.
  • Audit is mentioned only as access logging.
Compare
Compare the audit needs in the requirements with the restatement policy: if published figures may be restated but must also be reproducible as reported, the design has to keep both versions.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 7: Test
Fail fast
Ask the finance or compliance owner: "Which number from this system could an auditor ask us to reproduce in a year's time?"
Related
SEC-04, DATA-05

User acceptance designed early with real users, recognisable numbers, data, environment and sign-off (3)

UAT-01User acceptance scenarios designed late, without the usershighBusiness analyst
Question
Were the user acceptance scenarios written during discovery, with the users who will test, and built around the questions they really ask and the figures they already know (last month's total for their region, a customer they can name)? When testing starts, will the testers know what to check, or will they click around and say it looks fine?
Why it matters
The scenarios were written by the test team in the last week of the build, from the requirements alone. They checked that screens loaded, not that numbers were right, and the first month-end after go-live showed revenue a few per cent out in every region.
Applies to
any · components: modelling, serving · patterns: any
Owner
Business analyst · also Test lead, Data consumers
Evidence
in: test-strategy, requirements
looks like: A first set of acceptance scenarios agreed with named users by discovery sign-off, each one a real question with the expected figure from a trusted report for a known period, covering every user group and the tricky cases (returns, late data, reorganisations).
How the gap shows
  • The test strategy says "user acceptance testing" with no scenarios, or "scenarios to be defined".
  • Scenarios describe screens and clicks, not figures and expected values.
  • Scenarios were written by the project team with no named business contributor.
Compare
Compare the acceptance scenarios in the test strategy with the decisions and user groups in the requirements: a decision or user group with no scenario will be accepted without being tested.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
Fail fast
Ask two real users to write down three questions they will ask the new system and the answers they expect for last month; those are the first scenarios.
Related
TEST-04, REQ-02, CUR-02, RES-04
UAT-02No data or environment for user testingmediumBusiness analyst
Question
What data will business testers use (real enough for them to recognise their numbers, covering a period they know), in which environment, with what access, and is personal data in it allowed? If testers cannot log in on the first day, or see numbers they cannot judge, how many testing days are lost?
Why it matters
User testing used invented data so that no personal data left production. The testers could not tell whether any figure was right, signed off out of politeness, and the first real month showed the errors.
Applies to
any · components: modelling, serving · patterns: any
Owner
Business analyst · also Test lead, Platform team, Security and governance
Evidence
in: test-strategy, platform
looks like: A statement of which period and data the testers will use, how it is made safe if it holds personal data, which environment they use, and that their access was requested and tested before testing starts.
How the gap shows
  • The test strategy says user testing will use "test data" or "sample data" with no period named.
  • Business testers' access to the test environment is not in the plan.
  • Nothing says whether personal data may be used in user testing.
Compare
Compare the user testing approach in the test strategy with the platform environments: user testing that needs production-like data in an environment that is not allowed to hold it cannot happen as planned.
Due by
planstage 4: Discovery sign-off
designedstage 6: Build
evidencedstage 7: Test
Fail fast
Ask one business tester to log in to the test environment this week and find last month's figure for their own area.
Related
TEST-02, ENV-01, POL-01
UAT-03Acceptance signed by the wrong people, or with no exit rulehighBusiness analyst
Question
Will acceptance be signed by the people who will use the output, and is it agreed in advance what counts as accepted: which differences from the trusted figure are tolerable, and which open defects may go live? If testing ends with twenty open defects, is the go or no-go already decided?
Why it matters
The project manager and the sponsor signed acceptance. The analysts who used the reports daily had found differences they could not explain, were not asked, and quietly went back to the old spreadsheets.
Applies to
any · components: modelling, serving · patterns: any
Owner
Business analyst · also Business owner and sponsor, Data consumers, Test lead
Evidence
in: test-strategy, decisions
looks like: Named acceptors from each user group, exit rules agreed before testing (tolerances for differences, severity of defects that may remain), and a dated sign-off record.
How the gap shows
  • Acceptance is signed only by the sponsor or the project team.
  • No exit criteria or tolerances are stated for user testing.
  • The sign-off template has no place to record open defects or conditions.
Compare
Compare the acceptors named in the test strategy with the user groups in the requirements: a user group with no acceptor has not accepted anything.
Due by
planstage 4: Discovery sign-off
designedstage 6: Build
evidencedstage 7: Test
Fail fast
Write the exit rule in three lines now and get the business owner and one real user to agree to it.
Related
REC-03, STK-05, REQ-05, RLC-01

Project management (64)

How the work runs: plan and dependencies, risks, decisions, budget, stakeholders, communication, governance, go-live and benefits.

Estimates and their assumptions (1)

EST-06Never corrected against actual progresshighTechnical anchor
Question
Is actual effort compared with the estimate at least every two weeks, and has the estimate been changed when the real rate differed? If we finish late, will we have known it two months before or two days before?
Why it matters
The thin slice took three times longer than planned per table. Nobody re-estimated, and the gap only showed when the last month of the plan arrived with half the tables still to build.
Applies to
any · components: any · patterns: any
Owner
Technical anchor · also Project manager
Evidence
in: estimate, status
looks like: A re-estimate after the thin slice and then every two weeks, showing items done, the real rate, and the forecast finish, with changes explained.
How the gap shows
  • The estimate document has one version and its date is before build started.
  • Status reports show percent complete but never a forecast end date or rate.
  • No mention of a thin slice or of the rate it achieved.
Compare
Compare status reports with the estimate: if progress reports show fewer items finished than the estimate's rate predicts, but the end date never moves, the estimate is not being corrected.
Due by
planstage 4: Discovery sign-off
evidencedstage 6: Build
Fail fast
Divide items finished so far by weeks spent, and project the finish date from that rate today.
Related
EST-01, PM-04

In scope, out of scope, change control (2)

SCOPE-03Scope creep with no change processhighProject manager
Question
Is there an agreed way to handle a new request: who can raise it, how it is sized, who decides, and how the plan and estimate change? If the project ends late, how much of the lateness will be work that was added without anyone deciding to add it?
Why it matters
Each request was small and agreed in a meeting. After three months, a fifth of the work had never been estimated, the date had not moved, and the team was blamed for the slip.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Business owner and sponsor, Technical anchor
Evidence
in: plan, decisions
looks like: A written change process and a change log where each change shows the request, its size, its effect on date and cost, the decision and who made it.
How the gap shows
  • No "change control", "change request" or "change process" in the plan.
  • Status reports mention new items that do not appear in the original scope or in any change log.
  • Meeting notes include "can we also add" with no follow-up decision.
Compare
Compare the latest status report or backlog with the signed scope: items in the backlog that are not in the scope and have no change record are unmanaged creep.
Due by
planstage 4: Discovery sign-off
evidencedstage 6: Build
Fail fast
Count backlog items added since sign-off and check how many have a recorded decision.
Related
SCOPE-04, PM-03
SCOPE-05MVP 1 is too bigmediumProject manager
Question
Is the first MVP the smallest slice that gives real value and proves the hardest parts work end to end? If the first release slips by months, will it be because it tried to do everything at once?
Why it matters
MVP 1 contained eight sources and every report. Nothing reached production for nine months, so no real feedback arrived until it was too late to change direction.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Technical anchor, Business owner and sponsor
Evidence
in: plan
looks like: An MVP 1 that can reach production within a few months, chosen for value and for testing the riskiest beliefs, with later MVPs listed in order of dependency and value.
How the gap shows
  • MVP 1 contains most or all of the in-scope items.
  • The first production release date is more than about three months after build starts.
  • The reason for the MVP order is not written down.
Compare
Compare the plan's MVP 1 with the estimate: if MVP 1 holds most of the estimated effort, it is the whole project with a different label.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
Fail fast
Ask what could be removed from MVP 1 while it still gives one group of users a real answer.
Related
SCOPE-04, PM-05

Plan, dependencies, risks, decisions, status (18)

PM-01Dates set before the dependencies were knownhighProject manager
Question
Were the milestones set after the dependencies (access, environments, source changes, other teams, approvals) were listed and confirmed with the people who own them? If the first milestone slips, will it be because of something outside the team that nobody had asked about?
Why it matters
The go-live date was announced first. The source team's change freeze, the security review and the network change each added weeks, and none had been asked about when the date was set.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Technical anchor, Source system owners, Platform team
Evidence
in: plan, raid
looks like: A dependency list with each owner, the date they committed to, and the plan tasks that wait on it.
How the gap shows
  • Milestone dates appear in the brief or charter before any dependency list exists.
  • The plan has no dependencies, or lists them without owners or dates.
  • External teams named in the architecture do not appear in the plan.
Compare
Compare the architecture with the plan: every system or team the architecture relies on should appear as a dependency in the plan; one that does not is an unplanned wait.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
Fail fast
Phone the owner of the biggest external dependency and ask for the date they can deliver it, in writing.
Related
PM-07, PM-10
PM-02Risks listed, not managedhighProject manager
Question
Does each top risk have an owner, a trigger that tells us it is happening, a mitigation that is being done, and a date it was last reviewed? Looking back after a failure, will the cause be on the risk log with nothing done about it?
Why it matters
"Source data quality may be poor" sat on the risk log for six months with no owner. It was exactly the reason the project missed its date.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Technical anchor
Evidence
in: raid
looks like: A short top-risks list where each risk has an owner, a trigger, a mitigation with a due date, and a recent review date.
How the gap shows
  • Risks with no owner, or with a team rather than a person as owner.
  • No triggers or mitigation actions, only descriptions.
  • The risk log has not changed in several weeks while status reports describe new problems.
Compare
Compare the RAID log with status reports: problems in the status reports that were never on the risk log, or risks that became issues without the log changing, mean the log is not used.
Due by
planstage 3: Discovery
evidencedstage 6: Build
Fail fast
Take the top three risks and ask each owner what they did about it this week.
Script signal
risks-without-owner (found without AI)
Related
PM-04, EST-03
PM-03Decisions not written downhighProject manager
Question
Is there a decision log recording the options, the choice, the reason, who decided and when? When a decision is reopened by someone who was not in the room, what will we show them?
Why it matters
The team agreed to keep only the latest customer address. Three months later a new manager asked why history was missing, and with no record the decision was reversed and the work redone.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Technical anchor, Architect
Evidence
in: decisions
looks like: A decision log or short decision records: context, options, decision, reason, owner, who signed, date.
How the gap shows
  • No decision log or decision records exist among the project documents.
  • Meeting notes contain "agreed" or "decided" but no decision log entry matches.
  • Architecture choices are stated with no record of the options considered.
Compare
Compare meeting notes or status reports with the decision log: a decision mentioned in notes but not in the log is one that can be lost.
Due by
planstage 3: Discovery
evidencedstage 6: Build
Fail fast
List the last five important choices from memory and check how many are in the log.
Script signal
no-decision-log (found without AI)
Related
PM-08, COM-03, HAND-02
PM-04Green until it is redhighProject manager
Question
Do status reports show amber or red when something is slipping, with the reason and the next step? If the project fails suddenly, will the reports from the month before have shown it as green?
Why it matters
Status stayed green for four months while the team quietly fell behind. It went red two weeks before go-live, when there was nothing left to do but move the date.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Technical anchor, Business owner and sponsor
Evidence
in: status
looks like: Status reports where the colour is based on stated rules (for example forecast versus baseline), and where amber appears early with a recovery action.
How the gap shows
  • Every status report is green.
  • The status colour does not change while the forecast date or open issues change.
  • No forecast finish date, only activities done.
Compare
Compare status reports with the RAID log and the estimate: a green status alongside rising issues, a rate below the estimate, or slipping dependencies means the colour is not telling the truth.
Due by
planstage 4: Discovery sign-off
evidencedstage 6: Build
Fail fast
Write down the rule for red, amber and green today and apply it to the last report.
Script signal
status-always-green (found without AI)
Related
PM-02, EST-06, COM-01
PM-05Testing and deployment left to the endhighProject manager
Question
Does the plan have testing and deployment to a production-like environment starting in the first weeks of build, rather than after it? If go-live slips, will it be because the first real deployment happened a month before the date?
Why it matters
Everything was built, then tested, then deployed. The first deployment found network, permission and performance problems that took longer to fix than the build had taken.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Technical anchor, Test lead, Release and DevOps
Evidence
in: plan, test-strategy, deployment
looks like: A test strategy and a release approach agreed in discovery, a first deployment of a thin slice in the first weeks, and "done" defined as tested and deployed.
How the gap shows
  • No test strategy or deployment document exists, or it says "to be defined".
  • The plan has testing and deployment as single phases after build.
  • The definition of done mentions coding but not testing or deployment.
Compare
Compare the plan with the test strategy: if the strategy describes continuous testing but the plan has one test phase at the end, the plan wins and testing will be squeezed.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 6: Build
Fail fast
Deploy the thin slice to the test environment this week, through the real release route.
Script signal
missing-doc-type (found without AI)
Related
EST-04, SCOPE-05
PM-06Key-person riskmediumProject manager
Question
Is there any part of the project that only one person understands or can do, and are the people the plan needs named and available when it needs them? If one person leaves next month, what stops?
Why it matters
One developer understood the source system. When they moved to another project mid-build, the team lost six weeks rediscovering what they knew.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Technical anchor
Evidence
in: plan, raid
looks like: Named people for each role with their availability, a second person for every critical area, and key-person risks on the RAID log.
How the gap shows
  • The same person's name appears as owner for most technical items.
  • Roles are listed as "to be hired" or "to be confirmed" for work in the next stage.
  • No mention of availability, leave or shared allocation.
Compare
Compare the plan with the stakeholder map or team list: a plan that needs a role the team list does not have, or a person allocated part-time who carries full-time tasks, will slip.
Due by
planstage 3: Discovery
evidencedstage 6: Build
Fail fast
Ask each team member which area they could not cover if a colleague were away for a month.
Related
HAND-03
PM-07The plan ignores the estimate's assumptionsmediumProject manager
Question
Does the plan use the same assumptions as the estimate (start of access, team size, environment availability, source readiness)? When the plan slips, will it be because it assumed something the estimate never did?
Why it matters
The estimate assumed access in week three; the plan assumed week one. Nobody compared them, and the plan was two weeks behind before work started.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Technical anchor
Evidence
in: plan, estimate
looks like: A plan that lists the estimate's key assumptions and shows where each one sits on the timeline.
How the gap shows
  • The plan does not mention the estimate or its assumptions.
  • The plan's team size or start dates differ from the estimate's.
Compare
Compare the estimate's assumptions with the plan's dates and staffing: any mismatch is a slip already built in.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
Fail fast
Put the estimate's assumptions next to the first four weeks of the plan and look for disagreements.
Related
EST-03, PM-01
PM-08Open decisions with no owner, deadline or defaultmediumProject manager
Question
Does every open decision have an owner, a date by which it is needed, and a default that applies if nobody decides? If the team is blocked next month, will it be waiting on a decision nobody owns?
Why it matters
The choice between daily and hourly loads stayed open for ten weeks because nobody owned it. The team built for both, and later threw half away.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Business owner and sponsor, Technical anchor
Evidence
in: decisions, raid
looks like: An open-decisions list with the question, options, owner, the date needed, the default and the effect of deciding late.
How the gap shows
  • Open questions listed with no owner or due date.
  • Items the build depends on are marked "TBD" or "to be confirmed".
  • The same open question appears in several consecutive status reports.
Compare
Compare the open decisions with the plan: a task scheduled to start before the decision it depends on is due will either stall or be redone.
Due by
planstage 3: Discovery
evidencedstage 5: MVP starts
Fail fast
Give every open decision an owner and a default today, and tell the owners the default applies on the due date.
Related
PM-03, STK-02
PM-09Discovery with no timeboxmediumProject manager
Question
Does discovery have a fixed length, a cost, and a list of exit criteria? If discovery runs for months or is squeezed into days, which risks will we carry into the build without having tested them?
Why it matters
Discovery was squeezed into three days to "start building faster". The riskiest beliefs were never tested, and they surfaced one by one during the build.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Technical anchor, Business owner and sponsor
Evidence
in: plan, discovery
looks like: A discovery plan of a few weeks with weekly goals, its own estimate, named reviewers and written exit criteria.
How the gap shows
  • No discovery stage in the plan, or no end date for it.
  • Discovery has no estimate of its own.
  • No exit criteria for discovery sign-off.
Compare
Compare the discovery exit criteria with the discovery document: criteria that are not met but sign-off still happened show the timebox beat the evidence.
Due by
planstage 2: Rough estimate
evidencedstage 4: Discovery sign-off
Fail fast
Write the discovery exit criteria now and agree the end date with the sponsor.
Related
EST-01
PM-10Access and environments treated as adminhighProject manager
Question
Were access to sources, environments and tools requested in the first week and tracked as dependencies with owners and dates? If the build starts late, will it be because the team waited weeks for access?
Why it matters
Access requests were raised when build started. Approvals took six weeks, so the team spent the first sprints on paper designs that the real data later contradicted.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Technical anchor, Platform team, Security and governance
Evidence
in: raid, plan
looks like: A list of every access and environment request with the date raised, the approver, the expected date and its current state.
How the gap shows
  • No access or environment requests are listed in the plan or RAID log.
  • Access is described as "assumed" or "will be arranged".
  • Discovery has no real data extract because access had not arrived.
Compare
Compare the plan's start of build with the access request dates: a build that starts before access is expected is planned to wait.
Due by
planstage 2: Rough estimate
evidencedstage 3: Discovery
Fail fast
Raise every access request today and record the date each one arrives.
Related
PM-01, EST-04
PM-11History load not on the plan as real taskshighProject manager
Question
Is the history load on the plan as its own tasks, with the elapsed time it needs (source extract windows, several trial runs, checking the old numbers), not folded into "build"? If go-live slips, will it be because the history load turned out to be a project of its own?
Why it matters
The plan gave the history load two days before go-live. The source could only be read at night, the first run found years with a different code list, and the full load took five weekends.
Applies to
any · components: ingestion · patterns: history load
Owner
Project manager · also Technical anchor, Source system owners
Evidence
in: plan, deployment
looks like: Plan tasks for the history extract, at least one full trial run, checking the old periods with the business and the final load, each with elapsed time, source windows and an owner.
How the gap shows
  • The history load appears only in the architecture, not in the plan.
  • The plan has one task, or a few days, for loading many years of history.
  • No trial run of the history load is planned before the cut-over.
Compare
Compare the history load approach in the architecture or deployment plan with the plan's tasks: an approach that needs night windows and several runs, given a couple of days in the plan, will slip.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 7: Test
Fail fast
Time a full extract of one year of history from the source now and multiply by the years in scope.
Related
HIST-01, HIST-02, EST-07
PM-12Data fixes with no owner and no timehighProject manager
Question
When profiling or testing finds bad data at the source (missing keys, wrong codes, duplicate customers), is it agreed who fixes it, where (at the source, by the business, or in the pipeline), and is that time on someone's plan? Whose plan holds the weeks of cleansing that the profile is already hinting at?
Why it matters
Profiling showed that a fifth of customers had no region. Everyone assumed someone else would fix it; the business had no time booked, and user acceptance failed on regional totals six weeks before go-live.
Applies to
any · components: ingestion, transformation · patterns: any
Owner
Project manager · also Business owner and sponsor, Source system owners, Technical anchor
Evidence
in: plan, raid
looks like: A data issues list from profiling, each with the decision on where it is fixed, an owner, the effort, and matching tasks in the project plan or the source team's plan.
How the gap shows
  • The data profile lists quality problems that appear nowhere in the plan or RAID log.
  • Fixes are assigned to "the business" or "the source" with no name or date.
  • No time for data fixing appears in the plan.
Compare
Compare the data profile or data quality findings with the plan: each problem that needs fixing, with no task and owner, is work that will appear in user acceptance instead.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 7: Test
Fail fast
Take the top five data problems from the profile and get a named owner and a date for each this week.
Related
DQ-05, DQ-01, EST-07
PM-13Reconciliation and user acceptance planned as one passhighProject manager
Question
Does the plan allow for more than one cycle of reconciliation and user acceptance (test, find differences, explain or fix, test again), with business time booked for each cycle? If the first round finds the numbers do not match, is there any time left to fix them?
Why it matters
One week of user acceptance was planned. The first day showed revenue three percent off the ledger, the explanation took two weeks, and there was no second test window before the go-live date.
Applies to
any · components: modelling, serving · patterns: any
Owner
Project manager · also Test lead, Business owner and sponsor, Technical anchor
Evidence
in: plan, test-strategy
looks like: At least two reconciliation and acceptance cycles on the plan, with time between them to explain and fix differences, and business testers booked for each.
How the gap shows
  • The plan has a single user acceptance window.
  • Reconciliation is not a task in the plan, or sits inside "testing".
  • No time between the end of user acceptance and go-live.
Compare
Compare the test strategy's reconciliation approach with the plan: a strategy that compares every layer with trusted figures, given one short window, will run out of time on the first difference.
Due by
planstage 4: Discovery sign-off
designedstage 6: Build
evidencedstage 7: Test
Fail fast
Reconcile one month of one key number against the business's trusted figure now and time how long it takes.
Related
REC-02, REC-03, TEST-04, RES-04
PM-14Cut-over weekend not plannedhighProject manager
Question
Is there an hour-by-hour cut-over plan covering the business freeze, the final loads, the checks before users are let in, the point after which there is no going back, the rollback steps, and who is on call? At three in the morning on cut-over night, if the final load fails, who decides what happens next?
Why it matters
Cut-over was "the weekend after UAT". The final history load overran by ten hours, nobody had agreed a point to stop, and on Monday half the users saw new numbers while the rest still had the old reports.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Technical anchor, Release and DevOps, Operations and support, Business owner and sponsor
Evidence
in: deployment, plan
looks like: A cut-over runbook with timed steps and owners, the business freeze window, checks before opening to users, a go-back deadline and rollback steps, a call list, and at least one rehearsal.
How the gap shows
  • No cut-over plan, or one line saying "deploy to production".
  • No business freeze, rollback window or decision point is mentioned.
  • No rehearsal of the cut-over is planned.
Compare
Compare the cut-over plan's timings with the history load and final load run times measured in test: a window shorter than the measured loads will overrun.
Due by
planstage 4: Discovery sign-off
designedstage 7: Test
evidencedstage 8: Deploy
Fail fast
Write the cut-over as timed steps on one page and add up the hours against the window you have.
Related
REL-07, HIST-03, RLC-01
PM-15No parallel run periodmediumProject manager
Question
Where the new solution replaces an existing report or feed, is there a period when both run side by side, with a named person comparing them, a length that covers at least one month-end, and criteria for ending it? When the new number and the old number differ in week one, who decides which is right?
Why it matters
The old report was switched off on go-live day. In the first month-end the new figures differed from what finance expected, there was nothing left to compare against, and trust in the new report never recovered.
Applies to
any · components: serving, modelling · patterns: any
Owner
Project manager · also Business owner and sponsor, Data consumers, Technical anchor
Evidence
in: plan, deployment
looks like: A parallel run on the plan with dates covering a month-end, a person who compares old and new, a tolerance, a decision owner for differences and written exit criteria.
How the gap shows
  • The old system or report is switched off on the go-live date.
  • Parallel running is mentioned with no length, owner or exit criteria.
  • The parallel run ends before the first month-end or quarter-end.
Compare
Compare the plan's parallel run dates with the business calendar: a parallel run that ends before the first month-end tests the easy days only.
Due by
planstage 4: Discovery sign-off
designedstage 7: Test
evidencedstage 9: Production sign-off
Fail fast
Ask the business owner which month-end they would want to see matched before switching off the old report.
Related
REC-04, RLC-03, BUD-06
PM-16Team rolled off before the first month-endmediumProject manager
Question
Does the plan and budget keep the people who built the solution available through early-life support, including the first month-end, quarter-end or other peak that the data will meet? When the first month-end load breaks, are the developers who wrote it still on the project?
Why it matters
The developers moved to a new project the Monday after go-live. The first month-end load hit late corrections the tests never had, and support spent four days rebuilding knowledge the team could have given in an hour.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Technical anchor, Operations and support, Business owner and sponsor
Evidence
in: plan, runbook
looks like: Named people booked for early-life support, an end date after the first month-end or other peak, and the exit criteria for handing over fully to support.
How the gap shows
  • The plan and budget end on the go-live date.
  • Early-life support is mentioned but nobody is named or funded for it.
  • Team members are already booked on other projects from go-live.
Compare
Compare the plan's end date and staffing with the support model's early-life support period: a support period that outlasts the team's booking has nobody to do it.
Due by
planstage 4: Discovery sign-off
designedstage 7: Test
evidencedstage 9: Production sign-off
Fail fast
Check each developer's next booking date against the first month-end after go-live.
Related
HAND-04, HAND-06, OPS-05
PM-17Buffers not where data surprises clustermediumProject manager
Question
Does the plan hold explicit buffer time after the points where data projects usually find surprises: the first real extract, the first full load, the history load and the first reconciliation? Where in this plan would a two-week surprise from the data fit without moving go-live?
Why it matters
Every task was planned back to back with no slack. The first full load showed that keys were not unique, and with nowhere to absorb the fix, every later date moved by the same three weeks.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Technical anchor
Evidence
in: plan, raid
looks like: Named buffer tasks after the first real loads, the history load and reconciliation, sized to the risks in the RAID log, with a rule for who may use them.
How the gap shows
  • No buffer appears in the plan, or one buffer sits at the very end.
  • Tasks after the first full load and reconciliation are planned with no gap before go-live.
  • The RAID log has data risks with no time set aside for them.
Compare
Compare the data risks in the RAID log with the plan: a high data risk that hits in the middle of the plan, with no buffer after it, will move every later date.
Due by
planstage 4: Discovery sign-off
evidencedstage 6: Build
Fail fast
Ask where a two-week slip after the first full load would land, and whether the go-live date survives it.
Related
PM-02, BUD-04, EST-02
PM-18The business calendar ignoredmediumProject manager
Question
Has the plan been checked against the business and data calendar: month-end and year-end close, audit, peak trading, source system change freezes and the platform's own release windows? Which key date in our plan falls in a week when the business or a source team cannot help us?
Why it matters
Go-live was set for the first week of January. Finance was closing the year, the source system was frozen for its own year-end, and the release had to wait until February.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Business owner and sponsor, Source system owners, Platform team
Evidence
in: plan, raid
looks like: Business close periods, peak periods and change freezes marked on the plan, with no testing, cut-over or go-live inside them unless agreed.
How the gap shows
  • The plan shows no business calendar, freezes or close periods.
  • Go-live, cut-over or user acceptance falls in a month-end, year-end or known freeze.
Compare
Compare the plan's key dates with source teams' and the platform's change freezes: a release planned inside a freeze will not be allowed to happen.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
Fail fast
Ask finance, the source owners and the platform team for their freeze and close dates for the next year.
Related
PM-01, RES-04, RLC-04

Who is involved, who decides, who can block (7)

STK-01Only the sponsor is managedhighProject manager
Question
Does the stakeholder map include the people outside the weekly meeting who can block go-live: source system owners, operations, security, the platform team and downstream users? Who could stop this project in its last month that we have not spoken to in its first?
Why it matters
The sponsor was kept happy every week. Security saw the design for the first time two weeks before go-live, required personal data to be masked, and the release slipped by two months.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Technical anchor
Evidence
in: stakeholders
looks like: A stakeholder map naming each person or group, what they care about, what they can approve or block, when they were last spoken to, and how they are kept informed.
How the gap shows
  • No stakeholder map or communication plan exists.
  • The map lists only the sponsor, the business owner and the project team.
  • Operations, security, source owners or downstream consumers are not named anywhere.
Compare
Compare the architecture with the stakeholder map: every source, platform and consumer in the architecture has an owner who should appear on the map; one who does not is an unmanaged blocker.
Due by
planstage 3: Discovery
evidencedstage 4: Discovery sign-off
Fail fast
List everyone who must say yes before go-live and book a thirty-minute meeting with each one you have not met.
Script signal
missing-doc-type (found without AI)
Related
REQ-05, COM-04
STK-02Nobody clearly owns the decisionshighProject manager
Question
For each kind of decision (scope, definitions, priorities, go-live), is one named person the decision owner? When two senior people disagree, who decides, and how long will we wait?
Why it matters
Two directors each believed they owned the project. Every decision was taken twice, the team built to whichever spoke last, and both rejected the result.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Business owner and sponsor
Evidence
in: stakeholders, decisions
looks like: A responsibility table (who decides, who is consulted, who is informed) for the main decision types, with one decider per type and a route for escalation.
How the gap shows
  • No responsibility table, or several people marked as the decider for the same thing.
  • Decision log entries with no named decider.
  • Phrases such as "the business will decide" without a name.
Compare
Compare the stakeholder map with the decision log: decisions signed by someone the map does not name as the decider are likely to be reopened.
Due by
planstage 3: Discovery
evidencedstage 4: Discovery sign-off
Fail fast
Ask the sponsor to name, in writing, the one person who decides on scope and definitions.
Related
PM-08, REQ-03
STK-03No real numbers shown to the businesshighTechnical anchor
Question
Have the business users seen real numbers from real data, regularly, before user acceptance testing? If they say "these numbers are wrong" at the end, will it be the first time they have looked?
Why it matters
Demos used made-up data that looked fine. The first time the business saw real numbers was user acceptance, and they did not recognise them.
Applies to
any · components: any · patterns: any
Owner
Technical anchor · also Business analyst, Data consumers
Evidence
in: status, plan
looks like: Regular demos, roughly every two weeks, with real data, and a record of what the business said and what changed as a result.
How the gap shows
  • No demos are planned or recorded before the test stage.
  • Demos are described as using sample, dummy or mock data.
  • The data profile was not shown to the business owner during discovery.
Compare
Compare the plan with the meeting notes or status reports: planned demos that do not appear in the notes did not happen.
Due by
planstage 4: Discovery sign-off
evidencedstage 6: Build
Fail fast
Show the business owner one real number from the thin slice and ask whether they recognise it.
Related
REQ-05, REQ-04
STK-04Conflicts left unsaidmediumProject manager
Question
Where stakeholders want different things (speed against cost, one department's definition against another's), is the conflict written down with options and taken to the decision owner? Which disagreement are we avoiding that will surface at sign-off?
Why it matters
Finance wanted monthly figures to match the ledger; sales wanted daily figures fast. Nobody named the conflict, the design pleased neither, and both refused to sign.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Technical anchor, Business analyst
Evidence
in: raid, decisions
looks like: Conflicts recorded as issues or open decisions, with each side's need, options, a recommendation and a deadline.
How the gap shows
  • Requirements from different groups that contradict each other with no note of the conflict.
  • Meeting notes saying a topic was "taken offline" repeatedly.
  • No issues or open decisions involve more than one stakeholder group.
Compare
Compare requirements from two different stakeholder groups: requirements that cannot both be met, with no decision recorded, are a hidden conflict.
Due by
planstage 3: Discovery
evidencedstage 4: Discovery sign-off
Fail fast
Put the two conflicting needs on one page with options and send it to the decision owner this week.
Related
STK-02, COM-01
STK-05Sign-off without readingmediumProject manager
Question
Did the people signing get the document in time to read it, and are their findings answered in writing? Is anyone signing something they have not read or do not agree with?
Why it matters
Reviewers saw the discovery document for the first time in the sign-off meeting and approved it to keep things moving. Their real objections arrived in the build, where they cost ten times more.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Technical anchor, Business owner and sponsor
Evidence
in: decisions, status
looks like: The document sent to reviewers several days before sign-off, a list of findings with written answers, and a sign-off record naming who approved what.
How the gap shows
  • No review comments or findings are recorded before sign-off.
  • The document was distributed on the same day as the sign-off meeting.
  • Sign-off is recorded for a group ("the steering committee") with no individual names.
Compare
Compare the sign-off record with later status reports or change requests: early objections to items that were signed suggest the signers had not read them.
Due by
planstage 4: Discovery sign-off
evidencedstage 4: Discovery sign-off
Fail fast
Ask each signer, before the meeting, for their single biggest concern in writing.
Related
SCOPE-06
STK-06Success not defined for each stakeholderlowProject manager
Question
Has each key stakeholder been asked what success looks like for them and what has gone wrong for them before? If a stakeholder calls the project a failure, will we know in advance what they were measuring?
Why it matters
The sponsor wanted lower cost, operations wanted fewer night calls, and users wanted faster reports. The project delivered only faster reports and was seen as a failure by two out of three.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Technical anchor, Business analyst
Evidence
in: stakeholders, discovery
looks like: For each key stakeholder, one line on what success means to them and a note of past problems they raised.
How the gap shows
  • The stakeholder map lists names and roles but not interests or success measures.
  • No record of initial interviews with stakeholders outside the business owner.
Compare
Compare each stakeholder's success measure with the scope: a measure that nothing in scope can move is a disappointment already planned.
Due by
planstage 3: Discovery
Fail fast
Ask each stakeholder one question this week: "What would make you say this project worked?"
Related
REQ-07, STK-01
STK-07Sponsor or owner changes and nobody re-confirmsmediumProject manager
Question
When the sponsor, business owner or a key data owner changes during the project, is the new person taken through the scope, definitions, decisions, budget and dates, and asked to confirm them in writing? If a reorganisation replaces our sponsor next month, which agreed decision is the new one most likely to reopen?
Why it matters
The business owner moved roles after discovery sign-off. Her successor had never seen the definitions, did not accept the agreed treatment of returns, and two months of modelling were redone.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Business owner and sponsor, Business analyst
Evidence
in: stakeholders, decisions
looks like: A record of each change in key stakeholders, the briefing given to the new person, and their written confirmation (or the decisions they reopened, handled as change requests).
How the gap shows
  • Names in recent status reports or meeting notes differ from those on the stakeholder map or sign-off record.
  • A new sponsor or owner appears with no record of a handover briefing.
  • Signed decisions are being questioned again in meeting notes after a change of people.
Compare
Compare the signatories on the discovery sign-off with the people now named as sponsor and owners: a decider who did not sign has not yet agreed.
Due by
planstage 4: Discovery sign-off
evidencedstage 6: Build
Fail fast
Check whether everyone who signed discovery still holds the same role, and brief anyone new this week.
Related
STK-02, STK-05, PM-03

How news, decisions and problems travel (7)

COM-01Bad news held backhighTechnical anchor
Question
When the team learns something bad (a slip, a data problem, a design that will not work), how quickly and to whom is it reported, with a next step? Is there anything the team knows today that the sponsor does not?
Why it matters
The team knew for six weeks that the source could not deliver hourly data. They hoped to find a workaround and said nothing, so the sponsor learned it from the source owner instead.
Applies to
any · components: any · patterns: any
Owner
Technical anchor · also Project manager
Evidence
in: status, raid
looks like: Status reports and the issue log that raise problems within days of their discovery, each with an impact and a proposed next step.
How the gap shows
  • Issues appear in the RAID log weeks before they appear in any status report.
  • Status reports contain only achievements.
  • Words such as "minor", "under control" or "being looked at" used for issues that later became major.
Compare
Compare the RAID log dates with status report dates: an issue logged long before it was reported is news that was held back.
Due by
planstage 4: Discovery sign-off
evidencedstage 6: Build
Fail fast
Ask the team: "What do we know that we have not told the sponsor?" and report it this week.
Related
PM-04, STK-04
COM-02Technical words used with business peoplemediumTechnical anchor
Question
Are documents and reports written for the business in plain words, with technical terms explained or left out? When the business owner signs something, do they understand what it means for them?
Why it matters
The design said history would be "type 1 for most dimensions". The business owner signed it, not realising it meant old addresses would be overwritten and last year's regional figures would change.
Applies to
any · components: any · patterns: any
Owner
Technical anchor · also Business analyst, Project manager
Evidence
in: status, discovery, decisions
looks like: Business-facing documents that state consequences in business terms ("last year's figures by region will change when a customer moves"), with a short glossary for unavoidable terms.
How the gap shows
  • Status or sign-off documents for the business use unexplained abbreviations and technical terms.
  • Decisions are described by their technical option, not by their effect on users.
  • No glossary in documents read by business stakeholders.
Compare
Compare a decision record with the business owner's own description of it (in notes or e-mail): if they describe a different consequence, the decision was not understood.
Due by
planstage 3: Discovery
evidencedstage 4: Discovery sign-off
Fail fast
Ask the business owner to explain the three biggest decisions back in their own words.
Related
STK-05
COM-03Agreed in a meeting, lost afterwardsmediumProject manager
Question
Does every important meeting end with written actions and decisions, sent to the people affected and copied into the decision log or RAID log? A month from now, will anyone remember what was agreed today?
Why it matters
The source owner agreed in a meeting to provide a daily extract by a set time. Nothing was written, the owner moved on, and the successor had never heard of it.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Technical anchor
Evidence
in: status, decisions
looks like: Short meeting notes with actions (owner and date) and decisions, sent the same day, and decisions copied into the decision log.
How the gap shows
  • No meeting notes, or notes without actions and owners.
  • Actions in notes that never appear again in later notes or status reports.
  • Phrases such as "as discussed" referring to something with no written record.
Compare
Compare meeting notes with the decision log and RAID log: decisions and actions in the notes that are missing from the logs have been lost.
Due by
planstage 3: Discovery
evidencedstage 5: MVP starts
Fail fast
After the next meeting, send three lines within an hour: decided, actions, open questions.
Related
PM-03
COM-04Nobody knows who to callmediumProject manager
Question
Is there a contact list for the project and the systems it depends on, including who to call when a source changes or a load fails, and how to escalate? If the pipeline fails at night in the first week of production, who is woken up and who do they call?
Why it matters
A source was changed without notice and the load failed. Nobody knew who owned the source, so it took three days to find the right person while the reports stayed empty.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Operations and support, Source system owners
Evidence
in: stakeholders, runbook
looks like: A contact and escalation list for the team, each source, the platform, security and operations, with named people and a deputy for each.
How the gap shows
  • No contact or escalation list in the stakeholder map or runbook.
  • Source systems named in the architecture with no named owner.
  • Escalation described as "raise with management".
Compare
Compare the architecture's list of sources and systems with the contact list: a system with no contact is one that will fail silently.
Due by
planstage 3: Discovery
evidencedstage 8: Deploy
Fail fast
For each source, find the named owner and confirm they will tell the team before changing it.
Related
STK-01, HAND-04
COM-05Assumptions never shared with the businessmediumProject manager
Question
Have the assumptions behind the estimate and plan been shown to the sponsor and the business owner, and do they know what changes if one of them breaks? If an assumption fails, will the business be surprised that the plan ever depended on it?
Why it matters
The estimate assumed the business would provide testers for two weeks. The business never knew; when no testers came, user acceptance slipped by a month and the team was blamed.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Technical anchor, Business owner and sponsor
Evidence
in: status, estimate
looks like: The key assumptions repeated in status reports and sign-off packs, with their current state (holding, at risk, broken).
How the gap shows
  • Status reports and steering packs do not mention assumptions.
  • Assumptions that require business action (testers, decisions, data) are not assigned to a business person.
Compare
Compare the estimate's assumptions with the status reports: assumptions in the estimate that never appear in any report have not been shared.
Due by
planstage 4: Discovery sign-off
evidencedstage 6: Build
Fail fast
Add a three-line "assumptions this plan depends on" box to the next status report.
Related
EST-03, PM-07
COM-06No regular rhythm for updateslowProject manager
Question
Is there an agreed rhythm for telling stakeholders what was learned, what changed, and what is needed from them? Which stakeholder will say they heard nothing for a month?
Why it matters
Updates went out only when there was good news. Stakeholders filled the silence with rumours and the sponsor started asking for daily reports, which took the team's time away from the work.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Technical anchor
Evidence
in: stakeholders, status
looks like: A communication plan stating who gets what and how often, and status reports that follow it.
How the gap shows
  • No communication plan.
  • Gaps of several weeks between status reports.
  • Status reports go to the sponsor only.
Compare
Compare the communication plan with the actual status reports: a planned weekly update that appears monthly is a rhythm that broke.
Due by
planstage 4: Discovery sign-off
evidencedstage 6: Build
Fail fast
Send a three-line update this Friday: what we learned, what changed, what we need from you.
Related
COM-01
COM-07Steering packs that ask for nothingmediumProject manager
Question
Does each steering pack open with the decisions and help the project needs from that meeting (a date, a budget release, a source team pushed, a definition settled), each with options and a deadline? After the next steering meeting, which blocker will still be blocked because we only reported progress?
Why it matters
Steering packs ran to twenty slides of activity. The one line that mattered, that a source team had not started its extract change, sat on slide fourteen, and nobody in the room acted on it for three meetings.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Technical anchor, Business owner and sponsor
Evidence
in: status, decisions
looks like: A first page listing the decisions and help needed, each with options, a recommendation and a deadline, and notes from the meeting recording what was decided.
How the gap shows
  • Steering packs contain progress and activity but no "decisions needed" section.
  • The same blocker appears in several packs with no request attached.
  • Steering notes record no decisions.
Compare
Compare open issues and decisions in the RAID log with the steering packs: an issue that needs steering action but never appears as a request in the pack is being reported, not escalated.
Due by
planstage 4: Discovery sign-off
evidencedstage 6: Build
Fail fast
Put "decisions we need from you today" as the first slide of the next steering pack.
Related
GOV-01, PM-08, COM-01

Documentation, knowledge transfer, support model (2)

HAND-04Support model not agreedhighProject manager
Question
Is it agreed who supports the system after go-live, in which hours, how fast they respond, how long the project team stays during the early weeks, and who pays? On the first weekend in production, who answers a failure?
Why it matters
The project assumed the central support team would take over; the support team assumed the project would. After go-live a failed weekend load went unnoticed until Monday morning.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Operations and support, Business owner and sponsor, Technical anchor
Evidence
in: runbook, plan
looks like: A signed support model: support team, hours, response times, escalation, early-life support period, and the criteria for ending it.
How the gap shows
  • No support model, or one that says "business as usual support".
  • The support team has not been asked to accept the system.
  • Support hours do not match the freshness promised to users.
Compare
Compare the support model with the non-functional requirements: data promised by 7 a.m. but support starting at 9 a.m. means nobody fixes a night failure in time.
Due by
planstage 4: Discovery sign-off
designedstage 6: Build
evidencedstage 9: Production sign-off
Fail fast
Ask the support team lead whether they have agreed to take this system, and on what terms.
Related
REQ-06, HAND-01, COM-04
HAND-05Benefits never measuredmediumBusiness owner and sponsor
Question
Is there a plan to measure, after go-live, the benefits promised at the start, against the baseline taken before? A year after closure, can anyone prove the project was worth its cost?
Why it matters
The business case promised hours saved each month. Nobody measured hours before or after, so the project could not show any return and the platform's next phase was refused.
Applies to
any · components: any · patterns: any
Owner
Business owner and sponsor · also Project manager, Business analyst
Evidence
in: discovery, status
looks like: A benefits list with baseline values, the date and method of measurement after go-live, and an owner who will report the result.
How the gap shows
  • The business case or goals have no baseline figures.
  • No benefits review is planned after go-live.
  • The project closure report does not mention benefits.
Compare
Compare the goals in the discovery document with the closure or status reports: goals that are never reported on were never measured.
Due by
planstage 3: Discovery
evidencedstage 10: Project sign-off
Fail fast
Record today's value of each promised benefit, so there is something to compare against later.
Related
REQ-07

Budget and its basis, spend against plan, contingency, running cost ownership, cost of change (6)

BUD-01A budget with no basishighProject manager
Question
Can every line of the budget be traced to something written down: the estimate range and its assumptions, day rates, environments, licences, running cost and contingency? If the budget runs out in month five, will anyone be able to say which assumption it was built on?
Why it matters
The budget was the number the sponsor had left at year end. It matched the low end of the rough estimate, held no running cost and no history load, and was spent before the first report reached production.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Business owner and sponsor, Technical anchor
Evidence
in: budget, estimate, plan
looks like: A budget broken into people, environments, licences, external services, running cost during the project and contingency, each line pointing to the estimate or a quote, with the estimate's range shown.
How the gap shows
  • The budget is one number with no breakdown.
  • The budget matches the low end of the estimate range, or the rough estimate from before discovery.
  • No line for running cost, environments or licences while the project is being built.
Compare
Compare the budget with the estimate: a budget below the estimate's likely figure, or built from an earlier version of the estimate, is already overspent.
Due by
planstage 2: Rough estimate
designedstage 4: Discovery sign-off
Fail fast
Rebuild the budget from the current estimate's middle figure and put the gap, if any, in front of the sponsor.
Related
EST-01, EST-02, BUD-04
BUD-02Spend not tracked against the planhighProject manager
Question
Is actual spend compared with planned spend at least monthly, with a forecast of the total at completion? If the money runs out early, will the status reports from three months before have shown it coming?
Why it matters
Contractor invoices went straight to finance. The project manager saw the total only at quarter end, when seventy percent of the budget was gone and forty percent of the work was done.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Business owner and sponsor
Evidence
in: budget, status, plan
looks like: A monthly table of planned spend, actual spend, forecast to complete and forecast at completion, set against the share of work done, shown in the steering pack.
How the gap shows
  • Status reports mention dates and scope but never money.
  • Spend is reported as a total so far, with no plan to compare it with and no forecast.
  • The forecast at completion equals the budget in every report.
Compare
Compare spend in the status reports with progress against the estimate: money spent faster than work is finished, while the forecast total never changes, means the forecast is not real.
Due by
planstage 4: Discovery sign-off
evidencedstage 6: Build
Fail fast
Divide spend so far by the share of work done and compare the answer with the budget today.
Related
EST-06, PM-04
BUD-03Nobody owns the running cost after go-livehighBusiness owner and sponsor
Question
Is the monthly cost of running the solution (compute, storage, licences, support) agreed with the person whose budget will pay it, for this year and next? When the first production bill arrives, whose cost centre is it charged to, and do they know?
Why it matters
The project budget ended at go-live. The platform bill landed on the central data team, who had not planned for it, and they switched off the hourly refresh to stay within their own budget.
Applies to
any · components: any · patterns: any
Owner
Business owner and sponsor · also Project manager, Technical anchor, Platform team
Evidence
in: discovery, plan
looks like: The running cost figure from the technical estimate, a named budget owner who has accepted it, the cost centre, and a forecast for the next two years that includes data growth.
How the gap shows
  • Running cost is described as "business as usual" or "absorbed by the platform".
  • The business case shows build cost only.
  • No named person has agreed to pay the running cost.
Compare
Compare the running cost estimate in the architecture or discovery document with the business case: a business case without that cost overstates the return.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 9: Production sign-off
Fail fast
Send the running cost figure to the person expected to pay it and ask them to confirm it in writing.
Related
COST-01, COST-03, HAND-04
BUD-04Contingency hidden or missingmediumProject manager
Question
Is contingency held as a separate, visible amount, sized to the riskiest assumptions, with a named person who can release it and a record of each use? When the data turns out worse than profiled, where will the money for fixing it come from?
Why it matters
Every estimator padded their own numbers quietly. The padding was spent on gold-plating early on, and when the history load needed three extra weeks there was nothing left and no one could approve more.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Business owner and sponsor, Technical anchor
Evidence
in: budget, estimate, raid
looks like: A contingency line linked to the top risks and assumptions, an owner who approves its use, and a log of drawdowns showing what each one paid for.
How the gap shows
  • No contingency in the budget, or a flat percentage with no link to risks.
  • Contingency described as "included in the estimates".
  • The risk log has high risks with costly mitigations that no budget line pays for.
Compare
Compare the risk log with the budget: high risks whose mitigation costs money, with no contingency to cover them, are unfunded risks.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 6: Build
Fail fast
Price the mitigation of the top three risks and compare the total with the contingency held.
Related
BUD-01, PM-02, EST-03
BUD-05Change requests with no costmediumProject manager
Question
Does every change request show what it adds in build cost, running cost and time, before anyone decides? If the project overspends, how much of it will be changes that were approved without a price?
Why it matters
The business asked for the report to refresh hourly instead of daily. It was agreed in a demo as a small change, and it tripled the monthly compute bill and added two weeks of testing nobody had priced.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Business owner and sponsor, Technical anchor
Evidence
in: decisions, status
looks like: A change log where each entry shows the build effort, the change to running cost, the effect on the date and the decider, with a running total of approved changes against contingency.
How the gap shows
  • Change records that state the request and the decision but not the cost.
  • Changes to freshness, history or volume approved without a running cost figure.
  • No running total of the cost of approved changes.
Compare
Compare the change log with the budget and forecast: changes approved since the baseline, with a forecast at completion that has not moved, means their cost was never added.
Due by
planstage 4: Discovery sign-off
evidencedstage 6: Build
Fail fast
Price the last three approved changes after the fact and add them to the forecast.
Related
SCOPE-03, COST-01
BUD-06Costs that sit in other teams' budgetsmediumProject manager
Question
Have the costs that land outside the project been found and agreed: the source team's work on a new extract, a vendor's fee for an interface or bulk export, extra environments, and running the old system during a parallel run? Who will refuse to pay for something the project cannot go live without?
Why it matters
The source vendor charged for every change to the extract, and the source team had no budget for the work. The project waited two months while the two managers argued over who would pay.
Applies to
any · components: ingestion, platform · patterns: any
Owner
Project manager · also Source system owners, Business owner and sponsor, Platform team
Evidence
in: budget, plan, raid
looks like: A list of costs the project causes in other teams or with vendors, with the amount, who pays, and the date each was agreed.
How the gap shows
  • Source changes, vendor interfaces or extra environments appear in the architecture with no cost or payer.
  • The old system's licence or support is due to end before the planned parallel run is over.
  • Phrases such as "the source team will provide" with no mention of their budget.
Compare
Compare the architecture's list of source changes and environments with the budget: each one that needs another team's work or a vendor's fee, with no agreed payer, is a stop waiting to happen.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
Fail fast
Ask each source owner and vendor whether the work the project needs from them is funded, and by whom.
Related
VEN-01, VEN-03, PM-15

Named people and their real availability, onboarding, booked time from source experts and business users (4)

RES-01Capacity counted in people, not in real hourshighProject manager
Question
Does the plan use each person's real availability: their share of time on this project, planned leave, public holidays, other commitments and support duties? If the team delivers two thirds of what was planned, will the missing third be leave and part-time allocations nobody subtracted?
Why it matters
The plan assumed five full-time developers. Two were shared with production support, one had three weeks of leave in the summer, and the real capacity was barely three people, so the first MVP slipped a month.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Technical anchor
Evidence
in: plan, estimate
looks like: A team table with each named person, their percentage on the project, known leave and the weeks they are needed, with the total compared against the effort in the estimate.
How the gap shows
  • The plan lists roles or head counts with no names or percentages.
  • No mention of leave, holidays or support duties.
  • The same person is allocated full time on this project and another one.
Compare
Compare the estimate's effort with the plan's available hours: if effort divided by real capacity gives a later date than the plan shows, the plan assumes people who do not exist.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 6: Build
Fail fast
Add up each person's real available days for the next eight weeks and compare them with the planned work.
Related
PM-06, PM-07, EST-03
RES-02New joiners assumed productive on day onemediumProject manager
Question
Does the plan allow for the time a new person needs before they are useful: laptop, accounts, access to sources and environments, security training and learning the domain? When a developer joins in week six, in which week will they first deliver?
Why it matters
Two contractors joined to catch up the plan. Their access to the source data took four weeks to approve, so they were paid for a month of reading documents while the slip grew.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Technical anchor, Platform team, Security and governance
Evidence
in: plan, raid
looks like: An onboarding checklist with the lead time of each item, requests raised before the start date, and the plan showing reduced output for each new person's first weeks.
How the gap shows
  • People are added to the plan at full output from their first week.
  • No onboarding checklist or access lead times are written down.
  • Recovery plans rely on adding people late in the build.
Compare
Compare the plan's joining dates with the access request lead times in the RAID log: a person planned to deliver before their access can arrive will sit idle.
Due by
planstage 4: Discovery sign-off
evidencedstage 6: Build
Fail fast
Ask the most recent joiner how many days passed before they could query real data, and use that number.
Related
PM-10, ENV-02, RES-01
RES-03Source experts' time never bookedhighProject manager
Question
Is the time the project needs from source system experts (to explain tables, deletes and odd values, to review mappings and to fix extracts) agreed with their manager, in hours per week and by date? When the team needs an answer about the source, how long will it wait?
Why it matters
The only person who understood the source's status codes worked on that system's own release. She gave the project an hour a week when she could, and every mapping question waited ten days for an answer.
Applies to
any · components: ingestion, transformation · patterns: any
Owner
Project manager · also Source system owners, Technical anchor, Business analyst
Evidence
in: plan, stakeholders
looks like: Named source experts with agreed hours per week through discovery, build and test, confirmed by their line manager, and a deputy for each.
How the gap shows
  • Source experts are named in the stakeholder map with no time commitment.
  • Open questions about source data stay open across several status reports.
  • The plan relies on "input from the source team" without names or dates.
Compare
Compare the number of sources in the architecture with the expert time booked in the plan: a source with no booked expert will be understood by guessing.
Due by
planstage 3: Discovery
evidencedstage 5: MVP starts
Fail fast
Ask each source expert's manager to confirm, in writing, the hours per week they will give the project.
Related
PM-06, REQ-04, DQ-05
RES-04Business testers and sign-off people not bookedhighProject manager
Question
Are the business users who will test, reconcile numbers and sign off named, with their days booked and clear of month-end, year-end and their own peak periods? If user acceptance slips, will it be because the testers were closing the books that week?
Why it matters
User acceptance was planned for the last week of the quarter. Every finance tester was on the quarter close, the test window passed with a handful of cases run, and go-live moved by six weeks.
Applies to
any · components: modelling, serving · patterns: any
Owner
Project manager · also Business owner and sponsor, Test lead, Data consumers
Evidence
in: plan, test-strategy
looks like: Named testers and approvers, the days each has agreed to give, their managers' agreement, and a test window checked against the business calendar.
How the gap shows
  • The test strategy says "business users" without names.
  • User acceptance dates fall in a month-end, quarter-end or year-end period.
  • No business manager has agreed to release testers.
Compare
Compare the plan's user acceptance window with the business calendar or the testers' own commitments: a window that overlaps their busiest week will not get the testing it needs.
Due by
planstage 4: Discovery sign-off
designedstage 6: Build
evidencedstage 7: Test
Fail fast
Ask each named tester's manager to put the test days in their calendar now.
Related
TEST-04, COM-05, PM-18

Source teams, vendors and suppliers as dependencies, with agreed dates, terms and escalation (4)

VEN-01Source-side changes not on the source team's planhighProject manager
Question
For each change the project needs from a source (a new extract, a new field, change tracking switched on, a service account, a firewall rule), is it on the source team's or vendor's own plan, with a release date, a named contact and an escalation route? If the source team's next release slot is in four months, when will we find out?
Why it matters
The project needed change tracking switched on in the source database. The source team agreed in a meeting, but their releases were quarterly and the next slot was full, so the build waited three months.
Applies to
any · components: ingestion · patterns: any
Owner
Project manager · also Source system owners, Technical anchor
Evidence
in: raid, plan
looks like: A list of source-side changes, each with the item reference on the source team's own plan, the release date they committed to, a named contact and the person to escalate to.
How the gap shows
  • Source changes appear in the architecture but not as dependencies in the RAID log.
  • Dependencies on source teams have no date, or a date the source team has not confirmed.
  • No mention of the source team's release calendar or change freezes.
Compare
Compare the architecture's needs from each source with the RAID log's dependencies: a change the design needs that is not on the source team's plan is a date nobody has promised.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 5: MVP starts
Fail fast
Ask each source team for the reference number of the project's request on their own plan, and its release date.
Related
PM-01, HIST-02, SCH-03
VEN-02No agreement on how data is deliveredhighProject manager
Question
Is there a written agreement with each source team or external supplier on what they deliver: by what time, in what format, how complete, how a late or broken delivery is reported and fixed, and in which support hours? When the file is late at 6 a.m., what does the agreement say happens?
Why it matters
An external partner sent a daily file "in the morning". Some days it came at nine, some days at two, and the reports promised for eight o'clock were wrong half the week with nothing in writing to point to.
Applies to
any · components: ingestion · patterns: file, api, database, cdc
Owner
Project manager · also Source system owners, Technical anchor, Operations and support
Evidence
in: stakeholders, runbook
looks like: A delivery agreement or contract schedule per source: delivery time, format, completeness check, notice of change, support hours, contact and escalation, and what happens when it is missed.
How the gap shows
  • Source delivery times written as "daily" or "overnight" with no clock time.
  • No support hours or contacts for external sources.
  • Freshness promised to users is tighter than anything a source has agreed to.
Compare
Compare the freshness promised to consumers with the delivery times agreed with sources: a report promised at 8 a.m. from a source that only promises "the morning" cannot keep its promise.
Due by
planstage 4: Discovery sign-off
designedstage 6: Build
evidencedstage 8: Deploy
Fail fast
Ask each source owner to state the latest time they will deliver and what they do when they cannot.
Related
SERV-01, SCH-03, PERF-02, DQ-04
VEN-03Licences and contract terms for getting the datamediumProject manager
Question
Do the contracts with source system vendors allow the project to extract the data in the way and volume the design needs, and are the licences for every tool and user covered? Which clause will we discover the week before go-live?
Why it matters
The design pulled history through the vendor's interface. The contract capped calls per day and charged for bulk export, which nobody had read, so the history load needed a new commercial agreement.
Applies to
any · components: ingestion, serving, platform · patterns: any
Owner
Project manager · also Source system owners, Business owner and sponsor, Security and governance
Evidence
in: raid, architecture
looks like: A note per vendor source and tool of the contract terms checked (extract rights, call limits, export fees, data use), the licence count needed against the count owned, and who checked it.
How the gap shows
  • Vendor-hosted sources in the architecture with no mention of contract terms or limits.
  • Licences for report users or tools are not mentioned in the budget.
  • The number of users planned is larger than the licences held, or the count is unknown.
Compare
Compare the history load and refresh approach in the architecture with the vendor's stated limits: a design that needs more calls or bulk export than the contract allows will be stopped.
Due by
planstage 3: Discovery
evidencedstage 4: Discovery sign-off
Fail fast
Send the vendor the planned extract approach and volumes and ask them to confirm it is allowed at no extra cost.
Related
BUD-06, HIST-02
VEN-04Supplier work with no acceptance or handover termsmediumProject manager
Question
If an outside supplier builds part of the solution, does their contract say what "done" means (tested, reconciled, documented), how it is accepted, who fixes defects after go-live and for how long, and what knowledge they must hand over? When the supplier's contract ends at go-live, who fixes the first broken load?
Why it matters
The supplier was paid on delivery of the pipelines. Their contract ended at go-live, the first month-end load failed, and the internal team had no documentation and no right to ask the supplier back.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Technical anchor, Operations and support
Evidence
in: plan, runbook
looks like: A statement of work with acceptance criteria tied to tests and reconciliation, a warranty period that covers the first month-end, and handover deliverables with dates.
How the gap shows
  • Supplier deliverables are described as code or pipelines, with no tests or documents.
  • The supplier's contract ends on or before the go-live date.
  • No warranty or defect-fix period is mentioned.
Compare
Compare the supplier's end date with the plan's early-life support period: a supplier who leaves before support ends leaves the defects behind.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 9: Production sign-off
Fail fast
Read the supplier's acceptance and end-date clauses against the go-live and month-end dates this week.
Related
HAND-03, HAND-06, PM-16

Steering group, stage gates and their exit criteria, escalation, approval boards, lessons per gate (6)

GOV-01A steering group that cannot decidehighProject manager
Question
Does the steering group include people who can decide on money, scope and dates, plus the owners of the data and the platform the project depends on, and is it written down what it decides and how often it meets? When the project needs a yes on extra budget or a later date, will the people in the room be able to give it?
Why it matters
The steering group was the project team's managers. When the source team refused to change their extract, nobody in the room had authority over them, and the issue went round the meeting for two months.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Business owner and sponsor
Evidence
in: stakeholders, status
looks like: Short terms of reference: members by name, what the group decides (budget, scope, dates, go-live), how often it meets, and attendance records showing the deciders actually came.
How the gap shows
  • No steering group or terms of reference is written down.
  • Members are all from the delivery side, with no data owner, source owner or budget holder.
  • Steering notes record updates but no decisions.
Compare
Compare the steering membership with the RAID log's biggest dependencies and issues: an issue owned by a team nobody on the steering group can direct will not be solved there.
Due by
planstage 3: Discovery
evidencedstage 4: Discovery sign-off
Fail fast
Take the top open issue to the next steering meeting and see whether anyone there can decide it.
Related
STK-02, GOV-03
GOV-02Stage gates with no exit criteriahighProject manager
Question
Do discovery sign-off, production sign-off and project sign-off each have written exit criteria and the evidence each needs (for example reconciled numbers, clean runs, an accepted support model, measured benefits), agreed before the gate? At each gate, are we deciding on evidence or on the calendar?
Why it matters
Production sign-off was a meeting on the planned date. It was signed with reconciliation still failing for two regions, and the project team was released, so the differences were never explained.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Business owner and sponsor, Technical anchor, Operations and support
Evidence
in: plan, decisions
looks like: For each gate, a short list of exit criteria with the document or test result that proves each one, who signs, and the possible outcomes (go, go with conditions, stop).
How the gap shows
  • Gates appear in the plan as dates only.
  • Production or project sign-off has no criteria, or criteria such as "all work complete".
  • Sign-off records do not mention which criteria were met.
Compare
Compare each gate's sign-off record with its exit criteria and the evidence available on that date: a gate signed while a criterion was unmet was passed on the date, not the evidence.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 9: Production sign-off
provenstage 10: Project sign-off
Fail fast
Write the production sign-off criteria now and check which of them the project could meet today.
Related
PM-09, HAND-06, REC-04
GOV-03Escalation with no time limitsmediumProject manager
Question
Is there a written escalation path for blockers, saying how long a problem may stay at each level before it goes to the next, up to someone who can direct other teams? How many days can a blocked access request or a source team's refusal sit before the sponsor hears about it?
Why it matters
A firewall change was blocked for seven weeks. The team kept chasing it politely, nobody had said when to escalate, and the sponsor learned of it when the date slipped.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Business owner and sponsor, Technical anchor
Evidence
in: stakeholders, raid
looks like: An escalation path with levels, named people and time limits (for example three working days at team level, a week at project manager level, then the steering group), and issues in the RAID log showing when they were escalated.
How the gap shows
  • Escalation described as "raise with management" with no names or times.
  • Issues in the RAID log open for weeks with no record of escalation.
  • Dependencies on other teams have slipped more than once with no escalation.
Compare
Compare the age of open issues in the RAID log with the escalation time limits: an issue older than its limit that was never escalated shows the path is not used.
Due by
planstage 3: Discovery
evidencedstage 5: MVP starts
Fail fast
List every issue open longer than two weeks and escalate each one by Friday.
Related
COM-04, PM-10, GOV-01
GOV-04Approval boards and their calendars not on the planhighProject manager
Question
Are the outside approvals the project needs (architecture review, security review, privacy assessment, data sharing agreement, change board) listed with their meeting dates, what they need submitted and how long they take? Which board meets monthly and has not yet heard of us?
Why it matters
The privacy assessment had to be approved before personal data could be copied to the new platform. The board met monthly, sent the first submission back, and the project lost two months it had not planned for.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Security and governance, Architect, Technical anchor
Evidence
in: plan, raid
looks like: A list of approvals with the board, its meeting calendar, the submission deadline, what it needs, the usual number of rounds, and the plan tasks that wait on it.
How the gap shows
  • No approvals or review boards appear in the plan.
  • Personal or sensitive data is in scope but no privacy assessment or data sharing agreement is mentioned.
  • Approvals are planned in the same week the work that needs them starts.
Compare
Compare the architecture's personal data flows and platform changes with the plan's approvals: each one that needs a board's approval, with no slot booked, is an unplanned wait.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 6: Build
Fail fast
Ask security and the architecture board which approvals this project needs and the next date each one meets.
Related
SEC-01, SEC-05, PM-01, STK-01
GOV-05Conditions from a gate never closedmediumProject manager
Question
When a gate is passed "with conditions", is each condition written down with an owner and a date, and checked at the next gate? Which condition from discovery sign-off is still open, and does anyone remember it?
Why it matters
Discovery was approved on condition that the source owner confirmed delete handling. Nobody tracked the condition, it was never confirmed, and deleted orders were still counted in production.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Business owner and sponsor, Technical anchor
Evidence
in: decisions, raid
looks like: Each gate condition recorded as an action in the RAID log with owner and date, and the next gate's pack showing the state of every earlier condition.
How the gap shows
  • Sign-off records say "approved with conditions" or "subject to" with no list of conditions.
  • Conditions from an earlier gate are not mentioned in later status reports or gate packs.
Compare
Compare the conditions in each sign-off record with the RAID log: a condition that never became a tracked action has been forgotten.
Due by
planstage 4: Discovery sign-off
evidencedstage 9: Production sign-off
Fail fast
Read the last sign-off record and list every condition with its owner and current state.
Related
GOV-02, STK-05, COM-03
GOV-06Lessons captured only at the endlowProject manager
Question
Are lessons gathered at each gate and after each MVP, with an action that changes how the next stage is run? If the same data surprise hits MVP 2 that hit MVP 1, will anyone have written down how to avoid it?
Why it matters
The first MVP lost three weeks to late source access. The lessons review was held at project closure, so MVP 2 and MVP 3 lost the same three weeks for the same reason.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Technical anchor, Business analyst
Evidence
in: status, plan
looks like: A short lessons list after each gate and MVP: what happened, what we will do differently, the owner of that change, and evidence in the next stage's plan that it was done.
How the gap shows
  • The only lessons review in the plan is at project closure.
  • The same kind of issue appears in the RAID log for more than one MVP.
  • Lessons are listed with no action or owner.
Compare
Compare the lessons from one stage with the plan for the next: a lesson with no matching change in the next plan has not been learned.
Due by
planstage 4: Discovery sign-off
evidencedstage 6: Build
provenstage 10: Project sign-off
Fail fast
Spend fifteen minutes at the next retrospective on one question: what will we change for the next MVP?
Related
PM-02, EST-06

Go or no-go, business readiness, old reports switched off, clashing releases, consumers told (5)

RLC-01Go or no-go decided on the day, not agreed in advancehighProject manager
Question
Are the go-live criteria (reconciled numbers within tolerance, no open critical defects, support ready, users trained, rollback rehearsed) written down and agreed with the business owner before the go or no-go meeting? If the history load finishes with two regions unreconciled, do we already know whether we go?
Why it matters
With no agreed criteria, the go or no-go meeting became a debate. The sponsor wanted the date, the test lead wanted another week, and they went live with known differences that took three months to explain away.
Applies to
any · components: any · patterns: any
Owner
Project manager · also Business owner and sponsor, Technical anchor, Test lead, Operations and support
Evidence
in: deployment, decisions
looks like: A go or no-go checklist agreed weeks before cut-over, with each criterion, the evidence for it, who confirms it, and what happens if it fails (delay, go with a workaround, or partial release).
How the gap shows
  • No go-live criteria are written, or they say "UAT signed off".
  • The go or no-go decision has no named decider.
  • Criteria do not mention reconciliation, support readiness or rollback.
Compare
Compare the go-live criteria with the test results and defect list on the decision date: a go decision taken while a criterion was unmet means the criteria were not the real basis.
Due by
planstage 4: Discovery sign-off
designedstage 7: Test
evidencedstage 8: Deploy
Fail fast
Draft the go-live checklist now and ask the business owner which line they would refuse to go without.
Related
GOV-02, PM-14, REC-03
RLC-02Users not ready on go-live daymediumProject manager
Question
Are the users ready: told what is changing and when, trained on the new reports and what the numbers now mean, and pointed at where to ask for help, with the service desk briefed? On the first Monday, will users know why their numbers look different from last week?
Why it matters
The new sales report used the agreed definition of revenue, which excluded returns. Nobody told the regional managers, who saw lower figures, assumed the report was broken, and kept using their old spreadsheets.
Applies to
any · components: serving, modelling · patterns: any
Owner
Project manager · also Business owner and sponsor, Business analyst, Data consumers
Evidence
in: plan, stakeholders
looks like: A business readiness plan: user groups, messages and dates, training sessions with attendance, a short note on changed definitions and why numbers differ, and a briefed service desk.
How the gap shows
  • No training, user communication or service desk briefing is in the plan.
  • Definitions changed during the project but no user-facing explanation exists.
  • The only user communication is a go-live announcement.
Compare
Compare definitions that changed in the decision log or data model with the user communications: a changed definition users were never told about will be reported as a defect.
Due by
planstage 4: Discovery sign-off
designedstage 7: Test
evidencedstage 8: Deploy
Fail fast
Ask three future users what will change for them on go-live day, and note what they do not know.
Related
COM-02, REQ-03, STK-03
RLC-03Old reports and feeds never switched offmediumProject manager
Question
Is there a list of the old reports, extracts and spreadsheets the new solution replaces, each with an owner who agrees to switch it off and a date? A year after go-live, how many people will still be using the old numbers, and paying to run them?
Why it matters
The old reports were left running "just in case". Two years later both sets were still in use, gave different answers in board meetings, and the cost of the old system was never saved.
Applies to
any · components: serving · patterns: any
Owner
Project manager · also Business owner and sponsor, Data consumers, Operations and support
Evidence
in: plan, deployment
looks like: A switch-off list naming each old report or feed, its users, its owner, the switch-off date after the parallel run, and confirmation that it was done.
How the gap shows
  • The plan has no switch-off or decommissioning tasks.
  • Benefits depend on retiring an old system, but no retirement date exists.
  • Old reports are listed as "to be reviewed after go-live".
Compare
Compare the benefits in the business case with the switch-off list: savings that depend on retiring the old system, with no switch-off date, will not arrive.
Due by
planstage 4: Discovery sign-off
designedstage 7: Test
evidencedstage 9: Production sign-off
provenstage 10: Project sign-off
Fail fast
List every old report the new one replaces and ask each owner for a switch-off date.
Related
PM-15, BEN-01, HAND-05
RLC-04Clashes with other releases and platform changesmediumTechnical anchor
Question
Has the release been checked against other changes in the same window: source system upgrades, platform upgrades, other projects changing shared tables, network or identity changes? Which other team's release could break ours the same weekend?
Why it matters
The project went live the same weekend the source system was upgraded. Two columns changed type, the first production loads failed, and nobody could tell which release had caused it.
Applies to
any · components: any · patterns: any
Owner
Technical anchor · also Project manager, Release and DevOps, Platform team, Source system owners
Evidence
in: deployment, raid
looks like: The release date checked against the change calendar for sources, the platform and shared data, with clashes recorded as dependencies or risks and agreed with the other teams.
How the gap shows
  • No mention of the change calendar or other releases in the deployment plan.
  • A shared platform or source upgrade is known but not on the RAID log.
  • Other projects write to tables this project reads, with no coordination recorded.
Compare
Compare the go-live date in the deployment plan with source and platform release dates in the RAID log or change calendar: two changes to the same system in one window will hide each other's faults.
Due by
planstage 4: Discovery sign-off
evidencedstage 8: Deploy
Fail fast
Ask the change board for every change planned to the sources and platform in the go-live week.
Related
PM-18, SCH-03, REL-07
RLC-05Consumers outside the steering group not toldhighProject manager
Question
Have all consumers of the data being replaced or changed been found and told, including other teams' extracts, downstream systems and spreadsheets fed from the old source, not only the users in the steering group? Who will discover our go-live because their own report broke?
Why it matters
The old warehouse table was replaced. A pricing team nobody had met read it every night for their own model, and their prices were wrong for three days before anyone linked it to the release.
Applies to
any · components: serving, modelling · patterns: any
Owner
Project manager · also Technical anchor, Data consumers, Operations and support
Evidence
in: stakeholders, deployment
looks like: A consumer list built from access logs and lineage, not only interviews, each consumer told the change and the date, and their confirmation recorded.
How the gap shows
  • The consumer list comes only from the steering group and requirements workshops.
  • No one has checked who reads the tables or feeds being replaced.
  • No notice period for consumers is in the deployment plan.
Compare
Compare the consumers in the stakeholder map with the readers found in access logs or lineage: a reader who is not on the map has not been told.
Due by
planstage 4: Discovery sign-off
evidencedstage 8: Deploy
Fail fast
Pull the last ninety days of reads on the tables being replaced and contact every reader not on the map.
Related
STK-01, SERV-03, SERV-01

Whether the promised value arrives after go-live - adoption and the business changes it depends on (2)

BEN-01Adoption never trackedmediumBusiness owner and sponsor
Question
After go-live, will anyone track who is using the new reports and data, compared with the old ones, and act when a user group does not move? Six months on, how will we know the people the project was built for are actually using it?
Why it matters
The new dashboards went live on time and the project closed. Usage logs, read a year later, showed most managers had opened them once and gone back to spreadsheets exported from the old system.
Applies to
any · components: serving · patterns: any
Owner
Business owner and sponsor · also Project manager, Data consumers
Evidence
in: plan, status
looks like: An adoption measure per user group (for example weekly active users of the new reports against use of the old ones), a target, a named person who reviews it monthly, and an action when a group lags.
How the gap shows
  • No adoption or usage measure in the plan or the benefits list.
  • The project closes on go-live with no review of usage afterwards.
  • Old reports keep running with no plan to compare their use with the new ones.
Compare
Compare the benefits in the business case with the measures planned after go-live: a benefit that depends on people using the new reports, with no usage measure, cannot be shown.
Due by
planstage 4: Discovery sign-off
evidencedstage 10: Project sign-off
Fail fast
Agree today which usage figure, read one month after go-live, would tell the sponsor adoption is failing.
Related
HAND-05, RLC-03, RLC-02
BEN-02Benefits depend on business changes nobody ownsmediumBusiness owner and sponsor
Question
For each promised benefit, is it written down what the business must change to get it (a process, a role, a meeting, a decision that now uses the new data), and who owns that change? If the data arrives perfectly and nothing improves, which business change did nobody make?
Why it matters
The business case promised two days saved at each month-end. The data arrived early as designed, but the finance close process was never changed, so the team still waited for the old sign-off steps.
Applies to
any · components: any · patterns: any
Owner
Business owner and sponsor · also Project manager, Business analyst
Evidence
in: discovery, plan
looks like: A benefits map linking each benefit to the data the project delivers and to the business change it needs, with an owner and date for each business change.
How the gap shows
  • Benefits are described as outcomes of the data alone, with no business change named.
  • Process or role changes are mentioned as "the business will adapt" with no owner.
  • No business change tasks appear in any plan.
Compare
Compare the benefits in the business case with the scope: a benefit that needs a process change that is outside scope and owned by nobody will not arrive.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 10: Project sign-off
Fail fast
For the biggest benefit, ask the business owner which process changes on day one and who will change it.
Related
HAND-05, REQ-07, STK-06

Technical (91)

Whether it will work: the things that make numbers wrong, loads fail or go-lives slip, plus the effort estimate and technical handover.

Same data arriving or being processed twice (4)

DUP-01Same batch or file arriving twicehighTechnical anchor
Question
How is a file, batch or extract that arrives a second time prevented from doubling the numbers?
Why it matters
After an outage the source resent yesterday's file, the pipeline loaded it again as new, and revenue on the dashboard doubled for a day before anyone noticed.
Applies to
any · components: ingestion · patterns: batch, file, api, database
Owner
Technical anchor · also Developers, Test lead
Evidence
in: architecture, test-strategy
looks like: A business key or file fingerprint, a record of what has already been loaded, and a test that loads the same input twice.
How the gap shows
  • The architecture describes loads as appends with no key, fingerprint or load register.
  • The test strategy has no test that resends an input.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
provenstage 9: Production sign-off
Fail fast
Load the same file twice in a test environment; row counts and key totals in every layer must not change after the second load.
Related
DUP-02, DUP-03, REC-01
On DatabricksIngest files with Auto Loader or COPY INTO, which both remember files already loaded. Write to silver with MERGE on the business key, not append.
Check it: Re-drop the same file and rerun; DESCRIBE HISTORY on the target shows no rows inserted (operationMetrics numTargetRowsInserted = 0).
Trap: A file renamed or rewritten by the source counts as new to Auto Loader. Its checkpoint is the memory: deleting or moving it reloads everything.
DUP-02Rerunning a job changes the resulthighTechnical anchor
Question
If a job is run again for the same period, does it produce exactly the same result (is every step idempotent)?
Why it matters
Operations reran a failed nightly job, but its first half had already appended rows, so the fact table held two copies of the night's orders until month-end reconciliation found them.
Applies to
any · components: ingestion, transformation, modelling · patterns: any
Owner
Technical anchor · also Developers, Operations and support
Evidence
in: architecture, test-strategy, runbook
looks like: Each write described as merge on a key, or replace of a whole partition or period, never a blind append; a rerun test per pipeline; the runbook says reruns are safe.
How the gap shows
  • Writes are described as "insert" or "append" with no key or partition overwrite.
  • The runbook warns operators not to rerun, or says to clean up by hand before rerunning.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
provenstage 9: Production sign-off
Fail fast
Run the pipeline twice in a row for the same day in test and compare a fingerprint of every output table (counts, sums of key measures); they must be identical.
Related
DUP-01, ORCH-03, TEST-05
On DatabricksUse MERGE on a key, or INSERT OVERWRITE / replaceWhere for a whole date range, for every write a rerun could repeat.
Check it: Run the job twice for the same period; compare counts and DESCRIBE HISTORY metrics between runs.
Trap: Repairing a failed Lakeflow Jobs run reruns failed tasks from the start; a task that appended before failing will append again.
DUP-03Duplicates already inside the sourcehighBusiness analyst
Question
Does each source have a reliable unique key, and what is the rule when the source itself sends two rows for the same thing?
Why it matters
The source table had no primary key and sent the same customer twice with different spellings; joins to it multiplied order rows and inflated sales by eight percent.
Applies to
any · components: ingestion, transformation · patterns: any
Owner
Business analyst · also Technical anchor, Source system owners
Evidence
in: discovery, data-model
looks like: A profile showing key uniqueness per source table, the chosen business key, and an agreed rule for duplicates (keep latest, keep all, reject), signed by the data owner.
How the gap shows
  • The data model lists tables without a key, or says "key to be confirmed".
  • No profiling results for key uniqueness are recorded.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 7: Test
Fail fast
Count rows per proposed key in a real extract of every source table; any key with more than one row needs a written rule before design continues.
Related
DQ-05, DES-04, SCD-03
On DatabricksProfile key uniqueness with a GROUP BY key HAVING count(*) > 1 query; deduplicate in silver with a window (row_number over key ordered by a change timestamp) before MERGE.
Check it: A uniqueness expectation or test on the silver key runs on every load.
Trap: MERGE fails when several source rows match one target row; deduplicate the source batch first. PRIMARY KEY constraints in Unity Catalog are informational and are not enforced.
DUP-04Replayed events in a streamhighTechnical anchor
Question
When the stream redelivers events after a restart or failure (at-least-once delivery), how does the target avoid counting them twice?
Why it matters
A consumer restarted from an older position and replayed two hours of events; the counts sink appended them again and the real-time dashboard overstated traffic for the day.
Applies to
any · components: ingestion, transformation · patterns: streaming, cdc
Owner
Technical anchor · also Developers
Evidence
in: architecture, test-strategy
looks like: An event id or key plus sequence used to deduplicate, a stated delivery guarantee end to end, and an idempotent or transactional write to the sink.
How the gap shows
  • The design assumes "exactly once" without saying which component provides it.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
provenstage 9: Production sign-off
Fail fast
In test, stop the stream mid-flow, rewind its position by a known number of events, restart, and confirm output counts match the input exactly.
Related
DUP-02, ORCH-06, LATE-03
On DatabricksIn Structured Streaming use dropDuplicatesWithinWatermark on an event id, and idempotent writes in foreachBatch (MERGE, or txnAppId plus txnVersion options).
Check it: Restart a stream from an older checkpoint copy in test and compare counts.
Trap: dropDuplicates without a watermark keeps state forever and grows until the job slows or fails.

Initial load, history, backfill (4)

HIST-01History load scope and approachhighTechnical anchor
Question
How much history will be loaded, from where, and by what approach (one go, in phases, or from archive), and who checks the old numbers?
Why it matters
The plan assumed history would come from the same feed as daily loads, but the source only kept ninety days; five years had to come from an archive in another format, adding six weeks.
Applies to
any · components: ingestion · patterns: any
Owner
Technical anchor · also Business owner and sponsor, Source system owners
Evidence
in: discovery, architecture, deployment
looks like: The number of years in scope, where each year comes from, the load approach and order, the expected volume, and who signs off that historical numbers are right.
How the gap shows
  • History is not mentioned, or "history to be loaded" appears with no years or source.
  • The plan has no task or time for the history load.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
provenstage 9: Production sign-off
Fail fast
Ask the source owner how far back the source really holds data, and pull one month of the oldest year in scope through the pipeline.
Related
HIST-02, HIST-03, REC-02
On DatabricksLoad history into the same bronze tables as daily loads, by date chunk, from archive files or a source extract; record the chunks in a control table.
Check it: Reconcile each historical year against the source or finance before switching on daily loads.
HIST-02Source can bear the history extractmediumTechnical anchor
Question
Can the source system take the load of the history extract, and is the extract split into chunks that can each be retried?
Why it matters
A single five-year query against the source database ran for nine hours, hit an undo limit and failed at ninety percent; it failed again the next night and slowed the source for its users.
Applies to
any · components: ingestion · patterns: database, cdc, api, history load
Owner
Technical anchor · also Source system owners
Evidence
in: architecture, deployment, raid
looks like: An extract split by date or key range, an agreed extract window with the source owner, and an estimate of run time per chunk from a real trial.
How the gap shows
  • The history load is one step with no chunking or restart point.
  • Source owners are not listed as reviewers of the extract approach.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
Fail fast
Time the extract of one realistic chunk against the real source in its normal busy window and multiply out; agree the result with the source owner.
Related
HIST-01, PERF-01
On DatabricksFor JDBC reads, set partitionColumn, lowerBound, upperBound and numPartitions and keep parallel connections within what the source owner agreed; or land extracts as files instead.
Check it: Time one chunk against the source in its busy window.
Trap: Too many partitions in a JDBC read opens that many connections at once against the source.
HIST-03Handing over from history load to incremental loadshighTechnical anchor
Question
How does the switch from the history or full load to incremental loads avoid both a gap and an overlap in the data?
Why it matters
The full snapshot finished at 10:00 but incremental capture was started at 11:00; an hour of changes was never loaded and the totals were quietly short for months.
Applies to
any · components: ingestion · patterns: cdc, database, streaming, history load
Owner
Technical anchor · also Developers
Evidence
in: architecture, deployment
looks like: A defined high-water mark or change position taken before the snapshot starts, incremental capture starting from it, and overlapping rows absorbed by a merge on key.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
provenstage 9: Production sign-off
Fail fast
In test, change rows in the source while the full load runs, then start incremental; every changed row must appear once with its latest value.
Related
HIST-01, DUP-01, REC-01
On DatabricksWith Lakeflow pipelines, AUTO CDC with SEQUENCE BY absorbs overlap between the snapshot and the change feed; otherwise MERGE with a sequence column so older changes never overwrite newer.
Check it: Change rows during the snapshot in test and check each ends with its latest value once.
HIST-04Reloading or backfilling a past periodmediumTechnical anchor
Question
Can a single past period (a day, a month) be reloaded or backfilled on demand without touching other periods or rebuilding everything?
Why it matters
A source bug corrupted three days in March; the only way to fix them was a full rebuild that took the platform down for a weekend.
Applies to
any · components: ingestion, transformation, modelling · patterns: batch, cdc, file, database, api
Owner
Technical anchor · also Operations and support, Developers
Evidence
in: architecture, runbook
looks like: Pipelines parameterised by period, writes that replace exactly that period, a runbook step for reloading a period, and a note on which downstream tables must also be rebuilt.
How the gap shows
  • The runbook has no reload or backfill procedure.
  • Jobs read "today" from the clock instead of taking the period as a parameter.
Due by
planstage 4: Discovery sign-off
designedstage 5: MVP starts
evidencedstage 7: Test
provenstage 10: Project sign-off
Fail fast
In test, reload one day from three days ago with a parameter and check that only that day's rows and its downstream aggregates changed.
Related
DUP-02, LATE-01, HAND-01
On DatabricksPass the period as a job parameter and write with replaceWhere on the date column; keep a separate backfill job or bundle target for large reloads.
Check it: Reload one past day in test; DESCRIBE HISTORY shows only that day's files were rewritten.
Trap: A full refresh of a Lakeflow pipeline streaming table re-reads all source data and clears the table first; check source retention before pressing it.

Late arriving and corrected data (4)

LATE-01Late and corrected rows rebuild the right periodshighTechnical anchor
Question
When rows arrive late or yesterday's figures are corrected next week, which periods are rebuilt, and how far back can corrections reach?
Why it matters
Returns were booked against sales from three weeks earlier, but the pipeline only rebuilt the last two days, so monthly net revenue never matched the finance ledger.
Applies to
any · components: ingestion, transformation, modelling · patterns: any
Owner
Technical anchor · also Business analyst, Source system owners
Evidence
in: discovery, architecture
looks like: A profile of how late data really arrives, an agreed correction window, and a design that reprocesses affected periods by business date rather than load date.
How the gap shows
  • Aggregates are described as built "for the load date" only.
  • Nothing in the discovery document says how late source data can arrive.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
provenstage 9: Production sign-off
Fail fast
In a real extract, compare business date against load date and find the longest lag; then insert a row two weeks late in test and check the monthly total updates.
Related
LATE-02, HIST-04, REC-01
On DatabricksRebuild aggregates for affected business dates with MERGE or replaceWhere driven by the dates present in the incoming batch; Change Data Feed on silver can tell gold which keys changed.
Check it: Insert a row with an old business date in test and check the gold total for that date changes.
LATE-02Facts arriving before their reference datahighTechnical anchor
Question
What happens to a fact row whose customer, product or other reference record has not arrived yet?
Why it matters
Orders for new products arrived hours before the product feed; an inner join silently dropped them, and the new-launch report showed zero sales on launch day.
Applies to
any · components: transformation, modelling · patterns: any
Owner
Technical anchor · also Developers
Evidence
in: architecture, data-model
looks like: An "unknown" or placeholder member, or a quarantine with automatic reprocessing, and a step that re-points facts once the reference record arrives.
How the gap shows
  • Joins between facts and reference tables are described as inner joins with no handling of misses.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
Fail fast
Load a fact row whose reference key does not exist yet, then load the reference row; the fact must be kept and end up attached to the right reference record.
Related
LATE-01, DQ-02, REC-01
On DatabricksInsert an unknown-member row in each dimension and left join facts to it; or quarantine misses to a table and reprocess it in a scheduled task.
Check it: An expectation or test counts facts pointing at the unknown member over time.
LATE-03Event time, watermarks and late events in streamsmediumTechnical anchor
Question
Do streaming aggregations use event time, how long do they wait for late events (the watermark), and what happens to events later than that?
Why it matters
The window waited ten minutes, but mobile clients uploaded events hours later after going offline; those events were silently discarded and daily active users were understated.
Applies to
any · components: transformation, modelling · patterns: streaming
Owner
Technical anchor · also Developers, Data consumers
Evidence
in: architecture, requirements
looks like: The lateness of real events measured, a watermark chosen from that, and a stated home for events beyond it (a correction batch or a side table) agreed with consumers.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
provenstage 9: Production sign-off
Fail fast
Measure the gap between event time and arrival time in a day of real events and see what share falls beyond the proposed watermark.
Related
LATE-01, DUP-04, PERF-02
On DatabricksSet withWatermark on the event-time column from measured lateness; route later events to a side table.
Check it: Check the streaming query progress metrics for numRowsDroppedByWatermark.
Trap: A watermark that is too long holds more state and costs memory; too short silently drops events.
LATE-04Time zones and business datesmediumBusiness analyst
Question
Which time zone does each source use, how are timestamps stored, and which time zone defines a business day?
Why it matters
One source sent local time and another sent universal time; orders placed late in the evening fell into the next day and daily sales never matched the stores' own reports.
Applies to
any · components: ingestion, transformation, modelling · patterns: any
Owner
Business analyst · also Technical anchor
Evidence
in: data-model, discovery
looks like: The time zone of every source timestamp, a single storage convention, a defined business-day time zone, and a rule for daylight-saving changes.
How the gap shows
  • The data dictionary lists timestamps without a time zone.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
Fail fast
Pick ten real transactions near midnight and check that their business date matches what the source application shows.
Related
DES-04, REC-02
On DatabricksStore timestamps as TIMESTAMP (universal time) and convert with from_utc_timestamp for the business-day time zone; set the session time zone deliberately rather than relying on defaults.
Check it: Compare business dates of near-midnight rows with the source application.
Trap: TIMESTAMP_NTZ has no time zone; mixing it with TIMESTAMP across sources shifts hours.

Deletes and soft deletes at the source (3)

DEL-01Detecting hard deletes at the sourcehighTechnical anchor
Question
When a record is physically deleted in the source, how does the platform find out?
Why it matters
Cancelled orders were deleted in the source, but the incremental extract only picked up new and changed rows, so the cancelled orders stayed in the warehouse and inflated sales.
Applies to
any · components: ingestion · patterns: batch, database, api, file, cdc
Owner
Technical anchor · also Source system owners
Evidence
in: discovery, architecture
looks like: For each source, whether deletes happen and how they are captured: change capture with delete events, a periodic full key comparison, or a delete feed from the source.
How the gap shows
  • Incremental loads use a modified-date filter and deletes are not mentioned.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
Fail fast
Ask the source owner whether rows are ever deleted, then delete a test row in a source test system and run the load; the platform must reflect it.
Related
DEL-02, DEL-03, REC-01
On DatabricksWith CDC feeds use AUTO CDC ... APPLY AS DELETE WHEN, or MERGE ... WHEN MATCHED AND op = 'D' THEN DELETE; without CDC, compare full key snapshots (AUTO CDC FROM SNAPSHOT treats missing keys as deleted) or use MERGE ... WHEN NOT MATCHED BY SOURCE.
Check it: Delete a row in a source test system and check it disappears or is flagged after the load.
Trap: WHEN NOT MATCHED BY SOURCE against a partial extract deletes everything not in that extract.
DEL-02What a delete means downstreammediumBusiness analyst
Question
When a source deletes or soft-deletes a record, must it disappear, be flagged, or stay in each layer and report, and who decided?
Why it matters
Engineers removed deleted customers from every table, but finance needed them kept for historic invoices, and last year's revenue dropped after a routine load.
Applies to
any · components: transformation, modelling, serving · patterns: any
Owner
Business analyst · also Technical anchor, Data consumers
Evidence
in: requirements, data-model
looks like: A rule per entity (remove, flag with a deleted date, or keep) agreed with the data owner, and how flagged rows are filtered in reports.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
Fail fast
Write the delete rule for the top three entities and get the data owner to sign them in one meeting.
Related
DEL-01, SCD-01, SEC-03
DEL-03Deletes flow through derived tablesmediumTechnical anchor
Question
When a source row is deleted, are the aggregates, summaries and extracts built from it recalculated too?
Why it matters
The detail table dropped a deleted order, but the daily summary built from it was only ever appended to, so detail and summary disagreed and users stopped trusting both.
Applies to
any · components: transformation, modelling, serving · patterns: any
Owner
Technical anchor · also Developers
Evidence
in: architecture, test-strategy
looks like: Derived tables rebuilt for the affected periods when deletes arrive, and a test that checks detail and summary still agree after a delete.
Due by
planstage 4: Discovery sign-off
designedstage 5: MVP starts
evidencedstage 7: Test
Fail fast
Delete one source row in test, run the full chain, and compare the summary total against the sum of the detail.
Related
DEL-01, LATE-01, REC-01
On DatabricksEnable Change Data Feed on silver so downstream jobs see deletes and rebuild affected aggregates.
Check it: After a test delete, gold totals equal the sum of silver detail.

Keeping history of changes (slowly changing dimensions) (4)

SCD-01Which tables keep history, and for how longmediumBusiness analyst
Question
For each table, has the business decided whether to keep the history of changes or only the latest value, and for how long?
Why it matters
Customer region was stored as latest only; when a sales team was reorganised, last year's sales by region were silently restated and the bonus calculation was disputed.
Applies to
any · components: modelling, transformation · patterns: any
Owner
Business analyst · also Technical anchor, Data consumers
Evidence
in: requirements, data-model, decisions
looks like: A list of entities and attributes with "history kept" or "latest only", a retention period, and the business owner who agreed it.
How the gap shows
  • The data model has no statement about history for dimensions.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 7: Test
Fail fast
Ask the main report owner "should last year's figures change when a customer moves region?" and record the answer as a decision.
Related
SCD-02, DEL-02
SCD-02History rows stay correct on reruns and out-of-order changeshighDevelopers
Question
Does the history logic (effective from and to dates, current flag) stay correct when a load is rerun, when two changes arrive in one batch, or when changes arrive out of order?
Why it matters
A rerun closed and reopened every current row, and an older change arriving late became "current", so customers showed last year's address with overlapping validity periods.
Applies to
any · components: modelling, transformation · patterns: any
Owner
Developers · also Technical anchor, Test lead
Evidence
in: architecture, test-strategy
looks like: Changes ordered by a source sequence or change timestamp, not arrival time; a hash to skip unchanged rows; and tests for rerun, two-changes-per-batch and out-of-order cases.
Due by
planstage 4: Discovery sign-off
designedstage 5: MVP starts
evidencedstage 7: Test
provenstage 9: Production sign-off
Fail fast
Feed three versions of one record in reverse order, then rerun the same batch; exactly one current row and no overlapping date ranges may remain.
Related
SCD-01, DUP-02, SCD-04
On DatabricksPrefer AUTO CDC ... STORED AS SCD TYPE 2 with SEQUENCE BY on a source sequence; it places out-of-order changes in the right history position. Hand-written MERGE needs a hash of tracked columns and ordering by sequence.
Check it: Feed reversed versions and a rerun in test; check no overlapping __START_AT / __END_AT ranges.
Trap: Sequencing by ingestion time instead of a source sequence breaks history when changes arrive out of order.
SCD-03A business key that can changemediumTechnical anchor
Question
Can a source's business key ever change (renumbering, merges, re-registration), and how is the identity of the thing kept across that change?
Why it matters
The source renumbered accounts during a migration; each renumbered account became a new customer, splitting its history and doubling the customer count.
Applies to
any · components: modelling, transformation · patterns: any
Owner
Technical anchor · also Source system owners, Business analyst
Evidence
in: data-model, discovery
looks like: Surrogate keys owned by the platform, a mapping from old to new source keys, and the source owner's confirmation of whether keys can change.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
Fail fast
Ask each source owner directly whether keys are ever reused, merged or renumbered, and check a real extract for one key mapping to two names over time.
Related
DUP-03, DES-04
On DatabricksGenerate surrogate keys with an IDENTITY column on the dimension; keep a source-key mapping table.
Trap: IDENTITY values are unique but not consecutive, and declaring one limits concurrent writes to the table.
SCD-04Facts joined to the right version of historymediumTechnical anchor
Question
When facts are joined to a dimension that keeps history, are they matched to the version valid at the time of the fact, and is that the rule consumers expect?
Why it matters
Facts were joined to the current customer row; reports that should show sales by the region at the time showed today's region instead, and nobody had written down which was wanted.
Applies to
any · components: modelling, serving · patterns: any
Owner
Technical anchor · also Data consumers
Evidence
in: data-model, requirements
looks like: For each report, whether it shows "as it was" or "as it is now", and joins that use the fact date within the validity range or the surrogate key captured at load.
Due by
planstage 4: Discovery sign-off
designedstage 5: MVP starts
evidencedstage 7: Test
Fail fast
Take one customer who changed region last year and check a sales report shows each sale under the region the business expects.
Related
SCD-02, SERV-04

Columns added, renamed or retyped at the source (3)

SCH-01A new column appears at the sourcemediumTechnical anchor
Question
When a source adds a column, does the pipeline fail, drop it silently, or carry it through, and is that the agreed behaviour per layer?
Why it matters
The source added a discount column; the raw layer dropped it silently, and when finance asked for it three months later the history could not be recovered.
Applies to
any · components: ingestion, transformation · patterns: any
Owner
Technical anchor · also Developers
Evidence
in: architecture
looks like: A schema policy per layer (for example: raw keeps everything including unknown columns, cleaned layers change only through a reviewed change), and an alert when a new column appears.
How the gap shows
  • The architecture does not mention schema changes at all.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
Fail fast
Add a column to a test source file or table and run the pipeline; see what each layer does and whether anyone is told.
Related
SCH-02, SCH-03
On DatabricksChoose an Auto Loader schemaEvolutionMode per source: addNewColumns (default when no schema is given; stream stops once, then continues with the new column), rescue (new columns go to _rescued_data), failOnNewColumns, or none. In silver, use MERGE WITH SCHEMA EVOLUTION only where agreed.
Check it: Drop a test file with an extra column and watch what the stream and the target do.
Trap: With addNewColumns the stream fails on purpose and must be restarted; make sure the job has retries, or it stays down overnight.
SCH-02A column renamed, removed or retypedhighTechnical anchor
Question
When a source renames, removes or changes the type of a column, does the pipeline stop loudly before writing wrong data?
Why it matters
A source changed an amount column from cents to a decimal in dollars; the pipeline cast it happily and revenue fell a hundredfold in the morning report.
Applies to
any · components: ingestion, transformation · patterns: any
Owner
Technical anchor · also Developers, Operations and support
Evidence
in: architecture, test-strategy
looks like: An expected schema checked on every load, breaking changes failing the load with a clear alert, and range or distribution checks on key measures.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
provenstage 9: Production sign-off
Fail fast
In test, rename one column and change one type in a source sample; the load must fail with a message that names the column.
Related
SCH-01, DQ-01, OPS-01
On DatabricksSupply an explicit schema or schema hints for critical columns; let Delta schema enforcement reject mismatched writes; add CHECK constraints for ranges.
Check it: A test file with a renamed or retyped column fails the load with a clear error.
Trap: Type mismatches can land in _rescued_data quietly; alert when that column is not null.
SCH-03Source owners warn before they changemediumTechnical anchor
Question
Is there an agreement (a data contract or a change notice) that source owners tell the data team before changing structure or meaning?
Why it matters
The source team released a new version on a Friday with restructured tables; nobody told the data team and the weekend loads failed, missing a regulatory deadline.
Applies to
any · components: ingestion · patterns: any
Owner
Technical anchor · also Source system owners, Project manager
Evidence
in: discovery, stakeholders, raid
looks like: A written agreement or contract per source naming the schema, the notice period for changes, and a contact; source release calendars known to the team.
How the gap shows
  • Source owners are not in the stakeholder map.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 9: Production sign-off
Fail fast
Ask each source owner when their next release is and whether it changes anything the project reads.
Related
SCH-02, SERV-03

Rules, bad rows, quarantine (5)

DQ-01Data quality rules written with ownershighBusiness analyst
Question
Which data quality rules apply to each important table, who owns each rule, and are they written down as checks that run on every load?
Why it matters
Everyone assumed someone else checked that order totals were positive; a source bug sent negative totals for a week and nobody's check caught it.
Applies to
any · components: ingestion, transformation, modelling · patterns: any
Owner
Business analyst · also Technical anchor, Test lead
Evidence
in: requirements, data-model, test-strategy
looks like: A rule list per table (not null, unique, valid values, ranges, referential integrity, business rules), each with a business owner and a severity, implemented as automated checks.
How the gap shows
  • The documents say "data quality checks will be applied" with no list of rules.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
provenstage 9: Production sign-off
Fail fast
Ask the business owner for the five things that would make them distrust the main report, and write each as a rule that runs on today's data.
Related
DQ-02, DQ-05, TEST-01
On DatabricksIn Lakeflow pipelines write rules as expectations (EXPECT ... ON VIOLATION DROP ROW / FAIL UPDATE); elsewhere use CHECK and NOT NULL constraints plus a test framework in jobs.
Check it: Query the pipeline event log or Data quality tab for pass and fail counts per rule.
DQ-02What happens to a row that fails a rulehighTechnical anchor
Question
When a row fails a rule, is the batch rejected, the row quarantined, or the row loaded and flagged, and who is told and fixes it?
Why it matters
Bad rows were filtered out silently; three thousand orders a day disappeared into a table no one looked at, and sales were understated for a quarter.
Applies to
any · components: ingestion, transformation · patterns: any
Owner
Technical anchor · also Business analyst, Operations and support
Evidence
in: architecture, runbook, requirements
looks like: An action per rule severity, a quarantine table with reasons, a count of quarantined rows reported to an owner, and a way to reprocess rows once fixed.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
provenstage 9: Production sign-off
Fail fast
Feed a sample with known bad rows in test and check each lands where the design says, with a reason, and that an owner is notified.
Related
DQ-01, DQ-03, OPS-01
On DatabricksUse warn or drop expectations, and write failing rows to a quarantine table with an inverted expectation so they are kept with a reason, not just counted.
Check it: Load known-bad rows in test and find each one in quarantine.
Trap: expect_or_drop only counts dropped rows; the rows themselves are gone unless you quarantine them.
DQ-03One poison record does not stop the whole loadmediumDevelopers
Question
Can a single malformed record (broken encoding, unparseable value, oversized field) stop the whole pipeline, and how is it isolated?
Why it matters
One record with a malformed date crashed the stream parser on every retry; the pipeline was stuck for six hours while all later data waited behind it.
Applies to
any · components: ingestion · patterns: any
Owner
Developers · also Technical anchor
Evidence
in: architecture, test-strategy
looks like: Parsing that captures unreadable records into a side table with the raw content and the error, and a test with deliberately malformed input.
Due by
planstage 4: Discovery sign-off
designedstage 5: MVP starts
evidencedstage 7: Test
Fail fast
Put one malformed record in the middle of a test input; the rest must load and the bad record must be kept somewhere readable.
Related
DQ-02, ORCH-02
On DatabricksRead files with a rescued data column (Auto Loader) or badRecordsPath, so malformed records are kept aside instead of failing the stream.
Check it: Put a malformed record in a test file; the rest loads and the bad record is readable.
DQ-04A source goes quiet or sends far too littlehighOperations and support
Question
Does anything detect a source that sends no data, an empty file, or a much smaller or larger volume than usual?
Why it matters
The source sent an empty file after a failed export; the load succeeded, every job was green, and the business saw a day with zero sales before any engineer did.
Applies to
any · components: ingestion · patterns: any
Owner
Operations and support · also Technical anchor
Evidence
in: architecture, runbook
looks like: Freshness and volume checks per source with expected ranges, which fail or alert rather than load quietly.
How the gap shows
  • Monitoring is described only as "job failure alerts".
Due by
planstage 4: Discovery sign-off
designedstage 5: MVP starts
evidencedstage 7: Test
provenstage 9: Production sign-off
Fail fast
Run a load with an empty input in test; it must not complete as a green success without someone being told.
Related
OPS-01, DQ-01
On DatabricksEnable data quality monitoring on the schema for freshness and completeness, and add a row-count check task that fails on empty or abnormal batches; alert with Databricks SQL alerts.
Check it: Load an empty file in test and confirm an alert fires.
Trap: Monitoring results arrive after the load; a gate inside the job is still needed to stop bad data reaching gold.
DQ-05Real data profiled before designhighTechnical anchor
Question
Has real data from every source been profiled (nulls, duplicates, broken keys, odd values, volumes), and was the profile shown to the business owner?
Why it matters
The design assumed every order had a customer; profiling in build week three found twenty percent did not, and the model and estimate both had to be redone.
Applies to
any · components: ingestion, transformation, modelling · patterns: any
Owner
Technical anchor · also Business analyst, Business owner and sponsor
Evidence
in: discovery
looks like: A profile per source table from a real extract (not a sample built by hand), with findings and what was decided about each.
How the gap shows
  • Discovery describes sources from documentation only, with no profiling results.
Due by
planstage 2: Rough estimate
designedstage 3: Discovery
evidencedstage 4: Discovery sign-off
Fail fast
Pull a real extract of the biggest source table this week and count nulls, duplicates and orphan keys in the columns the design relies on.
Related
DUP-03, DQ-01, DES-01
On DatabricksProfile real extracts with SQL in a notebook or with data profiling on the bronze tables.

Proving numbers match from source to report (4)

REC-01Counts and totals reconciled layer by layerhighTechnical anchor
Question
Are row counts and key totals compared automatically between the source and each layer on every load, with the differences explained?
Why it matters
The report was four percent short of the source and nobody could say in which layer rows had been lost, so a week went into tracing one join that dropped orphan rows.
Applies to
any · components: ingestion, transformation, modelling · patterns: any
Owner
Technical anchor · also Test lead, Developers
Evidence
in: test-strategy, architecture
looks like: Counts and sums of key measures captured at the source and after each layer for every run, stored, compared, and every expected difference (filters, quarantined rows) accounted for.
How the gap shows
  • Reconciliation appears only as a one-off test activity, not as part of every run.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
provenstage 9: Production sign-off
Fail fast
For one day of real data, write down the source count and total, then the count and total in each layer, and explain every difference.
Related
REC-02, REC-03, DUP-01
On DatabricksWrite per-run counts and sums per layer to a reconciliation table from each task; DESCRIBE HISTORY operationMetrics give rows written per commit.
Check it: A dashboard or query compares source and each layer per run.
REC-02Matched against a number the business already trustshighTest lead
Question
Are the final numbers checked against an independent figure the business already trusts (a finance ledger, an existing report, a source application screen)?
Why it matters
Source-to-target counts all matched, yet the new revenue figure disagreed with the finance ledger because the definition of revenue differed; it was found only at user acceptance.
Applies to
any · components: modelling, serving · patterns: any
Owner
Test lead · also Business owner and sponsor, Data consumers
Evidence
in: test-strategy, requirements
looks like: A named trusted figure for each key measure, the comparison method, and an agreed explanation of known differences.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
provenstage 9: Production sign-off
Fail fast
Ask the business owner which existing number they would compare against, and compare one month of thin-slice output with it now.
Related
REC-01, SERV-04, TEST-04
REC-03Tolerances and who signs off differencesmediumBusiness owner and sponsor
Question
What size of difference is acceptable for each measure, and who decides when a difference is explained well enough to go live?
Why it matters
Test found a 0.3 percent gap; engineering called it rounding, finance called it a blocker, and go-live slipped two weeks while they argued.
Applies to
any · components: modelling, serving · patterns: any
Owner
Business owner and sponsor · also Test lead, Technical anchor
Evidence
in: test-strategy, decisions
looks like: A tolerance per measure and a named person who accepts reconciliation results.
Due by
planstage 4: Discovery sign-off
designedstage 5: MVP starts
evidencedstage 7: Test
Fail fast
Put the tolerance question on the next sign-off agenda and record a number for the top three measures.
Related
REC-01, REC-02
REC-04First production loads reconciledhighTechnical anchor
Question
Were the first production loads, including the history load, reconciled against the source and trusted figures before users were told to rely on them?
Why it matters
Production used a different source instance with extra test records; nobody reconciled the first loads, and dummy orders appeared in the board pack.
Applies to
any · components: ingestion, modelling · patterns: any
Owner
Technical anchor · also Operations and support, Business owner and sponsor
Evidence
in: deployment, status
looks like: A reconciliation result for each of the first production runs, with sign-off, recorded in the hypercare or go-live report.
Due by
planstage 6: Build
designedstage 8: Deploy
evidencedstage 9: Production sign-off
provenstage 9: Production sign-off
Fail fast
Add "reconcile first production load" as a go-live checklist item with an owner and a tolerance.
Related
REC-01, HIST-01, OPS-05

Volume, speed, freshness (5)

PERF-01Volumes and growth statedmediumTechnical anchor
Question
What are the real volumes per source today (rows, size, files per day), and how much are they expected to grow over the next two to three years?
Why it matters
The design was sized from a test sample of a million rows; production sent eight hundred million on month-end and the nightly window was blown on the first close.
Applies to
any · components: any · patterns: any
Owner
Technical anchor · also Source system owners, Architect
Evidence
in: discovery, architecture
looks like: Measured volumes per source from real data, peak days (month-end, seasonal), and a growth assumption.
How the gap shows
  • Volumes are missing, or given as "large" or "medium" rather than numbers.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 7: Test
Fail fast
Ask each source owner for the row count of their biggest table on their busiest day, and check one against a real extract.
Related
PERF-03, COST-01, HIST-02
PERF-02Freshness target measured end to endhighTechnical anchor
Question
What freshness or latency is promised (for example, ready by 07:00, or within one minute), and has the full path from source to consumer been timed against it?
Why it matters
Each step met its own target, but source export, transfer, three layers and the report refresh added up to ninety minutes against a promised thirty.
Applies to
any · components: any · patterns: any
Owner
Technical anchor · also Data consumers, Architect
Evidence
in: requirements, architecture, test-strategy
looks like: A freshness target per consumer, a time budget split across every hop, and a measured end to end time from the thin slice or a test run.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 7: Test
provenstage 9: Production sign-off
Fail fast
Time one record from source change to visible in the report in the thin slice, and compare with the target.
Related
PERF-03, SERV-02, DES-06
PERF-03Tested at production volumehighTest lead
Question
Has the pipeline been run at production volume, including a peak day and the history load, before go-live?
Why it matters
Every test ran on a tenth of the data; the first month-end in production ran out of memory in the biggest join and the finance close waited a day.
Applies to
any · components: ingestion, transformation, modelling · patterns: any
Owner
Test lead · also Technical anchor, Platform team
Evidence
in: test-strategy
looks like: A performance test at peak production volume with run times, compute size and cost recorded, and the result compared with the freshness target.
How the gap shows
  • The test strategy has no performance or volume test, or uses only small samples.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
provenstage 9: Production sign-off
Fail fast
Run the heaviest transformation on one full peak day of real (or realistically generated) data and time it.
Related
PERF-01, PERF-04, COST-01
On DatabricksRun on a production-sized copy (a deep clone in a test catalog, or generated data) on the production compute configuration; read the Spark UI or query profile for spill and skew.
Check it: Record run time, compute size and cost from system.lakeflow.job_run_timeline and system.billing.usage.
PERF-04Skewed keys and very large joinsmediumDevelopers
Question
Are there hot keys (one customer, one default value, nulls) or joins between two very large tables that will make some steps run far slower than the rest?
Why it matters
Half of all rows carried a placeholder customer id; in the big join one worker processed that key for an hour while the others sat idle, and the job missed its window.
Applies to
any · components: transformation, modelling · patterns: any
Owner
Developers · also Technical anchor
Evidence
in: architecture, discovery
looks like: Key distributions profiled for the large join keys, and a strategy for skewed keys and large joins (filtering defaults, splitting hot keys, pre-aggregating).
Due by
planstage 4: Discovery sign-off
designedstage 5: MVP starts
evidencedstage 7: Test
Fail fast
Count rows per join key for the two largest joins and look at the top ten keys' share.
Related
PERF-03, DQ-05
On DatabricksRely on adaptive query execution for skewed joins, broadcast small tables, filter null and placeholder keys before joins, and salt only when AQE is not enough.
Check it: In the Spark UI, compare the longest task against the median for the big join stage.
PERF-05Storage layout and file sizes kept healthymediumDevelopers
Question
How are table layout (partitioning or clustering) and file sizes chosen and maintained, so frequent small writes do not slow reads over time?
Why it matters
A streaming load wrote thousands of tiny files an hour; reads slowed every week until a simple query took ten minutes and cost ten times more.
Applies to
any · components: ingestion, transformation, modelling · patterns: any
Owner
Developers · also Platform team
Evidence
in: architecture, runbook
looks like: Layout chosen from the main filter and join columns, a scheduled compaction and cleanup job, and a metric for average file size.
Due by
planstage 4: Discovery sign-off
designedstage 5: MVP starts
evidencedstage 7: Test
provenstage 10: Project sign-off
Fail fast
After a week of test loads, count files and average file size for the busiest table.
Related
COST-04, PERF-03, SEC-03
On DatabricksUse liquid clustering on the main filter or merge columns for new tables; let predictive optimization run OPTIMIZE and VACUUM on Unity Catalog managed tables.
Check it: DESCRIBE DETAIL shows numFiles and sizeInBytes; DESCRIBE HISTORY shows OPTIMIZE runs.
Trap: Predictive optimization covers managed tables only; external tables need scheduled OPTIMIZE and VACUUM. Check that clustering keys match the MERGE condition, or merges scan the whole table.

Running cost and its controls (4)

COST-01Running cost estimated from a real runmediumTechnical anchor
Question
What will the solution cost to run each month (compute, storage, transfer, licences), and is the figure based on a measured run rather than a guess?
Why it matters
The business case assumed a few hundred a month; the first production bill was eleven thousand because the streaming jobs ran around the clock, and the project was paused for review.
Applies to
any · components: any · patterns: any
Owner
Technical anchor · also Platform team, Business owner and sponsor
Evidence
in: discovery, architecture, estimate
looks like: A monthly running cost estimate broken down by job and storage, derived from the thin slice or a test run scaled to production volume, agreed with the budget owner.
How the gap shows
  • Cost appears only as build effort, with no running cost.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 7: Test
provenstage 9: Production sign-off
Fail fast
Read the actual cost of the thin slice's runs and multiply by production volume and schedule.
Related
COST-02, PERF-01, DES-06
On DatabricksRead the thin slice's cost from system.billing.usage joined to system.billing.list_prices, by job.
Check it: Compare the estimate with the first month of production usage from the same tables.
Trap: List price is not your contract price; agree discounts with whoever holds the contract.
COST-02Platform cost guardrailshighPlatform team
Question
What stops a single team, job or user from running up an unexpected bill (budgets, alerts, limits on compute size, automatic shutdown)?
Why it matters
A developer left a large compute cluster running over a holiday weekend, and the month's budget was gone by the fourth.
Applies to
new platform · components: platform · patterns: any
Owner
Platform team · also Security and governance, Business owner and sponsor
Evidence
in: platform, runbook
looks like: Budgets with alerts per team or project, policies that cap compute size and enforce automatic shutdown, and a named person who receives cost alerts.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 6: Build
provenstage 9: Production sign-off
Fail fast
Try to create the largest compute allowed as an ordinary developer in the development environment; the policy should stop you.
Related
COST-03, COST-04, ENV-02
On DatabricksUse compute policies to cap node types and worker counts and to force auto-termination and tags; set account budgets with alerts per workspace or tag.
Check it: Try to create oversized compute as a developer in development; the policy should block it.
Trap: Budgets alert, they do not cap general spend; serverless usage policies only tag usage for attribution. Real limits come from policies and permissions.
COST-03Cost can be traced to a team and a joblowPlatform team
Question
Can the running cost be broken down by project, team, environment and job (by tags or separate accounts), so overruns can be traced?
Why it matters
The platform bill doubled in one month and nobody could say which project caused it, so every team was asked to cut costs instead of the one that needed to.
Applies to
any · components: platform, orchestration · patterns: any
Owner
Platform team · also Technical anchor
Evidence
in: platform, architecture
looks like: A mandatory tagging standard (project, owner, environment), enforced on compute and storage, and a cost report by tag.
Due by
planstage 4: Discovery sign-off
designedstage 5: MVP starts
evidencedstage 7: Test
provenstage 9: Production sign-off
Fail fast
Pull last week's cost broken down by project tag; check how much is untagged.
Related
COST-02, OPS-03
On DatabricksEnforce custom tags through compute policies for classic compute and serverless usage policies for serverless; report from system.billing.usage custom_tags.
Check it: Query the share of last week's usage with no project tag.
COST-04No compute running when nothing needs itmediumDevelopers
Question
Is compute sized from real runs and does it stop when idle, rather than staying on or oversized "to be safe"?
Why it matters
A job that needed ten minutes of compute held a large cluster for two hours each run because it waited on an upstream file, and nobody noticed for six months.
Applies to
any · components: ingestion, transformation, modelling, orchestration · patterns: any
Owner
Developers · also Platform team, Operations and support
Evidence
in: architecture, runbook
looks like: Compute sizes justified by measured runs, automatic stop or job-scoped compute, waits on upstream data done outside paid compute, and a periodic review of utilisation.
Due by
planstage 4: Discovery sign-off
designedstage 5: MVP starts
evidencedstage 7: Test
provenstage 9: Production sign-off
Fail fast
For the three most expensive jobs, compare the time compute was running against the time it was busy.
Related
COST-01, COST-02, PERF-05
On DatabricksUse job compute or serverless for scheduled work instead of all-purpose clusters; set auto-termination; wait for upstream data with file-arrival or table-update triggers, not sleeping tasks.
Check it: Compare job duration with compute uptime in system tables for the top jobs.
Trap: A continuous Lakeflow pipeline or stream runs compute all day; use triggered mode when freshness allows.

Access, masking, retention, regulations (5)

SEC-01Personal and sensitive data found and classifiedhighSecurity and governance
Question
Which columns hold personal or sensitive data, in which sources and layers, and is each classified?
Why it matters
Free-text notes held customer phone numbers and health details; nobody had classified them, so they were copied unmasked into a table hundreds of analysts could read.
Applies to
any · components: any · patterns: any
Owner
Security and governance · also Business analyst, Technical anchor
Evidence
in: data-model, discovery, platform
looks like: A classification for every column in scope (public, internal, confidential, personal, special category), including free-text fields, recorded in the data dictionary or catalog.
How the gap shows
  • Personal data is not mentioned, or mentioned only as "will follow policy".
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 7: Test
Fail fast
Scan a real extract of each source for patterns like emails, phone numbers and national ids, including in free-text columns.
Related
SEC-02, SEC-03, SEC-05
On DatabricksApply governed tags for classification on columns in Unity Catalog; data classification can suggest tags.
Check it: Query information_schema column tags to list sensitive columns and their owners.
SEC-02Who can see whathighSecurity and governance
Question
Who may see which tables, rows and columns, and how is that enforced (masking, row filters, separate views) rather than trusted?
Why it matters
Regional managers were meant to see only their own region, but the report filtered it in the front end; anyone querying the table directly saw every region's salaries.
Applies to
any · components: modelling, serving, governance · patterns: any
Owner
Security and governance · also Technical anchor, Data consumers
Evidence
in: platform, architecture, requirements
looks like: An access matrix by group, enforced in the data platform itself with masking and row-level rules, and tests that run as each persona.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
provenstage 9: Production sign-off
Fail fast
Log in as a test user of the lowest-privilege persona and query the most sensitive table directly.
Related
SEC-01, ENV-02, SEC-04
On DatabricksGrant to groups in Unity Catalog; apply row filters and column masks, or ABAC policies on governed tags at catalog or schema level for many tables.
Check it: Query the table as a test user from the lowest persona.
Trap: Row filters and masks need recent runtimes (older compute returns no data), cannot be put on views, and block time travel and clones on that table.
SEC-03Retention and the right to be forgottenhighSecurity and governance
Question
How long is each kind of data kept, and how is a person's data erased everywhere it lives, including history tables, old table versions, backups and extracts?
Why it matters
A customer's erasure request deleted their current row, but the history table, old file versions kept for recovery and a weekly extract to a partner still held their data.
Applies to
any · components: governance, modelling, serving, ingestion · patterns: any
Owner
Security and governance · also Technical anchor, Operations and support
Evidence
in: platform, architecture, runbook
looks like: A retention period per data class, an erasure procedure that covers every layer, history, versions and copies, and the time it takes to complete.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
provenstage 10: Project sign-off
Fail fast
Trace one test person through every table, version and extract in test and count where their data would remain after the erasure procedure.
Related
SEC-01, DEL-02, PERF-05
On DatabricksDELETE the person's rows in every layer, then VACUUM so old files are removed; deletion vectors need a REORG ... APPLY (PURGE) before VACUUM to physically remove rows.
Check it: After the procedure, time travel to an earlier version and confirm the data is gone once retention has passed.
Trap: Time travel, deep clones and exports keep copies until each is cleaned too.
SEC-04Access is auditedmediumSecurity and governance
Question
Is every read and change of sensitive data logged with who, what and when, and does someone review it?
Why it matters
After a data leak, the team could not say who had queried the customer table in the last month, and the regulator treated the whole table as exposed.
Applies to
new platform · components: platform, governance · patterns: any
Owner
Security and governance · also Platform team
Evidence
in: platform, runbook
looks like: Audit logs switched on for data access, kept for the regulated period, and a named reviewer with a review frequency.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
provenstage 9: Production sign-off
Fail fast
Query a sensitive test table, then find that query in the audit log within an hour.
Related
SEC-02, OPS-03
On DatabricksQuery system.access.audit for data access events; keep it for the regulated period.
Check it: Find a test query against a sensitive table in system.access.audit.
SEC-05Regulations and where data may livehighSecurity and governance
Question
Which regulations apply to the data (privacy, industry, residency), and does the design keep data, copies and processing in permitted locations?
Why it matters
A disaster-recovery copy was configured to another region outside the permitted jurisdiction; the audit found it a week before go-live and the platform design had to change.
Applies to
any · components: platform, governance · patterns: any
Owner
Security and governance · also Architect, Platform team
Evidence
in: platform, decisions
looks like: A list of applicable regulations, permitted regions for storage, processing and backups, and sign-off from the data protection or compliance owner.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 8: Deploy
Fail fast
Ask the compliance owner which locations are permitted, and list every place the design puts a copy of the data.
Related
SEC-01, ORCH-05

Environments, permissions, network (5)

ENV-01Separate environments with separate datahighPlatform team
Question
Are there separate development, test and production environments, isolated so work in one cannot read or overwrite another's data?
Why it matters
A developer ran a cleanup script pointed at what they thought was development; the paths were shared and production tables were wiped the night before a board meeting.
Applies to
new platform · components: platform · patterns: any
Owner
Platform team · also Architect, Security and governance
Evidence
in: platform
looks like: Named environments with separate storage, catalogs or schemas, identities and compute, and a rule that non-production identities cannot write to production.
How the gap shows
  • The platform document describes one workspace or one storage location for all stages.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 5: MVP starts
Fail fast
Using a development identity, try to write to a production table; it must be refused.
Related
ENV-02, REL-04, TEST-02
On DatabricksSeparate workspaces or at least separate catalogs per environment, bound to their workspaces with workspace-catalog binding, and separate storage locations.
Check it: A development service principal cannot write to the production catalog.
ENV-02Access through groups and service identitieshighPlatform team
Question
Is access granted to groups by role with least privilege, and do scheduled jobs run as service identities rather than as a person?
Why it matters
Production jobs ran under the lead developer's own account; when she left the company her account was disabled and every nightly job failed.
Applies to
any · components: platform, governance, orchestration · patterns: any
Owner
Platform team · also Security and governance, Technical anchor
Evidence
in: platform, architecture
looks like: A role-to-group access model, service identities for jobs in each environment, no direct user grants in production, and who approves access requests.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
provenstage 9: Production sign-off
Fail fast
List who owns and runs each production job; any job owned or run by a named person is a gap.
Related
ENV-01, SEC-02, REL-05
On DatabricksGrant privileges to account groups, make groups (not people) owners of catalogs, schemas and jobs, and run jobs as service principals.
Check it: Check job run_as and object owners; none should be a named person in production.
ENV-03Network path to every source proven earlyhighPlatform team
Question
Has a connection from the platform to every source and target been made in a real environment, through firewalls, private links and credentials?
Why it matters
The source sat in a data centre behind a firewall; the network change took nine weeks of tickets and MVP 1 waited for it with the team idle.
Applies to
any · components: ingestion, serving, platform · patterns: any
Owner
Platform team · also Technical anchor, Source system owners
Evidence
in: platform, raid, discovery
looks like: Each source and target with its connection method, the network changes required, their lead time, and evidence a test connection succeeded.
How the gap shows
  • Connectivity appears as an assumption in the risk log with no date.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 5: MVP starts
Fail fast
Raise the network and credential requests in week one of discovery and pull one row from every source.
Related
DES-01, ENV-01
On DatabricksFor private sources, plan private connectivity (for serverless, network connectivity configurations) and Unity Catalog connections or credentials early.
Trap: Serverless and classic compute reach the network differently; prove the route for the compute you will really use.
ENV-04Platform standards that projects followmediumArchitect
Question
Are naming, layer, folder and table standards written down for the platform so every project builds the same way?
Why it matters
Three projects named and layered their tables differently; consumers could not tell raw from cleaned data, and a shared lineage view became impossible.
Applies to
new platform · components: platform, governance · patterns: any
Owner
Architect · also Platform team, Developers
Evidence
in: platform, decisions
looks like: A short standards document covering naming, layers, table types, ownership tags and code structure, with examples, applied by the first project.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
Fail fast
Ask two developers to name the same new table independently; if the names differ, the standard is missing or unclear.
Related
DES-05, ENV-01
ENV-05Project fits the existing platform's standardsmediumTechnical anchor
Question
Does the project follow the existing platform's environments, standards and shared frameworks, and are any exceptions agreed with the platform team?
Why it matters
The project built its own ingestion code and its own environment layout; the platform team refused to support it at go-live and it had to be rebuilt.
Applies to
project on existing · components: any · patterns: any
Owner
Technical anchor · also Platform team, Architect
Evidence
in: architecture, decisions
looks like: References to the platform standards and frameworks used, and a decision record for each deviation, signed by the platform team.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 7: Test
Fail fast
Walk the platform team through the architecture for thirty minutes and list every point where they say "we don't do it that way".
Related
DES-05, ENV-04

Version control, branching, promotion between environments (7)

REL-01Everything that runs is in version controlhighRelease and DevOps
Question
Is everything that runs in production (pipeline code, SQL, job definitions, schedules, data quality rules, permissions) kept in version control?
Why it matters
The schedule and parameters were edited by hand in the production interface; when the job was recreated after an incident, nobody knew the real settings and it ran with wrong ones for a week.
Applies to
any · components: any · patterns: any
Owner
Release and DevOps · also Developers, Technical anchor
Evidence
in: deployment, architecture
looks like: A repository layout that includes code, job and schedule definitions and configuration, and a rule that production is never changed by hand.
How the gap shows
  • Jobs are described as "configured in the user interface".
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 6: Build
provenstage 9: Production sign-off
Fail fast
Pick one production job and try to recreate it in a fresh environment only from the repository.
Related
REL-02, REL-03, REL-06
On DatabricksKeep code in Git folders and jobs, pipelines and schedules as bundle resources in the repository.
Check it: Recreate a job from the repository into an empty workspace with a bundle deploy.
Trap: Jobs edited in the UI drift from the bundle; restrict edit rights in production.
REL-02Branching and code reviewmediumRelease and DevOps
Question
What is the branching approach, and must every change be reviewed by another person and pass automated checks before it merges?
Why it matters
A developer pushed a join change straight to the main branch on a Friday; it removed a filter, and the weekend loads double-counted refunds.
Applies to
any · components: any · patterns: any
Owner
Release and DevOps · also Developers, Technical anchor
Evidence
in: deployment
looks like: A short branching model, protected main branch, required reviewer, and automated tests and linting run on every pull request.
Due by
planstage 4: Discovery sign-off
designedstage 5: MVP starts
evidencedstage 6: Build
Fail fast
Try to push directly to the main branch; it should be refused.
Related
REL-01, TEST-03
REL-03Automated promotion of the same buildhighRelease and DevOps
Question
Is the same tested build promoted automatically from development to test to production, with an approval step before production?
Why it matters
Each environment was deployed by copying scripts by hand; production ended up with an older version of one script, and test results said nothing about what really ran.
Applies to
any · components: any · patterns: any
Owner
Release and DevOps · also Platform team
Evidence
in: deployment
looks like: A pipeline that builds once, deploys the same version to each environment with environment settings injected, and records which version is where.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 6: Build
provenstage 9: Production sign-off
Fail fast
Deploy the thin slice to test using only the pipeline, with nobody clicking in the target environment.
Related
REL-01, REL-04, REL-07
On DatabricksUse Declarative Automation Bundles with one target per environment; CI runs databricks bundle validate and deploy with a service principal.
Check it: The production deployment history shows only the CI service principal deploying.
REL-04Configuration per environment, not in codemediumDevelopers
Question
Are environment-specific values (paths, connection names, schedules, compute sizes) kept in configuration per environment rather than written into the code?
Why it matters
A storage path to production was hard-coded in a shared module; tests in development read and then overwrote production data.
Applies to
any · components: any · patterns: any
Owner
Developers · also Release and DevOps
Evidence
in: deployment, architecture
looks like: A configuration file or variable set per environment, and code that reads all environment values from it.
Due by
planstage 4: Discovery sign-off
designedstage 5: MVP starts
evidencedstage 6: Build
Fail fast
Search the code for production host names, paths and catalog names; each hit is a defect.
Related
REL-03, REL-05, ENV-01
On DatabricksUse bundle variables and target overrides for catalog names, paths and compute sizes.
Check it: Search the code for production catalog names outside the bundle configuration.
REL-05No secrets in the repositoryhighSecurity and governance
Question
Are passwords, keys and tokens kept in a secret store and read at run time, never in code, configuration files or the repository's history?
Why it matters
A database password was committed in a script; the repository was later shared with a vendor, and the security team had to rotate credentials across four systems.
Applies to
any · components: any · patterns: any
Owner
Security and governance · also Developers, Release and DevOps
Evidence
in: platform, deployment
looks like: A named secret store per environment, code that reads secrets by name, and an automated secret scan on the repository.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 6: Build
Fail fast
Run a secret scanner over the whole repository history today.
Related
REL-04, ENV-02
On DatabricksKeep secrets in secret scopes (optionally backed by the cloud key store) and read them with dbutils.secrets; prefer Unity Catalog connections and service credentials over passwords in code.
Trap: Secrets printed in a notebook cell are redacted, but derived strings and logs can still leak them.
REL-06Infrastructure defined as codemediumPlatform team
Question
Are the platform's environments, storage, compute policies, identities and permissions created from code, so another environment can be built identically?
Why it matters
The test environment was built by hand months after production and differed in network and permission settings, so tests passed that failed in production.
Applies to
new platform · components: platform · patterns: any
Owner
Platform team · also Release and DevOps
Evidence
in: platform, deployment
looks like: Infrastructure code in version control covering workspaces, storage, networks, identities, policies and grants, applied through a pipeline.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 6: Build
provenstage 9: Production sign-off
Fail fast
Build a throwaway environment from the infrastructure code alone and compare its settings with test.
Related
REL-01, ENV-01
On DatabricksDefine workspaces, catalogs, grants, policies and storage credentials with the Databricks Terraform provider; bundles for jobs and pipelines.
REL-07Rolling back a release, including its datahighRelease and DevOps
Question
If a release goes wrong, how are both the code and any data or table structure it changed put back, and has that been rehearsed?
Why it matters
A release changed a table's structure and backfilled it; the code could be rolled back in minutes, but the data could not, and reports were broken for two days.
Applies to
any · components: any · patterns: any
Owner
Release and DevOps · also Technical anchor, Operations and support
Evidence
in: deployment, runbook
looks like: A rollback plan per release covering code, schema and data (restore points or table versions), and a rehearsal in a production-like environment.
How the gap shows
  • The cut-over plan has no rollback section, or says "roll back the code" only.
Due by
planstage 5: MVP starts
designedstage 6: Build
evidencedstage 8: Deploy
Fail fast
Deploy a release that changes a table in test, then roll it back fully, and time it.
Related
REL-03, ORCH-05
On DatabricksUse RESTORE TABLE ... TO VERSION AS OF for data rollback, within the retention window; redeploy the previous bundle version for code.
Check it: Rehearse RESTORE on a test table after a schema-changing release.
Trap: RESTORE only works for versions VACUUM has not cleaned up.

Scheduling, dependencies, retries, restart (6)

ORCH-01Jobs wait for their data, not the clockhighTechnical anchor
Question
Does each job start when its upstream data is complete, rather than at a fixed time that assumes the upstream has finished?
Why it matters
The report job ran at 06:00 assuming the load finished at 05:00; one night the load ran late, and the report published half a day's figures to the whole company.
Applies to
any · components: orchestration · patterns: batch, file, api, database, cdc
Owner
Technical anchor · also Developers, Operations and support
Evidence
in: architecture
looks like: Dependencies declared between jobs or on completion markers, cross-team dependencies named, and what happens when an upstream is late.
How the gap shows
  • The schedule is a list of fixed times with no dependencies.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
Fail fast
Delay an upstream job in test and check that the downstream waits instead of running on stale or partial data.
Related
ORCH-02, SERV-02
On DatabricksUse task dependencies inside a job, and table-update or file-arrival triggers across jobs.
Check it: Delay an upstream table in test and check the downstream job does not start.
ORCH-02Retries for passing failures onlymediumDevelopers
Question
Which failures are retried automatically (timeouts, throttling), how many times and with what wait, and which fail straight away (bad data, logic errors)?
Why it matters
Every failure was retried five times; a bad-data error retried for three hours, burning compute and delaying the alert until the morning.
Applies to
any · components: orchestration · patterns: any
Owner
Developers · also Operations and support
Evidence
in: architecture, runbook
looks like: A retry policy per job with limits and back-off, failures classified as transient or permanent, and an alert after the last retry.
Due by
planstage 4: Discovery sign-off
designedstage 5: MVP starts
evidencedstage 7: Test
Fail fast
In test, cause one timeout and one data error; the first should retry and succeed, the second should fail fast and alert.
Related
ORCH-01, DQ-03, OPS-01
On DatabricksSet task retries and minimum retry interval, plus job and task timeouts and duration warnings.
Trap: Retrying a non-idempotent task duplicates data; fix idempotency before adding retries.
ORCH-03Restart after failing in the middlehighTechnical anchor
Question
If a job fails halfway through writing, is the target left untouched or fully consistent, and can the job restart from the failed step without redoing everything?
Why it matters
A job failed after updating three of five tables; reports joined new facts to old dimensions, and rerunning from the start re-applied the first three updates.
Applies to
any · components: orchestration, transformation, modelling · patterns: any
Owner
Technical anchor · also Developers, Operations and support
Evidence
in: architecture, runbook, test-strategy
looks like: Atomic writes per table, a defined restart point per step, consumers reading only completed runs, and a failure-injection test.
Due by
planstage 4: Discovery sign-off
designedstage 5: MVP starts
evidencedstage 7: Test
provenstage 9: Production sign-off
Fail fast
Kill a job in test while it is writing, then restart it; outputs must match an uninterrupted run.
Related
DUP-02, TEST-05, ORCH-04
On DatabricksEach Delta write is atomic per table; split multi-table updates into tasks and use repair runs to rerun only failed tasks and those after them.
Check it: Cancel a run mid-task in test, repair it, and compare outputs with a clean run.
Trap: A repair reruns the whole failed task from the beginning, not from the failed row.
ORCH-04Two runs writing the same table at oncemediumDevelopers
Question
What stops two runs of the same job, or two different jobs, writing to the same table at the same time?
Why it matters
A manual rerun overlapped the scheduled run; both merged into the same table, one failed on a conflict, and the other left half-updated rows.
Applies to
any · components: orchestration, transformation, modelling · patterns: any
Owner
Developers · also Operations and support
Evidence
in: architecture, runbook
looks like: One writer per table by design, a limit of one concurrent run per job, and a runbook rule for manual reruns.
Due by
planstage 4: Discovery sign-off
designedstage 5: MVP starts
evidencedstage 7: Test
Fail fast
Start the same job twice in test and see what happens to the second run and the table.
Related
ORCH-03
On DatabricksKeep maximum concurrent runs at 1 (the default for new jobs) and use queueing for triggered jobs.
Trap: Two different jobs writing one table can still conflict; Delta raises concurrent modification errors on overlapping files.
ORCH-05Recovery time and data loss limitshighArchitect
Question
How quickly must the service recover after a major failure, how much data may be lost, and has recovery to those limits been tried?
Why it matters
A storage account was deleted by mistake; the backups existed but nobody had tried a restore, and it took four days instead of the four hours the business assumed.
Applies to
any · components: platform, orchestration · patterns: any
Owner
Architect · also Business owner and sponsor, Operations and support, Platform team
Evidence
in: requirements, platform, runbook
looks like: An agreed recovery time and acceptable data loss, a backup or replication design that meets them, and the result of a restore test.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 8: Deploy
provenstage 10: Project sign-off
Fail fast
Restore one important table from backup into a test environment and time it.
Related
REL-07, SEC-05
On DatabricksDecide which tables need deep clones or storage replication to another region; time travel is not a backup, because VACUUM removes old files.
Check it: Restore a table from the chosen backup into a test catalog and time it.
ORCH-06Stream restarts from where it stoppedhighDevelopers
Question
When a streaming job stops or is redeployed, does it resume from its saved position without losing or repeating data, and how is that state protected?
Why it matters
A redeploy changed the stream's checkpoint location; the job restarted from the earliest available data, reprocessed a week and flooded downstream tables.
Applies to
any · components: ingestion, transformation, orchestration · patterns: streaming, cdc
Owner
Developers · also Operations and support, Technical anchor
Evidence
in: architecture, runbook
looks like: Where progress is saved, how it survives redeploys and code changes, the source retention versus the longest expected outage, and a runbook for resetting a stream.
Due by
planstage 4: Discovery sign-off
designedstage 5: MVP starts
evidencedstage 7: Test
provenstage 9: Production sign-off
Fail fast
Stop the stream for longer than a typical outage in test, redeploy a changed version, restart, and compare counts with the source.
Related
DUP-04, ORCH-03
On DatabricksGive every stream a fixed, environment-specific checkpoint location and keep it with the table; in Lakeflow pipelines checkpoints are managed for you.
Check it: Redeploy a changed stream in test and confirm it resumes from its position.
Trap: Some query changes are not allowed with an existing checkpoint (for example changes to stateful operations); plan a reset and backfill when they are needed.

Monitoring, alerting, support, runbooks (3)

OPS-01Alerts reach someone who can acthighOperations and support
Question
When a job fails, runs late or data looks wrong, who is alerted, through which channel, and within how long?
Why it matters
Failure emails went to a shared mailbox nobody read; a load had failed for four days before a user asked why the report had not changed.
Applies to
any · components: orchestration, platform · patterns: any
Owner
Operations and support · also Technical anchor
Evidence
in: runbook, architecture
looks like: Alerts for failure, lateness against the freshness target and data checks, routed to a named team or on-call rota, with an expected response time.
Due by
planstage 4: Discovery sign-off
designedstage 5: MVP starts
evidencedstage 8: Deploy
provenstage 9: Production sign-off
Fail fast
Make a job fail in test and time how long it takes for a person to know.
Related
DQ-04, HAND-01, HAND-04
On DatabricksConfigure job notifications to a team channel or paging tool, plus duration-warning notifications for lateness.
Check it: Fail a test job and time the alert.
OPS-03Every run leaves a recordmediumDevelopers
Question
Does every run record what it read, what it wrote, row counts, duration and outcome, in a place that can be queried?
Why it matters
A user reported a wrong figure from two weeks ago; with no run records nobody could say which load produced it or which input it came from.
Applies to
any · components: ingestion, transformation, modelling, orchestration · patterns: any
Owner
Developers · also Operations and support
Evidence
in: architecture, runbook
looks like: A run log or audit table with run id, inputs, outputs, counts, timings and status, kept for a stated period.
Due by
planstage 4: Discovery sign-off
designedstage 5: MVP starts
evidencedstage 7: Test
Fail fast
Ask which source file produced a given row in yesterday's table, and see how long it takes to answer.
Related
REC-01, OPS-01, COST-03
On DatabricksUse system.lakeflow job and task run tables and DESCRIBE HISTORY for run records; add an audit table for business counts.
Check it: Trace a row in gold to its job run and source file (_metadata.file_path captured in bronze).
OPS-05Production watched closely after go-livemediumTechnical anchor
Question
Were the first weeks in production watched closely (hypercare), with defects closed and operations running it alone before sign-off?
Why it matters
The team moved to the next project on go-live day; a slow memory leak failed the job at month-end and nobody who understood it was available.
Applies to
any · components: any · patterns: any
Owner
Technical anchor · also Operations and support, Business owner and sponsor
Evidence
in: deployment, status
looks like: A hypercare period with daily checks, a log of production issues and their fixes, and a handover point when operations run it alone.
Due by
planstage 6: Build
designedstage 8: Deploy
evidencedstage 9: Production sign-off
provenstage 9: Production sign-off
Fail fast
Put hypercare dates, people and exit criteria in the plan now.
Related
REC-04, HAND-04

Test levels, test data, environments (5)

TEST-01Test levels that include data testshighTest lead
Question
Does the test strategy cover code tests, data tests (rules, reconciliation), integration end to end, performance and user acceptance, each with an owner?
Why it matters
Tests checked that each function worked on hand-made rows; nobody tested the real data end to end, and the first integrated run produced reports nobody recognised.
Applies to
any · components: any · patterns: any
Owner
Test lead · also Technical anchor
Evidence
in: test-strategy
looks like: A test strategy naming each test level, what it proves, who runs it, where, and with what data, including data quality and reconciliation tests.
How the gap shows
  • The test strategy says "to be defined", or covers only unit tests and user acceptance.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 7: Test
Fail fast
For each top requirement, write the one data test that would prove it, and see which ones you cannot write.
Related
TEST-02, TEST-03, DQ-01
TEST-02Realistic, safe test datahighTest lead
Question
Where does test data come from, is it realistic enough to find real problems (volumes, edge cases, bad rows), and is personal data masked or synthetic?
Why it matters
Test data was a tidy hand-made sample with no nulls or duplicates; every test passed, and the first real load failed on problems the sample could never have shown.
Applies to
any · components: any · patterns: any
Owner
Test lead · also Security and governance, Source system owners
Evidence
in: test-strategy, platform
looks like: A source of test data per environment (masked copy, sampled production, generated), the edge cases it includes, and security approval.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 6: Build
Fail fast
Compare the null, duplicate and orphan-key rates of the test data with the real-data profile.
Related
TEST-01, SEC-01, DQ-05
On DatabricksUse shallow or deep clones of masked production tables into a test catalog, or generated data.
Trap: Clones of tables with row filters or masks are not supported; mask before cloning.
TEST-03Automated tests run on every changemediumDevelopers
Question
Do automated tests (code and data) run on every change in the release pipeline, and does a failure stop the release?
Why it matters
A small refactor changed a join; the tests existed but were run by hand and nobody ran them, so the regression reached production.
Applies to
any · components: any · patterns: any
Owner
Developers · also Release and DevOps, Test lead
Evidence
in: test-strategy, deployment
looks like: Tests wired into the pull request and deployment pipeline as required gates, with coverage of the transformations that carry business rules.
Due by
planstage 4: Discovery sign-off
designedstage 5: MVP starts
evidencedstage 6: Build
provenstage 9: Production sign-off
Fail fast
Break one business rule deliberately on a branch; the pipeline must fail before merge.
Related
REL-02, TEST-01
On DatabricksRun unit tests with pytest locally or with Databricks Connect, and data tests in a CI job deployed by the bundle to a test target.
TEST-04Users test with numbers they recognisehighTest lead
Question
Will business users accept the solution by checking real numbers they know, against agreed criteria, rather than by looking at screens?
Why it matters
User acceptance was a demo of the dashboard layout; the first time finance compared it with their own numbers was after go-live, and they rejected it.
Applies to
any · components: modelling, serving · patterns: any
Owner
Test lead · also Business owner and sponsor, Data consumers
Evidence
in: test-strategy, requirements
looks like: User acceptance scenarios with expected figures supplied by the business, named testers, dates, and sign-off criteria.
Due by
planstage 4: Discovery sign-off
designedstage 6: Build
evidencedstage 7: Test
Fail fast
Ask the business owner for three numbers they know by heart and check them in the thin slice.
Related
REC-02, TEST-01
TEST-05Failure, rerun and recovery tested on purposehighTest lead
Question
Does the test plan include deliberate failures: reruns, duplicate input, late data, schema changes, empty input and jobs killed mid-run?
Why it matters
Every test assumed the happy path; in the first month the pipeline met a resent file, a late batch and a crash mid-write, and each one corrupted data.
Applies to
any · components: ingestion, transformation, modelling, orchestration · patterns: any
Owner
Test lead · also Technical anchor, Developers
Evidence
in: test-strategy
looks like: Negative test cases for each failure the design claims to handle, with expected outcomes.
Due by
planstage 4: Discovery sign-off
designedstage 5: MVP starts
evidencedstage 7: Test
Fail fast
List every failure the architecture says it handles and check each has a test case.
Related
DUP-02, ORCH-03, SCH-02

Architecture decisions, patterns, non-functional needs (5)

DES-01A thin slice proved the design end to endhighTechnical anchor
Question
Has one source table been pushed through every layer to one report or query in a production-like environment, before the design was signed off?
Why it matters
A clean architecture diagram was signed off without ever running; in MVP 1 the team found the source format, network and permissions all differed from the assumptions.
Applies to
any · components: any · patterns: any
Owner
Technical anchor · also Architect
Evidence
in: discovery, architecture
looks like: A description of the thin slice, where it ran, what it proved, what it found, and how the design changed as a result.
How the gap shows
  • The architecture shows only target-state diagrams with no evidence of anything having run.
Due by
planstage 3: Discovery
designedstage 3: Discovery
evidencedstage 4: Discovery sign-off
Fail fast
Choose the riskiest source and the most important report, and connect them with the simplest possible pipeline this week.
Related
DQ-05, ENV-03, PERF-02
DES-02Non-functional needs written as numbershighArchitect
Question
Are the non-functional needs (freshness, volume, recovery, availability, retention, concurrency) written as measurable numbers and agreed with the business?
Why it matters
"Near real time" meant five seconds to the business and an hour to the engineers; the gap was found at user acceptance and the design had to be rebuilt.
Applies to
any · components: any · patterns: any
Owner
Architect · also Business analyst, Business owner and sponsor
Evidence
in: requirements, architecture
looks like: A table of non-functional needs, each with a number, a measurement method and who agreed it.
How the gap shows
  • Words like "fast", "near real time", "scalable" or "highly available" with no numbers.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 7: Test
Fail fast
Rewrite each non-functional sentence as a number and ask the business owner to confirm it.
Related
PERF-02, ORCH-05, DES-06
DES-04Grain and keys of every tablehighTechnical anchor
Question
Is the grain (what one row means) and the key of every target table written down and agreed?
Why it matters
The orders table mixed order headers and order lines; summing it double-counted shipping, and two teams built different fixes.
Applies to
any · components: transformation, modelling, serving · patterns: any
Owner
Technical anchor · also Business analyst, Data consumers
Evidence
in: data-model
looks like: For each table: one sentence for the grain, the key columns, and a uniqueness test on that key.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 7: Test
Fail fast
For each target table, finish the sentence "one row is one …" and test the key for uniqueness.
Related
DUP-03, SERV-01, SCD-03
DES-05Shared frameworks instead of one-off codemediumArchitect
Question
Does the platform provide shared, configuration-driven patterns for common work (ingestion, quality checks, logging, history handling), so each pipeline is not hand-built?
Why it matters
Forty pipelines were each written differently; a fix to the deduplication logic had to be made forty times, and twelve were missed.
Applies to
new platform · components: platform, ingestion, transformation · patterns: any
Owner
Architect · also Technical anchor, Developers
Evidence
in: platform, architecture, decisions
looks like: A small set of shared patterns or frameworks with documented configuration, proved by the first two projects, and a decision on what is built versus reused.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
Fail fast
Onboard a second source table with the framework and count how many lines of new code it took.
Related
ENV-04, ENV-05
DES-06Batch or streaming chosen for a reasonmediumArchitect
Question
Is the choice between batch, micro-batch and streaming justified by a stated freshness need and its cost, not by preference?
Why it matters
The team built continuous streaming for a report read once a day; it cost eight times a nightly batch and was harder to support.
Applies to
any · components: ingestion, transformation, orchestration · patterns: batch, streaming, cdc
Owner
Architect · also Business owner and sponsor, Technical anchor
Evidence
in: decisions, requirements
looks like: A decision record linking the processing mode to a freshness number and a running cost comparison.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
Fail fast
Ask the consumer what decision they would take differently with data one minute old versus one hour old.
Related
DES-02, COST-01, PERF-02

Contracts with consumers, grain, freshness promises (5)

SERV-01A contract with each consumerhighTechnical anchor
Question
For each consumer (report, extract, application, model), is there an agreed contract: the tables or fields they use, their meaning, grain, freshness and how they will receive it?
Why it matters
A downstream team built a pricing model on an internal staging table; a routine refactor renamed it and the pricing model failed on a sale day.
Applies to
any · components: serving, modelling · patterns: any
Owner
Technical anchor · also Data consumers, Business analyst
Evidence
in: requirements, data-model, architecture
looks like: A list of consumers and, for each, the published interface, its fields and meaning, grain, freshness promise and a named contact.
How the gap shows
  • Consumers are listed by name only, with no interface or freshness.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
provenstage 9: Production sign-off
Fail fast
Ask the main consumer which exact fields they will read; check each is in the data model with a definition.
Related
SERV-03, DES-04, SERV-04
SERV-02Consumers can tell whether data is completemediumTechnical anchor
Question
Can a consumer see when the data was last refreshed and whether today's load is complete, before they rely on it?
Why it matters
The load failed overnight but the dashboard looked normal; the sales director presented yesterday's numbers as today's to the board.
Applies to
any · components: serving · patterns: any
Owner
Technical anchor · also Data consumers, Operations and support
Evidence
in: architecture, requirements
looks like: A visible "data as of" time on reports, a completeness flag or status table, and a way to warn consumers when a load is late.
Due by
planstage 4: Discovery sign-off
designedstage 5: MVP starts
evidencedstage 7: Test
provenstage 9: Production sign-off
Fail fast
Stop the load in test and look at the report as a user would; can you tell it is stale?
Related
ORCH-01, PERF-02, OPS-01
On DatabricksPublish a status table or use table health in Catalog Explorer from data quality monitoring; show last refresh time on AI/BI dashboards.
SERV-03Changes to what consumers use are announcedmediumTechnical anchor
Question
How are breaking changes to published tables or interfaces versioned and announced, so consumers have time to adapt?
Why it matters
A column was renamed for consistency in a minor release; three reports and a partner extract broke the next morning.
Applies to
any · components: serving, modelling · patterns: any
Owner
Technical anchor · also Data consumers, Release and DevOps
Evidence
in: architecture, deployment
looks like: A rule separating published interfaces from internal tables, a versioning or deprecation approach with a notice period, and a list of who to tell.
Due by
planstage 4: Discovery sign-off
designedstage 5: MVP starts
evidencedstage 9: Production sign-off
Fail fast
Find out how the team would learn which consumers use a given column today.
Related
SERV-01, SCH-03
SERV-04Business definitions live in one placehighBusiness analyst
Question
Is each business measure (revenue, active customer) defined and calculated once, in a shared model, rather than separately in each report?
Why it matters
Three dashboards calculated margin three ways; the leadership meeting spent an hour arguing about which number was right instead of deciding anything.
Applies to
any · components: modelling, serving · patterns: any
Owner
Business analyst · also Technical anchor, Data consumers
Evidence
in: data-model, requirements
looks like: A definitions list with the owner of each measure, implemented once in a shared model or semantic layer that reports reuse.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
Fail fast
Ask two consumers to write down how they calculate the top measure; compare the answers.
Related
REC-02, SERV-01, SCD-04
On DatabricksDefine shared measures once as Unity Catalog metric views and point dashboards and Genie spaces at them.
Trap: Metric views are newer; check the feature's status in your workspace before standardising on it.
SERV-05Consumer queries fast enough under real loadmediumTechnical anchor
Question
Will reports, interfaces and extracts respond fast enough when the expected number of users or calls hit them at the same time?
Why it matters
The dashboard was quick in testing with one user; on Monday morning two hundred managers opened it at once and it took two minutes per page.
Applies to
any · components: serving · patterns: any
Owner
Technical anchor · also Data consumers, Platform team
Evidence
in: requirements, test-strategy
looks like: Expected concurrent users or calls and response targets, the serving design (aggregates, caching, sizing), and a concurrency test result.
Due by
planstage 4: Discovery sign-off
designedstage 5: MVP starts
evidencedstage 7: Test
provenstage 9: Production sign-off
Fail fast
Run the slowest report query twenty times at once against realistic data and time it.
Related
PERF-03, DES-02
On DatabricksServe reports from a SQL warehouse sized and scaled for concurrency; pre-aggregate heavy gold tables or use materialized views.
Check it: Run concurrent queries and read the warehouse's query history for queueing time.

Estimates and their assumptions (6)

EST-01The rough guess became the promisehighTechnical anchor
Question
Has the estimate been redone since the first rough guess, and does everyone quoting a date or a budget know which version they are quoting? Imagine it is six months from now and the project is late: was the date we are judged against the one somebody guessed before anyone looked at the data?
Why it matters
A number given in a corridor before discovery was written into the budget. Discovery found twice the work, but the plan was expected to confirm the guess, so the project was "late" from its first week.
Applies to
any · components: any · patterns: any
Owner
Technical anchor · also Project manager, Business owner and sponsor
Evidence
in: estimate, plan, status
looks like: An estimate history showing the rough estimate, the discovery estimate and later re-estimates, each dated, with the baseline that sign-off accepted clearly marked.
How the gap shows
  • The same number appears in the first brief, the discovery document and the plan, unchanged.
  • The estimate has no date or version, so nobody can tell whether it came before or after discovery.
  • Status reports measure progress against a figure that is older than the discovery document.
  • Phrases such as "as agreed at the outset" or "the approved budget" with no link to how it was built.
Compare
Compare the estimate with the plan: if the plan's end date or budget matches the stage-2 rough estimate rather than the discovery estimate, the guess has become the promise.
Due by
planstage 2: Rough estimate
evidencedstage 4: Discovery sign-off
Fail fast
Put the rough estimate and the discovery estimate side by side in front of the sponsor and ask which one the plan is built on.
Related
EST-02, EST-06
EST-02One number instead of a rangehighTechnical anchor
Question
Is the estimate given as a range, with the reason for its width? If the project overruns, will anyone be able to say whether the overrun was inside the range we said was possible?
Why it matters
"Six weeks" was heard as a commitment. "Five to eight weeks if access arrives in week one" would have been a plan, and the slip to seven weeks would have been expected rather than a crisis.
Applies to
any · components: any · patterns: any
Owner
Technical anchor · also Project manager
Evidence
in: estimate
looks like: A low and a high figure (or a most-likely figure with a stated spread), and a sentence on what would move it toward either end.
How the gap shows
  • The estimate is a single figure in days, weeks or money with no low and high value.
  • The width of the range does not narrow from the rough estimate to the discovery estimate.
  • The range is the same percentage on every item, which suggests it was added, not reasoned.
Compare
Compare the estimate with the status reports: a range in the estimate but a single committed date in every status report means the range was dropped on the way to the stakeholders.
Due by
planstage 2: Rough estimate
evidencedstage 4: Discovery sign-off
Fail fast
Ask the person who wrote the estimate for the worst realistic case. If they can name one without hesitating, it belongs in the document.
Script signal
single-number-estimate (found without AI)
Related
EST-01, EST-03
EST-03An estimate with no written assumptionshighTechnical anchor
Question
Which assumptions does the estimate depend on, and which one, if wrong, would move it the most? If the estimate turns out wrong, will we be able to point to the assumption that broke?
Why it matters
The estimate assumed the source data was clean and that one person would test. Neither was written down, so when both failed, the overrun looked like the team's fault instead of a known risk.
Applies to
any · components: any · patterns: any
Owner
Technical anchor · also Project manager, Business owner and sponsor
Evidence
in: estimate, raid
looks like: A numbered list of assumptions next to the estimate, the most sensitive ones marked, and each one also logged in the RAID log with an owner.
How the gap shows
  • No "assumptions" heading or list anywhere in or near the estimate.
  • Assumptions are written as one line, such as "standard assumptions apply".
  • Assumptions exist in the estimate but none of them appear in the RAID log.
Compare
Compare the estimate's assumptions with the RAID log: an assumption in one but not the other is either unmanaged or was never the real basis of the estimate.
Due by
planstage 2: Rough estimate
evidencedstage 4: Discovery sign-off
Fail fast
Pick the riskiest assumption and test it this week, for example by requesting the access or pulling one real extract.
Script signal
no-assumptions (found without AI)
Related
EST-02, COM-05, PM-07
EST-04Invisible work left out of the estimatehighTechnical anchor
Question
Does the estimate include the work that is not building: getting access, setting up environments, testing, fixing data, rehearsing the cut-over, documentation and handover? If we run out of time, will it be because the build took longer, or because this work was never counted?
Why it matters
The build finished on time. Then access approvals, test data, two cut-over rehearsals and the runbook took another five weeks that nobody had planned for.
Applies to
any · components: any · patterns: any
Owner
Technical anchor · also Project manager, Test lead, Operations and support
Evidence
in: estimate, plan
looks like: Separate lines for access and onboarding, environments, testing (including reconciliation and user acceptance), data fixes, history load, cut-over and rollback rehearsal, runbook, handover and hypercare.
How the gap shows
  • The estimate lists only build items (pipelines, tables, reports).
  • Testing is a fixed percentage added at the end rather than counted per item.
  • Words such as "cut-over", "handover", "access", "environment" or "hypercare" do not appear in the estimate.
Compare
Compare the estimate with the plan: plan tasks for testing, cut-over or handover that have no matching line in the estimate are unfunded work.
Due by
planstage 2: Rough estimate
evidencedstage 4: Discovery sign-off
Fail fast
Read the estimate against the ten lifecycle stages and ask, for each stage after build, which line pays for it.
Script signal
no-invisible-work (found without AI)
Related
EST-05, EST-07, PM-05
EST-05Guessed, not countedmediumTechnical anchor
Question
Was the estimate built by counting the things the project will deliver (sources, tables, rules, reports, interfaces) and multiplying by a rate, or was it a total someone felt was right? Could another person rebuild the number from the document?
Why it matters
The estimate was "about three months" for the whole project. When it was finally counted there were forty tables and twelve reports, which at the team's real rate was nearer six months.
Applies to
any · components: any · patterns: any
Owner
Technical anchor · also Project manager
Evidence
in: estimate
looks like: A table of counted items by type and complexity, a rate per item, where the rate came from, and the total.
How the gap shows
  • The estimate is a total with no breakdown, or a breakdown by phase only.
  • No counts of sources, tables, rules or reports appear anywhere in the documents.
  • The rate per item is not stated or not explained.
Compare
Compare the estimate with the requirements or data model: if they list more sources, tables or reports than the estimate counted, the estimate is short.
Due by
planstage 2: Rough estimate
designedstage 4: Discovery sign-off
Fail fast
Count the items in the requirements yourself and divide the estimate by the count. If the effort per item looks implausibly small, it probably is.
Related
EST-04, EST-06
EST-07History load and data fixing underestimatedhighTechnical anchor
Question
Has the work of loading history and dealing with poor source data been estimated separately, using a profile of the real data? If the history load is what delays go-live, will that surprise anyone?
Why it matters
The pipelines worked on a month of data. Ten years of history held old codes, broken keys and a format change, and checking the old numbers with the business took longer than building the pipelines.
Applies to
any · components: any · patterns: any
Owner
Technical anchor · also Business analyst, Source system owners
Evidence
in: estimate, discovery
looks like: Separate estimate lines for the history load (volume, number of runs, reconciliation with the business) and for data fixes, sized from a profile of real data.
How the gap shows
  • The history load is not mentioned in the estimate, or it is one line with a small number.
  • The estimate says data quality is "assumed acceptable" with no profile behind it.
  • There is no data profiling section in the discovery document.
Compare
Compare the discovery profiling results with the estimate: if the profile found quality problems but the estimate has no line for fixing or handling them, the estimate ignored the evidence.
Due by
planstage 3: Discovery
evidencedstage 4: Discovery sign-off
Fail fast
Load one year of real history in a test environment and time it, and count the rows that fail basic rules.
Related
EST-04, REQ-04

Documentation, knowledge transfer, support model (4)

HAND-01Runbook missing or written at the endhighTechnical anchor
Question
Is there a runbook that operations have read and used: what runs when, how to tell it failed, how to rerun safely, and who to call? If a load fails on the first night after the team has gone, can operations fix it?
Why it matters
The runbook was written in the last week and never tried. On the first failure operations reran the whole load, which doubled a month of figures.
Applies to
any · components: any · patterns: any
Owner
Technical anchor · also Operations and support
Evidence
in: runbook
looks like: A runbook started in build and finished before deploy, covering schedule, monitoring, alerts, common failures with fixes, safe rerun steps and contacts, used in at least one failure drill.
How the gap shows
  • No runbook or operations guide among the documents.
  • The runbook is planned only for the last week before go-live.
  • The runbook has headings but sections marked "to be completed".
Compare
Compare the runbook with the architecture: jobs, sources or failure points in the architecture that the runbook does not mention will be unsupported.
Due by
planstage 4: Discovery sign-off
designedstage 6: Build
evidencedstage 8: Deploy
provenstage 9: Production sign-off
Fail fast
Break one job in the test environment and ask operations to recover it using only the runbook.
Script signal
missing-doc-type (found without AI)
Related
HAND-03, HAND-04
HAND-02Decision log and data dictionary not ready to hand overmediumTechnical anchor
Question
Will the support team receive a current decision log and data dictionary that explain why things are the way they are? When someone asks "why does this table work like this?" after the team has left, where is the answer?
Why it matters
The support team "fixed" a rule that looked wrong. It had been agreed with finance a year before, but the decision was only in an old e-mail thread, and the change broke month-end reporting.
Applies to
any · components: any · patterns: any
Owner
Technical anchor · also Project manager, Operations and support
Evidence
in: decisions, data-model
looks like: A decision log and a data dictionary that match what is in production, reviewed as part of project closure.
How the gap shows
  • The decision log stops well before the end of build.
  • The data dictionary does not cover tables or rules added during build.
  • Handover lists code and access but not decisions or definitions.
Compare
Compare the data dictionary with the delivered data model or test results: rules in production that are not in the dictionary are unexplained.
Due by
planstage 4: Discovery sign-off
evidencedstage 9: Production sign-off
provenstage 10: Project sign-off
Fail fast
Ask someone outside the team to explain one business rule using only the handed-over documents.
Related
PM-03, HAND-03
HAND-03Knowledge transfer never testedhighTechnical anchor
Question
Has the support team made at least one real change and one real recovery while the project team watched? If the project team left tomorrow, could the support team run and change this system?
Why it matters
Knowledge transfer was two presentation sessions. The first change request after closure took the support team a month, and the project team had to be called back.
Applies to
any · components: any · patterns: any
Owner
Technical anchor · also Operations and support, Project manager
Evidence
in: plan, runbook
looks like: Planned handover sessions with hands-on tasks, and a record of a change and a recovery done by the support team themselves.
How the gap shows
  • Knowledge transfer is a single line or a few presentation sessions in the plan.
  • The support team is not named, or joins only at the end.
  • No hands-on tasks are planned for the support team.
Compare
Compare the plan's handover time with the size of the system in the architecture: a short handover for a large system means much will not transfer.
Due by
planstage 4: Discovery sign-off
evidencedstage 9: Production sign-off
provenstage 10: Project sign-off
Fail fast
Ask the support team to make one small real change this sprint, with the project team only watching.
Related
PM-06, HAND-01
HAND-06No exit criteria for early-life supportmediumTechnical anchor
Question
Are there written criteria for ending the early weeks of close support after go-live (for example a number of clean runs, reconciled loads and no critical defects)? How will we know the system is ready to be left alone, rather than the team simply running out of time?
Why it matters
The project team left after two weeks because the budget ended. Month-end processing, which had never run in production, failed the following week with nobody who knew the system.
Applies to
any · components: any · patterns: any
Owner
Technical anchor · also Operations and support, Project manager
Evidence
in: deployment, plan
looks like: Written exit criteria for early-life support, covering at least one full business cycle, reconciled production loads and agreed defect levels.
How the gap shows
  • The plan ends at go-live, or has a fixed support period with no criteria.
  • Period-end or yearly processes are not mentioned in the early-life support period.
Compare
Compare the plan's early-life support period with the business calendar in the requirements: if the period ends before a month-end or other key cycle has run, the riskiest run happens unsupported.
Due by
planstage 4: Discovery sign-off
designedstage 8: Deploy
evidencedstage 9: Production sign-off
Fail fast
Ask: "Which business cycle has not yet run in production?" and keep support in place until it has.
Related
HAND-04

Databricks only (12)

Checks that only make sense on Databricks. Every common check above that works differently on Databricks also has an "On Databricks" box. Checked against docs.databricks.com on 2026-11-01. Product names used here: Unity Catalog, Lakeflow Jobs, Lakeflow pipelines (formerly Delta Live Tables / DLT), AUTO CDC (formerly APPLY CHANGES), Declarative Automation Bundles (formerly Databricks Asset Bundles), data quality monitoring (formerly anomaly detection) and data profiling (formerly Lakehouse Monitoring). Names move fast; re-check before quoting them to a client.

DBX-01Unity Catalog objects owned by groupsmediumPlatform team
Question
Are catalogs, schemas, tables, jobs and pipelines owned by groups or service principals, with privileges granted to account groups rather than individual users?
Why it matters
A catalog owned by a contractor's user account could not be changed after the contractor left, until an admin reassigned ownership in the middle of an incident.
Applies to
any · components: platform, governance · patterns: any
Owner
Platform team · also Security and governance
Evidence
in: platform
looks like: An ownership and grant model by group per catalog and schema, defined in code.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 7: Test
provenstage 9: Production sign-off
Fail fast
List owners of production catalogs, schemas and jobs from information_schema and the jobs list; flag every individual user.
Related
ENV-02, SEC-02
DBX-02Compute policies cap size and force shutdownhighPlatform team
Question
Do compute policies limit node types and worker counts, force auto-termination and required tags, and are users unable to create compute without a policy?
Why it matters
Without policies a developer created a large all-purpose cluster with no auto-termination, and it ran idle for two weeks.
Applies to
new platform · components: platform · patterns: any
Owner
Platform team · also Security and governance
Evidence
in: platform
looks like: Named policies per workload type, who may use each, and unrestricted cluster creation removed.
Due by
planstage 3: Discovery
designedstage 5: MVP starts
evidencedstage 6: Build
Fail fast
As a normal user, try to create compute without a policy and above the policy's size.
Related
COST-02, COST-04
DBX-03Serverless usage tagged for attributionmediumPlatform team
Question
Is serverless compute (notebooks, jobs, pipelines) attributed to projects through serverless usage policies, and is the team aware these tag usage rather than cap it?
Why it matters
Serverless spend grew quickly across teams with no tags, so it showed as one untraceable line on the bill.
Applies to
any · components: platform · patterns: any
Owner
Platform team
Evidence
in: platform
looks like: Serverless usage policies per project or team, assigned to the right groups, and a cost report by their tags.
Due by
planstage 4: Discovery sign-off
designedstage 5: MVP starts
evidencedstage 7: Test
provenstage 9: Production sign-off
Fail fast
Query system.billing.usage for serverless usage in the last week with no project tag.
Related
COST-03, DBX-09
DBX-04Table retention matches time travel and recovery needsmediumTechnical anchor
Question
Is delta.deletedFileRetentionDuration (and log retention) set per table to match how far back the team needs time travel, RESTORE and stream recovery, balanced against storage cost and erasure rules?
Why it matters
A bad release was found eight days later; VACUUM had already removed files older than the default seven days, so RESTORE to the good version was impossible.
Applies to
any · components: transformation, modelling, platform · patterns: any
Owner
Technical anchor · also Security and governance, Operations and support
Evidence
in: architecture, runbook
looks like: A retention setting per table class, justified by recovery needs and personal-data erasure deadlines.
Due by
planstage 4: Discovery sign-off
designedstage 5: MVP starts
evidencedstage 7: Test
Fail fast
Run SHOW TBLPROPERTIES on key tables and compare retention with how long defects usually take to find.
Related
REL-07, SEC-03, PERF-05
DBX-05Auto Loader schema evolution mode chosen per sourcemediumDevelopers
Question
For each Auto Loader source, is the schema evolution mode (addNewColumns, rescue, failOnNewColumns, none) chosen deliberately, with the job set to restart after an intentional stop?
Why it matters
With the default addNewColumns mode, a new source column stopped the stream as designed, but the job had no retries and the stream stayed down all night.
Applies to
any · components: ingestion · patterns: file, streaming, cdc
Owner
Developers · also Technical anchor
Evidence
in: architecture
looks like: The mode per source, where rescued data is monitored, and retries on the ingestion task.
Due by
planstage 4: Discovery sign-off
designedstage 5: MVP starts
evidencedstage 7: Test
Fail fast
Drop a file with a new column in test and see whether the job recovers without a person.
Related
SCH-01, SCH-02
DBX-06Job retries, concurrency and repair are safehighDevelopers
Question
Are task retries, timeouts, maximum concurrent runs and queueing set deliberately for each job, and is every task safe to rerun from the start, as a repair run does?
Why it matters
Operations repaired a failed run; the failed task had appended half its rows before failing, and the repair appended them again.
Applies to
any · components: orchestration · patterns: any
Owner
Developers · also Operations and support
Evidence
in: architecture, runbook
looks like: Retry and timeout settings per task in the bundle, concurrency left at one, and a runbook note on repair.
Due by
planstage 4: Discovery sign-off
designedstage 5: MVP starts
evidencedstage 7: Test
Fail fast
Cancel a task halfway in test, repair the run, and compare the output with a clean run.
Related
DUP-02, ORCH-02, ORCH-03
DBX-07Pipeline expectations with the right actionmediumDevelopers
Question
In Lakeflow pipelines, does each expectation use the right action (warn, drop row, fail update), and are dropped rows kept somewhere for reprocessing?
Why it matters
Every rule was set to drop; thousands of rows a day were silently removed and only a count was logged.
Applies to
any · components: ingestion, transformation · patterns: any
Owner
Developers · also Business analyst
Evidence
in: architecture, data-model
looks like: A rule list mapping each expectation to an action, plus a quarantine table for dropped rows.
Due by
planstage 4: Discovery sign-off
designedstage 5: MVP starts
evidencedstage 7: Test
provenstage 9: Production sign-off
Fail fast
Count expectations by action in the pipeline code and check each drop has a quarantine.
Related
DQ-01, DQ-02
DBX-08Promotion through bundles, run as service principalsmediumRelease and DevOps
Question
Are jobs, pipelines and dashboards deployed to each environment through Declarative Automation Bundles (formerly Asset Bundles) from CI, with production jobs running as a service principal?
Why it matters
Production jobs were created by hand in the UI and ran as the developer; a later bundle deploy created duplicates and both copies wrote to the same tables.
Applies to
any · components: platform, orchestration · patterns: any
Owner
Release and DevOps · also Platform team
Evidence
in: deployment
looks like: A bundle with dev, test and prod targets, run_as a service principal in prod, deployed only by CI.
Due by
planstage 4: Discovery sign-off
designedstage 5: MVP starts
evidencedstage 6: Build
provenstage 9: Production sign-off
Fail fast
Deploy the thin slice to the test target with databricks bundle deploy from a CI pipeline.
Related
REL-03, REL-04, ENV-02
DBX-09System tables used for cost and run monitoringlowPlatform team
Question
Are system tables (billing usage and prices, job run timeline, audit, query history) available to the people who need them, with a cost and reliability view built on them?
Why it matters
Nobody had been granted access to the system catalog, so a cost spike was investigated for a week from the invoice instead of in an hour from the usage table.
Applies to
new platform · components: platform · patterns: any
Owner
Platform team · also Operations and support
Evidence
in: platform, runbook
looks like: Grants on the system catalog to platform and finance groups, and a dashboard by job, tag and day.
Due by
planstage 4: Discovery sign-off
designedstage 5: MVP starts
evidencedstage 7: Test
provenstage 9: Production sign-off
Fail fast
Query yesterday's cost by job from system.billing.usage and list prices.
Related
COST-01, COST-03, OPS-03
DBX-10Managed or external tables decidedlowArchitect
Question
Is the choice between Unity Catalog managed and external tables, and the storage location for each catalog, decided and recorded?
Why it matters
Tables were created as external by habit; predictive optimization did not run on them, small files built up, and dropping a table left its data behind in storage.
Applies to
new platform · components: platform, governance · patterns: any
Owner
Architect · also Platform team
Evidence
in: platform, decisions
looks like: A decision record on table types and managed storage locations per catalog.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
Fail fast
Count production tables by table type in information_schema.tables.
Related
PERF-05, ENV-04
DBX-11AUTO CDC sequenced by a real source orderhighDevelopers
Question
Where AUTO CDC builds SCD tables, is SEQUENCE BY a column that truly reflects source change order (with a tie-breaker), not the time the row landed?
Why it matters
Sequencing used the file arrival time; a delayed file replayed older updates as newest, and SCD type 2 history showed customers moving back to old addresses.
Applies to
any · components: transformation, modelling · patterns: cdc
Owner
Developers · also Source system owners
Evidence
in: architecture, data-model
looks like: The sequence column per source, how ties are broken (STRUCT of columns), and the delete condition.
Due by
planstage 4: Discovery sign-off
designedstage 5: MVP starts
evidencedstage 7: Test
Fail fast
Ask the source owner which column orders changes, and check for ties in a real change extract.
Related
SCD-02, HIST-03
DBX-12Workspace and catalog layout per environmenthighArchitect
Question
Is it decided how environments map to workspaces and catalogs (for example one catalog per environment, bound to its workspace), and how layers map to schemas?
Why it matters
All environments shared one metastore catalog with prefixed schema names; a test job pointed at the wrong prefix and overwrote a production schema.
Applies to
new platform · components: platform · patterns: any
Owner
Architect · also Platform team, Security and governance
Evidence
in: platform, decisions
looks like: A diagram of workspaces, catalogs and schemas per environment, with workspace-catalog bindings.
Due by
planstage 3: Discovery
designedstage 4: Discovery sign-off
evidencedstage 5: MVP starts
Fail fast
From the development workspace, try to read and write a production catalog.
Related
ENV-01, ENV-04