Home › Learn › The technical anchor › Project review › The checks
Project review · the catalog
The checks: all 210, by category
These are the questions the project review skills ask of a data project's documents, in three packs, one per discipline: 43 business analysis, 64 project management, 91 technical, plus 12 that only apply on Databricks. Each pack is also a skill on its own, and the full review checks the seams between them. Open a check to see why it matters, when it applies, what good evidence looks like, by which stage it is due, and the cheapest test that could prove a team wrong.
To review them: mark each check Keep, Change or Drop and add a note. Your marks stay in this browser. When you're done, Copy my review puts them on your clipboard as plain text to send back. The checks come from the repository, so this page always matches the skill.
Business analysis (43)
What the business needs, in terms that can be tested against data: requirements, definitions and metrics, data requirements, scope, acceptance.
Read first: Source semantic understanding: a key to data program success. What to ask about the meaning of each source (grain, statuses, corrections, dates, amounts, the number the business trusts) before anything is built.
Requirements, definitions, acceptance criteria (10)
REQ-01Vague words in the requirementshighBusiness analyst
- Question
- Has every vague word in the requirements, such as "real-time", "fast", "all data", "user-friendly", "accurate" or "complete", been replaced with a number or a test? If the users reject the system at the end, will it be because we each read those words differently?
- Why it matters
- "Real-time" meant within a second to the sponsor and by the next morning to the team. The design was built for daily loads and had to be rebuilt late in the project.
- Applies to
- any · components: any · patterns: any
- Owner
- Business analyst · also Technical anchor, Test lead
- Evidence
- in: requirements
looks like: Every quality word paired with a measure, for example "available by 7 a.m. on working days" instead of "timely", or "the last three years" instead of "all history". - How the gap shows
- Words such as real-time, near real-time, fast, quickly, timely, all data, complete, accurate, user-friendly, intuitive, seamless, scalable or robust appear without a number next to them.
- Lists of required data or reports end with "etc." or "and so on".
- Compare
- Compare the requirements with the architecture: if the requirement says "real-time" and the architecture schedules a daily batch, someone's reading of the word is wrong.
- Due by
plan stage 1: Requirement received designed stage 4: Discovery sign-off - Fail fast
- Ask the requester to put a number on each vague word in a fifteen-minute call, and write the answers down.
- Script signal
vague-words(found without AI)- Related
- REQ-02, REQ-06
REQ-02No acceptance criteriahighBusiness analyst
- Question
- Does every requirement have acceptance criteria that the data can be tested against? At user acceptance, how will we and the business agree that a requirement is met?
- Why it matters
- Requirements said "show monthly sales by region". At user acceptance the business compared it with their own report, got a different number, and nobody had agreed in advance which one was right.
- Applies to
- any · components: any · patterns: any
- Owner
- Business analyst · also Test lead, Technical anchor
- Evidence
- in: requirements, test-strategy
looks like: For each requirement, one or more criteria in the form "given this data, the result is this", ideally with a known figure the business already trusts. - How the gap shows
- Requirements or user stories have no "acceptance criteria" or "done when" section.
- Acceptance criteria repeat the requirement in other words rather than describing a test.
- Backlog items are marked ready for build without criteria.
- Compare
- Compare the requirements with the test strategy: requirements that have no matching test, or tests that trace to no requirement, show the two were written apart.
- Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test - Fail fast
- Take the three most important requirements and write a test for each with the business owner, using a figure from a report they already trust.
- Script signal
no-acceptance-criteria(found without AI)- Related
- REQ-01, REQ-03
REQ-03Business definitions not agreed and signedhighBusiness analyst
- Question
- Are the key business definitions (for example revenue, active customer, order date) written down and signed by the person who owns each one? If two departments see different numbers after go-live, which signed definition settles it?
- Why it matters
- Sales counted revenue when the order was booked; finance counted it when it was paid. Both signed the requirements, which said "revenue", and the dashboard was disowned by one of them in the first week.
- Applies to
- any · components: modelling, serving · patterns: any
- Owner
- Business analyst · also Business owner and sponsor, Data consumers, Technical anchor
- Evidence
- in: data-model, decisions
looks like: A definitions list with the exact rule for each term, the grain, the owner who signed it and the date. - How the gap shows
- The data dictionary has names but no rules, owners or sign-off.
- The same term is used in requirements from different departments with no definition.
- Definitions are marked "to be confirmed" in a document that has passed discovery sign-off.
- Compare
- Compare the requirements with the data model: if a term in the requirements has a different rule, or no rule, in the data model, the definition was never agreed.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off - Fail fast
- Ask two of the main consumers to write their own definition of the top five terms separately, then compare the answers.
- Related
- REQ-02, STK-02
REQ-04Requirements written without looking at the datahighBusiness analyst
- Question
- Were the requirements checked against a profile of the real source data? If a requirement turns out to be impossible, will we find out in discovery or in testing?
- Why it matters
- The requirement asked for sales by product category. The source had no category for a third of products, which only came to light when the report was tested.
- Applies to
- any · components: any · patterns: any
- Owner
- Business analyst · also Technical anchor, Source system owners
- Evidence
- in: discovery, requirements
looks like: A data profile per source (volumes, keys, nulls, odd values) and a note against each requirement saying which source fields support it and any gaps found. - How the gap shows
- No data profiling section or results in the discovery document.
- Requirements name fields or attributes that do not appear in any source description.
- Phrases such as "data to be confirmed" or "assumed available" against required fields.
- Compare
- Compare the requirements with the data model or source mapping: a required field with no source mapped to it is a requirement nobody checked against the data.
- Due by
plan stage 3: Discovery evidenced stage 4: Discovery sign-off - Fail fast
- Pull one real extract from each source and map the top ten requirements to actual columns in an afternoon.
- Related
- EST-07, REQ-03
REQ-05The users are not the requestersmediumBusiness analyst
- Question
- Have the people who will use the output every day been asked what they need, or only the people who requested it? If nobody uses the system three months after go-live, will it be because it answered the requester's question and not the users'?
- Why it matters
- A director requested a dashboard and approved the design. The analysts who were meant to use it needed row-level detail to act on, kept their old reports, and the dashboard went unused.
- Applies to
- any · components: serving, modelling · patterns: any
- Owner
- Business analyst · also Data consumers, Business owner and sponsor
- Evidence
- in: requirements, stakeholders
looks like: Named users or user groups, how they will use the output, which decisions it supports, and evidence that some of them took part in workshops and will take part in user acceptance testing. - How the gap shows
- Requirements name only a sponsor or requester, not users.
- No user workshops, interviews or user personas are mentioned.
- User acceptance testers are not named, or are the same person as the requester.
- Compare
- Compare the stakeholder map with the requirements sources: user groups on the map who are not listed as contributors to the requirements were never asked.
- Due by
plan stage 1: Requirement received designed stage 4: Discovery sign-off evidenced stage 7: Test - Fail fast
- Show a rough mock-up or a real sample of the output to three actual users and ask what decision they would make with it.
- Related
- STK-01, STK-03
REQ-06Non-functional needs missinghighBusiness analyst
- Question
- Are freshness, volume and growth, recovery time and acceptable data loss, retention, and who may see what written as requirements with numbers? If the system works but is rejected, will it be because it is too slow, too stale, or unrecoverable?
- Why it matters
- Nobody asked how fresh the data had to be or how quickly it had to recover. After the first overnight failure the business found out it would take a day to recover, and the system was not accepted.
- Applies to
- any · components: any · patterns: any
- Owner
- Business analyst · also Technical anchor, Operations and support, Security and governance
- Evidence
- in: requirements, architecture
looks like: A short non-functional list: freshness deadline, expected volume and growth, how long it may be down, how much data may be lost, how long data is kept, and access rules. - How the gap shows
- No "non-functional" heading in the requirements.
- Freshness, recovery or retention are mentioned only as "to be agreed".
- Volume is described in words ("large") rather than numbers.
- Compare
- Compare the requirements with the runbook or support model: if the runbook promises recovery within hours but no requirement says what is needed, or the reverse, the two disagree on what was promised.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 7: Test - Fail fast
- Ask the business owner: "If this is down for a day, what happens?" and write down the answer as a number.
- Related
- REQ-01, HAND-04
REQ-07Goals with no measure of successmediumBusiness analyst
- Question
- What business result is this project for, how will it be measured, and what is the measure today? A year after go-live, how will anyone know whether it was worth it?
- Why it matters
- The goal was "better insight into customers". No measure was agreed, so at closure the project could not show any benefit, and the next request for funding was refused.
- Applies to
- any · components: any · patterns: any
- Owner
- Business analyst · also Business owner and sponsor
- Evidence
- in: discovery, requirements
looks like: Two or three goals, each with a measure, today's value (the baseline), a target, and where the data for measuring it will come from. - How the gap shows
- Goals are written as activities ("build a data platform") rather than results.
- No baseline figures are given for any goal.
- Success measures need data the project does not collect.
- Compare
- Compare the goals in the discovery document with the scope: a goal whose measure depends on data or reports that are out of scope cannot be measured.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off proven stage 10: Project sign-off - Fail fast
- Ask the sponsor to state the one number they hope will change, and check today whether that number can be produced.
- Related
- HAND-05
REQ-08Everything is a "must"highBusiness analyst
- Question
- Are the requirements prioritised (for example must, should, could and won't), with real items in the "won't" list and fewer musts than the first release can deliver? If time runs out, has the business already said which requirements go first?
- Why it matters
- Every one of sixty requirements was marked essential. When the build ran late the team cut by effort instead of value, dropped the one report the finance director actually used, and the release was rejected.
- Applies to
- any · components: any · patterns: any
- Owner
- Business analyst · also Business owner and sponsor, Project manager
- Evidence
- in: requirements, plan
looks like: Each requirement with a priority agreed by the business owner, a "won't this time" list with real items in it, and the musts sized to fit the first release with room to spare. - How the gap shows
- No priority column in the requirements, or every requirement has the same priority.
- The "won't" or "not this release" list is empty.
- Musts make up most of the requirements.
- Compare
- Compare the must-have requirements with the first release in the plan: musts that are not in the first release, or a first release full of shoulds and coulds, mean priority and plan disagree.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off - Fail fast
- Ask the business owner: "If we could deliver only five of these, which five?" and record the answer as the priority.
- Related
- SCOPE-05, SCOPE-01
REQ-09No trace from goal to requirement to testmediumBusiness analyst
- Question
- Can every requirement be traced back to a business goal or decision and forward to the data it needs, the thing that builds it and the test that proves it? When someone asks why a table exists, or whether a goal has been tested, can the team show it in minutes?
- Why it matters
- A requirement for weekly churn by segment was dropped when the backlog was re-planned. Nothing linked it to the retention goal it served, so nobody noticed until the sponsor asked for the churn figure at closure.
- Applies to
- any · components: any · patterns: any
- Owner
- Business analyst · also Test lead, Technical anchor
- Evidence
- in: requirements, test-strategy
looks like: A simple trace table: goal, requirement, sources and fields, backlog item, test case, and status, kept up to date as items are added, changed or dropped. - How the gap shows
- Requirements have no ids, or ids are not used in the backlog or test cases.
- Goals and requirements live in separate documents with no link between them.
- Test cases do not say which requirement they prove.
- Compare
- Compare the requirement ids in the requirements with those referenced in the test strategy: requirements with no test, and tests pointing at no requirement, show where the trace is broken.
- Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test - Fail fast
- Pick the main goal and try to list, in ten minutes, every requirement and test that serves it.
- Related
- REQ-02, REQ-07
REQ-10Requirements change without a version or a trailmediumBusiness analyst
- Question
- When a requirement, rule or definition changes, is the change versioned, dated, approved by its owner, and carried through to the data model, the backlog and the tests? If a tester tests against last month's version of a rule, will anyone know?
- Why it matters
- The definition of an active customer changed twice in workshops. The data model held the first version, the tests the second, and the report the original, and user testing failed on all three.
- Applies to
- any · components: any · patterns: any
- Owner
- Business analyst · also Test lead, Developers, Project manager
- Evidence
- in: requirements, decisions
looks like: Requirements with a version and date, a change history per requirement or rule, the approver of each change, and a note of which designs and tests were updated. - How the gap shows
- The requirements document has no version, date or change history.
- Definitions or rules differ between the requirements, the data model and the test cases.
- Changes are agreed in meetings or chat and not reflected in the requirements.
- Compare
- Compare the rule or definition text in the requirements with the same rule in the data model and test cases: any difference means a change reached some documents and not others.
- Due by
plan stage 3: Discovery designed stage 5: MVP starts - Fail fast
- Take the three most-discussed definitions and check that the requirements, data model and tests say the same thing today.
- Related
- SCOPE-03, REQ-09, OWN-02
In scope, out of scope, change control (4)
SCOPE-01Scope written only as "in"highBusiness analyst
- Question
- Is there a written list of what is out of scope, as well as what is in? When someone says "I assumed that was included", which document will we point to?
- Why it matters
- The scope listed the tables to build. The business assumed the reports on top were included too, and the argument cost a month and the sponsor's trust.
- Applies to
- any · components: any · patterns: any
- Owner
- Business analyst · also Project manager, Technical anchor, Business owner and sponsor
- Evidence
- in: discovery, requirements
looks like: Three lists, in scope, out of scope and later, each item specific enough that a stranger could say which list a new request belongs to. - How the gap shows
- No "out of scope" heading or list in the discovery, scope or requirements document.
- The out-of-scope list is generic ("anything not listed above").
- Compare
- Compare the scope with the requirements: a requirement that is neither in the in-scope list nor explicitly out is a future argument.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off - Fail fast
- Ask the business owner to read only the out-of-scope list and say which item surprises them.
- Script signal
no-out-of-scope(found without AI)- Related
- SCOPE-02, SCOPE-06
SCOPE-02The data scope traps left silenthighTechnical anchor
- Question
- Has each of the ten data scope traps been written down as in, out or later: history and backfill, deletes at the source, late and corrected data, changes over time, bad rows, source data quality, personal data, reports and dashboards, access and onboarding, and recovery? Which of them will someone assume was included?
- Why it matters
- Nobody wrote down whether deleted customers should disappear from the reports. The business assumed yes, the team assumed no, and the gap was found by an auditor.
- Applies to
- any · components: any · patterns: any
- Owner
- Technical anchor · also Business analyst, Business owner and sponsor
- Evidence
- in: discovery, requirements
looks like: A table with one row per trap, the decision (in, out or later), who decided and why. - How the gap shows
- Several of the words history, backfill, delete, late, correction, bad rows, quality, personal data, reports, access and recovery do not appear in the scope section.
- Traps are mentioned but marked "to be confirmed" after discovery sign-off.
- Compare
- Compare the scope with the architecture: a trap the architecture handles (for example history kept) that the scope does not mention, or the reverse, shows the two were never reconciled.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off - Fail fast
- Walk the ten traps with the business owner in thirty minutes and record each answer.
- Related
- SCOPE-01, EST-07
SCOPE-04"Phase 2" used as a hiding placemediumProject manager
- Question
- Is everything labelled "phase 2" or "later" actually planned, funded and owned, and does the business know that phase 2 may never happen? If phase 2 is cancelled, will the first phase still be useful on its own?
- Why it matters
- Hard items such as history and data quality were moved to phase 2 to make the first phase look affordable. Phase 2 was never funded, and phase 1 was unusable without them.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Business owner and sponsor, Business analyst
- Evidence
- in: plan, discovery
looks like: A "later" list in which each item states whether phase 1 depends on it, an owner, and whether funding exists, plus a statement that phase 1 is usable without them. - How the gap shows
- Many items marked "phase 2", "future phase" or "later" with no dates or owners.
- Phase 2 has no estimate or budget.
- Items that the main goal depends on are listed under phase 2.
- Compare
- Compare the goals or success measures with the phase-2 list: a goal that can only be met by phase-2 items means phase 1 alone does not deliver the value promised.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off - Fail fast
- Ask the sponsor: "If there is no phase 2, would you still pay for phase 1 as written?"
- Related
- SCOPE-05, REQ-07
SCOPE-06Scope not signed by the business ownermediumProject manager
- Question
- Has the person who will judge success signed the in, out and later lists, and is that signature recorded with a date? If scope is disputed later, whose signature ends the argument?
- Why it matters
- The scope was agreed with a delegate. The real business owner saw it for the first time at user acceptance and rejected half of the out-of-scope list.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Business owner and sponsor, Business analyst
- Evidence
- in: decisions, discovery
looks like: A dated sign-off record naming the business owner, the version of the scope signed, and any conditions. - How the gap shows
- The scope document has no approver, approval date or version.
- Approval is recorded as "agreed in meeting" without a name.
- The approver is not the business owner named in the stakeholder map.
- Compare
- Compare the scope sign-off with the stakeholder map: if the person who signed is not the one the map names as the decision owner, the sign-off may not hold.
- Due by
plan stage 4: Discovery sign-off evidenced stage 4: Discovery sign-off - Fail fast
- Send the out-of-scope list to the business owner today and ask for a written yes.
- Related
- SCOPE-01, STK-05
How the business gets the numbers today - reports replaced, trusted figures, manual adjustments, decisions served (4)
CUR-01The current process and reports not written downhighBusiness analyst
- Question
- Is the way the business produces this information today written down: which reports, spreadsheets, extracts and manual steps, who runs them, how often, and who reads them? If an old report keeps being used after go-live, will it be one we never knew existed?
- Why it matters
- The project replaced the monthly sales pack. Nobody had mapped the twelve spreadsheets that fed it, and one of them carried a manual rebate adjustment, so the new numbers were wrong by exactly that amount and the old pack stayed in use.
- Applies to
- any · components: modelling, serving · patterns: any
- Owner
- Business analyst · also Data consumers, Business owner and sponsor
- Evidence
- in: discovery, requirements
looks like: A current-state inventory: one row per report, spreadsheet or extract being replaced, with its producer, frequency, readers, the sources it pulls from and the manual steps in between. - How the gap shows
- The discovery document describes the future solution but has no "current state" or "as is" section.
- Reports to be replaced are mentioned by name only, with no producer, frequency or readers.
- Phrases such as "replaces existing reporting" with no list of what that reporting is.
- Compare
- Compare the current-state inventory in the discovery document with the in-scope list: a report or spreadsheet in use today that is neither replaced nor declared out of scope will keep running beside the new system.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off - Fail fast
- Sit for one hour with the person who produces today's main report and write down every step, file and manual change they make.
- Related
- CUR-02, CUR-04, RLC-03
CUR-02The number the business trusts today not identifiedhighBusiness analyst
- Question
- For each key figure, which report or person does the business trust today, and has the business agreed that the new system will be judged against it? When the new number differs from the old one, who will decide which is wrong?
- Why it matters
- The new revenue figure was two per cent lower than finance's ledger. Nobody had agreed in advance that the ledger was the reference, so three weeks were lost arguing about whose number was right before anyone looked for the cause.
- Applies to
- any · components: modelling, serving · patterns: any
- Owner
- Business analyst · also Business owner and sponsor, Data consumers, Test lead
- Evidence
- in: requirements, test-strategy
looks like: For each key figure, the trusted reference (report, ledger, application screen), who owns it, a sample value for a known period, and whether the business believes that reference is itself correct. - How the gap shows
- No reference report or trusted figure is named anywhere in the requirements or test strategy.
- The requirements say the new figures must be "accurate" or "correct" without saying compared with what.
- The trusted figure is named but nobody has captured a value for a known period.
- Compare
- Compare the requirements with the test strategy: if the requirements name a trusted report but the reconciliation tests compare against something else, or nothing, acceptance will be argued, not tested.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 7: Test - Fail fast
- Ask the business owner for last month's value of the top three figures and where they got it, and store the screenshots with the requirements.
- Related
- REC-02, TEST-04, UAT-01
CUR-03Decisions the data will drive not namedhighBusiness analyst
- Question
- Which decisions will people make with this data, how often, and what do they do today instead? If the project delivers every table and report on time but nothing changes in the business, will it be because nobody said which decision it was for?
- Why it matters
- The team built a full customer model because "the business needs a single view of the customer". The one decision it was actually needed for, which customers to call each Monday, required a field that was never captured.
- Applies to
- any · components: any · patterns: any
- Owner
- Business analyst · also Business owner and sponsor, Data consumers
- Evidence
- in: discovery, requirements
looks like: A short list of decisions, each with who makes it, how often, what information they need to make it, what they use today, and what is painful about that. - How the gap shows
- Requirements list data and reports but never the decision or action they support.
- The problem statement is about technology ("legacy reporting", "move to the cloud") rather than a business pain.
- Words such as "insight", "visibility" or "single view" appear without a named decision.
- Compare
- Compare the decisions in the discovery document with the freshness and grain in the requirements: a daily decision served by monthly data, or a decision about individual customers served by totals, means the two were not connected.
- Due by
plan stage 1: Requirement received designed stage 3: Discovery - Fail fast
- Ask three intended users: "What will you do differently on the day this is live?" and write the answers against the requirements.
- Related
- REQ-07, REQ-05, DATA-04
CUR-04Manual adjustments hidden in today's numbershighBusiness analyst
- Question
- Do today's trusted figures include manual adjustments, hand-kept mappings, overrides or exclusions that live in someone's spreadsheet or head, and has each one been listed with a decision to automate it, keep it manual, or drop it? If the new numbers never match the old ones, will it be because of a correction nobody told us about?
- Why it matters
- Each month an analyst removed two intercompany customers and reclassified a product line by hand before the board pack went out. The new system did neither, and it took a month of reconciliation to find out why.
- Applies to
- any · components: transformation, modelling · patterns: any
- Owner
- Business analyst · also Data consumers, Technical anchor
- Evidence
- in: discovery, requirements
looks like: A list of every manual adjustment, override, exclusion and hand-kept mapping in today's process, with its size, who makes it, why, and whether the new system will automate it, keep it as a governed manual input, or stop it. - How the gap shows
- The current state is described as "extracted from the source system" with no manual steps.
- Spreadsheets are mentioned as inputs to today's report with no description of what they change.
- No requirement covers manual inputs, overrides or exclusions.
- Compare
- Compare the current-state description with the business rules or data model: an adjustment people make today that has no rule, no manual input and no "dropped" decision will show up as an unexplained difference.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off - Fail fast
- Take last month's trusted figure and the raw source total for the same period; ask the report owner to explain the difference line by line.
- Related
- CUR-01, CUR-02, RULE-01
The business side of the data needed - sources and owners, grain, history, freshness, late data, source of truth (7)
DATA-01Sources not named, each with an ownerhighBusiness analyst
- Question
- Is every source the requirements depend on named, with the system, the data needed from it, and a named person who owns it and can answer questions about it? If a source turns out to be unavailable in the build, will we find out then, or now?
- Why it matters
- The requirements said "customer and order data". In the build the team learned that returns lived in a separate warehouse system nobody had listed, owned by a team that had no time for the project that quarter.
- Applies to
- any · components: ingestion · patterns: any
- Owner
- Business analyst · also Source system owners, Technical anchor
- Evidence
- in: requirements, discovery
looks like: A source list: one row per source with the system, the business data it provides, the requirements that need it, the source owner's name, and whether access has been requested. - How the gap shows
- Data needs are described as subjects ("customer data", "sales data") with no system named.
- Sources are listed without an owner, or with a team name instead of a person.
- A source appears in the architecture that is not in the requirements, or the reverse.
- Compare
- Compare the source list in the requirements with the sources drawn in the architecture: a source in one and not the other is either unneeded work or a missing dependency.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off - Fail fast
- For each requirement, write the source system it comes from; any requirement you cannot fill in is a gap to raise this week.
- Related
- REQ-04, OWN-01
DATA-02Grain the users need not compared with the grain the sources holdhighBusiness analyst
- Question
- For each requirement, what level of detail do the users need (each transaction, each customer per day, each store per month), and does the source hold data at that level? If users ask to drill into a total after go-live, will the detail exist?
- Why it matters
- Users needed sales by individual product. The source only kept daily totals by product group, which nobody checked until the first drill-down in user testing.
- Applies to
- any · components: modelling, serving · patterns: any
- Owner
- Business analyst · also Data consumers, Technical anchor
- Evidence
- in: requirements, data-model
looks like: For each key requirement, the grain needed in plain words (for example "one row per order line per day"), the grain available at the source, and how any gap was resolved. - How the gap shows
- Requirements say "by product", "by customer" or "detail" without saying the lowest level needed.
- Drill-down or "detail" is requested with no statement of how far down.
- The data model's grain is finer or coarser than anything stated in the requirements.
- Compare
- Compare the grain in the requirements with the grain of the tables in the data model: a requirement needing more detail than its table holds cannot be met.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 7: Test - Fail fast
- Show users one real row from the source and ask: "Is this the smallest thing you will ever need to see?"
- Related
- DES-04, REQ-04
DATA-03History depth asked for without a reasonmediumBusiness analyst
- Question
- How many years of history does the business need, for which data, and what use requires that depth (year-on-year comparison, trend models, regulation)? If the history load doubles the estimate, will we know which part of it the business could live without?
- Why it matters
- The business asked for "all history". Ten years were loaded at great cost, the oldest seven had different product codes nobody could map, and the only use was a two-year comparison.
- Applies to
- any · components: ingestion, modelling · patterns: any
- Owner
- Business analyst · also Business owner and sponsor, Technical anchor
- Evidence
- in: requirements, discovery
looks like: For each data area, the years of history needed, the use that needs them, whether old data must match today's definitions, and who will check the old numbers. - How the gap shows
- History is requested as "all", "full" or "as much as possible".
- A number of years is given with no reason.
- Nobody is named to check that the loaded history matches what the business remembers.
- Compare
- Compare the history depth in the requirements with the history load in the estimate or plan: a different number of years, or none in the estimate, means the cost of history was never seen by the business.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off - Fail fast
- Ask: "Which report or decision needs data older than two years?" and price each extra year before agreeing.
- Related
- HIST-01, EST-07, SCD-01
DATA-04Freshness not tied to a decisionhighBusiness analyst
- Question
- How fresh must the data be, as a time of day or a delay in minutes or hours, and which decision or action needs it that fresh? If the business later asks for faster data, will we know whether the decision really needs it?
- Why it matters
- The sponsor asked for hourly data. The only decision it served was a weekly stock order, so the team built and paid for hourly loads that nobody looked at between Mondays.
- Applies to
- any · components: any · patterns: any
- Owner
- Business analyst · also Business owner and sponsor, Data consumers, Technical anchor
- Evidence
- in: requirements
looks like: For each consumer or report, a freshness target such as "ready by 7 a.m. local time on working days" or "within fifteen minutes of the sale", paired with the decision that needs it and what happens if it is late. - How the gap shows
- Freshness is stated as "real-time", "up to date" or "latest" with no number.
- One freshness target is stated for everything, with no difference between consumers.
- No reason or decision is given for the freshness asked for.
- Compare
- Compare the freshness in the requirements with the decisions in the discovery document: data needed faster than the decision it serves is cost with no benefit, and slower is a decision made on stale data.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 7: Test - Fail fast
- Ask the user: "If this arrived at noon instead of 7 a.m., what would you do differently?" and price the two options side by side.
- Related
- REQ-01, REQ-06, PERF-02, CUR-03
DATA-05No business rule for late and corrected datahighBusiness analyst
- Question
- When data arrives late or a past figure is corrected, what should the business see: should last month change, which date decides the period (order, shipment, invoice or posting date), and when is a period closed and never restated? If a published figure changes after the board has seen it, has the business agreed that it may?
- Why it matters
- Returns posted in March for February sales changed February's revenue after it had gone to the board. The finance director had assumed closed months never change, the team had assumed they always do, and neither was written down.
- Applies to
- any · components: modelling, serving · patterns: any
- Owner
- Business analyst · also Business owner and sponsor, Data consumers, Technical anchor
- Evidence
- in: requirements, decisions
looks like: A short restatement policy: which date assigns a record to a period, how far back corrections may change published figures, when a period closes, and how users are told that a past figure changed. - How the gap shows
- The requirements never mention late data, corrections, restatement, closing or period end.
- The date that decides the reporting period is not stated for any measure.
- Late and corrected data is marked as a technical matter only.
- Compare
- Compare the restatement policy in the requirements with the late-data handling in the architecture: if the architecture rebuilds past periods but the business expects closed periods to be frozen, or the reverse, one of them is wrong.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 7: Test - Fail fast
- Show the business owner one real correction from last month and ask which month it should change and whether anyone must be told.
- Related
- LATE-01, LATE-04, SCOPE-02
DATA-06No source of truth when systems disagreehighBusiness analyst
- Question
- When two sources hold the same thing differently (a customer's address in the sales system and the billing system, a product's price in two catalogues), which one wins, for which fields, and who decided? When a user finds the two disagree after go-live, which written decision answers them?
- Why it matters
- The sales and billing systems had different customer segments. The team picked one, the business assumed the other, and every segment report was disputed until the decision was made at steering level.
- Applies to
- any · components: transformation, modelling · patterns: any
- Owner
- Business analyst · also Business owner and sponsor, Source system owners, Technical anchor
- Evidence
- in: decisions, data-model
looks like: For each shared business entity, a recorded decision on which source wins for which fields, what happens when a record exists in only one, and who owns the decision. - How the gap shows
- The same entity (customer, product, employee, site) comes from more than one source with no rule for which wins.
- Phrases such as "merge customer data" or "single customer view" with no matching rule.
- The decision log has no entry about sources of truth.
- Compare
- Compare the source list with the decision log: every entity supplied by two or more sources needs a decision recorded; any without one will be decided silently in code.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off - Fail fast
- Pull twenty customers that exist in both systems, show the differences to the business owner, and ask which column wins for each field.
- Related
- DATA-01, OWN-01, DUP-03
DATA-07As it was, or as it is now, not decidedmediumBusiness analyst
- Question
- When something changes over time (a customer moves region, a product changes category, a salesperson changes team), should past figures be reported under the old value or the new one, and for which reports? When last year's numbers move because of a reorganisation, will the business call it a bug?
- Why it matters
- After a sales reorganisation, last year's revenue by region changed overnight because history was shown under the new regions. Finance needed it as it was originally reported, and the model had to be rebuilt.
- Applies to
- any · components: modelling, serving · patterns: any
- Owner
- Business analyst · also Data consumers, Technical anchor
- Evidence
- in: requirements, data-model
looks like: For each attribute that can change (region, category, team, status), whether reports should show the value at the time, the current value, or both, with a worked example the business has agreed. - How the gap shows
- The requirements never say whether history is shown "as it was" or "as it is now".
- Hierarchies or organisation structures are in scope with no mention of reorganisations.
- The data model keeps history of changes for attributes the requirements never ask about, or the reverse.
- Compare
- Compare the requirements with the data model: an attribute the business wants "as it was" that the model stores only as the latest value has lost its history the day it changes.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 7: Test - Fail fast
- Take one real customer who changed region last year and ask the users which region last year's sales should appear under.
- Related
- SCD-01, SCD-04
Each metric fully defined, departmental conflicts resolved, a business glossary, roll-up rules (4)
MET-01Metric defined by name, not by formulahighBusiness analyst
- Question
- Does every key metric have a complete definition: the formula, what is included and excluded, the grain, which date puts it in a period, the calendar used, the currency or unit and how amounts are converted? If two developers built it separately from the definition, would they get the same number?
- Why it matters
- "Net sales" was defined as "sales minus returns". It did not say whether tax, discounts or shipping were included, or which exchange rate applied, so the number came out three different ways in three reports.
- Applies to
- any · components: modelling, serving · patterns: any
- Owner
- Business analyst · also Data consumers, Technical anchor, Test lead
- Evidence
- in: data-model, requirements
looks like: A definition card per metric: formula, inclusions and exclusions, grain, the date that decides the period, calendar (fiscal or calendar), currency or unit and conversion rule, a worked example with real numbers, an owner and a sign-off date. - How the gap shows
- Metrics are listed by name with a one-line description and no formula.
- Definitions never mention tax, discounts, returns, cancellations, test accounts or currency.
- No metric has a worked example with numbers.
- Compare
- Compare the metric definitions in the data model with the calculations described in report specifications or test cases: a calculation that adds a filter or rule the definition does not have means the definition is incomplete.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 7: Test - Fail fast
- Give the definition of one key metric to two people separately, ask each to calculate it from the same sample, and compare.
- Related
- REQ-03, SERV-04, MET-02
MET-02Conflicting definitions across departments left hiddenhighBusiness analyst
- Question
- Have the departments that use the same term (customer, revenue, headcount, active) been asked for their definition, and are the differences written down and resolved, either into one definition or into clearly different names? When sales and finance show different numbers for the same word, has that argument already happened?
- Why it matters
- Marketing counted an active customer as anyone who opened an email in ninety days; finance as anyone who paid in twelve months. The dashboard used finance's rule, marketing stopped trusting it, and built their own.
- Applies to
- any · components: modelling, serving · patterns: any
- Owner
- Business analyst · also Business owner and sponsor, Data consumers
- Evidence
- in: data-model, decisions
looks like: For each shared term, the definition each department uses today, the differences, and the decision: one shared definition, or separate named measures (for example "booked revenue" and "invoiced revenue"). - How the gap shows
- Only one department was consulted on definitions that several departments use.
- The same word appears in requirements from different teams with no note on whether it means the same.
- No decision log entry records a resolved definition conflict.
- Compare
- Compare the stakeholder map with the definition sign-offs: a department that uses a term but did not sign its definition has not agreed to it.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off - Fail fast
- Email the top five terms to one person in each department that uses them, ask for their definition in one sentence, and put the answers side by side.
- Related
- REQ-03, STK-04, OWN-02
MET-03No business glossarymediumBusiness analyst
- Question
- Is there one glossary of the business terms used in the requirements, reports and data model, each with a plain definition and an owner, kept where users and the team both look? When a new analyst asks what a term means, is there one place to point them?
- Why it matters
- Three documents used "order", "booking" and "sale" for overlapping things. The developers treated them as the same, the users did not, and the confusion only came out in user testing.
- Applies to
- any · components: modelling, serving, governance · patterns: any
- Owner
- Business analyst · also Data consumers, Technical anchor
- Evidence
- in: data-model, requirements
looks like: A glossary listing each business term, its plain definition, synonyms people use, the owner, and the fields or metrics it maps to. - How the gap shows
- Requirements use several words for what may be the same thing (order, booking, sale).
- Abbreviations and internal jargon appear with no definition.
- The data dictionary describes columns but not business terms.
- Compare
- Compare the words used in the requirements with the names in the data model: a business term with no match in the model, or a model name no business person would recognise, shows the two speak different languages.
- Due by
plan stage 3: Discovery designed stage 5: MVP starts - Fail fast
- List every noun in the top ten requirements and ask the business owner which are the same thing.
- Related
- MET-01, HAND-02
MET-04Totals that cannot simply be added upmediumBusiness analyst
- Question
- For each metric that is a count of distinct things, a ratio, an average, a percentage or a balance at a point in time, has the business said how it rolls up across days, regions and products? When a user adds up the regional figures and gets a different total, will the definition explain why?
- Why it matters
- Daily unique customers were summed to give a monthly figure, counting a customer who bought on ten days ten times. Margin percentages were averaged instead of recalculated. Both errors reached the board pack.
- Applies to
- any · components: modelling, serving · patterns: any
- Owner
- Business analyst · also Data consumers, Technical anchor, Test lead
- Evidence
- in: data-model, requirements
looks like: Each metric marked as additive, partly additive (for example a stock balance that adds across stores but not across days) or not additive, with the rule for totals and a worked example. - How the gap shows
- Ratios, averages, distinct counts or balances are defined with no rule for totals.
- Requirements ask for the same metric "by day, week and month" with no word on how it rolls up.
- Test cases check only the lowest level, not totals.
- Compare
- Compare the metric definitions with the report designs: a report that shows totals or subtotals for a metric with no roll-up rule will show a number nobody defined.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 7: Test - Fail fast
- For each ratio or distinct count, calculate the monthly total two ways on a real sample and show the business owner the difference.
- Related
- MET-01
Business rules as testable rules with examples, exceptions, and what users see when data is bad or missing (3)
RULE-01Business rules written as prose, not as testable ruleshighBusiness analyst
- Question
- Is every business rule (how a customer is classified, when an order counts, how a discount is applied) written as a precise rule with at least one worked example of input and expected result? If a tester and a developer read the rule separately, would they agree on the answer for a tricky case?
- Why it matters
- The rule said "VIP customers get priority". Nobody wrote down that VIP meant spend over a threshold in the last twelve rolling months, so developers used the calendar year and a fifth of VIPs were missing.
- Applies to
- any · components: transformation, modelling · patterns: any
- Owner
- Business analyst · also Test lead, Developers, Data consumers
- Evidence
- in: requirements, test-strategy
looks like: Each rule numbered, stated as a condition and a result, with two or three worked examples using real values, including at least one edge case, and the person who confirmed it. - How the gap shows
- Rules are embedded in long paragraphs rather than listed and numbered.
- Rules have no examples, or only the obvious case.
- Words such as "usually", "normally", "where appropriate" or "as needed" appear in rules.
- Compare
- Compare the business rules in the requirements with the test cases: a rule with no test, or a test that checks something no rule states, shows the rules were never made testable.
- Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test - Fail fast
- Pick the three most complex rules and ask the business owner to work one real record through each by hand.
- Related
- REQ-02, RULE-03
RULE-02What the user sees when data is bad or missing not decided by the businesshighBusiness analyst
- Question
- When a value is missing, a record fails a check, or a source is late, what should the user see: a blank, a placeholder such as "unknown", yesterday's figure, a warning, or nothing at all? Has the business chosen, for each key report? If a manager makes a decision on incomplete data, will the report have told them?
- Why it matters
- A store's sales file failed to load. The dashboard showed the store at zero without any warning, and a regional manager phoned the store manager to ask why nothing had been sold.
- Applies to
- any · components: modelling, serving · patterns: any
- Owner
- Business analyst · also Data consumers, Technical anchor
- Evidence
- in: requirements
looks like: For each key report or consumer, what is shown for missing values, rejected rows, late sources and partial periods, agreed with the users, with how they are told the data is incomplete. - How the gap shows
- The requirements describe only the happy path.
- Rules for bad data exist in the technical design but not in the requirements.
- Words such as "unknown", "missing", "incomplete", "partial" or "late" never appear in the requirements.
- Compare
- Compare the requirements with the data quality design: if the design quarantines or drops rows but the requirements never say what users should see when that happens, users will see silent gaps.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 7: Test - Fail fast
- Show users a mock-up of a key report with one region missing and ask what they expect it to look like.
- Related
- DQ-02, SERV-02, DQ-04
RULE-03The exceptions nobody mentionedmediumBusiness analyst
- Question
- Have the unusual cases been asked about and given rules: returns, cancellations, refunds, credit notes, intercompany sales, test and internal accounts, free items, split shipments, merged customers? Which exception will the users spot in the first week of testing?
- Why it matters
- Nobody asked about staff purchases. They were twelve per cent of transactions at one site, were excluded from every report the business trusted, and were included in every number the new system produced.
- Applies to
- any · components: transformation, modelling · patterns: any
- Owner
- Business analyst · also Data consumers, Source system owners
- Evidence
- in: requirements, discovery
looks like: A list of exception types found in the data profile and in interviews, each with how often it occurs, and a rule for whether and how it counts in each metric. - How the gap shows
- No requirement mentions returns, cancellations, internal accounts, test records or adjustments.
- The data profile shows unusual record types or status codes that no requirement explains.
- Rules cover only the standard case.
- Compare
- Compare the status codes and record types in the data profile with the business rules: every code with no rule will be treated by default, and the default is rarely what the business wants.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off - Fail fast
- List every distinct status or type code in the main source and ask the business owner what each one means and whether it counts.
- Related
- RULE-01, DQ-05, CUR-04
Named business owners for data domains, definitions and mappings, and who approves changes to meaning (3)
OWN-01No named business owner for each data domainhighBusiness analyst
- Question
- For each data domain the project uses (customer, product, finance, employee), is there a named business person who decides what the data means, what good quality is, and who may use it? When the team has a question about a field, who answers, and how long does it take?
- Why it matters
- Product data had no business owner. Every question about categories went to the source system's technical team, who did not know, and the product hierarchy was argued about until the week before go-live.
- Applies to
- any · components: any · patterns: any
- Owner
- Business analyst · also Business owner and sponsor, Security and governance
- Evidence
- in: stakeholders, data-model
looks like: A table of data domains, each with a named business owner, a day-to-day steward if different, what they decide, and their agreement to the role. - How the gap shows
- Data owners are listed as teams or systems rather than people.
- The technical owner of a source system is named as the owner of the business meaning of its data.
- Some domains in scope have no owner, or the owner is "to be confirmed".
- Compare
- Compare the data domains in the data model with the stakeholder map: a domain whose owner is not on the stakeholder map will not be consulted when its meaning is decided.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off - Fail fast
- List the domains and ask the sponsor to put one name against each in a ten-minute call, then ask each named person to confirm.
- Related
- STK-02, DATA-01, OWN-02
OWN-02Nobody approves changes to a definitionmediumBusiness analyst
- Question
- After go-live, who may change a business definition or rule, how is the change requested, who approves it, and how are users told that past figures will move? When a manager asks the team to "just exclude" something, who has to say yes first?
- Why it matters
- A regional director asked a developer to exclude one large customer from the margin report. The change went in quietly, the group margin moved, and finance found out when the board asked why.
- Applies to
- any · components: modelling, serving, governance · patterns: any
- Owner
- Business analyst · also Business owner and sponsor, Data consumers, Operations and support
- Evidence
- in: decisions, runbook
looks like: A definition change process: who can request, who approves (the definition's owner), how other users are consulted, how the change is versioned and dated, and how users are told. - How the gap shows
- Definitions have owners but no process for changing them.
- The support model handles technical changes but says nothing about changes to meaning.
- Change requests to metrics go straight to developers.
- Compare
- Compare the definition owners in the data model with the support model's change process: if the support model lets the team change logic without the definition owner's approval, ownership exists only on paper.
- Due by
plan stage 4: Discovery sign-off designed stage 6: Build - Fail fast
- Ask the support lead: "If a director asks you tomorrow to change how revenue is counted, what do you do?"
- Related
- REQ-03, SERV-03, MET-02, REQ-10
OWN-03Mappings and hierarchies with no business maintainermediumBusiness analyst
- Question
- Who in the business maintains the mappings and hierarchies the reports depend on (product to category, store to region, account to reporting line, cost centre to department), how do they change them, and how quickly? When a new product or store appears, who notices it is unmapped?
- Why it matters
- New products launched every month. The mapping to categories lived in a spreadsheet the analyst who left had maintained, so new products fell into "unknown" and the category report drifted away from reality.
- Applies to
- any · components: transformation, modelling, governance · patterns: any
- Owner
- Business analyst · also Data consumers, Operations and support
- Evidence
- in: requirements, runbook
looks like: A list of business-maintained mappings and hierarchies, each with a named maintainer, where it is edited, how often it changes, and what happens to records that are not yet mapped. - How the gap shows
- Mappings or hierarchies are described as "reference data" with no owner.
- Mappings are kept in personal spreadsheets or e-mailed files.
- Nothing says what happens to a new code that is not in the mapping.
- Compare
- Compare the reference data in the data model with the support model: a mapping the reports depend on that no one in the support model maintains will go stale.
- Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test - Fail fast
- Count how many codes in last month's source data are missing from today's mapping, and ask who would fix them.
- Related
- OWN-01, RULE-02, CUR-04
Every consumer and channel named with what it needs, and self-service expectations (2)
USE-01Not every consumer and channel namedhighBusiness analyst
- Question
- Is every consumer of the output named, with the channel they use (dashboard, scheduled extract, file to a partner, application interface, machine learning model, AI assistant answering questions) and what each needs from it? Who will turn up at go-live asking where their feed is?
- Why it matters
- The project built dashboards for the sales team. Two weeks before go-live the pricing team said their model read a nightly extract from the old system, which nobody had listed, and the cut-over was postponed.
- Applies to
- any · components: serving · patterns: any
- Owner
- Business analyst · also Data consumers, Technical anchor
- Evidence
- in: requirements, stakeholders
looks like: A consumer list: one row per consumer or channel with who they are, how they receive the data, what data and grain they need, how fresh, and any needs such as languages, accessibility or mobile use. - How the gap shows
- Requirements describe only dashboards or reports, with no extracts, interfaces, models or assistants.
- Downstream systems that read today's data are not listed.
- Consumers are named as "the business" or "users" without groups.
- Compare
- Compare the consumer list in the requirements with the outputs drawn in the architecture: an output with no named consumer is unowned work, and a consumer with no output will be missed at go-live.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off - Fail fast
- Ask the source and current-report owners for the list of everyone who receives or queries today's data, and check each one against the requirements.
- Related
- SERV-01, REQ-05, CUR-01, RLC-05
USE-02Self-service expectations not statedmediumBusiness analyst
- Question
- Does the business expect users to build their own reports and answer their own questions, and if so, which users, with what skills, on which data, with what training and support? If users say at go-live "we thought we could build our own", will that be in scope?
- Why it matters
- The sponsor expected analysts to explore the data freely. The team built fixed reports only, with no documented model for users to build on, and the analysts went back to exporting to spreadsheets.
- Applies to
- any · components: serving, modelling · patterns: any
- Owner
- Business analyst · also Business owner and sponsor, Data consumers
- Evidence
- in: requirements, discovery
looks like: A clear statement of who will only read reports and who will build their own, what data the builders may use, what training and documentation they will get, and who supports them. - How the gap shows
- Words such as "self-service", "explore" or "ad hoc" appear with no named users or scope.
- No training, documentation or support for business users is in the plan.
- The out-of-scope list says nothing about self-service.
- Compare
- Compare the self-service promises in the discovery document with the plan and handover: self-service promised but no user documentation, training or support planned will not happen.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off - Fail fast
- Ask the sponsor to name three people who will build their own reports, and ask those three what they plan to build.
- Related
- USE-01, SCOPE-01
Privacy purpose, retention and audit needs stated by the business (the tech pack covers implementation) (3)
POL-01Use of personal data not confirmed as allowedhighBusiness analyst
- Question
- For each use of personal data in the requirements, has the business confirmed with its privacy or legal team that the use is allowed: the purpose, any consent it depends on, and who may see it? If the privacy team stops the project a month before go-live, will it be over a use nobody asked them about?
- Why it matters
- The marketing dashboard joined customer purchases to support-call notes. Customers had never been told their call notes would be used for marketing, and the privacy officer stopped the release in user testing.
- Applies to
- any · components: any · patterns: any
- Owner
- Business analyst · also Security and governance, Business owner and sponsor
- Evidence
- in: requirements, decisions
looks like: A list of the personal data each requirement needs, the business purpose, the basis that allows it (consent, contract, legal duty), who may see it, and the privacy team's recorded answer. - How the gap shows
- Requirements need customer, employee or patient data with no mention of purpose or permission.
- The privacy or legal team is not on the stakeholder map.
- Personal data is treated only as a security or masking question.
- Compare
- Compare the personal data named in the requirements with the stakeholder map and decision log: personal data in use with no privacy sign-off recorded is a go-live blocker waiting to be found.
- Due by
plan stage 3: Discovery evidenced stage 4: Discovery sign-off - Fail fast
- Send the privacy team a one-page list of the personal data and its purposes this week and ask for a yes, a no or a condition.
- Related
- SEC-01, SEC-05, STK-01
POL-02Retention needs not stated by the businessmediumBusiness analyst
- Question
- For each kind of data, how long does the business need to keep it, how long does law or policy require it to be kept, and when must it be deleted? When a user asks for seven-year trends and policy says delete after three, who decides?
- Why it matters
- The business asked for ten years of customer history. Company policy required customer data to be deleted three years after the relationship ended, and the history load had to be cut back after it was done.
- Applies to
- any · components: governance, modelling · patterns: any
- Owner
- Business analyst · also Security and governance, Business owner and sponsor
- Evidence
- in: requirements, decisions
looks like: A retention table per data kind: the business need, the legal or policy minimum and maximum, the agreed period, and who signed it, including what happens to aggregated figures after detail is deleted. - How the gap shows
- Retention is not mentioned in the requirements, or is "to be agreed".
- The history requested is longer than any retention period stated.
- No one from records management, legal or compliance is named.
- Compare
- Compare the history depth requested in the requirements with the retention periods: history the business wants beyond the period policy allows must be resolved before it is loaded.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off - Fail fast
- Ask the records or compliance owner for the retention rule for the two most sensitive data kinds, and compare it with the history asked for.
- Related
- SEC-03, DATA-03
POL-03What must be proven to an auditor not statedmediumBusiness analyst
- Question
- Will auditors, regulators or internal control need to see how a reported figure was produced, reproduce a past figure exactly as it was reported, or see who changed what and when? Has the business written down those needs, so the design can meet them?
- Why it matters
- A regulator asked the bank to reproduce a capital figure reported six months earlier. The data had been corrected since, nobody had kept what was reported, and the bank could not show where the number came from.
- Applies to
- any · components: modelling, serving, governance · patterns: any
- Owner
- Business analyst · also Security and governance, Business owner and sponsor, Technical anchor
- Evidence
- in: requirements
looks like: A list of audit and control needs in business terms: which figures must be reproducible as reported and for how long, which changes need a record of who and when, and which reports need sign-off before release. - How the gap shows
- Figures go to regulators, auditors or external parties but no audit need is stated.
- The requirements never mention reproducing past figures, sign-off of published numbers, or change history.
- Audit is mentioned only as access logging.
- Compare
- Compare the audit needs in the requirements with the restatement policy: if published figures may be restated but must also be reproducible as reported, the design has to keep both versions.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 7: Test - Fail fast
- Ask the finance or compliance owner: "Which number from this system could an auditor ask us to reproduce in a year's time?"
- Related
- SEC-04, DATA-05
User acceptance designed early with real users, recognisable numbers, data, environment and sign-off (3)
UAT-01User acceptance scenarios designed late, without the usershighBusiness analyst
- Question
- Were the user acceptance scenarios written during discovery, with the users who will test, and built around the questions they really ask and the figures they already know (last month's total for their region, a customer they can name)? When testing starts, will the testers know what to check, or will they click around and say it looks fine?
- Why it matters
- The scenarios were written by the test team in the last week of the build, from the requirements alone. They checked that screens loaded, not that numbers were right, and the first month-end after go-live showed revenue a few per cent out in every region.
- Applies to
- any · components: modelling, serving · patterns: any
- Owner
- Business analyst · also Test lead, Data consumers
- Evidence
- in: test-strategy, requirements
looks like: A first set of acceptance scenarios agreed with named users by discovery sign-off, each one a real question with the expected figure from a trusted report for a known period, covering every user group and the tricky cases (returns, late data, reorganisations). - How the gap shows
- The test strategy says "user acceptance testing" with no scenarios, or "scenarios to be defined".
- Scenarios describe screens and clicks, not figures and expected values.
- Scenarios were written by the project team with no named business contributor.
- Compare
- Compare the acceptance scenarios in the test strategy with the decisions and user groups in the requirements: a decision or user group with no scenario will be accepted without being tested.
- Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test - Fail fast
- Ask two real users to write down three questions they will ask the new system and the answers they expect for last month; those are the first scenarios.
- Related
- TEST-04, REQ-02, CUR-02, RES-04
UAT-02No data or environment for user testingmediumBusiness analyst
- Question
- What data will business testers use (real enough for them to recognise their numbers, covering a period they know), in which environment, with what access, and is personal data in it allowed? If testers cannot log in on the first day, or see numbers they cannot judge, how many testing days are lost?
- Why it matters
- User testing used invented data so that no personal data left production. The testers could not tell whether any figure was right, signed off out of politeness, and the first real month showed the errors.
- Applies to
- any · components: modelling, serving · patterns: any
- Owner
- Business analyst · also Test lead, Platform team, Security and governance
- Evidence
- in: test-strategy, platform
looks like: A statement of which period and data the testers will use, how it is made safe if it holds personal data, which environment they use, and that their access was requested and tested before testing starts. - How the gap shows
- The test strategy says user testing will use "test data" or "sample data" with no period named.
- Business testers' access to the test environment is not in the plan.
- Nothing says whether personal data may be used in user testing.
- Compare
- Compare the user testing approach in the test strategy with the platform environments: user testing that needs production-like data in an environment that is not allowed to hold it cannot happen as planned.
- Due by
plan stage 4: Discovery sign-off designed stage 6: Build evidenced stage 7: Test - Fail fast
- Ask one business tester to log in to the test environment this week and find last month's figure for their own area.
- Related
- TEST-02, ENV-01, POL-01
UAT-03Acceptance signed by the wrong people, or with no exit rulehighBusiness analyst
- Question
- Will acceptance be signed by the people who will use the output, and is it agreed in advance what counts as accepted: which differences from the trusted figure are tolerable, and which open defects may go live? If testing ends with twenty open defects, is the go or no-go already decided?
- Why it matters
- The project manager and the sponsor signed acceptance. The analysts who used the reports daily had found differences they could not explain, were not asked, and quietly went back to the old spreadsheets.
- Applies to
- any · components: modelling, serving · patterns: any
- Owner
- Business analyst · also Business owner and sponsor, Data consumers, Test lead
- Evidence
- in: test-strategy, decisions
looks like: Named acceptors from each user group, exit rules agreed before testing (tolerances for differences, severity of defects that may remain), and a dated sign-off record. - How the gap shows
- Acceptance is signed only by the sponsor or the project team.
- No exit criteria or tolerances are stated for user testing.
- The sign-off template has no place to record open defects or conditions.
- Compare
- Compare the acceptors named in the test strategy with the user groups in the requirements: a user group with no acceptor has not accepted anything.
- Due by
plan stage 4: Discovery sign-off designed stage 6: Build evidenced stage 7: Test - Fail fast
- Write the exit rule in three lines now and get the business owner and one real user to agree to it.
- Related
- REC-03, STK-05, REQ-05, RLC-01
Project management (64)
How the work runs: plan and dependencies, risks, decisions, budget, stakeholders, communication, governance, go-live and benefits.
Estimates and their assumptions (1)
EST-06Never corrected against actual progresshighTechnical anchor
- Question
- Is actual effort compared with the estimate at least every two weeks, and has the estimate been changed when the real rate differed? If we finish late, will we have known it two months before or two days before?
- Why it matters
- The thin slice took three times longer than planned per table. Nobody re-estimated, and the gap only showed when the last month of the plan arrived with half the tables still to build.
- Applies to
- any · components: any · patterns: any
- Owner
- Technical anchor · also Project manager
- Evidence
- in: estimate, status
looks like: A re-estimate after the thin slice and then every two weeks, showing items done, the real rate, and the forecast finish, with changes explained. - How the gap shows
- The estimate document has one version and its date is before build started.
- Status reports show percent complete but never a forecast end date or rate.
- No mention of a thin slice or of the rate it achieved.
- Compare
- Compare status reports with the estimate: if progress reports show fewer items finished than the estimate's rate predicts, but the end date never moves, the estimate is not being corrected.
- Due by
plan stage 4: Discovery sign-off evidenced stage 6: Build - Fail fast
- Divide items finished so far by weeks spent, and project the finish date from that rate today.
- Related
- EST-01, PM-04
In scope, out of scope, change control (2)
SCOPE-03Scope creep with no change processhighProject manager
- Question
- Is there an agreed way to handle a new request: who can raise it, how it is sized, who decides, and how the plan and estimate change? If the project ends late, how much of the lateness will be work that was added without anyone deciding to add it?
- Why it matters
- Each request was small and agreed in a meeting. After three months, a fifth of the work had never been estimated, the date had not moved, and the team was blamed for the slip.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Business owner and sponsor, Technical anchor
- Evidence
- in: plan, decisions
looks like: A written change process and a change log where each change shows the request, its size, its effect on date and cost, the decision and who made it. - How the gap shows
- No "change control", "change request" or "change process" in the plan.
- Status reports mention new items that do not appear in the original scope or in any change log.
- Meeting notes include "can we also add" with no follow-up decision.
- Compare
- Compare the latest status report or backlog with the signed scope: items in the backlog that are not in the scope and have no change record are unmanaged creep.
- Due by
plan stage 4: Discovery sign-off evidenced stage 6: Build - Fail fast
- Count backlog items added since sign-off and check how many have a recorded decision.
- Related
- SCOPE-04, PM-03
SCOPE-05MVP 1 is too bigmediumProject manager
- Question
- Is the first MVP the smallest slice that gives real value and proves the hardest parts work end to end? If the first release slips by months, will it be because it tried to do everything at once?
- Why it matters
- MVP 1 contained eight sources and every report. Nothing reached production for nine months, so no real feedback arrived until it was too late to change direction.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Technical anchor, Business owner and sponsor
- Evidence
- in: plan
looks like: An MVP 1 that can reach production within a few months, chosen for value and for testing the riskiest beliefs, with later MVPs listed in order of dependency and value. - How the gap shows
- MVP 1 contains most or all of the in-scope items.
- The first production release date is more than about three months after build starts.
- The reason for the MVP order is not written down.
- Compare
- Compare the plan's MVP 1 with the estimate: if MVP 1 holds most of the estimated effort, it is the whole project with a different label.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off - Fail fast
- Ask what could be removed from MVP 1 while it still gives one group of users a real answer.
- Related
- SCOPE-04, PM-05
Plan, dependencies, risks, decisions, status (18)
PM-01Dates set before the dependencies were knownhighProject manager
- Question
- Were the milestones set after the dependencies (access, environments, source changes, other teams, approvals) were listed and confirmed with the people who own them? If the first milestone slips, will it be because of something outside the team that nobody had asked about?
- Why it matters
- The go-live date was announced first. The source team's change freeze, the security review and the network change each added weeks, and none had been asked about when the date was set.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Technical anchor, Source system owners, Platform team
- Evidence
- in: plan, raid
looks like: A dependency list with each owner, the date they committed to, and the plan tasks that wait on it. - How the gap shows
- Milestone dates appear in the brief or charter before any dependency list exists.
- The plan has no dependencies, or lists them without owners or dates.
- External teams named in the architecture do not appear in the plan.
- Compare
- Compare the architecture with the plan: every system or team the architecture relies on should appear as a dependency in the plan; one that does not is an unplanned wait.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off - Fail fast
- Phone the owner of the biggest external dependency and ask for the date they can deliver it, in writing.
- Related
- PM-07, PM-10
PM-02Risks listed, not managedhighProject manager
- Question
- Does each top risk have an owner, a trigger that tells us it is happening, a mitigation that is being done, and a date it was last reviewed? Looking back after a failure, will the cause be on the risk log with nothing done about it?
- Why it matters
- "Source data quality may be poor" sat on the risk log for six months with no owner. It was exactly the reason the project missed its date.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Technical anchor
- Evidence
- in: raid
looks like: A short top-risks list where each risk has an owner, a trigger, a mitigation with a due date, and a recent review date. - How the gap shows
- Risks with no owner, or with a team rather than a person as owner.
- No triggers or mitigation actions, only descriptions.
- The risk log has not changed in several weeks while status reports describe new problems.
- Compare
- Compare the RAID log with status reports: problems in the status reports that were never on the risk log, or risks that became issues without the log changing, mean the log is not used.
- Due by
plan stage 3: Discovery evidenced stage 6: Build - Fail fast
- Take the top three risks and ask each owner what they did about it this week.
- Script signal
risks-without-owner(found without AI)- Related
- PM-04, EST-03
PM-03Decisions not written downhighProject manager
- Question
- Is there a decision log recording the options, the choice, the reason, who decided and when? When a decision is reopened by someone who was not in the room, what will we show them?
- Why it matters
- The team agreed to keep only the latest customer address. Three months later a new manager asked why history was missing, and with no record the decision was reversed and the work redone.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Technical anchor, Architect
- Evidence
- in: decisions
looks like: A decision log or short decision records: context, options, decision, reason, owner, who signed, date. - How the gap shows
- No decision log or decision records exist among the project documents.
- Meeting notes contain "agreed" or "decided" but no decision log entry matches.
- Architecture choices are stated with no record of the options considered.
- Compare
- Compare meeting notes or status reports with the decision log: a decision mentioned in notes but not in the log is one that can be lost.
- Due by
plan stage 3: Discovery evidenced stage 6: Build - Fail fast
- List the last five important choices from memory and check how many are in the log.
- Script signal
no-decision-log(found without AI)- Related
- PM-08, COM-03, HAND-02
PM-04Green until it is redhighProject manager
- Question
- Do status reports show amber or red when something is slipping, with the reason and the next step? If the project fails suddenly, will the reports from the month before have shown it as green?
- Why it matters
- Status stayed green for four months while the team quietly fell behind. It went red two weeks before go-live, when there was nothing left to do but move the date.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Technical anchor, Business owner and sponsor
- Evidence
- in: status
looks like: Status reports where the colour is based on stated rules (for example forecast versus baseline), and where amber appears early with a recovery action. - How the gap shows
- Every status report is green.
- The status colour does not change while the forecast date or open issues change.
- No forecast finish date, only activities done.
- Compare
- Compare status reports with the RAID log and the estimate: a green status alongside rising issues, a rate below the estimate, or slipping dependencies means the colour is not telling the truth.
- Due by
plan stage 4: Discovery sign-off evidenced stage 6: Build - Fail fast
- Write down the rule for red, amber and green today and apply it to the last report.
- Script signal
status-always-green(found without AI)- Related
- PM-02, EST-06, COM-01
PM-05Testing and deployment left to the endhighProject manager
- Question
- Does the plan have testing and deployment to a production-like environment starting in the first weeks of build, rather than after it? If go-live slips, will it be because the first real deployment happened a month before the date?
- Why it matters
- Everything was built, then tested, then deployed. The first deployment found network, permission and performance problems that took longer to fix than the build had taken.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Technical anchor, Test lead, Release and DevOps
- Evidence
- in: plan, test-strategy, deployment
looks like: A test strategy and a release approach agreed in discovery, a first deployment of a thin slice in the first weeks, and "done" defined as tested and deployed. - How the gap shows
- No test strategy or deployment document exists, or it says "to be defined".
- The plan has testing and deployment as single phases after build.
- The definition of done mentions coding but not testing or deployment.
- Compare
- Compare the plan with the test strategy: if the strategy describes continuous testing but the plan has one test phase at the end, the plan wins and testing will be squeezed.
- Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 6: Build - Fail fast
- Deploy the thin slice to the test environment this week, through the real release route.
- Script signal
missing-doc-type(found without AI)- Related
- EST-04, SCOPE-05
PM-06Key-person riskmediumProject manager
- Question
- Is there any part of the project that only one person understands or can do, and are the people the plan needs named and available when it needs them? If one person leaves next month, what stops?
- Why it matters
- One developer understood the source system. When they moved to another project mid-build, the team lost six weeks rediscovering what they knew.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Technical anchor
- Evidence
- in: plan, raid
looks like: Named people for each role with their availability, a second person for every critical area, and key-person risks on the RAID log. - How the gap shows
- The same person's name appears as owner for most technical items.
- Roles are listed as "to be hired" or "to be confirmed" for work in the next stage.
- No mention of availability, leave or shared allocation.
- Compare
- Compare the plan with the stakeholder map or team list: a plan that needs a role the team list does not have, or a person allocated part-time who carries full-time tasks, will slip.
- Due by
plan stage 3: Discovery evidenced stage 6: Build - Fail fast
- Ask each team member which area they could not cover if a colleague were away for a month.
- Related
- HAND-03
PM-07The plan ignores the estimate's assumptionsmediumProject manager
- Question
- Does the plan use the same assumptions as the estimate (start of access, team size, environment availability, source readiness)? When the plan slips, will it be because it assumed something the estimate never did?
- Why it matters
- The estimate assumed access in week three; the plan assumed week one. Nobody compared them, and the plan was two weeks behind before work started.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Technical anchor
- Evidence
- in: plan, estimate
looks like: A plan that lists the estimate's key assumptions and shows where each one sits on the timeline. - How the gap shows
- The plan does not mention the estimate or its assumptions.
- The plan's team size or start dates differ from the estimate's.
- Compare
- Compare the estimate's assumptions with the plan's dates and staffing: any mismatch is a slip already built in.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off - Fail fast
- Put the estimate's assumptions next to the first four weeks of the plan and look for disagreements.
- Related
- EST-03, PM-01
PM-08Open decisions with no owner, deadline or defaultmediumProject manager
- Question
- Does every open decision have an owner, a date by which it is needed, and a default that applies if nobody decides? If the team is blocked next month, will it be waiting on a decision nobody owns?
- Why it matters
- The choice between daily and hourly loads stayed open for ten weeks because nobody owned it. The team built for both, and later threw half away.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Business owner and sponsor, Technical anchor
- Evidence
- in: decisions, raid
looks like: An open-decisions list with the question, options, owner, the date needed, the default and the effect of deciding late. - How the gap shows
- Open questions listed with no owner or due date.
- Items the build depends on are marked "TBD" or "to be confirmed".
- The same open question appears in several consecutive status reports.
- Compare
- Compare the open decisions with the plan: a task scheduled to start before the decision it depends on is due will either stall or be redone.
- Due by
plan stage 3: Discovery evidenced stage 5: MVP starts - Fail fast
- Give every open decision an owner and a default today, and tell the owners the default applies on the due date.
- Related
- PM-03, STK-02
PM-09Discovery with no timeboxmediumProject manager
- Question
- Does discovery have a fixed length, a cost, and a list of exit criteria? If discovery runs for months or is squeezed into days, which risks will we carry into the build without having tested them?
- Why it matters
- Discovery was squeezed into three days to "start building faster". The riskiest beliefs were never tested, and they surfaced one by one during the build.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Technical anchor, Business owner and sponsor
- Evidence
- in: plan, discovery
looks like: A discovery plan of a few weeks with weekly goals, its own estimate, named reviewers and written exit criteria. - How the gap shows
- No discovery stage in the plan, or no end date for it.
- Discovery has no estimate of its own.
- No exit criteria for discovery sign-off.
- Compare
- Compare the discovery exit criteria with the discovery document: criteria that are not met but sign-off still happened show the timebox beat the evidence.
- Due by
plan stage 2: Rough estimate evidenced stage 4: Discovery sign-off - Fail fast
- Write the discovery exit criteria now and agree the end date with the sponsor.
- Related
- EST-01
PM-10Access and environments treated as adminhighProject manager
- Question
- Were access to sources, environments and tools requested in the first week and tracked as dependencies with owners and dates? If the build starts late, will it be because the team waited weeks for access?
- Why it matters
- Access requests were raised when build started. Approvals took six weeks, so the team spent the first sprints on paper designs that the real data later contradicted.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Technical anchor, Platform team, Security and governance
- Evidence
- in: raid, plan
looks like: A list of every access and environment request with the date raised, the approver, the expected date and its current state. - How the gap shows
- No access or environment requests are listed in the plan or RAID log.
- Access is described as "assumed" or "will be arranged".
- Discovery has no real data extract because access had not arrived.
- Compare
- Compare the plan's start of build with the access request dates: a build that starts before access is expected is planned to wait.
- Due by
plan stage 2: Rough estimate evidenced stage 3: Discovery - Fail fast
- Raise every access request today and record the date each one arrives.
- Related
- PM-01, EST-04
PM-11History load not on the plan as real taskshighProject manager
- Question
- Is the history load on the plan as its own tasks, with the elapsed time it needs (source extract windows, several trial runs, checking the old numbers), not folded into "build"? If go-live slips, will it be because the history load turned out to be a project of its own?
- Why it matters
- The plan gave the history load two days before go-live. The source could only be read at night, the first run found years with a different code list, and the full load took five weekends.
- Applies to
- any · components: ingestion · patterns: history load
- Owner
- Project manager · also Technical anchor, Source system owners
- Evidence
- in: plan, deployment
looks like: Plan tasks for the history extract, at least one full trial run, checking the old periods with the business and the final load, each with elapsed time, source windows and an owner. - How the gap shows
- The history load appears only in the architecture, not in the plan.
- The plan has one task, or a few days, for loading many years of history.
- No trial run of the history load is planned before the cut-over.
- Compare
- Compare the history load approach in the architecture or deployment plan with the plan's tasks: an approach that needs night windows and several runs, given a couple of days in the plan, will slip.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 7: Test - Fail fast
- Time a full extract of one year of history from the source now and multiply by the years in scope.
- Related
- HIST-01, HIST-02, EST-07
PM-12Data fixes with no owner and no timehighProject manager
- Question
- When profiling or testing finds bad data at the source (missing keys, wrong codes, duplicate customers), is it agreed who fixes it, where (at the source, by the business, or in the pipeline), and is that time on someone's plan? Whose plan holds the weeks of cleansing that the profile is already hinting at?
- Why it matters
- Profiling showed that a fifth of customers had no region. Everyone assumed someone else would fix it; the business had no time booked, and user acceptance failed on regional totals six weeks before go-live.
- Applies to
- any · components: ingestion, transformation · patterns: any
- Owner
- Project manager · also Business owner and sponsor, Source system owners, Technical anchor
- Evidence
- in: plan, raid
looks like: A data issues list from profiling, each with the decision on where it is fixed, an owner, the effort, and matching tasks in the project plan or the source team's plan. - How the gap shows
- The data profile lists quality problems that appear nowhere in the plan or RAID log.
- Fixes are assigned to "the business" or "the source" with no name or date.
- No time for data fixing appears in the plan.
- Compare
- Compare the data profile or data quality findings with the plan: each problem that needs fixing, with no task and owner, is work that will appear in user acceptance instead.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 7: Test - Fail fast
- Take the top five data problems from the profile and get a named owner and a date for each this week.
- Related
- DQ-05, DQ-01, EST-07
PM-13Reconciliation and user acceptance planned as one passhighProject manager
- Question
- Does the plan allow for more than one cycle of reconciliation and user acceptance (test, find differences, explain or fix, test again), with business time booked for each cycle? If the first round finds the numbers do not match, is there any time left to fix them?
- Why it matters
- One week of user acceptance was planned. The first day showed revenue three percent off the ledger, the explanation took two weeks, and there was no second test window before the go-live date.
- Applies to
- any · components: modelling, serving · patterns: any
- Owner
- Project manager · also Test lead, Business owner and sponsor, Technical anchor
- Evidence
- in: plan, test-strategy
looks like: At least two reconciliation and acceptance cycles on the plan, with time between them to explain and fix differences, and business testers booked for each. - How the gap shows
- The plan has a single user acceptance window.
- Reconciliation is not a task in the plan, or sits inside "testing".
- No time between the end of user acceptance and go-live.
- Compare
- Compare the test strategy's reconciliation approach with the plan: a strategy that compares every layer with trusted figures, given one short window, will run out of time on the first difference.
- Due by
plan stage 4: Discovery sign-off designed stage 6: Build evidenced stage 7: Test - Fail fast
- Reconcile one month of one key number against the business's trusted figure now and time how long it takes.
- Related
- REC-02, REC-03, TEST-04, RES-04
PM-14Cut-over weekend not plannedhighProject manager
- Question
- Is there an hour-by-hour cut-over plan covering the business freeze, the final loads, the checks before users are let in, the point after which there is no going back, the rollback steps, and who is on call? At three in the morning on cut-over night, if the final load fails, who decides what happens next?
- Why it matters
- Cut-over was "the weekend after UAT". The final history load overran by ten hours, nobody had agreed a point to stop, and on Monday half the users saw new numbers while the rest still had the old reports.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Technical anchor, Release and DevOps, Operations and support, Business owner and sponsor
- Evidence
- in: deployment, plan
looks like: A cut-over runbook with timed steps and owners, the business freeze window, checks before opening to users, a go-back deadline and rollback steps, a call list, and at least one rehearsal. - How the gap shows
- No cut-over plan, or one line saying "deploy to production".
- No business freeze, rollback window or decision point is mentioned.
- No rehearsal of the cut-over is planned.
- Compare
- Compare the cut-over plan's timings with the history load and final load run times measured in test: a window shorter than the measured loads will overrun.
- Due by
plan stage 4: Discovery sign-off designed stage 7: Test evidenced stage 8: Deploy - Fail fast
- Write the cut-over as timed steps on one page and add up the hours against the window you have.
- Related
- REL-07, HIST-03, RLC-01
PM-15No parallel run periodmediumProject manager
- Question
- Where the new solution replaces an existing report or feed, is there a period when both run side by side, with a named person comparing them, a length that covers at least one month-end, and criteria for ending it? When the new number and the old number differ in week one, who decides which is right?
- Why it matters
- The old report was switched off on go-live day. In the first month-end the new figures differed from what finance expected, there was nothing left to compare against, and trust in the new report never recovered.
- Applies to
- any · components: serving, modelling · patterns: any
- Owner
- Project manager · also Business owner and sponsor, Data consumers, Technical anchor
- Evidence
- in: plan, deployment
looks like: A parallel run on the plan with dates covering a month-end, a person who compares old and new, a tolerance, a decision owner for differences and written exit criteria. - How the gap shows
- The old system or report is switched off on the go-live date.
- Parallel running is mentioned with no length, owner or exit criteria.
- The parallel run ends before the first month-end or quarter-end.
- Compare
- Compare the plan's parallel run dates with the business calendar: a parallel run that ends before the first month-end tests the easy days only.
- Due by
plan stage 4: Discovery sign-off designed stage 7: Test evidenced stage 9: Production sign-off - Fail fast
- Ask the business owner which month-end they would want to see matched before switching off the old report.
- Related
- REC-04, RLC-03, BUD-06
PM-16Team rolled off before the first month-endmediumProject manager
- Question
- Does the plan and budget keep the people who built the solution available through early-life support, including the first month-end, quarter-end or other peak that the data will meet? When the first month-end load breaks, are the developers who wrote it still on the project?
- Why it matters
- The developers moved to a new project the Monday after go-live. The first month-end load hit late corrections the tests never had, and support spent four days rebuilding knowledge the team could have given in an hour.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Technical anchor, Operations and support, Business owner and sponsor
- Evidence
- in: plan, runbook
looks like: Named people booked for early-life support, an end date after the first month-end or other peak, and the exit criteria for handing over fully to support. - How the gap shows
- The plan and budget end on the go-live date.
- Early-life support is mentioned but nobody is named or funded for it.
- Team members are already booked on other projects from go-live.
- Compare
- Compare the plan's end date and staffing with the support model's early-life support period: a support period that outlasts the team's booking has nobody to do it.
- Due by
plan stage 4: Discovery sign-off designed stage 7: Test evidenced stage 9: Production sign-off - Fail fast
- Check each developer's next booking date against the first month-end after go-live.
- Related
- HAND-04, HAND-06, OPS-05
PM-17Buffers not where data surprises clustermediumProject manager
- Question
- Does the plan hold explicit buffer time after the points where data projects usually find surprises: the first real extract, the first full load, the history load and the first reconciliation? Where in this plan would a two-week surprise from the data fit without moving go-live?
- Why it matters
- Every task was planned back to back with no slack. The first full load showed that keys were not unique, and with nowhere to absorb the fix, every later date moved by the same three weeks.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Technical anchor
- Evidence
- in: plan, raid
looks like: Named buffer tasks after the first real loads, the history load and reconciliation, sized to the risks in the RAID log, with a rule for who may use them. - How the gap shows
- No buffer appears in the plan, or one buffer sits at the very end.
- Tasks after the first full load and reconciliation are planned with no gap before go-live.
- The RAID log has data risks with no time set aside for them.
- Compare
- Compare the data risks in the RAID log with the plan: a high data risk that hits in the middle of the plan, with no buffer after it, will move every later date.
- Due by
plan stage 4: Discovery sign-off evidenced stage 6: Build - Fail fast
- Ask where a two-week slip after the first full load would land, and whether the go-live date survives it.
- Related
- PM-02, BUD-04, EST-02
PM-18The business calendar ignoredmediumProject manager
- Question
- Has the plan been checked against the business and data calendar: month-end and year-end close, audit, peak trading, source system change freezes and the platform's own release windows? Which key date in our plan falls in a week when the business or a source team cannot help us?
- Why it matters
- Go-live was set for the first week of January. Finance was closing the year, the source system was frozen for its own year-end, and the release had to wait until February.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Business owner and sponsor, Source system owners, Platform team
- Evidence
- in: plan, raid
looks like: Business close periods, peak periods and change freezes marked on the plan, with no testing, cut-over or go-live inside them unless agreed. - How the gap shows
- The plan shows no business calendar, freezes or close periods.
- Go-live, cut-over or user acceptance falls in a month-end, year-end or known freeze.
- Compare
- Compare the plan's key dates with source teams' and the platform's change freezes: a release planned inside a freeze will not be allowed to happen.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off - Fail fast
- Ask finance, the source owners and the platform team for their freeze and close dates for the next year.
- Related
- PM-01, RES-04, RLC-04
Who is involved, who decides, who can block (7)
STK-01Only the sponsor is managedhighProject manager
- Question
- Does the stakeholder map include the people outside the weekly meeting who can block go-live: source system owners, operations, security, the platform team and downstream users? Who could stop this project in its last month that we have not spoken to in its first?
- Why it matters
- The sponsor was kept happy every week. Security saw the design for the first time two weeks before go-live, required personal data to be masked, and the release slipped by two months.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Technical anchor
- Evidence
- in: stakeholders
looks like: A stakeholder map naming each person or group, what they care about, what they can approve or block, when they were last spoken to, and how they are kept informed. - How the gap shows
- No stakeholder map or communication plan exists.
- The map lists only the sponsor, the business owner and the project team.
- Operations, security, source owners or downstream consumers are not named anywhere.
- Compare
- Compare the architecture with the stakeholder map: every source, platform and consumer in the architecture has an owner who should appear on the map; one who does not is an unmanaged blocker.
- Due by
plan stage 3: Discovery evidenced stage 4: Discovery sign-off - Fail fast
- List everyone who must say yes before go-live and book a thirty-minute meeting with each one you have not met.
- Script signal
missing-doc-type(found without AI)- Related
- REQ-05, COM-04
STK-02Nobody clearly owns the decisionshighProject manager
- Question
- For each kind of decision (scope, definitions, priorities, go-live), is one named person the decision owner? When two senior people disagree, who decides, and how long will we wait?
- Why it matters
- Two directors each believed they owned the project. Every decision was taken twice, the team built to whichever spoke last, and both rejected the result.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Business owner and sponsor
- Evidence
- in: stakeholders, decisions
looks like: A responsibility table (who decides, who is consulted, who is informed) for the main decision types, with one decider per type and a route for escalation. - How the gap shows
- No responsibility table, or several people marked as the decider for the same thing.
- Decision log entries with no named decider.
- Phrases such as "the business will decide" without a name.
- Compare
- Compare the stakeholder map with the decision log: decisions signed by someone the map does not name as the decider are likely to be reopened.
- Due by
plan stage 3: Discovery evidenced stage 4: Discovery sign-off - Fail fast
- Ask the sponsor to name, in writing, the one person who decides on scope and definitions.
- Related
- PM-08, REQ-03
STK-03No real numbers shown to the businesshighTechnical anchor
- Question
- Have the business users seen real numbers from real data, regularly, before user acceptance testing? If they say "these numbers are wrong" at the end, will it be the first time they have looked?
- Why it matters
- Demos used made-up data that looked fine. The first time the business saw real numbers was user acceptance, and they did not recognise them.
- Applies to
- any · components: any · patterns: any
- Owner
- Technical anchor · also Business analyst, Data consumers
- Evidence
- in: status, plan
looks like: Regular demos, roughly every two weeks, with real data, and a record of what the business said and what changed as a result. - How the gap shows
- No demos are planned or recorded before the test stage.
- Demos are described as using sample, dummy or mock data.
- The data profile was not shown to the business owner during discovery.
- Compare
- Compare the plan with the meeting notes or status reports: planned demos that do not appear in the notes did not happen.
- Due by
plan stage 4: Discovery sign-off evidenced stage 6: Build - Fail fast
- Show the business owner one real number from the thin slice and ask whether they recognise it.
- Related
- REQ-05, REQ-04
STK-04Conflicts left unsaidmediumProject manager
- Question
- Where stakeholders want different things (speed against cost, one department's definition against another's), is the conflict written down with options and taken to the decision owner? Which disagreement are we avoiding that will surface at sign-off?
- Why it matters
- Finance wanted monthly figures to match the ledger; sales wanted daily figures fast. Nobody named the conflict, the design pleased neither, and both refused to sign.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Technical anchor, Business analyst
- Evidence
- in: raid, decisions
looks like: Conflicts recorded as issues or open decisions, with each side's need, options, a recommendation and a deadline. - How the gap shows
- Requirements from different groups that contradict each other with no note of the conflict.
- Meeting notes saying a topic was "taken offline" repeatedly.
- No issues or open decisions involve more than one stakeholder group.
- Compare
- Compare requirements from two different stakeholder groups: requirements that cannot both be met, with no decision recorded, are a hidden conflict.
- Due by
plan stage 3: Discovery evidenced stage 4: Discovery sign-off - Fail fast
- Put the two conflicting needs on one page with options and send it to the decision owner this week.
- Related
- STK-02, COM-01
STK-05Sign-off without readingmediumProject manager
- Question
- Did the people signing get the document in time to read it, and are their findings answered in writing? Is anyone signing something they have not read or do not agree with?
- Why it matters
- Reviewers saw the discovery document for the first time in the sign-off meeting and approved it to keep things moving. Their real objections arrived in the build, where they cost ten times more.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Technical anchor, Business owner and sponsor
- Evidence
- in: decisions, status
looks like: The document sent to reviewers several days before sign-off, a list of findings with written answers, and a sign-off record naming who approved what. - How the gap shows
- No review comments or findings are recorded before sign-off.
- The document was distributed on the same day as the sign-off meeting.
- Sign-off is recorded for a group ("the steering committee") with no individual names.
- Compare
- Compare the sign-off record with later status reports or change requests: early objections to items that were signed suggest the signers had not read them.
- Due by
plan stage 4: Discovery sign-off evidenced stage 4: Discovery sign-off - Fail fast
- Ask each signer, before the meeting, for their single biggest concern in writing.
- Related
- SCOPE-06
STK-06Success not defined for each stakeholderlowProject manager
- Question
- Has each key stakeholder been asked what success looks like for them and what has gone wrong for them before? If a stakeholder calls the project a failure, will we know in advance what they were measuring?
- Why it matters
- The sponsor wanted lower cost, operations wanted fewer night calls, and users wanted faster reports. The project delivered only faster reports and was seen as a failure by two out of three.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Technical anchor, Business analyst
- Evidence
- in: stakeholders, discovery
looks like: For each key stakeholder, one line on what success means to them and a note of past problems they raised. - How the gap shows
- The stakeholder map lists names and roles but not interests or success measures.
- No record of initial interviews with stakeholders outside the business owner.
- Compare
- Compare each stakeholder's success measure with the scope: a measure that nothing in scope can move is a disappointment already planned.
- Due by
plan stage 3: Discovery - Fail fast
- Ask each stakeholder one question this week: "What would make you say this project worked?"
- Related
- REQ-07, STK-01
STK-07Sponsor or owner changes and nobody re-confirmsmediumProject manager
- Question
- When the sponsor, business owner or a key data owner changes during the project, is the new person taken through the scope, definitions, decisions, budget and dates, and asked to confirm them in writing? If a reorganisation replaces our sponsor next month, which agreed decision is the new one most likely to reopen?
- Why it matters
- The business owner moved roles after discovery sign-off. Her successor had never seen the definitions, did not accept the agreed treatment of returns, and two months of modelling were redone.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Business owner and sponsor, Business analyst
- Evidence
- in: stakeholders, decisions
looks like: A record of each change in key stakeholders, the briefing given to the new person, and their written confirmation (or the decisions they reopened, handled as change requests). - How the gap shows
- Names in recent status reports or meeting notes differ from those on the stakeholder map or sign-off record.
- A new sponsor or owner appears with no record of a handover briefing.
- Signed decisions are being questioned again in meeting notes after a change of people.
- Compare
- Compare the signatories on the discovery sign-off with the people now named as sponsor and owners: a decider who did not sign has not yet agreed.
- Due by
plan stage 4: Discovery sign-off evidenced stage 6: Build - Fail fast
- Check whether everyone who signed discovery still holds the same role, and brief anyone new this week.
- Related
- STK-02, STK-05, PM-03
How news, decisions and problems travel (7)
COM-01Bad news held backhighTechnical anchor
- Question
- When the team learns something bad (a slip, a data problem, a design that will not work), how quickly and to whom is it reported, with a next step? Is there anything the team knows today that the sponsor does not?
- Why it matters
- The team knew for six weeks that the source could not deliver hourly data. They hoped to find a workaround and said nothing, so the sponsor learned it from the source owner instead.
- Applies to
- any · components: any · patterns: any
- Owner
- Technical anchor · also Project manager
- Evidence
- in: status, raid
looks like: Status reports and the issue log that raise problems within days of their discovery, each with an impact and a proposed next step. - How the gap shows
- Issues appear in the RAID log weeks before they appear in any status report.
- Status reports contain only achievements.
- Words such as "minor", "under control" or "being looked at" used for issues that later became major.
- Compare
- Compare the RAID log dates with status report dates: an issue logged long before it was reported is news that was held back.
- Due by
plan stage 4: Discovery sign-off evidenced stage 6: Build - Fail fast
- Ask the team: "What do we know that we have not told the sponsor?" and report it this week.
- Related
- PM-04, STK-04
COM-02Technical words used with business peoplemediumTechnical anchor
- Question
- Are documents and reports written for the business in plain words, with technical terms explained or left out? When the business owner signs something, do they understand what it means for them?
- Why it matters
- The design said history would be "type 1 for most dimensions". The business owner signed it, not realising it meant old addresses would be overwritten and last year's regional figures would change.
- Applies to
- any · components: any · patterns: any
- Owner
- Technical anchor · also Business analyst, Project manager
- Evidence
- in: status, discovery, decisions
looks like: Business-facing documents that state consequences in business terms ("last year's figures by region will change when a customer moves"), with a short glossary for unavoidable terms. - How the gap shows
- Status or sign-off documents for the business use unexplained abbreviations and technical terms.
- Decisions are described by their technical option, not by their effect on users.
- No glossary in documents read by business stakeholders.
- Compare
- Compare a decision record with the business owner's own description of it (in notes or e-mail): if they describe a different consequence, the decision was not understood.
- Due by
plan stage 3: Discovery evidenced stage 4: Discovery sign-off - Fail fast
- Ask the business owner to explain the three biggest decisions back in their own words.
- Related
- STK-05
COM-03Agreed in a meeting, lost afterwardsmediumProject manager
- Question
- Does every important meeting end with written actions and decisions, sent to the people affected and copied into the decision log or RAID log? A month from now, will anyone remember what was agreed today?
- Why it matters
- The source owner agreed in a meeting to provide a daily extract by a set time. Nothing was written, the owner moved on, and the successor had never heard of it.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Technical anchor
- Evidence
- in: status, decisions
looks like: Short meeting notes with actions (owner and date) and decisions, sent the same day, and decisions copied into the decision log. - How the gap shows
- No meeting notes, or notes without actions and owners.
- Actions in notes that never appear again in later notes or status reports.
- Phrases such as "as discussed" referring to something with no written record.
- Compare
- Compare meeting notes with the decision log and RAID log: decisions and actions in the notes that are missing from the logs have been lost.
- Due by
plan stage 3: Discovery evidenced stage 5: MVP starts - Fail fast
- After the next meeting, send three lines within an hour: decided, actions, open questions.
- Related
- PM-03
COM-04Nobody knows who to callmediumProject manager
- Question
- Is there a contact list for the project and the systems it depends on, including who to call when a source changes or a load fails, and how to escalate? If the pipeline fails at night in the first week of production, who is woken up and who do they call?
- Why it matters
- A source was changed without notice and the load failed. Nobody knew who owned the source, so it took three days to find the right person while the reports stayed empty.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Operations and support, Source system owners
- Evidence
- in: stakeholders, runbook
looks like: A contact and escalation list for the team, each source, the platform, security and operations, with named people and a deputy for each. - How the gap shows
- No contact or escalation list in the stakeholder map or runbook.
- Source systems named in the architecture with no named owner.
- Escalation described as "raise with management".
- Compare
- Compare the architecture's list of sources and systems with the contact list: a system with no contact is one that will fail silently.
- Due by
plan stage 3: Discovery evidenced stage 8: Deploy - Fail fast
- For each source, find the named owner and confirm they will tell the team before changing it.
- Related
- STK-01, HAND-04
COM-05Assumptions never shared with the businessmediumProject manager
- Question
- Have the assumptions behind the estimate and plan been shown to the sponsor and the business owner, and do they know what changes if one of them breaks? If an assumption fails, will the business be surprised that the plan ever depended on it?
- Why it matters
- The estimate assumed the business would provide testers for two weeks. The business never knew; when no testers came, user acceptance slipped by a month and the team was blamed.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Technical anchor, Business owner and sponsor
- Evidence
- in: status, estimate
looks like: The key assumptions repeated in status reports and sign-off packs, with their current state (holding, at risk, broken). - How the gap shows
- Status reports and steering packs do not mention assumptions.
- Assumptions that require business action (testers, decisions, data) are not assigned to a business person.
- Compare
- Compare the estimate's assumptions with the status reports: assumptions in the estimate that never appear in any report have not been shared.
- Due by
plan stage 4: Discovery sign-off evidenced stage 6: Build - Fail fast
- Add a three-line "assumptions this plan depends on" box to the next status report.
- Related
- EST-03, PM-07
COM-06No regular rhythm for updateslowProject manager
- Question
- Is there an agreed rhythm for telling stakeholders what was learned, what changed, and what is needed from them? Which stakeholder will say they heard nothing for a month?
- Why it matters
- Updates went out only when there was good news. Stakeholders filled the silence with rumours and the sponsor started asking for daily reports, which took the team's time away from the work.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Technical anchor
- Evidence
- in: stakeholders, status
looks like: A communication plan stating who gets what and how often, and status reports that follow it. - How the gap shows
- No communication plan.
- Gaps of several weeks between status reports.
- Status reports go to the sponsor only.
- Compare
- Compare the communication plan with the actual status reports: a planned weekly update that appears monthly is a rhythm that broke.
- Due by
plan stage 4: Discovery sign-off evidenced stage 6: Build - Fail fast
- Send a three-line update this Friday: what we learned, what changed, what we need from you.
- Related
- COM-01
COM-07Steering packs that ask for nothingmediumProject manager
- Question
- Does each steering pack open with the decisions and help the project needs from that meeting (a date, a budget release, a source team pushed, a definition settled), each with options and a deadline? After the next steering meeting, which blocker will still be blocked because we only reported progress?
- Why it matters
- Steering packs ran to twenty slides of activity. The one line that mattered, that a source team had not started its extract change, sat on slide fourteen, and nobody in the room acted on it for three meetings.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Technical anchor, Business owner and sponsor
- Evidence
- in: status, decisions
looks like: A first page listing the decisions and help needed, each with options, a recommendation and a deadline, and notes from the meeting recording what was decided. - How the gap shows
- Steering packs contain progress and activity but no "decisions needed" section.
- The same blocker appears in several packs with no request attached.
- Steering notes record no decisions.
- Compare
- Compare open issues and decisions in the RAID log with the steering packs: an issue that needs steering action but never appears as a request in the pack is being reported, not escalated.
- Due by
plan stage 4: Discovery sign-off evidenced stage 6: Build - Fail fast
- Put "decisions we need from you today" as the first slide of the next steering pack.
- Related
- GOV-01, PM-08, COM-01
Documentation, knowledge transfer, support model (2)
HAND-04Support model not agreedhighProject manager
- Question
- Is it agreed who supports the system after go-live, in which hours, how fast they respond, how long the project team stays during the early weeks, and who pays? On the first weekend in production, who answers a failure?
- Why it matters
- The project assumed the central support team would take over; the support team assumed the project would. After go-live a failed weekend load went unnoticed until Monday morning.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Operations and support, Business owner and sponsor, Technical anchor
- Evidence
- in: runbook, plan
looks like: A signed support model: support team, hours, response times, escalation, early-life support period, and the criteria for ending it. - How the gap shows
- No support model, or one that says "business as usual support".
- The support team has not been asked to accept the system.
- Support hours do not match the freshness promised to users.
- Compare
- Compare the support model with the non-functional requirements: data promised by 7 a.m. but support starting at 9 a.m. means nobody fixes a night failure in time.
- Due by
plan stage 4: Discovery sign-off designed stage 6: Build evidenced stage 9: Production sign-off - Fail fast
- Ask the support team lead whether they have agreed to take this system, and on what terms.
- Related
- REQ-06, HAND-01, COM-04
HAND-05Benefits never measuredmediumBusiness owner and sponsor
- Question
- Is there a plan to measure, after go-live, the benefits promised at the start, against the baseline taken before? A year after closure, can anyone prove the project was worth its cost?
- Why it matters
- The business case promised hours saved each month. Nobody measured hours before or after, so the project could not show any return and the platform's next phase was refused.
- Applies to
- any · components: any · patterns: any
- Owner
- Business owner and sponsor · also Project manager, Business analyst
- Evidence
- in: discovery, status
looks like: A benefits list with baseline values, the date and method of measurement after go-live, and an owner who will report the result. - How the gap shows
- The business case or goals have no baseline figures.
- No benefits review is planned after go-live.
- The project closure report does not mention benefits.
- Compare
- Compare the goals in the discovery document with the closure or status reports: goals that are never reported on were never measured.
- Due by
plan stage 3: Discovery evidenced stage 10: Project sign-off - Fail fast
- Record today's value of each promised benefit, so there is something to compare against later.
- Related
- REQ-07
Budget and its basis, spend against plan, contingency, running cost ownership, cost of change (6)
BUD-01A budget with no basishighProject manager
- Question
- Can every line of the budget be traced to something written down: the estimate range and its assumptions, day rates, environments, licences, running cost and contingency? If the budget runs out in month five, will anyone be able to say which assumption it was built on?
- Why it matters
- The budget was the number the sponsor had left at year end. It matched the low end of the rough estimate, held no running cost and no history load, and was spent before the first report reached production.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Business owner and sponsor, Technical anchor
- Evidence
- in: budget, estimate, plan
looks like: A budget broken into people, environments, licences, external services, running cost during the project and contingency, each line pointing to the estimate or a quote, with the estimate's range shown. - How the gap shows
- The budget is one number with no breakdown.
- The budget matches the low end of the estimate range, or the rough estimate from before discovery.
- No line for running cost, environments or licences while the project is being built.
- Compare
- Compare the budget with the estimate: a budget below the estimate's likely figure, or built from an earlier version of the estimate, is already overspent.
- Due by
plan stage 2: Rough estimate designed stage 4: Discovery sign-off - Fail fast
- Rebuild the budget from the current estimate's middle figure and put the gap, if any, in front of the sponsor.
- Related
- EST-01, EST-02, BUD-04
BUD-02Spend not tracked against the planhighProject manager
- Question
- Is actual spend compared with planned spend at least monthly, with a forecast of the total at completion? If the money runs out early, will the status reports from three months before have shown it coming?
- Why it matters
- Contractor invoices went straight to finance. The project manager saw the total only at quarter end, when seventy percent of the budget was gone and forty percent of the work was done.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Business owner and sponsor
- Evidence
- in: budget, status, plan
looks like: A monthly table of planned spend, actual spend, forecast to complete and forecast at completion, set against the share of work done, shown in the steering pack. - How the gap shows
- Status reports mention dates and scope but never money.
- Spend is reported as a total so far, with no plan to compare it with and no forecast.
- The forecast at completion equals the budget in every report.
- Compare
- Compare spend in the status reports with progress against the estimate: money spent faster than work is finished, while the forecast total never changes, means the forecast is not real.
- Due by
plan stage 4: Discovery sign-off evidenced stage 6: Build - Fail fast
- Divide spend so far by the share of work done and compare the answer with the budget today.
- Related
- EST-06, PM-04
BUD-03Nobody owns the running cost after go-livehighBusiness owner and sponsor
- Question
- Is the monthly cost of running the solution (compute, storage, licences, support) agreed with the person whose budget will pay it, for this year and next? When the first production bill arrives, whose cost centre is it charged to, and do they know?
- Why it matters
- The project budget ended at go-live. The platform bill landed on the central data team, who had not planned for it, and they switched off the hourly refresh to stay within their own budget.
- Applies to
- any · components: any · patterns: any
- Owner
- Business owner and sponsor · also Project manager, Technical anchor, Platform team
- Evidence
- in: discovery, plan
looks like: The running cost figure from the technical estimate, a named budget owner who has accepted it, the cost centre, and a forecast for the next two years that includes data growth. - How the gap shows
- Running cost is described as "business as usual" or "absorbed by the platform".
- The business case shows build cost only.
- No named person has agreed to pay the running cost.
- Compare
- Compare the running cost estimate in the architecture or discovery document with the business case: a business case without that cost overstates the return.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 9: Production sign-off - Fail fast
- Send the running cost figure to the person expected to pay it and ask them to confirm it in writing.
- Related
- COST-01, COST-03, HAND-04
BUD-04Contingency hidden or missingmediumProject manager
- Question
- Is contingency held as a separate, visible amount, sized to the riskiest assumptions, with a named person who can release it and a record of each use? When the data turns out worse than profiled, where will the money for fixing it come from?
- Why it matters
- Every estimator padded their own numbers quietly. The padding was spent on gold-plating early on, and when the history load needed three extra weeks there was nothing left and no one could approve more.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Business owner and sponsor, Technical anchor
- Evidence
- in: budget, estimate, raid
looks like: A contingency line linked to the top risks and assumptions, an owner who approves its use, and a log of drawdowns showing what each one paid for. - How the gap shows
- No contingency in the budget, or a flat percentage with no link to risks.
- Contingency described as "included in the estimates".
- The risk log has high risks with costly mitigations that no budget line pays for.
- Compare
- Compare the risk log with the budget: high risks whose mitigation costs money, with no contingency to cover them, are unfunded risks.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 6: Build - Fail fast
- Price the mitigation of the top three risks and compare the total with the contingency held.
- Related
- BUD-01, PM-02, EST-03
BUD-05Change requests with no costmediumProject manager
- Question
- Does every change request show what it adds in build cost, running cost and time, before anyone decides? If the project overspends, how much of it will be changes that were approved without a price?
- Why it matters
- The business asked for the report to refresh hourly instead of daily. It was agreed in a demo as a small change, and it tripled the monthly compute bill and added two weeks of testing nobody had priced.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Business owner and sponsor, Technical anchor
- Evidence
- in: decisions, status
looks like: A change log where each entry shows the build effort, the change to running cost, the effect on the date and the decider, with a running total of approved changes against contingency. - How the gap shows
- Change records that state the request and the decision but not the cost.
- Changes to freshness, history or volume approved without a running cost figure.
- No running total of the cost of approved changes.
- Compare
- Compare the change log with the budget and forecast: changes approved since the baseline, with a forecast at completion that has not moved, means their cost was never added.
- Due by
plan stage 4: Discovery sign-off evidenced stage 6: Build - Fail fast
- Price the last three approved changes after the fact and add them to the forecast.
- Related
- SCOPE-03, COST-01
BUD-06Costs that sit in other teams' budgetsmediumProject manager
- Question
- Have the costs that land outside the project been found and agreed: the source team's work on a new extract, a vendor's fee for an interface or bulk export, extra environments, and running the old system during a parallel run? Who will refuse to pay for something the project cannot go live without?
- Why it matters
- The source vendor charged for every change to the extract, and the source team had no budget for the work. The project waited two months while the two managers argued over who would pay.
- Applies to
- any · components: ingestion, platform · patterns: any
- Owner
- Project manager · also Source system owners, Business owner and sponsor, Platform team
- Evidence
- in: budget, plan, raid
looks like: A list of costs the project causes in other teams or with vendors, with the amount, who pays, and the date each was agreed. - How the gap shows
- Source changes, vendor interfaces or extra environments appear in the architecture with no cost or payer.
- The old system's licence or support is due to end before the planned parallel run is over.
- Phrases such as "the source team will provide" with no mention of their budget.
- Compare
- Compare the architecture's list of source changes and environments with the budget: each one that needs another team's work or a vendor's fee, with no agreed payer, is a stop waiting to happen.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off - Fail fast
- Ask each source owner and vendor whether the work the project needs from them is funded, and by whom.
- Related
- VEN-01, VEN-03, PM-15
Named people and their real availability, onboarding, booked time from source experts and business users (4)
RES-01Capacity counted in people, not in real hourshighProject manager
- Question
- Does the plan use each person's real availability: their share of time on this project, planned leave, public holidays, other commitments and support duties? If the team delivers two thirds of what was planned, will the missing third be leave and part-time allocations nobody subtracted?
- Why it matters
- The plan assumed five full-time developers. Two were shared with production support, one had three weeks of leave in the summer, and the real capacity was barely three people, so the first MVP slipped a month.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Technical anchor
- Evidence
- in: plan, estimate
looks like: A team table with each named person, their percentage on the project, known leave and the weeks they are needed, with the total compared against the effort in the estimate. - How the gap shows
- The plan lists roles or head counts with no names or percentages.
- No mention of leave, holidays or support duties.
- The same person is allocated full time on this project and another one.
- Compare
- Compare the estimate's effort with the plan's available hours: if effort divided by real capacity gives a later date than the plan shows, the plan assumes people who do not exist.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 6: Build - Fail fast
- Add up each person's real available days for the next eight weeks and compare them with the planned work.
- Related
- PM-06, PM-07, EST-03
RES-02New joiners assumed productive on day onemediumProject manager
- Question
- Does the plan allow for the time a new person needs before they are useful: laptop, accounts, access to sources and environments, security training and learning the domain? When a developer joins in week six, in which week will they first deliver?
- Why it matters
- Two contractors joined to catch up the plan. Their access to the source data took four weeks to approve, so they were paid for a month of reading documents while the slip grew.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Technical anchor, Platform team, Security and governance
- Evidence
- in: plan, raid
looks like: An onboarding checklist with the lead time of each item, requests raised before the start date, and the plan showing reduced output for each new person's first weeks. - How the gap shows
- People are added to the plan at full output from their first week.
- No onboarding checklist or access lead times are written down.
- Recovery plans rely on adding people late in the build.
- Compare
- Compare the plan's joining dates with the access request lead times in the RAID log: a person planned to deliver before their access can arrive will sit idle.
- Due by
plan stage 4: Discovery sign-off evidenced stage 6: Build - Fail fast
- Ask the most recent joiner how many days passed before they could query real data, and use that number.
- Related
- PM-10, ENV-02, RES-01
RES-03Source experts' time never bookedhighProject manager
- Question
- Is the time the project needs from source system experts (to explain tables, deletes and odd values, to review mappings and to fix extracts) agreed with their manager, in hours per week and by date? When the team needs an answer about the source, how long will it wait?
- Why it matters
- The only person who understood the source's status codes worked on that system's own release. She gave the project an hour a week when she could, and every mapping question waited ten days for an answer.
- Applies to
- any · components: ingestion, transformation · patterns: any
- Owner
- Project manager · also Source system owners, Technical anchor, Business analyst
- Evidence
- in: plan, stakeholders
looks like: Named source experts with agreed hours per week through discovery, build and test, confirmed by their line manager, and a deputy for each. - How the gap shows
- Source experts are named in the stakeholder map with no time commitment.
- Open questions about source data stay open across several status reports.
- The plan relies on "input from the source team" without names or dates.
- Compare
- Compare the number of sources in the architecture with the expert time booked in the plan: a source with no booked expert will be understood by guessing.
- Due by
plan stage 3: Discovery evidenced stage 5: MVP starts - Fail fast
- Ask each source expert's manager to confirm, in writing, the hours per week they will give the project.
- Related
- PM-06, REQ-04, DQ-05
RES-04Business testers and sign-off people not bookedhighProject manager
- Question
- Are the business users who will test, reconcile numbers and sign off named, with their days booked and clear of month-end, year-end and their own peak periods? If user acceptance slips, will it be because the testers were closing the books that week?
- Why it matters
- User acceptance was planned for the last week of the quarter. Every finance tester was on the quarter close, the test window passed with a handful of cases run, and go-live moved by six weeks.
- Applies to
- any · components: modelling, serving · patterns: any
- Owner
- Project manager · also Business owner and sponsor, Test lead, Data consumers
- Evidence
- in: plan, test-strategy
looks like: Named testers and approvers, the days each has agreed to give, their managers' agreement, and a test window checked against the business calendar. - How the gap shows
- The test strategy says "business users" without names.
- User acceptance dates fall in a month-end, quarter-end or year-end period.
- No business manager has agreed to release testers.
- Compare
- Compare the plan's user acceptance window with the business calendar or the testers' own commitments: a window that overlaps their busiest week will not get the testing it needs.
- Due by
plan stage 4: Discovery sign-off designed stage 6: Build evidenced stage 7: Test - Fail fast
- Ask each named tester's manager to put the test days in their calendar now.
- Related
- TEST-04, COM-05, PM-18
Source teams, vendors and suppliers as dependencies, with agreed dates, terms and escalation (4)
VEN-01Source-side changes not on the source team's planhighProject manager
- Question
- For each change the project needs from a source (a new extract, a new field, change tracking switched on, a service account, a firewall rule), is it on the source team's or vendor's own plan, with a release date, a named contact and an escalation route? If the source team's next release slot is in four months, when will we find out?
- Why it matters
- The project needed change tracking switched on in the source database. The source team agreed in a meeting, but their releases were quarterly and the next slot was full, so the build waited three months.
- Applies to
- any · components: ingestion · patterns: any
- Owner
- Project manager · also Source system owners, Technical anchor
- Evidence
- in: raid, plan
looks like: A list of source-side changes, each with the item reference on the source team's own plan, the release date they committed to, a named contact and the person to escalate to. - How the gap shows
- Source changes appear in the architecture but not as dependencies in the RAID log.
- Dependencies on source teams have no date, or a date the source team has not confirmed.
- No mention of the source team's release calendar or change freezes.
- Compare
- Compare the architecture's needs from each source with the RAID log's dependencies: a change the design needs that is not on the source team's plan is a date nobody has promised.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 5: MVP starts - Fail fast
- Ask each source team for the reference number of the project's request on their own plan, and its release date.
- Related
- PM-01, HIST-02, SCH-03
VEN-02No agreement on how data is deliveredhighProject manager
- Question
- Is there a written agreement with each source team or external supplier on what they deliver: by what time, in what format, how complete, how a late or broken delivery is reported and fixed, and in which support hours? When the file is late at 6 a.m., what does the agreement say happens?
- Why it matters
- An external partner sent a daily file "in the morning". Some days it came at nine, some days at two, and the reports promised for eight o'clock were wrong half the week with nothing in writing to point to.
- Applies to
- any · components: ingestion · patterns: file, api, database, cdc
- Owner
- Project manager · also Source system owners, Technical anchor, Operations and support
- Evidence
- in: stakeholders, runbook
looks like: A delivery agreement or contract schedule per source: delivery time, format, completeness check, notice of change, support hours, contact and escalation, and what happens when it is missed. - How the gap shows
- Source delivery times written as "daily" or "overnight" with no clock time.
- No support hours or contacts for external sources.
- Freshness promised to users is tighter than anything a source has agreed to.
- Compare
- Compare the freshness promised to consumers with the delivery times agreed with sources: a report promised at 8 a.m. from a source that only promises "the morning" cannot keep its promise.
- Due by
plan stage 4: Discovery sign-off designed stage 6: Build evidenced stage 8: Deploy - Fail fast
- Ask each source owner to state the latest time they will deliver and what they do when they cannot.
- Related
- SERV-01, SCH-03, PERF-02, DQ-04
VEN-03Licences and contract terms for getting the datamediumProject manager
- Question
- Do the contracts with source system vendors allow the project to extract the data in the way and volume the design needs, and are the licences for every tool and user covered? Which clause will we discover the week before go-live?
- Why it matters
- The design pulled history through the vendor's interface. The contract capped calls per day and charged for bulk export, which nobody had read, so the history load needed a new commercial agreement.
- Applies to
- any · components: ingestion, serving, platform · patterns: any
- Owner
- Project manager · also Source system owners, Business owner and sponsor, Security and governance
- Evidence
- in: raid, architecture
looks like: A note per vendor source and tool of the contract terms checked (extract rights, call limits, export fees, data use), the licence count needed against the count owned, and who checked it. - How the gap shows
- Vendor-hosted sources in the architecture with no mention of contract terms or limits.
- Licences for report users or tools are not mentioned in the budget.
- The number of users planned is larger than the licences held, or the count is unknown.
- Compare
- Compare the history load and refresh approach in the architecture with the vendor's stated limits: a design that needs more calls or bulk export than the contract allows will be stopped.
- Due by
plan stage 3: Discovery evidenced stage 4: Discovery sign-off - Fail fast
- Send the vendor the planned extract approach and volumes and ask them to confirm it is allowed at no extra cost.
- Related
- BUD-06, HIST-02
VEN-04Supplier work with no acceptance or handover termsmediumProject manager
- Question
- If an outside supplier builds part of the solution, does their contract say what "done" means (tested, reconciled, documented), how it is accepted, who fixes defects after go-live and for how long, and what knowledge they must hand over? When the supplier's contract ends at go-live, who fixes the first broken load?
- Why it matters
- The supplier was paid on delivery of the pipelines. Their contract ended at go-live, the first month-end load failed, and the internal team had no documentation and no right to ask the supplier back.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Technical anchor, Operations and support
- Evidence
- in: plan, runbook
looks like: A statement of work with acceptance criteria tied to tests and reconciliation, a warranty period that covers the first month-end, and handover deliverables with dates. - How the gap shows
- Supplier deliverables are described as code or pipelines, with no tests or documents.
- The supplier's contract ends on or before the go-live date.
- No warranty or defect-fix period is mentioned.
- Compare
- Compare the supplier's end date with the plan's early-life support period: a supplier who leaves before support ends leaves the defects behind.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 9: Production sign-off - Fail fast
- Read the supplier's acceptance and end-date clauses against the go-live and month-end dates this week.
- Related
- HAND-03, HAND-06, PM-16
Steering group, stage gates and their exit criteria, escalation, approval boards, lessons per gate (6)
GOV-01A steering group that cannot decidehighProject manager
- Question
- Does the steering group include people who can decide on money, scope and dates, plus the owners of the data and the platform the project depends on, and is it written down what it decides and how often it meets? When the project needs a yes on extra budget or a later date, will the people in the room be able to give it?
- Why it matters
- The steering group was the project team's managers. When the source team refused to change their extract, nobody in the room had authority over them, and the issue went round the meeting for two months.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Business owner and sponsor
- Evidence
- in: stakeholders, status
looks like: Short terms of reference: members by name, what the group decides (budget, scope, dates, go-live), how often it meets, and attendance records showing the deciders actually came. - How the gap shows
- No steering group or terms of reference is written down.
- Members are all from the delivery side, with no data owner, source owner or budget holder.
- Steering notes record updates but no decisions.
- Compare
- Compare the steering membership with the RAID log's biggest dependencies and issues: an issue owned by a team nobody on the steering group can direct will not be solved there.
- Due by
plan stage 3: Discovery evidenced stage 4: Discovery sign-off - Fail fast
- Take the top open issue to the next steering meeting and see whether anyone there can decide it.
- Related
- STK-02, GOV-03
GOV-02Stage gates with no exit criteriahighProject manager
- Question
- Do discovery sign-off, production sign-off and project sign-off each have written exit criteria and the evidence each needs (for example reconciled numbers, clean runs, an accepted support model, measured benefits), agreed before the gate? At each gate, are we deciding on evidence or on the calendar?
- Why it matters
- Production sign-off was a meeting on the planned date. It was signed with reconciliation still failing for two regions, and the project team was released, so the differences were never explained.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Business owner and sponsor, Technical anchor, Operations and support
- Evidence
- in: plan, decisions
looks like: For each gate, a short list of exit criteria with the document or test result that proves each one, who signs, and the possible outcomes (go, go with conditions, stop). - How the gap shows
- Gates appear in the plan as dates only.
- Production or project sign-off has no criteria, or criteria such as "all work complete".
- Sign-off records do not mention which criteria were met.
- Compare
- Compare each gate's sign-off record with its exit criteria and the evidence available on that date: a gate signed while a criterion was unmet was passed on the date, not the evidence.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 9: Production sign-off proven stage 10: Project sign-off - Fail fast
- Write the production sign-off criteria now and check which of them the project could meet today.
- Related
- PM-09, HAND-06, REC-04
GOV-03Escalation with no time limitsmediumProject manager
- Question
- Is there a written escalation path for blockers, saying how long a problem may stay at each level before it goes to the next, up to someone who can direct other teams? How many days can a blocked access request or a source team's refusal sit before the sponsor hears about it?
- Why it matters
- A firewall change was blocked for seven weeks. The team kept chasing it politely, nobody had said when to escalate, and the sponsor learned of it when the date slipped.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Business owner and sponsor, Technical anchor
- Evidence
- in: stakeholders, raid
looks like: An escalation path with levels, named people and time limits (for example three working days at team level, a week at project manager level, then the steering group), and issues in the RAID log showing when they were escalated. - How the gap shows
- Escalation described as "raise with management" with no names or times.
- Issues in the RAID log open for weeks with no record of escalation.
- Dependencies on other teams have slipped more than once with no escalation.
- Compare
- Compare the age of open issues in the RAID log with the escalation time limits: an issue older than its limit that was never escalated shows the path is not used.
- Due by
plan stage 3: Discovery evidenced stage 5: MVP starts - Fail fast
- List every issue open longer than two weeks and escalate each one by Friday.
- Related
- COM-04, PM-10, GOV-01
GOV-04Approval boards and their calendars not on the planhighProject manager
- Question
- Are the outside approvals the project needs (architecture review, security review, privacy assessment, data sharing agreement, change board) listed with their meeting dates, what they need submitted and how long they take? Which board meets monthly and has not yet heard of us?
- Why it matters
- The privacy assessment had to be approved before personal data could be copied to the new platform. The board met monthly, sent the first submission back, and the project lost two months it had not planned for.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Security and governance, Architect, Technical anchor
- Evidence
- in: plan, raid
looks like: A list of approvals with the board, its meeting calendar, the submission deadline, what it needs, the usual number of rounds, and the plan tasks that wait on it. - How the gap shows
- No approvals or review boards appear in the plan.
- Personal or sensitive data is in scope but no privacy assessment or data sharing agreement is mentioned.
- Approvals are planned in the same week the work that needs them starts.
- Compare
- Compare the architecture's personal data flows and platform changes with the plan's approvals: each one that needs a board's approval, with no slot booked, is an unplanned wait.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 6: Build - Fail fast
- Ask security and the architecture board which approvals this project needs and the next date each one meets.
- Related
- SEC-01, SEC-05, PM-01, STK-01
GOV-05Conditions from a gate never closedmediumProject manager
- Question
- When a gate is passed "with conditions", is each condition written down with an owner and a date, and checked at the next gate? Which condition from discovery sign-off is still open, and does anyone remember it?
- Why it matters
- Discovery was approved on condition that the source owner confirmed delete handling. Nobody tracked the condition, it was never confirmed, and deleted orders were still counted in production.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Business owner and sponsor, Technical anchor
- Evidence
- in: decisions, raid
looks like: Each gate condition recorded as an action in the RAID log with owner and date, and the next gate's pack showing the state of every earlier condition. - How the gap shows
- Sign-off records say "approved with conditions" or "subject to" with no list of conditions.
- Conditions from an earlier gate are not mentioned in later status reports or gate packs.
- Compare
- Compare the conditions in each sign-off record with the RAID log: a condition that never became a tracked action has been forgotten.
- Due by
plan stage 4: Discovery sign-off evidenced stage 9: Production sign-off - Fail fast
- Read the last sign-off record and list every condition with its owner and current state.
- Related
- GOV-02, STK-05, COM-03
GOV-06Lessons captured only at the endlowProject manager
- Question
- Are lessons gathered at each gate and after each MVP, with an action that changes how the next stage is run? If the same data surprise hits MVP 2 that hit MVP 1, will anyone have written down how to avoid it?
- Why it matters
- The first MVP lost three weeks to late source access. The lessons review was held at project closure, so MVP 2 and MVP 3 lost the same three weeks for the same reason.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Technical anchor, Business analyst
- Evidence
- in: status, plan
looks like: A short lessons list after each gate and MVP: what happened, what we will do differently, the owner of that change, and evidence in the next stage's plan that it was done. - How the gap shows
- The only lessons review in the plan is at project closure.
- The same kind of issue appears in the RAID log for more than one MVP.
- Lessons are listed with no action or owner.
- Compare
- Compare the lessons from one stage with the plan for the next: a lesson with no matching change in the next plan has not been learned.
- Due by
plan stage 4: Discovery sign-off evidenced stage 6: Build proven stage 10: Project sign-off - Fail fast
- Spend fifteen minutes at the next retrospective on one question: what will we change for the next MVP?
- Related
- PM-02, EST-06
Go or no-go, business readiness, old reports switched off, clashing releases, consumers told (5)
RLC-01Go or no-go decided on the day, not agreed in advancehighProject manager
- Question
- Are the go-live criteria (reconciled numbers within tolerance, no open critical defects, support ready, users trained, rollback rehearsed) written down and agreed with the business owner before the go or no-go meeting? If the history load finishes with two regions unreconciled, do we already know whether we go?
- Why it matters
- With no agreed criteria, the go or no-go meeting became a debate. The sponsor wanted the date, the test lead wanted another week, and they went live with known differences that took three months to explain away.
- Applies to
- any · components: any · patterns: any
- Owner
- Project manager · also Business owner and sponsor, Technical anchor, Test lead, Operations and support
- Evidence
- in: deployment, decisions
looks like: A go or no-go checklist agreed weeks before cut-over, with each criterion, the evidence for it, who confirms it, and what happens if it fails (delay, go with a workaround, or partial release). - How the gap shows
- No go-live criteria are written, or they say "UAT signed off".
- The go or no-go decision has no named decider.
- Criteria do not mention reconciliation, support readiness or rollback.
- Compare
- Compare the go-live criteria with the test results and defect list on the decision date: a go decision taken while a criterion was unmet means the criteria were not the real basis.
- Due by
plan stage 4: Discovery sign-off designed stage 7: Test evidenced stage 8: Deploy - Fail fast
- Draft the go-live checklist now and ask the business owner which line they would refuse to go without.
- Related
- GOV-02, PM-14, REC-03
RLC-02Users not ready on go-live daymediumProject manager
- Question
- Are the users ready: told what is changing and when, trained on the new reports and what the numbers now mean, and pointed at where to ask for help, with the service desk briefed? On the first Monday, will users know why their numbers look different from last week?
- Why it matters
- The new sales report used the agreed definition of revenue, which excluded returns. Nobody told the regional managers, who saw lower figures, assumed the report was broken, and kept using their old spreadsheets.
- Applies to
- any · components: serving, modelling · patterns: any
- Owner
- Project manager · also Business owner and sponsor, Business analyst, Data consumers
- Evidence
- in: plan, stakeholders
looks like: A business readiness plan: user groups, messages and dates, training sessions with attendance, a short note on changed definitions and why numbers differ, and a briefed service desk. - How the gap shows
- No training, user communication or service desk briefing is in the plan.
- Definitions changed during the project but no user-facing explanation exists.
- The only user communication is a go-live announcement.
- Compare
- Compare definitions that changed in the decision log or data model with the user communications: a changed definition users were never told about will be reported as a defect.
- Due by
plan stage 4: Discovery sign-off designed stage 7: Test evidenced stage 8: Deploy - Fail fast
- Ask three future users what will change for them on go-live day, and note what they do not know.
- Related
- COM-02, REQ-03, STK-03
RLC-03Old reports and feeds never switched offmediumProject manager
- Question
- Is there a list of the old reports, extracts and spreadsheets the new solution replaces, each with an owner who agrees to switch it off and a date? A year after go-live, how many people will still be using the old numbers, and paying to run them?
- Why it matters
- The old reports were left running "just in case". Two years later both sets were still in use, gave different answers in board meetings, and the cost of the old system was never saved.
- Applies to
- any · components: serving · patterns: any
- Owner
- Project manager · also Business owner and sponsor, Data consumers, Operations and support
- Evidence
- in: plan, deployment
looks like: A switch-off list naming each old report or feed, its users, its owner, the switch-off date after the parallel run, and confirmation that it was done. - How the gap shows
- The plan has no switch-off or decommissioning tasks.
- Benefits depend on retiring an old system, but no retirement date exists.
- Old reports are listed as "to be reviewed after go-live".
- Compare
- Compare the benefits in the business case with the switch-off list: savings that depend on retiring the old system, with no switch-off date, will not arrive.
- Due by
plan stage 4: Discovery sign-off designed stage 7: Test evidenced stage 9: Production sign-off proven stage 10: Project sign-off - Fail fast
- List every old report the new one replaces and ask each owner for a switch-off date.
- Related
- PM-15, BEN-01, HAND-05
RLC-04Clashes with other releases and platform changesmediumTechnical anchor
- Question
- Has the release been checked against other changes in the same window: source system upgrades, platform upgrades, other projects changing shared tables, network or identity changes? Which other team's release could break ours the same weekend?
- Why it matters
- The project went live the same weekend the source system was upgraded. Two columns changed type, the first production loads failed, and nobody could tell which release had caused it.
- Applies to
- any · components: any · patterns: any
- Owner
- Technical anchor · also Project manager, Release and DevOps, Platform team, Source system owners
- Evidence
- in: deployment, raid
looks like: The release date checked against the change calendar for sources, the platform and shared data, with clashes recorded as dependencies or risks and agreed with the other teams. - How the gap shows
- No mention of the change calendar or other releases in the deployment plan.
- A shared platform or source upgrade is known but not on the RAID log.
- Other projects write to tables this project reads, with no coordination recorded.
- Compare
- Compare the go-live date in the deployment plan with source and platform release dates in the RAID log or change calendar: two changes to the same system in one window will hide each other's faults.
- Due by
plan stage 4: Discovery sign-off evidenced stage 8: Deploy - Fail fast
- Ask the change board for every change planned to the sources and platform in the go-live week.
- Related
- PM-18, SCH-03, REL-07
RLC-05Consumers outside the steering group not toldhighProject manager
- Question
- Have all consumers of the data being replaced or changed been found and told, including other teams' extracts, downstream systems and spreadsheets fed from the old source, not only the users in the steering group? Who will discover our go-live because their own report broke?
- Why it matters
- The old warehouse table was replaced. A pricing team nobody had met read it every night for their own model, and their prices were wrong for three days before anyone linked it to the release.
- Applies to
- any · components: serving, modelling · patterns: any
- Owner
- Project manager · also Technical anchor, Data consumers, Operations and support
- Evidence
- in: stakeholders, deployment
looks like: A consumer list built from access logs and lineage, not only interviews, each consumer told the change and the date, and their confirmation recorded. - How the gap shows
- The consumer list comes only from the steering group and requirements workshops.
- No one has checked who reads the tables or feeds being replaced.
- No notice period for consumers is in the deployment plan.
- Compare
- Compare the consumers in the stakeholder map with the readers found in access logs or lineage: a reader who is not on the map has not been told.
- Due by
plan stage 4: Discovery sign-off evidenced stage 8: Deploy - Fail fast
- Pull the last ninety days of reads on the tables being replaced and contact every reader not on the map.
- Related
- STK-01, SERV-03, SERV-01
Whether the promised value arrives after go-live - adoption and the business changes it depends on (2)
BEN-01Adoption never trackedmediumBusiness owner and sponsor
- Question
- After go-live, will anyone track who is using the new reports and data, compared with the old ones, and act when a user group does not move? Six months on, how will we know the people the project was built for are actually using it?
- Why it matters
- The new dashboards went live on time and the project closed. Usage logs, read a year later, showed most managers had opened them once and gone back to spreadsheets exported from the old system.
- Applies to
- any · components: serving · patterns: any
- Owner
- Business owner and sponsor · also Project manager, Data consumers
- Evidence
- in: plan, status
looks like: An adoption measure per user group (for example weekly active users of the new reports against use of the old ones), a target, a named person who reviews it monthly, and an action when a group lags. - How the gap shows
- No adoption or usage measure in the plan or the benefits list.
- The project closes on go-live with no review of usage afterwards.
- Old reports keep running with no plan to compare their use with the new ones.
- Compare
- Compare the benefits in the business case with the measures planned after go-live: a benefit that depends on people using the new reports, with no usage measure, cannot be shown.
- Due by
plan stage 4: Discovery sign-off evidenced stage 10: Project sign-off - Fail fast
- Agree today which usage figure, read one month after go-live, would tell the sponsor adoption is failing.
- Related
- HAND-05, RLC-03, RLC-02
BEN-02Benefits depend on business changes nobody ownsmediumBusiness owner and sponsor
- Question
- For each promised benefit, is it written down what the business must change to get it (a process, a role, a meeting, a decision that now uses the new data), and who owns that change? If the data arrives perfectly and nothing improves, which business change did nobody make?
- Why it matters
- The business case promised two days saved at each month-end. The data arrived early as designed, but the finance close process was never changed, so the team still waited for the old sign-off steps.
- Applies to
- any · components: any · patterns: any
- Owner
- Business owner and sponsor · also Project manager, Business analyst
- Evidence
- in: discovery, plan
looks like: A benefits map linking each benefit to the data the project delivers and to the business change it needs, with an owner and date for each business change. - How the gap shows
- Benefits are described as outcomes of the data alone, with no business change named.
- Process or role changes are mentioned as "the business will adapt" with no owner.
- No business change tasks appear in any plan.
- Compare
- Compare the benefits in the business case with the scope: a benefit that needs a process change that is outside scope and owned by nobody will not arrive.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 10: Project sign-off - Fail fast
- For the biggest benefit, ask the business owner which process changes on day one and who will change it.
- Related
- HAND-05, REQ-07, STK-06
Technical (91)
Whether it will work: the things that make numbers wrong, loads fail or go-lives slip, plus the effort estimate and technical handover.
Same data arriving or being processed twice (4)
DUP-01Same batch or file arriving twicehighTechnical anchor
- Question
- How is a file, batch or extract that arrives a second time prevented from doubling the numbers?
- Why it matters
- After an outage the source resent yesterday's file, the pipeline loaded it again as new, and revenue on the dashboard doubled for a day before anyone noticed.
- Applies to
- any · components: ingestion · patterns: batch, file, api, database
- Owner
- Technical anchor · also Developers, Test lead
- Evidence
- in: architecture, test-strategy
looks like: A business key or file fingerprint, a record of what has already been loaded, and a test that loads the same input twice. - How the gap shows
- The architecture describes loads as appends with no key, fingerprint or load register.
- The test strategy has no test that resends an input.
- Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test proven stage 9: Production sign-off - Fail fast
- Load the same file twice in a test environment; row counts and key totals in every layer must not change after the second load.
- Related
- DUP-02, DUP-03, REC-01
Check it: Re-drop the same file and rerun; DESCRIBE HISTORY on the target shows no rows inserted (operationMetrics numTargetRowsInserted = 0).
Trap: A file renamed or rewritten by the source counts as new to Auto Loader. Its checkpoint is the memory: deleting or moving it reloads everything.
DUP-02Rerunning a job changes the resulthighTechnical anchor
- Question
- If a job is run again for the same period, does it produce exactly the same result (is every step idempotent)?
- Why it matters
- Operations reran a failed nightly job, but its first half had already appended rows, so the fact table held two copies of the night's orders until month-end reconciliation found them.
- Applies to
- any · components: ingestion, transformation, modelling · patterns: any
- Owner
- Technical anchor · also Developers, Operations and support
- Evidence
- in: architecture, test-strategy, runbook
looks like: Each write described as merge on a key, or replace of a whole partition or period, never a blind append; a rerun test per pipeline; the runbook says reruns are safe. - How the gap shows
- Writes are described as "insert" or "append" with no key or partition overwrite.
- The runbook warns operators not to rerun, or says to clean up by hand before rerunning.
- Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test proven stage 9: Production sign-off - Fail fast
- Run the pipeline twice in a row for the same day in test and compare a fingerprint of every output table (counts, sums of key measures); they must be identical.
- Related
- DUP-01, ORCH-03, TEST-05
Check it: Run the job twice for the same period; compare counts and DESCRIBE HISTORY metrics between runs.
Trap: Repairing a failed Lakeflow Jobs run reruns failed tasks from the start; a task that appended before failing will append again.
DUP-03Duplicates already inside the sourcehighBusiness analyst
- Question
- Does each source have a reliable unique key, and what is the rule when the source itself sends two rows for the same thing?
- Why it matters
- The source table had no primary key and sent the same customer twice with different spellings; joins to it multiplied order rows and inflated sales by eight percent.
- Applies to
- any · components: ingestion, transformation · patterns: any
- Owner
- Business analyst · also Technical anchor, Source system owners
- Evidence
- in: discovery, data-model
looks like: A profile showing key uniqueness per source table, the chosen business key, and an agreed rule for duplicates (keep latest, keep all, reject), signed by the data owner. - How the gap shows
- The data model lists tables without a key, or says "key to be confirmed".
- No profiling results for key uniqueness are recorded.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 7: Test - Fail fast
- Count rows per proposed key in a real extract of every source table; any key with more than one row needs a written rule before design continues.
- Related
- DQ-05, DES-04, SCD-03
Check it: A uniqueness expectation or test on the silver key runs on every load.
Trap: MERGE fails when several source rows match one target row; deduplicate the source batch first. PRIMARY KEY constraints in Unity Catalog are informational and are not enforced.
DUP-04Replayed events in a streamhighTechnical anchor
- Question
- When the stream redelivers events after a restart or failure (at-least-once delivery), how does the target avoid counting them twice?
- Why it matters
- A consumer restarted from an older position and replayed two hours of events; the counts sink appended them again and the real-time dashboard overstated traffic for the day.
- Applies to
- any · components: ingestion, transformation · patterns: streaming, cdc
- Owner
- Technical anchor · also Developers
- Evidence
- in: architecture, test-strategy
looks like: An event id or key plus sequence used to deduplicate, a stated delivery guarantee end to end, and an idempotent or transactional write to the sink. - How the gap shows
- The design assumes "exactly once" without saying which component provides it.
- Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test proven stage 9: Production sign-off - Fail fast
- In test, stop the stream mid-flow, rewind its position by a known number of events, restart, and confirm output counts match the input exactly.
- Related
- DUP-02, ORCH-06, LATE-03
Check it: Restart a stream from an older checkpoint copy in test and compare counts.
Trap: dropDuplicates without a watermark keeps state forever and grows until the job slows or fails.
Initial load, history, backfill (4)
HIST-01History load scope and approachhighTechnical anchor
- Question
- How much history will be loaded, from where, and by what approach (one go, in phases, or from archive), and who checks the old numbers?
- Why it matters
- The plan assumed history would come from the same feed as daily loads, but the source only kept ninety days; five years had to come from an archive in another format, adding six weeks.
- Applies to
- any · components: ingestion · patterns: any
- Owner
- Technical anchor · also Business owner and sponsor, Source system owners
- Evidence
- in: discovery, architecture, deployment
looks like: The number of years in scope, where each year comes from, the load approach and order, the expected volume, and who signs off that historical numbers are right. - How the gap shows
- History is not mentioned, or "history to be loaded" appears with no years or source.
- The plan has no task or time for the history load.
- Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test proven stage 9: Production sign-off - Fail fast
- Ask the source owner how far back the source really holds data, and pull one month of the oldest year in scope through the pipeline.
- Related
- HIST-02, HIST-03, REC-02
Check it: Reconcile each historical year against the source or finance before switching on daily loads.
HIST-02Source can bear the history extractmediumTechnical anchor
- Question
- Can the source system take the load of the history extract, and is the extract split into chunks that can each be retried?
- Why it matters
- A single five-year query against the source database ran for nine hours, hit an undo limit and failed at ninety percent; it failed again the next night and slowed the source for its users.
- Applies to
- any · components: ingestion · patterns: database, cdc, api, history load
- Owner
- Technical anchor · also Source system owners
- Evidence
- in: architecture, deployment, raid
looks like: An extract split by date or key range, an agreed extract window with the source owner, and an estimate of run time per chunk from a real trial. - How the gap shows
- The history load is one step with no chunking or restart point.
- Source owners are not listed as reviewers of the extract approach.
- Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test - Fail fast
- Time the extract of one realistic chunk against the real source in its normal busy window and multiply out; agree the result with the source owner.
- Related
- HIST-01, PERF-01
Check it: Time one chunk against the source in its busy window.
Trap: Too many partitions in a JDBC read opens that many connections at once against the source.
HIST-03Handing over from history load to incremental loadshighTechnical anchor
- Question
- How does the switch from the history or full load to incremental loads avoid both a gap and an overlap in the data?
- Why it matters
- The full snapshot finished at 10:00 but incremental capture was started at 11:00; an hour of changes was never loaded and the totals were quietly short for months.
- Applies to
- any · components: ingestion · patterns: cdc, database, streaming, history load
- Owner
- Technical anchor · also Developers
- Evidence
- in: architecture, deployment
looks like: A defined high-water mark or change position taken before the snapshot starts, incremental capture starting from it, and overlapping rows absorbed by a merge on key. - Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test proven stage 9: Production sign-off - Fail fast
- In test, change rows in the source while the full load runs, then start incremental; every changed row must appear once with its latest value.
- Related
- HIST-01, DUP-01, REC-01
Check it: Change rows during the snapshot in test and check each ends with its latest value once.
HIST-04Reloading or backfilling a past periodmediumTechnical anchor
- Question
- Can a single past period (a day, a month) be reloaded or backfilled on demand without touching other periods or rebuilding everything?
- Why it matters
- A source bug corrupted three days in March; the only way to fix them was a full rebuild that took the platform down for a weekend.
- Applies to
- any · components: ingestion, transformation, modelling · patterns: batch, cdc, file, database, api
- Owner
- Technical anchor · also Operations and support, Developers
- Evidence
- in: architecture, runbook
looks like: Pipelines parameterised by period, writes that replace exactly that period, a runbook step for reloading a period, and a note on which downstream tables must also be rebuilt. - How the gap shows
- The runbook has no reload or backfill procedure.
- Jobs read "today" from the clock instead of taking the period as a parameter.
- Due by
plan stage 4: Discovery sign-off designed stage 5: MVP starts evidenced stage 7: Test proven stage 10: Project sign-off - Fail fast
- In test, reload one day from three days ago with a parameter and check that only that day's rows and its downstream aggregates changed.
- Related
- DUP-02, LATE-01, HAND-01
Check it: Reload one past day in test; DESCRIBE HISTORY shows only that day's files were rewritten.
Trap: A full refresh of a Lakeflow pipeline streaming table re-reads all source data and clears the table first; check source retention before pressing it.
Late arriving and corrected data (4)
LATE-01Late and corrected rows rebuild the right periodshighTechnical anchor
- Question
- When rows arrive late or yesterday's figures are corrected next week, which periods are rebuilt, and how far back can corrections reach?
- Why it matters
- Returns were booked against sales from three weeks earlier, but the pipeline only rebuilt the last two days, so monthly net revenue never matched the finance ledger.
- Applies to
- any · components: ingestion, transformation, modelling · patterns: any
- Owner
- Technical anchor · also Business analyst, Source system owners
- Evidence
- in: discovery, architecture
looks like: A profile of how late data really arrives, an agreed correction window, and a design that reprocesses affected periods by business date rather than load date. - How the gap shows
- Aggregates are described as built "for the load date" only.
- Nothing in the discovery document says how late source data can arrive.
- Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test proven stage 9: Production sign-off - Fail fast
- In a real extract, compare business date against load date and find the longest lag; then insert a row two weeks late in test and check the monthly total updates.
- Related
- LATE-02, HIST-04, REC-01
Check it: Insert a row with an old business date in test and check the gold total for that date changes.
LATE-02Facts arriving before their reference datahighTechnical anchor
- Question
- What happens to a fact row whose customer, product or other reference record has not arrived yet?
- Why it matters
- Orders for new products arrived hours before the product feed; an inner join silently dropped them, and the new-launch report showed zero sales on launch day.
- Applies to
- any · components: transformation, modelling · patterns: any
- Owner
- Technical anchor · also Developers
- Evidence
- in: architecture, data-model
looks like: An "unknown" or placeholder member, or a quarantine with automatic reprocessing, and a step that re-points facts once the reference record arrives. - How the gap shows
- Joins between facts and reference tables are described as inner joins with no handling of misses.
- Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test - Fail fast
- Load a fact row whose reference key does not exist yet, then load the reference row; the fact must be kept and end up attached to the right reference record.
- Related
- LATE-01, DQ-02, REC-01
Check it: An expectation or test counts facts pointing at the unknown member over time.
LATE-03Event time, watermarks and late events in streamsmediumTechnical anchor
- Question
- Do streaming aggregations use event time, how long do they wait for late events (the watermark), and what happens to events later than that?
- Why it matters
- The window waited ten minutes, but mobile clients uploaded events hours later after going offline; those events were silently discarded and daily active users were understated.
- Applies to
- any · components: transformation, modelling · patterns: streaming
- Owner
- Technical anchor · also Developers, Data consumers
- Evidence
- in: architecture, requirements
looks like: The lateness of real events measured, a watermark chosen from that, and a stated home for events beyond it (a correction batch or a side table) agreed with consumers. - Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test proven stage 9: Production sign-off - Fail fast
- Measure the gap between event time and arrival time in a day of real events and see what share falls beyond the proposed watermark.
- Related
- LATE-01, DUP-04, PERF-02
Check it: Check the streaming query progress metrics for numRowsDroppedByWatermark.
Trap: A watermark that is too long holds more state and costs memory; too short silently drops events.
LATE-04Time zones and business datesmediumBusiness analyst
- Question
- Which time zone does each source use, how are timestamps stored, and which time zone defines a business day?
- Why it matters
- One source sent local time and another sent universal time; orders placed late in the evening fell into the next day and daily sales never matched the stores' own reports.
- Applies to
- any · components: ingestion, transformation, modelling · patterns: any
- Owner
- Business analyst · also Technical anchor
- Evidence
- in: data-model, discovery
looks like: The time zone of every source timestamp, a single storage convention, a defined business-day time zone, and a rule for daylight-saving changes. - How the gap shows
- The data dictionary lists timestamps without a time zone.
- Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test - Fail fast
- Pick ten real transactions near midnight and check that their business date matches what the source application shows.
- Related
- DES-04, REC-02
Check it: Compare business dates of near-midnight rows with the source application.
Trap: TIMESTAMP_NTZ has no time zone; mixing it with TIMESTAMP across sources shifts hours.
Deletes and soft deletes at the source (3)
DEL-01Detecting hard deletes at the sourcehighTechnical anchor
- Question
- When a record is physically deleted in the source, how does the platform find out?
- Why it matters
- Cancelled orders were deleted in the source, but the incremental extract only picked up new and changed rows, so the cancelled orders stayed in the warehouse and inflated sales.
- Applies to
- any · components: ingestion · patterns: batch, database, api, file, cdc
- Owner
- Technical anchor · also Source system owners
- Evidence
- in: discovery, architecture
looks like: For each source, whether deletes happen and how they are captured: change capture with delete events, a periodic full key comparison, or a delete feed from the source. - How the gap shows
- Incremental loads use a modified-date filter and deletes are not mentioned.
- Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test - Fail fast
- Ask the source owner whether rows are ever deleted, then delete a test row in a source test system and run the load; the platform must reflect it.
- Related
- DEL-02, DEL-03, REC-01
Check it: Delete a row in a source test system and check it disappears or is flagged after the load.
Trap: WHEN NOT MATCHED BY SOURCE against a partial extract deletes everything not in that extract.
DEL-02What a delete means downstreammediumBusiness analyst
- Question
- When a source deletes or soft-deletes a record, must it disappear, be flagged, or stay in each layer and report, and who decided?
- Why it matters
- Engineers removed deleted customers from every table, but finance needed them kept for historic invoices, and last year's revenue dropped after a routine load.
- Applies to
- any · components: transformation, modelling, serving · patterns: any
- Owner
- Business analyst · also Technical anchor, Data consumers
- Evidence
- in: requirements, data-model
looks like: A rule per entity (remove, flag with a deleted date, or keep) agreed with the data owner, and how flagged rows are filtered in reports. - Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test - Fail fast
- Write the delete rule for the top three entities and get the data owner to sign them in one meeting.
- Related
- DEL-01, SCD-01, SEC-03
DEL-03Deletes flow through derived tablesmediumTechnical anchor
- Question
- When a source row is deleted, are the aggregates, summaries and extracts built from it recalculated too?
- Why it matters
- The detail table dropped a deleted order, but the daily summary built from it was only ever appended to, so detail and summary disagreed and users stopped trusting both.
- Applies to
- any · components: transformation, modelling, serving · patterns: any
- Owner
- Technical anchor · also Developers
- Evidence
- in: architecture, test-strategy
looks like: Derived tables rebuilt for the affected periods when deletes arrive, and a test that checks detail and summary still agree after a delete. - Due by
plan stage 4: Discovery sign-off designed stage 5: MVP starts evidenced stage 7: Test - Fail fast
- Delete one source row in test, run the full chain, and compare the summary total against the sum of the detail.
- Related
- DEL-01, LATE-01, REC-01
Check it: After a test delete, gold totals equal the sum of silver detail.
Keeping history of changes (slowly changing dimensions) (4)
SCD-01Which tables keep history, and for how longmediumBusiness analyst
- Question
- For each table, has the business decided whether to keep the history of changes or only the latest value, and for how long?
- Why it matters
- Customer region was stored as latest only; when a sales team was reorganised, last year's sales by region were silently restated and the bonus calculation was disputed.
- Applies to
- any · components: modelling, transformation · patterns: any
- Owner
- Business analyst · also Technical anchor, Data consumers
- Evidence
- in: requirements, data-model, decisions
looks like: A list of entities and attributes with "history kept" or "latest only", a retention period, and the business owner who agreed it. - How the gap shows
- The data model has no statement about history for dimensions.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 7: Test - Fail fast
- Ask the main report owner "should last year's figures change when a customer moves region?" and record the answer as a decision.
- Related
- SCD-02, DEL-02
SCD-02History rows stay correct on reruns and out-of-order changeshighDevelopers
- Question
- Does the history logic (effective from and to dates, current flag) stay correct when a load is rerun, when two changes arrive in one batch, or when changes arrive out of order?
- Why it matters
- A rerun closed and reopened every current row, and an older change arriving late became "current", so customers showed last year's address with overlapping validity periods.
- Applies to
- any · components: modelling, transformation · patterns: any
- Owner
- Developers · also Technical anchor, Test lead
- Evidence
- in: architecture, test-strategy
looks like: Changes ordered by a source sequence or change timestamp, not arrival time; a hash to skip unchanged rows; and tests for rerun, two-changes-per-batch and out-of-order cases. - Due by
plan stage 4: Discovery sign-off designed stage 5: MVP starts evidenced stage 7: Test proven stage 9: Production sign-off - Fail fast
- Feed three versions of one record in reverse order, then rerun the same batch; exactly one current row and no overlapping date ranges may remain.
- Related
- SCD-01, DUP-02, SCD-04
Check it: Feed reversed versions and a rerun in test; check no overlapping __START_AT / __END_AT ranges.
Trap: Sequencing by ingestion time instead of a source sequence breaks history when changes arrive out of order.
SCD-03A business key that can changemediumTechnical anchor
- Question
- Can a source's business key ever change (renumbering, merges, re-registration), and how is the identity of the thing kept across that change?
- Why it matters
- The source renumbered accounts during a migration; each renumbered account became a new customer, splitting its history and doubling the customer count.
- Applies to
- any · components: modelling, transformation · patterns: any
- Owner
- Technical anchor · also Source system owners, Business analyst
- Evidence
- in: data-model, discovery
looks like: Surrogate keys owned by the platform, a mapping from old to new source keys, and the source owner's confirmation of whether keys can change. - Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test - Fail fast
- Ask each source owner directly whether keys are ever reused, merged or renumbered, and check a real extract for one key mapping to two names over time.
- Related
- DUP-03, DES-04
Trap: IDENTITY values are unique but not consecutive, and declaring one limits concurrent writes to the table.
SCD-04Facts joined to the right version of historymediumTechnical anchor
- Question
- When facts are joined to a dimension that keeps history, are they matched to the version valid at the time of the fact, and is that the rule consumers expect?
- Why it matters
- Facts were joined to the current customer row; reports that should show sales by the region at the time showed today's region instead, and nobody had written down which was wanted.
- Applies to
- any · components: modelling, serving · patterns: any
- Owner
- Technical anchor · also Data consumers
- Evidence
- in: data-model, requirements
looks like: For each report, whether it shows "as it was" or "as it is now", and joins that use the fact date within the validity range or the surrogate key captured at load. - Due by
plan stage 4: Discovery sign-off designed stage 5: MVP starts evidenced stage 7: Test - Fail fast
- Take one customer who changed region last year and check a sales report shows each sale under the region the business expects.
- Related
- SCD-02, SERV-04
Columns added, renamed or retyped at the source (3)
SCH-01A new column appears at the sourcemediumTechnical anchor
- Question
- When a source adds a column, does the pipeline fail, drop it silently, or carry it through, and is that the agreed behaviour per layer?
- Why it matters
- The source added a discount column; the raw layer dropped it silently, and when finance asked for it three months later the history could not be recovered.
- Applies to
- any · components: ingestion, transformation · patterns: any
- Owner
- Technical anchor · also Developers
- Evidence
- in: architecture
looks like: A schema policy per layer (for example: raw keeps everything including unknown columns, cleaned layers change only through a reviewed change), and an alert when a new column appears. - How the gap shows
- The architecture does not mention schema changes at all.
- Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test - Fail fast
- Add a column to a test source file or table and run the pipeline; see what each layer does and whether anyone is told.
- Related
- SCH-02, SCH-03
Check it: Drop a test file with an extra column and watch what the stream and the target do.
Trap: With addNewColumns the stream fails on purpose and must be restarted; make sure the job has retries, or it stays down overnight.
SCH-02A column renamed, removed or retypedhighTechnical anchor
- Question
- When a source renames, removes or changes the type of a column, does the pipeline stop loudly before writing wrong data?
- Why it matters
- A source changed an amount column from cents to a decimal in dollars; the pipeline cast it happily and revenue fell a hundredfold in the morning report.
- Applies to
- any · components: ingestion, transformation · patterns: any
- Owner
- Technical anchor · also Developers, Operations and support
- Evidence
- in: architecture, test-strategy
looks like: An expected schema checked on every load, breaking changes failing the load with a clear alert, and range or distribution checks on key measures. - Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test proven stage 9: Production sign-off - Fail fast
- In test, rename one column and change one type in a source sample; the load must fail with a message that names the column.
- Related
- SCH-01, DQ-01, OPS-01
Check it: A test file with a renamed or retyped column fails the load with a clear error.
Trap: Type mismatches can land in _rescued_data quietly; alert when that column is not null.
SCH-03Source owners warn before they changemediumTechnical anchor
- Question
- Is there an agreement (a data contract or a change notice) that source owners tell the data team before changing structure or meaning?
- Why it matters
- The source team released a new version on a Friday with restructured tables; nobody told the data team and the weekend loads failed, missing a regulatory deadline.
- Applies to
- any · components: ingestion · patterns: any
- Owner
- Technical anchor · also Source system owners, Project manager
- Evidence
- in: discovery, stakeholders, raid
looks like: A written agreement or contract per source naming the schema, the notice period for changes, and a contact; source release calendars known to the team. - How the gap shows
- Source owners are not in the stakeholder map.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 9: Production sign-off - Fail fast
- Ask each source owner when their next release is and whether it changes anything the project reads.
- Related
- SCH-02, SERV-03
Rules, bad rows, quarantine (5)
DQ-01Data quality rules written with ownershighBusiness analyst
- Question
- Which data quality rules apply to each important table, who owns each rule, and are they written down as checks that run on every load?
- Why it matters
- Everyone assumed someone else checked that order totals were positive; a source bug sent negative totals for a week and nobody's check caught it.
- Applies to
- any · components: ingestion, transformation, modelling · patterns: any
- Owner
- Business analyst · also Technical anchor, Test lead
- Evidence
- in: requirements, data-model, test-strategy
looks like: A rule list per table (not null, unique, valid values, ranges, referential integrity, business rules), each with a business owner and a severity, implemented as automated checks. - How the gap shows
- The documents say "data quality checks will be applied" with no list of rules.
- Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test proven stage 9: Production sign-off - Fail fast
- Ask the business owner for the five things that would make them distrust the main report, and write each as a rule that runs on today's data.
- Related
- DQ-02, DQ-05, TEST-01
Check it: Query the pipeline event log or Data quality tab for pass and fail counts per rule.
DQ-02What happens to a row that fails a rulehighTechnical anchor
- Question
- When a row fails a rule, is the batch rejected, the row quarantined, or the row loaded and flagged, and who is told and fixes it?
- Why it matters
- Bad rows were filtered out silently; three thousand orders a day disappeared into a table no one looked at, and sales were understated for a quarter.
- Applies to
- any · components: ingestion, transformation · patterns: any
- Owner
- Technical anchor · also Business analyst, Operations and support
- Evidence
- in: architecture, runbook, requirements
looks like: An action per rule severity, a quarantine table with reasons, a count of quarantined rows reported to an owner, and a way to reprocess rows once fixed. - Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test proven stage 9: Production sign-off - Fail fast
- Feed a sample with known bad rows in test and check each lands where the design says, with a reason, and that an owner is notified.
- Related
- DQ-01, DQ-03, OPS-01
Check it: Load known-bad rows in test and find each one in quarantine.
Trap: expect_or_drop only counts dropped rows; the rows themselves are gone unless you quarantine them.
DQ-03One poison record does not stop the whole loadmediumDevelopers
- Question
- Can a single malformed record (broken encoding, unparseable value, oversized field) stop the whole pipeline, and how is it isolated?
- Why it matters
- One record with a malformed date crashed the stream parser on every retry; the pipeline was stuck for six hours while all later data waited behind it.
- Applies to
- any · components: ingestion · patterns: any
- Owner
- Developers · also Technical anchor
- Evidence
- in: architecture, test-strategy
looks like: Parsing that captures unreadable records into a side table with the raw content and the error, and a test with deliberately malformed input. - Due by
plan stage 4: Discovery sign-off designed stage 5: MVP starts evidenced stage 7: Test - Fail fast
- Put one malformed record in the middle of a test input; the rest must load and the bad record must be kept somewhere readable.
- Related
- DQ-02, ORCH-02
Check it: Put a malformed record in a test file; the rest loads and the bad record is readable.
DQ-04A source goes quiet or sends far too littlehighOperations and support
- Question
- Does anything detect a source that sends no data, an empty file, or a much smaller or larger volume than usual?
- Why it matters
- The source sent an empty file after a failed export; the load succeeded, every job was green, and the business saw a day with zero sales before any engineer did.
- Applies to
- any · components: ingestion · patterns: any
- Owner
- Operations and support · also Technical anchor
- Evidence
- in: architecture, runbook
looks like: Freshness and volume checks per source with expected ranges, which fail or alert rather than load quietly. - How the gap shows
- Monitoring is described only as "job failure alerts".
- Due by
plan stage 4: Discovery sign-off designed stage 5: MVP starts evidenced stage 7: Test proven stage 9: Production sign-off - Fail fast
- Run a load with an empty input in test; it must not complete as a green success without someone being told.
- Related
- OPS-01, DQ-01
Check it: Load an empty file in test and confirm an alert fires.
Trap: Monitoring results arrive after the load; a gate inside the job is still needed to stop bad data reaching gold.
DQ-05Real data profiled before designhighTechnical anchor
- Question
- Has real data from every source been profiled (nulls, duplicates, broken keys, odd values, volumes), and was the profile shown to the business owner?
- Why it matters
- The design assumed every order had a customer; profiling in build week three found twenty percent did not, and the model and estimate both had to be redone.
- Applies to
- any · components: ingestion, transformation, modelling · patterns: any
- Owner
- Technical anchor · also Business analyst, Business owner and sponsor
- Evidence
- in: discovery
looks like: A profile per source table from a real extract (not a sample built by hand), with findings and what was decided about each. - How the gap shows
- Discovery describes sources from documentation only, with no profiling results.
- Due by
plan stage 2: Rough estimate designed stage 3: Discovery evidenced stage 4: Discovery sign-off - Fail fast
- Pull a real extract of the biggest source table this week and count nulls, duplicates and orphan keys in the columns the design relies on.
- Related
- DUP-03, DQ-01, DES-01
Proving numbers match from source to report (4)
REC-01Counts and totals reconciled layer by layerhighTechnical anchor
- Question
- Are row counts and key totals compared automatically between the source and each layer on every load, with the differences explained?
- Why it matters
- The report was four percent short of the source and nobody could say in which layer rows had been lost, so a week went into tracing one join that dropped orphan rows.
- Applies to
- any · components: ingestion, transformation, modelling · patterns: any
- Owner
- Technical anchor · also Test lead, Developers
- Evidence
- in: test-strategy, architecture
looks like: Counts and sums of key measures captured at the source and after each layer for every run, stored, compared, and every expected difference (filters, quarantined rows) accounted for. - How the gap shows
- Reconciliation appears only as a one-off test activity, not as part of every run.
- Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test proven stage 9: Production sign-off - Fail fast
- For one day of real data, write down the source count and total, then the count and total in each layer, and explain every difference.
- Related
- REC-02, REC-03, DUP-01
Check it: A dashboard or query compares source and each layer per run.
REC-02Matched against a number the business already trustshighTest lead
- Question
- Are the final numbers checked against an independent figure the business already trusts (a finance ledger, an existing report, a source application screen)?
- Why it matters
- Source-to-target counts all matched, yet the new revenue figure disagreed with the finance ledger because the definition of revenue differed; it was found only at user acceptance.
- Applies to
- any · components: modelling, serving · patterns: any
- Owner
- Test lead · also Business owner and sponsor, Data consumers
- Evidence
- in: test-strategy, requirements
looks like: A named trusted figure for each key measure, the comparison method, and an agreed explanation of known differences. - Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test proven stage 9: Production sign-off - Fail fast
- Ask the business owner which existing number they would compare against, and compare one month of thin-slice output with it now.
- Related
- REC-01, SERV-04, TEST-04
REC-03Tolerances and who signs off differencesmediumBusiness owner and sponsor
- Question
- What size of difference is acceptable for each measure, and who decides when a difference is explained well enough to go live?
- Why it matters
- Test found a 0.3 percent gap; engineering called it rounding, finance called it a blocker, and go-live slipped two weeks while they argued.
- Applies to
- any · components: modelling, serving · patterns: any
- Owner
- Business owner and sponsor · also Test lead, Technical anchor
- Evidence
- in: test-strategy, decisions
looks like: A tolerance per measure and a named person who accepts reconciliation results. - Due by
plan stage 4: Discovery sign-off designed stage 5: MVP starts evidenced stage 7: Test - Fail fast
- Put the tolerance question on the next sign-off agenda and record a number for the top three measures.
- Related
- REC-01, REC-02
REC-04First production loads reconciledhighTechnical anchor
- Question
- Were the first production loads, including the history load, reconciled against the source and trusted figures before users were told to rely on them?
- Why it matters
- Production used a different source instance with extra test records; nobody reconciled the first loads, and dummy orders appeared in the board pack.
- Applies to
- any · components: ingestion, modelling · patterns: any
- Owner
- Technical anchor · also Operations and support, Business owner and sponsor
- Evidence
- in: deployment, status
looks like: A reconciliation result for each of the first production runs, with sign-off, recorded in the hypercare or go-live report. - Due by
plan stage 6: Build designed stage 8: Deploy evidenced stage 9: Production sign-off proven stage 9: Production sign-off - Fail fast
- Add "reconcile first production load" as a go-live checklist item with an owner and a tolerance.
- Related
- REC-01, HIST-01, OPS-05
Volume, speed, freshness (5)
PERF-01Volumes and growth statedmediumTechnical anchor
- Question
- What are the real volumes per source today (rows, size, files per day), and how much are they expected to grow over the next two to three years?
- Why it matters
- The design was sized from a test sample of a million rows; production sent eight hundred million on month-end and the nightly window was blown on the first close.
- Applies to
- any · components: any · patterns: any
- Owner
- Technical anchor · also Source system owners, Architect
- Evidence
- in: discovery, architecture
looks like: Measured volumes per source from real data, peak days (month-end, seasonal), and a growth assumption. - How the gap shows
- Volumes are missing, or given as "large" or "medium" rather than numbers.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 7: Test - Fail fast
- Ask each source owner for the row count of their biggest table on their busiest day, and check one against a real extract.
- Related
- PERF-03, COST-01, HIST-02
PERF-02Freshness target measured end to endhighTechnical anchor
- Question
- What freshness or latency is promised (for example, ready by 07:00, or within one minute), and has the full path from source to consumer been timed against it?
- Why it matters
- Each step met its own target, but source export, transfer, three layers and the report refresh added up to ninety minutes against a promised thirty.
- Applies to
- any · components: any · patterns: any
- Owner
- Technical anchor · also Data consumers, Architect
- Evidence
- in: requirements, architecture, test-strategy
looks like: A freshness target per consumer, a time budget split across every hop, and a measured end to end time from the thin slice or a test run. - Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 7: Test proven stage 9: Production sign-off - Fail fast
- Time one record from source change to visible in the report in the thin slice, and compare with the target.
- Related
- PERF-03, SERV-02, DES-06
PERF-03Tested at production volumehighTest lead
- Question
- Has the pipeline been run at production volume, including a peak day and the history load, before go-live?
- Why it matters
- Every test ran on a tenth of the data; the first month-end in production ran out of memory in the biggest join and the finance close waited a day.
- Applies to
- any · components: ingestion, transformation, modelling · patterns: any
- Owner
- Test lead · also Technical anchor, Platform team
- Evidence
- in: test-strategy
looks like: A performance test at peak production volume with run times, compute size and cost recorded, and the result compared with the freshness target. - How the gap shows
- The test strategy has no performance or volume test, or uses only small samples.
- Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test proven stage 9: Production sign-off - Fail fast
- Run the heaviest transformation on one full peak day of real (or realistically generated) data and time it.
- Related
- PERF-01, PERF-04, COST-01
Check it: Record run time, compute size and cost from system.lakeflow.job_run_timeline and system.billing.usage.
PERF-04Skewed keys and very large joinsmediumDevelopers
- Question
- Are there hot keys (one customer, one default value, nulls) or joins between two very large tables that will make some steps run far slower than the rest?
- Why it matters
- Half of all rows carried a placeholder customer id; in the big join one worker processed that key for an hour while the others sat idle, and the job missed its window.
- Applies to
- any · components: transformation, modelling · patterns: any
- Owner
- Developers · also Technical anchor
- Evidence
- in: architecture, discovery
looks like: Key distributions profiled for the large join keys, and a strategy for skewed keys and large joins (filtering defaults, splitting hot keys, pre-aggregating). - Due by
plan stage 4: Discovery sign-off designed stage 5: MVP starts evidenced stage 7: Test - Fail fast
- Count rows per join key for the two largest joins and look at the top ten keys' share.
- Related
- PERF-03, DQ-05
Check it: In the Spark UI, compare the longest task against the median for the big join stage.
PERF-05Storage layout and file sizes kept healthymediumDevelopers
- Question
- How are table layout (partitioning or clustering) and file sizes chosen and maintained, so frequent small writes do not slow reads over time?
- Why it matters
- A streaming load wrote thousands of tiny files an hour; reads slowed every week until a simple query took ten minutes and cost ten times more.
- Applies to
- any · components: ingestion, transformation, modelling · patterns: any
- Owner
- Developers · also Platform team
- Evidence
- in: architecture, runbook
looks like: Layout chosen from the main filter and join columns, a scheduled compaction and cleanup job, and a metric for average file size. - Due by
plan stage 4: Discovery sign-off designed stage 5: MVP starts evidenced stage 7: Test proven stage 10: Project sign-off - Fail fast
- After a week of test loads, count files and average file size for the busiest table.
- Related
- COST-04, PERF-03, SEC-03
Check it: DESCRIBE DETAIL shows numFiles and sizeInBytes; DESCRIBE HISTORY shows OPTIMIZE runs.
Trap: Predictive optimization covers managed tables only; external tables need scheduled OPTIMIZE and VACUUM. Check that clustering keys match the MERGE condition, or merges scan the whole table.
Running cost and its controls (4)
COST-01Running cost estimated from a real runmediumTechnical anchor
- Question
- What will the solution cost to run each month (compute, storage, transfer, licences), and is the figure based on a measured run rather than a guess?
- Why it matters
- The business case assumed a few hundred a month; the first production bill was eleven thousand because the streaming jobs ran around the clock, and the project was paused for review.
- Applies to
- any · components: any · patterns: any
- Owner
- Technical anchor · also Platform team, Business owner and sponsor
- Evidence
- in: discovery, architecture, estimate
looks like: A monthly running cost estimate broken down by job and storage, derived from the thin slice or a test run scaled to production volume, agreed with the budget owner. - How the gap shows
- Cost appears only as build effort, with no running cost.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 7: Test proven stage 9: Production sign-off - Fail fast
- Read the actual cost of the thin slice's runs and multiply by production volume and schedule.
- Related
- COST-02, PERF-01, DES-06
Check it: Compare the estimate with the first month of production usage from the same tables.
Trap: List price is not your contract price; agree discounts with whoever holds the contract.
COST-02Platform cost guardrailshighPlatform team
- Question
- What stops a single team, job or user from running up an unexpected bill (budgets, alerts, limits on compute size, automatic shutdown)?
- Why it matters
- A developer left a large compute cluster running over a holiday weekend, and the month's budget was gone by the fourth.
- Applies to
- new platform · components: platform · patterns: any
- Owner
- Platform team · also Security and governance, Business owner and sponsor
- Evidence
- in: platform, runbook
looks like: Budgets with alerts per team or project, policies that cap compute size and enforce automatic shutdown, and a named person who receives cost alerts. - Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 6: Build proven stage 9: Production sign-off - Fail fast
- Try to create the largest compute allowed as an ordinary developer in the development environment; the policy should stop you.
- Related
- COST-03, COST-04, ENV-02
Check it: Try to create oversized compute as a developer in development; the policy should block it.
Trap: Budgets alert, they do not cap general spend; serverless usage policies only tag usage for attribution. Real limits come from policies and permissions.
COST-03Cost can be traced to a team and a joblowPlatform team
- Question
- Can the running cost be broken down by project, team, environment and job (by tags or separate accounts), so overruns can be traced?
- Why it matters
- The platform bill doubled in one month and nobody could say which project caused it, so every team was asked to cut costs instead of the one that needed to.
- Applies to
- any · components: platform, orchestration · patterns: any
- Owner
- Platform team · also Technical anchor
- Evidence
- in: platform, architecture
looks like: A mandatory tagging standard (project, owner, environment), enforced on compute and storage, and a cost report by tag. - Due by
plan stage 4: Discovery sign-off designed stage 5: MVP starts evidenced stage 7: Test proven stage 9: Production sign-off - Fail fast
- Pull last week's cost broken down by project tag; check how much is untagged.
- Related
- COST-02, OPS-03
Check it: Query the share of last week's usage with no project tag.
COST-04No compute running when nothing needs itmediumDevelopers
- Question
- Is compute sized from real runs and does it stop when idle, rather than staying on or oversized "to be safe"?
- Why it matters
- A job that needed ten minutes of compute held a large cluster for two hours each run because it waited on an upstream file, and nobody noticed for six months.
- Applies to
- any · components: ingestion, transformation, modelling, orchestration · patterns: any
- Owner
- Developers · also Platform team, Operations and support
- Evidence
- in: architecture, runbook
looks like: Compute sizes justified by measured runs, automatic stop or job-scoped compute, waits on upstream data done outside paid compute, and a periodic review of utilisation. - Due by
plan stage 4: Discovery sign-off designed stage 5: MVP starts evidenced stage 7: Test proven stage 9: Production sign-off - Fail fast
- For the three most expensive jobs, compare the time compute was running against the time it was busy.
- Related
- COST-01, COST-02, PERF-05
Check it: Compare job duration with compute uptime in system tables for the top jobs.
Trap: A continuous Lakeflow pipeline or stream runs compute all day; use triggered mode when freshness allows.
Access, masking, retention, regulations (5)
SEC-01Personal and sensitive data found and classifiedhighSecurity and governance
- Question
- Which columns hold personal or sensitive data, in which sources and layers, and is each classified?
- Why it matters
- Free-text notes held customer phone numbers and health details; nobody had classified them, so they were copied unmasked into a table hundreds of analysts could read.
- Applies to
- any · components: any · patterns: any
- Owner
- Security and governance · also Business analyst, Technical anchor
- Evidence
- in: data-model, discovery, platform
looks like: A classification for every column in scope (public, internal, confidential, personal, special category), including free-text fields, recorded in the data dictionary or catalog. - How the gap shows
- Personal data is not mentioned, or mentioned only as "will follow policy".
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 7: Test - Fail fast
- Scan a real extract of each source for patterns like emails, phone numbers and national ids, including in free-text columns.
- Related
- SEC-02, SEC-03, SEC-05
Check it: Query information_schema column tags to list sensitive columns and their owners.
SEC-02Who can see whathighSecurity and governance
- Question
- Who may see which tables, rows and columns, and how is that enforced (masking, row filters, separate views) rather than trusted?
- Why it matters
- Regional managers were meant to see only their own region, but the report filtered it in the front end; anyone querying the table directly saw every region's salaries.
- Applies to
- any · components: modelling, serving, governance · patterns: any
- Owner
- Security and governance · also Technical anchor, Data consumers
- Evidence
- in: platform, architecture, requirements
looks like: An access matrix by group, enforced in the data platform itself with masking and row-level rules, and tests that run as each persona. - Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test proven stage 9: Production sign-off - Fail fast
- Log in as a test user of the lowest-privilege persona and query the most sensitive table directly.
- Related
- SEC-01, ENV-02, SEC-04
Check it: Query the table as a test user from the lowest persona.
Trap: Row filters and masks need recent runtimes (older compute returns no data), cannot be put on views, and block time travel and clones on that table.
SEC-03Retention and the right to be forgottenhighSecurity and governance
- Question
- How long is each kind of data kept, and how is a person's data erased everywhere it lives, including history tables, old table versions, backups and extracts?
- Why it matters
- A customer's erasure request deleted their current row, but the history table, old file versions kept for recovery and a weekly extract to a partner still held their data.
- Applies to
- any · components: governance, modelling, serving, ingestion · patterns: any
- Owner
- Security and governance · also Technical anchor, Operations and support
- Evidence
- in: platform, architecture, runbook
looks like: A retention period per data class, an erasure procedure that covers every layer, history, versions and copies, and the time it takes to complete. - Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test proven stage 10: Project sign-off - Fail fast
- Trace one test person through every table, version and extract in test and count where their data would remain after the erasure procedure.
- Related
- SEC-01, DEL-02, PERF-05
Check it: After the procedure, time travel to an earlier version and confirm the data is gone once retention has passed.
Trap: Time travel, deep clones and exports keep copies until each is cleaned too.
SEC-04Access is auditedmediumSecurity and governance
- Question
- Is every read and change of sensitive data logged with who, what and when, and does someone review it?
- Why it matters
- After a data leak, the team could not say who had queried the customer table in the last month, and the regulator treated the whole table as exposed.
- Applies to
- new platform · components: platform, governance · patterns: any
- Owner
- Security and governance · also Platform team
- Evidence
- in: platform, runbook
looks like: Audit logs switched on for data access, kept for the regulated period, and a named reviewer with a review frequency. - Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test proven stage 9: Production sign-off - Fail fast
- Query a sensitive test table, then find that query in the audit log within an hour.
- Related
- SEC-02, OPS-03
Check it: Find a test query against a sensitive table in system.access.audit.
SEC-05Regulations and where data may livehighSecurity and governance
- Question
- Which regulations apply to the data (privacy, industry, residency), and does the design keep data, copies and processing in permitted locations?
- Why it matters
- A disaster-recovery copy was configured to another region outside the permitted jurisdiction; the audit found it a week before go-live and the platform design had to change.
- Applies to
- any · components: platform, governance · patterns: any
- Owner
- Security and governance · also Architect, Platform team
- Evidence
- in: platform, decisions
looks like: A list of applicable regulations, permitted regions for storage, processing and backups, and sign-off from the data protection or compliance owner. - Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 8: Deploy - Fail fast
- Ask the compliance owner which locations are permitted, and list every place the design puts a copy of the data.
- Related
- SEC-01, ORCH-05
Environments, permissions, network (5)
ENV-01Separate environments with separate datahighPlatform team
- Question
- Are there separate development, test and production environments, isolated so work in one cannot read or overwrite another's data?
- Why it matters
- A developer ran a cleanup script pointed at what they thought was development; the paths were shared and production tables were wiped the night before a board meeting.
- Applies to
- new platform · components: platform · patterns: any
- Owner
- Platform team · also Architect, Security and governance
- Evidence
- in: platform
looks like: Named environments with separate storage, catalogs or schemas, identities and compute, and a rule that non-production identities cannot write to production. - How the gap shows
- The platform document describes one workspace or one storage location for all stages.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 5: MVP starts - Fail fast
- Using a development identity, try to write to a production table; it must be refused.
- Related
- ENV-02, REL-04, TEST-02
Check it: A development service principal cannot write to the production catalog.
ENV-02Access through groups and service identitieshighPlatform team
- Question
- Is access granted to groups by role with least privilege, and do scheduled jobs run as service identities rather than as a person?
- Why it matters
- Production jobs ran under the lead developer's own account; when she left the company her account was disabled and every nightly job failed.
- Applies to
- any · components: platform, governance, orchestration · patterns: any
- Owner
- Platform team · also Security and governance, Technical anchor
- Evidence
- in: platform, architecture
looks like: A role-to-group access model, service identities for jobs in each environment, no direct user grants in production, and who approves access requests. - Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test proven stage 9: Production sign-off - Fail fast
- List who owns and runs each production job; any job owned or run by a named person is a gap.
- Related
- ENV-01, SEC-02, REL-05
Check it: Check job run_as and object owners; none should be a named person in production.
ENV-03Network path to every source proven earlyhighPlatform team
- Question
- Has a connection from the platform to every source and target been made in a real environment, through firewalls, private links and credentials?
- Why it matters
- The source sat in a data centre behind a firewall; the network change took nine weeks of tickets and MVP 1 waited for it with the team idle.
- Applies to
- any · components: ingestion, serving, platform · patterns: any
- Owner
- Platform team · also Technical anchor, Source system owners
- Evidence
- in: platform, raid, discovery
looks like: Each source and target with its connection method, the network changes required, their lead time, and evidence a test connection succeeded. - How the gap shows
- Connectivity appears as an assumption in the risk log with no date.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 5: MVP starts - Fail fast
- Raise the network and credential requests in week one of discovery and pull one row from every source.
- Related
- DES-01, ENV-01
Trap: Serverless and classic compute reach the network differently; prove the route for the compute you will really use.
ENV-04Platform standards that projects followmediumArchitect
- Question
- Are naming, layer, folder and table standards written down for the platform so every project builds the same way?
- Why it matters
- Three projects named and layered their tables differently; consumers could not tell raw from cleaned data, and a shared lineage view became impossible.
- Applies to
- new platform · components: platform, governance · patterns: any
- Owner
- Architect · also Platform team, Developers
- Evidence
- in: platform, decisions
looks like: A short standards document covering naming, layers, table types, ownership tags and code structure, with examples, applied by the first project. - Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test - Fail fast
- Ask two developers to name the same new table independently; if the names differ, the standard is missing or unclear.
- Related
- DES-05, ENV-01
ENV-05Project fits the existing platform's standardsmediumTechnical anchor
- Question
- Does the project follow the existing platform's environments, standards and shared frameworks, and are any exceptions agreed with the platform team?
- Why it matters
- The project built its own ingestion code and its own environment layout; the platform team refused to support it at go-live and it had to be rebuilt.
- Applies to
- project on existing · components: any · patterns: any
- Owner
- Technical anchor · also Platform team, Architect
- Evidence
- in: architecture, decisions
looks like: References to the platform standards and frameworks used, and a decision record for each deviation, signed by the platform team. - Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 7: Test - Fail fast
- Walk the platform team through the architecture for thirty minutes and list every point where they say "we don't do it that way".
- Related
- DES-05, ENV-04
Version control, branching, promotion between environments (7)
REL-01Everything that runs is in version controlhighRelease and DevOps
- Question
- Is everything that runs in production (pipeline code, SQL, job definitions, schedules, data quality rules, permissions) kept in version control?
- Why it matters
- The schedule and parameters were edited by hand in the production interface; when the job was recreated after an incident, nobody knew the real settings and it ran with wrong ones for a week.
- Applies to
- any · components: any · patterns: any
- Owner
- Release and DevOps · also Developers, Technical anchor
- Evidence
- in: deployment, architecture
looks like: A repository layout that includes code, job and schedule definitions and configuration, and a rule that production is never changed by hand. - How the gap shows
- Jobs are described as "configured in the user interface".
- Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 6: Build proven stage 9: Production sign-off - Fail fast
- Pick one production job and try to recreate it in a fresh environment only from the repository.
- Related
- REL-02, REL-03, REL-06
Check it: Recreate a job from the repository into an empty workspace with a bundle deploy.
Trap: Jobs edited in the UI drift from the bundle; restrict edit rights in production.
REL-02Branching and code reviewmediumRelease and DevOps
- Question
- What is the branching approach, and must every change be reviewed by another person and pass automated checks before it merges?
- Why it matters
- A developer pushed a join change straight to the main branch on a Friday; it removed a filter, and the weekend loads double-counted refunds.
- Applies to
- any · components: any · patterns: any
- Owner
- Release and DevOps · also Developers, Technical anchor
- Evidence
- in: deployment
looks like: A short branching model, protected main branch, required reviewer, and automated tests and linting run on every pull request. - Due by
plan stage 4: Discovery sign-off designed stage 5: MVP starts evidenced stage 6: Build - Fail fast
- Try to push directly to the main branch; it should be refused.
- Related
- REL-01, TEST-03
REL-03Automated promotion of the same buildhighRelease and DevOps
- Question
- Is the same tested build promoted automatically from development to test to production, with an approval step before production?
- Why it matters
- Each environment was deployed by copying scripts by hand; production ended up with an older version of one script, and test results said nothing about what really ran.
- Applies to
- any · components: any · patterns: any
- Owner
- Release and DevOps · also Platform team
- Evidence
- in: deployment
looks like: A pipeline that builds once, deploys the same version to each environment with environment settings injected, and records which version is where. - Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 6: Build proven stage 9: Production sign-off - Fail fast
- Deploy the thin slice to test using only the pipeline, with nobody clicking in the target environment.
- Related
- REL-01, REL-04, REL-07
Check it: The production deployment history shows only the CI service principal deploying.
REL-04Configuration per environment, not in codemediumDevelopers
- Question
- Are environment-specific values (paths, connection names, schedules, compute sizes) kept in configuration per environment rather than written into the code?
- Why it matters
- A storage path to production was hard-coded in a shared module; tests in development read and then overwrote production data.
- Applies to
- any · components: any · patterns: any
- Owner
- Developers · also Release and DevOps
- Evidence
- in: deployment, architecture
looks like: A configuration file or variable set per environment, and code that reads all environment values from it. - Due by
plan stage 4: Discovery sign-off designed stage 5: MVP starts evidenced stage 6: Build - Fail fast
- Search the code for production host names, paths and catalog names; each hit is a defect.
- Related
- REL-03, REL-05, ENV-01
Check it: Search the code for production catalog names outside the bundle configuration.
REL-05No secrets in the repositoryhighSecurity and governance
- Question
- Are passwords, keys and tokens kept in a secret store and read at run time, never in code, configuration files or the repository's history?
- Why it matters
- A database password was committed in a script; the repository was later shared with a vendor, and the security team had to rotate credentials across four systems.
- Applies to
- any · components: any · patterns: any
- Owner
- Security and governance · also Developers, Release and DevOps
- Evidence
- in: platform, deployment
looks like: A named secret store per environment, code that reads secrets by name, and an automated secret scan on the repository. - Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 6: Build - Fail fast
- Run a secret scanner over the whole repository history today.
- Related
- REL-04, ENV-02
Trap: Secrets printed in a notebook cell are redacted, but derived strings and logs can still leak them.
REL-06Infrastructure defined as codemediumPlatform team
- Question
- Are the platform's environments, storage, compute policies, identities and permissions created from code, so another environment can be built identically?
- Why it matters
- The test environment was built by hand months after production and differed in network and permission settings, so tests passed that failed in production.
- Applies to
- new platform · components: platform · patterns: any
- Owner
- Platform team · also Release and DevOps
- Evidence
- in: platform, deployment
looks like: Infrastructure code in version control covering workspaces, storage, networks, identities, policies and grants, applied through a pipeline. - Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 6: Build proven stage 9: Production sign-off - Fail fast
- Build a throwaway environment from the infrastructure code alone and compare its settings with test.
- Related
- REL-01, ENV-01
REL-07Rolling back a release, including its datahighRelease and DevOps
- Question
- If a release goes wrong, how are both the code and any data or table structure it changed put back, and has that been rehearsed?
- Why it matters
- A release changed a table's structure and backfilled it; the code could be rolled back in minutes, but the data could not, and reports were broken for two days.
- Applies to
- any · components: any · patterns: any
- Owner
- Release and DevOps · also Technical anchor, Operations and support
- Evidence
- in: deployment, runbook
looks like: A rollback plan per release covering code, schema and data (restore points or table versions), and a rehearsal in a production-like environment. - How the gap shows
- The cut-over plan has no rollback section, or says "roll back the code" only.
- Due by
plan stage 5: MVP starts designed stage 6: Build evidenced stage 8: Deploy - Fail fast
- Deploy a release that changes a table in test, then roll it back fully, and time it.
- Related
- REL-03, ORCH-05
Check it: Rehearse RESTORE on a test table after a schema-changing release.
Trap: RESTORE only works for versions VACUUM has not cleaned up.
Scheduling, dependencies, retries, restart (6)
ORCH-01Jobs wait for their data, not the clockhighTechnical anchor
- Question
- Does each job start when its upstream data is complete, rather than at a fixed time that assumes the upstream has finished?
- Why it matters
- The report job ran at 06:00 assuming the load finished at 05:00; one night the load ran late, and the report published half a day's figures to the whole company.
- Applies to
- any · components: orchestration · patterns: batch, file, api, database, cdc
- Owner
- Technical anchor · also Developers, Operations and support
- Evidence
- in: architecture
looks like: Dependencies declared between jobs or on completion markers, cross-team dependencies named, and what happens when an upstream is late. - How the gap shows
- The schedule is a list of fixed times with no dependencies.
- Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test - Fail fast
- Delay an upstream job in test and check that the downstream waits instead of running on stale or partial data.
- Related
- ORCH-02, SERV-02
Check it: Delay an upstream table in test and check the downstream job does not start.
ORCH-02Retries for passing failures onlymediumDevelopers
- Question
- Which failures are retried automatically (timeouts, throttling), how many times and with what wait, and which fail straight away (bad data, logic errors)?
- Why it matters
- Every failure was retried five times; a bad-data error retried for three hours, burning compute and delaying the alert until the morning.
- Applies to
- any · components: orchestration · patterns: any
- Owner
- Developers · also Operations and support
- Evidence
- in: architecture, runbook
looks like: A retry policy per job with limits and back-off, failures classified as transient or permanent, and an alert after the last retry. - Due by
plan stage 4: Discovery sign-off designed stage 5: MVP starts evidenced stage 7: Test - Fail fast
- In test, cause one timeout and one data error; the first should retry and succeed, the second should fail fast and alert.
- Related
- ORCH-01, DQ-03, OPS-01
Trap: Retrying a non-idempotent task duplicates data; fix idempotency before adding retries.
ORCH-03Restart after failing in the middlehighTechnical anchor
- Question
- If a job fails halfway through writing, is the target left untouched or fully consistent, and can the job restart from the failed step without redoing everything?
- Why it matters
- A job failed after updating three of five tables; reports joined new facts to old dimensions, and rerunning from the start re-applied the first three updates.
- Applies to
- any · components: orchestration, transformation, modelling · patterns: any
- Owner
- Technical anchor · also Developers, Operations and support
- Evidence
- in: architecture, runbook, test-strategy
looks like: Atomic writes per table, a defined restart point per step, consumers reading only completed runs, and a failure-injection test. - Due by
plan stage 4: Discovery sign-off designed stage 5: MVP starts evidenced stage 7: Test proven stage 9: Production sign-off - Fail fast
- Kill a job in test while it is writing, then restart it; outputs must match an uninterrupted run.
- Related
- DUP-02, TEST-05, ORCH-04
Check it: Cancel a run mid-task in test, repair it, and compare outputs with a clean run.
Trap: A repair reruns the whole failed task from the beginning, not from the failed row.
ORCH-04Two runs writing the same table at oncemediumDevelopers
- Question
- What stops two runs of the same job, or two different jobs, writing to the same table at the same time?
- Why it matters
- A manual rerun overlapped the scheduled run; both merged into the same table, one failed on a conflict, and the other left half-updated rows.
- Applies to
- any · components: orchestration, transformation, modelling · patterns: any
- Owner
- Developers · also Operations and support
- Evidence
- in: architecture, runbook
looks like: One writer per table by design, a limit of one concurrent run per job, and a runbook rule for manual reruns. - Due by
plan stage 4: Discovery sign-off designed stage 5: MVP starts evidenced stage 7: Test - Fail fast
- Start the same job twice in test and see what happens to the second run and the table.
- Related
- ORCH-03
Trap: Two different jobs writing one table can still conflict; Delta raises concurrent modification errors on overlapping files.
ORCH-05Recovery time and data loss limitshighArchitect
- Question
- How quickly must the service recover after a major failure, how much data may be lost, and has recovery to those limits been tried?
- Why it matters
- A storage account was deleted by mistake; the backups existed but nobody had tried a restore, and it took four days instead of the four hours the business assumed.
- Applies to
- any · components: platform, orchestration · patterns: any
- Owner
- Architect · also Business owner and sponsor, Operations and support, Platform team
- Evidence
- in: requirements, platform, runbook
looks like: An agreed recovery time and acceptable data loss, a backup or replication design that meets them, and the result of a restore test. - Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 8: Deploy proven stage 10: Project sign-off - Fail fast
- Restore one important table from backup into a test environment and time it.
- Related
- REL-07, SEC-05
Check it: Restore a table from the chosen backup into a test catalog and time it.
ORCH-06Stream restarts from where it stoppedhighDevelopers
- Question
- When a streaming job stops or is redeployed, does it resume from its saved position without losing or repeating data, and how is that state protected?
- Why it matters
- A redeploy changed the stream's checkpoint location; the job restarted from the earliest available data, reprocessed a week and flooded downstream tables.
- Applies to
- any · components: ingestion, transformation, orchestration · patterns: streaming, cdc
- Owner
- Developers · also Operations and support, Technical anchor
- Evidence
- in: architecture, runbook
looks like: Where progress is saved, how it survives redeploys and code changes, the source retention versus the longest expected outage, and a runbook for resetting a stream. - Due by
plan stage 4: Discovery sign-off designed stage 5: MVP starts evidenced stage 7: Test proven stage 9: Production sign-off - Fail fast
- Stop the stream for longer than a typical outage in test, redeploy a changed version, restart, and compare counts with the source.
- Related
- DUP-04, ORCH-03
Check it: Redeploy a changed stream in test and confirm it resumes from its position.
Trap: Some query changes are not allowed with an existing checkpoint (for example changes to stateful operations); plan a reset and backfill when they are needed.
Monitoring, alerting, support, runbooks (3)
OPS-01Alerts reach someone who can acthighOperations and support
- Question
- When a job fails, runs late or data looks wrong, who is alerted, through which channel, and within how long?
- Why it matters
- Failure emails went to a shared mailbox nobody read; a load had failed for four days before a user asked why the report had not changed.
- Applies to
- any · components: orchestration, platform · patterns: any
- Owner
- Operations and support · also Technical anchor
- Evidence
- in: runbook, architecture
looks like: Alerts for failure, lateness against the freshness target and data checks, routed to a named team or on-call rota, with an expected response time. - Due by
plan stage 4: Discovery sign-off designed stage 5: MVP starts evidenced stage 8: Deploy proven stage 9: Production sign-off - Fail fast
- Make a job fail in test and time how long it takes for a person to know.
- Related
- DQ-04, HAND-01, HAND-04
Check it: Fail a test job and time the alert.
OPS-03Every run leaves a recordmediumDevelopers
- Question
- Does every run record what it read, what it wrote, row counts, duration and outcome, in a place that can be queried?
- Why it matters
- A user reported a wrong figure from two weeks ago; with no run records nobody could say which load produced it or which input it came from.
- Applies to
- any · components: ingestion, transformation, modelling, orchestration · patterns: any
- Owner
- Developers · also Operations and support
- Evidence
- in: architecture, runbook
looks like: A run log or audit table with run id, inputs, outputs, counts, timings and status, kept for a stated period. - Due by
plan stage 4: Discovery sign-off designed stage 5: MVP starts evidenced stage 7: Test - Fail fast
- Ask which source file produced a given row in yesterday's table, and see how long it takes to answer.
- Related
- REC-01, OPS-01, COST-03
Check it: Trace a row in gold to its job run and source file (_metadata.file_path captured in bronze).
OPS-05Production watched closely after go-livemediumTechnical anchor
- Question
- Were the first weeks in production watched closely (hypercare), with defects closed and operations running it alone before sign-off?
- Why it matters
- The team moved to the next project on go-live day; a slow memory leak failed the job at month-end and nobody who understood it was available.
- Applies to
- any · components: any · patterns: any
- Owner
- Technical anchor · also Operations and support, Business owner and sponsor
- Evidence
- in: deployment, status
looks like: A hypercare period with daily checks, a log of production issues and their fixes, and a handover point when operations run it alone. - Due by
plan stage 6: Build designed stage 8: Deploy evidenced stage 9: Production sign-off proven stage 9: Production sign-off - Fail fast
- Put hypercare dates, people and exit criteria in the plan now.
- Related
- REC-04, HAND-04
Test levels, test data, environments (5)
TEST-01Test levels that include data testshighTest lead
- Question
- Does the test strategy cover code tests, data tests (rules, reconciliation), integration end to end, performance and user acceptance, each with an owner?
- Why it matters
- Tests checked that each function worked on hand-made rows; nobody tested the real data end to end, and the first integrated run produced reports nobody recognised.
- Applies to
- any · components: any · patterns: any
- Owner
- Test lead · also Technical anchor
- Evidence
- in: test-strategy
looks like: A test strategy naming each test level, what it proves, who runs it, where, and with what data, including data quality and reconciliation tests. - How the gap shows
- The test strategy says "to be defined", or covers only unit tests and user acceptance.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 7: Test - Fail fast
- For each top requirement, write the one data test that would prove it, and see which ones you cannot write.
- Related
- TEST-02, TEST-03, DQ-01
TEST-02Realistic, safe test datahighTest lead
- Question
- Where does test data come from, is it realistic enough to find real problems (volumes, edge cases, bad rows), and is personal data masked or synthetic?
- Why it matters
- Test data was a tidy hand-made sample with no nulls or duplicates; every test passed, and the first real load failed on problems the sample could never have shown.
- Applies to
- any · components: any · patterns: any
- Owner
- Test lead · also Security and governance, Source system owners
- Evidence
- in: test-strategy, platform
looks like: A source of test data per environment (masked copy, sampled production, generated), the edge cases it includes, and security approval. - Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 6: Build - Fail fast
- Compare the null, duplicate and orphan-key rates of the test data with the real-data profile.
- Related
- TEST-01, SEC-01, DQ-05
Trap: Clones of tables with row filters or masks are not supported; mask before cloning.
TEST-03Automated tests run on every changemediumDevelopers
- Question
- Do automated tests (code and data) run on every change in the release pipeline, and does a failure stop the release?
- Why it matters
- A small refactor changed a join; the tests existed but were run by hand and nobody ran them, so the regression reached production.
- Applies to
- any · components: any · patterns: any
- Owner
- Developers · also Release and DevOps, Test lead
- Evidence
- in: test-strategy, deployment
looks like: Tests wired into the pull request and deployment pipeline as required gates, with coverage of the transformations that carry business rules. - Due by
plan stage 4: Discovery sign-off designed stage 5: MVP starts evidenced stage 6: Build proven stage 9: Production sign-off - Fail fast
- Break one business rule deliberately on a branch; the pipeline must fail before merge.
- Related
- REL-02, TEST-01
TEST-04Users test with numbers they recognisehighTest lead
- Question
- Will business users accept the solution by checking real numbers they know, against agreed criteria, rather than by looking at screens?
- Why it matters
- User acceptance was a demo of the dashboard layout; the first time finance compared it with their own numbers was after go-live, and they rejected it.
- Applies to
- any · components: modelling, serving · patterns: any
- Owner
- Test lead · also Business owner and sponsor, Data consumers
- Evidence
- in: test-strategy, requirements
looks like: User acceptance scenarios with expected figures supplied by the business, named testers, dates, and sign-off criteria. - Due by
plan stage 4: Discovery sign-off designed stage 6: Build evidenced stage 7: Test - Fail fast
- Ask the business owner for three numbers they know by heart and check them in the thin slice.
- Related
- REC-02, TEST-01
TEST-05Failure, rerun and recovery tested on purposehighTest lead
- Question
- Does the test plan include deliberate failures: reruns, duplicate input, late data, schema changes, empty input and jobs killed mid-run?
- Why it matters
- Every test assumed the happy path; in the first month the pipeline met a resent file, a late batch and a crash mid-write, and each one corrupted data.
- Applies to
- any · components: ingestion, transformation, modelling, orchestration · patterns: any
- Owner
- Test lead · also Technical anchor, Developers
- Evidence
- in: test-strategy
looks like: Negative test cases for each failure the design claims to handle, with expected outcomes. - Due by
plan stage 4: Discovery sign-off designed stage 5: MVP starts evidenced stage 7: Test - Fail fast
- List every failure the architecture says it handles and check each has a test case.
- Related
- DUP-02, ORCH-03, SCH-02
Architecture decisions, patterns, non-functional needs (5)
DES-01A thin slice proved the design end to endhighTechnical anchor
- Question
- Has one source table been pushed through every layer to one report or query in a production-like environment, before the design was signed off?
- Why it matters
- A clean architecture diagram was signed off without ever running; in MVP 1 the team found the source format, network and permissions all differed from the assumptions.
- Applies to
- any · components: any · patterns: any
- Owner
- Technical anchor · also Architect
- Evidence
- in: discovery, architecture
looks like: A description of the thin slice, where it ran, what it proved, what it found, and how the design changed as a result. - How the gap shows
- The architecture shows only target-state diagrams with no evidence of anything having run.
- Due by
plan stage 3: Discovery designed stage 3: Discovery evidenced stage 4: Discovery sign-off - Fail fast
- Choose the riskiest source and the most important report, and connect them with the simplest possible pipeline this week.
- Related
- DQ-05, ENV-03, PERF-02
DES-02Non-functional needs written as numbershighArchitect
- Question
- Are the non-functional needs (freshness, volume, recovery, availability, retention, concurrency) written as measurable numbers and agreed with the business?
- Why it matters
- "Near real time" meant five seconds to the business and an hour to the engineers; the gap was found at user acceptance and the design had to be rebuilt.
- Applies to
- any · components: any · patterns: any
- Owner
- Architect · also Business analyst, Business owner and sponsor
- Evidence
- in: requirements, architecture
looks like: A table of non-functional needs, each with a number, a measurement method and who agreed it. - How the gap shows
- Words like "fast", "near real time", "scalable" or "highly available" with no numbers.
- Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 7: Test - Fail fast
- Rewrite each non-functional sentence as a number and ask the business owner to confirm it.
- Related
- PERF-02, ORCH-05, DES-06
DES-04Grain and keys of every tablehighTechnical anchor
- Question
- Is the grain (what one row means) and the key of every target table written down and agreed?
- Why it matters
- The orders table mixed order headers and order lines; summing it double-counted shipping, and two teams built different fixes.
- Applies to
- any · components: transformation, modelling, serving · patterns: any
- Owner
- Technical anchor · also Business analyst, Data consumers
- Evidence
- in: data-model
looks like: For each table: one sentence for the grain, the key columns, and a uniqueness test on that key. - Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 7: Test - Fail fast
- For each target table, finish the sentence "one row is one …" and test the key for uniqueness.
- Related
- DUP-03, SERV-01, SCD-03
DES-05Shared frameworks instead of one-off codemediumArchitect
- Question
- Does the platform provide shared, configuration-driven patterns for common work (ingestion, quality checks, logging, history handling), so each pipeline is not hand-built?
- Why it matters
- Forty pipelines were each written differently; a fix to the deduplication logic had to be made forty times, and twelve were missed.
- Applies to
- new platform · components: platform, ingestion, transformation · patterns: any
- Owner
- Architect · also Technical anchor, Developers
- Evidence
- in: platform, architecture, decisions
looks like: A small set of shared patterns or frameworks with documented configuration, proved by the first two projects, and a decision on what is built versus reused. - Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test - Fail fast
- Onboard a second source table with the framework and count how many lines of new code it took.
- Related
- ENV-04, ENV-05
DES-06Batch or streaming chosen for a reasonmediumArchitect
- Question
- Is the choice between batch, micro-batch and streaming justified by a stated freshness need and its cost, not by preference?
- Why it matters
- The team built continuous streaming for a report read once a day; it cost eight times a nightly batch and was harder to support.
- Applies to
- any · components: ingestion, transformation, orchestration · patterns: batch, streaming, cdc
- Owner
- Architect · also Business owner and sponsor, Technical anchor
- Evidence
- in: decisions, requirements
looks like: A decision record linking the processing mode to a freshness number and a running cost comparison. - Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off - Fail fast
- Ask the consumer what decision they would take differently with data one minute old versus one hour old.
- Related
- DES-02, COST-01, PERF-02
Contracts with consumers, grain, freshness promises (5)
SERV-01A contract with each consumerhighTechnical anchor
- Question
- For each consumer (report, extract, application, model), is there an agreed contract: the tables or fields they use, their meaning, grain, freshness and how they will receive it?
- Why it matters
- A downstream team built a pricing model on an internal staging table; a routine refactor renamed it and the pricing model failed on a sale day.
- Applies to
- any · components: serving, modelling · patterns: any
- Owner
- Technical anchor · also Data consumers, Business analyst
- Evidence
- in: requirements, data-model, architecture
looks like: A list of consumers and, for each, the published interface, its fields and meaning, grain, freshness promise and a named contact. - How the gap shows
- Consumers are listed by name only, with no interface or freshness.
- Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test proven stage 9: Production sign-off - Fail fast
- Ask the main consumer which exact fields they will read; check each is in the data model with a definition.
- Related
- SERV-03, DES-04, SERV-04
SERV-02Consumers can tell whether data is completemediumTechnical anchor
- Question
- Can a consumer see when the data was last refreshed and whether today's load is complete, before they rely on it?
- Why it matters
- The load failed overnight but the dashboard looked normal; the sales director presented yesterday's numbers as today's to the board.
- Applies to
- any · components: serving · patterns: any
- Owner
- Technical anchor · also Data consumers, Operations and support
- Evidence
- in: architecture, requirements
looks like: A visible "data as of" time on reports, a completeness flag or status table, and a way to warn consumers when a load is late. - Due by
plan stage 4: Discovery sign-off designed stage 5: MVP starts evidenced stage 7: Test proven stage 9: Production sign-off - Fail fast
- Stop the load in test and look at the report as a user would; can you tell it is stale?
- Related
- ORCH-01, PERF-02, OPS-01
SERV-03Changes to what consumers use are announcedmediumTechnical anchor
- Question
- How are breaking changes to published tables or interfaces versioned and announced, so consumers have time to adapt?
- Why it matters
- A column was renamed for consistency in a minor release; three reports and a partner extract broke the next morning.
- Applies to
- any · components: serving, modelling · patterns: any
- Owner
- Technical anchor · also Data consumers, Release and DevOps
- Evidence
- in: architecture, deployment
looks like: A rule separating published interfaces from internal tables, a versioning or deprecation approach with a notice period, and a list of who to tell. - Due by
plan stage 4: Discovery sign-off designed stage 5: MVP starts evidenced stage 9: Production sign-off - Fail fast
- Find out how the team would learn which consumers use a given column today.
- Related
- SERV-01, SCH-03
SERV-04Business definitions live in one placehighBusiness analyst
- Question
- Is each business measure (revenue, active customer) defined and calculated once, in a shared model, rather than separately in each report?
- Why it matters
- Three dashboards calculated margin three ways; the leadership meeting spent an hour arguing about which number was right instead of deciding anything.
- Applies to
- any · components: modelling, serving · patterns: any
- Owner
- Business analyst · also Technical anchor, Data consumers
- Evidence
- in: data-model, requirements
looks like: A definitions list with the owner of each measure, implemented once in a shared model or semantic layer that reports reuse. - Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test - Fail fast
- Ask two consumers to write down how they calculate the top measure; compare the answers.
- Related
- REC-02, SERV-01, SCD-04
Trap: Metric views are newer; check the feature's status in your workspace before standardising on it.
SERV-05Consumer queries fast enough under real loadmediumTechnical anchor
- Question
- Will reports, interfaces and extracts respond fast enough when the expected number of users or calls hit them at the same time?
- Why it matters
- The dashboard was quick in testing with one user; on Monday morning two hundred managers opened it at once and it took two minutes per page.
- Applies to
- any · components: serving · patterns: any
- Owner
- Technical anchor · also Data consumers, Platform team
- Evidence
- in: requirements, test-strategy
looks like: Expected concurrent users or calls and response targets, the serving design (aggregates, caching, sizing), and a concurrency test result. - Due by
plan stage 4: Discovery sign-off designed stage 5: MVP starts evidenced stage 7: Test proven stage 9: Production sign-off - Fail fast
- Run the slowest report query twenty times at once against realistic data and time it.
- Related
- PERF-03, DES-02
Check it: Run concurrent queries and read the warehouse's query history for queueing time.
Estimates and their assumptions (6)
EST-01The rough guess became the promisehighTechnical anchor
- Question
- Has the estimate been redone since the first rough guess, and does everyone quoting a date or a budget know which version they are quoting? Imagine it is six months from now and the project is late: was the date we are judged against the one somebody guessed before anyone looked at the data?
- Why it matters
- A number given in a corridor before discovery was written into the budget. Discovery found twice the work, but the plan was expected to confirm the guess, so the project was "late" from its first week.
- Applies to
- any · components: any · patterns: any
- Owner
- Technical anchor · also Project manager, Business owner and sponsor
- Evidence
- in: estimate, plan, status
looks like: An estimate history showing the rough estimate, the discovery estimate and later re-estimates, each dated, with the baseline that sign-off accepted clearly marked. - How the gap shows
- The same number appears in the first brief, the discovery document and the plan, unchanged.
- The estimate has no date or version, so nobody can tell whether it came before or after discovery.
- Status reports measure progress against a figure that is older than the discovery document.
- Phrases such as "as agreed at the outset" or "the approved budget" with no link to how it was built.
- Compare
- Compare the estimate with the plan: if the plan's end date or budget matches the stage-2 rough estimate rather than the discovery estimate, the guess has become the promise.
- Due by
plan stage 2: Rough estimate evidenced stage 4: Discovery sign-off - Fail fast
- Put the rough estimate and the discovery estimate side by side in front of the sponsor and ask which one the plan is built on.
- Related
- EST-02, EST-06
EST-02One number instead of a rangehighTechnical anchor
- Question
- Is the estimate given as a range, with the reason for its width? If the project overruns, will anyone be able to say whether the overrun was inside the range we said was possible?
- Why it matters
- "Six weeks" was heard as a commitment. "Five to eight weeks if access arrives in week one" would have been a plan, and the slip to seven weeks would have been expected rather than a crisis.
- Applies to
- any · components: any · patterns: any
- Owner
- Technical anchor · also Project manager
- Evidence
- in: estimate
looks like: A low and a high figure (or a most-likely figure with a stated spread), and a sentence on what would move it toward either end. - How the gap shows
- The estimate is a single figure in days, weeks or money with no low and high value.
- The width of the range does not narrow from the rough estimate to the discovery estimate.
- The range is the same percentage on every item, which suggests it was added, not reasoned.
- Compare
- Compare the estimate with the status reports: a range in the estimate but a single committed date in every status report means the range was dropped on the way to the stakeholders.
- Due by
plan stage 2: Rough estimate evidenced stage 4: Discovery sign-off - Fail fast
- Ask the person who wrote the estimate for the worst realistic case. If they can name one without hesitating, it belongs in the document.
- Script signal
single-number-estimate(found without AI)- Related
- EST-01, EST-03
EST-03An estimate with no written assumptionshighTechnical anchor
- Question
- Which assumptions does the estimate depend on, and which one, if wrong, would move it the most? If the estimate turns out wrong, will we be able to point to the assumption that broke?
- Why it matters
- The estimate assumed the source data was clean and that one person would test. Neither was written down, so when both failed, the overrun looked like the team's fault instead of a known risk.
- Applies to
- any · components: any · patterns: any
- Owner
- Technical anchor · also Project manager, Business owner and sponsor
- Evidence
- in: estimate, raid
looks like: A numbered list of assumptions next to the estimate, the most sensitive ones marked, and each one also logged in the RAID log with an owner. - How the gap shows
- No "assumptions" heading or list anywhere in or near the estimate.
- Assumptions are written as one line, such as "standard assumptions apply".
- Assumptions exist in the estimate but none of them appear in the RAID log.
- Compare
- Compare the estimate's assumptions with the RAID log: an assumption in one but not the other is either unmanaged or was never the real basis of the estimate.
- Due by
plan stage 2: Rough estimate evidenced stage 4: Discovery sign-off - Fail fast
- Pick the riskiest assumption and test it this week, for example by requesting the access or pulling one real extract.
- Script signal
no-assumptions(found without AI)- Related
- EST-02, COM-05, PM-07
EST-04Invisible work left out of the estimatehighTechnical anchor
- Question
- Does the estimate include the work that is not building: getting access, setting up environments, testing, fixing data, rehearsing the cut-over, documentation and handover? If we run out of time, will it be because the build took longer, or because this work was never counted?
- Why it matters
- The build finished on time. Then access approvals, test data, two cut-over rehearsals and the runbook took another five weeks that nobody had planned for.
- Applies to
- any · components: any · patterns: any
- Owner
- Technical anchor · also Project manager, Test lead, Operations and support
- Evidence
- in: estimate, plan
looks like: Separate lines for access and onboarding, environments, testing (including reconciliation and user acceptance), data fixes, history load, cut-over and rollback rehearsal, runbook, handover and hypercare. - How the gap shows
- The estimate lists only build items (pipelines, tables, reports).
- Testing is a fixed percentage added at the end rather than counted per item.
- Words such as "cut-over", "handover", "access", "environment" or "hypercare" do not appear in the estimate.
- Compare
- Compare the estimate with the plan: plan tasks for testing, cut-over or handover that have no matching line in the estimate are unfunded work.
- Due by
plan stage 2: Rough estimate evidenced stage 4: Discovery sign-off - Fail fast
- Read the estimate against the ten lifecycle stages and ask, for each stage after build, which line pays for it.
- Script signal
no-invisible-work(found without AI)- Related
- EST-05, EST-07, PM-05
EST-05Guessed, not countedmediumTechnical anchor
- Question
- Was the estimate built by counting the things the project will deliver (sources, tables, rules, reports, interfaces) and multiplying by a rate, or was it a total someone felt was right? Could another person rebuild the number from the document?
- Why it matters
- The estimate was "about three months" for the whole project. When it was finally counted there were forty tables and twelve reports, which at the team's real rate was nearer six months.
- Applies to
- any · components: any · patterns: any
- Owner
- Technical anchor · also Project manager
- Evidence
- in: estimate
looks like: A table of counted items by type and complexity, a rate per item, where the rate came from, and the total. - How the gap shows
- The estimate is a total with no breakdown, or a breakdown by phase only.
- No counts of sources, tables, rules or reports appear anywhere in the documents.
- The rate per item is not stated or not explained.
- Compare
- Compare the estimate with the requirements or data model: if they list more sources, tables or reports than the estimate counted, the estimate is short.
- Due by
plan stage 2: Rough estimate designed stage 4: Discovery sign-off - Fail fast
- Count the items in the requirements yourself and divide the estimate by the count. If the effort per item looks implausibly small, it probably is.
- Related
- EST-04, EST-06
EST-07History load and data fixing underestimatedhighTechnical anchor
- Question
- Has the work of loading history and dealing with poor source data been estimated separately, using a profile of the real data? If the history load is what delays go-live, will that surprise anyone?
- Why it matters
- The pipelines worked on a month of data. Ten years of history held old codes, broken keys and a format change, and checking the old numbers with the business took longer than building the pipelines.
- Applies to
- any · components: any · patterns: any
- Owner
- Technical anchor · also Business analyst, Source system owners
- Evidence
- in: estimate, discovery
looks like: Separate estimate lines for the history load (volume, number of runs, reconciliation with the business) and for data fixes, sized from a profile of real data. - How the gap shows
- The history load is not mentioned in the estimate, or it is one line with a small number.
- The estimate says data quality is "assumed acceptable" with no profile behind it.
- There is no data profiling section in the discovery document.
- Compare
- Compare the discovery profiling results with the estimate: if the profile found quality problems but the estimate has no line for fixing or handling them, the estimate ignored the evidence.
- Due by
plan stage 3: Discovery evidenced stage 4: Discovery sign-off - Fail fast
- Load one year of real history in a test environment and time it, and count the rows that fail basic rules.
- Related
- EST-04, REQ-04
Documentation, knowledge transfer, support model (4)
HAND-01Runbook missing or written at the endhighTechnical anchor
- Question
- Is there a runbook that operations have read and used: what runs when, how to tell it failed, how to rerun safely, and who to call? If a load fails on the first night after the team has gone, can operations fix it?
- Why it matters
- The runbook was written in the last week and never tried. On the first failure operations reran the whole load, which doubled a month of figures.
- Applies to
- any · components: any · patterns: any
- Owner
- Technical anchor · also Operations and support
- Evidence
- in: runbook
looks like: A runbook started in build and finished before deploy, covering schedule, monitoring, alerts, common failures with fixes, safe rerun steps and contacts, used in at least one failure drill. - How the gap shows
- No runbook or operations guide among the documents.
- The runbook is planned only for the last week before go-live.
- The runbook has headings but sections marked "to be completed".
- Compare
- Compare the runbook with the architecture: jobs, sources or failure points in the architecture that the runbook does not mention will be unsupported.
- Due by
plan stage 4: Discovery sign-off designed stage 6: Build evidenced stage 8: Deploy proven stage 9: Production sign-off - Fail fast
- Break one job in the test environment and ask operations to recover it using only the runbook.
- Script signal
missing-doc-type(found without AI)- Related
- HAND-03, HAND-04
HAND-02Decision log and data dictionary not ready to hand overmediumTechnical anchor
- Question
- Will the support team receive a current decision log and data dictionary that explain why things are the way they are? When someone asks "why does this table work like this?" after the team has left, where is the answer?
- Why it matters
- The support team "fixed" a rule that looked wrong. It had been agreed with finance a year before, but the decision was only in an old e-mail thread, and the change broke month-end reporting.
- Applies to
- any · components: any · patterns: any
- Owner
- Technical anchor · also Project manager, Operations and support
- Evidence
- in: decisions, data-model
looks like: A decision log and a data dictionary that match what is in production, reviewed as part of project closure. - How the gap shows
- The decision log stops well before the end of build.
- The data dictionary does not cover tables or rules added during build.
- Handover lists code and access but not decisions or definitions.
- Compare
- Compare the data dictionary with the delivered data model or test results: rules in production that are not in the dictionary are unexplained.
- Due by
plan stage 4: Discovery sign-off evidenced stage 9: Production sign-off proven stage 10: Project sign-off - Fail fast
- Ask someone outside the team to explain one business rule using only the handed-over documents.
- Related
- PM-03, HAND-03
HAND-03Knowledge transfer never testedhighTechnical anchor
- Question
- Has the support team made at least one real change and one real recovery while the project team watched? If the project team left tomorrow, could the support team run and change this system?
- Why it matters
- Knowledge transfer was two presentation sessions. The first change request after closure took the support team a month, and the project team had to be called back.
- Applies to
- any · components: any · patterns: any
- Owner
- Technical anchor · also Operations and support, Project manager
- Evidence
- in: plan, runbook
looks like: Planned handover sessions with hands-on tasks, and a record of a change and a recovery done by the support team themselves. - How the gap shows
- Knowledge transfer is a single line or a few presentation sessions in the plan.
- The support team is not named, or joins only at the end.
- No hands-on tasks are planned for the support team.
- Compare
- Compare the plan's handover time with the size of the system in the architecture: a short handover for a large system means much will not transfer.
- Due by
plan stage 4: Discovery sign-off evidenced stage 9: Production sign-off proven stage 10: Project sign-off - Fail fast
- Ask the support team to make one small real change this sprint, with the project team only watching.
- Related
- PM-06, HAND-01
HAND-06No exit criteria for early-life supportmediumTechnical anchor
- Question
- Are there written criteria for ending the early weeks of close support after go-live (for example a number of clean runs, reconciled loads and no critical defects)? How will we know the system is ready to be left alone, rather than the team simply running out of time?
- Why it matters
- The project team left after two weeks because the budget ended. Month-end processing, which had never run in production, failed the following week with nobody who knew the system.
- Applies to
- any · components: any · patterns: any
- Owner
- Technical anchor · also Operations and support, Project manager
- Evidence
- in: deployment, plan
looks like: Written exit criteria for early-life support, covering at least one full business cycle, reconciled production loads and agreed defect levels. - How the gap shows
- The plan ends at go-live, or has a fixed support period with no criteria.
- Period-end or yearly processes are not mentioned in the early-life support period.
- Compare
- Compare the plan's early-life support period with the business calendar in the requirements: if the period ends before a month-end or other key cycle has run, the riskiest run happens unsupported.
- Due by
plan stage 4: Discovery sign-off designed stage 8: Deploy evidenced stage 9: Production sign-off - Fail fast
- Ask: "Which business cycle has not yet run in production?" and keep support in place until it has.
- Related
- HAND-04
Databricks only (12)
Checks that only make sense on Databricks. Every common check above that works differently on Databricks also has an "On Databricks" box. Checked against docs.databricks.com on 2026-11-01. Product names used here: Unity Catalog, Lakeflow Jobs, Lakeflow pipelines (formerly Delta Live Tables / DLT), AUTO CDC (formerly APPLY CHANGES), Declarative Automation Bundles (formerly Databricks Asset Bundles), data quality monitoring (formerly anomaly detection) and data profiling (formerly Lakehouse Monitoring). Names move fast; re-check before quoting them to a client.
DBX-01Unity Catalog objects owned by groupsmediumPlatform team
- Question
- Are catalogs, schemas, tables, jobs and pipelines owned by groups or service principals, with privileges granted to account groups rather than individual users?
- Why it matters
- A catalog owned by a contractor's user account could not be changed after the contractor left, until an admin reassigned ownership in the middle of an incident.
- Applies to
- any · components: platform, governance · patterns: any
- Owner
- Platform team · also Security and governance
- Evidence
- in: platform
looks like: An ownership and grant model by group per catalog and schema, defined in code. - Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 7: Test proven stage 9: Production sign-off - Fail fast
- List owners of production catalogs, schemas and jobs from information_schema and the jobs list; flag every individual user.
- Related
- ENV-02, SEC-02
DBX-02Compute policies cap size and force shutdownhighPlatform team
- Question
- Do compute policies limit node types and worker counts, force auto-termination and required tags, and are users unable to create compute without a policy?
- Why it matters
- Without policies a developer created a large all-purpose cluster with no auto-termination, and it ran idle for two weeks.
- Applies to
- new platform · components: platform · patterns: any
- Owner
- Platform team · also Security and governance
- Evidence
- in: platform
looks like: Named policies per workload type, who may use each, and unrestricted cluster creation removed. - Due by
plan stage 3: Discovery designed stage 5: MVP starts evidenced stage 6: Build - Fail fast
- As a normal user, try to create compute without a policy and above the policy's size.
- Related
- COST-02, COST-04
DBX-03Serverless usage tagged for attributionmediumPlatform team
- Question
- Is serverless compute (notebooks, jobs, pipelines) attributed to projects through serverless usage policies, and is the team aware these tag usage rather than cap it?
- Why it matters
- Serverless spend grew quickly across teams with no tags, so it showed as one untraceable line on the bill.
- Applies to
- any · components: platform · patterns: any
- Owner
- Platform team
- Evidence
- in: platform
looks like: Serverless usage policies per project or team, assigned to the right groups, and a cost report by their tags. - Due by
plan stage 4: Discovery sign-off designed stage 5: MVP starts evidenced stage 7: Test proven stage 9: Production sign-off - Fail fast
- Query system.billing.usage for serverless usage in the last week with no project tag.
- Related
- COST-03, DBX-09
DBX-04Table retention matches time travel and recovery needsmediumTechnical anchor
- Question
- Is delta.deletedFileRetentionDuration (and log retention) set per table to match how far back the team needs time travel, RESTORE and stream recovery, balanced against storage cost and erasure rules?
- Why it matters
- A bad release was found eight days later; VACUUM had already removed files older than the default seven days, so RESTORE to the good version was impossible.
- Applies to
- any · components: transformation, modelling, platform · patterns: any
- Owner
- Technical anchor · also Security and governance, Operations and support
- Evidence
- in: architecture, runbook
looks like: A retention setting per table class, justified by recovery needs and personal-data erasure deadlines. - Due by
plan stage 4: Discovery sign-off designed stage 5: MVP starts evidenced stage 7: Test - Fail fast
- Run SHOW TBLPROPERTIES on key tables and compare retention with how long defects usually take to find.
- Related
- REL-07, SEC-03, PERF-05
DBX-05Auto Loader schema evolution mode chosen per sourcemediumDevelopers
- Question
- For each Auto Loader source, is the schema evolution mode (addNewColumns, rescue, failOnNewColumns, none) chosen deliberately, with the job set to restart after an intentional stop?
- Why it matters
- With the default addNewColumns mode, a new source column stopped the stream as designed, but the job had no retries and the stream stayed down all night.
- Applies to
- any · components: ingestion · patterns: file, streaming, cdc
- Owner
- Developers · also Technical anchor
- Evidence
- in: architecture
looks like: The mode per source, where rescued data is monitored, and retries on the ingestion task. - Due by
plan stage 4: Discovery sign-off designed stage 5: MVP starts evidenced stage 7: Test - Fail fast
- Drop a file with a new column in test and see whether the job recovers without a person.
- Related
- SCH-01, SCH-02
DBX-06Job retries, concurrency and repair are safehighDevelopers
- Question
- Are task retries, timeouts, maximum concurrent runs and queueing set deliberately for each job, and is every task safe to rerun from the start, as a repair run does?
- Why it matters
- Operations repaired a failed run; the failed task had appended half its rows before failing, and the repair appended them again.
- Applies to
- any · components: orchestration · patterns: any
- Owner
- Developers · also Operations and support
- Evidence
- in: architecture, runbook
looks like: Retry and timeout settings per task in the bundle, concurrency left at one, and a runbook note on repair. - Due by
plan stage 4: Discovery sign-off designed stage 5: MVP starts evidenced stage 7: Test - Fail fast
- Cancel a task halfway in test, repair the run, and compare the output with a clean run.
- Related
- DUP-02, ORCH-02, ORCH-03
DBX-07Pipeline expectations with the right actionmediumDevelopers
- Question
- In Lakeflow pipelines, does each expectation use the right action (warn, drop row, fail update), and are dropped rows kept somewhere for reprocessing?
- Why it matters
- Every rule was set to drop; thousands of rows a day were silently removed and only a count was logged.
- Applies to
- any · components: ingestion, transformation · patterns: any
- Owner
- Developers · also Business analyst
- Evidence
- in: architecture, data-model
looks like: A rule list mapping each expectation to an action, plus a quarantine table for dropped rows. - Due by
plan stage 4: Discovery sign-off designed stage 5: MVP starts evidenced stage 7: Test proven stage 9: Production sign-off - Fail fast
- Count expectations by action in the pipeline code and check each drop has a quarantine.
- Related
- DQ-01, DQ-02
DBX-08Promotion through bundles, run as service principalsmediumRelease and DevOps
- Question
- Are jobs, pipelines and dashboards deployed to each environment through Declarative Automation Bundles (formerly Asset Bundles) from CI, with production jobs running as a service principal?
- Why it matters
- Production jobs were created by hand in the UI and ran as the developer; a later bundle deploy created duplicates and both copies wrote to the same tables.
- Applies to
- any · components: platform, orchestration · patterns: any
- Owner
- Release and DevOps · also Platform team
- Evidence
- in: deployment
looks like: A bundle with dev, test and prod targets, run_as a service principal in prod, deployed only by CI. - Due by
plan stage 4: Discovery sign-off designed stage 5: MVP starts evidenced stage 6: Build proven stage 9: Production sign-off - Fail fast
- Deploy the thin slice to the test target with databricks bundle deploy from a CI pipeline.
- Related
- REL-03, REL-04, ENV-02
DBX-09System tables used for cost and run monitoringlowPlatform team
- Question
- Are system tables (billing usage and prices, job run timeline, audit, query history) available to the people who need them, with a cost and reliability view built on them?
- Why it matters
- Nobody had been granted access to the system catalog, so a cost spike was investigated for a week from the invoice instead of in an hour from the usage table.
- Applies to
- new platform · components: platform · patterns: any
- Owner
- Platform team · also Operations and support
- Evidence
- in: platform, runbook
looks like: Grants on the system catalog to platform and finance groups, and a dashboard by job, tag and day. - Due by
plan stage 4: Discovery sign-off designed stage 5: MVP starts evidenced stage 7: Test proven stage 9: Production sign-off - Fail fast
- Query yesterday's cost by job from system.billing.usage and list prices.
- Related
- COST-01, COST-03, OPS-03
DBX-10Managed or external tables decidedlowArchitect
- Question
- Is the choice between Unity Catalog managed and external tables, and the storage location for each catalog, decided and recorded?
- Why it matters
- Tables were created as external by habit; predictive optimization did not run on them, small files built up, and dropping a table left its data behind in storage.
- Applies to
- new platform · components: platform, governance · patterns: any
- Owner
- Architect · also Platform team
- Evidence
- in: platform, decisions
looks like: A decision record on table types and managed storage locations per catalog. - Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off - Fail fast
- Count production tables by table type in information_schema.tables.
- Related
- PERF-05, ENV-04
DBX-11AUTO CDC sequenced by a real source orderhighDevelopers
- Question
- Where AUTO CDC builds SCD tables, is SEQUENCE BY a column that truly reflects source change order (with a tie-breaker), not the time the row landed?
- Why it matters
- Sequencing used the file arrival time; a delayed file replayed older updates as newest, and SCD type 2 history showed customers moving back to old addresses.
- Applies to
- any · components: transformation, modelling · patterns: cdc
- Owner
- Developers · also Source system owners
- Evidence
- in: architecture, data-model
looks like: The sequence column per source, how ties are broken (STRUCT of columns), and the delete condition. - Due by
plan stage 4: Discovery sign-off designed stage 5: MVP starts evidenced stage 7: Test - Fail fast
- Ask the source owner which column orders changes, and check for ties in a real change extract.
- Related
- SCD-02, HIST-03
DBX-12Workspace and catalog layout per environmenthighArchitect
- Question
- Is it decided how environments map to workspaces and catalogs (for example one catalog per environment, bound to its workspace), and how layers map to schemas?
- Why it matters
- All environments shared one metastore catalog with prefixed schema names; a test job pointed at the wrong prefix and overwrote a production schema.
- Applies to
- new platform · components: platform · patterns: any
- Owner
- Architect · also Platform team, Security and governance
- Evidence
- in: platform, decisions
looks like: A diagram of workspaces, catalogs and schemas per environment, with workspace-catalog bindings. - Due by
plan stage 3: Discovery designed stage 4: Discovery sign-off evidenced stage 5: MVP starts - Fail fast
- From the development workspace, try to read and write a production catalog.
- Related
- ENV-01, ENV-04