Data PlatformZoom level 2
Recovery, retention and housekeeping
Sometimes a batch that was loaded and published turns out to be wrong, a week later. Recovery is how it is taken out of every layer. Retention is how long each copy of the data is kept, so that a correct copy is still there to rebuild from. Housekeeping is the job that removes what retention no longer needs.
The three belong together. A platform can only recover by rerun if the older copy it reruns from still exists, and it can only delete old data safely if it knows what a rerun would need.
Recovery: withdrawing a bad batch
To void a batch is to mark its record in the file log as withdrawn, with a reason and the name of who approved it. Every reader skips a void batch from then on. A backfill plan is a record in the metadata store that names windows gold must rebuild regardless of its usual lookback. A window is the slice of a fact that is rebuilt as a unit, such as one promised date.
src/dataplatform/recovery.py:18The whole of recovery's own code.def withdraw(store: MetadataStore, source: str, dataset: str, batch_key: str, *, reason: str, approved_by: str,
today: date) -> BackfillPlan:
store.void_batch(source, dataset, batch_key, reason, approved_by)
plan = BackfillPlan(f"void-{batch_key[:40]}", source, dataset, (EVERY_WINDOW,), today.isoformat(),
f"VOID-01: batch {batch_key} withdrawn: {reason}", approved_by)
store.record_corrections(plan, [])
return plan- Void the batch. The store marks the batch void with the reason and the approver. Silver's reader never takes a void batch.
- Record a plan for every window. A backfill plan is recorded whose window list is the special value meaning every window. It carries no corrections.
- Rebuild silver. The next silver run is started with this dataset named for rebuild. It drops the silver table and replays bronze from the first batch, without the void one.
- Run gold as usual. Gold sees the plan, recomputes every window of the facts that read the dataset, replaces only the windows whose numbers changed, and logs each as a restatement.
A worked example
The values are from the test. Two orders are loaded for 9 November. One has a quantity of 1. The other, in a corrupt batch, has a quantity of 9. A week later the batch is withdrawn. The window is by then outside gold's lookback of five days.
| Case | Quantity gold shows for 9 November | Windows restated |
|---|---|---|
| The batch is withdrawn | 1 | 2026-11-09 |
| No withdrawal | 10 | None. A window outside the lookback is left alone. |
tests/dataplatform/test_gold_harness.py:156The two cases of the withdrawal test.@pytest.mark.parametrize(("case", "withdrawn", "day9_qty", "restated"), [
("a withdrawn batch: the old window is rebuilt and restated", True, 1, ["2026-11-09"]),
("no withdrawal: a window outside the lookback is left alone", False, 10, []),
])Retention: how long each copy is kept
Three copies of the data exist before silver. Landing holds each delivery as it arrived. The landing archive is colder storage for older deliveries. Bronze holds every batch as a table. The table lists the settings with their defaults from the code.
| Setting | Meaning | Default |
|---|---|---|
landing_hot_days | Days a delivery stays in landing before it moves to the archive. | 30 |
landing_archive_days | Days a delivery stays in the archive after that. | The same as bronze. 90 when the dataset holds personal data. |
bronze_retention_days | Days bronze keeps a batch. | 1095 |
bronze_baseline | Whether a baseline batch is written before old bronze is removed. | true |
version_history_days | Days of table versions kept for undo. | 7 |
All five are in the settings registry with status proposed. The personal data limit is a ceiling, not a default: landing holds personal data in clear, so the archive is never kept longer than 90 days for such a dataset whatever the setting says. A dataset holds personal data when any contract column is classified as restricted, personal, pii or sensitive.
src/dataplatform/retention.py:30Reading the five settings, with the ceiling for personal data. @classmethod
def from_settings(cls, settings: dict[str, Any], personal: bool = False) -> Retention:
def num(key: str, default: int) -> int:
value = settings.get(key)
return int(value) if value not in (None, "") else default
bronze = num("bronze_retention_days", cls.bronze_days)
archive_default = PERSONAL_ARCHIVE_DAYS if personal else bronze
archive = num("landing_archive_days", archive_default)
if personal: # never longer than the personal-data limit, whatever the setting says
archive = min(archive, PERSONAL_ARCHIVE_DAYS)
return cls(
landing_hot_days=num("landing_hot_days", cls.landing_hot_days),
landing_archive_days=archive,
bronze_days=bronze,
bronze_baseline=str(settings.get("bronze_baseline", "true")).lower() != "false",
version_history_days=num("version_history_days", cls.version_history_days),
)src/dataplatform/retention.py:56Keep, archive or delete, decided for each delivery.def landing_actions(deliveries: list[tuple[str, str, str]], taken: set[tuple[str, str]], today: date,
r: Retention) -> list[tuple[str, str, str]]:
"""For each (window, name, place) with place 'hot' or 'archive': (window, name, action) with action keep, archive
or delete. A delivery the lander never took (not in `taken`) is always kept: it may be the only copy."""
out = []
for window, name, place in deliveries:
age = (today - window_day(window)).days
if (window, name) not in taken:
action = "keep"
elif place == "hot" and age > r.landing_hot_days + r.landing_archive_days:
action = "delete"
elif place == "hot" and age > r.landing_hot_days:
action = "archive"
elif place == "archive" and age > r.landing_hot_days + r.landing_archive_days:
action = "delete"
else:
action = "keep"
out.append((window, name, action))
return outThe first branch is the safety rule. A delivery the lander never finished with is always kept, however old, because it may be the only copy. The lander is finished with a delivery when its status is loaded, skipped as a duplicate, or void. A held or rejected delivery waits for a fix.
tests/dataplatform/test_retention.py:33The landing cases, with today fixed at 31 December 2026, 30 hot days and 90 archive days.ACTION_CASES = [
("young and hot: keep", ("2026-12-20", "a.csv", "hot"), True, "keep"),
("older than hot: archive", ("2026-11-20", "a.csv", "hot"), True, "archive"),
("archived and young enough: keep", ("2026-11-01", "a.csv", "archive"), True, "keep"),
("past hot + archive: delete", ("2026-09-01", "a.csv", "archive"), True, "delete"),
("hot but past both limits: delete at once", ("2026-09-01", "a.csv", "hot"), True, "delete"),
("never taken by the lander: always kept", ("2026-01-01", "a.csv", "hot"), False, "keep"),
("a month window counts from its first day", ("2026-11", "m.csv", "hot"), True, "archive"),
]Housekeeping: removing old bronze safely
Bronze cannot simply drop batches older than a cut-off. For a change feed, the old batches are the only record of rows that have not changed since. So housekeeping needs a baseline: a complete picture of the table at a point in time. A baseline is either a full copy from the source, or a batch the platform writes from silver and checks against silver. The correction window is how far back a rerun must still be possible.
The keep rule in one sentence: keep everything from the newest checked baseline that is older than the correction window, and remove what is older than that baseline.
src/dataplatform/housekeeping.py:53The decision for one dataset. It is pure: batches and dates in, a plan out.def plan_removals(dataset: str, batches: Iterable[Batch], today: date, correction_days: int) -> Plan:
if correction_days < 0:
raise ValueError(f"{dataset}: correction window {correction_days} days is negative")
bad = sorted({b.load_type for b in batches if b.load_type not in LOAD_TYPES})
if bad:
raise ValueError(f"{dataset}: load type(s) {bad} are not {LOAD_TYPES}")
ordered = sorted(batches, key=lambda b: (b.delivered, b.batch_key))
cutoff = today - timedelta(days=correction_days)
baselines = [b for b in ordered if b.load_type in ("full", "baseline") and b.checked and b.delivered <= cutoff]
if not ordered or ordered[0].delivered >= cutoff:
return Plan(dataset, None, (), False, "nothing older than the correction window")
if not baselines:
return Plan(dataset, None, (), True,
f"no checked baseline on or before {cutoff}: write and check one before removing anything")
base = baselines[-1] # the newest checked baseline inside the allowed range
keep_from = base.delivered
remove = tuple(b.batch_key for b in ordered if b.delivered < keep_from)
return Plan(dataset, keep_from, remove, False,
f"keep from {keep_from} (checked {base.load_type} {base.batch_key}); {len(remove)} batch(es) older")- Nothing old enough. If the oldest batch is inside the correction window, nothing is removed.
- No baseline yet. If no checked baseline exists on or before the cut-off, the plan says a baseline is needed and removes nothing in this run.
- Remove what is older than the baseline. Otherwise the newest qualifying baseline is the keep-from point. Every batch delivered before it is listed for removal, oldest first. The baseline itself stays.
A worked example
The values are from the test. Today is 15 November 2026 and the correction window is 90 days. The first column is the batch, the second how many days ago it was delivered, the third its kind.
| Batch | Days ago | Kind | Decision |
|---|---|---|---|
| b1 | 200 | incremental | Removed |
| b2 | 150 | incremental | Removed |
| base | 120 | baseline, checked | Kept. It is the keep-from point. |
| b3 | 100 | incremental | Kept |
| b4 | 5 | incremental | Kept |
tests/dataplatform/test_housekeeping.py:20The decision's case table.@pytest.mark.parametrize(
("case", "batches", "window", "remove", "needs_baseline"),
[
("everything inside the window stays", [inc("b1", 10), inc("b2", 5)], 30, [], False),
("change feed with no baseline: write one first, remove nothing",
[inc("b1", 120), inc("b2", 100), inc("b3", 5)], 90, [], True),
("checked baseline before the cut-off: older batches go",
[inc("b1", 200), inc("b2", 150), Batch("base", d(120), "baseline", True), inc("b3", 100), inc("b4", 5)],
90, ["b1", "b2"], False),
("an unchecked baseline does not count",
[inc("b1", 200), Batch("base", d(120), "baseline", False), inc("b3", 5)], 90, [], True),
("a baseline inside the window does not count yet",
[inc("b1", 200), Batch("base", d(30), "baseline", True), inc("b3", 5)], 90, [], True),
("the newest qualifying baseline wins",
[Batch("f1", d(300), "full", True), inc("b1", 250), Batch("f2", d(100), "full", True), inc("b2", 5)],
90, ["f1", "b1"], False),
("daily full copies: each day is its own baseline",
[Batch(f"c{n}", d(n), "full", True) for n in (40, 39, 36, 35, 34)], 35, ["c40", "c39", "c36"], False),
("no batches at all", [], 90, [], False),
],src/dataplatform/housekeeping.py:158The job for one dataset, run while holding the dataset's bronze lease.def _sweep_leased(store: MetadataStore, bronze: BronzeRemover, run_id: str, today: date,
what: tuple[str, dict[str, Any], int], *, baseliner: Baseliner | None) -> Swept:
name, view, days = what
source, dataset = view["source"], view["dataset"]
plan = plan_removals(name, batches_of(store.dataset_files(source, dataset)), today, days)
got = None
if plan.needs_baseline and baseliner is not None:
got = baseliner.write(store, run_id, view, today - timedelta(days=days), today)
status = {"written": "succeeded", "refused": "failed"}.get(got.status, "skipped")
store.log_step(StepRecord(run_id, source, dataset, BASELINE_STEP, status, got.counts, got.message))
if got.status == "written":
plan = plan_removals(name, batches_of(store.dataset_files(source, dataset)), today, days)
removed = bronze.remove(name, list(plan.remove)) if plan.remove else 0
reason = REASON.format(base=plan.keep_from, days=days)
for key in plan.remove:
store.void_batch(source, dataset, key, reason, "housekeeping")
status = "warned" if plan.needs_baseline else "succeeded"
counts = {"batches_removed": len(plan.remove), "rows_removed": removed, "correction_days": days}
store.log_step(StepRecord(run_id, source, dataset, STEP, status, counts, plan.reason))
store.finish_dataset_run(run_id, source, dataset, "succeeded", counts=counts, message=plan.reason)
return Swept(name, plan, removed, got)Walk through it. The plan is made from the file log. If a baseline is needed and a baseliner was given, it writes one, checks it, and logs the step housekeeping.baseline. If the baseline was written, the plan is made again. The listed batches are removed from bronze in one transaction, and each is then voided in the file log with the reason housekeeping. The file log stays append-only, and every later reader, a rebuild included, skips the removed batches. The step housekeeping.bronze records the plan, and is warned when the dataset still waits for a baseline.
The job takes the same lease as the lander, so it never removes while a load is writing. A lease is a short lock with a time limit, kept in the metadata store.
The run script can prove a rebuild afterwards. With the option --prove-rebuild it rebuilds silver for every dataset that received a baseline and compares the result with silver as it stood.
How it is tested
tests/dataplatform/test_housekeeping.py has the decision table shown above, a refusal table, and a table for the whole job run against in-memory bronze. tests/dataplatform/test_retention.py has a table for the settings, the table for landing actions, and one test that moves and deletes real files in a temporary landing folder. tests/dataplatform/test_baseline_stage.py covers writing and checking a baseline. The withdrawal is tested in tests/dataplatform/test_gold_harness.py.
- The function that carries out landing retention is called only by its test. No job or run script applies it, and on a cloud the same rules would be lifecycle rules that are not generated from these settings.
- Withdrawing a batch is a function with no command or screen around it. It is called only by its test. The order of the three steps is a convention that the caller must follow.
- Expiring old table versions is left to the table format's own upkeep.
version_history_daysis read into the retention object but nothing in this code acts on it. - The five retention settings are proposed in the registry, not registered.
- A loaded full copy counts as a checked baseline without a separate check, on the reasoning that silver already took it through the key comparison and the mass delete guard.