Data PlatformZoom level 2
The five pipelines and how one is chosen
A pipeline is a function that answers one question for one dataset: which batches are new since last time. It hands each new batch to the entry steps and reports the position to remember.
There are five because sources offer five different things to hold on to: a list of files, a list of copies, a page cursor, a position in a stream. The pipelines are short because all the checking and writing is somewhere else.
src/dataplatform/ingest/pipelines.py:386The whole chooser.PIPELINES: dict[tuple[str, str], Callable[[Ctx, Sources, Plan], Result]] = {
("push", "change_feed"): push,
("push", "files"): push,
("pull", "full_copy"): pull_copy,
("pull", "pages"): pull_pages,
("stream", "events"): stream_events,
}The five at a glance
| Pipeline | What the source gives | How new data is found | What is remembered | How deletes are seen | Tyre datasets |
|---|---|---|---|---|---|
| Push, change feed | Files in landing, each a set of change rows | Deliveries in landing that the file log has not recorded | Bookmark feed_pos, the watermark | A change row with operation D | Sales order lines, deliveries, production |
| Push, files | Files in landing, bytes as the partner sent them | The same, limited to names that match file_pattern | Bookmark last_file, one bookmark per feed | Not seen | Inspection results, dealer sell-out |
| Pull, full copy | A whole copy of a table, published each night | The newest copy later than the bookmark | Bookmarks copy_extracted_at and copy_batch | A key that is no longer in the copy. Silver marks it. The guard here holds a copy that loses too many. | Materials, presses |
| Pull, pages | Records changed after a cursor, a page at a time | The stored cursor | Bookmark cursor, the watermark | Not seen | Dealers |
| Stream, events | Records per shard, each with a sequence number | The stored sequence number per shard | One bookmark per shard, the watermark | Not seen | Press readings |
A shard is one of the parallel parts a stream is split into. A cursor is a marker an interface returns that means the caller has seen everything up to here.
What every pipeline receives and returns
src/dataplatform/ingest/pipelines.py:39The run type and, for a reload or a backfill, the windows to go over.@dataclass
class Plan:
"""What a run does for one dataset: the run type and, for reload and backfill, the windows."""
run_type: str = "normal"
window: str = "" # the business date of a normal run (stamped on pulled batches)
windows: tuple[str, ...] = ()src/dataplatform/ingest/pipelines.py:70What a pipeline returns: the outcome of each batch, the positions to store, and counts of what it noticed.@dataclass
class Result:
outcomes: list[Outcome] = field(default_factory=list)
bookmarks: dict[str, str] = field(default_factory=dict)
watermark: str | None = None
noticed: dict[str, int] = field(default_factory=dict) # detector -> count (DEL-01, SCD-02, LATE-03, DUP-04)
notes: list[str] = field(default_factory=list)
def notice(self, detector: str, n: int) -> None:
if n:
self.noticed[detector] = self.noticed.get(detector, 0) + n1. Push, change feed
A change feed is a list of the inserts, updates and deletes made in a source table, in the order they were committed. A change capture mover writes pieces of the feed into landing as files. The tyre company's sales order lines, deliveries and production records arrive this way.
src/dataplatform/ingest/pipelines.py:130The push pipeline. It serves both change feed deliveries and plain files.def push(ctx: Ctx, src: Sources, plan: Plan) -> Result:
s, res = ctx.spec, Result()
files = ctx.store.dataset_files(s.source, s.dataset)
seen, loaded = _pick(files)
pattern = s.settings.get("file_pattern")
for d in sorted(src.landing.deliveries(s.source, s.dataset), key=lambda d: (d.window, d.name)):
if pattern and s.read == "files" and not fnmatch(d.name, pattern):
continue
if plan.run_type == "normal" and (d.window, d.name) in seen:
continue
if plan.run_type != "normal" and (
d.window not in plan.windows or (plan.run_type == "backfill" and (d.window, d.name) in loaded)
):
continue
data = src.landing.read_bytes(d)
rows: list[Row] | None = None
if s.read == "change_feed":
names = feed_columns(s.settings)
rows = [{names.get(k, k): _op_value(k, v, names) for k, v in r.items()} for r in parse(data, s.settings)]
_feed_notices(ctx, rows, res)
ctx.rewrite = plan.run_type == "reload"
o = enter(Batch(d.window, d.name, data, rows, d.declared_rows), ctx)
res.outcomes.append(o)
if o.status == "loaded" and rows and all(isinstance(r.get("_feed_pos"), int) for r in rows):
res.bookmarks["feed_pos"] = (
_max_mark(
res.bookmarks.get("feed_pos") or ctx.store.bookmarks(s.source, s.dataset).get("feed_pos"),
[r["_feed_pos"] for r in rows],
)
or ""
)
res.watermark = _watermark(ctx, rows, res.watermark)
elif o.status == "loaded" and rows: # a change feed with no integer position (a change-capture service)
res.bookmarks["last_file"] = d.name
res.watermark = _watermark(ctx, rows, res.watermark)
elif o.status == "loaded":
res.bookmarks["last_file"] = d.name
_feed_bookmarks(ctx, src, o.feeds, res)
return resWalk through it. The file log of the dataset is read once and turned into two sets: the deliveries recorded at all, and the deliveries loaded. The deliveries in landing are then taken in window and name order. In a normal run a delivery the file log already has is passed over. That list is how new data is found. The pipeline does not use the watermark to choose.
For a change feed the bytes are parsed here, because three columns have to be renamed first. The source's operation, feed position and commit time become _op, _feed_pos and _committed_at. The batch then goes to enter. When it loads, the pipeline works out the highest feed position and the highest value of the watermark column in the rows, and reports both.
src/dataplatform/ingest/pipelines.py:207What the pipeline counts in a change delivery: deletes, and changes that arrived out of order.def _feed_notices(ctx: Ctx, rows: list[Row], res: Result) -> None:
"""Deletes arrive as op D (DEL-01). A change whose sequence is older than one already seen for its key in the
delivery arrived out of order (SCD-02); silver orders history by the sequence (W1c)."""
res.notice("DEL-01", sum(r.get("_op") == "D" for r in rows))
order = ctx.spec.load.get("history_order_by")
if not order:
return
newest: dict[tuple[Any, ...], Any] = {}
late = 0
for r in rows:
k = tuple(r.get(c) for c in ctx.spec.primary_key)
if k in newest and r.get(order) is not None and int(r[order]) < int(newest[k]):
late += 1
else:
newest[k] = r.get(order)
res.notice("SCD-02", late)Deletes are seen as rows whose operation is D. They are counted and written to bronze like any other row. Applying the delete is silver's job. See Deletes. A change is counted as out of order when its sequence is lower than one already seen for the same key in the same delivery.
src/dataplatform/ingest/pipelines.py:171A change capture service may name its three columns differently. Three settings map them.def feed_columns(settings: dict[str, str]) -> dict[str, str]:
"""Delivery column -> standard feed column. A change-capture service names its columns its own way (ADR-031):
cdc_op_column (default op), cdc_position_column (default feed_pos; empty when the service has none) and
cdc_commit_column (default committed_at). Unset keys keep the defaults W1's own feed uses."""
out = {}
for key, default, std in (("cdc_op_column", "op", "_op"), ("cdc_position_column", "feed_pos", "_feed_pos"),
("cdc_commit_column", "committed_at", "_committed_at")):
name = settings.get(key, default)
if name:
out[name] = std
return out2. Push, files
Partners and the laboratory drop CSV files. A file mover copies each file into landing byte for byte. The same push function handles them, with three differences. A delivery whose name does not match the dataset's file_pattern is left alone. The bytes are not parsed in the pipeline. And the bookmark is the name of the last file loaded.
src/dataplatform/ingest/pipelines.py:191One bookmark per feed. For the dealer sell-out files, a feed is a dealer.def _feed_bookmarks(ctx: Ctx, src: Sources, feeds: dict[str, Any], res: Result) -> None:
"""One bookmark per feed (a dealer), holding its newest data time as an instant in the business time zone; it
only moves forward. Freshness compares feeds with their siblings; gold reads them for completeness (ADR-018)."""
if not feeds:
return
stored = ctx.store.bookmarks(ctx.spec.source, ctx.spec.dataset)
for feed, value in feeds.items():
if isinstance(value, datetime):
t = value if value.tzinfo else value.replace(tzinfo=ZoneInfo(src.business_tz))
else:
t = datetime(value.year, value.month, value.day, tzinfo=ZoneInfo(src.business_tz))
old = res.bookmarks.get(feed) or stored.get(feed)
if not old or datetime.fromisoformat(old) < t:
res.bookmarks[feed] = t.isoformat()A dealer sends one file a month. If every row of a file shares one dealer code, the pipeline keeps a bookmark named after that dealer with the newest month in the file. A dealer that has not sent its file can then be seen as behind the others. Files have no way to say that a row was deleted, and the two tyre manifests say so with deletes.detect: none.
3. Pull, full copy
Small master tables are published whole each night. The run takes only the newest copy. An older copy that was never taken is of no use once a newer one exists.
src/dataplatform/ingest/pipelines.py:245The full copy pipeline.def pull_copy(ctx: Ctx, src: Sources, plan: Plan) -> Result:
"""The newest full copy only: a newer copy supersedes an older one (and a held one)."""
s, res = ctx.spec, Result()
system, table = src.systems[s.source], _table_spec(ctx, "full_copy")
marks = ctx.store.bookmarks(s.source, s.dataset)
files = ctx.store.dataset_files(s.source, s.dataset)
if plan.run_type == "normal":
done = {f.file_name for f in files}
new = [
at
for at in _read(ctx, src, lambda: src.db.copies(system, table))
if f"copy:{at}" not in done and at > marks.get("copy_extracted_at", "")
]
todo = [(_window_of(new[-1], src.business_tz), new[-1])] if new else []
else:
todo = [(f.window, f.file_name.split(":", 1)[1]) for f in _replay_names(files, plan)]
last_ok = marks.get("copy_batch") or next((f.batch_key for f in reversed(files) if f.status == "loaded"), "")
for window, at in todo:
ctx.previous_keys = _guard_keys(ctx, last_ok)
ctx.rewrite = plan.run_type == "reload"
name = f"copy:{at}"
rows = _read(ctx, src, lambda at=at: src.db.copy(system, table, at))
o = enter(Batch(window, name, f"{s.table}/{name}".encode(), rows), ctx)
res.outcomes.append(o)
if o.status == "loaded" and plan.run_type == "normal":
res.bookmarks.update(copy_extracted_at=at, copy_batch=o.batch_key)
return resWalk through it. The source lists the extraction times of its copies. A copy is new when the file log has no record of it and its time is later than the bookmark copy_extracted_at. Of the new ones only the last is kept. Before the copy is read, the pipeline fetches the keys to compare it with and puts them on the context for the mass-delete guard. The batch is named copy: followed by the extraction time. When it loads, two bookmarks are reported: the extraction time and the batch key.
src/dataplatform/ingest/pipelines.py:234What the guard compares a copy with: silver's current keys if silver exists, else the last accepted copy in bronze.def _guard_keys(ctx: Ctx, last_ok: str) -> set[tuple[Any, ...]] | None:
"""What the mass-delete guard compares a full copy with: silver's current keys (DPC-0054), or the last accepted
copy in bronze while silver has never been written (the first days, or a binding without silver)."""
s = ctx.spec
if ctx.silver is not None:
keys = ctx.silver.current_keys(s.source, s.dataset, s.primary_key)
if keys is not None:
return keys
return ctx.writer.keys(s.table, s.primary_key, {"_batch_key": last_ok}) if last_ok else NoneA full copy shows a delete by absence. A material that was in yesterday's copy and is not in today's was deleted. Ingestion does not mark anything. It writes the copy to bronze, and silver compares keys. Ingestion's part is the guard that refuses a copy which would delete too much at once.
4. Pull, pages
The dealer system is an interface that returns the records changed after a cursor, oldest first, 200 at a time.
src/dataplatform/ingest/pipelines.py:274The page pipeline. One page is one batch.def pull_pages(ctx: Ctx, src: Sources, plan: Plan) -> Result:
"""Pages since the cursor, one batch per page; the batch name holds the cursor range so a reload can re-read it."""
s, res = ctx.spec, Result()
system = src.systems[s.source]
files = ctx.store.dataset_files(s.source, s.dataset)
if plan.run_type == "reload":
res.notes.append(
"an interface serves only current records, so a page cannot be read again as it was: "
"bronze is this dataset's replay point (ADR-002); nothing re-read"
)
return res
if plan.run_type == "backfill": # pages held earlier: read again, keeping only records up to the page's end
for f in _replay_names(files, plan):
start, end = f.file_name.removeprefix("page:").split(">", 1)
items, _ = _read(ctx, src, lambda start=start: src.api.page(system, s.dataset, start, PAGE_SIZE))
keep = [dict(r) for r in items if _cursor_of(r, s) <= end]
res.outcomes.append(enter(Batch(f.window, f.file_name, f"{s.table}/{f.file_name}".encode(), keep), ctx))
return res
cursor = _resume_cursor(ctx.store.bookmarks(s.source, s.dataset).get("cursor", ""), files)
while True:
items, nxt = _read(ctx, src, lambda cursor=cursor: src.api.page(system, s.dataset, cursor, PAGE_SIZE))
if not items:
break
name = f"page:{cursor}>{nxt}"
o = enter(Batch(plan.window, name, f"{s.table}/{name}".encode(), [dict(r) for r in items]), ctx)
res.outcomes.append(o)
if o.status not in ("loaded", "skipped_duplicate"):
break # held: the next run starts again at this page
res.watermark = _watermark(ctx, [dict(r) for r in items], res.watermark)
cursor = res.bookmarks["cursor"] = nxt
return resWalk through the normal path at the bottom. The starting cursor is the stored bookmark, moved forward if the file log shows pages loaded after it. The loop asks for a page, enters it as a batch named page: with the cursor range, and moves the cursor. It stops on an empty page. It also stops when a page is held, so the next run starts again at that page.
A reload does nothing here and says why. The interface only serves records as they are now, so a page cannot be read again as it was. Bronze is the only place the old page still exists.
src/dataplatform/ingest/pipelines.py:312Why a failed run does not cause a page to be read twice.def _resume_cursor(bookmark: str, files: list[FileRecord]) -> str:
"""The cursor after the last page loaded, if a failed run loaded pages its bookmark does not show."""
ends = [f.file_name.split(">", 1)[1] for f in files if f.status == "loaded" and f.file_name.startswith("page:")]
return max([bookmark, *ends])5. Stream, events
Press readings are appended to a stream all day. A run reads each shard from the last sequence number it loaded to the end of the shard, makes one batch per shard, and stops.
src/dataplatform/ingest/pipelines.py:319The stream pipeline.def stream_events(ctx: Ctx, src: Sources, plan: Plan) -> Result:
"""Trigger-once: each shard from its checkpoint to the end, one batch per shard, then stop (BP-044). Bronze keeps
every reading as delivered (RAW-01); replays and late readings are counted here and dropped or placed by event
time in silver (DUP-04, LATE-03; W1e)."""
s, res = ctx.spec, Result()
files = ctx.store.dataset_files(s.source, s.dataset)
if plan.run_type != "normal":
for f in _replay_names(files, plan):
shard, rng = f.file_name.rsplit(":", 1)
first, last = (int(x) for x in rng.split("-"))
recs = src.stream.read(s.dataset, shard, first - 1, last - first + 1)
ctx.rewrite = plan.run_type == "reload"
res.outcomes.append(
enter(Batch(f.window, f.file_name, f"{s.table}/{f.file_name}".encode(), _stream_rows(shard, recs)), ctx)
)
return res
marks = ctx.store.bookmarks(s.source, s.dataset)
loaded_to: dict[str, int] = {}
for f in files:
if f.status == "loaded" and f.file_name.startswith("shard-"):
shard, rng = f.file_name.rsplit(":", 1)
loaded_to[shard] = max(loaded_to.get(shard, 0), int(rng.split("-")[1]))
seen: set[str] = set()
for shard in src.stream.shards(s.dataset):
after = max(int(marks.get(shard, "0")), loaded_to.get(shard, 0))
recs = []
while chunk := src.stream.read(s.dataset, shard, after, STREAM_CHUNK):
recs += chunk
after = chunk[-1].seq
if not recs:
continue
rows = _stream_rows(shard, recs)
_stream_notices(rows, seen, res)
name = f"{shard}:{recs[0].seq}-{recs[-1].seq}"
o = enter(Batch(plan.window, name, f"{s.table}/{name}".encode(), rows), ctx)
res.outcomes.append(o)
if o.status == "loaded":
res.bookmarks[shard] = str(recs[-1].seq)
res.watermark = _watermark(ctx, rows, res.watermark)
return resWalk through the normal path. The stored bookmarks hold one sequence number per shard. The file log is also scanned for the highest sequence loaded per shard, and the larger of the two is where reading starts. Records are read in chunks of 200,000 until the shard is empty. The batch is named with the shard and the first and last sequence number, which is what lets a reload read the same range again. Every reading is written to bronze, replays included.
src/dataplatform/ingest/pipelines.py:365Replayed readings and late readings are counted, not removed.def _stream_notices(rows: list[Row], seen: set[str], res: Result) -> None:
"""DUP-04: a reading id read before in this run is a replay. LATE-03: a reading whose event time is more than the
allowance behind the newest already read on its shard arrived out of order."""
newest, replays, late = "", 0, 0
for r in rows:
rid, ts = str(r.get("reading_id")), str(r.get("event_ts"))
if rid in seen:
replays += 1
continue
seen.add(rid)
if (
newest
and ts < newest
and (datetime.fromisoformat(newest) - datetime.fromisoformat(ts)).total_seconds() > LATE_ALLOWANCE_S
):
late += 1
newest = max(newest, ts)
res.notice("DUP-04", replays)
res.notice("LATE-03", late)Where a run starts after a failure
Bookmarks move only when a dataset run succeeds. But a batch is committed by its file log record, and a run can fail after some batches loaded. Each pipeline therefore reads both the bookmark and the file log and starts from whichever is further on. The duplicate check in the entry steps is the second line of defence.
Reload and backfill
src/dataplatform/ingest/pipelines.py:107Which recorded batches a reload or a backfill goes over again.def _replay_names(files: list[FileRecord], plan: Plan) -> list[FileRecord]:
"""The recorded batches a reload (all) or backfill (not loaded) re-enters, latest record per batch."""
latest: dict[tuple[str, str], FileRecord] = {}
for f in files:
if f.window in plan.windows:
latest[(f.window, f.file_name)] = f
keep = [f for f in latest.values() if plan.run_type == "reload" or f.status not in ("loaded", "skipped_duplicate")]
return sorted(keep, key=lambda f: (f.window, f.file_name))| Run type | What it takes | Does it move positions |
|---|---|---|
| normal | Batches not seen before. | Yes, on success. |
| reload | Every batch recorded for the given windows, loaded or not. The write replaces the rows of the batch key, so nothing changes if the data is the same. | No. |
| backfill | The batches of the given windows that are not loaded yet. A batch that was held is checked again. | No. |
A worked example: three runs of the sales order lines
The values are small and made up to show the mechanics. The first column is the run, the second what is in landing, the third what the pipeline does, the fourth what the store remembers.
| Run | Landing holds | Pipeline | Stored afterwards |
|---|---|---|---|
| Monday | File A with feed positions 1 to 250 | A is not in the file log. It is entered and loads. | File log: A loaded. Bookmark feed_pos 250. |
| Tuesday | A, and file B with positions 251 to 300 | A is in the file log and is passed over. B loads. | File log: A, B loaded. Bookmark 300. |
| Tuesday again | A and B | Both are in the file log. Nothing is entered. | The dataset run is skipped. Nothing moves. |
The settings the pipelines read
| Key | Read by | Meaning | Allowed values and default |
|---|---|---|---|
ingest.mode, ingest.read | The chooser | Which pipeline runs. | The five pairs above. No default. |
file_pattern | Push, files | Only deliveries whose name matches are taken. | A file name pattern. No default, which means every delivery. |
watermark_column | Push, pages, stream | The column whose highest loaded value is recorded as the watermark. | A column name. No default, which means no watermark is recorded. |
cdc_op_column, cdc_position_column, cdc_commit_column | Push, change feed | The names of the operation, position and commit time columns in the delivery. | Text. The dictionary has no default. The code falls back to op, feed_pos and committed_at. |
load.history_order_by | Push, change feed | The sequence column used to count changes that arrived out of order. | A contract column. From the manifest's load block. |
retry_count | Pull | How many times a failed source call is tried again. | Whole number. Default 2. |
retry_interval_seconds | Pull | The wait before the first retry. Each later wait is twice the one before, up to 600 seconds. | Whole number. Default 60. |
src/dataplatform/ingest/pipelines.py:62Every pull goes through the source guard when the binding supplies one.def _read(ctx: Ctx, src: Sources, fn: Callable[[], T]) -> T:
"""One call to a source through the run's guard: retried on temporary errors per the dataset's retry settings,
refused while the source's circuit is open, limited by the source's connection cap."""
if src.guard is None:
return fn()
return src.guard.call(ctx.spec.source, fn, RetryPolicy.from_settings(ctx.spec.settings))src/dataplatform/resilience.py:42The two retry settings become a retry policy. @classmethod
def from_settings(cls, settings: dict[str, Any]) -> RetryPolicy:
return cls(retries=int(settings.get("retry_count", 2) or 0),
interval_s=float(settings.get("retry_interval_seconds", 60) or 0))How it is tested
tests/dataplatform/test_ingest_pipelines.py:185A change feed with an out-of-order change and a delete: one batch, five rows, both noticed, bookmark and watermark set, every change kept in bronze.def test_change_feed_lands_and_notices(tmp_path):
r = Rig(tmp_path)
r.change("I", "A", 5)
r.change("I", "B", 1)
r.change("U", "A", 7, seq=10)
r.change("U", "A", 6, seq=9) # an older change committed after a newer one: SCD-02
r.change("D", "B", 1) # a hard delete: DEL-01
d = r.run()["erp.line"]
assert (d.status, d.counts["loaded"], d.counts["rows_loaded"]) == ("succeeded", 1, 5)
assert d.noticed == {"DEL-01": 1, "SCD-02": 1}
assert r.store.bookmarks("erp", "line") == {"feed_pos": "5"} and r.store.watermark("erp", "line") == "10"
ops = sorted(x["_op"] for x in r.bronze.tables["erp.line"])
assert ops == ["D", "I", "I", "U", "U"] # bronze keeps every change as delivered (RAW-01)tests/dataplatform/test_ingest_pipelines.py:396A stream run that fails after loading. The bookmarks did not move, yet the next run reads nothing twice.def test_a_failed_run_never_makes_the_next_read_a_batch_twice(tmp_path):
"""ORCH-03: the file log commits each batch, so after a run fails midway the next run starts after the last batch
loaded (though bookmarks stayed) and nothing is loaded twice."""
r = Rig(tmp_path)
r.readings([(f"R{i}", ts(1, i), ts(1, i)) for i in range(10)])
view = r.store.dataset_view("bus", "reading")
r.store.start_run("crash", "t", "dev")
r.store.start_dataset_run("crash", "bus", "reading", "fp", view)
stream_events(
Ctx(Spec.from_view(view), r.store, r.bronze, KeyedTokenizer(b"k" * 32), "crash"), r.sources, Plan("normal", W1)
)
r.store.finish_dataset_run("crash", "bus", "reading", "failed") # bookmarks stay where they were
assert r.store.bookmarks("bus", "reading") == {}
assert r.run()["bus.reading"].status == "skipped" and r.rows("bus.reading") == 10- The watermark is recorded but no pipeline uses it to decide what to read. New data is found from the file log, the copy list, the cursor or the stream position.
- The dealer manifest sets
rest_endpoint_path,rest_methodand thepaginationkeys. The page pipeline does not read them. It calls the interface port with the dataset name and a page size of 200 that is fixed in the code. - The circuit breaker's threshold and reset time are registered as proposed settings and the source connection cap as a source setting. The guard is built with its code defaults, three failures and 30 minutes, and reads none of them.
- The push pipeline is not wrapped by the source guard. Only the pulls are. The stream reads are not wrapped either.
- The stream pipeline reads a whole shard's new records into memory before it enters them as one batch.
- A reload of a page dataset is a no-op by design.
- The stream pipeline is tested with the in-memory stream. By reading the code I see three things that would need checking against a managed stream whose shard names and sequence numbers look different: the scan of the file log looks for names that begin with
shard-, a reload works out how many records to read by subtracting two sequence numbers, and the sequence number is stored in an integer column. I did not run this and could not verify it either way.