Data PlatformZoom level 2
Source ports and movers: database, change feed, files, interface, stream
Ingestion never names a product. It talks to four small interfaces, called ports, that describe a source by what it is: a database, a file drop, a record interface, an event stream. An adapter is the code that puts a real system, or a stand-in for one, behind a port.
This page lists every port and every adapter, and says plainly what each adapter is. That matters here, because most of them were built to let a simulated tyre company run on a laptop, and a reader should not mistake them for connectors to production systems.
The honest summary
| Kind of source | Adapters | What it really is |
|---|---|---|
| Database with a change feed and full copies | MemorySourceDb, SqliteSource, PostgresSource | A stand-in. All three read tables that the simulated company itself writes in a layout of our own: the live table, a table of changes and a table of copies. None connects to a real ERP or reads a database's own change log. |
| Change feed mover | ChangeFeedMover, DmsMover | A stand-in for a change capture service. The second one imitates the file layout such a managed service writes, with Parquet files and its column names. No managed service is called. |
| File drop | MemoryFileDrop, LocalFileDrop, S3FileDrop | The first two are local. The third makes real calls to a cloud object storage service through the vendor's library. |
| Drop mover | DropMover | Real but simple code: it copies new files from a drop to landing. It stands in for a file transfer service. |
| Record interface | MemoryRecords, LocalRecordStore with serve, HttpRecordClient | A stand-in. A small HTTP server on the same machine serves the simulated dealer system. The client refuses any address that is not the local machine. |
| Event stream | MemoryStream, LocalStream, KinesisStream | The first two are local. The third makes real calls to a managed streaming service through the vendor's library. |
| Landing | FileLanding, S3Landing | A folder, which is also how a storage volume of the engine is reached, and cloud object storage. |
For the two cloud adapters that make real calls, the tests and the recorded run in the cloud realm's README use a local stand-in for the cloud service. I found no record of a run against the real services and could not verify one.
Why the ports have write methods
src/dataplatform/source_ports.py:1The module's own opening. It calls them mock-source ports."""The four mock-source ports (goal G4, data-platform-catalog dpc/ports.yaml: source systems). The world writes into source systems only through these shapes, so the same simulated company feeds any cloud: a database on the laptop today, a managed one later; a local folder today, object storage later; a local log of shards today, a managed stream later. Each is a separate adapter (M2) and each passes the same contract tests (M5). Names here say what a system is (a database, a file drop, an interface, a stream), never which product it is (R1). """
A connector to a real system would only read. These ports also have methods that write: apply, publish_copy, put, upsert and append. They are there because the simulated company uses the same ports to act as the source systems. Ingestion calls only the reading methods.
Port 1: the source database
src/dataplatform/source_ports.py:56A database with a change feed and a nightly full copy.class SourceDatabase(Protocol):
"""A database with a change feed (a change-capture source) and a nightly full copy for master tables."""
def ensure(self, system: str, tables: list[TableSpec], reset: bool = False) -> None: ...
def apply(self, system: str, table: TableSpec, changes: list[SourceChange]) -> None:
"""Apply changes to the live table (newest change sequence wins) and append the in-feed ones to the feed."""
def publish_copy(self, system: str, table: TableSpec, extracted_at: str, rows: list[Row]) -> None: ...
def rows(self, system: str, table: TableSpec) -> list[Row]:
"""The live table now, sorted by key."""
def feed(self, system: str, table: TableSpec, after_pos: int = 0) -> list[Row]:
"""Feed rows in commit order after a feed position, each with feed_pos, op and committed_at beside the row's
own columns. Readers keep a feed position, not a change sequence: an old change can commit late."""
def copy(self, system: str, table: TableSpec, extracted_at: str) -> list[Row]: ...
def copies(self, system: str, table: TableSpec) -> list[str]:
"""The extraction times of the full copies published so far, oldest first (a puller takes the newest)."""| Method | Used by | What it returns |
|---|---|---|
feed | The change feed mover | The change rows after a feed position, in commit order, each with its position, operation and commit time. |
copies, copy | The full copy pipeline | The extraction times of the copies published so far, and the rows of one copy. |
rows | The platform script, when it compares the live source with silver | The live table as it stands. |
ensure, apply, publish_copy | The simulated company | Nothing. They create tables and write changes and copies. |
The docstring of feed states a rule worth keeping: a reader keeps a feed position, not a change sequence, because an old change can be committed late.
src/dataplatform/adapters/sql_source.py:151The feed on a SQL database: a plain query on a changes table that the simulated company fills. def feed(self, system: str, table: TableSpec, after_pos: int = 0) -> list[Row]:
return self._select(
f"SELECT * FROM {self.d.qualify(system, table.name + '__changes')} " # nosec B608
f"WHERE feed_pos > {self.d.placeholder} ORDER BY feed_pos",
(after_pos,),
)This is the clearest sign of what the adapter is. The feed is a table named after the source table with __changes added, and the full copies are a table with __copy added. A real database does not have these. A real connector would read the database's own change log, or a managed change capture service would, and write files into landing.
The movers: from a source to landing
src/dataplatform/adapters/local_movers.py:23The change feed mover. Its checkpoint is the highest feed position among the deliveries it has already landed.class ChangeFeedMover:
def __init__(self, db: SourceDatabase, landing: Landing):
self.db, self.landing = db, landing
def checkpoint(self, source: str, dataset: str) -> int:
landed = [self.landing.manifest(d) for d in self.landing.deliveries(source, dataset)]
return max((int(m.get("feed_pos_to", 0)) for m in landed), default=0)
def move(self, system: str, source: str, table: TableSpec, window: str) -> Delivery | None:
rows = self.db.feed(system, table, self.checkpoint(source, table.name))
if not rows:
return None # nothing changed: a change-capture service writes no empty file
first, last = rows[0]["feed_pos"], rows[-1]["feed_pos"]
name = f"{table.name}_{window}_{first:010d}-{last:010d}.jsonl"
try:
return self.landing.put(
source,
table.name,
window,
name,
to_bytes(rows),
len(rows),
{"feed_pos_from": first, "feed_pos_to": last},
)
except DeliveryExists:
return NoneA mover copies and never changes data. This one asks the database for feed rows after its checkpoint and lands them as one JSON lines delivery, named with the first and last feed position. It writes no empty file. If landing refuses the write because the delivery exists, the mover treats it as already done.
src/dataplatform/adapters/local_movers.py:51The drop mover: every file not yet in landing is copied byte for byte, under the business date of its arrival.class DropMover:
def __init__(self, drop: FileDrop, landing: Landing, business_tz: str):
self.drop, self.landing, self.tz = drop, landing, ZoneInfo(business_tz)
def move(self, system: str, drop_dataset: str, source: str, dataset: str) -> list[Delivery]:
landed = {d.name for d in self.landing.deliveries(source, dataset)}
out = []
for f in self.drop.list(system, drop_dataset):
if f.name in landed:
continue
data = self.drop.get(system, drop_dataset, f.name)
window = datetime.fromisoformat(f.arrived_at).astimezone(self.tz).date().isoformat()
rows = max(0, data.count(b"\n") - 1) # a header line, then one line per row
try:
out.append(self.landing.put(source, dataset, window, f.name, data, rows, {"arrived_at": f.arrived_at}))
except DeliveryExists:
continue
return outsrc/dataplatform/adapters/local_movers.py:111The settings a dataset needs when its deliveries have the managed service's shape.DMS_SETTINGS = {"file_format_data": "parquet", "cdc_op_column": "Op", "cdc_position_column": "",
"cdc_commit_column": "ar_commit_ts"}Port 2: the file drop
src/dataplatform/source_ports.py:79A folder that partners and the laboratory drop files into.class FileDrop(Protocol):
"""A folder partners and lab systems drop files into. Arrival time is part of the file (object storage: LastModified)."""
def put(self, system: str, dataset: str, name: str, data: bytes, arrived_at: str) -> DropFile: ...
def list(self, system: str, dataset: str) -> list[DropFile]:
"""Files in arrival order."""
def get(self, system: str, dataset: str, name: str) -> bytes: ...The local adapter keeps a file's arrival time as its modified time. The object storage adapter keeps it as a piece of metadata on the object, because the storage service stamps its own upload time. Both refuse a name that contains a path.
Port 3: the record interface
src/dataplatform/source_ports.py:90An interface that returns records changed after a cursor, in pages.class RecordInterface(Protocol):
"""A request-response interface over a set of records, read in pages since a cursor (the dealer system)."""
def upsert(self, system: str, resource: str, key: str, record: Row, updated_at: str) -> None: ...
def page(self, system: str, resource: str, cursor: str, limit: int) -> tuple[list[Row], str]:
"""Records changed after `cursor` ('' = from the start), oldest first; returns them and the next cursor."""src/dataplatform/adapters/http_api.py:106The HTTP client. Read the check on the address in the page method.class HttpRecordClient:
"""Reads the interface over HTTP. Writes go to the store: a real dealer system is not written by its readers."""
def __init__(self, base_url: str, store: LocalRecordStore | None = None):
self.base = base_url.rstrip("/")
self.store = store
def upsert(self, system: str, resource: str, key: str, record: dict[str, Any], updated_at: str) -> None:
if self.store is None:
raise PermissionError("this interface is read-only over HTTP")
self.store.upsert(system, resource, key, record, updated_at)
def page(self, system: str, resource: str, cursor: str, limit: int) -> tuple[list[dict[str, Any]], str]:
qs = urllib.parse.urlencode({"cursor": cursor, "limit": limit})
url = f"{self.base}/{urllib.parse.quote(system)}/{urllib.parse.quote(resource)}?{qs}"
if not url.startswith(("http://127.0.0.1", "http://localhost")):
raise ValueError("the local client only reads the local interface")
# no proxy: the stand-in runs on this machine, and a proxy set for the environment (Databricks serverless sets
# one) refused the loopback call on 13 Nov
with _DIRECT.open(url, timeout=10) as resp: # nosec B310 - local http only, checked above
body = json.loads(resp.read())
return body["items"], body["next_cursor"]The cursor is the record's update time and its key joined by a bar. Records come back ordered by that pair, so a page never skips or repeats a record even when many share a time. The client has no authentication, no rate limit and no way to describe another interface's paths or paging style. It is the simulated dealer system's client and nothing more.
Port 4: the event stream
src/dataplatform/source_ports.py:99An append-only stream split into shards. The reader keeps its own position per shard.class EventStream(Protocol):
"""An append-only stream split into shards by a partition key; readers keep their own position per shard."""
def append(self, stream: str, records: list[tuple[str, str, Row]]) -> None:
"""(partition key, arrived_at, record) in arrival order."""
def shards(self, stream: str) -> list[str]: ...
def read(self, stream: str, shard: str, after_seq: int, limit: int) -> list[StreamRecord]: ...src/dataplatform/adapters/kinesis.py:54Reading a shard of the managed stream from after a sequence number. def read(self, stream: str, shard: str, after_seq: int, limit: int) -> list[StreamRecord]:
start: dict[str, Any] = {"ShardIteratorType": "TRIM_HORIZON"}
if after_seq > 0:
start = {"ShardIteratorType": "AFTER_SEQUENCE_NUMBER", "StartingSequenceNumber": str(after_seq)}
it = self.k.get_shard_iterator(StreamName=self._name(stream), ShardId=shard, **start)["ShardIterator"]
out: list[StreamRecord] = []
while it and len(out) < limit:
page = self.k.get_records(ShardIterator=it, Limit=min(10_000, limit - len(out)))
for r in page["Records"]:
body = json.loads(r["Data"])
out.append(StreamRecord(shard, int(r["SequenceNumber"]), body["arrived_at"], body["data"]))
if not page["Records"] or page.get("MillisBehindLatest", 0) == 0:
break
it = page.get("NextShardIterator")
return out[:limit]The local stream is a set of compressed files, one per shard. A record's shard is chosen by a hash of its key, the way managed streams spread records. The managed adapter lets the service assign shard names and sequence numbers.
The guard around source calls
src/dataplatform/resilience.py:123Three protections in one call: a circuit breaker per source, an optional cap on parallel calls, and retries with growing waits. def call(self, source: str, fn: Callable[[], T], policy: RetryPolicy | None = None) -> T:
breaker = self.breaker(source)
breaker.before(self.clock(), source)
sem = self._sem(source)
def once() -> T:
if sem is None:
return fn()
with sem:
return fn()
def noted(attempt: int, exc: BaseException, wait: float) -> None:
self.retries_seen.append((source, attempt, type(exc).__name__))
try: # the breaker counts calls that failed after their retries, not each failed try
out = call_with_retry(once, policy or RetryPolicy(), self.sleep, noted)
except BaseException:
breaker.failure(self.clock())
raise
breaker.success()
return outA circuit breaker stops calling a source that has failed several times in a row, and tries again after a pause. Only errors an adapter declares as temporary are retried. The pull pipelines send every source call through this guard when the binding supplies one.
Discovery: looking at what arrives before onboarding a feed
src/dataplatform/ingest/discovery.py:21The format of a file from its first bytes, not from its name.def sniff(head: bytes) -> str:
"""The format of a file from its first bytes: parquet, avro, orc, gzip, zip, json_lines, csv, or unknown."""
for magic, name in MAGIC:
if head.startswith(magic):
return name
text = head.lstrip()[:1]
if text in (b"{", b"["):
return "json_lines"
try:
first = head.decode("utf-8").splitlines()[0] if head else ""
except UnicodeDecodeError:
return "unknown"
return "csv" if any(d in first for d in (",", "|", "\t")) else "unknown"src/dataplatform/ingest/discovery.py:52Formats, files and rows per day, and the days with no file.def discover(files: list[tuple[str, bytes, int | None]]) -> Discovery:
"""`files` = (window, first bytes, declared rows or None) for every delivery seen. Days are the windows' first
ten characters; a day between the first and the last with no file is missing."""
d = Discovery()
for window, head, rows in files:
day = window[:10]
d.formats[sniff(head)] += 1
d.files_per_day[day] = d.files_per_day.get(day, 0) + 1
d.rows_per_day[day] = d.rows_per_day.get(day, 0) + (rows or 0)
if d.files_per_day:
start, end = date.fromisoformat(d.first), date.fromisoformat(d.last)
days = (end - start).days
d.missing_days = [str(start + timedelta(n)) for n in range(days + 1)
if str(start + timedelta(n)) not in d.files_per_day]
return dDiscovery is read-only and uses only the standard library. It answers the questions asked before a feed is set up: which formats come, how much per day, and which days are missing.
A worked example: one order line through the stand-in database
The simulated company changes one sales order line three times. The table shows what the source database stand-in holds afterwards. The first column is the change, the second the live table, the third the feed.
| Change applied | Live table | Feed rows |
|---|---|---|
| Insert line 10 with change sequence 1 | Line 10, sequence 1 | Position 1: I, sequence 1 |
| Update the promised date with sequence 3 | Line 10, sequence 3 | Position 2: U, sequence 3 |
| An older update with sequence 2 is committed late | Line 10, still sequence 3. The older change does not overwrite. | Position 3: U, sequence 2 |
The feed keeps the late change at position 3, after the newer one. That is why the mover and the pipeline keep the feed position as their checkpoint, and why silver orders history by the change sequence and not by arrival.
The settings this part reads
| Name | Meaning | Default |
|---|---|---|
WORLD_PG_DSN | Environment variable with the connection string of the stand-in source database on PostgreSQL. | dbname=tyre_world |
systems in Sources | A map from a manifest's source name to the system that holds it, passed by the binding. | None |
business_tz in Sources | The business time zone, used to turn an arrival time into a business date. | None. The binding passes it. |
retry_count, retry_interval_seconds | The retry policy for pulls. | 2 and 60 |
| Shard count of the local stream | How many shards the local stream has. | 4 |
| Page size limit of the local interface | The most records one page returns. | 200 |
The dictionary registers 43 source-scope settings, such as a server name or an authentication method. None of the adapters here reads them, and the store has no place to keep them. Settings and the merge explains that gap.
How it is tested
tests/dataplatform/test_source_ports.py is the contract test for the four source ports. Every adapter of a port must pass the same cases. The PostgreSQL adapter runs when a local server answers. The two cloud adapters run against a local stand-in for the cloud when its library is installed. Otherwise they are skipped.
tests/dataplatform/test_source_ports.py:125The source database cases: changes applied, the live rows expected, and the feed expected.DB_CASES = {
"insert": ([ch("I", row())], [row()], [("I", 1)]),
"newer change wins": (
[ch("I", row()), ch("U", row(promised="2026-11-13", seq=2), 2)],
[row(promised="2026-11-13", seq=2)],
[("I", 1), ("U", 2)],
),
"older change arriving late does not overwrite": (
[ch("I", row()), ch("U", row(promised="2026-11-14", seq=3), 2), ch("U", row(promised="2026-11-13", seq=2), 3)],
[row(promised="2026-11-14", seq=3)],
[("I", 1), ("U", 3), ("U", 2)],
),
"hard delete leaves the feed a D": ([ch("I", row()), ch("D", row(seq=2), 2)], [], [("I", 1), ("D", 2)]),
"silent loss: in the table, not in the feed": (
[ch("I", row()), ch("I", row(line=20, seq=2), 2, in_feed=False)],
[row(), row(line=20, seq=2)],
[("I", 1)],
),
}tests/dataplatform/test_discovery.py:26Discovery on four files: two formats, three days with files, two missing days.def test_discover_finds_formats_and_missing_days() -> None:
d = discover([("2026-11-01", b"PAR1", 100), ("2026-11-01", b"PAR1", 50), ("2026-11-02", b"PAR1", 120),
("2026-11-05T10", b'{"a":1}', None)])
assert dict(d.formats) == {"parquet": 3, "json_lines": 1}
assert d.files_per_day == {"2026-11-01": 2, "2026-11-02": 1, "2026-11-05": 1}
assert d.rows_per_day["2026-11-01"] == 150
assert d.missing_days == ["2026-11-03", "2026-11-04"]
assert (d.first, d.last) == ("2026-11-01", "2026-11-05")- There is no connector that reads a real database's change log. Change capture is expected to come from a managed service that writes into landing. Only its file shape is imitated.
- There is no general HTTP connector. The client reads only the local stand-in and has no authentication, rate limit or configurable paging.
- There is no connector for a message queue product other than the one managed stream, and none for file transfer protocols or other databases.
- The two cloud adapters are tested against a local stand-in for the cloud service. I could not verify a run against the real services.
- The source database port mixes reading and writing because the simulated company shares it. A production binding would need a read-only adapter.
- Discovery is not wired into any runner or command. It is a function with tests.
- The object storage landing adapter downloads every delivery to compute its checksum when it lists deliveries, like the folder adapter.