cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Ingestion › Sources and movers

Data PlatformZoom level 2

Source ports and movers: database, change feed, files, interface, stream

Ingestion never names a product. It talks to four small interfaces, called ports, that describe a source by what it is: a database, a file drop, a record interface, an event stream. An adapter is the code that puts a real system, or a stand-in for one, behind a port.

This page lists every port and every adapter, and says plainly what each adapter is. That matters here, because most of them were built to let a simulated tyre company run on a laptop, and a reader should not mistake them for connectors to production systems.

SourceDatabasefeed and full copyFileDropfiles as droppedRecordInterfacepages since a cursorEventStreamshards and sequenceChange feed moverfeed to landingDrop moverdrop to landingOpen: LandingLandingzoom in ›Open: PipelinesPipelinesthe five readerszoom in ›pushpull pagesstreama full copy is pulled from the database
The four boxes on the left are ports. The dashed boxes are movers, which only copy. Push goes through landing. Pull and stream are read directly.

The honest summary

Kind of sourceAdaptersWhat it really is
Database with a change feed and full copiesMemorySourceDb, SqliteSource, PostgresSourceA stand-in. All three read tables that the simulated company itself writes in a layout of our own: the live table, a table of changes and a table of copies. None connects to a real ERP or reads a database's own change log.
Change feed moverChangeFeedMover, DmsMoverA stand-in for a change capture service. The second one imitates the file layout such a managed service writes, with Parquet files and its column names. No managed service is called.
File dropMemoryFileDrop, LocalFileDrop, S3FileDropThe first two are local. The third makes real calls to a cloud object storage service through the vendor's library.
Drop moverDropMoverReal but simple code: it copies new files from a drop to landing. It stands in for a file transfer service.
Record interfaceMemoryRecords, LocalRecordStore with serve, HttpRecordClientA stand-in. A small HTTP server on the same machine serves the simulated dealer system. The client refuses any address that is not the local machine.
Event streamMemoryStream, LocalStream, KinesisStreamThe first two are local. The third makes real calls to a managed streaming service through the vendor's library.
LandingFileLanding, S3LandingA folder, which is also how a storage volume of the engine is reached, and cloud object storage.

For the two cloud adapters that make real calls, the tests and the recorded run in the cloud realm's README use a local stand-in for the cloud service. I found no record of a run against the real services and could not verify one.

Why the ports have write methods

src/dataplatform/source_ports.py:1The module's own opening. It calls them mock-source ports.
"""The four mock-source ports (goal G4, data-platform-catalog dpc/ports.yaml: source systems).

The world writes into source systems only through these shapes, so the same simulated company feeds any cloud:
a database on the laptop today, a managed one later; a local folder today, object storage later; a local log of
shards today, a managed stream later. Each is a separate adapter (M2) and each passes the same contract tests (M5).

Names here say what a system is (a database, a file drop, an interface, a stream), never which product it is (R1).
"""

A connector to a real system would only read. These ports also have methods that write: apply, publish_copy, put, upsert and append. They are there because the simulated company uses the same ports to act as the source systems. Ingestion calls only the reading methods.

Port 1: the source database

src/dataplatform/source_ports.py:56A database with a change feed and a nightly full copy.
class SourceDatabase(Protocol):
    """A database with a change feed (a change-capture source) and a nightly full copy for master tables."""

    def ensure(self, system: str, tables: list[TableSpec], reset: bool = False) -> None: ...

    def apply(self, system: str, table: TableSpec, changes: list[SourceChange]) -> None:
        """Apply changes to the live table (newest change sequence wins) and append the in-feed ones to the feed."""

    def publish_copy(self, system: str, table: TableSpec, extracted_at: str, rows: list[Row]) -> None: ...

    def rows(self, system: str, table: TableSpec) -> list[Row]:
        """The live table now, sorted by key."""

    def feed(self, system: str, table: TableSpec, after_pos: int = 0) -> list[Row]:
        """Feed rows in commit order after a feed position, each with feed_pos, op and committed_at beside the row's
        own columns. Readers keep a feed position, not a change sequence: an old change can commit late."""

    def copy(self, system: str, table: TableSpec, extracted_at: str) -> list[Row]: ...

    def copies(self, system: str, table: TableSpec) -> list[str]:
        """The extraction times of the full copies published so far, oldest first (a puller takes the newest)."""
MethodUsed byWhat it returns
feedThe change feed moverThe change rows after a feed position, in commit order, each with its position, operation and commit time.
copies, copyThe full copy pipelineThe extraction times of the copies published so far, and the rows of one copy.
rowsThe platform script, when it compares the live source with silverThe live table as it stands.
ensure, apply, publish_copyThe simulated companyNothing. They create tables and write changes and copies.

The docstring of feed states a rule worth keeping: a reader keeps a feed position, not a change sequence, because an old change can be committed late.

src/dataplatform/adapters/sql_source.py:151The feed on a SQL database: a plain query on a changes table that the simulated company fills.
    def feed(self, system: str, table: TableSpec, after_pos: int = 0) -> list[Row]:
        return self._select(
            f"SELECT * FROM {self.d.qualify(system, table.name + '__changes')} "  # nosec B608
            f"WHERE feed_pos > {self.d.placeholder} ORDER BY feed_pos",
            (after_pos,),
        )

This is the clearest sign of what the adapter is. The feed is a table named after the source table with __changes added, and the full copies are a table with __copy added. A real database does not have these. A real connector would read the database's own change log, or a managed change capture service would, and write files into landing.

The movers: from a source to landing

src/dataplatform/adapters/local_movers.py:23The change feed mover. Its checkpoint is the highest feed position among the deliveries it has already landed.
class ChangeFeedMover:
    def __init__(self, db: SourceDatabase, landing: Landing):
        self.db, self.landing = db, landing

    def checkpoint(self, source: str, dataset: str) -> int:
        landed = [self.landing.manifest(d) for d in self.landing.deliveries(source, dataset)]
        return max((int(m.get("feed_pos_to", 0)) for m in landed), default=0)

    def move(self, system: str, source: str, table: TableSpec, window: str) -> Delivery | None:
        rows = self.db.feed(system, table, self.checkpoint(source, table.name))
        if not rows:
            return None  # nothing changed: a change-capture service writes no empty file
        first, last = rows[0]["feed_pos"], rows[-1]["feed_pos"]
        name = f"{table.name}_{window}_{first:010d}-{last:010d}.jsonl"
        try:
            return self.landing.put(
                source,
                table.name,
                window,
                name,
                to_bytes(rows),
                len(rows),
                {"feed_pos_from": first, "feed_pos_to": last},
            )
        except DeliveryExists:
            return None

A mover copies and never changes data. This one asks the database for feed rows after its checkpoint and lands them as one JSON lines delivery, named with the first and last feed position. It writes no empty file. If landing refuses the write because the delivery exists, the mover treats it as already done.

src/dataplatform/adapters/local_movers.py:51The drop mover: every file not yet in landing is copied byte for byte, under the business date of its arrival.
class DropMover:
    def __init__(self, drop: FileDrop, landing: Landing, business_tz: str):
        self.drop, self.landing, self.tz = drop, landing, ZoneInfo(business_tz)

    def move(self, system: str, drop_dataset: str, source: str, dataset: str) -> list[Delivery]:
        landed = {d.name for d in self.landing.deliveries(source, dataset)}
        out = []
        for f in self.drop.list(system, drop_dataset):
            if f.name in landed:
                continue
            data = self.drop.get(system, drop_dataset, f.name)
            window = datetime.fromisoformat(f.arrived_at).astimezone(self.tz).date().isoformat()
            rows = max(0, data.count(b"\n") - 1)  # a header line, then one line per row
            try:
                out.append(self.landing.put(source, dataset, window, f.name, data, rows, {"arrived_at": f.arrived_at}))
            except DeliveryExists:
                continue
        return out
src/dataplatform/adapters/local_movers.py:111The settings a dataset needs when its deliveries have the managed service's shape.
DMS_SETTINGS = {"file_format_data": "parquet", "cdc_op_column": "Op", "cdc_position_column": "",
                "cdc_commit_column": "ar_commit_ts"}

Port 2: the file drop

src/dataplatform/source_ports.py:79A folder that partners and the laboratory drop files into.
class FileDrop(Protocol):
    """A folder partners and lab systems drop files into. Arrival time is part of the file (object storage: LastModified)."""

    def put(self, system: str, dataset: str, name: str, data: bytes, arrived_at: str) -> DropFile: ...

    def list(self, system: str, dataset: str) -> list[DropFile]:
        """Files in arrival order."""

    def get(self, system: str, dataset: str, name: str) -> bytes: ...

The local adapter keeps a file's arrival time as its modified time. The object storage adapter keeps it as a piece of metadata on the object, because the storage service stamps its own upload time. Both refuse a name that contains a path.

Port 3: the record interface

src/dataplatform/source_ports.py:90An interface that returns records changed after a cursor, in pages.
class RecordInterface(Protocol):
    """A request-response interface over a set of records, read in pages since a cursor (the dealer system)."""

    def upsert(self, system: str, resource: str, key: str, record: Row, updated_at: str) -> None: ...

    def page(self, system: str, resource: str, cursor: str, limit: int) -> tuple[list[Row], str]:
        """Records changed after `cursor` ('' = from the start), oldest first; returns them and the next cursor."""
src/dataplatform/adapters/http_api.py:106The HTTP client. Read the check on the address in the page method.
class HttpRecordClient:
    """Reads the interface over HTTP. Writes go to the store: a real dealer system is not written by its readers."""

    def __init__(self, base_url: str, store: LocalRecordStore | None = None):
        self.base = base_url.rstrip("/")
        self.store = store

    def upsert(self, system: str, resource: str, key: str, record: dict[str, Any], updated_at: str) -> None:
        if self.store is None:
            raise PermissionError("this interface is read-only over HTTP")
        self.store.upsert(system, resource, key, record, updated_at)

    def page(self, system: str, resource: str, cursor: str, limit: int) -> tuple[list[dict[str, Any]], str]:
        qs = urllib.parse.urlencode({"cursor": cursor, "limit": limit})
        url = f"{self.base}/{urllib.parse.quote(system)}/{urllib.parse.quote(resource)}?{qs}"
        if not url.startswith(("http://127.0.0.1", "http://localhost")):
            raise ValueError("the local client only reads the local interface")
        # no proxy: the stand-in runs on this machine, and a proxy set for the environment (Databricks serverless sets
        # one) refused the loopback call on 13 Nov
        with _DIRECT.open(url, timeout=10) as resp:  # nosec B310 - local http only, checked above
            body = json.loads(resp.read())
        return body["items"], body["next_cursor"]

The cursor is the record's update time and its key joined by a bar. Records come back ordered by that pair, so a page never skips or repeats a record even when many share a time. The client has no authentication, no rate limit and no way to describe another interface's paths or paging style. It is the simulated dealer system's client and nothing more.

Port 4: the event stream

src/dataplatform/source_ports.py:99An append-only stream split into shards. The reader keeps its own position per shard.
class EventStream(Protocol):
    """An append-only stream split into shards by a partition key; readers keep their own position per shard."""

    def append(self, stream: str, records: list[tuple[str, str, Row]]) -> None:
        """(partition key, arrived_at, record) in arrival order."""

    def shards(self, stream: str) -> list[str]: ...

    def read(self, stream: str, shard: str, after_seq: int, limit: int) -> list[StreamRecord]: ...
src/dataplatform/adapters/kinesis.py:54Reading a shard of the managed stream from after a sequence number.
    def read(self, stream: str, shard: str, after_seq: int, limit: int) -> list[StreamRecord]:
        start: dict[str, Any] = {"ShardIteratorType": "TRIM_HORIZON"}
        if after_seq > 0:
            start = {"ShardIteratorType": "AFTER_SEQUENCE_NUMBER", "StartingSequenceNumber": str(after_seq)}
        it = self.k.get_shard_iterator(StreamName=self._name(stream), ShardId=shard, **start)["ShardIterator"]
        out: list[StreamRecord] = []
        while it and len(out) < limit:
            page = self.k.get_records(ShardIterator=it, Limit=min(10_000, limit - len(out)))
            for r in page["Records"]:
                body = json.loads(r["Data"])
                out.append(StreamRecord(shard, int(r["SequenceNumber"]), body["arrived_at"], body["data"]))
            if not page["Records"] or page.get("MillisBehindLatest", 0) == 0:
                break
            it = page.get("NextShardIterator")
        return out[:limit]

The local stream is a set of compressed files, one per shard. A record's shard is chosen by a hash of its key, the way managed streams spread records. The managed adapter lets the service assign shard names and sequence numbers.

The guard around source calls

src/dataplatform/resilience.py:123Three protections in one call: a circuit breaker per source, an optional cap on parallel calls, and retries with growing waits.
    def call(self, source: str, fn: Callable[[], T], policy: RetryPolicy | None = None) -> T:
        breaker = self.breaker(source)
        breaker.before(self.clock(), source)
        sem = self._sem(source)

        def once() -> T:
            if sem is None:
                return fn()
            with sem:
                return fn()

        def noted(attempt: int, exc: BaseException, wait: float) -> None:
            self.retries_seen.append((source, attempt, type(exc).__name__))

        try:  # the breaker counts calls that failed after their retries, not each failed try
            out = call_with_retry(once, policy or RetryPolicy(), self.sleep, noted)
        except BaseException:
            breaker.failure(self.clock())
            raise
        breaker.success()
        return out

A circuit breaker stops calling a source that has failed several times in a row, and tries again after a pause. Only errors an adapter declares as temporary are retried. The pull pipelines send every source call through this guard when the binding supplies one.

Discovery: looking at what arrives before onboarding a feed

src/dataplatform/ingest/discovery.py:21The format of a file from its first bytes, not from its name.
def sniff(head: bytes) -> str:
    """The format of a file from its first bytes: parquet, avro, orc, gzip, zip, json_lines, csv, or unknown."""
    for magic, name in MAGIC:
        if head.startswith(magic):
            return name
    text = head.lstrip()[:1]
    if text in (b"{", b"["):
        return "json_lines"
    try:
        first = head.decode("utf-8").splitlines()[0] if head else ""
    except UnicodeDecodeError:
        return "unknown"
    return "csv" if any(d in first for d in (",", "|", "\t")) else "unknown"
src/dataplatform/ingest/discovery.py:52Formats, files and rows per day, and the days with no file.
def discover(files: list[tuple[str, bytes, int | None]]) -> Discovery:
    """`files` = (window, first bytes, declared rows or None) for every delivery seen. Days are the windows' first
    ten characters; a day between the first and the last with no file is missing."""
    d = Discovery()
    for window, head, rows in files:
        day = window[:10]
        d.formats[sniff(head)] += 1
        d.files_per_day[day] = d.files_per_day.get(day, 0) + 1
        d.rows_per_day[day] = d.rows_per_day.get(day, 0) + (rows or 0)
    if d.files_per_day:
        start, end = date.fromisoformat(d.first), date.fromisoformat(d.last)
        days = (end - start).days
        d.missing_days = [str(start + timedelta(n)) for n in range(days + 1)
                          if str(start + timedelta(n)) not in d.files_per_day]
    return d

Discovery is read-only and uses only the standard library. It answers the questions asked before a feed is set up: which formats come, how much per day, and which days are missing.

A worked example: one order line through the stand-in database

The simulated company changes one sales order line three times. The table shows what the source database stand-in holds afterwards. The first column is the change, the second the live table, the third the feed.

Change appliedLive tableFeed rows
Insert line 10 with change sequence 1Line 10, sequence 1Position 1: I, sequence 1
Update the promised date with sequence 3Line 10, sequence 3Position 2: U, sequence 3
An older update with sequence 2 is committed lateLine 10, still sequence 3. The older change does not overwrite.Position 3: U, sequence 2

The feed keeps the late change at position 3, after the newer one. That is why the mover and the pipeline keep the feed position as their checkpoint, and why silver orders history by the change sequence and not by arrival.

The settings this part reads

NameMeaningDefault
WORLD_PG_DSNEnvironment variable with the connection string of the stand-in source database on PostgreSQL.dbname=tyre_world
systems in SourcesA map from a manifest's source name to the system that holds it, passed by the binding.None
business_tz in SourcesThe business time zone, used to turn an arrival time into a business date.None. The binding passes it.
retry_count, retry_interval_secondsThe retry policy for pulls.2 and 60
Shard count of the local streamHow many shards the local stream has.4
Page size limit of the local interfaceThe most records one page returns.200

The dictionary registers 43 source-scope settings, such as a server name or an authentication method. None of the adapters here reads them, and the store has no place to keep them. Settings and the merge explains that gap.

How it is tested

tests/dataplatform/test_source_ports.py is the contract test for the four source ports. Every adapter of a port must pass the same cases. The PostgreSQL adapter runs when a local server answers. The two cloud adapters run against a local stand-in for the cloud when its library is installed. Otherwise they are skipped.

tests/dataplatform/test_source_ports.py:125The source database cases: changes applied, the live rows expected, and the feed expected.
DB_CASES = {
    "insert": ([ch("I", row())], [row()], [("I", 1)]),
    "newer change wins": (
        [ch("I", row()), ch("U", row(promised="2026-11-13", seq=2), 2)],
        [row(promised="2026-11-13", seq=2)],
        [("I", 1), ("U", 2)],
    ),
    "older change arriving late does not overwrite": (
        [ch("I", row()), ch("U", row(promised="2026-11-14", seq=3), 2), ch("U", row(promised="2026-11-13", seq=2), 3)],
        [row(promised="2026-11-14", seq=3)],
        [("I", 1), ("U", 3), ("U", 2)],
    ),
    "hard delete leaves the feed a D": ([ch("I", row()), ch("D", row(seq=2), 2)], [], [("I", 1), ("D", 2)]),
    "silent loss: in the table, not in the feed": (
        [ch("I", row()), ch("I", row(line=20, seq=2), 2, in_feed=False)],
        [row(), row(line=20, seq=2)],
        [("I", 1)],
    ),
}
tests/dataplatform/test_discovery.py:26Discovery on four files: two formats, three days with files, two missing days.
def test_discover_finds_formats_and_missing_days() -> None:
    d = discover([("2026-11-01", b"PAR1", 100), ("2026-11-01", b"PAR1", 50), ("2026-11-02", b"PAR1", 120),
                  ("2026-11-05T10", b'{"a":1}', None)])
    assert dict(d.formats) == {"parquet": 3, "json_lines": 1}
    assert d.files_per_day == {"2026-11-01": 2, "2026-11-02": 1, "2026-11-05": 1}
    assert d.rows_per_day["2026-11-01"] == 150
    assert d.missing_days == ["2026-11-03", "2026-11-04"]
    assert (d.first, d.last) == ("2026-11-01", "2026-11-05")
Not built yet, or known gaps
  • There is no connector that reads a real database's change log. Change capture is expected to come from a managed service that writes into landing. Only its file shape is imitated.
  • There is no general HTTP connector. The client reads only the local stand-in and has no authentication, rate limit or configurable paging.
  • There is no connector for a message queue product other than the one managed stream, and none for file transfer protocols or other databases.
  • The two cloud adapters are tested against a local stand-in for the cloud service. I could not verify a run against the real services.
  • The source database port mixes reading and writing because the simulated company shares it. A production binding would need a read-only adapter.
  • Discovery is not wired into any runner or command. It is a function with tests.
  • The object storage landing adapter downloads every delivery to compute its checksum when it lists deliveries, like the folder adapter.