Data PlatformZoom level 3
Who calls what: ingest
Every class, function and method in the ingest package, with who calls it. Found by reading the code. A call written as object.method() is matched by the method's name, so a same-named method elsewhere can appear: those rows say by name.
92 definitions in 6 modules. Jump to a module: daybyday.py · discovery.py · pipelines.py · ports.py · runner.py · steps.py
ingest/daybyday.py
The world and the platform, day by day: a world day is written into the mock sources, the push movers fill landing, one ingestion run takes everything new to bronze, and bronze is compared with the oracle's bronze totals for that day. Ingestion
ingest/discovery.py
Source discovery (B3): before a feed is onboarded, look at what actually arrives. For the client's own drops (format unknown, history completeness unknown) this answers: which formats, which windows, how many files and rows a day, and which Ingestion
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
snifffunction, line 21 | The format of a file from its first bytes: parquet, avro, orc, gzip, zip, json_lines, csv, or unknown. | discover · ingest/discovery.py:58 | test_discovery.py |
Discoveryclass, line 37 | discover · ingest/discovery.py:55 | no test names it | |
Discovery.firstmethod, line 44 | discover · ingest/discovery.py:62 (value)create · realms/databricks/engine/tasks/probe.py:49 (by name)module level · realms/databricks/engine/tasks/probe.py:57 (by name)module level · realms/databricks/pipelines/silver.py:16 (by name)run_checks · realms/databricks/tasks/checks.py:39 (by name)metric_view · realms/databricks/tasks/checks.py:102 (by name)and 15 more | test_semantic_governance.py, silver_helpers.py and 42 more | |
Discovery.lastmethod, line 48 | discover · ingest/discovery.py:62 (value) | no test names it | |
discoverfunction, line 52 | `files` = (window, first bytes, declared rows or None) for every delivery seen. Days are the windows' first ten characters; a day between the first and the last with no file is missing. | none found in the code | test_discovery.py, test_cli.py and 3 more |
ingest/pipelines.py
One ingestion pipeline per source type (ADR-002, D11). Each picks its batches, hands every one to the lander steps (steps.enter) and reports bookmarks, a watermark and what it noticed. They differ only in where batches come from: Ingestion
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
Planclass, line 40 | What a run does for one dataset: the run type and, for reload and backfill, the windows. | run_ingest · scripts/run_hop.py:61run_days · ingest/daybyday.py:129module level · ingest/pipelines.py:386 (value) | no test names it |
Sourcesclass, line 49 | The source side as the binding hands it over: landing for push, the source systems for pull and stream. `systems` maps a manifest source_name to the system that holds it (erp -> the ERP database). | run_ingest · scripts/run_hop.py:56main · scripts/run_platform.py:441module level · ingest/pipelines.py:386 (value) | test_ingest_pipelines.py, test_ingest_world.py and 3 more |
_readfunction, line 62 | One call to a source through the run's guard: retried on temporary errors per the dataset's retry settings, refused while the source's circuit is open, limited by the source's connection cap. | pull_copy · ingest/pipelines.py:255pull_pages · ingest/pipelines.py:288 | no test names it |
Resultclass, line 71 | push · ingest/pipelines.py:131pull_copy · ingest/pipelines.py:247pull_pages · ingest/pipelines.py:276stream_events · ingest/pipelines.py:323module level · ingest/pipelines.py:386 (value) | test_obs_steps.py | |
Result.noticemethod, line 78 | _feed_notices · ingest/pipelines.py:210 (by name)_stream_notices · ingest/pipelines.py:382 (by name) | no test names it | |
_window_offunction, line 84 | pull_copy · ingest/pipelines.py:258 | no test names it | |
_table_specfunction, line 88 | pull_copy · ingest/pipelines.py:248 | no test names it | |
_pickfunction, line 99 | (window, name) of batches recorded at all, and of batches loaded. | push · ingest/pipelines.py:133 | no test names it |
_replay_namesfunction, line 107 | The recorded batches a reload (all) or backfill (not loaded) re-enters, latest record per batch. | pull_copy · ingest/pipelines.py:260pull_pages · ingest/pipelines.py:286stream_events · ingest/pipelines.py:326 | no test names it |
_max_markfunction, line 117 | The larger of a stored mark and new values: whole numbers compare as numbers, everything else as ISO text. | push · ingest/pipelines.py:155_watermark · ingest/pipelines.py:230 | no test names it |
pushfunction, line 130 | module level · ingest/pipelines.py:387 (value) | no test names it | |
feed_columnsfunction, line 171 | Delivery column -> standard feed column. A change-capture service names its columns its own way (ADR-031): cdc_op_column (default op), cdc_position_column (default feed_pos; empty when the service has none) and cdc_commit_column (default co | push · ingest/pipelines.py:147 | test_ingest_formats.py |
_op_valuefunction, line 184 | Operation letters upper-cased (I, U, D); a full-load row with no operation stays None (history reads it as U). | push · ingest/pipelines.py:148 | no test names it |
_feed_bookmarksfunction, line 191 | One bookmark per feed (a dealer), holding its newest data time as an instant in the business time zone; it only moves forward. Freshness compares feeds with their siblings; gold reads them for completeness (ADR-018). | push · ingest/pipelines.py:167 | no test names it |
_feed_noticesfunction, line 207 | Deletes arrive as op D (DEL-01). A change whose sequence is older than one already seen for its key in the delivery arrived out of order (SCD-02); silver orders history by the sequence (W1c). | push · ingest/pipelines.py:149 | no test names it |
_watermarkfunction, line 225 | The dataset's watermark column, moved forward to the largest value loaded (never back). | push · ingest/pipelines.py:161pull_pages · ingest/pipelines.py:302stream_events · ingest/pipelines.py:357 | no test names it |
_guard_keysfunction, line 234 | What the mass-delete guard compares a full copy with: silver's current keys (DPC-0054), or the last accepted copy in bronze while silver has never been written (the first days, or a binding without silver). | pull_copy · ingest/pipelines.py:263 | no test names it |
pull_copyfunction, line 245 | The newest full copy only: a newer copy supersedes an older one (and a held one). | module level · ingest/pipelines.py:389 (value) | no test names it |
pull_pagesfunction, line 274 | Pages since the cursor, one batch per page; the batch name holds the cursor range so a reload can re-read it. | module level · ingest/pipelines.py:390 (value) | no test names it |
_cursor_offunction, line 307 | A record's position in the interface's order: '<updated_at>|<key>' (adapters/http_api.py). | pull_pages · ingest/pipelines.py:289 | no test names it |
_resume_cursorfunction, line 312 | The cursor after the last page loaded, if a failed run loaded pages its bookmark does not show. | pull_pages · ingest/pipelines.py:292 | no test names it |
stream_eventsfunction, line 319 | Trigger-once: each shard from its checkpoint to the end, one batch per shard, then stop (BP-044). Bronze keeps every reading as delivered (RAW-01); replays and late readings are counted here and dropped or placed by event time in silver (DU | module level · ingest/pipelines.py:391 (value) | test_ingest_pipelines.py |
_stream_rowsfunction, line 361 | stream_events · ingest/pipelines.py:332 | no test names it | |
_stream_noticesfunction, line 365 | DUP-04: a reading id read before in this run is a replay. LATE-03: a reading whose event time is more than the allowance behind the newest already read on its shard arrived out of order. | stream_events · ingest/pipelines.py:351 | no test names it |
ingest/ports.py
The ports ingestion talks to (data-platform-catalog dpc/ports.yaml: landing_store, table_writer, tokenizer). Ingestion
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
check_batch_keyfunction, line 19 | DeltaBronze.write · adapters/delta_bronze.py:60DeltaBronze.write_many · adapters/delta_bronze.py:80DeltaBronze.remove · adapters/delta_bronze.py:94MemoryBronze.write · adapters/memory_bronze.py:19MemoryBronze.remove · adapters/memory_bronze.py:43SparkBronzeWriter.write · adapters/spark_delta.py:152and 5 more | no test names it | |
DeliveryExistsclass, line 25 | Landing refuses a second write of the same delivery: a delivery is immutable once its manifest exists (landing_store invariant, DUP-02). The mover treats it as already landed. | AzureLanding.put · adapters/azure_blob.py:72FileLanding.put · adapters/files.py:44S3Landing.put · adapters/s3.py:59 | test_azure_adapters.py, test_cloud_adapters.py and 1 more |
Landingclass, line 30 | landing_store for push deliveries: the bytes exactly as delivered, with a manifest beside them. | none found in the code | no test names it |
Landing.putmethod, line 33 | Store a delivery and then its manifest. Raises DeliveryExists if (window, name) was landed before. | ChangeFeedMover.move · adapters/local_movers.py:38 (by name)DropMover.move · adapters/local_movers.py:65 (by name)DmsMover.move · adapters/local_movers.py:105 (by name)World.write · world/engine.py:133 (by name) | no test names it |
Landing.deliveriesmethod, line 45 | Deliveries that have a manifest, in window then name order. | step_prepare · realms/databricks/tasks/main.py:81 (by name)ChangeFeedMover.checkpoint · adapters/local_movers.py:28 (by name)DropMover.move · adapters/local_movers.py:56 (by name)push · ingest/pipelines.py:135 (by name)land · lander.py:31 (by name) | gold_helpers.py, test_cloud_adapters.py and 5 more |
Landing.read_bytesmethod, line 48 | LocalFileDrop.list · adapters/file_drop.py:38 (by name)LocalFileDrop.get · adapters/file_drop.py:41 (by name)FileLanding.deliveries · adapters/files.py:56 (by name)FileLanding.read_bytes · adapters/files.py:64 (by name)push · ingest/pipelines.py:144 (by name) | test_ingest_formats.py, test_ingest_ports.py and 3 more | |
Landing.manifestmethod, line 50 | The manifest the mover wrote, including any extra fields (feed positions, arrival time). | ChangeFeedMover.checkpoint · adapters/local_movers.py:28 (by name) | test_bundle.py, gold_helpers.py and 34 more |
BronzeWriterclass, line 54 | table_writer for bronze: append-only as delivered, idempotent by batch key. | none found in the code | no test names it |
BronzeWriter.writemethod, line 57 | Write one batch in one transaction, replacing whatever the table holds for this batch key: the same batch written twice leaves the table unchanged (DUP-01, DPC-0024) and a failed write leaves no trace (ORCH-03). `replace` is the lander sayi | none found in the code | gold_helpers.py, test_cloud_adapters.py and 31 more |
BronzeWriter.write_manymethod, line 63 | Write several new batches of one table in one transaction (one commit instead of one per batch); all or nothing. The lander logs them as loaded only after this returns. Returns the rows written. | flush · ingest/steps.py:365 (by name) | no test names it |
BronzeWriter.fingerprintmethod, line 67 | {"row_count": n, <column>: total, ...} for the table, or for the rows where every column equals its value. A table never written has row_count 0. | _one · ingest/runner.py:149 (value)FixtureRunner.run_day · obs/fixture.py:117 (value)FixtureRunner._rerun · obs/fixture.py:134 (value)run_days · ingest/daybyday.py:133 (by name) | gold_helpers.py, silver_helpers.py and 14 more |
BronzeWriter.keysmethod, line 73 | The distinct values of `columns` in the rows matching `where` (the mass-delete guard's comparison). | none found in the code | no test names it |
CurrentKeysclass, line 77 | Silver's current keys, for the mass-delete guard (DPC-0054: compare a full copy with silver's current keys). | none found in the code | test_ingest_pipelines.py |
CurrentKeys.current_keysmethod, line 80 | The keys of the current, not deleted silver rows, or None when silver was never written. | _guard_keys · ingest/pipelines.py:239 (by name) | test_ingest_pipelines.py, test_silver_ports.py |
Tokenizerclass, line 84 | tokenizer: a personal value in, a token out, before anything reaches bronze (DPC-0070). | none found in the code | no test names it |
Tokenizer.tokenmethod, line 87 | main.on_day · scripts/run_platform.py:582 (value)instance_token · adapters/lakebase.py:65 (value)main.live_rows · scripts/run_platform.py:489 (by name)mass_delete_problem.stored · ingest/steps.py:256 (by name)protect · ingest/steps.py:291 (by name) | test_bundle.py, test_semantic_governance.py and 18 more |
ingest/runner.py
One ingestion run over a domain, every step recorded in the metadata store (ADR-010): Ingestion
ingest/steps.py
The lander steps: one library every way into bronze calls (ING-01, ADR-002, DPC-0037). A pushed file, a pulled copy, a page from an interface and a chunk of a stream all pass the same steps in the same order: Ingestion
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
CastErrorclass, line 50 | parse · ingest/steps.py:151_parquet_rows · ingest/steps.py:162_cast_int · ingest/steps.py:194_cast_bool · ingest/steps.py:205cast_rows · ingest/steps.py:230 (value)enter · ingest/steps.py:329 (value) | no test names it | |
content_keyfunction, line 54 | The batch key: the same bytes give the same key whatever the file is called or when it came (DUP-01). | enter · ingest/steps.py:317 | test_ingest_steps.py |
Specclass, line 60 | One dataset as the lander needs it, from the metadata store's merged view (never from files). | none found in the code | no test names it |
Spec.tablemethod, line 75 | none found in the code | test_lakeformation.py, test_semantic_governance.py and 84 more | |
Spec.bronze_columnsmethod, line 79 | enter · ingest/steps.py:355 (value)flush · ingest/steps.py:365 (value) | no test names it | |
Spec.from_viewmethod, line 83 | _one · ingest/runner.py:148 (by name) | test_ingest_pipelines.py | |
Batchclass, line 101 | One batch on its way in: a delivery's bytes (push) or rows read from a source (pull, stream). | push · ingest/pipelines.py:151pull_copy · ingest/pipelines.py:267pull_pages · ingest/pipelines.py:290stream_events · ingest/pipelines.py:332 | test_housekeeping.py, test_ingest_steps.py and 2 more |
Outcomeclass, line 113 | enter · ingest/steps.py:323module level · ingest/steps.py:137 (value) | test_obs_steps.py | |
Ctxclass, line 125 | _one_leased · ingest/runner.py:179module level · ingest/pipelines.py:386 (value) | no test names it | |
parsefunction, line 141 | A delivery's bytes as rows, by the dataset's file_format_data (ADR-031): json_lines (default), csv/tsv/pipe with a header, or parquet (change-capture and buffered-stream deliveries; types and nulls kept). | push · ingest/pipelines.py:148enter · ingest/steps.py:328 | test_ingest_formats.py, test_ingest_steps.py and 19 more |
_parquet_rowsfunction, line 156 | Rows of a columnar file; needs pyarrow (the `local` extra). Timestamps come back as datetimes, so the cast step sees the same types as from text. | parse · ingest/steps.py:146 | no test names it |
check_columnsfunction, line 166 | (reason the batch stops, or ''; new columns seen). An empty batch has no columns to judge: the volume step holds it. | enter · ingest/steps.py:331 | test_ingest_steps.py, test_slice1_local.py |
_cast_tsfunction, line 187 | module level · ingest/steps.py:212 (value) | no test names it | |
_cast_intfunction, line 192 | module level · ingest/steps.py:210 (value) | no test names it | |
_cast_boolfunction, line 200 | module level · ingest/steps.py:214 (value) | no test names it | |
cast_rowsfunction, line 218 | Cast every value to its column's type, in place. Returns why the batch stops (naming the column), or ''. An empty text value is null for every type but string (CSV has no null). | enter · ingest/steps.py:337 | test_ingest_steps.py |
volume_problemfunction, line 235 | enter · ingest/steps.py:340 | test_ingest_steps.py | |
mass_delete_problemfunction, line 244 | A full copy that would delete more than the declared share of the last accepted copy's keys (DEL-04). `previous` holds the keys as stored (a restricted key column holds its token), so the copy's keys are compared as stored too: with a `toke | enter · ingest/steps.py:343 | test_ingest_steps.py |
mass_delete_problem.storedfunction, line 254 | mass_delete_problem · ingest/steps.py:259 | test_semantic_governance.py, test_azure_adapters.py and 6 more | |
feed_marksfunction, line 269 | A file that feeds one member (a dealer's monthly file): every row shares one value of a text column of the primary key, and the key holds a date or time column. Returns {that value: the newest date or time in the file}, else {}. One bookmar | enter · ingest/steps.py:352 | no test names it |
protectfunction, line 284 | Tokenise restricted columns in place; returns how many values were replaced. | enter · ingest/steps.py:346 | test_ingest_steps.py |
stampfunction, line 296 | enter · ingest/steps.py:348 | no test names it | |
enterfunction, line 315 | Run the steps for one batch and record the outcome. Never raises for a bad batch: it is held and logged. | push · ingest/pipelines.py:151pull_copy · ingest/pipelines.py:267pull_pages · ingest/pipelines.py:290stream_events · ingest/pipelines.py:332 | test_ingest_steps.py |
flushfunction, line 359 | Write the run's pending new batches in one commit, then log each as loaded: the file log never says loaded before bronze holds the rows. If the write fails nothing is logged and the next run takes the batches again. | _one_leased · ingest/runner.py:181 | no test names it |
_logfunction, line 372 | enter · ingest/steps.py:323 | no test names it | |
_recordfunction, line 379 | flush · ingest/steps.py:367_log · ingest/steps.py:376 | no test names it |