cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Code map › Who calls what › ingest

Data PlatformZoom level 3

Who calls what: ingest

Every class, function and method in the ingest package, with who calls it. Found by reading the code. A call written as object.method() is matched by the method's name, so a same-named method elsewhere can appear: those rows say by name.

92 definitions in 6 modules. Jump to a module: daybyday.py · discovery.py · pipelines.py · ports.py · runner.py · steps.py

ingest/daybyday.py

The world and the platform, day by day: a world day is written into the mock sources, the push movers fill landing, one ingestion run takes everything new to bronze, and bronze is compared with the oracle's bronze totals for that day. Ingestion

DefinitionWhat it does (its own first sentence)Called byNamed in tests
DayWorld
class, line 38
none found in the codeno test names it
DayWorld.prepare
method, line 39
run_days · ingest/daybyday.py:119 (by name)test_world_tyre.py
DayWorld.step
method, line 41
MemoryMetaStore.log_step · adapters/memory_meta.py:198 (value)
PostgresMetaStore.log_step · adapters/postgres_meta.py:158 (value)
SparkRunLog.write · adapters/spark_delta.py:117 (value)
steps_table · obs/ops.py:188 (value)
step_durations_table · obs/ops.py:230 (value)
run_signals · obs/ops.py:343 (value)
and 4 more
no test names it
DayWorld.finish
method, line 43
none found in the codetest_gold_engine.py, test_meta_store.py and 2 more
Verdict
class, line 47
judge.v · ingest/daybyday.py:75
judge.v · silver/daybyday.py:114
ClinicalGoldJudge._v · world/domains/clinical_gold.py:166
TyreGoldJudge._v · world/domains/tyre_gold.py:175
test_silver_runner.py
Day
class, line 57
run_days · ingest/daybyday.py:136
run_days · ingest/daybyday.py:113 (value)
no test names it
Day.bronze_ok
method, line 66
write_ops_page · scripts/run_platform.py:133 (value)test_ingest_world.py
judge
function, line 70
run_days · ingest/daybyday.py:136test_end_to_end.py, test_explain.py and 3 more
judge.v
function, line 74
judge · ingest/daybyday.py:79no test names it
run_days
function, line 102
`run_ingest(run_id, window)` runs the day's ingestion somewhere else (its own process or job) and returns its report; without it the ingestion runs here.main · scripts/run_platform.py:615test_explainer.py, test_ingest_world.py and 3 more

ingest/discovery.py

Source discovery (B3): before a feed is onboarded, look at what actually arrives. For the client's own drops (format unknown, history completeness unknown) this answers: which formats, which windows, how many files and rows a day, and which Ingestion

DefinitionWhat it does (its own first sentence)Called byNamed in tests
sniff
function, line 21
The format of a file from its first bytes: parquet, avro, orc, gzip, zip, json_lines, csv, or unknown.discover · ingest/discovery.py:58test_discovery.py
Discovery
class, line 37
discover · ingest/discovery.py:55no test names it
Discovery.first
method, line 44
discover · ingest/discovery.py:62 (value)
create · realms/databricks/engine/tasks/probe.py:49 (by name)
module level · realms/databricks/engine/tasks/probe.py:57 (by name)
module level · realms/databricks/pipelines/silver.py:16 (by name)
run_checks · realms/databricks/tasks/checks.py:39 (by name)
metric_view · realms/databricks/tasks/checks.py:102 (by name)
and 15 more
test_semantic_governance.py, silver_helpers.py and 42 more
Discovery.last
method, line 48
discover · ingest/discovery.py:62 (value)no test names it
discover
function, line 52
`files` = (window, first bytes, declared rows or None) for every delivery seen. Days are the windows' first ten characters; a day between the first and the last with no file is missing.none found in the codetest_discovery.py, test_cli.py and 3 more

ingest/pipelines.py

One ingestion pipeline per source type (ADR-002, D11). Each picks its batches, hands every one to the lander steps (steps.enter) and reports bookmarks, a watermark and what it noticed. They differ only in where batches come from: Ingestion

DefinitionWhat it does (its own first sentence)Called byNamed in tests
Plan
class, line 40
What a run does for one dataset: the run type and, for reload and backfill, the windows.run_ingest · scripts/run_hop.py:61
run_days · ingest/daybyday.py:129
module level · ingest/pipelines.py:386 (value)
no test names it
Sources
class, line 49
The source side as the binding hands it over: landing for push, the source systems for pull and stream. `systems` maps a manifest source_name to the system that holds it (erp -> the ERP database).run_ingest · scripts/run_hop.py:56
main · scripts/run_platform.py:441
module level · ingest/pipelines.py:386 (value)
test_ingest_pipelines.py, test_ingest_world.py and 3 more
_read
function, line 62
One call to a source through the run's guard: retried on temporary errors per the dataset's retry settings, refused while the source's circuit is open, limited by the source's connection cap.pull_copy · ingest/pipelines.py:255
pull_pages · ingest/pipelines.py:288
no test names it
Result
class, line 71
push · ingest/pipelines.py:131
pull_copy · ingest/pipelines.py:247
pull_pages · ingest/pipelines.py:276
stream_events · ingest/pipelines.py:323
module level · ingest/pipelines.py:386 (value)
test_obs_steps.py
Result.notice
method, line 78
_feed_notices · ingest/pipelines.py:210 (by name)
_stream_notices · ingest/pipelines.py:382 (by name)
no test names it
_window_of
function, line 84
pull_copy · ingest/pipelines.py:258no test names it
_table_spec
function, line 88
pull_copy · ingest/pipelines.py:248no test names it
_pick
function, line 99
(window, name) of batches recorded at all, and of batches loaded.push · ingest/pipelines.py:133no test names it
_replay_names
function, line 107
The recorded batches a reload (all) or backfill (not loaded) re-enters, latest record per batch.pull_copy · ingest/pipelines.py:260
pull_pages · ingest/pipelines.py:286
stream_events · ingest/pipelines.py:326
no test names it
_max_mark
function, line 117
The larger of a stored mark and new values: whole numbers compare as numbers, everything else as ISO text.push · ingest/pipelines.py:155
_watermark · ingest/pipelines.py:230
no test names it
push
function, line 130
module level · ingest/pipelines.py:387 (value)no test names it
feed_columns
function, line 171
Delivery column -> standard feed column. A change-capture service names its columns its own way (ADR-031): cdc_op_column (default op), cdc_position_column (default feed_pos; empty when the service has none) and cdc_commit_column (default copush · ingest/pipelines.py:147test_ingest_formats.py
_op_value
function, line 184
Operation letters upper-cased (I, U, D); a full-load row with no operation stays None (history reads it as U).push · ingest/pipelines.py:148no test names it
_feed_bookmarks
function, line 191
One bookmark per feed (a dealer), holding its newest data time as an instant in the business time zone; it only moves forward. Freshness compares feeds with their siblings; gold reads them for completeness (ADR-018).push · ingest/pipelines.py:167no test names it
_feed_notices
function, line 207
Deletes arrive as op D (DEL-01). A change whose sequence is older than one already seen for its key in the delivery arrived out of order (SCD-02); silver orders history by the sequence (W1c).push · ingest/pipelines.py:149no test names it
_watermark
function, line 225
The dataset's watermark column, moved forward to the largest value loaded (never back).push · ingest/pipelines.py:161
pull_pages · ingest/pipelines.py:302
stream_events · ingest/pipelines.py:357
no test names it
_guard_keys
function, line 234
What the mass-delete guard compares a full copy with: silver's current keys (DPC-0054), or the last accepted copy in bronze while silver has never been written (the first days, or a binding without silver).pull_copy · ingest/pipelines.py:263no test names it
pull_copy
function, line 245
The newest full copy only: a newer copy supersedes an older one (and a held one).module level · ingest/pipelines.py:389 (value)no test names it
pull_pages
function, line 274
Pages since the cursor, one batch per page; the batch name holds the cursor range so a reload can re-read it.module level · ingest/pipelines.py:390 (value)no test names it
_cursor_of
function, line 307
A record's position in the interface's order: '<updated_at>|<key>' (adapters/http_api.py).pull_pages · ingest/pipelines.py:289no test names it
_resume_cursor
function, line 312
The cursor after the last page loaded, if a failed run loaded pages its bookmark does not show.pull_pages · ingest/pipelines.py:292no test names it
stream_events
function, line 319
Trigger-once: each shard from its checkpoint to the end, one batch per shard, then stop (BP-044). Bronze keeps every reading as delivered (RAW-01); replays and late readings are counted here and dropped or placed by event time in silver (DUmodule level · ingest/pipelines.py:391 (value)test_ingest_pipelines.py
_stream_rows
function, line 361
stream_events · ingest/pipelines.py:332no test names it
_stream_notices
function, line 365
DUP-04: a reading id read before in this run is a replay. LATE-03: a reading whose event time is more than the allowance behind the newest already read on its shard arrived out of order.stream_events · ingest/pipelines.py:351no test names it

ingest/ports.py

The ports ingestion talks to (data-platform-catalog dpc/ports.yaml: landing_store, table_writer, tokenizer). Ingestion

DefinitionWhat it does (its own first sentence)Called byNamed in tests
check_batch_key
function, line 19
DeltaBronze.write · adapters/delta_bronze.py:60
DeltaBronze.write_many · adapters/delta_bronze.py:80
DeltaBronze.remove · adapters/delta_bronze.py:94
MemoryBronze.write · adapters/memory_bronze.py:19
MemoryBronze.remove · adapters/memory_bronze.py:43
SparkBronzeWriter.write · adapters/spark_delta.py:152
and 5 more
no test names it
DeliveryExists
class, line 25
Landing refuses a second write of the same delivery: a delivery is immutable once its manifest exists (landing_store invariant, DUP-02). The mover treats it as already landed.AzureLanding.put · adapters/azure_blob.py:72
FileLanding.put · adapters/files.py:44
S3Landing.put · adapters/s3.py:59
test_azure_adapters.py, test_cloud_adapters.py and 1 more
Landing
class, line 30
landing_store for push deliveries: the bytes exactly as delivered, with a manifest beside them.none found in the codeno test names it
Landing.put
method, line 33
Store a delivery and then its manifest. Raises DeliveryExists if (window, name) was landed before.ChangeFeedMover.move · adapters/local_movers.py:38 (by name)
DropMover.move · adapters/local_movers.py:65 (by name)
DmsMover.move · adapters/local_movers.py:105 (by name)
World.write · world/engine.py:133 (by name)
no test names it
Landing.deliveries
method, line 45
Deliveries that have a manifest, in window then name order.step_prepare · realms/databricks/tasks/main.py:81 (by name)
ChangeFeedMover.checkpoint · adapters/local_movers.py:28 (by name)
DropMover.move · adapters/local_movers.py:56 (by name)
push · ingest/pipelines.py:135 (by name)
land · lander.py:31 (by name)
gold_helpers.py, test_cloud_adapters.py and 5 more
Landing.read_bytes
method, line 48
LocalFileDrop.list · adapters/file_drop.py:38 (by name)
LocalFileDrop.get · adapters/file_drop.py:41 (by name)
FileLanding.deliveries · adapters/files.py:56 (by name)
FileLanding.read_bytes · adapters/files.py:64 (by name)
push · ingest/pipelines.py:144 (by name)
test_ingest_formats.py, test_ingest_ports.py and 3 more
Landing.manifest
method, line 50
The manifest the mover wrote, including any extra fields (feed positions, arrival time).ChangeFeedMover.checkpoint · adapters/local_movers.py:28 (by name)test_bundle.py, gold_helpers.py and 34 more
BronzeWriter
class, line 54
table_writer for bronze: append-only as delivered, idempotent by batch key.none found in the codeno test names it
BronzeWriter.write
method, line 57
Write one batch in one transaction, replacing whatever the table holds for this batch key: the same batch written twice leaves the table unchanged (DUP-01, DPC-0024) and a failed write leaves no trace (ORCH-03). `replace` is the lander sayinone found in the codegold_helpers.py, test_cloud_adapters.py and 31 more
BronzeWriter.write_many
method, line 63
Write several new batches of one table in one transaction (one commit instead of one per batch); all or nothing. The lander logs them as loaded only after this returns. Returns the rows written.flush · ingest/steps.py:365 (by name)no test names it
BronzeWriter.fingerprint
method, line 67
{"row_count": n, <column>: total, ...} for the table, or for the rows where every column equals its value. A table never written has row_count 0._one · ingest/runner.py:149 (value)
FixtureRunner.run_day · obs/fixture.py:117 (value)
FixtureRunner._rerun · obs/fixture.py:134 (value)
run_days · ingest/daybyday.py:133 (by name)
gold_helpers.py, silver_helpers.py and 14 more
BronzeWriter.keys
method, line 73
The distinct values of `columns` in the rows matching `where` (the mass-delete guard's comparison).none found in the codeno test names it
CurrentKeys
class, line 77
Silver's current keys, for the mass-delete guard (DPC-0054: compare a full copy with silver's current keys).none found in the codetest_ingest_pipelines.py
CurrentKeys.current_keys
method, line 80
The keys of the current, not deleted silver rows, or None when silver was never written._guard_keys · ingest/pipelines.py:239 (by name)test_ingest_pipelines.py, test_silver_ports.py
Tokenizer
class, line 84
tokenizer: a personal value in, a token out, before anything reaches bronze (DPC-0070).none found in the codeno test names it
Tokenizer.token
method, line 87
main.on_day · scripts/run_platform.py:582 (value)
instance_token · adapters/lakebase.py:65 (value)
main.live_rows · scripts/run_platform.py:489 (by name)
mass_delete_problem.stored · ingest/steps.py:256 (by name)
protect · ingest/steps.py:291 (by name)
test_bundle.py, test_semantic_governance.py and 18 more

ingest/runner.py

One ingestion run over a domain, every step recorded in the metadata store (ADR-010): Ingestion

DefinitionWhat it does (its own first sentence)Called byNamed in tests
DatasetReport
class, line 43
_ingest · hop_reports.py:41
_one · ingest/runner.py:154
_one_leased · ingest/runner.py:177
RunReport.by_name · ingest/runner.py:58 (value)
test_hop_reports.py
RunReport
class, line 53
_ingest · hop_reports.py:41
ingest · ingest/runner.py:135
kind_of · hop_reports.py:26 (value)
run_days · ingest/daybyday.py:115 (value)
test_hop_reports.py
RunReport.by_name
method, line 58
run_days · ingest/daybyday.py:131 (by name)
judge · silver/daybyday.py:148 (by name)
ClinicalGoldJudge._enrolment · world/domains/clinical_gold.py:130 (by name)
ClinicalGoldJudge._later · world/domains/clinical_gold.py:206 (by name)
TyreGoldJudge._sales · world/domains/tyre_gold.py:146 (by name)
TyreGoldJudge._later · world/domains/tyre_gold.py:223 (by name)
test_gold_harness.py, test_hop_reports.py and 11 more
_profile_step
function, line 62
The run's profile (B9, DPC-0051): logged as a step and compared with the last one the store kept; drift makes the step `warned` and names what moved. A normal run's profile is kept for the next comparison._one_leased · ingest/runner.py:208no test names it
counts_of
function, line 78
_one_leased · ingest/runner.py:187no test names it
status_of
function, line 86
_one_leased · ingest/runner.py:187no test names it
detect_rows
function, line 92
(detector, status, counts, message) per detector that met this run: a held batch is `failed` (blocked), a skipped duplicate or anything noticed is `warned` (handled or flagged for the layer that applies it)._detect_steps · ingest/runner.py:108test_obs_steps.py
_detect_steps
function, line 107
_one_leased · ingest/runner.py:207no test names it
ingest
function, line 112
Run every enabled dataset of a domain (or the named `source.dataset`s) once, by the plan's run type.run_ingest · scripts/run_hop.py:60
run_days · ingest/daybyday.py:129
test_bundle.py, silver_helpers.py and 44 more
_one
function, line 138
ingest · ingest/runner.py:133no test names it
_one_leased
function, line 161
_one · ingest/runner.py:156no test names it

ingest/steps.py

The lander steps: one library every way into bronze calls (ING-01, ADR-002, DPC-0037). A pushed file, a pulled copy, a page from an interface and a chunk of a stream all pass the same steps in the same order: Ingestion

DefinitionWhat it does (its own first sentence)Called byNamed in tests
CastError
class, line 50
parse · ingest/steps.py:151
_parquet_rows · ingest/steps.py:162
_cast_int · ingest/steps.py:194
_cast_bool · ingest/steps.py:205
cast_rows · ingest/steps.py:230 (value)
enter · ingest/steps.py:329 (value)
no test names it
content_key
function, line 54
The batch key: the same bytes give the same key whatever the file is called or when it came (DUP-01).enter · ingest/steps.py:317test_ingest_steps.py
Spec
class, line 60
One dataset as the lander needs it, from the metadata store's merged view (never from files).none found in the codeno test names it
Spec.table
method, line 75
none found in the codetest_lakeformation.py, test_semantic_governance.py and 84 more
Spec.bronze_columns
method, line 79
enter · ingest/steps.py:355 (value)
flush · ingest/steps.py:365 (value)
no test names it
Spec.from_view
method, line 83
_one · ingest/runner.py:148 (by name)test_ingest_pipelines.py
Batch
class, line 101
One batch on its way in: a delivery's bytes (push) or rows read from a source (pull, stream).push · ingest/pipelines.py:151
pull_copy · ingest/pipelines.py:267
pull_pages · ingest/pipelines.py:290
stream_events · ingest/pipelines.py:332
test_housekeeping.py, test_ingest_steps.py and 2 more
Outcome
class, line 113
enter · ingest/steps.py:323
module level · ingest/steps.py:137 (value)
test_obs_steps.py
Ctx
class, line 125
_one_leased · ingest/runner.py:179
module level · ingest/pipelines.py:386 (value)
no test names it
parse
function, line 141
A delivery's bytes as rows, by the dataset's file_format_data (ADR-031): json_lines (default), csv/tsv/pipe with a header, or parquet (change-capture and buffered-stream deliveries; types and nulls kept).push · ingest/pipelines.py:148
enter · ingest/steps.py:328
test_ingest_formats.py, test_ingest_steps.py and 19 more
_parquet_rows
function, line 156
Rows of a columnar file; needs pyarrow (the `local` extra). Timestamps come back as datetimes, so the cast step sees the same types as from text.parse · ingest/steps.py:146no test names it
check_columns
function, line 166
(reason the batch stops, or ''; new columns seen). An empty batch has no columns to judge: the volume step holds it.enter · ingest/steps.py:331test_ingest_steps.py, test_slice1_local.py
_cast_ts
function, line 187
module level · ingest/steps.py:212 (value)no test names it
_cast_int
function, line 192
module level · ingest/steps.py:210 (value)no test names it
_cast_bool
function, line 200
module level · ingest/steps.py:214 (value)no test names it
cast_rows
function, line 218
Cast every value to its column's type, in place. Returns why the batch stops (naming the column), or ''. An empty text value is null for every type but string (CSV has no null).enter · ingest/steps.py:337test_ingest_steps.py
volume_problem
function, line 235
enter · ingest/steps.py:340test_ingest_steps.py
mass_delete_problem
function, line 244
A full copy that would delete more than the declared share of the last accepted copy's keys (DEL-04). `previous` holds the keys as stored (a restricted key column holds its token), so the copy's keys are compared as stored too: with a `tokeenter · ingest/steps.py:343test_ingest_steps.py
mass_delete_problem.stored
function, line 254
mass_delete_problem · ingest/steps.py:259test_semantic_governance.py, test_azure_adapters.py and 6 more
feed_marks
function, line 269
A file that feeds one member (a dealer's monthly file): every row shares one value of a text column of the primary key, and the key holds a date or time column. Returns {that value: the newest date or time in the file}, else {}. One bookmarenter · ingest/steps.py:352no test names it
protect
function, line 284
Tokenise restricted columns in place; returns how many values were replaced.enter · ingest/steps.py:346test_ingest_steps.py
stamp
function, line 296
enter · ingest/steps.py:348no test names it
enter
function, line 315
Run the steps for one batch and record the outcome. Never raises for a bad batch: it is held and logged.push · ingest/pipelines.py:151
pull_copy · ingest/pipelines.py:267
pull_pages · ingest/pipelines.py:290
stream_events · ingest/pipelines.py:332
test_ingest_steps.py
flush
function, line 359
Write the run's pending new batches in one commit, then log each as loaded: the file log never says loaded before bronze holds the rows. If the write fails nothing is logged and the next run takes the batches again._one_leased · ingest/runner.py:181no test names it
_log
function, line 372
enter · ingest/steps.py:323no test names it
_record
function, line 379
flush · ingest/steps.py:367
_log · ingest/steps.py:376
no test names it