Data PlatformZoom level 3
Who calls what: Top-level modules
Every class, function and method in Top-level modules, with who calls it. Found by reading the code. A call written as object.method() is matched by the method's name, so a same-named method elsewhere can appear: those rows say by name.
123 definitions in 17 modules. Jump to a module: cloud_ports.py · contract.py · hop_reports.py · housekeeping.py · keys.py · lander.py · leases.py · manifest.py · model.py · ports.py · profiling.py · reconcile.py · recovery.py · resilience.py · retention.py · source_ports.py · timing.py
cloud_ports.py
Ports only a cloud fills (data-platform-catalog dpc/ports.yaml, binds_in: cloud). The kernel imports only these shapes; product names stay in the adapters (R1). The source-side ports (file drop, event stream, database, record interface) are none yet
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
ObjectStoreclass, line 12 | object_store: the bytes under landing and every table, one location per environment and layer (ADR-012). | none found in the code | no test names it |
ObjectStore.locationmethod, line 15 | The storage location for an environment and a layer, e.g. a bucket URL. | none found in the code | test_azure_adapters.py, test_cloud_adapters.py and 2 more |
ObjectStore.versionedmethod, line 18 | True when the location keeps old versions (invariant ORCH-05: storage meets the recovery targets). | none found in the code | test_azure_adapters.py, test_cloud_adapters.py and 2 more |
SecretNotFoundclass, line 25 | A secret reference the store does not hold. The message names the reference, never a value. | EnvSecrets.get · adapters/secrets.py:32AwsSecretsManager.get · adapters/secrets.py:46AzureKeyVault.get · adapters/secrets.py:85 | test_secrets.py |
SecretStoreclass, line 29 | secret_store (DPC-0029): credentials come only from here, by reference (secret://<name>), resolved at run time and never written to metadata, logs or generated files. | none found in the code | no test names it |
SecretStore.getmethod, line 33 | none found in the code | no test names it | |
secret_namefunction, line 36 | The name inside a secret://<name> reference; anything else is refused (a value is never a reference). | EnvSecrets.get · adapters/secrets.py:26AwsSecretsManager.get · adapters/secrets.py:41AzureKeyVault.get · adapters/secrets.py:79 | test_secrets.py, test_adf_generator.py and 1 more |
contract.py
Reading an ODCS v3.2.0 contract: the columns a delivery must carry, and the quality rules as row predicates. none yet
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
ColumnVerdictclass, line 17 | check_columns · contract.py:49 | no test names it | |
ColumnVerdict.reasonmethod, line 23 | main.housekeeping · scripts/run_platform.py:514 (value)main · scripts/run_platform.py:680 (value)MemoryMetaStore.log_file · adapters/memory_meta.py:207 (value)PostgresMetaStore.log_file · adapters/postgres_meta.py:164 (value)_sweep_leased · housekeeping.py:176 (value)_record · ingest/steps.py:382 (value)and 4 more | test_semantic_governance.py, test_dac_coverage.py and 38 more | |
propertiesfunction, line 30 | columns · contract.py:35primary_key · contract.py:39check_columns · contract.py:46row_rules · contract.py:64 | gold_helpers.py, silver_helpers.py and 9 more | |
columnsfunction, line 34 | none found in the code | test_lakeformation.py, test_semantic_governance.py and 67 more | |
primary_keyfunction, line 38 | none found in the code | no test names it | |
check_columnsfunction, line 43 | A missing required column stops the load (SCH-02). New columns are allowed through to bronze only (manifest load.new_columns = hold_in_bronze), so they are reported, not fatal. | none found in the code | test_ingest_steps.py, test_slice1_local.py |
row_rulesfunction, line 55 | Column-level ODCS rules as SQL predicates a good row satisfies: {rule_name: predicate}. | none found in the code | no test names it |
hop_reports.py
A hop's report as plain data, so a hop that runs as its own job can hand its report to whoever judges it later. none yet
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
HopReportErrorclass, line 21 | kind_of · hop_reports.py:32load · hop_reports.py:60 | test_hop_reports.py | |
kind_offunction, line 25 | dump · hop_reports.py:37 | test_gold_harness.py, test_hop_reports.py and 4 more | |
dumpfunction, line 35 | The report as JSON-ready data, with its kind and a format version. | main · scripts/run_hop.py:125 | no test names it |
_ingestfunction, line 40 | module level · hop_reports.py:54 (value) | no test names it | |
_silverfunction, line 44 | module level · hop_reports.py:54 (value) | no test names it | |
_goldfunction, line 48 | module level · hop_reports.py:54 (value) | no test names it | |
loadfunction, line 57 | The report object back from `dump`'s data. The kind asked for must be the kind that was written. | main.run_hop · scripts/run_platform.py:558 | no test names it |
housekeeping.py
What the housekeeping job may remove from bronze (B8; DAC RET-01, DPC-0116). Pure and product-free: it decides, adapters act (archive, delete by batch key, expire snapshots) and the run log records. Recovery and housekeeping
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
Batchclass, line 37 | batches_of · housekeeping.py:109batches_of · housekeeping.py:105 (value) | test_housekeeping.py, test_ingest_steps.py and 2 more | |
Planclass, line 45 | plan_removals · housekeeping.py:63_sweep · housekeeping.py:151 | no test names it | |
plan_removalsfunction, line 53 | _sweep_leased · housekeeping.py:162 | test_housekeeping.py | |
BronzeRemoverclass, line 74 | The bronze writer's removal edge (MemoryBronze, DeltaBronze, StoreBronzeWriter). | none found in the code | no test names it |
BronzeRemover.removemethod, line 77 | Delete the rows of these batches in one transaction; returns the rows removed. | main.housekeeping · scripts/run_platform.py:506 (value)_sweep_leased · housekeeping.py:170 (value)_sweep_leased · housekeeping.py:170 (by name)RunHandle.dataset · obs/recorder.py:160 (by name)SilverBaseliner.write · silver/baseline_writer.py:70 (by name) | test_lakeformation.py, test_housekeeping.py and 2 more |
BaselineOutcomeclass, line 82 | SilverBaseliner.write · silver/baseline_writer.py:52 | test_housekeeping.py | |
Baselinerclass, line 88 | Writes and checks a baseline batch for one dataset (silver/baseline_writer.py: a compaction of bronze, replayed through the processors against silver). Only a checked baseline is logged to the file log, as load type baseline, window = the c | none found in the code | no test names it |
Baseliner.writemethod, line 93 | Write, check and log one baseline; a baseline that fails its check is removed again and refused. | none found in the code | gold_helpers.py, test_cloud_adapters.py and 31 more |
batches_offunction, line 103 | Loaded batches, one per batch key (the first loaded record), void ones left out. | _sweep_leased · housekeeping.py:162SilverBaseliner.write · silver/baseline_writer.py:57 | test_housekeeping.py |
Sweptclass, line 114 | _sweep · housekeeping.py:151_sweep_leased · housekeeping.py:178 | no test names it | |
run_housekeepingfunction, line 121 | One housekeeping run over a domain's source datasets. Each dataset gets its plan; removals go through the bronze port, then each removed batch is voided; one step per dataset records the plan. A dataset that needs a baseline gets one from t | main.housekeeping · scripts/run_platform.py:501 | test_housekeeping.py |
_sweepfunction, line 138 | run_housekeeping · housekeeping.py:132 | no test names it | |
_sweep_leasedfunction, line 158 | _sweep · housekeeping.py:153 | no test names it |
keys.py
Batch keys (DPC-0024) and config fingerprints (CFG-02). Landing to bronze
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
checksumfunction, line 9 | AzureLanding.put · adapters/azure_blob.py:86AzureLanding.deliveries · adapters/azure_blob.py:95AzureFileDrop.put · adapters/azure_blob.py:129LocalFileDrop.put · adapters/file_drop.py:31LocalFileDrop.list · adapters/file_drop.py:38FileLanding.put · adapters/files.py:50and 6 more | test_azure_adapters.py, test_cloud_adapters.py and 4 more | |
batch_keyfunction, line 13 | Same window and same bytes give the same key, whatever the file is called. That is what makes a resend a no-op. | land · lander.py:32 | silver_helpers.py, test_housekeeping.py and 3 more |
config_fingerprintfunction, line 18 | A short, stable hash of the merged config a run used. Key order does not matter. | step_prepare · realms/databricks/tasks/main.py:67land · lander.py:58dataset_doc · meta/loader.py:98compile_domain · meta/loader.py:132load · meta/loader.py:173gold_doc · meta/loader.py:207 | test_ingest_pipelines.py, test_meta_store.py and 2 more |
lander.py
The lander: landing -> bronze. Our own code (decision D11); the mover that filled landing is swappable. Landing to bronze
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
landfunction, line 23 | step_land · realms/databricks/tasks/main.py:113 | no test names it | |
_decidefunction, line 63 | land · lander.py:44 | no test names it |
leases.py
Writer leases (ORCH-04): two runs never write the same table at once. Product-free; the metadata store keeps the lease (runtime.table_lease), so it holds across jobs, clusters and laptops. none yet
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
busyfunction, line 20 | Record that this run did not write because another holds the lease; the dataset run must be started. | run_gold · gold/harness.py:143_sweep · housekeeping.py:150_one · ingest/runner.py:153_one · silver/runner.py:124 | no test names it |
manifest.py
DPM manifest -> compiled config. Runs locally or in CI (needs PyYAML); the platform itself reads only the result, after the register step has loaded it into the metadata DB (agreed 27 Oct: YAML in git -> validating loader -> DB). none yet
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
ManifestErrorclass, line 19 | compile_slice · manifest.py:53compile_domain · meta/loader.py:150load · meta/loader.py:167load_gold · meta/loader.py:233 | test_bundle.py, test_meta_loader.py and 1 more | |
validatefunction, line 23 | Problems that stop a compile. An empty list means it is valid. | compile_slice · manifest.py:51compile_domain · meta/loader.py:139 | test_obs_dashboards.py, test_slice1_local.py and 3 more |
compile_slicefunction, line 44 | module level · scripts/dp_compile.py:21 | no test names it | |
fillfunction, line 74 | Replace ${bronze} etc. with this environment's schema names. | step_prepare · realms/databricks/tasks/main.py:65 | no test names it |
model.py
Plain records shared by the kernel and the adapters. none yet
ports.py
The ports slice 1 uses (data-platform-catalog dpc/ports.yaml). The kernel imports only these shapes. none yet
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
LandingStoreclass, line 13 | landing_store: files exactly as delivered, kept so a bad read can be redone. | none found in the code | no test names it |
LandingStore.deliveriesmethod, line 16 | step_prepare · realms/databricks/tasks/main.py:81 (by name)ChangeFeedMover.checkpoint · adapters/local_movers.py:28 (by name)DropMover.move · adapters/local_movers.py:56 (by name)push · ingest/pipelines.py:135 (by name)land · lander.py:31 (by name) | gold_helpers.py, test_cloud_adapters.py and 5 more | |
LandingStore.readmethod, line 18 | none found in the code | no test names it | |
LandingStore.writemethod, line 20 | none found in the code | gold_helpers.py, test_cloud_adapters.py and 31 more | |
MetadataStoreclass, line 23 | metadata_service: config is read from here at run time, never from YAML (agreed 27 Oct). | none found in the code | no test names it |
MetadataStore.dataset_configmethod, line 26 | step_land · realms/databricks/tasks/main.py:112 (by name)step_finish · realms/databricks/tasks/main.py:123 (by name) | test_exceptions.py, test_runtime_sql_views.py | |
MetadataStore.batch_statusmethod, line 28 | StampingRunLog.batch_status · adapters/memory_runlog.py:96 (by name)StoreBatchRegistry.batch_status · adapters/store_registry.py:24 (by name)enter · ingest/steps.py:318 (by name)land · lander.py:34 (by name) | test_meta_store.py, test_slice1_local.py | |
MetadataStore.record_batchmethod, line 30 | land · lander.py:38 (by name) | no test names it | |
TableWriterclass, line 33 | table_writer: append rows to a bronze table. Silver is written by the pipeline product, behind the same port. | none found in the code | no test names it |
TableWriter.appendmethod, line 36 | none found in the code | azure_fakes.py, gold_helpers.py and 45 more | |
RunLogclass, line 39 | run_log: one row per step per run, with the config fingerprint (DPC-0020, CFG-02). | none found in the code | no test names it |
RunLog.writemethod, line 42 | none found in the code | gold_helpers.py, test_cloud_adapters.py and 31 more | |
CheckSinkclass, line 45 | Where check verdicts go. Keyed by run id, so the evidence view can join everything for one load. | none found in the code | no test names it |
CheckSink.writemethod, line 48 | none found in the code | gold_helpers.py, test_cloud_adapters.py and 31 more |
profiling.py
Profiling (X5, DPC-0051, B9): every loaded batch profiled at the lander, the dataset run's profile compared with the last one, and drift reported in the run log. Standard library only. Reconciliation and profiling
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
_textfunction, line 21 | profile_rows · profiling.py:34 | no test names it | |
profile_rowsfunction, line 25 | enter · ingest/steps.py:347 | test_profiling.py | |
mergefunction, line 42 | One profile for a run's batches: rows and nulls add up, distinct is the largest batch's (a lower bound), min and max widen. | none found in the code | test_baseline_stage.py, test_meta_store.py and 7 more |
null_sharefunction, line 58 | drift · profiling.py:76 | no test names it | |
driftfunction, line 63 | Findings, each one plain sentence; empty when nothing moved beyond the limits or there is no earlier profile. | none found in the code | test_lakeformation.py, test_governance_catalog.py and 4 more |
reconcile.py
Reconciliation between hops (DPC-0015, DPC-0016): the same measure, counted at each hop, must agree. Reconciliation and profiling
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
comparefunction, line 7 | measures = {hop: {measure: value}}, hops in spine order. Each hop is compared with the first hop. | none found in the code | test_profiling.py, test_silver_stages.py and 4 more |
bucket_offunction, line 31 | The bucket of a business key: the first 8 hex digits of sha256 over the key's text parts, mod `buckets`. The engine side computes the same (sha2 over concat_ws of the parts cast to text), so both sides agree on buckets. | bucket_totals · reconcile.py:47reconcile_live · silver/live_recon.py:67 | test_live_recon.py, test_reconcile_buckets.py |
bucket_totalsfunction, line 41 | {bucket: {measure: total}} for rows held in memory (a source's current rows). `measures` maps a measure name to the column it sums, or None for a row count. | reconcile_live · silver/live_recon.py:59 | test_live_recon.py, test_reconcile_buckets.py and 1 more |
compare_bucketsfunction, line 53 | (bucket, measure, left, right) for every bucket and measure that differ beyond the tolerance (REC-05: the finding names where it drifted, not only that it did). A bucket on one side only counts as zero on the other. | reconcile_live · silver/live_recon.py:62 | test_reconcile_buckets.py |
recovery.py
Withdrawing a bad batch after it was published (VOID-01, ADR-025). Product-free. Recovery and housekeeping
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
withdrawfunction, line 18 | none found in the code | test_gold_harness.py |
resilience.py
Resilience for calls to sources (X18, B7): retry with backoff, a circuit breaker per source, a connection cap per source. Standard library only; time comes in as functions (DPC-0032: time only from the clock port), so tests drive it. Resilience
retention.py
Retention (ADR-029, B8): how long each copy is kept, so a fault found late still has a correct copy to reload from. Pure functions over dates and settings; the landing adapter or the cloud's lifecycle rules carry the actions out. Recovery and housekeeping
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
Retentionclass, line 23 | none found in the code | test_retention.py | |
Retention.from_settingsmethod, line 31 | _sweep · housekeeping.py:145 (by name)_read · ingest/pipelines.py:67 (by name)apply_landing_retention · retention.py:92 (by name) | test_resilience.py, test_retention.py | |
Retention.from_settings.numfunction, line 32 | Retention.from_settings · retention.py:36 | no test names it | |
window_dayfunction, line 50 | A delivery window's first day: 2026-11-09, 2026-11-09T10, 2026-11 (the month's first day). | batches_of · housekeeping.py:109landing_actions · retention.py:62 | test_retention.py |
landing_actionsfunction, line 56 | For each (window, name, place) with place 'hot' or 'archive': (window, name, action) with action keep, archive or delete. A delivery the lander never took (not in `taken`) is always kept: it may be the only copy. | apply_landing_retention · retention.py:97 | test_retention.py |
bronze_cutofffunction, line 77 | Windows before this day may leave bronze, and only after a baseline batch holds silver's state as of it (ADR-029), unless the dataset keeps bronze forever (bronze_baseline false and a very long retention). | none found in the code | test_retention.py |
apply_landing_retentionfunction, line 87 | Carry out landing_actions for one dataset through a landing archive adapter (LandingArchive locally, lifecycle rules on a cloud). Personal data is decided from the contract's column classes. Returns counts per action. | none found in the code | test_retention.py |
source_ports.py
The four mock-source ports (goal G4, data-platform-catalog dpc/ports.yaml: source systems). Sources and movers
timing.py
End-to-end and hop-by-hop delay (ADR-034, B4): every hop stamps its own time; this turns the stamps into the 50th, 95th and 99th percentile per hop and end to end, each labelled measured or modeled. none yet
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
HopDelayclass, line 21 | hop_delays_frame · adapters/spark_silver.py:381hop_delay · timing.py:40 | test_timing.py | |
percentilefunction, line 30 | Nearest-rank percentile of already sorted values (0 for none). | hop_delay · timing.py:40 | test_timing.py |
hop_delayfunction, line 38 | none found in the code | test_timing.py | |
breachesfunction, line 43 | The hops whose delay at `at` is over its promise (seconds), each named, with its basis (ADR-034: a breach points at its hop, and a modeled number is never passed off as measured). | none found in the code | test_obs_dashboards.py, test_timing.py |