Data PlatformZoom level 2
Code index: every module, what it is for, and where it is explained
Every Python module of the platform, grouped by package. Each row gives the module's own one-line description, its size, a link to the file, and the page that explains it. The list is generated from the code when the site is built, so it cannot drift.
192 modules in 16 groups.
| Group | Modules | Lines | Caller pages |
|---|---|---|---|
| access | 2 | 212 | access |
| adapters | 47 | 5,581 | adapters |
| Top-level modules | 18 | 1,260 | Top-level modules |
| des | 2 | 57 | des |
| gold | 9 | 1,190 | gold |
| governance | 5 | 457 | governance |
| ingest | 7 | 1,316 | ingest |
| meta | 4 | 601 | meta |
| obs | 23 | 3,223 | obs |
| Deployable bundles (realms) | 22 | 2,436 | Deployable bundles (realms) |
| Scripts | 11 | 3,002 | Scripts |
| semantic | 4 | 481 | semantic |
| silver | 12 | 1,327 | silver |
| stages | 7 | 932 | stages |
| twin | 6 | 682 | twin |
| world | 13 | 3,326 | world |
access
| Module | What it is for (its own first sentence) | Lines | Explained on |
|---|---|---|---|
access/__init__.py | Access model as code (DPC-0066; SEC-02, ENV-02; dpc/ports.yaml access_control). | 7 | Semantic layer |
access/model.py | Groups and classification tags -> grants, column masks and row filters. | 205 | Semantic layer |
adapters
| Module | What it is for (its own first sentence) | Lines | Explained on |
|---|---|---|---|
adapters/__init__.py | Adapters: one package per product family. Only adapters import a product. | 9 | none yet |
adapters/azure_blob.py | Object storage on Azure Blob Storage (cloud layer): landing_store, the partner file drop and object_store. Needs azure-storage-blob (the `azure` extra) only when a real client is built; nothing here imports an Azure SDK. The container is pa | 171 | none yet |
adapters/customer_runlog.py | EXAMPLE SKELETON: a customer's own run log on the run-log reader port (level 2 of ADR-024 rule 6). | 37 | none yet |
adapters/databricks_clone.py | The release gate's ports (dataplatform.twin.gate) on Databricks: Delta SHALLOW CLONE and one measuring query. | 70 | none yet |
adapters/databricks_native.py | Databricks bindings for the adopted platform-native screens and notifications (dataplatform.obs.native). | 66 | none yet |
adapters/databricks_notify.py | The notifier port (dataplatform.obs.monitor.Notifier) through the workspace's notification destinations. | 46 | none yet |
adapters/databricks_ops_views.py | The ops metric views (dataplatform.obs.views) as Databricks metric views: one `CREATE ... WITH METRICS` per view. | 54 | none yet |
adapters/databricks_pipeline_events.py | The pipeline event-log port (dataplatform.obs.pipeline_events) on Lakeflow Declarative Pipelines' event log. | 67 | none yet |
adapters/databricks_system.py | The system-records port (dataplatform.obs.system) on Databricks system tables. | 105 | none yet |
adapters/delta_bronze.py | The bronze writer on Delta tables in a local folder, through delta-rs (`deltalake`, Apache 2.0) and pyarrow: no JVM, no Spark, the same approach as LakeIngestionFramework's LocalDeltaWriter (core/writers/local_delta.py), which only appends; | 136 | Landing to bronze |
adapters/duckdb_frames.py | DuckDB as a frame engine for the pure stage and build code, through SQLFrame (the PySpark DataFrame API compiled to SQL by sqlglot): one logic, several engines (docs/DUCKDB-SPIKE.md). For laptop and CI runs only; nothing here ever writes a | 61 | none yet |
adapters/duckdb_gold.py | Gold's engine operations on DuckDB through SQLFrame (docs/DUCKDB-SPIKE.md): the Spark adapter's operations, with the five engine-specific pieces replaced. For laptop and CI runs only; Databricks keeps adapters/spark_gold.py. | 165 | The harness |
adapters/duckdb_silver.py | Silver on DuckDB (through SQLFrame) and local Delta tables (through delta-rs): the same stages (dataplatform.stages) as the Spark adapter, other IO (docs/DUCKDB-SPIKE.md). For laptop and CI runs only: nothing here writes a governed table on | 137 | Writer |
adapters/file_drop.py | FileDrop on a local folder: the partner and lab drop zones. Object storage is a later adapter with the same port. | 45 | Sources and movers |
adapters/files.py | landing_store on a plain folder. On Databricks the folder is a Unity Catalog volume path, which serverless Python reads and writes as ordinary files, so the same adapter serves both. | 100 | none yet |
adapters/http_api.py | RecordInterface as a small local HTTP service: the dealer management system. | 127 | Sources and movers |
adapters/keyed_tokens.py | The tokenizer as a keyed hash (HMAC-SHA256, standard library): the same value always gives the same token, so joins and duplicate checks still work, and nobody without the key can reverse or recompute it. The key is the binding's secret (DP | 37 | none yet |
adapters/kinesis.py | stream_bus on Amazon Kinesis Data Streams (cloud layer, dbs/cloud/aws.yaml). Needs boto3; the client is passed in, so tests use a local stand-in (moto) and the platform passes one built from the job's role. | 68 | Sources and movers |
adapters/lakebase.py | The metadata store on Lakebase, Databricks' managed PostgreSQL (ADR-010, binding dbs/engine/databricks.yaml). | 92 | Metadata store |
adapters/lakeview.py | The ops dashboards (dataplatform.obs.dashboards) as Databricks AI/BI (Lakeview) dashboard files. | 159 | none yet |
adapters/local_movers.py | The push movers for the local stack: they stand in for what a cloud supplies (a change-capture service, a file transfer service) and meet the landing contract every push mover meets (DPC-0038): each delivery written once, under a unique nam | 112 | Sources and movers |
adapters/local_stream.py | EventStream on local files: a log of shards, each an append-only JSON-lines file. A managed stream is a later adapter with the same port; the shard count and the partition-key hashing mirror how such streams spread records. | 73 | Sources and movers |
adapters/memory.py | In-memory adapters for tests. Same ports as the Spark/Delta ones. | 41 | none yet |
adapters/memory_bronze.py | The bronze writer in memory, for tests: the reference behaviour the table-format adapters are tested against (M5). | 48 | none yet |
adapters/memory_catalog.py | Catalog state in memory: the reference adapter for the CatalogState port (tests, dry runs, the first deploy). | 20 | none yet |
adapters/memory_meta.py | The metadata store in memory, for unit tests. Same port and the same refusals as the PostgreSQL adapter; the contract test table (tests/dataplatform/test_meta_store.py) runs every row against both (M5). | 391 | Metadata store |
adapters/memory_pipeline_events.py | The pipeline event-log port (dataplatform.obs.pipeline_events) in memory: rows added by tests. | 16 | none yet |
adapters/memory_runlog.py | StampingRunLog: any metadata store, plus the run-log reader port (dataplatform.obs.runlog) in process. | 201 | none yet |
adapters/memory_sources.py | The four source ports in memory: the reference behaviour every real adapter is tested against (M5). | 109 | none yet |
adapters/memory_system.py | The system-records port (dataplatform.obs.system) in memory: rows added by tests or the fixture runner. | 26 | none yet |
adapters/memory_tables.py | The release gate's ports (dataplatform.twin.gate) in memory: tables are lists of rows; a clone is a copy of the list (a shallow clone shares data until written; in memory a copy behaves the same for the gate). | 44 | none yet |
adapters/null_runlog.py | The run-log reader port (dataplatform.obs.runlog) with no run log connected: level 1 of ADR-024 rule 6. | 16 | none yet |
adapters/otel_tracer.py | The telemetry port (dataplatform.obs.trace) on the OpenTelemetry API. | 77 | none yet |
adapters/postgres_meta.py | The metadata store on PostgreSQL (ADR-010). Every read and write is one call to a function in adapters/sql/postgres_meta.sql; this file holds no table names (DPC-0033, checked by tests/dataplatform/test_meta_store.py). | 258 | Metadata store |
adapters/postgres_runlog.py | The run-log reader port (dataplatform.obs.runlog) on PostgreSQL: one call to runtime.ops_run_log (sql/postgres_obs.sql), installed beside the metadata store's own schema (postgres_meta.sql). No table names here (DPC-0033). Timestamps are th | 61 | none yet |
adapters/postgres_source.py | SourceDatabase on PostgreSQL: the mock ERP and MES databases, one schema per system. | 55 | Sources and movers |
adapters/s3.py | Object storage on Amazon S3 (cloud layer, dbs/cloud/aws.yaml): landing_store, the partner file drop and object_store. Needs boto3 (the `s3` extra). The client is passed in, so tests use a local stand-in (moto) and the platform passes one bu | 142 | Sources and movers |
adapters/secrets.py | secret_store adapters (cloud_ports.SecretStore, DPC-0029). The value is returned to the caller only; nothing here logs it, and an error names the reference, never the value. | 88 | none yet |
adapters/spark_delta.py | The slice 1 ports on Spark + Delta tables: Databricks (serverless, Spark Connect) or open-source Spark locally. | 203 | none yet |
adapters/spark_gold.py | Gold on Spark and Delta tables: the harness's engine operations (PySpark) and the gold read and write edges. | 499 | The harness |
adapters/spark_ops_tables.py | Publish the ops tables (obs/ops_export.py file) as catalog tables, then the ops metric views over them (B13). | 47 | none yet |
adapters/spark_silver.py | Silver on Spark and Delta tables: the stage runner (dataplatform.stages on PySpark) and the read and write edges. | 382 | Writer |
adapters/sql_source.py | SourceDatabase over any DB-API connection. Two thin adapters use it: postgres_source.py and sqlite_source.py. | 184 | none yet |
adapters/sqlite_source.py | SourceDatabase on SQLite (standard library): for tests and for a laptop with no Postgres. SQLite has no schemas, so a system's tables are prefixed with its name: erp_db__sales_order_line. | 30 | Sources and movers |
adapters/store_registry.py | Slice 1's batch registry (dataplatform.ports.MetadataStore: batch_status(key), record_batch) on the W1a metadata store, so slice 1's lander (dataplatform.lander) records into the same file log as W1b's ingestion, through the store's procedu | 42 | none yet |
adapters/table_store.py | The table port's engine side (data-platform-catalog dpc/ports.yaml table_writer, ADR-027): one small adapter per open table format, so silver, gold and bronze code never names a format. | 598 | none yet |
adapters/uc_catalog.py | Catalog state read from Unity Catalog's information schema (Databricks; dbs/engine/databricks.yaml data_catalog). | 66 | none yet |
Top-level modules
| Module | What it is for (its own first sentence) | Lines | Explained on |
|---|---|---|---|
__init__.py | dataplatform: the realm-agnostic kernel for the Clean Data Engineering specs. | 17 | none yet |
cloud_ports.py | Ports only a cloud fills (data-platform-catalog dpc/ports.yaml, binds_in: cloud). The kernel imports only these shapes; product names stay in the adapters (R1). The source-side ports (file drop, event stream, database, record interface) are | 48 | none yet |
contract.py | Reading an ODCS v3.2.0 contract: the columns a delivery must carry, and the quality rules as row predicates. | 77 | none yet |
hop_reports.py | A hop's report as plain data, so a hop that runs as its own job can hand its report to whoever judges it later. | 68 | none yet |
housekeeping.py | What the housekeeping job may remove from bronze (B8; DAC RET-01, DPC-0116). Pure and product-free: it decides, adapters act (archive, delete by batch key, expire snapshots) and the run log records. | 178 | Recovery and housekeeping |
keys.py | Batch keys (DPC-0024) and config fingerprints (CFG-02). | 20 | Landing to bronze |
lander.py | The lander: landing -> bronze. Our own code (decision D11); the mover that filled landing is swappable. | 73 | Landing to bronze |
leases.py | Writer leases (ORCH-04): two runs never write the same table at once. Product-free; the metadata store keeps the lease (runtime.table_lease), so it holds across jobs, clusters and laptops. | 25 | none yet |
manifest.py | DPM manifest -> compiled config. Runs locally or in CI (needs PyYAML); the platform itself reads only the result, after the register step has loaded it into the metadata DB (agreed 27 Oct: YAML in git -> validating loader -> DB). | 79 | none yet |
model.py | Plain records shared by the kernel and the adapters. | 52 | none yet |
ports.py | The ports slice 1 uses (data-platform-catalog dpc/ports.yaml). The kernel imports only these shapes. | 48 | none yet |
profiling.py | Profiling (X5, DPC-0051, B9): every loaded batch profiled at the lander, the dataset run's profile compared with the last one, and drift reported in the run log. Standard library only. | 81 | Reconciliation and profiling |
reconcile.py | Reconciliation between hops (DPC-0015, DPC-0016): the same measure, counted at each hop, must agree. | 65 | Reconciliation and profiling |
recovery.py | Withdrawing a bad batch after it was published (VOID-01, ADR-025). Product-free. | 24 | Recovery and housekeeping |
resilience.py | Resilience for calls to sources (X18, B7): retry with backoff, a circuit breaker per source, a connection cap per source. Standard library only; time comes in as functions (DPC-0032: time only from the clock port), so tests drive it. | 143 | Resilience |
retention.py | Retention (ADR-029, B8): how long each copy is kept, so a fault found late still has a correct copy to reload from. Pure functions over dates and settings; the landing adapter or the cloud's lifecycle rules carry the actions out. | 103 | Recovery and housekeeping |
source_ports.py | The four mock-source ports (goal G4, data-platform-catalog dpc/ports.yaml: source systems). | 107 | Sources and movers |
timing.py | End-to-end and hop-by-hop delay (ADR-034, B4): every hop stamps its own time; this turns the stamps into the 50th, 95th and 99th percentile per hop and end to end, each labelled measured or modeled. | 52 | none yet |
des
| Module | What it is for (its own first sentence) | Lines | Explained on |
|---|---|---|---|
des/__init__.py | Data expectation services (DES, ADR-024 set c): whether the data is right. Quality results, reconciliation, profiling, quarantine and release. Owned here, filled by session W1e. | 9 | Reconciliation and profiling |
des/dashboards.py | The DES screens (ADR-024 rule 1, set c): owned by data expectation services, not by observability. | 48 | Reconciliation and profiling |
gold
| Module | What it is for (its own first sentence) | Lines | Explained on |
|---|---|---|---|
gold/__init__.py | Gold (W1d): the gold harness (ADR-003, -004, -005, -006, -017, -018; D18), product-free. | 12 | Silver to gold |
gold/calendar.py | The date dimension (B15; gold modelling: one shared, role-playing date dimension). Pure and product-free. | 57 | Silver to gold |
gold/daybyday.py | Gold per world day: after the day's silver run, one gold run for the day's business date, then gold is compared with the oracle's gold for the day and the incidents gold owns are judged. | 67 | Silver to gold |
gold/harness.py | The gold harness (ADR-003): one run builds the domain's gold tables, dimensions first, then facts. | 480 | Silver to gold |
gold/inspect.py | Inspect mode (INSP-01): rebuild one window of a past gold run from what its run log names. Product-free. | 84 | Silver to gold |
gold/keys.py | Deterministic hash surrogate keys with reserved members (ADR-004, DPC-0081, DPC-0046). | 50 | Landing to bronze |
gold/plan.py | A gold table's plan, read from its DPM gold manifest (manifest/gold/<name>.yaml) and its ODCS contract. | 217 | Silver to gold |
gold/ports.py | The ports gold talks to (dpc/ports.yaml: table_writer for gold, and the engine operations the harness asks for). | 118 | Silver to gold |
gold/windows.py | Windows of a fact (ADR-006), when one is complete (ADR-018) and when a rebuild is a restatement (ADR-017). Pure. | 105 | Silver to gold |
governance
| Module | What it is for (its own first sentence) | Lines | Explained on |
|---|---|---|---|
governance/__init__.py | Governance (goal G6, concern X13): what the catalog must say, read from the contracts and the manifest. | 9 | Semantic layer |
governance/catalog.py | Catalog entries filled from the contracts and the manifest (DPC-0075; SERV-01, SEC-01, CAT-01). | 113 | Semantic layer |
governance/coverage.py | DAC coverage (B20 C20): every platform rule has a passing test or a written waiver. Pure. | 76 | Semantic layer |
governance/port.py | The catalog-state port (dpc/ports.yaml data_catalog): what the catalog holds now, so the sync plan can diff it. | 38 | Semantic layer |
governance/tables.py | Published tables, data products and the tag standard. | 221 | Semantic layer |
ingest
| Module | What it is for (its own first sentence) | Lines | Explained on |
|---|---|---|---|
ingest/__init__.py | Ingestion (W1b; ADR-002, D11): sources -> bronze, driven by the metadata store, every run logged in it. | 14 | Ingestion |
ingest/daybyday.py | The world and the platform, day by day: a world day is written into the mock sources, the push movers fill landing, one ingestion run takes everything new to bronze, and bronze is compared with the oracle's bronze totals for that day. | 141 | Ingestion |
ingest/discovery.py | Source discovery (B3): before a feed is onboarded, look at what actually arrives. For the client's own drops (format unknown, history completeness unknown) this answers: which formats, which windows, how many files and rows a day, and which | 66 | Ingestion |
ingest/pipelines.py | One ingestion pipeline per source type (ADR-002, D11). Each picks its batches, hands every one to the lander steps (steps.enter) and reports bookmarks, a watermark and what it noticed. They differ only in where batches come from: | 392 | Ingestion |
ingest/ports.py | The ports ingestion talks to (data-platform-catalog dpc/ports.yaml: landing_store, table_writer, tokenizer). | 87 | Ingestion |
ingest/runner.py | One ingestion run over a domain, every step recorded in the metadata store (ADR-010): | 219 | Ingestion |
ingest/steps.py | The lander steps: one library every way into bronze calls (ING-01, ADR-002, DPC-0037). A pushed file, a pulled copy, a page from an interface and a chunk of a stream all pass the same steps in the same order: | 397 | Ingestion |
meta
| Module | What it is for (its own first sentence) | Lines | Explained on |
|---|---|---|---|
meta/__init__.py | The metadata store (ADR-010): the port, the settings registry and the loader that fills the store from a manifest. | 9 | Metadata store |
meta/loader.py | The validating loader: a domain's manifest folder -> the metadata store (M3: domains are data; REL-01: only the loader writes configuration, and only from manifest files in version control). | 236 | Metadata store |
meta/port.py | The metadata store port (ADR-010; dpc/ports.yaml metadata_service and run_log). The kernel imports only these shapes. | 317 | Metadata store |
meta/registry.py | The settings registry (DPC-0034): every key a manifest may set, with its scope, type and default. | 39 | Metadata store |
obs
| Module | What it is for (its own first sentence) | Lines | Explained on |
|---|---|---|---|
obs/__init__.py | Observability (W3, goals G8 and G10; ADR-024): runs and promises. One run id everywhere, the ops tables and their metric views, freshness, pushed alerts, five dashboards. Owners judge (DES in dataplatform/des), observability reports. | 24 | Operations |
obs/alerts.py | Alerts that people still read (OPS-01, OPS-06, DPC-0019, DPC-0056). Rules are configuration; evaluation is pure. | 297 | Operations |
obs/clock.py | The clock port (DPC-0032): observability reads time only through this, so a simulated week runs in a second. | 40 | Operations |
obs/dashboards.py | The ops dashboards as definitions (G8, DPC-0057, ADR-024). Product-free: a realm compiles them to its BI tool's file (a dashboard-file adapter writes them into the realm's dashboards folder), and `evaluate` runs any of them on the ops table | 313 | Operations |
obs/explain.py | A run explained in plain words, deterministically: the run's report and ops tables in, a Markdown narrative out. | 180 | Operations |
obs/explain_meta.py | A dataset's metadata and the repository's checks explained in plain words, deterministically: no model, no network. | 229 | Operations |
obs/fixture.py | FIXTURE - a tiny fake runner that fills the run log until W1b-W1e produce real runs. | 221 | Operations |
obs/freshness.py | Freshness against the manifest's targets (FRESH-01, OPS-01, G10). Pure functions, plus one reader of the store. | 161 | Operations |
obs/lineage.py | Lineage before the engine's catalog (X7, LIN-01, B9): the engine's catalog records lineage from bronze on; the hops before it (source system -> mover -> landing -> bronze) are only in our file log. This turns file-log records into run event | 50 | Operations |
obs/monitor.py | Monitoring, pushed by runs (ADR-024 rule 4): nothing here runs on a fixed interval. | 123 | Operations |
obs/native.py | Platform-native screens and notifications, adopted, not rebuilt (ADR-024 rule 1, set a). | 46 | Operations |
obs/ops.py | The ops builder (G8, DPC-0057): run log + platform telemetry + freshness + alerts -> the compact ops tables. Pure. | 358 | Operations |
obs/ops_export.py | The ops tables as a typed, portable file, so a realm can publish them where its dashboards read (B13). | 84 | Operations |
obs/page.py | The local ops page (B10): one self-contained HTML file written after a local run, so a person can see what happened without a workspace. It reads only the ops tables (obs/ops.py, the same rows the ops metric views and dashboards read) plus | 121 | Operations |
obs/pipeline_events.py | Declarative pipelines report themselves (ADR-024 rule 5): the pipeline event log, behind one read-only port. | 38 | Operations |
obs/recorder.py | The run recorder: one id from start to finish (G8, OPS-03, DPC-0057). | 209 | Operations |
obs/runlog.py | The run log, read back for operations (DPC-0057): every run in a time window with its timestamps. | 106 | Operations |
obs/steps.py | The one step format (ADR-024 rule 2): every owner writes one result row per check to the run log. | 62 | Operations |
obs/success.py | The platform's success criteria and the scorecard a run computes against them (B20; CCA-P D4 4.1: define what good means first, one metric per dimension with a target, no blended score). Pure. | 168 | Operations |
obs/system.py | The platform's own records, behind one port: job runs, lineage, and access (ADR-024: platform telemetry adopted). | 102 | Operations |
obs/trace.py | The telemetry port: one id for a run, everywhere (G8). The run id IS the trace id. | 101 | Operations |
obs/traced_store.py | Live wiring of observability (B10, ADR-024): a metadata store that traces every run as it is recorded and pushes "run finished" to its hooks. The engine runners already record through the store port; wrapping the store wires them to the tra | 67 | Operations |
obs/views.py | Certified ops definitions (ADR-024 rule 7): ops metric views over the compact ops tables. Product-free. | 123 | Operations |
Deployable bundles (realms)
| Module | What it is for (its own first sentence) | Lines | Explained on |
|---|---|---|---|
realms/aws-databricks/generate_bundle.py | Generate the Databricks Asset Bundle for Databricks on AWS from the compiled manifests (M4: jobs are generated, never hand-written). | 213 | none yet |
realms/aws-databricks/integration.py | Integration test on a deployed environment (CI: after deploy dev, before the release gate). | 46 | none yet |
realms/aws-databricks/lakeformation.py | Lake Formation grants from the same access plan as Unity Catalog (ADR-028 rule 4, B12): AWS engines (Redshift, Athena) read the Unity Catalog tables through a Glue federated catalog, and Lake Formation decides what each reader sees. Permiss | 102 | none yet |
realms/aws-databricks/meta_setup.py | Install the metadata store (ADR-010) on a Lakebase instance and load a domain's manifest into it. | 94 | none yet |
realms/azure-databricks/generate_bundle.py | Generate the Databricks Asset Bundle that deploys the platform's engine job to an Azure Databricks workspace. | 96 | none yet |
realms/azure-databricks/tests/test_realm.py | Offline checks of the Azure Databricks realm: names, layers, secrets, budget, isolation, pinned versions, bundle. | 292 | none yet |
realms/databricks/dashboards/enable_anomaly_detection.py | Turn on the platform's anomaly detection (freshness and completeness, limits learned per table) on every schema of the silver and gold catalogs: the adopted half of "freshness" (obs/native.py `anomaly_detection`). Idempotent: a schema alrea | 49 | Operations page and dashboards |
realms/databricks/dashboards/generate.py | Write (or check) the ops dashboards, the ops metric views and the adopted native set (ADR-024). | 72 | Operations page and dashboards |
realms/databricks/declarative/compare.py | Compare the declarative silver (schema declarative_proof) with the job-built silver, dataset by dataset: current live rows (history) or all rows (append), version counts, and rows that differ on the business columns. | 29 | Declarative engine |
realms/databricks/declarative/silver.py | The declarative silver engine on Databricks (ADR-035): one generic definition for every dataset whose silver_engine is `declarative` (or the datasets named in dp.only, for proofs). | 87 | Declarative engine |
realms/databricks/engine/tasks/probe.py | What Databricks serverless allows the engine (13 Nov probe). Each check prints one line: ok or FAIL with the reason. Throwaway schema, dropped at the end. | 90 | Orchestration, end to end |
realms/databricks/engine/tasks/run_days.py | The chosen slice's world (--slice, default the tyre company), N days, on Databricks: the same day loop as the laptop (scripts/run_platform.py), on the job's own Spark session, Unity Catalog managed Iceberg tables (<env>_<layer>.<schema>.<ta | 100 | Orchestration, end to end |
realms/databricks/engine/tasks/semantic_layer.py | After a proof run: create the business metric views (the semantic generator's committed SQL for this environment) and ask each certified known answer of the DEPLOYED views, the way a dashboard tile or a Genie answer would. | 49 | Orchestration, end to end |
realms/databricks/pipelines/silver.py | Silver, as a Lakeflow Declarative Pipeline: maximum use of what Databricks offers, driven by the metadata DB. | 52 | none yet |
realms/databricks/semantic_governance/deploy_genie.py | Deploy the generated Genie space (B12, SERV-06): create it, or update the space with the same title. | 66 | Semantic layer |
realms/databricks/semantic_governance/generate.py | Generate the semantic layer, BI, catalog sync, access model and audit files for one domain's slice (cost belongs to the separate cost tool since 4 Nov 2026; the jobs keep the tag convention, governance.tables.COMPUTE_TAGS). | 200 | Semantic layer |
realms/databricks/semantic_governance/render.py | Databricks rendering of the kernel's semantic, BI, governance and access definitions (product names live here). | 379 | Semantic layer |
realms/databricks/tasks/checks.py | The day's checks on Databricks. Every result carries the run id, so the evidence view joins them to the run log. | 109 | none yet |
realms/databricks/tasks/main.py | Slice 1 job steps on Databricks. One entry file; the job calls it once per step. | 166 | none yet |
realms/databricks/v1/tasks/driver.py | The proof driver: the simulated tyre world, N days, against the DEPLOYED platform job. Each day it writes the day into the mock sources, plays the client's movers into the landing volume, triggers the platform job (ingestion, silver, gold a | 60 | none yet |
realms/databricks/v1/tasks/hop.py | One hop of the platform as a job task: ingestion, silver or gold for one business date (scripts/run_hop.py on the job's own Spark session, Unity Catalog managed Iceberg tables, the metadata store on Lakebase as the job's identity). Deliveri | 54 | none yet |
realms/databricks/v1/tasks/paths.py | Where the V1 jobs meet: the landing volume the feeder writes and ingestion reads, and the folders the proof driver and the platform tasks pass files through. One catalog per environment and layer (ADR-012). | 31 | none yet |
Scripts
| Module | What it is for (its own first sentence) | Lines | Explained on |
|---|---|---|---|
scripts/dac_coverage.py | DAC coverage report (B20 C20): which platform rules have a passing test, a waiver, or neither. | 55 | none yet |
scripts/demo_end_to_end.py | End-to-end local demo — Source → Bronze → Silver → Gold + lineage. | 1084 | none yet |
scripts/dp_compile.py | Compile a slice's DPM manifest, ODCS contract and DWD world into the JSON the platform loads. | 22 | none yet |
scripts/dp_compile_domain.py | Compile what the job generator needs to know about a domain from its DPM manifest folder. | 62 | none yet |
scripts/explain.py | The platform explains itself in plain words. No model and no network: the same inputs always give the same text. | 157 | none yet |
scripts/export_settings_registry.py | Export the settings registry for the metadata store (DPC-0034) from LakeIngestionFramework's key registry. | 64 | none yet |
scripts/platform_binding.py | The adapters a run picks, shared by the day loop (run_platform.py) and by one hop running as its own process (run_hop.py), so both bind the same engine, tables and metadata store from the same options. | 118 | none yet |
scripts/run_hop.py | One hop of the platform as its own process: ingestion, silver or gold for one business date. | 131 | Orchestration, end to end |
scripts/run_platform.py | Run the tyre world and the platform together, day by day, locally: world -> sources -> landing -> bronze -> silver. (--slice 03-clinical runs the clinical trial instead: the world in the slice picks the domain, models and gold judge.) | 745 | Orchestration, end to end |
scripts/run_world.py | Run the tyre world locally: N simulated days into the four mock sources, with the oracle's totals per day. | 214 | none yet |
scripts/show_postgres_delta.py | Demo inspection script: postgres → ADF Mimic → local Delta, with row dump. | 350 | none yet |
semantic
| Module | What it is for (its own first sentence) | Lines | Explained on |
|---|---|---|---|
semantic/__init__.py | The semantic layer and what reads it (goal G6, ADR-022): metric definitions and the experiences built on them. | 8 | Semantic layer |
semantic/answers.py | Ask a metric view's question of published gold (ADR-022 known answers, one number everywhere). | 45 | Semantic layer |
semantic/experiences.py | Dashboards and the conversation space, generated from the certified metric views (DPC-0076, SERV-04, SERV-06). | 133 | Semantic layer |
semantic/metrics.py | Metric views from the manifest's metric files (DPM v0.2 kind metric_view; ADR-022, DPC-0073, SERV-04, SERV-06). | 295 | Semantic layer |
silver
| Module | What it is for (its own first sentence) | Lines | Explained on |
|---|---|---|---|
silver/__init__.py | Silver (W1c): the thirteen standard stages from bronze to silver (ADR-007, -008, -013, -016), product-free. | 9 | Bronze to silver |
silver/baseline_writer.py | The baseline batch writer (B8, assumed decision A3): compact bronze up to the cut-off, write it back to bronze as one batch, replay it through the processors and compare with silver; log it only when it matches. | 115 | Bronze to silver |
silver/contract_rules.py | The contract's row rules as quality-stage rules (DAC DQ-01: rules written in the contract run; ADR-009: severity sets the action). Pure and product-free; the predicates are the engine-neutral SQL subset both engines take. | 67 | Bronze to silver |
silver/daybyday.py | Silver per world day: after the day's ingestion, one silver run takes the new bronze batches, and silver is compared with the oracle's silver totals for the day (current rows, versions, deletes marked, and the manifest's reconciliation meas | 211 | Bronze to silver |
silver/declarative.py | The declarative silver engine's recipe for one dataset (ADR-035, DAC ENG-02). Pure and product-free. | 70 | Bronze to silver |
silver/engines.py | The bronze-to-silver engines and what each can honour (ADR-035, DAC ENG-02). | 67 | Bronze to silver |
silver/history_contract.py | The SCD Type 2 contract every bronze-to-silver engine must pass (ADR-035, DAC ENG-02). Pure data, product-free. | 65 | Bronze to silver |
silver/live_recon.py | Live source against silver (DPC-0052, DAC REC-05): a source can lose rows silently (a change feed that skips changes during maintenance), so every day the live table is compared with silver's current rows. Pure and product-free. | 69 | Bronze to silver |
silver/plan.py | The silver plan: what the thirteen standard stages (ADR-007) do for one dataset, read from the metadata store's merged view of the dataset (manifest + contract columns). Pure and product-free (R1): no dataframe here, only settings. | 341 | Bronze to silver |
silver/ports.py | The ports silver talks to (data-platform-catalog dpc/ports.yaml: stage_runner, and table_writer for silver). | 80 | Bronze to silver |
silver/privacy_scan.py | Run the privacy scan over every enabled dataset's bronze and silver (B20 C15). Engine-free: frames come through the silver ports. Returns (layer, dataset.column, values in clear) per restricted column. | 35 | Bronze to silver |
silver/runner.py | The silver run: for each enabled dataset, the bronze batches loaded since silver last ran go through the thirteen stages and into silver, and every step is recorded in the metadata store under the run id (ADR-010). | 198 | Bronze to silver |
stages
| Module | What it is for (its own first sentence) | Lines | Explained on |
|---|---|---|---|
stages/__init__.py | The thirteen standard bronze-to-silver stages (ADR-007) as pure PySpark functions: DataFrames in, DataFrames out, no reads or writes (ADR-013). This is the in-house part of the stage_runner port the engine binding names (dbs/engine/databric | 79 | The thirteen stages |
stages/baseline.py | The baseline batch (B8, ADR-029, assumed decision A3): a compaction of bronze, not a copy of silver. | 83 | The thirteen stages |
stages/clean.py | Stages 1-9: contract check, projection, hygiene, parse and type, time, standardise, personal data, row filter, quality. Each is a pure function over a DataFrame (ADR-013); the settings come from the plan (ADR-007). | 230 | The thirteen stages |
stages/common.py | What every stage shares: its inputs, its output, and the quarantine record shape. | 114 | The thirteen stages |
stages/history.py | Stage 12: history by dataset kind (ADR-016), deletes soft. A pure function: new rows and silver as it stands in, the rows to write out. Every output row has a `_row_key` and an `_action`: `upsert` (write it) or `delete` (a stale row of a ke | 318 | The thirteen stages |
stages/privacy.py | The privacy scan (B20 C15, SEC-01): restricted values in clear. Pure PySpark (Spark, DuckDB, Databricks). | 20 | The thirteen stages |
stages/rows.py | Stages 10, 11 and 13: duplicates, keys, provenance. Pure functions over DataFrames (ADR-013). | 88 | The thirteen stages |
twin
| Module | What it is for (its own first sentence) | Lines | Explained on |
|---|---|---|---|
twin/__init__.py | DataTwin (goal G10): the tyre world's incidents as end-to-end scenarios, and the release gate. | 8 | Tests and proof |
twin/engine.py | An engine layer's run of the world, attached to the DataTwin harness so the scenarios' `engine` checks run. | 74 | Tests and proof |
twin/gate.py | The release gate (G10, TEST-03): run a change on clones of prod, compare against the oracle and prod, pass or block. | 210 | Tests and proof |
twin/learning.py | The learning loop (G10): every production failure becomes one more scenario. | 83 | Tests and proof |
twin/runner.py | The DataTwin scenario runner and its oracle checks (G10, TEST-05, SIM-01). | 197 | Tests and proof |
twin/scenario.py | A DataTwin scenario as a file: `apiVersion: dwd-scenario/v0` (goal G10, TEST-05). | 110 | Tests and proof |
world
| Module | What it is for (its own first sentence) | Lines | Explained on |
|---|---|---|---|
world/__init__.py | The data world (DWD): a pretend source that sends data on a clock and plants incidents on a calendar. | 1 | The simulated company |
world/clock.py | The simulated clock (DPC-0032: jobs read time only through the clock port). A world day is one business day in the business time zone; day 1 starts at the DWD's clock.start. | 38 | The simulated company |
world/domains/__init__.py | Domain models: how one simulated company's source systems behave. The DWD's `name` picks the model. | 43 | The simulated company |
world/domains/clinical.py | The clinical trial (data-platform-catalog slices/03-clinical/world.yaml): six source systems, day by day. | 890 | The simulated company |
world/domains/clinical_gold.py | The clinical trial's gold judge: published gold against the oracle's gold for the day, and the incidents gold owns. | 228 | The simulated company |
world/domains/clinical_oracle.py | The clinical trial's oracle: per simulated day, what a correct platform holds at bronze, silver and gold, and what each certified metric must say. Computed only from the world's own truth (R9, SIM-01). | 256 | The simulated company |
world/domains/tyre.py | The tyre company (data-platform-catalog slices/02-tyre/world.yaml): six source systems, day by day. | 839 | The simulated company |
world/domains/tyre_gold.py | The tyre world's gold judge: published gold against the oracle's gold for the day, and the incidents gold owns. | 245 | The simulated company |
world/domains/tyre_oracle.py | The tyre world's oracle: per simulated day, what a correct platform holds at bronze, silver and gold, and what each certified metric must say. Computed only from the world's own truth (R9, SIM-01). | 282 | The simulated company |
world/dwd.py | Reads a DWD world (dwd/v0) and the specification it cites: dataset manifests and their ODCS contracts. | 155 | The simulated company |
world/engine.py | The world engine: runs the clock a day at a time, asks the domain model what its source systems did that day, writes it into the mock sources through their ports, and saves the oracle's totals for the day. | 151 | The simulated company |
world/oracle.py | The oracle: what silver must hold, computed from the world's own record, never from the platform's logs. | 43 | The simulated company |
world/shop.py | The clothing shop's orders database, as a change feed, for slice 1 (data-platform-catalog slices/01-orders/world.yaml). | 155 | The simulated company |