cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Code map › Code index

Data PlatformZoom level 2

Code index: every module, what it is for, and where it is explained

Every Python module of the platform, grouped by package. Each row gives the module's own one-line description, its size, a link to the file, and the page that explains it. The list is generated from the code when the site is built, so it cannot drift.

192 modules in 16 groups.

GroupModulesLinesCaller pages
access2212access
adapters475,581adapters
Top-level modules181,260Top-level modules
des257des
gold91,190gold
governance5457governance
ingest71,316ingest
meta4601meta
obs233,223obs
Deployable bundles (realms)222,436Deployable bundles (realms)
Scripts113,002Scripts
semantic4481semantic
silver121,327silver
stages7932stages
twin6682twin
world133,326world

access

ModuleWhat it is for (its own first sentence)LinesExplained on
access/__init__.pyAccess model as code (DPC-0066; SEC-02, ENV-02; dpc/ports.yaml access_control).7Semantic layer
access/model.pyGroups and classification tags -> grants, column masks and row filters.205Semantic layer

adapters

ModuleWhat it is for (its own first sentence)LinesExplained on
adapters/__init__.pyAdapters: one package per product family. Only adapters import a product.9none yet
adapters/azure_blob.pyObject storage on Azure Blob Storage (cloud layer): landing_store, the partner file drop and object_store. Needs azure-storage-blob (the `azure` extra) only when a real client is built; nothing here imports an Azure SDK. The container is pa171none yet
adapters/customer_runlog.pyEXAMPLE SKELETON: a customer's own run log on the run-log reader port (level 2 of ADR-024 rule 6).37none yet
adapters/databricks_clone.pyThe release gate's ports (dataplatform.twin.gate) on Databricks: Delta SHALLOW CLONE and one measuring query.70none yet
adapters/databricks_native.pyDatabricks bindings for the adopted platform-native screens and notifications (dataplatform.obs.native).66none yet
adapters/databricks_notify.pyThe notifier port (dataplatform.obs.monitor.Notifier) through the workspace's notification destinations.46none yet
adapters/databricks_ops_views.pyThe ops metric views (dataplatform.obs.views) as Databricks metric views: one `CREATE ... WITH METRICS` per view.54none yet
adapters/databricks_pipeline_events.pyThe pipeline event-log port (dataplatform.obs.pipeline_events) on Lakeflow Declarative Pipelines' event log.67none yet
adapters/databricks_system.pyThe system-records port (dataplatform.obs.system) on Databricks system tables.105none yet
adapters/delta_bronze.pyThe bronze writer on Delta tables in a local folder, through delta-rs (`deltalake`, Apache 2.0) and pyarrow: no JVM, no Spark, the same approach as LakeIngestionFramework's LocalDeltaWriter (core/writers/local_delta.py), which only appends;136Landing to bronze
adapters/duckdb_frames.pyDuckDB as a frame engine for the pure stage and build code, through SQLFrame (the PySpark DataFrame API compiled to SQL by sqlglot): one logic, several engines (docs/DUCKDB-SPIKE.md). For laptop and CI runs only; nothing here ever writes a 61none yet
adapters/duckdb_gold.pyGold's engine operations on DuckDB through SQLFrame (docs/DUCKDB-SPIKE.md): the Spark adapter's operations, with the five engine-specific pieces replaced. For laptop and CI runs only; Databricks keeps adapters/spark_gold.py.165The harness
adapters/duckdb_silver.pySilver on DuckDB (through SQLFrame) and local Delta tables (through delta-rs): the same stages (dataplatform.stages) as the Spark adapter, other IO (docs/DUCKDB-SPIKE.md). For laptop and CI runs only: nothing here writes a governed table on137Writer
adapters/file_drop.pyFileDrop on a local folder: the partner and lab drop zones. Object storage is a later adapter with the same port.45Sources and movers
adapters/files.pylanding_store on a plain folder. On Databricks the folder is a Unity Catalog volume path, which serverless Python reads and writes as ordinary files, so the same adapter serves both.100none yet
adapters/http_api.pyRecordInterface as a small local HTTP service: the dealer management system.127Sources and movers
adapters/keyed_tokens.pyThe tokenizer as a keyed hash (HMAC-SHA256, standard library): the same value always gives the same token, so joins and duplicate checks still work, and nobody without the key can reverse or recompute it. The key is the binding's secret (DP37none yet
adapters/kinesis.pystream_bus on Amazon Kinesis Data Streams (cloud layer, dbs/cloud/aws.yaml). Needs boto3; the client is passed in, so tests use a local stand-in (moto) and the platform passes one built from the job's role.68Sources and movers
adapters/lakebase.pyThe metadata store on Lakebase, Databricks' managed PostgreSQL (ADR-010, binding dbs/engine/databricks.yaml).92Metadata store
adapters/lakeview.pyThe ops dashboards (dataplatform.obs.dashboards) as Databricks AI/BI (Lakeview) dashboard files.159none yet
adapters/local_movers.pyThe push movers for the local stack: they stand in for what a cloud supplies (a change-capture service, a file transfer service) and meet the landing contract every push mover meets (DPC-0038): each delivery written once, under a unique nam112Sources and movers
adapters/local_stream.pyEventStream on local files: a log of shards, each an append-only JSON-lines file. A managed stream is a later adapter with the same port; the shard count and the partition-key hashing mirror how such streams spread records.73Sources and movers
adapters/memory.pyIn-memory adapters for tests. Same ports as the Spark/Delta ones.41none yet
adapters/memory_bronze.pyThe bronze writer in memory, for tests: the reference behaviour the table-format adapters are tested against (M5).48none yet
adapters/memory_catalog.pyCatalog state in memory: the reference adapter for the CatalogState port (tests, dry runs, the first deploy).20none yet
adapters/memory_meta.pyThe metadata store in memory, for unit tests. Same port and the same refusals as the PostgreSQL adapter; the contract test table (tests/dataplatform/test_meta_store.py) runs every row against both (M5).391Metadata store
adapters/memory_pipeline_events.pyThe pipeline event-log port (dataplatform.obs.pipeline_events) in memory: rows added by tests.16none yet
adapters/memory_runlog.pyStampingRunLog: any metadata store, plus the run-log reader port (dataplatform.obs.runlog) in process.201none yet
adapters/memory_sources.pyThe four source ports in memory: the reference behaviour every real adapter is tested against (M5).109none yet
adapters/memory_system.pyThe system-records port (dataplatform.obs.system) in memory: rows added by tests or the fixture runner.26none yet
adapters/memory_tables.pyThe release gate's ports (dataplatform.twin.gate) in memory: tables are lists of rows; a clone is a copy of the list (a shallow clone shares data until written; in memory a copy behaves the same for the gate).44none yet
adapters/null_runlog.pyThe run-log reader port (dataplatform.obs.runlog) with no run log connected: level 1 of ADR-024 rule 6.16none yet
adapters/otel_tracer.pyThe telemetry port (dataplatform.obs.trace) on the OpenTelemetry API.77none yet
adapters/postgres_meta.pyThe metadata store on PostgreSQL (ADR-010). Every read and write is one call to a function in adapters/sql/postgres_meta.sql; this file holds no table names (DPC-0033, checked by tests/dataplatform/test_meta_store.py).258Metadata store
adapters/postgres_runlog.pyThe run-log reader port (dataplatform.obs.runlog) on PostgreSQL: one call to runtime.ops_run_log (sql/postgres_obs.sql), installed beside the metadata store's own schema (postgres_meta.sql). No table names here (DPC-0033). Timestamps are th61none yet
adapters/postgres_source.pySourceDatabase on PostgreSQL: the mock ERP and MES databases, one schema per system.55Sources and movers
adapters/s3.pyObject storage on Amazon S3 (cloud layer, dbs/cloud/aws.yaml): landing_store, the partner file drop and object_store. Needs boto3 (the `s3` extra). The client is passed in, so tests use a local stand-in (moto) and the platform passes one bu142Sources and movers
adapters/secrets.pysecret_store adapters (cloud_ports.SecretStore, DPC-0029). The value is returned to the caller only; nothing here logs it, and an error names the reference, never the value.88none yet
adapters/spark_delta.pyThe slice 1 ports on Spark + Delta tables: Databricks (serverless, Spark Connect) or open-source Spark locally.203none yet
adapters/spark_gold.pyGold on Spark and Delta tables: the harness's engine operations (PySpark) and the gold read and write edges.499The harness
adapters/spark_ops_tables.pyPublish the ops tables (obs/ops_export.py file) as catalog tables, then the ops metric views over them (B13).47none yet
adapters/spark_silver.pySilver on Spark and Delta tables: the stage runner (dataplatform.stages on PySpark) and the read and write edges.382Writer
adapters/sql_source.pySourceDatabase over any DB-API connection. Two thin adapters use it: postgres_source.py and sqlite_source.py.184none yet
adapters/sqlite_source.pySourceDatabase on SQLite (standard library): for tests and for a laptop with no Postgres. SQLite has no schemas, so a system's tables are prefixed with its name: erp_db__sales_order_line.30Sources and movers
adapters/store_registry.pySlice 1's batch registry (dataplatform.ports.MetadataStore: batch_status(key), record_batch) on the W1a metadata store, so slice 1's lander (dataplatform.lander) records into the same file log as W1b's ingestion, through the store's procedu42none yet
adapters/table_store.pyThe table port's engine side (data-platform-catalog dpc/ports.yaml table_writer, ADR-027): one small adapter per open table format, so silver, gold and bronze code never names a format.598none yet
adapters/uc_catalog.pyCatalog state read from Unity Catalog's information schema (Databricks; dbs/engine/databricks.yaml data_catalog).66none yet

Top-level modules

ModuleWhat it is for (its own first sentence)LinesExplained on
__init__.pydataplatform: the realm-agnostic kernel for the Clean Data Engineering specs.17none yet
cloud_ports.pyPorts only a cloud fills (data-platform-catalog dpc/ports.yaml, binds_in: cloud). The kernel imports only these shapes; product names stay in the adapters (R1). The source-side ports (file drop, event stream, database, record interface) are48none yet
contract.pyReading an ODCS v3.2.0 contract: the columns a delivery must carry, and the quality rules as row predicates.77none yet
hop_reports.pyA hop's report as plain data, so a hop that runs as its own job can hand its report to whoever judges it later.68none yet
housekeeping.pyWhat the housekeeping job may remove from bronze (B8; DAC RET-01, DPC-0116). Pure and product-free: it decides, adapters act (archive, delete by batch key, expire snapshots) and the run log records.178Recovery and housekeeping
keys.pyBatch keys (DPC-0024) and config fingerprints (CFG-02).20Landing to bronze
lander.pyThe lander: landing -> bronze. Our own code (decision D11); the mover that filled landing is swappable.73Landing to bronze
leases.pyWriter leases (ORCH-04): two runs never write the same table at once. Product-free; the metadata store keeps the lease (runtime.table_lease), so it holds across jobs, clusters and laptops.25none yet
manifest.pyDPM manifest -> compiled config. Runs locally or in CI (needs PyYAML); the platform itself reads only the result, after the register step has loaded it into the metadata DB (agreed 27 Oct: YAML in git -> validating loader -> DB).79none yet
model.pyPlain records shared by the kernel and the adapters.52none yet
ports.pyThe ports slice 1 uses (data-platform-catalog dpc/ports.yaml). The kernel imports only these shapes.48none yet
profiling.pyProfiling (X5, DPC-0051, B9): every loaded batch profiled at the lander, the dataset run's profile compared with the last one, and drift reported in the run log. Standard library only.81Reconciliation and profiling
reconcile.pyReconciliation between hops (DPC-0015, DPC-0016): the same measure, counted at each hop, must agree.65Reconciliation and profiling
recovery.pyWithdrawing a bad batch after it was published (VOID-01, ADR-025). Product-free.24Recovery and housekeeping
resilience.pyResilience for calls to sources (X18, B7): retry with backoff, a circuit breaker per source, a connection cap per source. Standard library only; time comes in as functions (DPC-0032: time only from the clock port), so tests drive it.143Resilience
retention.pyRetention (ADR-029, B8): how long each copy is kept, so a fault found late still has a correct copy to reload from. Pure functions over dates and settings; the landing adapter or the cloud's lifecycle rules carry the actions out.103Recovery and housekeeping
source_ports.pyThe four mock-source ports (goal G4, data-platform-catalog dpc/ports.yaml: source systems).107Sources and movers
timing.pyEnd-to-end and hop-by-hop delay (ADR-034, B4): every hop stamps its own time; this turns the stamps into the 50th, 95th and 99th percentile per hop and end to end, each labelled measured or modeled.52none yet

des

ModuleWhat it is for (its own first sentence)LinesExplained on
des/__init__.pyData expectation services (DES, ADR-024 set c): whether the data is right. Quality results, reconciliation, profiling, quarantine and release. Owned here, filled by session W1e.9Reconciliation and profiling
des/dashboards.pyThe DES screens (ADR-024 rule 1, set c): owned by data expectation services, not by observability.48Reconciliation and profiling

gold

ModuleWhat it is for (its own first sentence)LinesExplained on
gold/__init__.pyGold (W1d): the gold harness (ADR-003, -004, -005, -006, -017, -018; D18), product-free.12Silver to gold
gold/calendar.pyThe date dimension (B15; gold modelling: one shared, role-playing date dimension). Pure and product-free.57Silver to gold
gold/daybyday.pyGold per world day: after the day's silver run, one gold run for the day's business date, then gold is compared with the oracle's gold for the day and the incidents gold owns are judged.67Silver to gold
gold/harness.pyThe gold harness (ADR-003): one run builds the domain's gold tables, dimensions first, then facts.480Silver to gold
gold/inspect.pyInspect mode (INSP-01): rebuild one window of a past gold run from what its run log names. Product-free.84Silver to gold
gold/keys.pyDeterministic hash surrogate keys with reserved members (ADR-004, DPC-0081, DPC-0046).50Landing to bronze
gold/plan.pyA gold table's plan, read from its DPM gold manifest (manifest/gold/<name>.yaml) and its ODCS contract.217Silver to gold
gold/ports.pyThe ports gold talks to (dpc/ports.yaml: table_writer for gold, and the engine operations the harness asks for).118Silver to gold
gold/windows.pyWindows of a fact (ADR-006), when one is complete (ADR-018) and when a rebuild is a restatement (ADR-017). Pure.105Silver to gold

governance

ModuleWhat it is for (its own first sentence)LinesExplained on
governance/__init__.pyGovernance (goal G6, concern X13): what the catalog must say, read from the contracts and the manifest.9Semantic layer
governance/catalog.pyCatalog entries filled from the contracts and the manifest (DPC-0075; SERV-01, SEC-01, CAT-01).113Semantic layer
governance/coverage.pyDAC coverage (B20 C20): every platform rule has a passing test or a written waiver. Pure.76Semantic layer
governance/port.pyThe catalog-state port (dpc/ports.yaml data_catalog): what the catalog holds now, so the sync plan can diff it.38Semantic layer
governance/tables.pyPublished tables, data products and the tag standard.221Semantic layer

ingest

ModuleWhat it is for (its own first sentence)LinesExplained on
ingest/__init__.pyIngestion (W1b; ADR-002, D11): sources -> bronze, driven by the metadata store, every run logged in it.14Ingestion
ingest/daybyday.pyThe world and the platform, day by day: a world day is written into the mock sources, the push movers fill landing, one ingestion run takes everything new to bronze, and bronze is compared with the oracle's bronze totals for that day.141Ingestion
ingest/discovery.pySource discovery (B3): before a feed is onboarded, look at what actually arrives. For the client's own drops (format unknown, history completeness unknown) this answers: which formats, which windows, how many files and rows a day, and which66Ingestion
ingest/pipelines.pyOne ingestion pipeline per source type (ADR-002, D11). Each picks its batches, hands every one to the lander steps (steps.enter) and reports bookmarks, a watermark and what it noticed. They differ only in where batches come from:392Ingestion
ingest/ports.pyThe ports ingestion talks to (data-platform-catalog dpc/ports.yaml: landing_store, table_writer, tokenizer).87Ingestion
ingest/runner.pyOne ingestion run over a domain, every step recorded in the metadata store (ADR-010):219Ingestion
ingest/steps.pyThe lander steps: one library every way into bronze calls (ING-01, ADR-002, DPC-0037). A pushed file, a pulled copy, a page from an interface and a chunk of a stream all pass the same steps in the same order:397Ingestion

meta

ModuleWhat it is for (its own first sentence)LinesExplained on
meta/__init__.pyThe metadata store (ADR-010): the port, the settings registry and the loader that fills the store from a manifest.9Metadata store
meta/loader.pyThe validating loader: a domain's manifest folder -> the metadata store (M3: domains are data; REL-01: only the loader writes configuration, and only from manifest files in version control).236Metadata store
meta/port.pyThe metadata store port (ADR-010; dpc/ports.yaml metadata_service and run_log). The kernel imports only these shapes.317Metadata store
meta/registry.pyThe settings registry (DPC-0034): every key a manifest may set, with its scope, type and default.39Metadata store

obs

ModuleWhat it is for (its own first sentence)LinesExplained on
obs/__init__.pyObservability (W3, goals G8 and G10; ADR-024): runs and promises. One run id everywhere, the ops tables and their metric views, freshness, pushed alerts, five dashboards. Owners judge (DES in dataplatform/des), observability reports.24Operations
obs/alerts.pyAlerts that people still read (OPS-01, OPS-06, DPC-0019, DPC-0056). Rules are configuration; evaluation is pure.297Operations
obs/clock.pyThe clock port (DPC-0032): observability reads time only through this, so a simulated week runs in a second.40Operations
obs/dashboards.pyThe ops dashboards as definitions (G8, DPC-0057, ADR-024). Product-free: a realm compiles them to its BI tool's file (a dashboard-file adapter writes them into the realm's dashboards folder), and `evaluate` runs any of them on the ops table313Operations
obs/explain.pyA run explained in plain words, deterministically: the run's report and ops tables in, a Markdown narrative out.180Operations
obs/explain_meta.pyA dataset's metadata and the repository's checks explained in plain words, deterministically: no model, no network.229Operations
obs/fixture.pyFIXTURE - a tiny fake runner that fills the run log until W1b-W1e produce real runs.221Operations
obs/freshness.pyFreshness against the manifest's targets (FRESH-01, OPS-01, G10). Pure functions, plus one reader of the store.161Operations
obs/lineage.pyLineage before the engine's catalog (X7, LIN-01, B9): the engine's catalog records lineage from bronze on; the hops before it (source system -> mover -> landing -> bronze) are only in our file log. This turns file-log records into run event50Operations
obs/monitor.pyMonitoring, pushed by runs (ADR-024 rule 4): nothing here runs on a fixed interval.123Operations
obs/native.pyPlatform-native screens and notifications, adopted, not rebuilt (ADR-024 rule 1, set a).46Operations
obs/ops.pyThe ops builder (G8, DPC-0057): run log + platform telemetry + freshness + alerts -> the compact ops tables. Pure.358Operations
obs/ops_export.pyThe ops tables as a typed, portable file, so a realm can publish them where its dashboards read (B13).84Operations
obs/page.pyThe local ops page (B10): one self-contained HTML file written after a local run, so a person can see what happened without a workspace. It reads only the ops tables (obs/ops.py, the same rows the ops metric views and dashboards read) plus 121Operations
obs/pipeline_events.pyDeclarative pipelines report themselves (ADR-024 rule 5): the pipeline event log, behind one read-only port.38Operations
obs/recorder.pyThe run recorder: one id from start to finish (G8, OPS-03, DPC-0057).209Operations
obs/runlog.pyThe run log, read back for operations (DPC-0057): every run in a time window with its timestamps.106Operations
obs/steps.pyThe one step format (ADR-024 rule 2): every owner writes one result row per check to the run log.62Operations
obs/success.pyThe platform's success criteria and the scorecard a run computes against them (B20; CCA-P D4 4.1: define what good means first, one metric per dimension with a target, no blended score). Pure.168Operations
obs/system.pyThe platform's own records, behind one port: job runs, lineage, and access (ADR-024: platform telemetry adopted).102Operations
obs/trace.pyThe telemetry port: one id for a run, everywhere (G8). The run id IS the trace id.101Operations
obs/traced_store.pyLive wiring of observability (B10, ADR-024): a metadata store that traces every run as it is recorded and pushes "run finished" to its hooks. The engine runners already record through the store port; wrapping the store wires them to the tra67Operations
obs/views.pyCertified ops definitions (ADR-024 rule 7): ops metric views over the compact ops tables. Product-free.123Operations

Deployable bundles (realms)

ModuleWhat it is for (its own first sentence)LinesExplained on
realms/aws-databricks/generate_bundle.pyGenerate the Databricks Asset Bundle for Databricks on AWS from the compiled manifests (M4: jobs are generated, never hand-written).213none yet
realms/aws-databricks/integration.pyIntegration test on a deployed environment (CI: after deploy dev, before the release gate).46none yet
realms/aws-databricks/lakeformation.pyLake Formation grants from the same access plan as Unity Catalog (ADR-028 rule 4, B12): AWS engines (Redshift, Athena) read the Unity Catalog tables through a Glue federated catalog, and Lake Formation decides what each reader sees. Permiss102none yet
realms/aws-databricks/meta_setup.pyInstall the metadata store (ADR-010) on a Lakebase instance and load a domain's manifest into it.94none yet
realms/azure-databricks/generate_bundle.pyGenerate the Databricks Asset Bundle that deploys the platform's engine job to an Azure Databricks workspace.96none yet
realms/azure-databricks/tests/test_realm.pyOffline checks of the Azure Databricks realm: names, layers, secrets, budget, isolation, pinned versions, bundle.292none yet
realms/databricks/dashboards/enable_anomaly_detection.pyTurn on the platform's anomaly detection (freshness and completeness, limits learned per table) on every schema of the silver and gold catalogs: the adopted half of "freshness" (obs/native.py `anomaly_detection`). Idempotent: a schema alrea49Operations page and dashboards
realms/databricks/dashboards/generate.pyWrite (or check) the ops dashboards, the ops metric views and the adopted native set (ADR-024).72Operations page and dashboards
realms/databricks/declarative/compare.pyCompare the declarative silver (schema declarative_proof) with the job-built silver, dataset by dataset: current live rows (history) or all rows (append), version counts, and rows that differ on the business columns.29Declarative engine
realms/databricks/declarative/silver.pyThe declarative silver engine on Databricks (ADR-035): one generic definition for every dataset whose silver_engine is `declarative` (or the datasets named in dp.only, for proofs).87Declarative engine
realms/databricks/engine/tasks/probe.pyWhat Databricks serverless allows the engine (13 Nov probe). Each check prints one line: ok or FAIL with the reason. Throwaway schema, dropped at the end.90Orchestration, end to end
realms/databricks/engine/tasks/run_days.pyThe chosen slice's world (--slice, default the tyre company), N days, on Databricks: the same day loop as the laptop (scripts/run_platform.py), on the job's own Spark session, Unity Catalog managed Iceberg tables (<env>_<layer>.<schema>.<ta100Orchestration, end to end
realms/databricks/engine/tasks/semantic_layer.pyAfter a proof run: create the business metric views (the semantic generator's committed SQL for this environment) and ask each certified known answer of the DEPLOYED views, the way a dashboard tile or a Genie answer would.49Orchestration, end to end
realms/databricks/pipelines/silver.pySilver, as a Lakeflow Declarative Pipeline: maximum use of what Databricks offers, driven by the metadata DB.52none yet
realms/databricks/semantic_governance/deploy_genie.pyDeploy the generated Genie space (B12, SERV-06): create it, or update the space with the same title.66Semantic layer
realms/databricks/semantic_governance/generate.pyGenerate the semantic layer, BI, catalog sync, access model and audit files for one domain's slice (cost belongs to the separate cost tool since 4 Nov 2026; the jobs keep the tag convention, governance.tables.COMPUTE_TAGS).200Semantic layer
realms/databricks/semantic_governance/render.pyDatabricks rendering of the kernel's semantic, BI, governance and access definitions (product names live here).379Semantic layer
realms/databricks/tasks/checks.pyThe day's checks on Databricks. Every result carries the run id, so the evidence view joins them to the run log.109none yet
realms/databricks/tasks/main.pySlice 1 job steps on Databricks. One entry file; the job calls it once per step.166none yet
realms/databricks/v1/tasks/driver.pyThe proof driver: the simulated tyre world, N days, against the DEPLOYED platform job. Each day it writes the day into the mock sources, plays the client's movers into the landing volume, triggers the platform job (ingestion, silver, gold a60none yet
realms/databricks/v1/tasks/hop.pyOne hop of the platform as a job task: ingestion, silver or gold for one business date (scripts/run_hop.py on the job's own Spark session, Unity Catalog managed Iceberg tables, the metadata store on Lakebase as the job's identity). Deliveri54none yet
realms/databricks/v1/tasks/paths.pyWhere the V1 jobs meet: the landing volume the feeder writes and ingestion reads, and the folders the proof driver and the platform tasks pass files through. One catalog per environment and layer (ADR-012).31none yet

Scripts

ModuleWhat it is for (its own first sentence)LinesExplained on
scripts/dac_coverage.pyDAC coverage report (B20 C20): which platform rules have a passing test, a waiver, or neither.55none yet
scripts/demo_end_to_end.pyEnd-to-end local demo — Source → Bronze → Silver → Gold + lineage.1084none yet
scripts/dp_compile.pyCompile a slice's DPM manifest, ODCS contract and DWD world into the JSON the platform loads.22none yet
scripts/dp_compile_domain.pyCompile what the job generator needs to know about a domain from its DPM manifest folder.62none yet
scripts/explain.pyThe platform explains itself in plain words. No model and no network: the same inputs always give the same text.157none yet
scripts/export_settings_registry.pyExport the settings registry for the metadata store (DPC-0034) from LakeIngestionFramework's key registry.64none yet
scripts/platform_binding.pyThe adapters a run picks, shared by the day loop (run_platform.py) and by one hop running as its own process (run_hop.py), so both bind the same engine, tables and metadata store from the same options.118none yet
scripts/run_hop.pyOne hop of the platform as its own process: ingestion, silver or gold for one business date.131Orchestration, end to end
scripts/run_platform.pyRun the tyre world and the platform together, day by day, locally: world -> sources -> landing -> bronze -> silver. (--slice 03-clinical runs the clinical trial instead: the world in the slice picks the domain, models and gold judge.)745Orchestration, end to end
scripts/run_world.pyRun the tyre world locally: N simulated days into the four mock sources, with the oracle's totals per day.214none yet
scripts/show_postgres_delta.pyDemo inspection script: postgres → ADF Mimic → local Delta, with row dump.350none yet

semantic

ModuleWhat it is for (its own first sentence)LinesExplained on
semantic/__init__.pyThe semantic layer and what reads it (goal G6, ADR-022): metric definitions and the experiences built on them.8Semantic layer
semantic/answers.pyAsk a metric view's question of published gold (ADR-022 known answers, one number everywhere).45Semantic layer
semantic/experiences.pyDashboards and the conversation space, generated from the certified metric views (DPC-0076, SERV-04, SERV-06).133Semantic layer
semantic/metrics.pyMetric views from the manifest's metric files (DPM v0.2 kind metric_view; ADR-022, DPC-0073, SERV-04, SERV-06).295Semantic layer

silver

ModuleWhat it is for (its own first sentence)LinesExplained on
silver/__init__.pySilver (W1c): the thirteen standard stages from bronze to silver (ADR-007, -008, -013, -016), product-free.9Bronze to silver
silver/baseline_writer.pyThe baseline batch writer (B8, assumed decision A3): compact bronze up to the cut-off, write it back to bronze as one batch, replay it through the processors and compare with silver; log it only when it matches.115Bronze to silver
silver/contract_rules.pyThe contract's row rules as quality-stage rules (DAC DQ-01: rules written in the contract run; ADR-009: severity sets the action). Pure and product-free; the predicates are the engine-neutral SQL subset both engines take.67Bronze to silver
silver/daybyday.pySilver per world day: after the day's ingestion, one silver run takes the new bronze batches, and silver is compared with the oracle's silver totals for the day (current rows, versions, deletes marked, and the manifest's reconciliation meas211Bronze to silver
silver/declarative.pyThe declarative silver engine's recipe for one dataset (ADR-035, DAC ENG-02). Pure and product-free.70Bronze to silver
silver/engines.pyThe bronze-to-silver engines and what each can honour (ADR-035, DAC ENG-02).67Bronze to silver
silver/history_contract.pyThe SCD Type 2 contract every bronze-to-silver engine must pass (ADR-035, DAC ENG-02). Pure data, product-free.65Bronze to silver
silver/live_recon.pyLive source against silver (DPC-0052, DAC REC-05): a source can lose rows silently (a change feed that skips changes during maintenance), so every day the live table is compared with silver's current rows. Pure and product-free.69Bronze to silver
silver/plan.pyThe silver plan: what the thirteen standard stages (ADR-007) do for one dataset, read from the metadata store's merged view of the dataset (manifest + contract columns). Pure and product-free (R1): no dataframe here, only settings.341Bronze to silver
silver/ports.pyThe ports silver talks to (data-platform-catalog dpc/ports.yaml: stage_runner, and table_writer for silver).80Bronze to silver
silver/privacy_scan.pyRun the privacy scan over every enabled dataset's bronze and silver (B20 C15). Engine-free: frames come through the silver ports. Returns (layer, dataset.column, values in clear) per restricted column.35Bronze to silver
silver/runner.pyThe silver run: for each enabled dataset, the bronze batches loaded since silver last ran go through the thirteen stages and into silver, and every step is recorded in the metadata store under the run id (ADR-010).198Bronze to silver

stages

ModuleWhat it is for (its own first sentence)LinesExplained on
stages/__init__.pyThe thirteen standard bronze-to-silver stages (ADR-007) as pure PySpark functions: DataFrames in, DataFrames out, no reads or writes (ADR-013). This is the in-house part of the stage_runner port the engine binding names (dbs/engine/databric79The thirteen stages
stages/baseline.pyThe baseline batch (B8, ADR-029, assumed decision A3): a compaction of bronze, not a copy of silver.83The thirteen stages
stages/clean.pyStages 1-9: contract check, projection, hygiene, parse and type, time, standardise, personal data, row filter, quality. Each is a pure function over a DataFrame (ADR-013); the settings come from the plan (ADR-007).230The thirteen stages
stages/common.pyWhat every stage shares: its inputs, its output, and the quarantine record shape.114The thirteen stages
stages/history.pyStage 12: history by dataset kind (ADR-016), deletes soft. A pure function: new rows and silver as it stands in, the rows to write out. Every output row has a `_row_key` and an `_action`: `upsert` (write it) or `delete` (a stale row of a ke318The thirteen stages
stages/privacy.pyThe privacy scan (B20 C15, SEC-01): restricted values in clear. Pure PySpark (Spark, DuckDB, Databricks).20The thirteen stages
stages/rows.pyStages 10, 11 and 13: duplicates, keys, provenance. Pure functions over DataFrames (ADR-013).88The thirteen stages

twin

ModuleWhat it is for (its own first sentence)LinesExplained on
twin/__init__.pyDataTwin (goal G10): the tyre world's incidents as end-to-end scenarios, and the release gate.8Tests and proof
twin/engine.pyAn engine layer's run of the world, attached to the DataTwin harness so the scenarios' `engine` checks run.74Tests and proof
twin/gate.pyThe release gate (G10, TEST-03): run a change on clones of prod, compare against the oracle and prod, pass or block.210Tests and proof
twin/learning.pyThe learning loop (G10): every production failure becomes one more scenario.83Tests and proof
twin/runner.pyThe DataTwin scenario runner and its oracle checks (G10, TEST-05, SIM-01).197Tests and proof
twin/scenario.pyA DataTwin scenario as a file: `apiVersion: dwd-scenario/v0` (goal G10, TEST-05).110Tests and proof

world

ModuleWhat it is for (its own first sentence)LinesExplained on
world/__init__.pyThe data world (DWD): a pretend source that sends data on a clock and plants incidents on a calendar.1The simulated company
world/clock.pyThe simulated clock (DPC-0032: jobs read time only through the clock port). A world day is one business day in the business time zone; day 1 starts at the DWD's clock.start.38The simulated company
world/domains/__init__.pyDomain models: how one simulated company's source systems behave. The DWD's `name` picks the model.43The simulated company
world/domains/clinical.pyThe clinical trial (data-platform-catalog slices/03-clinical/world.yaml): six source systems, day by day.890The simulated company
world/domains/clinical_gold.pyThe clinical trial's gold judge: published gold against the oracle's gold for the day, and the incidents gold owns.228The simulated company
world/domains/clinical_oracle.pyThe clinical trial's oracle: per simulated day, what a correct platform holds at bronze, silver and gold, and what each certified metric must say. Computed only from the world's own truth (R9, SIM-01).256The simulated company
world/domains/tyre.pyThe tyre company (data-platform-catalog slices/02-tyre/world.yaml): six source systems, day by day.839The simulated company
world/domains/tyre_gold.pyThe tyre world's gold judge: published gold against the oracle's gold for the day, and the incidents gold owns.245The simulated company
world/domains/tyre_oracle.pyThe tyre world's oracle: per simulated day, what a correct platform holds at bronze, silver and gold, and what each certified metric must say. Computed only from the world's own truth (R9, SIM-01).282The simulated company
world/dwd.pyReads a DWD world (dwd/v0) and the specification it cites: dataset manifests and their ODCS contracts.155The simulated company
world/engine.pyThe world engine: runs the clock a day at a time, asks the domain model what its source systems did that day, writes it into the mock sources through their ports, and saves the oracle's totals for the day.151The simulated company
world/oracle.pyThe oracle: what silver must hold, computed from the world's own record, never from the platform's logs.43The simulated company
world/shop.pyThe clothing shop's orders database, as a change feed, for slice 1 (data-platform-catalog slices/01-orders/world.yaml).155The simulated company