Data PlatformZoom level 3
Who calls what: adapters
Every class, function and method in the adapters package, with who calls it. Found by reading the code. A call written as object.method() is matched by the method's name, so a same-named method elsewhere can appear: those rows say by name.
663 definitions in 46 modules. Jump to a module: azure_blob.py · customer_runlog.py · databricks_clone.py · databricks_native.py · databricks_notify.py · databricks_ops_views.py · databricks_pipeline_events.py · databricks_system.py · delta_bronze.py · duckdb_frames.py · duckdb_gold.py · duckdb_silver.py · file_drop.py · files.py · http_api.py · keyed_tokens.py · kinesis.py · lakebase.py · lakeview.py · local_movers.py · local_stream.py · memory.py · memory_bronze.py · memory_catalog.py · memory_meta.py · memory_pipeline_events.py · memory_runlog.py · memory_sources.py · memory_system.py · memory_tables.py · null_runlog.py · otel_tracer.py · postgres_meta.py · postgres_runlog.py · postgres_source.py · s3.py · secrets.py · spark_delta.py · spark_gold.py · spark_ops_tables.py · spark_silver.py · sql_source.py · sqlite_source.py · store_registry.py · table_store.py · uc_catalog.py
adapters/azure_blob.py
Object storage on Azure Blob Storage (cloud layer): landing_store, the partner file drop and object_store. Needs azure-storage-blob (the `azure` extra) only when a real client is built; nothing here imports an Azure SDK. The container is pa none yet
adapters/customer_runlog.py
EXAMPLE SKELETON: a customer's own run log on the run-log reader port (level 2 of ADR-024 rule 6). none yet
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
CustomerRunLogclass, line 32 | none found in the code | no test names it | |
CustomerRunLog.__init__method, line 33 | none found in the code | no test names it | |
CustomerRunLog.snapshotmethod, line 36 | Monitor.run_finished · obs/monitor.py:99 (by name)Monitor.check_lateness · obs/monitor.py:108 (by name)Monitor.tables · obs/monitor.py:120 (by name) | test_meta_loader.py, test_obs_recorder.py and 10 more |
adapters/databricks_clone.py
The release gate's ports (dataplatform.twin.gate) on Databricks: Delta SHALLOW CLONE and one measuring query. none yet
adapters/databricks_native.py
Databricks bindings for the adopted platform-native screens and notifications (dataplatform.obs.native). none yet
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
render_native_yamlfunction, line 53 | The adopted set as a deterministic YAML file (no YAML library: plain, sorted, quoted). | files · realms/databricks/dashboards/generate.py:35 | no test names it |
adapters/databricks_notify.py
The notifier port (dataplatform.obs.monitor.Notifier) through the workspace's notification destinations. none yet
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
UnroutedChannelclass, line 22 | An alert's channel has no destination bound: refused at the first send, not dropped silently. | DestinationNotifier.__init__ · adapters/databricks_notify.py:39DestinationNotifier.send · adapters/databricks_notify.py:45 | test_obs_notify.py |
payloadfunction, line 26 | DestinationNotifier.send · adapters/databricks_notify.py:46 | test_genie_deploy.py, test_governance_catalog.py and 6 more | |
DestinationNotifierclass, line 35 | none found in the code | test_obs_notify.py | |
DestinationNotifier.__init__method, line 36 | none found in the code | no test names it | |
DestinationNotifier.sendmethod, line 42 | Monitor._evaluate · obs/monitor.py:81 (by name) | no test names it |
adapters/databricks_ops_views.py
The ops metric views (dataplatform.obs.views) as Databricks metric views: one `CREATE ... WITH METRICS` per view. none yet
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
_plainfunction, line 19 | measure_expr · adapters/databricks_ops_views.py:32view_yaml · adapters/databricks_ops_views.py:40view_sql · adapters/databricks_ops_views.py:52 | no test names it | |
_qfunction, line 25 | view_yaml · adapters/databricks_ops_views.py:39 | no test names it | |
measure_exprfunction, line 29 | view_yaml · adapters/databricks_ops_views.py:45 | no test names it | |
view_yamlfunction, line 38 | view_sql · adapters/databricks_ops_views.py:54 | no test names it | |
view_sqlfunction, line 51 | files · realms/databricks/dashboards/generate.py:32publish · adapters/spark_ops_tables.py:43 | test_obs_dashboards.py, test_runtime_sql_views.py |
adapters/databricks_pipeline_events.py
The pipeline event-log port (dataplatform.obs.pipeline_events) on Lakeflow Declarative Pipelines' event log. none yet
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
_tsfunction, line 47 | DatabricksPipelineEvents.table_updates · adapters/databricks_pipeline_events.py:64 | no test names it | |
DatabricksPipelineEventsclass, line 51 | none found in the code | test_obs_pipeline_events.py | |
DatabricksPipelineEvents.__init__method, line 52 | `pipelines`: pipeline name -> pipeline id, from the bundle (silver and gold pipelines of a domain). | none found in the code | no test names it |
DatabricksPipelineEvents.table_updatesmethod, line 56 | Monitor.tables · obs/monitor.py:121 (by name) | test_obs_pipeline_events.py |
adapters/databricks_system.py
The system-records port (dataplatform.obs.system) on Databricks system tables. none yet
adapters/delta_bronze.py
The bronze writer on Delta tables in a local folder, through delta-rs (`deltalake`, Apache 2.0) and pyarrow: no JVM, no Spark, the same approach as LakeIngestionFramework's LocalDeltaWriter (core/writers/local_delta.py), which only appends; Landing to bronze
adapters/duckdb_frames.py
DuckDB as a frame engine for the pure stage and build code, through SQLFrame (the PySpark DataFrame API compiled to SQL by sqlglot): one logic, several engines (docs/DUCKDB-SPIKE.md). For laptop and CI runs only; nothing here ever writes a none yet
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
connectionfunction, line 20 | activate · adapters/duckdb_frames.py:36 | test_lakebase.py, test_resilience.py and 9 more | |
activatefunction, line 26 | Point every later `import pyspark...` at SQLFrame on DuckDB (process-wide). Returns the connection. | none found in the code | conftest.py |
local_sessionfunction, line 41 | A SQLFrame DuckDB session (the SparkSession stand-in) on the activated connection, or a new one. | none found in the code | conftest.py, test_obs_steps.py |
checkpointfunction, line 48 | localCheckpoint for a SQLFrame frame: run its query once into a DuckDB table and read that table (a plain table in the session's in-memory database: SQLFrame finds a temporary table's columns in no catalog it reads). Without it a frame buil | DuckGoldEngine.hold · adapters/duckdb_gold.py:68DuckStageRunner.run · adapters/duckdb_silver.py:65 | silver_helpers.py |
adapters/duckdb_gold.py
Gold's engine operations on DuckDB through SQLFrame (docs/DUCKDB-SPIKE.md): the Spark adapter's operations, with the five engine-specific pieces replaced. For laptop and CI runs only; Databricks keeps adapters/spark_gold.py. The harness
adapters/duckdb_silver.py
Silver on DuckDB (through SQLFrame) and local Delta tables (through delta-rs): the same stages (dataplatform.stages) as the Spark adapter, other IO (docs/DUCKDB-SPIKE.md). For laptop and CI runs only: nothing here writes a governed table on Writer
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
to_arrowfunction, line 31 | A SQLFrame frame's rows as an Arrow table (the frame's query run on its DuckDB connection). | DeltaRsGoldTables.write_dimension · adapters/duckdb_gold.py:111DeltaRsGoldTables.replace_windows · adapters/duckdb_gold.py:123DeltaRsSilverTables._append · adapters/duckdb_silver.py:106DeltaRsSilverTables._merge · adapters/duckdb_silver.py:113 | no test names it |
read_deltafunction, line 37 | A Delta folder as a frame (at `version` when given: time travel, INSP-01): delta-rs reads it, DuckDB holds the rows in a table of the session. | DeltaRsGoldTables._read · adapters/duckdb_gold.py:78DeltaRsGoldTables.silver_at · adapters/duckdb_gold.py:102DeltaRsSilverTables._read · adapters/duckdb_silver.py:97 | no test names it |
DuckStageRunnerclass, line 55 | stage_runner on DuckDB: the same run_stages, the rows run once into a table before they are counted and written (the counts and the write read the same rows). | _duckdb · scripts/platform_binding.py:69 | silver_helpers.py |
DuckStageRunner.__init__method, line 59 | none found in the code | no test names it | |
DuckStageRunner.runmethod, line 62 | none found in the code | no test names it | |
DuckStageRunner.releasemethod, line 89 | none found in the code | test_meta_store.py, test_silver_ports.py and 2 more | |
DeltaRsSilverTablesclass, line 93 | The read and write edges on Delta folders, by delta-rs; the frames are DuckDB's (SQLFrame). | _duckdb · scripts/platform_binding.py:68 | no test names it |
DeltaRsSilverTables._readmethod, line 96 | none found in the code | no test names it | |
DeltaRsSilverTables._appendmethod, line 99 | step_prepare · realms/databricks/tasks/main.py:70 (by name) | no test names it | |
DeltaRsSilverTables._mergemethod, line 110 | none found in the code | no test names it | |
DeltaRsSilverTables.count_by_hourmethod, line 126 | hourly · silver/daybyday.py:73 (by name) | test_silver_ports.py |
adapters/file_drop.py
FileDrop on a local folder: the partner and lab drop zones. Object storage is a later adapter with the same port. Sources and movers
adapters/files.py
landing_store on a plain folder. On Databricks the folder is a Unity Catalog volume path, which serverless Python reads and writes as ordinary files, so the same adapter serves both. none yet
adapters/http_api.py
RecordInterface as a small local HTTP service: the dealer management system. Sources and movers
adapters/keyed_tokens.py
The tokenizer as a keyed hash (HMAC-SHA256, standard library): the same value always gives the same token, so joins and duplicate checks still work, and nobody without the key can reverse or recompute it. The key is the binding's secret (DP none yet
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
KeyedTokenizerclass, line 16 | none found in the code | test_ingest_pipelines.py, test_ingest_ports.py and 4 more | |
KeyedTokenizer.__init__method, line 19 | none found in the code | no test names it | |
KeyedTokenizer.tokenmethod, line 24 | main.on_day · scripts/run_platform.py:582 (value)instance_token · adapters/lakebase.py:65 (value)main.live_rows · scripts/run_platform.py:489 (by name)mass_delete_problem.stored · ingest/steps.py:256 (by name)protect · ingest/steps.py:291 (by name) | test_bundle.py, test_semantic_governance.py and 18 more | |
KeyedTokenizer.from_env_or_filemethod, line 29 | The key from the environment, else from a local file made once with a random key (local runs only). | run_ingest · scripts/run_hop.py:60 (by name)main · scripts/run_platform.py:443 (by name) | test_ingest_ports.py |
adapters/kinesis.py
stream_bus on Amazon Kinesis Data Streams (cloud layer, dbs/cloud/aws.yaml). Needs boto3; the client is passed in, so tests use a local stand-in (moto) and the platform passes one built from the job's role. Sources and movers
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
KinesisStreamclass, line 20 | aws_sinks · scripts/run_world.py:59 | test_cloud_adapters.py, test_source_ports.py | |
KinesisStream.__init__method, line 21 | none found in the code | no test names it | |
KinesisStream._namemethod, line 24 | KinesisStream.append · adapters/kinesis.py:35KinesisStream.shards · adapters/kinesis.py:46KinesisStream.read · adapters/kinesis.py:58 | no test names it | |
KinesisStream.appendmethod, line 27 | none found in the code | azure_fakes.py, gold_helpers.py and 45 more | |
KinesisStream.shardsmethod, line 44 | read_back · scripts/run_world.py:137 (by name)stream_events · ingest/pipelines.py:342 (by name) | test_cloud_adapters.py, test_ingest_pipelines.py and 2 more | |
KinesisStream.readmethod, line 54 | none found in the code | no test names it |
adapters/lakebase.py
The metadata store on Lakebase, Databricks' managed PostgreSQL (ADR-010, binding dbs/engine/databricks.yaml). Metadata store
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
_project_endpointfunction, line 22 | (endpoint resource name, host) of a Lakebase project's read-write endpoint on its default branch, or None when the workspace has no such project. Lakebase on Azure Databricks is the project (autoscaling) kind, found over REST so an older SD | instance_dsn · adapters/lakebase.py:46instance_token · adapters/lakebase.py:59 | no test names it |
instance_dsnfunction, line 44 | host, port, database and user for the instance; no password (the token is passed separately). | connect_lakebase · adapters/lakebase.py:81 | test_lakebase.py |
instance_tokenfunction, line 58 | connect_lakebase · adapters/lakebase.py:81 | test_lakebase.py | |
connect_lakebasefunction, line 70 | A metadata store on a Lakebase instance, connected as the caller's own identity. | attach_meta · scripts/platform_binding.py:36lakebase_fresh_store · scripts/run_platform.py:87meta_store_from_env · adapters/lakebase.py:91 | test_lakebase.py |
meta_store_from_envfunction, line 86 | The binding's choice, from the environment: DP_META_INSTANCE (+ DP_META_DATABASE) means Lakebase as this identity; otherwise DP_META_DSN (plain PostgreSQL, the laptop and other clouds); one adapter either way. | module level · realms/databricks/declarative/silver.py:71lakebase · realms/databricks/engine/tasks/probe.py:79 | test_lakebase.py |
adapters/lakeview.py
The ops dashboards (dataplatform.obs.dashboards) as Databricks AI/BI (Lakeview) dashboard files. none yet
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
_identfunction, line 29 | _measure_sql · adapters/lakeview.py:47query_sql · adapters/lakeview.py:55_field · adapters/lakeview.py:85 | no test names it | |
_literalfunction, line 35 | query_sql · adapters/lakeview.py:69 | no test names it | |
_measure_sqlfunction, line 43 | query_sql · adapters/lakeview.py:61 | no test names it | |
query_sqlfunction, line 51 | The query as SQL lines (Lakeview keeps a dataset's query as a list of lines). | compile_dashboard · adapters/lakeview.py:135 | test_obs_dashboards.py |
_fieldfunction, line 84 | _widget · adapters/lakeview.py:107 | no test names it | |
_encodingsfunction, line 88 | _widget · adapters/lakeview.py:109 | no test names it | |
_widgetfunction, line 102 | _layout · adapters/lakeview.py:126 | no test names it | |
_layoutfunction, line 114 | compile_dashboard · adapters/lakeview.py:136 | no test names it | |
compile_dashboardfunction, line 131 | render_file · adapters/lakeview.py:141 | test_obs_dashboards.py | |
render_filefunction, line 140 | files · realms/databricks/dashboards/generate.py:31 | no test names it | |
bundle_resourcesfunction, line 144 | The bundle resource file: one dashboard resource per file, published to the bundle's SQL warehouse. `native` names the platform's own dashboards kept unchanged under native/<id>.lvdash.json (adopted, not rebuilt). | files · realms/databricks/dashboards/generate.py:34 | no test names it |
adapters/local_movers.py
The push movers for the local stack: they stand in for what a cloud supplies (a change-capture service, a file transfer service) and meet the landing contract every push mover meets (DPC-0038): each delivery written once, under a unique nam Sources and movers
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
ChangeFeedMoverclass, line 23 | mover · scripts/run_platform.py:232 | test_ingest_pipelines.py, test_ingest_world.py and 2 more | |
ChangeFeedMover.__init__method, line 24 | none found in the code | no test names it | |
ChangeFeedMover.checkpointmethod, line 27 | ChangeFeedMover.move · adapters/local_movers.py:32DmsMover.move · adapters/local_movers.py:87 (by name) | silver_helpers.py | |
ChangeFeedMover.movemethod, line 31 | mover.move · scripts/run_platform.py:242 (by name) | no test names it | |
DropMoverclass, line 51 | mover · scripts/run_platform.py:233 | test_ingest_pipelines.py, test_ingest_world.py and 2 more | |
DropMover.__init__method, line 52 | none found in the code | no test names it | |
DropMover.movemethod, line 55 | mover.move · scripts/run_platform.py:242 (by name) | no test names it | |
DmsMoverclass, line 71 | A change-capture mover shaped like a managed database migration service writing to object storage (B4, ADR-031): the same changes as ChangeFeedMover, delivered the way such a service writes them. The first delivery is a full-load file (LOAD | mover · scripts/run_platform.py:232 | test_ingest_formats.py |
DmsMover.movemethod, line 83 | mover.move · scripts/run_platform.py:242 (by name) | no test names it |
adapters/local_stream.py
EventStream on local files: a log of shards, each an append-only JSON-lines file. A managed stream is a later adapter with the same port; the shard count and the partition-key hashing mirror how such streams spread records. Sources and movers
adapters/memory.py
In-memory adapters for tests. Same ports as the Spark/Delta ones. none yet
adapters/memory_bronze.py
The bronze writer in memory, for tests: the reference behaviour the table-format adapters are tested against (M5). none yet
adapters/memory_catalog.py
Catalog state in memory: the reference adapter for the CatalogState port (tests, dry runs, the first deploy). none yet
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
MemoryCatalogStateclass, line 11 | catalog_actions · realms/databricks/semantic_governance/generate.py:128 | test_governance_catalog.py | |
MemoryCatalogState.__init__method, line 12 | none found in the code | no test names it | |
MemoryCatalogState.entriesmethod, line 15 | catalog_actions · realms/databricks/semantic_governance/generate.py:129 (by name) | test_governance_catalog.py, test_silver_plan.py | |
MemoryCatalogState.applymethod, line 18 | Stands in for running the realm's statements. | World.write · world/engine.py:125 (by name) | silver_helpers.py, test_baseline_stage.py and 15 more |
adapters/memory_meta.py
The metadata store in memory, for unit tests. Same port and the same refusals as the PostgreSQL adapter; the contract test table (tests/dataplatform/test_meta_store.py) runs every row against both (M5). Metadata store
adapters/memory_pipeline_events.py
The pipeline event-log port (dataplatform.obs.pipeline_events) in memory: rows added by tests. none yet
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
MemoryPipelineEventsclass, line 10 | none found in the code | test_obs_pipeline_events.py | |
MemoryPipelineEvents.__init__method, line 11 | none found in the code | no test names it | |
MemoryPipelineEvents.table_updatesmethod, line 14 | Monitor.tables · obs/monitor.py:121 (by name) | test_obs_pipeline_events.py |
adapters/memory_runlog.py
StampingRunLog: any metadata store, plus the run-log reader port (dataplatform.obs.runlog) in process. none yet
adapters/memory_sources.py
The four source ports in memory: the reference behaviour every real adapter is tested against (M5). none yet
adapters/memory_system.py
The system-records port (dataplatform.obs.system) in memory: rows added by tests or the fixture runner. none yet
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
MemorySystemTablesclass, line 10 | main · scripts/run_platform.py:423FixtureRunner.__init__ · obs/fixture.py:94 | test_obs_dashboards.py, test_obs_system.py | |
MemorySystemTables.__init__method, line 11 | none found in the code | no test names it | |
MemorySystemTables.jobsmethod, line 16 | read_all · obs/system.py:69 (by name) | no test names it | |
MemorySystemTables.lineagemethod, line 20 | read_all · obs/system.py:69 (by name) | test_obs_dashboards.py, test_obs_lineage.py and 10 more | |
MemorySystemTables.accessmethod, line 24 | generate · realms/databricks/semantic_governance/generate.py:173 (value)read_all · obs/system.py:69 (by name) | test_lakeformation.py, test_semantic_governance.py and 5 more |
adapters/memory_tables.py
The release gate's ports (dataplatform.twin.gate) in memory: tables are lists of rows; a clone is a copy of the list (a shallow clone shares data until written; in memory a copy behaves the same for the gate). none yet
adapters/null_runlog.py
The run-log reader port (dataplatform.obs.runlog) with no run log connected: level 1 of ADR-024 rule 6. none yet
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
NullRunLogclass, line 14 | none found in the code | test_obs_dashboards.py | |
NullRunLog.snapshotmethod, line 15 | Monitor.run_finished · obs/monitor.py:99 (by name)Monitor.check_lateness · obs/monitor.py:108 (by name)Monitor.tables · obs/monitor.py:120 (by name) | test_meta_loader.py, test_obs_recorder.py and 10 more |
adapters/otel_tracer.py
The telemetry port (dataplatform.obs.trace) on the OpenTelemetry API. none yet
adapters/postgres_meta.py
The metadata store on PostgreSQL (ADR-010). Every read and write is one call to a function in adapters/sql/postgres_meta.sql; this file holds no table names (DPC-0033, checked by tests/dataplatform/test_meta_store.py). Metadata store
adapters/postgres_runlog.py
The run-log reader port (dataplatform.obs.runlog) on PostgreSQL: one call to runtime.ops_run_log (sql/postgres_obs.sql), installed beside the metadata store's own schema (postgres_meta.sql). No table names here (DPC-0033). Timestamps are th none yet
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
installfunction, line 24 | Install (or update) the read procedure. Needs the store's schema first (postgres_meta.install). | none found in the code | test_meta_store.py, test_obs_runlog.py and 5 more |
_tsfunction, line 30 | parse · adapters/postgres_runlog.py:42 | no test names it | |
_atfunction, line 34 | parse · adapters/postgres_runlog.py:42 | no test names it | |
parsefunction, line 38 | The procedure's document as port rows (pure; tested on its own). | PostgresRunLogReader.snapshot · adapters/postgres_runlog.py:61 | test_ingest_formats.py, test_ingest_steps.py and 19 more |
PostgresRunLogReaderclass, line 55 | none found in the code | test_obs_runlog.py | |
PostgresRunLogReader.__init__method, line 56 | none found in the code | no test names it | |
PostgresRunLogReader.snapshotmethod, line 59 | Monitor.run_finished · obs/monitor.py:99 (by name)Monitor.check_lateness · obs/monitor.py:108 (by name)Monitor.tables · obs/monitor.py:120 (by name) | test_meta_loader.py, test_obs_recorder.py and 10 more |
adapters/postgres_source.py
SourceDatabase on PostgreSQL: the mock ERP and MES databases, one schema per system. Sources and movers
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
_create_databasefunction, line 33 | PostgresSource.connect · adapters/postgres_source.py:51 | no test names it | |
PostgresSourceclass, line 42 | none found in the code | test_source_ports.py | |
PostgresSource.__init__method, line 43 | none found in the code | no test names it | |
PostgresSource.connectmethod, line 47 | connect_lakebase · adapters/lakebase.py:80 (value)main · realms/aws-databricks/meta_setup.py:78 (by name)_seed_postgres · scripts/demo_end_to_end.py:266 (by name)_ingest_table_to_bronze · scripts/demo_end_to_end.py:372 (by name)attach_meta · scripts/platform_binding.py:39 (by name)meta_store · scripts/run_platform.py:71 (by name)and 12 more | test_lakebase.py, test_meta_store.py and 21 more |
adapters/s3.py
Object storage on Amazon S3 (cloud layer, dbs/cloud/aws.yaml): landing_store, the partner file drop and object_store. Needs boto3 (the `s3` extra). The client is passed in, so tests use a local stand-in (moto) and the platform passes one bu Sources and movers
adapters/secrets.py
secret_store adapters (cloud_ports.SecretStore, DPC-0029). The value is returned to the caller only; nothing here logs it, and an error names the reference, never the value. none yet
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
EnvSecretsclass, line 21 | none found in the code | test_secrets.py | |
EnvSecrets.__init__method, line 22 | none found in the code | no test names it | |
EnvSecrets.getmethod, line 25 | none found in the code | no test names it | |
AwsSecretsManagerclass, line 35 | none found in the code | test_secrets.py | |
AwsSecretsManager.__init__method, line 36 | none found in the code | no test names it | |
AwsSecretsManager.getmethod, line 40 | none found in the code | no test names it | |
_is_not_foundfunction, line 51 | Key Vault's "no such secret" error. The SDK is imported only here and only when an error has happened, and a missing SDK falls back to the class name, so this module never needs the `azure` extra at import. | AzureKeyVault.get · adapters/secrets.py:84 | no test names it |
key_vault_namefunction, line 61 | The Key Vault secret name for a port name. Key Vault allows letters, digits and hyphens only (1 to 127), so the "/" separator becomes "--". That is reversible only if no name already holds "--" (the port refuses it) and no hyphen sits next | test_secret_references_map_to_key_vault_names · realms/azure-databricks/tests/test_realm.py:119AzureKeyVault.get · adapters/secrets.py:82 | no test names it |
AzureKeyVaultclass, line 73 | none found in the code | test_secrets.py, test_adf_generator.py | |
AzureKeyVault.__init__method, line 74 | none found in the code | no test names it | |
AzureKeyVault.getmethod, line 78 | none found in the code | no test names it |
adapters/spark_delta.py
The slice 1 ports on Spark + Delta tables: Databricks (serverless, Spark Connect) or open-source Spark locally. none yet
adapters/spark_gold.py
Gold on Spark and Delta tables: the harness's engine operations (PySpark) and the gold read and write edges. The harness
adapters/spark_ops_tables.py
Publish the ops tables (obs/ops_export.py file) as catalog tables, then the ops metric views over them (B13). none yet
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
publishfunction, line 19 | module level · realms/databricks/engine/tasks/run_days.py:58 | test_gold_harness.py, test_source_ports.py |
adapters/spark_silver.py
Silver on Spark and Delta tables: the stage runner (dataplatform.stages on PySpark) and the read and write edges. Writer
adapters/sql_source.py
SourceDatabase over any DB-API connection. Two thin adapters use it: postgres_source.py and sqlite_source.py. none yet
adapters/sqlite_source.py
SourceDatabase on SQLite (standard library): for tests and for a laptop with no Postgres. SQLite has no schemas, so a system's tables are prefixed with its name: erp_db__sales_order_line. Sources and movers
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
SqliteSourceclass, line 28 | sinks · scripts/run_world.py:82 | test_ingest_world.py, test_semantic_metrics.py and 5 more | |
SqliteSource.__init__method, line 29 | none found in the code | no test names it |
adapters/store_registry.py
Slice 1's batch registry (dataplatform.ports.MetadataStore: batch_status(key), record_batch) on the W1a metadata store, so slice 1's lander (dataplatform.lander) records into the same file log as W1b's ingestion, through the store's procedu none yet
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
StoreBatchRegistryclass, line 16 | none found in the code | test_slice1_local.py | |
StoreBatchRegistry.__init__method, line 17 | none found in the code | no test names it | |
StoreBatchRegistry.dataset_configmethod, line 20 | step_land · realms/databricks/tasks/main.py:112 (by name)step_finish · realms/databricks/tasks/main.py:123 (by name) | test_exceptions.py, test_runtime_sql_views.py | |
StoreBatchRegistry.batch_statusmethod, line 23 | StampingRunLog.batch_status · adapters/memory_runlog.py:96 (by name)enter · ingest/steps.py:318 (by name)land · lander.py:34 (by name) | test_meta_store.py, test_slice1_local.py | |
StoreBatchRegistry.record_batchmethod, line 26 | land · lander.py:38 (by name) | no test names it |
adapters/table_store.py
The table port's engine side (data-platform-catalog dpc/ports.yaml table_writer, ADR-027): one small adapter per open table format, so silver, gold and bronze code never names a format. none yet
adapters/uc_catalog.py
Catalog state read from Unity Catalog's information schema (Databricks; dbs/engine/databricks.yaml data_catalog). none yet
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
UcCatalogStateclass, line 40 | none found in the code | test_governance_catalog.py | |
UcCatalogState.__init__method, line 41 | none found in the code | no test names it | |
UcCatalogState.entriesmethod, line 44 | catalog_actions · realms/databricks/semantic_governance/generate.py:129 (by name) | test_governance_catalog.py, test_silver_plan.py |