Home › Learn › Databricks interview prep › Egress and ingress
Databricks interview prep · data engineering
Cheapest way to move data in and out
The cheapest way to egress and ingress data on Databricks: avoid, shrink, then pick the path, with hidden network costs and a way to find them.
Defaults, limits and preview status change often. Check the current Databricks docs before quoting a number.
The cheapest transfer is the one you never do.
Egress
The cheapest way to move data out
Prices vary by cloud, region and contract. Check your provider's price sheet before quoting a number.
Cloud providers charge for data leaving a region or the cloud and charge little or nothing for movement inside the same region.
| Step | Options |
|---|---|
| Do not move it | Run compute in the same cloud and region as the storage; query in place and return only results; use Delta Sharing for live reads; move the consumer to the data. Cross-region sharing still incurs egress; Databricks supports a Cloudflare R2 replica for sharing, because R2 has no egress fees (check current support). |
| Move less | Filter and aggregate first; select only needed columns; compress (Parquet with zstd or snappy); send increments with change data feed; export columnar not CSV or JSON; cache on the consumer side; cap result sizes so nobody pulls a whole table to a laptop. |
| Cheapest path | Same region is usually free; cross-region is cheaper than internet; private circuits (Direct Connect, ExpressRoute, Interconnect) have lower per-GB rates but a fixed fee, so they pay only at steady high volume; free-egress stores or a CDN for widely downloaded data; physical transfer devices for one-off petabyte moves; negotiated discounts at scale. |
| Hidden egress costs in Databricks | Detail |
|---|---|
| Cross-AZ traffic | Shuffle between zones is charged by some clouds |
| NAT gateway processing | Per-GB fees can exceed the transfer itself; use storage gateway endpoints |
| Cross-region storage reads | Every query from a workspace in another region is egress |
| Disaster recovery replication | Continuous cross-region copies are an ongoing bill; replicate critical tables incrementally |
| Federation queries | Pull data from external databases, possibly across regions |
| Result downloads and exports | From notebooks and warehouses to local machines |
To find the source: use the cloud cost tool filtered on data-transfer lines by region pair, storage access or flow logs, and Databricks system tables (billing, audit, query history). Fix the top one or two sources first.
Ingress
The cheapest way to bring data in
Inbound transfer to AWS, Azure and Google Cloud is generally free. The cost is the source's egress, the network path you build, the compute that loads, and storage requests.
| Source | Cheapest approach |
|---|---|
| Files in cloud storage, same cloud | Auto Loader, or COPY INTO for simple batch |
| Files in another region or cloud | Copy once in bulk, compressed, then ingest locally |
| Operational database | Log-based CDC (Lakeflow Connect where supported) or incremental queries on an indexed watermark; avoid full reloads |
| SaaS apps | Managed connector if supported, else scheduled incremental API pulls using bulk endpoints |
| Event streams | Structured Streaming; availableNow batches if freshness allows |
| On-premises data | Compress and upload in bulk; transfer devices for huge one-offs |
| Another lakehouse or warehouse | Federation or Delta Sharing, no copy |
| Big historical backfill | One run on a right-sized job cluster with spot workers |
| Technique | Effect |
|---|---|
| Incremental loads | Only new or changed data |
| Compressed columnar files | Parquet beats CSV and JSON |
| Files of 128 MB to 1 GB | Millions of tiny files raise request costs and slow Spark |
| Land raw, transform later | Cheap, rerunnable ingest job |
| Scheduled instead of continuous | availableNow runs when latency allows |
| Serverless or job clusters with auto-termination | Pay only while loading |
| Spot workers | Fault-tolerant batch ingestion |
| File notification mode at large scale | Avoids repeated listing of huge directories |
| Lifecycle rules on the landing zone | Move old raw files to cheaper tiers |
Hidden costs: NAT gateway processing, private link hourly and per-GB fees, dedicated circuits, cross-AZ and cross-region traffic, third-party ELT tools that charge by rows, request charges, source load, and failed reruns that reload everything (use checkpoints and idempotent writes).
Practice
Questions to rehearse
- How would you reduce egress cost?
- Is Delta Sharing free of egress?
- What hidden costs come with private networking?
- What does ingress cost, really?
- How do you ingest cheaply at scale?
- When would you not ingest at all?
FAQ
Questions people ask
Is Delta Sharing free of egress?
No. Data read across regions or clouds is still billed, unless it sits in a store with no egress fee.
Why is a NAT gateway a risk?
It charges per GB processed, which can exceed the transfer cost; storage endpoints avoid it.
Next
Where to go next
All Databricks interview topics · All data engineering · Databricks on this site