Home › Data engineering › Fundamentals › Security
Data engineering fundamentals · free guide
Security fundamentals for data engineers
Security in data work is mostly a few habits applied everywhere: know who is asking, give the minimum access, protect secrets, know where sensitive data lives, and keep a record. This page lays them out as a mind map so you can check a pipeline or platform branch by branch. It teaches principles; follow your organisation's policy and your platform's documentation for specifics.
Start here
The data physics
Five plain facts that explain almost everything below. Read these and the rest follows.
The mind map
The whole topic on one screen
Each card is a branch. Walk them in order when you are stuck: the question tells you what the branch is about, and the first check tells you where to look.
1. Identity and access
Who is asking?
Single sign-on, service principals, groups, roles, least privilege, short-lived tokens
First check: Who can read this table?
2. Secrets
Where do the keys live?
Secret stores, rotation, no secrets in code or logs, managed identities
First check: Search code and logs for credentials
3. Protection in place
Is the data protected at rest and in motion?
Encryption, key management, TLS, private networking
First check: Public endpoints and unencrypted stores
4. Know the data
Which data is sensitive?
Classification, tagging, PII discovery, lineage, data contracts
First check: Is every sensitive column labelled?
5. Limit what is seen
Who sees which rows and columns?
Row filters, column masks, views, anonymisation, tokenisation
First check: What does a junior analyst see?
6. Prove it
Can we show what happened?
Audit logs, access reviews, lineage, change history, alerts
First check: Can you answer 'who read this last week'?
7. Share and retire safely
Where does data go, and when does it go away?
Sharing, extracts, retention, deletion, backup, environments
First check: Production data in test?
Identity and access
Who can do what
Most data breaches are access mistakes, not clever attacks: an over-wide group, a shared account, a forgotten service account.
Layers of access control. Each gate narrows what is visible; the audit log proves it.
| Principle | What it means | Good practice | Common mistake | Verdict |
|---|---|---|---|---|
| Least privilege | Only the access the task needs | Grant on groups and roles, not individuals | 'Just give them admin to unblock' | Always |
| Separate identities for people and jobs | Pipelines run as service identities, not as a person | One service principal per pipeline or domain | Jobs running under an engineer's login | Always |
| Short-lived credentials | Tokens that expire | Managed identities, federated login | Long-lived tokens in config | Pursue |
| Access by group, reviewed | Membership is the control point | Periodic access review | Accounts of leavers still active | Always |
| Separation of duties | Builder is not approver | Review before production | One person writes and ships alone | Pursue for sensitive systems |
| Environment separation | Dev, test, prod have separate access | No production write access from dev | Same credentials everywhere | Always |
Secrets
Keys, passwords and tokens
Treat any credential that has been pasted in a chat, committed to code or printed in a log as compromised.
| Do | Do not | Why |
|---|---|---|
| Keep secrets in a secret store and fetch at run time | Hard-code them in code, notebooks or config | Code spreads; secret stores have access control and audit |
| Rotate on a schedule and on any suspicion | Keep one key for years | Limits the window of misuse |
| Use managed identities where the platform allows | Share passwords between systems | No secret to leak |
| Scan code and logs for credentials | Print connection strings in logs | Logs are widely readable |
Protection in place
Encryption and network paths
These controls are mostly platform settings. A data engineer's job is to check they are on and not bypassed.
| Control | What it covers | Check | Verdict |
|---|---|---|---|
| Encryption at rest | Stored files and disks | Enabled on storage; know who holds the keys | Default on; verify |
| Encryption in transit | Data on the network | TLS for every connection, including to sources | Always |
| Customer-managed keys | You control the key | Where policy requires; adds operational duty | When required |
| Private networking | Traffic stays off the public internet | Private endpoints, restricted egress | Pursue for sensitive data |
| Public buckets and open endpoints | Anyone can reach it | Scan for public access | Always fix |
Know the data
Classification before control
Controls are applied to columns and tables, so you need a reliable list of which ones are sensitive.
| Step | What | Where it lives | Verdict |
|---|---|---|---|
| Classify | Mark personal, financial, health, secret | Tags on columns and tables | First step |
| Discover | Scan for sensitive values in unknown columns | Automated scans, reviewed by people | Pursue |
| Trace | Follow sensitive columns downstream | Lineage | Pursue |
| Agree | Record what each dataset may be used for | Data contracts and catalog entries | Pursue |
| Minimise | Do not collect or copy what you do not need | Design reviews | Pursue first |
Limit what is seen
Row, column and value controls
Different people may query the same table and see different things.
| Technique | What it does | Use when | Trade-off |
|---|---|---|---|
| Row filters | Hide rows a user may not see | Region or tenant separation | Policy must be maintained |
| Column masks | Show partial or hashed values | Analysts need shape, not identity | Masked data cannot be joined on raw values |
| Views as a gate | Expose only chosen columns | Simple, quick control | Easy to bypass if base table is open |
| Tokenisation / pseudonymisation | Replace identifiers with stable tokens | Join across datasets without identity | Token store becomes sensitive |
| Aggregation and minimum group size | Share totals, not individuals | Reporting | Small groups can leak |
Prove it
Audit and review
A control you cannot evidence did not happen, as far as an auditor is concerned.
| Evidence | Answers | Keep for | Verdict |
|---|---|---|---|
| Access audit log | Who read or changed what and when | As policy requires | Always on |
| Lineage | Where a sensitive column went | Life of the data | Pursue |
| Change history | Who changed a pipeline or permission | Life of the system | Always |
| Access reviews | Who still needs access | Each cycle | Pursue |
| Alerts on unusual access | A burst of reads or new exports | Ongoing | Pursue |
In practice
CostSherlock rules for this topic
CostSherlock is our open set of cost and performance rules. Each rule below has a plain meaning here; the rule id is its name in CostSherlock. They are grouped by the same branches as the mind map, so a finding tells you which branch to open.
Identity and access
| CostSherlock rule | What it catches, in plain words |
|---|---|
uc_cluster_security_mode_mismatch | Compute mode does not enforce the intended access model |
uc_cluster_unauthorized_runtime_change | Compute changed outside policy |
uc_no_cluster_policy_enforced | No policy governs compute creation |
uc_cluster_policy_permissive_instance_types | Policy allows almost anything |
uc_personal_compute_proliferation | Personal compute multiplying |
uc_orphaned_cluster_creator_inactive | Resources whose creator has left |
Prove it
| CostSherlock rule | What it catches, in plain words |
|---|---|
uc_audit_burst_anomaly | Unusual burst of audited activity |
uc_cluster_creation_anomaly_by_user | Unusual compute creation by a user |
uc_maintenance_history_stale | Maintenance history out of date |
Know the data and keep it tidy
| CostSherlock rule | What it catches, in plain words |
|---|---|
uc_cluster_no_cost_tags | Resources without ownership tags |
uc_external_bi_tool_unbounded_refresh | External tool pulling unbounded data |
FAQ
Questions people ask
What is the single biggest security habit for a data engineer?
Least privilege on service identities plus no secrets in code. Together they remove the most common causes of data exposure.
Do I need to encrypt data myself?
Mostly the platform does at rest and in transit. Your job is to confirm it is on, understand who holds the keys, and never turn it off to unblock a problem.
What is the difference between masking and removing a column?
Masking keeps the column but hides or alters values per user. Removing keeps the data out of reach entirely. Use masking when the shape is useful.
Why does lineage matter for security?
When a sensitive column is found, lineage tells you every downstream table, dashboard and export that now carries it.
Is this page legal or compliance advice?
No. It explains engineering principles. Follow your organisation's policy, local law and your platform's current documentation.
Does CostSherlock check security?
Only a few signals that overlap with cost and governance, such as unmanaged compute and unusual activity. It is not a security scanner.
Related
Where it connects
Performance fundamentals · Cost fundamentals · Processing fundamentals · Storage fundamentals · Database families · Where data engineering time goes · Who maintains your tables · All data engineering
The numbers and thresholds on this page are rules of thumb and orders of magnitude to build intuition, not benchmarks. Check your platform's current documentation and measure your own workload.