Home › Data engineering › Fundamentals › Security

Data engineering fundamentals · free guide

Security fundamentals for data engineers

Security in data work is mostly a few habits applied everywhere: know who is asking, give the minimum access, protect secrets, know where sensitive data lives, and keep a record. This page lays them out as a mind map so you can check a pipeline or platform branch by branch. It teaches principles; follow your organisation's policy and your platform's documentation for specifics.

Start here

The data physics

Five plain facts that explain almost everything below. Read these and the rest follows.

Every access is a question: who, what, why.Identity says who. Permissions say what. Purpose and audit say why, and prove it later.
Least privilege is the default.Start with no access and grant exactly what a role needs. Wide access is easy to give and very hard to take back.
Secrets are the keys to everything else.A password in code, a notebook or a log defeats every other control.
You cannot protect what you cannot find.Sensitive columns must be known, labelled and traced downstream, otherwise controls miss them.
Copies multiply risk.Every extra copy of sensitive data is another place to secure, audit and delete.

The mind map

The whole topic on one screen

Each card is a branch. Walk them in order when you are stuck: the question tells you what the branch is about, and the first check tells you where to look.

Security

1. Identity and access

Who is asking?

Single sign-on, service principals, groups, roles, least privilege, short-lived tokens

First check: Who can read this table?

2. Secrets

Where do the keys live?

Secret stores, rotation, no secrets in code or logs, managed identities

First check: Search code and logs for credentials

3. Protection in place

Is the data protected at rest and in motion?

Encryption, key management, TLS, private networking

First check: Public endpoints and unencrypted stores

4. Know the data

Which data is sensitive?

Classification, tagging, PII discovery, lineage, data contracts

First check: Is every sensitive column labelled?

5. Limit what is seen

Who sees which rows and columns?

Row filters, column masks, views, anonymisation, tokenisation

First check: What does a junior analyst see?

6. Prove it

Can we show what happened?

Audit logs, access reviews, lineage, change history, alerts

First check: Can you answer 'who read this last week'?

7. Share and retire safely

Where does data go, and when does it go away?

Sharing, extracts, retention, deletion, backup, environments

First check: Production data in test?

Identity and access

Who can do what

Most data breaches are access mistakes, not clever attacks: an over-wide group, a shared account, a forgotten service account.

Youperson or jobWho are you?identityAllowed?permissionsWhich rows?filters, masksDataAudit log: who read what, and whenSecrets stay in a secret store, never in code.

Layers of access control. Each gate narrows what is visible; the audit log proves it.

PrincipleWhat it meansGood practiceCommon mistakeVerdict
Least privilegeOnly the access the task needsGrant on groups and roles, not individuals'Just give them admin to unblock'Always
Separate identities for people and jobsPipelines run as service identities, not as a personOne service principal per pipeline or domainJobs running under an engineer's loginAlways
Short-lived credentialsTokens that expireManaged identities, federated loginLong-lived tokens in configPursue
Access by group, reviewedMembership is the control pointPeriodic access reviewAccounts of leavers still activeAlways
Separation of dutiesBuilder is not approverReview before productionOne person writes and ships alonePursue for sensitive systems
Environment separationDev, test, prod have separate accessNo production write access from devSame credentials everywhereAlways

Secrets

Keys, passwords and tokens

Treat any credential that has been pasted in a chat, committed to code or printed in a log as compromised.

DoDo notWhy
Keep secrets in a secret store and fetch at run timeHard-code them in code, notebooks or configCode spreads; secret stores have access control and audit
Rotate on a schedule and on any suspicionKeep one key for yearsLimits the window of misuse
Use managed identities where the platform allowsShare passwords between systemsNo secret to leak
Scan code and logs for credentialsPrint connection strings in logsLogs are widely readable

Protection in place

Encryption and network paths

These controls are mostly platform settings. A data engineer's job is to check they are on and not bypassed.

ControlWhat it coversCheckVerdict
Encryption at restStored files and disksEnabled on storage; know who holds the keysDefault on; verify
Encryption in transitData on the networkTLS for every connection, including to sourcesAlways
Customer-managed keysYou control the keyWhere policy requires; adds operational dutyWhen required
Private networkingTraffic stays off the public internetPrivate endpoints, restricted egressPursue for sensitive data
Public buckets and open endpointsAnyone can reach itScan for public accessAlways fix

Know the data

Classification before control

Controls are applied to columns and tables, so you need a reliable list of which ones are sensitive.

StepWhatWhere it livesVerdict
ClassifyMark personal, financial, health, secretTags on columns and tablesFirst step
DiscoverScan for sensitive values in unknown columnsAutomated scans, reviewed by peoplePursue
TraceFollow sensitive columns downstreamLineagePursue
AgreeRecord what each dataset may be used forData contracts and catalog entriesPursue
MinimiseDo not collect or copy what you do not needDesign reviewsPursue first

Limit what is seen

Row, column and value controls

Different people may query the same table and see different things.

TechniqueWhat it doesUse whenTrade-off
Row filtersHide rows a user may not seeRegion or tenant separationPolicy must be maintained
Column masksShow partial or hashed valuesAnalysts need shape, not identityMasked data cannot be joined on raw values
Views as a gateExpose only chosen columnsSimple, quick controlEasy to bypass if base table is open
Tokenisation / pseudonymisationReplace identifiers with stable tokensJoin across datasets without identityToken store becomes sensitive
Aggregation and minimum group sizeShare totals, not individualsReportingSmall groups can leak

Prove it

Audit and review

A control you cannot evidence did not happen, as far as an auditor is concerned.

EvidenceAnswersKeep forVerdict
Access audit logWho read or changed what and whenAs policy requiresAlways on
LineageWhere a sensitive column wentLife of the dataPursue
Change historyWho changed a pipeline or permissionLife of the systemAlways
Access reviewsWho still needs accessEach cyclePursue
Alerts on unusual accessA burst of reads or new exportsOngoingPursue

Share and retire safely

Copies, extracts and deletion

Data leaves the platform in extracts, exports, test copies and screenshots. Plan for the exit as well as the entry.

SituationRiskGood practiceVerdict
Production data in testReal personal data in weak environmentsUse synthetic or masked dataAlways
Extracts to files and external toolsLose control and auditShare in place with access controlPursue
RetentionKeeping data foreverSet a period and delete after itPursue
Right to erasureCannot find all copiesKnow every location; design deletes inPlan early
Backups and historyDeleted data lives on in old versionsAlign clean-up with deletion dutiesPlan early

In practice

CostSherlock rules for this topic

CostSherlock is our open set of cost and performance rules. Each rule below has a plain meaning here; the rule id is its name in CostSherlock. They are grouped by the same branches as the mind map, so a finding tells you which branch to open.

Identity and access

CostSherlock ruleWhat it catches, in plain words
uc_cluster_security_mode_mismatchCompute mode does not enforce the intended access model
uc_cluster_unauthorized_runtime_changeCompute changed outside policy
uc_no_cluster_policy_enforcedNo policy governs compute creation
uc_cluster_policy_permissive_instance_typesPolicy allows almost anything
uc_personal_compute_proliferationPersonal compute multiplying
uc_orphaned_cluster_creator_inactiveResources whose creator has left

Prove it

CostSherlock ruleWhat it catches, in plain words
uc_audit_burst_anomalyUnusual burst of audited activity
uc_cluster_creation_anomaly_by_userUnusual compute creation by a user
uc_maintenance_history_staleMaintenance history out of date

Know the data and keep it tidy

CostSherlock ruleWhat it catches, in plain words
uc_cluster_no_cost_tagsResources without ownership tags
uc_external_bi_tool_unbounded_refreshExternal tool pulling unbounded data

FAQ

Questions people ask

What is the single biggest security habit for a data engineer?

Least privilege on service identities plus no secrets in code. Together they remove the most common causes of data exposure.

Do I need to encrypt data myself?

Mostly the platform does at rest and in transit. Your job is to confirm it is on, understand who holds the keys, and never turn it off to unblock a problem.

What is the difference between masking and removing a column?

Masking keeps the column but hides or alters values per user. Removing keeps the data out of reach entirely. Use masking when the shape is useful.

Why does lineage matter for security?

When a sensitive column is found, lineage tells you every downstream table, dashboard and export that now carries it.

Is this page legal or compliance advice?

No. It explains engineering principles. Follow your organisation's policy, local law and your platform's current documentation.

Does CostSherlock check security?

Only a few signals that overlap with cost and governance, such as unmanaged compute and unusual activity. It is not a security scanner.

Related

Where it connects

Performance fundamentals · Cost fundamentals · Processing fundamentals · Storage fundamentals · Database families · Where data engineering time goes · Who maintains your tables · All data engineering

The numbers and thresholds on this page are rules of thumb and orders of magnitude to build intuition, not benchmarks. Check your platform's current documentation and measure your own workload.