Home › AWS › AWS data services › Macie
AWS data services · Security, identity and network · about 11 min read
Amazon Macie
Scans S3 buckets for personal and sensitive data and flags buckets that are public, unencrypted or shared outside your account.
In plain words
What it is
Amazon Macie is an AWS service that looks inside your Amazon S3 buckets and tells you where personal or sensitive data is sitting. Personal data means anything that identifies a person: a name, an email address, a phone number, a national ID number such as an Indian PAN or Aadhaar number, a bank account or a credit card number. Macie reads the files, matches what it sees against a library of built-in detectors called managed data identifiers, and writes a finding for every file where it finds a match.
Macie also does a second, simpler job without reading any file. It keeps an inventory of every bucket in the account and checks its settings: is the bucket public, is default encryption on, is it shared with another AWS account, is replication going somewhere unexpected. When a setting is risky it writes a policy finding.
A concrete example: a retail company receives a nightly CSV of loyalty-card orders from a partner over SFTP. The contract says the file holds only a hashed customer key. One night the partner changes its export and the file now carries full names, PAN numbers and delivery addresses. The landing bucket accepts it as it accepts every file. Macie, scanning the landing bucket, finds names and PAN numbers in a column labelled pan and raises a finding. That finding reaches the privacy team through EventBridge before the data has spread into silver and gold tables.
Macie works only on S3. It does not scan databases, Kafka topics or Delta tables as tables; it scans the files underneath. It reads common formats such as CSV, JSON, Parquet, Avro, Excel, PDF, Word documents and compressed archives.
A customs officer who opens a sample of the parcels arriving at the warehouse door and shouts when one holds something that should not be there.
Big picture
Where it fits
The lit lane is where Macie works. Boxes on the left feed it; boxes on the right use what it produces.
On the reference platform, Macie sits beside the landing zone, the first S3 hop where movers such as DMS, AppFlow and Transfer Family drop files exactly as received. The design already keeps personal data out of the lake: the lander swaps names, phone numbers and IDs for tokens as data enters bronze, and the real values live only in a separate token vault. Macie is the check that this promise holds. It scans the landing buckets weekly for personal data in places where the contract says there should be none.
Macie is a detective control, not a preventive one. It does not stop a file from landing. Its findings go to EventBridge and Security Hub, and from there to the privacy team's queue. The usual fix is upstream: the source owner corrects the export, the dataset gets a personal-data tag in its YAML so the lander tokenises the new columns, and the bad landing file is deleted.
Unity Catalog handles the table side. Column tags, masks and data classification on tables cover silver and gold, where data is already typed and governed. Comprehend finds names in free text, such as warranty notes, where a table classifier sees only a long string. Macie covers the gap neither of them reaches: raw files in S3 before any table exists.
How it works
The ideas you need
| Idea | What it means |
|---|---|
| Managed data identifier | A built-in detector that AWS maintains for one kind of sensitive data, such as credit card numbers, email addresses or a country's tax ID. Many are country-specific. |
| Custom data identifier | A detector you write yourself as a regular expression plus optional keywords near the match, for example an internal employee number or a loyalty-card format. |
| Allow list | Text or patterns Macie should ignore, such as the company's own head-office phone number or a test email domain, so they stop raising findings. |
| Automated sensitive data discovery | A background mode where Macie samples objects across your buckets every day and builds a sensitivity score per bucket, without you defining jobs. |
| Sensitive data discovery job | A scan you define: which buckets, which identifiers, run once or on a schedule (daily, weekly, monthly). Gives full coverage of the chosen objects. |
| Sensitive data finding | A report that one S3 object holds a given type of sensitive data, with counts and, for many formats, the location (row, column, page) of each match. |
| Policy finding | A report that a bucket's settings got riskier, such as public access allowed, encryption turned off or the bucket shared with an outside account. |
| Discovery results repository | An S3 bucket you must configure where Macie writes a detailed record for every object it analysed, including objects with no findings. It is your evidence of coverage. |
Choosing
When to use it, and when not
Use it when
- Use it to scan landing buckets that receive files from outside parties, where you cannot fully trust what arrives.
- Use it when an auditor or privacy officer asks you to prove that personal data is only where the register says it is.
- Use it to keep an inventory of every bucket's public access, encryption and sharing settings across all accounts in the organisation, at a low per-bucket price.
- Use it after a migration or acquisition, when a pile of old buckets arrives and nobody knows what is in them.
- Use it with custom identifiers when your own formats, such as dealer codes or employee numbers, count as personal data.
Avoid it when
- Avoid scanning every bronze, silver and gold Delta table on a schedule; at a per-GB price, a large lake scanned weekly becomes one of the bigger security bills, and Unity Catalog classification covers tables more cheaply.
- Avoid treating it as prevention. It finds data after it has landed; tokenising at the door and data contracts with sources stop it from landing.
- Avoid it for data outside S3, such as databases, streams or SaaS systems; it simply cannot see them.
- Avoid relying on it alone for free text in local languages; detection is tuned for structured fields and common formats, and recall on names in Hindi or Marathi text is limited.
Production grade
What goes wrong in production
| Problem | What breaks | What to do |
|---|---|---|
| It cannot read the files | Every landing object is encrypted with a customer-managed KMS key, and the key policy does not allow Macie, so Macie records only metadata and reports nothing useful. | Add a statement to each landing key's policy allowing the Macie service-linked role kms:Decrypt; check the 'unclassifiable objects' counts after the first run. |
| Bill grows with the lake | Someone enables automated discovery and broad jobs on every bucket; Macie samples terabytes of Parquet that hold only tokens. | Exclude bronze to gold buckets from automated discovery, scope jobs to landing prefixes, and set the monthly analysis quota as a spend cap. |
| Too many false positives | Built-in identifiers flag test data, sample addresses or the company's own phone numbers, and the privacy team starts ignoring findings. | Add allow lists, suppress known patterns with suppression rules, and review the top noisy identifiers monthly. |
| Landing already deleted | Landing files with personal data are deleted after one day, so a weekly job finds nothing even when bad data passed through. | For sources flagged as personal, run a daily job on the new prefix, or scan a quarantine copy; keep the discovery results as evidence of coverage. |
| Files too large or unsupported | Huge archives or odd formats are skipped silently, which looks like 'no findings'. | Watch the skipped and unclassifiable counts in job statistics; split large exports at the source. |
| Findings nobody owns | Findings sit in the Macie console where no one looks. | Route findings through EventBridge to Security Hub and a ticket queue with a named owner per source, and track time to close. |
| Multi-account blind spots | Macie is on in the platform account but not in the account where a team created its own landing bucket. | Use a delegated administrator account under Organizations and auto-enable Macie for new member accounts. |
Alternatives
What to compare it with
| Alternative | Pick the alternative when | Pick Macie when |
|---|---|---|
| Unity Catalog data classification | You want sensitive columns tagged on governed tables in silver and gold, with masks applied from the tags. | You need to inspect raw files in S3 before any table exists. |
| Amazon Comprehend PII detection | You need to find names and addresses inside free-text fields, such as warranty notes or chat logs, inside a pipeline. | You want bucket-wide scanning and an inventory without writing code. |
| Microsoft Purview (Azure) | The platform runs on Azure Data Lake Storage and you want one scanner across Azure and Microsoft 365. | The data lives in S3 on AWS. |
| Google Sensitive Data Protection | The data lives in Cloud Storage or BigQuery on Google Cloud. | The data lives in S3 on AWS. |
| Presidio or Spark regular expressions | You want full control, local-language recognisers, and to scan inside the pipeline at no per-GB fee. | You do not want to build, tune and run your own scanner and its evidence trail. |
Cost
How you pay
Macie bills on three meters. Bucket inventory and monitoring is a small monthly charge per bucket. Automated sensitive data discovery adds a charge per object monitored and per GB inspected. Discovery jobs charge per GB inspected, plus the normal S3 GET and LIST request charges. Each newly enabled account gets a 30-day free trial for bucket monitoring and up to 150 GB of automated discovery; jobs are not part of the trial. As a guide, the US East list price is about one dollar per GB inspected (list price, check your region).
What drives the bill
- Gigabytes inspected, which is almost all of the bill on a data platform.
- How many buckets and objects automated discovery watches.
- How often jobs run, and whether they re-scan objects already scanned.
- The number of accounts with Macie enabled across the organisation.
Cost traps
- A full job on the whole lake every week rescans the same terabytes again and again; use incremental scheduled jobs that only scan new objects.
- Automated discovery left on for all buckets samples gold tables full of tokens and finds nothing.
- Raising the 5 TB monthly job quota without a reason removes your only built-in spend cap.
- S3 request charges from jobs over many small files can exceed the Macie charge itself.
Prices change; check the AWS pricing page for your region before deciding.
Security and governance
Keeping it safe
- Macie uses a service-linked role, AWSServiceRoleForAmazonMacie, to read buckets and objects; keep it, and grant it kms:Decrypt on the landing keys only, not on the vault or recipe keys.
- Findings can reveal samples of the sensitive data itself; restrict the macie2:GetSensitiveDataOccurrences permission to the privacy team.
- The discovery results repository must be an S3 bucket encrypted with its own KMS key, in the security account, with versioning and tight access.
- Run Macie from a delegated administrator in the security account, so platform engineers cannot switch it off; an SCP can deny macie2:DisableMacie in member accounts.
- Every Macie API call is logged in CloudTrail, giving an audit trail of who changed jobs, allow lists or suppression rules.
Operating it
What to watch, and the limits
| Watch | Why |
|---|---|
| Sensitive data findings per landing bucket | The main signal: new findings in buckets where the register says no personal data should be. |
| Policy findings (Policy:IAMUser/S3BucketPublic and similar) | A bucket became public, lost encryption or was shared outside the account. |
| Job status and statistics | Whether scheduled jobs ran, how many objects were analysed, skipped or unclassifiable. |
| Unclassifiable objects by reason | Objects Macie could not read, often because of a KMS key policy, which means a blind spot. |
| Usage against the monthly quota | How close you are to the 5 TB job quota and the expected bill. |
| Time to close a finding | How long from finding to fixed source, the number the privacy team should report. |
| Limit | Value and notes |
|---|---|
| Data analysed by jobs | 5 TB per account per month by default, raisable to 1 PB through Service Quotas |
| Buckets fully monitored | 10,000 per account; beyond that only the most recently created or changed are monitored |
| Custom data identifiers | 10,000 per account, and up to 30 per job |
| Allow lists | 10 per account (up to 5 text lists and 5 regular-expression lists) |
| Single file analysed | Up to 8 GB for Parquet, Avro, gzip and ZIP; larger files are skipped entirely |
| Findings per job run | 100,000 in the console and API, plus 5 per cent of the rest; all are still sent to EventBridge |
Limits and quotas checked 2026-11-11; many can be raised through Service Quotas.
Decisions
Situations and verdicts
| Situation | Choice | Verdict |
|---|---|---|
| Weekly scan of landing buckets for personal data in the wrong place | Scheduled Macie job on landing prefixes, findings to Security Hub | Pursue: the site's design; small data volume, high value, evidence for audits. |
| Scan every Delta table in bronze, silver and gold | Macie automated discovery on the whole lake | Drop: tables are tokenised and tagged; use Unity Catalog classification and masks instead. |
| Names hidden in warranty notes and service tickets | Comprehend PII detection inside the pipeline | Pursue with care: Macie sees the file, not the meaning of free text; test recall on your languages. |
| Bucket inventory across all accounts | Macie bucket monitoring from the delegated admin | Pursue: a few cents per bucket for a live view of public, unencrypted or shared buckets. |
FAQ
Questions people ask
Does Amazon Macie scan Delta Lake or Iceberg tables?
It scans the Parquet files underneath them as plain files. It does not understand the table, so it scans deleted and old file versions too. For governed tables, classify columns in Unity Catalog instead.
Can Macie read files encrypted with my own KMS key?
Yes, but only if the key policy allows the Macie service-linked role to use kms:Decrypt. Without that, Macie reports only metadata. Files encrypted on the client side or with SSE-C cannot be read.
How much does Amazon Macie cost?
A small monthly fee per bucket, a fee per object monitored by automated discovery, and a fee per GB inspected, which dominates on a data platform. Scope scans to landing buckets and use the monthly quota as a cap.
Does Macie stop personal data from landing?
No. It is a detective control: it reports after the file is in S3. Prevention comes from data contracts and tokenising at the door.
Does Macie detect Indian identifiers such as PAN or Aadhaar?
Yes: there are managed identifiers for Aadhaar, PAN and Indian driving licences, and they need a keyword such as a column name nearby. There is no built-in identifier for Indian phone numbers or Indian postal addresses, and names are found only in Latin script, so add custom identifiers for those.
Sources
Checked against
- AWS: Amazon Macie pricing
- AWS docs: Quotas for Macie
- AWS docs: Analyzing encrypted S3 objects
- AWS docs: What is Amazon Macie
- AWS docs: Managed data identifiers by type
Checked 2026-11-11.
Related
Where it connects
AWS and Databricks reference platform · Ingestion design · Amazon Comprehend · AWS KMS · Amazon S3 · All AWS data services