Home › AWS › AWS data services › DataSync
AWS data services · Ingestion and CDC · about 11 min read
AWS DataSync
A managed copy service that moves files and objects between on-premises shares, other clouds and AWS storage, checking every file it copies.
In plain words
What it is
Before a data platform can analyse anything, the data has to be in the cloud. A lot of it sits in places that were never meant to be read by a lakehouse: a Windows file share in a plant full of CSV exports from the machine historian, a NetApp filer with ten years of quality lab reports, an HDFS cluster from an old Hadoop project, or a bucket in another cloud that a retail team filled with clickstream files. Copying all of that by hand with scripts is slow, fragile and hard to prove complete.
AWS DataSync is a managed copy service for exactly this. You define a source location, such as an NFS or SMB share, HDFS, an object store or another cloud's bucket, and a destination location, such as an S3 bucket, EFS or an FSx file system. You then create a task that says what to copy, with include and exclude filters, and how to treat files that already exist. DataSync copies in parallel, keeps file metadata where the destination supports it, checks data integrity during the transfer and can verify the destination afterwards, and writes a report of every file it moved, skipped or failed.
For sources outside AWS that it cannot reach directly, you install a small DataSync agent, which is a virtual machine you run on VMware, Hyper-V, KVM, or as an EC2 instance, close to the data. For several other clouds' object stores, DataSync can now copy without any agent at all. There are two task modes. Basic mode is the original. Enhanced mode lists, prepares, transfers and verifies in parallel, removes the per-task file count limit and gives more detailed metrics.
A concrete example: a tyre maker wants three years of curing-press readings that sit as daily CSV files on a file server in each plant. A DataSync agent at each plant copies the share into s3://landing/historian/plant-07/ once, over a weekend, with full verification. After that, the live readings arrive through SiteWise Edge and Kinesis, and DataSync is no longer needed for that source.
A professional moving company that packs every box, counts it on both ends and hands you an inventory list, but is not the postal service you use every day.
Big picture
Where it fits
The lit lane is where DataSync works. Boxes on the left feed it; boxes on the right use what it produces.
DataSync sits in the move-in lane, on the files hop, beside Transfer Family. On the reference platform the reference table lists both under files and SFTP. Transfer Family is the daily door for partners who push files. DataSync is the tool you reach for when you need to pull a large body of existing files into the landing bucket, once or on a modest schedule.
Its typical jobs on this platform are backfills and migrations. It loads history that predates the streaming lane, moves an old Hadoop or file-server data set into S3 before switching it off, copies an archive from another cloud during a consolidation, and copies between AWS storage services when a team needs files from EFS or FSx in S3.
Whatever DataSync writes is just files in S3 landing. The Databricks lander reads them with Auto Loader exactly as it reads any other files feed, records them in its file log and writes bronze. That keeps DataSync behind the same port as every other mover: a migration does not need its own pipeline, only a YAML entry and a landing prefix.
How it works
The ideas you need
| Idea | What it means |
|---|---|
| Location | A source or destination DataSync knows how to reach, such as an NFS share, an S3 bucket with a prefix, or an Azure Blob container. |
| Agent | A virtual machine you deploy near the source so DataSync can read storage it cannot reach from AWS. One agent can drive transfers at up to about 10 Gbps per task. |
| Task | A saved copy job: source, destination, filters, options and schedule. Running it once is a task execution. |
| Basic and Enhanced mode | The two task modes. Enhanced mode runs in parallel stages, has no practical file count limit per execution and gives richer metrics, but is billed differently and supports a set of locations that keeps growing. |
| Verification | After copying, DataSync can compare the transferred data, or all data, at source and destination, so you can show the copy is complete. |
| Transfer mode | Whether to copy only changed data or everything, and whether to keep or delete files at the destination that no longer exist at the source. |
| Manifest | A file listing exactly which files or objects to copy, so a task can move a known set rather than walking a whole share. |
| Task report | A per-file record of what was transferred, skipped, verified or failed, written to S3 for audit and retry. |
Choosing
When to use it, and when not
Use it when
- You need to copy a large existing file share, HDFS cluster or object store into S3 once, with proof that every file arrived.
- You are retiring an on-premises Hadoop or file server and want its history in the lakehouse landing area first.
- You are consolidating data from another cloud's buckets into S3 and want an AWS-run copy without building scripts.
- A team must keep a share and an S3 prefix in step on a daily or weekly schedule for a limited period while a real feed is built.
- You need to copy between AWS storage services, such as EFS or FSx to S3, with metadata and verification.
Avoid it when
- Partners push files to you every day; Transfer Family gives them logins and folders, which DataSync does not.
- The data is in a database; use DMS or Lakeflow Connect for change capture, not file copies of exports.
- Data must arrive within seconds or minutes; DataSync is a batch copier, not a stream.
- The link is too slow for the volume, for example hundreds of terabytes over a thin plant line; an offline device or a Direct Connect upgrade comes first.
- You are just copying between two S3 buckets; S3 Replication or S3 Batch Operations is usually simpler and cheaper.
Production grade
What goes wrong in production
| Problem | What breaks | What to do |
|---|---|---|
| The plant link is saturated | A large copy fills the plant's WAN link and slows the systems that run production. | Set a bandwidth limit on the task, schedule runs at night, and size the copy against the line before starting. |
| Files change during the copy | Historian exports are still being written while DataSync reads them, so the copy has partial or mismatched files. | Exclude open or recent files with filters, copy from a snapshot where the filer supports it, and run a final changed-only pass after the source is frozen. |
| Millions of tiny files | Small files make the listing and preparation stages slow, and Basic mode tasks hit per-execution file count quotas. | Use Enhanced mode where the location is supported, split Basic mode tasks by folder with include filters, and give the agent enough memory. |
| Agent sizing and placement | An undersized agent, or one far from the share, becomes the bottleneck. | Follow the agent CPU and memory guidance for large file counts, place it on the same network as the storage, and use several agents for big jobs. |
| A scheduled task becomes a hidden feed | A one-off migration task keeps running weekly for years, with no owner, no contract and no alerts. | Register every DataSync task in the source register with an owner and an end date, and alert on failed executions. |
| Deletes propagate | Transfer mode set to remove deleted files wipes destination objects when someone tidies the source share. | Keep deleted files at the destination for landing copies, turn on S3 versioning, and review transfer options before each scheduled task. |
| Unexpected S3 request charges | Copying millions of small objects, or into Glacier classes, adds request charges on top of the DataSync fee. | Land in S3 Standard and let lifecycle rules move data later; estimate request counts before large copies. |
Alternatives
What to compare it with
| Alternative | Pick the alternative when | Pick DataSync when |
|---|---|---|
| AWS Transfer Family | Outside partners push files to you on their own schedule over SFTP, FTPS or AS2. | You are pulling a large body of existing files that you control. |
| AWS Snowball Edge | Hundreds of terabytes or more must move and the network would take weeks. | The network can carry the data in a reasonable time window. |
| S3 Replication or S3 Batch Operations | Both ends are S3 buckets. | The source is a file share, HDFS or another cloud. |
| Azure Storage Mover and AzCopy | Your platform runs on Azure and the destination is Blob or ADLS. | Your landing zone is S3. |
| Google Cloud Storage Transfer Service | Your platform runs on Google Cloud. | Your landing zone is S3. |
| rclone or rsync scripts | A small, one-time copy where you are happy to own the script. | You need parallel speed, verification, per-file reports and retries without writing them yourself. |
| Databricks reading the source directly | The source is already an object store Databricks can read with a storage credential. | The source is a file share or HDFS behind a plant firewall. |
Cost
How you pay
You pay a flat fee per GB copied, with a different rate for Basic and Enhanced mode, and Enhanced mode tasks also carry a fee per task execution. The AWS pricing examples show about 0.0125 US dollars per GB for Basic mode, 0.015 per GB for Enhanced mode and 0.55 per Enhanced execution; list price, check your region. On top of that you pay for S3 requests DataSync makes, data transfer out when copying out of AWS or across Regions, the agent's compute if it runs on EC2, and CloudWatch logs.
What drives the bill
- Total GB copied, including repeated full copies if transfer mode is set to copy everything.
- Number of task executions in Enhanced mode, especially frequent small scheduled runs.
- S3 request counts, driven by the number of files and by the storage class you land in.
- Data transfer out when the destination is outside AWS or in another Region.
Cost traps
- Running Enhanced mode tasks every few minutes on tiny changes, so the per-execution fee dominates.
- Landing directly in Glacier classes, where request and retrieval charges are much higher.
- Re-copying everything each run instead of only changed data.
- Forgetting the other cloud's egress charges, which are billed by that cloud, not AWS.
Prices change; check the AWS pricing page for your region before deciding.
Security and governance
Keeping it safe
- Give DataSync an IAM role scoped to the landing prefix only, and use a separate role per task or per source where practical.
- Encrypt the destination bucket with the catalog's KMS key; data in flight is encrypted with TLS between the agent and AWS.
- Activate agents to use VPC endpoints so traffic goes over PrivateLink and Direct Connect or VPN, not the public internet.
- Store share passwords and other clouds' access keys in Secrets Manager, and use read-only credentials at the source.
- Record task and location changes with CloudTrail and send task logs to CloudWatch; keep task reports as the migration's audit trail.
- Assume migrated archives contain personal data: Macie scans landing before the lander tokenises into bronze.
Operating it
What to watch, and the limits
| Watch | Why |
|---|---|
| Task execution status | SUCCESS or ERROR per run; alert on any failed or partially failed execution through EventBridge. |
| BytesTransferred and FilesTransferred | How much moved per execution; compare with the expected size of the source. |
| FilesVerified and verification errors | Whether the destination matches the source; a mismatch means the copy cannot be signed off. |
| Task report skipped and failed entries | Per-file reasons for skips and failures, used to plan a retry run. |
| Agent status | ONLINE or OFFLINE; an offline agent stops every task that uses it. |
| Limit | Value and notes |
|---|---|
| Tasks per account per Region | 100 by default, adjustable |
| Files per Basic mode execution | 50 million from on-premises or other clouds, 25 million between AWS services; Enhanced mode is virtually unlimited |
| Throughput per task | 10 Gbps with an agent, 5 Gbps without one |
| Concurrent Enhanced mode executions | 120 |
| Task execution history kept | 30 days |
Limits and quotas checked 2026-11-11; many can be raised through Service Quotas.
Decisions
Situations and verdicts
| Situation | Choice | Verdict |
|---|---|---|
| Load three years of historian CSV files from plant file servers | One agent per plant, Enhanced mode, verify all, run once | Pursue: built for this, with proof every file arrived. |
| Retire an old Hadoop cluster with 400 TB | DataSync from HDFS to S3 in phases, or Snowball if the link is slow | Pursue with care: plan bandwidth and freeze windows first. |
| A team asks for a daily share sync with no end date | A real feed through Transfer Family or the source system | Drop: a permanent DataSync task becomes an unowned feed; use it only as a bridge with an end date. |
| Copy a retail clickstream archive from another cloud's bucket | Agentless DataSync cross-cloud task | Pursue: no agent to run, and it lands straight in S3 landing. |
| Copy between two S3 buckets in the same account | S3 Replication or Batch Operations | Drop DataSync here: the native S3 tools are simpler. |
FAQ
Questions people ask
Do I need a DataSync agent?
You need one for on-premises shares and HDFS, and for some self-managed object stores. Copies between AWS storage services, and from several other clouds' object storage, can run without an agent.
What is the difference between Basic and Enhanced mode?
Enhanced mode runs listing, transfer and verification in parallel, removes the per-execution file count limit and gives more detail in metrics. It supports a growing set of locations and adds a per-execution fee.
Can DataSync keep a share and S3 in sync continuously?
It can run on a schedule and copy only changes, but it is a batch copier. For an ongoing business feed, build a proper ingestion route with an owner instead.
Does DataSync check that files are copied correctly?
Yes. It checks integrity during transfer and can verify transferred files, or all files, at the destination afterwards. Task reports list each file's result.
What happened to DataSync Discovery?
DataSync Discovery, which assessed on-premises storage before migration, reached end of support on 20 May 2025 and can no longer be used. DataSync's copy features are unaffected.
Sources
Checked against
- AWS docs: What is AWS DataSync
- AWS docs: DataSync quotas
- AWS: DataSync pricing
- AWS What's New: Enhanced mode adds HDFS, Azure Blob and object storage (Jul 2026)
- AWS What's New: cross-cloud transfers without agents (May 2025)
- AWS docs: services in full shutdown (DataSync Discovery)
Checked 2026-11-11.
Related
Where it connects
AWS Transfer Family · Amazon S3 · Common Ingestion Framework · Data platform on AWS and Databricks · Direct Connect, VPN and Transit Gateway · All AWS data services
← Data Exchange · IoT Core →