Home › AWS › AWS data services › DataSync

AWS data services · Ingestion and CDC · about 11 min read

AWS DataSync

A managed copy service that moves files and objects between on-premises shares, other clouds and AWS storage, checking every file it copies.

RoleIngestion and CDC
StatusGenerally available. Enhanced mode keeps widening (on-premises NFS and SMB in Dec 2025; HDFS, Azure Blob, EFS and FSx for Lustre in Jul 2026). DataSync Discovery reached end of support on 20 May 2025.
Verdict on this sitePursue with care
AzureAzure Storage Mover (and AzCopy)
Google CloudStorage Transfer Service
Databricksnone: Databricks reads what DataSync lands in S3 with Auto Loader
Open sourcerclone or rsync
Verdict: Pursue with care. Use it for bulk and one-off copies, such as moving years of historian exports from a plant file server or another cloud's bucket into S3. It is not the daily path for partner files (Transfer Family) or databases (DMS), and a scheduled DataSync task is no substitute for a real feed with an owner and a contract.

In plain words

What it is

Before a data platform can analyse anything, the data has to be in the cloud. A lot of it sits in places that were never meant to be read by a lakehouse: a Windows file share in a plant full of CSV exports from the machine historian, a NetApp filer with ten years of quality lab reports, an HDFS cluster from an old Hadoop project, or a bucket in another cloud that a retail team filled with clickstream files. Copying all of that by hand with scripts is slow, fragile and hard to prove complete.

AWS DataSync is a managed copy service for exactly this. You define a source location, such as an NFS or SMB share, HDFS, an object store or another cloud's bucket, and a destination location, such as an S3 bucket, EFS or an FSx file system. You then create a task that says what to copy, with include and exclude filters, and how to treat files that already exist. DataSync copies in parallel, keeps file metadata where the destination supports it, checks data integrity during the transfer and can verify the destination afterwards, and writes a report of every file it moved, skipped or failed.

For sources outside AWS that it cannot reach directly, you install a small DataSync agent, which is a virtual machine you run on VMware, Hyper-V, KVM, or as an EC2 instance, close to the data. For several other clouds' object stores, DataSync can now copy without any agent at all. There are two task modes. Basic mode is the original. Enhanced mode lists, prepares, transfers and verifies in parallel, removes the per-task file count limit and gives more detailed metrics.

A concrete example: a tyre maker wants three years of curing-press readings that sit as daily CSV files on a file server in each plant. A DataSync agent at each plant copies the share into s3://landing/historian/plant-07/ once, over a weekend, with full verification. After that, the live readings arrive through SiteWise Edge and Kinesis, and DataSync is no longer needed for that source.

A professional moving company that packs every box, counts it on both ends and hands you an inventory list, but is not the postal service you use every day.

Big picture

Where it fits

SourcesMove inStreamStoreProcessServeUseAcross everything: catalog · security · orchestration · observability and cost · infrastructure as codegets data fromhands data toOn-premises NFS, SMB and HDFS sharesOther clouds' object storageAmazon EFS and FSx file systemsDataSyncIngestion and CDCS3Databricks Auto Loader (the lander)S3 storage classes

The lit lane is where DataSync works. Boxes on the left feed it; boxes on the right use what it produces.

DataSync sits in the move-in lane, on the files hop, beside Transfer Family. On the reference platform the reference table lists both under files and SFTP. Transfer Family is the daily door for partners who push files. DataSync is the tool you reach for when you need to pull a large body of existing files into the landing bucket, once or on a modest schedule.

Its typical jobs on this platform are backfills and migrations. It loads history that predates the streaming lane, moves an old Hadoop or file-server data set into S3 before switching it off, copies an archive from another cloud during a consolidation, and copies between AWS storage services when a team needs files from EFS or FSx in S3.

Whatever DataSync writes is just files in S3 landing. The Databricks lander reads them with Auto Loader exactly as it reads any other files feed, records them in its file log and writes bronze. That keeps DataSync behind the same port as every other mover: a migration does not need its own pipeline, only a YAML entry and a landing prefix.

How it works

The ideas you need

IdeaWhat it means
LocationA source or destination DataSync knows how to reach, such as an NFS share, an S3 bucket with a prefix, or an Azure Blob container.
AgentA virtual machine you deploy near the source so DataSync can read storage it cannot reach from AWS. One agent can drive transfers at up to about 10 Gbps per task.
TaskA saved copy job: source, destination, filters, options and schedule. Running it once is a task execution.
Basic and Enhanced modeThe two task modes. Enhanced mode runs in parallel stages, has no practical file count limit per execution and gives richer metrics, but is billed differently and supports a set of locations that keeps growing.
VerificationAfter copying, DataSync can compare the transferred data, or all data, at source and destination, so you can show the copy is complete.
Transfer modeWhether to copy only changed data or everything, and whether to keep or delete files at the destination that no longer exist at the source.
ManifestA file listing exactly which files or objects to copy, so a task can move a known set rather than walking a whole share.
Task reportA per-file record of what was transferred, skipped, verified or failed, written to S3 for audit and retry.

Choosing

When to use it, and when not

Use it when

  • You need to copy a large existing file share, HDFS cluster or object store into S3 once, with proof that every file arrived.
  • You are retiring an on-premises Hadoop or file server and want its history in the lakehouse landing area first.
  • You are consolidating data from another cloud's buckets into S3 and want an AWS-run copy without building scripts.
  • A team must keep a share and an S3 prefix in step on a daily or weekly schedule for a limited period while a real feed is built.
  • You need to copy between AWS storage services, such as EFS or FSx to S3, with metadata and verification.

Avoid it when

  • Partners push files to you every day; Transfer Family gives them logins and folders, which DataSync does not.
  • The data is in a database; use DMS or Lakeflow Connect for change capture, not file copies of exports.
  • Data must arrive within seconds or minutes; DataSync is a batch copier, not a stream.
  • The link is too slow for the volume, for example hundreds of terabytes over a thin plant line; an offline device or a Direct Connect upgrade comes first.
  • You are just copying between two S3 buckets; S3 Replication or S3 Batch Operations is usually simpler and cheaper.

Production grade

What goes wrong in production

ProblemWhat breaksWhat to do
The plant link is saturatedA large copy fills the plant's WAN link and slows the systems that run production.Set a bandwidth limit on the task, schedule runs at night, and size the copy against the line before starting.
Files change during the copyHistorian exports are still being written while DataSync reads them, so the copy has partial or mismatched files.Exclude open or recent files with filters, copy from a snapshot where the filer supports it, and run a final changed-only pass after the source is frozen.
Millions of tiny filesSmall files make the listing and preparation stages slow, and Basic mode tasks hit per-execution file count quotas.Use Enhanced mode where the location is supported, split Basic mode tasks by folder with include filters, and give the agent enough memory.
Agent sizing and placementAn undersized agent, or one far from the share, becomes the bottleneck.Follow the agent CPU and memory guidance for large file counts, place it on the same network as the storage, and use several agents for big jobs.
A scheduled task becomes a hidden feedA one-off migration task keeps running weekly for years, with no owner, no contract and no alerts.Register every DataSync task in the source register with an owner and an end date, and alert on failed executions.
Deletes propagateTransfer mode set to remove deleted files wipes destination objects when someone tidies the source share.Keep deleted files at the destination for landing copies, turn on S3 versioning, and review transfer options before each scheduled task.
Unexpected S3 request chargesCopying millions of small objects, or into Glacier classes, adds request charges on top of the DataSync fee.Land in S3 Standard and let lifecycle rules move data later; estimate request counts before large copies.

Alternatives

What to compare it with

AlternativePick the alternative whenPick DataSync when
AWS Transfer FamilyOutside partners push files to you on their own schedule over SFTP, FTPS or AS2.You are pulling a large body of existing files that you control.
AWS Snowball EdgeHundreds of terabytes or more must move and the network would take weeks.The network can carry the data in a reasonable time window.
S3 Replication or S3 Batch OperationsBoth ends are S3 buckets.The source is a file share, HDFS or another cloud.
Azure Storage Mover and AzCopyYour platform runs on Azure and the destination is Blob or ADLS.Your landing zone is S3.
Google Cloud Storage Transfer ServiceYour platform runs on Google Cloud.Your landing zone is S3.
rclone or rsync scriptsA small, one-time copy where you are happy to own the script.You need parallel speed, verification, per-file reports and retries without writing them yourself.
Databricks reading the source directlyThe source is already an object store Databricks can read with a storage credential.The source is a file share or HDFS behind a plant firewall.

Cost

How you pay

You pay a flat fee per GB copied, with a different rate for Basic and Enhanced mode, and Enhanced mode tasks also carry a fee per task execution. The AWS pricing examples show about 0.0125 US dollars per GB for Basic mode, 0.015 per GB for Enhanced mode and 0.55 per Enhanced execution; list price, check your region. On top of that you pay for S3 requests DataSync makes, data transfer out when copying out of AWS or across Regions, the agent's compute if it runs on EC2, and CloudWatch logs.

What drives the bill

  • Total GB copied, including repeated full copies if transfer mode is set to copy everything.
  • Number of task executions in Enhanced mode, especially frequent small scheduled runs.
  • S3 request counts, driven by the number of files and by the storage class you land in.
  • Data transfer out when the destination is outside AWS or in another Region.

Cost traps

  • Running Enhanced mode tasks every few minutes on tiny changes, so the per-execution fee dominates.
  • Landing directly in Glacier classes, where request and retrieval charges are much higher.
  • Re-copying everything each run instead of only changed data.
  • Forgetting the other cloud's egress charges, which are billed by that cloud, not AWS.

Prices change; check the AWS pricing page for your region before deciding.

Security and governance

Keeping it safe

  • Give DataSync an IAM role scoped to the landing prefix only, and use a separate role per task or per source where practical.
  • Encrypt the destination bucket with the catalog's KMS key; data in flight is encrypted with TLS between the agent and AWS.
  • Activate agents to use VPC endpoints so traffic goes over PrivateLink and Direct Connect or VPN, not the public internet.
  • Store share passwords and other clouds' access keys in Secrets Manager, and use read-only credentials at the source.
  • Record task and location changes with CloudTrail and send task logs to CloudWatch; keep task reports as the migration's audit trail.
  • Assume migrated archives contain personal data: Macie scans landing before the lander tokenises into bronze.

Operating it

What to watch, and the limits

WatchWhy
Task execution statusSUCCESS or ERROR per run; alert on any failed or partially failed execution through EventBridge.
BytesTransferred and FilesTransferredHow much moved per execution; compare with the expected size of the source.
FilesVerified and verification errorsWhether the destination matches the source; a mismatch means the copy cannot be signed off.
Task report skipped and failed entriesPer-file reasons for skips and failures, used to plan a retry run.
Agent statusONLINE or OFFLINE; an offline agent stops every task that uses it.
LimitValue and notes
Tasks per account per Region100 by default, adjustable
Files per Basic mode execution50 million from on-premises or other clouds, 25 million between AWS services; Enhanced mode is virtually unlimited
Throughput per task10 Gbps with an agent, 5 Gbps without one
Concurrent Enhanced mode executions120
Task execution history kept30 days

Limits and quotas checked 2026-11-11; many can be raised through Service Quotas.

Decisions

Situations and verdicts

SituationChoiceVerdict
Load three years of historian CSV files from plant file serversOne agent per plant, Enhanced mode, verify all, run oncePursue: built for this, with proof every file arrived.
Retire an old Hadoop cluster with 400 TBDataSync from HDFS to S3 in phases, or Snowball if the link is slowPursue with care: plan bandwidth and freeze windows first.
A team asks for a daily share sync with no end dateA real feed through Transfer Family or the source systemDrop: a permanent DataSync task becomes an unowned feed; use it only as a bridge with an end date.
Copy a retail clickstream archive from another cloud's bucketAgentless DataSync cross-cloud taskPursue: no agent to run, and it lands straight in S3 landing.
Copy between two S3 buckets in the same accountS3 Replication or Batch OperationsDrop DataSync here: the native S3 tools are simpler.

FAQ

Questions people ask

Do I need a DataSync agent?

You need one for on-premises shares and HDFS, and for some self-managed object stores. Copies between AWS storage services, and from several other clouds' object storage, can run without an agent.

What is the difference between Basic and Enhanced mode?

Enhanced mode runs listing, transfer and verification in parallel, removes the per-execution file count limit and gives more detail in metrics. It supports a growing set of locations and adds a per-execution fee.

Can DataSync keep a share and S3 in sync continuously?

It can run on a schedule and copy only changes, but it is a batch copier. For an ongoing business feed, build a proper ingestion route with an owner instead.

Does DataSync check that files are copied correctly?

Yes. It checks integrity during transfer and can verify transferred files, or all files, at the destination afterwards. Task reports list each file's result.

What happened to DataSync Discovery?

DataSync Discovery, which assessed on-premises storage before migration, reached end of support on 20 May 2025 and can no longer be used. DataSync's copy features are unaffected.

Sources

Checked against

Checked 2026-11-11.

Related

Where it connects

AWS Transfer Family · Amazon S3 · Common Ingestion Framework · Data platform on AWS and Databricks · Direct Connect, VPN and Transit Gateway · All AWS data services

← Data Exchange · IoT Core →