Home › Key concepts › Idempotency
Key concept · Crawl (beginner) · the same on every platform
Idempotency
Running the same thing twice gives the same result as running it once.
What it is
An operation is idempotent if repeating it does no extra harm. Pipelines, APIs and agents retry after failures, so every step must be safe to run again: no duplicate rows, no second payment, no second email.
See it happen: DataTwin Studio: the same file sent twice (simulation), recorded from a real run.
Why it matters
Retries are everywhere. Without idempotency, every retry is a chance to double-count or double-charge.
An example
A load uses MERGE on the order id instead of INSERT, so re-running it updates the same rows instead of adding copies.
The common mistake
Appending rows on every run, so one failed-then-retried job doubles yesterday's revenue.
Where you meet it on each platform
The idea stays the same; only the product name changes.
| Job | AWS | Azure | Google Cloud | Databricks | Open source |
|---|---|---|---|---|---|
| Open table formatfiles that behave like a database table, with history | Apache Iceberg (S3 Tables) | Delta Lake in OneLake | Iceberg on BigLake | Delta Lake | Delta Lake or Iceberg |
| App databasewhere an app keeps its own rows | RDS Postgres / DynamoDB | Azure Database for PostgreSQL / Cosmos DB | Cloud SQL / Firestore | Lakebase (Postgres) | Postgres |
Related ideas
Where it shows up
Part of the Crawl level on these paths: Data Engineering (moving data) · AI Engineering (engineering basics) · Software Engineering (building services).