Home › Key concepts › Idempotency

Key concept · Crawl (beginner) · the same on every platform

Idempotency

Running the same thing twice gives the same result as running it once.

What it is

An operation is idempotent if repeating it does no extra harm. Pipelines, APIs and agents retry after failures, so every step must be safe to run again: no duplicate rows, no second payment, no second email.

See it happen: DataTwin Studio: the same file sent twice (simulation), recorded from a real run.

Why it matters

Retries are everywhere. Without idempotency, every retry is a chance to double-count or double-charge.

An example

A load uses MERGE on the order id instead of INSERT, so re-running it updates the same rows instead of adding copies.

The common mistake

Appending rows on every run, so one failed-then-retried job doubles yesterday's revenue.

Where you meet it on each platform

The idea stays the same; only the product name changes.

JobAWSAzureGoogle CloudDatabricksOpen source
Open table formatfiles that behave like a database table, with historyApache Iceberg (S3 Tables)Delta Lake in OneLakeIceberg on BigLakeDelta LakeDelta Lake or Iceberg
App databasewhere an app keeps its own rowsRDS Postgres / DynamoDBAzure Database for PostgreSQL / Cosmos DBCloud SQL / FirestoreLakebase (Postgres)Postgres

Related ideas

Where it shows up

Part of the Crawl level on these paths: Data Engineering (moving data) · AI Engineering (engineering basics) · Software Engineering (building services).