cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Operations › Resilience

Data PlatformZoom level 2

Resilience: retry, circuit breaker and connection limit

Sources fail in passing ways. A database times out. An interface answers that it is busy. Resilience is the small part that decides what to do then: try again, wait, or stop asking.

Without it, one slow minute at a source fails a whole run. With too much of it, a broken source is hammered by retries. Three protections together handle both.

Pipelinewants to read a sourceCircuit breakeropen: refuse at onceConnection capwait for a free slotRetrywait, then try againguard.callall three togetherSource
Every pull from a source goes through the guard. The three boxes before it are what it applies.
ProtectionWhat it doesNumbers from the code
Retry with backoffOn an error declared temporary, waits and tries again. Each wait is longer than the last. Any other error is raised at once.2 retries, first wait 60 seconds, each wait twice the last, no wait longer than 600 seconds
Circuit breakerAfter several failed calls in a row, the source is not called at all for a while. Then one trial call is let through.Opens after 3 failures, stays open 1800 seconds
Connection capAt most a set number of calls to one source run at the same time.No cap unless one is given for the source

A temporary error is a timeout, a connection error, or the platform's own TransientError, which an adapter raises for failures it knows will pass. A wrong column name is a bug, and a bug is never retried.

A worked example

The dealer interface is read every six hours. On one read it fails twice with passing errors, then answers. With a first wait of 10 seconds, as in the test:

TryResultWait before the next try
1A temporary error10 seconds
2Connection error20 seconds
3The page of dealersNone

The breaker counts this as one successful call. It counts calls that failed after all their retries, not each failed try.

The settings that drive it

SettingScopeMeaningDefaultRead by the code?
retry_countDatasetTries after the first.2Yes
retry_interval_secondsDatasetWait before the first retry.60Yes
max_concurrent_datasetsSourceThe connection cap.NoneNamed in the module's notes as the cap. The run scripts build the guard without caps.
circuit_failure_thresholdDataset, proposedFailures in a row before the circuit opens.3No. The guard uses its own default.
circuit_reset_secondsDataset, proposedHow long the circuit stays open.1800No. The guard uses its own default.

The code

src/dataplatform/resilience.py:51The retry loop. Sleeping is a function passed in, so tests do not wait.
def call_with_retry(fn: Callable[[], T], policy: RetryPolicy, sleep: Callable[[float], None],
                    on_retry: Callable[[int, BaseException, float], None] | None = None) -> T:
    """Call fn; on a declared temporary error wait and try again, up to policy.retries times. The last error is raised
    when every try failed; any other error is raised at once."""
    waits = policy.waits()
    for attempt in range(len(waits) + 1):
        try:
            return fn()
        except policy.retry_on as exc:
            if attempt == len(waits):
                raise
            if on_retry is not None:
                on_retry(attempt + 1, exc, waits[attempt])
            sleep(waits[attempt])
    raise AssertionError("unreachable")  # pragma: no cover

The waits are worked out once from the policy. The loop tries one more time than there are waits. Only the error types in the policy are caught. On the last try the error is raised as it is. Because sleep is an argument, a test passes a function that records the wait and returns at once.

src/dataplatform/resilience.py:68The breaker's three states, worked out from two numbers.
@dataclass
class CircuitBreaker:
    """closed: calls go through. open: calls are refused until reset_s has passed since it opened. half-open: one trial
    call; success closes it, failure opens it again."""

    threshold: int = 3
    reset_s: float = 1800.0
    failures: int = 0
    opened_at: float | None = None

    def state(self, now: float) -> str:
        if self.opened_at is None:
            return "closed"
        return "half-open" if now - self.opened_at >= self.reset_s else "open"

    def before(self, now: float, source: str) -> None:
        if self.state(now) == "open":
            raise CircuitOpen(f"{source}: {self.failures} failures in a row; not called again until "
                              f"{self.reset_s:.0f}s after the last one")

    def success(self) -> None:
        self.failures, self.opened_at = 0, None

    def failure(self, now: float) -> None:
        self.failures += 1
        if self.failures >= self.threshold or self.opened_at is not None:
            self.opened_at = now

The breaker stores only a count of failures and the time it opened. Closed means it never opened or was reset by a success. Open means the reset time has not passed. Half-open means it has: one call is let through, a success closes the breaker, and a failure opens it again from now.

src/dataplatform/resilience.py:123The three together. This is what an ingestion pipeline calls.
    def call(self, source: str, fn: Callable[[], T], policy: RetryPolicy | None = None) -> T:
        breaker = self.breaker(source)
        breaker.before(self.clock(), source)
        sem = self._sem(source)

        def once() -> T:
            if sem is None:
                return fn()
            with sem:
                return fn()

        def noted(attempt: int, exc: BaseException, wait: float) -> None:
            self.retries_seen.append((source, attempt, type(exc).__name__))

        try:  # the breaker counts calls that failed after their retries, not each failed try
            out = call_with_retry(once, policy or RetryPolicy(), self.sleep, noted)
        except BaseException:
            breaker.failure(self.clock())
            raise
        breaker.success()
        return out
src/dataplatform/ingest/pipelines.py:62Where ingestion uses it: every read of a source goes through the run's guard.
def _read(ctx: Ctx, src: Sources, fn: Callable[[], T]) -> T:
    """One call to a source through the run's guard: retried on temporary errors per the dataset's retry settings,
    refused while the source's circuit is open, limited by the source's connection cap."""
    if src.guard is None:
        return fn()
    return src.guard.call(ctx.spec.source, fn, RetryPolicy.from_settings(ctx.spec.settings))

How it is tested

tests/dataplatform/test_resilience.py uses a fake clock and a recorded sleep. The retry table gives the errors a call raises in order, the policy, the expected result, the number of calls made and the waits.

RETRY_CASES = [
    ("works first time: no wait", [], RetryPolicy(2, 10), "ok", 1, []),
    ("a timeout then success: one wait", [TimeoutError("slow")], RetryPolicy(2, 10), "ok", 2, [10]),
    ("two temporary errors: backoff doubles", [TransientError("x"), ConnectionError("y")], RetryPolicy(2, 10), "ok", 3,
     [10, 20]),
    ("out of retries: the last error is raised", [TimeoutError("a"), TimeoutError("b"), TimeoutError("c")],
     RetryPolicy(2, 10), TimeoutError, 3, [10, 20]),
    ("a bug is never retried", [ValueError("bad column")], RetryPolicy(5, 10), ValueError, 1, []),
    ("waits are capped", [TimeoutError()] * 3, RetryPolicy(3, 100, factor=10, max_wait_s=300), "ok", 4,
     [100, 300, 300]),
    ("no retries configured", [TimeoutError("a")], RetryPolicy(0, 10), TimeoutError, 1, []),
]
Not built yet, or known gaps
  • The breaker's state lives in the guard for one run. A source that fails in every run is tried again in each new run. The module's notes say a breaker that must outlive runs belongs in the metadata store, and that is not built.
  • The two circuit settings are proposed in the registry and are not read from a dataset's settings.
  • The run scripts create the guard with no connection caps, so the cap is tested but not in effect in the runs this site describes.
  • Retries are kept in a list on the guard for the run log. I did not find the code that writes them to the run log.
  • Push sources, where a mover delivers files to landing, do not go through the guard. It protects pulls.