Data PlatformZoom level 2
Resilience: retry, circuit breaker and connection limit
Sources fail in passing ways. A database times out. An interface answers that it is busy. Resilience is the small part that decides what to do then: try again, wait, or stop asking.
Without it, one slow minute at a source fails a whole run. With too much of it, a broken source is hammered by retries. Three protections together handle both.
| Protection | What it does | Numbers from the code |
|---|---|---|
| Retry with backoff | On an error declared temporary, waits and tries again. Each wait is longer than the last. Any other error is raised at once. | 2 retries, first wait 60 seconds, each wait twice the last, no wait longer than 600 seconds |
| Circuit breaker | After several failed calls in a row, the source is not called at all for a while. Then one trial call is let through. | Opens after 3 failures, stays open 1800 seconds |
| Connection cap | At most a set number of calls to one source run at the same time. | No cap unless one is given for the source |
A temporary error is a timeout, a connection error, or the platform's own TransientError, which an adapter raises for failures it knows will pass. A wrong column name is a bug, and a bug is never retried.
A worked example
The dealer interface is read every six hours. On one read it fails twice with passing errors, then answers. With a first wait of 10 seconds, as in the test:
| Try | Result | Wait before the next try |
|---|---|---|
| 1 | A temporary error | 10 seconds |
| 2 | Connection error | 20 seconds |
| 3 | The page of dealers | None |
The breaker counts this as one successful call. It counts calls that failed after all their retries, not each failed try.
The settings that drive it
| Setting | Scope | Meaning | Default | Read by the code? |
|---|---|---|---|---|
retry_count | Dataset | Tries after the first. | 2 | Yes |
retry_interval_seconds | Dataset | Wait before the first retry. | 60 | Yes |
max_concurrent_datasets | Source | The connection cap. | None | Named in the module's notes as the cap. The run scripts build the guard without caps. |
circuit_failure_threshold | Dataset, proposed | Failures in a row before the circuit opens. | 3 | No. The guard uses its own default. |
circuit_reset_seconds | Dataset, proposed | How long the circuit stays open. | 1800 | No. The guard uses its own default. |
The code
src/dataplatform/resilience.py:51The retry loop. Sleeping is a function passed in, so tests do not wait.def call_with_retry(fn: Callable[[], T], policy: RetryPolicy, sleep: Callable[[float], None],
on_retry: Callable[[int, BaseException, float], None] | None = None) -> T:
"""Call fn; on a declared temporary error wait and try again, up to policy.retries times. The last error is raised
when every try failed; any other error is raised at once."""
waits = policy.waits()
for attempt in range(len(waits) + 1):
try:
return fn()
except policy.retry_on as exc:
if attempt == len(waits):
raise
if on_retry is not None:
on_retry(attempt + 1, exc, waits[attempt])
sleep(waits[attempt])
raise AssertionError("unreachable") # pragma: no coverThe waits are worked out once from the policy. The loop tries one more time than there are waits. Only the error types in the policy are caught. On the last try the error is raised as it is. Because sleep is an argument, a test passes a function that records the wait and returns at once.
src/dataplatform/resilience.py:68The breaker's three states, worked out from two numbers.@dataclass
class CircuitBreaker:
"""closed: calls go through. open: calls are refused until reset_s has passed since it opened. half-open: one trial
call; success closes it, failure opens it again."""
threshold: int = 3
reset_s: float = 1800.0
failures: int = 0
opened_at: float | None = None
def state(self, now: float) -> str:
if self.opened_at is None:
return "closed"
return "half-open" if now - self.opened_at >= self.reset_s else "open"
def before(self, now: float, source: str) -> None:
if self.state(now) == "open":
raise CircuitOpen(f"{source}: {self.failures} failures in a row; not called again until "
f"{self.reset_s:.0f}s after the last one")
def success(self) -> None:
self.failures, self.opened_at = 0, None
def failure(self, now: float) -> None:
self.failures += 1
if self.failures >= self.threshold or self.opened_at is not None:
self.opened_at = nowThe breaker stores only a count of failures and the time it opened. Closed means it never opened or was reset by a success. Open means the reset time has not passed. Half-open means it has: one call is let through, a success closes the breaker, and a failure opens it again from now.
src/dataplatform/resilience.py:123The three together. This is what an ingestion pipeline calls. def call(self, source: str, fn: Callable[[], T], policy: RetryPolicy | None = None) -> T:
breaker = self.breaker(source)
breaker.before(self.clock(), source)
sem = self._sem(source)
def once() -> T:
if sem is None:
return fn()
with sem:
return fn()
def noted(attempt: int, exc: BaseException, wait: float) -> None:
self.retries_seen.append((source, attempt, type(exc).__name__))
try: # the breaker counts calls that failed after their retries, not each failed try
out = call_with_retry(once, policy or RetryPolicy(), self.sleep, noted)
except BaseException:
breaker.failure(self.clock())
raise
breaker.success()
return outsrc/dataplatform/ingest/pipelines.py:62Where ingestion uses it: every read of a source goes through the run's guard.def _read(ctx: Ctx, src: Sources, fn: Callable[[], T]) -> T:
"""One call to a source through the run's guard: retried on temporary errors per the dataset's retry settings,
refused while the source's circuit is open, limited by the source's connection cap."""
if src.guard is None:
return fn()
return src.guard.call(ctx.spec.source, fn, RetryPolicy.from_settings(ctx.spec.settings))How it is tested
tests/dataplatform/test_resilience.py uses a fake clock and a recorded sleep. The retry table gives the errors a call raises in order, the policy, the expected result, the number of calls made and the waits.
tests/dataplatform/test_resilience.py:35The retry cases.RETRY_CASES = [
("works first time: no wait", [], RetryPolicy(2, 10), "ok", 1, []),
("a timeout then success: one wait", [TimeoutError("slow")], RetryPolicy(2, 10), "ok", 2, [10]),
("two temporary errors: backoff doubles", [TransientError("x"), ConnectionError("y")], RetryPolicy(2, 10), "ok", 3,
[10, 20]),
("out of retries: the last error is raised", [TimeoutError("a"), TimeoutError("b"), TimeoutError("c")],
RetryPolicy(2, 10), TimeoutError, 3, [10, 20]),
("a bug is never retried", [ValueError("bad column")], RetryPolicy(5, 10), ValueError, 1, []),
("waits are capped", [TimeoutError()] * 3, RetryPolicy(3, 100, factor=10, max_wait_s=300), "ok", 4,
[100, 300, 300]),
("no retries configured", [TimeoutError("a")], RetryPolicy(0, 10), TimeoutError, 1, []),
]- The breaker's state lives in the guard for one run. A source that fails in every run is tried again in each new run. The module's notes say a breaker that must outlive runs belongs in the metadata store, and that is not built.
- The two circuit settings are proposed in the registry and are not read from a dataset's settings.
- The run scripts create the guard with no connection caps, so the cap is tested but not in effect in the runs this site describes.
- Retries are kept in a list on the guard for the run log. I did not find the code that writes them to the run log.
- Push sources, where a mover delivers files to landing, do not go through the guard. It protects pulls.