SC ReconciliationReconciliation Automation
  1. Home
  2. Exceptions
  3. Dead-Letter Queues and Retry Orchestration
  4. Capping Retries with Exponential Backoff and Jitter

Capping Retries with Exponential Backoff and Jitter

↑ Part of Dead-Letter Queues and Retry Orchestration.

A single reconciliation record failing on a missing FX rate or a timed-out 3PL status call is a non-event — retry it once the dependency recovers and move on. The failure mode this page exists to prevent is what happens when a few hundred records fail for the same reason at the same moment: an FX feed that hasn’t published today’s rate, or a carrier API mid-outage. If every one of those records computes an identical delay and retries at the identical instant, the retry storm re-creates — or worsens — the exact load spike that caused the original failure. Capping the backoff delay bounds worst-case latency; adding jitter breaks the synchronization between records that failed together. Neither one alone is sufficient, and getting the cap or the jitter formula wrong is the difference between a retry layer that quietly absorbs an outage and one that amplifies it.

Operational Trigger Signals

Build capped backoff with jitter into any retry path where these conditions hold, rather than defaulting every dependency to a fixed-delay retry:

  1. Shared-dependency failures cluster in time. A missing FX rate or a 3PL outage fails many records within the same processing window, not one record in isolation — the retry mechanism must assume correlated failure, not independent failure.
  2. Recovery time is unknown and variable. An FX rate might publish in five minutes or five hours; a carrier API might recover in seconds or after a prolonged incident. A fixed retry interval either wastes attempts on a slow recovery or under-utilizes a fast one.
  3. Attempt count without a cap threatens queue depth. Uncapped exponential growth (2n2^n) reaches multi-day delays within a handful of attempts, which is functionally indistinguishable from never retrying — the record should instead be promoted to the dead-letter queue with a clear reason code well before that point.
  4. Synchronized retry spikes are visible in logs. If retry timestamps cluster into narrow bursts instead of spreading across the delay window, jitter is either missing or not actually randomizing — a symptom covered in more depth below.

Step-by-Step Implementation

Implement the retry as a decorator so any call into a transient dependency — the FX rate lookup, the 3PL status poll — gets identical backoff, jitter, attempt tracking, and DLQ handoff without duplicating the loop at every call site:

  1. Define the backoff ceiling — compute dnd_n, the maximum possible delay for attempt nn, as a capped exponential of a per-dependency base delay.
  2. Sample the actual delay from the ceiling — apply a jitter strategy (full or equal) so the realized wait, not just the ceiling, is randomized per record.
  3. Track attempts against a hard cap — increment a counter per call; once it reaches max_attempts, stop retrying unconditionally.
  4. Raise a typed exhaustion error carrying a reason code — the caller (or the orchestrator described in Dead-Letter Queues and Retry Orchestration) uses that reason code to build the DLQ entry.
  5. Log every attempt and the final disposition — the sampled delay, the attempt number, and the outcome, so a retry storm or a miscalibrated cap is visible in logs before it shows up as a paging alert.
PYTHON
import functools
import logging
import random
import time
from dataclasses import dataclass
from enum import Enum
from typing import Callable, Optional, Tuple, Type, TypeVar

logger = logging.getLogger("recon.retry.backoff")

T = TypeVar("T")


class JitterStrategy(str, Enum):
    NONE = "none"
    FULL = "full"
    EQUAL = "equal"


class DlqReasonCode(str, Enum):
    RETRY_BUDGET_EXHAUSTED = "RETRY_BUDGET_EXHAUSTED"


class RetryBudgetExhausted(Exception):
    """Raised once max_attempts is reached; carries the reason code and the
    last underlying error so the caller can build a DLQ entry without
    re-deriving why the record failed."""

    def __init__(self, reason_code: DlqReasonCode, attempts: int, last_error: Exception) -> None:
        self.reason_code = reason_code
        self.attempts = attempts
        self.last_error = last_error
        super().__init__(f"{reason_code.value} after {attempts} attempts: {last_error}")


@dataclass(frozen=True)
class BackoffConfig:
    """Per-dependency retry policy. FX lookups and 3PL calls should each get
    their own instance, not share one global default."""
    base_delay_s: float = 1.0
    cap_s: float = 60.0
    max_attempts: int = 5
    jitter: JitterStrategy = JitterStrategy.FULL


def compute_delay(attempt: int, cfg: BackoffConfig, rng: Optional[random.Random] = None) -> float:
    """Capped exponential ceiling, then jittered per cfg.jitter.

    attempt is zero-indexed: attempt=0 is the delay before the *second* try.
    """
    rng = rng or random
    ceiling = min(cfg.cap_s, cfg.base_delay_s * (2 ** attempt))
    if cfg.jitter is JitterStrategy.NONE:
        return ceiling
    if cfg.jitter is JitterStrategy.FULL:
        return rng.uniform(0, ceiling)
    # Equal jitter: half the ceiling is guaranteed, the other half is randomized.
    return (ceiling / 2) + rng.uniform(0, ceiling / 2)


def capped_backoff_retry(
    cfg: BackoffConfig,
    retryable: Tuple[Type[Exception], ...],
    sleep_fn: Callable[[float], None] = time.sleep,
) -> Callable[[Callable[..., T]], Callable[..., T]]:
    """Decorator factory: wraps fn so calls raising a retryable exception are
    retried with capped, jittered backoff up to cfg.max_attempts."""

    def decorator(fn: Callable[..., T]) -> Callable[..., T]:
        @functools.wraps(fn)
        def wrapper(*args: object, **kwargs: object) -> T:
            last_error: Optional[Exception] = None
            for attempt in range(cfg.max_attempts):
                try:
                    return fn(*args, **kwargs)
                except retryable as exc:
                    last_error = exc
                    if attempt == cfg.max_attempts - 1:
                        break
                    delay = compute_delay(attempt, cfg)
                    logger.warning(
                        "retry_attempt fn=%s attempt=%d/%d delay_s=%.2f jitter=%s error=%s",
                        fn.__name__, attempt + 1, cfg.max_attempts, delay, cfg.jitter.value, exc,
                    )
                    sleep_fn(delay)
            logger.error(
                "retry_budget_exhausted fn=%s attempts=%d reason=%s last_error=%s",
                fn.__name__, cfg.max_attempts, DlqReasonCode.RETRY_BUDGET_EXHAUSTED.value, last_error,
            )
            raise RetryBudgetExhausted(
                DlqReasonCode.RETRY_BUDGET_EXHAUSTED, cfg.max_attempts, last_error  # type: ignore[arg-type]
            )
        return wrapper
    return decorator

Applied to the two dependencies this page opened with — a base delay measured in minutes for a daily-cadence rate, a base delay measured in seconds for a request/response API:

PYTHON
from decimal import Decimal


class RateNotYetPublishedError(Exception):
    """Today's FX rate has not landed in the rate dimension yet."""


class CarrierApiUnreachableError(Exception):
    """3PL tracking/status endpoint timed out or returned a 5xx."""


fx_cfg = BackoffConfig(base_delay_s=900.0, cap_s=21_600.0, max_attempts=4, jitter=JitterStrategy.FULL)
carrier_cfg = BackoffConfig(base_delay_s=1.0, cap_s=30.0, max_attempts=6, jitter=JitterStrategy.FULL)


@capped_backoff_retry(fx_cfg, retryable=(RateNotYetPublishedError,))
def fetch_fx_rate(pair: str, as_of: str) -> Decimal:
    """Raises RateNotYetPublishedError until the rate publishes for as_of."""
    ...  # rate-source lookup


@capped_backoff_retry(carrier_cfg, retryable=(CarrierApiUnreachableError,))
def poll_carrier_status(shipment_id: str) -> dict:
    """Raises CarrierApiUnreachableError on timeout or 5xx from the 3PL."""
    ...  # HTTP call to the 3PL status endpoint


def resolve_shipment(shipment_id: str, fx_pair: str, as_of: str) -> dict:
    """Caller-level handoff: an exhausted retry budget becomes a DLQ entry,
    not an unhandled exception."""
    try:
        rate = fetch_fx_rate(fx_pair, as_of)
        status = poll_carrier_status(shipment_id)
    except RetryBudgetExhausted as exc:
        logger.error(
            "promoting_to_dlq shipment_id=%s reason=%s attempts=%d",
            shipment_id, exc.reason_code.value, exc.attempts,
        )
        raise  # orchestrator layer builds the DLQ envelope from exc.reason_code
    return {"shipment_id": shipment_id, "fx_rate": rate, "status": status}

RetryBudgetExhausted deliberately carries a closed DlqReasonCode rather than a free-text string — the orchestrator described in Dead-Letter Queues and Retry Orchestration tags every DLQ entry with a reason code, and a decorator that raised a bare exception would force that code to be re-derived from the error message, which is brittle. If a dependency needs more than one exhaustion reason (say, distinguishing “rate never published” from “rate source unreachable”), extend the enum rather than encoding meaning into last_error text.

Attempt timeline: growing jittered delays capped, then DLQ promotion Six retry attempts are shown left to right. Each attempt has a shaded band from zero up to its delay ceiling d_n, which doubles per attempt until it hits the cap at attempt five and stays flat at attempt six. A dot inside each band marks the actual jittered delay sampled for that attempt, at a different position each time, showing that identical ceilings still produce de-correlated actual waits. A dashed horizontal line marks the cap. After attempt six the record has exhausted max_attempts and is promoted to a dead-letter queue box tagged with reason code RETRY_BUDGET_EXHAUSTED. cap (d_max) d1=1s Attempt 1 d2=2s Attempt 2 d3=4s Attempt 3 d4=8s Attempt 4 d5=20s (capped) Attempt 5 d6=20s (capped) Attempt 6 max_attempts reached RETRY_BUDGET_EXHAUSTED

The shaded band on each attempt is the full range the jittered delay could fall in; the dot is the one value actually sampled that run. Two records that failed at the identical timestamp with the identical ceiling still land at different points inside that band — which is the entire mechanism that prevents a retry storm.

Configuration Reference

dn=min⁡ ⁣(dmax⁡,  d0⋅2 n)d_n = \min\!\left(d_{\max},\; d_0 \cdot 2^{\,n}\right) tnfull∼Uniform(0, dn)tnequal=dn2+Uniform ⁣(0, dn2)t_n^{\text{full}} \sim \mathrm{Uniform}(0,\, d_n) \qquad\qquad t_n^{\text{equal}} = \frac{d_n}{2} + \mathrm{Uniform}\!\left(0,\, \frac{d_n}{2}\right)

d0d_0 is the per-dependency base delay, dmax⁡d_{\max} the cap, nn the zero-indexed attempt, and dnd_n the ceiling for that attempt. Full jitter draws the whole realized wait tnt_n uniformly from zero to that ceiling; equal jitter guarantees at least half the ceiling as a floor and randomizes only the remainder. Full jitter de-correlates a retry storm harder — its lower bound is zero, so some records retry almost immediately — while equal jitter trades some de-correlation for a guaranteed minimum spacing, which matters against a dependency that penalizes very-short-interval retries specifically.

Strategy Delay formula De-correlation strength Best fit
No jitter tn=dnt_n = d_n None — every record with the same ceiling retries in lockstep Never for shared-dependency failures; acceptable only for a single isolated retry
Full jitter tn∼Uniform(0,dn)t_n \sim \mathrm{Uniform}(0, d_n) Strongest — spread covers the entire range down to zero Correlated bursts: FX feed outage, 3PL incident, feed-wide timeout
Equal jitter tn=dn/2+Uniform(0,dn/2)t_n = d_n/2 + \mathrm{Uniform}(0, d_n/2) Moderate — floor at half the ceiling Dependencies that penalize very short retry intervals (rate-limited APIs)
Decorrelated jitter tn=min⁡(dmax⁡,Uniform(d0,tn−1⋅3))t_n = \min(d_{\max}, \mathrm{Uniform}(d_0, t_{n-1} \cdot 3)) Strong, grows from the prior sampled value rather than the ceiling High-volume systems wanting wide spread without full jitter’s near-zero waits

Per-dependency parameters, sized to each failure’s natural recovery cadence rather than one shared default:

Parameter FX rate lookup 3PL status call
base_delay_s 900 (15 min) 1
cap_s 21,600 (6 h) 30
max_attempts 4 6
jitter full full
DLQ reason code on exhaustion RETRY_BUDGET_EXHAUSTED RETRY_BUDGET_EXHAUSTED

A daily-published rate that hasn’t landed after four capped attempts spanning up to six hours is not going to land in a fifth; a carrier timeout that hasn’t cleared after six attempts inside roughly a minute and a half is signaling an outage, not a blip. Both cases are better served by promoting to the dead-letter queue than by raising max_attempts further — the ingestion layer’s backpressure controls exist for exactly this reason: slowing the rate of new work into a struggling dependency is a separate lever from how long any one record’s retry budget runs.

Debugging & Recovery

  • Retries arriving in synchronized bursts. If logged retry_attempt timestamps cluster into narrow spikes instead of spreading across each delay window, jitter isn’t actually randomizing — check for a random.Random instance seeded once and reused across the whole process, or a sleep_fn override in tests that leaked into production and short-circuits the delay entirely.
  • Records exhausting budget faster than the dependency recovers. Compare cap_s and max_attempts against the failure’s real recovery cadence. A carrier incident that takes ten minutes to clear will out-live a 3PL config with a 30-second cap and six attempts; either raise the cap for that specific dependency or accept that the record correctly belongs in the dead-letter queue until an operator forces a manual replay.
  • DLQ filling with RETRY_BUDGET_EXHAUSTED from one dependency only. A concentrated spike in one reason code from one call site is a leading indicator of an upstream outage, not a retry-policy defect — this is the signal to page the FX feed or 3PL owner, and it is exactly the kind of queue-depth-by-source breakdown covered in Monitoring and Alerting for Reconciliation Pipelines.
  • Confirm idempotency before widening the cap. Because capped_backoff_retry re-invokes the wrapped function on every attempt, any side effect inside it (a partial write, an API call with side effects on the far end) must be safe to repeat. Widening max_attempts on a non-idempotent call multiplies the blast radius of that risk instead of just the wait time.
  • Escalate the whole section, not just this decorator, when triage volume grows. A sustained rise in exhausted-budget promotions across dependencies is a workload problem for Exception Handling & Discrepancy Resolution as a whole, not something a backoff tweak fixes on its own.

FAQ

Why cap the delay instead of letting exponential backoff grow indefinitely?

Because uncapped 2n2^n growth reaches multi-day waits within a handful of attempts, which behaves the same as never retrying but without the clarity of an explicit decision. Capping the ceiling at a value close to the dependency’s realistic worst-case recovery time keeps every attempt meaningful; once the cap is reached repeatedly, the record should exhaust its max_attempts and promote to the dead-letter queue with a reason code, rather than keep computing longer and longer delays that no one is waiting on.

Should I use full jitter or equal jitter?

Default to full jitter — it de-correlates a retry storm the hardest because the sampled delay can land anywhere from zero up to the ceiling. Switch to equal jitter only when the dependency itself penalizes very short retry intervals, such as an API that counts near-immediate retries against a rate limit separately from spaced-out ones; the guaranteed half-ceiling floor in equal jitter avoids that penalty at the cost of slightly weaker de-correlation.

Does every retryable dependency need the same max_attempts and cap?

No, and sharing one global config is a common misconfiguration. A daily-cadence dependency like an FX rate needs a base delay and cap measured in hours, while a request/response API needs one measured in seconds; using an API-sized cap against a once-daily rate exhausts the budget hours before the rate could plausibly publish, and using a rate-sized cap against an API timeout delays DLQ promotion — and the operator page it should trigger — far longer than the outage warrants.