SC ReconciliationReconciliation Automation
  1. Home
  2. Exceptions
  3. Dead-Letter Queues and Retry Orchestration

Dead-Letter Queues and Retry Orchestration

↑ Part of Exception Handling & Discrepancy Resolution.

A reconciliation pipeline fails transactions for two structurally different reasons, and conflating them is what causes either lost records or infinite loops. A missing FX rate, an unreachable third-party-logistics API, or a counterpart record that has not yet arrived are all transient — the same input will very likely succeed on a later attempt, once a rate publishes, a carrier endpoint recovers, or a late invoice lands. A malformed payload that fails schema validation, or a currency pair that will never resolve, is permanent — no amount of retrying changes the outcome. A pipeline that retries everything forever eventually wedges on the permanent failures, burning attempt budget and delaying every record queued behind them. A pipeline that gives up after one failure silently drops transactions that would have cleared on retry two.

This page treats retry orchestration as the layer that sits directly downstream of failed delivery or processing attempts and decides, deterministically, whether a record gets another chance, how long it waits, and when it stops waiting. It complements Exception Queue Design and Triage, which handles the human-facing queue of confirmed exceptions — this page handles the machine-facing loop that runs before a record is confirmed to be a real exception at all. Everything below assumes at-least-once delivery semantics: a message may be redelivered after a failure that occurred downstream of a side effect, so every retryable operation in the pipeline must be idempotent or the retry loop itself becomes a source of duplicate postings.

Core Concept & Decision Criteria

Two constructs do different jobs and should never be merged into one queue. The retry queue is a short-lived holding area for records that failed for a reason expected to resolve on its own; it requeues each record with a computed delay and a bounded attempt counter. The dead-letter queue (DLQ) is a terminal store for records that either exhausted their retry budget or were classified as non-retryable on first failure; nothing in the DLQ is automatically reprocessed — it waits for a human or an upstream fix, then an explicit replay. Treating the DLQ as “just another retry with a longer delay” is the most common design error in this layer: it lets poison messages consume the same monitoring and paging budget as genuinely time-sensitive transient failures, and it hides the moment a record needs root-cause attention behind an infinite polite retry.

The second decision this layer forces is retry spacing. Retrying immediately assumes the failure was a single transient blip; retrying on a fixed schedule assumes recovery time is roughly constant; retrying with exponential backoff and jitter assumes recovery time is unknown and that many records are failing for the same root cause at once — which is exactly the profile of an unreachable 3PL API or a not-yet-published daily FX rate. The spacing strategy should be chosen per failure class, not globally: a rate that publishes once daily needs a cap measured in hours, while a carrier API timeout needs a cap measured in minutes.

Strategy Spacing behavior Best fit Primary risk
Immediate retry Re-attempt with no delay, 1–3 times Sub-second transient blips (connection reset) Thundering herd against a struggling dependency; wastes attempts on outages that need seconds, not milliseconds
Fixed delay Constant wait between attempts (e.g. every 60s) Predictable, short recovery windows Synchronized retries from many failed records collide on the same interval, re-creating the load spike that caused the failure
Exponential + jitter Delay grows geometrically, capped, randomized Unknown recovery time; correlated failures (feed/API outage) None structural — main cost is longer worst-case latency to eventual success, which is acceptable for async reconciliation

The retry-queue-versus-DLQ distinction and the spacing decision compound: a record only belongs in the retry queue if its failure was classified transient and it has not exhausted max_attempts under the chosen spacing strategy.

Property Retry queue Dead-letter queue
Lifetime of an entry Minutes to hours, bounded by attempt cap Indefinite, until explicit replay or closure
Population Transient-classified failures under the attempt cap Poison-classified failures, or attempts exhausted
Processing Automatic — orchestrator requeues on schedule Manual/triggered — human review or root-cause replay
Payload shape Original record + attempt counter + next-run time Original record + full attempt/error history + failure class
What “success” looks like Record clears and leaves the queue on a later attempt Record is fixed upstream and explicitly replayed, or written off

Attempt-level backoff configuration for the retry queue is detailed separately in Capping Retries with Exponential Backoff and Jitter; this page treats that mechanism as one stage inside the larger orchestration loop.

Implementation

The orchestrator below classifies a failure as transient or poison, computes a capped exponential delay with full jitter, and produces either an updated retry envelope or a typed dead-letter envelope. Both envelope types are pydantic models so downstream consumers — the scheduler, the DLQ store, the alerting job — get validated, self-describing records instead of loose dictionaries.

PYTHON
import logging
import random
from datetime import datetime, timedelta, timezone
from enum import Enum
from typing import Optional, Union

from pydantic import BaseModel, Field

logger = logging.getLogger("recon.retry")


class FailureClass(str, Enum):
    TRANSIENT = "transient"
    POISON = "poison"


class RetryableError(Exception):
    """Base class for failures expected to resolve on a later attempt."""


class RateNotYetPublishedError(RetryableError):
    """Today's FX rate has not landed in the rate dimension yet."""


class CarrierApiUnreachableError(RetryableError):
    """3PL tracking/status API timed out or returned a 5xx."""


class CounterpartNotArrivedError(RetryableError):
    """The matching document (invoice/ASN) has not been ingested yet."""


class PoisonError(Exception):
    """Base class for failures that will never resolve by retrying."""


class SchemaValidationError(PoisonError):
    """Payload fails structural validation — retrying changes nothing."""


# Map concrete exception types to a failure class without needing every
# call site to know the taxonomy; new error types register here.
_CLASSIFICATION: dict[type[Exception], FailureClass] = {
    RateNotYetPublishedError: FailureClass.TRANSIENT,
    CarrierApiUnreachableError: FailureClass.TRANSIENT,
    CounterpartNotArrivedError: FailureClass.TRANSIENT,
    SchemaValidationError: FailureClass.POISON,
}


def classify_exception(exc: Exception) -> FailureClass:
    """Look up the failure class; unknown exception types are treated as
    poison, because retrying an unclassified error is an unproven bet."""
    for exc_type, cls in _CLASSIFICATION.items():
        if isinstance(exc, exc_type):
            return cls
    logger.warning("unclassified_exception type=%s -> treating as poison", type(exc).__name__)
    return FailureClass.POISON


def compute_backoff_delay(
    attempt: int, base_delay_s: float = 2.0, cap_s: float = 900.0, rng: Optional[random.Random] = None
) -> float:
    """Full-jitter capped exponential backoff (AWS architecture-blog form).

    d_n = min(cap, base * 2^n); the actual wait is drawn uniformly from
    [0, d_n], which decorrelates retries that failed together far better
    than a deterministic or half-jitter delay does.
    """
    rng = rng or random
    ceiling = min(cap_s, base_delay_s * (2 ** attempt))
    return rng.uniform(0, ceiling)


class RetryEnvelope(BaseModel):
    """A record currently living in the retry queue."""
    message_id: str
    payload: dict
    attempt: int = 0
    max_attempts: int = 6
    first_seen_at: datetime
    last_error: Optional[str] = None
    next_retry_at: Optional[datetime] = None


class DLQEnvelope(BaseModel):
    """A terminal record — no further automatic processing."""
    message_id: str
    payload: dict
    attempt: int
    failure_class: FailureClass
    last_error: str
    first_seen_at: datetime
    promoted_at: datetime = Field(default_factory=lambda: datetime.now(timezone.utc))
    source_stage: str


def handle_failure(
    envelope: RetryEnvelope, exc: Exception, source_stage: str
) -> Union[RetryEnvelope, DLQEnvelope]:
    """Decide the fate of a failed record: requeue with backoff, or
    promote to the dead-letter queue."""
    failure_class = classify_exception(exc)
    attempt = envelope.attempt + 1

    if failure_class is FailureClass.POISON:
        logger.error(
            "poison_message id=%s stage=%s error=%s -> DLQ (no retry attempted)",
            envelope.message_id, source_stage, exc,
        )
        return DLQEnvelope(
            message_id=envelope.message_id, payload=envelope.payload, attempt=attempt,
            failure_class=failure_class, last_error=str(exc),
            first_seen_at=envelope.first_seen_at, source_stage=source_stage,
        )

    if attempt >= envelope.max_attempts:
        logger.warning(
            "attempts_exhausted id=%s stage=%s attempt=%d/%d -> DLQ",
            envelope.message_id, source_stage, attempt, envelope.max_attempts,
        )
        return DLQEnvelope(
            message_id=envelope.message_id, payload=envelope.payload, attempt=attempt,
            failure_class=failure_class, last_error=str(exc),
            first_seen_at=envelope.first_seen_at, source_stage=source_stage,
        )

    delay_s = compute_backoff_delay(attempt)
    next_retry_at = datetime.now(timezone.utc) + timedelta(seconds=delay_s)
    logger.info(
        "requeued id=%s stage=%s attempt=%d/%d delay_s=%.1f next_retry_at=%s",
        envelope.message_id, source_stage, attempt, envelope.max_attempts, delay_s, next_retry_at,
    )
    return envelope.model_copy(
        update={"attempt": attempt, "last_error": str(exc), "next_retry_at": next_retry_at}
    )

handle_failure is the single decision point every failing stage calls into — the FX conversion stage, the carrier-status poller, the counterpart-matching stage — so attempt tracking, classification, and DLQ promotion behave identically no matter which stage produced the failure. RetryEnvelope.model_copy returns a new immutable snapshot rather than mutating in place, which keeps the retry state safe to persist and re-read from a queue table without racing a concurrent worker.

Retry-to-DLQ decision flow A message is dequeued from the retry queue and an attempt at delivery or processing is made. If it succeeds, the orchestrator acknowledges it and removes it from retry state. If it fails, the failure is classified as poison or transient. A poison classification promotes the record straight to the dead-letter queue with no retry attempted. A transient classification checks whether attempt is below max_attempts: if the cap is reached, the record is promoted to the dead-letter queue as exhausted; otherwise the orchestrator computes a capped exponential backoff delay with full jitter, sets next_retry_at, and requeues the record, which loops back around to another attempt once next_retry_at elapses. Both dead-letter paths — poison and exhausted — feed into a single dead-letter queue store that carries a typed envelope, is depth-alerted, and supports explicit replay. Message dequeued from retry queue Attempt delivery or processing Delivery succeeded? Acknowledge & complete removed from retry state Classify failure: poison or transient? DLQ — poison non-retryable, no attempt attempt < max_attempts? DLQ — exhausted attempt cap reached Compute backoff + jitter d_n = min(cap, base·2^n) requeue at next_retry_at Dead-letter queue typed envelope depth-alerted · replayable yes no poison transient no yes requeued → retried when next_retry_at elapses

Configuration & Threshold Calibration

The four knobs below define the behavior of the retry loop. They should be set per failure class, not once globally, because a rate-publication delay and a carrier API blip recover on entirely different timescales.

dn=min⁡ ⁣(dmax⁡,  d0⋅2 n),tn∼Uniform(0, dn)d_n = \min\!\left(d_{\max},\; d_0 \cdot 2^{\,n}\right), \qquad t_n \sim \mathrm{Uniform}(0,\, d_n)

d0d_0 is the base delay, dmax⁡d_{\max} the cap, nn the zero-indexed attempt number, and tnt_n the actual wait — drawn uniformly rather than used directly, which is the “full jitter” form. Without the uniform draw, every record that failed for the same root cause (a single carrier outage, one missing daily rate) retries in lockstep and re-creates the exact load spike that caused the outage.

Parameter Recommended value Rationale
max_attempts 5–6 for API/network failures; 3–4 for daily-cadence lookups (FX rate) Bounds worst-case time-to-DLQ; more attempts than the failure’s natural recovery cadence just delays diagnosis
base_delay_s (d0) 1–2s for API timeouts; 900s+ for daily-published data Should approximate the fastest plausible recovery time for that failure class
cap_s (d_max) 60–300s for API/network; 6–24h for daily-cadence lookups Prevents unbounded wait on a record that will eventually be re-classified anyway
Jitter form Full jitter (Uniform(0, d_n)) Decorrelates simultaneous retries better than equal or decorrelated jitter for bursty, correlated failure sets
poison_default Unclassified exception → poison, not transient An unproven failure type should surface for review, not consume retry budget silently

A counterpart-not-arrived failure deserves its own cadence entirely: rather than a fixed attempt cap, tie its max_attempts to a business-calendar cutoff (e.g. stop retrying, and promote to DLQ, once the matching window in Matching & Reconciliation Algorithms closes for that document pair) so records don’t sit retrying past the point where a late arrival is even still eligible to match.

Orchestration & Integration

Retry orchestration sits between the stage that raised the failure and the stage that would have consumed a successful result — it never blocks the pipeline’s forward progress. A failed record is detached from its batch immediately, given a next_retry_at, and the pipeline continues with the remaining rows; this is why the ingestion layer described in Async Batch Processing for High-Volume Feeds treats retry as an out-of-band side channel rather than a blocking step inside the main batch loop. A slow-moving retry queue for one supplier’s feed should never stall ingestion for every other supplier’s feed.

Scheduling the actual requeue can use either a broker’s native delayed-delivery feature (a visibility timeout or a delay-queue primitive) or a simple polling worker that selects rows where next_retry_at <= now() from a retry-state table and re-dispatches them. The polling approach is easier to make idempotent and auditable in a reconciliation context, since the same table doubles as the attempt-history record referenced during triage. Either way, the dispatch step must re-check max_attempts at pop time, not just at enqueue time — a configuration change to lower the cap should apply to records already waiting in the queue, not just newly failing ones.

Because delivery is at-least-once, a record can be redelivered after the side effect it triggered already committed downstream (a partial write that succeeded just before a network failure severed the response). Every retryable operation must therefore key its writes on the same idempotency approach used elsewhere in the pipeline — a stable hash or natural key checked before any insert — so a retried attempt that duplicates prior work is a no-op rather than a double-posting.

Debugging & Pipeline Recovery

DLQ depth is the single most useful health signal this layer produces, and it belongs on the same dashboards described in Monitoring and Alerting for Reconciliation Pipelines: alert when depth exceeds a rolling baseline, and separately alert on the rate of new DLQ arrivals, since a sudden spike in exhausted-attempt promotions is a leading indicator of an upstream outage even before any single record’s cap is reached. Track depth by failure_class and source_stage so a spike in CarrierApiUnreachableError promotions is immediately distinguishable from a spike in SchemaValidationError promotions — the former needs an infrastructure fix, the latter needs a mapping fix, and paging the wrong owner delays recovery.

  • Retry queue growing but not draining: check whether next_retry_at values are clustering — if jitter isn’t actually randomizing (a shared random.Random instance seeded once, or a broker feature that ignores the jitter parameter), retries synchronize and repeatedly collide with the same outage window.
  • DLQ filling with records that should have retried: confirm the exception taxonomy in _CLASSIFICATION covers the failure type; an unclassified exception silently defaults to poison and skips retry entirely, which is safe but should be rare — recurring “unclassified_exception” log lines mean the taxonomy is out of date.
  • Records stuck at max_attempts with a transient-looking error: the cap may be miscalibrated for that failure class — a daily FX rate lookup with a 5-minute cap will always exhaust before the rate publishes. Recalibrate per class rather than raising the global cap.
  • Replay: once the root cause is fixed (rate hydrated, API restored, counterpart ingested), replay pulls the original payload out of the DLQ envelope and resubmits it as a fresh retry envelope with attempt=0, preserving the DLQ record itself as history rather than deleting it. Route root-cause tagging on replay through Discrepancy Root-Cause Classification so recurring failure patterns are visible across replays, not just within one incident.

FAQ

Why not just retry forever until the record succeeds?

Because “forever” has a cost even when it’s cheap per attempt: a permanently-failing record occupies queue depth, appears in every DLQ-adjacent metric as noise, and — for genuinely poison payloads — will never succeed no matter how many attempts run. A bounded max_attempts converts an indefinite unknown into a definite signal: either the record clears, or it becomes a DLQ entry someone can act on. Unbounded retry hides the second outcome behind an infinite first one.

Why does jitter matter if exponential backoff already spreads out the delay?

Exponential backoff spreads delay for one record, but many records typically fail from the same root cause at the same moment — one FX feed outage or one carrier API incident fails hundreds of records simultaneously. Without jitter, all of them compute the identical deterministic delay and retry in the same instant, recreating the load spike (or hitting the same still-recovering dependency) that caused the original failure. Full jitter draws each record’s actual wait from a uniform distribution up to that ceiling, which decorrelates the retry storm.

How do you tell a poison message from a slow transient failure?

By the exception type, not by how many times it has failed. A schema validation failure will produce the identical error on every attempt regardless of timing — that’s the signature of poison. A missing rate, an unreachable API, or an unarrived counterpart are failures whose error condition is expected to change independent of anything the retry does. Classifying by exception type at the moment of failure, rather than waiting to observe repeated identical failures, means poison messages never consume retry budget in the first place.

What happens to a record when it reaches the dead-letter queue — is it lost?

No — the DLQ is a durable store, not a discard bin. It holds the full original payload plus the complete attempt and error history, and it stays there until an explicit replay or an explicit closure decision. The record is out of the automatic retry loop, which is the point: it needs a human or an upstream fix, not another silent attempt.

How many attempts is “enough”?

It should match the failure class’s natural recovery cadence, not an arbitrary round number. An API-timeout class recovers in seconds to minutes, so five or six capped-exponential attempts spanning a few minutes is usually sufficient. A once-daily FX rate publication needs a cap measured in hours, not minutes, or every record will exhaust before the rate ever lands — in which case a lower max_attempts with a much larger base_delay_s and cap_s is the correct fix, not a higher attempt count.