SC ReconciliationReconciliation Automation
  1. Home
  2. Exceptions
  3. Exception Queue Design and Triage

Exception Queue Design and Triage

↑ Part of Exception Handling & Discrepancy Resolution.

Every reconciliation pipeline that tolerates any variance at all eventually produces records it cannot resolve automatically — a price that falls outside the tolerance band, a match with low confidence, a rate lookup that timed out. Those records need a durable, queryable home, and the schema of that home is not an afterthought bolted on after the matcher runs; it is the contract between the Matching & Reconciliation Algorithms engine that routes exceptions here and every downstream process that resolves them. A queue that stores only “unmatched” and a timestamp forces every record through the same undifferentiated backlog, so a six-figure invoice variance waits behind a one-cent rounding artifact simply because the rounding artifact arrived first.

This page treats the exception queue as a first-class data structure with an explicit record shape, a deterministic priority-scoring function, and a triage policy that partitions incoming records into three lanes — auto-adjust, manual review, and retry — before a human ever looks at the backlog. Getting this boundary wrong in either direction is expensive: routing too much to auto-adjust silently posts erroneous journal entries, and routing too much to manual review buries reviewers in noise until they start rubber-stamping the queue instead of reading it.

Core Concept & Decision Criteria

The exception queue’s central job is to answer one question for every unresolved record the moment it lands: does this get fixed by a rule, fixed by a person, or fixed by trying again? Each lane has a different cost profile and a different audit posture, so the routing decision has to be made from a small set of signals already available at ingestion time — the variance value, the matcher’s confidence score, the record’s age against its service-level target, and whether the underlying failure looks transient.

Auto-adjust is reserved for variances that are both small and well-understood: a rounding residue, a shipping-fee delta, an FX-drift artifact that a documented rule can post without a human decision. Manual review is for variances that are material, ambiguous, or carry a low-confidence match — anything where an automated posting would create audit risk. Retry is categorically different from the other two: it is not a judgment about the variance itself but a recognition that the failure was transient — a timeout, a rate-limited upstream call, a lock contention error — and the record was never actually evaluated for variance in the first place.

Decision axis Auto-adjust Manual review Retry
Variance value ≤ configured ceiling (e.g. $250 or 0.1% of line value) Above the auto-adjust ceiling, or ceiling not evaluable Not evaluated — failure precedes scoring
Match confidence ≥ confidence floor (e.g. 0.95) Below the floor, or root cause ambiguous N/A
Age / SLA Fires immediately on ingestion, no waiting Ranked by priority score, oldest-material first Capped attempts with exponential backoff + jitter
Typical root cause Rounding residue, fee delta, FX rounding Price/quantity tolerance breach, vendor drift, low confidence Upstream timeout, rate limit, transient 5xx
Write-back Immediate posting to the ERP Analyst-approved posting only after resolution None — record re-enters the matcher unchanged
Audit weight Low, cited to a deterministic rule High, a logged human decision Low, a logged system retry

The deep-dive on how the materiality half of this table is calibrated per supplier tier and commodity class lives in Prioritizing Exceptions by Financial Materiality; this page treats materiality as one input among several to the router. The confidence and root-cause signals themselves are produced upstream by the classification work in Discrepancy Root-Cause Classification — the triage router below consumes a root_cause_hint field rather than re-deriving it.

Within the manual-review lane, records are not reviewed first-in-first-out. They are ranked by a single deterministic priority score that blends materiality, uncertainty, and age so reviewers always work the highest-risk record next:

P=wm⋅∣Δ∣Δcap  +  wu⋅(1−c)  +  wa⋅min⁡ ⁣(agehrsSLAhrs, 1),wm+wu+wa=1P = w_m \cdot \frac{|\Delta|}{\Delta_{\text{cap}}} \;+\; w_u \cdot (1 - c) \;+\; w_a \cdot \min\!\left(\frac{\text{age}_{\text{hrs}}}{\text{SLA}_{\text{hrs}}},\, 1\right), \qquad w_m + w_u + w_a = 1

Here Δ\Delta is the record’s signed variance amount, Δcap\Delta_{\text{cap}} is a normalization cap above which any variance is treated as maximally material, cc is the matcher’s confidence score, and the age term is clamped to 1 once a record blows past its SLA — so an ancient trivial exception cannot outrank a fresh, large one, but it does climb steadily toward parity as it stales out.

Implementation

The reference implementation defines the queue record as a frozen dataclass — every field a downstream consumer (write-back, audit, monitoring) can depend on — plus a pure priority-scoring function and a router that turns signals into a routing decision. Both functions are deterministic: identical inputs at identical now always yield identical output, which is what lets a triage replay reproduce a prior day’s exact routing for an audit.

PYTHON
import logging
from dataclasses import dataclass
from datetime import datetime, timezone
from decimal import Decimal
from typing import Literal, Optional

logger = logging.getLogger("recon.exceptions.triage")

RoutingDecision = Literal["auto_adjust", "manual_review", "retry"]

# Weights must sum to 1.0; each term below is pre-normalized to [0, 1].
PRIORITY_WEIGHTS: dict[str, float] = {"materiality": 0.5, "uncertainty": 0.3, "age": 0.2}


@dataclass(frozen=True)
class ExceptionRecord:
    """One row of the exception queue — the durable shape every unresolved record takes."""
    exception_id: str
    source_txn_id: str
    variance_amount: Decimal          # signed, settlement-currency, Decimal-safe
    match_confidence: float           # 0.0-1.0 emitted by the matching engine
    root_cause_hint: Optional[str]    # e.g. "PRICE_TOLERANCE_EXCEEDED"; None if unclassified
    retry_count: int
    first_seen: datetime
    last_attempted: Optional[datetime] = None
    status: str = "OPEN"


def _clamp(value: float, lo: float = 0.0, hi: float = 1.0) -> float:
    """Bound a normalized term so no single signal can dominate the score past its weight."""
    return max(lo, min(hi, value))


def priority_score(
    record: ExceptionRecord,
    variance_cap: Decimal,
    sla_hours: float,
    now: Optional[datetime] = None,
) -> float:
    """Deterministic 0-1 triage priority. Higher sorts first in the manual-review lane."""
    now = now or datetime.now(timezone.utc)
    materiality = _clamp(float(abs(record.variance_amount)) / float(variance_cap))
    uncertainty = _clamp(1.0 - record.match_confidence)
    age_hours = (now - record.first_seen).total_seconds() / 3600.0
    age_term = _clamp(age_hours / sla_hours)

    score = (
        PRIORITY_WEIGHTS["materiality"] * materiality
        + PRIORITY_WEIGHTS["uncertainty"] * uncertainty
        + PRIORITY_WEIGHTS["age"] * age_term
    )
    logger.debug(
        "priority exc=%s materiality=%.3f uncertainty=%.3f age_term=%.3f score=%.3f",
        record.exception_id, materiality, uncertainty, age_term, score,
    )
    return round(score, 6)


def route_exception(
    record: ExceptionRecord,
    *,
    auto_adjust_ceiling: Decimal,
    confidence_floor: float,
    max_retries: int,
    transient_reasons: frozenset[str],
) -> RoutingDecision:
    """Deterministic router: identical inputs always produce the identical routing decision."""
    if record.root_cause_hint in transient_reasons and record.retry_count < max_retries:
        logger.info(
            "route exc=%s -> retry attempt=%d/%d",
            record.exception_id, record.retry_count + 1, max_retries,
        )
        return "retry"

    if abs(record.variance_amount) <= auto_adjust_ceiling and record.match_confidence >= confidence_floor:
        logger.info(
            "route exc=%s -> auto_adjust variance=%s confidence=%.3f",
            record.exception_id, record.variance_amount, record.match_confidence,
        )
        return "auto_adjust"

    logger.info(
        "route exc=%s -> manual_review variance=%s confidence=%.3f reason=%s",
        record.exception_id, record.variance_amount, record.match_confidence, record.root_cause_hint,
    )
    return "manual_review"

The router deliberately checks retry eligibility before the auto-adjust gate. A record whose root_cause_hint marks it as a transient infrastructure failure — a timeout or a rate limit — was never actually evaluated against a variance threshold, so scoring it for auto-adjust would be scoring noise. Only once a record has exhausted its retry budget, or its failure reason is not on the transient list, does the router treat the variance and confidence values as meaningful signals.

Triage Routing Flow

Exception queue triage routing decision flow An exception record enters the queue and is first checked for retry eligibility: if its root cause is on the transient-failure list and its retry count is below the cap, it routes to retry with exponential backoff and loops back into the matcher, incrementing retry_count. Otherwise it proceeds to the auto-adjust gate: if the absolute variance is at or below the auto-adjust ceiling and match confidence is at or above the confidence floor, it routes to auto-adjust and posts immediately to the ERP write-back stage. Otherwise it routes to manual review, where it is ranked by the priority score P and queued for an analyst, and after resolution flows to the same write-back and audit stage. Exception record enters Transient reason & retry_count < max? yes no Retry backoff + jitter retry_count += 1 loop back to matcher |variance| ≤ ceiling & confidence ≥ floor? yes no Auto-adjust deterministic rule posts Manual review ranked by priority score P analyst resolves write-back + audit log

Configuration & Threshold Calibration

The router’s behavior is entirely governed by a small parameter set. Set these per business unit or supplier tier — a global flat threshold either lets high-volume noisy suppliers flood manual review or lets low-volume strategic suppliers auto-post variances that deserve a human read.

Parameter Recommended value Rationale
auto_adjust_ceiling $250, or 0.1% of line value, whichever is lower Caps the blast radius of any single unattended posting
confidence_floor 0.95 Below this the matcher itself is uncertain; a rule should not post on its behalf
max_retries 3, exponential backoff with jitter Beyond this a “transient” failure is systemic, not flaky — stop masking it
manual_review_sla_hours 24 (highest-priority band), 72 (mid), 168 (low) Tiered by priority band, not a single flat clock for every record
priority_weights materiality 0.5, uncertainty 0.3, age 0.2 Materiality dominates ranking; the age term exists purely to prevent starvation
variance_cap (Δcap\Delta_{\text{cap}}) 95th percentile of trailing 90-day variance Keeps the materiality term meaningful without one outlier flattening the rest of the score
aging_escalation_threshold 2× SLA hours Forces reassignment or escalation before a record goes fully stale

Calibrate auto_adjust_ceiling and confidence_floor together, never independently: a low ceiling paired with a low confidence floor still lets a group of small, low-confidence variances auto-post and accumulate into a material misstatement over a period-close. Review the pair quarterly against actual auto-adjust volume and reversal rate, not on a fixed schedule divorced from observed drift.

Orchestration & Integration

The exception queue sits downstream of matching and upstream of every resolution path. Its single upstream producer is the Matching & Reconciliation Algorithms engine, which emits an ExceptionRecord the moment a candidate match fails exact equality, fails a tolerance check, or fails to resolve at all — carrying the variance amount, the matcher’s confidence score, and a root_cause_hint populated by the Discrepancy Root-Cause Classification taxonomy. The queue itself never re-derives root cause; it consumes whatever the upstream classifier attached and treats an unclassified hint as its own signal for manual review.

Downstream, the three lanes fan out to different consumers. Auto-adjust and analyst-approved manual-review resolutions both flow into the same write-back path documented in Variance Write-Back and ERP Adjustment Posting — the queue record’s status transitions to RESOLVED only after that write-back confirms the posting, never optimistically at routing time. Retry-routed records do not touch write-back at all; they loop back through the backoff and dead-lettering mechanics detailed in Dead-Letter Queues and Retry Orchestration, and only arrive back at this queue’s router once a retry attempt itself produces a real variance outcome or exhausts its budget. Every routing decision, priority score, and status transition is emitted as a structured log line so the queue’s health — depth by lane, aging distribution, auto-adjust volume — is directly observable through the Monitoring and Alerting for Reconciliation Pipelines stack rather than requiring an ad-hoc query against production tables.

Treat the queue table itself as append-only for status transitions: never overwrite a prior routing decision in place, insert a new state row instead. This is what lets an auditor reconstruct exactly why a given record was auto-adjusted on a given day, even after the thresholds that governed it have since been recalibrated.

Debugging & Pipeline Recovery

Most exception-queue incidents fall into a small number of recognizable failure shapes:

  • Unbounded queue growth: depth climbs steadily with no corresponding rise in resolution throughput. Check whether auto_adjust_ceiling or confidence_floor were tightened without a corresponding staffing adjustment for manual review, or whether an upstream root-cause classifier regression is emitting None for root_cause_hint on records that should be routing to retry.
  • Priority inversion: reviewers report working low-value records ahead of high-value ones. Verify variance_cap has not gone stale relative to actual variance distribution — a cap set months ago against a smaller trailing window flattens the materiality term for every new, larger variance, letting the uncertainty and age terms dominate the score inappropriately.
  • Retry storms: retry_count climbing rapidly across many records in a short window without resolution usually means the “transient” classification is wrong — a systemic upstream outage is being retried as if it were a flaky, isolated failure. Cap total retry volume per time window independent of the per-record max_retries, and escalate a storm to a page rather than letting it exhaust budgets silently.
  • Stale confidence scores after a matcher redeploy: if the matching engine’s confidence calibration changes, confidence_floor may no longer mean what it meant when it was tuned. Re-validate the floor against a labeled sample after any matcher model or scoring-logic change, not just after a queue-side configuration change.
  • Orphaned records stuck OPEN: a record whose last_attempted timestamp is old but whose status never transitioned usually indicates a router crash between the routing decision and the write-back confirmation. Because the queue table is append-only, a reconciliation sweep can safely re-run the router for any record whose newest status row is older than its own SLA without risking a double-post.

Alert on three leading indicators specifically: manual-review depth crossing its rolling baseline, the share of records exceeding aging_escalation_threshold, and any single-day spike in auto-adjust volume relative to trailing average — the last of which is the fastest signal that a threshold was miscalibrated in a way that is quietly posting bad adjustments.

FAQ

Why not just prioritize the exception queue by dollar variance alone?

Because dollar variance alone ignores confidence and age. A large variance with high matcher confidence and a well-understood root cause may need less review time than a smaller variance the matcher flagged as ambiguous. The priority score blends all three signals so the queue surfaces genuinely high-risk records — material and uncertain and aging — rather than simply the largest numbers, which can overweight a single high-value but well-explained outlier.

What happens when a record’s priority score changes while it is sitting in the queue?

The score is recomputed, not cached, every time the queue is read for ranking — it is a pure function of the record’s current age, confidence, and variance against the live threshold configuration. This means a record’s position in the manual-review lane naturally rises over time purely from the age term, even if nothing else about it changes, which is the mechanism that prevents low-priority records from starving indefinitely.

How many retry attempts happen before a record moves to manual review instead of retry?

max_retries governs this, typically 3 attempts with exponential backoff and jitter. Once a record’s retry_count reaches that cap, the router no longer treats its failure as transient regardless of the root_cause_hint, and it falls through to the auto-adjust and manual-review gates like any other record. This prevents a genuinely broken upstream dependency from retrying indefinitely while masquerading as a healthy queue.

Can an auto-adjust decision be reversed after write-back?

Yes, but never by mutating the original record. A reversal is posted as a new, linked adjustment through the same Variance Write-Back and ERP Adjustment Posting path, referencing the original exception_id. The append-only design of the queue and the ledger means both the original auto-adjust and its reversal remain fully visible in the audit trail — reversing an entry is itself an auditable event, not an erasure.

Should retry and manual review ever compete for the same worker pool?

No. Retry is a system-driven, unattended process — it should never consume analyst time. Mixing the two lanes in the same operational dashboard tends to make retries look like backlog when they are actually just waiting out a backoff window. Keep retry visibility on the pipeline-health surface covered by Monitoring and Alerting for Reconciliation Pipelines and keep the analyst-facing queue scoped strictly to manual review.