Monitoring and Alerting for Reconciliation Pipelines Permalink to this section#

↑ Part of Core Architecture & Data Mapping for Reconciliation.

A reconciliation run that finishes without error is not the same as a reconciliation run that finished correctly. A batch job can exit zero while its auto-match rate has quietly collapsed from 94% to 61%, or while its exception queue has tripled in depth and started aging past the close deadline. Exit codes and row counts tell you the pipeline ran; they say nothing about whether the output is trustworthy enough to hand to finance. The gap between “the job succeeded” and “the numbers are right” is exactly the gap that observability is supposed to close, and for a reconciliation pipeline that gap has a dollar value attached to it every single close cycle.

The engineering problem is that most reconciliation pipelines are not long-running services — they are cron-triggered or orchestrator-triggered batch processes that start, do work, and exit. That process shape breaks the default Prometheus model, which assumes a scrape target is alive to be polled. It also means the four signals operations actually cares about — the auto-match rate, exception-queue depth and age, per-tier resolution counts, and stage latency against the close deadline — have to be emitted deliberately, from inside the run, and pushed somewhere durable before the process exits. This page treats monitoring as a design decision made at instrumentation time, not a dashboard bolted on afterward, and builds the metric emission, threshold calibration, and alert routing needed to catch a degrading run before it reaches the ledger.

Core Concept & Decision Criteria Permalink to this section#

Prometheus exposes three metric primitives, and picking the wrong one for a given reconciliation signal produces a metric that cannot answer the question you actually need answered at 2 a.m. A counter only increases (or resets to zero on restart) and is correct for anything you are tallying across the run — records matched, records rejected, retries attempted. A gauge holds a value that can move up or down and is correct for anything that describes current state — how many rows sit in the exception queue right now, how many workers are active. A histogram buckets observed values and is correct for anything you need a distribution over — how long each pipeline stage took, how old exception-queue rows are when they finally resolve. Applying a gauge where a histogram belongs (or vice versa) silently discards the percentile information an alert threshold needs.

The auto-match rate is not itself a metric primitive — it is a derived ratio computed from two counters (auto_matched_total / records_total), which means the alerting rule, not the instrumentation, owns the division. This distinction matters for the pull-vs-push decision that governs how these metrics leave the process. A long-running exporter or API service can expose /metrics and let Prometheus scrape it on its own interval, which is the natural fit for gauges that need to reflect live state. A batch job that starts, runs for four minutes, and exits has no live process for Prometheus to poll — by the time a scrape would fire, the job is gone. That shape requires the push model: the job computes its final counter, gauge, and histogram values and pushes them to a Pushgateway before exiting, and Prometheus scrapes the Pushgateway instead of the ephemeral job. The concrete counter, gauge, and histogram definitions used across a run — including label cardinality limits — are covered in Instrumenting Custom Pipeline Metrics with Prometheus; this page treats that instrumentation layer as a given and focuses on turning it into alerts that reach a human.

Implementation Permalink to this section#

The instrumentation lives inside the run itself: every stage records its outcome into typed metric objects, and a single finalize step pushes the accumulated state to the Pushgateway exactly once, right before the process exits. Keeping the push at the very end — rather than streaming updates mid-run — means a crashed run either pushes nothing (visible as a missing run in Grafana) or pushes a complete, consistent snapshot, never a half-updated one.

import logging
import time
from dataclasses import dataclass
from typing import Optional

from prometheus_client import (
    CollectorRegistry,
    Counter,
    Gauge,
    Histogram,
    push_to_gateway,
)

logger = logging.getLogger("recon.monitoring")

STAGE_LATENCY_BUCKETS = (0.5, 1, 2, 5, 10, 30, 60, 120, 300, 600)
QUEUE_AGE_BUCKETS = (60, 300, 900, 3600, 14400, 43200, 86400, 172800)


@dataclass
class ReconciliationMetrics:
    """Typed handle to every metric emitted by one reconciliation run."""
    records_total: Counter
    auto_matched_total: Counter
    tolerance_adjusted_total: Counter
    exception_routed_total: Counter
    exception_queue_depth: Gauge
    stage_latency_seconds: Histogram
    exception_age_seconds: Histogram
    registry: CollectorRegistry


def build_metrics() -> ReconciliationMetrics:
    """Register a fresh metric set scoped to a single push-gateway job run."""
    registry = CollectorRegistry()
    return ReconciliationMetrics(
        records_total=Counter(
            "recon_records_total", "Records evaluated this run",
            ["tier"], registry=registry,
        ),
        auto_matched_total=Counter(
            "recon_auto_matched_total", "Records auto-matched without human input",
            ["tier"], registry=registry,
        ),
        tolerance_adjusted_total=Counter(
            "recon_tolerance_adjusted_total", "Records matched within tolerance",
            ["tier"], registry=registry,
        ),
        exception_routed_total=Counter(
            "recon_exception_routed_total", "Records routed to the exception queue",
            ["tier", "reason"], registry=registry,
        ),
        exception_queue_depth=Gauge(
            "recon_exception_queue_depth", "Unresolved rows in the exception queue",
            ["tier"], registry=registry,
        ),
        stage_latency_seconds=Histogram(
            "recon_stage_latency_seconds", "Wall time per pipeline stage",
            ["stage"], buckets=STAGE_LATENCY_BUCKETS, registry=registry,
        ),
        exception_age_seconds=Histogram(
            "recon_exception_age_seconds", "Age of exception rows at scrape time",
            ["tier"], buckets=QUEUE_AGE_BUCKETS, registry=registry,
        ),
        registry=registry,
    )


def record_match_outcome(metrics: ReconciliationMetrics, tier: str, outcome: str) -> None:
    """Tally one record's disposition: auto_matched, tolerance_adjusted, or exception."""
    metrics.records_total.labels(tier=tier).inc()
    if outcome == "auto_matched":
        metrics.auto_matched_total.labels(tier=tier).inc()
    elif outcome == "tolerance_adjusted":
        metrics.tolerance_adjusted_total.labels(tier=tier).inc()
    else:
        logger.debug("non-terminal outcome=%s tier=%s recorded as pending", outcome, tier)


def record_exception(metrics: ReconciliationMetrics, tier: str, reason: str) -> None:
    """Route a failed record to the exception queue and log its reason for triage."""
    metrics.exception_routed_total.labels(tier=tier, reason=reason).inc()
    logger.warning("exception routed tier=%s reason=%s", tier, reason)


def observe_stage_latency(metrics: ReconciliationMetrics, stage: str, seconds: float) -> None:
    """Record wall-clock duration for a named pipeline stage."""
    metrics.stage_latency_seconds.labels(stage=stage).observe(seconds)
    logger.info("stage=%s duration_s=%.2f", stage, seconds)


def sample_queue_state(
    metrics: ReconciliationMetrics, tier: str, depth: int, ages_seconds: list[float]
) -> None:
    """Set current queue depth and observe the age of every unresolved row."""
    metrics.exception_queue_depth.labels(tier=tier).set(depth)
    for age in ages_seconds:
        metrics.exception_age_seconds.labels(tier=tier).observe(age)
    logger.info("queue tier=%s depth=%d oldest_s=%.0f", tier, depth, max(ages_seconds, default=0.0))


def finalize_run(
    metrics: ReconciliationMetrics,
    gateway_url: str,
    job_name: str,
    run_id: str,
    grouping_key: Optional[dict[str, str]] = None,
) -> None:
    """Push the completed run's metrics once, right before the process exits."""
    started = time.monotonic()
    grouping_key = grouping_key or {"run_id": run_id}
    push_to_gateway(gateway_url, job=job_name, registry=metrics.registry, grouping_key=grouping_key)
    logger.info(
        "pushed metrics job=%s run_id=%s gateway=%s elapsed_s=%.3f",
        job_name, run_id, gateway_url, time.monotonic() - started,
    )

record_match_outcome and record_exception are called from inside the matching loop, observe_stage_latency wraps each pipeline stage boundary, and sample_queue_state runs once after the exception queue is finalized for the run. finalize_run is the only function that talks to the network — everything else only mutates in-process metric objects, so a Pushgateway outage degrades observability, never the reconciliation itself. Grouping by run_id means a re-run of a failed batch does not silently overwrite the prior run’s pushed metrics under the same series; it is a distinct labeled instance, which keeps history intact for trend alerts.

Configuration & Threshold Calibration Permalink to this section#

An alert rule only earns its place if it fires early enough to matter and rarely enough not to be muted. The auto-match rate is the single highest-signal metric because it collapses match, tolerance, and exception outcomes into one number operations already understands, but it must be evaluated as a ratio of two counters over the run window, not as a raw gauge:

Here $\rho_{\min}$ is the floor auto-match rate below which a run needs human eyes before it reaches close, and $a_{\max}$ is the maximum tolerated p95 age of rows sitting in the exception queue — both are read off the histogram and counters emitted during instrumentation, never computed inline in application code. Calibrate both per supplier tier: a strategic-supplier tier with tight tolerance windows will naturally auto-match lower than a high-volume commodity tier, and a single global $\rho_{\min}$ either pages constantly on the strict tier or misses real degradation on the loose one. The exception-queue-specific thresholds — depth, age percentile, and per-reason breakdowns — are calibrated in detail in Alerting on Exception Queue Depth and Age; the table below is the run-level starting point.

The “missing run” row exists because a pipeline that crashes before instrumentation runs produces no metrics at all, which looks identical to silence from a healthy system unless you alert on the absence of an expected push. Pair every threshold with a rolling baseline rather than a hardcoded constant wherever volume is seasonal — month-end and quarter-end batches routinely run larger than mid-month runs, and a static depth threshold that is correct on day 3 will false-page on day 30.

Orchestration & Integration Permalink to this section#

The Pushgateway sits between the batch job and Prometheus, but it is not a metrics database — it is a durable landing pad that Prometheus scrapes on its own schedule. Configure the orchestrator (Airflow, Dagster, or a plain cron wrapper) to invoke finalize_run in a finally block so a failed run still pushes whatever partial counters it accumulated before the exception, tagged with status="failed" in the grouping key. Without that, a crashed run disappears from monitoring entirely instead of showing up as a visibly incomplete one. Once a run’s metrics have been scraped and its outcome is durable in Prometheus, delete the pushed group from the Pushgateway; leaving it in place means the next scrape re-reports a stale run’s numbers as if they were current, which is one of the most common sources of phantom alerts in a push-based setup.

Alertmanager routing should split on two axes: severity, and whether the alert concerns the run as a whole (auto-match rate, missing run) or a specific tier or stage (per-tier resolution, individual stage latency). Route run-level criticals to on-call paging with a short repeat interval near the close deadline, and route tier-level warnings to a chat channel the reconciliation team already watches — paging on every tier-level dip trains people to ignore pages. Silence rules matter as much as routing rules: a scheduled backfill or a deliberate historical re-run will spike counters in ways that look like anomalies, so tag those runs with a run_type="backfill" label at instrumentation time and exclude that label from the standard alert rules rather than manually silencing in Alertmanager after the fact.

This monitoring layer is downstream of two other subsystems it should stay loosely coupled to. The auto-match rate it alerts on is produced by the matching engine described in Matching & Reconciliation Algorithms — monitoring measures that engine’s output, it does not participate in the match decision. Rows that fail to match flow into the exception queue whose lifecycle — intake, triage, resolution — is owned by Exception Handling & Discrepancy Resolution, and specifically by the queue schema and priority rules in Exception Queue Design and Triage; this page only observes that queue’s depth and age from the outside. Once metrics land in Prometheus, they are the data source for the dashboards covered in Building Grafana Dashboards for Reconciliation Health, which give on-call staff the drill-down view an alert notification cannot.

Debugging & Pipeline Recovery Permalink to this section#

Most monitoring incidents are instrumentation bugs wearing a “pipeline is broken” costume, so triage the metrics path before assuming the reconciliation logic itself regressed.

• Stale Pushgateway groups. If a dashboard shows a run from hours ago as still “current,” the prior run’s group was never deleted after a successful scrape. Confirm the orchestrator’s cleanup step actually ran, and add a pushed_at timestamp gauge so staleness is directly queryable instead of inferred.
• Label cardinality explosion. Adding a high-cardinality label — a raw supplier_id or po_number instead of a bounded tier — turns a handful of time series into millions and can silently degrade or crash the Prometheus server. Audit every label against a known-bounded value set before it ships.
• Alert fatigue from static thresholds. A depth or latency threshold that was correct at initial rollout becomes noise as volume grows month over month. Rolling-baseline thresholds (as in the table above) age far better than constants; revisit any constant threshold that has fired more than twice in a quarter.
• Missing-run false negatives. If the “missing run” alert itself depends on the same Pushgateway path that failed, it will not fire. Back it with an independent orchestrator-level heartbeat (a dead man’s switch metric scraped from the scheduler, not the job) so a total pipeline failure is still caught.
• Correlating an alert to root cause. An Alertmanager notification tells you what crossed a threshold, not why. Use the run_id label to pivot from the alert into the Grafana dashboard for that run, then into the append-only records covered in Audit Trail and Compliance for Reconciliation to see the exact rows and reasons behind the metric that fired — the metric tells you where to look, the audit trail tells you what happened.

Recovery from a firing alert almost always means letting the affected run finish, then re-running the finalize step against corrected inputs rather than hand-editing a pushed metric value; a manually patched gauge breaks the audit trail’s guarantee that every observed number traces back to an actual pipeline execution.

FAQ Permalink to this section#

Why push metrics instead of letting Prometheus scrape the batch job directly? Permalink to this section#

Because the batch job is not alive long enough to be polled. Prometheus scraping assumes a target that stays up between scrape intervals; a reconciliation run that starts, processes a batch, and exits within a few minutes will usually be gone before the next scheduled scrape fires. Pushing the final metric snapshot to a Pushgateway before the process exits guarantees the numbers survive long enough for Prometheus to collect them on its own schedule.

Do pushed metrics need to be cleaned up after a run finishes? Permalink to this section#

Yes. The Pushgateway holds whatever was last pushed under a given job and grouping key until something deletes or overwrites it, so a completed run’s metrics will keep being re-reported as “current” on every subsequent scrape unless the orchestrator explicitly deletes that group once Prometheus has scraped it. Skipping cleanup is the most common cause of a dashboard that looks healthy hours after a pipeline actually failed.

Should exception-queue depth be a gauge, or should I derive it from a counter? Permalink to this section#

A gauge. Depth is current state — it goes up when rows enter the queue and down when they resolve — and a counter can only increase, so it cannot represent that. Reserve counters for cumulative totals like how many rows were ever routed to the queue, and use the gauge for how many sit there right now.

How do I avoid alert fatigue on a nightly close job whose volume varies by day of month? Permalink to this section#

Calibrate thresholds against a rolling baseline (a 14-run trailing median, for example) instead of a fixed constant. Month-end and quarter-end runs are legitimately larger than mid-month runs, and a static depth or latency threshold tuned for a typical day will either page constantly during month-end or miss a real regression during a quiet week. Tag deliberate backfills with a distinct label so they are excluded from the standard alert rules entirely rather than tuned around.

Is this different from just grepping application logs for error strings? Permalink to this section#

Yes, in what it can answer before someone reads a log line. Log-based alerting tells you an individual event happened; metric-based alerting on counters, gauges, and histograms tells you about aggregate state — a rate, a distribution, a threshold crossing — computed cheaply at query time across an entire run or fleet of runs. Logs remain essential for root-cause detail once an alert fires, but they do not scale as a primary signal for “is this run’s auto-match rate degrading” the way a counter ratio does.

Related Permalink to this section#

• Instrumenting Custom Pipeline Metrics with Prometheus
• Alerting on Exception Queue Depth and Age
• Building Grafana Dashboards for Reconciliation Health
• Exception Queue Design and Triage
• ← Back to Core Architecture & Data Mapping for Reconciliation