Building Grafana Dashboards for Reconciliation Health Permalink to this section#

↑ Part of Monitoring and Alerting for Reconciliation Pipelines.

An engineer paged for a reconciliation incident at 2 a.m. should not have to write PromQL from scratch to find out whether the problem is upstream ingestion, the matching engine, or a stuck exception queue. That diagnosis needs a single dashboard that answers “what changed and where” in under thirty seconds. This page specifies exactly five panels, the metric contract each one assumes, the row layout that keeps them scannable, and a provisioning script that treats the dashboard as a versioned artifact rather than something an on-call engineer edits by hand during an incident and forgets to save.

Operational Trigger Signals Permalink to this section#

Build or overhaul the reconciliation dashboard when any of the following becomes true. Each is observable from your own incident history or Prometheus retention, not a subjective call:

1. Mean time to detect (MTTD) exceeds 15 minutes: incidents are first noticed by finance asking why the close hasn’t finished, not by an engineer looking at a graph. If the metrics already exist (see Instrumenting Custom Pipeline Metrics with Prometheus) but nobody has a board that surfaces them, MTTD is a visualization gap, not an instrumentation gap.
2. Alerts fire with no diagnostic path: an exception queue depth or age alert pages someone, and the first thing they do is open five separate ad hoc Prometheus queries to figure out which vendor tier or pipeline stage is responsible. That triage sequence belongs on one screen.
3. Per-tier blind spots: an aggregate auto-match rate looks healthy at 91% while one vendor tier has silently dropped to 60%, because the aggregate is a weighted average across a stable majority. Nobody notices until that tier’s exceptions overflow.
4. Stage latency invisible against the close deadline: pipeline stages (ingest, normalize, match, write-back) each have their own duration, and none of them is visible relative to the hard close-of-books deadline until the deadline is already missed.
5. Dead-letter queue (DLQ) growth is discovered retroactively: DLQ depth climbs for days before anyone checks it, because it lives in a log line instead of a graph with a trend.

When two or more of these are true, stop querying Prometheus ad hoc and build the board described below.

Step-by-Step Implementation Permalink to this section#

Treat the dashboard as five panels with a fixed metric contract, laid out in three rows by information density: an at-a-glance top row, a trend row for the metric most predictive of an SLA breach, and a diagnostic row for root-causing once the top row shows red.

1. Row 1 — status row (stat panels): auto-match rate, exception queue depth, and DLQ depth, each as a single current-value stat with a sparkline and threshold coloring. This row alone should let someone say “healthy” or “not healthy” in five seconds.
2. Row 2 — trend row (time series, full width): per-stage pipeline latency plotted against the close deadline as a threshold line. This is the panel that predicts a missed close before it happens, so it gets the full width and the most screen real estate after the status row.
3. Row 3 — diagnostic row (two panels): per-tier resolution rate as a bar panel and exception queue age (p95) as a time series. These answer “which tier” and “how stale,” the two questions that follow immediately after the status row goes red.
4. Pin PromQL and thresholds to the same source of truth as your alert rules — a dashboard that shows a panel “green” while Alertmanager is actively paging on the same metric erodes trust in both.
5. Provision the board as JSON checked into the same repository as the pipeline code, applied through Grafana’s HTTP API or the sidecar provisioning directory, so a dashboard edit goes through the same review as a metric name change.
6. Validate against a live Prometheus instance before merging — an expression that references a metric name that was renamed during a refactor fails silently in the UI as an empty panel, not an error.

The auto-match rate panel is the anchor of the status row. It is a ratio of two counters, not a raw gauge, so it must always be computed as a rate over a window rather than read from a point-in-time value that a single slow scrape can spike:

PromQL for the five panels Permalink to this section#

# Panel 1 — Auto-match rate (stat, unit: percentunit)
sum(rate(reconciliation_matches_total{match_type="auto"}[15m]))
/
sum(rate(reconciliation_lines_processed_total[15m]))

# Panel 2 — Exception queue depth (stat, unit: short)
sum(reconciliation_exception_queue_depth)

# Panel 3 — Dead-letter queue depth by failure reason (stat + breakdown table)
sum by (failure_reason) (reconciliation_dlq_depth)

# Panel 4 — Stage latency vs close deadline (time series, unit: seconds)
histogram_quantile(
  0.95,
  sum by (stage, le) (rate(reconciliation_stage_duration_seconds_bucket[5m]))
)
# threshold overlay, same panel:
max(reconciliation_close_deadline_seconds)

# Panel 5a — Per-tier resolution rate (bar gauge, unit: percentunit)
sum by (vendor_tier) (rate(reconciliation_matches_total[1h]))
/
sum by (vendor_tier) (rate(reconciliation_lines_processed_total[1h]))

# Panel 5b — Exception queue age, p95 (time series, unit: seconds)
histogram_quantile(
  0.95,
  sum by (le) (rate(reconciliation_exception_age_seconds_bucket[5m]))
)

Provisioning the board as code means the panel definitions above become the single source of truth, checked in next to the exporters that emit reconciliation_matches_total and reconciliation_stage_duration_seconds. The following builds the Grafana dashboard JSON model programmatically rather than hand-editing it in the UI:

import json
import logging
from dataclasses import dataclass, field

logger = logging.getLogger(__name__)

PANEL_WIDTH = 8   # grafana grid units, 24 total per row


@dataclass
class Panel:
    """One panel in the reconciliation health dashboard."""
    title: str
    expr: str
    panel_type: str = "timeseries"
    unit: str = "short"
    width: int = PANEL_WIDTH
    height: int = 8
    thresholds: list[float] = field(default_factory=list)


def build_panel_json(panel: Panel, grid_x: int, grid_y: int, panel_id: int) -> dict:
    """Render one Panel dataclass into a Grafana panel JSON object."""
    return {
        "id": panel_id,
        "title": panel.title,
        "type": panel.panel_type,
        "gridPos": {"x": grid_x, "y": grid_y, "w": panel.width, "h": panel.height},
        "fieldConfig": {"defaults": {"unit": panel.unit}},
        "targets": [{"expr": panel.expr, "refId": "A"}],
    }


def build_reconciliation_dashboard(panels_by_row: list[list[Panel]]) -> dict:
    """Lay out panels row by row and return a full Grafana dashboard JSON model.

    panels_by_row: rows in top-to-bottom order; panel widths within a row
    must sum to 24 or less. Row 0 is expected to be the status row.
    """
    dashboard_panels: list[dict] = []
    panel_id = 1
    grid_y = 0
    for row_index, row in enumerate(panels_by_row):
        row_width = sum(p.width for p in row)
        if row_width > 24:
            # A misconfigured row silently truncates in the Grafana UI, so fail loudly here.
            logger.error("row %d exceeds 24 grid units (%d) — fix panel widths", row_index, row_width)
            raise ValueError(f"row {row_index} width {row_width} exceeds 24 units")
        grid_x = 0
        row_height = max((p.height for p in row), default=8)
        for panel in row:
            dashboard_panels.append(build_panel_json(panel, grid_x, grid_y, panel_id))
            grid_x += panel.width
            panel_id += 1
        grid_y += row_height
        logger.info("row %d: placed %d panels, next grid_y=%d", row_index, len(row), grid_y)

    logger.info("built dashboard with %d panels across %d rows", panel_id - 1, len(panels_by_row))
    return {
        "title": "Reconciliation Health",
        "schemaVersion": 39,
        "refresh": "30s",
        "panels": dashboard_panels,
    }


if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO)
    status_row = [
        Panel("Auto-Match Rate", 'sum(rate(reconciliation_matches_total{match_type="auto"}[15m])) / '
                                  'sum(rate(reconciliation_lines_processed_total[15m]))',
              panel_type="stat", unit="percentunit", width=8),
        Panel("Exception Queue Depth", "sum(reconciliation_exception_queue_depth)",
              panel_type="stat", unit="short", width=8),
        Panel("DLQ Depth", "sum by (failure_reason) (reconciliation_dlq_depth)",
              panel_type="stat", unit="short", width=8),
    ]
    trend_row = [
        Panel("Stage Latency vs Close Deadline",
              "histogram_quantile(0.95, sum by (stage, le) "
              "(rate(reconciliation_stage_duration_seconds_bucket[5m])))",
              panel_type="timeseries", unit="s", width=24, height=9),
    ]
    diagnostic_row = [
        Panel("Per-Tier Resolution Rate",
              "sum by (vendor_tier) (rate(reconciliation_matches_total[1h])) / "
              "sum by (vendor_tier) (rate(reconciliation_lines_processed_total[1h]))",
              panel_type="bargauge", unit="percentunit", width=12),
        Panel("Exception Queue Age p95",
              "histogram_quantile(0.95, sum by (le) "
              "(rate(reconciliation_exception_age_seconds_bucket[5m])))",
              panel_type="timeseries", unit="s", width=12),
    ]
    model = build_reconciliation_dashboard([status_row, trend_row, diagnostic_row])
    print(json.dumps(model, indent=2))

Apply the generated JSON through Grafana’s dashboard API (or drop it into the provisioning directory read by the Grafana sidecar) rather than pasting it into the UI import dialog — the API path is what makes the dashboard reproducible from a clean environment during a disaster-recovery drill.

Configuration Reference Permalink to this section#

Debugging & Recovery Permalink to this section#

A dashboard fails you in different ways than a pipeline does — mostly by lying quietly rather than crashing loudly:

• Blank panel after a metric rename — a Prometheus expression referencing a metric or label that changed during an exporter refactor renders an empty panel with no error banner. Run a validation pass against a live Prometheus instance as part of the same pull request that changes an exporter’s metric names, before the dashboard JSON merges.
• Stale panel with a healthy-looking flat line — if a panel’s refresh interval is shorter than the scrape interval on the underlying job, the line appears to update but is actually re-rendering the same stale sample. Confirm the panel refresh is never faster than scrape_interval.
• Cardinality explosion from the per-tier panel — adding vendor_tier as a by() label is safe at a few dozen tiers; adding raw vendor_id to the same query multiplies series count and can push a shared Prometheus instance into memory pressure. Keep per-tier breakdowns on a bounded, low-cardinality label and push per-vendor detail into a table panel with a topk() cap instead of an unbounded time series.
• Provisioning drift — someone edits a panel threshold directly in the Grafana UI during an incident and never ports the change back into the checked-in JSON. The next automated provisioning apply silently reverts it. Treat any UI edit as a draft; the JSON in the repository is the only durable copy.
• Dashboard shows green while Alertmanager is paging — this almost always means the dashboard query window differs from the alert rule’s for duration, so the panel is smoothing over a spike the alert correctly caught. Pin both to the same PromQL fragment rather than maintaining them independently.

The following validates a generated panel’s expression against a live Prometheus instance before a deploy, catching the blank-panel failure mode above:

import logging
from typing import Any

import requests

logger = logging.getLogger(__name__)


def validate_panel_expr(prometheus_url: str, expr: str, timeout_s: float = 5.0) -> bool:
    """Query Prometheus with a panel's PromQL and confirm it returns data.

    Returns False (and logs an error) if the expression is malformed or the
    metric it references currently has zero series — the two causes of a
    silently blank panel in the Grafana UI.
    """
    response = requests.get(
        f"{prometheus_url}/api/v1/query",
        params={"query": expr},
        timeout=timeout_s,
    )
    response.raise_for_status()
    payload: dict[str, Any] = response.json()

    if payload.get("status") != "success":
        logger.error("query failed: %s — %s", expr, payload.get("error"))
        return False

    result = payload["data"]["result"]
    if not result:
        logger.warning("query returned zero series, panel will render blank: %s", expr)
        return False

    logger.info("query ok: %s series returned", len(result))
    return True

FAQ Permalink to this section#

Should the dashboard also fire alerts, or is that a separate concern? Permalink to this section#

Keep them separate. The dashboard is for human triage once something has already paged or is being actively investigated; alert rules belong in Alertmanager, evaluated against the same underlying metrics as described in Alerting on Exception Queue Depth and Age. A panel can carry a visual threshold color for context, but the page-worthy decision should never live only in a dashboard that nobody is required to be looking at.

How many panels can a single reconciliation health board hold before it stops being useful? Permalink to this section#

Five to eight is the practical ceiling for a board someone opens mid-incident. The five panels here cover detection (auto-match rate), backlog (queue depth and age), root cause (per-tier resolution), and forward risk (latency vs deadline). Anything beyond that belongs on a secondary, more detailed board linked from this one — for example a per-vendor drill-down — rather than crowding the primary view.

Do I need a separate dashboard per vendor tier or per pipeline stage? Permalink to this section#

Not as a default. Start with the aggregate five-panel board and only fork a tier-specific or stage-specific board once a specific tier or stage has shown a sustained, recurring problem that the aggregate view keeps masking. A proliferation of near-identical boards, each drifting slightly from the others, is harder to trust than one board that always reflects current PromQL and current metric names.

Related Permalink to this section#

• Monitoring and Alerting for Reconciliation Pipelines — the parent monitoring approach this dashboard visualizes
• Instrumenting Custom Pipeline Metrics with Prometheus — where the counters and histograms behind these panels come from
• Alerting on Exception Queue Depth and Age — the paging rules that should share PromQL with this board’s thresholds
• Matching & Reconciliation Algorithms — the engine whose auto-match and per-tier rates this board surfaces
• ↑ Parent: Monitoring and Alerting for Reconciliation Pipelines