SC ReconciliationReconciliation Automation
  1. Home
  2. Matching
  3. Algorithm Performance Optimization for Supply Chain Reconciliation
  4. Blocking Strategy Selection for High-Volume Supplier Feeds

Blocking Strategy Selection for High-Volume Supplier Feeds

↑ Part of Algorithm Performance Optimization.

A fuzzy scoring stage that compares every invoice line against every open PO line is a full cross product: at 50,000 invoice lines and 50,000 PO lines that is 2.5 billion candidate pairs, and no scorer — however fast — finishes that inside an overnight window. Blocking is the fix, but the choice of blocking key is the entire engineering decision: too coarse a key and buckets stay huge, so the cross product barely shrinks; too aggressive a key and true matching pairs land in different buckets and silently disappear before they ever reach the scorer. This page is narrowly about picking and validating that key for a high-volume supplier feed, not about the scorer itself — scorer selection for the within-block comparisons is covered separately in Levenshtein vs Jaro-Winkler for PO Number Matching.

Blocking sits directly upstream of the tiered exact-then-fuzzy design used across the matching engine: the exact pass already removes the cleanest records, and blocking is what keeps the residual fuzzy pass affordable once that residual is still in the tens of thousands of rows. Skip it and a pipeline that matches correctly in staging will time out the moment a real supplier feed hits production volume.

Operational Trigger Signals

Blocking strategy work is triggered by measurable symptoms in the pipeline’s own candidate-pair telemetry, not by a general sense that matching feels slow:

  1. Logged candidate-pair count exceeds the batch budget. If the pipeline emits a full_cross_product figure (as shown in the implementation below) above roughly 50–100 million pairs for a single run, unblocked scoring will not finish inside a standard overnight window on commodity hardware.
  2. Reduction ratio metric falls below 0.90. A blocking pass that only removes 70–80% of candidate pairs is usually using a key with too few distinct values — vendor_id alone on a feed dominated by two or three suppliers, for example — and needs a compound key.
  3. A single block holds a disproportionate share of rows. Skewed block sizes (one vendor_id + period bucket holding 40% of the day’s volume) mean the O(n) win from blocking collapses back toward O(n²) inside that one bucket, even though the aggregate reduction ratio looks fine.
  4. Spot-audited true matches are landing in different blocks. If a sample audit of known-good invoice/PO pairs shows more than roughly 1–2% split across blocks, the key is too strict and pair completeness is silently degrading the matched ledger — a failure mode blocking metrics alone will not surface.

Any one of these signals is reason enough to revisit the blocking key rather than tune the scorer, because scorer tuning cannot fix a candidate set that never contained the correct pair.

Step-by-Step Implementation

Treat blocking as a partitioning step that runs before scoring and is auditable on its own, independent of match quality:

  1. Pick a blocking key shaped to the data. Use a compound key with high cardinality but low drift — vendor_id + billing_period is the default for governed feeds because both fields are structured and rarely typo’d.
  2. Build the key as a vectorized column, never inside a loop, so key construction cost stays linear in row count.
  3. Partition both frames with groupby on the key and take only the buckets that exist on both sides — a block with invoice rows but no PO rows (or vice versa) can never produce a match and should be routed straight to the exception queue.
  4. Form the within-block cross product only. For each shared bucket, compare every invoice row against every PO row in that bucket and nowhere else.
  5. Log the candidate-pair count and reduction ratio for every run so a regression in blocking effectiveness is visible in monitoring before it shows up as a missed SLA.
  6. Periodically audit pair completeness against a labeled sample of known-good matches to catch a key that is quietly dropping true pairs, since reduction ratio alone cannot detect that failure.
PYTHON
import logging

import pandas as pd
from rapidfuzz import fuzz

logger = logging.getLogger("recon.match.blocking")


def make_block_key(df: pd.DataFrame, id_col: str, period_col: str) -> pd.Series:
    """Composite standard-blocking key: vendor_id + billing period.

    Two records can only ever be compared once they land in the same
    (vendor_id, period) bucket, so this key is applied before any
    scoring happens — it never touches the description or price fields.
    """
    vendor = df[id_col].astype("string").str.strip().str.upper()
    period = df[period_col].astype("string").str.strip()
    return vendor.str.cat(period, sep="|")


def score_within_blocks(
    invoice_df: pd.DataFrame,
    po_df: pd.DataFrame,
    block_col: str = "block_key",
    desc_col: str = "line_description",
    sim_floor: float = 0.85,
) -> pd.DataFrame:
    """Score candidate pairs only within shared blocks, never across them.

    Groups both frames on block_col and forms the cross product only
    inside each shared bucket, so total scoring work is the sum of
    per-block products instead of the full |invoice| x |po| product.
    """
    inv_groups = dict(tuple(invoice_df.groupby(block_col)))
    po_groups = dict(tuple(po_df.groupby(block_col)))
    shared_blocks = inv_groups.keys() & po_groups.keys()
    orphan_inv = len(invoice_df) - sum(len(inv_groups[k]) for k in shared_blocks)

    logger.info(
        "blocking_summary invoice_blocks=%d po_blocks=%d shared_blocks=%d orphan_invoice_rows=%d",
        len(inv_groups), len(po_groups), len(shared_blocks), orphan_inv,
    )

    results: list[dict[str, object]] = []
    candidate_pairs = 0
    for key in shared_blocks:
        inv_block = inv_groups[key]
        po_block = po_groups[key]
        candidate_pairs += len(inv_block) * len(po_block)
        for inv_row in inv_block.itertuples():
            for po_row in po_block.itertuples():
                score = fuzz.token_set_ratio(
                    getattr(inv_row, desc_col), getattr(po_row, desc_col)
                ) / 100.0
                if score >= sim_floor:
                    results.append({
                        "block_key": key,
                        "invoice_idx": inv_row.Index,
                        "po_idx": po_row.Index,
                        "similarity": score,
                    })

    full_cross_product = len(invoice_df) * len(po_df)
    reduction_ratio = 1.0 - (candidate_pairs / max(1, full_cross_product))
    logger.info(
        "candidate_count blocked_pairs=%d full_cross_product=%d reduction_ratio=%.4f",
        candidate_pairs, full_cross_product, reduction_ratio,
    )
    return pd.DataFrame(results)

The orphan_invoice_rows figure and the candidate_count log line are the two numbers worth wiring into monitoring: the first tells you how many rows the blocking key excluded from scoring entirely (a missing or malformed vendor_id/period value), and the second is the raw input to the reduction-ratio trigger signal above.

Full cross product versus a blocked partition Two six-by-six grids of invoice-line versus PO-line pairs. In the left grid every cell is filled, representing all 36 pairs being scored — the full cross product. In the right grid the same 36 cells are split into three 2x2 diagonal blocks by a shared vendor and period key; only the 12 cells inside those blocks are filled and scored, and the remaining 24 off-diagonal cells are left unscored because the key proves they cannot match. Unblocked: full cross product 36 of 36 pairs scored Blocked: vendor_id + period 12 of 36 pairs scored scored pair (both feeds compared) scored pair (within block) skipped — different block key Same 36 pairs, three vendor/period blocks: 24 comparisons eliminated before scoring runs

The reduction achieved by any blocking key is measured directly against the unblocked baseline. For an invoice feed AA and a PO feed BB partitioned into kk shared blocks, the reduction ratio is the share of the full cross product that blocking eliminates:

RR=1−∑i=1k∣Ai∣×∣Bi∣∣A∣×∣B∣RR = 1 - \frac{\sum_{i=1}^{k} |A_i| \times |B_i|}{|A| \times |B|}

Reduction ratio alone says nothing about correctness, because a key that puts every record in its own singleton block scores RR≈1RR \approx 1 while destroying every match. The counterweight is pair completeness, the share of the true matching pairs StrueS_{\text{true}} that the blocking key still leaves in the same block and therefore reachable by the scorer:

PC=∣{(a,b)∈Strue:block(a)=block(b)}∣∣Strue∣PC = \frac{\bigl|\{(a,b) \in S_{\text{true}} : \text{block}(a) = \text{block}(b)\}\bigr|}{|S_{\text{true}}|}

A usable blocking key maximizes RRRR subject to PCPC staying above a floor the business can tolerate — typically 0.98–0.995 for financial reconciliation, since every point lost from PCPC is a true match that silently never reaches scoring at all, as distinct from a match the scorer sees and rejects.

Configuration Reference

The three blocking families trade reduction ratio against pair-completeness risk differently, and the right default depends on how clean the feed’s structured fields are versus how much the matching signal actually lives in free text:

Strategy Blocking key Typical reduction ratio Pair-completeness risk Best for
Standard blocking Exact match on vendor_id + billing_period 0.95–0.999 High if the key itself drifts (a stray vendor_id typo splits a true pair permanently) Governed feeds with clean, structured identifiers
Sorted-neighborhood Sort on a normalized key, slide a fixed window w over the sorted list 0.85–0.97 Moderate; too small a w misses pairs separated by sort noise, too large erodes the reduction Feeds with minor key drift (whitespace, casing) but no reliable exact partition
Canopy / q-gram Overlapping canopies from loose and tight q-gram similarity thresholds 0.70–0.95 Low; overlapping canopies deliberately let borderline records join more than one block High-drift free text such as line descriptions with no stable identifier

For most supplier feeds the right architecture layers these: standard blocking on vendor_id + billing_period as the primary partition (cheapest, highest reduction ratio), with a canopy pass on line_description inside the largest blocks when the standard key alone still leaves a bucket too big to score in full. Sorted-neighborhood is worth adding only when the primary key has no reliable exact form to partition on at all — a legacy feed with inconsistent vendor identifiers, for instance.

Debugging & Recovery

A blocking layer fails silently by default — the pipeline runs faster and nobody notices the missed matches until a downstream audit finds them — so recovery has to be built around the two metrics above, not around runtime alone:

  • Track reduction ratio and pair completeness as paired metrics, never one without the other. A dashboard that only shows candidate-pair count falling will not distinguish a healthy tuning change from a key that started silently excluding true pairs.
  • Route orphan rows to the exception queue, not to a bigger fallback block. Records with a missing or malformed blocking field (null vendor_id, unparseable period) should never be dumped into one giant catch-all bucket — that reintroduces the O(n²) cost the blocking layer exists to remove, concentrated in a single block.
  • Audit block-size skew per run. Log the largest block’s row count alongside the total; a block holding more than a few percent of total volume is a signal that the key needs a secondary discriminator (add a line_category or contract_id component) before it dominates the scoring stage’s wall-clock.
  • Replay against a labeled sample regularly. Because a bad blocking key produces a false negative the scorer never even evaluates, the only reliable detector is a periodic sample audit comparing known-good matched pairs against the block assignments, feeding the pair-completeness figure back into monitoring.
  • Version the blocking key definition alongside the scorer. A key change (adding a component, widening a sorted-neighborhood window) changes which pairs are even reachable, so it must be logged and reproducible the same way scorer and threshold changes are — this is the same audit discipline covered in Algorithm Performance Optimization.

FAQ

How small should a block get before blocking stops helping?

Once a block’s average size drops to single digits, the marginal gain from further splitting is negligible and the overhead of managing more blocks (groupby cost, key construction, monitoring more buckets) starts to dominate. The useful target is not the smallest possible block but the largest block size your scoring budget can absorb inside the batch window — size blocks to that ceiling rather than to zero.

Can standard blocking and canopy blocking run together?

Yes, and for high-volume feeds this is usually the right design: use standard blocking on vendor_id + billing_period as the cheap primary partition, then apply a canopy pass on line_description only inside blocks that are still too large after the primary split. Running canopy blocking across the whole feed from the start is more expensive than necessary once a cheap exact key has already done most of the reduction.

What happens to invoice rows that don’t match any PO block?

They should be logged with the orphan count shown in the implementation above and routed to the exception queue rather than silently dropped or forced into a fallback comparison against the entire PO set. A rising orphan count usually points to a missing or malformed vendor_id/billing_period value upstream — the same class of issue that drives a pipeline toward fuzzy matching in the first place, so the two problems are worth triaging together.