SC ReconciliationReconciliation Automation
  1. Home
  2. Ingestion
  3. Parsing EDI X12 Envelopes into Line Items
  4. Parsing X12 810 Functional Groups into Line Items

Parsing X12 810 Functional Groups into Line Items

↑ Part of Parsing EDI X12 Envelopes into Line Items.

An X12 810 invoice is not a flat record — it is a nested envelope where a single functional group (GS) can carry multiple transaction sets (ST), and a single transaction set can carry an arbitrary number of repeating IT1 loops, each one a candidate invoice line. A parser that treats IT1 as a fixed-position row instead of a repeating loop bounded by sentinel segments will silently drop trailing lines, miscount elements when a vendor omits an optional qualifier pair, or merge two invoices’ line items together when a functional group batches several ST transactions in one file. This page walks the GS → ST(810) → IT1 descent segment by segment, maps the positional IT1 elements onto a typed line-item record, and validates the ST/SE segment count so a truncated transmission fails loudly instead of loading a partial invoice.

Operational Trigger Signals

Build a dedicated IT1-loop parser — rather than reusing a generic delimited-file reader — when the inbound 810 feed shows any of the following:

  1. Multiple ST transactions per GS envelope. Trading partners that batch a day’s invoices into one interchange will nest several ST 810 transaction sets inside a single functional group. A parser keyed only on file boundaries, not ST/SE pairs, will concatenate line items from different invoices.
  2. Variable-length IT1 segments. Vendors routinely omit the basis-of-unit-price element (IT105) or send only one product ID qualifier pair instead of the two or three the spec allows. Fixed-width positional parsing throws index errors instead of treating the missing element as optional.
  3. Repeating loops with no explicit count. Unlike SE01, nothing in the IT1 loop itself declares how many lines are coming — the loop only ends when the segment stream stops emitting IT1 (and its subordinate PID/SAC segments) and reaches the invoice summary area (TDS, CAD). A parser that assumes a fixed number of lines per transaction set will truncate or overrun.
  4. SE01 segment-count failures on a subset of files. If your gateway or downstream loader intermittently rejects 810 files with a segment-count mismatch, the root cause is almost always the IT1 loop boundary being detected incorrectly, not corruption in transit.

When any of these show up in gateway logs or rejection reports, replace ad hoc splitting with an explicit state machine that tracks loop membership per segment, the same discipline the parent Parsing EDI X12 Envelopes into Line Items cluster applies to X12 envelopes generally. The typed output this parser produces is also the input the How to Map EDI 810 Invoices to Internal PO Schemas procedure assumes already exists — schema mapping is a separate concern from segment walking.

Step-by-Step Implementation

Treat the transaction set as a segment stream and track loop membership explicitly rather than trying to slice the raw payload with fixed offsets:

  1. Split on delimiters, not assumptions. Split the raw payload on the segment terminator, then split each segment on the element separator. Do not hardcode ~ and * — read them from the ISA header when the interchange declares custom delimiters.
  2. Walk to ST and confirm the transaction type. Reject early if ST01 is not 810; record ST02 as the control number you will reconcile against SE02.
  3. Capture header context before the loop. BIG carries the invoice number, invoice date, and PO reference; header-level ITD carries terms of sale. Both apply to every line item that follows and should be attached to the parsed invoice, not repeated per line.
  4. Enter the IT1 loop and map each occurrence. For every IT1 segment, map the positional elements onto named fields (quantity, unit of measure, unit price, product ID qualifier pairs) and construct a typed record. Log a warning, not a silent default, when an optional element is absent.
  5. Detect the loop boundary from TDS. The first TDS segment (invoice total) marks the end of the IT1 loop and the start of the summary area; anything from that point forward — including CAD carrier detail — belongs to the invoice summary, not to a line item.
  6. Validate the segment count at SE. Count every segment from ST through SE inclusive and compare it against SE01. A mismatch means the parser walked the loop incorrectly or the transmission was truncated — never load the line items from a transaction set that fails this check.
PYTHON
import logging
from dataclasses import dataclass, field
from decimal import Decimal
from typing import Optional

from pydantic import BaseModel, field_validator

logger = logging.getLogger(__name__)

# Product/service ID qualifiers this pipeline recognizes (X12 element 235 code list).
KNOWN_ID_QUALIFIERS = {"VP", "UP", "BP", "SK", "IN", "EN"}


class X12LineItem(BaseModel):
    """One IT1 loop occurrence: a single invoiced line on an 810 transaction set."""

    line_number: str
    quantity_invoiced: Decimal
    unit_of_measure: str
    unit_price: Decimal
    basis_of_unit_price: Optional[str] = None
    product_id_qualifier: str
    product_id: str
    secondary_id_qualifier: Optional[str] = None
    secondary_id: Optional[str] = None

    @field_validator("product_id_qualifier")
    @classmethod
    def qualifier_must_be_known(cls, value: str) -> str:
        """Flag qualifier drift without failing the line — see Debugging & Recovery."""
        if value not in KNOWN_ID_QUALIFIERS:
            logger.warning("unrecognized IT1 product id qualifier %r; check qualifier map", value)
        return value


@dataclass
class Parsed810:
    control_number: str
    invoice_number: str
    line_items: list[X12LineItem] = field(default_factory=list)


def _split_segments(raw: str, seg_term: str = "~", elem_sep: str = "*") -> list[list[str]]:
    """Split a raw X12 payload into segments, then elements within each segment."""
    segments = [seg.strip() for seg in raw.split(seg_term) if seg.strip()]
    return [seg.split(elem_sep) for seg in segments]


def _map_it1(elements: list[str]) -> dict[str, Optional[str]]:
    """Map positional IT1 elements onto named line-item fields, tolerating gaps."""

    def elem(pos: int) -> Optional[str]:
        return elements[pos] if len(elements) > pos and elements[pos] else None

    return {
        "line_number": elem(1) or "",
        "quantity_invoiced": elem(2) or "0",
        "unit_of_measure": elem(3) or "",
        "unit_price": elem(4) or "0",
        "basis_of_unit_price": elem(5),
        "product_id_qualifier": elem(6) or "",
        "product_id": elem(7) or "",
        "secondary_id_qualifier": elem(8),
        "secondary_id": elem(9),
    }


def parse_810(raw: str) -> Parsed810:
    """Walk GS -> ST(810) -> IT1 loop -> SE, emitting typed line items.

    The IT1 loop ends at the first TDS segment (invoice total), which marks
    the transition from repeating line items to the invoice summary area.
    """
    segments = _split_segments(raw)
    control_number = ""
    invoice_number = ""
    line_items: list[X12LineItem] = []
    in_summary = False
    st_index: Optional[int] = None

    for idx, elements in enumerate(segments):
        seg_id = elements[0]

        if seg_id == "ST":
            if elements[1] != "810":
                logger.error("ST01 %r is not an 810 transaction set; aborting parse", elements[1])
                raise ValueError(f"unsupported transaction set type {elements[1]!r}")
            control_number = elements[2]
            st_index = idx

        elif seg_id == "BIG":
            invoice_number = elements[2] if len(elements) > 2 else ""

        elif seg_id == "TDS":
            # Sentinel: everything from here on is summary, not a line item.
            in_summary = True

        elif seg_id == "IT1" and not in_summary:
            fields = _map_it1(elements)
            try:
                line_items.append(X12LineItem(**fields))
            except Exception as exc:  # noqa: BLE001 - surfaced, never silenced
                logger.error("IT1 line %s failed validation: %s", elements[1], exc)
                raise

        elif seg_id == "SE":
            declared_count = int(elements[1])
            counted = idx - st_index + 1  # inclusive of ST and SE
            if declared_count != counted:
                logger.error(
                    "SE01 declares %d segments but %d were counted between ST and SE",
                    declared_count, counted,
                )
                raise ValueError("ST/SE segment count mismatch")

    logger.info("parsed %d IT1 line items from ST%s", len(line_items), control_number)
    return Parsed810(control_number=control_number, invoice_number=invoice_number, line_items=line_items)

The segment-count check enforces the same invariant the X12 spec requires of every compliant transaction set:

SE01=nsegments(ST…SE)SE_{01} = n_{\text{segments}}(ST \ldots SE)

where nsegmentsn_{\text{segments}} counts every segment from ST through SE, inclusive. This check costs one integer comparison and catches truncation that a schema validator downstream — such as the models described in Schema Validation Using Pydantic — would only discover indirectly, as missing line items rather than a malformed envelope.

Walking an X12 810 transaction set from GS to typed line items The parser descends from the GS functional group header through the ST 810 transaction set header and header-level BIG and ITD segments into a repeating IT1 loop. Each IT1 occurrence is mapped onto a typed line item and emitted to the output record. The loop checks whether the next segment is TDS; if not, it continues consuming IT1 occurrences, and if so, it moves into the TDS and CAD summary area. The parser then validates that SE01 equals the counted segments between ST and SE: on a match it commits the parsed invoice at SE and GE, and on a mismatch it routes to a reject path tagged ST_SE_COUNT_MISMATCH. GS functional group ST transaction set ST01=810 · ST02=ctrl # BIG + ITD (header) invoice # · PO # · terms IT1 loop: baseline item qty · UOM · unit price product ID qualifier pairs Next segment = TDS? TDS + CAD (summary) invoice total · carrier detail SE01 = counted segments? SE / GE commit parsed invoice ready X12LineItem typed record, one per IT1 Reject batch ST_SE_COUNT_ MISMATCH no partial load emit no, loop yes yes no

Configuration Reference

The IT1 segment carries a fixed, positionally addressed element sequence. Map every accepted position explicitly rather than assuming a segment always arrives at full length:

Element Field Required Example Notes
IT101 line_number Yes 1 Assigned identification; use as the natural sort key within the invoice.
IT102 quantity_invoiced Yes 240 Parse as Decimal, never float, before any downstream tolerance math.
IT103 unit_of_measure Yes EA UOM code list 355; must resolve against your internal UOM table.
IT104 unit_price Yes 4.375 Decimal precision varies by vendor; do not round on ingest.
IT105 basis_of_unit_price No PE Frequently omitted; default to None, not 0 or empty string.
IT106 product_id_qualifier Yes VP First qualifier of a repeating pair; see qualifier drift below.
IT107 product_id Yes SKU-88421 The identifier the qualifier in IT106 describes.
IT108 secondary_id_qualifier No UP Second qualifier pair; up to three pairs are legal per the 4010 spec.
IT109 secondary_id No 00012345678905 Paired with IT108; commonly a UPC or buyer part number.

For how these fields ultimately reconcile against a purchase order rather than just being extracted, see How to Map EDI 810 Invoices to Internal PO Schemas, which covers the EDI 810 vs 850 Schema Mapping decision this parser’s output feeds into.

Debugging & Recovery

Two failure modes account for nearly every 810 line-item parsing defect reported from production:

  • Segment-count mismatch at SE. When SE01 does not equal the counted segments between ST and SE, the transmission is either truncated in transit or the loop boundary was detected incorrectly upstream of the count. Never load partial line items from a transaction set that fails this check — reject the whole ST and let the retry or resend process re-deliver it. Log the declared count, the counted count, and the last successfully parsed segment ID so triage does not require re-running the parser with a debugger attached.
  • Product ID qualifier drift. Vendors switch between VP (vendor part number), UP (UPC), BP (buyer’s part number), and SK (SKU) across contract renewals or system migrations without notice, and some send a qualifier your KNOWN_ID_QUALIFIERS set does not yet recognize. Treat an unrecognized qualifier as a warning, not a hard failure — the line is still structurally valid — but route it into a review queue so the qualifier map gets updated before the drift silently degrades matching quality downstream. If you are converting the same feed to an intermediate document form instead of parsing it positionally, Converting Legacy EDI XML to Structured JSON covers the equivalent drift problem for XML-wrapped EDI.
  • Multiple ST transactions in one GS envelope. If line items from two invoices appear merged, confirm the parser resets st_index, control_number, and line_items on every new ST, and that it never carries state across the SE boundary into the next transaction set.
  • TDS never reached. If a transaction set ends at SE without ever setting the in_summary flag, the file is missing its invoice total — treat this as a malformed transaction set, not an invoice with zero-dollar total, and reject it the same way as a segment-count failure.

FAQ

Why terminate the IT1 loop on TDS instead of just detecting when IT1 segments stop appearing?

Because PID, SAC, and other subordinate segments can legally appear between IT1 occurrences without ending the loop, a parser that stops on “first non-IT1 segment” will truncate the last line item’s descriptive detail. TDS (the invoice total) is a reliable, spec-defined sentinel that only appears once the line-item area is complete, so anchoring the loop boundary to it is more robust than inferring the boundary from the absence of IT1.

Should a segment-count mismatch fail the whole file or just the affected transaction set?

Just the transaction set, provided the GS/GE functional group boundaries are intact. Since a single GS envelope can carry multiple ST transactions, rejecting the entire file over one malformed 810 would also drop invoices that parsed and validated correctly. Reject at the ST/SE boundary, log the control number, and let the rest of the functional group continue processing.

How many product ID qualifier pairs should the parser expect per IT1 segment?

Up to three pairs are legal under the X12 4010 implementation (IT106/IT107, IT108/IT109, IT110/IT111), but most trading partners send one or two. Model every pair beyond the first as optional fields defaulting to None, and never assume a fixed count — a vendor adding a third qualifier pair mid-relationship should not require a parser code change.