Agentic Document Processing for Insurance Claims

The constraint

A claims pipeline built over a decade as nested conditionals. 1,400 special cases, no test coverage, and one person who understood roughly two-thirds of it. Every new document format added rules; nothing was ever removed.

The tempting move is to replace the whole thing with an LLM. That is also how you turn a slow, auditable system into a fast, unauditable one.

Approach

Split the problem by whether judgement is actually required:

• Structured extraction — form fields, tables, dates, amounts — goes to Document AI. Deterministic, cheap, high accuracy, no hallucination surface at all.
• Classification and routing — which claim type, which queue — stays as rules where the rules are correct. Most of the 1,400 were, they were just undocumented.
• Genuine ambiguity — free-text descriptions, inconsistent attachments, conflicting fields — goes to an agent.

That last category turned out to be about 8% of documents. Sending 100% through an LLM to handle 8% would have been slower, more expensive, and less accurate on the easy 92%.

Architecture

Pub/Sub decouples ingestion from processing, so a spike in submissions queues rather than failing. Each stage is a Cloud Run service: extract, classify, then an agent stage only for documents flagged ambiguous by the classifier.

The agent produces a structured decision plus a confidence score and a reasoning trace. Below a confidence threshold, the claim routes to a human queue with the agent's reasoning attached — so the reviewer starts from an informed position rather than a blank form. Above threshold it proceeds, with the trace retained for audit.

Firestore holds per-claim state, which makes the pipeline resumable. A failure at the agent stage does not re-run extraction.

What broke

The confidence score from the model was close to useless — poorly calibrated and clustered near the top, as generative models generally are. Nearly everything looked confident.

Replaced with an ensemble agreement signal: run the extraction path and the agent path, compare structured outputs, and treat disagreement as low confidence. Cruder, far better calibrated, and it uses the deterministic path you already have as a check on the probabilistic one.

Outcome

Special-case rules reduced from 1,400 to around 40 genuinely irreducible ones. Human review targeted at real ambiguity rather than sampling randomly. And — the part the client cared about most — every automated decision has a retained reasoning trace.