Insurance Agent Benchmark (IAB)

Why we built an insurance agent benchmark

At Cooper, we're building an AI coworker for insurance. Doing that has forced us to think deeply about a deceptively simple question: how should we measure whether an AI agent is actually getting better at the work insurance professionals do?

Insurance already has useful benchmarks. InsuranceQA evaluates question answering in the insurance domain. INS-MMBench evaluates multimodal understanding and reasoning across insurance scenarios. InsureBench measures language models on document-grounded underwriting and claims work.

The Insurance Agent Benchmark (IAB) aims to evaluate whether AI agents can complete insurance workflows from start to finish.

In practice, insurance work rarely arrives as an isolated question. It arrives as an email with attachments, a scanned ACORD with handwritten notes, a loss run that needs to be reconciled against an application, a policy hundreds of pages long, or a workbook with dozens of tabs. Sometimes the most important thing the system can do is recognize that a value is missing rather than confidently invent one.

This first phase focuses on document understanding. Later phases will cover whole workflows, then long-horizon tasks and browser-use capabilities.

Key Findings

• The same model scores higher with Cooper's harness. Across all 17 models, Cooper improves the median score by 9.4 percentage points. The lift ranges from about a point for Grok and GPT-5.6 Luna, well inside single-run noise, to roughly 15 points for Claude Haiku 4.5 and Meta Muse Spark 1.3. The biggest gains show up on the hardest files: long documents, oversized files and broken inputs that a raw call struggles to ingest.
• Gemini 3.8 Flash moves the efficiency frontier. At 85.6%, it posts the highest accuracy point estimate in the benchmark, within the statistical band of the top Claude runs, while costing about $68 for the full corpus. Gemini 3.7 Flash remains close behind at 83.3% for $33 and ties for the highest reliability at 98.2%.
• AI is still bad at saying “I don’t know.” When a field is blank, redacted or unreadable, most models still invent values often enough to matter. Claude Sonnet 5 and Fable 5.1 are lowest at 10.6%; Opus 5 and Meta Muse Spark are next at 14.9%.
• Prompt injection looks more manageable than long policies. Sixteen of 17 models resist document-embedded injection at 90% or better with Cooper. Long-policy clause retrieval remains far more uneven: Fable 5.1 reaches 89.7%, but most Cooper runs remain below 80%. The capability is there, but it isn't consistent across models.
• Reliability is a system property. Model-alone calls fail to produce a usable answer on 7–28% of cases, mostly on very large or broken files. With Cooper, every model stays above 90% reliability. The highest reliability is 98.2%, reached by Gemini 3.5 Flash Lite, Gemini 3.7 Flash and Fable 5.1.

How does the harness impact model performance?

Cooper raised accuracy for every model we tested, without a model upgrade. Its value is clearest on the files that raw model calls struggle to process: long documents, oversized files and damaged inputs. Better document handling lets us get more accurate answers from the models we already have.

For a model to answer a question about a policy, it has to be able to read the file in the first place. A scanned application may need rendering; a large workbook may need to be split into chunks before the model can use it. The harness makes those decisions, which affect what information reaches the model.

We ran every model across the same 166 cases twice. In the model-alone run, it received the raw file and question in a single call. With Cooper, file-type routing chose the extraction or rendering path, and large files went through chunked map-reduce, with the candidate model doing all the reading and reduction. The model, document and question stayed the same across the two runs.

How we built IAB

The corpus is built to resemble the pile of files that lands on a commercial-lines desk. The 166 documents come from live brokerage and carrier workflows under existing consent agreements. They range from a single-page certificate of insurance and photographed auto ID cards to full policy wordings with endorsement schedules, program books hundreds of pages long, large multi-location statements of values, and underwriting workbooks with more than fifty sheets. The two largest workbooks each pushed Cooper through nearly 16 million tokens to answer a single question. In between are multi-year loss runs, quotes, binders, endorsements, and broker submission emails that arrive as Outlook .msg files with the application, SOV and loss run still attached.

Much of it arrives exactly as messy as desks receive it: faxed and low-quality scans, pages rotated or stamped over, handwritten annotations, checkbox-heavy ACORDs photographed rather than scanned. One scanned application alone expands to 1.9 million tokens of extracted text. A few files lie about themselves: a wrong extension, a corrupt file truncated partway through, a “policy” with nothing inside.

None of it is synthetic. Of the 166 documents, 150 are untouched originals; the other 16 are originals modified to set a specific trap: an embedded instruction, a blacked-out name, a password. A team of insurance professionals prepared and checked the ground truth for every case against the source document. Where a value is absent or unreadable, the answer key says so, and reporting any value at all counts as a failure.

Eleven ways to test document understanding

What the corpus does not cover: personal lines, non-US markets, languages beyond a three-document bilingual slice, handwriting-only documents, and any judgment task (appetite, pricing, coverage adequacy). It measures reading, not underwriting.

Same model. Two systems.

With Cooper, the document runs through Cooper’s production document system: type-and-size routing, format-specific extraction or rendering, and chunked map-reduce for large files, with the candidate model doing all reading and reduction. Model alone sends the raw file and the question to the same model in a single call, with no scaffold. System and model are reported separately, and every number in this report names both.

How answers are graded

A pinned frontier LLM grades each answer against the human-verified ground truth on meaning, not string match: $1M equals $1,000,000, date formats are interchangeable, field order is irrelevant. It returns correct, partial or incorrect plus a 0–1 score; partial credit applies on multi-field answers. Accuracy throughout this report is the mean judge score over all 166 cases, shown as a percentage. Structured-extraction cases also get field-level F1 against the expected JSON, and absent-value cases record whether the model invented a value (hallucination rate). Grading against a human-written answer key is the main defence against the known ways an LLM judge can fail.

What counts as a failure

A benchmark case can fail for two different reasons, and they are scored differently. Environment faults of the evaluation run, failures in the surrounding infrastructure that say nothing about the model, are excluded from that run's denominator, the standard treatment for infrastructure errors. No run needed the exclusion: five Cooper runs hit environment faults on a first attempt and were re-run to completion, so every denominator is the full 166. Genuine model and pipeline failures, timeouts, empty extractions and refusals, stay in: they score zero and count against reliability, because a model that cannot process the document has failed the task.

How to read small differences

Every run is a single pass over n=166. Treating scores as Bernoulli (a conservative upper bound on variance), the 95% interval on an accuracy of 80% is ±6.1 percentage points. Treat ranks as bands: differences of about 6 points or less should not be treated as definitive rankings. The interval above describes a single run; it does not establish whether a difference between runs is statistically significant. Per-track slices (n between 8 and 42) are directional only.

Results in detail

5.1 · Full leaderboard

Accuracy is the mean judge score across a run's scored cases, shown as a percentage. The Cases column gives that run's denominator, the full 166 for every run (§4). The Failures column counts genuine model and pipeline errors, which score zero. Est. cost uses each run's token count and OpenRouter list rates (method in §4). Rows keep the same model order as you switch between With Cooper and Model alone. Click a column to reorder the model groups.

5.2 · Accuracy, cost and the efficiency frontier

5.3 · What models can and cannot do yet

5.4 · The trust problem: knowing when not to answer

5.5 · Speed in production

Where AI still breaks

Aggregate scores hide what failure actually looks like. These three de-identified cases show the failure classes behind the numbers. No single example is representative; business names are constructed or replaced under the corpus PII policy.

How we judge the answers

A pinned frontier LLM grades each answer against the case's human-verified ground truth on semantic equivalence, with partial credit where appropriate. The human-written answer key is the main defence against known LLM-judge failure modes: the judge checks the answer against that key.

Scope and limitations

• Corpus scope: US commercial lines, English (a 3-document bilingual slice), no personal lines. Property-casualty submission documents dominate.
• Point in time. Models are OpenRouter-pinned IDs measured 13 Aug – 9 Sep 2026; vendor-side updates shift results silently. Cost estimates use OpenRouter list prices of August 18th 2026. The corpus is private: real documents from live workflows, used under consent.

From documents to the whole job

Documents come first because later tasks depend on reading them correctly. The later phases apply the same method, human-verified ground truth and disclosed grading, to more of the job.

• Phase two: whole workflows. A submission taken from inbox to underwriter-ready: read every attachment, notice what is missing, draft the chase email, fill the forms, reconcile the SOV against the application. Scored end to end, the way a desk manager would score a new hire.
• Phase three: long-horizon tasks and browser use. Tasks that stretch over hours or days and leave the chat window entirely: working carrier portals in a browser, tracking a renewal to its deadline, following an email thread until the file is complete.

Citation

This first release covers 166 cases from corpus iab-docs-2026-08.