MIT Sloan / BMW Group

Continual Prompt Optimization for Document Extraction

MIT Sloan Generative AI Lab × BMW Group · Spring 2026 · team of four

We taught an AI system to get better at reading BMW repair orders without retraining it. After each attempt, it studies its own mistakes and rewrites its instructions.

Mean extraction score across 6 repair orders: 0.33 → 0.83; Mean extraction score across 6 repair orders
Model weights changed. Only the prompt evolves: 0; Model weights changed. Only the prompt evolves
Prompts kept on the Pareto front, each best at something: 1 → 5; Prompts kept on the Pareto front, each best at something

PythonVision LLMsGEPAPareto OptimizationLLM-as-Judge

01 · The problem

Scanned PDFs in, structured JSON out

Each dealership repair order is a multi-page PDF, often scanned, containing several copies of the order (ASI, BWO, CSI…). BMW needed it as clean, nested JSON for its downstream systems.

CSI

BWO

ASI

ASIBWOCSIISIJSIWSI

Multi-page scanned PDF

{
"prefix": "CSI",
"header": { "vin": "WBA…", "ro_number": 629903 },
"content": {
"labor": [ {…}, {…}, {…} ],
"parts": [ {…}, {…} ]
},
"footer": { "total_charges": 1284.6 }
}

Structured JSON, one object per section

Pages are scanned images
6 section types, each with its own fields
No model retraining allowed

02 · The loop

The model stays frozen. Only the prompt evolves

The model's weights never change. Each round, the system tries, gets graded, figures out what went wrong, and writes a better prompt.

1
Extract. A vision LLM reads the PDF pages and returns JSON.
2
Evaluate. Each field is compared with the correct answer.
3
Reflect. An LLM reads the mistakes and explains the pattern.
4
Mutate. It rewrites the prompt with targeted new rules.
5
Select. The Pareto frontier decides which prompts survive.

03 · The evaluator

A score isn't enough. The grader has to explain the mistake

A single score like 0.41 says nothing about how to improve. The evaluator returns a list of exact mistakes, and the reflection step turns that list into specific prompt fixes.

evaluation reportIllustrative example

score 0.41 · 3 issues shown

CSI.content.labor[1]−0.08 structure

missingrow not extracted

ASI.footer.labor_amount−0.06 numbers

type"142.50" should be 142.5

WSI.header.advisor−0.04 text

value"J. SMlTH" should be "J. SMITH"

LLM diagnosis: Only the first labor row is extracted, and dollar amounts come back as strings.

What the score is made of

BMW's scoring weights

Structure45%
Numbers40%
Text15%

Two fixes that made the grader fair

Before these fixes, the loop was learning to please a flawed grader instead of reading documents better.

Match sections by code, not by position

Before

TruthModel

ASIASI

BWOCSI

CSIJSI

After

TruthModel

ASIASI

BWOmissing–

CSICSI

–extraJSI

One section out of order used to make every section after it look wrong.

Normalize numbers before comparing

Before"10.50"≠10.5

After10.5=10.5

Formatting slips no longer drown out real extraction errors.

04 · Pareto frontier

Why keep more than one prompt?

Repair orders fail in different ways. A prompt that fixes warranty sections might break customer sections. Instead of crowning a single winner, GEPA keeps a Pareto frontier of prompts.

aThe best-on-average prompt still loses somewhere

Five candidate prompts, scored on six documents. p4 has the best average, yet another prompt beats it on four of the six documents.

Score per prompt and document

Illustrative example

	Doc A	Doc B	Doc C	Doc D	Doc E	Doc F	Avg
p0	0.30	0.35	0.28	0.40	0.25	0.38	0.33
p1	0.62	0.58	0.40	0.55	0.42	0.50	0.51
p2	0.48	0.45	0.72	0.50	0.68	0.44	0.55
p3	0.55	0.70	0.50	0.62	0.45	0.58	0.57
p4	0.78	0.52	0.55	0.48	0.50	0.74	0.60

Outlined cell: best prompt for that document. Darker cells mean higher scores.

bKeep every prompt that is best at something

One prompt dominates another only if it is at least as good on every document and strictly better on at least one. The non-dominated prompts form the Pareto frontier, and all of them are kept.

x:y:Illustrative example

On the frontier: kept

Dominated: dropped

Hover or tap a prompt to see where it stands.

Try other document pairs. With all six documents at once, only p0 is dominated.

cPick the next parent by how many documents it wins

Each frontier prompt gets one vote per document where it scores best. A small bonus keeps zero-win prompts in play, so the search doesn't collapse onto a single idea.

Chance of being chosen as the next parent

Illustrative example

p0
dominated, not eligible
0%
p1
0 wins, exploration bonus
8%
p2
wins 2 docs
31%
p3
wins 2 docs
31%
p4
wins 2 docs
31%

Then: the reflection step studies one of the parent's two weakest documents, picked at random, so the loop doesn't keep fixing the same failure.

dGreedy replacement vs. a Pareto pool

Greedy (our baseline)

One prompt at a time

p1p2p3

Each new prompt replaces the last one, even when it fixes one document and breaks another. What the old prompt did well is lost.

Pareto (GEPA)

A pool of specialists

p1p2p3p0

Every prompt that is best at something stays in the pool. Only prompts beaten everywhere are dropped, so nothing that worked is forgotten.

In our run, the pool kept growing

Prompts on the Pareto front, per iteration (6 BMW repair orders)

1Iter 1
1Iter 2
2Iter 3
3Iter 4
5Iter 5

By iteration 5, all five candidate prompts were non-dominated. Each one was the best on a different mix of documents.

05 · Results

From 0.33 to 0.83 in five iterations

Score per GEPA iteration, 6 repair ordersMean scoreWorst document

The iteration-5 jump reflects the whole pipeline. Fixes to the evaluator (section alignment, number normalization) landed alongside the prompt changes.

Fewer missing sections
All labor & parts rows captured, not just the first
Numbers returned as numbers

06 · Takeaways

What I'd tell the next team

The evaluator sets the ceiling

The optimizer only learns what the evaluator can see. We had to fix the grader before the loop improved anything real.

A better model beats a longer prompt

Upgrading the base extractor gave the biggest jump. Prompt evolution added a real, smaller gain on top.

Six documents is a start

The next step is a held-out test set, to check whether the evolved rules transfer to unseen repair orders.