A UK driving school uses an AI agent to answer its support email, 3,480 cases a month. Most of those cases turn on a few decisions: whether a refund is due, whether a medical request came with evidence, and whether to change a booking, open a task for staff or hold the reply for a person. Before any changes, the agent sometimes softened a firm no on a refund, said it had done things it hadn’t, or failed to pass a case to a person.
Over six rounds, we used Permafrost™ to test changes to the agent’s instructions and to measure what each one did to those decisions. For the main comparison we kept the model fixed, Claude Sonnet 4.6, and ran the original and the new instructions on the same 300 synthetic cases. Cases graded send-ready nearly doubled, from 48 to 92. Cases with a serious error fell from 94 to 44.
Evaluation setup
two instruction sets · one model · 300 shared cases
Only the instructions changed. No model was retrained.
Same model, new instructions
The send-ready rate rose from 16.0% to 30.7%, a gain of 14.7 percentage points, or 92% in relative terms. The paired 95% interval runs from +9.2 to +20.5 points.
“Send-ready” is a grade. It means the grader found no serious or major errors. A correct decision to hold a case, or not to reply, counts too. It does not mean the email could have gone out without review.
Same-model evaluation
300 paired cases · Claude Sonnet 4.6 · same grader
No serious or major errors; correct holds and no-reply decisions count too.
Cases with a serious error fell by 53%. The rubric score, which weights each requirement by severity, rose from 43.3 to 57.3 out of 100. The score gives partial credit; the send-ready rate only asks whether a case cleared the bar for serious and major errors. Neither one measures customer satisfaction or how the agent does with real customers.
Six rounds of changes
In each round we proposed changes, tested them and decided which to keep. Most were left out. One round kept nothing new, and the first version turned out to be a side branch that the final agent does not descend from.
Version history
six rounds · 2–13 October 2026
Ask for the evidence first.
Medical requests that came without evidence needed a clearer next step. The kept change tells the agent to ask for it first.
Decision and evidence Small gain
Keep that change alone. The other candidates did not hold up on the holdout set, or missed the threshold we set in advance.
On the matched GLM runs: +2.39 score points over V4, with a 95% interval of +0.47 to +4.41. The changes in the send-ready and error rates were inconclusive.
One case short
One V5 candidate had the agent check each draft against the rubric and then repair it. Its average score went up, but only 46 of 66 paired cases scored the same or better. Our rule, set before the test, was 70%, which meant 47. We left it out.
The 70% rule
check-and-repair candidate · 66 paired development cases
One more case scoring the same or better would have passed. The candidate was dropped.
From V5 we kept only the change about medical evidence. On the matched GLM runs its gain over V4 was small, but the paired interval supports it. V6 then added an ordered checklist for policy and actions. Its extra gain over V5 is inconclusive, though V6 is far ahead of the original agent.
Scores by version
rubric score / 100 · GLM-5.3 Flash · 300 cases
What got better, and what got worse
The net gain of 44 send-ready cases hides some losses. 55 cases became send-ready, and 11 that had been send-ready fell back. The grid groups every case by its original grade.
Paired outcomes
one square per case · grouped by original grade
55 cases became send-ready and 11 fell back. Net gain: 44.
Show the counts
| Original → V6 | Send-ready | Needs edits | Serious error |
|---|---|---|---|
| Send-ready | 37 | 8 | 3 |
| Needs edits | 29 | 120 | 9 |
| Serious error | 26 | 36 | 32 |
Serious errors moved both ways too. In 62 cases a serious error went away, 12 other cases gained one, and 32 still had one. So the 53% drop shows the agent got better. On its own, it is not a reason to ship it.
Results by category
The test covers 15 kinds of support request, with 20 cases each. Groups that small show what happened, but they can’t prove an effect in any one category.
Support categories
same-model comparison · 20 synthetic cases per category
Higher is better. A rust-coloured line marks a change for the worse. Hover or tap a row for its numbers.
Changing the model too
With the V6 instructions frozen, we ran them on three models. The result above isolates the change in instructions. Swapping the model as well makes a different comparison, so we report it separately.
On GLM-5.3 Flash, V6 produced 108 send-ready cases at about $0.0103 of recorded model usage per case. Against the original agent on Sonnet 4.6, that is 2.25 times as many send-ready cases at 88% lower model cost. Both the instructions and the model changed.
Model trade-offs
send-ready rate · recorded model usage · 300 synthetic cases
Recorded model usage per 1,000 evaluated cases · USD
V6 · Sonnet 4.692/300 ready$0.0992/case42.4 s median
Show the numbers
| Configuration | Ready / 300 | Score / 100 | $/case |
|---|---|---|---|
| Original · Sonnet 4.6 | 48 | 43.3 | $0.0862 |
| V6 · Sonnet 4.6 | 92 | 57.3 | $0.0992 |
| V6 · Sonnet 5.5 | 94 | 61.7 | $0.0476 |
| V6 · GLM-5.3 Flash | 108 | 63.5 | $0.0103 |
Same model as the original. A separate $3.02 of usage we could not account for is left out of the cost.
On the same model, V6 cost about 15% more in recorded usage than the original. Its Sonnet 4.6 run also has $3.02 of usage we could not account for. We list it separately; it is not confirmed billing. Between GLM and Sonnet 5.5 we found no clear difference in mean score or send-ready rate at 95% confidence.