Datasets, labels and the hold-out

A row

Upload CSV or JSONL. Each row is a flat set of columns: the inputs your placeholders need, and one column holding the right answer. For ticket triage that is message and queue. Fifty rows is enough to learn something; a few hundred is comfortable.

Then two mappings: each placeholder in your prompts to a column, and the label column to whichever column holds the answer. Check data validates both, plus the things that are easy to get wrong — a placeholder no column feeds, a label column that is empty on some rows, a column mapped twice.

Label quality caps everything. If two of your rows disagree about the same question, no prompt can score above the disagreement, and the optimizer will spend your budget discovering that. It is worth reading twenty rows by hand before spending anything.

The hold-out

A fraction of the rows — 20% by default — is set aside before the search begins. The optimizer never sees it: not for scoring candidates, not for the gate, not in the failures shown to the reflection model. At the end, the prompt that won is evaluated on those rows once, and that is the number reported.

Two things follow, and they are the whole reason for the split:

The best score during a run is not a prediction. It is the maximum of many noisy evaluations, and the maximum of noise is biased upward. It will be too high, reliably.
Training up and hold-out down means overfitting, and it is common enough to be worth looking for every time. A run in our own testing gained 7.3 points on training and lost 4.2 on the hold-out; the entire training gain sat in one class going from 0/7 to 4/7, while another class stayed 0/8 throughout. The average moved. Nothing got better.

That is why the run page has a per-class table as well as a score. An aggregate that moved for one reason is not an improvement.

The pilot

Before any run that spends real money, the pilot evaluates your prompts as written on a dozen rows, twice, and tries a single rewrite. It costs a fraction of a cent and it answers three questions:

Baseline

What your current prompts score. Every later claim is relative to this. Sometimes it is already at the ceiling and the honest move is to stop.

Noise

Two identical evaluations of the same prompt, differenced. If that gap is as big as the improvement you are hoping for, you need more rows before you need more rounds.

Cost

Projected calls and dollars for the full run, from your actual row count, program shape and settings — including any external step's per-call price.

A run also carries a hard budget in dollars. It stops there, whatever round it is in.

Enrichment

If your program uses an external step, its answers are written onto the dataset as columns, once, at capture time — called freezing. Evaluation reads those frozen columns; production calls the step live. The columns are named {step_field} and can be used as placeholders like any other. Why it works that way →