Evaluate: scorers, metrics and the objective

Scorers

A scorer compares the terminal prompt's output with the row's label and returns a number. Pick the output field to compare, or leave it on whole output.

Scorer	Compares	Use when
Exact match	the field equals the label, optionally normalizing case, punctuation and articles	classification into known categories
Contains	the label appears somewhere in the output	the answer is embedded in prose
Token F1	word overlap with the label	short free-text answers, QA-style
Regex	a pattern extracts the answer, or just must be present	structured strings: ids, amounts, dates
JSON field equality	one field of a JSON label	the label is an object and only part of it matters
Numeric	numbers within a relative tolerance	quantities, where 4.99 and 5.0 are the same answer
LLM judge, vs reference	a judge model grades the output against the label	free text with a right answer but many wordings
LLM judge, no reference	a judge model grades against a rubric alone	there is no single right answer — tone, helpfulness

You can add more than one. Each produces its own named metric, and they all land in the same vector.

A judge costs money and has opinions. Every judged row is an extra model call on every evaluation of every candidate, and the judge's bias becomes the thing you optimize. Where an exact-match scorer will do, it is cheaper, faster and more honest.

The metric vector

A run does not reduce a row to one number. Alongside your scorers it records how the program behaved:

template_tokens — the prompts themselves, summed over prompts. This is the recurring bill.
prompt_tokens and output_tokens — rendered with the data, and what came back.
steps — how many prompt calls the row took, which matters when there are loops.
parse_fail.* — per prompt, how often structured output could not be parsed.
step.<id>.* — an external step's latency, cost, cache hits and failures.

Nothing is collapsed inside a scorer. The scorers report; the objective decides.

The objective

The objective is a set of weights over those metrics. accuracy: 1.0 alone means what it says. Adding a negative weight on template_tokens makes the search pay for length:

accuracy: 1.0 template_tokens: -0.002

At that weight, a prompt may spend ten extra tokens if they buy two points of accuracy, and not otherwise. Choosing the Compress goal in the optimizer sets this for you and asks the reflection model for shorter prompts that keep the same answers; it is the same machinery with a different weight and a different rewriting instruction.

The run also reports a Pareto front: the candidates that are not beaten on both accuracy and length at once. Reading the front is usually more useful than reading the winner — a prompt one point behind the best and half the length is frequently the one to ship, and only the front shows you it exists.

Pre-program step cost is reported, not traded off. Every candidate sees the same enriched rows, so a step's fetch cost is the same constant for all of them and shifts every score equally. What is optimizable is which of the step's fields are worth pasting into a prompt — pulling five fields and using two is a finding the compression objective will make for you.