The same compiled program runs in evaluation and in production. That is the invariant the whole serving path exists to protect.
Publishing a node from a run creates a version: an immutable snapshot of the prompts, the model each one runs on, the external steps pinned to the exact manifest they were scored against, the evaluation settings, and the score it earned. Versions are never edited. Changing anything means publishing another one.
Each version carries a fingerprint — a hash over the behavioural fields only, so moving a node on the canvas does not produce a “new” version while repointing a step does. Two versions with the same fingerprint will behave identically; that is what makes the list trustworthy enough to roll back from.
A published version has an endpoint. Create an API key for it and send the inputs the program needs:
curl -X POST https://impromptune.com/api/v/<version>/run \
-H "Authorization: Bearer <your api key>" \
-H "Content-Type: application/json" \
-d '{"message": "my package never arrived"}'Which inputs are required is derived from the program, not declared by hand: the placeholders every prompt needs, minus the ones an earlier prompt fills, minus the ones an external step provides, plus whatever the step itself needs from the row. Ask for the wrong thing and the error names the field.
We store only a hash of an API key. If you lose it, you make a new one.
Every served request is recorded: the inputs, which version answered, what each prompt produced, what any external step was asked and what it returned, the tokens and the latency. A step's inputs and outputs are on the trace deliberately — live data moves, and six weeks later the trace is the only remaining evidence of what the model was actually looking at.
A trace says what the program answered. An outcome says what happened next: an agent reassigned the ticket, a customer replied, a reviewer corrected it. A trace plus an outcome is a label.
Through the API, or by hand in the traces list. The latest correction on a trace is the one that counts.
Turn labelled traces into a dataset: real inputs, real answers, each row carrying the trace it came from, the version that produced it, when it was captured and where the label came from.
Optimize against the traffic you actually get rather than the fifty examples you wrote at the start. This is the only part of the product that gets better while you are not looking.