> For the complete documentation index, see [llms.txt](https://docs.promptshuttle.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.promptshuttle.com/platform/run-verification.md).

# Run Verification

Run verification answers a question the status field cannot: **did this run actually get anywhere?**

A run records `Succeeded` when we returned an answer. It says nothing about whether the answer was any good, and for an agent that called twelve tools over four minutes that is usually the only thing you wanted to know. Verification has a small, cheap model read the trajectory and grade it — while the run is still executing.

It is **off by default and opt-in in two places**, because each check is an extra inference billed to you.

## What you get

Two numbers, and they answer different questions.

|               |       |                                                   |
| ------------- | ----- | ------------------------------------------------- |
| **Progress**  | 0 – 1 | How complete the verifier judges the run to be    |
| **Certainty** | 0 – 1 | How concentrated the verifier's own judgement was |

Read them together. A run at 0.5 with a certainty of 0.99 is *confidently half-finished*. The same 0.5 at a certainty of 0.2 means the verifier **could not tell** — which is itself worth knowing, and is invisible if you only look at the grade.

Alongside those, a run records the **lowest** score it ever reached and **how many readings** were taken. A run that dipped to 0.1 and recovered to 0.9 looks identical to one that sailed through, if you only keep the last value.

> **An absent score is not a zero.** Zero is a real verdict: the verifier judging that the run achieved nothing. A run that was never verified has no score at all. Never treat one as the other.

## Turning it on

Both switches must be on.

1. **Tenant** — *Tenant Profile → Progress verification*. This is a master switch. It enables nothing by itself; turning it off stops everything at once.
2. **Flow template** — *Verify progress*, in the template editor. Versioned with the template, like the model and the reasoning effort, so a change is diffable and revertible.

### The criterion

Give the verifier one sentence saying what counts as done:

> Did the agent confirm the fix by running a test and reporting its output?

This is worth writing. The generic fallback only asks whether the stated task appears complete, and a criterion written for the prompt grades measurably better. It lives on the template, because what counts as progress is a property of the prompt, not of whoever calls it.

## From the OpenAI-compatible endpoint

Callers on `/api/v1/chat/completions` have no template to configure, so they use a header:

```bash
curl https://api.promptshuttle.com/api/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -H "X-Shuttle-Verify: Did the agent confirm the fix by running a test?" \
  -d '{"model":"claude-sonnet-4-6","messages":[...]}'
```

`X-Shuttle-Verify: true` uses the generic criterion. `false`, `no` and `0` mean off. Any other value is read as the criterion itself.

The result comes back as response headers, since the OpenAI body schema has no room for it:

```
X-PromptShuttle-Progress: 0.170
X-PromptShuttle-Progress-Certainty: 0.662
X-PromptShuttle-Verifier: gpt-4o-mini
```

The tenant switch still applies. A header cannot start billing verifier calls against a tenant that has the feature turned off.

## Where the scores show up

* **Invocation log** — a percentage badge beside the status badge, on the list and the run detail. A low-certainty reading is drawn greyed out, because it should not look authoritative.
* **Sub-agent tree** — each child carries its own score. When a tree scores badly it is usually one sub-agent doing so, and that is invisible from a root-level number.
* **API** — `progressScore`, `progressCertainty`, `minProgressScore`, `progressScoreCount` and `verifierModel` on the log entry; `progressScore` / `progressCertainty` on the flow-run response.
* **MCP** — `Run.Progress` on `get_run`.

## Warnings

A run that scores low on two consecutive checks records a `ProgressStalled` [warning](/platform/analytics.md). It means the model has been working without visibly getting closer to the goal — the signature of a loop that will run to its iteration ceiling and bill the whole way there.

Two consecutive checks, not one, on purpose. A single low score is a normal mid-run state: a run dips whenever it discovers its first approach was wrong, which is a run working correctly.

**Verification never stops a run.** It observes and reports. Stopping a run is [cancellation](/platform/analytics.md#stopping-a-run), which is yours to trigger.

## What it costs

About **$0.00006 per check** with the default verifier on a short trajectory, and a run is checked a handful of times. Cost scales with how long the trajectory is, not with how expensive the model doing the work is — each check re-reads the transcript so far.

Verification cost is **absorbed, not billed onward**: it is overhead you chose, not work your own customer asked for, so it is recorded separately and does not inflate the per-run and per-customer cost figures you bill from.

## Choosing a verifier

Leave it empty unless you have a reason. If you set one, it must be a model that:

* **returns token-level logprobs** — the grade distribution *is* the signal, and a model that only returns the sampled letter turns a twenty-point scale into a twenty-way coin flip; and
* **does not reason before answering** — a thinking model settles its uncertainty inside the trace and then states its conclusion at near-certainty whatever the evidence was, so its certainty number becomes a constant.

Being capable is not the same as being calibrated: we measured one cheap model that produces perfectly graded distributions and ranks a verified success *below* an unsupported claim.

> **Scores are not comparable across verifier models.** Three models we measured graded the same trajectory 0.015, 0.105 and 0.256. Changing the verifier makes new scores incomparable with old ones, which is why every score records the model that produced it.