Completion is not the metric

Daniel Weiss
Daniel Weiss
1 min read
Completion is not the metric

Every agent platform reports completions, because completions are trivial to count. The number is almost useless on its own.

The detail

What matters is what happened next: was the output used as-is, edited, discarded, or was the task re-run with a different instruction. Those four outcomes tell you more than any confidence score.

Assembly infers them from behaviour rather than asking people to report, because in practice nobody reports.

Share