Back to Research

The Spinrun team · October 3, 2026

Designing a fair model comparison for agent work

A rubric for comparing models on the work your team needs, without inventing benchmark results.

Start with a task set

Select tasks that represent the actual workflows your team wants to delegate. Include known successful cases, missing-data cases, and a case that must stop for approval. Freeze the source material so each model sees the same information.

Keep the harness constant

Use the same tool schemas, access rules, runtime behavior, and output requirements. Record model configuration and any retries. A change in available context or permission can dominate a model difference and make an apparently precise score misleading.

Score observable outcomes

A reviewer should check correctness, source attribution, appropriate actions, and explicit handling of uncertainty. Add a criterion for stopping at the required review boundary. Measure preparation, review, and corrections together rather than timing only the first response.

Record cost and variance

Run enough representative examples to observe inconsistency. Record input size, tool activity, latency, and billed usage under the current plan. Report the spread of results and the failure cases alongside any average.

Report only what was tested

This page provides a method, not a published Spinrun score for GPT-5.6 or Claude. A responsible comparison names the task set, configuration, dates, and limitations. Check current model availability in your workspace before applying the method.

Before anything loads

We would like to switch on analytics, so we can see which pages are worth keeping. Nothing has loaded yet and nothing will until you choose. The cookies that keep you signed in and remember your language are not part of this.

Read the Cookie Policy