Cobalt Industries: The new version was better on speed and worse on the two task types that mattered most to those users.

The problem

A rollout of a new agent version stalled after early users said it felt worse.

What we did

Cohort comparison held the task mix constant across old and new versions on live traffic.

The result

The new version was better on speed and worse on the two task types that mattered most to those users.

Perception was right, but not for the reason anyone guessed.