Start with one comparable batch
Keep the comparison standard consistent. A faster result under an easier or different standard is not a speed advantage for the same job.
Keep the route visible. Endpoint queues, regional network conditions, and service tiers are part of the observed configuration even when the underlying model name matches.
Use successful elapsed time, not a misleading average
A request that failed quickly did not finish the work. Separate hard failures and retries from successful completion time before comparing speed.
Time to first token is useful for interactive feel, but a coding task also depends on when the complete answer and terminal event arrive. Use the timing measure that matches the decision you are making.
Missing cost is unknown, not zero
Reference cost requires token usage and a matching pricing snapshot. Custom plans, subscriptions, cached input, or incomplete usage fields can leave cost uncovered. Do not rank an uncovered configuration as the cheapest.
- Quality mode: choose the highest measured quality that clears your time and budget limits.
- Balanced mode: choose the fastest or least costly configuration inside a narrow quality band.
- Worker mode: use a task-specific quality floor rather than a site-wide average.
Keep reference cost separate from your invoice
A public reference cost estimates the measured API usage under its recorded price snapshot. Your invoice may use a subscription, token plan, cache discount, regional price, or third-party markup. Use the public value for like-for-like comparison, then verify the commercial route you actually pay for.