PRODUCTION SCENARIO
A team pairs a lower-cost executor with a frontier advisor to save money on chart-reading tasks. Shadow runs show the executor calls the advisor on nearly every task, and the bill is now higher than the frontier model alone.
What should the team do?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: Run the advisor's model alone, since consulting on most tasks removes the savings
The multi-model guidance reports this outcome in its chart-reading benchmark, where the pairing scored about the same as the advisor's model alone but cost roughly 2.6 times as much per task because the advisor was consulted on nearly every task. Its advice is to measure the consult rate and run the advisor's model itself when the executor asks on most tasks.