PRODUCTION SCENARIO
A Generative AI Engineer must serve a support assistant at 40 requests per second inside a two-second budget. Scored on 900 held-out tickets, an 8B model reached 0.81 with a median latency of 0.4 seconds and a 70B model reached 0.86 at 2.6 seconds.
Which model should the engineer serve?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: The 8B model, whose median latency sits inside the two-second budget
The two-second budget is a requirement, not a preference, and a 2.6-second median means the 70B model misses it on more than half of requests, so its higher score is unusable. Reserving the 8B model for peak inverts the problem, since peak is when latency is worst and the slow model is serving everything else. Fine-tuning on the 900 held-out tickets consumes the only evaluation set the team has, leaving no honest way to measure whether the gap closed.
Official Source: https://docs.databricks.com/aws/en/mlflow3/genai/eval-monitor/ ↗