PRODUCTION SCENARIO
A Generative AI Engineer's agent endpoint answers inside its 900 ms target at 5 queries per second. At a 40 queries-per-second peak, requests queue while answer quality stays unchanged.
How should the engineer size the endpoint?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: Load test the endpoint, then set concurrency to target QPS times average latency
Databricks sizes serving capacity from measured behaviour: load test to measure actual latency, then required concurrency equals target QPS times average latency in seconds, which here is 40 times 0.9. A lower rate limit empties the queue by rejecting traffic rather than serving it, a larger workload type attacks per-request latency that already meets its target, and an offline evaluation run has no concurrent load, so its latency says nothing about queueing.