PRODUCTION SCENARIO
A Generative AI Engineer finds that a contractor's service principal runs a nightly batch job through a shared pay-per-token endpoint and crowds out interactive internal traffic during business hours. The contractor's job must keep running, and internal teams must keep their current throughput.
What should the engineer configure?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: A custom queries-per-minute limit for that contractor service principal
AI Gateway rate limits come in two forms: a User (Default) limit that applies to all users, and custom limits for individual users or service principals, in queries or tokens per minute. A custom limit on the contractor's principal throttles one caller while everyone else keeps their throughput. Lowering the default throttles the internal teams too, a spend budget caps cost without slowing the request rate that causes the crowding, and a second endpoint has no rate control of its own unless one is configured on it.