PRODUCTION SCENARIO
Sable Freight's quoting service sends a 15,000-token tariff document with every request and hits its tokens-per-minute limit with 429 errors at peak. Cost and latency are acceptable, and the team plans to rely on prompt caching to get more requests through.
Which approach raises the number of requests the service can make per minute?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: Retrieve only the relevant tariff sections from a vector store and send those
Prompt caching is on by default and lowers cost and input-processing latency, but cached input tokens still count toward tokens-per-minute limits and caching does not change how rate limits are calculated. Only sending fewer tokens per request, such as retrieving the relevant sections instead of the whole document, increases throughput within the limit.
Official Source: https://developers.openai.com/api/docs/guides/prompt-caching ↗