PRODUCTION SCENARIO
A law firm's assistant lets a user upload a 60-page contract and then ask a series of questions about it. Every question resends the full contract to the model. Each answer takes several seconds and consumes thousands of input tokens.
Which approach will reduce both the latency and the input cost of the follow-up questions?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: Prompt caching
Prompt caching keeps a static prompt prefix, such as an uploaded document, available across requests. Cached tokens are not reprocessed, which cuts latency, and are billed at the model's lower cache-read rate. A larger context window does not help: the contract already fits and the cost is re-reading it. Cross-Region inference changes where the request runs, not how many tokens are processed. Batch inference is asynchronous while a lawyer waits, and AWS does not support caching with the batch API.