Cutting input token cost on a manual sent with every question
MLA-C02
Operating, Monitoring, and Securing ML and AI Solutions
Medium
Question 14 of 30
A field service app sends a 40,000-token equipment manual to a foundation model on Amazon Bedrock, followed by a technician's short question. Technicians ask several questions about the same manual within a few minutes, and they expect answers in seconds. Input token charges make up most of the monthly bill. The model supports explicit prompt caching.
Which solution will meet these requirements?
Show answer and explanation
CPut the manual first with a cache checkpoint after it, and append each question after the checkpoint
Explicit prompt caching meets the requirements. It is built for long contexts that are reused across queries: static content goes first, a cache checkpoint marks the end of the reusable prefix, and tokens read from the cache are billed at the model's cache-read rate and need not be reprocessed. Each cache hit resets the time to live, which is 5 minutes on many models, so questions minutes apart keep hitting the cache. Batch inference runs asynchronously, so technicians would not get answers in seconds. A question placed before the manual changes the prompt prefix on every request, which causes cache misses. Splitting the manual still sends every token, and each request loses the rest of the manual's context.
Source: docs.aws.amazon.com/
Anonymous, no account needed. Your tally stays in this browser.
Practice by topic: Amazon Bedrock questions