Large-model accuracy at small-model speed for claim triage
MLA-C02ML Model and Foundation Model (FM) DevelopmentMediumQuestion 11 of 30
An insurance company classifies incoming claim descriptions with a large foundation model on Amazon Bedrock. Accuracy meets the target, but latency and inference cost per request are too high for the planned volume. A smaller model in the same family is fast enough, but its accuracy with the same prompt is 11 points lower. Model invocation logging has captured 6 months of the large model's prompts and responses. The company has no budget for manual labeling.
Which solution will meet these requirements?
Show answer and explanation
DRun an Amazon Bedrock Model Distillation job with the large model as teacher and the invocation logs
Model distillation meets the requirements. It transfers knowledge from a larger teacher model to a smaller, faster and cheaper student, and it can build its training data from the invocation logs the company already has, so no one labels anything by hand. Provisioned Throughput buys more capacity for the same large model at an hourly price; the model does not get smaller, so it does not gain the smaller model's speed. Supervised fine-tuning on adjuster-labeled claims needs the manual labeling the company cannot pay for. Routing to the large model when the small model is wrong assumes the application can tell a wrong answer at request time, which it cannot without a label.