PRODUCTION SCENARIO
A fraud team can investigate 200 alerts per day. At the current threshold the model raises 140 alerts and catches 62 percent of fraud. Lowering the threshold to 0.30 raises 900 alerts and catches 88 percent.
How should the ML engineer choose the operating threshold?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: The threshold with the highest recall whose volume fits the review capacity
Recall counts only if an analyst opens the alert, so the operating point is the highest-recall threshold whose daily volume fits the review budget. Maximising F1 and taking the point nearest the corner both optimise the shape of the curve while ignoring how many alerts a day that produces. Fixing recall at 80 percent constrains the wrong side of the tradeoff and still overruns a 200-alert queue.