PRODUCTION SCENARIO
A payments company trains a fraud classifier on 2 million transactions. Fraud accounts for 0.4% of those transactions. A candidate model reports 99.6% accuracy, but analysts notice that it flags almost nothing as fraud.
Which metric should the company use to compare candidate models on this dataset?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: Balanced accuracy
Balanced accuracy normalizes true positives and true negatives by the size of each class, so a model that flags almost no fraud scores poorly instead of 99.6 percent. Accuracy is dominated by the majority class and rewards the very model the analysts distrust. Macro-averaged precision stays high for a model that flags almost nothing, because it measures how often a flag is correct rather than how much fraud is caught, and AWS scopes it to multiclass problems. RMSE is a regression metric and does not apply to a classifier.