PRODUCTION SCENARIO
A payments company trains a fraud classifier on 2 million transactions. Fraud accounts for 0.4% of those transactions. A candidate model reports 99.6% accuracy, but analysts notice that it flags almost nothing as fraud.
Which metric should the company use to compare candidate models on this dataset?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: Balanced accuracy
Accuracy is dominated by the majority class, so a model that predicts no fraud at all still scores 99.6%. Balanced accuracy normalizes true positives and true negatives by the size of each class, which exposes a model that misses the rare class. Mean squared error and R2 are regression metrics and do not apply to a classifier.