Choosing an evaluation metric for rare, costly card fraud

AIF-C01 Fundamentals of AI and ML Medium Question 8 of 65

A financial services company is building an ML model to detect fraudulent credit card transactions. Fraudulent transactions are rare. Each fraudulent transaction that the model misses causes large financial losses and regulatory penalties.

Which evaluation metric should the company prioritize?

Show answer and explanation

BRecall (true positive rate)

Recall measures how well an algorithm correctly predicts all of the true positives in a dataset, making it the essential evaluation metric when the business cost of a false negative is severe, such as undetected fraudulent transactions. Precision measures the quality of positive predictions by calculating the proportion of flagged cases that are true positives, which is prioritized when the cost of false positives is high. Accuracy measures the ratio of all correct predictions to total predictions, which is misleading in imbalanced datasets because a model predicting only legitimate transactions achieves high accuracy while missing every fraudulent event. Log loss evaluates the quality of predicted probabilities against binary outcomes rather than measuring how completely positive class members are identified.

Source: docs.aws.amazon.com/sagemaker/latest/dg/autopilot-metrics-v…

Anonymous, no account needed. Your tally stays in this browser.

Next question Start 5 timed questions Browse all 65 original AIF-C01 practice questions