Compute False Positive Rate, Precision and Recall by Hand from a Fraud Confusion Matrix

A team has built a model that flags orders as likely fraudulent so they can be held for manual review. A slide shows its results on 1,000 orders from a holdout period whose true outcome is known, and you are asked to work out the model's metrics by hand, with only a calculator. The report says the slide showed a simple confusion matrix and asked for false positives and similar quantities to be computed by hand. It gives neither the scenario nor the counts, so both are illustrative here. | | Predicted fraud | Predicted legitimate | Total | | --- | --- | --- | --- | | **Actually fraud** | 40 | 10 | 50 | | **Actually legitimate** | 60 | 890 | 950 | | **Total** | 100 | 900 | 1,000 | Rows are the true outcome and columns are the model's prediction. The positive class is fraud. ### Clarifying Questions - Does "false positive" on the slide mean the number of false positives or the false positive rate? - Were these orders held out from training, and are their fraud labels final, or can chargebacks still arrive? - What happens to a flagged order, and what does a missed fraudulent order cost compared with wrongly holding a legitimate one? ### Part 1 — Read the matrix Name the four cells (true positives, false positives, false negatives and true negatives) and give each count. ```hint Truth first Before naming any cell, say which axis holds the truth and which holds the prediction; a false positive is wrong in one specific direction. ``` #### What This Part Should Cover - The orientation of the matrix - The four counts, each labeled correctly ### Part 2 — Compute the metrics Compute each of the following as a fraction and as a percentage to one decimal place: the false positive rate, the false negative rate, recall (sensitivity), specificity, precision and accuracy. Also give the F1 score. ```hint Name the denominator Every rate here is one cell divided by a row total or a column total. Decide which total before dividing. ``` #### What This Part Should Cover - The correct denominator for each metric - Exact values for every metric - The difference between the false positive rate and the share of flagged orders that are legitimate ### Part 3 — Interpret the result A colleague sums up the slide by saying the model is 93% accurate. Is it a good model? Explain what these numbers mean for the review team and for customers, and what else you would need to know. ```hint A rule that does nothing Before judging the headline number, work out how the simplest possible rule would score on the same 1,000 orders. ``` #### What This Part Should Cover - Why accuracy misleads at this class balance - What precision and recall mean for the review queue and for fraud that gets through - The information that would decide whether the model is good enough ### What a Strong Answer Covers - Correct cell labels and exact arithmetic, with each denominator stated - A clear distinction between the false positive rate and the false discovery rate - A comparison with the base rate and with a trivial baseline - Precision and recall translated into review workload and missed fraud - Recognition that the right operating point depends on the relative costs of the two errors, and that the counts carry sampling uncertainty ### Follow-up Questions - If fraud fell to 1% of orders while recall and the false positive rate stayed the same, what would precision be? - If the team lowers the model's score threshold, which cells grow and which shrink, and which metric does not move in a fixed direction? - Suppose a missed fraudulent order costs much more than reviewing a legitimate one. How would you use matrices like this one to choose between two thresholds?