Precision vs Recall: Why Accuracy Misleads (With a Worked Example)

Precision vs Recall: Why Accuracy Misleads (With a Worked Example)

Precision answers: of everything the model flagged as positive, how much really was positive? Recall answers: of all the real positives, how many did the model catch? High precision means few false alarms. High recall means few misses. On imbalanced problems like fraud, churn or disease detection, accuracy can look excellent while the model is useless, which is why precision and recall matter.

Classification metrics are where many new data scientists get caught out, in interviews and in real projects. A model that's "99% accurate" can miss nearly every case you actually care about. One worked example makes the difference clear.

The Scenario: Fraud Detection

A payment company checks 10,000 transactions. 100 of them are fraudulent (1%). A model flags 150 transactions as suspicious, and 80 of those really are fraud.

The Confusion Matrix

Every classification result falls into one of four boxes:

Actually fraud Actually legitimate
Flagged as fraud True positive (TP) = 80 False positive (FP) = 70
Not flagged False negative (FN) = 20 True negative (TN) = 9,830

Check: 80 + 70 = 150 flagged. 80 + 20 = 100 actual frauds. All four add up to 10,000.

The Metrics

Accuracy = (TP + TN) ÷ total = (80 + 9,830) ÷ 10,000 = 99.1%

That sounds excellent, until you notice that a "model" which flags nothing at all would score 99.0%, because 99% of transactions are legitimate. Accuracy barely separates a useful model from doing nothing.

Precision = TP ÷ (TP + FP) = 80 ÷ 150 = 53.3% About half of the flagged transactions are real fraud. The other half are legitimate customers whose payments get blocked or reviewed.

Recall = TP ÷ (TP + FN) = 80 ÷ 100 = 80% The model catches 80% of the fraud and misses 20 cases.

F1 score combines the two (their harmonic mean): 2 × P × R ÷ (P + R) = 2 × 0.533 × 0.80 ÷ 1.333 ≈ 0.64

Now you can have a real conversation: "we catch 80% of fraud, and about half our alerts are false alarms". That's something the business can act on, which "99.1% accurate" is not.

The Trade-off: Moving the Threshold

Most classifiers output a probability, and you choose the cut-off above which a case gets flagged. Changing it moves precision and recall in opposite directions.

Threshold Flagged True fraud caught Precision Recall
Lower (flag more) 150 80 53.3% 80%
Higher (flag less) 60 50 83.3% 50%

With a higher threshold, alerts become much more trustworthy (83% are real), but half the fraud gets through. There's no free lunch. Which point is right depends on what each kind of error costs.

Which One Should You Prioritize?

Situation Costlier error Prioritize Example
Missing a positive is very costly False negative Recall Fraud, disease screening, safety defects
A false alarm is very costly False positive Precision Spam filters (losing a real email), automatic account bans
Both matter, and classes are imbalanced Both F1, or choose a threshold using business costs Churn campaigns with limited budget

The best approach is to put money on each box. If a missed fraud costs $500 on average and reviewing a false alarm costs $5 in staff time plus some customer annoyance, you can calculate the expected cost at each threshold and pick the cheapest, instead of arguing over abstract percentages.

Other Terms You'll Meet

  • Sensitivity is another name for recall, common in medicine.
  • Specificity = TN ÷ (TN + FP): how many negatives are correctly left alone. Here it's 9,830 ÷ 9,900 ≈ 99.3%.
  • Precision–recall curve: plots precision against recall across all thresholds. It's more informative than accuracy for imbalanced data.
  • ROC curve and AUC: plot recall against the false positive rate. AUC summarizes how well the model ranks positives above negatives. It's useful, but can look optimistic when positives are very rare, so check the precision–recall curve too.

Common Mistakes

  1. Reporting accuracy on imbalanced data. Always compare against the "predict the majority class" baseline.
  2. Using the default 0.5 threshold without thinking. The default has no business meaning. Choose it based on error costs and capacity (for example, how many alerts the review team can handle per day).
  3. Tuning the threshold on the test set. Pick it on validation data, then confirm on the test set, or you'll overfit the threshold itself.
  4. Forgetting that base rates change. If fraud rises from 1% to 3%, precision changes even if the model doesn't. Monitor metrics in production.

Why It Matters Beyond Data Science

Precision and recall are also a way of asking any decision process "what are we missing, and what are we wrongly flagging?" That includes marketing target lists, credit rules and even AI assistants that search documents. For a retrieval system, recall asks whether it found all the relevant documents, and precision asks how much of what it returned was actually relevant.

What to Learn Next

These metrics build on the basics of supervised learning. Supervised Learning with scikit-learn (DataCamp) has hands-on exercises on confusion matrices and thresholds. The Machine Learning Specialization explains precision, recall and skewed datasets very clearly (see our review). Browse all Machine Learning courses.

Frequently Asked Questions

What's a good precision or recall? There's no universal number. It depends on base rates and on what each error costs. Compare against a simple baseline and against the business cost of each type of error.

Can I have both high precision and high recall? Only if the model separates the classes well. Better features and better models improve both. For a given model, moving the threshold trades one for the other.

When should I use F1? When you need one number to compare models and both error types matter roughly equally. If one error is clearly more expensive, optimize for that directly instead.

Is recall the same as sensitivity? Yes. They're two names for the same calculation, TP ÷ (TP + FN).

Bottom Line

Accuracy asks "how often is the model right overall?", and on imbalanced data that question hides the failures that matter. Precision asks how trustworthy the alerts are, and recall asks how many real cases are caught. Build the confusion matrix, put a cost on each type of error, and choose the threshold that makes business sense.

Enjoyed this article?

Share it with your network

Listings related to Precision vs Recall: Why Accuracy Misleads (With a Worked Example)