My classification model has 95% accuracy but is useless — why?

Question

I'm predicting fraudulent transactions. The model reaches 95% accuracy on the test set but catches almost no fraud. What's going on?

Accepted Answer

Classic **class imbalance**. If 95% of transactions are legitimate, a model that always predicts "not fraud" gets 95% accuracy while catching zero fraud. Accuracy is the wrong metric here.

1. **Use the right metrics**: precision, recall and F1 **for the fraud class**, the confusion matrix, and **PR-AUC** (more informative than ROC-AUC for rare classes).

```python
from sklearn.metrics import classification_report
print(classification_report(y_test, y_pred, digits=3))
```

2. **Handle the imbalance**: `class_weight='balanced'` (or `scale_pos_weight` in XGBoost/LightGBM), or resampling on the **training set only**.
3. **Tune the decision threshold**. The default 0.5 is arbitrary. Pick the threshold from the precision-recall curve based on business cost: how expensive is a missed fraud versus a false alarm that blocks a customer?
4. Use a **stratified** split, and for fraud a **time-based** split, so you don't train on the future.

1 Answer