Train/validation/test split: why do I need three sets?

Question

I split my data into train and test, tune hyperparameters until test accuracy is best, and report that number. My mentor says the reported number is too optimistic. Why?

Accepted Answer

Because you **used the test set to make decisions**. Each time you pick the hyperparameters that score best on test, a little information about the test set leaks into your model. After many trials, the test score is biased upward: you've overfit to the test set.

The fix:
- **Train**: fit model parameters.
- **Validation**: compare models and tune hyperparameters (or use **cross-validation** on the training data).
- **Test**: touch it **once**, at the very end, to estimate real-world performance.

```python
from sklearn.model_selection import train_test_split, GridSearchCV

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
search = GridSearchCV(model, params, cv=5).fit(X_train, y_train)  # tuning uses CV on train only
print(search.score(X_test, y_test))                                # final, honest estimate
```

For time-dependent data, split **by time** (train on the past, test on the future).

1 Answer