Train, Validation and Test Sets (and Cross-Validation) Explained
Train, Validation and Test Sets (and Cross-Validation) Explained
Machine learning data is split so that a model is judged on data it has never seen. The training set is used to fit the model, the validation set to compare models and tune settings, and the test set only once, at the end, to estimate real-world performance. Cross-validation rotates the validation role across several folds to get a more reliable estimate from limited data.
A model that scores 99% on the data it learned from tells you almost nothing. It may simply have memorized that data, which is overfitting. The whole point of splitting data is to simulate the future: the model will meet new customers, new transactions and new images, so it must be tested on examples it hasn't seen.
Why It Matters in a Real Job
If you evaluate a model on its training data, you will ship models that look excellent in a notebook and disappoint in production. Proper splitting is how a data scientist can honestly say, "We expect around 82% accuracy on new customers," and be roughly right. It's also one of the first things a reviewer checks in any ML project, interview take-home or ML engineering code review.
The Three Sets
Set
Purpose
Used when
Typical share
Training
Fit the model's parameters
Every time a model is trained
60–80%
Validation
Compare models, tune hyperparameters, choose a threshold
Repeatedly during development
10–20%
Test
Final, unbiased estimate of performance
Once, at the very end
10–20%
Think of it like exam preparation. Training data is the textbook. Validation data is the practice exams you use to adjust how you study. The test set is the real exam: if you peek at it while studying, your score no longer measures what you know.
A Worked Example
A bank has 10,000 historical loan applications and wants to predict default.
Step 1: Split once, up front. Using a 70/15/15 split:
7,000 rows for training
1,500 rows for validation
1,500 rows for testing
Because defaults are rare (say 8% of loans), use a stratified split so each set keeps roughly 8% defaults. Otherwise one set could end up with very few positive cases.
Step 2: Develop using training and validation. Train three candidate models on the 7,000 rows and score them on the 1,500 validation rows:
Model
Validation recall on defaults
Logistic regression
0.62
Random forest
0.71
Gradient boosting
0.74
You tune the gradient boosting model's settings and choose a decision threshold, still only looking at validation results. Recall is the right lens here because missing a default is costly; see Precision vs Recall.
Step 3: Test once. Retrain the chosen model, typically on training plus validation data, and score it on the 1,500 untouched test rows. If test recall is 0.72, that is your honest estimate. If it is far below validation (say 0.55), you have probably overfitted to the validation set through too much tuning.
Cross-Validation
With a small dataset, a single validation split is noisy: a lucky or unlucky 1,500 rows can swing the result. k-fold cross-validation solves this.
Set aside the test set first.
Split the remaining data into k equal folds (often 5 or 10).
Train on k − 1 folds and validate on the remaining fold.
Repeat k times so every fold serves as validation once.
Average the k scores and look at their spread.
Example. Five-fold cross-validation gives accuracy scores of 0.81, 0.79, 0.83, 0.80 and 0.82. The mean is 0.81 and the standard deviation is about 0.016. A small spread means the estimate is stable. A spread of ±0.10 would warn you that performance depends heavily on which data the model sees.
Single validation split
k-fold cross-validation
Compute cost
Low: train once per setting
k times higher
Reliability
Noisy on small data
More stable estimate
Best for
Large datasets, deep learning
Small to medium tabular data
Variants worth knowing:
Stratified k-fold keeps the class balance in every fold, which is essential for imbalanced classification.
Group k-fold keeps all rows from the same customer or patient in one fold, so the model can't "recognize" a person.
Time series split always trains on the past and validates on the future. Never shuffle time-ordered data; see Time Series Forecasting Explained.
Common Mistakes
Data leakage from preprocessing. Scaling, imputing or selecting features on the full dataset before splitting lets information from the test set leak in. Fit preprocessing on training data only, ideally inside a scikit-learn Pipeline.
Tuning on the test set. Every time you check the test score and adjust, the test set becomes another validation set and your final number becomes optimistic.
Shuffling time series. Predicting March using a model trained on April data is impossible in real life, so it produces fake accuracy.
Duplicates across splits. The same customer or near-identical rows in both train and test inflate scores. Deduplicate first; see the Data Cleaning Checklist.
Ignoring class imbalance. A random split can leave a rare class almost absent from validation. Stratify.
Reporting only the mean. Always report the spread of cross-validation scores as well.
Where It Shows Up in Interviews
Expect questions such as: "Why do we need a separate test set if we already have a validation set?", "What is data leakage? Give an example." and "How would you validate a forecasting model?" Good answers mention tuning bias, preprocessing leakage and time-based splits. Take-home assignments are also routinely judged on whether the candidate split the data correctly before doing anything else.
What is a good train/test split ratio?
70/15/15 or 80/10/10 are common starting points. With very large datasets you can give a smaller share to validation and test, because even 1% of ten million rows is plenty.
Do I always need a validation set?
You need some way to choose between models without touching the test set. That can be a separate validation set or cross-validation on the training data.
What value of k should I use?
Five or ten folds are standard. More folds use more data for training each time but cost more compute.
Is cross-validation used in deep learning?
Less often. Deep learning datasets are usually large and training is expensive, so a single train/validation/test split is the norm.
Bottom Line
Train on one part of the data, make decisions with a second and judge the final model on a third that you touch only once. Use cross-validation when data is limited, stratify when classes are imbalanced, split by time for time series and keep preprocessing inside the training folds. Get this right and your reported accuracy will mean something.
