A model can look excellent during training and still fail in real life. That is why evaluation, generalization, and sampling are core machine learning skills.
The practical question is not “Did the model memorize the training data?” It is “Will this model work on new data?”
For practical ML, evaluation is the difference between a model that looks good in an experiment and a model that can be trusted in a real workflow.
Quick Answer
Evaluate machine learning models by separating training, validation, and test data, comparing metrics across those splits, checking for overfitting, and using repeatable sampling methods so experiments can be reproduced.
A useful evaluation process should also compare the model against a simple benchmark, review errors by segment, and confirm that the test data represents the real prediction scenario.
Key Takeaways
- Low training error does not guarantee a useful model.
- Generalization means performance stays reliable on new data.
- Validation data helps tune models and detect overfitting.
- Test data should be reserved for final evaluation.
- Repeatable sampling matters when experiments need to be trusted.
- Benchmarks help decide whether a metric is actually good.
- Segment-level error review can reveal failures hidden by average metrics.
- The test set should not be used repeatedly during tuning.
Training Error Is Not Enough
A complex model can sometimes fit the training data almost perfectly. That may feel good, but it can be a warning sign.
If the model memorizes training examples, it may fail on new examples.
This is overfitting.
On the other side, a model can be too simple and miss important patterns.
This is underfitting.
The goal is a model that fits the real pattern well enough without memorizing noise.
Train, Validation, And Test Sets
| Split | Purpose | Used for |
|---|---|---|
| Training | Teach the model | Fitting model parameters |
| Validation | Tune choices | Hyperparameters, early stopping, model comparison |
| Test | Final check | Independent performance estimate |
Do not use the test set repeatedly during tuning. If you do, it becomes part of the experiment and is no longer independent.
Choosing The Right Split Strategy
The split should match the real prediction scenario.
| Scenario | Better split strategy | Why |
|---|---|---|
| General tabular data with independent rows | Random split with fixed seed or stable hash | Simple and repeatable |
| Time-based prediction | Time-based split | Tests whether the model works on future data |
| Customer-level prediction | Split by customer or account ID | Prevents the same customer appearing in train and test |
| Highly imbalanced classes | Stratified split | Keeps class proportions similar across splits |
| Small dataset | Cross-validation | Gives a more stable estimate |
Bad splits can make a model look better than it really is. For example, if support tickets from the same customer appear in both training and test data, the model may learn customer-specific patterns instead of general behavior.
What Generalization Looks Like
A model generalizes when performance on new data is similar to performance during training and validation.
Warning signs:
- training error is low but validation error is high,
- validation error starts increasing while training error decreases,
- test performance is much worse than validation performance,
- the model performs well only on a narrow subset of examples.
Choosing Evaluation Metrics
The metric should match the decision the model supports.
| Problem type | Useful metrics | Practical note |
|---|---|---|
| Regression | MAE, RMSE, R-squared | RMSE penalizes large errors more strongly |
| Binary classification | Precision, recall, F1, AUC | Accuracy can mislead when classes are imbalanced |
| Multiclass classification | Macro F1, weighted F1, confusion matrix | Review which classes are confused |
| Ranking or scoring | AUC, lift, precision at top K | Useful when only top results are acted on |
| Forecasting | MAE, RMSE, MAPE | Choose based on business tolerance for forecast error |
Do not choose a metric only because it is common. A fraud model may care more about recall. A support routing model may care about precision for high-priority tickets. A demand forecast may care about large misses more than small errors.
Early Stopping
Early stopping means stopping training when validation performance stops improving.
During training:
- training loss usually decreases,
- validation loss should also decrease at first,
- if validation loss rises while training loss keeps falling, overfitting may have started.
Early stopping helps preserve the last model state before overfitting becomes worse.
Regularization
Regularization discourages overly complex models.
Common ideas:
- L1 regularization can encourage sparsity,
- L2 regularization can keep weights smaller,
- dropout can reduce overdependence in neural networks,
- simpler models can sometimes generalize better.
The best regularization choice depends on the model and validation results.
Benchmarks
A metric is only meaningful when compared with something.
Benchmarks can be:
- a simple rule,
- historical average,
- median prediction,
- previous production model,
- human baseline,
- business threshold.
For example, an RMSE of 3 may be good or bad depending on whether a simple rule gets RMSE 8 or RMSE 2.5.
Error Review By Segment
Average metrics can hide important failures. After checking overall performance, review errors by meaningful groups.
Examples:
- customer type,
- product category,
- geography,
- support channel,
- transaction size,
- language,
- time period,
- new vs returning users.
| Segment review question | Why it matters |
|---|---|
| Does the model fail for one customer group? | Average metrics may hide uneven performance |
| Are errors worse for rare categories? | The model may not have enough examples |
| Does performance change over time? | Data drift or seasonality may be present |
| Are high-value cases predicted poorly? | Business impact may be larger than the metric suggests |
| Are false positives and false negatives acceptable? | Different errors can have different costs |
This review is especially important when predictions affect prioritization, customer handling, approvals, or risk decisions.
Repeatable Sampling
Random sampling is easy, but naive random sampling can make experiments hard to reproduce.
If a query uses a random function, the selected rows may change each time. That makes it harder to compare experiments fairly.
For repeatable sampling in large datasets, use a stable key and a deterministic split.
Example pattern in BigQuery:
WHERE MOD(ABS(FARM_FINGERPRINT(stable_id)), 10) < 8
This gives a repeatable split based on a stable field.
Choosing A Split Field
Choose a split field carefully.
Good split fields are:
- stable,
- available in every row,
- not the target label,
- not a feature you must use for training if splitting would remove it,
- aligned with the real prediction scenario.
Bad splitting choices can create leakage or biased evaluation.
Real-World Example
Imagine a team building a churn model. The first model shows strong accuracy, but churn is rare. Most customers do not churn, so a model can look accurate by predicting “no churn” most of the time.
The team reviews precision, recall, and F1 instead of accuracy alone. It also compares the model against a simple business rule: customers with falling usage and multiple support tickets. The model beats the simple rule overall, but segment review shows weaker recall for smaller customers.
Then the team checks the split. If the same customer appears in both training and test rows across different months, the evaluation may be too optimistic. A better split may hold out customers or use a time-based split depending on the prediction workflow.
This example shows why evaluation is not just a number. The team needs to know whether the model works on future or unseen examples, whether the metric matches the decision, and whether important groups are being missed.
Cross-Validation
Cross-validation repeats the train/validation process across multiple splits. It is useful when datasets are smaller or when you want a more stable estimate.
The model trains and validates several times, then you review the average and spread of metrics.
Practical Evaluation Checklist
Before trusting a model:
- Confirm the label is correct.
- Split data into train, validation, and test.
- Choose a split strategy that matches the real prediction scenario.
- Train a baseline model.
- Choose metrics that match the business decision.
- Compare training and validation metrics.
- Watch for overfitting or underfitting.
- Tune using validation data.
- Evaluate once on test data.
- Compare with a benchmark.
- Review false positives and false negatives.
- Review errors by segment.
- Document the result and known limitations.
- Decide whether the model is useful for the business decision.
Common Evaluation Mistakes
- using the test set repeatedly during tuning
- choosing accuracy for an imbalanced classification problem
- not comparing against a simple benchmark
- using a random split when a time-based split is needed
- leaking the same customer or account into both train and test data
- reporting only one average metric
- ignoring false positives and false negatives
- not reviewing errors with business users
- treating a good notebook score as production approval
Official Resources
- Google Machine Learning Crash Course: Training and test sets
- Scikit-learn cross-validation guide
- BigQuery FARM_FINGERPRINT function
- Vertex AI model evaluation documentation
Related AI Charcha Reading
- Launching Into Machine Learning: A Practical Learning Path
- Data Quality And EDA For Machine Learning
- Vertex AI AutoML Regression Guide
- BigQuery ML Beginner Guide
- Launching Into Machine Learning: A Practical Learning Path
- Data Preprocessing Options for Enterprise ML
- Enterprise Machine Learning Workflow Guide
FAQ
What is overfitting?
Overfitting happens when a model learns the training data too closely and performs worse on new data.
Why not train on all available data?
If you train on all data, you lose an independent way to estimate how the model performs on unseen examples. Cross-validation can help when data is limited.
What makes a sampling method repeatable?
A sampling method is repeatable when the same rows go into the same split every time the experiment is run.
Bottom Line
Good evaluation protects you from false confidence. A useful model is not the one with the lowest training loss. It is the one that performs reliably on new data and beats a meaningful benchmark.
Evaluation should leave evidence. A team should be able to explain how the data was split, which metric was used, what benchmark was beaten, where the model fails, and why the model is or is not ready for real use.