Training a model is only one part of machine learning. After a model is trained and validated, it must serve predictions in a real workflow. Then the team must monitor whether the model continues to behave well after users, systems, and data start changing.
This guide explains batch prediction, online prediction, serving containers, and model monitoring in simple production-minded terms.
Quick Answer
Use batch prediction when you need many predictions at once and do not need an instant response. Use online prediction when an application needs a fast response from a deployed endpoint. Use model monitoring to detect training-serving skew, feature drift, data quality changes, unusual prediction patterns, and behavior that may reduce model trust.
The practical goal is not only to get predictions. The goal is to know when predictions are still reliable, when they need review, and when the model should be retrained, rolled back, or improved.
Key Takeaways
- Batch prediction is better for offline or scheduled prediction jobs.
- Online prediction is better for real-time applications.
- Pre-built containers can simplify prediction serving.
- Custom containers are useful when serving needs custom logic.
- Model monitoring helps detect skew and drift after launch.
- Alert thresholds should match the business risk of the model.
- Monitoring only works if someone owns the alerts and knows what action to take.
Batch Prediction
Batch prediction is used when many prediction requests can be processed together.
Examples:
- score all customers overnight,
- predict demand for next week,
- classify a large set of documents,
- update risk scores in a table,
- generate recommendations for many users.
- refresh churn scores before a customer success review.
Batch prediction is usually asynchronous. You submit a job and review results when the job finishes.
This is useful when the prediction result does not need to appear immediately inside an application. For example, a retail team may run demand predictions every night and use the results in tomorrow’s inventory planning. A bank may refresh risk segments daily before analysts review accounts. A support team may classify old tickets to find recurring product issues.
The main design questions are:
- Where does the input data come from?
- Where should prediction results be written?
- How often should the job run?
- Who reviews failed records?
- What happens if the batch job is late or incomplete?
Online Prediction
Online prediction is used when an application needs a quick response.
Examples:
- show a recommendation while a user is on a site,
- classify a support request as it arrives,
- predict fraud risk during a transaction,
- estimate delivery time during checkout.
- rank search results inside an application,
- personalize an in-product message.
Online prediction usually uses a deployed model endpoint.
The main design questions are different from batch prediction:
- What latency is acceptable?
- How much traffic should the endpoint handle?
- What happens if the endpoint is unavailable?
- Should the application use a fallback rule?
- Which inputs are safe to send at request time?
- How will errors and slow responses be monitored?
Online prediction is powerful, but it adds operational responsibility. If the endpoint is part of checkout, fraud review, customer support routing, or product recommendations, the team must think about availability, cost, scaling, and rollback.
Batch vs Online Prediction
| Question | Batch prediction | Online prediction |
|---|---|---|
| Response needed immediately? | No | Yes |
| Works well for many records? | Yes | Sometimes |
| Used by applications? | Usually indirectly | Yes |
| Common pattern | Scheduled job | API endpoint |
| Main concern | Throughput and cost | Latency and availability |
| Failure handling | Retry job or reprocess records | Fallback, timeout, or service recovery |
| Best example | Nightly customer scoring | Real-time fraud check |
How To Choose The Right Prediction Pattern
Choose batch prediction when the work can wait and the value comes from processing many records together. Choose online prediction when the user or business process needs an answer during the workflow.
| Situation | Better choice | Why |
|---|---|---|
| Daily churn scoring | Batch prediction | Results can be refreshed on a schedule |
| Real-time fraud check | Online prediction | The decision happens during the transaction |
| Weekly demand forecast | Batch prediction | Large data volume and no instant response needed |
| Support ticket routing | Online prediction | The ticket needs classification when it arrives |
| Document archive classification | Batch prediction | Many records can be processed together |
| Personalized app recommendation | Online prediction | The recommendation depends on the current session |
If the same model supports both patterns, treat them as separate production workflows. Batch scoring and online serving may use different data paths, different latency expectations, and different monitoring needs.
Serving Containers
Vertex AI can use pre-built containers or custom containers for serving.
Pre-built containers are helpful when the model format and framework are supported. They reduce setup work.
Custom containers are useful when:
- the model needs custom preprocessing,
- the serving logic is special,
- the framework is not supported by a pre-built container,
- the container must handle custom health checks or prediction routes.
- the model depends on extra packages or runtime behavior,
- the team needs tighter control over request and response handling.
The practical rule is simple: use a pre-built container when it fits cleanly. Use a custom container when the production workflow needs behavior that the pre-built option does not support.
What Model Monitoring Checks
Model monitoring helps track whether production behavior changes.
Important signals:
- training-serving skew,
- feature drift,
- prediction distribution changes,
- missing input values,
- unusual input categories,
- data quality issues,
- prediction volume changes,
- latency and error rate,
- changes in business outcomes.
Monitoring is not the same as knowing the model is correct. It is an early warning system. It helps the team notice that production inputs or outputs are changing and decide whether a deeper review is needed.
| Monitoring signal | What it can suggest | Practical review |
|---|---|---|
| Training-serving skew | Production data differs from training data | Check feature calculations and data pipelines |
| Feature drift | Input patterns changed over time | Compare recent data with training or baseline data |
| Missing values | Source system or ingestion issue | Review upstream data quality |
| New categories | Business process or user behavior changed | Decide whether the model can handle new values |
| Prediction shift | Model output pattern changed | Review examples and business impact |
| Latency increase | Endpoint or dependency problem | Check traffic, scaling, and serving logs |
| Error rate increase | Application or model serving issue | Review request format, endpoint health, and recent changes |
Alert Thresholds
Alert thresholds should not be copied blindly. They depend on the use case.
A model used for marketing recommendations may tolerate more drift than a model used for risk review or safety-sensitive decisions.
Set thresholds based on:
- business impact,
- data volatility,
- model importance,
- review capacity,
- past monitoring results,
- acceptable false alarms,
- expected seasonal behavior,
- downstream decision risk.
Too many alerts can make teams ignore monitoring. Too few alerts can hide real problems. Start with sensible thresholds, review early alerts carefully, and adjust based on the actual production workflow.
Practical Monitoring Workflow
- Define the model owner.
- Decide which features, predictions, and service metrics should be monitored.
- Capture a baseline from training, validation, or trusted production data.
- Set initial alert thresholds based on business risk.
- Capture serving inputs and prediction outputs where appropriate.
- Review alerts on a regular schedule.
- Investigate skew, drift, missing values, latency, or error changes.
- Compare alerts with real business outcomes where possible.
- Retrain, rollback, update preprocessing, or adjust thresholds when needed.
- Document the decision so future reviewers understand what changed.
This workflow matters because an alert alone does not improve the model. The improvement comes from ownership, investigation, and action.
Real-World Example
Imagine a company using a model to prioritize customer support tickets. During training, most tickets came from email and a help center form. After deployment, the company adds chat support and mobile app feedback. The new messages are shorter, less structured, and use different words than the original training data.
At first, the model still returns predictions, so the system appears healthy. But after a few weeks, monitoring shows feature drift and a change in prediction distribution. More tickets are being classified as low priority even though some of them involve account access, billing, or service outages.
In this case, the problem is not simply “the model is bad.” The workflow changed. The input channel changed. The text style changed. The business process changed. Monitoring gives the team evidence that the production environment no longer looks like the original model validation environment.
A practical response could be:
- review examples from the new chat and mobile channels
- confirm whether labels still match support priorities
- update preprocessing for shorter messages
- retrain using newer examples
- adjust escalation rules while the new model is validated
- keep human review for sensitive ticket categories
That is the real value of model monitoring. It helps the team notice when the world around the model has changed.
When To Retrain, Roll Back, Or Investigate
Not every alert means the model should be retrained immediately.
| Situation | First action | Possible next step |
|---|---|---|
| Small expected seasonal drift | Review examples | Adjust thresholds if behavior is acceptable |
| Missing input fields | Check data pipeline | Fix upstream source before retraining |
| Sudden prediction shift | Inspect recent requests | Roll back if business impact is high |
| Gradual feature drift | Compare recent and baseline data | Retrain if model quality drops |
| Higher latency | Check endpoint and traffic | Scale endpoint or optimize serving path |
| More wrong predictions | Review labeled examples | Retrain or improve features |
Retraining is useful when the model needs newer examples. Rollback is useful when a recent deployment is causing harm. Investigation is useful when the signal may come from data quality, pipeline, or application changes rather than model quality alone.
Common Mistakes
- deploying without monitoring
- ignoring input data changes
- using online prediction when batch would be simpler
- setting alert thresholds too tight or too loose
- not assigning alert ownership
- monitoring technical metrics but not business impact
- retraining before checking whether the data pipeline is broken
- treating monitoring alerts as noise instead of review signals
- forgetting to document why a model was retrained or rolled back
Official Resources
- Vertex AI prediction documentation
- Vertex AI Model Monitoring documentation
- Google Cloud MLOps guidance
Related AI Charcha Reading
- Vertex AI Pipelines and ML Artifacts Guide
- Vertex AI Custom Training Guide
- Vertex AI Feature Store Guide
- Enterprise Machine Learning Workflow Guide
- Data Quality and EDA for Machine Learning
Bottom Line
Prediction makes the model useful, but monitoring keeps it trustworthy. Choose batch or online prediction based on the workflow, then monitor production data, prediction behavior, latency, errors, and business impact so the team knows when the model needs attention.
A production model should not only return predictions. It should also leave enough operational evidence for teams to understand whether those predictions can still be trusted.