Data Quality And EDA For Machine Learning

Before choosing an algorithm, start with the data. Machine learning models learn from examples. If those examples are incomplete, inconsistent, mislabeled, or poorly formatted, the model can produce unreliable predictions.

Data quality and exploratory data analysis, or EDA, are the first real skills to build in practical machine learning.

The practical goal is not only to make a dataset look clean. The goal is to understand whether the data can support the prediction task, what risks it contains, and what should be fixed before the model learns from it.

Quick Answer

Data quality means making sure the dataset is accurate, complete, consistent, timely, and usable. EDA means exploring the dataset with summaries and visualizations so you can find missing values, outliers, correlations, patterns, and potential modeling problems before training.

Key Takeaways

• Data quality problems often become model quality problems.
• Missing values need a deliberate strategy.
• Date, text, categorical, and numeric fields often need cleaning.
• EDA helps identify patterns, anomalies, and influential variables.
• Cleaning and exploration are usually iterative, not one-time steps.
• Label quality and target leakage should be checked before trusting model results.
• EDA should create notes and evidence, not just charts.

What Good Data Quality Looks Like

These checks are not just administrative. They directly affect whether the model can learn useful relationships.

Data Quality Issue Response

Finding a data issue is only useful if the team knows what to do next.

Common Data Quality Problems

Missing values

Missing values can appear because a field was not collected, a system failed, a customer skipped a form, or a value is not applicable.

Common strategies:

• remove rows when missingness is rare and safe,
• fill numeric values with a median or domain-based default,
• fill categories with unknown,
• create a missingness flag,
• investigate whether missingness itself is predictive.

Wrong data types

Dates stored as text, numbers stored as strings, and categories mixed with numeric codes can break analysis.

Check:

• date columns,
• currency columns,
• boolean fields,
• categorical fields,
• numeric ranges.

Unwanted characters

Data may include symbols, prefixes, whitespace, inconsistent casing, or special markers.

Examples:

• <2006 in a year field,
• N/A mixed with real categories,
• extra spaces in category names,
• inconsistent capitalization.

Categorical values

Many ML models require numeric inputs. Categorical columns often need encoding.

Common approaches:

• one-hot encoding,
• label encoding when order exists,
• grouping rare categories,
• using embeddings for high-cardinality categories.

Label Quality And Target Leakage

Two issues deserve special attention: bad labels and target leakage.

Label quality means the target value is correct and meaningful. If a churn label, fraud label, defect label, or risk label is inconsistent, the model may learn noise. Before training, confirm how the label is created, when it becomes available, and whether business users trust it.

Target leakage happens when the training data includes information that would not be available at prediction time. For example, a model predicting whether a customer will cancel should not use fields that are populated after cancellation.

Practical checks:

• define the prediction moment,
• list which columns are available before that moment,
• remove future-looking fields,
• review suspiciously strong features,
• compare validation performance with and without risky columns,
• ask domain experts whether the feature would really exist in production.

If a model looks too good too quickly, leakage is one of the first things to investigate.

What EDA Does

Exploratory data analysis helps you understand what the dataset is telling you before you build a model.

EDA helps answer:

• What values are common?
• What values are missing?
• Are there outliers?
• Are columns correlated?
• Which features may influence the target?
• Are there unusual groups?
• Does the label look usable?

Useful EDA Methods

The goal is not to make beautiful charts. The goal is to learn whether the data can support the prediction task.

Practical EDA Workflow

1. Confirm the prediction goal and target label.
2. Inspect row counts, columns, and data types.
3. Check missing values by column.
4. Review duplicate records.
5. Check numeric ranges and outliers.
6. Review category counts and rare values.
7. Inspect date ranges and freshness.
8. Visualize the target distribution.
9. Compare important features against the target.
10. Look for leakage or future-looking fields.
11. Document issues, assumptions, and cleanup decisions.
12. Recheck the dataset after cleaning.

EDA should be iterative. The first pass helps you find obvious issues. Later passes help you understand relationships, risks, and whether the prepared data is good enough for modeling.

Univariate And Bivariate Analysis

Univariate analysis looks at one variable at a time.

Use it to find:

• distribution shape,
• extreme values,
• missing values,
• category imbalance.

Bivariate analysis looks at relationships between two variables.

Use it to find:

• feature-target relationships,
• correlations,
• category differences,
• possible interaction effects.

Example EDA Checklist

Use this checklist before model training:

1. Confirm the target label is present.
2. Count missing values by column.
3. Check numeric ranges.
4. Review category counts.
5. Convert dates into usable date/time features.
6. Look for duplicates.
7. Visualize the target distribution.
8. Check correlations between numeric fields.
9. Compare features against the target.
10. Note data issues before training.

Real-World Example

Imagine a team building a model to predict whether support tickets will become escalations. The raw dataset includes ticket text, customer tier, product area, support channel, creation date, priority, assigned team, resolution time, and escalation status.

At first, the model may look strong because resolution time is highly related to escalation. But resolution time is usually known after the ticket has been handled. If the goal is to predict escalation when the ticket is created, resolution time is leakage and should not be used.

EDA also reveals practical issues:

• some support channels are missing,
• customer tier is spelled inconsistently,
• older tickets use different product names,
• escalation labels changed after a policy update,
• a few customers create many duplicate tickets,
• ticket text length varies heavily by channel.

These findings matter more than the first model score. The team may need to standardize product names, remove leakage fields, split data by time, and review whether escalation labels are consistent before training. This is how EDA protects the project from false confidence.

EDA Output To Save

EDA should produce notes that future reviewers can understand.

EDA In BigQuery And Python

Python is useful for visual exploration with tools such as Pandas, Matplotlib, and Seaborn.

BigQuery is useful when the dataset is large or already stored in a warehouse. SQL can help you count missing values, group categories, inspect ranges, and create training tables.

In real projects, both are common:

• use SQL to select and aggregate,
• use Python to visualize and experiment,
• return to SQL for repeatable data preparation.

Enterprise Review Questions

For business or production ML projects, EDA should include ownership and usage questions:

• Who owns the source data?
• Is the data approved for this ML use case?
• Does the data include sensitive or regulated fields?
• Is the target label trusted by business users?
• Are features available at prediction time?
• Will the same cleaning logic be used later?
• Who reviews data quality failures?
• How often should data quality be rechecked?

These questions keep EDA connected to real workflow risk. A dataset can be statistically interesting but still unsuitable for a production model.

Official Resources

• BigQuery documentation
• BigQuery ML documentation
• Pandas missing data guide
• TensorFlow Data Validation

Related AI Charcha Reading

• Launching Into Machine Learning: A Practical Learning Path
• Feature Engineering For Machine Learning
• How To Choose Good Machine Learning Features
• Model Evaluation, Generalization, And Sampling
• Data Preprocessing Options for Enterprise ML
• BigQuery ML Beginner Guide
• Enterprise Machine Learning Workflow Guide

FAQ

Should data cleaning happen before or after EDA?

Both. Start with basic cleaning so the data can be inspected, then use EDA to discover deeper cleaning needs.

Is EDA only for data scientists?

No. Analysts, engineers, product owners, and ML practitioners all benefit from understanding the dataset before trusting a model.

Can AutoML fix bad data?

AutoML can automate model training, but it cannot fully fix unclear labels, missing business context, or poor data quality.

Bottom Line

Good models start with trustworthy data. Data quality checks and EDA help you find problems before they become expensive model failures.

EDA is not just an early notebook activity. It is the process of understanding whether the data, label, features, assumptions, and business context are strong enough to support a model people can trust.