Feature engineering can happen before training, inside the model pipeline, or inside a data warehouse. Two practical options are Keras preprocessing layers and BigQuery ML transformations.
This guide explains when to use each approach and what patterns learners should understand first.
The practical question is not which tool is more advanced. The question is where preprocessing should live so the feature logic is repeatable, testable, and available when predictions are made.
Quick Answer
Use Keras preprocessing layers when you want preprocessing packaged with a TensorFlow model. Use BigQuery ML feature engineering when your data already lives in BigQuery and you want SQL-based transformations close to the warehouse.
Use Keras when the model pipeline should carry preprocessing with it. Use BigQuery ML when SQL transformations inside the warehouse are the clearest and most repeatable path.
Key Takeaways
- Keras preprocessing layers can normalize numbers, encode categories, vectorize text, and create embeddings.
- BigQuery ML can transform features with SQL and built-in ML preprocessing functions.
- The same transformations used during training must also be applied during prediction.
- BigQuery ML
TRANSFORMhelps keep training and prediction transformations consistent. - Always compare feature engineering changes against a baseline metric.
- Feature logic should avoid leakage and be available at prediction time.
- The best choice depends on data location, team skills, serving workflow, and governance needs.
Feature Engineering In Keras
When To Use Keras Or BigQuery ML
| Situation | Better starting point | Why |
|---|---|---|
| Data already lives in BigQuery | BigQuery ML | SQL transformations stay close to warehouse data |
| Model is built in TensorFlow/Keras | Keras preprocessing layers | Preprocessing can ship with the model |
| Team is mostly analysts | BigQuery ML | SQL is easier to review and repeat |
| Team is mostly Python/ML engineers | Keras | Fits model pipeline development |
| Text or high-cardinality categories need embeddings | Keras | Embeddings fit naturally in neural models |
| SQL transformations are enough | BigQuery ML | Simpler and easier to govern |
| Serving must use exactly the model preprocessing | Keras | Exported model can include preprocessing layers |
Prediction uses ML.PREDICT in BigQuery | BigQuery ML TRANSFORM | Training and prediction logic stay connected |
Neither option is automatically better. The right choice depends on where the data lives, who maintains the workflow, and how predictions will be served.
Feature Engineering In Keras
Keras preprocessing layers help build models that accept raw or lightly processed data and transform it inside the model pipeline.
Common layers include:
| Layer | Use |
|---|---|
Normalization | Standardize numeric features |
StringLookup | Map string categories to indexes |
CategoryEncoding | Convert categories to encoded vectors |
TextVectorization | Convert raw text to tokenized representations |
Embedding | Learn dense representations for categories or tokens |
Keras Workflow
A practical workflow:
- Create a training dataset.
- Identify numeric, categorical, and text columns.
- Build preprocessing layers for each feature type.
- Adapt layers on training data where needed.
- Connect preprocessing outputs to the model.
- Train and evaluate.
- Export the model with preprocessing included.
This reduces the chance that training and serving use different transformations.
Keras preprocessing is especially useful when the model will be exported and used outside the training notebook. If preprocessing lives inside the model graph, the serving path is less likely to forget an important normalization, lookup, or text vectorization step.
Example Keras Patterns
For numeric features:
normalizer = tf.keras.layers.Normalization()
normalizer.adapt(train_numeric_values)
For string categories:
lookup = tf.keras.layers.StringLookup(output_mode="one_hot")
lookup.adapt(train_category_values)
For text:
vectorizer = tf.keras.layers.TextVectorization(max_tokens=10000)
vectorizer.adapt(train_text_values)
The exact implementation depends on the dataset, but the concept is the same: convert raw values into model-ready tensors.
Before using a Keras preprocessing layer, check:
- whether the layer should be adapted only on training data,
- whether the vocabulary or normalization statistics need to be saved,
- whether the same raw input format will arrive during serving,
- whether rare categories or unknown values are handled,
- whether preprocessing increases model complexity.
Feature Engineering In BigQuery ML
BigQuery ML is useful when the training data already lives in BigQuery and the team wants feature engineering in SQL.
Useful patterns include:
- extracting dates and times,
- calculating distances,
- creating ratios,
- bucketizing numeric values,
- crossing categorical features,
- expanding polynomial features,
- filtering bad training examples,
- using
TRANSFORMfor repeatable preprocessing.
BigQuery ML is practical when analysts and data teams already trust the warehouse. It also makes feature logic easier to inspect because transformations are written in SQL.
BigQuery ML Feature Functions
| Function | Use |
|---|---|
ML.FEATURE_CROSS | Combine categorical features |
ML.BUCKETIZE | Convert numeric values into buckets |
ML.POLYNOMIAL_EXPAND | Create polynomial combinations |
ML.NGRAMS | Create text n-grams |
ML.STANDARD_SCALER | Standardize values |
Why TRANSFORM Matters
The TRANSFORM clause can define feature transformations as part of the model. That helps ensure the same logic is used when training and when calling ML.PREDICT.
This reduces training-serving skew, which happens when the model sees features one way during training and a different way during prediction.
This matters because a model can appear strong during evaluation but fail in production if the prediction workflow builds features differently. TRANSFORM is useful when BigQuery ML is both the training and prediction environment.
Example BigQuery ML Ideas
For a taxi fare model, useful engineered features might include:
- trip distance,
- pickup hour,
- pickup day of week,
- pickup and dropoff location buckets,
- pickup-hour feature cross,
- geographic distance,
- passenger count,
- toll-adjusted fare.
The feature set should be tested against a baseline model using a metric such as RMSE for regression.
Leakage And Availability Checks
Whether you use Keras or BigQuery ML, feature logic must avoid leakage.
| Check | Why it matters |
|---|---|
| Feature exists before prediction | Prevents future information from leaking into training |
| Transformation is repeatable | Keeps experiments comparable |
| Unknown categories are handled | Prevents serving failures |
| Training statistics are saved | Keeps normalization and lookup logic stable |
| Feature improves validation results | Avoids adding complexity without benefit |
| Feature is allowed for the use case | Supports privacy and governance |
For example, a feature such as days_since_last_purchase may be useful for churn prediction. A feature such as cancellation_date would leak the answer.
Real-World Example
Imagine a company building a model to predict monthly customer spend.
If the data lives in BigQuery, the team might use SQL to create features such as:
- purchases in the last 30 days,
- average order value,
- days since last purchase,
- customer segment,
- product category counts,
- support ticket count.
BigQuery ML is a good starting point because the transformations are easy to review and the model can be trained close to the data.
If the team later builds a TensorFlow model that consumes raw customer records and needs text, embeddings, or more complex preprocessing, Keras preprocessing layers may be a better fit. The model can include normalization, category lookup, and text vectorization as part of the exported model.
The decision is not about tool preference. It is about where the feature logic can be maintained safely and repeated during prediction.
Keras vs BigQuery ML
| Decision | Keras | BigQuery ML |
|---|---|---|
| Best for | TensorFlow model pipelines | SQL-first ML workflows |
| Data location | Files, tensors, pipelines | BigQuery tables |
| Preprocessing | Model layers | SQL and ML functions |
| Serving consistency | Export with model | Use TRANSFORM |
| Learner fit | Python/TensorFlow users | SQL/data warehouse users |
Practical Workflow
- Define the prediction goal.
- List raw fields available at prediction time.
- Create a simple baseline model.
- Add a small number of feature transformations.
- Compare validation metrics against the baseline.
- Check for leakage and serving mismatch.
- Keep only features that improve performance or trust.
- Document the feature logic.
- Decide whether preprocessing belongs in Keras, BigQuery ML, or a shared pipeline.
Official Resources
- Keras preprocessing layers
- TensorFlow structured data preprocessing
- BigQuery ML TRANSFORM clause
- BigQuery ML preprocessing functions
Related AI Charcha Reading
- Feature Engineering for Machine Learning
- How to Choose Good Machine Learning Features
- Vertex AI Feature Store Guide
- BigQuery ML Beginner Guide
- Data Preprocessing Options for Enterprise ML
- Model Evaluation, Generalization, And Sampling Guide
FAQ
How can Keras do feature engineering?
Keras can do feature engineering with preprocessing layers such as Normalization, StringLookup, CategoryEncoding, TextVectorization, and Embedding.
How can BigQuery ML do feature engineering?
BigQuery ML can do feature engineering with SQL transformations, preprocessing functions such as ML.FEATURE_CROSS, ML.BUCKETIZE, ML.POLYNOMIAL_EXPAND, and the TRANSFORM clause.
Bottom Line
Keras and BigQuery ML both support practical feature engineering. Use Keras when preprocessing belongs inside the model pipeline. Use BigQuery ML when SQL-first feature engineering close to warehouse data is the simplest path.
The best feature engineering choice is the one that keeps transformations useful, repeatable, available at prediction time, and easy for the right team to maintain.