Chapter 20 — Deep Learning with TensorFlow/Keras

Deep learning uses multi-layer neural networks to learn hierarchical representations. This chapter introduces neural networks with TensorFlow/Keras: building, training, evaluating, and visualizing learning curves on a classification task.

Learning Objectives

Explain what a neural network and a dense layer are.
Build a model with the Keras Sequential API.
Compile with loss, optimizer, and metrics.
Train with fit and monitor loss/accuracy.
Evaluate on a test set and predict.
Plot training curves to diagnose learning.
Understand overfitting and the train/validation split.

Prerequisites / Imports

We silence TensorFlow's verbose C++ logs before importing.

In [1]:

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
os.environ['TF_ENABLE_ONEDNN_OPTS'] = '0'
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

1 What Is a Neural Network?

A neural network stacks layers of weighted sums followed by nonlinear activations. A dense layer connects every input to every neuron. Deep = many layers.

2 Prepare Data

We synthesize a binary classification problem and scale features. Neural networks benefit from scaled inputs and a held-out validation split.

In [1]:

X, y = make_classification(n_samples=1000, n_features=10, n_informative=6, n_redundant=2, random_state=42)
X = StandardScaler().fit_transform(X)
X_train, X_tmp, y_train, y_tmp = train_test_split(X, y, test_size=0.3, random_state=42)
X_val, X_test, y_val, y_test = train_test_split(X_tmp, y_tmp, test_size=0.5, random_state=42)
print('train/val/test:', X_train.shape, X_val.shape, X_test.shape)

train/val/test: (700, 10) (150, 10) (150, 10)

3 Build the Model

Two hidden layers with ReLU activation, one sigmoid output for binary classification.

In [1]:

model = Sequential([
    Dense(16, activation='relu', input_shape=(10,)),
    Dense(8, activation='relu'),
    Dense(1, activation='sigmoid'),
])
model.summary()

Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Layer (type)        ┃ Output Shape   ┃  Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ dense (Dense)       │ (None, 16)     │      176 │
├─────────────────────┼────────────────┼──────────┤
│ dense_1 (Dense)     │ (None, 8)      │      136 │
├─────────────────────┼────────────────┼──────────┤
│ dense_2 (Dense)     │ (None, 1)      │        9 │
└─────────────────────┴────────────────┴──────────┘
 Total params: 321 (1.25 KB)
 Trainable params: 321 (1.25 KB)
 Non-trainable params: 0 (0.00 B)

C:\Users\DELL\anaconda3\Lib\site-packages\keras\src\layers\core\dense.py:86: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)

4 Compile the Model

Choose a loss, an optimizer, and metrics to track.

In [1]:

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

5 Train the Model

fit runs mini-batch gradient descent for a number of epochs, validating on held-out data.

In [1]:

history = model.fit(X_train, y_train, validation_data=(X_val, y_val),
                    epochs=25, batch_size=32, verbose=0)
print('final train acc:', round(float(history.history['accuracy'][-1]), 3))
print('final val acc:', round(float(history.history['val_accuracy'][-1]), 3))

final train acc: 0.891
final val acc: 0.867

6 Learning Curves

Plotting loss and accuracy over epochs reveals whether the model is learning or overfitting.

In [1]:

fig, ax = plt.subplots(1, 2, figsize=(12,4))
ax[0].plot(history.history['loss'], label='train')
ax[0].plot(history.history['val_loss'], label='val')
ax[0].set_title('Loss'); ax[0].set_xlabel('epoch'); ax[0].legend()
ax[1].plot(history.history['accuracy'], label='train')
ax[1].plot(history.history['val_accuracy'], label='val')
ax[1].set_title('Accuracy'); ax[1].set_xlabel('epoch'); ax[1].legend()
plt.tight_layout(); plt.show()

7 Evaluate and Predict

Final performance on the test set; predict probabilities for a few samples.

In [1]:

test_loss, test_acc = model.evaluate(X_test, y_test, verbose=0)
print('test loss:', round(float(test_loss), 3))
print('test accuracy:', round(float(test_acc), 3))
probs = model.predict(X_test[:5], verbose=0).flatten()
preds = (probs > 0.5).astype(int)
print('predicted:', preds, 'true:', y_test[:5])

test loss: 0.315
test accuracy: 0.847
predicted: [1 0 0 1 0] true: [1 0 0 1 0]

8 Overfitting and How to Fight It

When validation loss rises while training loss falls, the model overfits. Common remedies: more data, dropout, early stopping, weight regularization, and smaller networks.

Case Study: Recognizing Handwritten Digits

Train a small MLP on the 8x8 digits dataset (64 pixel inputs, 10 classes).

In [1]:

from sklearn.datasets import load_digits
from tensorflow.keras.utils import to_categorical
d = load_digits()
Xd = StandardScaler().fit_transform(d.data)
yd = to_categorical(d.target, num_classes=10)
Xtr, Xte, ytr, yte = train_test_split(Xd, yd, test_size=0.2, random_state=42)
dig_model = Sequential([Dense(64, activation='relu', input_shape=(64,)),
                      Dense(32, activation='relu'),
                      Dense(10, activation='softmax')])
dig_model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
h = dig_model.fit(Xtr, ytr, epochs=20, batch_size=32, validation_split=0.2, verbose=0)
print('digits test acc:', round(float(dig_model.evaluate(Xte, yte, verbose=0)[1]), 3))

digits test acc: 0.964

C:\Users\DELL\anaconda3\Lib\site-packages\keras\src\layers\core\dense.py:86: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)

Exercises

Describe the role of an activation function in a neural network.
Build a 3-layer Sequential model for 8-feature input and binary output.
Compile a regression model with MSE loss.
Train a classifier for 15 epochs and plot the loss curve.
Explain what validation_split does.
Add a third hidden layer to the digits model and compare test accuracy.
Identify overfitting from a learning-curve plot.
Use model.predict to classify 10 new samples.
Explain why softmax is used for the output layer of a multiclass model.
Name two techniques to reduce overfitting.

Python Data Science: From Foundations to Applications — Chapter 20