Chapter 12 — Linear Algebra and Calculus for Data Science

Linear algebra is the language of data; calculus is the language of optimization. This chapter reviews the vectors, matrices, derivatives, and gradients that power machine learning, from regression to neural networks.

Learning Objectives

Represent and operate on vectors and matrices with NumPy.
Compute dot products, matrix products, transposes, and inverses.
Solve linear systems and find eigenvalues/eigenvectors.
Measure vectors with norms and cosine similarity.
Approximate derivatives numerically.
Understand gradients and a basic gradient-descent step.
See how eigendecomposition enables PCA.

Prerequisites / Imports

In [1]:

import numpy as np
import matplotlib.pyplot as plt

1 Vectors

A vector is an ordered list of numbers. NumPy represents it as a 1-D array.

In [1]:

v = np.array([1, 2, 3])
w = np.array([4, 0, -1])
print('v + w =', v + w)
print('2 * v =', 2 * v)
print('dot(v, w) =', np.dot(v, w))

v + w = [5 2 2]
2 * v = [2 4 6]
dot(v, w) = 1

2 Matrices and Matrix Operations

Matrices are 2-D arrays. Multiply with @, transpose with .T.

In [1]:

A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])
print('A @ B ='); print(A @ B)
print('A transposed ='); print(A.T)

A @ B =
[[19 22]
 [43 50]]
A transposed =
[[1 3]
 [2 4]]

3 Inverse and Determinant

A square matrix with non-zero determinant has an inverse.

In [1]:

print('det(A) =', np.linalg.det(A).round(4))
A_inv = np.linalg.inv(A)
print('A_inv ='); print(np.round(A_inv, 4))
print('A @ A_inv (should be I) ='); print(np.round(A @ A_inv, 4))

det(A) = -2.0
A_inv =
[[-2.   1. ]
 [ 1.5 -0.5]]
A @ A_inv (should be I) =
[[1. 0.]
 [0. 1.]]

4 Solving Linear Systems

Solve $Ax = b$ directly with np.linalg.solve — more stable than computing the inverse.

In [1]:

A = np.array([[3.0, 2.0], [1.0, 4.0]])
b = np.array([7.0, 6.0])
x = np.linalg.solve(A, b)
print('solution x =', x)
print('check A @ x =', A @ x)

solution x = [1.6 1.1]
check A @ x = [7. 6.]

5 Vector Norms and Cosine Similarity

The $L^2$ norm measures length; cosine similarity measures direction agreement.

In [1]:

v = np.array([1, 2, 3])
print('L2 norm of v =', np.linalg.norm(v).round(4))

u = np.array([1, 0, 1])
cos_sim = np.dot(u, v) / (np.linalg.norm(u) * np.linalg.norm(v))
print('cosine similarity =', cos_sim.round(4))

L2 norm of v = 3.7417
cosine similarity = 0.7559

6 Eigenvalues and Eigenvectors

For a square matrix $A$, an eigenvector $v$ satisfies $Av = \lambda v$.

In [1]:

A = np.array([[4, -2], [1, 1]])
evals, evecs = np.linalg.eig(A)
print('eigenvalues:', np.round(evals, 4))
print('eigenvectors (columns):'); print(np.round(evecs, 4))
# verify A v = lambda v for the first pair
print('check:', np.round(A @ evecs[:, 0], 4), '=?', np.round(evals[0] * evecs[:, 0], 4))

eigenvalues: [3. 2.]
eigenvectors (columns):
[[0.8944 0.7071]
 [0.4472 0.7071]]
check: [2.6833 1.3416] =? [2.6833 1.3416]

7 Numerical Derivatives

The central difference approximates $f'(x) \approx \frac{f(x+h) - f(x-h)}{2h}$.

In [1]:

def f(x):
    return x ** 2 + 3 * x

def deriv(f, x, h=1e-5):
    return (f(x + h) - f(x - h)) / (2 * h)

print("f'(2) numeric =", round(deriv(f, 2.0), 6))
print("f'(2) exact   =", (2*2 + 3), "  (derivative 2x+3 at x=2)")

f'(2) numeric = 7.0
f'(2) exact   = 7   (derivative 2x+3 at x=2)

8 Gradients and Gradient Descent

A gradient collects partial derivatives. Gradient descent steps opposite the gradient to minimize a function. We minimize $g(x,y) = (x-3)^2 + (y+1)^2$, whose minimum is $(3, -1)$.

In [1]:

def g(p):
    x, y = p
    return (x - 3) ** 2 + (y + 1) ** 2

def grad_g(p, h=1e-5):
    x, y = p
    gx = (g([x + h, y]) - g([x - h, y])) / (2 * h)
    gy = (g([x, y + h]) - g([x, y - h])) / (2 * h)
    return np.array([gx, gy])

p = np.array([0.0, 0.0])       # start
lr = 0.1                       # learning rate
for i in range(60):
    p = p - lr * grad_g(p)
print('minimum found at:', np.round(p, 4), '(true: [3, -1])')

minimum found at: [ 3. -1.] (true: [3, -1])

Case Study: Principal Component Analysis by Hand

PCA finds directions of greatest variance via the eigendecomposition of the covariance matrix. We project 2-D synthetic data onto its first principal component.

In [1]:

rng = np.random.default_rng(0)
X = rng.normal(size=(100, 2)) @ np.array([[0.9, 0.6], [0.1, 0.3]])
Xc = X - X.mean(axis=0)
cov = np.cov(Xc, rowvar=False)
evals, evecs = np.linalg.eigh(cov)
order = np.argsort(evals)[::-1]
evals, evecs = evals[order], evecs[:, order]
print('explained variance ratio:', (evals / evals.sum()).round(3))
proj = Xc @ evecs[:, 0]
print('first PC shape:', proj.shape)

plt.figure(figsize=(6,6))
plt.scatter(Xc[:, 0], Xc[:, 1], alpha=0.5, label='data')
for i, ev in enumerate(evecs.T):
    plt.quiver(0, 0, ev[0]*np.sqrt(evals[i]), ev[1]*np.sqrt(evals[i]),
               angles='xy', scale_units='xy', scale=1, color=['red','green'][i])
plt.axis('equal'); plt.title('PCA: principal components'); plt.legend(['data','PC1','PC2'])
plt.show()

explained variance ratio: [0.977 0.023]
first PC shape: (100,)

Exercises

Compute the dot product of [1,2,3] and [4,5,6].
Multiply two 2x3 and 3x2 matrices and report the resulting shape.
Find the inverse of [[2,0],[0,3]] and verify with matrix multiplication.
Solve the system $2x + y = 5$, $x - 3y = -8$.
Compute the $L^2$ norm and cosine similarity of two vectors.
Find the eigenvalues of a 2x2 matrix and verify $Av = \lambda v$.
Approximate the derivative of $\sin(x)$ at $x=0$ with a central difference.
Run gradient descent on $g(x,y)=(x-1)^2+(y-2)^2$ and confirm the minimum.
Explain why solve is preferred over inv for $Ax=b$.
Describe in one sentence how PCA uses eigenvectors.

Python Data Science: From Foundations to Applications — Chapter 12