Chapter 14 — Statistical Inference and Hypothesis Testing

Inference generalizes from samples to populations. This chapter covers sampling, confidence intervals, hypothesis tests, p-values, and A/B testing — the tools used to decide whether an observed effect is real or noise.

Learning Objectives

Prerequisites / Imports

In [1]:
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats
import statsmodels.api as sm

1 Sampling and Sampling Distributions

The sample mean varies across samples; its standard error is $\sigma/\sqrt{n}$.

In [1]:
rng = np.random.default_rng(0)
pop = rng.normal(loc=50, scale=10, size=100000)
sample_means = [rng.choice(pop, 30).mean() for _ in range(2000)]
print('population mean:', pop.mean().round(3))
print('mean of sample means:', np.mean(sample_means).round(3))
print('std of sample means (SE):', np.std(sample_means, ddof=1).round(3))
print('theoretical SE (sigma/sqrt(n)):', (10/np.sqrt(30)).round(3))

population mean: 49.991
mean of sample means: 50.016
std of sample means (SE): 1.796
theoretical SE (sigma/sqrt(n)): 1.826

2 Confidence Intervals

A 95% CI for a mean is $\bar{x} \pm t_{0.975}\cdot s/\sqrt{n}$.

In [1]:
rng = np.random.default_rng(1)
sample = rng.normal(loc=70, scale=8, size=40)
xbar, s, n = sample.mean(), sample.std(ddof=1), len(sample)
t_crit = stats.t.ppf(0.975, df=n-1)
margin = t_crit * s / np.sqrt(n)
print(f'95% CI: [{xbar - margin:.2f}, {xbar + margin:.2f}]')
print('sample mean:', round(xbar, 2))

95% CI: [67.61, 72.43]
sample mean: 70.02

3 Hypothesis Testing Framework

State $H_0$ (no effect) and $H_1$ (an effect), choose $\alpha$ (often 0.05), compute a test statistic and p-value, and decide.

4 One-Sample t-Test

Test whether a sample mean differs from a claimed value.

In [1]:
rng = np.random.default_rng(2)
sample = rng.normal(loc=102, scale=15, size=36)
result = stats.ttest_1samp(sample, popmean=100)
print('t-statistic:', result.statistic.round(3))
print('p-value:', result.pvalue.round(4))
print('reject H0 at alpha=0.05:', result.pvalue < 0.05)

t-statistic: 0.997
p-value: 0.3257
reject H0 at alpha=0.05: False

5 Two-Sample t-Test

Compare the means of two independent groups.

In [1]:
rng = np.random.default_rng(3)
group_a = rng.normal(loc=65, scale=10, size=40)
group_b = rng.normal(loc=70, scale=10, size=40)
result = stats.ttest_ind(group_a, group_b)
print('mean A:', group_a.mean().round(2), 'mean B:', group_b.mean().round(2))
print('p-value:', result.pvalue.round(4))
print('reject H0:', result.pvalue < 0.05)

mean A: 64.59 mean B: 68.51
p-value: 0.1154
reject H0: False

6 Chi-Square Test of Independence

Test whether two categorical variables are associated.

In [1]:
# Survey: device preference by region (rows=region, cols=device)
table = np.array([[30, 20, 10], [10, 25, 35]])
chi2, p, dof, expected = stats.chi2_contingency(table)
print('chi-square:', round(chi2, 3))
print('p-value:', round(p, 4))
print('degrees of freedom:', dof)
print('reject H0 (associated):', p < 0.05)

chi-square: 23.816
p-value: 0.0
degrees of freedom: 2
reject H0 (associated): True

Case Study: A/B Testing a Website Button

Version B should raise click-through. We simulate impressions and clicks, then test whether the difference is significant.

In [1]:
rng = np.random.default_rng(4)
n_a, n_b = 5000, 5000
ctr_a, ctr_b = 0.10, 0.12
clicks_a = rng.binomial(1, ctr_a, n_a)
clicks_b = rng.binomial(1, ctr_b, n_b)

# Two-proportion z-test via ttest_ind on 0/1 outcomes
res = stats.ttest_ind(clicks_a, clicks_b)
print(f'CTR A: {clicks_a.mean():.4f}  CTR B: {clicks_b.mean():.4f}')
print(f'lift: {((clicks_b.mean()/clicks_a.mean())-1)*100:.1f}%')
print('p-value:', res.pvalue.round(6))
print('significant at 0.05:', res.pvalue < 0.05)

CTR A: 0.1032  CTR B: 0.1224
lift: 18.6%
p-value: 0.002406
significant at 0.05: True

7 Regression Inference with statsmodels

statsmodels adds inferential detail (standard errors, t-stats, p-values, $R^2$) to regression.

In [1]:
rng = np.random.default_rng(5)
x = rng.uniform(0, 10, 60)
y = 2.5 * x + 1.0 + rng.normal(0, 2, 60)
X = sm.add_constant(x)
model = sm.OLS(y, X).fit()
print(model.summary())

                            OLS Regression Results                            
==============================================================================
Dep. Variable:                      y   R-squared:                       0.954
Model:                            OLS   Adj. R-squared:                  0.954
Method:                 Least Squares   F-statistic:                     1214.
Date:                Wed, 16 Sep 2026   Prob (F-statistic):           1.36e-40
Time:                        20:28:28   Log-Likelihood:                -115.10
No. Observations:                  60   AIC:                             234.2
Df Residuals:                      58   BIC:                             238.4
Df Model:                           1                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
const          0.7051      0.420      1.680      0.098      -0.135       1.545
x1             2.4988      0.072     34.849      0.000       2.355       2.642
==============================================================================
Omnibus:                        1.338   Durbin-Watson:                   2.027
Prob(Omnibus):                  0.512   Jarque-Bera (JB):                1.346
Skew:                           0.332   Prob(JB):                        0.510
Kurtosis:                       2.687   Cond. No.                         11.6
==============================================================================

Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

Exercises

  1. Compute a 99% confidence interval for the mean of a sample.
  2. Run a one-sample t-test testing whether a sample mean equals 50.
  3. Compare two groups with a two-sample t-test and report the p-value.
  4. Build a 2x3 contingency table and run a chi-square test.
  5. Simulate an A/B test where both versions have the same CTR; confirm you usually do not reject.
  6. Explain what a p-value is and what it is not.
  7. Fit an OLS model with statsmodels and identify the feature's p-value.
  8. Describe the relationship between confidence intervals and hypothesis tests.
  9. Why does increasing sample size shrink the standard error?
  10. Outline the steps of an A/B test from design to decision.

Python Data Science: From Foundations to Applications — Chapter 14