Chapter 21 — Natural Language Processing

Text is the most abundant data type. NLP turns unstructured text into numeric features models can use. This chapter covers tokenization, vectorization, text classification, and visualization — using scikit-learn so everything runs offline without downloads.

Learning Objectives

Tokenize and vectorize text with bag-of-words and TF-IDF.
Build a text classification pipeline.
Remove stopwords and reason about vocabulary size.
Inspect the most informative features.
Generate a word cloud.
Understand the landscape: from bag-of-words to word embeddings and transformers.

Prerequisites / Imports

In [1]:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, accuracy_score
from wordcloud import WordCloud

1 A Small Labeled Corpus

We create a tiny corpus of documents labeled by topic so we can run a complete pipeline.

In [1]:

docs = [
    'the new phone has a great camera and fast processor',
    'laptop with fast cpu and lots of memory',
    'phone review the battery lasts all day',
    'tablet with a bright screen and fast chip',
    'the team scored a goal in the final minutes',
    'a great match with two goals and a save',
    'the striker scored a hat trick goal',
    'the goalkeeper made a great save',
    'a new gadget with a bright screen',
    'the midfielder scored a great goal',
]
labels = ['tech','tech','finance','tech','sports','sports','sports','sports','finance','sports']
corpus = pd.DataFrame({'text': docs, 'label': labels})
corpus

	text	label
0	the new phone has a great camera and fast proc...	tech
1	laptop with fast cpu and lots of memory	tech
2	phone review the battery lasts all day	finance
3	tablet with a bright screen and fast chip	tech
4	the team scored a goal in the final minutes	sports
5	a great match with two goals and a save	sports
6	the striker scored a hat trick goal	sports
7	the goalkeeper made a great save	sports
8	a new gadget with a bright screen	finance
9	the midfielder scored a great goal	sports

2 Bag-of-Words

CountVectorizer tokenizes and builds a term-document matrix.

In [1]:

cv = CountVectorizer()
X_counts = cv.fit_transform(corpus['text'])
print('vocabulary size:', len(cv.vocabulary_))
print('sample terms:', list(cv.vocabulary_.keys())[:10])
print('matrix shape:', X_counts.shape)

vocabulary size: 41
sample terms: ['the', 'new', 'phone', 'has', 'great', 'camera', 'and', 'fast', 'processor', 'laptop']
matrix shape: (10, 41)

3 TF-IDF

TF-IDF down-weights common words and highlights distinctive ones.

In [1]:

tfidf = TfidfVectorizer(stop_words='english')
X_tfidf = tfidf.fit_transform(corpus['text'])
print('tf-idf shape:', X_tfidf.shape)
print('features:', list(tfidf.get_feature_names_out()))

tf-idf shape: (10, 32)
features: ['battery', 'bright', 'camera', 'chip', 'cpu', 'day', 'fast', 'final', 'gadget', 'goal', 'goalkeeper', 'goals', 'great', 'hat', 'laptop', 'lasts', 'lots', 'match', 'memory', 'midfielder', 'minutes', 'new', 'phone', 'processor', 'review', 'save', 'scored', 'screen', 'striker', 'tablet', 'team', 'trick']

4 Text Classification

A pipeline of TF-IDF + a classifier predicts the topic of a document.

In [1]:

X_train, X_test, y_train, y_test = train_test_split(corpus['text'], corpus['label'], test_size=0.3, random_state=42)
clf = Pipeline([('tfidf', TfidfVectorizer(stop_words='english')), ('nb', MultinomialNB())])
clf.fit(X_train, y_train)
pred = clf.predict(X_test)
print('accuracy:', round(accuracy_score(y_test, pred), 3))
print(classification_report(y_test, pred, zero_division=0))

accuracy: 0.333
              precision    recall  f1-score   support

     finance       0.00      0.00      0.00         1
      sports       0.50      1.00      0.67         1
        tech       0.00      0.00      0.00         1

    accuracy                           0.33         3
   macro avg       0.17      0.33      0.22         3
weighted avg       0.17      0.33      0.22         3

5 Inspecting Features

A linear model exposes which words drive each class — a window into model reasoning.

In [1]:

clf2 = Pipeline([('tfidf', TfidfVectorizer(stop_words='english')), ('lr', LogisticRegression(max_iter=1000))])
clf2.fit(X_train, y_train)
features = clf2.named_steps['tfidf'].get_feature_names_out()
coefs = clf2.named_steps['lr'].coef_
for i, cls in enumerate(clf2.named_steps['lr'].classes_):
    top = np.argsort(coefs[i])[-5:][::-1]
    print(f'{cls}:', [features[j] for j in top])

finance: ['battery', 'day', 'lasts', 'review', 'phone']
sports: ['goal', 'scored', 'goalkeeper', 'save', 'midfielder']
tech: ['fast', 'camera', 'processor', 'new', 'tablet']

6 Word Cloud

A word cloud visualizes term frequency in a corpus.

In [1]:

text = ' '.join(corpus['text'])
wc = WordCloud(width=700, height=300, background_color='white').generate(text)
plt.figure(figsize=(8,3))
plt.imshow(wc, interpolation='bilinear'); plt.axis('off')
plt.title('Word cloud of the corpus'); plt.show()

7 The Wider NLP Landscape

Bag-of-words ignores word order. Modern NLP uses word embeddings (dense vectors capturing meaning) and transformer models (e.g., BERT/GPT) that understand context. Libraries like NLTK, spaCy, and Hugging Face Transformers power these; they typically require downloaded models, unlike the self-contained examples here.

Case Study: Classifying News Headlines

A slightly larger synthetic headline dataset demonstrates a realistic end-to-end pipeline.

In [1]:

headlines = [
    'stocks surge on strong earnings report', 'markets fall amid recession fears',
    'central bank raises interest rates', 'tech ipo oversubscribed',
    'new film breaks box office records', 'actor wins top award',
    'music festival lineup announced', 'streaming platform renews hit series',
    'quarterly gdp growth beats expectations', 'inflation eases as fuel prices drop',
    'director wins prize at film festival', 'album tops the charts this week',
] * 2
labels = (['finance']*5 + ['entertainment']*4 + ['finance']*3) * 2
df = pd.DataFrame({'headline': headlines, 'topic': labels})
clf = Pipeline([('tfidf', TfidfVectorizer(stop_words='english')), ('lr', LogisticRegression(max_iter=1000))])
scores = []
for _ in range(5):
    Xtr, Xte, ytr, yte = train_test_split(df['headline'], df['topic'], test_size=0.3)
    clf.fit(Xtr, ytr)
    scores.append(accuracy_score(yte, clf.predict(Xte)))
print('mean accuracy over 5 splits:', round(float(np.mean(scores)), 3))
print('example predictions:', clf.predict(['market rally continues', 'singer releases new album']))

mean accuracy over 5 splits: 0.525
example predictions: ['finance' 'finance']

Exercises

Build a CountVectorizer on five sentences and print the vocabulary.
Convert the same corpus to a TF-IDF matrix.
Train a MultinomialNB text classifier and report accuracy.
Use stop_words='english' and note the change in vocabulary size.
List the top 5 features for each class in a linear model.
Generate a word cloud from a paragraph of text.
Explain why TF-IDF differs from raw counts.
Describe one limitation of bag-of-words.
Outline how a transformer model differs from bag-of-words.
Build a pipeline that classifies your own set of 12 documents.

Python Data Science: From Foundations to Applications — Chapter 21