Lesson 3

Boosting Text Classification Power with Gradient Boosting Classifier


Greetings learners! Prepare to immerse yourself in advanced text classification techniques as we explore an advanced ensemble method: the Gradient Boosting Classifier. By the end of this lesson, you will have a sound understanding of this ensemble method and also gain practical experience in applying it using Python and Scikit-learn.

Quick Recap on Dataset Preparation

First, let's review a few steps that should already be familiar: loading required libraries and preparing the dataset, which is the Reuters-21578 Text Categorization Collection here.

1# import required libraries 2from sklearn.ensemble import GradientBoostingClassifier 3from sklearn.feature_extraction.text import CountVectorizer 4from sklearn.preprocessing import LabelEncoder 5from sklearn.model_selection import train_test_split 6from sklearn.metrics import accuracy_score 7from nltk.corpus import reuters 8import nltk 9 10nltk.download('reuters', quiet=True) 11 12# Limiting the data for quick execution 13categories = reuters.categories()[:3] 14documents = reuters.fileids(categories) 15 16# Preparing the dataset 17text_data = [" ".join([word for word in reuters.words(fileid)]) for fileid in documents] 18categories_data = [reuters.categories(fileid)[0] for fileid in documents] 19 20# Using CountVectorizer for feature extraction 21count_vectorizer = CountVectorizer(max_features=500) 22X = count_vectorizer.fit_transform(text_data) 23y = LabelEncoder().fit_transform(categories_data) 24 25# Split the data for train and test 26X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=1)

This code prepares the dataset, using CountVectorizer for feature extraction, LabelEncoder for changing categories into numeric format, and splitting our data into training and test sets.

Inside the Gradient Boosting Classifier

Gradient Boosting Classifier is an ensemble learning technique that fine-tunes its accuracy iteratively by addressing the inaccuracies of prior models, predominantly employing decision trees as its weak learners. The process unfolds through several critical stages:

  1. Initial Prediction: It starts with a simple model, often predicting a constant value (like the mean of the target variable), setting the stage for improvement.

  2. Iterative Correction: The essence of Gradient Boosting is its ability to learn from the mistakes of previous iterations. It focuses on the residuals - the differences between the predicted and actual values. Each new tree in the ensemble attempts to correct these residuals, aiming to minimize a loss function reflective of these errors.

  3. Learning Rate: This parameter moderates the contribution of each new tree. A smaller learning rate demands more trees to achieve high accuracy but fosters a model that's less prone to overfitting. Conversely, a larger learning rate can hasten learning but increase the risk of overfitting by overly adjusting to the training data.

  4. Controlling Complexity: To prevent overfitting, Gradient Boosting limits each tree's complexity, primarily using the max_depth parameter. This control ensures that individual trees do not grow too complex and start modeling the noise within the training data.

  5. Optimal Number of Trees: The algorithm iteratively adds trees until it reaches the specified number (n_estimators) or until adding new trees does not significantly reduce the error. This balance is crucial as too few trees might not capture all the data patterns, while too many could lead to overfitting.

In summary, Gradient Boosting sequentially builds upon previous trees to correct errors, with careful adjustments of parameters like the learning rate and max depth to ensure a robust model. Its adaptive nature makes it exceptionally powerful for tasks including text classification, albeit requiring thoughtful parameter tuning to balance complexity with generalization.

Implementing Gradient Boosting Classifier for Text Classification

The main attraction is the Gradient Boosting Classifier. Let's set up and implement it now.

1# Instantiate the GradientBoostingClassifier with tuned parameters 2gb_clf = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1, max_depth=3) 3 4# Train the classifier 5gb_clf.fit(X_train, y_train) 6 7# Make predictions 8y_pred = gb_clf.predict(X_test)

Here, we create an instance of the GradientBoostingClassifier. We set n_estimators (boosting stages) to 100, the learning_rate (model learning speed) to 0.1, and max_depth (tree depth) to 3. After this setup, the model is trained using fit and finally, predictions are made about our test data.

Performance Evaluation

With our model trained and having made some predictions, let's assess the model's performance.

1# Evaluate the performance of the classifier 2print("Accuracy: ", accuracy_score(y_test, y_pred))

Running this gives us:

Plain text
1Accuracy: 0.9852150537634409

The accuracy_score function compares predicted values (y_pred) to actual test categories (y_test). The result means that our Gradient Boosting model predicts around 98.5% of the instances correctly.


Today, you learned about the Gradient Boosting Classifier and its workings, implemented it, and evaluated its performance. Advanced ensemble methods like this offer you a significant upper hand in NLP tasks.

Remember, theory without practice is empty. Sharpen your skills using the tasks that follow this lesson. Don't hesitate to re-read any section you want more clarity on. Onwards!

Enjoy this lesson? Now it's time to practice with Cosmo!

Practice is how you turn knowledge into actual skills.