Hello there! In this lesson, we'll dive into the fascinating world of machine learning ensemble methods. Ensemble methods are based on a simple but powerful concept: a team of learners, or algorithms, can achieve better results working together than any individual learner on its own.
Bagging, which stands for Bootstrap Aggregating, is a prime example of an ensemble method. In the context of this course, where we are working with the Reuters-21578 Text Categorization Collection, our goal is to train a model that can accurately predict the category of a document based on its text. Bagging helps us achieve this by building multiple base learners (for instance, Decision Trees) on random subsets (bootstrapped samples) of the original dataset. Then, it aggregates their predictions to yield a final verdict. For classification tasks—like the text classification scenario we're addressing here—the aggregation occurs by taking the mode of the predictions from each model. This means we look for the most frequently predicted category across all models for any given observation. The beauty of Bagging lies in its ability to enhance model robustness by diminishing overfitting risks, effectively reducing variance without significantly increasing bias.
In text classification tasks, using Bagging can lead to marked improvements in model performance. By applying Bagging to our text data, we increase the predictive generalization capabilities of our model. Let's embark on this journey and put Bagging into action with text data, focusing on its mechanism and benefits in the sections to come.
Let's start by loading our dataset. We'll be using the Reuters-21578 Text Categorization Collection, a widely-used text dataset for document categorization and classification tasks. It is available via the NLTK (Natural Language Toolkit) library, which is the go-to library for natural language processing in Python.
Let's load the data and print the number of categories and documents:
Python1import nltk 2from nltk.corpus import reuters 3 4nltk.download('reuters', quiet=True) 5 6categories = reuters.categories()[:5] # limiting it to just 5 categories for quicker execution 7documents = reuters.fileids(categories) 8 9print(len(categories)) 10print(len(documents))
The output of the above code will be:
Plain text15 22648
This output indicates that we have limited our dataset to 5 categories, and there are a total of 2648 documents within these categories.
The Reuters-21578 dataset is a crucial resource in the field of text classification, consisting of news documents categorized by Reuters in the late 1980s. With its multitude of topics, it serves as an excellent resource for connected learning experiences related to supervised learning tasks.
Let’s delve into the dataset for an understanding of its content. We’ll look at the categories we’ve selected for this exercise and then explore the content of one document to understand its text:
Python1# Printing the categories 2print("Selected Categories:", categories) 3 4# Printing the content of one document 5doc_id = documents[0] 6print("\nDocument ID:", doc_id) 7print("Category:", reuters.categories(doc_id)) 8print("Content excerpt:\n", " ".join(reuters.words(doc_id)[:50]))
The output will be:
Plain text1Selected Categories: ['acq', 'alum', 'barley', 'bop', 'carcass'] 2 3Document ID: test/14843 4Category: ['acq'] 5Content excerpt: 6 SUMITOMO BANK AIMS AT QUICK RECOVERY FROM MERGER Sumitomo Bank Ltd & lt ; SUMI . T > is certain to lose its status as Japan ' s most profitable bank as a result of its merger with the Heiwa Sogo Bank , financial analysts said . Osaka - based
In this result, the 'acq' category signifies Acquisitions, focusing on articles about business mergers, acquisitions, and corporate deals.
Before applying any machine learning method, we first need to transform our raw text data into a format that our algorithms can work with. The CountVectorizer from the scikit-learn library offers a convenient way to both tokenize a collection of text documents and build a vocabulary of known words, as well as encode new documents using that vocabulary.
Python1from sklearn.feature_extraction.text import CountVectorizer 2from sklearn.preprocessing import LabelEncoder 3 4# Preparing the dataset 5text_data = [" ".join([word for word in reuters.words(fileid)]) for fileid in documents] 6categories_data = [reuters.categories(fileid)[0] for fileid in documents] 7 8# Using count vectorizer for feature extraction 9count_vectorizer = CountVectorizer(max_features=1000) 10X = count_vectorizer.fit_transform(text_data) 11 12# Encoding the category data 13label_encoder = LabelEncoder() 14y = label_encoder.fit_transform(categories_data) 15 16print("Categories:\n", categories_data[:5]) 17print("Encoded Categories:\n", y[:5])
The output will be:
Plain text1Categories: 2 ['acq', 'acq', 'carcass', 'bop', 'acq'] 3Encoded Categories: 4 [0 0 4 3 0]
We limit the number of features to 1000 for more sustainable computations. Feel free to experiment with this number. The encoded categories represent our categories mapped to numerical values, which makes it easier for our machine learning model to understand and process.
Following feature extraction with CountVectorizer, the variable X represents a sparse matrix of shape (number_of_documents, 1000). Each row corresponds to a document, while each column represents one of the 1000 most frequent words across all documents in our reduced dataset. In this matrix, the element at position (i, j) contains the frequency of the j-th word in the i-th document. This compact, numerical representation of our text data is what enables machine learning algorithms to process and learn from text.
As we journey deeper into ensemble learning, let's concentrate on the essence of our lesson - employing the Bagging Classifier in text classification. We're at a stage where the aim is to categorize documents based on their content. To accomplish this, we will train our model on a selected portion of our dataset, enabling it to make accurate category predictions for new, unseen documents.
Python1from sklearn.ensemble import BaggingClassifier 2from sklearn.tree import DecisionTreeClassifier 3from sklearn.model_selection import train_test_split 4 5# Split the data for training and testing 6X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1) 7 8# Initiating the BaggingClassifier with DecisionTree classifiers as the base learners 9bag_classifier = BaggingClassifier(DecisionTreeClassifier(), n_estimators=100, random_state=1) 10bag_classifier.fit(X_train.toarray(), y_train) 11 12# Generate predictions on the test data 13y_pred = bag_classifier.predict(X_test.toarray()) 14 15# Displaying the predicted category for the first document in our test set 16print("Predicted Category: ", label_encoder.inverse_transform([y_pred[0]])[0])
The predicted category for the first document in the test set is:
Plain text1Predicted Category: acq
In this context, what stands out is the Bagging method's approach to prediction. For each document in our dataset, our ensemble of Decision Trees makes individual category predictions. The Bagging algorithm then aggregates these predictions by selecting the category most frequently predicted (the mode) among all the trees for each document. This aggregation strategy, aiming to select the most common outcome, helps bolster the model's accuracy and reliability.
Finally, after the model is trained, we would like to evaluate its performance. To do that, we'll use the model to predict the labels for our test set and then print a classification report:
Python1from sklearn.metrics import classification_report 2 3# Checking the performance of the model on test data 4y_pred = bag_classifier.predict(X_test.toarray()) 5print(classification_report(y_test, y_pred, zero_division=1))
The output will be:
Plain text1 precision recall f1-score support 2 3 0 0.99 0.99 0.99 601 4 1 0.82 0.93 0.87 15 5 2 1.00 1.00 1.00 12 6 3 0.91 0.95 0.93 22 7 4 0.90 0.75 0.82 12 8 9 accuracy 0.99 662 10 macro avg 0.93 0.93 0.92 662 11weighted avg 0.99 0.99 0.99 662
This classification report summarizes the precision, recall, and F1-score for each category in our test dataset. High precision and recall values indicate our Bagging Classifier model performed exceptionally well, demonstrating the effectiveness of ensemble methods in text classification tasks.
Leveraging the concept of ensemble methods and specifically Bagging, you've successfully applied an advanced classification technique to textual data. You learned about the importance of feature extraction and used sklearn's CountVectorizer to convert text data into numerical features. You applied a Bagging Classifier with Decision Trees as base estimators in a text classification task. Furthermore, you understood how to evaluate your model using a classification report and deal with potential division by zero issues.
In the upcoming exercises, you'll get a chance to apply what you've learned and reinforce these concepts. Happy coding!