Welcome to your lesson on Basic TF-IDF Vectorization! In the world of Natural Language Processing (NLP), converting text into numerical representation such as vectors is crucial. Today, we will explore one of the popular techniques, Term Frequency-Inverse Document Frequency (TF-IDF), and see how we can implement it in Python using the scikit-learn library. By the end of the lesson, you will know how to vectorize a real-world text dataset, the SMS Spam Collection.

In the era of data-driven decision making, much of the valuable information comes in textual form — think about social media posts, medical records, news articles. Text mining is the practice of deriving valuable insights from text. However, machine learning algorithms, which help process and understand this text, usually require input data in numerical format.

Here is where text vectorization steps in, converting text into a set of numerical values, each corresponding to a particular word or even a group of words. This conversion opens the doors for performing various operations such as finding similarity between documents, document classification, sentiment analysis, and many more.

Term Frequency-Inverse Document Frequency (TF-IDF) is a statistical measure used to evaluate the importance of a word in the context of a set of documents, known as a corpus. Its importance increases proportionally to the number of times a word appears in the document but is offset by the frequency of the word in the corpus.

1. Term Frequency (TF): It measures the frequency of a word's occurrence in a document. Different ways to calculate TF include:

   • Raw Count: The basic measure where TF is the number of times the word appears in a document.
   • Frequency Adjusted for Document Length: The raw count of occurrences divided by the number of words in the document. For example, if the word 'learn' appears 5 times in a document of 100 words, the TF for 'learn' is 5/100 = 0.05.
   • Logarithmically Scaled Frequency: To diminish the importance of frequency, log scaling is applied, e.g., log(1 + raw count). This approach ensures that words that appear more frequently do not excessively dominate the vector representation of the document.
   • Boolean Frequency: Where the TF is set to 1 if the term occurs in the document, and 0 if it does not, disregarding how many times the term appears.

2. Inverse Document Frequency (IDF): It measures the importance of a word in the entire corpus. The beauty of IDF is that it diminishes the weight of terms that occur very frequently in the data set and increases the weight of terms that occur rarely. Traditionally, the IDF of a word 'learn', for example, is calculated using the formula:

$IDF(t) = log\left(\frac{N}{df_t}\right)$

where:

• $N$ is the total number of documents in the corpus.
• $df_t$ is the number of documents in the corpus that contain the term (t).

However, this straightforward calculation might lead to a division-by-zero issue if a term does not exist in any document. To avoid this, scikit-learn applies a modified version of the formula:

$IDF(t) = log\left(\frac{1 + N}{1 + df_t}\right) + 1$

This adjusted formula ensures no term ends up with an IDF of zero by adding 1 to both the total number of documents and the document frequency of the term. Such an approach not only prevents mathematical errors but also reflects an intuitive understanding: every term should have a minimum level of importance simply by existing in the language used across the corpus.

Finally, the TF-IDF value for a word in a document is calculated as the product of its TF (with one of the variations mentioned above chosen based on the application) and its IDF. This balanced mix serves to highlight words that are uniquely significant to individual documents, providing a robust way to represent text data numerically for various machine learning and natural language processing applications.

TfidfVectorizer, offered by the sklearn.feature_extraction.text module, is a comprehensive tool for transforming text into meaningful numerical representation. It not only calculates TF-IDF scores but also streamlines text preprocessing by:

• Automatically tokenizing the text, which involves separating the text into tokens or words.
• Converting text to lowercase by default, ensuring uniformity across the dataset.
• Removing punctuation, as it considers sequences of alphanumeric characters as tokens and discards other characters, like punctuation marks.
• Optionally removing stop words if specified, to filter out common words that might not contribute significant meaning to the document's context.

The tokenization process thus plays a critical role in filtering and preparing the text data, making it ready for vectorization without extensive manual preprocessing.

Example of using TfidfVectorizer:

Python
Copy to clipboard
1# Example corpus
2corpus = ['The car is driven on the road.',  # Note the punctuation
3          'The bus is driven on the street!']
4
5from sklearn.feature_extraction.text import TfidfVectorizer
6vectorizer = TfidfVectorizer()
7
8# Vectorize the corpus, including lowercasing, tokenization and punctuation removal
9X_tfidf = vectorizer.fit_transform(corpus)
10
11# Display the feature names and the TF-IDF matrix
12print(vectorizer.get_feature_names_out())
13print(X_tfidf.toarray())

In the output, the results form a matrix where each row represents a different document from our corpus, and each column corresponds to a unique token identified across all documents. Important to note, punctuation has been removed, ensuring our focus is solely on the words themselves:

Plain text
Copy to clipboard
1['bus' 'car' 'driven' 'is' 'on' 'road' 'street' 'the']
2
3[[0. 0.42471719 0.30218978 0.30218978 0.30218978 0.42471719 0. 0.60437955]
4 [0.42471719 0. 0.30218978 0.30218978 0.30218978 0. 0.42471719 0.60437955]]

The TfidfVectorizer's ability to remove punctuation and tokenize the text simplifies the preprocessing requirements, making it straightforward to turn raw text into a format that's ready for analysis or machine learning models.

Let's apply TF-IDF to the SMS Spam Collection text data:

Python
Copy to clipboard
1import pandas as pd
2from datasets import load_dataset
3from sklearn.feature_extraction.text import TfidfVectorizer
4
5# Load the dataset
6sms_spam = load_dataset('codesignal/sms-spam-collection')
7
8# Convert to Dataframe
9df = pd.DataFrame(sms_spam['train'])
10
11# Initialize a TF-IDF Vectorizer
12vectorizer = TfidfVectorizer()
13
14# Convert the 'message' column to TF-IDF vectors
15X_tfidf = vectorizer.fit_transform(df['message'])
16
17print(X_tfidf.shape)

The output of the above code will be:

Plain text
Copy to clipboard
1(5572, 8713)

This output shows that the TF-IDF vectorization transformed the messages into a matrix with 5572 rows, each corresponding to a message, and 8713 columns, each representing a unique word. This transformation effectively converts the textual data into a numerical format suitable for machine learning algorithms.

Congratulations on completing the lesson on Basic TF-IDF Vectorization! You have taken a step further into the world of NLP by learning about TF-IDF vectorization, a popular method to transform text into numerical representation. With this knowledge, you can now prepare your textual data for machine learning algorithms. Remember, the more you practise, the stronger your understanding will be. So dive into the exercises and keep learning!