Hi there! Today, we're diving into a significant concept in machine learning known as cross-validation. Imagine you're baking a cake. You wouldn't just taste one slice, right? You'd want to taste slices from different parts to ensure they are evenly good. That's what cross-validation does for machine learning models. It ensures our models work well on different sections of the data.

By the end of this lesson, you'll understand cross-validation, perform it using Scikit-Learn, and interpret the results. Let's get started!

What is cross-validation?

Cross-validation evaluates a machine learning model by splitting the data in multiple ways. Instead of just one split into training and testing sets, we split it multiple times, each time in a different way, and train and test the model on these splits. This gives a more reliable performance estimate.

Think of it like trying different slices of your cake to ensure it's consistently good.

In cross-validation, a fold refers to a single iteration of splitting the data into training and validation sets. For example, in 5-fold cross-validation, the entire dataset is divided into 5 parts (called folds). Each fold takes a turn being the validation set while the remaining folds together form the training set. This process repeats 5 times.

Let's see how to do this in Python.

First, we need a real-world dataset. We'll use the "wine dataset" from Scikit-Learn.

Python
Copy to clipboard
1from sklearn.datasets import load_wine
2from sklearn.preprocessing import StandardScaler
3
4# Load the wine dataset
5X, y = load_wine(return_X_y=True)
6X = StandardScaler().fit_transform(X)

Here, X contains the features (input data), and y contains the target (output labels). Note that we scale the features to improve the model's convergence.

Next, we'll split the data into training and testing sets. Even when using cross-validation, it's essential to hold back a portion of the data for final testing. Cross-validation will be performed only on the training data.

Python
Copy to clipboard
1from sklearn.model_selection import train_test_split
2
3# Split the data into training and testing sets
4X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

And we'll use DecisionTreeClassifier, a simple yet well-performing model.

Python
Copy to clipboard
1from sklearn.tree import DecisionTreeClassifier
2
3# Create a Decision Tree model
4decision_tree = DecisionTreeClassifier(random_state=42)

Now, let's perform cross-validation on the training data.

Python
Copy to clipboard
1from sklearn.model_selection import cross_val_score
2
3# Perform 5-fold cross-validation on the training data
4scores = cross_val_score(decision_tree, X_train, y_train, cv=5)
5print(f"Cross-validation scores: {scores}") 
6print(f"Mean cross-validation score: {scores.mean():.2f}") 
7# Cross-validation scores: [0.93103448 0.93103448 0.89285714 0.92857143 0.89285714]
8# Mean cross-validation score: 0.92

The cross_val_score function splits the training data into 5 parts (using cv=5), trains the model on 4 parts, and tests it on the remaining part. This is repeated 5 times.

Each time, we get a score that shows the model's performance. Finally, we print these individual scores and their average.

The scoring parameter in cross_val_score allows you to specify the metric used to evaluate the model's performance. By default, it uses the scoring method based on the model type (e.g., accuracy for classification models). However, you can specify other metrics such as 'f1', 'precision', 'recall', etc.

For example, to use F1-score, you can modify the cross_val_score function call as follows:

Python
Copy to clipboard
1scores = cross_val_score(decision_tree, X_train, y_train, cv=5, scoring='f1_weighted')

This flexibility allows you to choose the metric that best aligns with your model's performance goals.

After performing cross-validation, it's crucial to evaluate the model on the test set to get an unbiased estimate of its final performance.

Python
Copy to clipboard
1# Train the model on the entire training data
2decision_tree.fit(X_train, y_train)
3
4# Evaluate the model on the test data
5test_score = decision_tree.score(X_test, y_test)
6print(f"Test score: {test_score:.2f}") 
7# Test score: 0.94

Here, we fit the model on the entire training data and then evaluate it on the test data to see how well it generalizes to unseen data.

Let's look at the output:

Plain text
Copy to clipboard
1Cross-validation scores: [0.909 0.896 0.909 0.909 0.85 ]
2Mean cross-validation score: 0.89
3Test score: 0.94

These scores show the model's performance on different data parts. It's like tasting various slices of the cake.

The mean score gives an overall performance measure. It's like averaging the taste scores from different slices.

A mean cross-validation score of 0.89 means our Decision Tree model correctly predicts about 89% of the time on average during cross-validation. The test score of 0.94 indicates that the model performs better, predicting correctly 94% of the time on unseen test data.

Great job! Today, we've covered:

1. What is Cross-Validation?

   • A method to ensure our machine learning model performs well on different data parts.
   • Introduction to folds in cross-validation.

2. How to Perform Cross-Validation Using Scikit-Learn

   • We used Python to load a dataset, split it into training and testing sets, create a Decision Tree model, and perform cross-validation only on the training set.
   • Explanation of the scoring parameter in cross-validation.

3. Evaluating the Model on the Test Set

   • We trained the model on the entire training data and evaluated its performance on the test data to ensure it generalizes well.

4. Interpreting the Results

   • We learned how to read individual scores and calculate the mean score to understand the model's performance.

Now it's your turn! You'll get hands-on experience with cross-validation in the upcoming practice. You'll use different models and datasets to see how cross-validation helps ensure your machine learning models are reliable and performant. Ready to give it a try? Let's get started!