Evaluating a predictive model's performance is crucial in machine learning. To understand how well our model predicts outcomes, we need effective metrics. In this lesson, we will explore three important metrics used in regression analysis: Mean Squared Error (MSE), Mean Absolute Error (MAE), and Root Mean Squared Error (RMSE). By the end of this lesson, you will know what these metrics are, how to calculate them, and understand their differences. Let's get started!

Before we dive into the specifics, let's briefly touch on regression metrics and their importance. These metrics help us quantify the difference between actual outcomes (true values) and model predictions. This allows us to assess our model's performance. The example code below shows how to calculate MSE, MAE, and RMSE.

Python
Copy to clipboard
1from sklearn.metrics import mean_squared_error, mean_absolute_error
2import numpy as np
3
4# Sample regression dataset
5y_true = np.array([3.0, -0.5, 2.0, 7.0])
6y_pred = np.array([2.5, 0.0, 2.0, 8.0])
7
8# Calculating MSE, MAE, and RMSE
9mse = mean_squared_error(y_true, y_pred)
10mae = mean_absolute_error(y_true, y_pred)
11rmse = np.sqrt(mean_squared_error(y_true, y_pred))
12
13print(f"MSE: {mse}, MAE: {mae}, RMSE: {rmse}")
14# Output: MSE: 0.375, MAE: 0.5, RMSE: 0.6123724356957945

This code calculates the three metrics for a set of true and predicted values. Let’s delve into each metric to understand their implications.

Mean Squared Error (MSE) measures the average squared difference between the actual and predicted values. The formula is:

$MSE = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$

Imagine predicting house prices: if our predictions are close to the actual values, the MSE will be low; if predictions are far off, the MSE will be high. Note that it squares the differences. Because of that, high differences will bring more value than low differences to the final metric value.

MSE is calculated by summing squared differences between true and predicted values, then dividing by the number of samples. We'll use the already-implemented mean_squared_error function from the sklearn to calculate it:

Python
Copy to clipboard
1from sklearn.metrics import mean_squared_error
2import numpy as np
3
4# True and predicted values
5y_true = [3.0, -0.5, 2.0, 7.0]
6y_pred = [2.5, 0.0, 2.0, 8.0]
7
8# Mean Squared Error
9mse = mean_squared_error(y_true, y_pred)
10print(f"MSE: {mse}")  # Output: MSE: 0.375

Mean Absolute Error (MAE) measures the average absolute difference between true and predicted values. The formula is:

$MAE = \frac{1}{n} \sum_{i=1}^{n} |y_i - \hat{y}_i|$

Unlike MSE, MAE takes the absolute value of the errors and does not exaggerate them.

MAE is calculated by summing the absolute differences between true and predicted values, then dividing by the number of samples. As well as with the MSE, sklearn has a built-in function for it:

Python
Copy to clipboard
1from sklearn.metrics import mean_absolute_error
2import numpy as np
3
4# True and predicted values
5y_true = [3.0, -0.5, 2.0, 7.0]
6y_pred = [2.5, 0.0, 2.0, 8.0]
7
8# Mean Absolute Error
9mae = mean_absolute_error(y_true, y_pred)
10print(f"MAE: {mae}")  # Output: MAE: 0.5

Root Mean Squared Error (RMSE) is the square root of the Mean Squared Error:

$RMSE = \sqrt{\frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2}$

RMSE is in the same units as the target variable, making it more interpretable.

For example, if we're predicting house prices, RMSE tells us the average error in dollars. RMSE is calculated by taking the square root of the MSE:

Python
Copy to clipboard
1from sklearn.metrics import mean_squared_error
2import numpy as np
3
4# True and predicted values
5y_true = [3.0, -0.5, 2.0, 7.0]
6y_pred = [2.5, 0.0, 2.0, 8.0]
7
8# Root Mean Squared Error
9rmse = np.sqrt(mean_squared_error(y_true, y_pred))
10print(f"RMSE: {rmse}")  # Output: RMSE: 0.6123724356957945

• MSE is better for highlighting significant errors in applications like stock price prediction because its squaring of errors penalizes larger deviations more, which is crucial for financial forecasting where large errors can be very costly.
• MAE is preferable in cases like delivery time estimates because it treats all errors equally and provides a straightforward average error, which is more reflective of typical real-world conditions without exaggerating any outlier delays.
• RMSE is ideal for when you need interpretability in the same units as the target variable, like in house price prediction, because it provides an error metric that is directly comparable to the predicted values, making it more intuitive for stakeholders to understand the model's performance.

In this lesson, we covered MSE, MAE, and RMSE, what they mean, how to compute them, and their differences. MSE emphasizes larger errors, MAE treats errors equally, and RMSE offers interpretable insights.

Now, it's your turn! You will get hands-on experience implementing these metrics using SciKit Learn. This practical exercise will reinforce your learning and help you gain deeper insights into model evaluation. Have fun!