Lesson 1

Evaluating a predictive model's performance is crucial in *machine learning*. To understand how well our model predicts outcomes, we need effective metrics. In this lesson, we will explore three important metrics used in regression analysis: **Mean Squared Error** (`MSE`

), **Mean Absolute Error** (`MAE`

), and **Root Mean Squared Error** (`RMSE`

). By the end of this lesson, you will know what these metrics are, how to calculate them, and understand their differences. Let's get started!

Before we dive into the specifics, let's briefly touch on regression metrics and their importance. These metrics help us quantify the difference between actual outcomes (true values) and model predictions. This allows us to assess our model's performance. The example code below shows how to calculate `MSE`

, `MAE`

, and `RMSE`

.

Python`1from sklearn.metrics import mean_squared_error, mean_absolute_error 2import numpy as np 3 4# Sample regression dataset 5y_true = np.array([3.0, -0.5, 2.0, 7.0]) 6y_pred = np.array([2.5, 0.0, 2.0, 8.0]) 7 8# Calculating MSE, MAE, and RMSE 9mse = mean_squared_error(y_true, y_pred) 10mae = mean_absolute_error(y_true, y_pred) 11rmse = np.sqrt(mean_squared_error(y_true, y_pred)) 12 13print(f"MSE: {mse}, MAE: {mae}, RMSE: {rmse}") 14# Output: MSE: 0.375, MAE: 0.5, RMSE: 0.6123724356957945`

This code calculates the three metrics for a set of true and predicted values. Let’s delve into each metric to understand their implications.

**Mean Squared Error** (`MSE`

) measures the average squared difference between the actual and predicted values. The formula is:

$MSE = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$

Imagine predicting house prices: if our predictions are close to the actual values, the `MSE`

will be low; if predictions are far off, the `MSE`

will be high. Note that it squares the differences. Because of that, high differences will bring more value than low differences to the final metric value.

`MSE`

is calculated by summing squared differences between true and predicted values, then dividing by the number of samples. We'll use the already-implemented `mean_squared_error`

function from the sklearn to calculate it:

Python`1from sklearn.metrics import mean_squared_error 2import numpy as np 3 4# True and predicted values 5y_true = [3.0, -0.5, 2.0, 7.0] 6y_pred = [2.5, 0.0, 2.0, 8.0] 7 8# Mean Squared Error 9mse = mean_squared_error(y_true, y_pred) 10print(f"MSE: {mse}") # Output: MSE: 0.375`

**Mean Absolute Error** (`MAE`

) measures the average absolute difference between true and predicted values. The formula is:

$MAE = \frac{1}{n} \sum_{i=1}^{n} |y_i - \hat{y}_i|$

Unlike `MSE`

, `MAE`

takes the absolute value of the errors and does not exaggerate them.

`MAE`

is calculated by summing the absolute differences between true and predicted values, then dividing by the number of samples. As well as with the MSE, `sklearn`

has a built-in function for it:

Python`1from sklearn.metrics import mean_absolute_error 2import numpy as np 3 4# True and predicted values 5y_true = [3.0, -0.5, 2.0, 7.0] 6y_pred = [2.5, 0.0, 2.0, 8.0] 7 8# Mean Absolute Error 9mae = mean_absolute_error(y_true, y_pred) 10print(f"MAE: {mae}") # Output: MAE: 0.5`

**Root Mean Squared Error** (`RMSE`

) is the square root of the Mean Squared Error:

$RMSE = \sqrt{\frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2}$

`RMSE`

is in the same units as the target variable, making it more interpretable.

For example, if we're predicting house prices, `RMSE`

tells us the average error in dollars. `RMSE`

is calculated by taking the square root of the `MSE`

:

Python`1from sklearn.metrics import mean_squared_error 2import numpy as np 3 4# True and predicted values 5y_true = [3.0, -0.5, 2.0, 7.0] 6y_pred = [2.5, 0.0, 2.0, 8.0] 7 8# Root Mean Squared Error 9rmse = np.sqrt(mean_squared_error(y_true, y_pred)) 10print(f"RMSE: {rmse}") # Output: RMSE: 0.6123724356957945`

**MSE**is better for highlighting significant errors in applications like stock price prediction because its squaring of errors penalizes larger deviations more, which is crucial for financial forecasting where large errors can be very costly.**MAE**is preferable in cases like delivery time estimates because it treats all errors equally and provides a straightforward average error, which is more reflective of typical real-world conditions without exaggerating any outlier delays.**RMSE**is ideal for when you need interpretability in the same units as the target variable, like in house price prediction, because it provides an error metric that is directly comparable to the predicted values, making it more intuitive for stakeholders to understand the model's performance.

In this lesson, we covered `MSE`

, `MAE`

, and `RMSE`

, what they mean, how to compute them, and their differences. `MSE`

emphasizes larger errors, `MAE`

treats errors equally, and `RMSE`

offers interpretable insights.

Now, it's your turn! You will get hands-on experience implementing these metrics using `SciKit Learn`

. This practical exercise will reinforce your learning and help you gain deeper insights into model evaluation. Have fun!