Lesson 2

Mastering Gradient Descent: Your Guide to Optimizing Machine Learning Models


Hello learners! Until now, we have delved deep into the world of supervised machine learning, using the Wine Quality Dataset as a primary resource. As we proceed, we plan to illuminate the inner workings of the learning process within machine learning models, particularly Gradient Descent.

Gradient Descent is a cornerstone of optimization in machine learning and deep learning. Its function enables the machine learning model to 'learn,' thereby improving itself based on its past performance. As we peel back layers of this lesson, we promise you a more profound understanding of Gradient Descent, its role in machine learning, and its implementation with Python. Buckle up for an exciting educational journey!

Gradient Descent Demystified

Have you ever hiked to the top of a hill and looked down to determine the best route of descent? One potentially disastrous step off a steep cliff is dangerous, while cautiously descending the gentle slopes might cause less harm. The concept of Gradient Descent mirrors this scenario — it, too, sees the value in finding and taking the optimal path or, more precisely, reaching the minimum point.

In machine learning, Gradient Descent can be visualized as a careful navigation downwards until we find the valley between hills. The 'hill' in this context is the cost function, which quantifies our model's error. Through a series of small steps, Gradient Descent refines the cost function by 'walking' down the hill towards the steepest descent until it reaches the lowest possible point at its optimal state.

Mathematics Behind Gradient Descent

Having conceptualized Gradient Descent, let’s delve deeper and uncover the mathematical mechanics that fuel it. At its core, Gradient Descent relies on two key mathematical mechanisms: the Cost Function and the Learning Rate.

The Cost Function (or Loss Function) quantifies the disparity between predicted and expected values, presenting it as a single float number. The type of cost function utilized depends on the challenge at hand. In our Wine Quality dataset, we can define a cost function that computes the difference between our model's predicted quality of wine and the actual quality.

The Learning Rate, symbolized by α\alpha, dictates the size of the steps we take downhill. A lower value of α\alpha results in smaller, more precise steps, while a high value could cause drastic, potentially unstable steps.

From our previous analogy, imagine the hill is symbolized by a function of position, g(x)g(x). Starting at the hill's pinnacle (x0x_0), we revise our position (xx) by moving a step proportional to the negative gradient at that location. The gradient g(x)g'(x) is simply the derivative of g(x)g(x), pointing toward the steepest ascent. Conversely, g(x)-g'(x) signifies the fastest descending path. We repeat this stepping process until the gradient becomes zero at the minimum point, indicating no further downhill path, i.e., no additional optimization is required.

Advancements in Gradient Descent

Here, an interesting question arises, "Do we always use all data to calculate the gradient?" The answer depends. Gradient Descent has evolved into various versions, depending on the amount of data used in computing the gradient: batch, stochastic, and mini-batch gradient descent.

The original version, batch gradient descent, uses the complete dataset at every step. While this may seem meticulous and comprehensive, it proves extremely inefficient when dealing with substantial datasets housing millions of entries. Imagine watching a movie frame by frame at a snail's pace — it can be painstakingly slow despite its precision.

Implementing Gradient Descent

Now, let's make the Gradient Descent implementation in Python. We start by assigning random values to our model’s parameters. Gradual adjustments to these parameters follow, in each instance computing the cost function, our error, and taking a step towards the steepest slope until our error is minimal or the state is optimized.

Here’s a general outline of how we would implement gradient descent in Python:

1def gradient_descent(x, y, theta, alpha, iterations): 2 """ 3 x -- input dataset 4 y -- target dataset 5 theta -- initial parameters 6 alpha -- learning rate 7 iterations -- the number of times to execute the algorithm 8 """ 9 10 m = y.size # number of data points 11 cost_list = [] # list to store the cost function value at each iteration 12 theta_list = [theta] # list to store the values of theta at each iteration 13 14 for i in range(iterations): 15 # calculate our prediction based on our current theta 16 prediction = np.dot(x, theta) 17 18 # compute the error between our prediction and the actual values 19 error = prediction - y 20 21 # calculate the cost function 22 cost = 1 / (2*m) * np.dot(error.T, error) 23 24 # append the cost to the cost_list 25 cost_list.append(np.squeeze(cost)) 26 27 # calculate the gradient descent and update the theta 28 theta = theta - (alpha * (1/m) * np.dot(x.T, error)) 29 30 # append the updated theta to the theta_list 31 theta_list.append(theta) 32 33 # return the final values of theta, list of all theta, and list of all costs, respectively 34 return theta, theta_list, cost_list

In this code snippet, x represents your input dataset, y is your target dataset, theta indicates your initialized parameters, alpha is your learning rate, and iterations denotes the number of times the optimization algorithm executes to fine-tune the parameters.

Gradient Descent for Wine Quality Prediction: A Hands-On Application

Are you eager to see Gradient Descent in action? Let’s apply it to the Wine Quality Dataset. Using the cost function that computes the error between the actual and predicted wine quality, we can represent this error as a 'hill.' As we journey further into the hill, our error diminishes, optimizing the model's prediction accuracy for wine quality.

Let's focus on one feature for simplicity's sake: alcohol. We will use Python to demonstrate how Gradient Descent can design a model that predicts wine quality based on its alcohol content.

1import numpy as np 2import pandas as pd 3from sklearn.model_selection import train_test_split 4import matplotlib.pyplot as plt 5import datasets 6 7# Load Wine Quality Dataset 8red_wine = datasets.load_dataset('codesignal/wine-quality', split='red') 9red_wine = pd.DataFrame(red_wine) 10 11# Only consider the 'alcohol' column as a predictive feature for now 12x = pd.DataFrame(red_wine['alcohol']) 13y = red_wine['quality'] 14 15# Splitting datasets into training and testing datasets 16x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.3, random_state=0) 17 18# We set our parameters to start at 0 19theta = np.zeros(x_train.shape[1]).reshape(-1, 1) 20 21# Define the number of iterations and alpha value 22alpha = 0.0001 23iters = 1000 24 25# Applying Gradient Descent 26y_train = np.array(y_train).reshape(-1, 1) 27g, theta_list, cost_list = gradient_descent(x_train, y_train, theta, alpha, iters) 28 29print(cost_list) 30plt.plot(range(1, iters + 1), cost_list, color='blue') 31plt.rcParams["figure.figsize"] = (10,6) 32plt.grid() 33plt.xlabel('Number of iterations') 34plt.ylabel('Cost (J)') 35plt.title('Convergence of gradient descent') 36plt.show()


In this code, we first extricated the predictor, alcohol, from our Wine Quality Dataset and proceeded to run our Gradient Descent function. In the output, you can see the cost function reducing with each iteration, depicting how Gradient Descent gradually descends the hill, alleviating the cost function and thus enhancing our model's predictions.

Assessing Gradient Descent Performance

The learning rate (alpha) is a critical component in the performance of gradient descent. Striking the right balance can be delicate: if alpha is too large, we might overshoot our optimal point, while if it's too small, we might require an excessive number of iterations to converge, or we might not converge at all.

While this can be adjusted in our code as per requirement, we will later discuss how the ideal alpha is determined empirically by testing various alpha values, leading to the best model performance.


Finally, we've traversed the heart of Gradient Descent, decoded its mathematical interpretation, implemented it with Python, and applied it to our Wine Quality Dataset. Moreover, we spent some time comprehending the significance of the learning rate and how it impacts our model's predictions.

Grasping that gradient descent is a fundamental aspect of machine learning, and artificial intelligence becomes crucial as it enables our models to learn from data. As the cornerstone for optimizing our machine learning algorithms, understanding how it works provides a deeper insight into the intricacies of training a machine learning model.

Practice Is Key!

As we proceed to the practice exercises, remember that Gradient Descent might seem overwhelming initially. Nevertheless, the best antidote to any such overwhelming feeling is practice. As you engage earnestly with the exercises, you'll become adept at Gradient Descent, forming the bedrock of your journey into machine learning. Let's get started, learners!

Enjoy this lesson? Now it's time to practice with Cosmo!

Practice is how you turn knowledge into actual skills.