Hello and welcome! In today's lesson, we'll explore techniques to prevent overfitting in neural networks using PyTorch. You will learn about regularization and dropout methods, specifically L2 Regularization and Dropout, and how to implement these techniques in PyTorch to enhance your model's performance.

To demonstrate overfitting prevention, we'll use the wine dataset, a classic dataset in machine learning. Let's load the data using the load_wine function from sklearn.datasets.

Here's the code to load and select the data:

Python
Copy
1from sklearn.datasets import load_wine
2import torch
3
4# Load and select the data
5wine = load_wine()
6X = wine.data
7y = wine.target
8
9# Convert to PyTorch tensors
10X = torch.tensor(X, dtype=torch.float32)
11y = torch.tensor(y, dtype=torch.long)

This code loads the wine dataset, attributes features and targets to different variables, and finally converts the data to PyTorch tensors.

Regularization is a technique used to prevent overfitting in machine learning models. Overfitting occurs when a model learns not only the underlying patterns in the training data but also the random noise, leading to poor performance on new, unseen data. Regularization techniques add additional constraints or penalties to the model's learning process, encouraging it to focus on the most important patterns in the data and improving its generalization ability.

Dropout is a regularization technique used to prevent overfitting in neural networks. During training, dropout layers randomly set a fraction of activations in the preceding layer to zero at each update. This prevents the network from becoming too dependent on any particular neurons, thereby promoting generalization.

The fraction of activations set to zero is determined by the dropout rate, typically a value between 0 and 1. For instance, a dropout rate of 0.2 implies that 20% of the neurons will be randomly dropped during each iteration of training. This technique helps in creating a more robust model that performs better on unseen data.

Now, let's define a neural network model using PyTorch. We'll build the model using nn.Sequential and focus specifically on adding dropout layers to reduce overfitting.

In the following code:

• nn.Linear: Defines a fully connected layer.
• nn.ReLU: Adds activation functions to introduce non-linearities.
• nn.Dropout: Adds dropout layers with rates of 20% and 10%, respectively. Each dropout layer randomly sets the specified proportion of nodes in the previous layer to zero during training.

Python
Copy
1import torch.nn as nn
2
3# Define the model with dropout
4model = nn.Sequential(
5    nn.Linear(13, 10),
6    nn.ReLU(),
7    nn.Dropout(0.2),  # Dropout applied to the previous layer
8    nn.Linear(10, 10),
9    nn.ReLU(),
10    nn.Dropout(0.1),  # Dropout applied to the previous layer
11    nn.Linear(10, 3)
12)
13
14# Print the model summary
15print(model)

The output shows the structure of our defined neural network, including layers, activation functions, and dropout rates:

Plain text
Copy
1Sequential(
2  (0): Linear(in_features=13, out_features=10, bias=True)
3  (1): ReLU()
4  (2): Dropout(p=0.2, inplace=False)
5  (3): Linear(in_features=10, out_features=10, bias=True)
6  (4): ReLU()
7  (5): Dropout(p=0.1, inplace=False)
8  (6): Linear(in_features=10, out_features=3, bias=True)
9)

A common regularization technique is called L2 regularization, which penalizes large weights by adding their squared values to the loss function. This discourages the model from relying too heavily on any particular feature, thereby enhancing generalization. In PyTorch, L2 regularization is typically implemented by specifying the weight_decay parameter in the optimizer.

In the following code we will start training a model without regularization and introduce it midway to observe its effect. Here, regularization is applied by adding weight decay to our criterion. We are also going to print the L2 norm of the weights in the first layer at different epochs to observe how they evolve over time. The L2 norm is a measure of the magnitude of the weights, calculated as the square root of the sum of the squared weights. It gives an indication of how large the weights are.

Python
Copy
1import torch.optim as optim
2
3# Defining criterion and optimizer without weight decay
4criterion = nn.CrossEntropyLoss()
5optimizer = optim.Adam(model.parameters(), lr=0.001)  
6
7for i in range(100):
8    model.train()
9    optimizer.zero_grad()  # Zero the gradient buffers
10    outputs = model(X)  # Forward pass
11    loss = criterion(outputs, y)  # Compute loss
12    loss.backward()  # Backward pass
13
14    if(i==50):
15        # Introducing weight decay from 50th epoch on
16        optimizer = optim.Adam(model.parameters(), lr=0.001, weight_decay=0.01) 
17        print("\nRegularization added to optimizer\n")
18
19    if (i+1) % 10 ==0:    
20        # L2 norm of weights of the first linear layer 
21        first_layer_weights = model[0].weight.norm(2).item()
22        print(f'{i+1} - L2 norm of weights: {first_layer_weights}')
23
24    optimizer.step()  # Update weights

In this code:

• Initially, the optimizer is set up without any L2 regularization.
• At epoch 50, the optimizer is updated to include L2 regularization by adding a weight_decay of 0.1. This introduces a penalty to the weights, helping prevent overfitting.
• The L2 norm of the weights in the first linear layer is printed every 10 epochs to monitor their magnitude.

Here's how the output of the previous code might look like:

Plain text
Copy
110 - L2 norm of weights: 1.8384225368499756
220 - L2 norm of weights: 1.8378220796585083
330 - L2 norm of weights: 1.8378770351409912
440 - L2 norm of weights: 1.839258074760437
550 - L2 norm of weights: 1.8409700393676758
6
7Regularization added to optimizer
8
960 - L2 norm of weights: 1.784838318824768
1070 - L2 norm of weights: 1.725461721420288
1180 - L2 norm of weights: 1.669530987739563
1290 - L2 norm of weights: 1.6180046796798706
13100 - L2 norm of weights: 1.570449948310852

Notice how the L2 norm of weights in the first layer evolves over epochs. Initially, the L2 norm increases slightly, indicating that the weights are growing. After reaching epoch 50, the L2 norm starts to decrease, showing the effect of the stronger L2 regularization by controlling the magnitude of the weights. This adaptive approach helps to ensure the model does not overfit.

In this lesson, we've explored techniques like L2 Regularization and Dropout to prevent overfitting. We built a neural network model using PyTorch, applied these techniques, and trained the model to see their effects.

Practice exercises will help solidify these concepts by allowing you to implement and experiment with these techniques using different settings. These skills are crucial for building robust and generalizable machine learning models in real-world applications.

Happy coding, and keep exploring the power of PyTorch!