Lesson 4

Welcome! In today's lesson, you'll be diving into the world of data cleaning to learn how to **Detect and Handle Outliers** using the Diamonds dataset from the seaborn library. Outliers can significantly affect the quality of your data analysis and models, so it's crucial to identify and manage them correctly.

By the end of this lesson, you'll be able to identify outliers using boxplots and remove them using interquartile range (`IQR`

) thresholds.

- Understanding Outliers
- Identifying Outliers using
`IQR`

- Visualizing Outliers with Boxplots
- Removing Outliers from the Dataset
- Verifying the Cleaning Process

First, let's define what an outlier is in the context of data analysis.

**Outliers** are data points that differ significantly from other observations. These can be errors in data, variability in measurement, or they may indicate a varying characteristic you might need to explore.

Handling outliers is critical because they can distort statistical analyses and models. For example, extreme values can skew the mean and standard deviation of your dataset, leading to inaccurate conclusions and poor model performance.

In simple terms, imagine if you were analyzing the average height of a population and included some incorrect measurements that were twice or half the normal height. Your analysis would be misleading.

Next, we will identify the outliers using the Interquartile Range (`IQR`

) method.

**What is IQR?**

The `IQR`

is a measure of statistical dispersion, which represents the middle 50% of the data. It is calculated as the difference between the third quartile (Q3) and the first quartile (Q1). Quartiles divide a ranked dataset into four equal parts.

**Q1 (First Quartile)**: This is the median of the first half of the dataset (25th percentile).**Q3 (Third Quartile)**: This is the median of the second half of the dataset (75th percentile).**IQR**: This is the difference between the third quartile (Q3) and the first quartile (Q1). It represents the range within which the central 50% of the values lie (IQR = Q3 - Q1).

**Why use IQR for detecting outliers?**

Using `IQR`

helps to define the range within which the most typical values fall. Values that lie significantly outside this range can be considered potential outliers. Specifically, an outlier is defined as a data point that lies below `Q1 - 1.5 * IQR`

or above `Q3 + 1.5 * IQR`

.

Let's calculate the quartiles and the `IQR`

.

Python`1import seaborn as sns 2 3diamonds = sns.load_dataset('diamonds') 4 5# Calculate Q1 (25th percentile) and Q3 (75th percentile) 6Q1 = diamonds['price'].quantile(0.25) 7Q3 = diamonds['price'].quantile(0.75) 8IQR = Q3 - Q1 9 10# Define the lower and upper threshold for outliers 11lower_bound = Q1 - 1.5 * IQR 12upper_bound = Q3 + 1.5 * IQR 13 14# Print the results 15print(f"Q1 (25th percentile): {Q1}") 16print(f"Q3 (75th percentile): {Q3}") 17print(f"IQR (Interquartile Range): {IQR}") 18print(f"Lower Bound: {lower_bound}") 19print(f"Upper Bound: {upper_bound}")`

Here, `Q1`

and `Q3`

represent the 25th and 75th percentiles of the `price`

column, respectively. The thresholds will help us identify outliers.

The output of the above code will be:

Plain text`1Q1 (25th percentile): 950.0 2Q3 (75th percentile): 5324.25 3IQR (Interquartile Range): 4374.25 4Lower Bound: -5556.375 5Upper Bound: 11830.625`

This output shows the calculation of the quartiles, the `IQR`

, and the thresholds for identifying outliers in the diamonds dataset. It provides a clear numerical basis for filtering outliers from the data.

To better understand outliers in the Diamonds dataset, let's use a boxplot to visualize the `price`

column.

Boxplots are an effective tool for visualizing outliers because they succinctly display the distribution of the data. The box represents the interquartile range (IQR), with the line inside the box indicating the median. The "whiskers" extend to 1.5 times the IQR from Q1 and Q3, and any points outside this range are considered outliers.

Here's how to create a boxplot using the seaborn library:

Python`1import matplotlib.pyplot as plt 2import seaborn as sns 3 4# Load the Diamonds dataset 5diamonds = sns.load_dataset('diamonds') 6 7# Create a boxplot for the price column to visualize outliers 8plt.figure(figsize=(8, 6)) 9sns.boxplot(x=diamonds['price']) 10plt.title('Boxplot of Diamond Prices Showing Outliers') 11plt.xlabel('Price') 12plt.show()`

Running this code will generate a boxplot that highlights the outliers in the `price`

column, showing points that fall outside the whiskers.

Once we have the thresholds, we can filter the dataset to remove these outliers.

Python`1# Filter out the outliers 2diamonds_no_outliers = diamonds[(diamonds['price'] >= lower_bound) & (diamonds['price'] <= upper_bound)]`

This will keep only the rows where the price is within the lower and upper bounds, effectively removing the outliers.

Finally, it's essential to verify that our dataset is correctly cleaned and no critical data was lost.

We will use the `info()`

method to check the dataset:

Python`1# Display dataset information to verify outliers are removed 2print(diamonds_no_outliers.info())`

The output of the above code will be:

Plain text`1<class 'pandas.core.frame.DataFrame'> 2Index: 50400 entries, 0 to 53939 3Data columns (total 10 columns): 4 # Column Non-Null Count Dtype 5--- ------ -------------- ----- 6 0 carat 50400 non-null float64 7 1 cut 50400 non-null category 8 2 color 50400 non-null category 9 3 clarity 50400 non-null category 104 depth 50400 non-null float64 115 table 50400 non-null float64 126 price 50400 non-null int64 137 x 50400 non-null float64 148 y 50400 non-null float64 159 z 50400 non-null float64 16dtypes: category(3), float64(6), int64(1) 17memory usage: 3.2 MB 18None`

This output confirms that after removing outliers, the dataset contains 50400 entries, ensuring that no critical data was lost during the cleaning process.

In this lesson, you learned how to detect and handle outliers using the Diamonds dataset. You visualized outliers with boxplots, identified them using the `IQR`

method, and removed them from the dataset.

**Next Steps:**
In the upcoming practice exercises, you'll apply these techniques to different datasets and scenarios. Detecting and handling outliers is crucial for data quality and analysis accuracy, and mastering this skill will greatly enhance your data science projects.

Now, it's time to put this knowledge into practice!