Hello! In today's lesson, we're diving into the world of error handling in web scraping. Error handling is crucial because it helps ensure that your scraping scripts run smoothly, even when they encounter issues such as HTTP errors, parsing errors, or missing data.

Before we begin, let's understand the common types of errors you might encounter while scraping the web:

1. HTTP Errors: These occur when there's a problem with fetching the web page, such as a 404 Not Found error or a 500 Internal Server Error.
2. Parsing Errors: These arise when the HTML content is malformed or unexpected, causing issues during parsing.
3. Missing Data/Attributes: Sometimes, the necessary HTML elements or attributes may be missing, leading to errors.

By handling these issues, you can build robust and reliable web scraping scripts that continue to perform well even in the face of challenges.

HTTP errors occur when there is a problem with the request made to the server. Common HTTP status codes include:

• 200 OK: The request was successful.
• 404 Not Found: The requested resource could not be found.
• 500 Internal Server Error: The server encountered an unexpected condition.

Handling these errors gracefully is essential to ensure your script can proceed effectively or log meaningful error messages.

In Python, the requests library makes it simple to handle HTTP errors using the response.raise_for_status() method. This method raises an HTTPError if the HTTP request returned an unsuccessful status code.

Here's how we can implement it:

Python
Copy to clipboard
1import requests
2
3def fetch_page(url):
4    try:
5        response = requests.get(url)
6        response.raise_for_status()  # Check for HTTP errors
7        return response.text
8    except requests.HTTPError as e:
9        print(f"HTTP error: {e}")
10        return None
11
12url = 'http://quotes.toscrape.com'
13html = fetch_page(url)
14if html:
15    print(html[:500])  # Print the first 500 characters of the page content
16else:
17    print("An HTTP Error Occurred")

In this code:

• We use a try block to attempt to fetch the page content.
• The response.raise_for_status() method checks for HTTP errors.
• In the except block, we catch requests.HTTPError and print an error message if an error occurs.

Parsing errors can occur if the HTML content is malformed or unexpected. By using try and except blocks, you can handle these errors gracefully.

Here's an example using BeautifulSoup to parse HTML content and extract quotes from a webpage:

Python
Copy to clipboard
1from bs4 import BeautifulSoup
2import requests
3
4html = fetch_page('http://quotes.toscrape.com/')
5
6def parse_and_extract_quotes(html):
7    try:
8        soup = BeautifulSoup(html, 'html.parser')
9        quotes = soup.find_all('div', class_='quote')
10        print(f'Found {len(quotes)} quotes')
11    except Exception as e:
12        print(f"Parsing error: {e}")
13
14if html:
15    parse_and_extract_quotes(html) # Will print the number of quotes found
16
17parse_and_extract_quotes({}) # Will raise a parsing error

This code demonstrates how to handle parsing errors when using BeautifulSoup. The try block attempts to parse the HTML content and extract quotes. If an error occurs during parsing, the except block catches the exception and prints an error message.

Attribute errors occur when an expected HTML element or attribute is missing. For instance, if a span tag with the class text is not found, an AttributeError will be raised.

We can use try and except blocks to handle missing attributes gracefully. Here's how:

Python
Copy to clipboard
1from bs4 import BeautifulSoup
2import requests
3
4html = fetch_page('http://quotes.toscrape.com/')
5
6def parse_and_extract_quotes(html):
7    try:
8        soup = BeautifulSoup(html, 'html.parser')
9        quotes = soup.find_all('div', class_='quote')
10        quote = quotes[0]
11        try:
12            text = quote.find('span', class_='text').get_text()
13            author = quote.find('small', class_='author').get_text()
14            tags = [tag.get_text() for tag in quote.find_all('a', class_='tag')]
15            invalid_attribute = quote.find('invalid', class_='invalid').get_text() # This will raise an AttributeError
16            print(text, author, tags, invalid_attribute)
17        except AttributeError as e:
18            print(f"Attribute error: {e}")
19    except Exception as e:
20        print(f"Parsing error: {e}")
21
22if html:
23    parse_and_extract_quotes(html)

In this code:

• The inner try block attempts to extract the text, author, and tags from each quote which are expected attributes. However, it also tries to extract an invalid attribute that doesn't exist.
• The except AttributeError block catches any missing attribute errors and logs the error message.

In this case, we catch the AttributeError and print an error message. This helps us identify and handle missing attributes without causing the script to crash.

In this lesson, we covered the basics of error handling in web scraping. We discussed how to handle HTTP errors, parsing errors, and missing attributes gracefully. By now, you should feel comfortable handling various issues that may arise during web scraping. This will make your scripts more robust and reliable.

Keep practicing these concepts to master error management in web scraping. Happy scraping!