Hello and welcome! In this lesson, we will focus on handling pagination in web scraping using Python
and Beautiful Soup
. Pagination is essential when scraping large datasets from websites that display their content over multiple pages. By the end of this lesson, learners will be equipped with the skills to navigate multiple web pages, extract necessary data, and handle pagination effectively.
Pagination is a web design technique used to divide extensive content into multiple pages, commonly seen in search results, blogs, and forums. Each page shows a subset of the total data, and navigation links (typically labeled "Next", "Previous", or page numbers) let users move through the data.
Understanding pagination is essential for effective web scraping since it allows you to gather comprehensive datasets.
Let's consider a scenario where we scrape quotes from a website that paginates its content. The website displays quotes on multiple pages, with a "Next" button to navigate to the next page. The following code demonstrates how to scrape quotes from multiple pages:
Python1import requests 2from bs4 import BeautifulSoup 3 4base_url = 'http://quotes.toscrape.com' 5next_url = '/page/1/' 6while next_url: 7 response = requests.get(f"{base_url}{next_url}") 8 soup = BeautifulSoup(response.text, 'html.parser') 9 10 quotes = soup.find_all("div", class_="quote") 11 for quote in quotes: 12 print(quote.find("span", class_="text").text) 13 14 next_button = soup.find("li", class_="next") 15 next_url = next_button.find("a")["href"] if next_button else None
next_url
is available, extracting the next URL dynamically from the "Next" button link.This code elegantly handles pagination by recursively following "Next" links until no more pages are available.
We use soup.find_all
to locate all div
tags with class quote
. Within each quote
div, we find the span with the class text
to extract the quote text.
The output of the above code will be:
Plain text1“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.” 2“It is our choices, Harry, that show what we truly are, far more than our abilities.” 3“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.” 4...
This shows the quotes extracted from the first page, providing a base for expanding the scraping to handle pagination.
In this lesson, we explored how to handle pagination while scraping web data using Python
and Beautiful Soup
. We started with the concept of pagination, broke down the example code, and implemented a full pagination logic to scrape multiple pages.
Let's practice and reinforce the concepts learned in the lesson. Happy scraping!