Scraping Yelp reviews with Python allows you to gather valuable data for market research, sentiment analysis, and competitor analysis efficiently. This guide provides a practical, step-by-step approach to help you automate the process of extracting Yelp reviews using Python.

You’ll learn about the benefits of scraping Yelp data, the necessary tools and technologies, the best paid Yelp scrapers, and how to set up your Python environment. This guide also includes practical examples and advanced techniques for handling pagination and avoiding blocks.

By the end, you’ll be equipped to scrape Yelp reviews with Python effectively.

TL; DR 

Scraping Yelp reviews with Python allows you to gather valuable data for market research, sentiment analysis, and competitor analysis efficiently. This guide provides a practical, step-by-step approach to help you automate the process of extracting Yelp reviews using Python.

• Understanding the Benefits: Scraping Yelp data provides deep insights into customer behavior, preferences, and market trends. This data can be leveraged for lead generation, reselling insights, building targeted databases, and competitor analysis.
• Best Paid Yelp Data Scrapers: Tools like V6Proxies, Apify, Octoparse, ParseHub, WebHarvy, and ScrapeStorm offer powerful solutions for scraping Yelp data, each with unique features and pricing models to suit different needs.
• Setting Up Your Python Environment: Properly setting up your Python environment with necessary libraries like Requests, BeautifulSoup, Selenium, and Scrapy is crucial. Using virtual environments ensures clean and manageable dependencies.
• Fetching and Scraping Yelp Data: Practical examples demonstrate how to fetch search result pages, navigate to individual business pages, handle pagination, and extract detailed business information and reviews.
• Avoiding Blocks and Bans: Using proxies, implementing delays, and rotating user agents are essential strategies to avoid detection and ensure smooth scraping operations.
• Building a Yelp Scraper Project: An example project walks through setting up the project structure, writing the scraper script, running the scraper, collecting data, and performing data analysis using Pandas and visualization tools like Matplotlib or Seaborn.

Why Would You Want To Scrape Yelp Reviews?

Yelp web scraping is the process of automatically extracting data from Yelp.com. By using specialized software or scripts, you can collect large amounts of data quickly and efficiently. But Why would you want to collect yelp review data?

Yelp data is a goldmine for businesses looking to gain insights into customer behavior, preferences, and trends. With millions of reviews covering a wide range of businesses, Yelp provides comprehensive data that can be analyzed to inform business strategies and decisions. Here is how professional web scraping geeks try to make money scraping Yelp:

Common Yelp Scraping Use Cases

• Lead Generation and Sales: Scraping Yelp reviews can generate high-quality leads for businesses. By collecting contact information and review details, you can create targeted marketing lists to drive sales and business development efforts.
• Resell Scraped Data: Companies pay for data on competitors, market trends, and customer sentiments. By scraping Yelp reviews and organizing the data, you can sell valuable insights and datasets to businesses looking for competitive intelligence.
• Building Targeted Databases: Create comprehensive databases of businesses, categorized by industry or location. These databases can be sold to companies for marketing purposes or used to create directory services, providing ongoing revenue streams.
• Competitor Analysis: By analyzing competitors’ Yelp reviews, businesses can identify their strengths and weaknesses, understand market positioning, and refine their own business strategies​.
• Sentiment Analysis and Brand Monitoring: Businesses can perform sentiment analysis on scraped reviews to gauge customer satisfaction and identify common themes or issues that need addressing.

Best Paid Yelp Data Scrapers 2024

The field of Yelp scraping has evolved significantly, with numerous paid tools now offering user-friendly interfaces and robust functionalities. In this section, we explore some of the top paid Yelp data scrapers for 2024 that can streamline your data extraction efforts.

1. V6Proxies

V6Proxies offers robust web scraping proxies that are ideal for extracting Yelp data. V6proxies residential and datacenter IP pool helps avoid IP bans and ensure smooth scraping operations by rotating IP addresses and bypassing anti-scraping measures.

Features:

• Supports both IPv4 and IPv6 proxies.
• High-speed proxies with reliable uptime.
• Easy integration with web scraping tools.

Pricing: Flexible pricing plans based on the number of proxies and usage requirements.

Best For: Businesses and developers who need reliable proxies to support their web scraping activities, particularly those focusing on high-volume data extraction from Yelp.

2. Apify

Apify offers a powerful Yelp data scraper that can extract business reviews, ratings, and other details. It supports both URL and keyword inputs, making it versatile for different scraping needs.

Features:

• Cloud-based with API access
• Multiple data export options (JSON, CSV)
• Supports scheduling and rotating proxies

Pricing: Starts at $49 per month for 1000 results with a pay-as-you-go model.

Best For: Businesses needing scalable and reliable Yelp data scraping with API integration capabilities​.

3. Octoparse

Octoparse provides a user-friendly desktop-based scraper with pre-built templates for scraping Yelp data. It supports advanced features like IP rotation and captcha solving for an extra cost.

Features:

• Visual scraper with auto-detect mode.
• Multiple export formats (CSV, JSON, Excel).
• Supports cloud scraping and scheduling.

Pricing: Free plan available; premium plans start at $90 per month.

Best For: Large companies and users who need advanced features and are comfortable with a steep learning curve​.

4. ParseHub

ParseHub is known for its ease of use and powerful scraping capabilities. It allows users to scrape Yelp reviews and business data using a visual interface.

Features:

• Visual scraper with no coding required.
• Data export in JSON and Excel.
• Handles pagination and complex web structures.

Pricing: Starts at $149 per month, with a free desktop version available with limitations.

Best For: Users who prefer a visual interface and need a robust tool for detailed data extraction from Yelp​​.

5. WebHarvy

Overview: WebHarvy offers a point-and-click interface for easy data scraping. It’s designed to handle modern web structures and anti-scraping measures.

Features:

• Visual web scraper.
• Multiple data export formats (CSV, JSON, Excel).
• Intelligent pattern detection.

Pricing: Starts at $139 per month.

Best For: Businesses focused on scraping detailed business and review data with minimal setup effort​.

6. ScrapeStorm

ScrapeStorm uses AI-based methods for data recognition, simplifying the scraping process. It supports various operating systems and cloud-based platforms.

Features:

• AI-based data recognition.
• Supports multiple data export formats (CSV, JSON, Excel).
• Available for desktop and cloud use.

Pricing: Starts at $49.99 per month.

Best For: Users looking for an advanced, AI-powered scraping solution that minimizes manual setup​​.

How to Scrape Yelp Reviews With Python?

When it comes to scraping Yelp reviews on your own with Python, selecting the right tools and libraries is crucial. Here’s a guide to help you choose the best tools and set up your environment.

Overview of Python Libraries

To scrape Yelp reviews effectively, the following Python libraries are commonly used:

• Requests: Simplifies HTTP requests to download web pages.
• BeautifulSoup: Parses HTML and XML documents, ideal for extracting specific elements from web pages.
• Selenium: Automates web browsers, essential for scraping dynamic content that requires interaction.
• Scrapy: A comprehensive framework for large-scale web scraping projects, offering robust features for handling requests and data pipelines.

Choosing The Right Tool Based On Requirements

For scraping Yelp reviews, the combination of Requests and BeautifulSoup is highly recommended. These tools are effective for extracting review content from Yelp’s static pages. For handling dynamic content, such as loading reviews on scroll, Selenium is preferred. If you’re dealing with large-scale scraping and need a more structured approach, Scrapy is the way to go.

Step A: Setting Up Your Yelp Scraping Python Environment

• Installing Python and Necessary Libraries: Ensure you have Python installed. Download it from the official Python website. Install the necessary libraries using pip:

pip install requests beautifulsoup4 selenium scrapy

• Setting Up a Virtual Environment: Creating a virtual environment helps manage dependencies:

python -m venv yelp_scraper_env
source yelp_scraper_env/bin/activate  # On Windows use `yelp_scraper_env\Scripts\activate`

• Installing Requests and BeautifulSoup: With your virtual environment activated, install Requests and BeautifulSoup:

pip install requests beautifulsoup4

• Basic Usage of Requests and BeautifulSoup: Here’s a simple example to scrape Yelp reviews:

import requests
from bs4 import BeautifulSoup

# Fetch the Yelp page
url = “https://www.yelp.com/biz/some-business”
response = requests.get(url)
html_content = response.content

# Parse the HTML content
soup = BeautifulSoup(html_content, ‘html.parser’)

# Extract data
business_name = soup.find(‘h1’).text
reviews = soup.find_all(‘p’, {‘class’: ‘comment__373c0__Nsutg’})
for review in reviews:
    print(review.text)

This script fetches a Yelp page, parses its HTML, and prints out the reviews. Customize it further based on your specific needs. 

Step B: Fetching Yelp Search Result Pages

To scrape Yelp reviews effectively, you need to understand how to fetch the search result pages. Here’s how you can get started with fetching Yelp pages using Python.

Understanding Yelp’s HTML Structure

Yelp’s HTML structure is designed to display business listings, reviews, and other details in a structured format. Key elements include:

• Business Listings: Each business is typically enclosed within a <div> tag with a specific class.
• Review Text: Reviews are often contained within <p> tags with distinctive classes.
• Pagination: Links to additional pages are usually at the bottom of the search results, within <a> tags.

By inspecting the HTML structure, you can identify the elements that contain the data you need.

Using Requests to Download Pages

The Requests library in Python is a powerful tool for making HTTP requests to fetch web pages. Here’s how you can use it to download Yelp search result pages:

import requests

# URL of the Yelp search results page
url = “https://www.yelp.com/search?find_desc=restaurants&find_loc=San+Francisco%2C+CA“

# Send a GET request to the URL
response = requests.get(url)

# Check if the request was successful
if response.status_code == 200:
    html_content = response.content
else:
    print(f”Failed to retrieve the page. Status code: {response.status_code}”)

Example Code for Fetching Search Result Pages

Here’s a complete example that fetches a Yelp search results page and prints out the HTML content:

import requests

def fetch_yelp_search_results(query, location):
    # Construct the URL
    url = f”https://www.yelp.com/search?find_desc={query}&find_loc={location}“
    
    # Send a GET request to the URL
    response = requests.get(url)
    
    # Check if the request was successful
    if response.status_code == 200:
        return response.content
    else:
        print(f”Failed to retrieve the page. Status code: {response.status_code}”)
        return None

# Fetch search results for “restaurants” in “San Francisco, CA”
html_content = fetch_yelp_search_results(“restaurants”, “San Francisco, CA”)

# Print the first 500 characters of the HTML content
if html_content:
    print(html_content[:500])

This script constructs the URL for a Yelp search query, sends a GET request using the Requests library, and prints the first 500 characters of the HTML content if the request is successful. 

Step C: Scraping Individual Yelp Business Pages

Once you have the search results, the next step is to scrape individual Yelp business pages to gather detailed information.

Navigating to Individual Business Pages

To scrape detailed information, you first need to navigate to the individual business pages. Typically, each business listing in the search results contains a link to its dedicated page. You can extract these links using BeautifulSoup and then make a request to each business page.

Extracting Detailed Business Information

Once on the individual business page, you can extract detailed business information such as the website, additional reviews, address, phone number, and more. This involves identifying the HTML elements that contain the desired data.

Example Code for Scraping Individual Business Pages

Here’s an example script to demonstrate how to scrape individual Yelp business pages:

import requests
from bs4 import BeautifulSoup

def get_business_links(search_html):
    soup = BeautifulSoup(search_html, ‘html.parser’)
    links = []
    for link in soup.find_all(‘a’, {‘class’: ‘css-166la90’}):  # The class may vary; inspect the page to find the correct one
        href = link.get(‘href’)
        if href and href.startswith(‘/biz/’):
            links.append(‘https://www.yelp.com’ + href)
    return links

def scrape_business_page(url):
    response = requests.get(url)
    if response.status_code == 200:
        soup = BeautifulSoup(response.content, ‘html.parser’)
        
        # Extract business name
        business_name = soup.find(‘h1’).text
        
        # Extract website (if available)
        website = soup.find(‘a’, {‘class’: ‘css-1um3nx’}).get(‘href’) if soup.find(‘a’, {‘class’: ‘css-1um3nx’}) else ‘N/A’
        
        # Extract reviews
        reviews = []
        for review in soup.find_all(‘p’, {‘class’: ‘comment__373c0__Nsutg’}):
            reviews.append(review.text)
        
        return {
            ‘name’: business_name,
            ‘website’: website,
            ‘reviews’: reviews
        }
    else:
        print(f”Failed to retrieve the business page. Status code: {response.status_code}”)
        return None

# Example usage
search_results_html = fetch_yelp_search_results(“restaurants”, “San Francisco, CA”)
business_links = get_business_links(search_results_html)

for link in business_links:
    business_info = scrape_business_page(link)
    if business_info:
        print(f”Business Name: {business_info[‘name’]}”)
        print(f”Website: {business_info[‘website’]}”)
        print(“Reviews:”)
        for review in business_info[‘reviews’]:
            print(f”- {review}”)

This Script Does The Following:

• get_business_links: This function extracts links to individual business pages from the search results.
• scrape_business_page: This function scrapes detailed information from an individual business page, such as the business name, website, and reviews.
• Example Usage: Combines the two functions to fetch search results, extract business links, and scrape detailed information from each business page.

Step D: Handling Pagination

When scraping Yelp, it’s common to encounter multiple pages of search results or reviews for a single business. Handling pagination effectively is crucial to ensure you scrape all available data. Here’s how to identify the pagination structure and loop through multiple pages.

Identifying Pagination Structure

The pagination structure includes links to specific page numbers and “Next” buttons. Each page URL can be constructed by modifying the query parameters, typically the start parameter.

• URL Example: https://www.yelp.com/biz/business-name?hrid=C1xgKlxZdr0VFtPKcLZBaA
• Second page URL: https://www.yelp.com/biz/business-name?hrid=C1xgKlxZdr0VFtPKcLZBaA?start=10

Below is an example code to scrape reviews from all pages of a specific Yelp business:

import requests
from bs4 import BeautifulSoup

def fetch_yelp_reviews(url, page=0):
    # Construct the URL for each review page
    paginated_url = f”{url}&start={page * 10}”
    response = requests.get(paginated_url)
    if response.status_code == 200:
        return response.content
    else:
        print(f”Failed to retrieve the page. Status code: {response.status_code}”)
        return None

def extract_reviews(page_html):
    soup = BeautifulSoup(page_html, ‘html.parser’)
    reviews = []
    for review in soup.find_all(‘p’, {‘class’: ‘comment__373c0__Nsutg’}):
        reviews.append(review.text)
    return reviews

def scrape_all_reviews(base_url, max_pages=5):
    all_reviews = []
    for page in range(max_pages):
        page_html = fetch_yelp_reviews(base_url, page)
        if page_html:
            reviews = extract_reviews(page_html)
            all_reviews.extend(reviews)
        else:
            break
    return all_reviews

# Example usage
base_url = “https://www.yelp.com/biz/business-name?hrid=C1xgKlxZdr0VFtPKcLZBaA”
all_reviews = scrape_all_reviews(base_url, max_pages=5)
print(f”Found {len(all_reviews)} reviews.”)
for review in all_reviews:
    print(review)

What This Script Will Do:

• fetch_yelp_reviews: This function constructs the URL for each page using the base URL and the page number, then fetches the HTML content.
• extract_reviews: This function extracts review texts from the HTML content using BeautifulSoup.
• scrape_all_reviews: This function loops through multiple pages, fetching and extracting reviews from each page up to a specified maximum number of pages.

Step E. Avoiding Blocks and Bans

When scraping data from websites like Yelp, it’s essential to implement strategies to avoid getting blocked or banned. Here are some techniques to help you achieve this.

Using Proxies to Avoid IP Bans

To prevent your IP address from being blocked by Yelp, you can use rotating web scraping proxies to rotate your IP address with each request. This helps distribute your requests across multiple IP addresses, reducing the chance of being detected as a bot. V6Proxies provides reliable web scraping proxies that can be used for this purpose. You can integrate Proxies into your scraping script to manage IP rotation effectively and we will show you this later.

Implementing Delays and Random User Agents

To mimic human behavior and avoid detection, implement delays between your requests and use random user agents. This makes it harder for websites to identify and block your scraping activities.

• Delays: Introduce random delays between requests to simulate human browsing patterns.
• Random User Agents: Rotate user agents to make your requests appear as though they are coming from different browsers and devices.

Example Code for Using Proxies with Requests

Here’s an example of how to use proxies and implement delays and random user agents in your Python script using the Requests library:

import requests
import time
import random
from bs4 import BeautifulSoup

# List of user agents to rotate
user_agents = [
    ‘Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36’,
    ‘Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.1 Safari/605.1.15’,
    ‘Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0’
]

# Function to fetch Yelp reviews with proxies and random user agents
def fetch_yelp_reviews(url, proxies):
    headers = {
        ‘User-Agent’: random.choice(user_agents)
    }
    response = requests.get(url, headers=headers, proxies=proxies)
    if response.status_code == 200:
        return response.content
    else:
        print(f”Failed to retrieve the page. Status code: {response.status_code}”)
        return None

# Function to extract reviews from the HTML
def extract_reviews(page_html):
    soup = BeautifulSoup(page_html, ‘html.parser’)
    reviews = []
    for review in soup.find_all(‘p’, {‘class’: ‘comment__373c0__Nsutg’}):
        reviews.append(review.text)
    return reviews

# List of proxy servers
proxies_list = [
    {‘http’: ‘http://proxy1:port’, ‘https’: ‘https://proxy1:port’},
    {‘http’: ‘http://proxy2:port’, ‘https’: ‘https://proxy2:port’},
    # Add more proxies as needed
]

def scrape_all_reviews(base_url, max_pages=5):
    all_reviews = []
    for page in range(max_pages):
        proxies = random.choice(proxies_list)
        page_html = fetch_yelp_reviews(f”{base_url}&start={page * 10}”, proxies)
        if page_html:
            reviews = extract_reviews(page_html)
            all_reviews.extend(reviews)
            # Random delay between requests
            time.sleep(random.uniform(2, 5))
        else:
            break
    return all_reviews

# Example usage
base_url = “https://www.yelp.com/biz/business-name?hrid=C1xgKlxZdr0VFtPKcLZBaA”
all_reviews = scrape_all_reviews(base_url, max_pages=5)
print(f”Found {len(all_reviews)} reviews.”)
for review in all_reviews:
    print(review)

What This Script Will Do

• Proxies and User Agents: The script rotates between different proxies and user agents to avoid detection.
• Delays: Introduces random delays between requests to mimic human behavior.
• Fetching and Extracting Reviews: Fetches reviews from each page and extracts the relevant data.

Example Project: Building a Yelp Scraper

Embarking on an example project will allow you to put your newly acquired skills to practical use. This example project walks you through the process of building a Yelp scraper, from setting up the project structure to running the scraper and analyzing the data. .

Project Goals and Objectives:

• Goal: Build a Python-based scraper to extract reviews from Yelp.
• Objective: Collect detailed reviews, including ratings and review text, from multiple business pages on Yelp.

Dataset Description:

• Business Information: Business name, address, and website.
• Reviews: User ratings, review text, and review date.

Step-by-Step Guide

1. Setting Up the Project Structure:

• Create a project directory: yelp_scraper_project.
• Inside, create subdirectories: scripts, data, and outputs.

2. Writing the Scraper Script:

• In the scripts directory, create scraper.py.
• Write the script to fetch and parse Yelp pages, handle pagination, and save the data.

3. Running the Scraper and Collecting Data:

• Execute scraper.py to start scraping.
• Save the extracted data into CSV files in the data directory.

4. Cleaning and Organizing the Data:

• Create a script clean_data.py to clean and format the scraped data.
• Ensure all review texts are properly extracted and organized.

Example Code for Scraper.py

import requests
from bs4 import BeautifulSoup
import csv
import time
import random

# Define user agents and proxies
user_agents = [/* list of user agents */]
proxies_list = [/* list of proxies */]

# Function to fetch Yelp reviews
def fetch_yelp_reviews(url, page=0):
    headers = {‘User-Agent’: random.choice(user_agents)}
    proxies = random.choice(proxies_list)
    response = requests.get(f”{url}&start={page * 10}”, headers=headers, proxies=proxies)
    if response.status_code == 200:
        return response.content
    else:
        print(f”Failed to retrieve the page. Status code: {response.status_code}”)
        return None

def extract_reviews(page_html):
    soup = BeautifulSoup(page_html, ‘html.parser’)
    reviews = []
    for review in soup.find_all(‘p’, {‘class’: ‘comment__373c0__Nsutg’}):
        reviews.append(review.text)
    return reviews

def scrape_all_reviews(base_url, max_pages=5):
    all_reviews = []
    for page in range(max_pages):
        page_html = fetch_yelp_reviews(base_url, page)
        if page_html:
            reviews = extract_reviews(page_html)
            all_reviews.extend(reviews)
            time.sleep(random.uniform(2, 5))  # Random delay
        else:
            break
    return all_reviews

def save_to_csv(data, filename):
    with open(filename, ‘w’, newline=”, encoding=’utf-8′) as file:
        writer = csv.writer(file)
        writer.writerow([“Review”])
        for row in data:
            writer.writerow([row])

# Example usage
base_url = “https://www.yelp.com/biz/business-name?hrid=C1xgKlxZdr0VFtPKcLZBaA”
all_reviews = scrape_all_reviews(base_url, max_pages=5)
save_to_csv(all_reviews, ‘data/reviews.csv’)

Analyzing Scraped Data

1. Basic Data Analysis with Pandas:

• Load the CSV file into a Pandas DataFrame.
• Perform basic analysis such as counting the number of reviews and calculating average ratings.

2. Visualizing Data with Matplotlib or Seaborn:

• Create visualizations such as bar charts for ratings distribution.
• Plot trends over time if review dates are available.

3. Generating Insights and Reports:

• Summarize key findings, such as common sentiments in reviews.
• Generate reports highlighting the strengths and weaknesses of the business based on review analysis.

Example Code for Data Analysis

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

# Load the data
df = pd.read_csv(‘data/reviews.csv’)

# Basic data analysis
print(f”Total number of reviews: {len(df)}”)

# Visualize the data (e.g., word frequency in reviews)
sns.set(style=”darkgrid”)
plt.figure(figsize=(10, 6))
sns.countplot(y=df[‘Review’].value_counts().index, palette=”viridis”)
plt.title(‘Frequency of Words in Reviews’)
plt.show()

More Scraping Resources

In previous guides, we covered Google Search Scraping with python, Instagram Scraping, Facebook Scraping, WhatsApp Scraping, Scraping Amazon, scraping Airbnb, and LinkedIn Scraping. You can read and learn more and more in our blog.

Related articles:

A Guide to Google Maps Scraping (2004 Edition)