Python Web Scraping – requests & BeautifulSoup

Setup

pip install requests beautifulsoup4 lxml

Basic Page Fetch

import requests
from bs4 import BeautifulSoup

url = "https://example.com"
response = requests.get(url, timeout=10)
response.raise_for_status()
soup = BeautifulSoup(response.text, "lxml")

print(soup.title.string)
for link in soup.find_all("a", href=True):
    print(link["href"], link.get_text(strip=True))

CSS Selectors

headings = soup.select("h2.article-title")
first_para = soup.select_one("div.content > p")

images = soup.find_all("img", attrs={"class": "thumbnail"})
for img in images:
    print(img["src"], img.get("alt", ""))

Handling Pagination

import time

def scrape_all_pages(base_url, max_pages=10):
    results = []
    for page in range(1, max_pages + 1):
        resp = requests.get(base_url, params={"page": page}, timeout=10)
        if resp.status_code != 200:
            break
        soup = BeautifulSoup(resp.text, "lxml")
        items = soup.select(".item-card")
        if not items:
            break
        for item in items:
            results.append({
                "title": item.select_one("h3").get_text(strip=True),
                "price": item.select_one(".price").get_text(strip=True),
            })
        time.sleep(1)
    return results

Saving Results

import json, csv

data = [{"name": "Alice", "score": 95}]

with open("results.json", "w") as f:
    json.dump(data, f, indent=2)

with open("results.csv", "w", newline="") as f:
    writer = csv.DictWriter(f, fieldnames=["name", "score"])
    writer.writeheader()
    writer.writerows(data)

Web Scraping: Extracting Data From HTML — Responsibly

Scraping means fetching a web page and pulling structured data out of its HTML. The typical stack: requests to download, BeautifulSoup to parse. But scraping carries legal/ethical duties most tutorials skip.

import requests
from bs4 import BeautifulSoup

html = requests.get(url, headers={"User-Agent": "my-bot"}, timeout=10).text
soup = BeautifulSoup(html, "html.parser")

soup.find("h1").text                     # first h1
[a["href"] for a in soup.select("a.product")]   # CSS selector

Scrape responsibly — this matters: check the site's robots.txt and Terms of Service (some prohibit scraping), identify your bot with a User-Agent, and rate-limit your requests (a time.sleep() between them) so you don't hammer the server — aggressive scraping can get your IP banned or cause real harm. Prefer an official API if one exists; it's stabler and sanctioned. Technical caveat: requests+BeautifulSoup only see the initial HTML — pages that build content with JavaScript need a browser-automation tool (Selenium, Playwright). Scrape only public data, respect copyright, and never collect personal data without a lawful basis.

Interview Questions on Web Scraping

• What is web scraping and what are common use cases?
• What libraries are commonly used for scraping in Python?
• What is the difference between requests + BeautifulSoup and a tool like Selenium?
• How do you handle pagination when scraping?
• What are the legal and ethical considerations (robots.txt, rate limiting)?
• How do you scrape pages that load content with JavaScript?
• How do you make a scraper robust against page changes?

Related Topics