IT Course Directory: VMware, Linux, Networking, and Raspberry Pi

Create a Web Crawler in Python

Learn to build a responsible Python web crawler that fetches HTML, extracts and normalizes links, respects robots.txt, avoids duplicates, and saves crawl reports.

A web crawler is a program that systematically visits web pages and follows links discovered on those pages. In this lesson, you will build a small, domain-limited crawler in Python. It will fetch HTML, extract links, normalize URLs, avoid duplicate requests, obey crawl limits, apply delays, and save results.

This lesson assumes familiarity with Python functions, classes, collections, exceptions, virtual environments, basic HTTP, HTML anchor tags, and JSON or CSV files.

Web Crawling Fundamentals

A web crawler discovers and visits pages. A web scraper extracts particular data, such as product names or prices, from pages. Scraping may use crawling, but the two tasks are not identical.

• Spider: another common name for a crawler.
• Browser automation tool: controls a browser and can execute JavaScript. Tools in this category are useful when links or content are generated after page load.
• Search-engine indexer: a large-scale system that crawls pages, analyzes content, stores indexes, and ranks results. A small educational crawler does not provide this scale or search functionality.

A typical crawl follows this cycle:

1. Start with one or more seed URLs, the initial addresses.
2. Place those URLs in the crawl frontier, the collection of pending URLs.
3. Fetch a URL with an HTTP client.
4. Check the response status and content type.
5. Parse eligible HTML.
6. Extract page information and links.
7. Resolve and canonicalize discovered URLs.
8. Apply scope, robots, file-type, and limit rules.
9. Schedule eligible URLs and persist the result.

A small crawler normally runs in one process, uses an in-memory queue and sets, and visits a limited number of pages. Large crawlers require distributed scheduling, persistent frontiers, duplicate-content detection, per-host fairness, monitoring, retries, storage systems, and extensive abuse prevention.

Crawler Components

Project Setup

Create a project and an isolated virtual environment. Activation commands differ by operating system:

mkdir python-crawler
cd python-crawler
python -m venv .venv

# macOS or Linux
source .venv/bin/activate

# Windows PowerShell
.venv\Scripts\Activate.ps1

python -m pip install requests beautifulsoup4
# Optional development tools
python -m pip install pytest ruff

A useful module layout separates responsibilities:

crawler_project/
    crawler/
        __init__.py
        config.py       # limits and policy
        fetcher.py      # HTTP requests
        parser.py       # HTML extraction
        urls.py         # resolution and canonicalization
        policy.py       # scope and robots decisions
        scheduler.py    # frontier management
        storage.py      # JSON, CSV, or SQLite output
        crawl.py        # orchestration
    tests/
        test_urls.py
        test_policy.py
    main.py

Keep behavior in configuration rather than scattering constants through the code. Useful settings include the User-Agent, timeout, retry count, delay, allowed hosts, maximum pages, maximum depth, output file, accepted content types, and robots compliance.

Fetching Pages with HTTP

The requests library provides a straightforward HTTP client. Always identify the client honestly and set a timeout. A timeout prevents one unavailable server from stopping the entire crawl indefinitely.

import requests

HEADERS = {
    "User-Agent": "ExampleEducationalCrawler/1.0 (contact: crawler-owner@example.invalid)"
}


def fetch(session, url, timeout=10):
    try:
        response = session.get(
            url,
            headers=HEADERS,
            timeout=timeout,
            allow_redirects=True,
        )
        content_type = response.headers.get("Content-Type", "").lower()
        return response, content_type, None
    except requests.exceptions.RequestException as exc:
        return None, "", f"{type(exc).__name__}: {exc}"


with requests.Session() as session:
    response, content_type, error = fetch(session, "https://example.invalid/")
    if error:
        print(error)
    elif response.status_code == 200 and "text/html" in content_type:
        print(response.url, response.text[:200])

Redirects are normally followed, but record both the requested URL and the final URL. Set a maximum redirect policy when the client or application requires it. Check the status before parsing: a successful response is usually in the 200–299 range, a redirect is in the 300–399 range, a client error is in the 400–499 range, and a server error is in the 500–599 range.

Connection failures, DNS errors, SSL errors, timeouts, malformed responses, and encoding problems should be isolated to the affected URL. Catch request exceptions, log the exception type, and continue when safe. Retries should be bounded and reserved mainly for transient failures. Use increasing backoff, and do not retry indefinitely.

Parsing HTML and Extracting Links

An HTML parser converts markup into a structure that code can query. Beautiful Soup can locate anchor elements and read their href attributes.

from bs4 import BeautifulSoup
from urllib.parse import urljoin


def parse_page(html, page_url):
    soup = BeautifulSoup(html, "html.parser")
    title = soup.title.get_text(" ", strip=True) if soup.title else ""
    links = []

    for anchor in soup.find_all("a", href=True):
        raw_href = anchor["href"].strip()
        if not raw_href:
            continue
        lowered = raw_href.lower()
        if lowered.startswith(("#", "mailto:", "tel:", "javascript:", "data:")):
            continue
        links.append(urljoin(page_url, raw_href))

    return {"title": title, "links": links}

You can also extract visible text, headings, metadata, and a canonical link:

canonical = soup.find("link", rel=lambda value: value and "canonical" in value)
canonical_url = urljoin(page_url, canonical.get("href")) if canonical and canonical.get("href") else None
headings = [tag.get_text(" ", strip=True) for tag in soup.find_all(["h1", "h2", "h3"])]

Raw HTTP HTML does not necessarily contain links created by JavaScript. If a page requires JavaScript rendering, first determine whether a non-rendered representation, sitemap, or documented endpoint is available. Use browser rendering only when it is permitted and genuinely necessary.

Resolving and Canonicalizing URLs

An absolute URL contains a scheme and host, such as https://site.test/about. A relative URL depends on the current page. Use urljoin instead of manually concatenating strings.

URL canonicalization means applying consistent rules so equivalent addresses can be recognized as one crawl target. Remove fragments, lowercase the scheme and hostname, and remove an empty port when appropriate. Do not automatically lowercase paths or query values: those may be case-sensitive. Do not remove query parameters unless the site-specific policy proves that doing so is safe.

from urllib.parse import urlsplit, urlunsplit


def canonicalize(raw_url):
    parts = urlsplit(raw_url.strip())
    if parts.scheme.lower() not in {"http", "https"} or not parts.netloc:
        return None

    scheme = parts.scheme.lower()
    hostname = (parts.hostname or "").lower()
    if not hostname:
        return None

    # Preserve path and query case; remove only the fragment.
    port = parts.port
    default_port = (scheme == "http" and port == 80) or (scheme == "https" and port == 443)
    host = hostname if not port or default_port else f"{hostname}:{port}"
    path = parts.path or "/"
    return urlunsplit((scheme, host, path, parts.query, ""))

Query parameters can create many distinct targets, including tracking URLs, searches, calendars, and infinite combinations. Decide explicitly whether to reject selected parameters, limit query length, or allow queries only for known paths.

The Crawl Frontier and Deduplication

A queue or deque produces a breadth-first crawl. A stack produces a depth-first crawl. Breadth-first traversal visits pages near the seeds first and is usually easier to reason about for a small crawler.

from collections import deque

frontier = deque([(seed_url, 0, None)])
discovered = {seed_url}  # Mark when queued
a = set()               # URLs already fetched

while frontier:
    url, depth, parent = frontier.popleft()
    if url in a:
        continue
    a.add(url)
    # Fetch and process url here
    # Add new canonical URLs to discovered before queueing them

Use separate sets for clarity: discovered contains URLs seen or queued, queued can represent pending URLs, and visited contains URLs whose fetch attempt has been processed. Mark a URL as discovered as soon as it is accepted for scheduling. Waiting until after the request allows duplicate links to enter the frontier.

Scope Control

Resolve and canonicalize a link before checking its scope. Compare normalized hostnames against an approved set. A hostname is the complete host portion, while a registrable domain is the organization-level portion identified using the public suffix rules. For example, treating every host ending in a text suffix as equivalent can accidentally allow an unrelated hostname. For a beginner crawler, an explicit approved-host set is safest.

from urllib.parse import urlsplit


def in_scope(url, allowed_hosts):
    host = (urlsplit(url).hostname or "").lower()
    return host in allowed_hosts


def allowed_path(url):
    path = urlsplit(url).path.lower()
    blocked_extensions = (".jpg", ".jpeg", ".png", ".gif", ".pdf", ".zip")
    blocked_paths = ("/logout", "/delete", "/remove")
    return not path.endswith(blocked_extensions) and not path.startswith(blocked_paths)

Also exclude non-HTTP schemes, known destructive actions, unwanted file extensions, sensitive paths, and query patterns that produce unbounded results. Never follow logout or state-changing URLs merely because they appear in an anchor.

Polite and Ethical Crawling

robots.txt is a site-hosted file containing crawler directives for specified user agents. It is a crawl policy mechanism, not an access-control system or permission to ignore legal and contractual restrictions.

Before fetching a host, retrieve and evaluate its robots rules with a meaningful User-Agent. Python's standard library includes a parser suitable for basic robots evaluation:

from urllib.robotparser import RobotFileParser
from urllib.parse import urlsplit


def robots_for(seed_url, user_agent):
    parts = urlsplit(seed_url)
    robots_url = f"{parts.scheme}://{parts.netloc}/robots.txt"
    parser = RobotFileParser(robots_url)
    parser.read()
    return parser

# allowed = parser.can_fetch(user_agent, candidate_url)

For robust software, handle an unavailable or malformed robots file according to the policy you have documented, cache rules per host, and log robots exclusions. Review the site's terms, applicable legal requirements, privacy implications, and likely server impact. Do not bypass authentication, anti-bot controls, robots directives, or access restrictions.

Apply a politeness delay per host and rate-limit requests. Avoid excessive parallelism. A single-threaded crawler with a one- or two-second delay is a useful starting point for a controlled practice site.

A Domain-Limited Breadth-First Crawler

import csv
import time
from collections import deque
from urllib.parse import urlsplit, urljoin

import requests
from bs4 import BeautifulSoup


class Crawler:
    def __init__(self, seed, allowed_hosts, max_pages=25, max_depth=2,
                 delay=1.0, timeout=10):
        self.frontier = deque([(seed, 0, None)])
        self.discovered = {seed}
        self.visited = set()
        self.allowed_hosts = {host.lower() for host in allowed_hosts}
        self.max_pages = max_pages
        self.max_depth = max_depth
        self.delay = delay
        self.timeout = timeout
        self.rows = []
        self.session = requests.Session()
        self.session.headers.update({
            "User-Agent": "ExampleEducationalCrawler/1.0 (contact: crawler-owner@example.invalid)"
        })

    def eligible(self, url):
        parts = urlsplit(url)
        if parts.scheme not in {"http", "https"}:
            return False
        if (parts.hostname or "").lower() not in self.allowed_hosts:
            return False
        return not parts.path.lower().endswith((".jpg", ".png", ".gif", ".pdf", ".zip"))

    def run(self):
        while self.frontier and len(self.visited) < self.max_pages:
            url, depth, parent = self.frontier.popleft()
            if url in self.visited or not self.eligible(url):
                continue

            if self.delay:
                time.sleep(self.delay)
            self.visited.add(url)
            row = {"url": url, "parent": parent, "depth": depth,
                   "status": None, "content_type": "", "title": "", "error": ""}

            try:
                response = self.session.get(url, timeout=self.timeout, allow_redirects=True)
                row["status"] = response.status_code
                row["content_type"] = response.headers.get("Content-Type", "")
                if response.status_code >= 400:
                    row["error"] = f"HTTP {response.status_code}"
                elif "text/html" in row["content_type"].lower():
                    soup = BeautifulSoup(response.text, "html.parser")
                    row["title"] = soup.title.get_text(" ", strip=True) if soup.title else ""
                    if depth < self.max_depth:
                        for anchor in soup.find_all("a", href=True):
                            candidate = canonicalize(urljoin(response.url, anchor["href"]))
                            if candidate and candidate not in self.discovered and self.eligible(candidate):
                                self.discovered.add(candidate)
                                self.frontier.append((candidate, depth + 1, url))
            except requests.exceptions.RequestException as exc:
                row["error"] = f"{type(exc).__name__}: {exc}"
            except Exception as exc:
                row["error"] = f"ParserError: {exc}"

            self.rows.append(row)
        return self.rows


def canonicalize(url):
    parts = urlsplit(url)
    if parts.scheme.lower() not in {"http", "https"} or not parts.netloc:
        return None
    return urljoin(url, urlsplit(url)._replace(fragment="").geturl())

The example demonstrates the architecture, but production code should use a dedicated, tested canonicalization function, cached robots evaluation, per-host pacing, structured logging, and carefully designed retry behavior.

Command-Line Configuration

Expose crawl behavior through command-line arguments rather than editing source code for every run:

python main.py \
  --seed https://site.test/ \
  --max-pages 50 \
  --max-depth 3 \
  --allowed-host site.test \
  --delay 1.5 \
  --timeout 10 \
  --output crawl.csv \
  --verbose

Useful options include a seed URL, maximum pages, maximum depth, approved domains, request delay, output path, verbose logging, retry count, backoff policy, accepted content types, maximum redirects, and a flag that enables robots compliance.

Storing Results and Reporting

Record at least the requested URL, final URL, parent URL, depth, status code, content type, title, timestamp, and error information. Saving parent-child relationships lets you inspect how a page was discovered.

import csv


def write_csv(rows, filename):
    fields = ["url", "parent", "depth", "status", "content_type", "title", "error"]
    with open(filename, "w", newline="", encoding="utf-8") as output:
        writer = csv.DictWriter(output, fieldnames=fields)
        writer.writeheader()
        writer.writerows(rows)


def summary(rows, discovered):
    fetched = len(rows)
    successes = sum(1 for row in rows if row["status"] and 200 <= row["status"] < 300)
    failures = sum(1 for row in rows if row["error"])
    return {
        "fetch_attempts": fetched,
        "successful_responses": successes,
        "failures": failures,
        "discovered_urls": len(discovered),
    }

JSON is convenient for nested relationships, CSV is easy to inspect, and SQLite provides structured queries without requiring a separate database server. Persist records incrementally for longer runs. A checkpoint can store the frontier, discovered set, visited set, and result records so a process can resume after interruption.

Log successful fetches, skipped URLs, robots exclusions, redirects, HTTP errors, network errors, and parser errors. A final report should include pages fetched, links discovered, skipped resources, robots exclusions, redirects, and failures grouped by category.

Error Handling and Reliability

Do not let one malformed page terminate the crawl. Catch request and parser failures around each URL, store the failure, and continue. Avoid treating every exception as retryable. A timeout or temporary server failure may merit a small number of retries; an invalid URL or a permanent client error generally does not.

Testing and Debugging

Develop against a controlled local or test website containing relative links, root-relative links, duplicate links, redirects, an external link, a missing page, a fragment, and a non-HTML resource. This makes behavior predictable and avoids placing unnecessary load on an unrelated public site.

Unit-test URL handling with cases such as:

• ../about from a nested page
• /contact from different pages
• ?page=2 and its query-policy result
• #section, which should not create another page target
• mailto:, tel:, javascript:, and data:, which should be rejected
• Different scheme and hostname casing
• Case-sensitive paths and query values, which should remain unchanged

Test that scope rules reject external hosts, that duplicate links enter the frontier once, that depth and page limits stop scheduling, that robots exclusions are honored, and that delays occur before requests. When debugging, inspect logs and compare requested URLs with canonical URLs, final redirect URLs, response status, content type, and parent relationships.

Extensions and Production Considerations

• Bounded concurrency: asynchronous or concurrent fetching can improve throughput, but use a strict limit and per-host rate control.
• Fair scheduling: maintain per-domain queues so one host does not monopolize the crawler.
• Duplicate-page detection: hash normalized content to find different URLs serving the same page.
• Sitemaps: discover and parse sitemap files when permitted; they can complement, but not replace, link crawling.
• Canonical links: record the HTML canonical hint as a preferred duplicate-content address, but treat it as a hint rather than unquestionable truth.
• JavaScript rendering: use a browser-capable tool only for pages whose required content cannot be obtained through ordinary HTML.
• Persistent frontiers: use a database-backed queue and checkpoints for larger or resumable crawls.
• SSRF protection: if users can submit URLs, validate schemes and hosts, block private and loopback address ranges, re-check destinations after DNS resolution and redirects, limit response size, and restrict network access. This prevents the crawler from being used to request internal services.

Practical Crawl Checklist

1. Use a controlled seed and approved host list.
2. Canonicalize every URL before deduplication.
3. Remove fragments and reject unsupported schemes.
4. Check robots rules and site policies.
5. Identify the crawler with a descriptive User-Agent.
6. Apply a timeout, bounded retries, and a per-host delay.
7. Parse only successful HTML responses.
8. Set page, depth, runtime, response-size, and redirect limits.
9. Persist results and errors as the crawl runs.
10. Review the summary and logs before increasing crawl scope.