IT Course Directory: VMware, Linux, Networking, and Raspberry Pi

Create a Web Crawler in Python

Learn to build a responsible Python web crawler that fetches pages, parses HTML, follows links, respects crawl rules, handles errors, and stores results.

A web crawler is a program that systematically retrieves web resources and follows eligible links. In this lesson, you will design and build a bounded Python crawler that downloads HTML pages, extracts links and page data, applies URL and robots.txt rules, handles failures, and writes results for later inspection.

This lesson assumes familiarity with Python functions, modules, exceptions, lists, dictionaries, sets, queues, basic command-line usage, package installation, URLs, HTTP responses, and HTML attributes. For the complete course context, see Create A Web Crawler In Python.

What a web crawler does

A crawler begins with one or more seed URLs. It fetches a resource, parses the response, discovers links, places eligible new URLs in a URL frontier, and repeats the process until the frontier is empty or a crawl limit is reached.

Term — Meaning

Web crawler — A program that systematically retrieves resources and follows eligible links.

Web scraper — A tool focused on extracting particular data from pages. A scraper may use crawling to find those pages.

Browser automation tool — Software that controls a browser, often executing JavaScript and interacting with rendered pages.

Search engine — A large indexing and ranking system that uses crawlers as one part of a broader pipeline.

API client — A program that calls structured application endpoints rather than discovering and parsing general web pages.

Typical crawler uses include site indexing, link auditing, monitoring public content, research, and structured data collection. Crawling is not automatically authorized merely because a page is publicly reachable. Always consider published access rules, terms, permissions, privacy, and applicable legal or organizational requirements.

The crawler lifecycle

Choose seed URLs.
Check whether each URL is allowed by domain, path, content-type, robots.txt, and crawl limits.
Fetch the resource with an explicit timeout and identifiable headers.
Inspect the status code, headers, redirect history, encoding, and response body.
Parse HTML responses.
Extract page fields such as title, headings, metadata, and canonical URL.
Extract hyperlinks and resolve relative links against the current page URL.
Normalize and deduplicate discovered URLs.
Add eligible URLs to the frontier.
Persist page, link, and error information.
Log progress and produce a summary.

Architecture and crawl boundaries

A small crawler is easier to reason about when each responsibility is explicit.

Component — Input — Responsibility — Output

Seed list — Starting URLs — Defines the initial crawl scope — URLs placed in the frontier.

URL frontier — Eligible discovered URLs — Schedules pending work — Next URL to fetch.

Fetcher — URL and request policy — Sends HTTP requests and applies timeouts, headers, redirects, and retries — Response or failure.

HTML parser — Response body — Builds a queryable document structure — Parsed document.

Link extractor — Parsed document — Reads anchor href values and source relationships — Candidate URLs.

URL normalizer — Candidate URL and base URL — Produces a consistent representation — Normalized URL.

Deduplication store — Normalized URLs — Prevents repeated scheduling and cycles — Seen-state updates.

Data pipeline — Page and link fields — Converts extracted values into records — JSON, CSV, or database rows.

Logger — Events and exceptions — Records progress, warnings, and failures — Diagnostic output.

Set boundaries before writing code. Useful limits include allowed_domains, allowed_paths, max_depth, max_pages, allowed content types, URL patterns, response-size limits, and a per-host request rate. A boundary is a safety feature as well as a performance feature.

Breadth-first and depth-first scheduling

Breadth-first search processes pages level by level: seed pages first, then their direct links, then links at the next depth. It is often useful for controlled site crawls because it gives predictable depth coverage and tends to discover important nearby pages early.

Depth-first search follows one link chain deeply before returning to sibling links. It can use less frontier memory, but it may spend a long time in one branch before covering other sections.

A queue produces breadth-first behavior. A stack produces depth-first behavior.

Prepare the Python environment

python -m venv .venv
source .venv/bin/activate
.venv\Scripts\activate
python -m pip install requests beautifulsoup4 lxml

Use the activation command appropriate for your operating system. During development, use a controlled local or test website rather than an unrestricted public crawl.

Making HTTP requests

Python's requests library provides a convenient HTTP client. A response contains the status code, headers, final URL, redirect history, encoding information, and body content.

import requests

headers = {
    'User-Agent': 'ExampleResearchCrawler/1.0 (contact: crawler@example.invalid)',
    'Accept': 'text/html,application/xhtml+xml'
}

try:
    response = requests.get(
        'https://example.test/',
        headers=headers,
        timeout=(5, 20),
        allow_redirects=True
    )
    print(response.status_code)
    print(response.headers.get('Content-Type'))
    print(response.url)
    print([item.status_code for item in response.history])
    print(response.encoding)
    print(response.text[:200])
except requests.exceptions.Timeout:
    print('The request timed out')
except requests.exceptions.RequestException as error:
    print(f'Request failed: {error}')

The timeout tuple gives separate connection and read limits. Never rely on an unlimited network request. A clear User-Agent identifies your client; do not impersonate a browser to evade restrictions.

A persistent requests.Session reuses connections and keeps common headers and settings together. It is useful when making multiple requests to related hosts, but it does not remove the need for throttling.

session = requests.Session()
session.headers.update(headers)
response = session.get(url, timeout=(5, 20))

Status category — Typical meaning — Crawler action — Retry guidance

2xx — Successful response — Process supported content — No retry normally.

3xx — Redirect — Inspect the final URL and redirect chain, then reapply scope rules — Usually let the HTTP client follow redirects; avoid long chains.

400–499 — Client-side problem, including not found or forbidden — Record and usually skip — Do not repeatedly retry permanent failures.

429 — Rate limit — Slow down and honor a server-provided retry indication when present — Retry sparingly with backoff.

500–599 — Server-side failure — Record the failure and retry within a strict limit — Exponential backoff may be appropriate.

Parsing HTML and extracting data

An HTML parser converts markup into a structure that can be queried. Beautiful Soup supports CSS selectors and convenient element methods; lxml also supports XPath.

from bs4 import BeautifulSoup

soup = BeautifulSoup(response.content, 'lxml')
title = soup.title.get_text(' ', strip=True) if soup.title else None
headings = [node.get_text(' ', strip=True) for node in soup.select('h1, h2, h3')]
paragraphs = [node.get_text(' ', strip=True) for node in soup.select('p')]
description_node = soup.select_one('meta[name="description"]')
description = description_node.get('content', '').strip() if description_node else None
canonical_node = soup.select_one('link[rel="canonical"]')
canonical = canonical_node.get('href') if canonical_node else None

Selectors must tolerate missing elements. A page may omit a title, contain malformed markup, return a login or consent page, or place content in a different structure. Save a sample response when debugging instead of assuming that the selector is wrong.

Extracting hyperlinks

for anchor in soup.select('a[href]'):
    href = anchor.get('href')
    label = anchor.get_text(' ', strip=True)
    print(href, label)

Ignore empty values, fragment-only links such as #details, schemes such as mailto: and javascript:, and resources that are not intended for this HTML crawler. Images, documents, feeds, and scripts may require separate handling.

URL resolution and normalization

A relative URL depends on a base page. For example, ../guide on https://example.test/docs/start resolves to https://example.test/guide. Use standard URL utilities rather than string concatenation.

from urllib.parse import urljoin, urldefrag, urlsplit, urlunsplit

def normalize_url(raw_url, base_url):
    absolute = urljoin(base_url, raw_url)
    without_fragment, _ = urldefrag(absolute)
    parts = urlsplit(without_fragment)
    scheme = parts.scheme.lower()
    hostname = (parts.hostname or '').lower()
    if not scheme or not hostname:
        return None
    if parts.port is None or (scheme == 'http' and parts.port == 80) or (scheme == 'https' and parts.port == 443):
        netloc = hostname
    else:
        netloc = f'{hostname}:{parts.port}'
    path = parts.path or '/'
    return urlunsplit((scheme, netloc, path, parts.query, ''))

URL characteristic — Example outcome — Reason

Relative path — Resolve with urljoin — The current page supplies the missing context.

Fragment — Remove #section — Fragments identify a location in a document, not a different HTTP resource.

Scheme or hostname case — Convert to lowercase — These components are case-insensitive.

Default port — Treat :80 as HTTP default and :443 as HTTPS default — Avoid equivalent host representations.

Trailing slash — Choose a consistent policy — Do not assume every server treats slash variants identically; test before collapsing them.

Tracking parameters — Remove only an explicitly approved set such as selected analytics parameters — Query strings can change content, so indiscriminate removal is unsafe.

Canonical URL — Record the page's declared preferred URL separately — A canonical link is a page signal, not automatic proof that two URLs are identical.

Naïve string comparison treats equivalent representations as different URLs. Normalize before adding a URL to the frontier and before checking the seen set. Redirect destinations must also be normalized and checked against the crawl boundary.

Queues, deduplication, and crawl state

Mark a URL as seen when it is scheduled, not only after fetching. Otherwise several pages can discover the same URL before the first request completes.

State — Description — Possible next state

Queued — Eligible URL is waiting in the frontier — Fetched, failed, or skipped.

Fetched — A response was received and processed — Queued descendants, or complete.

Failed — Fetching or parsing ended with an error — Retried, skipped, or complete.

Skipped — URL was rejected by scope, robots, content type, size, or another policy — Complete.

Use a dictionary when you need detailed state and a set when you only need fast membership checks. Store the source page for each discovered link when constructing a link graph or auditing navigation.

A bounded same-domain crawler

The following example uses a breadth-first queue, a session, URL normalization, a host boundary, a page limit, a delay, robots.txt checks, content-type filtering, and JSON output. Replace the seed with a permitted test URL.

import json
import logging
import time
from collections import deque
from datetime import datetime, timezone
from urllib.parse import urljoin, urldefrag, urlsplit, urlunsplit
from urllib.robotparser import RobotFileParser

import requests
from bs4 import BeautifulSoup

SEED_URLS = ['https://example.test/']
ALLOWED_DOMAINS = {'example.test'}
MAX_PAGES = 25
MAX_DEPTH = 2
DELAY_SECONDS = 1.0
TIMEOUT = (5, 20)
USER_AGENT = 'ExampleResearchCrawler/1.0 (contact: crawler@example.invalid)'

logging.basicConfig(level=logging.INFO, format='%(levelname)s %(message)s')


def normalize_url(raw, base):
    absolute = urljoin(base, raw)
    absolute, _ = urldefrag(absolute)
    parts = urlsplit(absolute)
    if parts.scheme.lower() not in {'http', 'https'} or not parts.hostname:
        return None
    scheme = parts.scheme.lower()
    host = parts.hostname.lower()
    port = parts.port
    if port is None or (scheme == 'http' and port == 80) or (scheme == 'https' and port == 443):
        netloc = host
    else:
        netloc = f'{host}:{port}'
    return urlunsplit((scheme, netloc, parts.path or '/', parts.query, ''))


def allowed(url):
    return urlsplit(url).hostname in ALLOWED_DOMAINS


def robots_for(url, session):
    parts = urlsplit(url)
    robots_url = f'{parts.scheme}://{parts.netloc}/robots.txt'
    parser = RobotFileParser()
    parser.set_url(robots_url)
    try:
        response = session.get(robots_url, timeout=TIMEOUT)
        if response.status_code == 404:
            parser.parse([])
        elif response.ok:
            parser.parse(response.text.splitlines())
        else:
            parser.parse([])
    except requests.RequestException:
        parser.parse([])
    return parser


def crawl():
    session = requests.Session()
    session.headers.update({'User-Agent': USER_AGENT, 'Accept': 'text/html,application/xhtml+xml'})
    frontier = deque()
    seen = set()
    records = []
    robots_cache = {}
    last_request = {}

    for seed in SEED_URLS:
        normalized = normalize_url(seed, seed)
        if normalized and allowed(normalized) and normalized not in seen:
            frontier.append((normalized, 0, None))
            seen.add(normalized)

    while frontier and len(records) < MAX_PAGES:
        url, depth, source = frontier.popleft()
        host = urlsplit(url).netloc
        wait = DELAY_SECONDS - (time.monotonic() - last_request.get(host, 0))
        if wait > 0:
            time.sleep(wait)

        robot = robots_cache.setdefault(host, robots_for(url, session))
        if not robot.can_fetch(USER_AGENT, url):
            logging.info('robots skip %s', url)
            records.append({'url': url, 'status': None, 'state': 'skipped', 'source': source})
            continue

        last_request[host] = time.monotonic()
        try:
            response = session.get(url, timeout=TIMEOUT, allow_redirects=True)
            final_url = normalize_url(response.url, url) or response.url
            content_type = response.headers.get('Content-Type', '').lower()
            record = {
                'url': url,
                'final_url': final_url,
                'status': response.status_code,
                'content_type': content_type,
                'fetched_at': datetime.now(timezone.utc).isoformat(),
                'source': source
            }
            if response.status_code >= 400:
                record['state'] = 'failed'
                records.append(record)
                continue
            if 'text/html' not in content_type and 'application/xhtml+xml' not in content_type:
                record['state'] = 'skipped'
                records.append(record)
                continue

            soup = BeautifulSoup(response.content, 'lxml')
            record['state'] = 'fetched'
            record['title'] = soup.title.get_text(' ', strip=True) if soup.title else None
            record['headings'] = [node.get_text(' ', strip=True) for node in soup.select('h1, h2, h3')]
            records.append(record)

            if depth >= MAX_DEPTH:
                continue
            for anchor in soup.select('a[href]'):
                child = normalize_url(anchor['href'], final_url)
                if child and allowed(child) and child not in seen:
                    seen.add(child)
                    frontier.append((child, depth + 1, url))
        except requests.exceptions.RequestException as error:
            logging.warning('request failed %s: %s', url, error)
            records.append({'url': url, 'state': 'failed', 'error': str(error), 'source': source})

    with open('crawl-results.json', 'w', encoding='utf-8') as output:
        json.dump(records, output, indent=2, ensure_ascii=False)
    logging.info('finished: %d records, %d URLs remaining', len(records), len(frontier))


if __name__ == '__main__':
    crawl()

python crawler.py

Robots.txt and polite crawling

robots.txt is a site-published file that communicates crawler access preferences for user agents and paths. It is not authentication and cannot grant permission to access private material. Read the rules for your named crawler user agent before fetching paths, and document how your implementation interprets them.

Apply throttling per host rather than only globally. A crawl delay is the waiting interval between requests. Conservative concurrency, bounded retries, reasonable page limits, and response-size limits reduce load. An official API, downloadable dataset, or explicit permission is preferable when available, especially for large or repeated collections.

Retries, backoff, and reliability

Retry only failures that may be temporary, such as connection resets, timeouts, selected 5xx responses, and 429 responses. Use a maximum attempt count and exponential backoff, for example delay = min(cap, base * 2 ** attempt), with optional jitter. Do not endlessly retry 404, 401, or other permanent client errors.

import random
import time

for attempt in range(3):
    try:
        response = session.get(url, timeout=TIMEOUT)
        if response.status_code not in {429, 500, 502, 503, 504}:
            break
    except requests.RequestException:
        if attempt == 2:
            raise
    delay = min(30, 2 ** attempt) + random.uniform(0, 0.5)
    time.sleep(delay)

Check Content-Type before parsing and reject responses larger than your configured maximum. Record status, final URL, redirect history, response headers of interest, timestamps, parsing errors, and request errors. Logging should show progress, warnings, failures, and final counts for fetched, failed, skipped, and remaining URLs.

Storing and exporting crawl results

A JSON record can contain the requested URL, final URL, status code, content type, title, headings, extracted fields, fetch timestamp, source URL, depth, state, and error message. CSV is convenient for flat reports such as URL, title, status, and timestamp. JSON is better for lists and nested metadata.

For larger crawls, use a database. A practical schema includes:

pages: normalized URL, final URL, canonical URL, title, status, content type, content hash, depth, fetched timestamp, and error.
links: source page ID, target normalized URL, anchor text, discovery timestamp, and whether the target was internal.
crawl_events: URL, event type, attempt number, timestamp, response status, elapsed time, and diagnostic message.

Persisting these fields makes output reproducible and auditable. A content hash supports incremental crawling: fetch a page again, compare its hash, and send only changed records through downstream processing.

Testing and debugging

Develop against a controlled local or test website containing relative links, duplicate links, fragments, redirects, missing titles, malformed markup, unsupported resources, and intentional error responses.

Test that relative links resolve against the correct response URL.
Test removal of fragments and your chosen trailing-slash and query-parameter policy.
Test domain and path filtering, including redirect destinations.
Test that duplicate discovery results in one scheduled URL.
Test maximum depth and maximum page limits.
Test parser behavior when title, metadata, headings, or href attributes are missing.
Test timeout, rate-limit, server-error, and unsupported-content handling with mocked responses.
Test that the crawler terminates when the frontier is empty or a configured limit is reached.

When expected links or fields are absent, inspect the received status code, final URL, content type, response body, and redirect history. The page may be generated by JavaScript, may require authentication, or may be an error, login, or consent page rather than the expected document.

Common crawler failures

Problem — Likely cause — Resolution

Same content is visited repeatedly — Missing normalization, fragments, tracking parameters, or late visited marking — Normalize before membership checks and mark URLs seen when scheduled.

Relative links are incorrect — href values are treated as standalone URLs — Resolve with urljoin using the current response URL.

The crawler leaves the site — No hostname allowlist or redirect validation — Check discovered URLs and final redirect destinations.

Requests are blocked or rate-limited — Requests are too frequent, the client is unclear, or automated access is restricted — Slow down, identify the client, follow published rules, and use a permitted API or request access.

Expected data is missing — Selector mismatch, client-side rendering, or an error/login response — Save and inspect the response, verify status and content type, revise selectors, or use an authorized rendering or data source.

The crawler never finishes — Unbounded pagination, calendars, search, or faceted URLs — Set page and depth limits, reject patterns, canonicalize URLs, and stop at the crawl budget.

JavaScript-rendered pages

Many pages return useful HTML directly. Others insert content after JavaScript runs. A normal HTTP client receives the initial response and does not execute that JavaScript. Before using browser automation, check whether the site provides an API, embedded structured data, or a permitted server-rendered endpoint. If rendering is necessary and authorized, a browser-based approach can execute scripts, but it uses more CPU, memory, and time and requires stricter concurrency limits.

Scaling beyond one process

A single-process crawler is simple but limited by network latency, one frontier, one failure domain, and local memory. Asynchronous or concurrent fetching can improve throughput, but concurrency must be bounded per host and coordinated with delays, robots rules, retries, and service limits.

For resumable systems, persist the frontier and state transitions. Add checkpoints, caching, leases for work items, retry counters, and graceful shutdown handling. Persistent frontiers allow a crawl to resume after interruption. Incremental crawling revisits selected URLs based on change frequency, cache validators such as ETag or Last-Modified, and content hashes rather than downloading everything every time.

Approach — Best for — Advantages — Tradeoffs

Requests plus Beautiful Soup — Small educational crawlers and focused collectors — Easy to understand and customize — You must implement scheduling, retries, persistence, and throttling.

Async HTTP client — Many permitted network requests with predictable limits — Efficient connection use and concurrency — More complex state, cancellation, and host-level rate control.

Browser automation — Authorized JavaScript-rendered pages — Executes scripts and models browser behavior — Heavy resource use and more complicated debugging.

Mature crawler framework — Production crawls, pipelines, and resumable jobs — Established middleware, scheduling, throttling, and exporters — Requires learning framework conventions and configuration.

Frameworks such as Scrapy become more appropriate when you need persistent scheduling, retries, item pipelines, middleware, concurrency controls, and large-scale exports. Building from scratch remains valuable for learning and for narrowly scoped tools where a framework would add unnecessary complexity.

Configuration checklist

seed_urls
allowed_domains
max_pages
max_depth
request_timeout_seconds
requests_per_second_per_host
user_agent
respect_robots_txt
allowed_content_types
output_path

Exam-relevant notes

A crawler discovers and schedules resources; a scraper focuses on extracting selected data.
Normalize URLs before deduplication, remove fragments, and define a deliberate query-parameter policy.
Use a queue for breadth-first traversal and a stack for depth-first traversal.
Mark URLs as seen when scheduled to prevent races and cycles.
Always configure explicit timeouts and handle request exceptions.
Do not treat all HTTP failures alike: 4xx, 429, redirects, and 5xx responses require different policies.
Check content type and size before parsing.
Respect robots.txt, published rules, permissions, rate limits, and applicable requirements.
Record final URLs, status codes, timestamps, errors, and source links for auditing.
Use strict page, depth, domain, path, and URL-pattern limits to guarantee termination.

Next steps

Run the example against a permitted test site, add unit tests for normalization and filtering, then extend the output with link records and structured metadata. Review the broader crawler curriculum for related activities.