VMware ESXi and vSphere Cluster Management

Create a Web Crawler in Python

Learn to build a responsible, domain-scoped Python web crawler with requests, Beautiful Soup, URL normalization, persistence, error handling, and resumable crawl state.

What a Web Crawler Does

A web crawler is a program that automatically discovers and retrieves web resources by following hyperlinks according to defined rules. Crawling usually begins with a seed URL, fetches that page, extracts links, and adds eligible links to a crawl frontier: the collection of URLs waiting to be processed.

A crawler is related to, but different from, several other tools:

This lesson builds a single-worker, domain-scoped crawler. It collects normalized internal URLs, external links, response metadata, and selected page information without entering authenticated or private areas.

Responsible and Permitted Crawling

Automated access may be restricted by law, contract, site policy, robots.txt, authentication requirements, or technical controls. Crawl only websites you are authorized to test, preferably a local fixture or a small explicitly permitted site.

• Identify the client with a meaningful User-Agent, including a contact address when appropriate.
• Read and follow applicable robots.txt rules. A robots file is a site-hosted convention describing which paths automated clients may access; it is not a permission to bypass other restrictions.
• Use a politeness delay, conservative concurrency, timeouts, and a bounded page/request limit.
• Do not collect passwords, personal data, private content, or authenticated pages without explicit authorization.
• Never bypass rate limits, access controls, CAPTCHAs, or other defenses.
• Stop or slow down after responses such as 403 or 429.

Crawler Architecture

Keep these categories separate:

• Pending: eligible URLs waiting in the frontier.
• Completed: URLs whose HTML page was successfully handled.
• Rejected: discovered URLs that violate scope, scheme, file-type, pattern, depth, or robots rules.
• Failed: requests or parsing operations that could not be completed.

The lifecycle is: seed URL → normalize → enqueue → fetch → validate the final redirect URL → check content type → parse HTML → resolve and normalize links → filter and deduplicate → persist results → repeat.

Project Setup

python -m venv .venv
# macOS/Linux
source .venv/bin/activate
# Windows PowerShell
.venv\Scripts\Activate.ps1
python -m pip install requests beautifulsoup4

A small project can use this structure:

crawler/
    __init__.py
    main.py
    crawler.py
    link_parser.py
    storage.py
    config.py
data/

Run the application with:

python -m crawler.main

Configuration

URL Resolution and Normalization

An absolute URL contains a scheme and host, such as https://example.test/about. A relative URL, such as ../products, needs the current page as context. Python's urljoin performs this resolution.

Normalization should lower-case the hostname, remove fragments, remove default ports, and preserve meaningful paths, query strings, and non-default ports. Do not blindly remove query strings: they may identify different resources. Tracking-parameter removal should be an explicit policy. Trailing-slash treatment, case-sensitive paths, canonical links, and redirect destinations also need deliberate policies.

from urllib.parse import parse_qsl, urlencode, urldefrag, urljoin, urlsplit, urlunsplit

TRACKING = {"utm_source", "utm_medium", "utm_campaign", "fbclid"}

def normalize_url(raw_url, base_url=None):
    if base_url:
        raw_url = urljoin(base_url, raw_url)
    if not raw_url:
        return None
    parts = urlsplit(raw_url.strip())
    if parts.scheme.lower() not in {"http", "https"} or not parts.hostname:
        return None
    host = parts.hostname.lower()
    port = parts.port
    default_port = (parts.scheme.lower() == "http" and port == 80) or (parts.scheme.lower() == "https" and port == 443)
    netloc = host if not port or default_port else f"{host}:{port}"
    query = urlencode([(k, v) for k, v in parse_qsl(parts.query, keep_blank_values=True)
                       if k not in TRACKING])
    path = parts.path or "/"
    return urlunsplit((parts.scheme.lower(), netloc, path, query, ""))

Domain and Crawl-Scope Control

Compare parsed hostnames. A test such as "example.com" in url incorrectly accepts example.com.attacker.test. A strict policy accepts the configured host, and optionally hosts whose names end with .allowed-host.

Scope boundaries should be enforced before scheduling and again after redirects. Useful boundaries include hostname, scheme, depth, maximum pages, maximum requests, path patterns, and file types. For example, reject a URL whose path ends in .pdf, .jpg, or .zip, or require a path to match /docs/. External HTTP links may be saved as output while remaining unscheduled.

Link Parsing

from bs4 import BeautifulSoup
from urllib.parse import urlsplit

def extract_links(html, page_url):
    soup = BeautifulSoup(html, "html.parser")
    title = soup.title.get_text(" ", strip=True) if soup.title else ""
    headings = [h.get_text(" ", strip=True) for h in soup.find_all(["h1", "h2", "h3"])]
    links = []
    for anchor in soup.find_all("a", href=True):
        raw = anchor["href"].strip()
        if not raw or raw.startswith(("#", "mailto:", "tel:", "javascript:")):
            continue
        absolute = urljoin(page_url, raw)
        scheme = urlsplit(absolute).scheme.lower()
        if scheme in {"http", "https"}:
            links.append(absolute)
    return {"title": title, "headings": headings, "links": links}

Use an HTML parser rather than regular expressions because HTML contains nesting, entities, optional elements, and malformed markup. The same parser can collect images, canonical links, metadata, or selected content.

A Resumable Crawler Implementation

The following compact implementation shows the important behavior in one class. In a larger project, move configuration, parsing, and storage into the modules shown earlier.

import csv, json, logging, time
from collections import deque
from datetime import datetime, timezone
from pathlib import Path
from urllib.parse import urlsplit, urljoin
import requests
from bs4 import BeautifulSoup

class Crawler:
    def __init__(self, cfg):
        self.cfg = cfg
        self.out = Path(cfg["output_directory"]); self.out.mkdir(parents=True, exist_ok=True)
        self.pending_file = self.out / "pending.jsonl"
        self.completed_file = self.out / "completed.txt"
        self.rejected_file = self.out / "rejected.jsonl"
        self.failed_file = self.out / "failed.jsonl"
        self.links_file = self.out / "links.csv"
        self.log = logging.getLogger("crawler")
        logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
        self.session = requests.Session()
        self.session.headers.update({"User-Agent": cfg["user_agent"]})
        self.session.max_redirects = cfg.get("max_redirects", 5)
        self.pending, self.queued, self.completed = deque(), set(), set()
        self.rejected, self.failed, self.links = [], [], []
        self.requests = 0; self.last_request = {}
        self.load_state()
        seed = self.normalize(cfg["seed_url"])
        if seed and self.in_scope(seed): self.enqueue(seed, 0)

    def normalize(self, raw, base=None):
        from urllib.parse import urldefrag, urljoin, urlsplit, urlunsplit
        value = urljoin(base, raw) if base else raw
        value, _ = urldefrag(value.strip())
        p = urlsplit(value)
        if p.scheme.lower() not in {"http", "https"} or not p.hostname: return None
        host, port = p.hostname.lower(), p.port
        default = (p.scheme.lower(), port) in {("http", 80), ("https", 443)}
        netloc = host if not port or default else f"{host}:{port}"
        return urlunsplit((p.scheme.lower(), netloc, p.path or "/", p.query, ""))

    def in_scope(self, url):
        p = urlsplit(url); host = (p.hostname or "").lower()
        allowed = self.cfg["allowed_host"].lower()
        host_ok = host == allowed or (self.cfg.get("include_subdomains", False) and host.endswith("." + allowed))
        if p.scheme not in {"http", "https"} or not host_ok: return False
        path = p.path.lower()
        blocked = tuple(self.cfg.get("blocked_extensions", []))
        allowed_ext = tuple(self.cfg.get("allowed_extensions", []))
        if blocked and path.endswith(blocked): return False
        if allowed_ext and path.rsplit("/", 1)[-1].find(".") >= 0 and not path.endswith(allowed_ext): return False
        patterns = self.cfg.get("allowed_patterns", [])
        return not patterns or any(pattern in p.path for pattern in patterns)

    def reject(self, url, source, reason):
        record = {"url": url, "source": source, "reason": reason, "time": self.now()}
        self.rejected.append(record); self.append_json(self.rejected_file, record)

    def enqueue(self, url, depth, source=""):
        if not url: return
        if not self.in_scope(url): self.reject(url, source, "out_of_scope")
        elif depth > self.cfg.get("max_depth", 3): self.reject(url, source, "depth_limit")
        elif url not in self.queued and url not in self.completed:
            self.pending.append((url, depth)); self.queued.add(url)
            self.append_json(self.pending_file, {"url": url, "depth": depth})

    def fetch(self, url):
        host = urlsplit(url).hostname
        wait = self.cfg.get("delay_seconds", 1.0) - (time.monotonic() - self.last_request.get(host, 0))
        if wait > 0: time.sleep(wait)
        self.last_request[host] = time.monotonic(); self.requests += 1
        try:
            response = self.session.get(url, timeout=self.cfg.get("request_timeout_seconds", 10), allow_redirects=True)
            final_url = self.normalize(response.url)
            if not final_url or not self.in_scope(final_url):
                self.reject(response.url, url, "redirect_out_of_scope")
                return None, final_url, "redirect_out_of_scope"
            return response, final_url, None
        except requests.RequestException as exc:
            return None, None, type(exc).__name__

    def run(self):
        started = time.monotonic(); max_pages = self.cfg.get("max_pages", 100)
        while self.pending and len(self.completed) < max_pages and self.requests < self.cfg.get("max_requests", max_pages * 2):
            url, depth = self.pending.popleft()
            if url in self.completed: continue
            response, final_url, error = self.fetch(url)
            if error:
                record = {"url": url, "error": error, "time": self.now()}
                self.failed.append(record); self.append_json(self.failed_file, record)
                self.log.warning("fetch failed source=%s error=%s", url, error); continue
            status = response.status_code
            if status >= 400:
                record = {"url": url, "status": status, "time": self.now()}
                self.failed.append(record); self.append_json(self.failed_file, record)
                self.log.warning("HTTP failure url=%s status=%s", url, status); continue
            content_type = response.headers.get("Content-Type", "").lower()
            if "text/html" not in content_type and "application/xhtml+xml" not in content_type:
                self.reject(final_url, url, "non_html_content"); continue
            try:
                soup = BeautifulSoup(response.text, "html.parser")
                title = soup.title.get_text(" ", strip=True) if soup.title else ""
                for tag in soup.find_all("a", href=True):
                    target = self.normalize(tag["href"], final_url)
                    if not target: continue
                    relation = "internal" if self.in_scope(target) else "external"
                    self.links.append({"source": final_url, "target": target, "relation": relation})
                    if relation == "internal": self.enqueue(target, depth + 1, final_url)
                    else: self.reject(target, final_url, "out_of_scope")
                self.completed.add(final_url)
                self.completed.add(url)
                self.append_line(self.completed_file, final_url)
                self.log.info("completed url=%s status=%s title=%s", final_url, status, title)
            except Exception as exc:
                record = {"url": final_url, "error": "parse:" + type(exc).__name__, "time": self.now()}
                self.failed.append(record); self.append_json(self.failed_file, record)
        self.export_links()
        summary = {"pages_completed": len(self.completed), "requests": self.requests, "pending": len(self.pending), "rejected": len(self.rejected), "failed": len(self.failed), "elapsed_seconds": round(time.monotonic() - started, 2)}
        (self.out / "summary.json").write_text(json.dumps(summary, indent=2), encoding="utf-8")
        print("Crawl summary:", json.dumps(summary, sort_keys=True))

    def export_links(self):
        with self.links_file.open("w", newline="", encoding="utf-8") as file:
            writer = csv.DictWriter(file, fieldnames=["source", "target", "relation"]); writer.writeheader(); writer.writerows(self.links)

    def load_state(self):
        if self.completed_file.exists(): self.completed.update(x.strip() for x in self.completed_file.read_text().splitlines() if x.strip())
        if self.pending_file.exists():
            for line in self.pending_file.read_text().splitlines():
                item = json.loads(line); self.enqueue(item["url"], item["depth"])

    @staticmethod
    def now(): return datetime.now(timezone.utc).isoformat()
    @staticmethod
    def append_line(path, value):
        with path.open("a", encoding="utf-8") as file: file.write(value + "\n")
    @staticmethod
    def append_json(path, value):
        with path.open("a", encoding="utf-8") as file: file.write(json.dumps(value) + "\n")

The important redirect detail is self.session.max_redirects. requests.Session.get() does not accept a max_redirects keyword argument. The code also checks response.url after redirects and rejects a final destination outside the permitted scope before parsing it.

Application Entry Point

from .crawler import Crawler

CONFIG = {
    "seed_url": "https://allowed.example/",
    "allowed_host": "allowed.example",
    "include_subdomains": False,
    "max_pages": 100,
    "max_requests": 150,
    "max_depth": 3,
    "request_timeout_seconds": 10,
    "delay_seconds": 1.0,
    "max_redirects": 5,
    "user_agent": "ExampleDomainCrawler/1.0 (contact: crawler-owner@example.test)",
    "output_directory": "data",
    "blocked_extensions": [".pdf", ".zip", ".jpg", ".jpeg", ".png", ".gif", ".css", ".js"],
    "allowed_extensions": [],
    "allowed_patterns": []
}

if __name__ == "__main__":
    Crawler(CONFIG).run()

Replace the example host only with a permitted target. In production, load this configuration from environment variables, a TOML file, or command-line arguments rather than editing source code.

HTTP Response Handling

Retries must be bounded. A typical policy retries transient network errors and selected 5xx responses two or three times with delays such as 1, 2, and 4 seconds. Never retry indefinitely or use retries to bypass access controls. A redirect loop is bounded by the session's redirect limit.

Persistent State and Restartability

Line-oriented files are easy to inspect and append, while SQLite is a better choice when state must be updated transactionally or queried at scale. Persistence should be idempotent: repeating a state write must not create incorrect duplicate work. For stronger crash recovery, use SQLite transactions or atomically replace snapshot files.

robots.txt and Politeness

Before fetching pages on a host, retrieve the host's /robots.txt, evaluate the rules for your User-Agent, and refuse disallowed paths. Cache the result for the crawl rather than downloading it for every page. If the site publishes a crawl delay, use the larger of that value and your configured delay.

robots.txt retrieval is itself an HTTP request. Apply the same per-host delay bookkeeping to it, or perform it before the first page request and update the host's last-request timestamp afterward. Rate limiting should normally be per host, not merely global, because different hosts have separate infrastructure. Begin with one worker; add concurrency only with per-host limits and explicit authorization.

Breadth-First and Depth-First Traversal

A queue processed from the front gives breadth-first search: pages close to the seed are processed before deeper pages. This is useful for representative site coverage. A stack gives depth-first search: one branch is followed deeply before other branches. It can use less frontier memory but may spend too long in one section. The implementation uses a queue and stores each URL in queued before processing, preventing repeated frontier entries.

Testing and Validation

Use local fixture pages or a small permitted test site. Include pages containing repeated links, fragment variations, relative links, external links, a redirect, a missing page, and a non-HTML asset.

assert normalize_url("/guide#part-2", "https://example.test/docs/start") == "https://example.test/guide"
assert normalize_url("../products", "https://example.test/docs/start") == "https://example.test/products"
assert in_scope("https://example.test/about") is True
assert in_scope("https://example.test.attacker.test/") is False
assert in_scope("https://example.test/file.pdf") is False

Also verify that:

• Only permitted hostnames, schemes, extensions, patterns, depths, and redirect destinations enter the frontier.
• /page, /page#one, and repeated occurrences of /page produce one crawl target.
• 404, timeout, malformed-response, and non-HTML cases do not stop the remaining queue.
• External links appear in links.csv but do not become page requests.
• Stopping and restarting reloads pending and completed state without recrawling completed pages.
• summary.json and the log contain useful page, request, rejection, failure, status, and elapsed-time information.

Final Crawl Workflow and Summary

A complete run creates state, takes the next eligible frontier item, fetches it with a timeout and identifiable headers, validates status and content type, parses HTML, resolves and normalizes links, applies domain and boundary filters, records internal and external relationships, persists each event, and continues until the queue or configured limits are exhausted.

The implementation prints a concrete summary similar to this:

Crawl summary: {"completed": 42, "elapsed_seconds": 38.71, "failed": 3, "pending": 0, "rejected": 27, "requests": 49}

The exact field names in the sample implementation are pages_completed, requests, pending, rejected, failed, and elapsed_seconds. Treat this summary and the CSV as the first useful output of a site link collector.

Troubleshooting

• The same page repeats: normalize before membership checks, add URLs to the queued set when scheduled, and remove fragments.
• Unrelated sites are crawled: compare parsed hostnames, including after redirects; do not use substring checks.
• Links are malformed: resolve relative links against the source page and ignore non-HTTP schemes.
• The crawler hangs: set timeouts, finite page/request/depth limits, bounded retries, and a redirect limit.
• 403 or 429 responses appear: stop or slow down, review permission and robots rules, and do not bypass the restriction.
• Downloads are parsed as pages: inspect Content-Type and enforce blocked or allowed extensions.
• Progress disappears after restart: append state incrementally, flush files, and reload pending and completed records.

Extensions and Production Considerations

• Add cached robots.txt evaluation and crawl-delay support.
• Apply per-host rate limits, configurable concurrency, and connection pooling.
• Discover sitemaps, record canonical URLs, and collect titles, headings, images, metadata, and content hashes.
• Use a browser-based tool when important content is generated only after JavaScript executes.
• Move the frontier and state into SQLite or a database for transactional recovery and querying.
• For large authorized crawls, use distributed workers, durable queues, monitoring, caching, and scalable deduplication.

Exam-Relevant Notes

• A seed URL starts the crawl; the frontier holds eligible pending work; a visited or completed set prevents duplicate processing.
• URL normalization is required before deduplication.
• Hostname comparison is safer than substring matching.
• Relative links require the source page as a base URL.
• Fragments usually identify a document location and should be removed before HTTP fetching.
• Redirect destinations must be checked against scope, not trusted merely because the original URL was in scope.
• HTTP status, content type, timeout, parsing, rejection, and retry behavior are separate concerns.
• Persistence makes a crawl inspectable and resumable.

For a related implementation reference, see Create a Web Crawler in Python.