IT Course Directory: VMware, Linux, Networking, and Raspberry Pi

Create a Multithreaded Web Crawler in Python

Learn to build a responsible multithreaded Python web crawler with HTML link extraction, domain filtering, persistent URL state, normalization, and worker queues.

A web crawler is a program that automatically visits web pages and discovers additional pages through links. Crawlers are useful for site audits, link checking, sitemap generation, and collecting pages for a later processing task.

Crawling is not the same as indexing or searching. Crawling retrieves pages and discovers URLs. Indexing analyzes retrieved content and stores information about it. A search engine uses an index to answer queries. This lesson focuses on crawling, not building a search index.

We will build a crawler that starts with a seed URL, stays within a selected hostname, saves its progress to files, and uses worker threads to fetch several pages concurrently.

How a Web Crawler Works

The initial page is called the seed URL. The URLs waiting to be visited form the crawl frontier. In this project, the frontier is persisted as a queue file and also loaded into a Python set when state is processed.

1. Read the seed URL and crawler configuration.
2. Fetch a page over HTTP or HTTPS.
3. Parse the downloaded HTML and inspect anchor href attributes.
4. Resolve relative links against the page that contained them.
5. Normalize links and discard fragments, unsupported schemes, malformed URLs, and links outside the target host.
6. Add unseen links to the pending URL collection.
7. Move the processed URL from pending state to the crawled set.
8. Repeat until no pending URLs remain.

Project Design

The implementation uses Python's standard library. urllib.request performs HTTP requests, urllib.parse parses and joins URLs, html.parser reads HTML, pathlib handles files, and threading plus queue.Queue provide concurrency.

Useful inputs are the project name, seed URL, target hostname, worker count, and queue-file paths. The main outputs are a pending URL list and a crawled URL list. A production crawler may also keep retry, error, HTTP status, and response-time records.

Create the Project

mkdir -p crawler_project/crawler crawler_project/data/example-site
cd crawler_project
touch crawler/__init__.py crawler/main.py crawler/spider.py crawler/parser.py crawler/urls.py crawler/state.py
touch data/example-site/pending.txt data/example-site/crawled.txt

Place one normalized URL per line in each state file. The pending file represents URLs not yet completed. The crawled file represents URLs that have been processed. A new project should put its seed URL in the pending file and leave the crawled file empty.

Persistent Queue and Crawled State

Durable state matters because network programs can stop unexpectedly. If the process is interrupted after several pages, the next run can reconstruct its frontier from disk rather than starting over.

Sets are ideal for URL state because membership checks and duplicate removal are efficient. The two sets should be disjoint: a URL should not remain pending after it has been recorded as crawled.

from pathlib import Path


def read_urls(path: Path) -> set[str]:
    if not path.exists():
        return set()
    return {
        line.strip()
        for line in path.read_text(encoding="utf-8").splitlines()
        if line.strip()
    }


def write_urls(path: Path, urls: set[str]) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    path.write_text("".join(url + "\n" for url in sorted(urls)), encoding="utf-8")


def initialize_project(seed: str, pending: Path, crawled: Path) -> None:
    pending_urls = read_urls(pending)
    completed_urls = read_urls(crawled)
    if not pending.exists() and not crawled.exists():
        pending_urls.add(seed)
    pending_urls -= completed_urls
    write_urls(pending, pending_urls)
    write_urls(crawled, completed_urls)

The set difference operation pending_urls -= completed_urls removes anything accidentally present in both files. Union, written as left | right, combines discovered URLs with existing pending URLs. Membership checks such as candidate not in pending_urls prevent duplicate insertion.

Normalize and Filter URLs

An absolute URL contains a scheme and host, such as https://example.test/about. A relative URL such as /about must be interpreted using the current page as its base. A fragment such as #contact identifies a location inside a document and normally should not create a second crawl job.

URL normalization converts equivalent representations into a consistent form before comparison. The policy below lowercases the scheme and hostname, removes default HTTP and HTTPS ports, converts an empty path to /, removes fragments, and removes a trailing slash from non-root paths. That last rule is a project policy; some servers treat /about and /about/ as different resources, so change it if that distinction matters.

from urllib.parse import urljoin, urlsplit, urlunsplit


def normalize_url(raw: str, base: str | None = None) -> str | None:
    if not raw:
        return None
    try:
        absolute = urljoin(base, raw) if base else raw
        parts = urlsplit(absolute)
        scheme = parts.scheme.lower()
        if scheme not in {"http", "https"}:
            return None
        hostname = parts.hostname
        if not hostname:
            return None
        hostname = hostname.lower().rstrip(".")
        try:
            port = parts.port
        except ValueError:
            return None
        if (scheme == "http" and port == 80) or (scheme == "https" and port == 443):
            port = None
        netloc = hostname if port is None else f"{hostname}:{port}"
        path = parts.path or "/"
        if path != "/":
            path = path.rstrip("/") or "/"
        return urlunsplit((scheme, netloc, path, parts.query, ""))
    except (TypeError, ValueError):
        return None


def same_host(url: str, target_hostname: str) -> bool:
    try:
        hostname = urlsplit(url).hostname
    except (TypeError, ValueError):
        return False
    return hostname is not None and hostname.lower().rstrip(".") == target_hostname.lower().rstrip(".")

This example uses an exact-host policy. A target such as www.example.test does not automatically include blog.example.test. If subdomains are authorized, define that policy explicitly and validate a hostname boundary, for example by accepting the exact host or a name ending in .example.test. Never accept arbitrary suffix matches because notexample.test would incorrectly pass a naive test.

Fetch HTML and Extract Links

An HTTP response has a status code, headers, encoding information, and a body. The body may not be HTML, so check the content type before parsing when possible. The basic crawler below decodes the response using the server-declared charset when available and falls back to UTF-8 with replacement characters.

from html.parser import HTMLParser
from urllib.request import Request, urlopen


class LinkParser(HTMLParser):
    def __init__(self) -> None:
        super().__init__()
        self.links: list[str] = []

    def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
        if tag.lower() != "a":
            return
        for name, value in attrs:
            if name.lower() == "href" and value:
                self.links.append(value)


def fetch_html(url: str) -> tuple[str, str] | None:
    request = Request(url, headers={"User-Agent": "ExampleCrawler/1.0"})
    try:
        with urlopen(request, timeout=10) as response:
            content_type = response.headers.get_content_type()
            if content_type != "text/html":
                return None
            charset = response.headers.get_content_charset() or "utf-8"
            body = response.read().decode(charset, errors="replace")
            return response.geturl(), body
    except (OSError, UnicodeError, ValueError):
        return None


def extract_links(page_url: str, html: str, target_hostname: str) -> set[str]:
    parser = LinkParser()
    parser.feed(html)
    found: set[str] = set()
    for raw_href in parser.links:
        candidate = normalize_url(raw_href, page_url)
        if candidate and same_host(candidate, target_hostname):
            found.add(candidate)
    return found

Redirects require an additional safety check. The server may redirect an in-scope URL to another host. The spider must validate response.geturl(), the final URL returned by the request, before parsing its HTML. If the final hostname is outside the target policy, discard the response body.

Build the Spider

A spider is the component responsible for processing one page. It fetches a URL, checks the final redirect destination, extracts eligible links, updates persistent state, and prevents repeated work. Link extraction is kept separate from state management so each responsibility can be tested independently.

from pathlib import Path
from threading import Lock


class Spider:
    def __init__(self, project: str, target_hostname: str, data_root: Path) -> None:
        self.project = project
        self.target_hostname = target_hostname
        directory = data_root / project
        self.pending_path = directory / "pending.txt"
        self.crawled_path = directory / "crawled.txt"
        self.state_lock = Lock()

    def pending_urls(self) -> set[str]:
        return read_urls(self.pending_path)

    def crawled_urls(self) -> set[str]:
        return read_urls(self.crawled_path)

    def crawl_one(self, url: str) -> None:
        with self.state_lock:
            pending = self.pending_urls()
            crawled = self.crawled_urls()
            if url in crawled or url not in pending:
                return

        result = fetch_html(url)
        discovered: set[str] = set()
        if result is not None:
            final_url, html = result
            if same_host(final_url, self.target_hostname):
                discovered = extract_links(final_url, html, self.target_hostname)

        with self.state_lock:
            pending = self.pending_urls()
            crawled = self.crawled_urls()
            discovered -= pending
            discovered -= crawled
            pending |= discovered
            pending.discard(url)
            crawled.add(url)
            pending -= crawled
            write_urls(self.pending_path, pending)
            write_urls(self.crawled_path, crawled)

Here, a failed request is marked as crawled so that a permanently broken URL does not block completion. A more advanced design can instead keep failed URLs in a retry file with a retry count and backoff delay.

Worker Threads and a Real Controller Loop

A worker thread repeatedly takes a URL from a thread-safe queue, calls the spider, and signals that the task is complete. Threads are useful here because waiting for network responses is mostly I/O. They do not remove the need for rate limits or host policies.

The controller must continue refilling the work queue after workers discover new links. Submitting only the initial pending URLs and then exiting is incomplete: those workers may add URLs after the producer has stopped, leaving the newly discovered frontier untouched.

import threading
from queue import Queue
from pathlib import Path


def run_workers(spider: Spider, worker_count: int = 4) -> None:
    jobs: Queue[str | None] = Queue()
    scheduled: set[str] = set()
    scheduled_lock = threading.Lock()

    def worker() -> None:
        while True:
            url = jobs.get()
            try:
                if url is None:
                    return
                try:
                    spider.crawl_one(url)
                except Exception as error:
                    print(f"worker error for {url}: {error}")
            finally:
                jobs.task_done()

    workers = [
        threading.Thread(target=worker, name=f"crawler-{number}", daemon=True)
        for number in range(worker_count)
    ]
    for thread in workers:
        thread.start()

    try:
        while True:
            pending = spider.pending_urls()
            crawled = spider.crawled_urls()
            candidates = pending - crawled
            with scheduled_lock:
                new_candidates = candidates - scheduled
                scheduled.update(new_candidates)
            for url in new_candidates:
                jobs.put(url)

            if not new_candidates:
                if jobs.unfinished_tasks == 0:
                    latest_pending = spider.pending_urls()
                    latest_crawled = spider.crawled_urls()
                    if not (latest_pending - latest_crawled - scheduled):
                        break
            jobs.join()

        jobs.join()
    finally:
        for _ in workers:
            jobs.put(None)
        for thread in workers:
            thread.join()

The controller repeatedly reads persisted state, computes URLs that are pending but neither completed nor already scheduled, and submits them. It calls jobs.join() after each batch, allowing workers to finish and persist newly discovered links before the next scan. The shared file state is protected by state_lock, while Queue safely coordinates producer and consumer threads.

Daemon threads are convenient because they do not keep the interpreter alive after the controller exits, but graceful shutdown is still preferable. The sentinel values in the finally block tell workers to stop after all submitted jobs finish.

Main Entry Point

from pathlib import Path
from urllib.parse import urlsplit


PROJECT = "example-site"
SEED_URL = "https://example.test/"
DATA_ROOT = Path("data")
WORKER_COUNT = 4


if __name__ == "__main__":
    seed = normalize_url(SEED_URL)
    if seed is None:
        raise SystemExit("Invalid seed URL")
    hostname = urlsplit(seed).hostname
    if hostname is None:
        raise SystemExit("Seed URL has no hostname")
    pending = DATA_ROOT / PROJECT / "pending.txt"
    crawled = DATA_ROOT / PROJECT / "crawled.txt"
    initialize_project(seed, pending, crawled)
    spider = Spider(PROJECT, hostname, DATA_ROOT)
    run_workers(spider, WORKER_COUNT)
    print(f"Pending: {len(spider.pending_urls())}")
    print(f"Crawled: {len(spider.crawled_urls())}")

Replace the configuration values with a site and project you are permitted to crawl. Run the entry point from the project directory:

python crawler/main.py

Watch the progress output and inspect data/example-site/pending.txt and data/example-site/crawled.txt. To resume, stop the program and run it again. Existing files reconstruct the state. The seed is added only when both state files represent a new project.

Understanding the Complete Crawl Lifecycle

For each job, the spider first confirms that the URL is still pending and not already crawled. It downloads the page, validates the final redirect host, extracts links, normalizes and filters them, and then performs one locked state transition. Newly discovered URLs are added to pending state; the current URL is removed from pending state and added to crawled state. The controller sees those new pending URLs during its next refill cycle.

Keeping the in-memory queue and file state conceptually separate is important. The thread queue contains jobs currently assigned or waiting for a worker. The pending file contains all URLs not yet completed, including URLs that have not been submitted in the current controller cycle. The scheduled set prevents the controller from submitting the same pending URL repeatedly while its job is still in flight.

Troubleshooting

Pages repeat endlessly

Check membership against both pending and crawled sets. Normalize before every comparison, remove fragments, and verify that the processed URL is removed from pending state and written to crawled state. Also check that workers do not update files with unsynchronized read-modify-write operations.

Links leave the target site

Confirm that every candidate is parsed with urlsplit and compared to the selected hostname. Also apply the same check to the final URL after redirects. Decide explicitly whether subdomains are included; an exact-host policy is safer by default.

Relative links fail

Call urljoin(current_page_url, href) before normalization and domain filtering. A link beginning with / is relative to the host root, while a link such as next.html is relative to the current page path.

No links are found

Check the HTTP status and content type, confirm that the parser receives the response body, and verify that it inspects anchor href attributes. A basic HTML crawler sees only links present in downloaded HTML; links inserted later by client-side JavaScript require a browser-based or rendering-capable approach.

Workers are idle or the program never finishes

Verify that the controller calls jobs.put, that every worker calls task_done in a finally block, and that the controller rescans state after jobs.join(). Log worker exceptions instead of allowing them to disappear. Be careful when checking queue counters: the persisted files, not only the in-memory queue, determine whether newly discovered work exists.

Progress disappears after stopping

State must be written during crawl updates, not only when the program exits. The example rewrites both files after each processed URL. For larger crawls, use a database or atomic temporary-file replacement to improve durability.

Downloads fail

Network errors, redirects, timeouts, invalid URLs, and server errors are normal crawler conditions. Use timeouts, catch request exceptions, report failures, and choose a retry policy. Do not retry indefinitely or immediately at high speed.

Exam-Relevant Notes

• A web crawler discovers and retrieves pages; an indexer analyzes and stores content for searching.
• The seed URL starts the crawl, and the crawl frontier contains URLs waiting for work.
• Sets provide unique URL collections and fast membership checks.
• Relative URLs must be resolved against the current page before filtering.
• Fragments usually should be removed before URL comparison.
• mailto, javascript, and tel are unsupported schemes for an HTTP page crawler.
• Hostname comparison must include a clearly documented subdomain policy.
• A thread-safe queue coordinates workers, but file-based state updates still need a lock.
• Every queued task needs one matching task_done() call.
• A concurrent controller must refill work after workers discover new URLs.
• Persistent pending and crawled files allow an interrupted crawl to resume.
• Redirect destinations must be checked before accepting their content.

Next Improvements

This crawler is a practical foundation, not a complete production crawler. Useful extensions include robots-policy handling, per-host rate limiting, exponential backoff, redirect and status recording, retry limits, canonical-link processing, maximum depth, breadth-first scheduling, database-backed state, and asynchronous networking. For JavaScript-heavy sites, a simple HTML parser may not observe the links visible in a browser.

Review the related web crawler course and its practice activity to reinforce the design.