Python Web Crawler

Create the Spider Crawler Class in Python

Learn to build a Python Spider class that fetches pages, extracts and filters links, manages queue and crawled sets, and persists crawl state.

A web crawler visits webpages, retrieves HTML, discovers links, and continues through selected links. A spider is a common name for the crawler class or process that performs this work.

In this lesson, you will create spider.py and define a Spider class. The class coordinates the crawl workflow, while other modules handle link parsing, domain checks, and file storage.

This lesson assumes that you understand Python modules and project structure, sets, classes, basic HTML, and the project's link_finder.py, domain.py, and general.py modules.

The crawler's role in the project

The crawler is an orchestrator. It decides which URL to process next and coordinates the other components; it should not contain all parsing and file-management logic itself.

1. Load unfinished URLs from the queue file.
2. Choose one URL from the in-memory queue.
3. Request the page and read its HTML response.
4. Pass the page URL and HTML to LinkFinder.
5. Check each discovered URL against the permitted domain and known URL sets.
6. Add acceptable new URLs to the queue.
7. Remove the completed URL from the queue and add it to the crawled set.
8. Write both sets back to their files.

The supporting modules must exist and be importable before the spider can run. For background, see spider concepts and web crawler requirements.

Create spider.py

Create a dedicated file named spider.py in the same project package or directory as the supporting modules. A class is useful here because one object can group the crawl configuration, current state, and operations that use that state.

from urllib.request import urlopen
from urllib.error import HTTPError, URLError
from urllib.parse import urlparse

from link_finder import LinkFinder
from domain import get_domain_name, is_valid_domain
from general import file_to_set, set_to_file


class Spider:
    project_name = "example_project"
    base_url = "https://example.com"
    domain_name = get_domain_name(base_url)
    queue_file = project_name + "/queue.txt"
    crawled_file = project_name + "/crawled.txt"

    queue = set()
    crawled = set()

    def __init__(self, project_name, base_url, domain_name,
                 queue_file, crawled_file):
        self.project_name = project_name
        self.base_url = base_url
        self.domain_name = domain_name
        self.queue_file = queue_file
        self.crawled_file = crawled_file

        # These are initialized before loading persistent state.
        self.queue = set()
        self.crawled = set()

        self.load_state()
        if not self.queue and not self.crawled:
            self.queue.add(self.base_url)
            self.save_state()

    def load_state(self):
        self.queue = file_to_set(self.queue_file)
        self.crawled = file_to_set(self.crawled_file)

    def save_state(self):
        set_to_file(self.queue, self.queue_file)
        set_to_file(self.crawled, self.crawled_file)

    def fetch(self, url):
        try:
            with urlopen(url, timeout=15) as response:
                body = response.read()
                effective_url = response.geturl()
                encoding = response.headers.get_content_charset() or "utf-8"
                html = body.decode(encoding, errors="replace")
                return effective_url, html
        except (HTTPError, URLError, TimeoutError, ValueError) as error:
            print(f"Could not fetch {url}: {type(error).__name__}: {error}")
            return None, None

    def crawl_page(self, url):
        effective_url, html = self.fetch(url)
        if html is None:
            return False

        finder = LinkFinder(effective_url, html)
        for link in finder.get_links():
            if self.is_new_allowed_link(link):
                self.queue.add(link)

        self.queue.discard(url)
        self.queue.discard(effective_url)
        self.crawled.add(url)
        if effective_url != url:
            self.crawled.add(effective_url)
        self.save_state()
        return True

    def is_new_allowed_link(self, url):
        return (
            is_valid_domain(url, self.domain_name)
            and url not in self.queue
            and url not in self.crawled
        )

    def crawl_once(self):
        if not self.queue:
            return False
        url = next(iter(self.queue))
        print(f"Crawling: {url}")
        return self.crawl_page(url)

Configuration and crawl state

The class needs configuration describing the project and its scope. A base URL is the initial page from which the crawl begins. A domain name identifies the site or allowed subdomain scope.

Class variables and instance variables

A class variable is defined on the class and can be accessed by all instances. A class-level set can therefore represent shared state for one project. This pattern is useful in a simple file-backed crawler that creates several Spider objects with the same project configuration.

An instance variable belongs to one object and is normally assigned through self. In the example, assigning self.queue = set() and self.crawled = set() creates independent sets for each object. This is safer when multiple projects or crawler instances may run at the same time.

Fetching webpage content with urlopen

urlopen opens an HTTP or HTTPS URL and returns a response object. The response body is initially bytes, so the crawler must call read() before parsing and decode those bytes into text.

• Use a with statement so the response is closed even when processing finishes with an error.
• Use the response's declared character set when available. Falling back to UTF-8 with replacement prevents one bad byte sequence from stopping the crawl.
• Use response.geturl() when the effective URL matters. A redirect can send the request from the original URL to a different final URL.
• Set a timeout so an unresponsive server does not hold the crawler forever.

Unexpected content is also possible: a URL may return a PDF, image, login page, or error document rather than HTML. A production crawler should inspect the response status and content type before sending the body to an HTML parser.

Integrate LinkFinder

The crawler supplies two values to the link finder: the current page URL and the decoded HTML. LinkFinder parses anchor elements and resolves relative links, allowing the crawler to work with the resulting URLs rather than embedding HTML parsing logic in spider.py.

effective_url, html = spider.fetch(current_url)
if html is not None:
    finder = LinkFinder(effective_url, html)
    discovered_links = finder.get_links()
    for link in discovered_links:
        # Validate and deduplicate before queueing.
        if spider.is_new_allowed_link(link):
            spider.queue.add(link)

Filter URLs before queueing

Every discovered URL must be checked before it enters the queue. Domain validation prevents the crawler from leaving the intended website. A valid URL can still be rejected if it is already queued or already crawled.

A set stores unique values and supports fast membership checks. This is why both queue and crawled are sets rather than lists.

URL normalization, also called canonicalization, reduces duplicate representations. Depending on project requirements, consider removing fragments such as #section, standardizing trailing slashes, comparing hostnames without case differences, handling default ports, and deciding how query strings should be treated. Do not remove query parameters automatically if they change page content. Also decide whether HTTP and HTTPS should be considered separate URLs.

Drive the queue and persist state

The queue is the collection of discovered URLs that have not yet been crawled. The crawled set contains URLs already processed. The queue file provides durable storage for unfinished work, and the crawled file provides durable storage for completed work.

The essential state transition is:

pending queue  --successful handling-->  crawled set
                         |
                         +--valid new links--> pending queue

Persist both files after the transition. If the program stops, the next run can reload the queue and crawled files and continue without repeating recorded work.

Initialize a new crawl and resume an existing one

When a project is new, configure its name, homepage, allowed domain, queue path, and crawled path. The constructor places the homepage in the queue if both state files are empty. The first crawl operation fetches that homepage, discovers internal links, queues them, and records the homepage as crawled.

spider = Spider(
    project_name="example_project",
    base_url="https://example.com/",
    domain_name="example.com",
    queue_file="example_project/queue.txt",
    crawled_file="example_project/crawled.txt",
)

spider.crawl_once()

For an existing project, initialization loads the queue and crawled files instead of blindly adding the homepage again. A larger program can call crawl_once() repeatedly:

while spider.queue:
    spider.crawl_once()

For a complete crawler project, review creating queue and crawled files, parsing HTML, and adding and deleting URLs.

Handle failures without stopping the crawl

Network failures are normal. Invalid URLs, HTTP errors, SSL problems, timeouts, redirects, unavailable servers, and unexpected content should not terminate the entire crawl.

• Catch request-related exceptions around urlopen.
• Report the current URL, exception type, and useful error details.
• Continue with another queued URL after a failed request.
• Choose a clear policy for failed URLs: leave them queued for retry, move them to a failure log, or mark them handled only if the project explicitly treats failure as completion.
• Do not add a URL to crawled before its processing policy has completed successfully.

The example leaves a failed URL in the queue by returning False before the queue-to-crawled transition. This supports a later retry, but a real crawler should prevent endless retries by recording attempt counts, backoff times, or a failure file.

Project layout and prerequisites

example_project/
    spider.py
    link_finder.py
    domain.py
    general.py
    queue.txt
    crawled.txt

• Use Python 3 and a valid HTTP or HTTPS starting URL.
• Ensure the supporting modules are in the Python import path.
• Ensure the project directory and state files are writable.
• Keep the configured paths consistent with the project directory.
• For responsible crawling, add rate limiting, request identification, robots.txt handling, and appropriate scope rules before crawling a real site.

Troubleshooting

Import errors

Confirm that link_finder.py, domain.py, and general.py exist, that their names and capitalization match the imports, and that the program runs from the intended project or package context.

The queue does not grow

Check that the response body is decoded and passed to LinkFinder, inspect the links returned by the extractor, verify the configured domain, and confirm that accepted links are added to the set before the queue file is written.

The same page is visited repeatedly

Verify the queue-to-crawled transition, including removal from queue, addition to crawled, and persistence of both sets. Normalize equivalent URL forms when necessary.

The crawler visits external sites

Validate every extracted link. Check the allowed domain derived from the base URL, and test the helper with an internal URL, an allowed subdomain, and an unrelated external domain.

Network errors stop execution

Catch exceptions around urlopen, log the failed URL and exception type, and continue processing other queued URLs. Add a defined retry or failure-tracking policy.

Files and memory disagree

Call the file-writing utilities immediately after state changes, verify that the paths point to the expected directory, check write permissions, and inspect the saved files while debugging.

Exam-relevant summary

• Spider: coordinates the crawl; it is not the HTML parser or file utility.
• Queue: URLs waiting for processing.
• Crawled set: URLs already processed.
• Sets: prevent duplicate queue entries and repeated work.
• urlopen: opens a URL and returns a response whose body must be read and decoded.
• Redirects: make the final response URL relevant to state tracking.
• Filtering: reject external, invalid, queued, and already-crawled URLs.
• Persistence: synchronize in-memory sets with queue and crawled files so a crawl can resume.
• Failure policy: never mark a page crawled before the chosen processing policy finishes.