VMware ESXi and vSphere Cluster Management

Build the Spider Class for a Python Web Crawler

Learn how to create a Spider class that coordinates crawling, stores project settings, tracks queued and visited URLs, and restricts links by domain.

What the Spider module does

A web crawler is a program that automatically visits web pages and follows selected links. A Spider is the crawler class responsible for maintaining crawl state and coordinating the work.

The spider module coordinates this lifecycle:

1. Take one URL from the collection of pending URLs.
2. Request the page with urlopen.
3. Pass the returned HTML and page URL to LinkFinder.
4. Filter discovered links so only URLs in the allowed domain are accepted.
5. Add eligible, unseen links to the pending collection.
6. Record the processed URL in the crawled collection.
7. Save the updated collections to their files.

This lesson establishes the Spider's configuration and state. The complete methods for initialization, page processing, error handling, and repeated crawling can be added afterward.

Create the spider.py file

Create a file named spider.py in the crawler project. It should sit alongside the modules that perform specialized tasks:

• link_finder.py parses HTML and extracts hyperlinks.
• domain.py identifies the domain of a URL and supports scope checks.
• general.py provides file and set utilities for saving and loading crawler state.
• spider.py coordinates these components and owns the crawler's shared state.

Use Spider as the main crawler abstraction. Keeping coordination in its own module prevents the HTML parser and file utilities from having to know about the entire crawl lifecycle.

Import the crawler dependencies

Begin spider.py with imports for page retrieval, link extraction, domain checks, and persistent state:

from urllib.request import urlopen

from link_finder import LinkFinder
from domain import get_domain_name
from general import file_to_set, set_to_file

These names describe a typical project API. If your existing utility modules expose different function names, use those exported names consistently.

• urlopen opens an HTTP or HTTPS resource. The later page-processing method will use it to retrieve HTML.
• LinkFinder receives HTML and the current page URL, then identifies hyperlinks. The later crawl step will use it for discovery.
• get_domain_name extracts a host or domain-like identity from a URL. The crawler will use it to compare discovered links with the configured scope.
• file_to_set loads one URL per line from a state file into a set.
• set_to_file writes a set back to a state file. These helpers allow progress to survive program restarts.

Define shared Spider state

A class attribute is an attribute stored on the class itself and shared by its instances unless an instance overrides it. For a simple file-based crawler, shared attributes are useful when every Spider instance belongs to the same project and must see the same queue and crawled data.

The following definition initializes textual configuration values as empty strings and URL collections as empty sets:

class Spider:
    project_name = ''
    base_url = ''
    domain_name = ''
    queue_file = ''
    crawled_file = ''

    queue = set()
    crawled = set()

The empty strings are placeholders. Project setup logic will later supply values such as the project identifier, starting URL, allowed domain, and state-file paths.

The sets are intentionally class attributes in this design. However, mutable class attributes are shared by all instances. That is appropriate only when all Spider objects should operate on one common queue and one common visited collection. If separate crawler instances need independent state, initialize self.queue and self.crawled in an instance initializer instead.

Spider Class Attributes

Understand the URL collections

A queue is the collection of URLs still waiting to be processed. A crawled set is the collection of URLs that the crawler has already visited. A Python set stores unique values and provides fast membership checks, making it well suited to both collections.

For example, initial setup for a site might conceptually produce:

Spider.project_name = 'example_site'
Spider.base_url = 'https://example.com/'
Spider.domain_name = 'example.com'
Spider.queue_file = 'projects/example_site/queue.txt'
Spider.crawled_file = 'projects/example_site/crawled.txt'

Spider.queue = {'https://example.com/'}
Spider.crawled = set()

The pending and visited sets should normally obey this invariant: after successful processing, a URL is removed from the queue and added to the crawled set. It should not remain in both collections. Before adding a discovered URL, check both sets so a page is not scheduled when it is already pending or completed.

The in-memory sets make current operations fast. The queue and crawled files provide persistent state: progress saved outside the running process so a later run can restore or inspect it.

How a page moves through the crawler

Processing lifecycle

1. Select one URL from Spider.queue.
2. Call urlopen for that URL and read the response body.
3. Decode the response into HTML text when necessary.
4. Create a LinkFinder for the page URL and HTML.
5. Inspect the discovered URLs.
6. Use the domain helper to accept only URLs inside the configured scope.
7. Reject URLs already in either queue or crawled.
8. Add new valid URLs to queue.
9. Add the processed page to crawled.
10. Write both sets to queue_file and crawled_file.

A later implementation might follow this shape. The exact LinkFinder constructor and discovery method depend on the link-finder module's API:

def crawl_page(page_url):
    response = urlopen(page_url)
    html = response.read().decode('utf-8', errors='ignore')

    finder = LinkFinder(page_url, html)
    for discovered_url in finder.page_links():
        if get_domain_name(discovered_url) != Spider.domain_name:
            continue
        if discovered_url in Spider.queue or discovered_url in Spider.crawled:
            continue
        Spider.queue.add(discovered_url)

    Spider.crawled.add(page_url)
    set_to_file(Spider.queue, Spider.queue_file)
    set_to_file(Spider.crawled, Spider.crawled_file)

This example illustrates the coordination responsibilities; it is not a complete crawl loop. Production code should also handle response closing, redirects, invalid encodings, HTTP errors, timeouts, and malformed URLs.

URL State Transitions

Keep the crawl inside one domain

A base URL is the starting address, such as https://example.com/docs/. A domain name is the site identity used for scope decisions, such as example.com. A discovered URL is a complete web address, such as https://example.com/docs/setup or https://other.example.net/.

A page often contains external navigation, advertisements, social links, or references to other sites. Following all of them would make a site-focused crawl leave its intended scope. For each discovered URL, compare its domain with Spider.domain_name before adding it to the queue.

discovered_domain = get_domain_name(discovered_url)
if discovered_domain == Spider.domain_name:
    if (discovered_url not in Spider.queue and
            discovered_url not in Spider.crawled):
        Spider.queue.add(discovered_url)

Domain rules need an explicit policy. Decide whether subdomains such as blog.example.com belong to a crawl configured for example.com. Also decide how to treat http versus https, trailing slashes, default ports, redirects, malformed links, and URL fragments. A simple string comparison may not be sufficient when these forms need normalization.

Practical scenarios

Initialize one site

For an example_site project, configure the homepage as the base URL, derive or set example.com as the allowed domain, assign queue and crawled file paths, place the homepage in the queue, and leave the crawled set empty.

Ignore external navigation

If a page contains one link to https://example.com/about and another to https://external.test/, the domain helper accepts the first and rejects the second when the configured domain is example.com.

Avoid duplicate work

If several pages reference the same internal URL, the queue set can contain that URL only once. Checking the crawled set as well prevents a completed page from being scheduled again.

Troubleshooting

An import cannot be resolved

Check that spider.py, link_finder.py, domain.py, and general.py are in the intended package or working directory. Verify filenames, class names, and exported function names. Run the project through its expected entry point so local module resolution uses the correct path.

The same page is visited repeatedly

Use sets for both URL states, add a successfully processed URL to crawled, remove it from queue, and reject discovered URLs already present in either set. Save the state after each transition or controlled batch of transitions.

Unrelated sites enter the queue

Validate every discovered URL before queueing it. Confirm that domain_name is correct and test the intended policy for subdomains, protocol variations, redirects, and normalized URLs.

Progress disappears when the program stops

In-memory sets vanish when the process ends. Write the current sets to queue_file and crawled_file, then reload them during project initialization with the file helpers.

A page request fails

Invalid URLs, unreachable servers, HTTP errors, blocks, and timeouts can all cause retrieval to fail. Surround urlopen with network and HTTP exception handling, log the failed URL, and choose whether to retry, skip, or record it as attempted.

Exam-relevant notes

• Spider: the class that maintains crawl state and coordinates retrieval and link discovery.
• Class versus instance attributes: class attributes are shared; instance attributes belong to one object.
• Mutable class attributes: sets declared on the class are shared intentionally in this single-project design.
• Sets: they remove duplicate URLs and make membership tests efficient.
• Scope filtering: compare every discovered URL with the configured allowed domain before queueing it.
• Persistence: queue and crawled files let the crawler restore progress after a restart.
• State invariant: a successfully processed URL should move from pending to crawled rather than remaining in both collections.

Summary

The dedicated spider.py module is the coordinator for a file-based crawler. Its Spider class stores project metadata, the base URL, the allowed domain, state-file paths, and shared queue and crawled sets. Later methods will use urlopen to fetch pages, LinkFinder to discover links, domain helpers to enforce scope, and general utilities to persist state. Establishing this state model first makes the complete crawl loop easier to implement and troubleshoot.