VMware ESXi and vSphere Cluster Management

Initialize a Python Web Crawler with Project and Site Information

Learn to initialize a Python Spider with a project name, base URL, domain, queue file, crawled file, boot sequence, and first page crawl.

A crawler needs configuration before it can visit pages. It must know which local project stores crawl state, which URL starts the crawl, and which domain limits link discovery. This lesson adds that setup to an existing Spider class.

The examples assume that the class already provides boot() and crawl_page() methods. The boot() method prepares storage, while crawl_page() processes a URL and records crawl activity.

Why crawler initialization matters

Initialization is the setup performed when a Spider object is created. A web crawler needs this setup before crawling because later methods must use consistent project paths, URL settings, and domain rules.

  • It identifies the local crawl project.
  • It records the first page to visit.
  • It defines the site boundary for eligible links.
  • It derives the files used to store pending and completed URLs.
  • It prepares storage before page processing starts.

Centralizing these operations in __init__ gives every newly created Spider the same startup sequence. Instead of relying on the caller to remember several assignments and method calls, the initializer performs them in the required order.

Some values come from the user or application entry point. In this lesson, project_name, base_url, and domain_name are supplied values. Other values, such as queue_file and crawled_file, are derived from the project name by the crawler.

Core crawler information

The three required inputs describe one crawl project:

  • project_name: A local identifier, normally used as the project directory name. It separates one crawl's files from another crawl's files.
  • base_url: The starting URL. The initial crawl begins by passing this value to crawl_page().
  • domain_name: The target site's domain. Link-filtering logic can use it to reject links that lead outside the intended site.

For example, a documentation crawl can use the project name docs-crawl, the starting page https://example.org/docs/, and the domain example.org. These values describe the same crawl rather than three unrelated settings.

Value: project_name | Source: user or entry point | Purpose: names the local crawl project directory | Example: docs-crawl

Value: base_url | Source: user or entry point | Purpose: identifies the first page to process | Example: https://example.org/docs/

Value: domain_name | Source: user or entry point | Purpose: defines the intended site boundary | Example: example.org

Value: queue_file | Source: derived from project_name | Purpose: stores URLs waiting to be processed | Example: docs-crawl/queue.txt

Value: crawled_file | Source: derived from project_name | Purpose: stores URLs already visited | Example: docs-crawl/crawled.txt

Adding an initializer to the Spider class

Python calls the __init__ method automatically when an object is created. The first parameter is conventionally named self; the remaining parameters receive the values supplied by the caller.

import os


class Spider:
    def __init__(self, project_name, base_url, domain_name):
        Spider.project_name = project_name
        Spider.base_url = base_url
        Spider.domain_name = domain_name

        Spider.queue_file = os.path.join(
            project_name, "queue.txt"
        )
        Spider.crawled_file = os.path.join(
            project_name, "crawled.txt"
        )

        Spider.boot()
        Spider.crawl_page("initial spider", base_url)

This initializer performs five related tasks:

  1. It stores the supplied project, URL, and domain values.
  2. It creates a project-specific path for queue.txt.
  3. It creates a project-specific path for crawled.txt.
  4. It calls boot() to create or prepare the project directory and files.
  5. It calls crawl_page() with a readable worker label and the configured base URL.

os.path.join() is preferable to manually joining strings because it constructs paths using the platform's path conventions. The important result is that both state files are beneath the selected project directory.

Class attributes and instance attributes

A class attribute is stored on the class itself. In the example, assignments such as Spider.project_name = project_name put configuration on the Spider class. Spider behavior can then read values such as Spider.queue_file and Spider.domain_name.

An instance attribute belongs to one particular object and is normally assigned with self:

self.project_name = project_name
self.base_url = base_url
self.domain_name = domain_name

These two styles are not interchangeable. Class-level assignments support a design in which one crawler configuration is shared by the Spider's behavior. Instance-level assignments allow different Spider objects to maintain independent settings.

Queue and crawled files

The crawl frontier is the collection of discovered URLs that remain to be visited. In this file-based design, the frontier is represented by queue.txt. Each queued URL can be read and eventually passed to page-processing logic.

crawled.txt records URLs that have already been processed. Comparing candidate links with this file helps prevent repeated work. Both files belong inside the project directory so that separate projects do not accidentally share state.

docs-crawl/
    queue.txt      # URLs waiting to be processed
    crawled.txt    # URLs already visited

For the documentation example, the initializer derives:

  • docs-crawl/queue.txt for pending URLs.
  • docs-crawl/crawled.txt for visited URLs.

The crawler startup sequence

Startup order matters. The first page must not be processed until its project directory and state files are ready.

Step: Instantiate Spider | Operation: supply project name, base URL, and domain | Dependency: the Spider class exists | Result: initialization begins

Step: Save configuration | Operation: assign the three user-controlled values | Dependency: constructor arguments | Result: crawl behavior can access the settings

Step: Build state-file paths | Operation: derive queue.txt and crawled.txt beneath the project directory | Dependency: project_name | Result: project-specific storage locations exist conceptually

Step: Boot project storage | Operation: call boot() | Dependency: file paths and a working setup method | Result: directory and required files are prepared

Step: Start first page crawl | Operation: call crawl_page() with a label and base_url | Dependency: bootstrapped storage and a working crawl method | Result: initial page processing and status output begin

The sequence can be summarized as:

user values
    -> store configuration
    -> derive queue and crawled paths
    -> boot()
    -> crawl_page("initial spider", base_url)

Starting a documentation-site crawl

The application entry point supplies the three user-controlled values when it creates the Spider:

Spider(
    "docs-crawl",
    "https://example.org/docs/",
    "example.org"
)

Object creation immediately runs the initializer. The expected effects are:

  • The project identifier becomes docs-crawl.
  • The first target becomes https://example.org/docs/.
  • Downstream link checks can compare links with example.org.
  • The state paths become docs-crawl/queue.txt and docs-crawl/crawled.txt.
  • boot() prepares the directory and files before page processing.
  • crawl_page("initial spider", "https://example.org/docs/") begins the first crawl.

The label passed to crawl_page() is not the URL. It is a readable spider or worker name useful for status messages, logs, or identifying the action that started the crawl.

Integration requirements

The initializer is only one part of the crawler. The surrounding implementation must provide the methods and behavior it relies on:

  • Spider class: must exist before the application creates an object.
  • boot(): must create or prepare the project directory and required data files.
  • crawl_page(): must accept the worker label and URL in the order used by the initializer, then process the supplied page.
  • Link filtering: downstream logic should use domain_name when deciding whether discovered links are eligible.
  • Entry point: must pass a valid project directory name, starting URL, and allowed domain.

Initialization does not itself download HTML, extract links, enforce rate limits, or implement robots.txt handling. Those responsibilities belong to the other crawler methods. The initializer supplies their shared configuration and ensures that persistent state is ready.

Troubleshooting initialization

State files cannot be created or found

  • Check that project_name is not empty and is valid as a directory name.
  • Confirm that boot() runs before crawl_page().
  • Print or inspect Spider.queue_file and Spider.crawled_file to verify that both include the intended project directory.
  • Check write permission in the current working directory.

The initial page is not crawled

  • Verify that the initializer calls crawl_page().
  • Confirm that the configured base_url, rather than an unrelated URL, is passed as the first target.
  • Check that the URL is well formed.
  • Ensure that boot() created any files required by crawl_page().

A later Spider uses another crawler's settings

This usually means that configuration is stored as class-level state. Creating a second object changes values such as Spider.base_url for code that uses the class. Use self.project_name, self.base_url, and related instance attributes when multiple independent Spider objects must coexist.

The crawler leaves the intended website

  • Verify that domain_name uses the format expected by the link-filtering code.
  • Ensure that the domain check actually uses the configured value.
  • Inspect redirects from the base URL, because a redirect may lead to another domain.
  • Normalize domains consistently before comparing them, especially when handling ports, schemes, or a leading www.

Exam-relevant notes

  • __init__ runs automatically when a Spider object is created.
  • project_name, base_url, and domain_name are input parameters.
  • queue_file and crawled_file are derived values.
  • queue.txt stores pending URLs; crawled.txt stores visited URLs.
  • boot() must run before crawl_page().
  • Using Spider.attribute creates shared class-level configuration, while self.attribute creates per-object configuration.
  • The initial call to crawl_page() must receive the configured base URL.

With this structure, the crawler has one clear setup point: user-supplied site information is stored, project-specific crawl files are derived, storage is bootstrapped, and the first page is handed to the page-processing method. See also the crawler information lesson for the complete initialization context.