VMware ESXi and vSphere Cluster Management

Bootstrapping a Python Web Crawler

Learn how to initialize a file-based Python web crawler by creating project storage, seeding URL state, and loading queue and crawled sets.

Before a crawler downloads a page, it needs a known starting state. Bootstrapping, or boot, is the startup operation that establishes the folders, files, and in-memory collections required for crawling.

A boot routine does not download HTML, parse links, or discover new URLs. It prepares the crawler so a later crawling operation can safely process a URL from the initialized queue.

What crawler bootstrapping does

A file-based crawler normally stores its progress in two places:

• Persistent storage: files that survive when the program stops.
• In-memory state: Python collections used while the program is running.

The startup routine should perform these operations in order:

1. Create a project directory for this crawl.
2. Create or preserve the queue and crawled state files.
3. Seed the queue with the base URL when the project is new.
4. Load both files into Python sets.
5. Leave the crawler ready for the next stage: fetching a page from the queue.

Core crawler terms

Class-level crawler configuration

The Spider class can keep configuration and state as class attributes. This is suitable for a simple crawler that operates on one project at a time.

The important configuration values are:

• project_name: identifies the crawl directory.
• base_url: supplies the first URL for a new crawl.
• queue_file: points to the persistent pending-URL file.
• crawled_file: points to the persistent completed-URL file.

The project name determines the directory, and the directory determines where the state files live. For example, a project named example-site can use example-site/queue.txt and example-site/crawled.txt.

Helper functions for project storage

The boot method is easier to understand when directory and file operations are placed in helper functions. The required helpers are:

• create_project_dir(project_name): creates the project folder.
• create_data_files(project_name, base_url): creates state files and seeds a new queue.
• file_to_set(file_path): reads one URL per line into a Python set.

Creating the project directory

Use exist_ok=True so starting the crawler again does not fail when the directory already exists.

import os


def create_project_dir(project_name):
    os.makedirs(project_name, exist_ok=True)

Creating or preserving state files

The queue file and crawled file should be created only when they do not exist. The queue is seeded with the base URL only during first-time setup. Opening an existing file with write mode would erase saved progress, so this code uses existence checks.

def create_data_files(project_name, base_url):
    queue_file = os.path.join(project_name, "queue.txt")
    crawled_file = os.path.join(project_name, "crawled.txt")

    if not os.path.exists(queue_file):
        with open(queue_file, "w", encoding="utf-8") as file:
            file.write(base_url + "\n")

    if not os.path.exists(crawled_file):
        open(crawled_file, "w", encoding="utf-8").close()

This function is idempotent: running it repeatedly produces the same valid storage state without discarding existing URLs.

Loading a file into a set

The file_to_set helper reads line-based URL storage. strip() removes the line ending and surrounding whitespace, while the set automatically removes duplicate entries.

def file_to_set(file_path):
    with open(file_path, "r", encoding="utf-8") as file:
        return {
            line.strip()
            for line in file
            if line.strip()
        }

Sets are appropriate for crawler state because membership testing is efficient and duplicate URLs are represented only once. For example, url in Spider.crawled can determine whether a URL has already been processed.

The Spider boot method

A static method is a method attached to a class that can be called without constructing an object instance. Marking the startup routine with @staticmethod communicates that booting uses class-level configuration rather than instance-specific data.

import os


class Spider:
    project_name = "example-site"
    base_url = "https://example.com/"

    queue_file = os.path.join(project_name, "queue.txt")
    crawled_file = os.path.join(project_name, "crawled.txt")

    queue = set()
    crawled = set()

    @staticmethod
    def boot():
        create_project_dir(Spider.project_name)
        create_data_files(Spider.project_name, Spider.base_url)

        Spider.queue = file_to_set(Spider.queue_file)
        Spider.crawled = file_to_set(Spider.crawled_file)

The method follows the required sequence:

1. create_project_dir makes sure the project folder exists.
2. create_data_files creates missing files and writes the base URL for a new project.
3. file_to_set loads pending URLs into Spider.queue.
4. file_to_set loads completed URLs into Spider.crawled.

Calling boot during startup

Call the static method before calling any page-crawling method.

if __name__ == "__main__":
    Spider.boot()

    print("Pending URLs:", Spider.queue)
    print("Completed URLs:", Spider.crawled)

    # The next stage can now crawl a URL from Spider.queue.

Because boot is static, both Spider.boot() and an instance-free startup design are possible. No Spider() object is required for this class-level version.

Crawler startup sequence

How the data flows

For a new project, the base URL progresses through three representations:

1. Configuration: Spider.base_url contains https://example.com/.
2. Persistent state: create_data_files writes that URL to example-site/queue.txt.
3. Runtime state: file_to_set reads the file and assigns the URL to Spider.queue.

The crawled file follows the same storage pattern, except it starts empty and is later updated as pages are processed.

First startup example

With this configuration:

Spider.project_name = "example-site"
Spider.base_url = "https://example.com/"

the first call to Spider.boot() produces the following logical result:

• An example-site directory is created.
• example-site/queue.txt contains https://example.com/.
• example-site/crawled.txt exists and is empty.
• Spider.queue contains the starting URL.
• Spider.crawled is an empty set.

Restarting an existing crawler

Suppose the crawler has already discovered several URLs and recorded completed pages. On the next startup, create_project_dir sees the existing folder, and create_data_files leaves both files unchanged.

The boot method then restores the saved state:

• Previously pending URLs return to Spider.queue.
• Previously visited URLs return to Spider.crawled.
• The base URL is not added again merely because the crawler restarted.
• Saved progress is not discarded.

Duplicate URL protection

If a queue file contains the same URL more than once, loading it into a set creates one in-memory entry:

# queue.txt
https://example.com/about
https://example.com/about
https://example.com/contact

queue = file_to_set("queue.txt")
print(queue)
# {'https://example.com/about', 'https://example.com/contact'}

Sets do not by themselves implement the complete crawl algorithm, but they provide the foundation for checks such as:

if url not in Spider.crawled:
    Spider.queue.add(url)

A crawler should also keep pending and completed state distinct. A URL in the queue is waiting; a URL in the crawled set has already been processed.

Booting versus crawling

Boot code and crawl code have separate responsibilities:

After boot completes, the next stage is to select a URL from the initialized queue, fetch its page, and update persistent state as crawling proceeds.

Troubleshooting startup problems

The queue is empty for a new project

• Check that base_url is passed to create_data_files.
• Confirm that a newly created queue file is seeded with the base URL.
• Make sure the queue file is initialized before file_to_set reads it.
• Reload Spider.queue after file initialization.

A restart loses crawl progress

• Do not open existing state files in write mode during every startup.
• Keep the project name and file names stable between runs.
• Load both the queue file and crawled file during boot.

The same URL is crawled repeatedly

• Use sets for the in-memory queue and crawled collections.
• Restore completed URLs from the crawled file.
• Check membership before adding or processing a URL.
• Keep queue and crawled files separate.

The boot method cannot find a helper

• Import create_project_dir, create_data_files, and file_to_set into the module containing Spider.
• Check that function names match their definitions exactly.
• Confirm that the helpers are defined in an accessible module or scope.
• Verify that the arguments passed by boot match the helper signatures.

Exam-relevant checklist

• Booting prepares crawler state before any page is fetched.
• The project name identifies the project directory.
• The base URL seeds a new queue file.
• Existing queue and crawled files must be preserved.
• file_to_set converts line-based files into sets.
• Spider.queue stores pending URLs in memory.
• Spider.crawled stores completed URLs in memory.
• A static boot method can initialize class-level crawler configuration without an instance.
• The required order is directory, files, then in-memory sets.
• Booting prepares the next operation; it does not download or parse pages.

Once this startup process is reliable, the crawler has a durable project location, initialized state files, and restored URL collections. Page-fetching and link-discovery logic can now operate on that prepared state.

Continue with crawler bootstrapping concepts when reviewing the complete initialization flow.