Create the Main Spider Controller for a Python Web Crawler

Learn how main.py configures a Python web crawler, derives its target domain, creates persistent crawl files, initializes a thread-safe Queue, and bootstraps the first Spider.

A web crawler is a program that starts with one or more URLs, retrieves pages, discovers links, and follows eligible links according to defined rules. In a modular crawler, main.py is the application entry point: the file you run to assemble the crawler and start its later stages.

This lesson focuses on setup and coordination. The main script does not need to contain all HTTP fetching, HTML parsing, or link-discovery logic. Those responsibilities belong in modules such as spider.py, while main.py supplies configuration and connects the parts.

What the Main Crawler Script Does

The controller prepares the information and objects that the rest of the crawler needs:

• It accepts or defines a project name and a homepage.
• It derives the target domain from the homepage.
• It builds paths for persistent crawl-state files.
• It creates a thread-safe in-memory work queue.
• It initializes the first Spider object.
• It later starts worker threads that consume URL jobs.

A spider is a crawler component or worker that processes pages and discovers additional URLs. Keeping orchestration in main.py makes the project easier to test and extend: fetching, parsing, URL filtering, file management, and threading can evolve independently.

Project Layout

Place the entry script beside the modules it imports:

crawler_project/
    main.py
    spider.py
    domain.py
    general.py

Required Crawler Settings

Two inputs are essential at startup:

• Project name: a unique label such as example_site. It identifies the output directory for one crawl and prevents state from different projects being mixed.
• Homepage: the initial seed URL, such as https://example.com/. A seed URL is the first URL placed into the crawl.

Before starting, validate that the homepage is a usable absolute URL. An absolute URL includes a scheme such as https:// and a host. A blank value, a relative path such as /about, or a string with no host should be rejected before domain parsing begins.

PROJECT_NAME = "example_site"
HOMEPAGE = "https://example.com/"
NUMBER_OF_THREADS = 2

Python uses uppercase names by convention for values intended to remain fixed during a run. These are often called constants, but Python does not enforce immutability. Code can still reassign PROJECT_NAME; uppercase communicates the programmer's intent.

Imports and Module Responsibilities

The controller needs concurrency support, queue management, and project-specific helpers. A maintainable version uses explicit imports:

import threading
from queue import Queue

from spider import Spider
from domain import get_domain_name
import general

• threading provides the tools for creating concurrent worker threads.
• Queue is Python's thread-safe queue implementation for sharing URL jobs.
• Spider is the crawler class from spider.py.
• get_domain_name represents a helper that extracts or normalizes the allowed domain.
• general represents shared directory and file-management utilities from general.py.

A wildcard import such as from general import * can make a short example look compact, but it hides where names came from and can cause name collisions. Prefer explicit imports such as from general import create_project_files, or import the module and use qualified names such as general.create_project_files().

Configuration and Initialization Order

Configuration has dependencies. Set the homepage before deriving its domain, and derive the domain before constructing project-relative paths that may depend on it or on the project name.

1. Set PROJECT_NAME and validate HOMEPAGE.
2. Derive DOMAIN_NAME from HOMEPAGE.
3. Build QUEUE_FILE and CRAWLED_FILE.
4. Set the worker count.
5. Create the runtime Queue.
6. Initialize the first Spider.
7. Populate or load pending work, then start workers.

Deriving the Domain and Defining Crawl Scope

A URL has several parts. The scheme is usually http or https. The host identifies the network location and can include a subdomain. The domain name is the host value used by the crawler's scope rule. A path identifies a resource below that host.

• Full URL: https://www.example.com/docs/start.html
• Hostname: www.example.com
• Domain or normalized target: commonly example.com, depending on the helper's policy
• Path: /docs/start.html

The crawler needs a target domain so it can reject links that lead to unrelated sites. A domain helper should consistently handle common homepage variants:

• http://example.com and https://example.com/ differ in scheme but may represent the same crawl target if the policy treats them together.
• www.example.com and example.com may be treated as equivalent or distinct. Choose and document one rule.
• A trailing slash usually does not change the resource identity, but URL normalization should be consistent.
• A homepage path such as https://example.com/store/ still has example.com as its host; the path may also be used to limit the crawl further.
• Ports, redirects, and malformed host values require explicit handling by the domain utility.

Subdomains require a deliberate decision. With exact-host matching, a crawl of example.com accepts that host but rejects blog.example.com. With a subdomain-inclusive policy, trusted subdomains can be accepted. Do not use a careless suffix test: a host such as example.com.attacker.test must not be accepted merely because its text contains example.com.

Persistent Queue and Crawled Files

A project should have separate state files for each crawl:

example_site/
    queue.txt
    crawled.txt

queue.txt is persistent storage for URLs still waiting to be processed. crawled.txt is persistent storage for URLs already visited. These files are different from the runtime Queue object:

• The in-memory Queue coordinates worker threads during the current process.
• queue.txt survives process termination and can be inspected or loaded during a later run.
• crawled.txt helps avoid revisiting URLs and can preserve progress between runs.

For a new project, queue.txt can initially contain the seed homepage while crawled.txt is empty. After the homepage is processed, it moves conceptually from pending to crawled, and newly discovered eligible links are added to the pending state. The exact file-update operations belong in the shared utilities or Spider implementation, but main.py must provide the paths.

Creating the Runtime Queue

Create one shared queue for URL jobs:

URL_QUEUE = Queue()

Queue implements safe producer-consumer coordination. A producer can add a URL with put(), and a worker can retrieve one with get(). The queue coordinates access between threads so workers do not need to manipulate a plain list at the same time.

The expected worker lifecycle is:

1. Wait for a URL in the shared queue.
2. Retrieve the URL.
3. Ask a Spider to crawl it and discover eligible links.
4. Record the URL as crawled and enqueue new work.
5. Call task_done() when that queue task is complete.
6. Continue until the crawler has no work left.

The queue object and queue.txt must be synchronized by the crawler design. Creating one does not automatically load the file into memory, and writing the file does not automatically notify worker threads.

Choosing the Worker Count

A worker thread is a concurrent execution unit that repeatedly processes URL jobs. A small crawler might begin with:

NUMBER_OF_THREADS = 2

Each worker can take one pending URL at a time from the shared queue while using the same crawl configuration. Start conservatively and consider:

• Available CPU and memory on the machine.
• Network bandwidth and connection limits.
• The target server's capacity and rate limits.
• Timeouts, retries, and error handling.
• Responsible crawling practices, including robots.txt and polite request pacing.

More threads are not always faster. Excessive concurrency can overload the target, trigger blocking, increase connection errors, and make shared-state bugs harder to diagnose.

Bootstrapping the First Spider

The first Spider instance receives the project identity and crawl scope:

PROJECT_NAME = "example_site"
HOMEPAGE = "https://example.com/"
DOMAIN_NAME = get_domain_name(HOMEPAGE)
QUEUE_FILE = PROJECT_NAME + "/queue.txt"
CRAWLED_FILE = PROJECT_NAME + "/crawled.txt"
NUMBER_OF_THREADS = 2

URL_QUEUE = Queue()

first_spider = Spider(PROJECT_NAME, HOMEPAGE, DOMAIN_NAME)

This constructor call assumes that the Spider class accepts the project name, homepage, and domain name in that order. Adapt the names to the actual class definition, but preserve the setup dependency order.

Initial construction should bootstrap the project: create the project directory, create the queue and crawled files if needed, and arrange for the seed homepage to become initial crawl work. Seed initialization must be idempotent, meaning that running setup again should not add duplicate copies of the homepage or erase existing progress.

Later worker threads should reuse the same crawl configuration and coordinate through the shared queue. They should not independently create conflicting project files or reset the crawled set.

A More Explicit Startup Skeleton

The following skeleton shows the controller's responsibilities without placing page-fetching logic in main.py:

import threading
from queue import Queue
from urllib.parse import urlparse

from spider import Spider
from domain import get_domain_name

PROJECT_NAME = "example_site"
HOMEPAGE = "https://example.com/"
NUMBER_OF_THREADS = 2


def validate_homepage(url):
    parsed = urlparse(url)
    if parsed.scheme not in {"http", "https"} or not parsed.netloc:
        raise ValueError("HOMEPAGE must be an absolute HTTP or HTTPS URL")


validate_homepage(HOMEPAGE)
DOMAIN_NAME = get_domain_name(HOMEPAGE)
QUEUE_FILE = PROJECT_NAME + "/queue.txt"
CRAWLED_FILE = PROJECT_NAME + "/crawled.txt"
URL_QUEUE = Queue()

first_spider = Spider(PROJECT_NAME, HOMEPAGE, DOMAIN_NAME)

# Later stages load pending URLs into URL_QUEUE,
# create workers, and join them when crawling finishes.

The standard library's urllib.parse.urlparse is used here only for basic validation. Domain normalization remains the responsibility of domain.py. A production crawler should also define behavior for redirects, credentials, ports, fragments, internationalized domains, and URL normalization.

How the Modules Connect

The setup establishes a chain of responsibilities:

• main.py obtains the homepage and project settings.
• domain.py converts the homepage into a target-domain value used during link filtering.
• general.py provides shared directory and file operations for project state.
• spider.py uses the project, seed, domain, queue state, and crawled state to process pages and discover links.
• Worker-creation and worker-execution functions later consume the initialized URL_QUEUE.

When Spider discovers a link, it should validate and normalize the URL, apply the domain rule, check whether it has already been crawled or queued, and then add eligible work. This is how domain parsing, persistent state, and the runtime queue work together to continue a crawl without repeatedly visiting the same pages.

Running the Entry Script

After placing the files in the same directory, configuring the project name and homepage, and confirming that the imported names match the module definitions, run:

python main.py

Run the command from the directory containing main.py, spider.py, domain.py, and general.py. The initial setup should result in a project directory containing the persistent crawl files before workers begin processing URLs.

Troubleshooting

The Derived Domain Is Empty or Incorrect

• Check that the homepage is not blank.
• Include a scheme such as https://.
• Test the domain helper with trailing slashes, paths, ports, and subdomains.
• Confirm whether the helper returns an exact host or a normalized registrable domain.

The Project Directory or Files Are Missing

Verify that the Spider constructor is actually called, that its bootstrap behavior creates the directory and both state files, and that the process can write to the selected working directory.

Workers Have No URLs

Check that the seed homepage was added to initial crawl state, that pending URLs were loaded from queue.txt into the runtime Queue, and that the homepage was not incorrectly marked as crawled before any worker received it.

External Links Are Being Followed

Review the domain comparison rule and URL normalization. Test an internal page, an allowed or disallowed subdomain, an unrelated domain, and a malformed value before enqueueing discovered links.

More Threads Cause Failures

Reduce NUMBER_OF_THREADS, retain Queue-based coordination, and add timeouts, rate controls, and retry limits in later crawler stages. Check both local resource limits and the target site's policies.

Imports Fail

Confirm that module files are present and correctly named, run the command from the project directory, and use explicit imports whose symbols match the actual class and function definitions.

Exam- and Interview-Relevant Notes

• Queue versus queue.txt: the first is an in-memory, thread-safe runtime object; the second is persistent file storage.
• Domain versus full URL: a full URL includes scheme, host, and possibly path; a domain value is used to decide crawl scope.
• Initialization order: homepage first, domain second, file paths third, runtime queue and Spider afterward.
• Thread safety: multiple workers must coordinate through a thread-safe Queue and synchronized crawl-state updates.
• Constants: uppercase names communicate fixed configuration but are not enforced as immutable by Python.
• Module boundaries: main.py coordinates; spider.py crawls; domain.py handles domain rules; general.py handles shared project and file utilities.
• Seed handling: the initial homepage must enter pending work exactly once and must not reset an existing project.

For a related walkthrough of this controller and its setup concepts, see creating the main spider controller.