VMware ESXi and vSphere Cluster Management

Python Web Crawler: Workflow and Project Overview

Learn the architecture and workflow of a simple multithreaded Python web crawler, including queues, crawled state, link extraction, scope filtering, and project organization.

A web crawler is a program that automatically visits web pages and discovers links. A simple crawler begins with one seed URL, follows eligible hyperlinks found in the page HTML, and continues until there are no more URLs waiting to be visited.

This lesson focuses on the planned architecture and processing workflow for a small Python crawler. It does not yet implement every HTTP, parsing, error-handling, or synchronization detail. Those components can be added in later stages.

For a related overview of the same project, see the crawler description.

Crawler Purpose and Scope

The crawler's job is to visit pages within a target website and collect discovered hyperlinks. It starts from an initial URL rather than knowing every page in advance. Each visited page can reveal additional pages, which are evaluated and possibly added to the crawler's waiting list.

A planned crawl normally has a website scope. Scope is the rule that decides which discovered URLs belong to the intended site. For example, a project might accept links on one target domain while rejecting external domains, file types, or paths outside the selected section.

The basic output is crawl state:

• A collection of URLs waiting to be visited.
• A collection of URLs that have already been processed.

This state lets you inspect what remains, see what the crawler has completed, and resume work after a restart.

Overall Crawler Workflow

1. Choose a project directory for the target website.
2. Create or verify the directory and initialize the crawl state files.
3. Load the files into an in-memory queue set and crawled set.
4. Add the seed URL to the waiting collection.
5. Select a pending URL for processing.
6. Retrieve the page and read its HTML.
7. Extract hyperlink targets from the HTML.
8. Normalize or convert discovered links into crawl-ready URLs.
9. Reject duplicates and URLs outside the website scope.
10. Add newly eligible URLs to the pending collection.
11. Mark the processed URL as crawled and persist the updated state.
12. Repeat until no pending URLs remain.

Every URL follows a simple state transition: it begins as a seed or discovered link, waits in the queue, is selected by a worker, and then moves to the crawled collection. A newly discovered URL enters the queue only when it is eligible and is not already known.

Per-Website Project Organization

Use a separate project directory for each target website. Before creating files, check whether the directory already exists. If it does not exist, create it; if it does exist, reuse it only when it represents the intended crawl.

project_directory = "site_project"

# Conceptual setup
# 1. Check whether project_directory exists.
# 2. Create it when necessary.
# 3. Store queue.txt and crawled.txt inside it.

Separating data by site prevents unrelated crawl runs from mixing. Without this boundary, URLs from one website could appear in another site's queue or crawled history, making counts, resuming, and debugging unreliable.

Persistent Crawl State

The crawler uses two text files inside the site-specific project directory:

• queue.txt stores URLs that still need to be crawled.
• crawled.txt stores URLs that have already been visited.

For a new project, initialize both files. The queue file receives the seed URL, while the crawled file starts empty. When the program starts, it reads both files and reconstructs its in-memory state.

Persisted files make progress inspectable: you can open them to see pending and completed work. They also make progress recoverable because a later run can reload the files instead of starting discovery from nothing.

Crawler State Files and Sets

The files and sets represent the same logical state in different forms. The files provide persistence; the sets provide fast checks while the program is running. Updates must keep both representations synchronized.

In-Memory URL Sets

A Python set is a collection designed for fast membership checks. It also naturally removes duplicate values. Maintain at least two sets:

• Queue set: URLs waiting to be processed.
• Crawled set: URLs already processed.

Before adding a discovered URL, check whether it is already in either set. This prevents the same URL from being queued repeatedly or crawled again after completion.

if eligible_url not in queue_set and eligible_url not in crawled_set:
    queue_set.add(eligible_url)
    # Persist the addition to queue.txt

With multiple workers, a membership check and the following insertion must be coordinated as one shared-state operation. Otherwise, two workers can check at the same time, both see that a URL is absent, and both queue it.

Seed URL Handling

The seed URL is the starting URL supplied by the user. It is the entry point from which the crawler discovers other pages.

1. Configure the seed URL for the target website.
2. Write it to queue.txt for a new crawl.
3. Load it into the in-memory queue set.
4. Leave the crawled set empty until the page is processed.

seed_url = "SEED_URL"
queue_set = {seed_url}
crawled_set = set()

If the seed is not placed in both the persistent queue state and the in-memory queue state, workers have no task and the crawler can stop immediately.

Page Retrieval and Link Extraction

A worker selects one URL from the pending collection and retrieves its page. The response body contains HTML, the markup used to structure the page. The crawler examines that HTML to locate hyperlinks, commonly represented by link elements with target attributes.

Link extraction is the process of finding those hyperlink targets and collecting their URLs. Extracted values may need conversion before they are suitable for crawling. For example, a relative link must be resolved against the page's base URL, and equivalent URL spellings may need normalization.

Every discovered URL should pass validation before entering the queue:

• Convert it to a consistent URL form when appropriate.
• Reject values that are malformed or unsupported.
• Apply the website-scope rule.
• Check both the queue set and crawled set for duplicates.
• Add only eligible, unknown URLs to the pending state.

Scope filtering is important because pages often link to unrelated websites, downloads, external services, or content that the project does not intend to process.

Multithreaded Crawling

Multithreading means using multiple concurrent worker threads. Crawling often waits for network responses, so several workers can make progress while different requests are waiting.

Each worker follows the same general responsibility:

1. Take one URL from the pending queue.
2. Retrieve its page HTML.
3. Extract and evaluate links.
4. Add new eligible links to shared pending state.
5. Move the processed URL to shared crawled state.
6. Persist the corresponding file changes.

The queue set, crawled set, and persistent files are shared resources. Workers must coordinate selecting URLs and updating state. A thread-safe queue can coordinate task selection, while a lock or another synchronization strategy can protect the membership check and state transition.

The key invariant is that a URL should not be claimed by two workers. A safe design reserves or removes a URL from pending state before processing it, then records it as crawled after processing. The exact failure policy should be defined separately: a failed request may be recorded, retried, or returned to a retry queue.

URL Lifecycle

Crawl Loop and Termination

The crawler repeatedly selects pending URLs. After processing one URL, it records that URL as crawled, adds any newly eligible links to the queue, and continues with the remaining pending work.

Completion occurs when the queue file and the in-memory queue contain no URLs left to visit. In a threaded crawler, an empty queue alone is not always enough to stop immediately: a worker may currently be processing a page and may discover more links. The coordinator should wait until the queue is empty and no workers still have active tasks.

while pending_work_exists:
    url = claim_pending_url()
    process_page(url)
    add_new_eligible_links()
    mark_as_crawled(url)

# Stop after the queue is empty and active workers are finished.

Project Requirements and Next Steps

Before implementation, prepare the following:

• A working Python environment.
• A new crawler project.
• A chosen project directory for the target website.
• A configured seed URL.
• Paths for queue.txt and crawled.txt.
• A planned number of worker threads.
• A synchronization strategy for shared queue and crawled state.

Later implementation stages can build the project directory setup, state-file management, HTML link parsing, URL normalization, website-scope filtering, HTTP retrieval, and threaded worker components.

Configuration Checklist

Troubleshooting Common Problems

The Same URL Is Crawled More Than Once

This usually means a URL was added without checking both sets, or workers changed shared state without coordination. Check membership in the queue and crawled sets before queueing, and make the check-and-add operation synchronized.

The Crawler Stops Immediately

The seed URL may not have been written to queue.txt or loaded into the in-memory queue. Verify both initialization steps and confirm that the queue is not being overwritten as an empty file during startup.

Unrelated URLs Are Added

Discovered links may be accepted without a website-scope check. Filter links according to the target domain, host, or path policy before adding them to the queue.

Progress Is Lost or Inconsistent After Restarting

The persistent files and in-memory sets may have diverged. Persist queue and crawled changes consistently, use a clear update order, and reload both files when the program starts.

Threads Produce Incorrect Counts or Duplicate Work

Multiple workers may be reading or modifying the URL collections at the same time. Use a thread-safe task queue and synchronization around shared membership checks, claims, and state updates.

Exam-Relevant Summary

• A web crawler starts with a seed URL and progressively discovers links.
• The queue contains URLs waiting to be processed; the crawled set contains URLs already processed.
• queue.txt and crawled.txt persist those two categories between runs.
• Python sets provide fast membership checks and help prevent duplicate URLs.
• Extracted links must be normalized, checked for scope, and deduplicated before queueing.
• Multiple workers improve concurrency but require synchronized access to shared state.
• The crawl ends when no pending URLs remain and all active workers have finished.