Python Web Crawler

Web Spider Concepts: Queue, Crawled URLs, Link Extraction, and Coordination

Learn how a Python web spider fetches pages, extracts links, prevents duplicates, maintains queue and crawled files, and coordinates multiple workers.

A web spider is the worker that visits web pages, discovers links, and updates the crawler's records. A crawler is the broader system around that worker. It coordinates URL storage, page retrieval, HTML parsing, and crawl state.

This lesson focuses on the repeated workflow used by a file-based Python crawler: select a pending URL, download its page, extract links, add eligible links to the queue, and record the processed URL as crawled.

What a Web Spider Does

A spider visits one URL at a time. It requests the page, receives the page's HTML, and passes that HTML to a link-extraction component. The extracted links become candidates for future visits.

Crawling is not the same as parsing. Crawling coordinates which URL should be visited and when its state changes. Parsing examines downloaded HTML and identifies hyperlinks. Storing crawl state is a separate responsibility: files or another storage system record which URLs are pending and which have already been processed.

A crawl is therefore a cycle, not a single request:

1. Choose a URL from pending storage.
2. Download the page.
3. Obtain its HTML.
4. Pass the HTML and current page URL to a link extractor.
5. Filter and queue newly discovered links.
6. Move the processed URL from pending state to crawled state.
7. Repeat while eligible pending URLs remain.

The Crawl Lifecycle

1. Select a Pending URL

A pending URL is a URL discovered by the crawler but not yet crawled. The spider selects one from the queue file before visiting a page. In a single-worker design, this may be a simple read-and-remove operation. In a multi-worker design, selection must also claim the URL safely so another worker cannot select it at the same time.

2. Request the Page

The page retrieval layer sends an HTTP request for the selected URL. A successful response contains content that the spider can inspect. The spider should distinguish a successful HTML response from an error response, a redirect that needs handling, or a non-HTML resource.

The exact retry and error policy is a later design decision. If the chosen workflow requires successful handling before completion, an unsuccessful attempt should not silently mark the URL as crawled. It can remain pending for retry or be recorded in a separate failure system.

3. Pass HTML to LinkFinder

HTML is the markup downloaded from a web server. A dedicated LinkFinder component parses that markup and identifies hyperlinks. It should receive both the HTML and the URL of the page that contained it.

The current page URL matters because links may be relative. For example, a page at https://example.test/catalog/index.html might contain ../contact.html. The parser or URL-resolution layer needs the current page address to resolve that link correctly. Link extraction should not guess the base URL independently of the spider.

4. Add Links to the Queue

Each extracted link is a candidate for future crawling. The crawler checks whether the candidate is already pending or already crawled. Only an eligible new URL is added to the queue.

Two different pages can contain the same destination. The first discovery places that destination in the queue. A later discovery finds it already pending and does not append another copy.

5. Record Completion

After the page has been handled according to the crawler's workflow, the current URL is removed from pending storage and written to crawled storage. A crawled URL is a URL whose page processing has been completed.

This update is what prevents the same page from being selected again. The exact point at which it occurs should be consistent. For example, a crawler may mark a URL complete after successful retrieval and link extraction, while a retry-oriented crawler may keep it pending when retrieval fails.

Pending URL Queue

A queue is a collection of URLs awaiting processing. In this file-based design, the queue file is persistent storage for pending URLs. Persistence means the work can survive process termination: a later run can read the remaining URLs instead of starting over.

The queue normally starts with one or more seed URLs. As the spider processes pages, it adds newly discovered eligible URLs. The queue can be implemented as a file containing one URL per line, although the details of file format and ordering belong to the implementation stage.

A URL should remain pending until the crawler has handled its page successfully according to the selected workflow. Removing a URL too early can lose work if the process fails before retrieval or parsing. Removing it only after processing can make the state transition easier to reason about, but concurrent workers still need a safe claim mechanism.

Crawled URL Record

The crawled file is persistent storage for URLs already processed. When a page is completed, its URL is removed from the pending queue and added to this record.

Conceptually, the crawler maintains two sets:

• Pending: URLs waiting to be processed.
• Crawled: URLs whose page processing is complete.

These sets should be disjoint. A URL should not simultaneously be waiting for processing and recorded as completed. A newly discovered link starts in pending state only if it appears in neither set.

Duplicate Prevention

A duplicate URL is a URL that is already pending or has already been crawled. Duplicate detection is essential because web pages commonly link to the same destination.

For every extracted candidate, the crawler should perform a check equivalent to:

if link not in pending and link not in crawled:
    add_to_pending(link)

This check handles both duplicate cases:

• The URL is already waiting in the queue. Do not add a second copy.
• The URL is already in the crawled record. Do not revisit it.

Without these checks, the queue can grow indefinitely as pages repeatedly rediscover the same destinations. Duplicate requests also waste bandwidth and processing time.

In practice, the same resource can appear in different URL forms. For example, differences in trailing slashes, fragments, or equivalent URL spellings may require normalization before comparison. Consistent normalization is a related concern, but duplicate prevention still depends on checking both state records.

HTML Retrieval and Link Extraction

The spider and the parser should have different responsibilities:

• The spider coordinates the crawl cycle and state transitions.
• The page retrieval layer obtains response content from a URL.
• LinkFinder parses HTML and finds links.
• Queue and crawled storage persist state.

A simplified coordination flow looks like this:

current_url = get_pending_url()
html = download_page(current_url)
links = LinkFinder(html, current_url).find_links()

for link in links:
    if link not in pending and link not in crawled:
        add_to_pending(link)

mark_as_crawled(current_url)

This is a conceptual example rather than a complete crawler. It omits response validation, URL normalization, domain restrictions, retries, logging, and safe concurrent file operations. Its purpose is to show the boundary between coordination and parsing.

Spider Architecture Boundaries

Separating these responsibilities makes the crawler easier to test and extend. LinkFinder can be tested with HTML samples without making network requests. Storage can be tested with temporary files. The spider can be tested with a fake retrieval layer and a controlled set of extracted links.

Multiple Spider Workers and Shared State

Multiple spider workers can process different pages concurrently. This can improve throughput when workers spend time waiting for network responses. However, concurrency changes the storage problem: all workers must use the same pending queue and crawled record.

Shared state is data used by more than one worker. Here, it includes the queue file, the crawled file, and any in-memory representation that is supposed to reflect them.

Race Conditions

A race condition occurs when the result depends on the timing of concurrent operations. Consider two workers that both read the queue before either updates it:

1. Worker A reads URL X as available.
2. Worker B reads the same queue and also sees URL X.
3. Both workers select and crawl URL X.

Another race can lose newly discovered links. Worker A reads the queue, Worker B reads the same old version, and then each writes an updated file. The later write may overwrite changes made by the other worker.

Coordinating Shared Files

Reading a file, changing its contents, and writing it back is a read-modify-write operation. Multiple workers must not perform that operation independently when their updates can overlap.

Safe designs use a coordinated claim and update strategy, such as:

• A file lock around operations that select or modify shared records.
• An atomic update that completes as one indivisible state change.
• A dedicated state manager, database, or message queue that provides concurrency guarantees.

An atomic update is a state change that other workers cannot observe halfway through. The goal is to ensure that claiming a pending URL, adding discovered URLs, and moving a completed URL to crawled storage do not produce contradictory records or discard another worker's changes.

Single-Page Example

Suppose the queue initially contains a seed page and the crawled file is empty.

1. The spider selects the seed URL from the pending queue.
2. The retrieval layer downloads its HTML.
3. LinkFinder receives the HTML and the seed URL, then extracts several links.
4. The spider compares each link with both pending and crawled records.
5. Links found in neither record are added to the pending queue.
6. The seed URL is removed from pending storage and added to crawled storage.
7. The spider selects one of the newly queued links and repeats the cycle.

Duplicate Discovery Example

Page A and Page B both contain a link to Page C. When Page A is processed, Page C is added to the pending queue. When Page B is processed, Page C is found in pending storage, so the crawler ignores the second discovery. If Page C has already been processed instead, its presence in crawled storage also causes the crawler to ignore it.

Troubleshooting

The Same Page Is Crawled Repeatedly

• Check that the URL is recorded in crawled storage after processing.
• Check both pending and crawled records before adding a discovered link.
• Use consistent URL normalization so equivalent URL forms compare equally.

The Queue Grows With Repeated Copies

• Do not append extracted links without checking existing state.
• Treat the queue as a unique collection of pending URLs, not an unrestricted list.
• Filter candidates against both pending and crawled URLs.

Two Workers Crawl the Same URL

• The workers may both read the queue before either records its claim.
• Use a coordinated claim or remove operation.
• Protect shared state with an appropriate locking or atomic-update strategy.

New Links Disappear With Multiple Workers

• A worker may overwrite a queue-file update made by another worker.
• Protect read-modify-write operations.
• Use shared-state management that preserves every worker's additions.

The Spider Downloads a Page but Finds No Links

• Confirm that the response is HTML rather than an image, document, or other resource.
• Confirm that the downloaded markup is actually passed to LinkFinder.
• Remember that links generated only by client-side JavaScript may not appear in the server-rendered HTML.
• Separate network diagnostics from parser diagnostics so the failing layer is clear.

Exam-Relevant Notes

• A spider is a worker that visits pages, discovers links, and updates crawl state; a crawler is the larger coordinating system.
• The queue file stores pending URLs, while the crawled file stores completed URLs.
• Every discovered link must be checked against both pending and crawled records.
• The normal state transition is pending to crawled after page processing is completed.
• Relative links must be resolved using the URL of the page where they were found.
• Multiple workers require coordinated access to shared queue and crawled state.
• A race condition can cause duplicate work or lost file updates.

Next Implementation Steps

These concepts prepare you for separating HTML parsing from crawler coordination and then implementing the crawler class. Continue with Parse HTML to focus on link extraction, or see Create The Crawler for the coordinating class. The broader project sequence is covered in Create A Web Crawler In Python.