VMware ESXi and vSphere Cluster Management

Python Web Crawler Spider Concepts

Learn how a Python web crawler spider fetches pages, extracts and validates links, manages queue and crawled files, and coordinates multiple workers.

A web crawler is a system that systematically visits web pages by following links. A spider is one worker in that system: it processes a page, discovers more URLs, and helps move the crawl forward.

This lesson focuses on the core workflow of a Python crawler that uses a pending URL queue, a crawled URL collection, an HTTP fetcher, and a LinkFinder-style HTML parser.

What a Web Crawler Spider Does

A spider usually processes one page at a time. It selects a pending URL, downloads its page, extracts links from the returned HTML, filters those links, and schedules eligible URLs for later work.

These responsibilities are related but separate:

• Page crawling: requesting a URL and receiving its response.
• Link parsing: examining HTML to find hyperlink targets such as values in <a href="#"> elements.
• URL storage: recording which URLs are waiting, completed, rejected, or failed.

Separating these jobs makes the crawler easier to test and extend. For example, the link parser can find raw href values, while the spider decides whether those values are safe and relevant to crawl.

Conceptual Crawler Architecture

The data flow is:

1. The spider obtains a URL from the pending queue.
2. The HTTP fetcher requests the URL.
3. The spider passes the response HTML and source URL to LinkFinder.
4. The parser discovers links.
5. The spider normalizes, validates, and deduplicates those links.
6. Eligible new links are written to the queue.
7. The processed URL is removed from active pending work and recorded as crawled.

The Crawl Lifecycle

For one URL, the crawler follows an iterative lifecycle. After finishing one iteration, it repeats the same process while pending work remains.

1. Read a URL from the pending queue

The crawl queue is the collection or file containing URLs that still need to be visited. A spider obtains its next task from this shared queue.

In a simple file-based design, each line might contain one URL:

https://example.com/
https://example.com/about
https://example.com/products

The queue is persistent storage, so pending work can survive a program restart. A URL selected for processing should be claimed or removed from active pending work in a way that prevents another worker from selecting it at the same time.

2. Request the URL

The spider connects to the selected URL through an HTTP client and retrieves the page response. A successful response may contain HTML, which becomes the input to the link parser.

Not every response is a page that should be parsed. The spider should distinguish successful HTML responses from situations such as:

• Network failures, timeouts, or connection errors.
• HTTP error responses.
• Successful responses containing images, documents, JSON, or other non-HTML content.

A crawler needs a failure policy. It might retry temporary failures and then record the URL as failed or skipped. A failure should not leave a queue entry stuck forever.

3. Pass HTML to LinkFinder

LinkFinder is a parser component responsible for finding links in a page's HTML. It examines anchor elements and extracts their href values.

<a href="#">About</a>
<a href="#">News</a>

The current page URL is important because an extracted value may be a relative URL. A relative URL does not contain a complete scheme and host, so it must be interpreted in relation to the source page.

4. Normalize and filter discovered links

Raw extracted values are not automatically ready for the queue. The spider should convert eligible values into a consistent form called a canonical or normalized URL representation.

• Resolve relative URLs against the source page URL.
• Accept only supported schemes, normally http and https.
• Reject mailto:, javascript:, and similar non-page schemes.
• Ignore fragment-only links such as #details.
• Remove URL fragments, or handle them consistently, when fragments do not identify separate server responses.
• Enforce the crawl scope, such as allowing only the target domain.
• Reject malformed or otherwise unusable URLs.

An absolute URL includes its scheme and host, for example https://example.com/guide.html. The queue should normally store absolute, normalized URLs rather than a mixture of raw and resolved forms.

5. Add unseen links to the queue

Deduplication means preventing the same URL from being placed into crawl work more than once. Before scheduling a discovered URL, check both the pending queue and the crawled set.

if candidate not in pending and candidate not in crawled:
    pending.add(candidate)

This check is necessary because two pages may link to the same destination. It is also necessary when a page links back to a page already completed.

6. Record the processed URL

After the page has been handled according to the crawler's success or failure policy, the URL transitions out of active pending work. A successfully processed page is added to the crawled set, the collection or file containing URLs that have already been processed.

A URL should normally be in one meaningful state at a time: pending, completed, or rejected/failed. Temporary implementation states such as crawling may also be needed while a worker owns a task.

Queue File for Pending URLs

A queue file provides durable storage for work that has been discovered but not processed. Every spider in the project must use the same logical queue if workers are intended to share a crawl.

The queue answers the question: Which URLs still need attention? It should not be treated as an unrestricted list. Before writing a URL, normalize it, validate its scope and scheme, and check for duplicates.

When a URL moves from queued to crawled, the state transition should be explicit:

1. Claim the URL so another worker cannot choose it simultaneously.
2. Fetch and process it.
3. Remove it from pending work, or mark it no longer pending.
4. Record it in the completed collection if processing succeeded.

Crawled File for Completed URLs

The crawled file is persistent storage for completed URLs. It prevents the crawler from repeatedly following cycles such as /a linking to /b and /b linking back to /a.

At startup, a crawler can load the crawled records into a set for fast membership checks. The same normalized representation must be used when adding and checking URLs; otherwise, equivalent values may appear different.

if url not in pending_urls and url not in crawled_urls:
    pending_urls.add(url)

In a robust design, a URL is not repeatedly present in both active pending work and completed work. If a separate in_progress collection is used, it should be clear whether that state is temporary ownership or durable crawl history.

HTML Link Extraction with LinkFinder

Link extraction is the process of locating hyperlink targets in HTML. A LinkFinder-style class can receive the HTML and source URL, inspect anchor elements, and return discovered href values.

It is useful to distinguish two stages:

• Raw extraction: LinkFinder reads values exactly as they appear in HTML, such as /docs or ../guide.html.
• Finalization: the spider resolves, normalizes, validates, scopes, and deduplicates those values before storage.

Keeping the final crawl decision in the spider gives one component control over policy. LinkFinder discovers links; the spider decides which links are valid and where they are stored.

URL Validation, Normalization, and Deduplication

Crawl scope defines the boundaries of the crawl. For a domain-restricted crawler, a candidate might be accepted only when its host matches the target domain. Other projects may allow a list of hosts or selected path prefixes.

Relative URL resolution

Suppose the current page is https://example.com/docs/start.html and LinkFinder returns ../guide.html. The spider resolves the value against the source URL and obtains:

https://example.com/guide.html

Only after resolution should the spider apply scope checks and store the candidate. Storing raw relative values can cause incorrect requests and duplicate representations.

Fragments and consistent forms

A fragment begins with # and identifies a location within a document. Fragments are not sent to the server in the HTTP request, so a page URL with #part-one commonly represents the same fetched resource as the URL without the fragment.

Choose a consistent policy, usually removing fragments before comparison and storage. Similar consistency may be needed for trailing slashes, case-sensitive hostnames, default ports, and escaped characters. The exact canonicalization policy should match the crawler's scope and application needs.

URL State Transitions

Multiple Spiders and Shared State

Multiple spider workers can crawl different pages in parallel, but they need a common view of pending and completed URLs. Every worker must read and update the same logical queue and crawled storage.

Without coordination, a race condition can occur:

1. Worker A reads a pending URL.
2. Before A records ownership, worker B reads the same URL.
3. Both workers fetch and process the page.
4. Both workers may write overlapping queue or crawled updates.

A concurrent crawler therefore needs an atomic claim-and-update operation. Claiming means selecting a pending URL and marking or removing it in one coordinated action. File-based implementations may require locking and careful rewrite rules; database or message-queue implementations commonly provide stronger atomic operations.

After claiming different URLs, workers can independently fetch pages and submit discovered links to shared storage. Updates that add links and record completion must still be synchronized so that duplicate work and lost writes do not occur.

Practical Example: Single-Page Crawl Flow

Assume the pending queue contains https://example.com/ and the crawled collection is empty.

1. A spider selects the homepage from the queue.
2. It downloads the homepage HTML.
3. LinkFinder finds /about, /products, and an external URL.
4. The spider resolves the relative paths to https://example.com/about and https://example.com/products.
5. The internal URLs are added if they are not already queued or crawled.
6. The homepage is removed from pending work and added to the crawled collection.

After this iteration, the two internal pages are pending, the homepage is completed, and the external URL has been rejected by the scope rule.

Practical Example: Duplicate-Link Prevention

Suppose two different pages both link to https://example.com/contact.

1. The first discovery checks the queue and crawled set, then adds the contact URL to the queue.
2. The second discovery performs the same checks.
3. Because the URL is already known, the second discovery does not add another copy.

This works only when both workers use the same normalized URL and, in a concurrent system, the check-and-add operation is coordinated.

Practical Example: Two Workers

With two spider workers, both workers use the same pending and completed URL storage. Each worker atomically claims a different pending URL before fetching it. Each then records newly discovered URLs in the shared queue and records its completed page in the shared crawled store.

The important principle is not merely running two loops. It is ensuring that shared state has coordinated ownership and update operations.

Troubleshooting Common Problems

The same page is crawled repeatedly

• Likely cause: the crawler checks neither both pending and completed stores nor consistent URL forms.
• Resolution: normalize every candidate first, then check both queue and crawled records before scheduling it.
• Also check: fragment removal and trailing-slash policy, so equivalent URLs compare consistently.

Relative links point to the wrong page

• Likely cause: raw href values are stored without using the source page URL.
• Resolution: resolve each relative value against the current page before validation and queuing.

Multiple spiders crawl the same URL

• Likely cause: workers read and update the queue independently.
• Resolution: use an atomic claim-and-update approach and ensure all workers use synchronized shared state.

The queue fills with unusable links

• Likely cause: there is no filtering for external domains, unsupported schemes, fragments, or malformed values.
• Resolution: resolve URLs, validate schemes, enforce scope, apply the fragment policy, and reject non-crawlable candidates before storage.

A failed request stops progress

• Likely cause: the failed URL remains indefinitely in pending work.
• Resolution: define retry limits and then mark the URL failed or skipped. Keep failure records distinct from successful crawled-page records when that distinction matters.

Exam-Relevant Notes

• A spider is a worker that processes pages and discovers URLs; it is not the same thing as the queue or the parser.
• HTML is the input to link extraction, while the source URL is required to resolve relative links.
• LinkFinder discovers candidates; the spider applies crawl scope, validation, normalization, deduplication, and storage policy.
• Check both pending and crawled URLs before scheduling a candidate.
• Multiple workers require shared state and coordinated queue claims to avoid race conditions.
• Successful HTML, non-HTML responses, network failures, and rejected URLs should be handled as different outcomes.

Summary

A Python crawler spider repeatedly takes a pending URL, fetches its HTML, passes that HTML and the source URL to a link finder, and evaluates the discovered links. Relative links become absolute URLs, unsupported or out-of-scope links are rejected, and unseen candidates enter the shared queue. The processed URL leaves pending work and enters the crawled set. With multiple spiders, atomic claims and synchronized updates are required to keep this workflow correct.

Continue with spider concepts and crawler design when reviewing how these pieces fit into a larger crawler project.