VMware ESXi and vSphere Cluster Management

Add Discovered Links to a Python Web Crawler Queue

Learn how to filter discovered URLs, reject duplicates and external links, update a Python crawler queue, and persist pending and crawled state.

A web crawler finds new pages by extracting links from pages it has already fetched. Each extracted URL becomes a candidate URL: a URL that must be checked before the crawler attempts to visit it.

This lesson adds a queue-update method to a simple Python Spider class. The method accepts discovered links, keeps only eligible same-domain URLs, prevents duplicates, and saves the updated crawler state to disk.

The crawler queue's role

A queue is the collection of URLs waiting to be crawled. The crawled set is the collection of URLs that have already been processed. These collections have different jobs and should not be treated as one list.

When a page is fetched, its links become future crawl candidates. A typical lifecycle is:

1. Fetch one URL from the pending queue.
2. Extract links from the fetched page.
3. Validate each discovered candidate.
4. Enqueue valid, not-yet-seen URLs.
5. Mark the fetched URL as crawled.
6. Persist the changed queue and crawled state.

A URL can therefore move conceptually from discovered, to pending, to crawled.

Required Spider state

The queue-update method needs shared crawler state: the target domain, the pending queue, the crawled set, and the file paths used to save both collections. A set is a good representation because it stores unique values and provides efficient membership tests.

class Spider:
    target_domain = "example.com"
    queue = {"https://example.com/start"}
    crawled = {"https://example.com/"}

    queue_file = "queue.txt"
    crawled_file = "crawled.txt"

The exact initialization code may differ in your crawler. The important requirement is that the pending and completed URL collections are available consistently in memory.

Adding a queue-update method

Create a method whose single responsibility is processing a collection of discovered links. It does not fetch pages or extract HTML links; it evaluates URLs and updates crawler state.

A static method is appropriate when the routine does not need a particular Spider instance. It can operate using shared class-level state and is called through the class rather than through an instance.

class Spider:
    target_domain = "example.com"
    queue = {"https://example.com/start"}
    crawled = {"https://example.com/"}
    queue_file = "queue.txt"
    crawled_file = "crawled.txt"

    @staticmethod
    def add_links_to_queue(discovered_links):
        for url in discovered_links:
            if url in Spider.queue:
                continue

            if url in Spider.crawled:
                continue

            if get_domain_name(url) != Spider.target_domain:
                continue

            Spider.queue.add(url)

        file_to_set(Spider.queue, Spider.queue_file)
        file_to_set(Spider.crawled, Spider.crawled_file)

Here, discovered_links can be a set or another iterable of URL strings. The method checks every candidate before calling the set's add operation.

Duplicate prevention

A candidate must be rejected if it is already in either state collection:

• If it is in the queue, another page has already discovered it and it is waiting to be crawled.
• If it is in the crawled set, the crawler has already processed it.

Checking both collections prevents redundant downloads and repeated link extraction. Sets make these checks concise and efficient:

if url in Spider.queue:
    continue

if url in Spider.crawled:
    continue

Sets also prevent the same URL from being inserted more than once. A list would allow duplicates unless the program performed separate searches and cleanup operations.

Single-domain crawl filtering

A single-domain crawler should not follow every URL it discovers. Pages commonly contain links to documentation providers, social networks, advertisements, or unrelated sites. The crawler first parses the candidate's domain name, meaning its website host, and compares that value with the configured target domain.

candidate_domain = get_domain_name(url)

if candidate_domain != Spider.target_domain:
    continue

An internal link belongs to the target domain under the crawler's chosen policy. An external link points outside that scope and is rejected.

Define the domain policy explicitly

Domain comparison depends on the policy chosen for the crawler:

• Protocol: http://example.com/page and https://example.com/page have different schemes but the same host. A host-only policy treats them as the same domain, while a scheme-restricted policy may not.
• Subdomains: Decide whether blog.example.com belongs to a crawl targeting example.com. Exact host matching excludes it; an explicit suffix policy may include it.
• Hostnames: Normalize case and remove an ending dot when appropriate. Decide how www.example.com and example.com should relate before comparing them.

For an exact-host policy, both the configured target and parsed candidate should be normalized into the same form. Do not silently use a broader subdomain rule unless that behavior is intended.

Queue insertion

Only a candidate that passes all checks should be inserted:

for url in discovered_links:
    if url in Spider.queue:
        continue
    if url in Spider.crawled:
        continue
    if get_domain_name(url) != Spider.target_domain:
        continue

    Spider.queue.add(url)

Validation comes before insertion so malformed, external, or already-seen values never become pending work. The queue's add operation is appropriate when the queue is represented as a set.

Persisting crawler state

Changing a set in memory does not automatically change its file. After queue or crawled state changes, write both collections to their configured files:

file_to_set(Spider.queue, Spider.queue_file)
file_to_set(Spider.crawled, Spider.crawled_file)

This is persistence: saving in-memory state to durable files. Persistence allows a later run to resume with pending URLs intact and prevents completed URLs from being revisited after the process stops.

Saving both collections together keeps the state model synchronized. If only the queue is saved, newly discovered work may survive but completion information may be lost. If only the crawled set is saved, pending work may disappear.

Connecting extraction to the crawl workflow

The queue-update method belongs after link extraction in the page-crawling workflow. The page crawler fetches a URL, extracts links, and passes the result to the method:

class Spider:
    @staticmethod
    def crawl_page(page_url):
        html = fetch_page(page_url)
        discovered_links = extract_links(html, page_url)
        Spider.add_links_to_queue(discovered_links)

The extraction component may need to resolve relative links and normalize URLs before queue processing. The queue method should receive usable URL strings rather than fragment-only references, mailto: addresses, JavaScript links, or unresolved relative paths.

Example with a mixed discovered-link set

Suppose the target domain is example.com, the queue already contains https://example.com/contact, and the crawled set contains https://example.com/. The extracted links are:

discovered = {
    "https://example.com/blog",
    "https://example.com/contact",
    "https://example.com/",
    "https://news.example.net/story",
}

Only https://example.com/blog is added. The contact URL is already pending, the root URL was already crawled, and the news URL is external.

Practical examples

• New internal page: https://example.com/tutorials is added when it matches the target domain and appears in neither state set.
• Already queued: https://example.com/tutorials is ignored when it is already pending.
• Previously crawled: https://example.com/about is ignored when it is in the crawled set.
• External page: https://other-site.test/article is rejected because its parsed domain differs from the target.

Troubleshooting

The crawler repeatedly visits the same pages

Check that every candidate is tested against both the queue and crawled set before insertion. Also verify that both collections are sets and that URLs are normalized consistently. Differences such as a trailing slash, case variation, or alternate hostname can make equivalent pages look different.

The crawler follows unrelated websites

Confirm that domain comparison occurs before queue insertion. Check that the domain parser and configured target use the same format, such as hostname-only values without a protocol.

New URLs disappear when the program stops

The queue may be updated only in memory. Call the file-writing helper after state changes and verify that the queue file path is writable and is the same path used during initialization.

Valid internal URLs are skipped

Compare normalized values and define the treatment of www, non-www, protocols, and subdomains. A mismatch between www.example.com and example.com can cause an intended internal URL to be rejected.

Malformed values enter the queue

Resolve relative links and remove or reject fragment-only, mailto:, JavaScript, unsupported-scheme, and malformed references before this method processes them. URL normalization is ideally part of link extraction or a dedicated validation step.

Exam-relevant notes

• The pending queue contains future work; the crawled set contains completed work.
• Discovered links must be validated before queue insertion.
• Check both pending and crawled collections to avoid redundant processing.
• Use a domain parser and compare its result with the configured target domain.
• Use a set for uniqueness and efficient membership testing.
• Persist both queue and crawled state so a crawl can resume safely.
• A static method is suitable when the operation uses shared class state rather than instance-specific data.

The complete pattern is: extract links, normalize and validate candidates, skip pending or completed URLs, reject candidates outside the domain policy, add eligible URLs to the queue, and persist the updated state.