VMware ESXi and vSphere Cluster Management

Crawl a Single Page in a Python Web Crawler

Learn how crawl_page processes one URL, discovers links, updates queue and crawled sets, prevents duplicates, and saves crawler state.

What the crawl-page operation does

A web crawler is a program that visits web pages, extracts links, and follows eligible links according to defined rules. A URL is the web address that identifies a page to fetch and process.

The page-crawling operation processes one URL at a time. It gathers links from the current page, places eligible new URLs into the crawl queue for later work, and marks the current URL as completed. This turns one page into possible future work without immediately crawling every discovered link.

In a file-backed crawler, the operation usually performs this sequence:

  1. Receive a worker label and a page URL.
  2. Check whether the URL has already been crawled.
  3. Report which worker is handling the URL.
  4. Gather links from the current page.
  5. Add eligible links to the queue.
  6. Remove the current URL from the queue.
  7. Add the current URL to the crawled set.
  8. Persist the updated collections to files.

Crawler URL state

The crawl queue, also called the frontier, is the collection of URLs waiting to be crawled. The crawled set contains URLs that the crawler has already processed.

Python sets are suitable for both collections because they store unique items and provide efficient membership checks. A duplicate URL added several times still appears only once in a set.

Stage | Current URL in Queue | Current URL in Crawled Set | Discovered Links | Queue Outcome Before processing | https://example.test/start | No | Not available | The URL is pending After processing | No | Yes | /about, /contact | Eligible new links are waiting

The important state invariant is that a successfully completed URL should not remain in both collections. During a carefully defined transition, the URL leaves the pending queue and enters the crawled set.

The Spider class and a static method

A static method is a method associated with a class namespace that does not receive an instance as its first argument. This is useful when the crawler stores shared state on the class itself and workers call the same operation.

The method can receive a worker identifier, such as a thread name, and the URL to process. Because it is static, it accesses shared collections through Spider.queue and Spider.crawled rather than through self.

class Spider:
    queue = set()
    crawled = set()

    @staticmethod
    def crawl_page(worker_name, page_url):
        if page_url in Spider.crawled:
            return

        print(
            f"{worker_name} crawling {page_url} "
            f"| queued: {len(Spider.queue)} "
            f"| crawled: {len(Spider.crawled)}"
        )

        discovered_links = Spider.gather_links(page_url)
        Spider.add_links_to_queue(discovered_links)

        Spider.queue.remove(page_url)
        Spider.crawled.add(page_url)
        Spider.update_files()

This method coordinates crawler state. It does not need to contain the details of downloading HTML or parsing anchor elements. Those responsibilities belong to the link-gathering operation.

Preventing duplicate crawls

A duplicate crawl means processing the same URL more than once. The membership guard must run before fetching or otherwise processing the page:

if page_url in Spider.crawled:
    return

This guard prevents repeated work when multiple pages reference the same URL. It is also essential for cycles. For example, if page A links to page B and page B links back to page A, the crawled set prevents the crawler from looping forever.

Duplicate prevention should also exist in add_links_to_queue. A discovered link should be inserted only when it is neither already crawled nor already waiting:

@staticmethod
def add_links_to_queue(links):
    for link in links:
        if link not in Spider.crawled and link not in Spider.queue:
            # Apply domain and other crawl-scope rules here.
            Spider.queue.add(link)

The queue method is the right place for scope rules such as restricting URLs to the target domain. Keeping those rules there gives every source of discovered links the same filtering behavior.

Reporting crawl progress

Progress output should identify the worker and active URL, along with the current queue and crawled counts:

print(
    f"{worker_name} crawling {page_url} "
    f"| queued: {len(Spider.queue)} "
    f"| crawled: {len(Spider.crawled)}"
)

A thread name is an identifier showing which worker is handling a URL. Progress output helps reveal whether links are being discovered, whether the queue is shrinking or growing, and whether duplicate protection is working.

In a multithreaded crawler, counts can change immediately after they are printed because another worker may update the shared collections. Treat the output as an observation at a particular moment, not as an atomic snapshot unless state access is synchronized.

Gathering links from the current page

Link extraction is the process of finding URLs referenced by a page's HTML. Keep it separate from queue management:

discovered_links = Spider.gather_links(page_url)
Spider.add_links_to_queue(discovered_links)

gather_links receives the active page URL and returns a collection of discovered links. These links are candidates for future crawling; they are not pages that crawl_page processes immediately.

Component | Input | Responsibility | Output or State Change crawl_page | Worker label and page URL | Coordinate duplicate checks, extraction, state transition, logging, and persistence | Current URL becomes completed gather_links | Current page URL | Download or read the page and extract referenced URLs | Collection of discovered links add_links_to_queue | Discovered links | Reject crawled, queued, or out-of-scope links | Eligible URLs enter the queue update_files | Shared queue and crawled collections | Write current state to configured files | Persistent state reflects memory

Moving a URL from queued to crawled

After discovered links have been handled, the current URL transitions from pending to completed:

Spider.queue.remove(page_url)
Spider.crawled.add(page_url)

The removal happens after link handling so that the active page is still represented as pending while its outgoing links are being processed. The addition to the crawled set records that processing has completed.

remove assumes that page_url exists in the queue. This is a useful invariant when the bootstrap code always selects URLs from that queue. If callers may invoke crawl_page with arbitrary URLs, use a deliberate policy such as validating membership before the call or using discard when missing URLs should not raise an exception. Do not hide an unexpected queue-management bug without deciding what the missing state means.

Persisting crawler state

Persistence means saving in-memory crawler state to files. After both collections change, call the file-update operation:

Spider.queue.remove(page_url)
Spider.crawled.add(page_url)
Spider.update_files()

Persistence provides visibility into progress, supports recovery after a process stops, and allows a later run to continue without treating completed pages as new work. The files should represent the state after the page's transition.

How crawl_page fits the crawler workflow

crawl_page is a worker-level operation used after crawler initialization. Bootstrap code creates or loads the queue and crawled collections, selects a URL from the queue, and passes that URL to the method.

while Spider.queue:
    next_url = next(iter(Spider.queue))
    Spider.crawl_page("worker-1", next_url)

A real multithreaded implementation would coordinate selection and updates so that two workers do not claim the same URL. The static method still represents the work performed for one claimed URL, while the bootstrap or scheduler controls how URLs are assigned.

The link-gathering method is a dependency of this workflow: it supplies the URLs that add_links_to_queue evaluates and stores for later worker calls.

Practical scenarios

Initial page

If the queue contains one starting URL and the crawled set is empty, the worker gathers links from the starting page, adds eligible links, removes the starting URL, adds it to the crawled set, and saves both collections.

Repeated discovery

If another page references a URL already in the crawled set, the queue method rejects it and the duplicate guard prevents processing it again.

Circular links

If two pages link to each other, the first page eventually enters the crawled set. When its URL is encountered again, the membership check returns immediately. Each page is therefore processed once.

Several queued URLs

When several URLs are waiting, progress output shows the active URL and the current counts. After completion, the active URL is no longer pending and appears in the completed collection, while newly eligible links may increase the queue.

Troubleshooting crawl_page

The crawler visits the same page repeatedly

  • Confirm that the crawled-set check occurs before fetching or processing.
  • Confirm that completed URLs are added to Spider.crawled.
  • Confirm that queue insertion rejects URLs already in either collection.
  • Inspect queue and crawled counts and verify that a processed URL appears in the completed collection.

Removing the active URL raises an error

  • Check that the URL was placed in the queue before crawl_page was called.
  • Check whether another worker changed the shared queue.
  • Compare the exact URL representation in the queue with the method argument.
  • Use consistent URL normalization before storing and comparing URLs.

New links are never crawled

  • Check whether gather_links returns any URLs.
  • Compare the gathered-link count with queue counts before and after insertion.
  • Inspect domain and other scope filters in add_links_to_queue.
  • Verify that the bootstrap loop selects new URLs from the queue.

State files are stale

  • Confirm that update_files() runs after the queue and crawled sets change.
  • Check the configured file paths.
  • Check file permissions and whether the project directory exists.

Progress counts seem unexpected

  • Document whether counts are printed before or after the state transition.
  • Check that duplicate URLs are not entering the queue.
  • For multiple workers, remember that another worker may change shared state between log statements.

Implementation checklist

  • Define crawl_page as a static method on Spider.
  • Accept a worker identifier and a page URL.
  • Reference shared queue and crawled collections through the class.
  • Return immediately when the URL is already crawled.
  • Log the worker, URL, queued count, and crawled count.
  • Call the link-gathering method with the active URL.
  • Pass discovered links to the queue-management method.
  • Ensure queue insertion rejects crawled, queued, and out-of-scope URLs.
  • Remove the completed URL from the queue.
  • Add it to the crawled set.
  • Call the persistence method after the state change.
  • Coordinate shared-state access when adding multiple workers.