VMware ESXi and vSphere Cluster Management

Create Crawl Jobs and Coordinate Worker Queues in a Python Web Crawler

Learn how to load pending URLs from a file, enqueue crawl jobs for worker threads, wait for completion, and repeat crawl passes safely in Python.

A crawler usually discovers more links while visiting pages. Those links need to be stored so the program can process them later. This lesson connects the saved URL state to the in-memory queue consumed by worker threads.

You will implement two functions: one that creates jobs for pending URLs and one that controls repeated crawl passes.

How Crawl-Job Creation Works

A pending URL queue file is a persistent file containing links that have been found but not yet crawled. A job queue is a thread-safe, in-memory queue used to hand URL work to worker threads.

These are different queues with different purposes:

The job-creation routine reads the persistent file, converts its contents into a set of unique URLs, and enqueues each URL in the runtime queue. It does not replace the file with the runtime queue. The file is the crawler's saved state; the thread queue is temporary work coordination.

Required Components

The functions below assume that earlier parts of the crawler already provide these values:

• QUEUE_FILE: the constant containing the path to the pending URL queue file.
• file_to_set(): a helper that reads a file and returns a set of unique URLs.
• work_queue: the shared thread-safe queue instance, commonly created with queue.Queue().
• Started worker threads that continuously retrieve URLs from work_queue.

For example, the main crawler module may already contain the following setup:

from queue import Queue

QUEUE_FILE = "queue.txt"
work_queue = Queue()

The exact constant and object names can differ. What matters is that the controller and all workers use the same queue object and the same pending-URL file.

Implement the Job-Creation Function

An enqueue operation places a URL into the shared work queue. The queue's put() method is the insertion operation. After inserting all URLs, join() waits until every submitted job has received a task-completion signal.

def create_jobs():
    pending_urls = file_to_set(QUEUE_FILE)

    for url in pending_urls:
        work_queue.put(url)

    work_queue.join()

This function performs four steps:

1. Read the pending URL file through the existing file-to-set helper.
2. Receive a set, so duplicate entries are removed for this loading pass.
3. Insert every unique URL into the shared runtime queue.
4. Block at work_queue.join() until workers finish all submitted jobs.

A queue tracks unfinished tasks. Each call to put() increases that count. A worker must call task_done() once for the corresponding item. When the count reaches zero, join() returns.

Worker Completion Requirements

A worker thread is a concurrent execution unit that waits for URL jobs and crawls assigned pages. A worker normally retrieves a URL with get(), passes it to the spider, and then signals completion.

def worker():
    while True:
        url = work_queue.get()
        try:
            spider(url)
        finally:
            work_queue.task_done()

The finally block is important. It sends the completion signal even when crawling the URL raises an exception. Without that signal, the main thread can wait forever at work_queue.join().

Implement the Crawl Controller

The controller decides whether another crawl pass is needed. A pass loads pending links, submits them to workers, waits for those workers, and then checks the persisted state again.

def crawl():
    pending_urls = file_to_set(QUEUE_FILE)

    if pending_urls:
        print(f"{len(pending_urls)} links remain")
        create_jobs()

The set is loaded inside the controller each time it runs. This is necessary because worker activity can update the queue file while a pass is in progress. The controller should not rely on an old set captured before workers finish.

When the set is empty, the condition is false. The function creates no jobs and the crawl cycle ends normally.

Repeated Crawl Passes

Workers may discover new URLs while crawling the current batch. The spider should update the crawler's persistent files: processed URLs should no longer remain pending, and newly discovered, in-scope URLs should be added to the pending file.

After create_jobs() returns, all jobs submitted for that pass have completed. The controller can then reload the pending file and decide whether newly discovered links require another pass.

A recursive controller expresses this directly:

def crawl():
    pending_urls = file_to_set(QUEUE_FILE)

    if pending_urls:
        print(f"{len(pending_urls)} links remain")
        create_jobs()
        crawl()

The control flow is:

1. Load the pending links from disk.
2. Check whether at least one URL remains.
3. Report the number of outstanding links.
4. Enqueue a job for each pending URL.
5. Let workers crawl the URLs concurrently.
6. Wait until every submitted job signals completion.
7. Call the controller again so it can inspect the updated pending file.
8. Stop when the reloaded set is empty.

Three-URL Example

Suppose the pending file contains three URLs. The controller loads them into a set and displays 3 links remain. The job creator calls put() three times. Available workers process the URLs concurrently, and the main flow waits at join().

During processing, the spider may remove the three visited URLs from the pending state and discover additional links. When the first pass finishes, the controller reads the file again. If new links remain, it creates another batch. If no links remain, the crawl ends.

Empty and Duplicate Queue Cases

No URLs Remain

If file_to_set(QUEUE_FILE) returns an empty set, the controller prints nothing unless you add an explicit completion message. It does not call create_jobs(), so no empty crawl batch is submitted.

Duplicate URLs Exist

A set removes repeated entries while loading the file. For example, three identical lines produce one URL in pending_urls, so only one runtime job is submitted during that pass. URL normalization is still important: strings such as a URL with and without a trailing slash may represent the same resource but remain different set values unless normalized.

Placement in the Crawler Application

Place create_jobs() and crawl() in the main crawler module after the prerequisite imports, constants, queue setup, file helpers, and worker startup logic. The worker startup must occur before the controller entry point is called.

start_workers()
crawl()

In this arrangement, crawl() is the entry point that begins processing queued links. It depends on the existing pending-file constant, file-to-set helper, and shared queue object. The spider remains responsible for crawling pages and safely updating queued and crawled URL records.

Troubleshooting

The Program Waits Forever After Submitting Jobs

• Check that every worker pairs each get() with exactly one task_done().
• Put the completion call in a finally block so worker failures do not leave unfinished tasks.
• Confirm that worker threads were created and started before crawl() ran.
• Add logging before retrieval, after retrieval, around the spider call, and before completion.

The Crawler Reports Links but Does Not Visit Them

• Confirm that create_jobs() calls work_queue.put(url) for every loaded URL.
• Confirm that workers consume the same work_queue instance used by the controller.
• Inspect the worker loop to ensure it has not exited unexpectedly.

The Crawl Repeats Without Finishing

• Verify that processed URLs are removed from the persisted pending queue.
• Filter discovered links against visited or crawled URLs.
• Check that the spider is not repeatedly adding equivalent URLs in different textual forms.
• Inspect the pending file and confirm that its contents eventually become empty.

The Link Count Is Unexpectedly High

• Inspect the pending file for stale URLs from an earlier run.
• Normalize URLs before storing and comparing them.
• Apply domain and crawl-scope restrictions in the spider.

Recursive Control Versus an Iterative Loop

Recursive control is easy to read: after one completed pass, crawl() calls itself. However, a very large number of passes can create a deep call stack. An iterative controller avoids that limitation while preserving the same behavior:

def crawl():
    while True:
        pending_urls = file_to_set(QUEUE_FILE)

        if not pending_urls:
            break

        print(f"{len(pending_urls)} links remain")
        create_jobs()

Both versions reload the persisted set before each pass, enqueue only when work remains, wait for workers, and stop when the pending set is empty. Use the form required by the surrounding crawler design, and prefer the loop for very large crawls.

Exam-Relevant Notes

• The file-backed pending queue and the in-memory thread queue are separate components.
• file_to_set() supplies unique pending URLs for a pass.
• put() enqueues a URL; join() waits for all unfinished queue tasks.
• Every worker job must receive exactly one completion signal through task_done().
• Workers must already be running before the controller submits jobs.
• The controller must reload the pending file after a pass because workers may discover new links.
• Repeated or recursive controller calls continue crawling while the persisted pending set is nonempty.

For the surrounding worker-thread concepts, continue with the worker creation and queue coordination lesson.