VMware ESXi and vSphere Cluster Management
Create Daemon Worker Threads for a Python Web Crawler
Learn to build Python daemon worker threads that consume URLs from a shared queue, crawl pages concurrently, and report task completion safely.
A crawler worker is a background thread that repeatedly takes a URL from a shared task queue and processes it. Instead of creating one thread for every URL, a crawler can create a fixed pool of workers and let each worker handle many URLs during its lifetime.
This design separates thread setup from crawl coordination. The startup code creates workers, while the crawl coordinator seeds or manages URL tasks. The workers consume those tasks and call the crawler's page-processing method.
How Worker Threads Fit into a Crawler
A queue is a thread-safe task container. In this design, the crawl coordinator places URLs into the queue, and worker threads retrieve them. The queue is the shared handoff point between URL discovery and URL processing.
Concurrency means that several tasks can make progress during overlapping periods. With multiple workers, one URL may be waiting for a network response while another worker processes a different URL.
Configure a Fixed Number of Workers
Define a setting such as NUMBER_OF_THREADS. The startup routine loops once for every requested worker, creates one threading.Thread object per iteration, assigns the worker function as its target, and starts it.
import queue
import threading
NUMBER_OF_THREADS = 4
work_queue = queue.Queue()
def create_workers():
for _ in range(NUMBER_OF_THREADS):
worker = threading.Thread(target=work)
worker.daemon = True
worker.start()
The target is the callable that the thread runs after start() is called. Pass the function itself, as in target=work; do not call it during construction with target=work().
A fixed pool avoids creating a separate thread for every URL. For example, four workers can process several queued links concurrently without allowing the number of threads to grow with the queue size.
Daemon Thread Behavior
A daemon thread is a thread that does not keep the Python process alive after the main program has finished. Set the daemon property before calling start():
worker = threading.Thread(target=work)
worker.daemon = True
worker.start()
Daemon workers are convenient for a crawler because the process can exit when the main workflow is done. However, daemon status is not a completion mechanism. If the main thread exits while URLs remain in the queue, outstanding work may be abandoned.
If the crawler must finish its queued work, keep the main workflow active and coordinate completion with work_queue.join() or another shutdown mechanism. Do not rely on daemon behavior to wait for tasks.
Write the Persistent Worker Loop
The worker function normally uses an indefinite loop. A single thread can therefore process many URLs rather than stopping after one task.
def work():
while True:
url = work_queue.get()
try:
name = threading.current_thread().name
Spider.crawl_page(name, url)
finally:
work_queue.task_done()
queue.get() retrieves the next URL. By default, it blocks while the queue is empty, so the worker sleeps while waiting instead of repeatedly polling the queue.
threading.current_thread().name identifies the thread currently executing the function. Passing that name to Spider.crawl_page makes concurrent activity visible in diagnostic output or logs.
Spider.crawl_page represents the crawler's page-processing method. It is responsible for visiting the supplied URL and performing the page-specific work, such as fetching content, parsing links, and adding discovered tasks according to the crawler's design.
Account for Completed Queue Tasks
Each successful work_queue.get() must have exactly one matching work_queue.task_done(). The call to task_done() tells the queue that processing for the retrieved item has finished.
Put task_done() in a finally block after obtaining the task. This preserves the accounting even if page processing raises an exception.
def work():
while True:
url = work_queue.get()
try:
worker_name = threading.current_thread().name
try:
Spider.crawl_page(worker_name, url)
except Exception as error:
print("Could not crawl", url, error)
finally:
work_queue.task_done()
The inner try catches an error for one URL so the worker can continue with later tasks. The outer finally guarantees that the queue receives its completion notification after the item has been acquired.
Initialize Workers Before Crawl Coordination
Workers should exist before the coordinator expects queued URLs to be consumed. A typical application startup sequence is:
- Import
threadingand create a shared thread-safe queue. - Set
NUMBER_OF_THREADSto a suitable positive integer. - Call the worker-creation routine.
- Call the crawl coordinator to seed or manage URL work.
- Wait for completion if the application is intended to finish after a defined crawl.
if __name__ == "__main__":
create_workers()
crawl()
In this sequence, create_workers() starts the consumers first. The crawl() coordinator then supplies or manages jobs. Depending on the crawler architecture, the coordinator may add an initial URL, wait for the queue to drain, or continue coordinating newly discovered links.
Complete Minimal Pattern
import queue
import threading
NUMBER_OF_THREADS = 4
work_queue = queue.Queue()
class Spider:
@staticmethod
def crawl_page(worker_name, url):
print(worker_name, "crawling", url)
# Fetch and parse the page here.
def work():
while True:
url = work_queue.get()
try:
worker_name = threading.current_thread().name
try:
Spider.crawl_page(worker_name, url)
except Exception as error:
print("Crawl failed for", url, error)
finally:
work_queue.task_done()
def create_workers():
for _ in range(NUMBER_OF_THREADS):
worker = threading.Thread(target=work)
worker.daemon = True
worker.start()
def crawl():
for url in ["https://example.invalid/one", "https://example.invalid/two"]:
work_queue.put(url)
work_queue.join()
if __name__ == "__main__":
create_workers()
crawl()
The example demonstrates the architecture, but the example host is only a placeholder. A real crawler should use its configured, permitted targets and should implement timeouts, error handling, duplicate detection, link extraction, and applicable access policies.
Understand the Architecture
Thread count controls how many workers may process tasks concurrently. Queue contents control which URLs are pending. Crawl coordination controls when URLs are added and when the application considers the crawl complete.
The worker function connects these parts: it reads a URL from the queue, passes it to Spider.crawl_page, records the current worker name, and marks the task complete.
Troubleshooting Worker Pools
The Program Exits While URLs Are Waiting
This usually means daemon workers are present but the main thread ends before pending work is coordinated. Check whether the coordinator calls work_queue.join() or uses another completion mechanism. Keep the main workflow active until the intended tasks finish.
queue.join() Waits Forever
A worker may have retrieved an item without calling task_done(), or an exception may have bypassed the completion call. Verify a one-to-one match between get() and task_done(), and place the latter in a finally block.
No Pages Are Crawled
Confirm that URLs were placed on the queue, the crawl coordinator was invoked, and every thread uses target=work before calling start(). Creating a thread object without starting it does not execute the worker.
Only One Worker Appears in the Output
Check NUMBER_OF_THREADS. There may also be too little queued work, or page processing may be so fast that concurrent activity is not obvious. Test with several URLs and include threading.current_thread().name in logging.
A Worker Stops After a Bad URL
An unhandled exception can escape the worker loop. Catch expected crawl or network exceptions around page processing, log the URL and error, and retain the finally block so task accounting remains correct.
Exam-Relevant Notes
threading.Threadcreates a thread object;start()begins its independent execution.targetnames the function the thread will execute.- Set
daemonbefore callingstart(). queue.get()normally blocks until a task is available.- Every successful
get()requires onetask_done(). queue.join()waits for all unfinished queue tasks to receive completion notifications.- Start workers before the coordinator begins supplying work.
- Daemon threads can abandon outstanding work if the main program exits too early.
For the next implementation step, connect this worker pool to URL discovery, duplicate tracking, request timeouts, logging, and a deliberate shutdown strategy such as sentinel values.