Create workers 2

In this section we will write the code for our workers. We will create worker threads that will be configured to shut down when the main program exits.

Write the following code in the main.py file:

def create_workers(): # the function that will create worker threads
    for _ in range(NUMBER_OF_THREADS): # specifies how many threads will be created
        t = threading.Thread(target=work) # creates the worker and passes it the work function as its job
        t.daemon = True # ensures that the thread dies when the main program exits
        t.start() # starts the thread


 # Do the next job in the queue
def work():
    while True:
        url = queue.get() # gets the next link in the queue
        Spider.crawl_page(threading.current_thread().name, url) # crawls the URL and displays the thread name
        queue.task_done() # specifies that the processing on the task is done

Add the following code at the end of the main.py file to call the functions:

create_workers()
crawl()