Create workers 2
In this section we will write the code for our workers. We will create worker threads that will be configured to shut down when the main program exits.
Write the following code in the main.py file:
def create_workers(): # the function that will create worker threads for _ in range(NUMBER_OF_THREADS): # specifies how many threads will be created t = threading.Thread(target=work) # creates the worker and passes it the work function as its job t.daemon = True # ensures that the thread dies when the main program exits t.start() # starts the thread # Do the next job in the queue def work(): while True: url = queue.get() # gets the next link in the queue Spider.crawl_page(threading.current_thread().name, url) # crawls the URL and displays the thread name queue.task_done() # specifies that the processing on the task is done
Add the following code at the end of the main.py file to call the functions: