Add links to queue

After we gather the links from a webpage, we need to add them to the queue so they can be crawled as well. In this section we will write a function that will add new links to the queue.

The code that goes in the file, under the Spider class:

    def add_links_to_queue(links): # the function that will take a set of links and add them to the waiting list
        for url in links: # loops through the set
            if (url in Spider.queue) or (url in Spider.crawled): # checks if links are already in the waiting or the crawled list
            if Spider.domain_name != get_domain_name(url): # checks if the domain name is present in the URL.
#This ensures that the crawler will crawl only pages on the targeted website, and not the external links present on the website.
            Spider.queue.add(url) # adds link to the waiting list

    def update_files(): # updates the files
        set_to_file(Spider.queue, Spider.queue_file)
        set_to_file(Spider.crawled, Spider.crawled_file)
Geek University 2022