Boot the crawler

We will start our crawler by defining and calling the boot() method. The first crawler we start will create the project directory and the two data files. The code that needs to be added to the file, under the Spider class:

    @staticmethod  # defines the method as static
    def boot():
        create_project_dir(Spider.project_name)  # creates the project folder using the name provided by the user
                          Spider.base_url)  # creates data files and add the homepage to the queue file
        Spider.queue = file_to_set(
            Spider.queue_file)  # converts the links from the queue file to a set for faster operation
        Spider.crawled = file_to_set(Spider.crawled_file)  # converts the links from the crawled file to a set
Geek University 2022