All posts by Tuna Peyo

Let’s now briefly discuss how our spider will actually work. The spider will grab a link that needs to be …

In this section we will start writing code for the actual link collection. We will create a new file called …

A set in Python is a collection type that contains an unordered collection of unique and immutable objects. Unlike lists …

Once we create queue and crawled files, we need to add data to them. We can do this by using …

We will create two files inside the project folder for each website we crawl: queue file – this file will …

We will start with creating a function that will create a folder for each new website. The function will be …