Create a web crawler in Python

Create a New Python Web Crawler Project

Learn how to initialize a Python web crawler project, create an idempotent project directory, and prepare queue.txt and crawled.txt for persistent URL tracking.

Before a web crawler can discover and process pages, it needs a reliable place to store its state. In this lesson, you will create a project directory for one target website and prepare the structure that later crawler functions will use.

This setup is the first persistent-storage step in a crawler workflow. Instead of keeping every URL only in memory, the crawler will eventually record pending and completed URLs in text files.

Why Initialize a Crawler Project?

A project directory is a dedicated folder that contains crawler data for one target website. Each target should have its own directory so that URLs and progress from different websites do not become mixed together.

For example, a crawler might use a folder named example-site for one website. Its queue, completed URL list, and later crawler data can all remain inside that folder. A separate target can use a different folder.

Project initialization creates the location where crawler state will be stored. This matters when the program stops, restarts, or is run repeatedly: the crawler can read its previous state instead of starting from nothing each time.

Use a Shared Utility Module

A Python module is a file containing reusable Python code. In this crawler, general.py will be a shared utility module for helper functions used by multiple crawler components.

Place project-creation logic in a function inside general.py, rather than putting filesystem code directly in the main crawler script. The main crawler can then request a project directory without needing to know the details of checking paths or creating folders.

This separation keeps responsibilities clear:

• Filesystem helpers create directories and manage state-file paths.
• Crawling logic discovers, selects, downloads, and processes URLs.
• Shared utilities provide reusable operations to both parts of the program.

Separating these responsibilities makes the crawler easier to test, reuse, and extend.

Create the Project Directory

Python's os module provides standard-library functionality for interacting with the operating system and filesystem. Import it in general.py:

import os

Now define a function that accepts a directory name or path. The function checks whether the requested path already exists. It creates the directory only when the path is absent.

import os


def create_project(directory):
    if not os.path.exists(directory):
        print("Creating project " + directory)
        os.makedirs(directory)

os.path.exists checks whether a filesystem path already exists. If the check returns False, os.makedirs creates the directory path.

The function accepts a parameter named directory, so the same helper works for different target websites:

create_project("example-site")

After this call, the expected layout begins as follows:

example-site/

Make Initialization Safe to Repeat

Setup code may run more than once during development. It may also run after a crawler restart. The existing project directory must be preserved rather than recreated or overwritten.

Because the function checks os.path.exists(directory) before calling os.makedirs(directory), running it repeatedly is safe:

create_project("example-site")
create_project("example-site")

The first call creates the directory and may print a status message. The second call finds that the directory already exists, so it performs no creation action. This behavior is called idempotent setup: initialization can be repeated without damaging or duplicating existing project resources.

Prepare the Crawler State Files

Every project should eventually contain two text files:

• queue.txt is the persistent list of URLs that still need to be processed. This is the file-based crawl queue.
• crawled.txt is the persistent list of URLs that have already been visited. It provides visited URL tracking.

The directory must be created before these files can be created inside it. Later helper functions can create the files and add the starting, or seed, URL to queue.txt.

example-site/
    queue.txt
    crawled.txt

These two lists support both progress tracking and duplicate avoidance. When the crawler finds a new link, it can place the link in queue.txt. After processing the link, the crawler can record it in crawled.txt. Before adding or requesting a URL, the crawler should compare it with the visited list so that the same page is not repeatedly processed.

How This Fits into the Next Setup Steps

1. Create the project directory with create_project.
2. Create and initialize queue.txt and crawled.txt inside that directory.
3. Add the starting URL to the crawl queue.
4. Read pending URLs, process them, and record completed URLs.

The directory-creation function is therefore the foundation for the later state-file helpers. It establishes the location that all URL-processing operations will use.

Practical Example: Initialize One Domain

Put the reusable helper in general.py:

import os


def create_project(directory):
    if not os.path.exists(directory):
        print("Creating project " + directory)
        os.makedirs(directory)

A separate startup script can import and call the helper:

from general import create_project

create_project("example-site")

After the first run, create the state files as the next setup operation. The exact file-writing helper belongs to the subsequent crawler setup step, but its paths should be built from the project directory:

project = "example-site"
queue_file = os.path.join(project, "queue.txt")
crawled_file = os.path.join(project, "crawled.txt")

Constructing both paths from the same project directory helps keep crawler data together and reduces the chance of writing files to unrelated locations.

Troubleshooting

The program says that the directory already exists

The likely cause is that directory creation is attempted without checking the path first. Use os.path.exists before os.makedirs, as shown in the helper. This preserves an existing project.

The crawler starts with no URLs

The queue file may not have been created or seeded. After creating the project directory, create queue.txt and add the crawler's starting URL.

Pages are crawled more than once

The crawler may not be recording or checking visited URLs. Maintain crawled.txt and make URL-selection logic exclude entries already recorded there.

Files appear in the wrong location

Relative paths are interpreted from the program's current working directory. Use a consistent project path and construct both state-file paths from that directory with os.path.join.

Directory creation fails with a permissions error

The parent location may not be writable by the current user or process. Choose a writable output location or correct the relevant filesystem permissions.

Key Takeaways

• Give each target website its own project directory.
• Use general.py for reusable filesystem helpers.
• Import the os module for filesystem operations.
• Check os.path.exists before calling os.makedirs.
• Make initialization idempotent so repeated runs preserve existing data.
• Store pending URLs in queue.txt and visited URLs in crawled.txt.
• Create the directory before creating and seeding the state files.

Continue with the crawler project setup guide when you are ready to create the queue and crawled files and add the seed URL.