Python Web Crawler

Create a New Python Web Crawler Project

Learn how to initialize a file-based Python web crawler with per-site project folders, shared utilities, queue.txt, and crawled.txt.

Before a crawler fetches or parses its first page, it needs a place to store its work. Project initialization creates a dedicated directory for one target website and prepares the persistent state that later crawler components will use.

This lesson uses a simple file-based design. A shared module named general.py contains reusable functions, while each target website gets its own project directory containing queue.txt and crawled.txt.

Why initialize a crawler project?

A project directory is a folder that stores crawler data for one target website or crawl run. The directory name identifies the crawler project or target site, such as example_site or site_a.

Giving each website its own directory keeps crawl state separate. URLs for one website cannot accidentally become mixed with URLs for another website, and a crawl can be paused and resumed using the files belonging to that project.

Initialization is the first persistent-state step. It should happen before fetching pages, extracting links, or parsing HTML. At this point, the program establishes where discovered and completed URLs will be stored.

Separate reusable code from crawl data

The crawler needs both program code and data that changes during a crawl. Reusable code belongs in modules such as general.py. Per-site data belongs inside the individual project directory.

general.py is a general-purpose Python utility module. It can contain filesystem helpers and, later, functions for reading URL lists, writing URL lists, or managing other shared crawler behavior. Keeping directory creation in this module prevents every crawler component from implementing its own setup logic.

Create a project directory safely

Python's os module provides standard-library interfaces for operating-system tasks, including filesystem operations. Import it before using filesystem functions.

import os

The helper below creates a directory only when the requested path does not already exist:

def create_project_dir(directory):
    if not os.path.exists(directory):
        print(f"Creating project {directory}")
        os.makedirs(directory)

How the function works

1. directory is a function parameter containing the requested project path.
2. os.path.exists(directory) checks whether that path already exists.
3. The body of the if statement runs only when the path is absent.
4. The optional print() call reports that a new project folder is being created.
5. os.makedirs(directory) creates the directory path, including missing parent directories when necessary.

The existence check avoids trying to create a directory that is already present. As a result, running initialization again retains the existing folder instead of unnecessarily attempting duplicate creation.

Example: initialize one site

create_project_dir("example_site")

If example_site does not exist relative to the program's current working directory, the function creates it and reports the creation. If it already exists, the function does nothing.

Running initialization twice

create_project_dir("example_site")
create_project_dir("example_site")

The first call creates the directory. The second call finds that the path exists and skips os.makedirs(). This makes the setup operation safe to repeat during development or when restarting a crawl.

Prepare the crawler's persistent state

After the project directory exists, the crawler can use two text files to represent URL state:

• queue.txt contains discovered URLs that are still waiting to be processed.
• crawled.txt contains URLs that the crawler has already processed.

These files must be inside the individual project directory. For example, the state for example_site belongs at example_site/queue.txt and example_site/crawled.txt, rather than in one global location shared by every website.

The queue is the file-based form of the crawl frontier: the current collection of discovered URLs awaiting crawling. A URL should move from the pending queue state to the completed crawled state after successful processing.

Expected initial project layout

A basic filesystem layout can look like this:

crawler_program/
├── general.py
└── example_site/
    ├── queue.txt
    └── crawled.txt

For two independent sites, the layout could instead contain separate state directories:

crawler_program/
├── general.py
├── site_a/
│   ├── queue.txt
│   └── crawled.txt
└── site_b/
    ├── queue.txt
    └── crawled.txt

Here, general.py is reusable program code. The two project directories are generated crawl data, and each site has independent pending and completed URL lists.

How separate state prevents duplicate crawling

A crawler discovers URLs while processing pages. Every discovered URL should be checked against stored state before it is scheduled. If the URL is already pending in queue.txt or already completed in crawled.txt, the crawler should not add or revisit it unnecessarily.

1. Begin with a seed URL or discover a link on a page.
2. Check whether the URL is already represented in the queue or completed state.
3. If it is new, add it to queue.txt.
4. Select a pending URL and process it.
5. After successful processing, remove it from the pending state and record it in crawled.txt.
6. When new links are found, repeat the state check before scheduling them.

A duplicate crawl is an unnecessary repeat visit to a URL that has already been processed. Keeping pending and completed URLs distinct gives the crawler a durable record that helps avoid this problem, including after the program stops and starts again.

Relative and absolute project paths

A name such as example_site is a relative path. Python resolves it from the program's current working directory, which may differ from the directory containing the script. If the folder appears in an unexpected location, inspect the working directory and choose a deliberate path.

An absolute path identifies the location from the filesystem root. Either style can work, but the choice should be consistent and documented. During development, verify the resulting path before placing important crawl data there.

Troubleshooting initialization

The setup fails when the folder already exists

The likely cause is calling os.makedirs() without first checking the path. Test with os.path.exists() before creation, as shown in the helper above.

Different websites share the same URL files

This happens when state files are stored in one global location. Create one project directory per target site and construct both state-file paths inside that directory.

The same pages are crawled more than once

Completed URLs may not be recorded, or the crawler may not consult its stored state before scheduling work. Maintain queue.txt for pending URLs, maintain crawled.txt for completed URLs, and check both before adding a URL.

The directory appears in an unexpected location

A relative path is resolved from the current working directory. Inspect that directory, use a deliberate project path, and print or otherwise verify the path during development.

Directory creation reports a permission error

The selected parent location is probably not writable by the current user. Choose a writable project location or correct the filesystem permissions.

Next steps

This lesson establishes the folder but does not yet require the state files to be populated. Continue with creating the queue and crawled files, then learn how to add and delete URLs from the file-based state.

For broader context, review what a web crawler is and the overall Python crawler workflow.