Python Web Crawler

Initialize Queue and Crawled URL Files for a Python Web Crawler

Learn to initialize persistent queue.txt and crawled.txt files for each Python web-crawler project without overwriting existing crawl state.

A web crawler needs to remember what work is waiting and what work has already been completed. This information is called crawl state. In this lesson, you will create two persistent text files for each crawler project: queue.txt and crawled.txt.

The setup function will accept a project name and a base URL, create portable file paths, initialize missing files, and preserve existing data when run again.

Why a crawler needs crawl-state files

A crawler usually discovers URLs one page at a time. Some URLs are waiting to be processed, while others have already been visited. Storing these lists in files allows the crawler to retain its progress between processing steps and between program runs.

• queue.txt: a list of URLs that have been found but have not yet been crawled.
• crawled.txt: a list of URLs that have already been processed by the crawler.

Keeping the lists separate supports orderly traversal. The crawler takes a URL from the queue, processes it, and records it in the crawled list. Later, the crawler can avoid adding or processing URLs that are already complete.

Per-site crawler project structure

Each crawl target should have its own directory. The project name is the directory name used to isolate data for one target website. For example, if the project name is ExampleSite, the initial layout is:

ExampleSite/
├── queue.txt
└── crawled.txt

The two files belong inside the project directory, not in the program's general working directory. This keeps the state for different websites separate.

Initialization function and its contract

Define create_data_files(project_name, base_url). Its parameters have these meanings:

• project_name is the folder containing data for one crawl.
• base_url is the starting homepage URL supplied by the user. This is also called the base URL.

For a new project, the function must place the base URL in queue.txt and leave crawled.txt empty. If either file already exists, the function must leave it unchanged.

This behavior is called idempotent initialization: setup can be run repeatedly without destroying existing crawl state.

Build portable file paths

Use os.path.join to combine the project directory with each filename:

import os

queue_file = os.path.join(project_name, 'queue.txt')
crawled_file = os.path.join(project_name, 'crawled.txt')

os.path.join constructs paths using the separator required by the operating system. Avoid manually writing paths such as project_name + '/' + 'queue.txt', because manually assembled separators are less portable.

Write file contents safely

A small helper can receive a path and text data, open the file in write mode, write the data, and close the file:

def write_file(path, data):
    with open(path, 'w', encoding='utf-8') as file:
        file.write(data)

The with statement creates a context manager. It closes the file automatically after the block finishes, including when an exception interrupts the write.

The mode 'w' means write mode. It creates a missing file, but it truncates existing contents. Therefore, this helper must only be called for initialization after checking whether the file already exists.

Implement idempotent data-file initialization

Use os.path.isfile to check each expected file independently. This utility returns true when the path refers to an existing regular file.

import os


def write_file(path, data):
    with open(path, 'w', encoding='utf-8') as file:
        file.write(data)


def create_data_files(project_name, base_url):
    queue_file = os.path.join(project_name, 'queue.txt')
    crawled_file = os.path.join(project_name, 'crawled.txt')

    if not os.path.isfile(queue_file):
        write_file(queue_file, base_url + '\n')

    if not os.path.isfile(crawled_file):
        write_file(crawled_file, '')

The newline after the base URL gives the queue its normal line-oriented format. Each later URL can occupy its own line.

Why each existence check matters

Checking only the directory is not enough. One file might exist while the other is missing, so each file needs its own check.

Call directory creation before file initialization

The project directory must exist before Python can create files inside it. Call the earlier directory function first, then initialize the data files.

create_project_dir('ExampleSite')
create_data_files('ExampleSite', 'https://www.example.com')

A complete example, including a simple directory function, looks like this:

import os


def create_project_dir(project_name):
    if not os.path.exists(project_name):
        os.makedirs(project_name)


def write_file(path, data):
    with open(path, 'w', encoding='utf-8') as file:
        file.write(data)


def create_data_files(project_name, base_url):
    queue_file = os.path.join(project_name, 'queue.txt')
    crawled_file = os.path.join(project_name, 'crawled.txt')

    if not os.path.isfile(queue_file):
        write_file(queue_file, base_url + '\n')

    if not os.path.isfile(crawled_file):
        write_file(crawled_file, '')


create_project_dir('ExampleSite')
create_data_files('ExampleSite', 'https://www.example.com')

Verification run

After running the example, verify that the directory contains both files:

ExampleSite/
├── queue.txt
└── crawled.txt

The expected contents are:

queue.txt    https://www.example.com
crawled.txt  (empty)

You can inspect the files with Python:

with open('ExampleSite/queue.txt', encoding='utf-8') as file:
    print(repr(file.read()))

with open('ExampleSite/crawled.txt', encoding='utf-8') as file:
    print(repr(file.read()))

The output should show the starting URL followed by a newline for the queue, and an empty string for the crawled file:

'https://www.example.com\n'
''

Now place additional test data in both files and run initialization again:

create_data_files('ExampleSite', 'https://www.example.com')

Because both files already exist, neither is recreated or overwritten. Existing queued URLs remain in queue.txt, and existing processed URLs remain in crawled.txt.

How the files fit into the crawler workflow

Initialization seeds the first pending URL. A typical later workflow is:

1. Read a URL from queue.txt.
2. Fetch and process that URL.
3. Discover links and add eligible new URLs to the queue.
4. Record the processed URL in crawled.txt.
5. Repeat until no pending URLs remain or the crawl is paused.

The initialization lesson creates the storage foundation; it does not yet implement URL removal, discovery, deduplication, or same-domain filtering. Those operations can be added in later crawler components, such as adding and deleting URLs and creating the crawler.

Troubleshooting

Files appear outside the project directory

Likely cause: the code used bare filenames or assembled separators incorrectly.

Fix: construct both paths explicitly:

queue_file = os.path.join(project_name, 'queue.txt')
crawled_file = os.path.join(project_name, 'crawled.txt')

The starting URL disappears on a second run

Likely cause: queue.txt is opened in write mode every time.

Fix: call write_file only when os.path.isfile(queue_file) is false. Write mode overwrites existing data.

FileNotFoundError occurs

Likely cause: the project directory was not created first.

Fix: call create_project_dir(project_name) before create_data_files(project_name, base_url).

The crawler repeats pages

Likely cause: completed URLs are not being tracked separately, or crawled.txt was never initialized.

Fix: ensure crawled.txt exists from the beginning, then use it as the record of processed URLs when implementing the crawl loop.

Files remain open after writing

Likely cause: manual open and close calls were interrupted or omitted.

Fix: use a context manager:

with open(path, 'w', encoding='utf-8') as file:
    file.write(data)

Exam-relevant points

• queue.txt contains pending URLs; crawled.txt contains processed URLs.
• The expected paths are project_name/queue.txt and project_name/crawled.txt.
• Use os.path.join instead of manually concatenating path separators.
• Use os.path.isfile before initializing each file.
• The initial queue contains the base URL, while the initial crawled file is empty.
• Write mode creates or truncates a file, so existence checks protect persistent state.
• Create the project directory before creating files inside it.
• Use with open(...) so the file closes safely.

Summary

The initialization contract is simple: create a directory for one crawl target, create queue.txt with the base URL if it is missing, and create an empty crawled.txt if it is missing. Using os.path.join, os.path.isfile, and a context-managed write helper makes the setup portable, safe, and idempotent.