VMware ESXi and vSphere Cluster Management

Create Queue and Crawled Files for a Python Web Crawler

Learn how to initialize queue.txt and crawled.txt safely for a file-based Python web crawler, preserve existing state, and verify the setup.

A web crawler needs to remember what work is waiting and what work has already finished. This lesson creates two persistent text files: queue.txt for pending URLs and crawled.txt for processed URLs.

Keeping this state in files allows a crawler to stop and restart without forgetting its progress. Separating pending and completed URLs also helps prevent the crawler from repeatedly processing the same address.

Crawler State Files

queue.txt is a text file containing URLs that have been discovered but have not yet been crawled. It starts with the crawl's homepage, or base_url.

crawled.txt is a text file containing URLs that the crawler has already processed. It starts empty and receives URLs as crawling progresses.

Use a Separate Project Directory

A project directory is a folder dedicated to one crawl target and its state files. For example, a project named GeekUniversity can contain:

GeekUniversity/
├── queue.txt
└── crawled.txt

Using a separate folder for each website keeps crawl data isolated. A second website can have its own project directory without mixing its URLs with the first site's URLs.

The project_name argument identifies this folder. It may be a folder name such as GeekUniversity or a path to a selected project location. The examples assume that the project directory is created before the data files are initialized.

Initialization Inputs

For this example, the inputs are:

project_name = "GeekUniversity"
base_url = "https://geek-university.com"

Build File Paths Safely

Use os.path.join to combine the project directory with each filename:

import os

queue_file = os.path.join(project_name, "queue.txt")
crawled_file = os.path.join(project_name, "crawled.txt")

os.path.join creates a platform-appropriate path. This is safer than manually concatenating strings such as project_name + "/queue.txt", because path separators differ between operating systems.

Create the Project Directory First

The data files cannot be created inside a directory that does not exist. A directory helper can create the project folder and leave it unchanged if it already exists:

import os

def create_project_dir(project_name):
    os.makedirs(project_name, exist_ok=True)

The exist_ok=True option makes repeated directory setup safe. It does not delete files already inside the directory.

Create a Reusable File-Writing Helper

A reusable write_file helper accepts a file path and text data, opens the file in write mode, writes the data, and closes the file.

Write mode, represented by "w", creates a missing file. If the file already exists, it replaces its contents. Therefore, use this mode only after checking that an initial file is absent when existing crawler state must be preserved.

def write_file(path, data):
    with open(path, "w", encoding="utf-8") as file:
        file.write(data)

The with statement is a context manager. It closes the file reliably after writing, including when an exception occurs. This is preferable to opening a file and relying on a later manual close() call.

Create queue.txt and crawled.txt

The initialization function should:

1. Build both file paths with os.path.join.
2. Check whether queue.txt exists.
3. Create the queue only when it is missing, writing base_url as its first URL.
4. Check whether crawled.txt exists.
5. Create an empty crawled file only when it is missing.

def create_data_files(project_name, base_url):
    queue_file = os.path.join(project_name, "queue.txt")
    crawled_file = os.path.join(project_name, "crawled.txt")

    if not os.path.isfile(queue_file):
        write_file(queue_file, base_url + "\n")

    if not os.path.isfile(crawled_file):
        write_file(crawled_file, "")

os.path.isfile checks whether a path points to an existing regular file. The existence checks are important: they prevent a repeated initialization call from erasing the current queue or crawl history.

Run the Initialization

Call the directory function before the data-file function:

create_project_dir("GeekUniversity")
create_data_files(
    "GeekUniversity",
    "https://geek-university.com"
)

The resulting layout should be:

GeekUniversity/
├── queue.txt       # contains https://geek-university.com
└── crawled.txt     # empty

This setup is idempotent: initialization can be run repeatedly without destroying existing state. On a second run, an existing queue.txt is preserved, and an existing crawled.txt keeps its processed URLs.

Verify the Files

After running initialization, confirm the directory and file contents. On a Unix-like shell, you can inspect them with:

ls GeekUniversity
cat GeekUniversity/queue.txt
cat GeekUniversity/crawled.txt

On Windows PowerShell, an equivalent check is:

Get-ChildItem GeekUniversity
Get-Content GeekUniversity\queue.txt
Get-Content GeekUniversity\crawled.txt

You should confirm all of the following:

• The GeekUniversity project folder exists.
• The folder contains queue.txt and crawled.txt.
• queue.txt contains https://geek-university.com.
• crawled.txt contains no URLs.

How the Files Support Crawling

Initialization places the homepage in the pending queue. Later crawling code reads a URL from queue.txt, requests and processes that page, and records the completed URL in crawled.txt.

When the page contains eligible links, later crawler logic can add new URLs to the queue. Before adding or processing a URL, that logic can compare it with the pending and completed records to avoid duplicate work.

Initial state:
queue.txt   = https://geek-university.com
crawled.txt = empty

After processing the homepage:
queue.txt   = newly discovered URLs
crawled.txt = https://geek-university.com

Conceptually, a URL moves from pending work to completed work:

1. Select a URL from queue.txt.
2. Process the URL.
3. Remove it from the pending queue.
4. Record it in crawled.txt.
5. Add newly discovered, eligible URLs to queue.txt.

The two-file design provides a simple foundation for future operations such as adding URLs, deleting processed URLs, reading pending URLs, and restarting a crawl after the program stops.

Troubleshooting

The files are not created

• Call create_project_dir(project_name) before create_data_files(...).
• Check that project_name points to the intended directory.
• Verify that the program has permission to write there.
• Read the Python exception for missing-path or permission details.

queue.txt is empty

• Check that base_url is not an empty string.
• Verify that write_file writes its data argument.
• Check whether an older queue file was intentionally preserved by the existence guard.
• For a genuinely new crawl, use a new project directory or an explicit reset operation rather than accidentally overwriting state.

Existing queue data disappears

This usually happens when the file is opened with "w" every time setup runs. Use os.path.isfile first and write the base URL only when the queue file does not exist. If overwriting is needed, make it a separate, deliberate reset action.

Files appear in the wrong location

• Construct paths with os.path.join(project_name, "queue.txt") and the equivalent crawled path.
• Check that the correct project name was passed.
• Remember that relative paths are based on the program's current working directory.
• Print the paths during testing, or use an absolute project path when the execution location can vary.

Exam- relevant Notes

• queue.txt stores pending URLs; crawled.txt stores processed URLs.
• project_name determines the storage directory.
• base_url becomes the first queue entry.
• Call directory creation before file creation.
• Use os.path.join for portable paths.
• Use os.path.isfile before writing initial state.
• Use a context manager with open(..., "w", encoding="utf-8") to write and close files reliably.
• Preserving existing files makes initialization idempotent and protects restart data.

With these files initialized, the crawler has a persistent starting point: the homepage waits in queue.txt, while crawled.txt is ready to record completed work.

Continue with the crawler queue and crawled-file lesson when you are ready to extend this state-management foundation.