Python Web Crawler

Add and Delete URLs in a Python Web Crawler

Learn how to append URLs to crawler queue files, clear file contents safely, and manage queue and crawled URL state with Python.

What queue and crawled URL files do

A file-based web crawler commonly keeps two separate files:

The URL queue is persistent storage for URLs waiting to be visited. Crawled URLs are URLs that the crawler has already processed. Keeping these files updated lets a crawler resume work and avoid losing its state.

This lesson focuses on low-level file operations: adding a record to a file and clearing file contents. It does not implement crawling, link extraction, URL normalization, or duplicate detection. For broader crawler concepts, see What Is A Web Crawler and Create The Crawler.

Appending a URL to a file

A file path is the location and filename supplied to a file operation, such as queue.txt or data/crawled.txt. Create a helper that receives a path and a data value:

def append_to_file(path, data):
    with open(path, 'a') as file:
        file.write(data + '\n')

Here, data can be a URL string such as https://example.com/products. The open() function returns a file object, named file in this example. The file object provides the write() operation.

The mode 'a' means append mode. It preserves existing content and writes new content at the end. The expression data + '\n' adds a newline character after the URL, so every URL occupies its own line.

Adding one newly discovered link

Suppose queue.txt already contains two URLs. Calling the helper once adds a third URL without replacing the first two:

append_to_file('queue.txt', 'https://example.com/new-page')

The resulting file has one record per line:

https://example.com/
https://example.com/about
https://example.com/new-page

To record multiple discovered links, call the helper once for each URL:

new_urls = [
    'https://example.com/blog',
    'https://example.com/contact',
    'https://example.com/help'
]

for url in new_urls:
    append_to_file('queue.txt', url)

One-URL-per-line formatting makes later reading straightforward: a program can read the file line by line, remove line endings, and convert the results into a list or set of URLs.

Using a context manager for safe file handling

The with statement creates a context manager block. In this pattern, Python opens the file before the block and closes it automatically when the block ends, including when an error occurs inside the block.

with open(path, 'a') as file:
    file.write(data + '\n')

file is the file object available inside the block. It is the object used to write data. The automatic cleanup is safer than manually opening a file and remembering to close it on every possible code path.

Clearing a file's contents

To remove the existing contents while retaining the file itself, define a clearing helper:

def delete_file_contents(path):
    with open(path, 'w'):
        pass

The mode 'w' means write mode. When an existing file is opened in this mode, Python truncates it: the contents are reduced to zero length. This is different from deleting the file itself. The file remains present, but it is empty.

Write mode also creates the file if the target path does not already exist, provided the parent directory exists. Therefore, this helper can prepare a state file before a later operation writes new records.

Why the block contains pass

A with statement requires an indented block. pass is a no-operation statement: it performs no action. It is useful here because opening the file in write mode already performs the desired side effect, which is truncation. No write call is needed.

def delete_file_contents(path):
    with open(path, 'w'):
        pass

Do not confuse clearing contents with removing a filesystem entry. This helper does not delete the filename or its directory entry; it only empties the file.

Comparing append and clear operations

Start with a file containing several URL lines, append one URL, and then clear the file:

queue_path = 'queue.txt'

append_to_file(queue_path, 'https://example.com/first')
append_to_file(queue_path, 'https://example.com/second')
append_to_file(queue_path, 'https://example.com/third')

# Existing lines remain, and this line is added at the end.
append_to_file(queue_path, 'https://example.com/fourth')

# The file still exists, but all contents are removed.
delete_file_contents(queue_path)

Before the clear operation, the file contains four lines. After it, the file has zero contents. To verify the result, inspect its size or read it:

delete_file_contents('test-queue.txt')

with open('test-queue.txt', 'r') as file:
    contents = file.read()

print(contents == '')  # True

How these helpers fit into crawler state

Appending is appropriate when the crawler discovers a new URL and needs to record it in the queue, or when it adds a URL to persistent crawler state. A crawler may also append a processed URL to a crawled-URL file after visiting it.

Clearing is appropriate when resetting test data, starting a new crawl, or preparing a file that will be completely rewritten. Always select the exact state file intentionally. Clearing the queue can discard all pending work, while clearing the crawled file can remove the history used to prevent revisits.

Troubleshooting

New URLs replace previous URLs

Cause: The file was opened with 'w' instead of 'a'.

Fix: Use append mode for incremental URL storage. Reserve write mode for deliberate resets or complete rewrites.

URLs run together on one line

Cause: The write operation did not add a newline character.

Fix: Write data + '\n' for every URL. This preserves the one-record-per-line format.

The file is unexpectedly empty

Cause: The clear helper or another write-mode operation ran against the file.

Fix: Check the target path and use append mode whenever existing crawler state must be preserved.

Python reports that an indented block is required

Cause: A with block was left empty.

Fix: Add pass when opening the file itself is the intended operation.

File changes are not reliably saved or the file remains open

Cause: File handling was performed without a context manager.

Fix: Use with open(...) as file. Python closes the file automatically when the block ends.

Exam-relevant summary

• 'a' is append mode: it preserves existing content and adds new content at the end.
• 'w' is write mode: it truncates an existing file to zero length and can create a missing file.
• A newline character, written as \n, stores each URL as a separate line.
• A context manager uses with to close the file automatically.
• pass fills an intentionally empty Python block without performing an operation.
• Clearing a file removes its contents but does not delete the file itself.
• Always distinguish the queue file from the crawled-URL file before clearing either one.

For the surrounding project setup, review Create A New Project and Create Queue And Crawled Files. The next file-oriented step is commonly reading URL lines and converting them into Python collections.