Adding and Clearing URL Records in a Python Web Crawler

Learn how to append URLs to a crawler queue file, clear tracking files with write mode, and use Python context managers safely.

A web crawler needs to remember which pages it still needs to visit and which pages it has already processed. A simple crawler can store this state in ordinary text files, with one URL on each line.

This lesson shows how to append newly discovered URLs, clear existing crawler files, and integrate both operations into a basic crawler workflow.

URL Tracking Files in a Crawler

A URL is a web address that a crawler can queue, visit, and record. A file-based crawler commonly uses two tracking files:

• Queue file: contains URLs that remain to be crawled.
• Crawled file: contains URLs that have already been processed.

For example, a queue file might contain:

https://example.com/
https://example.com/about
https://example.com/contact

After the crawler processes a URL, a later component can remove or mark it as handled and record it in the crawled file:

https://example.com/
https://example.com/about

Keeping these lists on disk gives a small crawler persistent state. If the program stops, the next run can read the files instead of losing all progress when its in-memory lists disappear. File updates therefore support continuity across program runs.

Queue and crawled files compared

Later crawler components can read these files and convert their lines into in-memory collections such as lists or sets. A set is especially useful when duplicate URLs must be removed.

Appending a URL to a Text File

Append mode is Python's file mode 'a'. It writes new data at the end of a file without removing the existing contents. This is the appropriate mode for adding one newly discovered URL to a queue or crawled file.

Create a reusable helper that accepts a file path and a text value:

def append_to_file(path, data):
    with open(path, 'a') as file:
        file.write(data + '\n')

The path parameter identifies the target file, and data is the URL or other text record to add. The object returned by open is assigned to file. That file object provides the write method used to send text to the file.

The expression data + '\n' adds a newline, which is the line-ending character that makes each URL occupy its own record.

Recording a discovered page

queue_path = 'queue.txt'
discovered_url = 'https://example.com/products'

append_to_file(queue_path, discovered_url)

If queue.txt already contains two URLs, the new URL appears after them rather than replacing them:

https://example.com/
https://example.com/about
https://example.com/products

You can append several records one at a time:

append_to_file('queue.txt', 'https://example.com/')
append_to_file('queue.txt', 'https://example.com/about')
append_to_file('queue.txt', 'https://example.com/contact')

Inspecting the text file should show one URL per line. In a production crawler, normalize and validate discovered links before recording them, and use a separate deduplication strategy when necessary.

Using Context Managers for Files

A context manager is the with-based pattern that manages a resource such as an open file. In this example:

with open(path, 'a') as file:
    file.write(data + '\n')

Python opens the file before entering the indented block and closes it automatically when the block ends. The file is also closed if an error occurs inside the block. This prevents file descriptors from remaining open and makes file handling safer than manually calling close() in every code path.

The name file is a variable referring to the file object returned by open. You can choose another variable name, but the object is still the one used for operations such as write.

Clearing an Existing Crawler Data File

Sometimes a crawler must start with a clean queue and crawled history. Define a helper that opens a file in write mode, represented by 'w':

def clear_file(path):
    with open(path, 'w') as file:
        pass

Opening an existing file in write mode immediately truncates it: file truncation removes its existing contents. Because the block does not write replacement text, the result is an empty file.

Write mode also creates the file if it does not already exist, provided that its parent directory already exists.

Why use pass?

pass is a Python statement that performs no action. It supplies a valid body for a function or block that would otherwise be empty. In clear_file, opening the file in write mode performs the reset as a side effect, so no explicit write call is needed.

The context manager still closes the file after the block:

def clear_file(path):
    with open(path, 'w') as file:
        pass  # Opening in 'w' already emptied the file

Append mode and write mode

Initializing and Restarting Crawler State

A fresh crawl commonly resets both tracking files and then adds the starting URL to the queue. Reset first, seed second:

queue_path = 'queue.txt'
crawled_path = 'crawled.txt'
start_url = 'https://example.com/'

clear_file(queue_path)
clear_file(crawled_path)
append_to_file(queue_path, start_url)

After initialization, the queue contains the seed URL and the crawled file is empty. As the crawler discovers links, it appends them to the queue. After processing a URL, another part of the crawler can append that URL to the crawled file.

def record_discovered_link(queue_path, url):
    append_to_file(queue_path, url)


def record_processed_url(crawled_path, url):
    append_to_file(crawled_path, url)

Do not clear the files on every ordinary loop iteration. Reset them only during explicit initialization or an intentional restart. Otherwise, the crawler may lose its seed URL or previously recorded progress.

Append Versus Overwrite: Practical Demonstration

Suppose a file initially contains:

https://example.com/first
https://example.com/second

Appending a third URL preserves the earlier records:

append_to_file('urls.txt', 'https://example.com/third')

https://example.com/first
https://example.com/second
https://example.com/third

Opening the same file in write mode instead removes the previous records before any new content is written:

with open('urls.txt', 'w') as file:
    file.write('https://example.com/replacement\n')

https://example.com/replacement

This difference explains why 'a' is used for incremental tracking and 'w' is used for deliberate resets or complete replacement.

Troubleshooting URL File Operations

URLs replace earlier URLs

Likely cause: The append helper uses 'w' instead of 'a'.

Resolution: Use append mode for incremental additions. Reserve write mode for intentional resets.

Multiple URLs appear on one line

Likely cause: The write operation does not add a line ending.

Resolution: Write data + '\n' for each record and check that the input does not contain unintended formatting.

The crawler starts with no saved URLs

Likely cause: The clearing helper ran after the queue was seeded, or it was called unintentionally.

Resolution: Clear files first, then append the seed URL. Call reset functions only during explicit initialization or restart workflows.

A file cannot be opened or updated

Likely causes: The path is incorrect, the parent directory does not exist, or the process lacks filesystem permissions.

Resolution: Check the supplied path, create required parent directories during setup, and verify that the running process can read and write the location.

An empty clear function causes a syntax error

Likely cause: A function or with block has no valid statement.

Resolution: Add pass as a deliberate no-operation statement. The file-opening side effect still performs the reset.

Exam-Relevant Notes

For the next crawler feature, connect these helpers to code that reads URL lines into an in-memory collection and prevents duplicate work. This page is available at Adding and Clearing URL Records.