VMware ESXi and vSphere Cluster Management

Using Python Sets to Manage Crawler URLs

Learn how Python sets help a web crawler remove duplicate URLs, track pending and crawled pages, and persist URL collections in text files.

Why a crawler needs sets

A web crawler discovers URLs while it processes pages. Some URLs are waiting to be processed, while others have already been processed. Keeping these groups separate makes the crawler's state clear.

• Queue set: URLs that remain to be crawled.
• Crawled set: URLs that have already been processed.

A crawler may discover the same URL from several pages. If it schedules every discovery, it can request the same page repeatedly. Sets prevent this duplicate work because each URL can appear only once in a set.

For example, adding the same URL to a queue set several times still leaves one entry. After processing a URL, the crawler can remove it from the queue set and add it to the crawled set. This helps avoid repeated requests and unnecessary processing.

What is a Python set?

A set is a collection of unique, hashable values. Sets do not provide meaningful positional order, so you should not treat a set like a list with a first or last item.

URL strings are suitable set elements because strings are hashable. A hashable value can be used as a set member and can be compared efficiently by Python.

urls = {
    "https://example.com/",
    "https://example.com/about",
    "https://example.com/"
}

print(urls)
print(len(urls))

The repeated home-page URL is automatically collapsed into one item, so the set contains two URLs.

Sets compared with lists and tuples

Use a set when membership and uniqueness matter. Use a list when order matters. A tuple is generally used for a fixed sequence rather than a collection that the crawler continually updates.

Creating an empty set

Call set() to create an empty set:

queue = set()
crawled = set()

Do not use {} for this purpose. An empty pair of braces creates an empty dictionary:

value = {}
print(type(value))  # <class 'dict'>

Braces can create a non-empty set when they contain values, but set() is the unambiguous syntax for an empty set.

Loading URL lines into a set

A state file can store one URL per line. The loading helper below accepts a file name, reads the file as text, removes line-ending characters, and adds each URL to a set.

def load_urls(file_name):
    links = set()

    with open(file_name, "r", encoding="utf-8") as file:
        for line in file:
            url = line.rstrip("\r\n")
            links.add(url)

    return links

The with open(...) statement is a context manager. It closes the file automatically after the indented block finishes, including when an error occurs. The "r" argument selects text read mode.

Each line normally ends with a newline character. Without removing it, the stored value might be "https://example.com/\n" rather than "https://example.com/". The expression rstrip("\r\n") removes line-ending characters from the right side while avoiding removal of other valid trailing URL characters.

The call to set.add() inserts one item. If the input file contains duplicate lines, the set retains only one copy.

# queue.txt
https://example.com/
https://example.com/about
https://example.com/

queue = load_urls("queue.txt")
print(len(queue))  # 2

If blank lines are not valid crawler entries, the helper can skip them explicitly:

def load_urls(file_name):
    links = set()

    with open(file_name, "r", encoding="utf-8") as file:
        for line in file:
            url = line.rstrip("\r\n")
            if url:
                links.add(url)

    return links

Writing a URL set to a file

The saving helper accepts a set of links and an output file name. It writes one URL per line.

def save_urls(links, file_name):
    with open(file_name, "w", encoding="utf-8") as file:
        for url in sorted(links):
            file.write(url + "\n")

The "w" argument selects write mode. It creates the file if necessary and replaces its previous contents. Therefore, the set passed to the function should contain the complete state that you intend to save.

sorted() produces a sorted sequence from the set. Sorting is not required for correctness, but it gives the file stable, readable ordering. Stable output makes files easier to inspect, compare, and debug between runs.

queue = {
    "https://example.com/zebra",
    "https://example.com/apple"
}

save_urls(queue, "queue.txt")

The resulting file is:

https://example.com/apple
https://example.com/zebra

Queue and crawled state files

A simple file-backed crawler can use two text files. Each file contains one URL per line.

Load both collections when the crawler starts:

queue = load_urls("queue.txt")
crawled = load_urls("crawled.txt")

The crawler can then select a pending URL, process it, and update both collections:

if queue:
    url = next(iter(queue))
    queue.remove(url)

    # Fetch and process the page at url here.
    crawled.add(url)

next(iter(queue)) selects an available set member. It does not promise a particular URL order. If the crawler needs a deliberate ordering policy, a separate queue structure may be more appropriate; the set remains useful for fast duplicate checks.

After updates, save the complete collections:

save_urls(queue, "queue.txt")
save_urls(crawled, "crawled.txt")

File helper responsibilities

Updating sets while crawling

In memory, crawler state can be updated with normal set operations:

queue.remove(url)
crawled.add(url)

remove() raises a KeyError if the item is absent. If absence is acceptable, discard() avoids that exception:

queue.discard(url)
crawled.add(url)

When a newly discovered URL should be scheduled only if it has not already been handled, check both collections:

if discovered_url not in queue and discovered_url not in crawled:
    queue.add(discovered_url)

This check prevents a URL from being scheduled when it is already waiting or has already been processed. It assumes that URLs have been normalized consistently; two text strings that differ slightly may still represent the same web address.

Periodic persistence

Persistence means saving in-memory state so it can be reused later. Keeping queue and crawled data in sets makes updates efficient, but changes exist only in memory until they are written to disk.

Saving after every individual set update provides a small recovery window, but frequent file writes can reduce performance. Saving after a batch of URLs is often faster, but more recent work can be lost if the program stops unexpectedly.

queue = load_urls("queue.txt")
crawled = load_urls("crawled.txt")

for count in range(100):
    if not queue:
        break

    url = next(iter(queue))
    queue.remove(url)

    # Process url here.
    crawled.add(url)

    if (count + 1) % 10 == 0:
        save_urls(queue, "queue.txt")
        save_urls(crawled, "crawled.txt")

# Final checkpoint for work since the last batch save.
save_urls(queue, "queue.txt")
save_urls(crawled, "crawled.txt")

Complete file-backed example

The following example shows the complete load, update, and save workflow without implementing HTTP fetching.

def load_urls(file_name):
    links = set()

    with open(file_name, "r", encoding="utf-8") as file:
        for line in file:
            url = line.rstrip("\r\n")
            if url:
                links.add(url)

    return links


def save_urls(links, file_name):
    with open(file_name, "w", encoding="utf-8") as file:
        for url in sorted(links):
            file.write(url + "\n")


queue = load_urls("queue.txt")
crawled = load_urls("crawled.txt")

if queue:
    current_url = next(iter(queue))
    queue.remove(current_url)

    # The crawler would fetch and process current_url here.
    crawled.add(current_url)

save_urls(queue, "queue.txt")
save_urls(crawled, "crawled.txt")

Troubleshooting

Loaded URLs do not match expected values

The line-ending characters may still be attached to the strings. Remove them before calling add(), preferably with line.rstrip("\r\n").

Output order changes between runs

Sets are not intended to provide a meaningful traversal order. Pass the set to sorted() before writing when deterministic output is needed.

Previously saved URLs disappear

Write mode replaces the destination file. Make sure the in-memory set contains the complete intended state before calling save_urls(). An append-based design is possible, but it must match the crawler's state-management strategy.

Duplicate lines remain in the text file

Loading data into a set removes duplicates in memory, but the original file is unchanged until it is rewritten. Load the file and save the resulting set back to the same state file.

An empty collection behaves like a dictionary

Use set(), not {}, to create an empty set.

Crawler state is lost after interruption

Updates that were not persisted existed only in memory. Add deliberate checkpoints, such as after each page or after a batch of pages, and perform a final save when processing ends.

Exam-relevant points

• A set stores unique hashable values and has no guaranteed positional order.
• URL strings are hashable and can be set elements.
• set() creates an empty set; {} creates an empty dictionary.
• set.add(value) inserts one value and ignores an already-present duplicate.
• Use a context manager such as with open(...) so files close automatically.
• Remove newline characters when converting file lines into URL strings.
• Use sorted() before saving if stable file order matters.
• Write mode replaces existing file contents.
• Keep pending and completed URLs in separate collections so crawler state is easy to manage.

For the next crawler-state lesson, see creating and managing crawler URL sets.