VMware ESXi and vSphere Cluster Management

Parse HTML Links with Python HTMLParser

Learn to build a Python HTMLParser LinkFinder that extracts, resolves, deduplicates, and filters anchor URLs for a web crawler.

A web crawler usually works in stages: it downloads a page, inspects the returned HTML, collects links, and schedules eligible URLs for later downloads. HTML is markup used to structure web pages. The process of finding URLs referenced by anchor elements is called link collection.

Link extraction is only one crawler component. A typical workflow is:

1. Fetch a page and obtain its page source, meaning the HTML text retrieved for that page.
2. Parse the HTML and inspect its anchor tags.
3. Resolve relative references into complete URLs.
4. Remove duplicates and apply crawl-scope rules.
5. Send eligible URLs to a crawl queue.

What HTMLParser does

HTMLParser is a Python standard-library class that processes HTML markup and calls handler methods when it encounters markup events. A callback method is a method called automatically by a framework or parser when a particular event occurs.

You create a custom parser by making a subclass of HTMLParser. The subclass inherits the parser's behavior and customizes callbacks such as handle_starttag, which runs when the parser reads an opening tag.

from html.parser import HTMLParser
from urllib import parse

The super().__init__() call initializes the parent parser. Omitting it can leave the inherited parser state uninitialized.

HTML anchors and attributes

An anchor element is represented by the HTML a tag. Its href attribute contains the destination URL:

<a href="#" class="nav" target="_blank">About</a>

HTML elements can have many attributes, which are name-value properties attached to an element. Common anchor attributes include:

The attrs argument passed to handle_starttag(tag, attrs) is a sequence of attribute-name and attribute-value pairs. Inspect each pair individually rather than assuming that href is the first or only attribute.

Build a LinkFinder class

Create a module named link_finder.py. The parser object stores its own state: the base URL, the current page URL, and the discovered links. Encapsulating state in an object avoids global variables, makes simultaneous parser instances safer, and lets each page have an independent result set.

from html.parser import HTMLParser
from urllib import parse


class LinkFinder(HTMLParser):
    def __init__(self, base_url, page_url):
        super().__init__()
        self.base_url = base_url
        self.page_url = page_url
        self.links = set()

    def handle_starttag(self, tag, attrs):
        if tag.lower() != "a":
            return

        for name, value in attrs:
            if name.lower() != "href":
                continue
            if value is None:
                continue

            href = value.strip()
            if not href:
                continue

            resolved = parse.urljoin(self.page_url, href)
            self.links.add(resolved)

    def page_links(self):
        return self.links

The callback first checks that the tag is an anchor. It then loops through all attributes, selects only the attribute named href, safely ignores missing or empty values, resolves the reference against the current page URL, and inserts the result into a set.

Why use a set?

A set is a Python collection that stores unique values. Navigation bars, footers, and repeated page references can contain the same destination several times. Calling set.add() keeps one copy, reducing repeated crawl work. A set does not guarantee output ordering, so do not rely on the order of values returned by page_links().

Resolve relative and absolute URLs

An absolute URL includes a scheme and host, such as https://example.test/about. A relative URL is interpreted in relation to a base page. urllib.parse.urljoin performs this resolution.

In this class, page_url is the exact page currently being parsed, so it is the correct resolution base. base_url represents the site's configured starting point and is useful later for crawl-scope checks, such as same-domain filtering. These values can be the same for a first page but should not be confused when crawling nested pages.

URL joining normalizes a reference against a base URL; it does not decide whether the result is inside your crawl scope. A link to another host can be resolved correctly and still be rejected by crawler policy.

Feed HTML into the parser

Downloading and parsing are separate responsibilities. An HTTP client or downloader obtains the HTML; HTMLParser receives that text through its feed method.

from link_finder import LinkFinder

base_url = "https://example.test"
page_url = "https://example.test/articles/python/page.html"
html_content = """
<a href="#" class="nav">About</a>
<a target="_blank" href="#">Guide</a>
<a href="#">About again</a>
<a class="button">No destination</a>
"""

finder = LinkFinder(base_url, page_url)
finder.feed(html_content)
links = finder.page_links()

print(links)

The result is a unique set containing https://example.test/about and https://docs.example.test/guide. The repeated /about anchor appears once, and the anchor without href contributes nothing. The class and target attributes do not affect extraction.

For a parser instance used for one page, retrieve its links after feeding the complete source. Call close() when the parser lifecycle is complete, especially if the application feeds data incrementally. Create a new instance for a new page, or explicitly reset all application state before reusing an object.

Filter candidates before scheduling

The parser extracts candidates. Separate crawler logic should decide whether each candidate is eligible to visit. Common non-page references include fragment-only links, mailto: email links, tel: telephone links, javascript: URLs, and data: URLs.

A URL fragment is the portion after #, usually identifying a location within an already downloaded document. Crawlers commonly remove fragments before deduplication because /guide#intro and /guide#examples normally refer to the same page resource. Fragment-only links may also be skipped entirely because they do not discover a new page.

def crawlable_url(href, base_url, page_url):
    value = href.strip()
    lowered = value.lower()

    if not value or lowered.startswith(("#", "mailto:", "tel:",
                                        "javascript:", "data:")):
        return None

    resolved = parse.urljoin(page_url, value)
    clean_url, _fragment = parse.urldefrag(resolved)
    scheme = parse.urlparse(clean_url).scheme.lower()

    if scheme not in {"http", "https"}:
        return None

    return clean_url

This policy function is best applied while the parser examines each href, or in a separate candidate-processing layer that retains the original href values. It deliberately does not enforce same-domain scope. That policy can use base_url later:

def same_domain(url, base_url):
    return parse.urlparse(url).netloc == parse.urlparse(base_url).netloc

Filtering example

candidate_hrefs = [
    "#section-two",
    "mailto:help@example.test",
    "tel:+15551234567",
    "javascript:void(0)",
    "/documentation",
]

eligible = {
    url
    for href in candidate_hrefs
    if (url := crawlable_url(href, base_url, page_url)) is not None
}

print(eligible)
# {'https://example.test/documentation'}

This demonstrates the boundary between extraction and scheduling: all values can be found in an anchor's href, but only the ordinary web link passes this crawler policy.

Use LinkFinder in a crawler pipeline

1. A downloader obtains HTML text for the current page.
2. Construct LinkFinder(base_url, page_url).
3. Pass the downloaded text to finder.feed(html_content).
4. Retrieve unique URLs with finder.page_links().
5. Remove fragments, reject unsupported schemes, apply scope rules, and send remaining URLs to a queue.

finder = LinkFinder(base_url, page_url)
finder.feed(html_content)
links = finder.page_links()

for url in links:
    if same_domain(url, base_url):
        crawl_queue.add(url)

The queue should generally maintain its own visited or scheduled set as well. Parser-level deduplication prevents repeated links within one document; queue-level deduplication prevents the same URL from being scheduled by multiple pages.

Compatibility and malformed HTML

Older subclass examples sometimes include a legacy error callback because older Python versions documented parser error handling differently. Modern HTMLParser is generally tolerant of malformed HTML, and current code should tailor error handling to the Python version and crawler's requirements rather than blindly copying an obsolete callback.

Real pages may contain missing attributes, invalid nesting, broken entities, or incomplete markup. Treat parser output as best-effort. Check values before joining them, log the page and value when useful, and skip malformed or unsupported references instead of allowing one bad page to stop the entire crawler.

Troubleshooting

• Relative links are incomplete: The code probably stores href directly. Resolve it with parse.urljoin(page_url, href).
• The same URL is processed repeatedly: Use a set in the parser and a separate visited or scheduled set in the crawler queue.
• No links are found: Confirm that HTML text was passed to feed, that the callback is spelled exactly handle_starttag, and that test markup contains a elements with nonempty href values.
• Unsupported URLs appear: Filter fragment, mail, telephone, JavaScript, data, and non-HTTP schemes after extraction.
• A relative URL resolves beside the wrong resource: Check the exact page_url. A file-like URL such as /docs/page.html behaves differently from a directory URL ending in /.
• Malformed pages produce incomplete results: Accept best-effort results, validate values, and log problematic pages for later inspection.

Exam-relevant notes

• HTMLParser is a standard-library class; a custom parser subclasses it.
• super().__init__() initializes the inherited parser.
• handle_starttag(tag, attrs) receives the tag name and a sequence of attribute pairs.
• Only an anchor's href identifies its destination; attributes such as class and target do not.
• urljoin resolves relative references but does not enforce crawl scope.
• A set provides deduplication but does not guarantee output order.
• Parsing HTML, filtering candidates, downloading pages, and scheduling work are separate crawler responsibilities.