VMware ESXi and vSphere Cluster Management

Gather Links from a Web Page in a Python Crawler

Learn to fetch HTML with urllib.request, validate Content-Type, decode response bytes, parse links with LinkFinder, and handle crawler failures safely.

What gather_links Does

A web crawler is a program that visits web pages and discovers additional URLs to visit. In this design, the Spider class coordinates page retrieval and link collection.

The Spider class can provide a static method named gather_links. A static method belongs to a class but does not need instance-specific state. It accepts the URL of the page to crawl, retrieves that page, prepares its HTML, and returns the links discovered on it.

The method should separate two responsibilities:

• Spider.gather_links: requests the URL, checks whether the response is HTML, reads the body, decodes it, and handles failures.
• LinkFinder: parses HTML, examines anchor elements and href attributes, resolves relative URLs, and applies crawler-specific filtering or normalization.

This separation keeps network code independent from HTML parsing code.

Important Data Types

• An HTTP response is the server result for a request. It provides headers and a response body.
• Bytes are raw binary data. Reading an HTTP response body initially produces bytes, not a Python text string.
• Decoding converts bytes into text using a character encoding such as UTF-8.
• An HTML parser reads HTML text and identifies elements and attributes.
• A set stores unique values, so duplicate discovered URLs are automatically removed.

Fetching a Page with urlopen

Python's urllib.request.urlopen function opens a URL and returns a response object. The response exposes headers through response.headers and body data through response.read().

from urllib.request import urlopen

# LinkFinder must be imported from the crawler's parser module.
from link_finder import LinkFinder

Use a context manager so the response is closed after processing:

with urlopen(page_url) as response:
    content_type = response.headers.get("Content-Type", "")
    body_bytes = response.read()

The value returned by read() is a bytes value. It must be decoded before being passed to a parser that expects a Python string.

Checking Content-Type

The Content-Type response header identifies the media type of the returned content. An HTML page commonly uses text/html.

Do not require an exact match. Servers often append a character-set parameter:

text/html; charset=UTF-8

Testing whether text/html occurs in the header accepts both the simple form and the form with parameters.

content_type = response.headers.get("Content-Type", "").lower()

if "text/html" not in content_type:
    return set()

This check prevents the HTML parser from receiving images, PDF documents, archives, downloads, or other binary resources that are not web pages.

Decoding the Response Body

After confirming that the response is HTML, read its body and decode it:

body_bytes = response.read()
html_text = body_bytes.decode("utf-8")

These are three distinct stages:

1. HTTP body bytes: raw data received from the server.
2. Decoded HTML text: a Python string produced using UTF-8.
3. Parsed links: URLs identified by LinkFinder after it processes the HTML string.

UTF-8 is a sensible initial expectation for web content. Some pages use another encoding, so a more advanced crawler can later inspect charset declarations or implement a carefully chosen fallback.

Complete Spider.gather_links Method

The following implementation assumes that LinkFinder accepts the crawler's base URL and the current page URL, has a feed method, and exposes discovered links through page_links.

from urllib.request import urlopen
from link_finder import LinkFinder


class Spider:
    BASE_URL = "configured crawler base URL"

    @staticmethod
    def gather_links(page_url):
        try:
            with urlopen(page_url) as response:
                content_type = response.headers.get("Content-Type", "").lower()

                if "text/html" not in content_type:
                    return set()

                body_bytes = response.read()
                html_text = body_bytes.decode("utf-8")

            finder = LinkFinder(Spider.BASE_URL, page_url)
            finder.feed(html_text)
            return set(finder.page_links)

        except Exception as error:
            print("Unable to gather links from", page_url, ":", error)
            return set()

The base URL identifies the crawl target. LinkFinder can use it to decide whether a discovered URL belongs to the intended site and to help normalize relative URLs. The current page URL provides the reference point for links such as a relative href.

Page Retrieval and Link Collection Flow

Possible Response Outcomes

Practical Examples

HTML with Absolute and Relative Links

Suppose a fetched HTML document contains two anchor elements: one with an absolute destination and one with a relative destination. The method reads the body, decodes it, and passes the complete string to LinkFinder.

finder = LinkFinder(Spider.BASE_URL, page_url)
finder.feed(html_text)
links = set(finder.page_links)

LinkFinder is responsible for resolving the relative link against page_url and returning the normalized destinations. The method returns those destinations as a set.

HTML with a Charset Parameter

For a header such as text/html; charset=UTF-8, this condition succeeds:

if "text/html" in content_type.lower():
    # Read, decode, and parse the body.

An exact comparison such as content_type == "text/html" would incorrectly reject the charset form.

Non-HTML Resource

If the server identifies the response as an image or PDF, the method returns set() immediately. It does not read the body into LinkFinder because binary content is not an HTML document containing useful anchor elements.

Duplicate Anchors

If multiple anchors resolve to the same URL, converting the parser's collection to a set produces one entry:

unique_links = set(finder.page_links)

This removes duplicates discovered on the same page and gives calling code a predictable collection of unique URLs.

Where the Method Fits in the Crawler

gather_links belongs between choosing a page to crawl and adding newly discovered URLs to the crawl queue.

1. Obtain a page URL from the queue or another selection mechanism.
2. Call Spider.gather_links(page_url).
3. Compare the returned links with URLs already known, queued, or crawled.
4. Enqueue eligible new URLs.

page_url = next_url_to_crawl
new_links = Spider.gather_links(page_url)

for link in new_links:
    if link not in known_links:
        known_links.add(link)
        crawl_queue.append(link)

Because unsuccessful operations also return an empty set, the queue-processing loop can continue after one URL fails.

Failure Handling and Predictable Returns

Network access, response reading, decoding, and parser preparation can all fail. Examples include DNS failures, unavailable servers, invalid URLs, interrupted reads, unsupported characters, or parser errors.

Use exception handling to report the problem and preserve the crawler's workflow:

try:
    # Open, validate, read, decode, and parse.
    ...
except Exception as error:
    print("Unable to gather links:", error)
    return set()

The method should have a consistent interface:

• Successful HTML processing returns a set containing discovered links.
• Non-HTML content returns an empty set.
• Request, read, decoding, or parsing failures return an empty set.

Troubleshooting

The Parser Receives Unreadable Content or Raises a Type Error

Likely cause: The response bytes were passed directly to a parser that expects text.

Resolution: Read the body and decode it before calling feed:

html_text = response.read().decode("utf-8")
finder.feed(html_text)

The Crawler Parses Files That Are Not Web Pages

Likely cause: No Content-Type check was performed.

Resolution: Inspect the header and require that it contains text/html before reading and parsing.

One Failed URL Stops the Crawl

Likely cause: Network and parsing operations are not protected by exception handling.

Resolution: Catch relevant failures, report them, and return set() so queue processing can continue.

The Method Sometimes Returns No Usable Value

Likely cause: Some non-HTML or exception branches have no return statement.

Resolution: Return an empty set on every unsuccessful path and a link set on success.

Characters Are Corrupted or Decoding Raises an Error

Likely cause: The response uses an encoding different from UTF-8.

Resolution: Start with UTF-8 for the standard case. Later, extend the crawler to inspect charset declarations or use a safe fallback strategy.

Exam-Relevant Notes

• urlopen returns a response object; read() returns bytes.
• Decode bytes into a string before passing content to an HTML parser.
• Accept text/html when it appears with parameters such as a charset.
• Do not parse non-HTML resources.
• Construct LinkFinder with both the base URL and current page URL.
• Call the parser's feed method with decoded HTML text, then retrieve its discovered links.
• Return a set consistently, including when a request fails or the resource is not HTML.
• Use a set to remove duplicate links from one page.

Next Steps

Once this method works, the crawler can be extended with HTTP status handling, character-encoding detection, domain restrictions, queue tracking, logging, rate limiting, request headers, and respectful robots.txt handling.