VMware ESXi and vSphere Cluster Management

Parsing Domains and Subdomains for a Python Web Crawler

Learn how to parse hosts, subdomains, and simple base domains with urllib.parse so a Python crawler can enforce its crawl scope safely.

A web crawler follows links discovered in pages. Those links may lead to the original host, a related subdomain, or an entirely different website. Domain parsing gives the crawler a consistent way to decide which links belong in its crawl queue.

This lesson uses Python's standard urllib.parse.urlparse function. It shows how to extract a host, remove a leading subdomain with a simple label-based rule, handle invalid input, and apply the result to crawler queue filtering.

Why Crawlers Need Domain Parsing

A crawler usually begins with one target URL. While downloading pages, it discovers links such as:

• https://example.com/about
• https://blog.example.com/post
• https://other.example.net/page

Comparing complete URLs is not enough. Two URLs can have different paths, queries, or fragments while still belonging to the same host. For example, https://example.com/about and https://example.com/contact are different URLs but have the same host.

Before adding a discovered link to the crawl queue, parse its host and compare it with the crawler's selected crawl scope. This prevents the crawler from unexpectedly leaving the intended site.

Exact-host scope versus base-domain scope

An exact-host policy allows only one host. If the target is example.com, then blog.example.com is rejected.

A base-domain policy groups a domain with its subdomains. If the selected base domain is example.com, then both example.com and blog.example.com can be accepted. This is broader and should be chosen deliberately because subdomains may contain separate applications or user-controlled content.

URL Components Relevant to Crawling

A URL is a web address made of several components. Consider:

https://mail.geek-university.com:8443/courses?level=beginner#overview

• Scheme: https, the protocol prefix.
• Host: mail.geek-university.com, the server name. It includes the subdomain mail.
• Port: 8443, an optional network port.
• Path: /courses, the resource location on the host.
• Query string: level=beginner, parameters after ?.
• Fragment: overview, the section after #.

For domain checks, the host is the important component. The path and query do not change which host the crawler would contact. A fragment is normally handled by the browser and is not sent in an HTTP request, but it can still create duplicate queue entries if it is not removed during URL normalization.

Parsing URLs with urllib.parse

Python's standard library provides urlparse for breaking an absolute URL into named components.

from urllib.parse import urlparse

url = "https://mail.geek-university.com/courses?level=beginner#overview"
parsed = urlparse(url)

print(parsed.scheme)  # https
print(parsed.netloc)  # mail.geek-university.com
print(parsed.path)   # /courses
print(parsed.query)  # level=beginner
print(parsed.fragment)  # overview
print(parsed.hostname)  # mail.geek-university.com

The result is a structured ParseResult. Its netloc value represents the network location and may contain credentials, a host, and a port. Its hostname attribute is usually better for domain comparisons because it excludes the port and credentials.

from urllib.parse import urlparse

parsed = urlparse("https://user:secret@example.com:8443/path")

print(parsed.netloc)     # user:secret@example.com:8443
print(parsed.hostname)   # example.com

An input such as example.com/page does not provide a scheme. In that case, urlparse treats the text as a path rather than recognizing example.com as a host. A crawler should normalize such links first or reject them before domain filtering.

Extracting a Host Including Its Subdomain

A subdomain is a label before the rest of a domain. In mail.geek-university.com, mail is the subdomain and the complete host is mail.geek-university.com.

The following helper returns the hostname in lowercase. It returns an empty string when the input is not an absolute URL with a usable host.

from urllib.parse import urlparse


def get_host(url: str) -> str:
    """Return the lowercase host, or an empty string if none is available."""
    if not isinstance(url, str) or not url.strip():
        return ""

    parsed = urlparse(url.strip())

    # A netloc is expected for an absolute URL such as https://example.com.
    if not parsed.scheme or not parsed.netloc:
        return ""

    try:
        hostname = parsed.hostname
    except ValueError:
        # For example, malformed bracketed IPv6 input can fail here.
        return ""

    if not hostname:
        return ""

    return hostname.rstrip(".").lower()

Examples:

get_host("https://mail.geek-university.com/courses")
# "mail.geek-university.com"

get_host("https://docs.api.example.com:8443/v1")
# "docs.api.example.com"

get_host("not a valid absolute URL")
# ""

The helper intentionally returns a predictable empty string rather than allowing malformed data to enter later domain logic. Callers must treat that empty result as ineligible for crawling.

Extracting a Simple Base Domain

For instructional code, a simple approach is to split the host on dots and keep the final two labels:

def get_simple_base_domain(url: str) -> str:
    """Return the final two host labels, or an empty string when unavailable."""
    host = get_host(url)
    if not host:
        return ""

    labels = host.split(".")
    if len(labels) < 2:
        return ""

    return ".".join(labels[-2:])

For mail.geek-university.com, splitting produces ["mail", "geek-university", "com"]. Joining the final two labels removes the leading mail label and returns geek-university.com.

get_simple_base_domain("https://mail.geek-university.com/courses")
# "geek-university.com"

get_simple_base_domain("https://www.example.com/articles?id=7")
# "example.com"

get_simple_base_domain("https://docs.api.example.com:8443/v1")
# "example.com"

get_simple_base_domain("not a valid absolute URL")
# ""

The length check is important. Without it, indexing labels[-2] or joining the final labels could fail for a host such as localhost or for an absent host.

URL Parsing Results

Limitations of the Last-Two-Labels Technique

The final-two-label rule is useful for learning string operations, but it is not globally correct. A public suffix is a suffix under which registrations can occur, such as com or co.uk. A registrable domain is the part a registrant can control beneath that suffix.

For shop.example.co.uk, the registrable domain is example.co.uk. The simple function returns co.uk, which is only the public suffix. Treating co.uk as a site's base domain could group unrelated organizations together.

• Multi-part public suffixes: Examples such as example.co.uk need public-suffix knowledge.
• IP addresses: https://192.0.2.10/page is not a dotted domain whose final two labels form a useful base domain.
• Local hosts: http://localhost:8000/ has only one host label.
• Internationalized domain names: Unicode names and their IDNA or punycode forms require consistent normalization.
• Ports: A port is part of the network location but not normally part of the hostname used for domain comparison.

For production-grade registrable-domain extraction, use a Public Suffix List-aware library or service, such as the Python package tldextract, rather than relying only on dot splitting. The simple function in this lesson should be described as an instructional base-domain approximation.

Error Handling and Return Conventions

Parsing helpers should have a documented result for every input category. This lesson uses the empty string for invalid, incomplete, or hostless input.

• Return "" when the URL is not a string or is blank.
• Return "" when a scheme or network location is missing.
• Return "" when hostname is unavailable or malformed.
• Check the number of host labels before using negative indexes.
• Catch only the specific ValueError that can arise from malformed parsed URL data.

A broad except Exception can hide programming errors and make debugging difficult. Explicit validation is preferable where the expected invalid cases are known.

Testing the Helper Functions

Small tests should cover ordinary domains, nested subdomains, ports, malformed input, and country-code examples.

tests = [
    (
        "https://mail.geek-university.com/courses",
        "mail.geek-university.com",
        "geek-university.com",
    ),
    (
        "https://www.example.com/articles?id=7",
        "www.example.com",
        "example.com",
    ),
    (
        "https://docs.api.example.com:8443/v1",
        "docs.api.example.com",
        "example.com",
    ),
    (
        "https://shop.example.co.uk/products",
        "shop.example.co.uk",
        "co.uk",  # Shows the simple method's limitation.
    ),
    ("https://example.com", "example.com", "example.com"),
    ("https://localhost:8000/", "localhost", ""),
    ("not a valid absolute URL", "", ""),
    ("example.com/page", "", ""),
]

for url, expected_host, expected_base in tests:
    assert get_host(url) == expected_host
    assert get_simple_base_domain(url) == expected_base

print("All tests passed")

The .co.uk expectation above is intentionally the result of the simple algorithm, not a claim that it is the correct registrable domain. A Public Suffix List-aware test should instead expect example.co.uk.

Integrating Domain Checks into a Crawler

Store the selected scope when the crawler starts. Then parse every discovered link before adding it to the queue. The comparison should use the same normalized representation on both sides.

target_url = "https://example.com/start"
target_host = get_host(target_url)
target_base = get_simple_base_domain(target_url)


def allowed_for_scope(url: str, policy: str) -> bool:
    host = get_host(url)
    if not host:
        return False

    if policy == "exact-host":
        return host == target_host

    if policy == "base-domain":
        base = get_simple_base_domain(url)
        return bool(base) and base == target_base

    raise ValueError("Unknown crawl policy")


crawl_queue = []
discovered_urls = [
    "https://blog.example.com/post",
    "https://example.com/about",
    "https://other.example.net/page",
]

for link in discovered_urls:
    if allowed_for_scope(link, policy="base-domain"):
        crawl_queue.append(link)

print(crawl_queue)
# ["https://blog.example.com/post", "https://example.com/about"]

With policy="exact-host", only the URL on example.com is accepted. With policy="base-domain", the blog subdomain is accepted as well. In a real crawler, resolve relative links before calling this function, and separately reject unsupported schemes such as mailto:.

Recommended module separation

Keep URL parsing separate from traversal logic. For example, create a module named domain.py containing get_host and get_simple_base_domain. Import those helpers from the spider or queue-management module.

# domain.py
from urllib.parse import urlparse

# Define get_host and get_simple_base_domain here.

# spider.py
from domain import get_host, get_simple_base_domain

This separation makes the parsing helpers easy to test and allows the crawler's enqueue logic to focus on scope decisions.

Troubleshooting Domain Parsing

The returned host includes a port number

Cause: The code reads netloc directly from a URL such as https://example.com:8443/path.

Fix: Use parsed.hostname when comparisons should ignore the port.

A URL without a scheme has no usable host

Cause: example.com/page is parsed as a path because it lacks https:// or another scheme.

Fix: Resolve or normalize the link as an absolute URL first, or reject scheme-less input before queue filtering.

The base domain for a .co.uk address is wrong

Cause: The code assumes every registrable domain consists of exactly two trailing labels.

Fix: Use a Public Suffix List-aware extractor for real-world domain boundaries.

Malformed links break queue processing

Cause: The code indexes split labels before checking whether a host was parsed successfully.

Fix: Validate the host and label count first, returning the defined empty result when necessary.

The crawler visits an unwanted external site

Cause: Queue logic compares full URLs, paths, or inconsistent host representations.

Fix: Apply one explicit domain-scope comparison before every enqueue operation.

The crawler rejects valid subdomain links

Cause: An exact-host policy is active even though the intended scope includes subdomains.

Fix: Choose and document either exact-host matching or base-domain matching.

Key Exam and Practice Notes

• urlparse() returns structured URL components; it does not validate every aspect of a URL for you.
• netloc may include credentials and a port, while hostname is the safer field for host comparisons.
• A path, query string, or fragment does not change the host.
• Always validate a parsed host before splitting it or indexing its labels.
• The final-two-label method is a teaching approximation, not a universal registrable-domain algorithm.
• Exact-host scope and base-domain scope have different security and coverage implications.
• Invalid or hostless results must be rejected before URLs enter the crawl queue.

Next Steps

After domain filtering, a crawler commonly needs to resolve relative links with urllib.parse.urljoin, remove fragments, reject non-HTTP schemes, detect duplicate URLs, honor robots.txt, and apply rate limits. Accurate public-suffix-aware parsing should be added before using base-domain scope on diverse real-world sites.

Continue practicing these ideas in the domain parsing lesson.