VMware ESXi and vSphere Cluster Management
What Is a Web Crawler? Concepts, Components, and a Python Implementation
Learn how web crawlers discover pages, manage URLs, respect crawl rules, and build a responsible same-domain crawler in Python.
A web crawler is software that systematically discovers and visits web pages or other web resources to collect or process information. A crawler can retrieve a page, inspect its HTML, extract hyperlinks, record metadata, and save selected public data.
You may also hear web spider, spider, bot, or simply crawler. A crawler is one kind of bot: it performs automated actions on the web. Unlike manual browsing, which depends on a person selecting each link, a crawler follows programmed rules to discover and process many URLs.
Why Web Crawlers Are Used
- Search engines: Crawlers discover documents and revisit them when they may have changed. The retrieved content can later be organized in a searchable index.
- Link inventories: A site owner can list internal links, external links, broken links, redirects, and pages that are difficult to reach.
- Site auditing: A crawler can collect titles, status codes, content types, selected HTML elements, and other permitted public information.
- Extraction workflows: A crawler can find pages that a later process examines for targeted data. This must be done within the site's permissions, terms, and applicable rules.
Crawling is not automatically the same as scraping. Crawling focuses on discovering and fetching URLs, while scraping focuses on extracting particular data from fetched pages.
Crawling, Scraping, and Indexing Compared
For example, a search engine may crawl a page, parse its links, store its content, and then add useful terms and document information to a search index. A site-audit tool may crawl the same page but only save links, titles, and HTTP outcomes.
How a Crawler Works
- Begin with one or more seed URLs. A seed URL is an initial address from which the crawl begins.
- Place the seeds in the crawl frontier, the collection of URLs waiting to be processed.
- Fetch one URL with an HTTP request and receive an HTTP response.
- Check the response status, content type, final redirect destination, and other relevant metadata.
- Parse HTML and perform link extraction: identify anchor elements and their
hrefattributes. - Resolve relative links against the current page URL.
- Normalize and filter candidate URLs. For example, remove fragments, reject unsupported schemes, and enforce a same-host rule.
- Add eligible URLs that have not already been scheduled to the frontier.
- Record results such as the source page, destination URL, title, status code, depth, and errors.
- Repeat until the frontier is empty or a limit such as maximum pages, depth, time, or user cancellation stops the crawl.
The visited set records URLs already processed, or more safely, URLs already scheduled or processed. Marking a URL as seen before queueing it prevents duplicate work and cycles such as page A linking to page B while page B links back to page A.
Crawler Workflow and Data Structures
Queue Versus Stack
A FIFO queue processes the oldest pending URL first. This produces broadly breadth-first traversal: pages near the seed are generally visited before deeper pages. A stack processes the newest URL first and produces broadly depth-first traversal: one link path is followed deeply before other pending links.
A queue is often a useful default for a site crawler because depth limits and page-level reporting are easy to reason about. A stack can be useful when a particular path should be explored deeply. Neither strategy by itself guarantees a particular order when redirects, errors, or concurrent workers are involved.
URLs, Normalization, and Scope
An absolute URL contains a complete location, such as https://example.test/docs/start.html. A relative URL omits some of that information, such as ../contact or /about. The crawler must resolve a relative URL using the current page as its base.
Important URL parts include the scheme such as http or https, the host such as example.test, an optional port, the path, an optional query string beginning with ?, and a fragment beginning with #. A fragment usually identifies a location inside the same document and is not sent as a separate server resource, so crawlers commonly remove it before deduplication.
URL normalization converts equivalent forms into a consistent representation. At minimum, parse the URL, resolve it, remove its fragment, require an HTTP or HTTPS scheme, and compare a stable form before adding it to the seen set. More advanced normalization may address default ports, host case, trailing slashes, percent encoding, or query parameter order. Do not remove query strings blindly: some represent different content.
A same-domain crawl is restricted to one website or host. Compare the parsed candidate host with the intended allowed host. Consider whether subdomains should count as in scope; docs.example.test and example.test are different hosts even though they share a parent domain.
Redirects require another scope check. A discovered URL may be in scope but redirect to a different host. Inspect the final response URL and do not parse or schedule it as an in-scope page unless it passes the scope policy.
Basic Python Crawler Design
A small crawler is easier to test when each responsibility is separate:
- A URL queue manages pending work.
- A seen or visited set prevents duplicate scheduling.
- An HTTP fetching function handles headers, timeouts, status codes, redirects, and errors.
- An HTML link extraction function parses anchor elements and reads
href. - A scope filter resolves URLs and decides whether they are allowed.
- An output collector stores page records and discovered links.
Environment Setup
python -m venv .venv
# macOS or Linux
source .venv/bin/activate
# Windows PowerShell
.venv\Scripts\Activate.ps1
python -m pip install requests beautifulsoup4The example below uses requests for HTTP and Beautiful Soup for tolerant HTML parsing. It also uses standard-library URL utilities instead of manually joining strings.
A Same-Host, Breadth-First Implementation
from collections import deque
from urllib.parse import urldefrag, urljoin, urlsplit, urlunsplit
import time
import requests
from bs4 import BeautifulSoup
def normalize_url(raw_url, base_url):
absolute = urljoin(base_url, raw_url)
without_fragment, _ = urldefrag(absolute)
parts = urlsplit(without_fragment)
if parts.scheme not in {"http", "https"} or not parts.hostname:
return None
# Lowercase the scheme and host for stable comparisons.
host = parts.hostname.lower()
port = parts.port
if port and not ((parts.scheme == "http" and port == 80) or
(parts.scheme == "https" and port == 443)):
netloc = f"{host}:{port}"
else:
netloc = host
return urlunsplit((parts.scheme.lower(), netloc,
parts.path or "/", parts.query, ""))
def in_scope(url, allowed_host):
return urlsplit(url).hostname == allowed_host
def extract_links(html, page_url):
soup = BeautifulSoup(html, "html.parser")
title = soup.title.get_text(" ", strip=True) if soup.title else ""
links = []
for anchor in soup.find_all("a", href=True):
candidate = normalize_url(anchor["href"], page_url)
if candidate:
links.append(candidate)
return title, links
def crawl(seed_url, max_pages=50, max_depth=2, delay=1.0,
timeout=10, user_agent="ExampleStudyCrawler/1.0"):
seed = normalize_url(seed_url, seed_url)
if not seed:
raise ValueError("The seed URL is not an HTTP or HTTPS URL")
allowed_host = urlsplit(seed).hostname
frontier = deque([(seed, 0)])
seen = {seed} # Mark before queueing to prevent duplicates.
results = []
session = requests.Session()
session.headers.update({"User-Agent": user_agent})
while frontier and len(results) < max_pages:
url, depth = frontier.popleft()
record = {"url": url, "depth": depth}
try:
response = session.get(url, timeout=timeout, allow_redirects=True)
record["status_code"] = response.status_code
record["content_type"] = response.headers.get("Content-Type", "")
record["final_url"] = response.url
final_url = normalize_url(response.url, url)
if not final_url or not in_scope(final_url, allowed_host):
record["error"] = "redirected outside allowed host"
elif response.ok and "text/html" in record["content_type"].lower():
title, links = extract_links(response.text, final_url)
record["title"] = title
record["links_found"] = len(links)
if depth < max_depth:
for link in links:
if in_scope(link, allowed_host) and link not in seen:
seen.add(link)
frontier.append((link, depth + 1))
else:
record["error"] = "not a successful HTML response"
except requests.RequestException as exc:
record["error"] = type(exc).__name__
results.append(record)
time.sleep(delay)
return resultsThis implementation uses a FIFO queue, removes fragments, rejects non-web schemes, limits the host, records redirects, skips non-HTML responses, catches common request failures, and stops at page or depth limits. In production, add robots.txt checks, retry policy, logging, and a more explicit query-string policy.
Configurable Crawler Settings
seed_url: the starting address.allowed_hostorallowed_domains: the crawl boundary.max_pagesandmax_depth: work limits.worker_count: the number of concurrent workers, if used.request_timeout: the maximum wait for a request.request_delay: the pause between requests or scheduled requests.user_agent: an honest client identity.retry_count: the maximum number of carefully controlled retries.output_file: a JSON or CSV destination.
Robots.txt and Responsible Crawling
robots.txt is a site policy file that can communicate crawler access preferences by user agent and path. Check it before crawling, use the applicable user-agent rules, and treat disallowed paths as out of scope. Robots.txt is not authentication and does not grant permission to access private data.
Identify the client with a descriptive User-Agent rather than pretending to be a browser or using an anonymous identity. Apply a conservative rate limit and request delay. Use timeouts, bounded retries, and exponential backoff for temporary failures. Stay within the intended host and avoid private, authenticated, sensitive, or explicitly disallowed content. Review website terms, access restrictions, and applicable rules before collecting or reusing data.
Multithreaded Crawling
Fetching pages is usually network-bound: a worker spends much of its time waiting for a remote server. A small group of concurrent threads can therefore improve throughput. A typical worker-pool design has one shared thread-safe queue, several workers that take URLs from it, and shared visited and results collections.
Queue operations such as get, put, and task_done should use a thread-safe queue such as Python's queue.Queue. Compound operations on shared sets and lists still need synchronization. For example, checking whether a URL is absent and then adding it must be protected by a lock so two workers cannot schedule it simultaneously.
from queue import Queue
from threading import Lock, Thread
frontier = Queue()
seen = set()
results = []
state_lock = Lock()
def schedule(url):
with state_lock:
if url in seen:
return False
seen.add(url)
frontier.put(url)
return True
def worker():
while True:
url = frontier.get()
if url is None: # Sentinel used for orderly shutdown.
frontier.task_done()
return
try:
record = fetch_and_process(url)
with state_lock:
results.append(record)
for link in record.get("eligible_links", []):
schedule(link)
finally:
frontier.task_done()
workers = [Thread(target=worker) for _ in range(3)]
for thread in workers:
thread.start()
# Schedule seeds before waiting for completion.
# schedule(seed_url)
frontier.join()
for _ in workers:
frontier.put(None)
for thread in workers:
thread.join()The example is a concurrency pattern, not a complete replacement for the single-threaded crawler. The worker must still perform robots checks, scope checks, timeouts, delays, response validation, and bounded retries. Choose a conservative worker count. More threads can increase throughput, but they can also overwhelm a server or violate politeness requirements. A production design should also coordinate rate limiting across workers.
HTTP Outcomes and Reliability
Handle connection failures, DNS failures, TLS problems, timeouts, invalid URLs, and malformed HTML without stopping the entire crawl. Check the Content-Type header before parsing. If the crawler is designed only for HTML links, skip images, PDFs, archives, and other non-HTML responses or report them separately.
Common sources of runaway crawls include cyclic links, calendar pages that generate unlimited dates, faceted navigation with many parameter combinations, and session identifiers in query strings. Use normalization, host restrictions, depth and page limits, URL pattern rules, and a deliberate query-string policy. Always log failures and statistics such as pages attempted, successful responses, redirects, skipped content types, duplicate candidates, and errors.
Outputs and Extensions
A minimal output is a unique list of discovered in-scope links. A useful report can record:
- source page URL;
- destination URL;
- page title;
- HTTP status code;
- crawl depth;
- final redirect URL;
- response content type; and
- an error state or failure reason.
Export records with Python's json or csv module. JSON preserves nested structures conveniently, while CSV is convenient for spreadsheets and simple audits.
import csv
with open("crawl-report.csv", "w", newline="", encoding="utf-8") as file:
fields = ["url", "final_url", "status_code", "depth", "title", "error"]
writer = csv.DictWriter(file, fieldnames=fields)
writer.writeheader()
for record in results:
writer.writerow({field: record.get(field, "") for field in fields})The same foundation can be extended to extract selected elements such as headings, canonical links, image sources, or structured metadata. Keep discovery, fetching, parsing, filtering, and storage separate so an extraction change does not accidentally alter scope or request behavior.
Troubleshooting Common Problems
The Crawler Repeats Pages
Usually URLs are being queued before a seen check, equivalent forms are not normalized, or fragments are treated as separate targets. Normalize first, remove fragments, and add a URL to a thread-safe seen collection before queueing it.
The Crawler Leaves the Intended Website
Apply a host or domain filter after URL resolution and again after redirects. Incorrect relative-link handling can also produce unexpected addresses; use urljoin with the current page as the base.
The Crawler Is Slow
Serial requests may be dominated by network latency. A small worker pool and connection reuse can help, but preserve delays and do not raise concurrency beyond a responsible level.
Responses Are 403, 429, or Repeatedly Failing
Check robots.txt, site policies, and access requirements. Reduce request frequency, identify the crawler honestly, use bounded backoff, and do not attempt to circumvent access controls.
Parsing Fails
Confirm that the response is HTML, account for unexpected encoding, and use a tolerant parser. Record and skip unsupported response types when the goal is HTML link extraction.
Multithreaded Results Are Inconsistent
Use a thread-safe queue, protect compound shared-state updates with a lock, wait for all queued tasks, and join workers before writing final output.
Exam-Relevant Summary
- A crawler systematically discovers and visits web resources; spider and web spider are alternate names, and a crawler is one type of bot.
- Crawling discovers and fetches URLs, scraping extracts targeted data, and indexing organizes content for search.
- Seeds start a crawl; the frontier holds pending URLs; the seen set prevents duplicate work; and the results store records outcomes.
- FIFO queues support breadth-first behavior, while stacks support depth-first behavior.
- Resolve relative URLs with URL utilities, remove fragments, filter schemes, normalize before deduplication, and recheck redirect destinations.
- Use limits, timeouts, clear user-agent identification, robots.txt checks, rate limiting, bounded retries, and honest scope control.
- Threads can improve I/O-bound throughput, but shared state requires synchronization and concurrency must remain polite.
For related study, see What Is a Web Crawler?.