Create a web crawler in Python

Follow links

Parse hrefs, join them to the base URL, and stay on your own host.

from urllib.parse import urljoin, urlparse
from html.parser import HTMLParser

# collect hrefs, then:
absolute = urljoin(base, href)
if urlparse(absolute).netloc == urlparse(base).netloc:
    queue.append(absolute)

Only enqueue http(s) links on the same host unless you have a reason and permission. Cap depth and total pages. Sleep between requests. Read robots.txt first — a crawler that “just ignores it for the lab” is how you learn a bad habit.