Create a web crawler in Python

Fetch a page

HTTP GET, status codes, and a timeout.

import requests

resp = requests.get("https://example.com", timeout=10)
resp.raise_for_status()
print(resp.text[:200])

Always set a timeout. Always check the status. A crawler that ignores 429 and 503 will get blocked and deserves it. Identify yourself with a User-Agent that includes a contact URL when you crawl anything beyond a personal lab.