Programming · Intermediate

Create a web crawler in Python

Fetch pages you are allowed to fetch, parse links, and stay inside a polite crawl policy.

Who it is for

Python users who already understand functions and HTTP at a basic level.

You should already know

Python 3, and a site you own or that publishes a crawl policy you will follow.

You will be able to

  • Fetch a URL with requests
  • Extract links
  • Stay on-domain and honor robots.txt

Lessons

  1. Create a New Python Web Crawler Project Learn how to initialize a Python web crawler project, create an idempotent project directory, and prepare queue.txt and crawled.txt for persistent URL tracking.
  2. Fetch a page HTTP GET, status codes, and a timeout.
  3. Follow links Parse hrefs, join them to the base URL, and stay on your own host.