Programming · Intermediate

Create a web crawler in Python

Fetch pages you are allowed to fetch, parse links, and stay inside a polite crawl policy.

Who it is for

Python users who already understand functions and HTTP at a basic level.

You should already know

Python 3, and a site you own or that publishes a crawl policy you will follow.

You will be able to

  • Fetch a URL with requests
  • Extract links
  • Stay on-domain and honor robots.txt

Lessons

This section is a topic index. Open a lesson from search or the courses list.