Create a web crawler in Python

About this course

This course cover the process of creating a web crawler in Python.  A web crawler is a program that browses the World Wide Web in a methodical fashion for the purpose of collecting information.

Audience

This tutorial is designed for people with prior experience in the world of programming and the Python programming language. If you are not familiar with Python, we recommend that you take our free Python course first.

Prerequisites

You should have a year or two Python programming experience. You should know what a function is, how to create a file in Python, what classes are, etc.

What you’ll learn

You will learn to create a fully functional web crawler in Python. After you finish the course, you should have a functional web crawler that you can use to gather all the links (or other elements) from a domain.

References

This course uses the web crawler code found on the GitHub website of Mr. Bucky Roberts, a well-known YouTube instructor. I encourage you to support his work by pledging an amount on Patreon.

Start the course