VMware ESXi and vSphere Cluster Management

Web Crawler Requirements and Setup

Learn the Python, IDE, interpreter, source files, and internet connectivity requirements needed before building and running a web crawler.

Why prepare the environment first?

A web crawler needs a configured programming environment before you can write or execute its code. Preparing the environment helps you identify whether a problem comes from the program itself, the Python installation, the selected project interpreter, or the network.

A web crawler is a program that automatically requests web pages, discovers links, and visits reachable pages according to defined rules. Crawling depends on network access and on the target pages being available on the web.

Crawling and scraping are related but different activities. Crawling focuses on discovering and navigating URLs. Scraping focuses on extracting useful data from the pages that are retrieved. A project can do both, but the setup requirements come first.

Local development requirements and runtime requirements

Local development requirements are the tools you use to create and test the crawler, such as Python, an IDE, source files, and a terminal. Runtime requirements are what must be available while the crawler is running, such as a working Python interpreter, the required project files, and network access to the target website.

For a small learning project, these requirements may be installed on the same computer. In a larger deployment, development may happen on one machine while the crawler runs on another, but both environments still need a compatible Python runtime and the required project files.

Requirements checklist

Requirement	Purpose	Recommended Option	How to Verify
Python runtime	Interprets and runs the crawler's Python source code.	Python 3.x from the official Python distribution site.	Run `python --version` or `python3 --version` and confirm a Python 3 version.
Development environment	Provides tools for writing, organizing, running, and debugging code.	PyCharm Community Edition, or IDLE as a simpler alternative.	Open a project or script and confirm that its interpreter is Python 3.
Internet connectivity	Allows the crawler to request pages and follow links on external websites.	An active network connection with access to the intended target.	Confirm that the computer can access the target website and that network policies permit requests.
Tutorial source files	Provides example scripts and supporting project files.	An extracted course source archive opened as a local project.	Check that the files are extracted and can be opened and run with Python 3.

Install Python 3

All examples in this tutorial require Python 3.x. Python 3.x is the major Python language family used by the crawler examples. A legacy Python release may not understand Python 3 syntax or provide compatible libraries, so it cannot be used interchangeably.

Obtain Python from the official Python distribution site and install a current Python 3 release supported by your operating system. Follow the installer instructions for your platform. On systems that offer an option to make Python available from the command line, enable that option if appropriate for your setup.

The installed interpreter is the Python executable that runs Python source code. After installation, open a terminal or command prompt and check the version:

python --version
python3 --version

One of these commands should report a version beginning with Python 3, such as Python 3.x.y. The exact minor and patch numbers can vary. Use the command that resolves to your installed Python 3 interpreter.

Exam-relevant note: A Python version number beginning with 2 is not a valid substitute for Python 3.x in this tutorial. Always verify the interpreter rather than assuming that the command named python points to the correct installation.

Run a basic script

To run a crawler script from a terminal, change to the directory containing the file or provide the file path:

python crawler.py
python3 crawler.py

Use only the command that starts the intended Python 3 interpreter. If the command is run from another directory, provide an appropriate path to crawler.py.

Choose a Python development environment

An IDE, or integrated development environment, is an application used to edit, organize, run, and debug code. A code editor can also be used, but a beginner-friendly IDE makes it easier to open the project, select an interpreter, and run a script.

PyCharm Community Edition

PyCharm Community Edition is a free Python IDE suitable for this project. When creating or opening the crawler project, select a Python 3 installation as the project interpreter. The IDE's run configuration should point to the intended crawler script.

The project interpreter is the specific Python installation that an IDE uses for one project. It may differ from the interpreter selected by a terminal. This difference is a common source of confusing results, so check the project interpreter before running code.

IDLE

IDLE is a basic Python editor and interactive shell often installed alongside Python. It is suitable for opening a small example, saving it locally, and executing it with the Python 3 shell. IDLE provides fewer project-management features than PyCharm, but it can be useful for a first run.

Interpreter configuration checklist

Install Python 3 if it is not already installed.
Create or open the crawler project in PyCharm Community Edition, or open the script in IDLE.
Set the project or application interpreter to the Python 3 installation.
Confirm that the run configuration executes the intended crawler script.
Compare the IDE's interpreter version with the version reported by the terminal.

Internet connectivity

An active internet connection is required when crawling external websites. The crawler uses network requests to retrieve pages, reads the returned content, discovers links, and sends further requests according to its rules.

Being connected to the internet does not guarantee that every request will succeed. A local firewall, proxy, DNS problem, organizational policy, or other network restriction can block access. The target server can also be unavailable or refuse requests.

Before testing a crawler, confirm that the computer can reach the intended target in a browser or by another approved method. Crawl only targets you are permitted to access, and follow applicable site rules, robots.txt guidance, and reasonable request rates.

Prepare tutorial source files

Accompanying course code may be supplied as a downloadable source archive. A source archive is a compressed download containing project code and related files.

Download the course code package to a known location.
Extract the archive completely into a project directory. Do not treat the compressed file itself as the project folder.
Open the extracted project directory in PyCharm Community Edition, or open an individual Python file in IDLE.
Select a Python 3 interpreter for the project or shell.
Run the example from the IDE or from a terminal using the appropriate Python 3 command.

Source examples are reference material. They should be run only after Python, the development environment, the project interpreter, and network access have been configured.

Python environment compatibility

Environment Item	Expected Setting	Common Incorrect Setting	Result
Python interpreter	A Python 3.x executable.	A legacy Python installation or an unavailable interpreter.	Python 3 syntax or libraries may fail to run.
IDE project configuration	PyCharm or another environment is configured to use the same Python 3 installation used for testing.	The IDE points to a different Python version or no interpreter is selected.	Code may work in one place but fail in the IDE, or the IDE may be unable to start it.
Code execution command	`python` or `python3` resolves to Python 3, and the command is run from the correct project directory.	The command resolves to an older interpreter, or the script path is incorrect.	The script may report syntax errors, use the wrong libraries, or fail because the file cannot be found.

Check both the system Python version and the project interpreter version. Matching them reduces the chance that an example behaves differently in the terminal and the IDE.

Practical setup examples

Verify Python availability

After installing Python, open a terminal or command prompt and run:

python --version

If that command does not report Python 3, try:

python3 --version

If the second command reports Python 3.x, use python3 when running the crawler on that system.

Configure a PyCharm project

Create or open the crawler project in PyCharm Community Edition. In the project interpreter settings, select an installed Python 3 executable. Then verify that the run configuration points to the crawler script, rather than to a different example or an empty project.

Use IDLE for a first run

Open the crawler example in IDLE, save it in a local project directory, and run it using the Python 3 shell. If the script makes external requests, keep the computer connected to the internet while it runs.

Confirm network readiness

Run a small request-based crawler example against an appropriate target while connected to the internet. Successful retrieval of the target page indicates that basic network access is available, but it does not prove that every URL or network environment will work.

Troubleshooting

The command line reports an older Python version

The system command may resolve to a legacy Python installation instead of Python 3. Install Python 3 if necessary, try the Python 3 command variant, or update path settings so the intended interpreter is selected. Also update the IDE's project interpreter.

Code works in the IDE but fails in the terminal

The IDE and terminal are probably using different interpreters. Compare the version shown by the IDE with the output of python --version or python3 --version. Run the script with the same Python 3 installation in both locations.

The crawler cannot retrieve a website

First verify the internet connection and the target address. If connectivity is available, check firewall rules, proxy configuration, DNS resolution, and organizational network policies. The target website may also be unavailable or restricting automated requests.

The downloaded project does not open or run

The archive may not have been extracted, the wrong folder may have been opened, or no Python 3 project interpreter may be selected. Extract the archive fully, open the extracted project directory, and configure the project to use Python 3.

PyCharm cannot find Python

Python may not be installed, or its interpreter location has not been selected. Install Python 3 from the official Python distribution site, then add or select that interpreter in the project's settings.

Final readiness checklist

Python 3.x is installed and its version can be verified from a terminal.
PyCharm Community Edition, IDLE, or another suitable editor is available.
The IDE project interpreter is set to Python 3.
The terminal and IDE use compatible Python installations.
The tutorial source archive, if supplied, has been extracted into a project directory.
The crawler script can be located and run with the correct command.
An active internet connection is available for external targets.
Firewall, proxy, DNS, and other network restrictions have been considered.

Once these checks pass, the environment is ready for crawler code. The next technical concerns are how HTTP requests work, how URLs are discovered, and how links are followed safely and responsibly.