VMware ESXi and vSphere Cluster Management

Configure and Run the Python Web Crawler

Learn how to configure PROJECT_NAME and HOMEPAGE, run main.py, interpret crawler output, and inspect queue.txt and crawled.txt safely.

Purpose of the execution step

After creating the crawler source files, you launch the application from its entry-point script, usually main.py. This script initializes the crawl project, adds the starting address to the work queue, and starts the crawler workers.

Each run is defined by two important values:

• PROJECT_NAME: labels the crawl and determines the name of the local output folder.
• HOMEPAGE: the seed URL, or initial web address, from which the crawler begins discovering links.

The crawler retrieves the seed page, extracts links according to its rules, and gives eligible links to worker threads for processing.

Configure the crawler variables

Set the project name

Open main.py, the main application script, and locate the project configuration near the beginning of the file. Set PROJECT_NAME to a clear identifier for this crawl.

The project name is used to create a local directory. For example, a project named GeekUniversity normally produces a directory named GeekUniversity in the crawler's current working directory. The directory stores crawl state and makes it possible to distinguish this run from other projects.

Set the homepage or seed URL

HOMEPAGE identifies the first page the crawler visits. This is also called the seed URL. The spider retrieves that page and extracts links to discover additional pages.

Use a complete URL that includes the protocol, such as https:// or http://. A hostname by itself is not a complete URL. Prefer HTTPS when the site supports it.

PROJECT_NAME = 'GeekUniversity'
HOMEPAGE = 'https://geek-university.com'

This is an illustrative setup. Replace GeekUniversity with your own project identifier and replace the sample homepage with a website URL that you are authorized to crawl.

Run the application

Run from a terminal

Open a terminal in the directory containing main.py. The current working directory matters because the crawler creates the project output folder there.

python main.py

Some systems use an explicit Python 3 launcher:

python3 main.py

Python must be installed and available in the active environment. If the command is not recognized, activate the environment used for the project or verify the Python installation before continuing.

Run from an IDE

Open the crawler project in your IDE, select main.py, and use the IDE's Run command. Check that the IDE's working directory is the project directory; otherwise, the generated folder may appear somewhere unexpected.

During startup, the program initializes the project before crawler workers begin processing URLs. Initialization may create the project directory, prepare the queue and crawled records, and place the homepage into the initial queue.

Interpret terminal output

The exact wording depends on the implementation, but status messages commonly describe initialization and progress. Read the output as a changing snapshot rather than as a single final result.

Project creation

A message about creating a project directory indicates that the application has recognized PROJECT_NAME and is preparing local storage for the run.

Spider or worker startup

The spider is the crawler component that retrieves pages and extracts links. An announcement naming the homepage usually means that the spider has received the seed URL and is beginning discovery.

Queue and crawled counters

The queue is the collection of discovered URLs waiting to be crawled. A queue count shows how much pending work is currently available. The crawled set is the collection of URLs already processed. A crawled count shows completed work.

For example, a nonzero queue count together with a growing crawled count means that the crawler is finding work while also completing work. The counts can rise and fall as workers take URLs from the queue and add newly discovered links.

Worker and thread messages

A worker thread is a concurrent execution unit that processes crawl tasks. Messages containing different worker or thread names indicate that concurrent crawling is underway. The number of active workers is controlled by the crawler's concurrency setting.

Reported link counts usually represent URLs discovered by the spider that are waiting or eligible for processing. They do not necessarily mean that every reported URL has already been visited.

Inspect the generated crawl files

After startup, look for a directory named after the configured project. With the example configuration, the expected directory is GeekUniversity. Inside it, the crawler keeps persistent text records.

Keeping pending URLs in queue.txt and completed URLs in crawled.txt lets the crawler track progress across its workers. The crawled record also helps it avoid repeatedly processing the same links.

Practical inspection example

1. Start the crawler with a permitted, small website.
2. Wait until the project directory appears.
3. Open the project directory while the program is running or after it stops.
4. Compare queue.txt with crawled.txt: the first represents pending work, while the second represents visited URLs.

Operate the crawler responsibly

Concurrency is the number of worker threads allowed to crawl simultaneously. A high worker-thread count can send many requests to a target server at once and place substantial load on it.

• Begin with a low concurrency setting, especially while testing.
• Crawl only websites for which you have permission or a clear legitimate authorization.
• Respect site policies, applicable robots.txt guidance, and rate limits.
• Use delays or rate limiting when the crawler supports them.
• Stop the run if the target site becomes slow or starts returning errors.

A crawler may discover authentication pages, password-reset pages, administrative interfaces, account areas, or other sensitive paths. Discovery does not mean that a path is appropriate to request or process. Do not automatically treat such URLs as crawl targets; exclude them and follow the site's rules and your authorization boundaries.

Troubleshooting

The project output folder does not appear

• Confirm that main.py ran without an immediate Python error.
• Verify that PROJECT_NAME is present, nonempty, and valid for a directory name.
• Check the terminal's current working directory.
• Check whether the process has permission to write in that directory.

No pages or links are added to the queue

• Open the configured homepage in a browser to check availability.
• Confirm that the value includes http:// or https://.
• Review the crawler's link-filtering and domain-filter rules.
• Check whether the target site blocks the request.

Activity appears slow or inconsistent

• Test the seed URL manually and check network connectivity.
• Remember that a slow target server can limit progress.
• Use a smaller worker count if the selected concurrency is unsuitable.
• Compare queue and crawled counts over time instead of relying on one status message.

The target website becomes slow or returns errors

• Stop the run if your crawler is affecting the site.
• Lower the concurrency before trying again.
• Add request delays or rate limiting in a later crawler improvement.
• Review the target's policies and ensure that your activity is authorized.

Execution checklist

1. Open main.py.
2. Set a meaningful PROJECT_NAME.
3. Set HOMEPAGE to a complete, authorized URL beginning with https:// or http://.
4. Start with low concurrency.
5. Run python main.py or python3 main.py from the project directory.
6. Watch for project initialization, spider startup, worker messages, and queue/crawled counters.
7. Inspect the project-named folder, queue.txt, and crawled.txt.