robots.txt generator
Build a robots.txt file in your browser. User-agent, allow, disallow, and sitemap. Nothing is uploaded.
Set allow/disallow paths, then Generate. Host the file at /robots.txt.
How to use the robots.txt generator
- Set the user-agent (usually *).
- List allow and disallow paths, one per line, and an optional sitemap URL.
- Copy the file and host it at https://your-domain/robots.txt.
What robots.txt is for
It hints to crawlers which paths to skip. It is not authentication. Disallow /admin does not hide the URL from someone who types it. Sitemap is a hint to where the URL list lives. This site already serves its own /robots.txt; use this page for a property you control.
Nothing is uploaded
The text is local. You still have to publish the file at the site root.
robots.txt generator FAQ
Blank allow and disallow?
The generator writes Allow: / so the file is not an accidental full block.
Google vs Bing?
Both read this format. Per-bot user-agents are extra groups you can paste after.