Where should robots.txt be placed?
It normally belongs at the root of the site, for example `https://example.com/robots.txt`.
How to Use Robots.txt Generator
Step 1: Configure User-Agents
Specify default rules for all bots (*) or specific bots like Googlebot or Bingbot.
Step 2: Define Allow / Disallow
Add private directories (/admin, /api) you wish to block from indexing.
Step 3: Download or Copy
Download the generated robots.txt file and place it in your website root.
Key Features & Capabilities
- Preset templates for WordPress, Next.js, and static sites
- Supports Crawl-Delay and Sitemap URL directives
- 1-click download of clean robots.txt file
Frequently Asked Questions
Can robots.txt protect private files?
No. It is not an authentication or authorization system. Private information should be protected on the server.
Should I block every AI crawler?
That is a site-owner policy decision. Review which crawlers you want to allow, block, or treat differently, and consider the effect on discovery and search visibility.
Related Tools
- Sitemap XML Generator — Create clean, valid XML sitemaps for search engines from a list of URLs.
- Canonical Tag Generator — Generate rel="canonical" link tags to prevent duplicate content SEO penalties.