SEO & Web ToolsFree

Robots.txt Generator

Create a robots.txt file visually. Select which crawlers to allow or block, specify paths to disallow, and set your sitemap URL. Preview the output in real time and download the file ready to upload to your server root. No signup required.

User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Frequently Asked Questions

What is robots.txt used for?▼
The robots.txt file tells search engine crawlers (Googlebot, Bingbot, etc.) which pages or sections of your site to crawl and which to skip. It is placed at the root of your domain (yourdomain.com/robots.txt).
Does robots.txt actually block bots?▼
It is a voluntary standard that reputable bots like Googlebot respect. Malicious bots and scrapers may completely ignore it. Use server-level access controls for security-critical blocking.
What is the correct format for a disallow rule?▼
Each rule starts with 'User-agent:' (the bot name) followed by 'Disallow:' and the path to block. For example: User-agent: * / Disallow: /admin/ blocks all bots from the /admin/ directory.
Should I block CSS and JavaScript files?▼
No. Google needs to render your CSS and JS to understand your page layout. Blocking these files can hurt your SEO. Only block content you genuinely do not want indexed.
How do I add my sitemap to robots.txt?▼
Add a line 'Sitemap: https://yourdomain.com/sitemap.xml' at the bottom of the file. This helps search engines discover your sitemap automatically.