What a robots.txt file does
robots.txt is a plain text file at the root of your site that tells crawlers which URLs they may request. Search engines read it before crawling, so it's the standard way to keep bots out of admin areas, internal search results, cart pages and other URLs that waste crawl time.
It controls crawling, not indexing. A disallowed page can still appear in Google if other sites link to it — Google just shows it without a description. To keep a page out of search results, let it be crawled and add a noindex meta tag instead.
The directives
A robots.txt file is made of groups. Each group names one or more crawlers and lists rules for them:
- User-agent — the crawler the group applies to. * means every crawler without a more specific group.
- Disallow — a path prefix the crawler shouldn't request. Disallow: /admin/ covers /admin/ and everything under it. An empty Disallow: allows everything.
- Allow — an exception inside a disallowed path, such as Allow: /wp-admin/admin-ajax.php.
- Sitemap — the full URL of your XML sitemap. It applies to all crawlers and can be listed more than once.
- Crawl-delay — seconds between requests. Bing and Yandex honour it; Google ignores it.
Blocking AI crawlers
AI companies collect training data with their own crawlers, which read robots.txt like search bots do. The Block AI crawlers preset adds a group for GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Gemini training), CCBot (Common Crawl), PerplexityBot, Applebot-Extended, Bytespider and meta-externalagent.
Blocking Google-Extended doesn't affect how your site ranks in Google Search — that's handled by Googlebot, which the preset leaves alone. Keep in mind that robots.txt is voluntary: reputable crawlers obey it, but it can't stop a scraper that chooses to ignore it.
Common mistakes
Most robots.txt problems come from a handful of errors:
- Disallow: / under User-agent: * — this blocks your entire site from search engines. It's often left over from a staging site.
- Blocking /css/, /js/ or /wp-includes/ — Google needs those files to render your pages.
- Putting the file anywhere except the root. Crawlers only check https://yourdomain.com/robots.txt, and each subdomain needs its own.
- Forgetting that paths are case-sensitive: /Admin/ and /admin/ are different.
- Using robots.txt to hide private pages. The file is public, so it advertises the paths you list.
Checking it works
After uploading, open yourdomain.com/robots.txt in a browser to confirm it's live. In Google Search Console, Settings → robots.txt shows the version Google last fetched and any lines it couldn't parse.