Free Robots.txt Generator
A robots.txt generator builds the file that tells crawlers which parts of your website they may visit. This free generator sets a default for all robots, allows or blocks search engines, AI crawlers such as GPTBot and ClaudeBot, and SEO bots, adds blocked paths and your sitemap, and gives you a finished robots.txt to copy.
Robots.txt Generator
Technical SEO
About this tool
A crawler obeys only the most specific group that names it. If you give Googlebot its own group, it ignores everything in the "User-agent: *" group, including your blocked paths. This generator handles that for you: any robot you set to Allow while the default blocks everything gets its own group with your blocked paths repeated, and robots left on "Same as default" simply follow the main group.
Robots.txt controls crawling, not indexing. A blocked page can still appear in search results if other sites link to it, so to keep a page out, let it be crawled and add a noindex tag instead. Blocking an AI crawler only affects its future visits. Google-Extended is a permission token rather than a crawler: blocking it keeps your pages out of Gemini training and grounding without changing how Google Search crawls or ranks you. Google ignores Crawl-delay, while Bing and some other crawlers honor it.
The costly mistake is a leftover "Disallow: /" from a staging site, which quietly blocks an entire live site. SEO SMO HUB checks robots.txt on every client site before anything else, and we decide on AI crawlers with each client: blocking them protects content, while allowing them keeps the brand eligible to be cited in AI answers. Our own robots.txt welcomes GPTBot, ClaudeBot and PerplexityBot for exactly that reason.
Frequently asked questions
How do I create a robots.txt file?
Choose whether all robots may crawl by default, set any search engine, AI or SEO crawler to allow or block, list the paths to keep crawlers out of, such as /admin/ or /cart/, and paste your sitemap address. Click generate, copy the result into a plain text file named robots.txt and upload it to the root of your domain.
Should I block AI crawlers like GPTBot and ClaudeBot?
It depends on what you want from AI tools. Blocking training crawlers keeps new content out of future training by companies that respect robots.txt, but assistants increasingly answer with citations, and answer-focused bots such as PerplexityBot decide whether your pages can be quoted. Many businesses block training bots and allow answer bots. Decide per bot, not all at once.
Where do I upload the robots.txt file?
Save the output as a plain text file named robots.txt and upload it to the root of your domain, so it opens at https://yourdomain.com/robots.txt. It only applies to that exact host and protocol, so a subdomain such as shop.yourdomain.com needs its own file. After uploading, open the address in a browser to confirm it loads.
Does blocking a page in robots.txt remove it from Google?
No. Robots.txt stops crawling, not indexing. If other pages link to a blocked URL, Google can still list it, usually without a description. To remove a page, allow crawling and add a noindex robots tag, or password protect it. Once it has dropped out of the index, you can block it again if you want.
What should a WordPress robots.txt contain?
For most WordPress sites: allow everything by default, disallow /wp-admin/ while allowing /wp-admin/admin-ajax.php, and list your sitemap, which WordPress serves at /wp-sitemap.xml or your SEO plugin serves at /sitemap_index.xml. Do not block /wp-content/ or /wp-includes/, because Google needs the CSS and JavaScript there to render your pages.