robots.txt generator with AI crawler rules
Write the paths that crawlers may or may not visit, add your sitemap, and tick the AI crawlers you want to block. The file is generated in your browser, ready to copy or download as robots.txt.
Runs in your browser. Nothing you type here is sent to Zefily or stored.
How to use it
-
1
Enter the paths to allow or disallow, one per line, starting with a slash, such as /admin/.
-
2
Add the full address of your sitemap, and tick the AI crawlers to block.
-
3
Copy or download the file and upload it to the root of your site, so it opens at /robots.txt.
Questions
Where must robots.txt be placed?
At the root of the host, so it opens at https://example.com/robots.txt. Crawlers ignore a robots.txt file in a subfolder.
Does robots.txt keep a page out of search results?
No. It asks crawlers not to fetch a path. To keep a page out of search results, use a noindex robots meta tag or an X-Robots-Tag header. A page blocked in robots.txt cannot show its noindex tag, so do not block it when you want it removed.
Does blocking GPTBot or ClaudeBot stop AI answers that cite my site?
Each AI company documents its own crawler tokens, and some are for training while others are for search or for a user request. Block the tokens you choose, and read each company crawler documentation to see what a token is used for.
Is Google-Extended the same as Googlebot?
No. Googlebot still indexes your pages for Google Search. Google-Extended is a separate token that controls whether your content is used to train Google AI models, as Google describes in its crawler documentation.
Related guides
Zefily does this for you on your servers
Connect a server and Zefily sets up cron jobs, permissions, redirects, PHP pools and robots rules for your sites, then keeps them in place.