Skip to content
zefıly

robots.txt generator with AI crawler rules

Write the paths that crawlers may or may not visit, add your sitemap, and tick the AI crawlers you want to block. The file is generated in your browser, ready to copy or download as robots.txt.

Block AI crawlers

Each token is blocked for the whole site. Tick only the ones you want.

robots.txt

    

Runs in your browser. Nothing you type here is sent to Zefily or stored.

How to use it

  1. 1

    Enter the paths to allow or disallow, one per line, starting with a slash, such as /admin/.

  2. 2

    Add the full address of your sitemap, and tick the AI crawlers to block.

  3. 3

    Copy or download the file and upload it to the root of your site, so it opens at /robots.txt.

Questions

Where must robots.txt be placed?

At the root of the host, so it opens at https://example.com/robots.txt. Crawlers ignore a robots.txt file in a subfolder.

Does robots.txt keep a page out of search results?

No. It asks crawlers not to fetch a path. To keep a page out of search results, use a noindex robots meta tag or an X-Robots-Tag header. A page blocked in robots.txt cannot show its noindex tag, so do not block it when you want it removed.

Does blocking GPTBot or ClaudeBot stop AI answers that cite my site?

Each AI company documents its own crawler tokens, and some are for training while others are for search or for a user request. Block the tokens you choose, and read each company crawler documentation to see what a token is used for.

Is Google-Extended the same as Googlebot?

No. Googlebot still indexes your pages for Google Search. Google-Extended is a separate token that controls whether your content is used to train Google AI models, as Google describes in its crawler documentation.

Zefily does this for you on your servers

Connect a server and Zefily sets up cron jobs, permissions, redirects, PHP pools and robots rules for your sites, then keeps them in place.

Start free

Black Friday & Cyber Monday

Before you go: lifetime plans are on sale

Pay once for a lifetime plan, or get 30% off the first year of any yearly plan.

Only 100 lifetime plans per plan. When they are gone, they are gone.

See the campaign

Amazing November

See the campaign →