# AI crawler robots.txt generator: run it free, or from your agent

Robots.txt lines that allow or block each AI crawler, grouped by training, AI search and user fetches.

## Prompt

Use looot to check which AI crawlers my robots.txt blocks for [your domain]. 1. Set up https://looot.ai/skill.md. 2. Run a web scrape endpoint on https://[your domain]/robots.txt. 3. List each AI bot (GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot and others) as allowed or blocked. 4. Flag any AI search bot I block by mistake.

## What it needs

Allow or block for each AI crawler, optional paths and sitemap

## What comes back

- robots.txt rules per bot
- disallowed paths
- sitemap line
- what each bot does

## Endpoints and providers

- context-dev: context-dev-web-scrape-markdown

## What it costs

Nothing. The tool on the web page runs in the visitor's browser: no account, no API call and no limit per hour. The prompt above runs the same job from your own looot account with current data, billed per call.

## Opt out of training

Block the training crawlers and keep your pages out of future training sets.

## Stay in AI search

Keep OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed so AI answers can still cite you.

## Protect private paths

Block only /account or /drafts for AI bots and leave the rest open.

## Audit a client site

Show a client what each bot does before they decide.

## How do I block GPTBot in robots.txt?

Add User-agent: GPTBot followed by Disallow: /. The generator writes that block for every bot you set to Block.

## Does blocking GPTBot remove my site from ChatGPT search?

No. OpenAI uses OAI-SearchBot for ChatGPT search and GPTBot for training. Block one and allow the other.

## What is Google-Extended?

A robots.txt token, not a crawler. It tells Google whether Gemini may train on or ground answers with your pages. Google says it does not change Google Search.

## Do AI crawlers obey robots.txt?

The training and search crawlers on this list say they follow it. Several operators say their fetchers that open a page for a user may not, including ChatGPT-User, Perplexity-User and Meta-ExternalFetcher. Use your CDN or server to enforce a block.

## Which bot list does this use?

Each token and what it does was checked on its operator's own page on 2026-10-14. Bytespider has no operator page we could find, so it is marked unconfirmed.

## Should I block all AI bots?

It depends on what you want. Blocking training bots keeps your text out of training sets. Blocking AI search bots also drops you from the answers that link to sources.

## Where does robots.txt go?

At the root of your domain, at https://your-domain/robots.txt. Add these lines next to your existing rules.

## Is this generator free?

Yes. It runs in your browser, sends nothing and has no limit.

## Related tools

- /free-tools/llms-txt-generator
- /free-tools/backlink-checker
- /free-tools/mcp-config-generator

## Last reviewed

2026-10-14.

## Agent skill for this page

https://looot.ai/free-tools/ai-crawler-robots-txt-generator/skill.md sets an agent up for exactly the jobs on this page, with the live price of each, and offers the page's starter task once the user has picked a job and seen the price. The line to give an agent is under "Give this to your agent" below.

## For agents

### Give this to your agent

```text
set up looot for AI crawler robots.txt generator - https://looot.ai/free-tools/ai-crawler-robots-txt-generator/skill.md
```