AI crawler robots.txtGenerators

AI crawler robots.txt. Block training, keep AI search.

Each AI bot sits in one of three groups: training, AI search and fetch on request. Toggle them one by one and paste the lines. robots.txt is a request, and some operators say their user fetchers may ignore it.

Cost to you

Free

Runs in your browser. No signup, no API call, no limit per hour.

browser
runs in: none
API calls: none
account

Use it now

Use the AI crawler robots.txt generator here

It runs in your browser. Nothing you type is sent anywhere, there is no signup and no limit.

Start from

Paths to block (optional)

Empty blocks the whole site for the bots you block.

Sitemap URL (optional)

Also write an explicit Allow line for the bots you allow

Model training

Blocking these keeps future training sets from copying your pages. It does not remove you from AI answers.

GPTBot OpenAI
Collects pages to train OpenAI models.
ClaudeBot Anthropic
Collects pages to train Anthropic models.
anthropic-ai Anthropic
Older Anthropic token, no longer on Anthropic's page. Harmless to keep.
Google-Extended Google
Not a crawler. Tells Google whether Gemini may train on or ground with your pages. Google Search is not affected.
Applebot-Extended Apple
Not a crawler. Tells Apple whether its AI models may train on pages Applebot fetched.
CCBot Common Crawl
Builds the open Common Crawl archive, which many model makers train on.
Meta-ExternalAgent Meta
Collects pages for Meta AI training and product indexing.
MistralAI-Training Mistral
Collects pages to train Mistral models.
Amazonbot Amazon
Improves Amazon products and may train Amazon AI models.
Bytespider ByteDance
No operator page found to confirm what it does or whether it obeys robots.txt. Block it at your CDN if it matters.

AI search indexes

These decide whether AI search tools can find, quote and link you. Block them and you drop out of those answers.

OAI-SearchBot OpenAI
Indexes pages so ChatGPT search can show and link them.
Claude-SearchBot Anthropic
Indexes pages for search inside Claude.
PerplexityBot Perplexity
Indexes pages for Perplexity answers. Perplexity says it does not train foundation models with it.
MistralAI-Index Mistral
Indexes pages for Mistral's search.
Amzn-SearchBot Amazon
Indexes pages for Amazon search features such as Alexa. Amazon says it is not used for training.

Fetch when a user asks

These open one page because a person asked about it. Several operators say robots.txt may not stop them.

ChatGPT-User OpenAI
Opens a page when a ChatGPT user asks. OpenAI says robots.txt rules may not apply.
Claude-User Anthropic
Opens a page when a Claude user asks. Anthropic says it follows robots.txt.
Perplexity-User Perplexity
Opens a page when a Perplexity user asks. Perplexity says it generally ignores robots.txt.
Meta-ExternalFetcher Meta
Opens a page when a Meta AI user asks. Meta says it may bypass robots.txt.
Amzn-User Amazon
Opens a page when an Amazon AI user asks. Amazon says it may not follow every rule.
MistralAI-User Mistral
Opens a page when a Mistral user asks.
DuckAssistBot DuckDuckGo
Fetches pages for DuckDuckGo's AI answers. DuckDuckGo says it is not used for training.

robots.txt rules, 10 of 22 bots blocked

# AI crawler rules, made with https://looot.ai/free-tools/ai-crawler-robots-txt-generator

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: MistralAI-Training
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: Bytespider
Disallow: /

Sitemap: https://example.com/sitemap.xml

Paste these lines into the robots.txt at the root of your site, next to your existing rules. robots.txt is a request, not a lock: a bot that ignores it needs a block at your server or CDN. Bot list checked on each operator's own page on 2026-10-14. Runs in your browser, nothing is sent anywhere.

Run it from your agent

Copy one prompt, give it to your agent.

Fill in the placeholder, paste it, and the agent runs the same job with current data on your looot account and shows the cost.

You give it: allow or block for each ai crawler, optional paths and sitemap.
Billed per call, nothing while your agent is idle.
The agent shows the price before it calls.

Prompt: AI crawler robots.txt generator

Use looot to check which AI crawlers my robots.txt blocks for [your domain]. 1. Set up https://looot.ai/skill.md. 2. Run a web scrape endpoint on https://[your domain]/robots.txt. 3. List each AI bot (GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot and others) as allowed or blocked. 4. Flag any AI search bot I block by mistake.

Give this to your agent

Setup command

set up looot for AI crawler robots.txt generator - https://looot.ai/free-tools/ai-crawler-robots-txt-generator/skill.md

What comes back

What your agent gets from ai crawler robots.txt

Field names as the providers return them. The exact set varies by endpoint, and your agent reads it from the catalog.

robots.txt rules per bot
disallowed paths
sitemap line
what each bot does

Providers

Who can do this job today

Your agent picks the endpoint. These providers offer one right now, with the live price of their cheapest.

Context.dev$0.0025 / call

Cheapest priced endpoint of each provider, read from the looot catalog. Cheapest overall: $0.0025 per call.

Use it for

What people do with ai crawler robots.txt

Opt out of training
Block the training crawlers and keep your pages out of future training sets.
Stay in AI search
Keep OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed so AI answers can still cite you.
Protect private paths
Block only /account or /drafts for AI bots and leave the rest open.
Audit a client site
Show a client what each bot does before they decide.

Set up once

Three steps. Your agent does the rest.

Paste one prompt into your agent

Set up looot

Done, looot is connected.

Copy one prompt from /connect into Claude Code, Cursor, or any MCP agent. No SDK, no dashboard click-through first.

Your agent finds the right operation

Hunter$0.0265

Exa$0.0090

Diffbot$0.0032

The agent finds the operation it needs and sees the price first. It calls nothing blind.

You watch what it cost

Hunter$0.0265

Exa$0.0090

Diffbot$0.0032

Total$0.0387

Every call settles for its quoted price and lands in the ledger. You always know what ran and what it cost.

Keep going

Generators
llms.txt generator
An llms.txt file in the llmstxt.org format from your name, summary and key links, ready to download.
Free, no signupno limit
SEO
Bulk backlink checker
Referring domains, backlinks and rank for many domains in one run.
Runs a real API call$0.0240 / call
Generators
MCP config generator
The MCP server block for Claude Code, Claude Desktop, Cursor, Codex, VS Code and Windsurf, from one form.
Free, no signupno limit

FAQ

Questions about ai crawler robots.txt

Add User-agent: GPTBot followed by Disallow: /. The generator writes that block for every bot you set to Block.

No. OpenAI uses OAI-SearchBot for ChatGPT search and GPTBot for training. Block one and allow the other.

A robots.txt token, not a crawler. It tells Google whether Gemini may train on or ground answers with your pages. Google says it does not change Google Search.

The training and search crawlers on this list say they follow it. Several operators say their fetchers that open a page for a user may not, including ChatGPT-User, Perplexity-User and Meta-ExternalFetcher. Use your CDN or server to enforce a block.

Each token and what it does was checked on its operator's own page on 2026-10-14. Bytespider has no operator page we could find, so it is marked unconfirmed.

It depends on what you want. Blocking training bots keeps your text out of training sets. Blocking AI search bots also drops you from the answers that link to sources.

At the root of your domain, at https://your-domain/robots.txt. Add these lines next to your existing rules.

Yes. It runs in your browser, sends nothing and has no limit.

For agents

tool: AI crawler robots.txt generator
category: Generators
input: Allow or block for each AI crawler, optional paths and sitemap
output: robots.txt rules per bot, disallowed paths, sitemap line, what each bot does
endpoint: context-dev: context-dev-web-scrape-markdown
price: per call, read live from the catalog on the web page
web tool: runs in the visitor's browser, no API call, free and unlimited
schedule: on demand
setup: https://looot.ai/free-tools/ai-crawler-robots-txt-generator/skill.md
llms.txt: https://looot.ai/llms.txt
updated: 2026-10-14

markdown /free-tools/ai-crawler-robots-txt-generator.md

Run ai crawler robots.txt from your own agent.

Start your workspace, top up, and paste the prompt into the agent you already use. It runs the job with live data and shows the cost first.

Start your workspace See all free tools