# Robots.txt generator: run it free, or from your agent

A robots.txt with disallow paths, a sitemap line and optional rules for AI crawlers, ready to download.

## Prompt

Use looot to review the robots.txt of [domain]. 1. Set up https://looot.ai/skill.md. 2. Search the catalog for an endpoint that reads a web page and check its price before you call. 3. Read [domain]/robots.txt and list which crawlers and paths it blocks. 4. Return the file, the problems you see and a corrected version.

## What it needs

Paths to block or allow, a sitemap URL and bot choices

## What comes back

- robots.txt text
- one group per user agent
- the sitemap line
- warnings for rules that block the whole site

## Endpoints and providers

None. The tool on the web page runs in the browser and calls no endpoint.

## What it costs

Nothing. The tool on the web page runs in the visitor's browser: no account, no API call and no limit per hour. The prompt above runs the same job from your own looot account with current data, billed per call.

## Keep pages out

Stop crawlers from wasting time on admin, cart or search result pages.

## AI crawler choices

Allow or block GPTBot, ClaudeBot and other AI crawlers one by one.

## Point to the sitemap

Add the Sitemap line so every crawler finds your page list.

## Staging sites

Block a whole test site in one click before it gets indexed.

## How do I create a robots.txt file?

Pick the default rule, list the paths to block and add your sitemap URL. The file builds as you type. Download it and upload it to the root of your site as robots.txt.

## Where does robots.txt go?

At the root of the host, for example https://example.com/robots.txt. A robots.txt in a sub-folder is ignored, and each subdomain needs its own file.

## Does robots.txt stop a page from being indexed?

No. It asks crawlers not to fetch a page, but a blocked page can still be indexed from links. Use a noindex meta tag on a page that crawlers may fetch to keep it out of results.

## How do I block AI crawlers?

Set the AI crawlers you want to keep out to block. The generator writes one User-agent group with Disallow: / for each. Crawlers that respect robots.txt follow it, and others may not.

## What is the difference between Allow and Disallow?

Disallow keeps a crawler out of a path. Allow opens a path inside a blocked folder. When two rules match, Google follows the longer, more specific one.

## Should I block my CSS and JavaScript files?

No. Search engines render pages with them. Blocking them can make a page look broken to the crawler.

## Is this robots.txt generator free?

Yes. It runs in your browser, needs no account and has no limit.

## Related tools

- /free-tools/sitemap-xml-generator
- /free-tools/meta-tag-generator
- /free-tools/schema-markup-generator

## Last reviewed

2026-10-14.