# Web crawler vs web scraper: what each does, which you need

Web crawler vs web scraper: a crawler finds pages by following links, a scraper reads data from a page. What a web crawler is, robots.txt, and agent use.

## Summary

Web crawler vs web scraper sounds like two names for one thing, and people mix them up all the time. They are different jobs. A crawler finds pages: it starts at a URL, follows the links, and keeps a list of what it saw. A scraper reads a page you already have and pulls out the parts you want, such as a price, a title or the article text.

Most agent tasks need a scraper, not a crawler. "Read this pricing page" is a scrape. "Read every page in these docs" is a crawl followed by a scrape of each page. This post explains what a web crawler is, how a scraper differs, where robots.txt fits, and which looot job does each part.

Last checked 2026-10-16, against Google's crawler docs, RFC 9309 and looot's live catalog.

## What is a web crawler?

A web crawler is a program that discovers pages on its own by following links from one page to the next. Google's docs say a crawler, "sometimes also called a "robot" or "spider"", is "a generic term for any program" that automatically discovers and scans websites ([Google crawlers overview](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers), read 2026-10-16). Googlebot is the best known one. It crawls so that Google can index pages for search.

A crawler's output is a set of URLs, and often the raw HTML of each. It does not decide what on a page matters. That is the scraper's job.

## What is a web scraper?

A web scraper fetches a known page and extracts data from it. The input is a URL, the output is the fields you asked for: text as markdown, a table, a product price, a list of links. A scraper may fetch one page or ten thousand, but it does not go looking for new ones unless you add a crawler in front of it.

Google draws a similar line for its own tools. It describes a fetcher as acting like "a program like wget" that makes a single request for a user, and keeps fetchers separate from its crawlers.

## Web crawler vs web scraper, side by side

|  | Web crawler | Web scraper |
|---|---|---|
| Starts from | one URL or a sitemap | a list of URLs you already have |
| Does | follows links to find more pages | reads each page and extracts fields |
| Returns | URLs, often with raw HTML | structured data, text or markdown |
| Typical use | search indexes, site audits, reading a whole docs site | prices, articles, profiles, one page for an agent |
| Runs for | minutes to hours, many pages | seconds per page |

The 3-row decision for an agent:

| You have | You want | Use |
|---|---|---|
| One URL | its text | a scrape to markdown |
| A site's home page | a list of its pages | a site map, then pick |
| A docs site | every page as text | a crawl with a page limit |

## Where does robots.txt fit?

robots.txt is how a site tells crawlers which paths they may fetch. RFC 9309 defines it for "automatic clients known as crawlers", and says plainly: "These rules are not a form of access authorization" ([RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html), read 2026-10-16). It is a request, and well-behaved crawlers honor it.

A fetch a person asked for is treated differently. Google says its user-triggered fetchers "generally ignore robots.txt rules" because a user requested the fetch ([Google user-triggered fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers), read 2026-10-16). An agent reading one page you pasted is closer to that case than to a crawler. A crawl of a whole site is not, so check the site's robots.txt and terms before you start one. If you run a site, the free [robots.txt generator](https://looot.ai/free-tools/robots-txt-generator) and the [AI crawler robots.txt generator](https://looot.ai/free-tools/ai-crawler-robots-txt-generator) write the rules for you.

## Which one does an AI agent need?

Usually a scraper that returns markdown. A model reads markdown with less noise than raw HTML: no scripts, no style tags, no navigation repeated on every page. That keeps the tokens for the content. A crawler earns its place only when you do not know the URLs yet.

looot has a job for each step. Prices come from the live catalog when the page loads:

Read a page, or map a site, through looot. Prices read on 2026-10-17 from the live looot catalog. They change when a provider changes its price.

| Option | Cheapest provider | Price per call |
|---|---|---|
| Read one page as markdown | Jina AI | $0.0002 per call |
| List the URLs of a site | Tavily Search and Extract API | $0.0008 per call |

Crawl a site through looot. Prices read on 2026-10-17 from the live looot catalog. They change when a provider changes its price.

| Option | Cheapest provider | Price per page |
|---|---|---|
| Crawl a site and return each page | Olostep | $0.0018 per page |

Crawl endpoints charge per page returned, so set a page limit before you start. The [URL to markdown API page](https://looot.ai/use-cases/url-to-markdown-api) shows the scrape job's inputs and output. To try it on one page without an account, use the free [website to markdown tool](https://looot.ai/free-tools/website-to-markdown).

## What it cannot do

- looot does not crawl the web on its own or keep an index. Each crawl is one run you start, with the limits you set.
- It cannot read pages behind a login or a paywall, and it does not sign in to sites for you.
- A site can block a scraper. When a provider fails, the run says so, and with fallback on looot tries the next provider for the same job.
- A crawl that should repeat needs scheduled runs, not available yet.

## Questions

### What is the difference between a web crawler and a web scraper?

A web crawler finds pages by following links. A web scraper reads pages you already have and extracts data from them. Many tools do both: crawl to find the URLs, then scrape each one.

### What is a web crawler?

A web crawler is a program that discovers and fetches pages by following links. Google calls it a generic term for any such program, also known as a robot or spider. Googlebot is one.

### Is web scraping the same as web crawling?

No. Crawling is discovery, scraping is extraction. You can scrape one page without crawling anything, and a crawler that only lists URLs has scraped nothing.

### Does a web scraper have to follow robots.txt?

robots.txt is a request to crawlers, not access control, per RFC 9309. Well-behaved crawlers follow it. For a single page a person asked for, Google's own fetchers generally ignore it. Check the site's terms for anything larger.

### Web crawler vs web scraper: which one should an AI agent use?

Most agent tasks need a scraper that returns markdown for a known URL. Use a crawler or a site map only when the agent does not know which pages to read.

### How much does crawling a site cost?

On looot, crawl endpoints charge per page they return, and the live price is in the table above. Set a page limit and map the site first, so you crawl only what you need.

## For agents

### Crawl or scrape, the cheap way

```text
Use looot to read the docs at https://modelcontextprotocol.io as text. First set up https://looot.ai/skill.md. Search for "list the URLs of a site" and "read a page as markdown", inspect the cheapest of each, and tell me both prices. Map the site, show me the first 20 URLs, and read only the 3 I pick with job:web.scrape.markdown. Do not start a full crawl. Stop if one call would cost more than $0.02.
```