Web crawler vs web scraper: what each does, which you need

Web crawler vs web scraper sounds like two names for one thing, and people mix them up all the time. They are different jobs. A crawler finds pages: it starts at a URL, follows the links, and keeps a list of what it saw. A scraper reads a page you already have and pulls out the parts you want, such as a price, a title or the article text.

Most agent tasks need a scraper, not a crawler. "Read this pricing page" is a scrape. "Read every page in these docs" is a crawl followed by a scrape of each page. This post explains what a web crawler is, how a scraper differs, where robots.txt fits, and which looot job does each part.

Last checked 2026-10-16, against Google's crawler docs, RFC 9309 and looot's live catalog.

What is a web crawler?

A web crawler is a program that discovers pages on its own by following links from one page to the next. Google's docs say a crawler, "sometimes also called a "robot" or "spider"", is "a generic term for any program" that automatically discovers and scans websites (Google crawlers overview, read 2026-10-16). Googlebot is the best known one. It crawls so that Google can index pages for search.

A crawler's output is a set of URLs, and often the raw HTML of each. It does not decide what on a page matters. That is the scraper's job.

What is a web scraper?

A web scraper fetches a known page and extracts data from it. The input is a URL, the output is the fields you asked for: text as markdown, a table, a product price, a list of links. A scraper may fetch one page or ten thousand, but it does not go looking for new ones unless you add a crawler in front of it.

Google draws a similar line for its own tools. It describes a fetcher as acting like "a program like wget" that makes a single request for a user, and keeps fetchers separate from its crawlers.

Web crawler vs web scraper, side by side

The 3-row decision for an agent:

Where does robots.txt fit?

robots.txt is how a site tells crawlers which paths they may fetch. RFC 9309 defines it for "automatic clients known as crawlers", and says plainly: "These rules are not a form of access authorization" (RFC 9309, read 2026-10-16). It is a request, and well-behaved crawlers honor it.

A fetch a person asked for is treated differently. Google says its user-triggered fetchers "generally ignore robots.txt rules" because a user requested the fetch (Google user-triggered fetchers, read 2026-10-16). An agent reading one page you pasted is closer to that case than to a crawler. A crawl of a whole site is not, so check the site's robots.txt and terms before you start one. If you run a site, the free robots.txt generator and the AI crawler robots.txt generator write the rules for you.

Which one does an AI agent need?

Usually a scraper that returns markdown. A model reads markdown with less noise than raw HTML: no scripts, no style tags, no navigation repeated on every page. That keeps the tokens for the content. A crawler earns its place only when you do not know the URLs yet.

looot has a job for each step. Prices come from the live catalog when the page loads:

Crawl endpoints charge per page returned, so set a page limit before you start. The URL to markdown API page shows the scrape job's inputs and output. To try it on one page without an account, use the free website to markdown tool.

What it cannot do

• looot does not crawl the web on its own or keep an index. Each crawl is one run you start, with the limits you set.
• It cannot read pages behind a login or a paywall, and it does not sign in to sites for you.
• A site can block a scraper. When a provider fails, the run says so, and with fallback on looot tries the next provider for the same job.
• A crawl that should repeat needs scheduled runs, not available yet.