Firecrawl · Web Data

Firecrawl API / Crawl multiple URLs based on options

Crawls a site from a start URL and returns its pages. limit sets the page count, up to 100; set it on every call, since Firecrawl's own default is much higher. maxDiscoveryDepth is capped at 5 and maxConcurrency at 10. Priced per page crawled, from Firecrawl's reported credits.

Private Gateway connection

Data below comes from the configured private Gateway. Provider activation remains governed by its evidence and policy gates.

unverifiedRequest shape unavailableUnverified

Published API evidence

Public metadata only. Response bodies, credentials, and internal review notes are never displayed.

No verification date is claimed. No response capture is claimed.

Request parameters

allowExternalLinks body · optional
Allows the crawler to follow links to external websites.
allowSubdomains body · optional
Allows the crawler to follow links to subdomains of the main domain.
crawlEntireDomain body · optional
Allows the crawler to follow internal links to sibling or parent URLs, not just child paths. false: Only crawls deeper (child) URLs. → e.g. /features/feature-1 → /features/feature-1/tips ✅ → Won't follow /pricing or / ❌ true: Crawls any internal links, including siblings and parents. → e.g. /features/feature-1 → /pricing, /, etc. ✅ Use true for broader internal coverage beyond nested paths.
delay body · optional
Delay in seconds between scrapes. This helps respect website rate limits.
excludePaths body · optional
URL pathname regex patterns that exclude matching URLs from the crawl. For example, if you set "excludePaths": ["blog/.*"] for the base URL firecrawl.dev, any results matching that pattern will be excluded, such as https://www.firecrawl.dev/blog/firecrawl-launch-week-1-recap.
ignoreQueryParameters body · optional
Do not re-scrape the same path with different (or none) query parameters
includePaths body · optional
URL pathname regex patterns that include matching URLs in the crawl. Only the paths that match the specified patterns will be included in the response. For example, if you set "includePaths": ["blog/.*"] for the base URL firecrawl.dev, only results matching that pattern will be included, such as https://www.firecrawl.dev/blog/firecrawl-launch-week-1-recap.
limit body · optional
Maximum number of pages to crawl. Default limit is 10000. Example: 10
maxConcurrency body · optional
Maximum number of concurrent scrapes. This parameter allows you to set a concurrency limit for this crawl. If not specified, the crawl adheres to your team's concurrency limit. Example: 10
maxDiscoveryDepth body · optional
Maximum depth to crawl based on discovery order. The root site and sitemapped pages has a discovery depth of 0. For example, if you set it to 1, and you set ignoreSitemap, you will only crawl the entered URL and all URLs that are linked on that page. Example: 2
prompt body · optional
A prompt to use to generate the crawler options (all the parameters below) from natural language. Explicitly set parameters will override the generated equivalents.
scrapeOptions body · optional
sitemap body · optional
Sitemap mode when crawling. If you set it to 'skip', the crawler will ignore the website sitemap and only crawl the entered URL and discover pages from there onwards.
url body · required
The base URL to start crawling from
webhook body · optional
A webhook specification object.
zeroDataRetention body · optional
If true, this will enable zero data retention for this crawl. To enable this feature, please contact help@firecrawl.dev

Provider docs How to run it