Olostep / Crawl website (start)

Request parameters

• exclude_urls body · optional

  URL path names in glob pattern to exclude. For example: `/careers/**`. Excluded URLs will supersede included URLs.

• follow_robots_txt body · optional

  Whether to respect robots.txt rules. If set to `false`, the crawler will scrape the website regardless of robots.txt disallow directives. `true` by default.

• include_external body · optional

  Crawl first-degree external links.

• include_subdomain body · optional

  Include subdomains of the website. `false` by default.

• include_urls body · optional

  URL path patterns to include in the crawl using glob syntax. Defaults to `/**` which includes all URLs. Use patterns like `/blog/**` to crawl specific sections (e.g., only blog pages), `/products/*.html` for product pages, or multiple patterns for different sections. Supports standard glob features like * (any characters) and ** (recursive matching).

• max_depth body · optional

  Maximum depth of the crawl. Useful to extract only up to n-degree of links.

• max_pages body · required

  Maximum number of pages to crawl. Recommended for most use cases like crawling an entire website.

• scrape_options body · optional

  Controls what each individual page scrape requests from the Olostep API. All fields are optional.

• search_query body · optional

  An optional search query to find specific links and also sort the results by relevance.

• start_url body · required

  The starting point of the crawl.

• timeout body · optional

  End the crawl after n seconds with the pages completed until then. May take ~10s extra from provided timeout.

• top_n body · optional

  An optional number to only crawl the top N most relevant links on every page as per search query.

• webhook body · optional

  HTTPS URL to receive a POST request when the crawl completes. Must be a publicly accessible URL using `http://` or `https://` protocol. Cannot point to localhost or private IP addresses. See [Webhooks](/api-reference/common/webhooks) for payload format and retry behavior.