Scrape and crawl

POST /v1/scrape

Read one page and return it, with no session to create, drive or release. The page is read on a browser session of its own that is destroyed with the request. Auth: Bearer API key. Request:

Field	Type	Default	Meaning
`url`	string	required	Public http or https URL, up to 4096 characters.
`formats`	array of `markdown`, `html`, `text`, `links`, `metadata`	`["markdown"]`	What to return. Must not be empty.
`links`	`inline`, `footnote`, `none`	`inline`	How hyperlinks render in the markdown.
`max_chars`	integer, 100 to 10000000	null	Cut markdown and text at a block boundary; `truncated` reports a cut.
`include_images`	boolean	true	`false` drops images from the markdown.
`wait_ms`	integer, 0 to 60000	8000	Ceiling for the page to finish loading before it is read.
`schema`, `prompt`	object, string	null	Structured extraction; not currently available (501).
`residential`, `region`	boolean, string	null	Accepted; the page is read from a session chosen for the site.

Request:

{"url": "https://example.com", "formats": ["markdown", "text", "links", "metadata"]}

Response:

Field	Type	Meaning
`url`	string	Where the page landed (the same value as `final_url`, kept for older callers).
`final_url`	string	Where the page landed.
`requested_url`	string	The URL you sent. `requested_url != final_url` is a redirect.
`title`	string or null	Page title.
`status`	string	`ok` or one of the extract outcomes (`source_empty`, `challenged`, `hard_blocked`, `requires_login`, `rate_limited`).
`blocked_reason`	string	Present when `status` is not `ok`.
`self_healed`	boolean	Present when known; true when the page was reached after a heal.
`markdown`, `text`, `html`, `links`, `metadata`	per `formats`	As on `/v1/extract`.
`truncated`	boolean	Present when `max_chars` was given.
`data`	object	Present with `schema` or `prompt`, where enabled.

The answer (text fields cut):

{
 "url": "https://example.com/",
 "final_url": "https://example.com/",
 "title": "Example Domain",
 "status": "ok",
 "requested_url": "https://example.com",
 "self_healed": false,
 "markdown": "This domain is for use in documentation examples without needing permission. ...",
 "text": "This domain is for use in documentation examples without needing permission. ...",
 "links": [
  "https://iana.org/help/example-domains"
 ],
 "metadata": {
  "title": "Example Domain",
  "description": null,
  "canonical": null,
  "lang": "en",
  "og_title": null,
  "og_description": null,
  "og_image": null,
  "og_site_name": null,
  "published_time": null,
  "modified_time": null,
  "author": null,
  "word_count": 133,
  "reading_time_min": 1
 }
}

Redirect detection: compare requested_url and final_url yourself; scrape has no redirected flag. Errors:

Status	`error` or message	When
400	`unsafe url: ...`	The URL is not public http or https.
402	`insufficient_credits`, `quota_exceeded`	No credit, or the plan allowance is used up.
422	validation error	Bad field, or an empty `formats` list.
429	rate limit, `concurrency_limit`	The per-key drive window, or no room for the session the scrape creates.
502	`scrape_failed`	The render raised; the body carries `retryable: true`.
504	`deadline_exceeded`	The call ran past the 57 second limit.

Notes:

Timing: 1 to 2 seconds for a plain page, more for a heavy one. The whole call must finish inside 57 seconds; there is no async_mode.
Billing: the session charge (2) plus 1 action plus egress per MB of markdown plus the session’s browser time, prorated. A non-ok page is still charged the session.

POST /v1/crawl

Walk a site from a starting URL and return every page reached, in one call on one session of its own. Auth: Bearer API key. Request:

Field	Type	Default	Meaning
`url`	string	required	Where to start. Public http or https.
`max_pages`	integer, 1 to 100	10	Stop after this many delivered pages.
`depth`	integer, 1 to 5	2	How many link hops from the start page to follow.
`include_patterns`	array of up to 20 strings	`[]`	When set, a link is followed only if its URL contains one of them (plain substring match).
`exclude_patterns`	array of up to 20 strings	`[]`	A link whose URL contains one of them is skipped.
`formats`	array of `markdown`, `html`, `text`	`["markdown"]`	What each page carries. Must not be empty.
`residential`, `region`	boolean, string	null	Accepted.

Rules: only HTTP or HTTPS links on the same host are followed; links can switch between those schemes, but a different subdomain is another site. Fragments are dropped, /index.html and a trailing slash are folded into the same page, and each page contributes up to 300 links, breadth first. A page that fails to load or comes back blocked is skipped and does not count towards max_pages. Pages are read one at a time. Response:

Field	Type	Meaning
`url`	string	The start URL as sent.
`count`	integer	Number of pages returned.
`pages`	array	One entry per delivered page: `{url, final_url, title, status: "ok"}` plus `markdown`, `text` (plain text) and `html` as asked for in `formats`.

{"url": "https://www.iana.org/help/example-domains", "max_pages": 3} (markdown cut):

{
 "url": "https://www.iana.org/help/example-domains",
 "count": 3,
 "pages": [
  {
   "url": "https://www.iana.org/help/example-domains",
   "final_url": "https://www.iana.org/help/example-domains",
   "title": "Example Domains",
   "status": "ok",
   "markdown": "# Example Domains\n\nA number of domains such as `example.com` and `example.org` are maint..."
  },
  {
   "url": "https://www.iana.org/",
   "final_url": "https://www.iana.org/",
   "title": "Internet Assigned Numbers Authority",
   "status": "ok",
   "markdown": "## [Domain Names](https://www.iana.org/domains)\n\nManagement of the DNS Root Zone (assign..."
  },
  {
   "url": "https://www.iana.org/domains",
   "final_url": "https://www.iana.org/domains",
   "title": "Domain Name Services",
   "status": "ok",
   "markdown": "# Domain Name Services\n\nWe operate and maintain a number of key aspects of the DNS, incl..."
  }
 ]
}

A crawl of https://example.com with max_pages: 3 returns count: 1, because its only link points at iana.org, which is another origin. Errors: as /v1/scrape (400 unsafe url, 402, 422 on a bad field or empty formats, 429, 504 deadline_exceeded past 57 seconds). Notes:

Timing: about a second per page on a fast site. The whole crawl must finish inside the 57 second limit and there is no async_mode, so keep max_pages to what the site can serve in under a minute; the pages read before the cut are lost with the 504.
Billing: the session charge (2) plus 1 credit per page plus egress per MB of markdown plus prorated browser time. A crawl is always charged at least one action.

Get started

Account

Sessions

Reading pages

Automation

MCP

Scrape and crawl

POST /v1/scrape

POST /v1/crawl

​POST /v1/scrape

​POST /v1/crawl

POST /v1/scrape

POST /v1/crawl