> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tilion.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Scrape and crawl

> Read one page and return it, with no session to create, drive or release. The page is read on a browser session of its own that is destroyed with the request.

## POST /v1/scrape

Read one page and return it, with no session to create, drive or release. The page is read on a browser session of its own that is destroyed with the request.

Auth: Bearer API key.

Request:

| Field | Type | Default | Meaning |
| - | - | - | - |
| `url` | string | required | Public http or https URL, up to 4096 characters. |
| `formats` | array of `markdown`, `html`, `text`, `links`, `metadata` | `["markdown"]` | What to return. Must not be empty. |
| `links` | `inline`, `footnote`, `none` | `inline` | How hyperlinks render in the markdown. |
| `max_chars` | integer, 100 to 10000000 | null | Cut markdown and text at a block boundary; `truncated` reports a cut. |
| `include_images` | boolean | true | `false` drops images from the markdown. |
| `wait_ms` | integer, 0 to 60000 | 8000 | Ceiling for the page to finish loading before it is read. |
| `schema`, `prompt` | object, string | null | Structured extraction; not currently available (501). |
| `residential`, `region` | boolean, string | null | Accepted; the page is read from a session chosen for the site. |

Request:

```json theme={"dark"}
{"url": "https://example.com", "formats": ["markdown", "text", "links", "metadata"]}
```

Response:

| Field | Type | Meaning |
| - | - | - |
| `url` | string | Where the page landed (the same value as `final_url`, kept for older callers). |
| `final_url` | string | Where the page landed. |
| `requested_url` | string | The URL you sent. `requested_url != final_url` is a redirect. |
| `title` | string or null | Page title. |
| `status` | string | `ok` or one of the extract outcomes (`source_empty`, `challenged`, `hard_blocked`, `requires_login`, `rate_limited`). |
| `blocked_reason` | string | Present when `status` is not `ok`. |
| `self_healed` | boolean | Present when known; true when the page was reached after a heal. |
| `markdown`, `text`, `html`, `links`, `metadata` | per `formats` | As on `/v1/extract`. |
| `truncated` | boolean | Present when `max_chars` was given. |
| `data` | object | Present with `schema` or `prompt`, where enabled. |

The answer (text fields cut):

```json theme={"dark"}
{
 "url": "https://example.com/",
 "final_url": "https://example.com/",
 "title": "Example Domain",
 "status": "ok",
 "requested_url": "https://example.com",
 "self_healed": false,
 "markdown": "This domain is for use in documentation examples without needing permission. ...",
 "text": "This domain is for use in documentation examples without needing permission. ...",
 "links": [
  "https://iana.org/help/example-domains"
 ],
 "metadata": {
  "title": "Example Domain",
  "description": null,
  "canonical": null,
  "lang": "en",
  "og_title": null,
  "og_description": null,
  "og_image": null,
  "og_site_name": null,
  "published_time": null,
  "modified_time": null,
  "author": null,
  "word_count": 133,
  "reading_time_min": 1
 }
}
```

Redirect detection: compare `requested_url` and `final_url` yourself; scrape has no `redirected` flag.

Errors:

| Status | `error` or message | When |
| - | - | - |
| 400 | `unsafe url: ...` | The URL is not public http or https. |
| 402 | `insufficient_credits`, `quota_exceeded` | No credit, or the plan allowance is used up. |
| 422 | validation error | Bad field, or an empty `formats` list. |
| 429 | rate limit, `concurrency_limit` | The per-key drive window, or no room for the session the scrape creates. |
| 502 | `scrape_failed` | The render raised; the body carries `retryable: true`. |
| 504 | `deadline_exceeded` | The call ran past the 57 second limit. |

Notes:

* Timing: 1 to 2 seconds for a plain page, more for a heavy one. The whole call must finish inside 57 seconds; there is no `async_mode`.
* Billing: the session charge (2) plus 1 action plus egress per MB of markdown plus the session's browser time, prorated. A non-`ok` page is still charged the session.

## POST /v1/crawl

Walk a site from a starting URL and return every page reached, in one call on one session of its own.

Auth: Bearer API key.

Request:

| Field | Type | Default | Meaning |
| - | - | - | - |
| `url` | string | required | Where to start. Public http or https. |
| `max_pages` | integer, 1 to 100 | 10 | Stop after this many delivered pages. |
| `depth` | integer, 1 to 5 | 2 | How many link hops from the start page to follow. |
| `include_patterns` | array of up to 20 strings | `[]` | When set, a link is followed only if its URL contains one of them (plain substring match). |
| `exclude_patterns` | array of up to 20 strings | `[]` | A link whose URL contains one of them is skipped. |
| `formats` | array of `markdown`, `html`, `text` | `["markdown"]` | What each page carries. Must not be empty. |
| `residential`, `region` | boolean, string | null | Accepted. |

Rules: only HTTP or HTTPS links on the same host are followed; links can switch between those schemes, but a different subdomain is another site. Fragments are dropped, `/index.html` and a trailing slash are folded into the same page, and each page contributes up to 300 links, breadth first. A page that fails to load or comes back blocked is skipped and does not count towards `max_pages`. Pages are read one at a time.

Response:

| Field | Type | Meaning |
| - | - | - |
| `url` | string | The start URL as sent. |
| `count` | integer | Number of pages returned. |
| `pages` | array | One entry per delivered page: `{url, final_url, title, status: "ok"}` plus `markdown`, `text` (plain text) and `html` as asked for in `formats`. |

`{"url": "https://www.iana.org/help/example-domains", "max_pages": 3}` (markdown cut):

```json theme={"dark"}
{
 "url": "https://www.iana.org/help/example-domains",
 "count": 3,
 "pages": [
  {
   "url": "https://www.iana.org/help/example-domains",
   "final_url": "https://www.iana.org/help/example-domains",
   "title": "Example Domains",
   "status": "ok",
   "markdown": "# Example Domains\n\nA number of domains such as `example.com` and `example.org` are maint..."
  },
  {
   "url": "https://www.iana.org/",
   "final_url": "https://www.iana.org/",
   "title": "Internet Assigned Numbers Authority",
   "status": "ok",
   "markdown": "## [Domain Names](https://www.iana.org/domains)\n\nManagement of the DNS Root Zone (assign..."
  },
  {
   "url": "https://www.iana.org/domains",
   "final_url": "https://www.iana.org/domains",
   "title": "Domain Name Services",
   "status": "ok",
   "markdown": "# Domain Name Services\n\nWe operate and maintain a number of key aspects of the DNS, incl..."
  }
 ]
}
```

A crawl of `https://example.com` with `max_pages: 3` returns `count: 1`, because its only link points at `iana.org`, which is another origin.

Errors: as `/v1/scrape` (400 unsafe url, 402, 422 on a bad field or empty `formats`, 429, 504 `deadline_exceeded` past 57 seconds).

Notes:

* Timing: about a second per page on a fast site. The whole crawl must finish inside the 57 second limit and there is no `async_mode`, so keep `max_pages` to what the site can serve in under a minute; the pages read before the cut are lost with the 504.
* Billing: the session charge (2) plus 1 credit per page plus egress per MB of markdown plus prorated browser time. A crawl is always charged at least one action.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.