Collect web data with Crawl
Choose the right operation for a page, a list of URLs, or an entire website.
Choose an operation
| Your task | Operation |
|---|---|
| Read one page | Scrape → |
| Read a list of pages | Batch scrape → |
| Discover URLs on a site | Map → |
| Collect a whole site | Crawl → |
Output formats
Choose markdown, HTML, raw HTML, links, or extraction. Extraction supports CSS, XPath, and regular-expression rules. Screenshots are not an output format.
Use the endpoint reference for supported options, required fields, and limits.
Source access
Crawl renders pages in Chrome and respects robots.txt, matching its rules against the token FactgetCrawl. It blocks private-network targets and does not solve CAPTCHAs. Blocked or failed targets return explicit errors.