factget
Request Access
Developer hub
Console
Developers /Crawl API

Crawl a website

Crawl a site within limits as an async job; poll url (or use webhook).

POST/crawl/v1/crawl
POST example
curl --request POST \
  --url 'https://factget.dev/crawl/v1/crawl' \
  --header "Authorization: Bearer $FACTGET_CRAWL_KEY" \
  --header 'Content-Type: application/json' \
  --data '{
  "url": "https://example.com",
  "limit": 20,
  "scrapeOptions": {
    "formats": [
      "markdown"
    ]
  }
}'

Replace IDs and sample values with your own. Examples are not executed in your browser.

Authorization

Authorizationstring · header · required

Bearer Crawl API key with the required scope. Learn about authentication →

Request body

application/json
urlstringrequired

The public URL to read.

originstring | null
maxLength: 200
limitinteger

Maximum number of results or pages for this operation.

Default: 100minimum: 1maximum: 10000
maxDepthinteger | null
minimum: 0maximum: 100
maxDiscoveryDepthinteger
Default: 10minimum: 0maximum: 100
includePathsstring[]
maxItems: 50
excludePathsstring[]
maxItems: 50
allowSubdomainsboolean
Default: false
crawlEntireDomainboolean
Default: false
allowBackwardLinksboolean
Default: false
sitemapstring
Default: "include"Allowed: "include", "skip", "only"
ignoreSitemapboolean
Default: false
ignoreQueryParametersboolean
Default: false
scrapeOptionsobject

The scrape options shared by /v1/scrape, /v1/batch/scrape and a crawl's ``scrapeOptions``.

Show nested properties
originstring | null
maxLength: 200
formatsstring[]

Choose the output formats you need. Each item must be one of the allowed values.

Allowed: "markdown", "html", "rawHtml", "links", "extract"minItems: 1
onlyMainContentboolean

Return the main page content, excluding common navigation and page furniture.

Default: true
includeTagsstring[] | null

Include the matching HTML elements in the result.

maxItems: 50
excludeTagsstring[] | null

Exclude the matching HTML elements from the result.

maxItems: 50
waitForinteger

Additional time to wait for the page, in milliseconds.

Default: 0minimum: 0maximum: 30000
jsCodestring | null
maxLength: 10000
extractobject | null
Show nested properties
typestringrequired
Allowed: "css", "xpath", "regex"
schemaobject | null
webhookobject | null
Show nested properties
urlstringrequired

The public URL to read.

headersobject
metadataobject
eventsstring[]
Allowed: "started", "page", "completed", "failed"minItems: 1

Responses

200Successful Response
application/json

The published schema does not define individual response fields for this endpoint. Inspect the returned JSON in your integration.

422Validation Error
application/json
detailobject[]
Show item properties
locstring | integer[]required
msgstringrequired
typestringrequired
inputobject
ctxobject

The API schema lists the responses above. Long-running operations can also return 202; see async jobs. For application errors, see errors & retries.

FactGet developer documentationBack to API overview