Docs / Website Integration

Custom Webhook Setup

Use this guide when your site does not have a first-party Crawlbaby integration and you want to send page change events from a CMS or custom publish hook.

Typical platforms: Strapi, custom CMSes, build hooks, and internal publishing systems.

Site Values

Showing placeholder values. Open this guide from a site settings page to pre-fill the endpoint and site values.

Request Contract

• Method: POST
• Headers: Content-Type: application/json and either X-Crawlbaby-Event-Token or Authorization: Bearer <token>
• Body: JSON object using the fields below

Event Types

• updated: the live page changed and should be verified by universal monitoring.
• created: a new live page exists and should be verified like an update.
• deleted: the page was removed or unpublished and should be verified as missing.

If eventType is omitted, Crawlbaby treats the event as updated.

Examples

Complete Payload

{
  "url": "https://example.com/blog/my-post",
  "eventType": "updated",
  "occurredAt": "2026-04-05T12:34:56Z",
  "source": "custom-cms",
  "sourceEventId": "custom-cms:page_123:publish:v57",
  "checksum": "sha256:9b0e4d3a...",
  "payload": {
    "entityType": "page",
    "entityId": "page_123",
    "contentType": "landingPage",
    "locale": "en-US",
    "slug": "my-post",
    "operation": "publish",
    "workspaceId": "workspace_abc",
    "environmentId": "production",
    "version": 57
  }
}

Deleted Page

{
  "url": "https://example.com/blog/my-post",
  "eventType": "deleted",
  "occurredAt": "2026-04-05T12:40:00Z",
  "source": "custom-cms",
  "sourceEventId": "custom-cms:page_123:delete:v58"
}

Validation-Only Test

{
  "url": "https://example.com/blog/my-post",
  "eventType": "updated",
  "source": "webhook",
  "sourceEventId": "test-1710153600",
  "test": true
}

When test is true, Crawlbaby validates the token, site, and URL, records the successful test for connection status, and returns 200 without enqueueing a monitoring probe or snapshot.

Missing Or Wrong Values

• If url is missing or malformed, the route returns invalid_payload or invalid_url.
• If the URL host does not match the site base URL host, the route returns url_outside_site.
• If occurredAt is omitted, Crawlbaby uses the current server time.
• If source is omitted, Crawlbaby uses webhook.
• If sourceEventId is omitted, the event is still accepted, but replayed deliveries are less likely to deduplicate cleanly.
• If the webhook token is missing or wrong, the route returns missing_event_token or unauthorized.

Responses

Accepted events return 202 after they are queued for universal-monitoring verification. A snapshot is created only when that verification confirms a material change.

{
  "ok": true,
  "accepted": true,
  "deduplicated": false,
  "eventId": "7a8d3f2a-4b68-4c1d-90ae-4c59d8d9f111",
  "normalizedUrl": "https://example.com/blog/my-post",
  "events": [
    {
      "id": "7a8d3f2a-4b68-4c1d-90ae-4c59d8d9f111",
      "normalizedUrl": "https://example.com/blog/my-post",
      "deduplicated": false
    }
  ],
  "processing": {
    "status": "queued",
    "reason": "queued_for_verification",
    "snapshotId": null,
    "mode": "monitoring_probe"
  }
}

Implementation Notes

• Prefer sending the canonical live page URL, not an internal CMS preview URL.
• Prefer stable provider event ids in sourceEventId, such as publish event ids, revision ids, or entry-version ids.
• Keep payload compact. Include tracing metadata, not a full CMS document body.
• For providers with body transformations, map native webhook fields into this generic contract before posting to Crawlbaby.
• Retry timeouts and 5xx responses with exponential backoff. Correct 400 and 401 responses before retrying.