Skip to content

feat(crawl): get notified by webhook when a crawl starts, completes, stops or fails - #63

Merged
StefanoGuerrini merged 1 commit into
mainfrom
feat/crawl-webhooks
Oct 30, 2026
Merged

StefanoGuerrini merged 1 commit into
mainfrom
feat/crawl-webhooks

Conversation

@zenrows-dev-pool

@zenrows-dev-pool zenrows-dev-pool Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Crawls can now notify an endpoint when they start, complete, stop or fail, so callers no longer have to poll. This adds the webhook option on crawl creation and the webhook field on every returned crawl.

What the task asked for

Bring this surface in line with the Crawl API, which now accepts a webhook on POST /v1/crawls and sends signed zenrows.webhook/v1 events. create() takes webhook?: CrawlWebhookParams (url, optional events, optional metadata). Crawl gains webhook?: CrawlWebhook | null. New types CrawlWebhookEventType and CrawlWebhookEvent, and a helper verifyCrawlWebhookSignature that checks the X-Signature header.

How the check confirmed it

  • pnpm check (biome) clean; pnpm test 514 passed, 7 of them new; tsc --noEmit clean; pnpm build succeeds.
  • Acceptance suite: 29 passed, 1 skipped (23, no page fetched yet). Check 24 (Webhook) read back {"url":"https://example.com/zenrows-hook","events":["crawl.completed"],"metadata":{"k":"v"}}.
  • Public-text check: clean on the branch name, the commit and every added line.

How to try it

import { ZenRowsCrawlClient } from "zenrows";

const client = new ZenRowsCrawlClient({ apiKey: "YOUR_KEY" });
const crawl = await client.create("https://www.scrapingcourse.com/ecommerce/", {
  maxPages: 1,
  webhook: { url: "https://your-endpoint.example/hook", events: ["crawl.completed"] },
});
console.log(crawl.webhook);

The crawl prints the webhook back. The account needs a webhook signing key, created in the dashboard under Settings, API Keys.

Provenance

Follows the module's existing option and response types. verifyCrawlWebhookSignature calls the Batch helper the library already ships, since the scheme and key are the same.

Review

One pass, the author's own checks above. No independent review ran.

Human must verify:

  • The names read naturally next to the rest of the crawl API in this package.
  • Merge only after the Crawl reference page lists webhook.

Deviations

  • Unit tests are a new file, tests/crawl-webhook.test.ts, outside src/crawl, because the test runner only reads tests/.
  • acceptance/crawl/surface/typecheck.ts now uses webhook, so the type check covers it.

Record owed

None.

…stops or fails

The Crawl API now accepts a webhook on create and sends signed events to it,
so a caller no longer has to poll to learn when a crawl ends. create() takes
a webhook (url, events, metadata), every returned crawl carries the webhook
it runs with, CrawlWebhookEvent types the event body, and
verifyCrawlWebhookSignature() checks an event's X-Signature header with the
account's webhook signing key.
@StefanoGuerrini
StefanoGuerrini self-requested a review October 30, 2026 10:50
@StefanoGuerrini
StefanoGuerrini marked this pull request as ready for review October 30, 2026 10:51
@StefanoGuerrini
StefanoGuerrini merged commit 114f53a into main Oct 30, 2026
3 checks passed
@StefanoGuerrini
StefanoGuerrini deleted the feat/crawl-webhooks branch October 30, 2026 10:51
@github-actions github-actions Bot mentioned this pull request Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant