Skip to content

feat(crawl): get notified by webhook when a crawl starts, completes, stops or fails - #72

Merged
StefanoGuerrini merged 1 commit into
mainfrom
feat/crawl-webhooks
Oct 30, 2026
Merged

StefanoGuerrini merged 1 commit into
mainfrom
feat/crawl-webhooks

Conversation

@zenrows-dev-pool

@zenrows-dev-pool zenrows-dev-pool Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Crawls can now notify an endpoint when they start, complete, stop or fail, so callers no longer have to poll. This adds the webhook option on crawl creation and the webhook field on every returned crawl.

What the task asked for

Bring this surface in line with the Crawl API, which now accepts a webhook on POST /v1/crawls and sends signed zenrows.webhook/v1 events. ZenRowsCrawlClient.create(..., webhook=...) takes a url, optional events and optional metadata. Every returned crawl carries Crawl.webhook. CrawlWebhookEvent models an event body, and zenrows.crawl exposes verify_webhook_signature to check the X-Signature header.

How the check confirmed it

  • make check (ty, ruff check, ruff format) clean; uv run pytest 311 passed, 7 of them new.
  • Acceptance suite: 29 passed. Check 25 (Webhook) created a crawl with a webhook and read back {"url":"https://example.com/zenrows-hook","events":["crawl.completed"],"metadata":{"k":"v"}}.
  • Public-text check: clean on the branch name, the commit and every added line.

How to try it

from zenrows.crawl import ZenRowsCrawlClient

client = ZenRowsCrawlClient(api_key="YOUR_KEY")
crawl = client.create(
    "https://www.scrapingcourse.com/ecommerce/",
    max_pages=1,
    webhook={"url": "https://your-endpoint.example/hook", "events": ["crawl.completed"]},
)
print(crawl.webhook)

The crawl prints the webhook back. The account needs a webhook signing key, created in the dashboard under Settings, API Keys.

Provenance

Follows the module's existing TypedDict request options and open enums for response values. The signature check re-exports the helper the library already ships for Batch, since the scheme and key are the same.

Review

One pass, the author's own checks above. No independent review ran.

Human must verify:

  • The names read naturally next to the rest of the crawl API in this package.
  • Merge only after the Crawl reference page lists webhook.

Deviations

  • README.md (outside the crawl package) gains a short "Get notified by webhook" section, because this repository documents crawl options there.
  • Unit tests live in tests/test_crawl_client.py, outside the crawl package, where the suite runs them.

Record owed

None.

…stops or fails

The Crawl API now accepts a webhook on create and sends signed events to it,
so a caller no longer has to poll to learn when a crawl ends. create() takes
webhook= (url, events, metadata), every returned crawl carries the webhook it
runs with as Crawl.webhook, CrawlWebhookEvent models the event body, and
zenrows.crawl exposes verify_webhook_signature() to check an event's
X-Signature header with the account's webhook signing key.
@StefanoGuerrini
StefanoGuerrini marked this pull request as ready for review October 30, 2026 10:37
@StefanoGuerrini
StefanoGuerrini self-requested a review October 30, 2026 10:49
@StefanoGuerrini
StefanoGuerrini merged commit 42a6e37 into main Oct 30, 2026
3 checks passed
@StefanoGuerrini
StefanoGuerrini deleted the feat/crawl-webhooks branch October 30, 2026 10:50
@github-actions github-actions Bot mentioned this pull request Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant