Skip to content

feat(webhooks): signed delivery with retries and dead-letter queue - #569

Closed
Simultech369 wants to merge 1 commit into
StableRoute-Org:mainfrom
Simultech369:feat/552-signed-webhook-delivery
Closed

Simultech369 wants to merge 1 commit into
StableRoute-Org:mainfrom
Simultech369:feat/552-signed-webhook-delivery

Conversation

@Simultech369

Copy link
Copy Markdown

Summary

Closes #552

This PR implements an end-to-end webhook delivery pipeline featuring cryptographic HMAC-SHA256 signing, timestamp replay attack defense, bounded retries with exponential backoff on transient errors (5xx and network timeouts), and an in-memory bounded Dead-Letter Queue (DLQ) with inspection and replay endpoints.


Architecture & Implementation

1. Cryptographic HMAC Signing (src/webhooks/signer.ts)

  • Signature Format: X-Signature: t=<timestamp_ms>,v1=<hmac_sha256_hex>.
  • Replay Protection: The HMAC digest is computed over ${timestamp}.${rawPayload}, binding the timestamp cryptographically to the exact body bytes.
  • Verification Engine: verifyWebhookSignature() uses constant-time timingSafeEqual to defeat timing attacks and enforces a configurable clock drift tolerance window (default 300s).
  • Tamper Resistance: Modifying any property in the delivered JSON payload immediately invalidates the signature digest.

2. Delivery Pipeline with Exponential Backoff (src/webhooks/delivery.ts)

  • Retry Mechanics:
    • 2xx Success: Delivery acknowledged immediately.
    • 4xx Client Error: Non-retryable error (e.g. 400 Bad Request, 401 Unauthorized, 404 Not Found); stops retries immediately and dead-letters.
    • 5xx Server Error / Network Timeout: Transient error; retries up to maxRetries (default 3) using exponential backoff: delayMs = initialBackoffMs * Math.pow(backoffFactor, attempt - 1).
  • Payload Bound: Payloads exceeding maxPayloadBytes (default 64 KB) are rejected and dead-lettered prior to network transmission.
  • Pluggable HTTP Client: Accepts dependency-injected fetchFn for fast, hermetic testing without network sockets.

3. Dead-Letter Queue (DLQ) Engine (src/webhooks/deadLetterQueue.ts)

  • In-Memory Store: Tracks failed events along with failure reason, last HTTP status code, total attempts, and timestamp.
  • Bounded Eviction: Fixed cap (default 1,000 records) using oldest-first eviction to prevent unbounded memory growth.
  • Replay Capabilities: Replaying dead-lettered items re-invokes delivery. On success, the record is cleanly purged from the DLQ; on continued failure, the record's attempt count and error metadata are updated in place without creating duplicate DLQ records.

4. REST API & Lifecycle Integration (src/index.ts & src/stores.ts)

  • Secret Lifecycle:
    • POST /api/v1/webhooks: Accepts optional caller secret (8–256 chars) or generates a secure whsec_<hex> secret. Secret is returned in the 201 Created response.
    • GET /api/v1/webhooks, GET /api/v1/webhooks/:id, PATCH /api/v1/webhooks/:id: Secrets are strictly stripped and never leaked in list or read responses.
  • DLQ Endpoints:
    • GET /api/v1/webhooks/dead-letter: Paginated list of dead-letter records with total count and optional webhookId filter.
    • GET /api/v1/webhooks/dead-letter/:id: Read a specific dead-letter record.
    • POST /api/v1/webhooks/dead-letter/:id/replay: Replays an event from the DLQ.
    • DELETE /api/v1/webhooks/dead-letter/:id: Purges a record from the DLQ.
  • OpenAPI Documentation: Added OpenAPI 3.0.3 paths for all dead-letter endpoints and updated webhook schemas.
  • Documentation: Updated docs/webhook-delivery.md with complete architecture specifications, header tables, and verification examples.

Edge Case Coverage

Every edge case specified in Issue #552 is implemented and tested:

  • Successful delivery -> signature verifies: X-Signature and X-Signature-Timestamp headers generated and verified by subscriber.
  • 5xx then success -> retried and delivered: Retries transient 500/503 and network timeouts with exponential backoff and succeeds on subsequent attempts.
  • Retries exhausted -> lands in DLQ with reason: Exhausted attempts move event to DLQ with detailed lastError and lastStatusCode.
  • Replay from DLQ -> re-attempts delivery: Calling replay re-invokes delivery; successful delivery removes item from queue.
  • Tampered payload -> signature check fails: Modifying payload contents causes signature verification to reject with false.

Verification & Test Results

  1. Unit Tests:
    • src/__tests__/webhookSignerAndDlq.unit.test.ts: 13/13 tests pass
  2. Integration Tests:
    • src/__tests__/webhookDelivery.test.ts: 23/23 tests pass
    • src/__tests__/webhooks.test.ts: 31/31 tests pass
    • src/__tests__/openapi.test.ts: 3/3 tests pass (zero route drift)
  3. Full Repository Regression:
    • npm test: 37/37 test suites passed, 1021/1021 tests passed
  4. Code Quality:
    • npm run lint: 0 errors
    • npm run build: tsc compiles clean with 0 errors

@mikewheeleer

Copy link
Copy Markdown
Contributor

Closing as part of a repository reset; new issues will be opened shortly. Thanks for contributing!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

signed webhook delivery with retries and a dead-letter queue

2 participants