Repository navigation
fix: sign outbound webhooks and make delivery idempotent #733
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weβll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -5,15 +5,18 @@ The Notification & Event Delivery API serves as the central communication backbo | |
|
|
||
| ## Core Components | ||
|
|
||
| ### 0. Outbound Webhook Delivery (PREWAVE-BE-059) | ||
| Outbound webhooks are signed, versioned, and idempotent. Every delivery is bound to canonical finalized contract events and to chain/release/artifact provenance; delivery fails closed on stale, ambiguous, unauthorized, or incomplete state. | ||
|
|
||
| ### 1. Event Queue | ||
| Built on **BullMQ (Redis)**, ensuring fault-tolerant, asynchronous processing of events. | ||
| - **Queue Name**: `notifications` | ||
| - * *Queue Name**: `notifications` | ||
| - **Retry Mechanism**: Exponential backoff (max 5 attempts, delays starting at 2s, 4s, 8s, 16s). | ||
|
|
||
| ### 2. Preference Engine | ||
| Users define their notification preferences (`NotificationPreference` entity): | ||
| - **Channels**: `IN_APP`, `EMAIL`, `PUSH`, `WEBHOOK` | ||
| - **Categories**: `CLAIM`, `VERIFICATION`, `DISPUTEa, `GOVERNANCE`, `REWARD`, etc. | ||
| - **Categories**: `CLAIM`, `VERIFICATION`, `DISPUTE`, `GOVERNANCE`, `REWARD`, etc. | ||
| - **Quiet Hours & Digest Mode**: Settings stored for intelligent delivery windows. | ||
|
|
||
| #### Quiet-Hours Timezone Handling | ||
|
|
@@ -81,3 +84,42 @@ Placeholders are filled from event payloads treated as untrusted input. Values a | |
|
|
||
| ## Future Extensibility | ||
| The modular architecture permits drop-in integrations for SMS, Slack, Discord, Telegram, and AI-generated summaries without modifying the core queue processor. | ||
|
|
||
| ## Outbound Webhook Contract (PREWAVE-BE-059) | ||
|
|
||
| ### Envelope | ||
| All outbound webhook payloads use a versioned envelope. The envelope version is stable per event type and bumped only on breaking changes. | ||
| - `version`: envelope schema version (e.g. `v1`). | ||
| - `event`: canonical finalized contract event type. | ||
| - `deliveryId`: stable, deterministic identifier for one logical event (see Idempotency). | ||
| - `timestamp`: Unix epoch seconds at signing time. | ||
| - `provenance`: `{ chainId, releaseId, artifactDigest }` binding the event to chain/release/artifact provenance. | ||
| - `data`: event-specific payload. | ||
|
|
||
| ### Signature, Timestamp, Delivery ID | ||
| - Every request carries `X-Webhook-Signature`, `X-Webhook-Timestamp`, and `X-Webhook-Delivery-Id` headers. | ||
| - Signature is an HMAC over `version + "." + timestamp + "." + deliveryId + "." + rawBody` using the endpoint's active signing key. | ||
| - Signing keys are versioned and rotatable; the active key ID is sent in `X-Webhook-Key-Id`. Receivers must support overlapping key rotation windows. | ||
| - Receivers verify integrity and reject replays outside the configured timestamp tolerance. | ||
|
Comment on lines
+90
to
+103
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ποΈ Data Integrity & Integration | π Major | ποΈ Heavy lift Align the documented webhook contract with the implementation. Receivers that follow this document will fail verification against
Lines 9 and 96 also claim binding to finalized events and provenance. The code never checks finality or provenance. Implement these items or remove the claims. The linked issue requires provenance binding and fail-closed behavior. Also applies to: 105-107 π§° Toolsπͺ markdownlint-cli2 (0.23.3)[warning] 90-90: Headings should be surrounded by blank lines (MD022, blanks-around-headings) [warning] 99-99: Headings should be surrounded by blank lines (MD022, blanks-around-headings) π€ Prompt for AI Agents |
||
|
|
||
| ### Idempotency | ||
| - `deliveryId` is derived deterministically from the canonical event identity (event type, chain, release, artifact, and event key) so one logical event always has the same delivery identity across retries. | ||
| - Receivers are expected to deduplicate on `deliveryId`. | ||
|
|
||
| ### Retry Policy | ||
| - Retries are bounded: max 5 attempts with exponential backoff (2s, 4s, 8s, 16s). | ||
| - Retries reuse the same `deliveryId`; only `timestamp` and signature are refreshed. | ||
| - Retries stop on 2xx. 4xx (except 408/429) are treated as terminal. 5xx, timeouts, and redirects are retried within the bound. | ||
| - Redirects are not followed; a redirect response is treated as a delivery failure. | ||
|
|
||
| ### Endpoint Validation (SSRF) | ||
| - Endpoints must be HTTPS and resolve to public addresses; loopback, link-local, private, and metadata ranges are rejected. | ||
| - DNS is re-resolved at delivery time to prevent rebinding; unsafe targets are rejected and recorded. | ||
| - Endpoint changes require re-validation before delivery resumes. | ||
|
|
||
| ### Audit History | ||
| - Every attempt records delivery ID, endpoint, status, latency, response class, key ID, and outcome. | ||
| - Audit history is retained for verification and is queryable per endpoint and per delivery ID. | ||
|
|
||
| ### Required Tests | ||
| - Receiver timeout, 2xx/4xx/5xx handling, redirect rejection, duplicate delivery, key rotation, SSRF target rejection, and bounded retry behavior. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -64,6 +64,8 @@ import { IndexerModule } from './indexer/indexer.module'; | |
| import { V2OutboxModule } from './v2/outbox/v2-outbox.module'; | ||
| import { CredentialRotationModule } from './security/credential-rotation.module'; | ||
| import { OperationsModule } from './operations/operations.module'; | ||
| import { WebhooksModule } from './webhooks/webhooks.module'; | ||
| import { NotificationsModule as OutboundWebhooksNotificationsModule } from './notifications/notifications.module'; | ||
|
Comment on lines
+67
to
+68
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. π― Functional Correctness | π Major | β‘ Quick win Fix Line 68. It imports the wrong module and leaves it unused.
Proposed fix-import { NotificationsModule as OutboundWebhooksNotificationsModule } from './notifications/notifications.module';
+import { OutboundWebhookModule } from './notifications/outbound-webhook.module';Then add π§° Toolsπͺ ESLint[error] 68-68: 'OutboundWebhooksNotificationsModule' is defined but never used. ( π€ Prompt for AI AgentsSource: Linters/SAST tools |
||
| import { AbuseSignalsModule } from './abuse-signals/abuse-signals.module'; | ||
| import { AbuseSignalInterceptor } from './abuse-signals/abuse-signal.interceptor'; | ||
| import { WebhooksModule } from './webhooks/webhooks.module'; | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,82 @@ | ||
| import { | ||
| Entity, | ||
| PrimaryGeneratedColumn, | ||
| Column, | ||
| CreateDateColumn, | ||
| UpdateDateColumn, | ||
| Index, | ||
| Unique, | ||
| } from 'typeorm'; | ||
|
|
||
| export enum WebhookDeliveryStatus { | ||
| PENDING = 'PENDING', | ||
| SUCCEEDED = 'SUCCEEDED', | ||
| FAILED = 'FAILED', | ||
| EXHAUSTED = 'EXHAUSTED', | ||
| } | ||
|
|
||
| /** | ||
| * Per-attempt audit record for an outbound webhook delivery. | ||
| * | ||
| * The combination of `deliveryId` and `attempt` is unique, and `deliveryId` | ||
| * is derived deterministically from the logical event so retries and duplicate | ||
| * enqueues share a stable identity. | ||
| */ | ||
| @Entity('webhook_deliveries') | ||
| @Unique(['deliveryId', 'attempt']) | ||
| @Index(['endpointId']) | ||
| @Index(['status']) | ||
| export class WebhookDeliveryEntity { | ||
| @PrimaryGeneratedColumn('uuid') | ||
| id: string; | ||
|
|
||
| /** Stable identity for the logical event (shared across retries). */ | ||
| @Column({ type: 'varchar', length: 128 }) | ||
| deliveryId: string; | ||
|
|
||
| @Column({ type: 'varchar', length: 128 }) | ||
| endpointId: string; | ||
|
|
||
| @Column({ type: 'varchar', length: 255 }) | ||
| eventType: string; | ||
|
|
||
| /** Attempt number starting at 1. */ | ||
| @Column({ type: 'int' }) | ||
| attempt: number; | ||
|
|
||
| @Column({ type: 'varchar', default: WebhookDeliveryStatus.PENDING }) | ||
| status: WebhookDeliveryStatus; | ||
|
|
||
| @Column({ type: 'int', nullable: true }) | ||
| responseStatus: number | null; | ||
|
|
||
| @Column({ type: 'text', nullable: true }) | ||
| responseBody: string | null; | ||
|
|
||
| @Column({ type: 'text', nullable: true }) | ||
| errorMessage: string | null; | ||
|
|
||
| /** Hmac signature sent with this attempt (for audit). */ | ||
| @Column({ type: 'varchar', length: 255, nullable: true }) | ||
| signature: string | null; | ||
|
|
||
| @Column({ type: 'int', nullable: true }) | ||
| secretVersion: number | null; | ||
|
|
||
| /** Timestamp embedded in the signed envelope (Unix seconds). */ | ||
| @Column({ type: 'bigint', nullable: true }) | ||
| timestamp: string | null; | ||
|
|
||
| @Column({ type: 'int', nullable: true }) | ||
| durationMs: number | null; | ||
|
|
||
| /** Next scheduled retry (if any). */ | ||
| @Column({ type: 'timestamp', nullable: true }) | ||
| nextRetryAt: Date | null; | ||
|
|
||
| @CreateDateColumn() | ||
| createdAt: Date; | ||
|
|
||
| @UpdateDateColumn() | ||
| updatedAt: Date; | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,79 @@ | ||
| import { | ||
| Entity, | ||
| PrimaryGeneratedColumn, | ||
| Column, | ||
| CreateDateColumn, | ||
| UpdateDateColumn, | ||
| Index, | ||
| } from 'typeorm'; | ||
|
|
||
| export enum WebhookEndpointStatus { | ||
| ACTIVE = 'ACTIVE', | ||
| DISABLED = 'DISABLED', | ||
| REVOKED = 'REVOKED', | ||
| } | ||
|
|
||
| /** | ||
| * Registered outbound webhook endpoint. | ||
| * | ||
| * Holds the signing secret and the current/previous key versions so that key | ||
| * rotation can be performed without invalidating in-flight deliveries. The | ||
| * `secret` is never returned to clients after creation. | ||
| */ | ||
| @Entity('webhook_endpoints') | ||
| @Index(['status']) | ||
| export class WebhookEndpointEntity { | ||
| @PrimaryGeneratedColumn('uuid') | ||
| id: string; | ||
|
|
||
| @Column({ type: 'varchar', length: 255 }) | ||
| url: string; | ||
|
|
||
| /** Owner (user/org) that registered the endpoint. */ | ||
| @Column({ type: 'varchar', nullable: true }) | ||
| ownerId: string | null; | ||
|
|
||
| /** Current signing secret (HMAC-SHA256). */ | ||
| @Column({ type: 'varchar', length: 255 }) | ||
| secret: string; | ||
|
|
||
| /** Monotonically increasing key version for rotation. */ | ||
| @Column({ type: 'int', default: 1 }) | ||
| secretVersion: number; | ||
|
|
||
| /** Previous secret retained during rotation grace window. */ | ||
| @Column({ type: 'varchar', length: 255, nullable: true }) | ||
| previousSecret: string | null; | ||
|
|
||
| @Column({ type: 'int', nullable: true }) | ||
| previousSecretVersion: number | null; | ||
|
Comment on lines
+36
to
+49
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. π Security & Privacy | π Major | ποΈ Heavy lift Encrypt The entity stores both HMAC signing secrets as plaintext Based on learnings: "When persisting sensitive secrets... flag storage of plaintext values. Instead require encryption at rest using a random IV per encryption operation." The path instructions also prioritize "secret handling". π€ Prompt for AI AgentsSources: Path instructions, Learnings |
||
|
|
||
| /** When the previous secret stops being accepted. */ | ||
| @Column({ type: 'timestamp', nullable: true }) | ||
| previousSecretExpiresAt: Date | null; | ||
|
|
||
| @Column({ type: 'varchar', default: WebhookEndpointStatus.ACTIVE }) | ||
| status: WebhookEndpointStatus; | ||
|
|
||
| /** Event types this endpoint subscribes to. Empty means all. */ | ||
| @Column({ type: 'simple-array', array: true, default: '{}' }) | ||
| eventTypes: string[]; | ||
|
|
||
| /** Maximum number of delivery attempts before giving up. */ | ||
| @Column({ type: 'int', default: 5 }) | ||
| maxAttempts: number; | ||
|
|
||
| /** Request timeout in milliseconds. */ | ||
| @Column({ type: 'int', default: 10000 }) | ||
| timeoutMs: number; | ||
|
|
||
| /** Optional static Headers sent with every delivery. */ | ||
| @Column({ type: 'jsonb', nullable: true }) | ||
| headers: Record<string, string> | null; | ||
|
|
||
| @CreateDateColumn() | ||
| createdAt: Date; | ||
|
|
||
| @UpdateDateColumn() | ||
| updatedAt: Date; | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| import { Module } from '@nestjs/common'; | ||
| import { TypeOrmModule } from '@nestjs/typeorm'; | ||
| import { OutboundWebhookService } from './outbound-webhook.service'; | ||
| import { WebhookEndpointEntity } from './entities/webhook-endpoint.entity'; | ||
| import { WebhookDeliveryEntity } from './entities/webhook-delivery.entity'; | ||
|
|
||
| @Module({ | ||
| imports: [TypeOrmModule.forFeature([WebhookEndpointEntity, WebhookDeliveryEntity])], | ||
| providers: [OutboundWebhookService], | ||
| exports: [OutboundWebhookService], | ||
| }) | ||
| export class OutboundWebhookModule {} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
π Maintainability & Code Quality | π‘ Minor | β‘ Quick win
Fix the broken bold markup.
- * *Queue Name**renders as a nested list item with literal asterisks.π Committable suggestion
π€ Prompt for AI Agents