[mikelmonie]

Summary

Webhook diagnostics, redelivery budget fix, bounded workers, and auth failure visibility

What was solved

#883 — Webhook delivery rows record only status and attempts, not the receiver's response, so a failed delivery can't be diagnosed

Persist webhook delivery diagnostics (receiver HTTP status, classified error reason, latency, optional truncated body) on webhook_deliveries, populate them across all three send paths via a shared signed-send helper, expose them in the delivery JSON and OpenAPI, and include delivery_id in delivery logs.

Addressed:

• Changed: README.md
• Add additive, idempotent migration in db::migrate (following the existing pragma_table_info pattern) adding nullable last_response_status INTEGER, last_error TEXT, and last_duration_ms INTEGER columns to webhook_deliveries; optionally a truncated (≤1 KiB) response body column gated by a config flag.
• Classify failures into short reasons (timeout, connect, tls, dns, blocked_by_ssrf_guard, http_status) with no raw internal error strings stored in last_error.
• Populate the new fields in all three send paths: dispatch, redrive_one, and redeliver_webhook; extract a shared send_signed(...) -> AttemptOutcome helper to remove duplicated request-building code.

#884 — Manual redelivery consumes the automatic retry budget again (regression of #235), and has no audit log, no metrics, and can overwrite a delivered record

Fix payments::redeliver_webhook so manual redelivery no longer consumes the automatic redrive budget (increment manual_attempts/last_attempt via a new db::record_manual_attempt instead of bumping attempts), never downgrades a delivered row to failed (even with ?force=true), emits an audit = true, action = "webhook.redeliver" log line using the currently-ignored ConnectInfo/headers, records delivered/failed outcomes and latency in state.webhook_metrics, and restores the #235 regression test proving N manual redeliveries leave attempts unchanged and the redrive worker still picks the row up.

Addressed:

• Changed: README.md
• Increment manual_attempts and last_attempt on manual redelivery; leave attempts untouched so list_redrivable_deliveries still selects the row
• Add db::record_manual_attempt rather than reusing update_webhook_delivery for the manual path
• Never downgrade a delivered delivery to failed, including when ?force=true; preserve the delivered status and surface the failed forced attempt in the response/log

#885 — Expiry sweeper and webhook redrive load unbounded result sets and process them serially, so a backlog after downtime stalls them

Bound the expiry sweeper and webhook redrive workers: add real batch-size config (EXPIRY_BATCH_SIZE, WEBHOOK_REDRIVE_BATCH_SIZE), make expire_overdue and list_redrivable_deliveries use LIMIT, loop in bounded batches with shutdown checks, expire a batch in one UPDATE ... RETURNING statement, enqueue expiry webhooks as outbox delivery rows instead of inline dispatch, and claim deliveries (in_flight + lease) before inline attempts to prevent duplicate concurrent sends.

Addressed:

• Changed: src/expiry.rs
• Add EXPIRY_BATCH_SIZE and WEBHOOK_REDRIVE_BATCH_SIZE config values (defaults matching .env.example, e.g. 500) and read them in the workers
• Change db::expire_overdue to expire a bounded batch in a single UPDATE ... WHERE id IN (SELECT ... LIMIT ?) AND status IN (...) RETURNING ... statement
• Change db::list_redrivable_deliveries to apply LIMIT ? so it never returns more than the batch size

#886 — Failed X-Admin-Secret attempts are neither logged nor counted, and admin key listing gets the 5× read quota, so brute force goes unnoticed

Add warn-level logging and Prometheus counters for failed admin/metrics authentication, and route all methods under /merchants (including GET) to the strict merchants rate-limit bucket so admin-secret brute force is visible and not given 5× quota.

Addressed:

• Changed: README.md
• Log every admin auth failure at warn with source_ip, request_id, path, method, and reason (missing/mismatch/disabled); never log the supplied secret value.
• Log every metrics auth failure at warn with the same fields and reason.
• Add stellargate_admin_auth_failures_total{reason} and stellargate_metrics_auth_failures_total{reason} counters incremented on each failure.

Changes

• README.md (modify)
• src/expiry.rs (modify)

Approach

1. Webhook delivery rows record only status and attempts, not the receiver's response, so a failed delivery can't be diagnosed #883 — Webhook delivery rows record only status and attempts, not the receiver's response, so a failed delivery can't be diagnosed (Changed: README.md)
2. Manual redelivery consumes the automatic retry budget again (regression of #235), and has no audit log, no metrics, and can overwrite a delivered record #884 — Manual redelivery consumes the automatic retry budget again (regression of Manual redelivery consumes the redrive budget and can permanently disable automatic retries #235), and has no audit log, no metrics, and can overwrite a delivered record (Changed: README.md)
3. Expiry sweeper and webhook redrive load unbounded result sets and process them serially, so a backlog after downtime stalls them #885 — Expiry sweeper and webhook redrive load unbounded result sets and process them serially, so a backlog after downtime stalls them (Changed: src/expiry.rs)
4. Failed X-Admin-Secret attempts are neither logged nor counted, and admin key listing gets the 5× read quota, so brute force goes unnoticed #886 — Failed X-Admin-Secret attempts are neither logged nor counted, and admin key listing gets the 5× read quota, so brute force goes unnoticed (Changed: README.md)

Issues

Closes #883
Closes #884
Closes #885
Closes #886

[drips-wave]

@mikelmonie Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits