Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions deploy/monitoring/prometheus/alert-rules.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -335,3 +335,42 @@ groups:
annotations:
summary: 'Sustained rate limiting across route groups'
description: '{{ $value }} route groups currently reporting blocked requests. Distinguishes an ongoing event from a historical total.'

- name: neurowealth_support
interval: 60s
rules:
- alert: SupportTicketSlaBreached
expr: sum(support_tickets_sla_breached) > 0
for: 5m
labels:
severity: warning
annotations:
summary: 'Support tickets are past their response SLA'
description: '{{ $value }} open tickets have breached their response SLA. See docs/SUPPORT_TICKETS.md.'

- alert: SupportUrgentTicketSlaBreached
expr: support_tickets_sla_breached{priority=~"URGENT|HIGH"} > 0
for: 1m
labels:
severity: critical
annotations:
summary: 'High-priority support ticket past response SLA'
description: '{{ $value }} {{ $labels.priority }} tickets have breached their response SLA.'

- alert: SupportTicketsStale
expr: support_tickets_stale > 0
for: 15m
labels:
severity: warning
annotations:
summary: 'Support tickets untouched beyond the stale threshold'
description: '{{ $value }} open tickets have had no activity for SUPPORT_STALE_HOURS.'

- alert: ProtocolRiskMetadataStale
expr: risk_metadata_stale_entries > 0
for: 5m
labels:
severity: critical
annotations:
summary: 'Protocol risk metadata is past its review date'
description: '{{ $value }} protocol risk metadata entries are stale. See docs/PROTOCOL_RISK_SCORING.md.'
12 changes: 6 additions & 6 deletions docs/API_KEYS.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,19 +76,19 @@ New keys are **read-only by default**. Write scopes must be explicitly requested
| `metrics:read` | read | enforced | — | Read operational metrics and dashboards. | — | `GET /api/v1/admin/jobs/dashboard`, `GET /api/v1/admin/stellar/metrics` |
| `outbox:read` | read | enforced | — | Read the durable outbox. | — | `GET /api/v1/admin/outbox`, `GET /api/v1/admin/outbox/:id`, `GET /api/v1/admin/outbox/stats` |
| `read` | read | enforced | — | General read access to admin resources. | — | `GET /api/v1/admin/erasure/preview/:userId`, `GET /api/v1/admin/erasure/requests`, `GET /api/v1/admin/referrals/review` |
| `referrals:read` | read | enforced | — | Read flagged referral conversions. | — | `GET /api/v1/admin/referrals/flagged` |
| `referrals:read` | read | enforced | — | Read flagged referral conversions. | — | `GET /api/v1/admin/referrals/flagged`, `GET /api/v1/admin/referrals/fraud/calibration`, `GET /api/v1/admin/referrals/fraud/queue` |
| `reserves:read` | read | enforced | — | Read reserve and sponsorship state. | — | `GET /api/v1/admin/reserves` |
| `risk-metadata:read` | read | declared-only | — | Read protocol risk metadata. | No route requires this scope today — the risk-metadata review workflow is administered from src/config/riskMetadataAdmin.ts, not through the API. Declared so a key can be provisioned ahead of a route that needs it. | — |
| `risk-metadata:read` | read | enforced | — | Read protocol risk metadata. | `GET /api/v1/admin/risk-metadata` lists entries with a stale flag. | `GET /api/v1/admin/risk-metadata` |
| `sessions:read` | read | enforced | — | Read user sessions. | — | `GET /api/v1/admin/users/:id/sessions` |
| `support:read` | read | enforced | — | Read support tickets. | — | `GET /api/v1/admin/support/tickets`, `GET /api/v1/admin/support/tickets/:id` |
| `treasury:read` | read | enforced | — | Read treasury sweep policies. | — | `GET /api/v1/admin/treasury/policies` |
| `treasury:read` | read | enforced | — | Read treasury sweep policies. | — | `GET /api/v1/admin/treasury/policies`, `POST /api/v1/admin/treasury/sweep/preview` |
| `approvals:write` | write | enforced | — | Cancel pending approval requests. | — | `POST /api/v1/admin/approvals/:id/cancel` |
| `sessions:write` | write | enforced | — | Revoke user sessions. | — | `POST /api/v1/admin/users/:id/sessions/revoke-all` |
| `support:write` | write | enforced | — | Reply to and update support tickets. | — | `PATCH /api/v1/admin/support/tickets/:id`, `POST /api/v1/admin/support/tickets/:id/reply` |
| `write` | write | enforced | — | General write access to admin resources. | — | `POST /api/v1/admin/erasure/requests`, `POST /api/v1/admin/referrals/review/:id/approve`, `POST /api/v1/admin/referrals/review/:id/reject` |
| `outbox:write` | money | enforced | — | Force-retry and cancel queued outbox operations. | — | `POST /api/v1/admin/outbox/:id/cancel`, `POST /api/v1/admin/outbox/:id/retry` |
| `referrals:write` | money | enforced | — | Approve or reject flagged referral conversions. | — | `POST /api/v1/admin/referrals/:id/review` |
| `treasury:write` | money | enforced | — | Change treasury sweep policies and run a sweep. | Sweep policies and signer rotations are written to the admin audit log. | `POST /api/v1/admin/treasury/emergency-sweep`, `POST /api/v1/admin/treasury/policies`, `POST /api/v1/admin/treasury/signer-rotations`, `POST /api/v1/admin/treasury/signer-rotations/:id/finalize` |
| `referrals:write` | money | enforced | — | Approve or reject flagged referral conversions. | — | `POST /api/v1/admin/referrals/:id/review`, `POST /api/v1/admin/referrals/fraud/bulk-review` |
| `treasury:write` | money | enforced | — | Change treasury sweep policies and run a sweep. | Sweep policies and signer rotations are written to the admin audit log. | `POST /api/v1/admin/treasury/emergency-sweep`, `POST /api/v1/admin/treasury/policies`, `POST /api/v1/admin/treasury/policies/:id/enable`, `POST /api/v1/admin/treasury/signer-rotations`, `POST /api/v1/admin/treasury/signer-rotations/:id/finalize` |
| `wallet` | money | declared-only | — | Custodial wallet operations. | No route requires this scope today — the wallet rotation-status route is gated by keys:read. Declared so a key can be provisioned ahead of a route that needs it. | — |
| `erasure:write` | destructive | enforced | — | Create, approve, reject and execute erasure requests. | Execution requires super — approval and execution are deliberately separated so one operator cannot both authorise and carry out a deletion. | `POST /api/v1/admin/erasure` |
| `keys:write` | destructive | enforced | — | Create, rotate and revoke admin API keys. | Minting an admin key is a takeover vector — every use is written to the admin audit log. | `DELETE /api/v1/admin/keys/:id`, `POST /api/v1/admin/keys`, `POST /api/v1/admin/keys/:id/rotate` |
Expand All @@ -101,7 +101,7 @@ New keys are **read-only by default**. Write scopes must be explicitly requested
| `fiat:write` | operational | enforced | — | Change fiat provider failover state. | — | `POST /api/v1/admin/fiat/providers/:name/failover` |
| `flags:write` | operational | enforced | — | Manage feature flags, staged rollouts and emergency rollbacks. | — | `POST /api/v1/admin/feature-flags/:key/reset`, `POST /api/v1/admin/feature-flags/:key/rollback`, `POST /api/v1/admin/feature-flags/:key/staged-rollout`, `POST /api/v1/admin/feature-flags/emergency-disable-all`, `PUT /api/v1/admin/feature-flags/:key` |
| `messages:write` | operational | enforced | — | Retry and recover failed message deliveries. | — | `DELETE /api/v1/admin/messages/:id`, `POST /api/v1/admin/messages/:id/retry`, `POST /api/v1/admin/messages/retry-dead-letters` |
| `risk-metadata:write` | operational | declared-only | — | Review and publish protocol risk metadata. | No route requires this scope today — the risk-metadata review workflow is administered from src/config/riskMetadataAdmin.ts, not through the API. Declared so a key can be provisioned ahead of a route that needs it. | — |
| `risk-metadata:write` | operational | enforced | — | Review and publish protocol risk metadata. | `PUT /api/v1/admin/risk-metadata/:protocolName` upserts an entry; each change writes a history row and an admin audit log row. | `PUT /api/v1/admin/risk-metadata/:protocolName` |
<!-- END GENERATED: admin-scope-catalog -->

### Withdrawal guardrails
Expand Down
20 changes: 20 additions & 0 deletions docs/PROTOCOL_RISK_SCORING.md
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,26 @@ from it automatically.
> placeholders, all marked `dataConfidence: UNVERIFIED` — honest about the
> fact that they have never actually been reviewed against a source.

## Metadata refresh, staleness alerting and admin updates (#621)

Sourcing expectation: every entry needs an `auditReference`, and `dataConfidence: VERIFIED` needs a
`sourceUrl`. `nextReviewDueAt` is the refresh deadline; re-review each protocol before it passes
(recommended cadence: every 90 days, or immediately after an audit or incident).

- Admin API: `GET /api/v1/admin/risk-metadata` (scope `risk-metadata:read`) and
`PUT /api/v1/admin/risk-metadata/:protocolName` (scope `risk-metadata:write`). The upsert validates
input, requires a future `nextReviewDueAt`, and writes the entry, an append-only
`ProtocolRiskMetadataHistory` row and an `AdminAuditLog` row (`RISK_METADATA_UPSERT`, with
before/after) in one transaction.
- `src/jobs/riskMetadataStaleness.ts` runs every `PROTOCOL_RISK_STALENESS_INTERVAL_MS` (default 1h).
When any entry is past `nextReviewDueAt` it sets `risk_metadata_stale_entries` and emits a
`critical` alert through `src/services/alerting.ts` (PagerDuty when `PAGERDUTY_ROUTING_KEY` is set;
repeat alerts are rate-limited by the alerting cooldown). Prometheus rule
`ProtocolRiskMetadataStale` mirrors it.
- Stale protocols are degraded visibly: `GET /api/protocols/risk` returns `riskMetadataStale: true`,
a `score` reduced by `PROTOCOL_RISK_STALE_SCORE_PENALTY` (default 20, floor 0) and the unmodified
`rawScore`. Risk-ceiling checks in the rebalancing router use the degraded score.

## Out of scope (v1)

- Automated on-chain audit-status verification — `auditStatus` is curated.
Expand Down
25 changes: 25 additions & 0 deletions docs/REFERRAL_PROGRAM.md
Original file line number Diff line number Diff line change
Expand Up @@ -163,3 +163,28 @@ Signup attribution is **not** a route — it is the `referralCode` field on
(deduplicated per conversion + leg).
- Because payouts are agent-signed and platform-funded, ensure the agent
account is funded for the configured reward asset.

## Fraud review queue and calibration (#619)

Admin endpoints (mounted at `/api/v1/admin/referrals/fraud`):

- `GET /queue` (scope `referrals:read`) lists scored conversions. Filters: `state`
(`pending` default, `approved`, `rejected`, `all`), `scoreMin`/`scoreMax`, `flag`, `from`/`to`;
paginated with `page` and `pageSize` (max 100); highest score first.
- `POST /bulk-review` (scope `referrals:write`) body `{ ids, decision: approve|reject, reason }`
(up to 100 ids; `reason` required for reject). Each conversion is decided and written to
`AdminAuditLog` (`REFERRAL_BULK_APPROVE`/`REFERRAL_BULK_REJECT`) in its own transaction; ids that
are unknown or not pending come back in `skipped`.
- `GET /calibration?month=YYYY-MM` (scope `referrals:read`) returns the same report as
`npx ts-node scripts/referral-calibration.ts [YYYY-MM]`.

Calibration report: score distribution in 20-point bands for conversions created in the month,
and, for conversions reviewed in the month, the false-positive rate
(`approved / reviewed`, since a flagged conversion that a human approves was a false positive) and a
suggested `REFERRAL_FRAUD_REVIEW_SCORE`: the lowest observed score at which rejections make up at
least `REFERRAL_FRAUD_TARGET_PRECISION` (default 0.7) of reviews at or above it.

Thresholds and weights are environment-driven, not hard-coded: `REFERRAL_FRAUD_BLOCK_SCORE` (100),
`REFERRAL_FRAUD_REVIEW_SCORE` (80), `REFERRAL_FRAUD_WEIGHT_{DUPLICATE_WALLET,DUPLICATE_EMAIL,
DUPLICATE_PHONE,SAME_IP,VELOCITY,RAPID_ACTIVATION}`, `REFERRAL_FRAUD_VELOCITY_COUNT` (5) and
`REFERRAL_FRAUD_RAPID_ACTIVATION_COUNT` (3). Defaults reproduce the previous behaviour.
25 changes: 25 additions & 0 deletions docs/SUPPORT_TICKETS.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,3 +29,28 @@ The General-Purpose Support Ticket System provides users with a structured, trac
- `POST /api/v1/admin/support/tickets/:id/reply` - Admin reply or internal note

Admin reads require support:read; updates and replies require support:write. Status transitions are enforced and status/assignment/priority changes commit with an audit record. SLA defaults are LOW 72h, MEDIUM 24h, HIGH 8h and URGENT 2h. Override these with SUPPORT_SLA_<PRIORITY>_HOURS. Breaches are computed from creation time and exclude resolved/closed tickets. Internal notes are filtered by the database on user reads. User replies to RESOLVED or AWAITING_USER move the ticket to IN_PROGRESS.

## SLA, transitions and alerting (#618)

Priorities map to the usual P-levels: `URGENT` = P0, `HIGH` = P1, `MEDIUM` = P2, `LOW` = P3.

| Priority | Response SLA (default) | Override |
|---|---|---|
| URGENT (P0) | 2h | `SUPPORT_SLA_URGENT_HOURS` |
| HIGH (P1) | 8h | `SUPPORT_SLA_HIGH_HOURS` |
| MEDIUM (P2) | 24h | `SUPPORT_SLA_MEDIUM_HOURS` |
| LOW (P3) | 72h | `SUPPORT_SLA_LOW_HOURS` |

- `firstResponseAt` is set once, on the first non-internal admin reply. The SLA is met when
`firstResponseAt <= createdAt + SLA`. A ticket with no response is breached once the deadline
passes and it is not yet `RESOLVED`/`CLOSED`. `resolvedAt` is set when the ticket reaches `RESOLVED`.
- Allowed transitions (anything else returns `409`):
`OPEN -> IN_PROGRESS | AWAITING_USER | RESOLVED`, `IN_PROGRESS -> AWAITING_USER | RESOLVED`,
`AWAITING_USER -> IN_PROGRESS | RESOLVED`, `RESOLVED -> IN_PROGRESS (reopen) | CLOSED`, `CLOSED` is terminal.
- `src/jobs/supportTicketSla.ts` runs every `SUPPORT_SLA_CHECK_INTERVAL_MS` (default 5 min). It sets
`slaBreached`, publishes `support_tickets_sla_breached{priority}` and `support_tickets_stale`, and
alerts through `alertingService` (critical for URGENT/HIGH breaches).
- A ticket is stale when it is open (not `AWAITING_USER`) and untouched for `SUPPORT_STALE_HOURS`
(default 24).
- Prometheus rules `SupportTicketSlaBreached`, `SupportUrgentTicketSlaBreached` and
`SupportTicketsStale` live in `deploy/monitoring/prometheus/alert-rules.yaml`.
20 changes: 20 additions & 0 deletions docs/TREASURY.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,3 +49,23 @@ Emergency and cadence sweeps on the same signer account never race: op
submission is claimed and serialized per `signerPublicKey` by the existing
outbox dispatcher (`src/outbox/signerLock.ts`), the same mechanism every
other money-moving outbox op already relies on.

## Sweep preview and enablement runbook (#620)

New policy versions start with `sweepsEnabled = false`: the cadence job ignores them and
`executeSweep` refuses to run. Creating a new version therefore pauses sweeps for that tier pair
until an operator re-enables it.

1. `POST /api/admin/treasury/policies` creates the new policy version (scope `treasury:write`).
2. `POST /api/admin/treasury/sweep/preview` (scope `treasury:read`) returns every sweep the active
policies would plan, with `policyEnabled` and `requiresApproval` per item. It reads balances but
creates no `TreasurySweep`, `MultisigEnvelope` or outbox op, so nothing is sent to the chain.
3. Review the plans. Then `POST /api/admin/treasury/policies/:id/enable` with
`{ "confirmation": "SWEEP" }` (scope `treasury:write`). Any other text is rejected. The enable is
recorded with `enabledBy`/`enabledAt` and in the admin audit log.
4. Only after step 3 does the cadence job plan sweeps and enqueue `TREASURY_SWEEP` outbox ops.
5. A failed sweep emits a `critical` alert on component `treasury-sweep` (PagerDuty/Slack when
configured) and rethrows. Emergency sweeps are unaffected by `sweepsEnabled`.

Existing active policies also default to disabled after migration `20261020090000`; enable each one
after previewing.
Loading