feat(api): add ETag caching, correlation ID tracing, standardized err…

listener/API_ERROR_REFERENCE.md

@@ -1,11 +1,57 @@
 # Notification API Error Reference
 
-This reference lists listener API error responses, common causes, and operator or client actions. Error bodies use JSON and generally follow:
+This reference lists listener API error responses, common causes, and operator or client actions.
+
+## Standard Error Envelope (#987)
+
+Errors produced through the shared response helpers use this envelope:
 
 ```json
-{ "error": "Human readable error message" }
+{
+  "success": false,
+  "error": {
+    "code": "INVALID_PERMISSIONS",
+    "message": "The provided credentials lack the necessary scope for this action.",
+    "details": [],
+    "correlation_id": "req-12345"
+  }
+}
 ```
 
+| Field | Type | Notes |
+|-------|------|-------|
+| `code` | string | Stable machine-readable code from the table below. **Branch on this, not on `message`.** |
+| `message` | string | Human-readable; wording may change between releases. |
+| `details` | array \| object | Validation issues or extra context. `[]` when there is nothing to add. |
+| `correlation_id` | string | Same value as the `X-Correlation-Id` response header. Quote it when reporting problems. |
+
+### Error Codes
+
+Codes are never renamed or reused once published. The source of truth is
+`src/errors/api-error-codes.ts` (`ERROR_CODE_CATALOG`).
+
+| Code | HTTP | Meaning |
+|------|------|---------|
+| `BAD_REQUEST` | 400 | Malformed request or invalid parameters. |
+| `VALIDATION_ERROR` | 400 | Body failed schema validation; `details` lists `{ field, message }` issues. |
+| `PARSE_ERROR` | 400 | Body is not valid JSON. |
+| `UNAUTHORIZED` | 401 | API key missing, unknown, revoked, or mismatched. |
+| `INVALID_PERMISSIONS` | 403 | Valid credential lacking a required scope; `details` lists `requiredScopes` / `missingScopes`. |
+| `FORBIDDEN` | 403 | Action not permitted for this caller. |
+| `NOT_FOUND` | 404 | Unknown route or resource. |
+| `METHOD_NOT_ALLOWED` | 405 | Resource does not support the method; see `Allow`. |
+| `CONFLICT` | 409 | Conflicts with current resource state. |
+| `PAYLOAD_TOO_LARGE` | 413 | Body exceeds the size limit. |
+| `UNSUPPORTED_MEDIA_TYPE` | 415 | `Content-Type` not accepted. |
+| `UNPROCESSABLE` | 422 | Well-formed but cannot be processed. |
+| `RATE_LIMITED` | 429 | Rate limit exceeded; honour `Retry-After`. |
+| `INTERNAL_ERROR` | 500 | Unexpected server failure. |
+| `SERVICE_UNAVAILABLE` | 503 | A required optional service is not configured. |
+
+> Some older endpoints documented below (e.g. `POST /api/schedule`) still
+> write ad-hoc bodies such as `{ "error": "...", "code": "..." }`. They will be
+> migrated to the standard envelope; the codes they emit are unchanged.
+
 Batch validation endpoints return structured validation results instead:
 
 ```json

listener/docs/API_HARDENING.md

@@ -0,0 +1,98 @@
+# API Caching, Tracing and Scoped Credentials
+
+Covers issues #985 (ETags), #986 (correlation IDs), #987 (error codes) and
+#988 (scoped credentials). Error codes are documented in
+[`API_ERROR_REFERENCE.md`](../API_ERROR_REFERENCE.md).
+
+## Conditional GET with ETags (#985)
+
+These endpoints return a strong `ETag` (SHA-256 of the response body) and
+`Cache-Control: private, no-cache`:
+
+- `GET /api/events`, `GET /api/events/history`
+- `GET /api/analytics` (not when `?reset=true`), `GET /api/analytics/history`
+- `GET /api/schedule/stats`, `/execution-metrics`, `/retry-distribution`, `/retry-statistics`
+
+Send the tag back in `If-None-Match`. If the data is unchanged the server
+replies `304 Not Modified` with an empty body:
+
+```bash
+etag=$(curl -s -D - -o /dev/null localhost:8787/api/events | grep -i etag | cut -d' ' -f2 | tr -d '\r')
+curl -i -H "If-None-Match: $etag" localhost:8787/api/events   # → 304
+```
+
+The tag is computed from the bytes that would be sent, so it changes as soon as
+the underlying data changes; there is no separate cache to invalidate.
+`W/"…"`, comma-separated lists and `*` are honoured.
+
+## Correlation IDs (#986)
+
+- Every response carries `X-Correlation-Id`.
+- A client-supplied `X-Correlation-Id` is reused only if it is 1–64 characters
+  of `[A-Za-z0-9._:-]` and starts with a letter or digit. Anything else
+  (whitespace, control characters, quotes, overlong values) is replaced with a
+  new UUID v4, so untrusted header content never reaches the logs.
+- The ID is held in AsyncLocalStorage for the life of the request. Every
+  `logger.*` call made while serving it gets `correlationId` / `requestId`
+  added automatically, and error envelopes include it as `correlation_id`.
+- `EventProcessingQueue` captures the ID at enqueue time and restores it when
+  the job runs. For other deferred work, wrap the callback with
+  `bindCorrelationContext()` from `src/utils/correlation-context.ts`.
+
+## Scoped API Credentials (#988)
+
+### Scopes
+
+| Scope | Grants |
+|-------|--------|
+| `events:read` | Read event data (needed with `notifications:read` for `/api/export`) |
+| `events:write` | Reserved for event write endpoints (none exist yet) |
+| `stats:read` | Reserved for statistics endpoints (currently public) |
+| `notifications:read` | `/api/notifications/history`, `/api/notifications/:id/receipts` |
+| `notifications:write` | `POST /api/notifications/import` |
+| `webhooks:manage` | Reserved for webhook management endpoints (none exist yet) |
+| `credentials:manage` | Manage credentials via `/api/credentials` |
+
+Static keys from the `apiKeys` config keep full access, so existing
+integrations keep working. Credentials issued through the API are limited to
+their scopes; a missing scope returns `403 INVALID_PERMISSIONS`.
+
+### Managing credentials
+
+All routes need `credentials:manage`. On a deployment with no static keys
+they always return `401`, so an open dev server can't be used to create the
+first credential.
+
+```bash
+# Create: the only response that ever includes apiKey
+curl -X POST localhost:8787/api/credentials \
+  -H "X-API-Key: $ADMIN_KEY" -H "Content-Type: application/json" \
+  -d '{"name":"grafana","scopes":["stats:read","events:read"]}'
+# → 201 { "success": true, "data": { "id": "…", "apiKey": "nck_<id>_<secret>", … } }
+
+curl localhost:8787/api/credentials -H "X-API-Key: $ADMIN_KEY"            # list (no secrets)
+curl localhost:8787/api/credentials/<id> -H "X-API-Key: $ADMIN_KEY"       # get (no secrets)
+curl -X DELETE localhost:8787/api/credentials/<id> -H "X-API-Key: $ADMIN_KEY"  # revoke
+```
+
+A scoped caller holding `credentials:manage` can only grant scopes it already
+holds.
+
+### Secret handling
+
+- Keys look like `nck_<16-hex id>_<43-char secret>`. The id is a lookup handle.
+  The secret is 256 bits from `crypto.randomBytes`.
+- Only an scrypt hash (N=16384, r=8, p=1, 16-byte per-key salt) is stored.
+  Comparisons use `timingSafeEqual`, and unknown ids still do the same scrypt
+  work so response timing doesn't reveal which ids exist.
+- Successful verifications are cached for 60 s under a SHA-256 of the key, so
+  scrypt doesn't run on every request. Revoking a key clears its cache entry
+  right away.
+
+### Limitations
+
+- The default `InMemoryApiCredentialStore` loses credentials when the process
+  restarts. For durable storage, implement `ApiCredentialStore` and pass it to
+  `new ApiCredentialService({ store })`.
+- Scope checks apply only to endpoints that already require authentication.
+  Public read endpoints such as `GET /api/events` stay unauthenticated.

listener/src/api/api-key-auth.ts

@@ -1,5 +1,11 @@
 import crypto from 'crypto';
 import http from 'http';
+import { ErrorCode } from '../errors/api-error-codes';
+import {
+  looksLikeScopedApiKey,
+  type ApiCredentialService,
+} from '../services/api-credential-service';
+import type { ApiScope } from '../types/api-credential';
 
 export interface ApiKeyEntry {
   key: string;
@@ -63,3 +69,113 @@ export const API_KEY_AUTH_MESSAGES: Record<'missing' | 'invalid', string> = {
   missing: 'Unauthorized: missing API key',
   invalid: 'Unauthorized: invalid API key',
 };
+
+// ---------------------------------------------------------------------------
+// Scoped authorization (#988)
+// ---------------------------------------------------------------------------
+
+export type AuthPrincipal =
+  /** No keys configured anywhere: open dev/local mode (backward compatible). */
+  | { type: 'anonymous' }
+  /** A key from the static `apiKeys` config: full access, as before scopes existed. */
+  | { type: 'legacy'; keyName?: string }
+  /** A scoped credential issued through the credentials API. */
+  | { type: 'credential'; credentialId: string; name: string; scopes: readonly ApiScope[] };
+
+export type AuthorizationResult =
+  | { allowed: true; principal: AuthPrincipal }
+  | {
+      allowed: false;
+      status: 401 | 403;
+      code: typeof ErrorCode.UNAUTHORIZED | typeof ErrorCode.INVALID_PERMISSIONS;
+      message: string;
+      details: unknown[];
+    };
+
+export interface AuthorizeOptions {
+  apiKeys?: ApiKeyEntry[];
+  credentialService?: ApiCredentialService | null;
+  /** Every listed scope must be granted. Legacy keys implicitly hold all scopes. */
+  requiredScopes?: readonly ApiScope[];
+  /**
+   * Reject even when no keys are configured. Used for credential management so
+   * an unconfigured deployment cannot be used to mint the first key.
+   */
+  requireAuthentication?: boolean;
+}
+
+export const INVALID_PERMISSIONS_MESSAGE =
+  'The provided credentials lack the necessary scope for this action.';
+
+function unauthorized(message: string): AuthorizationResult {
+  return { allowed: false, status: 401, code: ErrorCode.UNAUTHORIZED, message, details: [] };
+}
+
+/** True when `principal` holds every scope in `required`. */
+export function principalHasScopes(
+  principal: AuthPrincipal,
+  required: readonly ApiScope[],
+): boolean {
+  if (principal.type !== 'credential') return true;
+  return required.every((scope) => principal.scopes.includes(scope));
+}
+
+/**
+ * Authenticates the `X-API-Key` header and enforces endpoint scopes
+ * server-side.
+ *
+ * - `nck_…` keys are verified against the credential service and must hold
+ *   every `requiredScopes` entry, else `403 INVALID_PERMISSIONS`.
+ * - Other keys are checked against the static `apiKeys` config and keep the
+ *   full access they always had.
+ * - With no static keys and no active credentials, requests are allowed
+ *   (unless `requireAuthentication`), preserving the existing dev behaviour.
+ */
+export async function authorizeRequest(
+  req: http.IncomingMessage,
+  options: AuthorizeOptions,
+): Promise<AuthorizationResult> {
+  const required = options.requiredScopes ?? [];
+  const header = req.headers['x-api-key'];
+  if (Array.isArray(header)) return unauthorized(API_KEY_AUTH_MESSAGES.invalid);
+
+  const hasStaticKeys = Boolean(options.apiKeys && options.apiKeys.length > 0);
+  const hasCredentials = options.credentialService?.hasActiveCredentials() ?? false;
+
+  if (typeof header === 'string' && header.length > 0 && looksLikeScopedApiKey(header)) {
+    const credential = options.credentialService
+      ? await options.credentialService.verify(header)
+      : null;
+    if (!credential) return unauthorized(API_KEY_AUTH_MESSAGES.invalid);
+
+    const principal: AuthPrincipal = {
+      type: 'credential',
+      credentialId: credential.id,
+      name: credential.name,
+      scopes: credential.scopes,
+    };
+    if (!principalHasScopes(principal, required)) {
+      const missing = required.filter((scope) => !credential.scopes.includes(scope));
+      return {
+        allowed: false,
+        status: 403,
+        code: ErrorCode.INVALID_PERMISSIONS,
+        message: INVALID_PERMISSIONS_MESSAGE,
+        details: [{ requiredScopes: [...required], missingScopes: missing }],
+      };
+    }
+    return { allowed: true, principal };
+  }
+
+  if (!hasStaticKeys) {
+    if (!hasCredentials && !options.requireAuthentication) {
+      return { allowed: true, principal: { type: 'anonymous' } };
+    }
+    const presented = typeof header === 'string' && header.length > 0;
+    return unauthorized(API_KEY_AUTH_MESSAGES[presented ? 'invalid' : 'missing']);
+  }
+
+  const legacy = authenticateApiKey(req, options.apiKeys);
+  if (!legacy.authenticated) return unauthorized(API_KEY_AUTH_MESSAGES[legacy.reason]);
+  return { allowed: true, principal: { type: 'legacy', keyName: legacy.keyName } };
+}