Skip to content

Add enterprise audit log with scoped visibility and local E2E harness - #140

Merged
shrsv merged 8 commits into
masterfrom
lr-audit
Oct 26, 2026
Merged

shrsv merged 8 commits into
masterfrom
lr-audit

Conversation

@shrsv

@shrsv shrsv commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This branch adds an enterprise audit log to LiveReview and a local end-to-end harness that verifies it with real roles, real API keys, and a real lrc review.

The audit module is a framework primitive, not a one-off page:

  • append-only, monthly-partitioned audit_events
  • automatic HTTP middleware capture plus semantic route specs
  • fail-closed audit writes for privileged changes
  • centralized tenant/role scoping for every audit read
  • retention, monthly archives, hash-chain integrity, and daily anchors
  • AI/model invocation records with prompt SHA-256 + size (never raw prompts)
  • audit-only API keys for SIEM pull
  • /reports/audit UI for browsing, filtering, live tail, exports, retention, and integrity

It also adds a safe local E2E harness so the audit behavior can be exercised against a real generated owner/member/super-admin matrix without touching the developer's ~/.lrc.toml.

Design

flowchart LR
  A[HTTP mutation] --> B[Echo audit middleware]
  B --> C{Route has audit.Spec?}
  C -- yes --> D[Semantic event]
  C -- no --> E[Generic HTTP event]
  D --> F[Sanitize details]
  E --> F
  F --> G[Recorder]
  G --> H[(audit_events<br/>monthly partitions)]
  G --> I[Per-org hash chain]
  G --> J[Audit-only key / SIEM]
  H --> K[Scoped query builder]
  K --> L[Org audit API]
  K --> M[Admin cross-org API]
  L --> N[/reports/audit]
  M --> N
Loading

Privileged changes are fail-closed: the business row and its audit row are written in the same transaction. If the audit write cannot be persisted, the business change rolls back.

sequenceDiagram
  participant U as User
  participant H as Handler
  participant DB as Postgres
  U->>H: PUT /orgs/:id/users/:user_id/role
  H->>DB: BEGIN
  H->>DB: update user_roles
  H->>DB: insert audit_events
  alt audit insert fails
    DB-->>H: error
    H->>DB: ROLLBACK
    H-->>U: 500, no business change
  else audit insert succeeds
    H->>DB: COMMIT
    H-->>U: 200
  end
Loading

Scoping model

Viewer Scope
Member org_id = current org AND actor_user_id = self
Owner org_id = current org, excluding actor_capacity=admin and actor_kind=super_admin
Super admin all orgs, all actors, including admin-capacity/support work
Audit-only API key audit routes only, inherits the creator's live role
Cross-tenant request 403/404, never a global fallback

Every super-admin action is written with actor_capacity=admin. Super-admin work is therefore recorded, visible to super admins, and hidden from owners.

Main changes

1. Storage and integrity

  • db/migrations/20261025120000_create_audit_events.sql
    • monthly range partitions
    • append-only trigger (UPDATE rejected; DELETE only for retention with app.audit_purge=on)
    • typed actor/action/resource/outcome/severity columns
    • seq, prev_hash, hash chain columns
  • db/migrations/20261025120100_add_api_key_kind.sql
    • standard vs audit_reader
  • db/migrations/20261025130000_backfill_super_admin_visibility.sql
    • normalizes historical super-admin rows to actor_capacity=admin
  • internal/auditlog/*
    • typed action catalog
    • sanitizer
    • transactional recorder
    • scoped query builder
    • integrity verifier
    • route spec registry / coverage checks

2. Automatic and semantic capture

  • Global middleware captures every non-GET /api/v1 request plus sensitive GETs.
  • s.audited(...) registers mutating routes and their semantic action spec together.
  • Explicit events cover auth, user/role, organization, API keys, settings, collections, reviews, chat, exports/downloads, and AI/model calls.
  • Action names are a stable SIEM contract in docs/audit-events.md.

3. Read API and exports

Org-scoped:

GET /api/v1/orgs/:org_id/audit-events
GET /api/v1/orgs/:org_id/audit-events/meta
GET /api/v1/orgs/:org_id/audit-events/facets
GET /api/v1/orgs/:org_id/audit-events/ai-usage
GET /api/v1/orgs/:org_id/audit-events/:id
GET /api/v1/orgs/:org_id/audit-events/export?format=csv|ndjson|package
GET /api/v1/orgs/:org_id/audit-events/archive?month=YYYY-MM

Admin:

GET  /api/v1/admin/audit-events
GET  /api/v1/admin/audit-events/export
GET  /api/v1/admin/audit-events/integrity
GET  /api/v1/admin/audit-settings
PUT  /api/v1/admin/audit-settings
POST /api/v1/admin/audit-maintenance/run

Exports, archive downloads, audit reads, and evidence packages emit their own sensitive-read events.

4. Retention and integrity

  • Hot Postgres window + monthly NDJSON archive + archive prune.
  • Daily maintenance job and per-org hash-chain anchor.
  • Admin integrity endpoint recomputes the chain and reports the first break.
  • Runtime retention policy lives in system_settings.audit_settings.

5. UI

  • New Reports → Audit Log page (ui/src/pages/Reports/AuditLog.tsx).
  • Live polling, range presets, category/action/outcome/actor/search filters.
  • Cursor pagination, detail drawer, AI usage tab.
  • CSV / NDJSON / evidence-package downloads, archive download.
  • Super-admin all-org selector, retention settings, integrity check.
  • Layout and table styling mirror the Reviews page.
  • Settings → API Keys supports creating audit_reader keys with a badge.
  • Mega-menu entry added.

6. Local E2E harness

  • make dev-up now starts an isolated lrc/E2E window.
  • Real account + ephemeral member/pure-owner/super-admin fixtures.
  • Sandboxed lrc home; the real ~/.lrc.toml is never touched by default.
  • Journaled backup/restore path for the opt-in real-config mode.
  • Capability preflight: API/UI/worker, auth, AI hello-world, git connector.
  • Real lrc review submission and polling.
  • Audit assertions for member/owner/super-admin visibility, failed-login visibility, cross-org denial, API-key kind gating, exports, AI usage, retention, integrity, and append-only rejection.
  • Playwright UI tests for the audit page and admin/member views.
  • Read-only smoke of all manual-talent* URLs; full suite runs on one selected target.

See docs/e2e-local-testing.md.

Tests / validation

  • go test ./internal/auditlog/... — pass
  • npx tsc --noEmit — pass
  • Go build for ./internal/... ./pkg/... ./storage/... ./network/... . — pass
  • Local full E2E:
    • 34 audit assertions passed
    • real lrc review completed
    • 4 Playwright UI tests passed
    • public URL smoke completed
  • Full E2E through https://manual-talent3.apps.hexmos.com — pass
  • make e2e-doctor — all checks OK
  • Real ~/.lrc.toml hash unchanged

A production UI build was not run as part of this PR validation.

Rollout notes

  • Apply the three migrations before deploying the new API/UI.
  • The super-admin visibility backfill takes an ACCESS EXCLUSIVE lock on audit partitions; run it in a maintenance window on busy instances.
  • Retention defaults are 90 days hot / 365 days total, super-admin controlled.
  • Rebuild/deploy the UI bundle to make /reports/audit available.
  • Existing GitHub connector PAT issues are environment data, not a code dependency; preflight reports them without blocking audit/review tests.

Known follow-ups

  • The historical super-admin visibility backfill changes hashed actor fields without recomputing historical chain hashes. Local integrity on the pre-existing owner org reports a break at seq 1; fresh rows verify correctly. If historical chain integrity on upgraded instances is required, a chain-rebuild/chain-version migration should follow before claiming historical tamper-evidence.
  • Billing/payment destructive E2E paths remain opt-in because they can move real money.
  • General API-key scope enforcement beyond audit_reader remains a separate hardening project.

Checklist

  • I kept the change scoped to audit + its local E2E harness.
  • I ran the most specific relevant validation and described it above.
  • I called out behavior and rollout changes.
  • I reviewed the security/scoping impact and updated related docs.
  • I added the new page to the mega menu and chatbot route guide.
  • GIF/video walkthrough attached — UI validated by Playwright tests and local screenshots instead.

Notes for reviewers

Focus areas:

  1. internal/auditlog/query.go — visibility rules and tenant isolation.
  2. internal/auditlog/recorder.go + privileged handlers — fail-closed transaction behavior.
  3. internal/api/audit_middleware.go + audit_allowlist.go — route coverage and sensitive-read handling.
  4. internal/api/audit_retention.go — partition/archive/anchor lifecycle.
  5. The historical hash-chain caveat above.

🔍 LiveReview

⚙️ Customize risk-based approval policy for this repository or your organization.

shrsv added 3 commits October 25, 2026 09:28
LiveReview Pre-Commit Check: skipped (iter:1, coverage:0%)
- Append-only, partitioned audit_events with hash-chain integrity,
  retention/archiving, and a scoped query layer.
- Automatic middleware capture plus semantic events for auth, user/role,
  API key, organization, sensitive reads, and AI usage.
- Super-admin/owner/member visibility rules, including a backfill for
  historical super-admin actions.
- New Reports > Audit Log page with filters, live tail, exports,
  integrity checks, and a Reviews-style table.
- Action catalog, routes guide, and audit documentation.

LiveReview Pre-Commit Check: skipped (iter:1, coverage:0%)
LiveReview Pre-Commit Check: skipped (iter:1, coverage:0%)
Comment thread scripts/e2e/control.py Fixed
Comment thread scripts/e2e/e2e_lib.py Dismissed
shrsv added 2 commits October 26, 2026 15:28
LiveReview Pre-Commit Check: skipped (iter:1, coverage:0%)
LiveReview Pre-Commit Check: skipped (iter:1, coverage:0%)
Comment thread scripts/e2e/e2e_lib.py Fixed
shrsv added 3 commits October 26, 2026 15:34
LiveReview Pre-Commit Check: skipped (iter:1, coverage:0%)
LiveReview Pre-Commit Check: skipped (iter:1, coverage:0%)
@shrsv
shrsv merged commit c7b784e into master Oct 26, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants