feat: ship logs to Loki, audit PR template, add test:changed script, …

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,17 +1,50 @@
 ## Description
-Please include a summary of the change and which issue is fixed.
 
-Fixes # (issue)
+Please include a summary of the change, the problem it solves, and why it is needed.
 
-## Type of change
-- [ ] Bug fix
-- [ ] New feature
-- [ ] Breaking change
-- [ ] Documentation update
+Fixes / Closes #(issue)
 
-## Checklist:
-- [ ] My code follows the coding conventions
+## Type of Change
+
+Select all that apply and ensure the matching PR label is set per [CONTRIBUTING.md](../CONTRIBUTING.md#pr-labels):
+
+- [ ] `bug` / `fix`: Bug fix (non-breaking change which fixes an issue)
+- [ ] `feature` / `enhancement`: New feature (non-breaking change which adds functionality)
+- [ ] `breaking-change`: Breaking change (fix or feature that causes existing functionality to not work as expected)
+- [ ] `documentation` / `docs`: Documentation update only
+- [ ] `performance`: Performance improvement
+- [ ] `security`: Security hardening or fix
+- [ ] `chore` / `refactor`: Refactoring, dependencies, or maintenance
+
+## Breaking Changes Disclosure
+
+Does this PR introduce any breaking change to APIs, database schemas, environment variables, or workflows?
+
+- [ ] **No** breaking changes are introduced.
+- [ ] **Yes**, this PR introduces breaking changes.
+
+_If yes, describe the breaking impact, reasoning, and migration steps for consumers/operators:_
+<!-- Describe breaking changes, affected endpoints, schema changes, or migration runbooks here -->
+
+## Test Plan & Verification
+
+Explain how you tested these changes. Provide commands executed, automated test files run, and manual verification results.
+
+### Automated Tests
+- Command(s) run: `npm test` / `npm run test:changed` / `npm run lint`
+- Relevant test file(s):
+
+### Manual Verification
+- Steps taken to verify in local or staging environment:
+- Verification results / evidence:
+
+## Checklist
+
+- [ ] My code follows the repository's coding style and conventions (`npm run lint`, `npm run format:check`)
 - [ ] I have performed a self-review of my own code
-- [ ] I have commented my code, particularly in hard-to-understand areas
-- [ ] I have added tests that prove my fix is effective or that my feature works
+- [ ] I have added or updated tests that prove my fix is effective or that my feature works
 - [ ] New and existing unit tests pass locally with my changes
+- [ ] I have commented my code, particularly in hard-to-understand or security-sensitive areas
+- [ ] I have updated corresponding documentation (`README.md`, `docs/`, `openapi.yaml`) where applicable
+- [ ] I have confirmed no secrets, private keys, or credentials are committed
+- [ ] I have applied the appropriate PR label per [CONTRIBUTING.md](../CONTRIBUTING.md#pr-labels)

CONTRIBUTING.md

@@ -46,7 +46,11 @@ npm run dev
 Run the test suite before pushing:
 
 ```bash
+# Full test suite (CI default)
 npm test
+
+# Run only tests affected by changed files (fast local iteration)
+npm run test:changed
 ```
 
 ### Pre-commit Hooks

README.md

@@ -658,6 +658,7 @@ Additional in-repo docs live under [`docs/`](docs/):
 - [Deployment & Scaling Guide](docs/deployment-and-scaling.md) — expected
   CPU/memory sizing for the API and the ffmpeg-backed worker, how to measure
   it, and how to scale each process (Issue #405)
+- [Logging & Aggregation Guide](docs/logging-and-aggregation.md) — centralized log shipping to Grafana Loki via Promtail, LogQL queries, and sensitive data redaction (Issue #380)
 - [Architecture](docs/ARCHITECTURE.md) — high-level module layout
 - [AI Feature Set](docs/AI_FEATURES.md) — AI capabilities, what data is sent
   where, and the per-artist opt-in/opt-out story
@@ -698,18 +699,18 @@ This brings up the **minimal stack** (API `localhost:4000` + Postgres `5432` + R
 
 | Command                                             | What it starts                                                          |
 | --------------------------------------------------- | ----------------------------------------------------------------------- |
-| `docker compose up --build`                         | `backend` + `db` + `redis` (minimal, no profile)                        |
-| `docker compose --profile full up --build`          | Everything: minimal + `rabbitmq` + `pgadmin` + `prometheus` + `grafana` |
-| `docker compose --profile queue up --build`         | Minimal + `rabbitmq` (song-processing queue)                            |
-| `docker compose --profile tools up --build`         | Minimal + `pgadmin` (`localhost:5050`)                                  |
-| `docker compose --profile monitoring up --build`    | Minimal + `prometheus` (`9090`) + `grafana` (`3000`)                    |
-| `docker compose --profile observability up --build` | Same as `monitoring` (alias)                                            |
+| `docker compose up --build`                         | `backend` + `db` + `redis` (minimal, no profile)                                                |
+| `docker compose --profile full up --build`          | Everything: minimal + `rabbitmq` + `pgadmin` + `prometheus` + `grafana` + `loki` + `promtail`   |
+| `docker compose --profile queue up --build`         | Minimal + `rabbitmq` (song-processing queue)                                                    |
+| `docker compose --profile tools up --build`         | Minimal + `pgadmin` (`localhost:5050`)                                                          |
+| `docker compose --profile monitoring up --build`    | Minimal + `prometheus` (`9090`) + `grafana` (`3000`) + `loki` (`3100`) + `promtail`            |
+| `docker compose --profile observability up --build` | Same as `monitoring` (alias)                                                                    |
 
 `backend` no longer `depends_on: rabbitmq` — the API starts without a queue and connects to RabbitMQ lazily when it appears (see `src/index.ts`), so the minimal stack stays self-contained. Any combination of profiles can be stacked, e.g.:
 
 ```bash
 docker compose --profile queue --profile monitoring up --build
-# → backend + db + redis + rabbitmq + prometheus + grafana
+# → backend + db + redis + rabbitmq + prometheus + grafana + loki + promtail
 ```
 
 The full observability stack (`--profile full`) is unchanged from the pre-profile `docker compose up` topology and is what CI and `docker-compose.prod.yml` extend.
@@ -757,6 +758,7 @@ npm run dev
 | `npm run migration:run`                               | Applies pending migrations to the database                                                                                      |
 | `npm run migration:revert`                            | Reverts the last applied migration                                                                                              |
 | `npm test`                                            | Runs the full Jest suite (`src/__tests__/**/*.test.ts`)                                                                         |
+| `npm run test:changed`                                | Runs only tests affected by files changed since `main` (for fast local iteration; Issue #397)                                   |
 | `npm test -- src/__tests__/health.test.ts`            | Runs a single test file (swap in any path under `src/__tests__`)                                                                |
 | `npm run test:watch`                                  | Runs Jest in watch mode                                                                                                         |
 | `npm run dev:bootstrap`                               | Bootstraps Docker dependencies, runs migrations, seeds genres and test accounts (Issue #369)                                    |

docker-compose.yml

@@ -157,10 +157,41 @@ services:
       - monitoring
       - observability
 
+  loki:
+    image: grafana/loki:latest
+    container_name: loki
+    restart: always
+    ports:
+      - '3100:3100'
+    command: -config.file=/etc/loki/local-config.yaml
+    volumes:
+      - loki_data:/loki
+    profiles:
+      - full
+      - monitoring
+      - observability
+
+  promtail:
+    image: grafana/promtail:latest
+    container_name: promtail
+    restart: always
+    volumes:
+      - ./monitoring/promtail.yml:/etc/promtail/promtail.yml:ro
+      - /var/lib/docker/containers:/var/lib/docker/containers:ro
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+    command: -config.file=/etc/promtail/promtail.yml
+    depends_on:
+      - loki
+    profiles:
+      - full
+      - monitoring
+      - observability
+
 volumes:
   pgdata:
   redisdata:
   pgadmin_data:
   rabbitmq_data:
   prometheus_data:
   grafana_data:
+  loki_data:

docs/logging-and-aggregation.md

@@ -0,0 +1,109 @@
+# Structured Logging & Centralized Log Aggregation (Issue #380)
+
+## Overview
+
+AudioBlock Backend implements structured JSON logging using [Pino](https://getpino.io). All logs carry service metadata (`audioblock-backend`), runtime environment, ISO timestamps, request correlation IDs (`reqId`), and HTTP context.
+
+Centralized log aggregation is wired to **Grafana Loki** via a **Promtail sidecar** in Docker Compose and Kubernetes environments, and also supports **direct HTTP log shipping** via the `LOG_AGGREGATOR_URL` environment variable.
+
+---
+
+## 1. Zero Sensitive Data Leakage (Redaction)
+
+To strictly enforce that **no credentials, tokens, or sensitive user data are shipped or persisted in logs**, Pino's native `redact` engine automatically sanitizes all matching fields before logs are emitted or shipped:
+
+- Passwords (`password`, `newPassword`, `oldPassword`, `confirmPassword`)
+- Authentication tokens & headers (`token`, `accessToken`, `refreshToken`, `authorization`, `headers.authorization`)
+- Secrets & keys (`apiKey`, `secret`, `jwtSecret`, `clientSecret`, `privateKey`, `twoFactorSecret`, `totpSecret`)
+- Financial & PII attributes (`creditCard`, `cardNumber`, `cvv`, `ssn`)
+
+All matching values are replaced with `[REDACTED]`.
+
+---
+
+## 2. Log Aggregation Topology
+
+### Local / Staging Docker Stack
+
+In `docker-compose.yml`, the observability stack provides:
+
+```
+┌─────────────────────────┐
+│   audioblocks-backend   │
+│ (stdout structured JSON)│
+└────────────┬────────────┘
+             │ Docker socket stream
+             ▼
+┌─────────────────────────┐
+│        Promtail         │
+│  (parses JSON & labels) │
+└────────────┬────────────┘
+             │ HTTP POST /loki/api/v1/push
+             ▼
+┌─────────────────────────┐         ┌─────────────────────────┐
+│       Grafana Loki      │◄────────┤         Grafana         │
+│ (centralized retention) │         │ (interactive log search)│
+└─────────────────────────┘         └─────────────────────────┘
+```
+
+1. **Backend** emits single-line JSON logs to `stdout`.
+2. **Promtail** sidecar automatically ingests container stdout via the Docker daemon socket (`unix:///var/run/docker.sock`), applies JSON parsing, and extracts labels (`service`, `level`, `statusCode`).
+3. **Loki** indexes labels and stores compressed log chunks on the `loki_data` volume.
+4. **Grafana** provisions Loki as a default log datasource (`http://loki:3100`), allowing developers to search, filter, and correlate logs against Prometheus metrics in Grafana Explore.
+
+---
+
+## 3. How to Run Locally
+
+Start the minimal backend stack with the monitoring & logging profile:
+
+```bash
+docker compose --profile monitoring up -d
+# or full stack
+docker compose --profile full up -d
+```
+
+Services started:
+- Backend: `http://localhost:4000`
+- Prometheus: `http://localhost:9090`
+- Grafana Loki: `http://localhost:3100`
+- Promtail: `http://localhost:9080`
+- Grafana: `http://localhost:3000` (Login: `admin` / `admin`)
+
+---
+
+## 4. Querying Logs in Grafana
+
+1. Open Grafana in your browser: `http://localhost:3000`.
+2. Navigate to **Explore** (compass icon on the left menu).
+3. Select datasource: **Loki**.
+4. Run LogQL queries:
+
+```logql
+# Stream all logs from AudioBlock backend
+{container_name="audioblocks-backend"}
+
+# Filter by log level
+{container_name="audioblocks-backend"} | json | level="error"
+
+# Search for specific request correlation ID
+{container_name="audioblocks-backend"} |= "reqId" | json | reqId="<uuid>"
+
+# Search for 5xx errors
+{container_name="audioblocks-backend"} | json | statusCode >= 500
+```
+
+---
+
+## 5. Direct HTTP Log Shipping (Non-Docker / Serverless)
+
+If running outside Docker (e.g. bare metal or serverless) without a sidecar, direct log shipping can be enabled by setting:
+
+```bash
+LOG_AGGREGATOR_URL=http://loki:3100/loki/api/v1/push
+```
+
+When `LOG_AGGREGATOR_URL` is set:
+- Logs continue writing to `stdout` for local visibility.
+- `logger` batches and asynchronously flushes log records in batches of up to 50 entries or every 2 seconds via `POST`.
+- Aggregator outages fail open to prevent application degradation.

docs/openapi.yaml

@@ -328,6 +328,41 @@ paths:
           description: The caller does not own the song.
         '404':
           description: Version not found.
+  /api/song/{id}/versions/{version}/rollback:
+    post:
+      summary: Roll back a song to a prior version
+      description: Owning artist only. Creates a new active version copying the specified historical revision's metadata and audio assets, preserving complete version history without mutation (Issue #441).
+      parameters:
+        - name: id
+          in: path
+          required: true
+          schema:
+            type: string
+            format: uuid
+        - name: version
+          in: path
+          required: true
+          description: 1-based historical version number to roll back to.
+          schema:
+            type: integer
+            minimum: 1
+      requestBody:
+        required: false
+        content:
+          application/json:
+            schema:
+              type: object
+              properties:
+                changeNote:
+                  type: string
+                  description: Optional explanation of why the rollback was performed.
+      responses:
+        '201':
+          description: Created new version copying the target revision's fields.
+        '403':
+          description: The caller does not own the song.
+        '404':
+          description: Song or target version not found.
   /api/song/{id}/report:
     post:
       summary: Report a song

monitoring/README.md

@@ -10,16 +10,17 @@ provisioned automatically on container startup.
 ```
 monitoring/
 ├── prometheus.yml        # Prometheus scrape config (targets the backend /metrics)
+├── promtail.yml          # Promtail log shipping config (targets Loki) (Issue #380)
 ├── grafana/
-│   └── provisioning.yml  # Grafana provisioning: Prometheus datasource + dashboards provider
+│   └── provisioning.yml  # Grafana provisioning: Prometheus & Loki datasources + dashboards provider
 └── dashboards/
     ├── audioblock-app.json   # Express app / HTTP / DB-pool / cache / royalties dashboard
     └── audioblock-node.json  # Host system (CPU / memory / disk / network) overview
 ```
 
 ## How it works
 
-`docker-compose.yml` starts a Prometheus instance and a Grafana instance.
+`docker-compose.yml` starts Prometheus, Grafana, Grafana Loki, and Promtail.
 
 - Prometheus scrapes the backend's `GET /metrics` endpoint (exposed by
   `prom-client` in `src/services/MetricsService.ts`, served by
@@ -28,10 +29,15 @@ monitoring/
 - The standalone indexer worker serves its process-local metrics on port 9464;
   the `monitoring`, `observability`, and `full` Compose profiles start it and
   let Prometheus scrape `indexer:9464`.
+- Log aggregation is handled by Grafana Loki and Promtail (Issue #380):
+  Promtail collects container stdout JSON logs, redacts any sensitive tokens,
+  and ships them to Loki at `http://loki:3100`. See
+  [`docs/logging-and-aggregation.md`](../docs/logging-and-aggregation.md) for
+  the complete runbook and LogQL query examples.
 - Grafana is provisioned entirely from disk:
   - `./monitoring/grafana/provisioning.yml` is mounted to
     `/etc/grafana/provisioning/provisioning.yml` — it defines the Prometheus
-    datasource and a file-based dashboard provider.
+    and Loki datasources and a file-based dashboard provider.
   - `./monitoring/dashboards/` is mounted to
     `/etc/grafana/provisioning/dashboards` — this is the provider's
     `path`, so every dashboard JSON in the directory is automatically loaded.

monitoring/grafana/provisioning.yml

@@ -14,6 +14,14 @@ datasources:
     url: http://prometheus:9090
     isDefault: true
 
+  - name: Loki
+    type: loki
+    access: proxy
+    url: http://loki:3100
+    isDefault: false
+    jsonData:
+      maxLines: 1000
+
 # Dashboard provider points at the version-controlled dashboards directory.
 providers:
   - name: "AudioBlock Backend"

monitoring/promtail.yml

@@ -0,0 +1,34 @@
+server:
+  http_listen_port: 9080
+  grpc_listen_port: 0
+
+positions:
+  filename: /tmp/positions.yaml
+
+clients:
+  - url: http://loki:3100/loki/api/v1/push
+
+scrape_configs:
+  - job_name: docker
+    docker_sd_configs:
+      - host: unix:///var/run/docker.sock
+        refresh_interval: 5s
+    relabel_configs:
+      - source_labels: ['__meta_docker_container_name']
+        regex: '/(.*)'
+        target_label: 'container_name'
+      - source_labels: ['__meta_docker_container_label_service']
+        target_label: 'service'
+    pipeline_stages:
+      - json:
+          expressions:
+            level: level
+            msg: msg
+            reqId: reqId
+            service: service
+            statusCode: statusCode
+            path: path
+      - labels:
+          level:
+          service:
+          statusCode: