[daniel6yi8-gif]

Summary

Add backup/restore consistency module with ingestion watermark and DR drill test

What was solved

#333 — Point-in-time consistent backup/restore across the datastore and cache

Create a new backend/backup/ module that coordinates point-in-time consistent backup/restore across the primary datastore and cache layer. The module will classify cache-held state as derivable vs. needs-real-backup, tag backups with an ingestion watermark at snapshot time, refuse service startup when watermarks disagree beyond a documented tolerance, treat derivable cache state as cold/empty on restore, and include a DR drill test that backs up, mutates both stores, restores, and asserts internal consistency.

Addressed:

• Changed: backend/backup/mod.rs, backend/backup/watermark.rs, backend/backup/classify.rs, backend/tests/dr_drill_test.rs
• Build backend/backup/ with mod.rs, watermark.rs, and classify.rs as suggested in the issue layout
• Classify every piece of cache-held state as pure cache/derivable (safe to cold-start empty) or needs-real-backup in classify.rs, with explicit documentation
• Tag anything in the needs-backup category and the primary datastore backup with the ingestion watermark at backup time

#334 — IP allowlisting that correctly resolves client IP behind a reverse proxy

Implement a new backend/ip_allowlist/ module that restricts admin/metrics routes to allowlisted source IPs, with correct trusted-proxy-hop handling. The client IP is derived by walking the X-Forwarded-For chain from the trusted end using an explicitly configured trusted-hop count (in trusted_hops.rs), so a forged header from an untrusted peer cannot spoof an allowlisted IP. Includes resolve.rs for chain-walking resolution, mod.rs for the middleware wiring, and two integration tests (spoofed_header_test.rs, legitimate_path_test.rs) plus documentation of the proxy topology assumptions.

Addressed:

• Changed: backend/ip_allowlist/mod.rs, backend/ip_allowlist/trusted_hops.rs, backend/ip_allowlist/resolve.rs, backend/ip_allowlist/trusted_hops.rs
• Create backend/ip_allowlist/ with mod.rs, trusted_hops.rs, and resolve.rs as suggested.
• Configure an explicit trusted-proxy-hop count in trusted_hops.rs (no implicit/unbounded trust).
• Derive the client IP in resolve.rs by walking the X-Forwarded-For chain from the trusted end — not naive first/last element.

#335 — Idempotent, dead-letter-aware background job runner

Implement a new Rust background job subsystem under backend/jobs/ with an explicit retryable-vs-terminal error taxonomy, a scheduler that retries retryable errors with backoff and immediately quarantines terminal errors, a dead-letter/quarantine module capturing full failure history plus alerting, and an upsert-based example backfill job. Add two integration tests: one proving a partially-completed job can be re-run without duplication, and one proving malformed/poison input is quarantined within the retry ceiling.

Addressed:

• Changed: backend/jobs/error.rs, backend/jobs/dead_letter.rs, backend/jobs/scheduler.rs, backend/jobs/mod.rs
• Create backend/jobs/mod.rs, error.rs, scheduler.rs, dead_letter.rs, and backfill.rs per the suggested layout.
• Define an explicit retryable vs. terminal error taxonomy in error.rs that job handlers return.
• Build scheduler.rs to retry retryable errors with backoff and immediately quarantine terminal errors.

Changes

• backend/backup/mod.rs (create)
• backend/backup/watermark.rs (create)
• backend/ip_allowlist/resolve.rs (create)
• backend/backup/classify.rs (create)
• backend/jobs/error.rs (create)
• backend/tests/dr_drill_test.rs (create)
• backend/jobs/dead_letter.rs (create)
• backend/ip_allowlist/mod.rs (create)
• backend/jobs/scheduler.rs (create)
• backend/ip_allowlist/trusted_hops.rs (create)
• backend/jobs/mod.rs (create)

Approach

1. Point-in-time consistent backup/restore across the datastore and cache #333 — Point-in-time consistent backup/restore across the datastore and cache (Changed: backend/backup/mod.rs, backend/backup/watermark.rs, backend/backup/classify.rs, backend/tests/dr_drill_test.rs)
2. IP allowlisting that correctly resolves client IP behind a reverse proxy #334 — IP allowlisting that correctly resolves client IP behind a reverse proxy (Changed: backend/ip_allowlist/mod.rs, backend/ip_allowlist/trusted_hops.rs, backend/ip_allowlist/resolve.rs, backend/ip_allowlist/trusted_hops.rs)
3. Idempotent, dead-letter-aware background job runner #335 — Idempotent, dead-letter-aware background job runner (Changed: backend/jobs/error.rs, backend/jobs/dead_letter.rs, backend/jobs/scheduler.rs, backend/jobs/mod.rs)

Issues

Closes #333
Closes #334
Closes #335

[vercel]

@daniel6yi8-gif is attempting to deploy a commit to the christabel888's projects Team on Vercel.

A member of the Team first needs to authorize it.