diff --git a/BACKLOG.md b/BACKLOG.md index cb9e55a..357179c 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -19,9 +19,9 @@ and vice versa. `secrets set|check|list|rotate|rm|rekey` ([#107](https://github.com/CryptoJones/FlatlineRoundtable/issues/107)) — shipped in PR #114 - [x] Lane config in the store: `lanes`, `defaults`, `globals`, `lane_versions`, `db export|import`, `import-yaml` ([#108](https://github.com/CryptoJones/FlatlineRoundtable/issues/108)) — shipped in PR #123 -- [ ] Run from the store: `load_config` reads the DB, `fetch_keys` reads secrets, - YAML retired ([#109](https://github.com/CryptoJones/FlatlineRoundtable/issues/109)) — on feat/sqlite-store-run-from-store -- [ ] `db backup|restore` and the telesto restic job ([#110](https://github.com/CryptoJones/FlatlineRoundtable/issues/110)) +- [x] Run from the store: `load_config` reads the DB, `fetch_keys` reads secrets, + YAML retired ([#109](https://github.com/CryptoJones/FlatlineRoundtable/issues/109)) — shipped in PR #124 +- [ ] `db backup|restore` and the telesto restic job ([#110](https://github.com/CryptoJones/FlatlineRoundtable/issues/110)) — on feat/sqlite-store-backup - [ ] Fleet rollout notes and the Python 3.11 floor ([#111](https://github.com/CryptoJones/FlatlineRoundtable/issues/111)) ### Epic: run the roundtable from inside Orca — local web UI, no fork ([#115](https://github.com/CryptoJones/FlatlineRoundtable/issues/115)) — DONE 2026-10-03 @@ -183,6 +183,12 @@ stub. `ps auxww` during `db init` shows no key (it goes to `pass insert` on stdin). - [ ] `./install.sh` then `./install.sh --uninstall` round-trips, and refuses to delete anything that is not a symlink. *(human — touches `$HOME`, no test)* +- [ ] **Restore drill:** `restic restore latest` from telesto → `db restore` into + a scratch `--config` → `db doctor` → `secrets check` → `--list`, and the + scratch store's `db export` equals the live one. *(human — needs telesto and + the real DB key; the commands are in the README.)* Last run 2026-10-06 on + makemake: all steps ok, 9/9 secrets present, export identical. + *Automated parts:* `TestBackup`. ## Done diff --git a/README.md b/README.md index 34e4b9b..b14aa27 100644 --- a/README.md +++ b/README.md @@ -126,6 +126,35 @@ copy. `--set` values are JSON when they parse and strings otherwise. [`examples/roster.example.json`](examples/roster.example.json) is the example YAML config after `import-yaml` and `db export`. +**Backups (#110).** `roundtable db backup` writes an integrity-checked `0600` +copy to `backups/` beside the store, with SQLite's online backup API, so it is +consistent while a run has the store open. It keeps the newest 14 (`--keep N`). +`roundtable db restore FILE` refuses a corrupt file, and a file from a newer +schema unless `--force`; the store it replaces is kept as +`roundtable.db.pre-restore-`. + +[`scripts/backup-telesto.sh`](scripts/backup-telesto.sh) is a one-shot job: it +takes a `db backup`, then pushes the backups and the gpg-encrypted DB key entry +to a restic repository on telesto (`sftp:telesto:/NAS/backups/flatline-roundtable/`, +password in the `pass` entry `flatline-roundtable/restic-telesto`), prunes to +14 daily, 8 weekly and 6 monthly snapshots, and runs `restic check` on Sundays. +A store without its DB key cannot decrypt its secrets, which is why the key goes +too; the key is in turn useless without your gpg private key. +[`examples/net.thenetwerk.roundtable-backup.plist`](examples/net.thenetwerk.roundtable-backup.plist) +runs it daily at 03:45 under launchd. + +The restore drill, run against a scratch path so the live store is untouched: + +```sh +export RESTIC_REPOSITORY=sftp:telesto:/NAS/backups/flatline-roundtable/$(hostname -s) +export RESTIC_PASSWORD_COMMAND="pass show flatline-roundtable/restic-telesto" +restic restore latest --target /tmp/drill +roundtable db restore /tmp/drill/.../backups/roundtable--.db --config /tmp/drill/roundtable.db +roundtable db doctor --config /tmp/drill/roundtable.db +roundtable secrets check --config /tmp/drill/roundtable.db +roundtable --config /tmp/drill/roundtable.db --list +``` + ## Cost **`harness` decides cost, not `model`.** diff --git a/examples/net.thenetwerk.roundtable-backup.plist b/examples/net.thenetwerk.roundtable-backup.plist new file mode 100644 index 0000000..a5b0eb7 --- /dev/null +++ b/examples/net.thenetwerk.roundtable-backup.plist @@ -0,0 +1,38 @@ + + + + + + Label + net.thenetwerk.roundtable-backup + ProgramArguments + + /Users/YOU/source/repos/FlatlineRoundtable/scripts/backup-telesto.sh + + EnvironmentVariables + + PATH + /opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin + + StartCalendarInterval + + Hour + 3 + Minute + 45 + + RunAtLoad + + StandardOutPath + /Users/YOU/.local/state/flatline-roundtable/backup.log + StandardErrorPath + /Users/YOU/.local/state/flatline-roundtable/backup.log + + diff --git a/roundtable b/roundtable index 86244b5..01b76cd 100755 --- a/roundtable +++ b/roundtable @@ -56,6 +56,7 @@ YAML is retired, and `import-yaml` moves an old config in once. roundtable lanes enable|disable|retire NAME | clone SRC NEW | rename OLD NEW roundtable defaults|globals show | set k=v ... | unset k ... roundtable db export [--out FILE] | import FILE [--merge|--replace] + roundtable db backup [--dir DIR] [--keep N] | restore FILE [--force] roundtable import-yaml PATH [--dry-run] [--pull-secrets] one-time move off YAML * Round 1 is blind by construction — a lane cannot see the others because @@ -891,7 +892,7 @@ def _referencing_lanes(conn: sqlite3.Connection, name: str) -> list[str]: # store admin commands # --------------------------------------------------------------------------- # ADMIN_VERBS = { - "db": ("init", "migrate", "doctor", "export", "import"), + "db": ("init", "migrate", "doctor", "export", "import", "backup", "restore"), "secrets": ("set", "check", "list", "rotate", "rm", "rekey"), "lanes": ("list", "show", "add", "edit", "enable", "disable", "retire", "clone", "rename", "history"), @@ -1884,6 +1885,125 @@ def cmd_import_yaml(a) -> int: return 0 +# --------------------------------------------------------------------------- # +# backup and restore (#110) +# +# A backup is a self-contained copy taken with SQLite's online backup API, so +# it is consistent even while a run holds the store open in WAL mode -- a file +# copy of roundtable.db alone would miss whatever still sits in -wal. The copy +# is useless without the DB key, which lives in `pass`; scripts/backup-telesto.sh +# ships both. +# --------------------------------------------------------------------------- # +def _sqlite_copy(src: sqlite3.Connection, dest: Path) -> None: + """Write `src` to a new, private `dest` as a rollback-journal database, so + the copy is one file with no -wal or -shm beside it.""" + os.close(os.open(dest, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)) + dst = sqlite3.connect(dest, isolation_level=None) + try: + src.backup(dst) + dst.execute("PRAGMA journal_mode=DELETE") + finally: + dst.close() + + +def _integrity(path: Path) -> str: + try: + c = sqlite3.connect(f"file:{path}?mode=ro", uri=True) + try: + return c.execute("PRAGMA integrity_check").fetchone()[0] + finally: + c.close() + except sqlite3.DatabaseError as e: + return str(e) + + +def cmd_db_backup(a) -> int: + conn = open_db(a.config) + out_dir = a.dir or a.config.parent / "backups" + out_dir.mkdir(parents=True, exist_ok=True, mode=0o700) + host = re.sub(r"[^A-Za-z0-9.-]", "_", socket.gethostname().split(".")[0]) + stamp = dt.datetime.now(dt.timezone.utc).strftime("%Y%m%dT%H%M%SZ") + dest = out_dir / f"roundtable-{host}-{stamp}.db" + n = 1 + while dest.exists(): # two backups in one second + dest = out_dir / f"roundtable-{host}-{stamp}-{n}.db" + n += 1 + try: + _sqlite_copy(conn, dest) + finally: + conn.close() + result = _integrity(dest) + if result != "ok": + dest.unlink() + die(f"backup copy failed integrity_check ({result}) — nothing kept") + # Newest first: by stamp, then by the -N a same-second backup gets. Plain + # name order would put "...Z-1.db" before "...Z.db", which is backwards. + pat = re.compile(rf"roundtable-{re.escape(host)}-(\d{{8}}T\d{{6}}Z)(?:-(\d+))?\.db") + mine = sorted((p for p in out_dir.iterdir() if pat.fullmatch(p.name)), + key=lambda p: (pat.fullmatch(p.name)[1], int(pat.fullmatch(p.name)[2] or 0)), + reverse=True) + pruned = mine[a.keep:] if a.keep > 0 else [] + for old in pruned: + old.unlink() + print(f"backup {dest} (integrity ok, kept {min(len(mine), a.keep)}" + + (f", pruned {len(pruned)}" if pruned else "") + ")") + return 0 + + +def cmd_db_restore(a) -> int: + src_path = a.file + if not src_path.exists(): + die(f"no backup at {src_path}") + result = _integrity(src_path) + if result != "ok": + die(f"{src_path} fails integrity_check ({result}) — refusing to restore it") + src = sqlite3.connect(f"file:{src_path}?mode=ro", uri=True) + try: + ver = schema_version(src) + kid = src.execute("SELECT value FROM meta WHERE key = 'key_id'").fetchone() + except sqlite3.DatabaseError: + kid = None + if kid is None: + src.close() + die(f"{src_path} is not a roundtable store (no key id in its meta table)") + # --force exists for this check only. A newer schema may hold tables this + # code does not know, which is a choice; a corrupt file never is. + if ver > SCHEMA_VERSION and not a.force: + src.close() + die(f"{src_path} is at schema {ver}, newer than this roundtable ({SCHEMA_VERSION}) — " + f"update roundtable, or pass --force") + target = a.config + kept = None + if target.exists(): + live = open_db(target) + # Fold -wal into the main file so the copy kept aside is complete. + live.execute("PRAGMA wal_checkpoint(TRUNCATE)") + live.close() + stamp = dt.datetime.now(dt.timezone.utc).strftime("%Y%m%dT%H%M%SZ") + kept = target.with_name(f"{target.name}.pre-restore-{stamp}") + target.rename(kept) + for side in ("-wal", "-shm"): + Path(f"{target}{side}").unlink(missing_ok=True) + else: + target.parent.mkdir(parents=True, exist_ok=True, mode=0o700) + try: + _sqlite_copy(src, target) + except BaseException: + if kept is not None and not target.exists(): + kept.rename(target) + raise + finally: + src.close() + restored = open_db(target) + restored.execute("PRAGMA journal_mode=WAL") + restored.close() + print(f"restored {src_path} -> {target} (schema {ver}, key id {kid[0]})") + if kept is not None: + print(f" the store it replaced is kept as {kept}") + print(" check it with: roundtable db doctor && roundtable secrets check") + return 0 + + def admin_main(argv: list[str]) -> int: common = argparse.ArgumentParser(add_help=False) common.add_argument("--config", type=Path, default=DEFAULT_DB, @@ -1911,6 +2031,17 @@ def admin_main(argv: list[str]) -> int: mode.add_argument("--replace", action="store_true", help="make the roster exactly the file; unlisted lanes are retired") p.set_defaults(fn=cmd_db_import) + p = dbv.add_parser("backup", parents=[common], + help="integrity-checked 0600 copy, safe while the store is in use") + p.add_argument("--dir", type=Path, help="where to write (default: backups/ beside the store)") + p.add_argument("--keep", type=int, default=14, help="newest copies to keep (default 14)") + p.set_defaults(fn=cmd_db_backup) + p = dbv.add_parser("restore", parents=[common], + help="replace the store with a backup; the old one is kept aside") + p.add_argument("file", type=Path) + p.add_argument("--force", action="store_true", + help="restore a backup from a newer schema (never a corrupt one)") + p.set_defaults(fn=cmd_db_restore) ln = groups.add_parser("lanes", help="the lane roster in the store") lv = ln.add_subparsers(dest="verb", required=True) diff --git a/scripts/backup-telesto.sh b/scripts/backup-telesto.sh new file mode 100755 index 0000000..f1c4d98 --- /dev/null +++ b/scripts/backup-telesto.sh @@ -0,0 +1,59 @@ +#!/usr/bin/env bash +# One-shot backup of the roundtable store to telesto, with restic (#110). +# +# Runs to completion and exits: a launchd timer starts it (see +# examples/net.thenetwerk.roundtable-backup.plist), which keeps the +# "no long-lived processes" rule in AGENTS.md. +# +# What goes over, and why both halves: +# * the store's backups dir -- `roundtable db backup` writes an +# integrity-checked copy there first, so restic never reads a live WAL file; +# * the gpg-encrypted pass entries under flatline-roundtable/ -- the DB key. +# Without it a restored store's secrets cannot be decrypted. It is still +# useless without your gpg private key, which this does not touch. +# +# Override any of these from the environment: +# RESTIC_REPOSITORY default sftp:telesto:/NAS/backups/flatline-roundtable/ +# RESTIC_PASSWORD_COMMAND default pass show flatline-roundtable/restic-telesto +# KEEP local backup copies to keep (default 14) +# CHECK=1 run `restic check` now (otherwise Sundays only) +# +# rsync is the fallback if restic is unavailable: +# rsync -a "$DATA/backups/" telesto:/NAS/backups/flatline-roundtable// +set -euo pipefail + +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)" +ROUNDTABLE="${SCRIPT_DIR}/../roundtable" +HOST="$(hostname -s)" +DATA="${XDG_DATA_HOME:-${HOME}/.local/share}/flatline-roundtable" +PASS_DIR="${PASSWORD_STORE_DIR:-${HOME}/.password-store}/flatline-roundtable" + +export RESTIC_REPOSITORY="${RESTIC_REPOSITORY:-sftp:telesto:/NAS/backups/flatline-roundtable/${HOST}}" +export RESTIC_PASSWORD_COMMAND="${RESTIC_PASSWORD_COMMAND:-pass show flatline-roundtable/restic-telesto}" + +log() { printf '%s %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*"; } +die() { log "ERROR: $*" >&2; exit 1; } + +command -v restic >/dev/null || die "restic is not installed (brew install restic)" +[[ -f "${PASS_DIR}/db-key.gpg" ]] || die "no DB key at ${PASS_DIR}/db-key.gpg — backing up a store without its key is pointless" + +log "snapshot the store" +"${ROUNDTABLE}" db backup --keep "${KEEP:-14}" + +if ! restic cat config >/dev/null 2>&1; then + log "initialise ${RESTIC_REPOSITORY}" + restic init +fi + +log "restic backup" +restic backup --quiet --tag flatline-roundtable --host "${HOST}" "${DATA}/backups" "${PASS_DIR}" + +log "restic forget" +restic forget --quiet --tag flatline-roundtable --host "${HOST}" \ + --keep-daily 14 --keep-weekly 8 --keep-monthly 6 --prune + +if [[ "$(date +%u)" == 7 || "${CHECK:-}" == 1 ]]; then + log "restic check" + restic check +fi +log "done" diff --git a/tests/test_roundtable.py b/tests/test_roundtable.py index e42b5e7..24dbe02 100644 --- a/tests/test_roundtable.py +++ b/tests/test_roundtable.py @@ -2154,6 +2154,102 @@ def test_roster_example_is_the_legacy_yaml_converted(self): self.assertEqual(doc, rt.yaml_to_roster(LEGACY_YAML)) +class TestBackup(StoreCase): + """db backup / db restore (#110).""" + + def ok(self, *args, **kw): + r = self.rt(*args, **kw) + self.assertEqual(r.returncode, 0, r.stderr) + return r + + def seeded(self): + self.init() + self.ok("db", "import", str(ROSTER_EXAMPLE), "--replace") + self.ok("secrets", "set", "or/x", "--stdin", input="sk-backup-canary\n") + return self.ok("db", "export").stdout + + def backups(self): + return sorted((self.db.parent / "backups").glob("roundtable-*.db")) + + def test_backup_is_private_complete_and_checked(self): + before = self.seeded() + holder = sqlite3.connect(self.db) # a live WAL reader, as during a run + holder.execute("SELECT count(*) FROM lanes").fetchall() + self.addCleanup(holder.close) + self.ok("db", "backup") + (b,) = self.backups() + self.assertEqual(b.stat().st_mode & 0o777, 0o600) + self.assertEqual(b.parent.stat().st_mode & 0o777, 0o700) + self.assertEqual(rt._integrity(b), "ok") + self.assertFalse(Path(f"{b}-wal").exists()) # one self-contained file + self.assertNotIn(b"sk-backup-canary", b.read_bytes()) + other = {**self.env, "XDG_DATA_HOME": str(self.d / "restored")} + r = self.ok("db", "restore", str(b), env=other) + self.assertEqual(self.ok("db", "export", env=other).stdout, before) + self.assertIn("16 chars", self.ok("secrets", "check", "or/x", env=other).stdout) + + def test_fifteen_backups_with_keep_14_leave_14(self): + self.seeded() + made = [] + for _ in range(15): + out = self.ok("db", "backup", "--keep", "14").stdout + made.append(Path(re.search(r"backup (\S+\.db)", out).group(1)).name) + kept = {p.name for p in self.backups()} + self.assertEqual(len(kept), 14) + # The oldest went, not an arbitrary one -- same-second names carry -N, + # and a plain name sort would have pruned the newest of those instead. + self.assertEqual(kept, set(made[1:])) + + def test_restore_refuses_a_corrupt_file(self): + self.seeded() + self.ok("db", "backup") + (b,) = self.backups() + data = bytearray(b.read_bytes()) + data[4096:8192] = os.urandom(4096) # trash page 2 + b.write_bytes(bytes(data)) + before = self.db.read_bytes() + r = self.rt("db", "restore", str(b), "--force") + self.assertNotEqual(r.returncode, 0) + self.assertIn("integrity", r.stderr) + self.assertEqual(self.db.read_bytes(), before) # --force does not override this + + def test_restore_refuses_a_newer_schema_unless_forced(self): + self.seeded() + self.ok("db", "backup") + (b,) = self.backups() + c = sqlite3.connect(b) + c.execute(f"PRAGMA user_version = {rt.SCHEMA_VERSION + 1}") + c.commit() + c.close() + r = self.rt("db", "restore", str(b)) + self.assertNotEqual(r.returncode, 0) + self.assertIn("newer than this roundtable", r.stderr) + r = self.ok("db", "restore", str(b), "--force") + self.assertIn("pre-restore", r.stdout) + + def test_restore_keeps_the_replaced_store_aside(self): + before = self.seeded() + self.ok("db", "backup") + (b,) = self.backups() + self.ok("lanes", "retire", "Skeptic") + self.ok("db", "restore", str(b)) + self.assertEqual(self.ok("db", "export").stdout, before) + (aside,) = self.db.parent.glob("roundtable.db.pre-restore-*") + self.assertEqual(aside.stat().st_mode & 0o777, 0o600) + c = sqlite3.connect(aside) + self.assertIsNotNone(c.execute( + "SELECT retired_at FROM lanes WHERE name = 'Skeptic'").fetchone()[0]) + c.close() + + def test_restore_refuses_a_file_that_is_not_a_store(self): + self.init() + f = self.d / "other.db" + sqlite3.connect(f).execute("CREATE TABLE t (x)").connection.close() + r = self.rt("db", "restore", str(f)) + self.assertNotEqual(r.returncode, 0) + self.assertIn("not a roundtable store", r.stderr) + + class TestAdminDispatch(unittest.TestCase): """The brief is free text, so the command guard must be narrow."""