fix(sqlite): one extenddb process per database file

[robinnsc]

What

A SQLite database file has one owner at a time. extenddb serve takes an exclusive flock(2) on <database>.lock before it opens the database and holds it for the life of the process; init and migrate take the same lock for as long as they run (through the bootstrapper's migration-lock hook), and destroy takes it before it unlinks the file. Each refuses, rather than waits, when another process holds it:

another extenddb process is already using /path/db.sqlite (lock held on /path/db.sqlite.lock);
stop it first, or point this command at a different database

Read-only commands (settings, manage, verify, status) take no lock and run alongside a server.

• crates/storage-sqlite/src/serve_lock.rs: ServeLock, and database_file(), which resolves the configured location to the file SQLite actually opens by parsing it with sqlx's own SqliteConnectOptions (so sqlite: URL forms, percent-encoding such as a%20b.sqlite, and file: URIs resolve as the engine resolves them) and returns nothing for in-memory databases. The lock path is the canonicalized database path plus .lock, so two spellings of one file, a .. path, or a symlink and its target share one lock.
• crates/storage-sqlite/src/lib.rs: the server factory takes the lock first, before it opens the database or runs startup recovery, and stores it on the engine (an Arc shared by everything in the server).
• crates/storage-sqlite/src/bootstrapper.rs: acquire_migration_lock / release_migration_lock take and drop the same lock; drop_databases holds it while removing the file.
• The kernel releases the lock when the process exits, including on kill -9, so a crash never leaves a stale lock. The lock file stays on disk; its presence means nothing.
• In-memory databases (sqlite-memory, dev mode) take no lock. On non-Unix platforms no lock is taken and the server logs a warning. flock is advisory; on network filesystems its behaviour depends on the server and mount options, so the docs say to keep SQLite databases on local disk.
• libc::flock because std's File::try_lock needs Rust 1.89 and the workspace MSRV is 1.88. libc is already a workspace dependency and listed in every license notices file.
• Docs: docs/troubleshooting.md entry for the error with its scope; README, docs/manuals/11-deployment-guide.md, and docs/manuals/02-design-guide.md now say that one instance per catalog is the supported deployment, that SQLite enforces it, and that PostgreSQL and MongoDB do not. The deployment guide previously said multiple PostgreSQL instances were consistent because there was no in-process cache; there are per-instance credential and policy caches with no cross-instance invalidation, every worker runs on every instance, and /health is liveness only.

Why

No issue filed. Raised while reviewing #384, which removes restore targets left CREATING at SQLite startup and is only safe if one server at a time holds the file.

Nothing stopped two servers from opening the same SQLite file, and the backend is not safe that way: writers are serialized by a lock inside each server process, and startup recovery (pending control-plane transitions, GSI and vector index rebuilds, abandoned restores) assumes no other server is running. Nothing stopped destroy from unlinking the file under a running server either, which left that server serving an unlinked inode while a later init created a fresh database at the same path.

This is the SQLite half of the readiness review's P0-7 ("state that exactly one instance may run against a catalog and enforce it"). PostgreSQL and MongoDB are now documented as single-instance but nothing enforces it there; P0-7 stays open for them.

Testing done

Unit tests in extenddb-storage-sqlite:

• database_file(): plain and relative paths, sqlite: and sqlite:// URLs with parameters, percent-encoded names, file: URIs (file:/abs, file:///abs, file://localhost/abs, file:rel; a foreign authority is an error), in-memory forms, and a disk file whose name contains mode=memory.
• ServeLock: a second lock on the same database is refused; free again once dropped; independent across databases; shared across a .. spelling of one file and across a file symlink and a directory symlink.
• The real server factory called twice on one file: the second call fails while the first server is up and succeeds once it is dropped.
• The bootstrapper: destroy and migrate are refused while a server holds the file (and destroy leaves the file in place); a server is refused while migrate holds it; both succeed once it is released.

By hand (previous revision), against a release build: a second serve on the same file exits 1 with the message above and the first server stays healthy; after kill -9 of the first server a new one starts at once; two dev-mode in-memory servers both start.

cargo fmt --all -- --check
cargo +1.97.0 clippy --workspace --all-targets -- -D warnings   # the CI toolchain
cargo test -p extenddb-storage-sqlite                           # 91 passed, 1 ignored
cargo +1.88.0 check --workspace --locked

Checklist

• I have read CONTRIBUTING.md
• All tests pass (cargo test --workspace)
• Code is formatted (cargo fmt --check)
• Clippy is clean (cargo clippy -- -W clippy::pedantic)
• I have added or updated tests for new functionality
• I have updated documentation if behavior changed
• Breaking changes are noted below (if any)
• If this changes the wire protocol, Storage trait, auth model, on-disk format, or public CLI surface, an RFC has been accepted or is linked below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: n/a. A startup check in the SQLite backend; no wire, trait, auth, on-disk format, or CLI flag change. The only new file on disk is the empty lock file next to the database.

Breaking changes

• A second extenddb serve against a SQLite file another server is using, or init/migrate/destroy against a file a server is using, now fails instead of proceeding. Running those concurrently was never safe.
• The directory holding a SQLite database must be writable so the lock file can be created; a database in a read-only directory now fails to start with cannot open lock file.

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0 and I agree to the Developer Certificate of Origin (DCO). See CONTRIBUTING.md for details.