Skip to content

feat(sync): persist the local cache and sync in the background - #209

Open
stevenwcarter wants to merge 1 commit into
romaintb:mainfrom
stevenwcarter:feat/non-blocking-sync
Open

feat(sync): persist the local cache and sync in the background#209
stevenwcarter wants to merge 1 commit into
romaintb:mainfrom
stevenwcarter:feat/non-blocking-sync

Conversation

@stevenwcarter

@stevenwcarter stevenwcarter commented Aug 23, 2026

Copy link
Copy Markdown

Summary

Every launch deleted the SQLite cache, then blocked the UI behind a centered "Loading data" overlay while it re-downloaded the entire account over four sequential API round-trips. The cache location was hardcoded, so users could not move it off a synced home directory, and cargo test deleted a real installation's database on every run. auto_sync_interval_minutes was parsed, defaulted and validated, but never read by any runtime code — no timer ever existed.

Storage:

  • New [storage] data_dir config key. A leading ~ is expanded and relative paths resolve against the working directory; unset keeps the platform default. An unusable path fails at startup with the path in the error rather than silently falling back, which would send writes to the database the user was trying to move away from.
  • The cache is no longer deleted at startup, and the schema is created idempotently, so it survives across launches.
  • The backend row's UUID is derived from (type, name) instead of being generated per launch. An existing row's UUID is adopted as-is, so a database written by an earlier version keeps its cache instead of having every project and task orphaned under a dead backend_uuid. Duplicate rows left by older debug runs are collapsed before the new unique index is built.
  • The database file is created 0600 on Unix; it holds the API token.

Sync:

  • Reconciles instead of rebuilding: rows are upserted on (backend_uuid, remote_id) and rows the remote no longer returns are deleted. The delete pass runs before parent re-linking, so completing a parent task cannot cascade its still-open subtasks out of the cache. Local UUIDs stay stable across syncs.
  • An empty fetch never wipes an entity table.
  • The four fetches run concurrently; per-resource error handling is unchanged, including sections degrading to empty rather than failing.

Interface:

  • Startup paints cached data immediately and syncs in the background.
  • Sync state appears as a toast in the lower right of the task list — syncing, synced, or failed — instead of a modal that ate keystrokes. A failed sync now reports as failed rather than as success.
  • auto_sync_interval_minutes drives a real timer; 0 disables it, and r still forces a sync. Every terminal outcome records the attempt, so a failing backend is retried once per interval rather than ten times a second.
  • Task selection follows the selected task across a sync-driven reload, so data arriving out of band cannot move the cursor under the user's hands. Reloads the user asked for — a task operation, a sidebar change, a debug refresh — leave the cursor on the row it was on, so marking an overdue task due today does not drag the cursor along as the task moves out of the Overdue section. Each reload carries its own policy, rather than the list guessing from shared state.

Behavior change: a locally completed or deleted task is removed from the cache at the next sync rather than lingering until the next launch, so it can be restored from the list for at most one auto-sync interval. Excluding those rows instead would make them permanent tombstones, since the remote never returns them again.

README and the architecture, configuration and development docs described a system that never existed — an in-memory database, a five-minute staleness check, and a dependency list naming crates the project does not use. They now describe the code.

Tests no longer touch any user path: storage is constructed against a tempfile directory throughout, and timing decisions are pure functions taking now, so nothing sleeps.

Claude-Session: https://claude.ai/code/session_01QddAyPzAbdC5bRK7nccDv5

Changes

  • Code changes
  • Tests added/updated
  • Docs updated (README/CHANGELOG)

Checklist

  • cargo fmt passes
  • cargo clippy -- -D warnings passes
  • cargo test passes

Screenshots / Logs (if UI/behavioral change)

Screenshots

Sync toasts now shown in the lower-right corner on refreshes instead of an input-blocking modal. (Note that if tasks are added in the current view for the user, their selection remains the same when the sync completed, even if new items are added to their current view/section)

image image

Logs

[13:59:25.750] Loading 1 backend(s) from database
[13:59:25.756] ✅ Loaded backend: My Todoist (todoist)
[13:59:25.756] Loaded 1 backend instance(s)
[13:59:25.927] ✅ Added backend: My Todoist (todoist)
[13:59:25.928] AppComponent: Starting initial sync
[13:59:25.928] AppComponent: Initial sync scheduled
[13:59:25.928] 🔄 Starting sync process...
[13:59:26.029] Background: Received action StartSync
[13:59:26.029] Background: Received action InitialDataLoaded { projects: ... redacted ... }
[13:59:26.030] Background: Cleaned up 1 finished tasks
[13:59:26.030] Sync already in progress, ignoring
[13:59:26.030] InitialData: Loaded 61 projects, 51 labels, 5 sections, 20 tasks
[13:59:26.030] AppComponent: Set initial sidebar selection to Today
[13:59:26.030] AppComponent: Set initial sidebar selection after initial data load
[13:59:26.030] AppComponent: Scheduled data fetch for initial sidebar selection
[13:59:26.030] InitialData: Updated all component data after initial data load
[13:59:26.131] Background: Received action DataLoaded { projects: [Model { ... redacted ... } ] }
[13:59:26.131] Background: Cleaned up 1 finished tasks
[13:59:26.131] Data: Loaded 61 projects, 51 labels, 5 sections, 20 tasks
[13:59:26.131] Data: Updated all component data after data load
[13:59:27.099] ✅ Fetched 61 projects from backend
[13:59:27.099] ✅ Fetched 316 tasks from backend
[13:59:27.099] ✅ Fetched 51 labels from backend
[13:59:27.099] ✅ Fetched 5 sections from backend
[13:59:27.099] 💾 Storing data in local database...
[13:59:27.119] ✅ Stored projects in database
[13:59:27.121] ✅ Stored labels in database
[13:59:27.121] ✅ Stored sections in database
[13:59:27.167] ✅ Stored tasks in database
[13:59:27.247] Background: Received action SyncCompleted(Success)
[13:59:27.247] Background: Cleaned up 1 finished tasks
[13:59:27.247] Sync: Completed with status Success
[13:59:27.348] Background: Received action DataLoaded { projects: ... redacted ... }
[13:59:27.348] Background: Cleaned up 1 finished tasks
[13:59:27.348] Data: Loaded 61 projects, 51 labels, 5 sections, 20 tasks
[13:59:27.349] Data: Updated all component data after data load
[13:59:51.643] Global key: 'q' - quitting application

Every launch deleted the SQLite cache, then blocked the UI behind a
centered "Loading data" overlay while it re-downloaded the entire
account over four sequential API round-trips. The cache location was
hardcoded, so users could not move it off a synced home directory, and
`cargo test` deleted a real installation's database on every run.
`auto_sync_interval_minutes` was parsed, defaulted and validated, but
never read by any runtime code — no timer ever existed.

Storage:

- New `[storage] data_dir` config key. A leading `~` is expanded and
  relative paths resolve against the working directory; unset keeps the
  platform default. An unusable path fails at startup with the path in
  the error rather than silently falling back, which would send writes
  to the database the user was trying to move away from.
- The cache is no longer deleted at startup, and the schema is created
  idempotently, so it survives across launches.
- The backend row's UUID is derived from (type, name) instead of being
  generated per launch. An existing row's UUID is adopted as-is, so a
  database written by an earlier version keeps its cache instead of
  having every project and task orphaned under a dead backend_uuid.
  Duplicate rows left by older debug runs are collapsed before the new
  unique index is built.
- The database file is created 0600 on Unix; it holds the API token.

Sync:

- Reconciles instead of rebuilding: rows are upserted on
  (backend_uuid, remote_id) and rows the remote no longer returns are
  deleted. The delete pass runs before parent re-linking, so completing
  a parent task cannot cascade its still-open subtasks out of the cache.
  Local UUIDs stay stable across syncs.
- An empty fetch never wipes an entity table.
- The four fetches run concurrently; per-resource error handling is
  unchanged, including sections degrading to empty rather than failing.

Interface:

- Startup paints cached data immediately and syncs in the background.
- Sync state appears as a toast in the lower right of the task list —
  syncing, synced, or failed — instead of a modal that ate keystrokes.
  A failed sync now reports as failed rather than as success.
- `auto_sync_interval_minutes` drives a real timer; 0 disables it, and
  `r` still forces a sync. Every terminal outcome records the attempt,
  so a failing backend is retried once per interval rather than ten
  times a second.
- Task selection follows the selected task across a sync-driven reload,
  so data arriving out of band cannot move the cursor under the user's
  hands. Reloads the user asked for — a task operation, a sidebar
  change, a debug refresh — leave the cursor on the row it was on, so
  marking an overdue task due today does not drag the cursor along as
  the task moves out of the Overdue section. Each reload carries its
  own policy, rather than the list guessing from shared state.

Behavior change: a locally completed or deleted task is removed from the
cache at the next sync rather than lingering until the next launch, so
it can be restored from the list for at most one auto-sync interval.
Excluding those rows instead would make them permanent tombstones, since
the remote never returns them again.

README and the architecture, configuration and development docs
described a system that never existed — an in-memory database, a
five-minute staleness check, and a dependency list naming crates the
project does not use. They now describe the code.

Tests no longer touch any user path: storage is constructed against a
tempfile directory throughout, and timing decisions are pure functions
taking `now`, so nothing sleeps.

Claude-Session: https://claude.ai/code/session_01QddAyPzAbdC5bRK7nccDv5
@romaintb

Copy link
Copy Markdown
Owner

Hello @stevenwcarter and thanks for the PR !
We are actually working on this exactly with @rsheyd, PR is #203
I will favor #203 over this one, as it's already a big change, but once we are done with this, I would gladly review your PR to see if there are more goodies we can integrate ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants