perf: avoid duplicate database row loads and handle backpressure

[appflowy]

Description

Opening a database fetched row snapshots in the page-view response even though the view loader only uses the page collab. Blob admission errors could also be decoded as protobuf failures and trigger individual row loads, amplifying server overload.

The view loader now requests include_rows=false and uses the existing paginated blob pipeline for rows. Blob requests recognize JSON application errors and retry admission overload with bounded, jittered backoff. When retries are exhausted, the database shows a Retry action and keeps row loading paused.

• Include the row-response option in request deduplication while preserving full-row responses for default callers and compatibility with older servers.
• Retry the same cursor/RID at most three times per page, honor server cooldown hints, and cancel unfinished requests/backoff when the last database owner closes.
• Preserve staged rows until the terminal page commits, and share work across concurrent database views.
• Restore readiness when returning to a successfully loaded view mode; ignore overload callbacks from an inactive mode so Retry cannot become stuck.

This preserves the existing 256-item / 16 MiB blob pages and complete seed walk needed for filters, sorting, grouping, and counts. It does not introduce viewport-only row semantics.

Validation

• All 101 tests passed across the view-loader, blob-prefetch, fetch-deduplication, HTTP collab API, and Database lifecycle suites.
• pnpm type-check passed.
• ESLint passed for all changed TypeScript/TSX files.
• git diff --check passed.
• A fresh multi-browser/live-server run was not performed for this PR.

Checklist

• Updated compatibility documentation and explanatory comments.
• Added regression coverage for request options, overload responses, retry limits, cancellation, shared prefetch, and view-mode transitions.
• Tested in multiple browsers/operating systems.

Feature preview: not included; validation above exercises transport behavior and the retry lifecycle.

Summary by Sourcery

Prevent duplicate database row loads and make blob prefetch resilient to admission backpressure.

Bug Fixes:

• Prevent database overload failures from being misinterpreted as protobuf errors and triggering individual row loads.
• Keep row loading paused after admission retries are exhausted, with an explicit Retry action and correct readiness across view-mode transitions.

Enhancements:

• Avoid duplicate database row snapshot loads by allowing page-view requests to omit rows while preserving full-row compatibility and option-aware request deduplication.
• Add bounded, jittered backoff with server cooldown support, cancellation, shared prefetch ownership, and atomic staged-page commits for database blob loading.

Documentation:

• Document the optimized database page-loading and blob admission-retry behavior, including compatibility and cancellation semantics.

Tests:

• Add regression coverage for row-fetch options, request deduplication, JSON application errors, retry limits, cancellation, shared prefetch, staged commits, and database lifecycle transitions.

[sourcery-ai]

Sorry @appflowy, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 16 hours and 6 minutes by commenting @sourcery-ai review. Upgrade to get a review now.

[sourcery-ai]

Reviewer's Guide

The PR removes redundant database row snapshots from optimized page-view loads, adds bounded and cancelable backpressure handling for paginated blob prefetches, and coordinates shared prefetch lifecycle state so overloads pause row loading with an explicit Retry path while preserving compatibility and complete seed-walk semantics.

Sequence diagram for optimized database page loading

sequenceDiagram
    participant ViewLoader
    participant FetchDedup as FetchDeduplication
    participant PageAPI as PageViewAPI
    participant BlobPrefetch as BlobPrefetchPipeline
    participant BlobAPI as DatabaseBlobAPI

    ViewLoader->>FetchDedup: fetchPageCollab(workspaceId, viewId, { includeRows: false })
    FetchDedup->>PageAPI: getPageCollab(..., { includeRows: false })
    PageAPI-->>ViewLoader: encoded_collab + metadata + empty row_data
    ViewLoader->>BlobPrefetch: prefetchDatabaseBlobDiff(workspaceId, databaseId)
    BlobPrefetch->>BlobAPI: databaseBlobDiff(page cursor, RID)
    BlobAPI-->>BlobPrefetch: paginated blob pages
    BlobPrefetch-->>ViewLoader: complete seed set after terminal Ready page

Loading

Sequence diagram for database blob backpressure retry

sequenceDiagram
    participant Database
    participant Prefetch as BlobPrefetch
    participant API as DatabaseBlobAPI
    participant Server

    Database->>Prefetch: prefetchDatabaseBlobDiff(...)
    Prefetch->>API: databaseBlobDiff(same cursor, RID, signal)
    API->>Server: POST /database/{databaseId}/blob/diff
    alt admission overload code 1079 or HTTP 429
        Server-->>API: JSON application error + retry hint
        API-->>Prefetch: APIError
        loop at most 3 retries
            Prefetch->>Prefetch: waitForDatabaseBlobRetry(backoff, signal)
            Prefetch->>API: databaseBlobDiff(same cursor, RID, signal)
        end
    else terminal page succeeds
        Server-->>API: binary protobuf page
        API-->>Prefetch: decoded diff page
        Prefetch-->>Database: commit staged pages and expose seeds
    end
    alt retries exhausted or cooldown exceeds 30 seconds
        Prefetch-->>Database: rejection
        Database-->>Database: keep seed gate closed and show Retry
    end

Loading

File-Level Changes

Change	Details	Files
Avoid duplicate row snapshot transport while preserving caller compatibility and safe request deduplication.	Added an explicit page-view option to omit row snapshots, defaulting to full-row responses. Included the normalized option in deduplication keys so optimized and legacy requests cannot share incompatible results. Updated the database view loader to use metadata plus the existing row seed/realtime pipeline.	src/application/services/js-services/http/collab-api.ts src/application/services/js-services/fetch.ts src/application/view-loader/index.ts src/application/services/js-services/__tests__/fetch.test.ts src/application/services/js-services/http/__tests__/collab-api.test.ts src/application/__tests__/view-loader.test.ts doc/web-server-compatibility.md
Make blob-diff loading resilient to admission backpressure without decoding application errors as protobuf data.	Detect JSON error envelopes from binary responses, including HTTP 200 errors and HTTP 429 overloads. Retry the same page cursor/RID up to three times with jittered exponential backoff and server cooldown hints. Stop retrying for permanent errors or cooldowns over 30 seconds and preserve existing permission/protocol handling.	src/application/services/js-services/http/collab-api.ts src/application/database-blob/request-retry.ts src/application/database-blob/index.ts src/application/services/js-services/http/__tests__/collab-api.test.ts src/application/database-blob/__tests__/prefetch-dedup.test.ts doc/web-server-compatibility.md
Coordinate shared prefetch ownership, cancellation, and atomic seed publication across concurrent database views.	Share in-flight and terminal prefetch work across database owners while retaining provisional pages until terminal commit. Abort unfinished transport or backoff work only after the final owner releases the database. Prevent canceled or late responses from publishing seeds or cached RIDs, and allow close/reopen to reuse valid work.	src/application/database-blob/index.ts src/application/database-blob/request-retry.ts src/application/database-blob/__tests__/prefetch-dedup.test.ts doc/web-server-compatibility.md
Expose overload recovery in the database UI while keeping row loading paused and handling view-mode transitions safely.	Show a Retry action after admission retries are exhausted without opening individual row syncs. Restore readiness when a successful prefetch becomes active again and ignore stale overload failures from inactive modes. Reset overload state for read-only transitions and lifecycle changes.	src/components/database/Database.tsx src/components/database/__tests__/Database.prefetch-lifecycle.test.tsx

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.