fix(knowledge): treat Google Workspace users without Gmail or Calendar as out of scope, not listing failures (#8168)

* fix(knowledge): treat Google Workspace users without Gmail or Calendar as out of scope, not listing failures

Admin-mode Gmail recorded a Directory user without a mailbox, and Calendar recorded a
403 notACalendarUser, as per-user listing failures. Both are standing account properties,
so the connector stayed partial, deletion reconciliation never ran, and the scheduler
re-probed the same accounts every retry window.

- Directory enumeration no longer schedules Gmail users whose mailbox is not set up; the
  hourly Directory refresh picks them up once provisioned. A partition queued earlier
  completes with an empty page instead of a failure.
- A Calendar 403 whose only reason is notACalendarUser completes the user's partition
  cleanly and re-probes it no sooner than the Directory refresh (permissions keep their
  own refresh cadence). Bare forbidden and mixed-reason 403s stay retryable failures.

* fix(knowledge): keep a Google Workspace user's visible documents until a missing service outlasts propagation

Google applies service and organizational-unit changes within 24 hours, so a missing
mailbox or notACalendarUser can be transient for a user whose documents are already
indexed, and a mid-listing answer does not prove the whole account lacks the service.

- The scheduler skips a service-not-enabled user only on their first provider page, and
  only when readers see none of their documents or the condition was first observed at
  least 24 hours ago. Otherwise it is a retained failure, as before, whose first
  observation is kept in the partition failure; after 24 hours a mid-listing user
  restarts from their first page.
- Permission passes skip on the first page, since a retained failure refreshes nothing.
- Gmail Directory enumeration keeps scheduling a user without a mailbox while readers
  still see their mail; the crawl reports the missing mailbox to the scheduler instead of
  completing the user.
- Visibility is read through doc_acl_gin_idx for the user's token behind an OFFSET 0
  fence, bounded by that user's grants.

* fix(knowledge): read a user's visible documents from the ACL index and cover the probe in PostgreSQL

The visibility probe moves to its own module so it can run against a real database.
Planned inline, LIMIT 1 made a sequential scan of the document table look cheaper than
doc_acl_gin_idx, because PostgreSQL cannot estimate array overlap. A materialized CTE now
reads the user's grants from the index first, bounding the probe by that user's grants.
The email goes through userToken, so a mixed-case or padded directory address matches the
normalized ACL token.

The new PostgreSQL integration test covers fresh, stale and missing permission evidence,
another user's grant, another connector, excluded, archived and deleted documents, and
email normalization, and runs in the Search progress PostgreSQL CI step.

Author: waleedlatif1

.github/workflows/test-build.yml

@@ -271,6 +271,7 @@ jobs:
           lib/knowledge/__integration__/connector-deferral.integration.ts
           lib/knowledge/__integration__/stored-document-recovery.integration.ts
           lib/knowledge/__integration__/connector-partition-work.integration.ts
+          lib/knowledge/__integration__/user-document-visibility.integration.ts
           lib/knowledge/__integration__/listing-continuation.integration.ts
           lib/knowledge/__integration__/member-scope-renewal.integration.ts
           lib/knowledge/__integration__/slack-empty-threads.integration.ts

apps/sim/connectors/google-workspace/company-crawl.test.ts

@@ -2,9 +2,11 @@
 import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
 import { GoogleApiError, readGoogleApiError } from '@/connectors/google-workspace/api-errors'
 import {
+  GoogleWorkspaceMailboxNotSetup,
   getGoogleWorkspaceDocument,
   InvalidGoogleWorkspaceCursor,
   listGoogleWorkspaceDocuments,
+  serviceNotEnabledFailure,
   validateGoogleWorkspaceConfig,
 } from '@/connectors/google-workspace/company-crawl'
 import type { ConnectorConfig, ExternalDocument } from '@/connectors/types'
@@ -483,44 +485,52 @@ describe('Google Workspace per-user central crawl', () => {
     }
   )
 
-  it('skips an explicitly unprovisioned Gmail mailbox before requesting a token', async () => {
+  it('leaves an explicitly unprovisioned Gmail mailbox to the scheduler before requesting a token', async () => {
     directory([USER('alice', undefined, { isMailboxSetup: false }), USER('bob')])
-    const ctx = context()
-    const first = await list(ctx)
-    expect(first.listingFailures?.samples[0]).toEqual({
-      scope: 'alice@corp.com',
+    const ctx: Record<string, unknown> = context()
+    const error = await list(ctx).catch((caught: unknown) => caught)
+    expect(error).toBeInstanceOf(GoogleWorkspaceMailboxNotSetup)
+    expect(serviceNotEnabledFailure(error)).toEqual({
       operation: 'directory.users.get',
       reasons: ['mailboxNotSetup'],
     })
+    expect(ctx.reconciliationUnsafe).toBeUndefined()
     expect(ctx.getDelegatedAccessToken).not.toHaveBeenCalled()
-    const second = await list(context(), first.nextCursor)
-    expect(second.documents[0].acl).toEqual(['u:bob@corp.com'])
+    expect(listUserDocuments).not.toHaveBeenCalled()
   })
 
   it('does not use Gmail mailbox eligibility for Calendar', async () => {
     directory([USER('alice', undefined, { isMailboxSetup: false })])
     expect((await list(context(), undefined, CONFIG, 'google_calendar')).documents).toHaveLength(1)
   })
 
-  it.each(['forbidden', 'notACalendarUser'])(
-    'isolates explicit Calendar list access failures (%s) without claiming a disabled service',
-    async (reason) => {
+  it.each([{ reasons: ['forbidden'] }, { reasons: ['forbidden', 'notACalendarUser'] }])(
+    'isolates explicit Calendar list access failures ($reasons) without claiming a disabled service',
+    async ({ reasons }) => {
       listUserDocuments.mockRejectedValueOnce(
-        new GoogleApiError('calendar.events.list', 403, [reason])
+        new GoogleApiError('calendar.events.list', 403, reasons)
       )
       const first = await list(context(), undefined, CONFIG, 'google_calendar')
       expect(first.listingFailures?.samples[0]).toEqual({
         scope: 'alice@corp.com',
         operation: 'calendar.events.list',
         status: 403,
-        reasons: [reason],
+        reasons,
       })
       const second = await list(context(), first.nextCursor, CONFIG, 'google_calendar')
       expect(second.documents[0].acl).toEqual(['u:bob@corp.com'])
       expect(second.reconciliationSafe).toBe(false)
     }
   )
 
+  it('leaves a user without the Calendar service to the scheduler instead of recording a failure', async () => {
+    const error = new GoogleApiError('calendar.events.list', 403, ['notACalendarUser'])
+    listUserDocuments.mockRejectedValueOnce(error)
+    const ctx: Record<string, unknown> = context()
+    await expect(list(ctx, undefined, CONFIG, 'google_calendar')).rejects.toBe(error)
+    expect(ctx.reconciliationUnsafe).toBeUndefined()
+  })
+
   it.each([{ error: { code: 403 } }, { error: { code: 403, errors: [], details: [] } }])(
     'propagates a Calendar 403 without reason codes: %j',
     async (body) => {
@@ -626,10 +636,9 @@ describe('Google Workspace per-user central crawl', () => {
   })
 
   it('bounds retained failure samples while counting every unavailable user', async () => {
-    directory(
-      Array.from({ length: 15 }, (_, index) =>
-        USER(`user-${index}`, undefined, { isMailboxSetup: false })
-      )
+    directory(Array.from({ length: 15 }, (_, index) => USER(`user-${index}`)))
+    listUserDocuments.mockRejectedValue(
+      new GoogleApiError('gmail.threads.list', 400, ['failedPrecondition'])
     )
     let cursor: string | undefined
     let final
@@ -643,7 +652,7 @@ describe('Google Workspace per-user central crawl', () => {
       reconciliationSafe: false,
     })
     expect(final?.listingFailures?.samples).toHaveLength(10)
-    expect(listUserDocuments).not.toHaveBeenCalled()
+    expect(listUserDocuments).toHaveBeenCalledTimes(15)
   })
 })
 

apps/sim/connectors/google-workspace/company-crawl.ts

@@ -161,23 +161,52 @@ function ownerDocument(document: ExternalDocument, access: DelegatedUser): Exter
   return { ...document, acl: [`u:${access.user.email}`] }
 }
 
-/** Isolates narrow user-list failures; delegation, known scope errors and quota errors still fail. */
+/** A Directory user without a Gmail mailbox; the user scheduler decides whether to skip them. */
+export class GoogleWorkspaceMailboxNotSetup extends Error {
+  constructor() {
+    super('Google Workspace user has no Gmail mailbox')
+    this.name = 'GoogleWorkspaceMailboxNotSetup'
+  }
+}
+
+/**
+ * Evidence that a user lacks the service itself: no Gmail mailbox, or a Calendar list 403 whose
+ * only reason is `notACalendarUser`. Admin changes to a user's services can take up to a day to
+ * settle, so the user scheduler, which can see indexed documents, decides when this is a skip.
+ */
+export function serviceNotEnabledFailure(
+  error: unknown
+): Omit<ExternalListingFailures['samples'][number], 'scope'> | null {
+  if (error instanceof GoogleWorkspaceMailboxNotSetup)
+    return { operation: 'directory.users.get', reasons: ['mailboxNotSetup'] }
+  return error instanceof GoogleApiError &&
+    error.reasonsComplete &&
+    error.status === 403 &&
+    error.diagnostic?.operation === 'calendar.events.list' &&
+    error.diagnostic.reasons.length === 1 &&
+    error.diagnostic.reasons[0] === 'notACalendarUser'
+    ? { operation: 'calendar.events.list', status: 403, reasons: ['notACalendarUser'] }
+    : null
+}
+
+/** Isolates narrow user-list failures; a missing Calendar service propagates to the scheduler. */
 function userListingFailure(
   error: unknown,
   provider: GoogleWorkspaceProvider
 ): Omit<ExternalListingFailures['samples'][number], 'scope'> | null {
   if (!(error instanceof GoogleApiError) || !error.diagnostic || !error.reasonsComplete) return null
   const reasons = error.diagnostic.reasons
   const isolated =
-    provider === 'gmail'
+    !serviceNotEnabledFailure(error) &&
+    (provider === 'gmail'
       ? error.diagnostic.operation === 'gmail.threads.list' &&
         error.status === 400 &&
         reasons.length > 0 &&
         reasons.every((reason) => reason === 'failedPrecondition')
       : error.diagnostic.operation === 'calendar.events.list' &&
         error.status === 403 &&
         reasons.length > 0 &&
-        reasons.every((reason) => reason === 'forbidden' || reason === 'notACalendarUser')
+        reasons.every((reason) => reason === 'forbidden' || reason === 'notACalendarUser'))
   return isolated
     ? { operation: error.diagnostic.operation, status: error.status, reasons: [...reasons] }
     : null
@@ -317,9 +346,8 @@ export async function listGoogleWorkspaceDocuments(
     })
     return emptyPage(advance())
   }
-  if (provider === 'gmail' && user.isMailboxSetup === false) {
-    return failedUser({ operation: 'directory.users.get', reasons: ['mailboxNotSetup'] })
-  }
+  if (provider === 'gmail' && user.isMailboxSetup === false)
+    throw new GoogleWorkspaceMailboxNotSetup()
   const access: PageAccess = {
     provider,
     user,

apps/sim/connectors/listing-failures.ts

@@ -4,17 +4,17 @@ import { CONNECTOR_SOURCE_REASON_STATES } from '@/connectors/source-error'
 export const MAX_LISTING_FAILURE_SAMPLES = 10
 
 /** Persist only bounded scope identifiers and provider-owned codes, never raw errors or content. */
+export const listingFailureSampleSchema = z.object({
+  scope: z.string().min(1).max(254),
+  operation: z.string().min(1).max(96),
+  status: z.number().int().min(100).max(599).optional(),
+  reasons: z.array(z.string().min(1).max(64)).max(16),
+  reasonState: z.enum(CONNECTOR_SOURCE_REASON_STATES).optional(),
+  /** When a standing account condition was first observed, bounding how long it is retained. */
+  since: z.string().datetime().optional(),
+})
+
 export const listingFailuresSchema = z.object({
   count: z.number().int().positive().max(Number.MAX_SAFE_INTEGER),
-  samples: z
-    .array(
-      z.object({
-        scope: z.string().min(1).max(254),
-        operation: z.string().min(1).max(96),
-        status: z.number().int().min(100).max(599).optional(),
-        reasons: z.array(z.string().min(1).max(64)).max(16),
-        reasonState: z.enum(CONNECTOR_SOURCE_REASON_STATES).optional(),
-      })
-    )
-    .max(MAX_LISTING_FAILURE_SAMPLES),
+  samples: z.array(listingFailureSampleSchema).max(MAX_LISTING_FAILURE_SAMPLES),
 })

apps/sim/connectors/types.ts

@@ -205,6 +205,8 @@ export interface ExternalListingFailures {
     status?: number
     reasons: string[]
     reasonState?: ConnectorSourceReasonState
+    /** When a standing account condition was first observed, bounding how long it is retained. */
+    since?: string
   }[]
 }
 

apps/sim/lib/knowledge/__integration__/user-document-visibility.integration.ts

@@ -0,0 +1,90 @@
+/** Real PostgreSQL coverage for the probe that keeps a user's visible documents from being skipped. */
+import { db } from '@sim/db'
+import { document, knowledgeConnector, organization, user, workspace } from '@sim/db/schema'
+import { generateId } from '@sim/utils/id'
+import { eq, inArray } from 'drizzle-orm'
+import { afterAll, afterEach, beforeEach, describe, expect, it } from 'vitest'
+import { seedKnowledgeAclFixture } from '@/lib/knowledge/__integration__/seed-source-access-fixture'
+import { hasVisibleUserDocuments } from '@/lib/knowledge/connectors/user-document-visibility'
+
+const DAY_MS = 24 * 60 * 60 * 1000
+
+describe('Visible user documents in PostgreSQL', () => {
+  let owner: Awaited<ReturnType<typeof seedKnowledgeAclFixture>>
+  let otherConnectorId: string
+  const email = () => `${owner.aliceId}@fixture.test`
+
+  beforeEach(async () => {
+    owner = await seedKnowledgeAclFixture(undefined, { connectorType: 'google_drive' })
+    otherConnectorId = generateId()
+    await db.insert(knowledgeConnector).values({
+      id: otherConnectorId,
+      knowledgeBaseId: owner.knowledgeBaseId,
+      connectorType: 'google_drive',
+      sourceConfig: {},
+      accessMode: 'admin',
+      status: 'active',
+      credentialId: owner.credentialId,
+    })
+  })
+  afterEach(async () => {
+    await db.delete(workspace).where(eq(workspace.id, owner.workspaceId))
+    await db.delete(organization).where(eq(organization.id, owner.organizationId))
+    await db.delete(user).where(inArray(user.id, [owner.aliceId, owner.bobId]))
+  })
+  afterAll(() => db.$client.end())
+
+  const insert = (overrides: Partial<typeof document.$inferInsert> = {}) =>
+    db.insert(document).values({
+      id: generateId(),
+      knowledgeBaseId: owner.knowledgeBaseId,
+      connectorId: owner.connectorId,
+      externalId: generateId(),
+      filename: 'Event.txt',
+      mimeType: 'text/plain',
+      fileUrl: '',
+      fileSize: 0,
+      acl: [`u:${email()}`],
+      aclVerifiedAt: new Date(),
+      ...overrides,
+    })
+
+  it('finds a live, included document granted to the user with fresh permission evidence', async () => {
+    await insert()
+    expect(await hasVisibleUserDocuments(owner.connectorId, email())).toBe(true)
+  })
+
+  it('matches a mixed-case, padded directory email to the normalized ACL token', async () => {
+    await insert()
+    expect(await hasVisibleUserDocuments(owner.connectorId, `  ${email().toUpperCase()} `)).toBe(
+      true
+    )
+  })
+
+  it.each([
+    {
+      label: 'permission evidence older than the freshness limit',
+      overrides: () => ({ aclVerifiedAt: new Date(Date.now() - DAY_MS - 60_000) }),
+    },
+    { label: 'no permission evidence', overrides: () => ({ aclVerifiedAt: null }) },
+    {
+      label: 'a grant to a different user',
+      overrides: () => ({ acl: [`u:${owner.bobId}@fixture.test`] }),
+    },
+    {
+      label: 'a document in another connector',
+      overrides: () => ({ connectorId: otherConnectorId }),
+    },
+    { label: 'a user-excluded document', overrides: () => ({ userExcluded: true }) },
+    { label: 'an archived document', overrides: () => ({ archivedAt: new Date() }) },
+    { label: 'a deleted document', overrides: () => ({ deletedAt: new Date() }) },
+  ])('ignores $label', async ({ overrides }) => {
+    await insert(overrides())
+    expect(await hasVisibleUserDocuments(owner.connectorId, email())).toBe(false)
+  })
+
+  it('refuses an address that cannot be a user token', async () => {
+    await insert()
+    expect(await hasVisibleUserDocuments(owner.connectorId, '   ')).toBe(false)
+  })
+})