feat(desktop): keep the machine awake for background chats and yield the page to the user (#8671)

* feat(desktop): keep the machine awake for background chats and yield the page to the user

- Prevent sleep: while any chat has desktop work claimed on this machine, the
  app holds a prevent-app-suspension power blocker, released as soon as the
  last result is delivered. On by default, with a switch in Desktop settings
  that notes a closed lid still sleeps the computer. The switch is a new
  optional bridge method, so older shells simply do not show it.
- Takeover: when the user clicks, scrolls or types in the page an agent drives,
  that chat's next browser action waits, marked as needing attention, until
  the user has left the page alone for a few seconds, then resumes in the same
  page. A Stop still cancels it while it waits.

* fix(desktop): give the page back to the user mid-action

- A browser action now rechecks, right before each input it sends, whether the
  user is working in the page. Between a batch's steps or during a click's
  target probes, the user wins and the action stops before its next input.
- The wait for the user is bounded inside the action's own deadline. A user
  who keeps working gets the model a clear "not run" instead of a timeout with
  an unknown outcome.
- The user's activity belongs to the tab they acted in. Once the agent drives
  another tab (the old one closed or replaced, the session reopened), that
  activity no longer holds anything up.
- `createSleepBlocker` is documented.

* test(desktop): read the blockers the OS holds instead of asserting the mock was called

* fix(desktop): yield the page only for background calls that act on it

- Only a call from the background executor yields the page to the user.
  A call from the chat view the user is watching acts exactly as before:
  there the user steers the agent directly.
- A background call that only reads the page (snapshot, read text, list
  tabs) never waits for the user, since it cannot collide with their input.
- Which browser tools only observe the page now lives in
  @sim/browser-protocol, shared by the chat view's replay policy and the
  driver.
- The prevent-sleep switch shows only when Sim runs chats on this device
  in the background.
- Tests: a chat-view call acts at once under user input; a background call
  waits, and a background read does not; a user who starts working
  mid-action stops it before its next input; sign-out releases the sleep
  blocker while a call is still running.

* test(desktop): give the notification tests the prevent-sleep setter the settings service now requires

* test(desktop): show the prevent-sleep switch only on a device that runs chats in the background

Author: waleedlatif1

apps/desktop/e2e/background-executor.spec.ts

@@ -359,6 +359,11 @@ async function launch(userData: string): Promise<{ app: ElectronApplication; win
       SIM_DESKTOP_USER_DATA: userData,
     },
   })
+  // A dialog listener stops Playwright auto-dismissing page dialogs, so the desktop's own handling
+  // decides their outcome exactly as it does in production.
+  const leaveDialogsToDesktop = (page: Page) => page.on('dialog', () => {})
+  app.context().pages().forEach(leaveDialogsToDesktop)
+  app.context().on('page', leaveDialogsToDesktop)
   const window = await app.firstWindow()
   return { app, window }
 }
@@ -730,6 +735,116 @@ test.describe('background executor', () => {
       expect(device).toEqual({ deviceId, protocolVersion: 1 })
     })
   })
+
+  test('I: the agent yields its page while the user works in it, then takes it back', async () => {
+    app = (await launch(mkdtempSync(join(tmpdir(), 'sim-executor-i-')))).app
+    const deviceId = await registeredDevice()
+    const opened = sim.issue(deviceId, CHAT_A, 'browser_open_url', {
+      url: `${sim.origin}/counter?chat=I`,
+    })
+    const outline = ((await settled(opened)).data?.snapshot as { outline: string }).outline
+    const button = refFor(outline, 'Count visit')
+
+    const typeInAgentPage = () =>
+      app?.evaluate(({ webContents }) => {
+        const page = webContents
+          .getAllWebContents()
+          .find((contents) => contents.getURL().includes('/counter?chat=I'))
+        page?.sendInputEvent({ type: 'keyDown', keyCode: 'Tab' })
+        page?.sendInputEvent({ type: 'keyUp', keyCode: 'Tab' })
+      })
+    await typeInAgentPage()
+    const click = sim.issue(deviceId, CHAT_A, 'browser_click', { elementId: button })
+    for (let i = 0; i < 4; i++) {
+      await sleep(500)
+      await typeInAgentPage()
+    }
+    const lastUserInputAt = Date.now()
+
+    await check('I: the click waits until the user stops, then runs once', async () => {
+      const completion = await settled(click, 30_000)
+      expect(completion.status, completion.message).toBe('success')
+      expect(completion.at - lastUserInputAt).toBeGreaterThanOrEqual(3_000)
+      await expect.poll(() => sim.hits.get('I')).toBe(1)
+    })
+  })
+
+  test('I: an action the user never stops working long enough for does not run', async () => {
+    app = (await launch(mkdtempSync(join(tmpdir(), 'sim-executor-i2-')))).app
+    const deviceId = await registeredDevice()
+    const opened = sim.issue(deviceId, CHAT_A, 'browser_open_url', {
+      url: `${sim.origin}/counter?chat=I2`,
+    })
+    const outline = ((await settled(opened)).data?.snapshot as { outline: string }).outline
+    const button = refFor(outline, 'Count visit')
+    const typeInAgentPage = () =>
+      app?.evaluate(({ webContents }) => {
+        const page = webContents
+          .getAllWebContents()
+          .find((contents) => contents.getURL().includes('/counter?chat=I2'))
+        page?.sendInputEvent({ type: 'keyDown', keyCode: 'Tab' })
+        page?.sendInputEvent({ type: 'keyUp', keyCode: 'Tab' })
+      })
+
+    await typeInAgentPage()
+    const click = sim.issue(deviceId, CHAT_A, 'browser_click', { elementId: button })
+    let typing = true
+    const keepTyping = (async () => {
+      while (typing) {
+        await typeInAgentPage()
+        await sleep(1_000)
+      }
+    })()
+
+    await check('I: the click reports it never ran, instead of timing out', async () => {
+      const completion = await settled(click, 60_000)
+      typing = false
+      await keepTyping
+      expect(completion.status).toBe('error')
+      expect(completion.message).toContain('Not run: the user kept working in this page')
+      expect(completion.data).not.toMatchObject({ outcomeUnknown: true })
+      expect(sim.hits.get('I2') ?? 0).toBe(0)
+    })
+  })
+
+  test('J: the machine stays awake only while a chat has work running', async () => {
+    app = (await launch(mkdtempSync(join(tmpdir(), 'sim-executor-j-')))).app
+    await app.evaluate(({ powerSaveBlocker }) => {
+      const log: string[] = []
+      const active = new Set<number>()
+      let next = 1
+      const target = globalThis as typeof globalThis & { __sleepBlocks?: string[] }
+      target.__sleepBlocks = log
+      powerSaveBlocker.start = (type) => {
+        log.push(`start:${type}`)
+        active.add(next)
+        return next++
+      }
+      powerSaveBlocker.stop = (id) => {
+        log.push('stop')
+        active.delete(id)
+        return true
+      }
+      powerSaveBlocker.isStarted = (id) => active.has(id)
+    })
+    const deviceId = await registeredDevice()
+    const sleepLog = () =>
+      app?.evaluate(() => (globalThis as { __sleepBlocks?: string[] }).__sleepBlocks ?? [])
+
+    const run = sim.issue(deviceId, CHAT_B, 'terminal', {
+      operation: 'run',
+      args: { command: 'sleep 3; echo awake', waitSeconds: 30 },
+    })
+    await check('J: a blocker is held while the command runs', async () => {
+      await expect.poll(sleepLog, { timeout: 15_000 }).toEqual(['start:prevent-app-suspension'])
+    })
+    await check('J: it is released once the result is delivered', async () => {
+      await settled(run)
+      await expect
+        .poll(sleepLog, { timeout: 10_000 })
+        .toEqual(['start:prevent-app-suspension', 'stop'])
+    })
+  })
 })
 
 function readFileSafe(path: string): string {

apps/desktop/src/main/browser-agent/driver.test.ts

@@ -79,6 +79,102 @@ describe('executeTool', () => {
     driver = freshDriver()
   })
 
+  /** The user presses a key in the page the agent drives, as the browser reports it. */
+  function typeInAgentPage(contents: WebContents) {
+    const calls: ReadonlyArray<readonly unknown[]> = vi.mocked(contents.on).mock.calls
+    const listener = calls.find(([event]) => event === 'before-input-event')?.[1]
+    if (typeof listener !== 'function') throw new Error('No input listener on the agent page')
+    listener({}, { type: 'keyDown', isAutoRepeat: false })
+  }
+
+  it('lets a call from the chat view act at once while the user works in the page', async () => {
+    await driver.executeTool('chat-test', 'browser_open_tab', {})
+    typeInAgentPage(session.requireTab().view.webContents)
+    expect(session.msSinceUserIntervention()).not.toBeNull()
+    vi.useFakeTimers()
+    try {
+      let settled = false
+      const opening = driver.executeTool('chat-test', 'browser_open_tab', {}).then((result) => {
+        settled = true
+        return result
+      })
+      await vi.advanceTimersByTimeAsync(100)
+
+      expect(settled).toBe(true)
+      expect((await opening).ok).toBe(true)
+    } finally {
+      vi.useRealTimers()
+    }
+  })
+
+  it('makes a background call wait until the user leaves the page alone', async () => {
+    await driver.executeTool('chat-test', 'browser_open_tab', {})
+    typeInAgentPage(session.requireTab().view.webContents)
+    vi.useFakeTimers()
+    try {
+      let settled = false
+      const opening = driver
+        .executeTool('chat-test', 'browser_open_tab', {}, 'background-1', undefined, {
+          background: true,
+        })
+        .then((result) => {
+          settled = true
+          return result
+        })
+      await vi.advanceTimersByTimeAsync(1_000)
+      expect(settled).toBe(false)
+
+      await vi.advanceTimersByTimeAsync(4_000)
+      expect((await opening).ok).toBe(true)
+    } finally {
+      vi.useRealTimers()
+    }
+  })
+
+  it('stops a background action before its next input once the user starts working mid-action', async () => {
+    await driver.executeTool('chat-test', 'browser_open_tab', {})
+    const contents = session.requireTab().view.webContents
+    // The user presses a key while the action reads the page before acting on it.
+    const probe = vi.mocked(contents.executeJavaScript).getMockImplementation()
+    vi.mocked(contents.executeJavaScript).mockImplementation(async (...args) => {
+      typeInAgentPage(contents)
+      return probe ? probe(...args) : undefined
+    })
+
+    const result = await driver.executeTool(
+      'chat-test',
+      'browser_press_key',
+      { key: 'Enter' },
+      'background-key',
+      undefined,
+      { background: true }
+    )
+
+    expect(result.ok).toBe(false)
+    expect(result.error).toContain('the user started working in this page')
+  })
+
+  it('never makes a background read wait for the user', async () => {
+    await driver.executeTool('chat-test', 'browser_open_tab', {})
+    typeInAgentPage(session.requireTab().view.webContents)
+    vi.useFakeTimers()
+    try {
+      let settled = false
+      void driver
+        .executeTool('chat-test', 'browser_list_tabs', {}, 'background-read', undefined, {
+          background: true,
+        })
+        .then(() => {
+          settled = true
+        })
+      await vi.advanceTimersByTimeAsync(100)
+
+      expect(settled).toBe(true)
+    } finally {
+      vi.useRealTimers()
+    }
+  })
+
   it('returns ok:false instead of throwing for tool-level failures', async () => {
     // No session exists, so any page-dependent tool fails with guidance.
     const result = await driver.executeTool('chat-test', 'browser_click', { elementId: 1 })

apps/desktop/src/main/browser-agent/driver.ts

@@ -19,6 +19,7 @@
 import {
   BROWSER_DATA_KINDS,
   BROWSER_NAVIGATION_NATIVE_WATCHDOG_MS,
+  BROWSER_TOOL_OBSERVES_ONLY,
   BROWSER_TOOL_QUEUE_WAIT_TIMEOUT_MS,
   BROWSER_UPLOAD_MAX_FILES,
   type BrowserDataKind,
@@ -32,7 +33,7 @@ import {
 import type { BrowserDownloadsState, BrowserToolbarCommand } from '@sim/desktop-bridge'
 import { createLogger } from '@sim/logger'
 import { getErrorMessage } from '@sim/utils/errors'
-import { sleep } from '@sim/utils/helpers'
+import { interruptibleSleep, sleep } from '@sim/utils/helpers'
 import { isRecordLike, omit, toArray, toRecord } from '@sim/utils/object'
 import type { BrowserWindow, MenuItemConstructorOptions, WebContents, WebFrameMain } from 'electron'
 import { Menu } from 'electron'
@@ -5094,12 +5095,64 @@ function withNotices(result: unknown): unknown {
   return { value: result, notices }
 }
 
+/** Quiet time after the user's last click or keystroke before the agent takes the page back. */
+const USER_TAKEOVER_IDLE_MS = 4_000
+/** The longest an action waits for the user to finish; past it, the action does not run. */
+const USER_TAKEOVER_MAX_WAIT_MS = 30_000
+const USER_KEPT_WORKING =
+  'Not run: the user kept working in this page, so this browser action never started and nothing was sent to the page. Ask the user whether they are done before acting in this page again.'
+const USER_TOOK_OVER_MID_ACTION =
+  'Stopped: the user started working in this page, so this browser action stopped before sending its next input. Earlier steps may have taken effect; inspect the page before continuing.'
+
+function isUserWorkingInPage(): boolean {
+  const since = session.msSinceUserIntervention()
+  return since !== null && since < USER_TAKEOVER_IDLE_MS
+}
+
+/**
+ * The user and the agent never act in the same page at once. While the user is clicking, typing
+ * or scrolling in the tab the agent drives, the agent's next action waits, marked as needing
+ * attention, and resumes once the user has left the page alone for a few seconds.
+ */
+async function yieldToUser(
+  toolCallId: string | undefined,
+  signal: AbortSignal,
+  maxWaitMs: number
+): Promise<void> {
+  session.setAutomationNeedsAttention(true)
+  logger.info('Browser automation yielding to the user', { toolCallId })
+  // Bounded well inside the action's own deadline, so a user who keeps working gets the model a
+  // clear "not run" rather than a timeout with an unknown outcome.
+  const deadline = Date.now() + maxWaitMs
+  try {
+    while (isUserWorkingInPage()) {
+      if (Date.now() >= deadline) throw new ToolError(USER_KEPT_WORKING)
+      await interruptibleSleep(250, signal)
+      if (signal.aborted) throw new ToolError('This browser action was cancelled.')
+    }
+  } finally {
+    session.setAutomationNeedsAttention(false)
+  }
+  logger.info('Browser automation resumed after the user stopped', { toolCallId })
+}
+
+/** How a browser call reached the driver. */
+interface BrowserToolExecutionOptions {
+  /**
+   * Run by the background executor while the user may be working in the same page, so it yields
+   * the page to them. A call from the chat view the user is watching never does: there the user
+   * steers the agent directly, as they always have.
+   */
+  background?: boolean
+}
+
 export async function executeTool(
   scopeId: string,
   tool: BrowserToolName,
   params: Record<string, unknown>,
   toolCallId?: string,
-  authorizationBoundary?: BrowserToolQueueBoundary
+  authorizationBoundary?: BrowserToolQueueBoundary,
+  options: BrowserToolExecutionOptions = {}
 ): Promise<{ ok: boolean; result?: unknown; error?: string }> {
   const resolvedScopeId = resolveDriverScopeId(scopeId)
   if (authorizationBoundary) {
@@ -5190,17 +5243,36 @@ export async function executeTool(
           session.setAutomationActive(true)
         }
         try {
+          // Only a background call acting on the page yields; reading it cannot collide with the user.
+          const yieldsToUser =
+            options.background === true &&
+            tool !== 'browser_request_takeover' &&
+            !BROWSER_TOOL_OBSERVES_ONLY[tool]
+          const watchdogMs = browserToolWatchdogMs(tool, params)
+          if (yieldsToUser && isUserWorkingInPage()) {
+            await yieldToUser(
+              toolCallId,
+              executionController.signal,
+              watchdogMs === null
+                ? USER_TAKEOVER_MAX_WAIT_MS
+                : Math.min(USER_TAKEOVER_MAX_WAIT_MS, Math.floor(watchdogMs / 2))
+            )
+          }
           const response = dialogResponse(tool, params)
           state.dialogResponse = response
             ? { contents: session.requireAutomationTab().view.webContents, response }
             : null
           const executionEpoch = ++state.toolExecutionEpoch
-          const watchdogMs = browserToolWatchdogMs(tool, params)
           const executionDeadline = watchdogMs === null ? undefined : Date.now() + watchdogMs
+          // Checked right before each input the action sends: a user who starts working in the
+          // page mid-action (between a batch's steps, during a click's target probes) wins it.
           const assertCurrentExecution = () => {
             if (state.toolExecutionEpoch !== executionEpoch) {
               throw new ToolError('This browser action expired before it could dispatch input.')
             }
+            if (yieldsToUser && isUserWorkingInPage()) {
+              throw new ToolError(USER_TOOK_OVER_MID_ACTION)
+            }
           }
           let actionOutcome: BrowserActionOutcome | undefined
           const execution = withPostActionObservation(