diff --git a/.papercuts/troubleshooting.md b/.papercuts/troubleshooting.md index 942dd4f40..9893aa1f5 100644 --- a/.papercuts/troubleshooting.md +++ b/.papercuts/troubleshooting.md @@ -1,5 +1,9 @@ # Troubleshooting +- 2026-09-17 release gate: do not run `npm test` and `npm run build` concurrently in one worktree. Both compile the universal `build/native/aiden-worktree-remover`, and `lipo` races its temporary output. Run build first, then the full suite serially. +- 2026-09-17 release gate: removing the final Live voice-approval sender left `chat:approval-withdrawn` in the preload notification allowlist. Focused Live tests do not own the global sender/allowlist equality contract; run the full suite before publishing renderer IPC changes. +- 2026-09-17 Live motion: do not key canvas layers by caption/action state; remounting restarts the animation and causes jumps. Kept stable duplex layers and separated visual activity from microphone activity so mic-off sessions retain Stop. Production macOS package must be rebuilt separately from the HTML review bundle. + ## 2026-09-12 — Listed upstream integration audit - This worktree has no `.memory/` or `node_modules/`. Read the main checkout's project memory as historical context, but use this worktree's exact HEAD/source as authority. The main checkout's `tsx` binary can execute dependency-free focused suites without installing packages here; suites with runtime package imports still fail module resolution (observed: `entities` in the subagent capability suite). Treat that as an environment limitation, not a product regression or passing test. @@ -505,7 +509,19 @@ symlink with this checkout's own npm ci. Full type-check and lint then passed. - The installed Google SDK exposes Live `interactionStatus` inside `serverContent`, while newer Extended Thinking examples describe status alongside tool-call lifecycle events; pin protocol handling to the installed typed wire contract and cover `IDLE` explicitly when adopting the new model. # Release coordination +- Screen-share work was based on an older Live contract. Integration must retain the exact Computer Use authorization token, extended-thinking model, async thread finalization, direct-action policy, device routing, and empty-envelope fix; add screen intent without replacing these later changes. Original screen worktree remains unchanged. + +- Live device picker visual check found two interacting issues: settings action-cluster CSS wraps every direct `.flex` child, including combobox triggers, and Radix Select.Value strips className/style. Scope a no-wrap override to Live device triggers and truncate their actual value spans; static markup tests alone did not catch geometry. + +- Live device routing must retain one audio dependency/player instance across capability refresh; otherwise start preflight can configure a different player from the hook's retained playback ref. Keep capabilities separate from audio ownership. + +- Audio selectors: Radix SelectValue has no selected-item text in static rendering until its item collection mounts; supply an explicit selected label so unavailable-device and initial-render states are readable and testable. + - Pullfrog took just over one hour to review PR #132 after first-party CI was green; keep exact-head checks separate so the long external review does not obscure test status. - Moving Live from default-on to acceptance-gated correctly hid the dock but invalidated the local UI E2E; keep default-off rendering covered separately and opt the isolated provider-free E2E harness into the experimental surface explicitly. - The hosted full Electron suite marked the unrelated chat-switch queue test flaky after it passed on retry, and `--fail-on-flaky-tests` failed the whole gate; rerun the exact failed job before changing unrelated product behavior, while preserving the strict gate if the flake repeats. - A distant Environment source-contract test asserted Aiden Live's two-argument command registration, so focused Live tests missed the intentional capability gate; search all source-contract assertions when changing a shared command signature. +- Signed Live test build: `isPackagedRuntime()` is false for an isolated development profile, causing Computer Use to search inside app.asar/build instead of Contents/Helpers. Resolve physical helper layout using `app.isPackaged`; profile identity must not determine package resource locations. +- Live cue tests must flush passive React effects after microphone and stop state updates before asserting audio feedback. The repo has no local Prettier binary; use its configured ESLint validation instead. +- Real signed Live session reached open/microphone-ready/input-first-packet, then Google emitted an empty top-level envelope at 00:20:01 UTC on 2026-09-17. Rejecting `{}` caused the observed silent disconnect. Admit exact empty envelopes as rate-limited no-ops without extending idle timeout; keep unknown populated fields rejected. Terminal error HUDs must remain visible after active becomes false. +- The Mac's default input was Bose Mini II while output was MacBook speakers. Packet flow alone does not prove intelligible user speech or audible playback; retain separate operator verification. diff --git a/docs/plans/README.md b/docs/plans/README.md index e430352f0..23e7f9acb 100644 --- a/docs/plans/README.md +++ b/docs/plans/README.md @@ -21,7 +21,7 @@ This directory is the source of truth for Aiden's implementation plans. The engi | [Dynamic Model Catalog](dynamic-model-catalog-plan.md) | Implemented | Validated pi.dev overlays, offline `0600` cache hydration, scoped setup refresh, four-hour launch refresh, force refresh, Pi metadata fallback, and Mac/iOS projection ship on pinned Pi 0.84.4. | | [Generative UI Artifacts](generative-ui-artifacts-plan.md) | Active | Phases 0–6 shipped: chat-scoped `render_artifact`, strict sandboxed preview/export hosts, verified vendored Chart.js/Plotly/KaTeX, permission-aware `/visualize`, crash-recoverable authoritative storage/copies, descriptor-relative workspace reads, one-iframe handoff/expansion, visible failure states, and route-stable Responding/Visualizing activity. Three-agent PR review findings are remediated with focused regression coverage. | | [Generation Progress Notes](generation-progress-notes-plan.md) | Planned | No implementation yet. | -| [Aiden Live Assistant](gemini-live-assistant-plan.md) | Partial | Phases 0–4, Extended Thinking spoken progress, strict voice-only per-action approval, the chrome-free blue orb, Google model visibility controls, and metadata-only per-session threads are implemented; native screen-picker and real-Google operator receipts remain open. | +| [Aiden Live Assistant](gemini-live-assistant-plan.md) | Partial | Extended Thinking, the blue orb, Google model visibility, and session threads are implemented. Local test builds add input/output device selection, connection sounds, and session-authorized direct actions without per-action prompts; native screen-picker and real-Google operator receipts remain open. | | [Libghostty workspace terminal](libghostty-terminal-plan.md) | Implemented | The workspace drawer uses Ghostty's official `libghostty-vt` WASM (T3-style runtime, PTY trampoline, canvas surface); node-pty sessions are unchanged. Packaged Mac acceptance remains. | | [Logging and Diagnostics Upgrade](logging-and-diagnostics-upgrade-plan.md) | Implemented | Phases 0–7 are implemented: bounded typed desktop journals, main-owned renderer evidence, local support export/delete, native categorical parity, and CI/release gates. Signed/notarized `v0.35.0` passed packaged diagnostics acceptance; physical-device termination receipts remain. | | [Long-thread payload upgrades](long-thread-payload-upgrade-plan.md) | Partial | Investigation complete: T3’s O(N²) stdout store does not exist here. No-op `toolRunning` timeline republish is skipped; Remote gzip, stream-journal debounce, chat JSON/attachments, and transcript windowing remain planned. | diff --git a/docs/plans/gemini-live-assistant-plan.md b/docs/plans/gemini-live-assistant-plan.md index 7c1fc2825..f9f60cccd 100644 --- a/docs/plans/gemini-live-assistant-plan.md +++ b/docs/plans/gemini-live-assistant-plan.md @@ -2,6 +2,13 @@ Status: Partial — Phases 0–4 plus the beta-labeled Aiden Live orb/setup shell and dedicated metadata-only session threads are implemented; authorized macOS screen capture and real Google beta receipts remain operator-owned Date: 2026-09-15 +Continuous orb follow-up (2026-09-17): active listening/thinking/speaking/acting now share stable Listening + Weaving canvas layers with a slow complementary blend and connection/rest crossfades. Removed visible dock status text (retained screen-reader status). First active click reveals Stop; the second stops, Escape restores the orb, and closing disables duplicate clicks. Sharing/error surfaces remain visible. Added mapping/click-policy regressions; all 230 Live tests, type-check, lint, and standalone browser interaction/motion checks pass. Downloads duplex-review.html uses the actual orb component; the installed notarized app has not been rebuilt for this change. +Combined artifact verification (2026-09-17): Apple accepted notarization submission `5b7f74d8-4740-47b9-82d8-66704cb75ec7`; the Downloads Combined Notarized app has a stapled ticket and passes the repository notarized-package/Gatekeeper check. Opened with the preserved isolated Test Profile and verified its renderer and Start Aiden Live control. Full Live suites, type-check, lint, audio/settings tests, and Electron selector-layout regression passed. Native-picker and real-provider acceptance remain separate and pending. +Combined build (2026-09-17): integrated the screen-sharing work from the 2f30 worktree into this audio-fix checkout without changing the source worktree. Preserved extended-thinking model selection, dedicated session threads, async thread finalization, direct actions, diagnostic markers, empty-envelope handling, device routing, and dropdown layout. Screen capture remains separately gated and off in the ordinary test launcher pending the exact-build native-picker acceptance receipt. Notarization and attended acceptance are separate gates; this is not a public release. +Audio selector layout follow-up: scoped single-line trigger styling and actual Radix value-span ellipsis fix long device labels pushing chevrons outside the control. Real Electron geometry regression passes at 1280, 600, and 390px; the full label remains available through the dropdown and hover title. +Audio selection (2026-09-16 local): Settings → Aiden Live now offers device-local input/output preferences, system defaults, hotplug list refresh, and a speaker test. Selections are snapshotted per session; explicit microphone constraints and output sink routing apply to voice capture, replies, and both cues. Missing selections fail with recovery instructions instead of silent startup fallback. Setup points users to these settings. The shared audio player remains stable across capability refresh; preparation is cancellation-fenced. Tests extend the existing registered Live/UI/audio suites. +Runtime diagnosis (2026-09-16 local): signed Live reached Google/open and microphone packet flow, but an empty top-level server envelope caused a fatal parser error. Exact empty envelopes now count against rate limits without extending idle liveness. Fixed lifecycle/audio/cue markers enter the local diagnostic journal without media/transcripts/credentials. The orb now labels microphone state, preserves terminal errors, and uses more audible connection tones. The signed Audio Fix build subsequently connected, captured microphone packets, received Google response audio, started playback, and stayed connected for 54 seconds until manually stopped. Both cue markers and the listening/idle labels were verified. Physical audibility still needs user confirmation; Computer Use acceptance remains unclaimed. Relevant tests, type-check, lint, and hardened development package verification passed. +Local test-build update (2026-09-16): signed development profiles resolve the bundled Computer Use helper by physical package layout. Live plays connection/disconnection cues. At the user's explicit request, starting Live with Computer Use enabled authorizes direct actions for that session without per-action prompts or voice approvals. The dock no longer mounts the voice-approval listener. Target binding, current capture requirements, cancellation, owner identity, and enablement checks remain enforced. Historical per-action approval sections below describe the superseded implementation; ordinary chat policy is unchanged. No new real-provider acceptance is claimed. Related: `aiden-assistant-plan.md`, `pi-provider-integration-plan.md`, and `../computer-use-integration.md` @@ -11,9 +18,9 @@ The bottom-right **Aiden orb** becomes the single entry point for a user-started Aiden Live session. Aiden streams microphone PCM and a user-approved screen/window as bounded JPEG frames to Gemini Live, plays Gemini native audio, and shows its input/output captions. When Gemini needs to act, it -invokes Aiden's existing `computer_use` tool; every input action keeps the -current global gate, per-chat activation, target binding, and fresh **Allow -once** approval. +invokes Aiden's existing `computer_use` tool. During Live, actions execute under +the user's session consent with global enablement and exact target binding. +Stopping Live revokes this authority. Ordinary chats retain their own policy. This is deliberately not a general screen recorder, an unattended assistant, or a second automation authority. diff --git a/main/handlers/assistant-live-parse.test.ts b/main/handlers/assistant-live-parse.test.ts index 1bf724ada..28d80b662 100644 --- a/main/handlers/assistant-live-parse.test.ts +++ b/main/handlers/assistant-live-parse.test.ts @@ -1,7 +1,10 @@ import assert from "node:assert/strict"; import test from "node:test"; +import { GEMINI_LIVE_MAX_JPEG_BYTES } from "../services/gemini-live/protocol.js"; import { parseAssistantLiveAudioIntent, + parseAssistantLiveEmptyIntent, + parseAssistantLiveFrameIntent, parseAssistantLiveStartIntent, parseAssistantLiveStopIntent, } from "./assistant-live-parse.js"; @@ -21,6 +24,9 @@ test("Assistant Live audio admission accepts only one exact 20 ms PCM chunk", () }); test("Assistant Live start intent accepts only an exact bounded authorization record", () => { + assert.equal(parseAssistantLiveStartIntent({ microphone: true, screen: true, computerUseAuthorization: null }).screen, true); + assert.throws(() => parseAssistantLiveStartIntent({ microphone: true, screen: "true", computerUseAuthorization: null })); + assert.throws(() => parseAssistantLiveStartIntent({ microphone: true, screen: true })); assert.deepEqual(parseAssistantLiveStartIntent({ microphone: true, computerUseAuthorization: null }), { microphone: true, computerUseAuthorization: null, @@ -51,3 +57,33 @@ test("Assistant Live stop intent accepts only an exact empty record", () => { /Invalid Assistant Live/u, ); }); + +test("Assistant Live frame admission accepts only one bounded copied JPEG byte range", () => { + const frame = new Uint8Array([0xff, 0xd8, 0x2a, 0xff, 0xd9]); + const parsed = parseAssistantLiveFrameIntent({ sessionId: "session-1", frame }); + assert.equal(parsed.sessionId, "session-1"); + assert.deepEqual(parsed.frame, frame); + assert.notEqual(parsed.frame, frame, "the parser copies admitted bytes"); + for (const value of [ + { sessionId: "session-1", frame: new Uint8Array(3) }, + { sessionId: "session-1", frame: new Uint8Array(GEMINI_LIVE_MAX_JPEG_BYTES + 1) }, + { sessionId: "", frame }, + { sessionId: "session-1", frame: frame.buffer }, + { sessionId: "session-1", frame, apiKey: "secret" }, + { frame }, + ]) assert.throws(() => parseAssistantLiveFrameIntent(value), /frame request/u); +}); + +test("Assistant Live display bind/release accept only an exact empty record", () => { + for (const name of ["display-bind", "display-release"]) { + assert.doesNotThrow(() => parseAssistantLiveEmptyIntent({}, name)); + assert.throws( + () => parseAssistantLiveEmptyIntent({ sessionId: "forged" }, name), + new RegExp(`Invalid Assistant Live ${name}`, "u"), + ); + assert.throws( + () => parseAssistantLiveEmptyIntent(null, name), + new RegExp(`Invalid Assistant Live ${name}`, "u"), + ); + } +}); diff --git a/main/handlers/assistant-live-parse.ts b/main/handlers/assistant-live-parse.ts index 210799592..11bcb2c86 100644 --- a/main/handlers/assistant-live-parse.ts +++ b/main/handlers/assistant-live-parse.ts @@ -1,4 +1,5 @@ import type { AssistantLiveStartIntent } from "../../renderer/shared/assistant-live.js"; +import { GEMINI_LIVE_MAX_JPEG_BYTES } from "../services/gemini-live/protocol.js"; function isRecord(value: unknown): value is Record { return Boolean(value && typeof value === "object" && !Array.isArray(value)); @@ -12,7 +13,8 @@ function exactKeys(value: Record, expected: readonly string[]): export function parseAssistantLiveStartIntent(value: unknown): AssistantLiveStartIntent { if ( !isRecord(value) || - !exactKeys(value, ["computerUseAuthorization", "microphone"]) || + !exactKeys(value, "screen" in value ? ["computerUseAuthorization", "microphone", "screen"] : ["computerUseAuthorization", "microphone"]) || + ("screen" in value && typeof value.screen !== "boolean") || typeof value.microphone !== "boolean" || !( value.computerUseAuthorization === null || @@ -25,6 +27,7 @@ export function parseAssistantLiveStartIntent(value: unknown): AssistantLiveStar } return { microphone: value.microphone, + ...("screen" in value ? { screen: value.screen as boolean } : {}), computerUseAuthorization: value.computerUseAuthorization, }; } @@ -54,3 +57,33 @@ export function parseAssistantLiveAudioIntent(value: unknown): AssistantLiveAudi } return { sessionId: value.sessionId, pcm: Uint8Array.from(value.pcm) }; } + +export interface AssistantLiveFrameIntent { + sessionId: string; + frame: Uint8Array; +} + +export function parseAssistantLiveFrameIntent(value: unknown): AssistantLiveFrameIntent { + if ( + !isRecord(value) || + !exactKeys(value, ["frame", "sessionId"]) || + typeof value.sessionId !== "string" || + value.sessionId.length < 1 || + value.sessionId.length > 128 || + !(value.frame instanceof Uint8Array) || + value.frame.byteLength < 4 || + value.frame.byteLength > GEMINI_LIVE_MAX_JPEG_BYTES + ) { + throw new Error("Invalid Assistant Live frame request."); + } + return { sessionId: value.sessionId, frame: Uint8Array.from(value.frame) }; +} + +export function parseAssistantLiveEmptyIntent( + value: unknown, + name: string, +): void { + if (!isRecord(value) || !exactKeys(value, [])) { + throw new Error(`Invalid Assistant Live ${name} request.`); + } +} diff --git a/main/handlers/assistant-live.ts b/main/handlers/assistant-live.ts index 6ae585373..8a27d8116 100644 --- a/main/handlers/assistant-live.ts +++ b/main/handlers/assistant-live.ts @@ -1,12 +1,19 @@ +import { session } from "electron"; import { ipcMain } from "../platform.js"; import { authorizeAidenLiveComputerUse, geminiLiveService, } from "../services/gemini-live/service-main.js"; import { rendererDocumentOwner } from "../services/renderer-document-owner.js"; +import { + bindGeminiLiveDisplayMediaDocument, + installGeminiLiveDisplayMediaGuards, +} from "../services/gemini-live/display-media-contract.js"; import { parseAssistantLiveStartIntent, parseAssistantLiveAudioIntent, + parseAssistantLiveEmptyIntent, + parseAssistantLiveFrameIntent, parseAssistantLiveStopIntent, } from "./assistant-live-parse.js"; import { invokeAssistantLiveStart } from "./assistant-live-start.js"; @@ -43,4 +50,24 @@ export function registerAssistantLiveHandlers(): void { const intent = parseAssistantLiveAudioIntent(input); return geminiLiveService.sendAudio(owner(event), intent.sessionId, intent.pcm); }); + ipcMain.handle("assistant-live:display-bind", (event, input: unknown) => { + parseAssistantLiveEmptyIntent(input, "display-bind"); + const binding = bindGeminiLiveDisplayMediaDocument(event); + if (!geminiLiveService.bindDisplayMedia(owner(event), binding)) return false; + // The guards consult the live binding set, so install them only after this + // document's binding is registered and only once per Electron session. + installGeminiLiveDisplayMediaGuards(session.defaultSession, () => + geminiLiveService.displayMediaBindings(), + ); + return true; + }); + ipcMain.handle("assistant-live:display-release", (event, input: unknown) => { + parseAssistantLiveEmptyIntent(input, "display-release"); + geminiLiveService.releaseDisplayMedia(owner(event)); + return true; + }); + ipcMain.handle("assistant-live:frame", (event, input: unknown) => { + const intent = parseAssistantLiveFrameIntent(input); + return geminiLiveService.sendFrame(owner(event), intent.sessionId, intent.frame); + }); } diff --git a/main/index.ts b/main/index.ts index 89f744277..ec291acb7 100644 --- a/main/index.ts +++ b/main/index.ts @@ -1290,6 +1290,13 @@ async function createMainWindow(): Promise { createdWindow.webContents.on("did-finish-load", () => { protectedAction = null; }); + let liveDiagnosticCount = 0; + createdWindow.webContents.on("console-message", (details) => { + // Only fixed local lifecycle markers; never forward arbitrary renderer console content. + if (liveDiagnosticCount >= 200 || !/^\[aiden-live\] (microphone-ready|input-first-packet|output-first-packet|playback-started|playback-failed|cue-connected|cue-disconnected|cue-failed)$/.test(details.message)) return; + liveDiagnosticCount += 1; + writeDiagnosticEvent({ level: "info", area: "voice", event: "legacy-log", fields: { message: details.message } }); + }); createdWindow.webContents.setWindowOpenHandler(({ url }) => { openExternalUrl(url); diff --git a/main/services/computer-use/computer-use-foundation.test.ts b/main/services/computer-use/computer-use-foundation.test.ts index a841d57e9..1bc1a8da3 100644 --- a/main/services/computer-use/computer-use-foundation.test.ts +++ b/main/services/computer-use/computer-use-foundation.test.ts @@ -30,6 +30,12 @@ import { } from "./session.js"; const fixture = fileURLToPath(new URL("./fixtures/fake-cua-driver.mjs", import.meta.url)); + +test("Computer Use helper layout follows physical packaging, not the runtime profile", async () => { + const source = await readFile(new URL("./runtime.ts", import.meta.url), "utf8"); + assert.match(source, /isPackaged:\s*app\.isPackaged/); + assert.doesNotMatch(source, /isPackagedRuntime/); +}); const SESSION_LESS_TOOLS = new Set([ "health_report", "check_permissions", diff --git a/main/services/computer-use/runtime.ts b/main/services/computer-use/runtime.ts index b89dcf03a..2536f0e7c 100644 --- a/main/services/computer-use/runtime.ts +++ b/main/services/computer-use/runtime.ts @@ -1,5 +1,4 @@ import { app } from "../../platform.js"; -import { isPackagedRuntime } from "../../runtime-mode.js"; import { resolveCuaDriverInstallation } from "./binary.js"; import { ComputerUseController } from "./controller.js"; import { CuaDriverHost } from "./host.js"; @@ -21,7 +20,8 @@ export async function createCuaDriverHost(signal: AbortSignal): Promise boolean | Promise; approve?: (signal: AbortSignal, summary: string) => Promise; @@ -83,6 +84,7 @@ function harness(overrides: { const approvalSummaries: string[] = []; const bridge = new GeminiLiveComputerUseBridge({ sessionId: "session-1", + actionPolicy: overrides.actionPolicy, controller, isAuthorized: overrides.authorized ?? (() => true), requestApproval: ({ signal, summary }) => { @@ -94,6 +96,26 @@ function harness(overrides: { return { bridge, controller, responses, approvalSummaries }; } +test("session actions execute without prompts and still bind a fresh target grant", async () => { + const h = harness({ actionPolicy: "session", approve: async () => { throw new Error("must not prompt"); } }); + h.bridge.enqueue({ id: "type", name: "computer_use", args: { action: "type", text: "hello" } }); + await tick(); + assert.deepEqual(h.approvalSummaries, []); + assert.deepEqual(h.controller.authorizations, ["type"]); + assert.deepEqual(h.controller.executions, ["type"]); + await h.bridge.close(); +}); + +test("session actions recheck revoked authority after resolving the target", async () => { + let checks = 0; + const h = harness({ actionPolicy: "session", authorized: () => ++checks === 1 }); + h.bridge.enqueue({ id: "type", name: "computer_use", args: { action: "type", text: "hello" } }); + await tick(); + assert.deepEqual(h.controller.authorizations, []); + assert.deepEqual(h.controller.executions, []); + await h.bridge.close(); +}); + test("Live bridge rejects unknown tools and fields before the controller", async () => { const h = harness(); h.bridge.enqueue({ id: "wrong", name: "shell", args: {} }); diff --git a/main/services/gemini-live/computer-use-bridge.ts b/main/services/gemini-live/computer-use-bridge.ts index 4164c88a3..db1f602a0 100644 --- a/main/services/gemini-live/computer-use-bridge.ts +++ b/main/services/gemini-live/computer-use-bridge.ts @@ -34,6 +34,8 @@ export interface GeminiLiveComputerUseCall { export interface GeminiLiveComputerUseBridgeOptions { sessionId: string; + /** Explicit Live-session consent replaces per-action prompts; target checks still apply. */ + actionPolicy?: "session" | "per-action"; controller: GeminiLiveComputerUseController; isAuthorized(): boolean | Promise; requestApproval(input: { @@ -205,7 +207,7 @@ export class GeminiLiveComputerUseBridge { entry.controller.signal, ); if (approval) { - const allowed = await this.options.requestApproval({ + const allowed = this.options.actionPolicy === "session" || await this.options.requestApproval({ streamId: `live:${this.options.sessionId}`, toolCallId: entry.call.id, toolName: COMPUTER_USE_TOOL_NAME, diff --git a/main/services/gemini-live/display-media-contract.test.ts b/main/services/gemini-live/display-media-contract.test.ts index ad1049399..7e9482899 100644 --- a/main/services/gemini-live/display-media-contract.test.ts +++ b/main/services/gemini-live/display-media-contract.test.ts @@ -4,6 +4,8 @@ import test from "node:test"; import { GEMINI_LIVE_SYSTEM_PICKER_OPTIONS, bindGeminiLiveDisplayMediaDocument, + installGeminiLiveDisplayMediaGuards, + type GeminiLiveDisplayMediaBinding, } from "./display-media-contract.js"; class FakeFrame { @@ -106,6 +108,126 @@ test("binds both custom-picker and system-picker permission admission to one exa ); }); +test("session guards gate display-capture only and install exactly once", () => { + const installed = { + checks: 0, + requests: 0, + displays: 0, + check: null as + | (( + webContents: Electron.WebContents | null, + permission: string, + requestingOrigin: string, + details: Electron.PermissionCheckHandlerHandlerDetails, + ) => boolean) + | null, + request: null as + | (( + webContents: Electron.WebContents, + permission: string, + callback: (granted: boolean) => void, + details: Electron.PermissionRequest, + ) => void) + | null, + display: null as + | (( + request: Electron.DisplayMediaRequestHandlerHandlerRequest, + callback: (streams: Electron.Streams) => void, + ) => void) + | null, + opts: undefined as Electron.DisplayMediaRequestHandlerOpts | undefined, + }; + const electronSession = { + setPermissionCheckHandler(handler: typeof installed.check) { + installed.checks += 1; + installed.check = handler; + }, + setPermissionRequestHandler(handler: typeof installed.request) { + installed.requests += 1; + installed.request = handler; + }, + setDisplayMediaRequestHandler( + handler: typeof installed.display, + opts?: Electron.DisplayMediaRequestHandlerOpts, + ) { + installed.displays += 1; + installed.display = handler; + installed.opts = opts; + }, + }; + const frame = new FakeFrame(10, 20, "document-one", "file:///Aiden/main-window.html"); + const sender = new FakeWebContents(7, frame); + const bindings: GeminiLiveDisplayMediaBinding[] = []; + installGeminiLiveDisplayMediaGuards(electronSession, () => bindings); + installGeminiLiveDisplayMediaGuards(electronSession, () => bindings); + assert.equal(installed.checks, 1, "re-installation must not stack handlers"); + assert.equal(installed.requests, 1); + assert.equal(installed.displays, 1); + assert.deepEqual(installed.opts, { useSystemPicker: true }); + + const details = { isMainFrame: true, requestingUrl: frame.url } as Electron.PermissionRequest; + // Without a Live binding every display-capture path denies. + assert.equal( + installed.check?.( + sender as unknown as Electron.WebContents, + "display-capture", + "file:///Aiden/", + details, + ), + false, + ); + let granted: boolean | null = null; + installed.request?.( + sender as unknown as Electron.WebContents, + "display-capture", + (next) => { + granted = next; + }, + details, + ); + assert.equal(granted, false); + + // A bound document admits display-capture; every other permission keeps the + // session's default allow so the guards never shrink unrelated authority. + bindings.push(bindGeminiLiveDisplayMediaDocument(invokeEvent(sender, frame))); + assert.equal( + installed.check?.( + sender as unknown as Electron.WebContents, + "display-capture", + "file:///Aiden/", + details, + ), + true, + ); + assert.equal( + installed.check?.(null, "media", "file:///Aiden/", details), + true, + "non-display permissions keep the default policy", + ); + granted = null; + installed.request?.(sender as unknown as Electron.WebContents, "media", (next) => { + granted = next; + }, details); + assert.equal(granted, true); + + // Any non-picker dispatch to the fallback handler is denied outright. + const streams: Electron.Streams[] = []; + installed.display?.(displayRequest(frame), (next) => streams.push(next)); + assert.deepEqual(streams, [{}]); + + // Releasing the binding closes capture again immediately. + bindings.pop(); + assert.equal( + installed.check?.( + sender as unknown as Electron.WebContents, + "display-capture", + "file:///Aiden/", + details, + ), + false, + ); +}); + test("navigation, replacement frames, and unrelated WebContents fail closed", () => { const frame = new FakeFrame(10, 20, "document-one", "file:///Aiden/main-window.html"); const sender = new FakeWebContents(7, frame); diff --git a/main/services/gemini-live/display-media-contract.ts b/main/services/gemini-live/display-media-contract.ts index af903e38c..bf5d9b678 100644 --- a/main/services/gemini-live/display-media-contract.ts +++ b/main/services/gemini-live/display-media-contract.ts @@ -86,3 +86,81 @@ export function bindGeminiLiveDisplayMediaDocument( details.requestingUrl === requestingUrl, }; } + +interface DisplayMediaGuardSession { + setPermissionCheckHandler( + handler: (( + webContents: Electron.WebContents | null, + permission: string, + requestingOrigin: string, + details: Electron.PermissionCheckHandlerHandlerDetails, + ) => boolean) | null, + ): void; + setPermissionRequestHandler( + handler: (( + webContents: Electron.WebContents, + permission: string, + callback: (permissionGranted: boolean) => void, + details: Electron.PermissionRequest, + ) => void) | null, + ): void; + setDisplayMediaRequestHandler( + handler: + | (( + request: Electron.DisplayMediaRequestHandlerHandlerRequest, + callback: (streams: Electron.Streams) => void, + ) => void) + | null, + opts?: Electron.DisplayMediaRequestHandlerOpts, + ): void; +} + +const guardedSessions = new WeakSet(); + +/** + * Installs the display-capture boundary once per Electron session. Every + * permission other than display-capture keeps Electron's default allow, and a + * display request succeeds only while a currently bound Live document admits + * it. The system-picker session never dispatches to the fallback handler; any + * non-picker dispatch is denied rather than trusted to select a source. + */ +export function installGeminiLiveDisplayMediaGuards( + electronSession: DisplayMediaGuardSession, + getBindings: () => readonly GeminiLiveDisplayMediaBinding[], +): void { + if (guardedSessions.has(electronSession)) return; + guardedSessions.add(electronSession); + electronSession.setPermissionCheckHandler( + (webContents, permission, _requestingOrigin, details) => { + if (String(permission) !== "display-capture" || !webContents) return true; + return getBindings().some((binding) => + binding.allowsPermissionRequest(webContents, "display-capture", { + isMainFrame: details.isMainFrame === true, + requestingUrl: details.requestingUrl ?? "", + }), + ); + }, + ); + electronSession.setPermissionRequestHandler( + (webContents, permission, callback, details) => { + if (String(permission) !== "display-capture") { + callback(true); + return; + } + callback( + getBindings().some((binding) => + binding.allowsPermissionRequest(webContents, "display-capture", { + isMainFrame: details.isMainFrame === true, + requestingUrl: details.requestingUrl ?? "", + }), + ), + ); + }, + ); + electronSession.setDisplayMediaRequestHandler( + (_request, callback) => { + callback({}); + }, + GEMINI_LIVE_SYSTEM_PICKER_OPTIONS, + ); +} diff --git a/main/services/gemini-live/feature-flag.test.ts b/main/services/gemini-live/feature-flag.test.ts index a31ad22df..0c31db4c6 100644 --- a/main/services/gemini-live/feature-flag.test.ts +++ b/main/services/gemini-live/feature-flag.test.ts @@ -1,6 +1,10 @@ import assert from "node:assert/strict"; import test from "node:test"; -import { experimentalGeminiLiveModel, geminiLiveEnabled } from "./feature-flag.js"; +import { + experimentalGeminiLiveModel, + geminiLiveEnabled, + geminiLiveScreenEnabled, +} from "./feature-flag.js"; test("Gemini 3.8 Live stays acceptance-gated behind an exact opt-in", () => { assert.equal(geminiLiveEnabled({}), false); @@ -25,3 +29,22 @@ test("Gemini 3.8 Live stays acceptance-gated behind an exact opt-in", () => { "gemini-3.8-live-extended-thinking", ); }); + +test("screen sharing needs both flags and can never outlive the Live gate", () => { + const live = { AIDEN_EXPERIMENTAL_GEMINI_LIVE: "1" }; + assert.equal(geminiLiveScreenEnabled({}), false); + assert.equal(geminiLiveScreenEnabled(live), false); + assert.equal( + geminiLiveScreenEnabled({ AIDEN_EXPERIMENTAL_GEMINI_LIVE_SCREEN: "1" }), + false, + "the screen flag alone must not admit capture", + ); + assert.equal( + geminiLiveScreenEnabled({ ...live, AIDEN_EXPERIMENTAL_GEMINI_LIVE_SCREEN: "true" }), + false, + ); + assert.equal( + geminiLiveScreenEnabled({ ...live, AIDEN_EXPERIMENTAL_GEMINI_LIVE_SCREEN: "1" }), + true, + ); +}); diff --git a/main/services/gemini-live/feature-flag.ts b/main/services/gemini-live/feature-flag.ts index c6e4c4c86..616d7eba1 100644 --- a/main/services/gemini-live/feature-flag.ts +++ b/main/services/gemini-live/feature-flag.ts @@ -1,5 +1,6 @@ export const GEMINI_LIVE_FEATURE_FLAG = "AIDEN_EXPERIMENTAL_GEMINI_LIVE"; export const GEMINI_LIVE_MODEL = "gemini-3.8-live-extended-thinking"; +export const GEMINI_LIVE_SCREEN_FLAG = "AIDEN_EXPERIMENTAL_GEMINI_LIVE_SCREEN"; /** * The exact Live + Computer Use contract stays opt-in until a credentialed @@ -16,6 +17,20 @@ export function geminiLiveEnabled( ); } +/** + * Screen sharing is a second attended-Live gate. It is enabled only after the + * packaged, Screen Recording-authorized native-picker acceptance has been + * recorded for the current build; it can never outlive the Live flag itself. + */ +export function geminiLiveScreenEnabled( + environment: Readonly> = process.env, +): boolean { + return ( + geminiLiveEnabled(environment) && + environment[GEMINI_LIVE_SCREEN_FLAG]?.trim() === "1" + ); +} + export function experimentalGeminiLiveModel( environment: Readonly> = process.env, ): string | null { diff --git a/main/services/gemini-live/protocol.test.ts b/main/services/gemini-live/protocol.test.ts index 87337f9a8..0b423bed5 100644 --- a/main/services/gemini-live/protocol.test.ts +++ b/main/services/gemini-live/protocol.test.ts @@ -149,7 +149,7 @@ test("extended-thinking sessions enable low thinking and a voice-first action co const config = subject.server.latest.params.config!; assert.deepEqual(config.thinkingConfig, { thinkingLevel: "LOW" }); assert.match(String(config.systemInstruction), /fully voice-first/u); - assert.match(String(config.systemInstruction), /Allow once.*Deny/u); + assert.match(String(config.systemInstruction), /without per-action approval prompts/u); }); test("accepts only bounded 16 kHz 20-40 ms PCM and emits an exact SDK audio blob", async () => { @@ -869,9 +869,17 @@ test("one compound event has a total decoded-content budget", async () => { ); }); -test("empty, unknown, and invalid aggregate events are transport-terminal", async () => { +test("empty provider envelopes are ignored without refreshing the idle deadline", async () => { + const subject = harness({ idleTimeoutMs: 100 }); + await subject.protocol.start(); + subject.server.latest.emit({}); + assert.equal(subject.protocol.state, "open"); + subject.clock.advance(101); + assert.equal(subject.protocol.state, "failed"); +}); + +test("unknown and invalid aggregate events are transport-terminal", async () => { for (const message of [ - {}, { unsupportedProviderField: {} }, { usageMetadata: { totalTokenCount: "1" } }, ]) { diff --git a/main/services/gemini-live/protocol.ts b/main/services/gemini-live/protocol.ts index 4ac3085ce..e1b786d20 100644 --- a/main/services/gemini-live/protocol.ts +++ b/main/services/gemini-live/protocol.ts @@ -1,4 +1,5 @@ import { Modality, ThinkingLevel } from "@google/genai"; +import { writeDiagnosticEvent } from "../diagnostic-journal.js"; import type { Content, FunctionResponse, @@ -30,7 +31,7 @@ const MAX_OUTPUT_AUDIO_BYTES_PER_EVENT = 512_000; const MAX_OUTPUT_AUDIO_BYTES_PER_SECOND = 768_000; const MAX_DECODED_SERVER_EVENT_BYTES = 640_000; export const GEMINI_LIVE_MAX_RENDERER_AUDIO_CHUNK_BYTES = 96_000; -const MAX_JPEG_BYTES = 1_500_000; +export const GEMINI_LIVE_MAX_JPEG_BYTES = 1_500_000; const MAX_TEXT_BYTES = 16_384; const MAX_HISTORY_TURNS = 16; const MAX_HISTORY_BYTES = 65_536; @@ -442,7 +443,7 @@ function audioChunkDurationMs(bytes: number): number { function validJpeg(bytes: Uint8Array): boolean { return ( bytes.length >= 4 && - bytes.length <= MAX_JPEG_BYTES && + bytes.length <= GEMINI_LIVE_MAX_JPEG_BYTES && bytes[0] === 0xff && bytes[1] === 0xd8 && bytes[bytes.length - 2] === 0xff && @@ -496,7 +497,7 @@ function liveConnectConfig( ? { thinkingConfig: { thinkingLevel: ThinkingLevel.LOW }, systemInstruction: - "Keep the conversation fully voice-first. Briefly tell the user what you are doing while background reasoning or tools are still running. Before any approval-gated Computer Use action, state the exact action and ask the user to say ‘Allow once’ or ‘Deny’. Never claim that an action completed until its tool result confirms it.", + "Keep the conversation fully voice-first. Briefly tell the user what you are doing while background reasoning or tools are still running. Execute the user's requested screen actions directly during this Live session without per-action approval prompts. Stop when the user cancels. Never claim that an action completed until its tool result confirms it.", } : {}), ...(handle @@ -876,10 +877,18 @@ export class GeminiLiveProtocol { this.malformedStage = "message"; if (!isRecord(message)) throw new Error("Invalid Live server message."); const keys = Object.keys(message); + // Observed from the real Gemini Developer API: an empty protobuf envelope + // carries no payload. Count it against rate limits without refreshing liveness. + if (keys.length === 0) { + this.admitServerEvent(); + return; + } if ( - keys.length === 0 || keys.some((key) => !SUPPORTED_SERVER_MESSAGE_KEYS.has(key)) ) { + writeDiagnosticEvent({ level: "warn", area: "voice", event: "legacy-log", fields: { + message: "Live rejected unsupported message fields", + } }); this.malformed(); } const toolSnapshot = new Map( @@ -1023,7 +1032,7 @@ export class GeminiLiveProtocol { } // Gemini's Developer API can serialize a serverContent envelope with all // optional protobuf fields omitted. Treat that exact shape as a bounded, - // rate-limited provider no-op; empty top-level messages still fail closed. + // rate-limited provider no-op. if (Object.keys(content).length === 0) { this.markMeaningful(); return; diff --git a/main/services/gemini-live/service-main.ts b/main/services/gemini-live/service-main.ts index 22665a810..952ae6b70 100644 --- a/main/services/gemini-live/service-main.ts +++ b/main/services/gemini-live/service-main.ts @@ -1,13 +1,12 @@ import { createOwnedGoogleGenAIConnector } from "./owned-sdk-connector.js"; import { piCredentialStore } from "../pi-credential-store.js"; import { GeminiLiveService } from "./service.js"; -import { experimentalGeminiLiveModel } from "./feature-flag.js"; +import { experimentalGeminiLiveModel, geminiLiveScreenEnabled } from "./feature-flag.js"; import { configStore } from "../config-store.js"; import { computerUseStatus } from "../computer-use/status.js"; import { createComputerUseController } from "../computer-use/runtime.js"; import { ComputerUseParameters } from "../computer-use/schema.js"; import { COMPUTER_USE_TOOL_NAME } from "../computer-use/tool.js"; -import { ToolApprovalCoordinator } from "../tool-approval.js"; import { GeminiLiveComputerUseBridge } from "./computer-use-bridge.js"; import { app } from "../../platform.js"; import { createGeminiLiveAcceptanceEvidenceRecorder } from "./acceptance-evidence.js"; @@ -69,7 +68,7 @@ export async function authorizeAidenLiveComputerUse( } const LIVE_COMPUTER_USE_DESCRIPTION = - "Use Aiden's approval-gated Computer Use controller. Capture an exact window first. You may operate Aiden itself to focus its main composer, choose the current web model or Actions menu, send a prompt, and create or review scheduled tasks. Keep speaking naturally while work is in progress. Before every click, key, type, drag, scroll, focus, or other mutation, briefly state the exact action and ask the user to say Allow once or Deny. Each mutation pauses for a fresh voice decision."; + "Use Aiden's Computer Use controller during this user-started Live session. Capture an exact window first. You may operate Aiden itself to focus its main composer, choose the current web model or Actions menu, send a prompt, and create or review scheduled tasks. Execute the user's requested actions directly without per-action approval prompts. Keep speaking naturally while work is in progress. Stop when the session ends or the user cancels. Never claim success before the tool result confirms it."; /** * The acceptance-gated beta resolves only the recorded @@ -95,6 +94,7 @@ export const geminiLiveService = new GeminiLiveService({ }, }, resolveModel: () => experimentalGeminiLiveModel(), + screenShareEnabled: () => geminiLiveScreenEnabled(), createConnector: (apiKey) => createOwnedGoogleGenAIConnector({ apiKey }), prepareComputerUse: async ({ authorization, owner, sessionId, signal }) => { if (!authorization || signal.aborted || owner.isDestroyed()) return null; @@ -123,10 +123,6 @@ export const geminiLiveService = new GeminiLiveService({ } const controller = createComputerUseController(`live:${sessionId}`, true); - const approvals = new ToolApprovalCoordinator( - (prompt) => owner.send("chat:approval", prompt), - (approvalId) => owner.send("chat:approval-withdrawn", { approvalId }), - ); // The bridge cannot receive a provider call before setup completes, while // bindSendResult runs synchronously before protocol.start(). let sendToolResult: @@ -142,14 +138,10 @@ export const geminiLiveService = new GeminiLiveService({ }; const bridge = new GeminiLiveComputerUseBridge({ sessionId, + actionPolicy: "session", controller, isAuthorized, - requestApproval: async ({ streamId, toolCallId, toolName, summary, signal: callSignal }) => - (await approvals.request( - { streamId, toolCallId, toolName, summary }, - callSignal, - owner.documentId, - )) === "allowed", + requestApproval: async () => false, onActivity: (active) => owner.send("assistant-live:event", { type: "computer_use_state", @@ -172,8 +164,7 @@ export const geminiLiveService = new GeminiLiveService({ ], }, ], - approve: (approvalId, allowed, ownerDocumentId) => - approvals.decide(approvalId, allowed, ownerDocumentId), + approve: () => false, bindSendResult: (send) => { sendToolResult = send; }, diff --git a/main/services/gemini-live/service.test.ts b/main/services/gemini-live/service.test.ts index 0b5017ac4..77af4b9ff 100644 --- a/main/services/gemini-live/service.test.ts +++ b/main/services/gemini-live/service.test.ts @@ -12,6 +12,7 @@ import { FakeGeminiLiveServer } from "./fake-live-server.js"; import { GeminiLiveService, GeminiLiveStartError } from "./service.js"; import { GeminiLiveComputerUseBridge } from "./computer-use-bridge.js"; import type { GeminiLiveComputerUseController } from "./computer-use-bridge.js"; +import type { GeminiLiveDisplayMediaBinding } from "./display-media-contract.js"; import { ComputerUseParameters } from "../computer-use/schema.js"; class FakeOwner implements RendererDocumentOwner { @@ -45,7 +46,15 @@ class FakeOwner implements RendererDocumentOwner { } } -function serviceHarness(credential: Credential | null = { type: "api_key", key: "KEY_SENTINEL" }) { +function serviceHarness( + credential: Credential | null = { type: "api_key", key: "KEY_SENTINEL" }, + overrides: Partial< + Pick< + ConstructorParameters[0], + "screenShareEnabled" | "prepareComputerUse" | "acceptanceEvidence" + > + > = {}, +) { const servers: FakeGeminiLiveServer[] = []; const connectorKeys: string[] = []; let sequence = 0; @@ -59,6 +68,7 @@ function serviceHarness(credential: Credential | null = { type: "api_key", key: servers.push(server); return server.connector; }, + ...overrides, }); return { connectorKeys, servers, service }; } @@ -152,6 +162,21 @@ test("shutdown waits for an in-flight thread begin and finalizes its record", as await shutdown; assert.deepEqual(finished, ["shutdown-during-begin"]); }); +/** + * A display-media binding shaped like what bindGeminiLiveDisplayMediaDocument + * records for the exact document that invoked the picker. + */ +function fakeDisplayBinding( + owner: FakeOwner, +): GeminiLiveDisplayMediaBinding { + return { + documentId: owner.documentId, + owner, + allowsDisplayRequest: () => true, + allowsPermissionRequest: () => true, + }; +} + test("explicit acceptance recorder corroborates ready, provider audio, and Stop teardown", async () => { const server = new FakeGeminiLiveServer(); @@ -739,6 +764,7 @@ test("credential-store failures are normalized without exposing private detail", assert.deepEqual(await service.availability(new FakeOwner(22, "22:1:doc")), { available: false, reason: "google_api_key_invalid", + screenShareAllowed: false, state: "idle", }); await assert.rejects(service.start(new FakeOwner(22, "22:2:doc"), intent), (error: unknown) => @@ -774,6 +800,7 @@ test("model resolver sync and async failures return one safe unavailable status" assert.deepEqual(status, { available: false, reason: "live_model_unverified", + screenShareAllowed: false, state: "idle", }); assert.doesNotMatch( @@ -838,9 +865,12 @@ test("the model gate is explicit-experimental and media has no persistence or lo ]); assert.match(mainSource, /resolveModel: \(\) => experimentalGeminiLiveModel\(\)/u); assert.match(mainSource, /behavior: Behavior\.NON_BLOCKING/u); - assert.match(mainSource, /say Allow once or Deny/u); + assert.match(mainSource, /actionPolicy: "session"/u); + assert.doesNotMatch(mainSource, /ToolApprovalCoordinator|say Allow once or Deny/u); assert.doesNotMatch(serviceSource, /DataStore|writeFile|appendFile|logger|writeDevLog/u); - assert.doesNotMatch(handlerSource, /jpeg|frame|credential|apiKey|tool/u); + // Bounded screen frames now cross a dedicated channel by name; credentials, + // raw tool data, and unbounded media still never reach the handler layer. + assert.doesNotMatch(handlerSource, /jpeg|credential|apiKey|tool/u); }); test("microphone PCM is admitted only for the current consenting exact-document session", async () => { @@ -904,3 +934,106 @@ test("Computer Use is absent unless the Live start explicitly requests it", asyn assert.equal(subject.servers[0]?.latest.params.config?.tools, undefined); await subject.service.stop(new FakeOwner(33, "33:1:doc")); }); +test("screen intent stays fail-closed without the screen gate or a bound document", async () => { + const subject = serviceHarness(); + await assert.rejects( + subject.service.start(new FakeOwner(33, "33:1:doc"), { microphone: true, screen: true, computerUseAuthorization: null }), + (error: unknown) => + error instanceof GeminiLiveStartError && error.reason === "live_start_failed", + ); + assert.equal(subject.servers.length, 0, "the ungated path must never connect"); + + // The gate alone is not authority: without a bound display document the + // intent is still rejected. + const gated = serviceHarness(undefined, { screenShareEnabled: () => true }); + await assert.rejects( + gated.service.start(new FakeOwner(34, "34:1:doc"), { microphone: true, screen: true, computerUseAuthorization: null }), + (error: unknown) => + error instanceof GeminiLiveStartError && error.reason === "live_start_failed", + ); + assert.equal(gated.servers.length, 0); + // The same gate still permits an ordinary voice-only session. + await gated.service.start(new FakeOwner(34, "34:1:doc"), intent); + gated.service.shutdown(); +}); + +test("screen frames are admitted only for a bound, gated, open session", async () => { + const subject = serviceHarness(undefined, { screenShareEnabled: () => true }); + const owner = new FakeOwner(35, "35:1:doc"); + const binding = fakeDisplayBinding(owner); + assert.equal(subject.service.bindDisplayMedia(owner, binding), true); + const session = await subject.service.start(owner, { + microphone: false, + screen: true, + computerUseAuthorization: null, + }); + assert.equal(session.screenShareAllowed, true); + + const first = new Uint8Array([0xff, 0xd8, 0x01, 0xff, 0xd9]); + const newest = new Uint8Array([0xff, 0xd8, 0x02, 0xff, 0xd9]); + const received = subject.servers[0]!.latest.received; + const videoInputs = () => + received.filter( + (message) => message.type === "realtime_input" && "video" in message.value, + ); + + // Latest-frame-wins buffering across a controlled resumption, before any + // frame has been sent, so the protocol's 1 FPS limiter sends it at once. + const owned = subject.service.sessions.get(owner)!; + owned.state = "resuming"; + assert.equal(subject.service.sendFrame(owner, session.sessionId!, first), true); + assert.equal(subject.service.sendFrame(owner, session.sessionId!, newest), true); + assert.equal(videoInputs().length, 0, "resumption buffers instead of sending"); + ( + subject.service as unknown as { + handleProtocolEvent( + session: typeof owned, + event: { type: "state"; state: "open" }, + ): void; + } + ).handleProtocolEvent(owned, { type: "state", state: "open" }); + const flushed = videoInputs(); + assert.equal(flushed.length, 1, "only one buffered frame flushes on reopen"); + assert.deepEqual( + Buffer.from( + (flushed[0]!.value as { video: { data: string } }).video.data, + "base64", + ), + Buffer.from(newest), + "the newest buffered frame wins", + ); + + // Wrong session, wrong owner, and non-screen sessions are all rejected. + assert.equal(subject.service.sendFrame(owner, "stale", newest), false); + assert.equal( + subject.service.sendFrame(new FakeOwner(36, "36:1:other"), session.sessionId!, newest), + false, + ); + + // Stopping releases both the session and the document's picker authority. + subject.service.stop(owner); + assert.deepEqual(subject.service.displayMediaBindings(), []); + await assert.rejects( + subject.service.start(owner, { microphone: false, screen: true, computerUseAuthorization: null }), + (error: unknown) => + error instanceof GeminiLiveStartError && error.reason === "live_start_failed", + ); +}); + +test("a display binding dies with its document and never survives shutdown", async () => { + const subject = serviceHarness(undefined, { screenShareEnabled: () => true }); + const owner = new FakeOwner(37, "37:1:doc"); + assert.equal(subject.service.bindDisplayMedia(owner, fakeDisplayBinding(owner)), true); + owner.navigate(); + assert.deepEqual(subject.service.displayMediaBindings(), []); + + const next = new FakeOwner(38, "38:1:doc"); + assert.equal(subject.service.bindDisplayMedia(next, fakeDisplayBinding(next)), true); + subject.service.shutdown(); + assert.deepEqual(subject.service.displayMediaBindings(), []); + assert.equal( + subject.service.bindDisplayMedia(next, fakeDisplayBinding(next)), + false, + "a shut-down service never binds new capture authority", + ); +}); diff --git a/main/services/gemini-live/service.ts b/main/services/gemini-live/service.ts index 266b1845b..f588343dd 100644 --- a/main/services/gemini-live/service.ts +++ b/main/services/gemini-live/service.ts @@ -1,4 +1,5 @@ import { randomUUID } from "node:crypto"; +import { writeDiagnosticEvent } from "../diagnostic-journal.js"; import type { Credential, CredentialStore } from "@earendil-works/pi-ai"; import type { AssistantLiveAvailabilityReason, @@ -15,6 +16,7 @@ import { type GeminiLiveProtocolOptions, } from "./protocol.js"; import type { GeminiLiveComputerUseBridge } from "./computer-use-bridge.js"; +import type { GeminiLiveDisplayMediaBinding } from "./display-media-contract.js"; import type { GeminiLiveAcceptanceEvidenceEvent, GeminiLiveAcceptanceEvidenceRecorder, @@ -94,6 +96,11 @@ export interface GeminiLiveServiceOptions { credentials: Pick; createConnector(apiKey: string): GeminiLiveConnector; resolveModel(): string | null | Promise; + /** + * True only after the packaged native-picker acceptance has been recorded + * for this build. When absent or false, every screen intent is rejected. + */ + screenShareEnabled?(): boolean; createSessionId?: () => string; acceptanceEvidence?: GeminiLiveAcceptanceEvidenceRecorder | null; threads?: { @@ -131,7 +138,9 @@ interface OwnedLiveSession { protocol: GeminiLiveProtocol | null; state: AssistantLiveSnapshot["state"]; microphone: boolean; + screen: boolean; resumptionAudio: Uint8Array[]; + resumptionFrame: Uint8Array | null; model?: string; computerUse: GeminiLiveComputerUseBridge | null; approveComputerUse: @@ -251,6 +260,10 @@ export class GeminiLiveSessionStore { export class GeminiLiveService { readonly sessions = new GeminiLiveSessionStore(); + private readonly displayBindings = new Map< + string, + { binding: GeminiLiveDisplayMediaBinding; dispose: () => void } + >(); private shuttingDown = false; private readonly pendingThreadFinalization = new Set(); private activeStarts = 0; @@ -258,6 +271,43 @@ export class GeminiLiveService { constructor(private readonly options: GeminiLiveServiceOptions) {} + /** + * Binds the exact renderer document that asked to share its screen. Binding + * alone authorizes nothing until a session also passes the screen gate. + */ + bindDisplayMedia( + owner: RendererDocumentOwner, + binding: GeminiLiveDisplayMediaBinding, + ): boolean { + if ( + this.shuttingDown || + this.options.screenShareEnabled?.() !== true || + owner.isDestroyed() + ) { + return false; + } + const key = documentKey(owner); + this.displayBindings.get(key)?.dispose(); + const dispose = owner.onInvalidated(() => { + if (this.displayBindings.get(key)?.binding === binding) { + this.displayBindings.delete(key); + } + }); + this.displayBindings.set(key, { binding, dispose }); + return true; + } + + releaseDisplayMedia(owner: RendererDocumentOwner): void { + const entry = this.displayBindings.get(documentKey(owner)); + entry?.dispose(); + this.displayBindings.delete(documentKey(owner)); + } + + /** Live bindings consulted by the session-level display-media guards. */ + displayMediaBindings(): readonly GeminiLiveDisplayMediaBinding[] { + return [...this.displayBindings.values()].map((entry) => entry.binding); + } + async availability( owner: RendererDocumentOwner, ): Promise { @@ -291,6 +341,16 @@ export class GeminiLiveService { if (this.shuttingDown) throw new GeminiLiveStartError("live_start_failed"); if (owner.isDestroyed()) throw new GeminiLiveStartError("live_start_failed"); + // A screen intent is valid only while the recorded-acceptance gate is open + // and this exact document bound the picker ahead of the session start. + if ( + _intent.screen && + (this.options.screenShareEnabled?.() !== true || + !this.displayBindings.has(documentKey(owner))) + ) { + throw new GeminiLiveStartError("live_start_failed"); + } + const session: OwnedLiveSession = { abort: new AbortController(), documentKey: documentKey(owner), @@ -300,7 +360,9 @@ export class GeminiLiveService { protocol: null, state: "connecting", microphone: _intent.microphone, + screen: _intent.screen === true, resumptionAudio: [], + resumptionFrame: null, computerUse: null, approveComputerUse: null, threadRecorded: false, @@ -474,9 +536,36 @@ export class GeminiLiveService { } } + sendFrame( + owner: RendererDocumentOwner, + sessionId: string, + jpeg: Uint8Array, + ): boolean { + const session = this.sessions.get(owner); + if (!session || session.sessionId !== sessionId || !session.screen) { + return false; + } + if (session.state === "resuming") { + // Frames are latest-wins; keep at most one until the replacement + // transport opens so a reconnect never replays stale screen state. + session.resumptionFrame = Uint8Array.from(jpeg); + return true; + } + if (session.state !== "open" || !session.protocol) return false; + try { + session.protocol.sendJpeg(jpeg); + return true; + } catch { + this.closeSession(session); + return false; + } + } + async shutdown(): Promise { if (this.shuttingDown) return; this.shuttingDown = true; + for (const entry of this.displayBindings.values()) entry.dispose(); + this.displayBindings.clear(); for (const session of this.sessions.values()) this.closeSession(session, false, "stopped"); await this.waitForStarts(); const sessions = [...this.pendingThreadFinalization]; @@ -499,6 +588,9 @@ export class GeminiLiveService { if (event.type === "audio") this.recordAcceptanceEvidence("provider_response", session.sessionId); if (event.type === "error") { + writeDiagnosticEvent({ level: "warn", area: "voice", event: "legacy-log", fields: { + message: `Live error: ${event.code}; stage: ${event.diagnostic ?? "unknown"}; detail: ${event.diagnosticDetail ?? "none"}`, + } }); // Content-free lifecycle evidence for development/runtime triage. Never // include provider payloads, close reasons, transcripts, or credentials. process.stderr.write( @@ -536,11 +628,17 @@ export class GeminiLiveService { session.computerUse?.interrupt(); } if (event.type === "state") { + writeDiagnosticEvent({ level: "info", area: "voice", event: "legacy-log", fields: { message: `Live state: ${event.state}` } }); session.state = event.state; - if (event.state === "open" && session.resumptionAudio.length > 0) { - const pending = session.resumptionAudio.splice(0); + if (event.state === "open") { + const pendingAudio = session.resumptionAudio.splice(0); + const pendingFrame = session.resumptionFrame; + session.resumptionFrame = null; try { - for (const pcm of pending) session.protocol?.sendAudio(pcm); + for (const pcm of pendingAudio) session.protocol?.sendAudio(pcm); + if (pendingFrame && session.screen) { + session.protocol?.sendJpeg(pendingFrame); + } } catch { this.closeSession(session); return; @@ -590,6 +688,10 @@ export class GeminiLiveService { session.approveComputerUse = null; session.state = "closed"; session.resumptionAudio = []; + session.resumptionFrame = null; + // Stopping or replacing a session also revokes this document's authority + // to admit display capture; a restart re-binds through the picker flow. + this.releaseDisplayMedia(session.owner); if (notifyRenderer && !session.owner.isDestroyed()) { try { session.owner.send(ASSISTANT_LIVE_EVENT_CHANNEL, { @@ -616,6 +718,7 @@ export class GeminiLiveService { reason, sessionId: session.sessionId, model: session.model, + screenShareAllowed: this.options.screenShareEnabled?.() === true, state: session.state, }; } @@ -628,6 +731,7 @@ export class GeminiLiveService { available: reason === "available", reason, ...(model ? { model } : {}), + screenShareAllowed: this.options.screenShareEnabled?.() === true, state: "idle", }; } diff --git a/package-lock.json b/package-lock.json index 8a3be6e98..518f6ab98 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "aiden-agent", - "version": "0.41.3", + "version": "0.41.4", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "aiden-agent", - "version": "0.41.3", + "version": "0.41.4", "hasInstallScript": true, "dependencies": { "@earendil-works/pi-agent-core": "0.84.4", diff --git a/package.json b/package.json index d78496c8e..678a3d7f7 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "aiden-agent", - "version": "0.41.3", + "version": "0.41.4", "private": true, "description": "A macOS AI workspace agent for local and hosted models", "keywords": [ diff --git a/renderer/components/aiden-orb.tsx b/renderer/components/aiden-orb.tsx index 9d64e7ab4..7b652f2f7 100644 --- a/renderer/components/aiden-orb.tsx +++ b/renderer/components/aiden-orb.tsx @@ -10,7 +10,7 @@ interface OrbAppearance { function readOrbAppearance(): OrbAppearance { if (typeof document === "undefined") return { paused: true, theme: "light" }; return { - paused: document.documentElement.dataset.reduceMotion === "true", + paused: document.documentElement.dataset.reduceMotion === "true" || window.matchMedia?.("(prefers-reduced-motion: reduce)").matches === true, theme: document.documentElement.dataset.appearanceScheme === "dark" ? "dark" : "light", }; } @@ -21,7 +21,12 @@ function useOrbAppearance(): OrbAppearance { React.useEffect(() => { const update = () => setAppearance(readOrbAppearance()); window.addEventListener(APPEARANCE_CHANGE_EVENT, update); - return () => window.removeEventListener(APPEARANCE_CHANGE_EVENT, update); + const motion = window.matchMedia?.("(prefers-reduced-motion: reduce)"); + motion?.addEventListener("change", update); + return () => { + window.removeEventListener(APPEARANCE_CHANGE_EVENT, update); + motion?.removeEventListener("change", update); + }; }, []); return appearance; diff --git a/renderer/components/assistant/aiden-live-orb.tsx b/renderer/components/assistant/aiden-live-orb.tsx index 0f816c500..a8cded963 100644 --- a/renderer/components/assistant/aiden-live-orb.tsx +++ b/renderer/components/assistant/aiden-live-orb.tsx @@ -1,5 +1,4 @@ import * as React from "react"; -import type { OrbState } from "thinking-orbs"; import { AidenOrb } from "../aiden-orb"; export type AidenLiveOrbState = @@ -13,21 +12,10 @@ export type AidenLiveOrbState = | "error" | "unavailable"; -const ORB_PRESENTATION: Record< - AidenLiveOrbState, - { state: OrbState; active: boolean } -> = { - ready: { state: "breathing", active: false }, - connecting: { state: "connecting", active: true }, - // Listening deliberately uses Libraries.dev's calm breathing treatment. - listening: { state: "breathing", active: true }, - thinking: { state: "solving", active: true }, - speaking: { state: "composing", active: true }, - acting: { state: "working", active: true }, - approval: { state: "shaping", active: true }, - error: { state: "breathing", active: false }, - unavailable: { state: "breathing", active: false }, -}; +export function aidenLiveOrbVisual(state: AidenLiveOrbState): "duplex" | "connecting" | "rest" { + if (["listening", "thinking", "speaking", "acting"].includes(state)) return "duplex"; + return state === "connecting" ? "connecting" : "rest"; +} export function AidenLiveOrb({ state, @@ -38,22 +26,28 @@ export function AidenLiveOrb({ level?: number; className?: string; }): React.ReactElement { - const presentation = ORB_PRESENTATION[state]; - const boundedLevel = Math.max(0, Math.min(1, level)); + const visual = aidenLiveOrbVisual(state); + const boundedLevel = Number.isFinite(level) ? Math.max(0, Math.min(1, level)) : 0; return ( ); } diff --git a/renderer/components/assistant/assistant-dock.tsx b/renderer/components/assistant/assistant-dock.tsx index 723f4b977..b5c1ea4a4 100644 --- a/renderer/components/assistant/assistant-dock.tsx +++ b/renderer/components/assistant/assistant-dock.tsx @@ -13,15 +13,26 @@ import { } from "./assistant-live"; import { AssistantComputerUseApproval } from "./assistant-computer-use-approval"; import { useAssistantLive, type AssistantLiveController } from "./use-assistant-live"; -import { - useAssistantLiveApprovals, - type AssistantLiveApprovals, -} from "./use-assistant-live-approvals"; +import type { AssistantLiveApprovals } from "./use-assistant-live-approvals"; + +const SESSION_ACTIONS: AssistantLiveApprovals = { + approvals: [], + decidingApprovalId: null, + decideApproval: async () => undefined, +}; const AIDEN_LOGO_URL = new URL("../../../resources/app-icon.png", import.meta.url).href; const AIDEN_LIVE_SETUP_COMPLETE_KEY = "aiden.live.setup-complete"; const LEGACY_GEMINI_LIVE_SETUP_COMPLETE_KEY = "aiden.gemini-live.setup-complete"; +export function liveDockClickAction(live: Pick, setupCompleted: boolean, stopRevealed: boolean) { + if (live.state === "closing") return "none"; + if (live.active) return stopRevealed ? "stop" : "reveal-stop"; + if (live.busy) return "none"; + if (!setupCompleted) return "setup"; + return live.setupComplete ? "start" : "settings"; +} + function storedSetupComplete(): boolean { try { return ( @@ -36,11 +47,6 @@ function storedSetupComplete(): boolean { export function AssistantDock({ rightInset = 0 }: { rightInset?: number }): React.ReactElement { const navigate = useNavigate(); const live = useAssistantLive(null); - const chat = useAssistantLiveApprovals( - live.voiceApprovalReceipts, - live.latestVoiceApprovalReceiptId, - live.retainVoiceApprovalReceiptsAfter, - ); const openSettings = React.useCallback( (section: SettingsSection) => { live.setSetupOpen(false); @@ -50,7 +56,7 @@ export function AssistantDock({ rightInset = 0 }: { rightInset?: number }): Reac ); return ( void; useCommand?: typeof useCommandHandler; }): React.ReactElement { - const [hudOpen, setHudOpen] = React.useState(false); + const [hudOpen, setHudOpen] = React.useState(Boolean(live.error)); + const [stopRevealed, setStopRevealed] = React.useState(false); const [setupCompleted, setSetupCompleted] = React.useState(storedSetupComplete); const triggerRef = React.useRef(null); const liveApproval = chat.approvals.find((approval) => approval.toolName === "computer_use"); const approvalPending = Boolean(liveApproval); const orbState = assistantLiveOrbState(live, approvalPending); + React.useEffect(() => { + if (!live.active) setStopRevealed(false); + }, [live.active]); + React.useEffect(() => { if (approvalPending || live.error) setHudOpen(true); - if (!live.active) setHudOpen(false); + else if (!live.active) setHudOpen(false); }, [approvalPending, live.active, live.error]); React.useEffect(() => { @@ -94,20 +105,20 @@ export function AssistantDockPresentation({ }, [live.active, live.microphoneActive, setupCompleted]); const openPanel = React.useCallback(() => { - if (!setupCompleted) { + const action = liveDockClickAction(live, setupCompleted, stopRevealed); + if (action === "none") return; + if (action === "stop") { void live.stop(); return; } + if (action === "reveal-stop") { setStopRevealed(true); return; } + if (action === "setup") { live.setSetupOpen(true); return; } - if (live.active) { - setHudOpen((open) => !open); - return; - } - if (live.setupComplete) { + if (action === "start") { void live.start(); return; } onOpenSettings(live.available ? "computerUse" : "providers"); - }, [live, onOpenSettings, setupCompleted]); + }, [live, onOpenSettings, setupCompleted, stopRevealed]); useCommand("assistant.open", openPanel, live.visible); if (!live.visible) return <>; @@ -116,7 +127,7 @@ export function AssistantDockPresentation({ className="pointer-events-none absolute bottom-4 z-40 flex flex-col items-end gap-2 transition-[right] duration-300 ease-out motion-reduce:transition-none" style={{ right: `calc(1rem + ${Math.max(0, rightInset)}px)` }} > - {live.active && hudOpen ? ( + {(live.active || live.error) && (hudOpen || live.screenActive) ? ( {liveApproval ? ( { if (event.key === "Escape") setStopRevealed(false); }} onClick={openPanel} > {setupCompleted ? ( @@ -149,8 +163,14 @@ export function AssistantDockPresentation({ )} +