Aiden Live screen sharing, audio polish, and 0.41.4

.papercuts/troubleshooting.md

@@ -1,5 +1,9 @@
 # Troubleshooting
 
+- 2026-09-17 release gate: do not run `npm test` and `npm run build` concurrently in one worktree. Both compile the universal `build/native/aiden-worktree-remover`, and `lipo` races its temporary output. Run build first, then the full suite serially.
+- 2026-09-17 release gate: removing the final Live voice-approval sender left `chat:approval-withdrawn` in the preload notification allowlist. Focused Live tests do not own the global sender/allowlist equality contract; run the full suite before publishing renderer IPC changes.
+- 2026-09-17 Live motion: do not key canvas layers by caption/action state; remounting restarts the animation and causes jumps. Kept stable duplex layers and separated visual activity from microphone activity so mic-off sessions retain Stop. Production macOS package must be rebuilt separately from the HTML review bundle.
+
 ## 2026-09-12 — Listed upstream integration audit
 
 - This worktree has no `.memory/` or `node_modules/`. Read the main checkout's project memory as historical context, but use this worktree's exact HEAD/source as authority. The main checkout's `tsx` binary can execute dependency-free focused suites without installing packages here; suites with runtime package imports still fail module resolution (observed: `entities` in the subagent capability suite). Treat that as an environment limitation, not a product regression or passing test.
@@ -505,7 +509,19 @@ symlink with this checkout's own npm ci. Full type-check and lint then passed.
 - The installed Google SDK exposes Live `interactionStatus` inside `serverContent`, while newer Extended Thinking examples describe status alongside tool-call lifecycle events; pin protocol handling to the installed typed wire contract and cover `IDLE` explicitly when adopting the new model.
 # Release coordination
 
+- Screen-share work was based on an older Live contract. Integration must retain the exact Computer Use authorization token, extended-thinking model, async thread finalization, direct-action policy, device routing, and empty-envelope fix; add screen intent without replacing these later changes. Original screen worktree remains unchanged.
+
+- Live device picker visual check found two interacting issues: settings action-cluster CSS wraps every direct `.flex` child, including combobox triggers, and Radix Select.Value strips className/style. Scope a no-wrap override to Live device triggers and truncate their actual value spans; static markup tests alone did not catch geometry.
+
+- Live device routing must retain one audio dependency/player instance across capability refresh; otherwise start preflight can configure a different player from the hook's retained playback ref. Keep capabilities separate from audio ownership.
+
+- Audio selectors: Radix SelectValue has no selected-item text in static rendering until its item collection mounts; supply an explicit selected label so unavailable-device and initial-render states are readable and testable.
+
 - Pullfrog took just over one hour to review PR #132 after first-party CI was green; keep exact-head checks separate so the long external review does not obscure test status.
 - Moving Live from default-on to acceptance-gated correctly hid the dock but invalidated the local UI E2E; keep default-off rendering covered separately and opt the isolated provider-free E2E harness into the experimental surface explicitly.
 - The hosted full Electron suite marked the unrelated chat-switch queue test flaky after it passed on retry, and `--fail-on-flaky-tests` failed the whole gate; rerun the exact failed job before changing unrelated product behavior, while preserving the strict gate if the flake repeats.
 - A distant Environment source-contract test asserted Aiden Live's two-argument command registration, so focused Live tests missed the intentional capability gate; search all source-contract assertions when changing a shared command signature.
+- Signed Live test build: `isPackagedRuntime()` is false for an isolated development profile, causing Computer Use to search inside app.asar/build instead of Contents/Helpers. Resolve physical helper layout using `app.isPackaged`; profile identity must not determine package resource locations.
+- Live cue tests must flush passive React effects after microphone and stop state updates before asserting audio feedback. The repo has no local Prettier binary; use its configured ESLint validation instead.
+- Real signed Live session reached open/microphone-ready/input-first-packet, then Google emitted an empty top-level envelope at 00:20:01 UTC on 2026-09-17. Rejecting `{}` caused the observed silent disconnect. Admit exact empty envelopes as rate-limited no-ops without extending idle timeout; keep unknown populated fields rejected. Terminal error HUDs must remain visible after active becomes false.
+- The Mac's default input was Bose Mini II while output was MacBook speakers. Packet flow alone does not prove intelligible user speech or audible playback; retain separate operator verification.

docs/plans/README.md

@@ -21,7 +21,7 @@ This directory is the source of truth for Aiden's implementation plans. The engi
 | [Dynamic Model Catalog](dynamic-model-catalog-plan.md)                                             | Implemented | Validated pi.dev overlays, offline `0600` cache hydration, scoped setup refresh, four-hour launch refresh, force refresh, Pi metadata fallback, and Mac/iOS projection ship on pinned Pi 0.84.4. |
 | [Generative UI Artifacts](generative-ui-artifacts-plan.md)                                         | Active  | Phases 0–6 shipped: chat-scoped `render_artifact`, strict sandboxed preview/export hosts, verified vendored Chart.js/Plotly/KaTeX, permission-aware `/visualize`, crash-recoverable authoritative storage/copies, descriptor-relative workspace reads, one-iframe handoff/expansion, visible failure states, and route-stable Responding/Visualizing activity. Three-agent PR review findings are remediated with focused regression coverage. |
 | [Generation Progress Notes](generation-progress-notes-plan.md)                                     | Planned | No implementation yet.                                                                                                                                                                                                    |
-| [Aiden Live Assistant](gemini-live-assistant-plan.md)                                              | Partial | Phases 0–4, Extended Thinking spoken progress, strict voice-only per-action approval, the chrome-free blue orb, Google model visibility controls, and metadata-only per-session threads are implemented; native screen-picker and real-Google operator receipts remain open. |
+| [Aiden Live Assistant](gemini-live-assistant-plan.md)                                              | Partial | Extended Thinking, the blue orb, Google model visibility, and session threads are implemented. Local test builds add input/output device selection, connection sounds, and session-authorized direct actions without per-action prompts; native screen-picker and real-Google operator receipts remain open. |
 | [Libghostty workspace terminal](libghostty-terminal-plan.md)                                        | Implemented | The workspace drawer uses Ghostty's official `libghostty-vt` WASM (T3-style runtime, PTY trampoline, canvas surface); node-pty sessions are unchanged. Packaged Mac acceptance remains. |
 | [Logging and Diagnostics Upgrade](logging-and-diagnostics-upgrade-plan.md)                           | Implemented | Phases 0–7 are implemented: bounded typed desktop journals, main-owned renderer evidence, local support export/delete, native categorical parity, and CI/release gates. Signed/notarized `v0.35.0` passed packaged diagnostics acceptance; physical-device termination receipts remain. |
 | [Long-thread payload upgrades](long-thread-payload-upgrade-plan.md)                                 | Partial | Investigation complete: T3’s O(N²) stdout store does not exist here. No-op `toolRunning` timeline republish is skipped; Remote gzip, stream-journal debounce, chat JSON/attachments, and transcript windowing remain planned. |

docs/plans/gemini-live-assistant-plan.md

@@ -2,6 +2,13 @@
 
 Status: Partial — Phases 0–4 plus the beta-labeled Aiden Live orb/setup shell and dedicated metadata-only session threads are implemented; authorized macOS screen capture and real Google beta receipts remain operator-owned
 Date: 2026-09-15
+Continuous orb follow-up (2026-09-17): active listening/thinking/speaking/acting now share stable Listening + Weaving canvas layers with a slow complementary blend and connection/rest crossfades. Removed visible dock status text (retained screen-reader status). First active click reveals Stop; the second stops, Escape restores the orb, and closing disables duplicate clicks. Sharing/error surfaces remain visible. Added mapping/click-policy regressions; all 230 Live tests, type-check, lint, and standalone browser interaction/motion checks pass. Downloads duplex-review.html uses the actual orb component; the installed notarized app has not been rebuilt for this change.
+Combined artifact verification (2026-09-17): Apple accepted notarization submission `5b7f74d8-4740-47b9-82d8-66704cb75ec7`; the Downloads Combined Notarized app has a stapled ticket and passes the repository notarized-package/Gatekeeper check. Opened with the preserved isolated Test Profile and verified its renderer and Start Aiden Live control. Full Live suites, type-check, lint, audio/settings tests, and Electron selector-layout regression passed. Native-picker and real-provider acceptance remain separate and pending.
+Combined build (2026-09-17): integrated the screen-sharing work from the 2f30 worktree into this audio-fix checkout without changing the source worktree. Preserved extended-thinking model selection, dedicated session threads, async thread finalization, direct actions, diagnostic markers, empty-envelope handling, device routing, and dropdown layout. Screen capture remains separately gated and off in the ordinary test launcher pending the exact-build native-picker acceptance receipt. Notarization and attended acceptance are separate gates; this is not a public release.
+Audio selector layout follow-up: scoped single-line trigger styling and actual Radix value-span ellipsis fix long device labels pushing chevrons outside the control. Real Electron geometry regression passes at 1280, 600, and 390px; the full label remains available through the dropdown and hover title.
+Audio selection (2026-09-16 local): Settings → Aiden Live now offers device-local input/output preferences, system defaults, hotplug list refresh, and a speaker test. Selections are snapshotted per session; explicit microphone constraints and output sink routing apply to voice capture, replies, and both cues. Missing selections fail with recovery instructions instead of silent startup fallback. Setup points users to these settings. The shared audio player remains stable across capability refresh; preparation is cancellation-fenced. Tests extend the existing registered Live/UI/audio suites.
+Runtime diagnosis (2026-09-16 local): signed Live reached Google/open and microphone packet flow, but an empty top-level server envelope caused a fatal parser error. Exact empty envelopes now count against rate limits without extending idle liveness. Fixed lifecycle/audio/cue markers enter the local diagnostic journal without media/transcripts/credentials. The orb now labels microphone state, preserves terminal errors, and uses more audible connection tones. The signed Audio Fix build subsequently connected, captured microphone packets, received Google response audio, started playback, and stayed connected for 54 seconds until manually stopped. Both cue markers and the listening/idle labels were verified. Physical audibility still needs user confirmation; Computer Use acceptance remains unclaimed. Relevant tests, type-check, lint, and hardened development package verification passed.
+Local test-build update (2026-09-16): signed development profiles resolve the bundled Computer Use helper by physical package layout. Live plays connection/disconnection cues. At the user's explicit request, starting Live with Computer Use enabled authorizes direct actions for that session without per-action prompts or voice approvals. The dock no longer mounts the voice-approval listener. Target binding, current capture requirements, cancellation, owner identity, and enablement checks remain enforced. Historical per-action approval sections below describe the superseded implementation; ordinary chat policy is unchanged. No new real-provider acceptance is claimed.
 Related: `aiden-assistant-plan.md`, `pi-provider-integration-plan.md`, and
 `../computer-use-integration.md`
 
@@ -11,9 +18,9 @@ The bottom-right **Aiden orb** becomes the single entry point for a
 user-started Aiden Live session. Aiden streams microphone PCM and a
 user-approved screen/window as bounded JPEG frames to Gemini Live, plays Gemini
 native audio, and shows its input/output captions. When Gemini needs to act, it
-invokes Aiden's existing `computer_use` tool; every input action keeps the
-current global gate, per-chat activation, target binding, and fresh **Allow
-once** approval.
+invokes Aiden's existing `computer_use` tool. During Live, actions execute under
+the user's session consent with global enablement and exact target binding.
+Stopping Live revokes this authority. Ordinary chats retain their own policy.
 
 This is deliberately not a general screen recorder, an unattended assistant,
 or a second automation authority.

main/handlers/assistant-live-parse.test.ts

@@ -1,7 +1,10 @@
 import assert from "node:assert/strict";
 import test from "node:test";
+import { GEMINI_LIVE_MAX_JPEG_BYTES } from "../services/gemini-live/protocol.js";
 import {
   parseAssistantLiveAudioIntent,
+  parseAssistantLiveEmptyIntent,
+  parseAssistantLiveFrameIntent,
   parseAssistantLiveStartIntent,
   parseAssistantLiveStopIntent,
 } from "./assistant-live-parse.js";
@@ -21,6 +24,9 @@ test("Assistant Live audio admission accepts only one exact 20 ms PCM chunk", ()
 });
 
 test("Assistant Live start intent accepts only an exact bounded authorization record", () => {
+  assert.equal(parseAssistantLiveStartIntent({ microphone: true, screen: true, computerUseAuthorization: null }).screen, true);
+  assert.throws(() => parseAssistantLiveStartIntent({ microphone: true, screen: "true", computerUseAuthorization: null }));
+  assert.throws(() => parseAssistantLiveStartIntent({ microphone: true, screen: true }));
   assert.deepEqual(parseAssistantLiveStartIntent({ microphone: true, computerUseAuthorization: null }), {
     microphone: true,
     computerUseAuthorization: null,
@@ -51,3 +57,33 @@ test("Assistant Live stop intent accepts only an exact empty record", () => {
     /Invalid Assistant Live/u,
   );
 });
+
+test("Assistant Live frame admission accepts only one bounded copied JPEG byte range", () => {
+  const frame = new Uint8Array([0xff, 0xd8, 0x2a, 0xff, 0xd9]);
+  const parsed = parseAssistantLiveFrameIntent({ sessionId: "session-1", frame });
+  assert.equal(parsed.sessionId, "session-1");
+  assert.deepEqual(parsed.frame, frame);
+  assert.notEqual(parsed.frame, frame, "the parser copies admitted bytes");
+  for (const value of [
+    { sessionId: "session-1", frame: new Uint8Array(3) },
+    { sessionId: "session-1", frame: new Uint8Array(GEMINI_LIVE_MAX_JPEG_BYTES + 1) },
+    { sessionId: "", frame },
+    { sessionId: "session-1", frame: frame.buffer },
+    { sessionId: "session-1", frame, apiKey: "secret" },
+    { frame },
+  ]) assert.throws(() => parseAssistantLiveFrameIntent(value), /frame request/u);
+});
+
+test("Assistant Live display bind/release accept only an exact empty record", () => {
+  for (const name of ["display-bind", "display-release"]) {
+    assert.doesNotThrow(() => parseAssistantLiveEmptyIntent({}, name));
+    assert.throws(
+      () => parseAssistantLiveEmptyIntent({ sessionId: "forged" }, name),
+      new RegExp(`Invalid Assistant Live ${name}`, "u"),
+    );
+    assert.throws(
+      () => parseAssistantLiveEmptyIntent(null, name),
+      new RegExp(`Invalid Assistant Live ${name}`, "u"),
+    );
+  }
+});

main/handlers/assistant-live-parse.ts

@@ -1,4 +1,5 @@
 import type { AssistantLiveStartIntent } from "../../renderer/shared/assistant-live.js";
+import { GEMINI_LIVE_MAX_JPEG_BYTES } from "../services/gemini-live/protocol.js";
 
 function isRecord(value: unknown): value is Record<string, unknown> {
   return Boolean(value && typeof value === "object" && !Array.isArray(value));
@@ -12,7 +13,8 @@ function exactKeys(value: Record<string, unknown>, expected: readonly string[]):
 export function parseAssistantLiveStartIntent(value: unknown): AssistantLiveStartIntent {
   if (
     !isRecord(value) ||
-    !exactKeys(value, ["computerUseAuthorization", "microphone"]) ||
+    !exactKeys(value, "screen" in value ? ["computerUseAuthorization", "microphone", "screen"] : ["computerUseAuthorization", "microphone"]) ||
+    ("screen" in value && typeof value.screen !== "boolean") ||
     typeof value.microphone !== "boolean" ||
     !(
       value.computerUseAuthorization === null ||
@@ -25,6 +27,7 @@ export function parseAssistantLiveStartIntent(value: unknown): AssistantLiveStar
   }
   return {
     microphone: value.microphone,
+    ...("screen" in value ? { screen: value.screen as boolean } : {}),
     computerUseAuthorization: value.computerUseAuthorization,
   };
 }
@@ -54,3 +57,33 @@ export function parseAssistantLiveAudioIntent(value: unknown): AssistantLiveAudi
   }
   return { sessionId: value.sessionId, pcm: Uint8Array.from(value.pcm) };
 }
+
+export interface AssistantLiveFrameIntent {
+  sessionId: string;
+  frame: Uint8Array;
+}
+
+export function parseAssistantLiveFrameIntent(value: unknown): AssistantLiveFrameIntent {
+  if (
+    !isRecord(value) ||
+    !exactKeys(value, ["frame", "sessionId"]) ||
+    typeof value.sessionId !== "string" ||
+    value.sessionId.length < 1 ||
+    value.sessionId.length > 128 ||
+    !(value.frame instanceof Uint8Array) ||
+    value.frame.byteLength < 4 ||
+    value.frame.byteLength > GEMINI_LIVE_MAX_JPEG_BYTES
+  ) {
+    throw new Error("Invalid Assistant Live frame request.");
+  }
+  return { sessionId: value.sessionId, frame: Uint8Array.from(value.frame) };
+}
+
+export function parseAssistantLiveEmptyIntent(
+  value: unknown,
+  name: string,
+): void {
+  if (!isRecord(value) || !exactKeys(value, [])) {
+    throw new Error(`Invalid Assistant Live ${name} request.`);
+  }
+}