Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .papercuts/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -528,3 +528,5 @@ symlink with this checkout's own npm ci. Full type-check and lint then passed.
- Pullfrog completed only after GitHub had already accepted the exact-head merge, and it found release-blocking screen-capture races. Keep publication cancellable until external review finishes, even when first-party CI and the merge gate are green.
- Electron exposes setters but no getters for session permission handlers. Scope the Live handler to the lifetime of exact display bindings, explicitly admit only display capture and microphone for that bound document, then restore the default handlers when the last binding disappears.
- Electron's `media` permission covers camera as well as microphone; check `mediaType` on permission checks and exact `mediaTypes: ["audio"]` on permission requests instead of treating the permission name as audio-only.
- A green external-review check can race a final inline comment by seconds. Re-read unresolved review threads after the check completes and make the merge command conditional on an empty result rather than chaining inspection and merge unconditionally.
- React effect cleanup marks the hook unmounted before revoking picker authority; reconfiguration needs the replacement effect setup to reset UI ownership, while final unmount must not schedule state recovery.
2 changes: 2 additions & 0 deletions docs/plans/gemini-live-assistant-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ Date: 2026-09-15
Continuous orb follow-up (2026-09-17): active listening/thinking/speaking/acting now share stable Listening + Weaving canvas layers with a slow complementary blend and connection/rest crossfades. Removed visible dock status text (retained screen-reader status). First active click reveals Stop; the second stops, Escape restores the orb, and closing disables duplicate clicks. Sharing/error surfaces remain visible. Added mapping/click-policy regressions; all 230 Live tests, type-check, lint, and standalone browser interaction/motion checks pass. Downloads duplex-review.html uses the actual orb component; the installed notarized app has not been rebuilt for this change.

Release hardening follow-up (2026-09-17): Pullfrog's post-merge review blocked 0.41.4 publication until screen authority is generation-fenced. Exact binding tokens now prevent stale releases from revoking replacements, late picker streams stop after teardown, rejected frame sends revoke capture, and temporary Electron permission handlers admit only audio-only media for the exact document before restoring defaults when the final binding closes. Source Change/Remove is locked across pending provider startup and start failure releases the exact current binding. Adversarial coverage was added for each path.

Final picker-state follow-up (2026-09-17): Live start is also blocked while the native picker is pending, and every authority-cancellation path synchronously clears the picker busy state before a later picker can begin. This prevents a superseded picker from leaving setup permanently disabled.
Combined artifact verification (2026-09-17): Apple accepted notarization submission `5b7f74d8-4740-47b9-82d8-66704cb75ec7`; the Downloads Combined Notarized app has a stapled ticket and passes the repository notarized-package/Gatekeeper check. Opened with the preserved isolated Test Profile and verified its renderer and Start Aiden Live control. Full Live suites, type-check, lint, audio/settings tests, and Electron selector-layout regression passed. Native-picker and real-provider acceptance remain separate and pending.
Combined build (2026-09-17): integrated the screen-sharing work from the 2f30 worktree into this audio-fix checkout without changing the source worktree. Preserved extended-thinking model selection, dedicated session threads, async thread finalization, direct actions, diagnostic markers, empty-envelope handling, device routing, and dropdown layout. Screen capture remains separately gated and off in the ordinary test launcher pending the exact-build native-picker acceptance receipt. Notarization and attended acceptance are separate gates; this is not a public release.
Audio selector layout follow-up: scoped single-line trigger styling and actual Radix value-span ellipsis fix long device labels pushing chevrons outside the control. Real Electron geometry regression passes at 1280, 600, and 390px; the full label remains available through the dropdown and hover title.
Expand Down
2 changes: 1 addition & 1 deletion renderer/components/assistant/assistant-live.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ export function AssistantLiveSetupDialog({
title="Set up Aiden Live"
description="Give Aiden only the access it needs. Nothing is captured until you start a Live session."
confirmLabel={live.busy ? "Starting…" : "Start Live"}
confirmDisabled={!live.setupComplete || live.busy || Boolean(live.startBlockedReason)}
confirmDisabled={!live.setupComplete || live.busy || live.screenBusy || Boolean(live.startBlockedReason)}
cancelLabel={live.busy ? "Stop" : "Not now"}
allowCancelWhileBusy
actionClassName={ASSISTANT_LIVE_FOCUS_CLASS}
Expand Down
1 change: 1 addition & 0 deletions renderer/components/assistant/assistant-ui.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -280,6 +280,7 @@ test("screen sharing is opt-in, source-labelled, and visibly active in the HUD",
assert.match(live, /live\.releaseScreen/u);
assert.match(live, /busy=\{live\.screenBusy \|\| live\.busy\}/u);
assert.match(live, /disabled=\{live\.screenBusy \|\| live\.busy\}/u);
assert.match(live, /confirmDisabled=\{!live\.setupComplete \|\| live\.busy \|\| live\.screenBusy/u);
assert.match(live, /Sharing \{live\.screenSourceLabel \?\? "screen"\}/u);
assert.match(live, /role="alert"[\s\S]*live\.screenError/u);
});
Expand Down
47 changes: 47 additions & 0 deletions renderer/components/assistant/use-assistant-live.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,7 @@ interface HookFixture {
player: RecordingPlayer;
startIntents(): Array<{ microphone: boolean; screen?: boolean; computerUseAuthorization: string | null }>;
refreshAvailability(): Promise<void>;
refreshApi(): Promise<void>;
unmount(): Promise<void>;
}

Expand Down Expand Up @@ -314,6 +315,15 @@ async function mountHook(overrides: Partial<AssistantLiveDependencies> = {}): Pr
await settle();
flushSync(() => undefined);
},
refreshApi: async () => {
dependencies = {
...dependencies,
api: { ...dependencies.api },
};
flushSync(() => root.render(<Harness />));
await settle();
flushSync(() => undefined);
},
unmount: async () => {
flushSync(() => root.unmount());
await settle();
Expand Down Expand Up @@ -1652,23 +1662,60 @@ test("releasing setup fences and stops a display stream returned by a late picke
const choosing = fixture.controller().chooseScreenSource();
await settle();
assert.deepEqual(state.binds, [1]);
assert.equal(fixture.controller().screenBusy, true);

await fixture.controller().start();
assert.equal(state.startCalls.length, 0, "Live cannot start while its picker is pending");

fixture.controller().releaseScreen();
await settle();
assert.equal(fixture.controller().screenBusy, false);
picker.resolve({ getTracks: () => [state.displayTrack] });
await choosing;
await settle();

assert.equal(state.displayTrack.stops, 1, "the late stream is stopped immediately");
assert.deepEqual(state.releases, ["binding-1"]);
assert.equal(fixture.controller().screenBusy, false);
assert.equal(fixture.controller().screenSourceLabel, null);
await fixture.unmount();
});

test("an API refresh cancels a pending picker and re-arms screen choice", async () => {
const { dependencies, state } = screenDependencies();
const firstPicker = deferred<DisplayMediaStream>();
let picks = 0;
const fixture = await mountHook({
...dependencies,
getDisplayMedia: async () => {
picks += 1;
if (picks === 1) return firstPicker.promise;
return { getTracks: () => [state.displayTrack] };
},
});
const staleChoice = fixture.controller().chooseScreenSource();
await settle();
assert.equal(fixture.controller().screenBusy, true);

await fixture.refreshApi();
assert.equal(fixture.controller().screenBusy, false);
await fixture.controller().chooseScreenSource();
await settle();
assert.equal(picks, 2, "the replacement effect accepts a new picker");

firstPicker.resolve({ getTracks: () => [new FakeDisplayTrack()] });
await staleChoice;
await settle();
assert.equal(fixture.controller().screenSourceLabel, "Aiden window");
await fixture.unmount();
});

test("a rejected screen frame stops capture and releases its exact binding", async () => {
const { dependencies, state } = screenDependencies();
state.admitFrames = false;
const fixture = await mountHook(dependencies);
await fixture.controller().chooseScreenSource();
await settle();
await fixture.controller().start();
await settle();
await settle();
Expand Down
7 changes: 7 additions & 0 deletions renderer/components/assistant/use-assistant-live.ts
Original file line number Diff line number Diff line change
Expand Up @@ -586,6 +586,7 @@ export function useAssistantLiveWithDependencies(
screenPickerGeneration.current += 1;
const bindingId = screenBindingRef.current;
screenBindingRef.current = null;
if (mounted.current) setScreenBusy(false);
Comment thread
pullfrog[bot] marked this conversation as resolved.
if (bindingId) {
void dependencies.api.releaseDisplay?.(bindingId).catch(() => undefined);
}
Expand Down Expand Up @@ -701,6 +702,10 @@ export function useAssistantLiveWithDependencies(

React.useEffect(() => {
mounted.current = true;
// A dependency refresh may have cancelled an in-flight picker during the
// previous effect's cleanup. The replacement effect now owns the hook and
// must make the picker re-armable; final unmount has no replacement setup.
setScreenBusy(false);
if (!dependencies.geminiLive)
return () => {
mounted.current = false;
Expand Down Expand Up @@ -1140,6 +1145,7 @@ export function useAssistantLiveWithDependencies(
const start = React.useCallback(async () => {
if (
busy ||
screenBusy ||
dependencies.ordinaryBusyReason ||
!snapshot.available ||
!["granted", "not-determined"].includes(microphonePermission) ||
Expand Down Expand Up @@ -1223,6 +1229,7 @@ export function useAssistantLiveWithDependencies(
}
}, [
busy,
screenBusy,
computerUseEnabled,
dependencies,
microphone,
Expand Down