MSGuide: live Teams recovery, local voice, and question-first UI

.github/workflows/ci.yml

@@ -0,0 +1,69 @@
+name: CI
+
+on:
+  push:
+  pull_request:
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  windows:
+    runs-on: windows-2025
+    timeout-minutes: 20
+    defaults:
+      run:
+        shell: pwsh
+    env:
+      MSGUIDE_GUIDANCE_PROVIDER: demo
+      MSGUIDE_SESSION_SCREEN_CONTEXT: "0"
+      MSGUIDE_SESSION_AUTOMATION: "0"
+      MSGUIDE_WHISPER_SYNTHETIC_TEST: "0"
+      MSGUIDE_SPEECH_SYNTHETIC_TEST: "0"
+      MSGUIDE_DIAGNOSTIC_LOG: ""
+      MSGUIDE_DESKTOP_LOG: ""
+      PYTHONUTF8: "1"
+      DOTNET_NOLOGO: "true"
+      DOTNET_CLI_TELEMETRY_OPTOUT: "1"
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+          clean: false
+      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.11.9"
+          architecture: x64
+          pip-version: "26.2.1"
+          cache: pip
+          cache-dependency-path: |
+            requirements.lock.txt
+            requirements-dev.lock.txt
+      - uses: actions/setup-dotnet@a98b56852c35b8e3190ac28c8c2271da59106c68 # v6.0.0
+        with:
+          global-json-file: global.json
+          cache: true
+          cache-dependency-path: |
+            desktop\packages.lock.json
+            desktop\CaptureProbe\packages.lock.json
+      - name: Install hash-locked Python dependencies
+        run: |
+          python -m venv --without-pip .venv &&
+            python -m pip --python .venv install --require-hashes --only-binary=:all: -r requirements-dev.lock.txt &&
+            python -m pip --python .venv check
+      - name: Backend and dependency-contract regressions
+        run: .\.venv\Scripts\python -m pytest -q -p no:cacheprovider
+      - name: Restore locked .NET dependencies
+        run: dotnet restore .\desktop\CaptureProbe\MSGuide.CaptureProbe.csproj --locked-mode --verbosity minimal
+      - name: Build desktop and diagnostic probe
+        run: dotnet build .\desktop\CaptureProbe\MSGuide.CaptureProbe.csproj --configuration Release --no-restore --nologo --verbosity minimal
+      - name: Desktop synthetic self-tests
+        run: |
+          $report = Join-Path $env:RUNNER_TEMP 'msguide-self-test.json'
+          $process = Start-Process -FilePath .\desktop\bin\Release\net10.0-windows10.0.19041.0\MSGuide.Desktop.exe `
+            -ArgumentList '--self-test', '--test-results', "`"$report`"" -Wait -PassThru
+          if ($process.ExitCode -ne 0) { throw "Desktop self-test failed: $($process.ExitCode)" }
+          $result = Get-Content -LiteralPath $report -Raw | ConvertFrom-Json
+          if ($result.passed -ne $true) { throw 'Desktop self-test report did not pass.' }
+          Get-Content -LiteralPath $report

.gitignore

@@ -6,6 +6,8 @@ __pycache__/
 htmlcov/
 desktop/bin/
 desktop/obj/
+desktop/CaptureProbe/bin/
+desktop/CaptureProbe/obj/
 .env
 .env.*
 !.env.example

Directory.Build.props

@@ -0,0 +1,5 @@
+<Project>
+  <PropertyGroup>
+    <RestorePackagesWithLockFile>true</RestorePackagesWithLockFile>
+  </PropertyGroup>
+</Project>

IMPLEMENTATION.md

@@ -1,25 +1,27 @@
 # Implementation status
 
-Updated September 14, 2026. This is a **local, single-user, guide-only desktop MVP**, not an enterprise service. Start with [README.md](README.md); evidence and remaining checks are in [docs/VALIDATION.md](docs/VALIDATION.md).
+Updated September 18, 2026. This is a **local, single-user desktop MVP with non-executing Guide mode and bounded, locally grounded plan execution**, not an enterprise service or unrestricted agent. Start with [README.md](README.md); historical evidence and remaining live checks are in [docs/VALIDATION.md](docs/VALIDATION.md).
 
 ## Implemented path
 
 1. [scripts/Start-MSGuide.ps1](scripts/Start-MSGuide.ps1) builds WPF, starts the loopback backend on port 8765 by default, and gives both children an ephemeral local bearer token. Only the backend inherits the model API key. Server lifetime is tied to desktop/launcher cleanup; no token file is created.
-2. [desktop/MainWindow.xaml.cs](desktop/MainWindow.xaml.cs) manages the hotkey, window selection, editable prompt, capture review, consent, explicit Send, fresh Check, and cancellation.
-3. [desktop/CaptureService.cs](desktop/CaptureService.cs) captures the selected HWND with `PrintWindow` and collects bounded UI Automation names/boxes. It provides a real local preview, not OCR or redaction. PNGs are bounded to 1600 pixels per side and 2,000,000 bytes.
+2. [desktop/CompanionWindow.cs](desktop/CompanionWindow.cs) provides the normal Clicky-style shell: a click-through Windows-logo buddy and response bubble follow the cursor at 60 FPS, while the hotkey opens a compact interactive prompt with monitor-edge clamping. [desktop/MainWindow.xaml.cs](desktop/MainWindow.xaml.cs) remains the orchestration and expanded review/approval surface.
+3. [desktop/CaptureService.cs](desktop/CaptureService.cs) uses selected-HWND Windows Graphics Capture, without desktop/PrintWindow fallback, and cached, bounded UIA evidence prioritized by actionability. Non-action context cropping does not invalidate a complete controls scan; actual traversal failures remain blocked. Logical control identity is separate from the exact reviewed-state target token. It provides a local preview, not OCR/redaction; PNGs are bounded to 1280 pixels on their longest side and 2,000,000 bytes.
 4. [desktop/ApiClient.cs](desktop/ApiClient.cs) calls health, sessions, and guidance only. It rejects non-loopback destinations, disables proxies/redirects, and does not call legacy assist/action routes.
 5. [src/main.py](src/main.py) checks the local boundary, session, consent, freshness, body limits, provider output, and target correspondence. It defines **10 application routes**, excluding FastAPI's four generated documentation/schema routes; see [docs/API.md](docs/API.md).
-6. [src/guidance.py](src/guidance.py) implements the deterministic built-in demo. [src/model_provider.py](src/model_provider.py) implements the opt-in OpenAI-compatible transport; neither provider can execute tools. Model setup is [separate from local security mode](docs/MODEL_SETUP.md).
-7. [desktop/OverlayWindow.cs](desktop/OverlayWindow.cs) draws a nonactivating target outline. [desktop/SpeechService.cs](desktop/SpeechService.cs) supplies installed Windows speech recognition/playback. The user clicks; Check begins another review cycle.
+6. [src/guidance.py](src/guidance.py) implements the deterministic built-in demo. [src/model_provider.py](src/model_provider.py) and [src/copilot_provider.py](src/copilot_provider.py) select only reviewed targets; neither provider executes desktop tools. Model setup is [separate from local security mode](docs/MODEL_SETUP.md).
+7. [desktop/ScreenTaskSession.cs](desktop/ScreenTaskSession.cs) validates/retains up to 32 steps per plan and executes continuously without eight-action or two-minute checkpoints. Every action is observed; suitable post-action evidence is reused for the next local binding. After progress, plan-limit/observation boundaries trigger a fresh approved capture and next model plan only if the same identified resource remains completely inspectable. Empty plans do not spin. [desktop/DesktopAction.cs](desktop/DesktopAction.cs) still reacquires/checks each exact target, value, capability and window before invoking. Deferred intents require unique fresh complete matches and deferred writes require empty fields. Resource changes, missing grounding and permission/info boundaries stop explicitly. Per-operation deadlines and the 10,000-decision protocol ceiling remain; unknown and cancelled queues cannot resume.
+8. The compact interactive prompt and Details share mode, continuation/reply and Stop handlers. Generic actions bring only the approved window forward from the companion, visibly place the Windows marker/outline on the target, then revalidate before invocation; unrelated foreground changes stop rather than cause background input. The cursor buddy/overlay remain click-through and non-activating. Speech transcript invalidation is separate from hardware shutdown acknowledgement; stopping/unknown input gates new recordings and input changes, and late closure never revives cancelled text. Whisper model-load, processor and inference timings are separate; no speculative factory cache was added.
 
 ## Implemented safeguards, not enterprise guarantees
 
 - The API accepts loopback clients/local Host values and rejects browser Origin headers. `/v1` and `/admin/` require the configured bearer token before body parsing. All authenticated calls share the local principal; no employee identity or cross-user authorization is established.
 - Requests are bounded to 3,000,000 bytes. Observations expire after 60 seconds, with at most five seconds of future clock skew. The desktop additionally checks window identity/bounds and response echo IDs.
-- [src/models.py](src/models.py) validates normalized target boxes and confidence. Targets must match reviewed UIA evidence; model output selects an existing element index, never arbitrary coordinates. Model completion claims become clarification with a verification warning.
+- [src/models.py](src/models.py) validates complete plan shapes and boundaries before any first action. Observed references become server-derived logical IDs; future intents are bounded exact descriptions, not invented IDs/selectors. Generic completion remains a model suggestion with `review_required` UI, never independently verified goal success.
+- Execution remains desktop-owned: supported UIA invocation, toggling, selection, expansion, bounded full-field replacement and small scrolling only. Password/read-only/value/identity/window checks remain live. No arbitrary typing, dragging, coordinate input, shell or filesystem tool execution is added. Supported Edge/Chrome windows use a locally checked browser-chrome address hash; other document trees without proven file/site identity require handoff; partial approved evidence can still support Guide descriptions.
 - [src/images.py](src/images.py) uses Pillow to decode/verify bounded PNGs and re-encode pixels without metadata. Valid PNGs are accepted even in demo mode, but deterministic guidance ignores them. Sharing pixels is unnecessary for that demo.
-- Capture/upload is manual. Prompt or approval changes, cancellation, and supersession invalidate pending work. Native capture runs on a worker with a 30-second waiting budget; it cannot be safely force-aborted. One stuck worker can block later captures until restart.
-- Evidence is not deliberately persisted. Clearing references/arrays does not guarantee erasure of all managed/native copies. Remote retention is the configured provider's policy, not controlled here.
+- Manual developer sessions require capture review and approval. A `-Copilot` launch uses its process-lifetime screen-context grant for foreground capture. Prompt or consent changes, cancellation, and supersession invalidate pending work. Native capture runs on a worker with a 30-second waiting budget; it cannot be safely force-aborted. One stuck worker can block later captures until restart.
+- Evidence is not deliberately persisted. Bounded rotating diagnostics under `%LOCALAPPDATA%\MSGuide\logs` contain operational metadata only, never screenshots, transcripts, tokens, UI text, prompts, or model output. Clearing references/arrays does not guarantee erasure of all managed/native copies. Remote retention is the configured provider's policy, not controlled here.
 
 ## Backend-only sample features
 
@@ -31,6 +33,11 @@ Sessions, previews, jobs, and optional audit events are bounded process-local st
 
 ## Evidence and gaps
 
-The parent validation run reports **149 pytest passes**, a clean dependency check, successful .NET build, and successful desktop self-test. Provider tests use synthetic evidence and mocked transport, not an approved live model. The real harness failed when `demo.Activate()` returned false after visible layout/rendering; foreground restrictions are only an unconfirmed explanation.
+The new offline suites exercise both fake providers/API, whole-plan rejection,
+three and 17 steps with one model call, batch continuation, scope/target drift,
+stable effects after label/position changes, compact handlers, and fake-input
+stop timeout/late acknowledgement. These are not live-model/app/device acceptance
+or native Whisper performance measurements. Older build/test counts and camera/
+voice observations in historical runbooks predate these changes.
 
-Do not infer capture, target placement, microphone, mixed-DPI behavior, or end-to-end desktop success from build/unit checks. See [docs/VALIDATION.md](docs/VALIDATION.md) and the unchecked milestones in [PLAN.md](PLAN.md). No OCR, enterprise access, internal-data pilot, full dependency lock, or production deployment is complete.
+Do not infer capture, target placement, microphone, mixed-DPI behavior, or end-to-end desktop success from build/unit checks. See [docs/VALIDATION.md](docs/VALIDATION.md) and [PLAN.md](PLAN.md). Python/NuGet locks and CI do not constitute live acceptance. OCR, enterprise access, an internal-data pilot and production deployment remain out of scope.