diff --git a/.github/release-notes/v1.1.9-rc.1.md b/.github/release-notes/v1.1.9-rc.1.md new file mode 100644 index 000000000..829f31ffc --- /dev/null +++ b/.github/release-notes/v1.1.9-rc.1.md @@ -0,0 +1,12 @@ +# Memmy v1.1.9-rc.1 + +## Integration candidate + +- Integrates the v1.1.9 Memory baseline with consent gated Computer History, personal WeChat History source controls, and reviewed Skill links. +- Adds a shared Agent browser session, Computer Use side panel and live macOS window preview, with Chrome and Edge extension pairing for explicitly claimed tabs. +- Adds per app desktop approval, site and download approval, browser annotations and manual autofill, plus a local Excel task pane bridge. +- Packages the integrated Mac and Windows desktop runtimes with their native Computer Use helpers. + +## Validation scope + +This is a local test candidate. Locked Mac use still requires a separate authorization plugin and guardian. Excel and Windows native application control require host testing before release. The Memory service keeps its independent version. diff --git a/.github/release-notes/v1.2.2-rc.1.md b/.github/release-notes/v1.2.2-rc.1.md new file mode 100644 index 000000000..3d83ce405 --- /dev/null +++ b/.github/release-notes/v1.2.2-rc.1.md @@ -0,0 +1,7 @@ +# Memmy v1.2.2-rc.1 + +本地候选测试版本,基于正式 1.2.1 汇总 9 月 23 日以来已确认的桌面、聊天、浏览器、Computer Use、History、微信、Excel 和桌宠改动。未合入另行发布的插件和文献综述分支。 + +修复记忆服务的测试版比较和并发安装、Mac 升级验证与失败恢复、Windows 安装器验证、微信读取运行组件和权限准备链路。Windows 微信使用安装包内的独立运行组件。 + +源码回归与构建检查和普通用户安装验收分别记录。最终签名安装包的 Mac 和 Windows 首次安装、从 1.2.1 升级及真实微信读取仍需实机验收。完整范围与测试顺序见 `docs/cn/desktop/local-release-test-20261009.md`。 diff --git a/.github/release-notes/v1.2.3-rc.2.md b/.github/release-notes/v1.2.3-rc.2.md new file mode 100644 index 000000000..fc8682864 --- /dev/null +++ b/.github/release-notes/v1.2.3-rc.2.md @@ -0,0 +1,11 @@ +# Memmy v1.2.3-rc.2 + +基于同事已定版的 `origin/v1.2.2`(`2069443c`),整合 2026-09-23 以来的本地开发与后续 Cursor 修复,供 Mac 与 Windows 安装、覆盖升级验收。 + +- 保留 1.2.2 的 Memory 构建身份识别、数据库版本保护、对话捕获期限和后台捕获队列。 +- 补入微信窗口上下文与聊天名修复、History 入库调度、Agent 历史增量扫描及证据引用校验。 +- 补入 Excel 工作簿选择与授权、Windows Excel 错误反馈、浏览器能力探测及桌面白屏恢复。 +- 补入初见报告采样与流读取期限,避免等待全量历史。 +- 同时检查 Mac 与 Windows 的老用户数据、设置、登录状态、Memory 运行时、微信游标及权限迁移。 + +本地测试包;Mac arm64 使用 Developer ID 签名及 Apple 公证,Windows x64 使用无签名安装包。真实安装、覆盖升级和 GUI 验收由用户执行,源码测试及包内容检查不替代这些验收。 diff --git a/.github/release-notes/v1.2.3-rc.3.md b/.github/release-notes/v1.2.3-rc.3.md new file mode 100644 index 000000000..c16172e50 --- /dev/null +++ b/.github/release-notes/v1.2.3-rc.3.md @@ -0,0 +1,13 @@ +# Memmy v1.2.3-rc.3 + +在同事已定版的 1.2.2 和上一轮 1.2.3-rc.2 整合结果上,补入截至 2026-10-12 的有效 Agent / Cursor 后续代码,供 Mac 与 Windows 覆盖升级验收。 + +- 微信原库读取改用独立 DB/WAL 副本,避免 SQLite 只读连接影响原库共享索引。 +- 浏览器截图按图片交给模型,兼容旧会话截图,并保持当前请求、截图和工具调用配对的输入预算。 +- 浏览器支持新标签、标签切换、会话最近标签及扩展能力检测;插件更新到 1.0.2。保留用户已保存的站点允许/拒绝规则,改善四种审批的可见性。 +- Excel 实时工作簿支持公式、日期和错误格反馈,写入结果未知时先读回核对。 +- 合齐 Finder / Windows 已安装应用发现、后台输入与显式前台点击/键盘能力;保留模型按任务选择操作方式的后续方案。 +- 修复 Mac 系统权限 Quit & Reopen、锁屏实际解锁和重新锁定;旧锁屏组件按构建身份检测更新。 +- 保留 rc.2 的 Memory 安装锁、Windows 迁移路径、旧 rc 迁移 ID、数据库 schema 和构建身份保护。 + +本地候选包:Mac arm64 Developer ID 签名及 Apple 公证,Windows x64 未签名。真实安装、普通用户覆盖升级、Windows 原生运行和 GUI 验收由用户执行;源码及包内容检查分别记录。 diff --git a/.github/release-notes/v1.2.3-rc.4.md b/.github/release-notes/v1.2.3-rc.4.md new file mode 100644 index 000000000..af7785a9e --- /dev/null +++ b/.github/release-notes/v1.2.3-rc.4.md @@ -0,0 +1,7 @@ +# Memmy v1.2.3-rc.4 + +在同事最终 1.2.2 基线及 9.23 至今 Agent 整合代码上更新候选。纳入最新 Cursor 扩展重连、Windows History、微信慢连接/WAL、Windows 启动和窗口操作、Excel 设置引导、浏览器自动安装,并按最新决定移除锁屏操作。 + +覆盖升级保留旧聊天、Memory、设置和微信游标;原生 helper 通过编译期版本与协议握手拒绝继续复用旧进程。旧锁屏系统组件由用户使用独立手动恢复文件清理,应用不自动修改登录策略。 + +源码和包内核验见本轮整合说明及交付 evidence。Mac arm64 签名公证,Windows x64 未签名;真实安装、旧用户覆盖升级、Windows ABI、GUI 和系统策略清理仍需用户验收。 diff --git a/.github/release-notes/v1.2.3-rc.5.md b/.github/release-notes/v1.2.3-rc.5.md new file mode 100644 index 000000000..0c12a59e8 --- /dev/null +++ b/.github/release-notes/v1.2.3-rc.5.md @@ -0,0 +1,11 @@ +# Memmy v1.2.3-rc.5 + +在同事定版 1.2.2 和 9.23 至今本地 Agent 整合代码上更新候选,补齐新会话中的已提交及已完成未提交改动。 + +本轮纳入 Windows 原生 History 常驻采集与隐私判断、微信连接风险确认、定时回复可靠投递与任务名称分隔、最终文件随回复发送、活动队列展开、Windows 窗口及预览布局、工作区 Git 修复、浏览器安装状态与真实连接分离、扩展恢复与接管提示。修复 Windows 扩展自动安装对不支持系统接口的引用,并对冲突保留后续确认逻辑。 + +Mac 微信 SQLCipher/OpenSSL 依赖改用同版本、明确部署目标的源码构建;打包检查拒绝最低系统版本高于 macOS 12 的依赖,避免当前开发机可加载但低版本老用户失败。 + +覆盖升级保留旧聊天、Memory、设置、微信游标和授权边界;独立 Memory 2.1.6 通过内容构建标识替换旧运行时,保留升级锁与 schema 防降级。原生 helper 仍校验编译期版本和协议。应用不自动修改旧锁屏系统组件或登录策略。 + +Mac arm64 签名公证,Windows x64 未签名。源码测试与安装包内容核验另附交付证据;真实安装、旧用户覆盖升级、Windows ABI、权限、微信和 GUI 行为需设备验收。 diff --git a/.github/release-notes/v1.2.3-rc.6.md b/.github/release-notes/v1.2.3-rc.6.md new file mode 100644 index 000000000..debe6433c --- /dev/null +++ b/.github/release-notes/v1.2.3-rc.6.md @@ -0,0 +1,9 @@ +# Memmy v1.2.3-rc.6 + +以同事定版 1.2.2 为基线,保留 9 月 23 日以来已整合的产品改动,并按后续会话的最终决策纳入新增代码。浏览器扩展改为手动安装:只在点击对应浏览器的“安装”后展开教程;已安装、已连接和断连状态分别显示,撤回此前的自动安装方案。 + +补入 Mac 屏幕权限重启后的会话恢复、Windows 首个窗口显示后再启动后台服务、聊天返回时保留未查看的内嵌浏览器标签、Windows 文件预览对盘符和共享文件夹路径的处理,以及定时回复分隔线的最终样式。 + +History 增加受隐私边界约束的可读文本采集、Windows 微信空 WAL 和授权起点修复;摘要使用用户配置的文本模型、计量及预算,生成的 Skill 展示来源与未验证边界。保留旧数据、设置、授权及 Memory 2.1.6 升级路径,运行时以内容构建标识更新,不降低数据库 schema。 + +源码、静态包内容和签名检查不能替代真实安装及升级验收。Mac、Windows 老用户应在各自设备上用现有数据做覆盖升级,重点检查聊天和 Memory、History 与微信授权、浏览器扩展、Excel 授权、首次启动和文件预览。 diff --git a/.github/release-notes/v1.2.3-rc.7.md b/.github/release-notes/v1.2.3-rc.7.md new file mode 100644 index 000000000..f9346630d --- /dev/null +++ b/.github/release-notes/v1.2.3-rc.7.md @@ -0,0 +1,9 @@ +# Memmy v1.2.3-rc.7 + +以同事定版 1.2.2 为基线,继承 rc.6 已整合的微信与 History、Excel 工作簿授权、白屏恢复、初见报告防卡顿、浏览器与老用户迁移修复。本轮仅加入后续已完成且确认的代码。 + +- macOS 从 Memmy 内打开完全磁盘访问或自动化设置,以及 Computer Use 的辅助功能、输入监控和录屏授权引导后,系统发出可信的“退出并重新打开”事件时保留短时重启意图;真正注销、关机及普通退出仍按原保护处理。 +- Excel 连接状态短暂检查失败时显示温和提示,持续失败再提示重试,恢复后清除匹配操作留下的旧错误;授权拒绝、锁屏、多个 Excel 进程等明确问题仍直接给出可操作提示。Excel 读写逻辑和旧工作簿授权未改。 +- 定时回复分隔线采用用户最终确认的 20px 上下间距,保留 rc.6 的浅绿背景与 1px 内线。 + +继续保留旧账号、聊天、附件、设置、History 排除规则、微信授权与游标、Memory 2.1.6 及 Windows 数据目录迁移路径。源码与安装包静态检查不能代替真实安装和覆盖升级;Mac、Windows 老用户请在各自设备上保留原数据进行验收。 diff --git a/.github/release-notes/v1.2.3.md b/.github/release-notes/v1.2.3.md new file mode 100644 index 000000000..1ba9bada2 --- /dev/null +++ b/.github/release-notes/v1.2.3.md @@ -0,0 +1,15 @@ +# Memmy v1.2.3 + +以 v1.2.3-rc.7 的已验证内容为基础,发布稳定版 v1.2.3;保留 rc.7 已整合的微信与 History、Excel 工作簿授权、白屏恢复、初见报告防卡顿、浏览器与老用户迁移修复。 + +- macOS 从 Memmy 内打开完全磁盘访问或自动化设置,以及 Computer Use 的辅助功能、输入监控和录屏授权引导后,系统发出可信的“退出并重新打开”事件时保留短时重启意图;真正注销、关机及普通退出仍按原保护处理。 +- Excel 连接状态短暂检查失败时显示温和提示,持续失败再提示重试,恢复后清除匹配操作留下的旧错误;授权拒绝、锁屏、多个 Excel 进程等明确问题仍直接给出可操作提示。Excel 读写逻辑和旧工作簿授权未改。 +- 定时回复分隔线采用用户最终确认的 20px 上下间距,保留 rc.6 的浅绿背景与 1px 内线。 +- 内置浏览器与 Chrome/Edge 插件共用网站授权,电脑操控总授权可覆盖浏览器操作;网页执行补充连续操作、子页面、弹窗与导航的处理。修复内置浏览器输入焦点错位、富文本填写核验和后台页面操作;外部浏览器保留真人接管并等待子页面输入监听就绪,补齐后台浏览器唤醒、慢握手、休眠与重连恢复,浏览器插件更新至 1.0.12。 +- 电脑历史补充网页语义记录,并将电脑历史记忆归在 Memmy 下展示;修复 Windows History 全量内容采集与 PowerShell 5.1 图标读取,保留采集缺口、质量和操作顺序。网页记录继续遵守暂停、排除网站及敏感内容过滤。 +- 微信连接补充 macOS 应用定位、私有 Python 和密钥获取路径,以及 Windows 数据目录发现、多账号识别与完整消息分库校验;Windows 读取失败后清理明文临时副本。连接效果仍须在实际用户设备上确认。 +- Windows 安装器在目录检查和启动回调期间及时显示“检查中”“启动中”,并记录启动结果和耗时。 +- Windows History 优先提取原生高清图标,保留透明度与快捷方式回退;技能审批按钮沿用技能卡片样式,并保留 History 各设置区之间的分隔线。 +- 浏览器扩展设置页区分未安装、停用和状态无法读取;系统拒绝读取浏览器配置时保留未知状态,不再误报未安装。实时连接仍以实际在线状态为准。明确使用正在运行的 Chrome/Edge 时,配置无法读取不再阻止按需唤醒,随后等待真实扩展握手;停用、缺失及任务取消边界保留。 + +继续保留旧账号、聊天、附件、设置、History 排除规则、微信授权与游标、Memory 2.1.6 及 Windows 数据目录迁移路径。源码与安装包静态检查不能代替真实安装和覆盖升级;Mac、Windows 老用户请在各自设备上保留原数据进行验收。 diff --git a/.github/workflows/computer-use-native.yml b/.github/workflows/computer-use-native.yml index c1e223c59..e7597e324 100644 --- a/.github/workflows/computer-use-native.yml +++ b/.github/workflows/computer-use-native.yml @@ -6,10 +6,12 @@ on: - ".github/workflows/computer-use-native.yml" - "App/memmy-agent/package*.json" - "App/memmy-agent/src/tools/computer-use/**" + - "App/native-computer-use/**" - "App/shell/desktop/electron-builder*.yml" - "scripts/internal/shared/check-open-computer-use.mjs" - "scripts/internal/linux/**" - "scripts/internal/mac/**" + - "scripts/internal/win/build-memmy-computer-use.sh" - "tests/ocu-dev-install.test.mjs" - "scripts/internal/win/build-nsis.sh" workflow_dispatch: @@ -22,7 +24,7 @@ jobs: strategy: fail-fast: false matrix: - runner: [ubuntu-24.04, ubuntu-24.04-arm, macos-15, windows-2025] + runner: [ubuntu-24.04, ubuntu-24.04-arm, macos-26, windows-2025] runs-on: ${{ matrix.runner }} timeout-minutes: 10 defaults: @@ -35,20 +37,38 @@ jobs: - uses: actions/setup-node@v4 with: node-version: 22 - - name: Install the pinned native package into an isolated directory + - uses: actions/setup-go@v5 + if: runner.os == 'Windows' || runner.os == 'Linux' + with: + go-version: '1.22' + - name: Build Memmy Computer Use for Linux + if: runner.os == 'Linux' run: | - version="$(node -p "require('./App/memmy-agent/package.json').dependencies['open-computer-use']")" - npm install --prefix "$RUNNER_TEMP/ocu-native" --ignore-scripts --no-audit --no-fund "open-computer-use@$version" - node --input-type=module <<'NODE' - import fs from 'node:fs'; - import path from 'node:path'; - const root = path.join(process.env.RUNNER_TEMP, 'ocu-native/node_modules/open-computer-use'); - const arch = process.arch === 'x64' ? 'amd64' : process.arch; - const relative = process.platform === 'darwin' - ? 'dist/Open Computer Use.app/Contents/MacOS/OpenComputerUse' - : `dist/${process.platform === 'win32' ? 'windows' : 'linux'}/${arch}/open-computer-use${process.platform === 'win32' ? '.exe' : ''}`; - fs.appendFileSync(process.env.GITHUB_ENV, `OCU_BINARY=${path.join(root, relative)}\n`); - NODE + (cd App/native-computer-use/apps/OpenComputerUseLinux && go test ./...) + bash scripts/internal/linux/build-memmy-computer-use.sh "$RUNNER_TEMP/native-computer-use/linux" + arch="$(node -p "process.arch === 'x64' ? 'amd64' : process.arch")" + echo "OCU_BINARY=$RUNNER_TEMP/native-computer-use/linux/$arch/memmy-computer-use" >> "$GITHUB_ENV" + - name: Build Memmy Computer Use for Windows + if: runner.os == 'Windows' + run: | + (cd App/native-computer-use/apps/OpenComputerUseWindows && go test ./...) + bash scripts/internal/win/build-memmy-computer-use.sh "$RUNNER_TEMP/memmy-computer-use.exe" amd64 + echo "OCU_BINARY=$RUNNER_TEMP/memmy-computer-use.exe" >> "$GITHUB_ENV" + - name: Check Windows runtime fixtures with native PowerShell 5.1 + if: runner.os == 'Windows' + shell: powershell + run: | + if ($PSVersionTable.PSVersion.Major -ne 5) { throw 'These fixtures must use Windows PowerShell 5.1.' } + & ./App/native-computer-use/apps/OpenComputerUseWindows/runtime-safety.tests.ps1 | Tee-Object -FilePath ocu-windows-fixtures.json + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + - name: Build Memmy Computer Use for macOS + if: runner.os == 'macOS' + run: | + bash scripts/internal/mac/build-memmy-computer-use.sh "$RUNNER_TEMP/Memmy Computer Use.app" "$(uname -m)" - cn.memtensor.memmy + echo "OCU_BINARY=$RUNNER_TEMP/Memmy Computer Use.app/Contents/MacOS/MemmyComputerUse" >> "$GITHUB_ENV" + - name: Test Memmy macOS native source + if: runner.os == 'macOS' + run: swift test --package-path App/native-computer-use -c release - name: Prepare an isolated Linux desktop session if: runner.os == 'Linux' run: | @@ -62,11 +82,11 @@ jobs: - name: Check Windows MCP and desktop backend if: runner.os == 'Windows' run: node scripts/internal/shared/check-open-computer-use.mjs "$OCU_BINARY" --list-apps | tee ocu-native-result.json - - name: Validate the published macOS bundle and installer + - name: Validate the Memmy macOS bundle and installer if: runner.os == 'macOS' run: | - codesign --verify --deep --strict "$RUNNER_TEMP/ocu-native/node_modules/open-computer-use/dist/Open Computer Use.app" - lipo "$OCU_BINARY" -verify_arch arm64 x86_64 + codesign --verify --deep --strict "$RUNNER_TEMP/Memmy Computer Use.app" + lipo "$OCU_BINARY" -verify_arch "$(uname -m)" node --test tests/ocu-dev-install.test.mjs - name: Check macOS MCP discovery if: runner.os == 'macOS' @@ -75,7 +95,9 @@ jobs: if: always() with: name: ocu-native-${{ matrix.runner }} - path: ocu-native-result.json + path: | + ocu-native-result.json + ocu-windows-fixtures.json if-no-files-found: warn - name: Record validation boundary if: always() diff --git a/.github/workflows/linux-cli-installer.yml b/.github/workflows/linux-cli-installer.yml index 0232031c2..6d9ed48cb 100644 --- a/.github/workflows/linux-cli-installer.yml +++ b/.github/workflows/linux-cli-installer.yml @@ -9,11 +9,11 @@ on: - "scripts/install.sh" - "scripts/internal/linux/**" - "App/memmy-agent/**" + - "App/native-computer-use/**" - "Memory/**" - "Migrations/**" - "App/backend/**" - "tests/linux-cli-packaging.test.mjs" - - "tests/linux-open-computer-use-bundle.test.mjs" - "tests/linux-computer-use-deps.test.mjs" - "scripts/internal/shared/check-open-computer-use.mjs" - "package.json" @@ -25,11 +25,11 @@ on: - "scripts/install.sh" - "scripts/internal/linux/**" - "App/memmy-agent/**" + - "App/native-computer-use/**" - "Memory/**" - "Migrations/**" - "App/backend/**" - "tests/linux-cli-packaging.test.mjs" - - "tests/linux-open-computer-use-bundle.test.mjs" - "tests/linux-computer-use-deps.test.mjs" - "scripts/internal/shared/check-open-computer-use.mjs" - "package.json" @@ -63,6 +63,9 @@ jobs: with: node-version: 22 cache: npm + - uses: actions/setup-go@v5 + with: + go-version: '1.22' - run: npm ci - run: npm ci --prefix App/memmy-agent - name: Resolve and verify version @@ -143,7 +146,7 @@ jobs: ocu_arch="$EXPECTED_ARCH" [[ "$ocu_arch" != "x64" ]] || ocu_arch="amd64" node scripts/internal/shared/check-open-computer-use.mjs \ - "$MEMMY_INSTALL_ROOT/current/App/memmy-agent/node_modules/open-computer-use/dist/linux/$ocu_arch/open-computer-use" + "$MEMMY_INSTALL_ROOT/current/App/memmy-agent/dist/native-computer-use/linux/$ocu_arch/memmy-computer-use" "$MEMMY_BIN_DIR/memmy" --version "$MEMMY_BIN_DIR/memmy" --help >/dev/null "$MEMMY_BIN_DIR/memmy-memory" health >/dev/null diff --git a/.github/workflows/memory-release.yml b/.github/workflows/memory-release.yml index 71c26a54e..91dde3a24 100644 --- a/.github/workflows/memory-release.yml +++ b/.github/workflows/memory-release.yml @@ -6,7 +6,7 @@ on: version: description: Memory version (X.Y.Z) required: true - default: 2.1.4 + default: 2.1.6 push: tags: - "memory-v*" diff --git a/.github/workflows/windows-computer-history-validation.yml b/.github/workflows/windows-computer-history-validation.yml new file mode 100644 index 000000000..724d19fc8 --- /dev/null +++ b/.github/workflows/windows-computer-history-validation.yml @@ -0,0 +1,164 @@ +name: Windows 11 Computer History validation + +on: + pull_request: + paths: + - ".github/workflows/windows-computer-history-validation.yml" + - "App/native-computer-history/**" + - "App/memmy-agent/src/tools/computer-history/**" + - "App/memmy-agent/tests/tools/computer-history/**" + - "App/memmy-agent/scripts/copy-build-assets.mjs" + - "App/shell/desktop/electron-builder.win*.yml" + - "scripts/internal/win/build-nsis.sh" + - "scripts/internal/win/*computer-history*" + - "App/shell/desktop/tests/packaged-runtime-boundary.test.ts" + - "tests/computer-history-packaging.test.mjs" + - "package.json" + - "package-lock.json" + workflow_dispatch: + +permissions: + contents: read + +jobs: + validate: + runs-on: windows-2025 + timeout-minutes: 30 + defaults: + run: + shell: pwsh + steps: + - uses: actions/checkout@v4 + with: + persist-credentials: false + - uses: actions/setup-node@v4 + with: + node-version: 22 + architecture: x64 + cache: npm + - uses: actions/setup-go@v5 + with: + go-version: "1.22.12" + cache-dependency-path: App/native-computer-history/apps/OpenComputerHistoryWindows/go.mod + - name: Require Windows 11 + run: | + $ErrorActionPreference = 'Stop' + $build = [Environment]::OSVersion.Version.Build + if ($build -lt 22000) { throw "Windows 11 build 22000 or later required; found $build" } + Write-Host "Windows build=$build; session=$((Get-Process -Id $PID).SessionId)" + - name: Install workspace dependencies + run: npm ci + - name: Install standalone agent dependencies + run: npm ci --prefix App/memmy-agent + - name: Validate native protocol and lifecycle without desktop access + run: | + $ErrorActionPreference = 'Stop' + Push-Location App/native-computer-history/apps/OpenComputerHistoryWindows + try { + go test ./... + if ($LASTEXITCODE -ne 0) { throw 'Native recorder pure tests failed' } + # Win32 hook LPARAM is a native record pointer carried as uintptr; + # unsafeptr cannot infer that callback ABI contract. Keep every other analyzer. + go vet -unsafeptr=false ./... + if ($LASTEXITCODE -ne 0) { throw 'Native recorder vet failed' } + } finally { Pop-Location } + - name: Build agent and native History recorder from source + run: npm --prefix App/memmy-agent run build + - name: Validate packaged app icons with Windows PowerShell 5.1 + run: | + $ErrorActionPreference = 'Stop' + $source = 'App/memmy-agent/src/tools/computer-history/win/app-icon.ps1' + $staged = 'App/memmy-agent/dist/tools/computer-history/win/app-icon.ps1' + if (-not (Test-Path -LiteralPath $staged)) { throw 'App icon helper was not copied into the distributable agent' } + if ((Get-FileHash $source).Hash -ne (Get-FileHash $staged).Hash) { throw 'Packaged app icon helper differs from source' } + $powershell = "$env:SystemRoot\System32\WindowsPowerShell\v1.0\powershell.exe" + foreach ($count in @(1, 2, 48)) { + $ids = [System.Collections.Generic.List[string]]::new() + $ids.Add('win32.powershell') + for ($index = 1; $index -lt $count; $index++) { $ids.Add("win32.iconprobe$index") } + $idArray = $ids.ToArray() + $json = ConvertTo-Json -InputObject $idArray -Compress + $encoded = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($json)) + $raw = & $powershell -NoProfile -NonInteractive -STA -ExecutionPolicy Bypass -File $staged -IdsBase64 $encoded + if ($LASTEXITCODE -ne 0) { throw "App icon helper failed for $count IDs with exit code $LASTEXITCODE" } + $icons = $raw | ConvertFrom-Json -AsHashtable + if ($icons.Count -ne $count) { throw "App icon helper returned $($icons.Count) entries for $count IDs" } + foreach ($id in $ids) { if ($icons.Keys -notcontains $id) { throw "App icon helper omitted $id" } } + $png = [Convert]::FromBase64String($icons['win32.powershell']) + if ($png.Length -lt 24) { throw 'App icon helper returned a truncated image' } + $signature = -join ($png[0..7] | ForEach-Object { $_.ToString('x2') }) + $dimensions = -join ($png[16..23] | ForEach-Object { $_.ToString('x2') }) + if ($signature -ne '89504e470d0a1a0a' -or $dimensions -ne '0000004000000040') { + throw "App icon helper did not return a 64x64 PNG for its own PowerShell process ($count IDs)" + } + } + - name: Validate native recorder identity, pure self-test and installed app catalog + run: | + $ErrorActionPreference = 'Stop' + $source = 'App/memmy-agent/src/tools/computer-history/win/win11-observer.ps1' + $staged = 'App/memmy-agent/dist/tools/computer-history/win/win11-observer.ps1' + if (-not (Test-Path -LiteralPath $staged)) { throw 'Observer was not copied into the distributable agent' } + if ((Get-FileHash $source).Hash -ne (Get-FileHash $staged).Hash) { throw 'Packaged observer differs from source' } + $powershell = "$env:SystemRoot\System32\WindowsPowerShell\v1.0\powershell.exe" + $native = 'App/memmy-agent/dist/native-computer-history/windows/amd64/memmy-computer-history.exe' + node scripts/internal/win/verify-computer-history-runtime.mjs $native x64 App/native-computer-history/apps/OpenComputerHistoryWindows/worker.ps1 + if ($LASTEXITCODE -ne 0) { throw 'Native recorder PE, identity or embedded source verification failed' } + $probe = & $native --self-test | ConvertFrom-Json + if ($LASTEXITCODE -ne 0 -or $probe.runtimeId -ne 'MemmyWindowsHistoryRuntimeV1' -or -not $probe.pureSelfTest -or $probe.desktopAccess -ne $false) { throw 'Pure native recorder self-test failed' } + # The native recorder writes its embedded UTF-8 worker with a BOM before + # launching Windows PowerShell 5.1. Exercise those exact on-disk bytes. + $workerBytes = [IO.File]::ReadAllBytes('App/native-computer-history/apps/OpenComputerHistoryWindows/worker.ps1') + $workerForTest = Join-Path $env:RUNNER_TEMP 'memmy-history-worker-self-test.ps1' + $materializedWorker = [byte[]]::new($workerBytes.Length + 3) + $materializedWorker[0] = 0xef; $materializedWorker[1] = 0xbb; $materializedWorker[2] = 0xbf + [Array]::Copy($workerBytes, 0, $materializedWorker, 3, $workerBytes.Length) + [IO.File]::WriteAllBytes($workerForTest, $materializedWorker) + $workerProbe = & $powershell -NoProfile -NonInteractive -MTA -ExecutionPolicy Bypass -File $workerForTest -SelfTest | ConvertFrom-Json + if ($LASTEXITCODE -ne 0 -or -not $workerProbe.workerApiCompiled -or -not $workerProbe.pureSelfTest -or $workerProbe.desktopAccess -ne $false) { throw 'Pure worker API compile and self-test failed' } + $apps = & $powershell -NoProfile -NonInteractive -ExecutionPolicy Bypass -File $staged -List | ConvertFrom-Json + if ($LASTEXITCODE -ne 0 -or @($apps).Count -lt 1) { throw 'Windows app catalog is unavailable' } + foreach ($app in @($apps)) { + if ($app.bundleId -notmatch '^win32\.[a-z0-9._-]+$' -or [string]::IsNullOrWhiteSpace($app.name)) { + throw "Invalid app catalog row: $($app | ConvertTo-Json -Compress)" + } + } + Write-Host "Validated $(@($apps).Count) installed or running app entries" + - name: Verify agent History platform contracts + run: | + $ErrorActionPreference = 'Stop' + $tests = @( + 'App/memmy-agent/tests/tools/computer-history/win/application-icon.test.ts' + 'App/memmy-agent/tests/tools/computer-history/win/win11.test.ts' + 'App/memmy-agent/tests/tools/computer-history/platform.test.ts' + 'App/memmy-agent/tests/tools/computer-history/mac/summary-writer.test.ts' + 'App/memmy-agent/tests/tools/computer-history/browser-privacy.test.ts' + 'App/memmy-agent/tests/tools/computer-history/browser-semantic-history.test.ts' + 'App/memmy-agent/tests/tools/computer-history/browser-history-bridge.test.ts' + 'App/memmy-agent/tests/tools/computer-history/win/native-event-adapter.test.ts' + 'App/memmy-agent/tests/tools/computer-history/win/native-recording-privacy.test.ts' + 'App/memmy-agent/tests/tools/computer-history/win/browser-authority-publisher.test.ts' + ) + foreach ($test in $tests) { + if (-not (Test-Path -LiteralPath $test -PathType Leaf)) { throw "Missing History test: $test" } + } + node node_modules/vitest/vitest.mjs run @tests + if ($LASTEXITCODE -ne 0) { throw 'History platform contracts failed' } + # On Windows, Vitest's Vite runner raises a collection-time SyntaxError for + # this valid ESM test; the Node ESM runner executes the same assertions. + - name: Verify packaged History assets + run: >- + node node_modules/vitest/vitest.mjs run + tests/computer-history-packaging.test.mjs + --experimental.viteModuleRunner=false + - name: Verify installer enforces the unpacked History observer + run: >- + node node_modules/vitest/vitest.mjs run + App/shell/desktop/tests/packaged-runtime-boundary.test.ts + -t "requires the native Windows 11 History recorder" + - name: Record validation scope + if: ${{ always() }} + run: | + @' + Validates source-built Windows History recorder identity, PE architecture, exact embedded worker source, pure recorder state/lifecycle self-tests, UTF-8 app catalog output, runtime asset copy, and packaging rules. This does not start live recording or prove real UI Automation/hook behavior. + A signed NSIS installer and a representative real-app capture matrix still require installation testing on a Windows 11 machine. + '@ | Add-Content -Path $env:GITHUB_STEP_SUMMARY diff --git a/AgentSourceCore/src/memory-token-budget.ts b/AgentSourceCore/src/memory-token-budget.ts index 241eb7cc0..7524c4e8f 100644 --- a/AgentSourceCore/src/memory-token-budget.ts +++ b/AgentSourceCore/src/memory-token-budget.ts @@ -8,6 +8,7 @@ export const MEMORY_WORKER_JOB_TYPES = [ "episode_idle_close", "episode_title", "trace_summary", + "observation_extract", "user_memory_embedding", "import_summary", "reflection", @@ -85,6 +86,7 @@ export function jobTypeBudgetRoles(jobType: string): MemoryBudgetModelRole[] { return []; case "episode_title": case "trace_summary": + case "observation_extract": case "import_summary": case "span_big_turn": case "work_memory_extract": diff --git a/AgentSourceCore/src/source-turn.ts b/AgentSourceCore/src/source-turn.ts index 01e67a0cf..a87eb3c1b 100644 --- a/AgentSourceCore/src/source-turn.ts +++ b/AgentSourceCore/src/source-turn.ts @@ -78,6 +78,27 @@ export function sourceTurnSkipBlocksWatermark(reason: string): boolean { return reason !== "turn_cancelled"; } +/** How long a skipped native turn may still finish and keep the source watermark behind it. */ +export const SOURCE_TURN_RETRY_WINDOW_MS = 24 * 60 * 60 * 1000; + +const CAPTURE_UNAVAILABLE_REASONS = new Set(["memory_add_disabled", "source_capture_activation_unresolved"]); + +/** + * `rescan`: Memory could not accept captures, so the whole source must be retried. + * `retry`: a recent turn may still finish; hold the watermark at its latest activity. + * `none`: a cancelled or long-abandoned turn must not force every later scan to start over. + */ +export function sourceTurnSkipWatermarkEffect( + reason: string, + latestActivityAt: string | null, + now = Date.now() +): "rescan" | "retry" | "none" { + if (!sourceTurnSkipBlocksWatermark(reason)) return "none"; + if (CAPTURE_UNAVAILABLE_REASONS.has(reason)) return "rescan"; + const activityAt = latestActivityAt ? Date.parse(latestActivityAt) : Number.NaN; + return Number.isFinite(activityAt) && now - activityAt <= SOURCE_TURN_RETRY_WINDOW_MS ? "retry" : "none"; +} + /** * A reader that stages native turns marks every message with its turn state, so the * scan can tell an unfinished native turn apart from a source that has no native diff --git a/AgentSourceCore/src/tests/memory-token-budget.test.ts b/AgentSourceCore/src/tests/memory-token-budget.test.ts index 5855b05bf..978463a59 100644 --- a/AgentSourceCore/src/tests/memory-token-budget.test.ts +++ b/AgentSourceCore/src/tests/memory-token-budget.test.ts @@ -28,6 +28,7 @@ describe("memory token budget rules", () => { it("pauses budgeted jobs and leaves idle close runnable", () => { expect(jobTypeConsumesMemoryBudget("trace_summary")).toBe(true); + expect(jobTypeConsumesMemoryBudget("observation_extract")).toBe(true); expect(jobTypeConsumesMemoryBudget("episode_title")).toBe(true); expect(jobTypeConsumesMemoryBudget("embedding")).toBe(true); expect(jobTypeConsumesMemoryBudget("new_future_job")).toBe(true); @@ -57,11 +58,13 @@ describe("memory token budget rules", () => { expect(jobTypeConsumesMemoryBudget("reflection", account)).toBe(false); expect(jobTypeConsumesMemoryBudget("trace_summary", account)).toBe(false); + expect(jobTypeConsumesMemoryBudget("observation_extract", account)).toBe(false); expect(jobTypeConsumesMemoryBudget("embedding", account)).toBe(false); expect(jobTypeConsumesMemoryBudget("episode_title", mixed)).toBe(true); expect(jobTypeConsumesMemoryBudget("span_big_turn", mixed)).toBe(true); expect(jobTypeConsumesMemoryBudget("work_memory_extract", mixed)).toBe(true); + expect(jobTypeConsumesMemoryBudget("observation_extract", mixed)).toBe(true); expect(jobTypeConsumesMemoryBudget("reflection", mixed)).toBe(true); expect(jobTypeConsumesMemoryBudget("reward", mixed)).toBe(true); expect(jobTypeConsumesMemoryBudget("embedding", mixed)).toBe(false); @@ -69,6 +72,7 @@ describe("memory token budget rules", () => { expect(jobTypeConsumesMemoryBudget("l2_association", mixed)).toBe(false); expect(jobTypeBudgetRoles("span_big_turn")).toEqual(["memory_summary"]); expect(jobTypeBudgetRoles("work_memory_extract")).toEqual(["memory_summary"]); + expect(jobTypeBudgetRoles("observation_extract")).toEqual(["memory_summary"]); expect(jobTypeBudgetRoles("reflection")).toEqual(["memory_summary", "memory_evolution"]); expect(jobTypeBudgetRoles("reward")).toEqual(["memory_summary", "memory_evolution"]); }); diff --git a/AgentSourceCore/src/tests/source-turn-review.test.ts b/AgentSourceCore/src/tests/source-turn-review.test.ts index 3b1371773..8f3201b91 100644 --- a/AgentSourceCore/src/tests/source-turn-review.test.ts +++ b/AgentSourceCore/src/tests/source-turn-review.test.ts @@ -5,6 +5,7 @@ import { readOpenclawSourceTurn, readOpencodeSourceTurn, sourceTurnSkipBlocksWatermark, + sourceTurnSkipWatermarkEffect, type CursorVscdbSource, type OpenclawTranscriptSource, type OpencodeSource, @@ -21,6 +22,18 @@ describe("review regressions for native source turns", () => { expect(sourceTurnSkipBlocksWatermark("source_turn_content_conflict")).toBe(true); }); + it("retries only recent unfinished turns and rescans only when Memory refused captures", () => { + const now = Date.parse("2026-10-10T12:00:00.000Z"); + expect(sourceTurnSkipWatermarkEffect("turn_incomplete", "2026-10-10T11:00:00.000Z", now)).toBe("retry"); + expect(sourceTurnSkipWatermarkEffect("source_turn_content_conflict", "2026-10-10T11:00:00.000Z", now)).toBe("retry"); + expect(sourceTurnSkipWatermarkEffect("turn_incomplete", "2026-10-09T11:59:59.000Z", now)).toBe("none"); + expect(sourceTurnSkipWatermarkEffect("identity_unresolved", "2026-08-18T03:00:00.000Z", now)).toBe("none"); + expect(sourceTurnSkipWatermarkEffect("turn_incomplete", null, now)).toBe("none"); + expect(sourceTurnSkipWatermarkEffect("turn_cancelled", "2026-10-10T11:00:00.000Z", now)).toBe("none"); + expect(sourceTurnSkipWatermarkEffect("memory_add_disabled", "2026-08-18T03:00:00.000Z", now)).toBe("rescan"); + expect(sourceTurnSkipWatermarkEffect("source_capture_activation_unresolved", null, now)).toBe("rescan"); + }); + it("uses the disk profile for both hook and scan requests", () => { const turn: SourceTurn = { source: "opencode", diff --git a/App/backend/local-api-contracts/src/browser-access-approval.ts b/App/backend/local-api-contracts/src/browser-access-approval.ts new file mode 100644 index 000000000..4f85bf4fb --- /dev/null +++ b/App/backend/local-api-contracts/src/browser-access-approval.ts @@ -0,0 +1,31 @@ +export const BROWSER_ACCESS_REQUEST = 'memmy:browser-access:request'; +export const BROWSER_ACCESS_RESULT = 'memmy:browser-access:result'; + +export type BrowserAccessRequest = { type: typeof BROWSER_ACCESS_REQUEST; requestId: string; origin: string; url: string }; +export type BrowserAccessResult = { type: typeof BROWSER_ACCESS_RESULT; requestId: string; decision: 'allow-once' | 'allow-always' | 'deny' }; + +function validOrigin(value: unknown): value is string { + if (typeof value !== 'string' || value.length > 320) return false; + try { + const parsed = new URL(value); + return ['http:', 'https:'].includes(parsed.protocol) && parsed.origin === value; + } catch { return false; } +} +export function isBrowserAccessRequest(value: unknown): value is BrowserAccessRequest { + if (!value || typeof value !== 'object' || Array.isArray(value)) return false; + const request = value as Record; + if (request.type !== BROWSER_ACCESS_REQUEST || typeof request.requestId !== 'string' + || !/^[a-f\d-]{36}$/i.test(request.requestId) || !validOrigin(request.origin) + || typeof request.url !== 'string' || request.url.length > 4096) return false; + try { if (new URL(request.url).origin !== request.origin) return false; } + catch { return false; } + return Object.keys(request).every(key => ['type', 'requestId', 'origin', 'url'].includes(key)); +} +export function isBrowserAccessResult(value: unknown): value is BrowserAccessResult { + if (!value || typeof value !== 'object' || Array.isArray(value)) return false; + const result = value as Record; + return result.type === BROWSER_ACCESS_RESULT && typeof result.requestId === 'string' + && /^[a-f\d-]{36}$/i.test(result.requestId) + && ['allow-once', 'allow-always', 'deny'].includes(String(result.decision)) + && Object.keys(result).every(key => ['type', 'requestId', 'decision'].includes(key)); +} diff --git a/App/backend/local-api-contracts/src/browser-capability-approval.ts b/App/backend/local-api-contracts/src/browser-capability-approval.ts new file mode 100644 index 000000000..f4b4cb086 --- /dev/null +++ b/App/backend/local-api-contracts/src/browser-capability-approval.ts @@ -0,0 +1,43 @@ +export const BROWSER_CAPABILITY_REQUEST = 'memmy:browser-capability:request'; +export const BROWSER_CAPABILITY_RESULT = 'memmy:browser-capability:result'; + +export type BrowserCapabilityKind = 'download' | 'upload' | 'debug' | 'debug-write'; +export type BrowserCapabilityRequest = { + type: typeof BROWSER_CAPABILITY_REQUEST; + requestId: string; + capability: BrowserCapabilityKind; + origin: string; + names: string[]; +}; +export type BrowserCapabilityResult = { + type: typeof BROWSER_CAPABILITY_RESULT; + requestId: string; + approved: boolean; +}; + +const requestIdPattern = /^[a-f\d-]{36}$/i; +export function isBrowserCapabilityRequest(value: unknown): value is BrowserCapabilityRequest { + if (!value || typeof value !== 'object' || Array.isArray(value)) return false; + const request = value as Record; + if (request.type !== BROWSER_CAPABILITY_REQUEST || typeof request.requestId !== 'string' + || !requestIdPattern.test(request.requestId) + || (request.capability !== 'download' && request.capability !== 'upload' + && request.capability !== 'debug' && request.capability !== 'debug-write') + || typeof request.origin !== 'string' || request.origin.length > 320 + || !Array.isArray(request.names) || request.names.length < 1 || request.names.length > 20 + || request.names.some(name => typeof name !== 'string' || !name || name.length > 120 + || name === '.' || name === '..' || /[\\/\x00-\x1f]/.test(name))) return false; + try { + const url = new URL(request.origin); + if (!['http:', 'https:'].includes(url.protocol) || url.origin !== request.origin) return false; + } catch { return false; } + return Object.keys(request).every(key => ['type', 'requestId', 'capability', 'origin', 'names'].includes(key)); +} + +export function isBrowserCapabilityResult(value: unknown): value is BrowserCapabilityResult { + if (!value || typeof value !== 'object' || Array.isArray(value)) return false; + const result = value as Record; + return result.type === BROWSER_CAPABILITY_RESULT && typeof result.requestId === 'string' + && requestIdPattern.test(result.requestId) && typeof result.approved === 'boolean' + && Object.keys(result).every(key => ['type', 'requestId', 'approved'].includes(key)); +} diff --git a/App/backend/local-api-contracts/src/browser-cdp-readonly.test.ts b/App/backend/local-api-contracts/src/browser-cdp-readonly.test.ts new file mode 100644 index 000000000..6a1a1eec6 --- /dev/null +++ b/App/backend/local-api-contracts/src/browser-cdp-readonly.test.ts @@ -0,0 +1,30 @@ +import { describe, expect, it } from 'vitest'; +import { normalizeBrowserCdpCommandOptions, normalizeBrowserCdpEventQuery, + normalizeBrowserCdpReadCall } from './browser-cdp-readonly.js'; + +describe('controlled browser CDP contract', () => { + it('restricts commands to bounded reads and forces side-effect checks', () => { + expect(normalizeBrowserCdpReadCall('Runtime.evaluate', { expression: 'document.title' })) + .toEqual({ method: 'Runtime.evaluate', params: { expression: 'document.title', + throwOnSideEffect: true, returnByValue: true, awaitPromise: false, userGesture: false } }); + expect(() => normalizeBrowserCdpReadCall('DOM.setFileInputFiles', { files: ['/private/file'] })) + .toThrow(/unavailable/); + expect(() => normalizeBrowserCdpReadCall('Runtime.evaluate', { + expression: 'document.title', throwOnSideEffect: false, + })).toThrow(/unavailable/); + expect(() => normalizeBrowserCdpReadCall('DOM.getDocument', { depth: -1 })).toThrow(/unavailable/); + }); + + it('bounds event cursor, page size and filters', () => { + expect(normalizeBrowserCdpEventQuery({ afterSequence: 3, methods: ['Network.requestWillBeSent'] })) + .toEqual({ afterSequence: 3, limit: 100, methods: ['Network.requestWillBeSent'] }); + expect(() => normalizeBrowserCdpEventQuery({ limit: 1001 })).toThrow(/Invalid/); + expect(normalizeBrowserCdpEventQuery({ target: { targetId: 'child-1' }, timeoutMs: 30000 })) + .toEqual({ limit: 100, target: { targetId: 'child-1' }, timeoutMs: 30000 }); + expect(normalizeBrowserCdpCommandOptions({ target: { sessionId: 'session-1' } })) + .toEqual({ target: { sessionId: 'session-1' } }); + expect(() => normalizeBrowserCdpEventQuery({ target: { targetId: 'a', sessionId: 'b' } })) + .toThrow(/Invalid/); + expect(() => normalizeBrowserCdpEventQuery({ timeoutMs: 30001 })).toThrow(/timeout/); + }); +}); diff --git a/App/backend/local-api-contracts/src/browser-cdp-readonly.ts b/App/backend/local-api-contracts/src/browser-cdp-readonly.ts new file mode 100644 index 000000000..4ea4c6d08 --- /dev/null +++ b/App/backend/local-api-contracts/src/browser-cdp-readonly.ts @@ -0,0 +1,109 @@ +/** Commands that observe the current page without changing page or browser state. */ +export const BROWSER_CDP_READ_METHODS = [ + 'Accessibility.getFullAXTree', 'DOM.getDocument', 'Page.getLayoutMetrics', + 'Page.captureScreenshot', 'Runtime.evaluate', +] as const; +export type BrowserCdpReadMethod = typeof BROWSER_CDP_READ_METHODS[number]; + +export type BrowserCdpTarget = { sessionId: string } | { targetId: string }; + +/** Internal approval snapshot for an attached iframe within the selected tab. */ +export type BrowserCdpTargetApproval = { + sessionId: string; + targetId: string; + url: string; + origin: string; + revision: number; +}; + +export function parseBrowserCdpTargetApproval(value: unknown): BrowserCdpTargetApproval | null { + if (!value || typeof value !== 'object' || Array.isArray(value)) return null; + const input = value as Record; + if (typeof input.sessionId !== 'string' || !/^[A-Za-z0-9._:-]{1,256}$/.test(input.sessionId) + || typeof input.targetId !== 'string' || !/^[A-Za-z0-9._:-]{1,256}$/.test(input.targetId) + || typeof input.url !== 'string' || input.url.length > 4096 + || typeof input.origin !== 'string' || !Number.isSafeInteger(input.revision) || (input.revision as number) < 0) return null; + try { + const url = new URL(input.url); + if (!['http:', 'https:'].includes(url.protocol) || url.origin !== input.origin) return null; + } catch { return null; } + return { sessionId: input.sessionId, targetId: input.targetId, url: input.url, + origin: input.origin, revision: input.revision as number }; +} + +export function normalizeBrowserCdpTarget(input: unknown): BrowserCdpTarget | undefined { + if (input === undefined) return undefined; + if (!input || typeof input !== 'object' || Array.isArray(input)) throw new Error('Invalid CDP target'); + const target = input as Record; + if (Object.keys(target).length !== 1) throw new Error('Invalid CDP target'); + const key = Object.keys(target)[0]; + if ((key !== 'sessionId' && key !== 'targetId') || typeof target[key] !== 'string' + || !/^[A-Za-z0-9._:-]{1,256}$/.test(target[key] as string)) throw new Error('Invalid CDP target'); + return { [key]: target[key] } as BrowserCdpTarget; +} + +export function normalizeBrowserCdpCommandOptions(input: unknown): { + target?: BrowserCdpTarget; timeoutMs?: number } { + if (!input || typeof input !== 'object' || Array.isArray(input)) throw new Error('Invalid CDP options'); + const options = input as Record; + const target = normalizeBrowserCdpTarget(options.target); + const timeoutMs = options.timeoutMs; + if (timeoutMs !== undefined && (!Number.isInteger(timeoutMs) + || (timeoutMs as number) < 0 || (timeoutMs as number) > 30_000)) throw new Error('Invalid CDP timeout'); + return { ...(target ? { target } : {}), ...(timeoutMs === undefined ? {} : { timeoutMs: timeoutMs as number }) }; +} + +export function normalizeBrowserCdpReadCall(method: unknown, input: unknown): { + method: BrowserCdpReadMethod; params: Record } { + if (typeof method !== 'string' || !BROWSER_CDP_READ_METHODS.includes(method as BrowserCdpReadMethod) + || !input || typeof input !== 'object' || Array.isArray(input) + || JSON.stringify(input).length > 24_000) throw new Error('CDP method or parameters are unavailable'); + const params = input as Record; + const noParams = () => { if (Object.keys(params).length) throw new Error('CDP method parameters are unavailable'); }; + switch (method) { + case 'Accessibility.getFullAXTree': + case 'Page.getLayoutMetrics': noParams(); return { method, params: {} }; + case 'DOM.getDocument': { + if (Object.keys(params).some(key => !['depth'].includes(key)) + || (params.depth !== undefined && (!Number.isInteger(params.depth) + || (params.depth as number) < 0 || (params.depth as number) > 3))) { + throw new Error('CDP DOM depth is unavailable'); + } + return { method, params: { depth: params.depth ?? 1, pierce: false } }; + } + case 'Page.captureScreenshot': { + if (Object.keys(params).some(key => key !== 'format') + || (params.format !== undefined && params.format !== 'png' && params.format !== 'jpeg')) { + throw new Error('CDP screenshot parameters are unavailable'); + } + return { method, params: { format: params.format ?? 'png', captureBeyondViewport: false } }; + } + case 'Runtime.evaluate': { + if (Object.keys(params).some(key => key !== 'expression') + || typeof params.expression !== 'string' || !params.expression + || params.expression.length > 20_000) throw new Error('CDP expression is unavailable'); + return { method, params: { expression: params.expression, + throwOnSideEffect: true, returnByValue: true, awaitPromise: false, userGesture: false } }; + } + default: throw new Error('CDP method is unavailable'); + } +} + +export function normalizeBrowserCdpEventQuery(input: unknown): { + afterSequence?: number; limit: number; methods?: string[]; + target?: BrowserCdpTarget; timeoutMs?: number } { + if (!input || typeof input !== 'object' || Array.isArray(input)) throw new Error('Invalid CDP event query'); + const query = input as Record; + if (Object.keys(query).some(key => !['afterSequence', 'limit', 'methods', 'target', 'timeoutMs'].includes(key)) + || (query.afterSequence !== undefined && (!Number.isSafeInteger(query.afterSequence) + || (query.afterSequence as number) < 0)) + || (query.limit !== undefined && (!Number.isInteger(query.limit) + || (query.limit as number) < 1 || (query.limit as number) > 1000)) + || (query.methods !== undefined && (!Array.isArray(query.methods) || query.methods.length < 1 + || query.methods.length > 20 || query.methods.some(method => typeof method !== 'string' + || !/^[A-Za-z]+\.[A-Za-z]+$/.test(method))))) throw new Error('Invalid CDP event query'); + const options = normalizeBrowserCdpCommandOptions(query); + return { ...(query.afterSequence === undefined ? {} : { afterSequence: query.afterSequence as number }), + limit: (query.limit as number | undefined) ?? 100, + ...(query.methods === undefined ? {} : { methods: query.methods as string[] }), ...options }; +} diff --git a/App/backend/local-api-contracts/src/browser-cdp-write.test.ts b/App/backend/local-api-contracts/src/browser-cdp-write.test.ts new file mode 100644 index 000000000..8a39ceaa3 --- /dev/null +++ b/App/backend/local-api-contracts/src/browser-cdp-write.test.ts @@ -0,0 +1,39 @@ +import { expect, it } from 'vitest'; +import { normalizeBrowserCdpWriteCall } from './browser-cdp-write.js'; +import { BROWSER_CAPABILITY_REQUEST, isBrowserCapabilityRequest } from './browser-capability-approval.js'; +import { EMBEDDED_BROWSER_REQUEST, isEmbeddedBrowserRequest } from './embedded-browser-control.js'; + +it('permits bounded page mutations but rejects prohibited raw CDP commands and extra parameters', () => { + expect(normalizeBrowserCdpWriteCall('Runtime.evaluate', { expression: 'document.title = "Updated"' })) + .toEqual({ method: 'Runtime.evaluate', params: { expression: 'document.title = "Updated"' }, + targetUrls: [] }); + expect(normalizeBrowserCdpWriteCall('DOM.setAttributeValue', { + nodeId: 3, name: 'data-status', value: 'ready', + })).toEqual({ method: 'DOM.setAttributeValue', params: { + nodeId: 3, name: 'data-status', value: 'ready', + }, targetUrls: [] }); + for (const method of ['DOM.setFileInputFiles', 'Input.dispatchKeyEvent', 'Page.navigate', + 'Target.attachToTarget']) { + expect(() => normalizeBrowserCdpWriteCall(method, {})).toThrow(/unavailable/); + } + expect(() => normalizeBrowserCdpWriteCall('Network.setCookie', {})).toThrow(/valid url/); + expect(normalizeBrowserCdpWriteCall('Runtime.evaluate', { + expression: 'document.title = "changed"', throwOnSideEffect: false, + }).params.throwOnSideEffect).toBe(false); + expect(() => normalizeBrowserCdpWriteCall('Fetch.enable', { + patterns: [{ resourceType: 'Document' }], + })).toThrow(/unavailable/); + expect(normalizeBrowserCdpWriteCall('Network.setCookie', { + name: 'test', value: '1', url: 'https://other.test/path', + }).targetUrls).toEqual(['https://other.test/path']); +}); + +it('accepts only a typed write approval and embedded command at the IPC boundary', () => { + expect(isBrowserCapabilityRequest({ type: BROWSER_CAPABILITY_REQUEST, + requestId: 'f6715e94-0c77-4796-96bb-1ebd0fc1700a', capability: 'debug-write', + origin: 'https://example.test', names: ['Runtime.evaluate'] })).toBe(true); + expect(isEmbeddedBrowserRequest({ type: EMBEDDED_BROWSER_REQUEST, + requestId: 'operation-1', tabId: 3, command: 'cdpWrite', args: { + method: 'Runtime.evaluate', params: { expression: 'document.title = "changed"' }, + } })).toBe(true); +}); diff --git a/App/backend/local-api-contracts/src/browser-cdp-write.ts b/App/backend/local-api-contracts/src/browser-cdp-write.ts new file mode 100644 index 000000000..2c3c54fbd --- /dev/null +++ b/App/backend/local-api-contracts/src/browser-cdp-write.ts @@ -0,0 +1,130 @@ +/** Browser-service CDP policy recreated from the installed package's observable gate. */ +const allowedDomains = new Set([ + 'Accessibility', 'Audits', 'Console', 'CSS', 'Debugger', 'DOM', 'DOMDebugger', + 'DOMSnapshot', 'Emulation', 'Fetch', 'IO', 'Input', 'Inspector', 'Log', 'Network', + 'Overlay', 'Page', 'Performance', 'Profiler', 'Runtime', 'Tracing', 'WebAudio', +]); + +const blockedMethods = new Set([ + 'DOM.getFileInfo', 'DOM.setFileInputFiles', 'Input.dispatchKeyEvent', 'Input.setInterceptDrags', + 'Network.clearBrowserCookies', 'Network.deleteDeviceBoundSession', + 'Network.enableDeviceBoundSessions', 'Network.getAllCookies', + 'Network.getResponseBodyForInterception', 'Network.setCookieControls', + 'Network.setExtraHTTPHeaders', 'Network.setRequestInterception', + 'Network.takeResponseBodyForInterceptionAsStream', 'Page.addScriptToEvaluateOnLoad', + 'Page.addScriptToEvaluateOnNewDocument', 'Page.crash', 'Page.disable', + 'Page.getNavigationHistory', 'Page.resetNavigationHistory', 'Page.setAdBlockingEnabled', + 'Page.setBypassCSP', 'Page.setDownloadBehavior', 'Page.setInterceptFileChooserDialog', + 'Page.setRPHRegistrationMode', 'Page.setSPCTransactionMode', 'Tracing.requestMemoryDump', + 'Fetch.disable', 'Input.dispatchDragEvent', 'Network.continueInterceptedRequest', + 'Network.getCertificate', 'Network.loadNetworkResource', 'Network.replayXHR', + 'Page.navigate', 'Page.navigateToHistoryEntry', +]); + +function urlField(method: string, value: Record, field: string, required = false): string[] { + const target = value[field]; + if (target === undefined && !required) return []; + if (typeof target !== 'string' || !target || target.length > 4096) { + throw new Error(`${method} requires a valid ${field} for origin approval`); + } + return [target]; +} + +function cookieUrls(method: string, value: Record): string[] { + if (['domain', 'sourcePort', 'sourceScheme'].some(field => Object.hasOwn(value, field))) { + throw new Error(`${method} requires an explicit URL instead of domain or source fields`); + } + const urls = urlField(method, value, 'url', true); + if (value.partitionKey !== undefined) { + if (!value.partitionKey || typeof value.partitionKey !== 'object' + || Array.isArray(value.partitionKey)) throw new Error('Invalid CDP cookie partition key'); + urls.push(...urlField(method, value.partitionKey as Record, 'topLevelSite')); + } + return urls; +} + +function destinationUrls(method: string, params: Record): string[] { + switch (method) { + case 'Fetch.continueRequest': return urlField(method, params, 'url'); + case 'Fetch.continueResponse': + case 'Fetch.fulfillRequest': { + if (params.responseHeaders === undefined) return []; + if (!Array.isArray(params.responseHeaders) || params.responseHeaders.length > 100) { + throw new Error('Invalid CDP response headers'); + } + const urls: string[] = []; + for (const header of params.responseHeaders) { + if (!header || typeof header !== 'object' || Array.isArray(header) + || typeof header.name !== 'string' || typeof header.value !== 'string') { + throw new Error('Invalid CDP response header'); + } + if (header.name.trim().toLowerCase() === 'location') urls.push(header.value); + } + return urls; + } + case 'Network.deleteCookies': + case 'Network.setCookie': return cookieUrls(method, params); + case 'Network.setCookies': { + if (params.cookies === undefined) return []; + if (!Array.isArray(params.cookies) || params.cookies.length > 20) throw new Error('Invalid CDP cookies'); + return params.cookies.flatMap(cookie => { + if (!cookie || typeof cookie !== 'object' || Array.isArray(cookie)) { + throw new Error('Invalid CDP cookie'); + } + return cookieUrls(method, cookie as Record); + }); + } + case 'Network.getCookies': { + if (!Array.isArray(params.urls) || params.urls.length < 1 || params.urls.length > 20 + || params.urls.some(url => typeof url !== 'string' || !url || url.length > 4096)) { + throw new Error('Network.getCookies requires explicit URLs for origin approval'); + } + return params.urls as string[]; + } + case 'Page.deleteCookie': return urlField(method, params, 'url', true); + default: return []; + } +} + +function validateSpecialParameters(method: string, params: Record): void { + const unavailable = () => { throw new Error(`${method} parameters are unavailable through raw CDP`); }; + if ((method === 'Fetch.continueResponse' || method === 'Fetch.fulfillRequest') + && params.binaryResponseHeaders != null) unavailable(); + if (method === 'Network.configureDurableMessages' && params.maxTotalBufferSize != null) unavailable(); + if (method === 'Network.enable' && params.enableDurableMessages === true) unavailable(); + if (method === 'Page.createIsolatedWorld' && params.grantUniveralAccess === true) unavailable(); + if (method === 'Page.reload' && params.scriptToEvaluateOnLoad != null) unavailable(); + if (method === 'Tracing.start') { + const options = typeof params.options === 'string' ? params.options.split(',').map(value => value.trim()) : []; + const trace = params.traceConfig && typeof params.traceConfig === 'object' + && !Array.isArray(params.traceConfig) ? params.traceConfig as Record : {}; + if (params.perfettoConfig != null || params.tracingBackend === 'system' + || options.includes('enable-systrace') || trace.enableSystrace === true + || trace.memoryDumpConfig != null) unavailable(); + } + if (method === 'Fetch.enable') { + if (!Array.isArray(params.patterns) || params.patterns.length > 50 + || params.patterns.some(pattern => !pattern || typeof pattern !== 'object' + || Array.isArray(pattern) || typeof pattern.resourceType !== 'string' + || pattern.resourceType === 'Document')) unavailable(); + } +} + +/** Returns every explicit destination that needs its own access and CDP decision. */ +export function normalizeBrowserCdpWriteCall(method: unknown, input: unknown): { + method: string; params: Record; targetUrls: string[] } { + if (typeof method !== 'string' || !/^[A-Za-z]+\.[A-Za-z]+$/.test(method) + || !allowedDomains.has(method.split('.')[0]!) || blockedMethods.has(method) + || !input || typeof input !== 'object' || Array.isArray(input)) { + throw new Error('CDP method or parameters are unavailable'); + } + let encoded: string; + try { encoded = JSON.stringify(input); } + catch { throw new Error('CDP parameters are unavailable'); } + if (!encoded || encoded.length > 24_000) throw new Error('CDP parameters are unavailable'); + const params = JSON.parse(encoded) as Record; + validateSpecialParameters(method, params); + const targetUrls = destinationUrls(method, params); + if (targetUrls.length > 20) throw new Error('Too many CDP destination URLs'); + return { method, params, targetUrls }; +} diff --git a/App/backend/local-api-contracts/src/browser-download-approval.ts b/App/backend/local-api-contracts/src/browser-download-approval.ts new file mode 100644 index 000000000..7d9171565 --- /dev/null +++ b/App/backend/local-api-contracts/src/browser-download-approval.ts @@ -0,0 +1,22 @@ +export const BROWSER_DOWNLOAD_REQUEST = 'memmy:browser-download:request'; +export const BROWSER_DOWNLOAD_RESULT = 'memmy:browser-download:result'; +export type BrowserDownloadRequest = { type: typeof BROWSER_DOWNLOAD_REQUEST; requestId: string; name: string; url: string }; +export type BrowserDownloadResult = { type: typeof BROWSER_DOWNLOAD_RESULT; requestId: string; filePath: string | null }; + +export function isBrowserDownloadRequest(value: unknown): value is BrowserDownloadRequest { + if (!value || typeof value !== 'object' || Array.isArray(value)) return false; + const request = value as Record; + return request.type === BROWSER_DOWNLOAD_REQUEST && typeof request.requestId === 'string' + && /^[a-f\d-]{36}$/i.test(request.requestId) + && typeof request.name === 'string' && request.name.length > 0 && request.name.length <= 255 + && typeof request.url === 'string' && request.url.length <= 4096 + && Object.keys(request).every(key => ['type', 'requestId', 'name', 'url'].includes(key)); +} +export function isBrowserDownloadResult(value: unknown): value is BrowserDownloadResult { + if (!value || typeof value !== 'object' || Array.isArray(value)) return false; + const result = value as Record; + return result.type === BROWSER_DOWNLOAD_RESULT && typeof result.requestId === 'string' + && /^[a-f\d-]{36}$/i.test(result.requestId) + && (result.filePath === null || (typeof result.filePath === 'string' && result.filePath.length <= 4096)) + && Object.keys(result).every(key => ['type', 'requestId', 'filePath'].includes(key)); +} diff --git a/App/backend/local-api-contracts/src/browser-global-approval.ts b/App/backend/local-api-contracts/src/browser-global-approval.ts new file mode 100644 index 000000000..82ca43863 --- /dev/null +++ b/App/backend/local-api-contracts/src/browser-global-approval.ts @@ -0,0 +1,9 @@ +/** The browser permission selected by the user and any effective Computer Use grant. */ +export type BrowserAllowAllState = { + enabled: boolean; + inherited: boolean; + effective: boolean; +}; + +/** Desktop provides the existing Computer Use approval path to the managed Agent. */ +export const BROWSER_NATIVE_ALLOW_ALL_FILE_ENV = 'MEMMY_NATIVE_APP_ALLOW_ALL_FILE'; diff --git a/App/backend/local-api-contracts/src/browser-playwright-control.test.ts b/App/backend/local-api-contracts/src/browser-playwright-control.test.ts new file mode 100644 index 000000000..6bb678de1 --- /dev/null +++ b/App/backend/local-api-contracts/src/browser-playwright-control.test.ts @@ -0,0 +1,33 @@ +import fs from 'node:fs'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { expect, it } from 'vitest'; +import { BROWSER_PLAYWRIGHT_CDP_METHODS } from './browser-playwright-control.js'; +import { EMBEDDED_BROWSER_EVENT, EMBEDDED_BROWSER_REQUEST, isEmbeddedBrowserEvent, isEmbeddedBrowserRequest } from './embedded-browser-control.js'; + +it('keeps external and embedded fixed executor CDP methods aligned without browser-wide APIs', () => { + const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../../..'); + const source = fs.readFileSync(path.join(root, 'App/shell/desktop/extensions/memmy-browser/background.js'), 'utf8'); + const body = source.match(/const PLAYWRIGHT_CDP_METHODS = new Set\(\[([\s\S]*?)\]\)/u)?.[1]; + expect(body).toBeDefined(); + const methods = [...body!.matchAll(/'([^']+)'/gu)].map(match => match[1]!); + expect(methods).toEqual([...BROWSER_PLAYWRIGHT_CDP_METHODS]); + expect(methods).not.toContain('DOM.setFileInputFiles'); + expect(methods.some(method => /^(?:Browser|Storage)\.|^Target\.(?:createTarget|closeTarget|attachToTarget|attachToBrowserTarget)$/u.test(method))).toBe(false); +}); + +it.each(['playwrightStart', 'playwrightCdp', 'playwrightEnd'] as const)('requires an explicit target tab for %s', command => { + const request = { type: EMBEDDED_BROWSER_REQUEST, requestId: 'request-1', command, args: {} }; + expect(isEmbeddedBrowserRequest(request)).toBe(false); + expect(isEmbeddedBrowserRequest({ ...request, tabId: 27 })).toBe(true); +}); + +it('accepts only a well-formed executor event for one tab and lease', () => { + const event = { type: EMBEDDED_BROWSER_EVENT, tabId: 27, leaseId: 'a'.repeat(32), method: 'Page.frameNavigated', params: {} }; + expect(isEmbeddedBrowserEvent(event)).toBe(true); + expect(isEmbeddedBrowserEvent({ ...event, sessionId: 'child' })).toBe(true); + expect(isEmbeddedBrowserEvent({ ...event, tabId: 0 })).toBe(false); + expect(isEmbeddedBrowserEvent({ ...event, leaseId: 'short' })).toBe(false); + expect(isEmbeddedBrowserEvent({ ...event, method: 'not a method' })).toBe(false); + expect(isEmbeddedBrowserEvent({ ...event, params: [] })).toBe(false); +}); diff --git a/App/backend/local-api-contracts/src/browser-playwright-control.ts b/App/backend/local-api-contracts/src/browser-playwright-control.ts new file mode 100644 index 000000000..605b43cd5 --- /dev/null +++ b/App/backend/local-api-contracts/src/browser-playwright-control.ts @@ -0,0 +1,21 @@ +/** Fixed local executor transport. These commands are not model-visible raw CDP tools. */ +export const BROWSER_PLAYWRIGHT_CDP_METHODS = [ + 'Runtime.enable', 'Runtime.disable', 'Runtime.evaluate', 'Runtime.callFunctionOn', + 'Runtime.getProperties', 'Runtime.releaseObject', 'Runtime.releaseObjectGroup', + 'Runtime.addBinding', 'Runtime.runIfWaitingForDebugger', + 'Page.enable', 'Page.disable', 'Page.getFrameTree', 'Page.getLayoutMetrics', + 'Page.setLifecycleEventsEnabled', 'Page.addScriptToEvaluateOnNewDocument', + 'Page.removeScriptToEvaluateOnNewDocument', 'Page.createIsolatedWorld', + 'Page.captureScreenshot', 'Page.handleJavaScriptDialog', + 'DOM.enable', 'DOM.getDocument', 'DOM.describeNode', 'DOM.resolveNode', + 'DOM.getContentQuads', 'DOM.getBoxModel', 'DOM.scrollIntoViewIfNeeded', + 'DOM.querySelector', 'DOM.querySelectorAll', + 'Network.enable', 'Network.disable', 'Network.setCacheDisabled', + 'Log.enable', 'Log.disable', + 'Input.dispatchMouseEvent', 'Input.dispatchKeyEvent', 'Input.insertText', + 'Emulation.setDeviceMetricsOverride', 'Emulation.clearDeviceMetricsOverride', + 'Target.setAutoAttach', 'Target.getTargetInfo', +] as const; + +export type BrowserPlaywrightEvent = { sequence: number; leaseId: string; method: string; + params: Record; sessionId?: string }; diff --git a/App/backend/local-api-contracts/src/browser-profile-control.test.ts b/App/backend/local-api-contracts/src/browser-profile-control.test.ts new file mode 100644 index 000000000..af4d69551 --- /dev/null +++ b/App/backend/local-api-contracts/src/browser-profile-control.test.ts @@ -0,0 +1,10 @@ +import { expect, it } from 'vitest'; +import { isBrowserProfileClearRequest, isBrowserProfileClearResult } from './browser-profile-control.js'; + +it('accepts only a bounded browser clear request and result', () => { + expect(isBrowserProfileClearRequest({ type: 'memmy:browser-profile:clear', requestId: 'a-123' })).toBe(true); + expect(isBrowserProfileClearRequest({ type: 'memmy:browser-profile:clear', requestId: 'a-123', path: '/tmp' })).toBe(false); + expect(isBrowserProfileClearResult({ type: 'memmy:browser-profile:clear-result', requestId: 'a-123', ok: true })).toBe(true); + expect(isBrowserProfileClearResult({ type: 'memmy:browser-profile:clear-result', requestId: 'a-123', ok: false, error: 'failed' })).toBe(true); + expect(isBrowserProfileClearResult({ type: 'memmy:browser-profile:clear-result', requestId: 'a-123', ok: true, error: 'x'.repeat(513) })).toBe(false); +}); diff --git a/App/backend/local-api-contracts/src/browser-profile-control.ts b/App/backend/local-api-contracts/src/browser-profile-control.ts new file mode 100644 index 000000000..580d6651a --- /dev/null +++ b/App/backend/local-api-contracts/src/browser-profile-control.ts @@ -0,0 +1,24 @@ +/** Desktop main process <-> managed Agent child browser data command. */ +export const BROWSER_PROFILE_CLEAR_REQUEST = 'memmy:browser-profile:clear'; +export const BROWSER_PROFILE_CLEAR_RESULT = 'memmy:browser-profile:clear-result'; + +export type BrowserProfileClearRequest = { type: typeof BROWSER_PROFILE_CLEAR_REQUEST; requestId: string }; +export type BrowserProfileClearResult = { type: typeof BROWSER_PROFILE_CLEAR_RESULT; requestId: string; ok: boolean; error?: string }; + +function validId(value: unknown): value is string { + return typeof value === 'string' && /^[a-zA-Z0-9-]{1,128}$/.test(value); +} +export function isBrowserProfileClearRequest(value: unknown): value is BrowserProfileClearRequest { + if (!value || typeof value !== 'object' || Array.isArray(value)) return false; + const item = value as Record; + return item.type === BROWSER_PROFILE_CLEAR_REQUEST && validId(item.requestId) + && Object.keys(item).every(key => ['type', 'requestId'].includes(key)); +} +export function isBrowserProfileClearResult(value: unknown): value is BrowserProfileClearResult { + if (!value || typeof value !== 'object' || Array.isArray(value)) return false; + const item = value as Record; + return item.type === BROWSER_PROFILE_CLEAR_RESULT && validId(item.requestId) + && typeof item.ok === 'boolean' + && (item.error === undefined || (typeof item.error === 'string' && item.error.length <= 512)) + && Object.keys(item).every(key => ['type', 'requestId', 'ok', 'error'].includes(key)); +} diff --git a/App/backend/local-api-contracts/src/browser-task-site-permission.ts b/App/backend/local-api-contracts/src/browser-task-site-permission.ts new file mode 100644 index 000000000..8db86e640 --- /dev/null +++ b/App/backend/local-api-contracts/src/browser-task-site-permission.ts @@ -0,0 +1,20 @@ +export const BROWSER_TASK_SITE_PERMISSIONS = ['camera', 'microphone'] as const; +export type BrowserTaskSitePermission = typeof BROWSER_TASK_SITE_PERMISSIONS[number]; +export type BrowserTaskSitePermissionRequest = { + origin: string; + permission: BrowserTaskSitePermission; + allow: boolean; +}; + +export function parseBrowserTaskSitePermission(value: unknown): BrowserTaskSitePermissionRequest | null { + if (!value || typeof value !== 'object' || Array.isArray(value)) return null; + const input = value as Record; + if (typeof input.origin !== 'string' || input.origin.length > 320 + || !BROWSER_TASK_SITE_PERMISSIONS.includes(input.permission as BrowserTaskSitePermission) + || typeof input.allow !== 'boolean') return null; + try { + const url = new URL(input.origin); + if (!['http:', 'https:'].includes(url.protocol) || url.origin !== input.origin) return null; + } catch { return null; } + return { origin: input.origin, permission: input.permission as BrowserTaskSitePermission, allow: input.allow }; +} diff --git a/App/backend/local-api-contracts/src/browser-use-site-policy.test.ts b/App/backend/local-api-contracts/src/browser-use-site-policy.test.ts new file mode 100644 index 000000000..42a7553ae --- /dev/null +++ b/App/backend/local-api-contracts/src/browser-use-site-policy.test.ts @@ -0,0 +1,71 @@ +import { describe, expect, it } from 'vitest'; +import { browserUsePatternMatches, evaluateBrowserUseSiteRule, + parseBrowserUsePattern, parseBrowserUseSitePolicyDocument, type BrowserUseSiteRule } from './browser-use-site-policy.js'; + +const rule = (pattern: string, values: Partial = {}): BrowserUseSiteRule => ({ + pattern, access: 'ask', downloads: 'ask', uploads: 'ask', fullCdp: 'block', ...values, +}); + +describe('browser use site policy', () => { + it('matches origin patterns by scheme, port, apex and subdomains', () => { + expect(parseBrowserUsePattern('https://*.Example.com')?.canonical).toBe('https://*.example.com'); + expect(browserUsePatternMatches('https://*.example.com', 'https://a.example.com/form')).toBe(true); + expect(browserUsePatternMatches('https://*.example.com', 'https://example.com/form')).toBe(false); + expect(browserUsePatternMatches('https://**.example.com', 'https://example.com/form')).toBe(true); + expect(browserUsePatternMatches('https://**.example.com', 'https://a.example.com/form')).toBe(true); + expect(browserUsePatternMatches('https://*.example.com', 'http://a.example.com/form')).toBe(false); + expect(browserUsePatternMatches('https://*.example.com:8443', 'https://a.example.com/form')).toBe(false); + expect(browserUsePatternMatches('https://*.example.com:8443', 'https://a.example.com:8443/form')).toBe(true); + expect(parseBrowserUsePattern('https://*.example.com/path')).toBeNull(); + expect(parseBrowserUsePattern('https://user@example.com')).toBeNull(); + expect(parseBrowserUsePattern('https://*.example.com ')).toBeNull(); + }); + + it('gives block priority across matches and prevents browsing block from allowing transfer', () => { + const rules = [rule('https://**.example.com', { uploads: 'allow' }), + rule('https://private.example.com', { uploads: 'block' })]; + expect(evaluateBrowserUseSiteRule(rules, 'https://private.example.com/', 'uploads')).toBe('block'); + expect(evaluateBrowserUseSiteRule(rules, 'https://public.example.com/', 'uploads')).toBe('allow'); + expect(evaluateBrowserUseSiteRule([rule('https://example.com', { access: 'block', uploads: 'allow' })], + 'https://example.com/', 'uploads')).toBe('block'); + }); + + it('asks for every capability on a site without a rule and honors explicit allow or block', () => { + for (const capability of ['access', 'downloads', 'uploads', 'fullCdp'] as const) { + expect(evaluateBrowserUseSiteRule([], 'https://example.com/', capability)).toBe('ask'); + } + const rules = [rule('https://example.com', { access: 'allow', fullCdp: 'allow' })]; + expect(evaluateBrowserUseSiteRule(rules, 'https://example.com/', 'fullCdp')).toBe('allow'); + expect(evaluateBrowserUseSiteRule(rules, 'https://other.example/', 'fullCdp')).toBe('ask'); + expect(evaluateBrowserUseSiteRule([rule('https://example.com', { access: 'block', fullCdp: 'allow' })], + 'https://example.com/', 'fullCdp')).toBe('block'); + }); + + it('rejects duplicate canonical patterns and corrupt documents', () => { + expect(parseBrowserUseSitePolicyDocument({ version: 1, rules: [rule('https://Example.com'), + rule('https://example.com')] })).toBeNull(); + expect(parseBrowserUseSitePolicyDocument({ version: 2, rules: [] })).toBeNull(); + }); + + it.each(['access', 'downloads', 'uploads', 'fullCdp'] as const)('rejects non-string persisted %s decisions', capability => { + for (const value of [['block'], ['ask'], ['allow'], null, undefined, true, 1, { toString: 'block' }]) { + expect(parseBrowserUseSitePolicyDocument({ version: 1, + rules: [{ ...rule('https://example.com'), [capability]: value }] })).toBeNull(); + } + }); + + it('retains valid schema 1 allow and block rules when the unconfigured default changes', () => { + const document = { version: 1, rules: [ + rule('https://allowed.example', { access: 'allow', fullCdp: 'allow' }), + rule('https://blocked.example', { access: 'block', uploads: 'allow', fullCdp: 'allow' }), + ] }; + const before = JSON.stringify(document); + const parsed = parseBrowserUseSitePolicyDocument(document); + expect(parsed).toEqual(document.rules); + expect(JSON.stringify(document)).toBe(before); + expect(evaluateBrowserUseSiteRule(parsed!, 'https://allowed.example/', 'fullCdp')).toBe('allow'); + expect(evaluateBrowserUseSiteRule(parsed!, 'https://blocked.example/', 'fullCdp')).toBe('block'); + expect(evaluateBrowserUseSiteRule(parsed!, 'https://blocked.example/', 'uploads')).toBe('block'); + expect(evaluateBrowserUseSiteRule(parsed!, 'https://unconfigured.example/', 'fullCdp')).toBe('ask'); + }); +}); diff --git a/App/backend/local-api-contracts/src/browser-use-site-policy.ts b/App/backend/local-api-contracts/src/browser-use-site-policy.ts new file mode 100644 index 000000000..bba0dae5c --- /dev/null +++ b/App/backend/local-api-contracts/src/browser-use-site-policy.ts @@ -0,0 +1,81 @@ +export type BrowserUseCapability = 'access' | 'downloads' | 'uploads' | 'fullCdp'; +export type BrowserUseDecision = 'block' | 'ask' | 'allow'; +export type BrowserUseSiteRule = { pattern: string } & Record; + +/** Sites without a user rule ask; a silent default block would deny without ever reaching the user. */ +export const BROWSER_USE_DEFAULT_DECISION: BrowserUseDecision = 'ask'; + +type ParsedPattern = { canonical: string; protocol: string; host: string; + port: string; kind: 'exact' | 'subdomains' | 'apex-and-subdomains' }; + +/** A conservative subset of the Codex origin pattern grammar. */ +export function parseBrowserUsePattern(value: string): ParsedPattern | null { + if (typeof value !== 'string' || value.length > 320 || value !== value.trim() || /\s/.test(value) + || !/^https?:\/\//.test(value)) return null; + const authority = value.slice(value.indexOf('://') + 3); + if (!authority || /[/?#@\\]/.test(authority)) return null; + try { + const url = new URL(value); + if (!['http:', 'https:'].includes(url.protocol) || url.username || url.password + || url.pathname !== '/' || url.search || url.hash) return null; + const rawHost = url.hostname.replace(/%2a/gi, '*').toLowerCase().replace(/\.$/, ''); + const kind = rawHost.startsWith('**.') ? 'apex-and-subdomains' + : rawHost.startsWith('*.') ? 'subdomains' : 'exact'; + const host = kind === 'apex-and-subdomains' ? rawHost.slice(3) + : kind === 'subdomains' ? rawHost.slice(2) : rawHost; + if (!host || /[\[\]{}\\]/.test(host) || (host.includes('*') && /[^\x00-\x7f]/.test(host))) return null; + if (host.split('.').some(label => !label)) return null; + const canonicalHost = kind === 'apex-and-subdomains' ? `**.${host}` + : kind === 'subdomains' ? `*.${host}` : host; + return { canonical: `${url.protocol}//${canonicalHost}${url.port ? `:${url.port}` : ''}`, + protocol: url.protocol, host, port: url.port, kind }; + } catch { return null; } +} + +function hostGlob(pattern: string, host: string): boolean { + const escaped = pattern.replace(/[.+^${}()|[\]\\]/g, '\\$&').replace(/\*/g, '.*'); + return new RegExp(`^${escaped}$`, 'i').test(host); +} + +export function browserUsePatternMatches(pattern: string, pageUrl: string): boolean { + const parsed = parseBrowserUsePattern(pattern); + if (!parsed) return false; + try { + const page = new URL(pageUrl); + if (page.protocol !== parsed.protocol || page.port !== parsed.port) return false; + const host = page.hostname.toLowerCase().replace(/\.$/, ''); + if (parsed.kind === 'exact') return hostGlob(parsed.host, host); + const subdomain = host.endsWith(`.${parsed.host}`) && host.length > parsed.host.length + 1; + return parsed.kind === 'subdomains' ? subdomain : host === parsed.host || subdomain; + } catch { return false; } +} + +export function isBrowserUseSiteRule(value: unknown): value is BrowserUseSiteRule { + if (!value || typeof value !== 'object' || Array.isArray(value)) return false; + const rule = value as Record; + return typeof rule.pattern === 'string' && parseBrowserUsePattern(rule.pattern) !== null + && (['access', 'downloads', 'uploads', 'fullCdp'] as const) + .every(key => typeof rule[key] === 'string' && ['block', 'ask', 'allow'].includes(rule[key] as string)); +} + +export function evaluateBrowserUseSiteRule(rules: readonly BrowserUseSiteRule[], pageUrl: string, + capability: BrowserUseCapability): BrowserUseDecision { + if (capability !== 'access' && evaluateBrowserUseSiteRule(rules, pageUrl, 'access') === 'block') return 'block'; + let result = BROWSER_USE_DEFAULT_DECISION; + for (const rule of rules) { + if (!browserUsePatternMatches(rule.pattern, pageUrl)) continue; + if (rule[capability] === 'block') return 'block'; + result = rule[capability]; + } + return result; +} + +export function parseBrowserUseSitePolicyDocument(raw: unknown): BrowserUseSiteRule[] | null { + if (!raw || typeof raw !== 'object' || Array.isArray(raw)) return null; + const document = raw as Record; + if (document.version !== 1 || !Array.isArray(document.rules) || document.rules.length > 200 + || !document.rules.every(isBrowserUseSiteRule)) return null; + const rules = document.rules as BrowserUseSiteRule[]; + const canonical = rules.map(rule => parseBrowserUsePattern(rule.pattern)!.canonical); + return new Set(canonical).size === canonical.length ? rules : null; +} diff --git a/App/backend/local-api-contracts/src/computer-use-contracts.test.ts b/App/backend/local-api-contracts/src/computer-use-contracts.test.ts index cbce01820..26c6e6690 100644 --- a/App/backend/local-api-contracts/src/computer-use-contracts.test.ts +++ b/App/backend/local-api-contracts/src/computer-use-contracts.test.ts @@ -21,7 +21,7 @@ describe("computer-use local API exports", () => { type: `${COMPUTER_USE_ONBOARDING_PREFIX}guide`, requestId: "guide-1", reason: "accessibility", - helperApp: "/Applications/Open Computer Use.app", + helperApp: "/Applications/Memmy Computer Use.app", canContinue: false, })).toBe(true); expect(isComputerUseOnboardingRequest({ diff --git a/App/backend/local-api-contracts/src/computer-use-onboarding.ts b/App/backend/local-api-contracts/src/computer-use-onboarding.ts index 2c07dd498..6ee818152 100644 --- a/App/backend/local-api-contracts/src/computer-use-onboarding.ts +++ b/App/backend/local-api-contracts/src/computer-use-onboarding.ts @@ -20,6 +20,7 @@ export type ComputerUseOnboardingRequest = { reason?: ComputerUseGuideReason; helperApp?: string; canContinue?: boolean; + canObserve?: boolean; }; export function isComputerUseGuideReason(value: unknown): value is ComputerUseGuideReason { return value === 'accessibility' || value === 'screenCaptureUnavailable'; @@ -27,11 +28,12 @@ export function isComputerUseGuideReason(value: unknown): value is ComputerUseGu export function isComputerUseOnboardingRequest(value: any): value is ComputerUseOnboardingRequest { return !!value && typeof value === 'object' && !Array.isArray(value) && typeof value.requestId === 'string' && /^[a-zA-Z0-9-]{1,80}$/.test(value.requestId) - && Object.keys(value).every(key => ['type', 'requestId', 'reason', 'helperApp', 'canContinue'].includes(key)) - && ((value.type === `${COMPUTER_USE_ONBOARDING_PREFIX}prepare` && value.reason === undefined && value.helperApp === undefined && value.canContinue === undefined) + && Object.keys(value).every(key => ['type', 'requestId', 'reason', 'helperApp', 'canContinue', 'canObserve'].includes(key)) + && ((value.type === `${COMPUTER_USE_ONBOARDING_PREFIX}prepare` && value.reason === undefined && value.helperApp === undefined && value.canContinue === undefined && value.canObserve === undefined) || (value.type === `${COMPUTER_USE_ONBOARDING_PREFIX}guide` && isComputerUseGuideReason(value.reason) && (value.canContinue === undefined || typeof value.canContinue === 'boolean') - && typeof value.helperApp === 'string' && value.helperApp.startsWith('/') && value.helperApp.endsWith('/Open Computer Use.app') + && (value.canObserve === undefined || typeof value.canObserve === 'boolean') + && typeof value.helperApp === 'string' && value.helperApp.startsWith('/') && value.helperApp.endsWith('/Memmy Computer Use.app') && value.helperApp.length <= 4096 && !/[\r\n\0]/.test(value.helperApp))); } export function isComputerUseProbeTarget(value: any): value is ComputerUseProbeTarget { diff --git a/App/backend/local-api-contracts/src/computer-use-surface.test.ts b/App/backend/local-api-contracts/src/computer-use-surface.test.ts new file mode 100644 index 000000000..66b1b82ab --- /dev/null +++ b/App/backend/local-api-contracts/src/computer-use-surface.test.ts @@ -0,0 +1,116 @@ +import { describe, expect, it } from 'vitest'; +import { isComputerUseSurfaceAction, isComputerUseSurfaceMessage } from './computer-use-surface.js'; + +const update = { + type: 'memmy:computer-use-surface:update', surface: 'browser', sessionKey: 's', + channel: 'gui', chatId: 'chat', targetId: 'tab', title: 'Example', + imageDataUrl: 'data:image/jpeg;base64,YWJj', +}; + +describe('computer use surface child IPC', () => { + it('accepts a bounded browser image update', () => { + expect(isComputerUseSurfaceMessage(update)).toBe(true); + expect(isComputerUseSurfaceMessage({ ...update, url: 'https://example.com/', + canGoBack: true, canGoForward: false })).toBe(true); + expect(isComputerUseSurfaceMessage({ ...update, canGoBack: 'yes' })).toBe(false); + expect(isComputerUseSurfaceMessage({ ...update, surface: 'computer', url: 'https://example.com/' })).toBe(false); + expect(isComputerUseSurfaceMessage({ ...update, surface: 'computer', targetWindowId: 42 })).toBe(true); + expect(isComputerUseSurfaceMessage({ ...update, surface: 'computer', targetWindowId: 4294967297 })).toBe(true); + expect(isComputerUseSurfaceMessage({ ...update, surface: 'computer', targetWindowId: Number.MAX_SAFE_INTEGER + 1 })).toBe(false); + expect(isComputerUseSurfaceMessage({ ...update, surface: 'computer', targetPid:321 })).toBe(true); + expect(isComputerUseSurfaceMessage({ ...update, targetPid:321 })).toBe(false); + expect(isComputerUseSurfaceMessage({ ...update, surface:'computer', targetPid:0 })).toBe(false); + expect(isComputerUseSurfaceMessage({ ...update, surface:'computer', targetPid:1.5 })).toBe(false); + expect(isComputerUseSurfaceMessage({ ...update, surface:'computer', targetPid:0x80000000 })).toBe(false); + expect(isComputerUseSurfaceMessage({ ...update, targetWindowId: 42 })).toBe(false); + expect(isComputerUseSurfaceMessage({ ...update, surface: 'computer', targetWindowId: 0 })).toBe(false); + }); + + it('rejects active content and oversized frames', () => { + expect(isComputerUseSurfaceMessage({ ...update, imageDataUrl: 'data:text/html;base64,YWJj' })).toBe(false); + expect(isComputerUseSurfaceMessage({ ...update, imageDataUrl: `data:image/png;base64,${'A'.repeat(3_000_001)}` })).toBe(false); + }); + + it('accepts bounded associated-window proof only with a native root and PID', () => { + const native = { ...update, surface: 'computer', targetWindowId: 52, targetPid: 20 }; + expect(isComputerUseSurfaceMessage({ ...native, targetAssociatedWindowIds: [53, 54] })).toBe(true); + expect(isComputerUseSurfaceMessage({ ...native, targetWindowId: 0x100000034, + targetAssociatedWindowIds: [0x100000035] })).toBe(true); + expect(isComputerUseSurfaceMessage({ ...native, targetAssociatedWindowIds: [] })).toBe(true); + for (const ids of [[0], [-1], [1.5], [Number.MAX_SAFE_INTEGER + 1], ['53'], [53, 53], Array(65).fill(53)]) + expect(isComputerUseSurfaceMessage({ ...native, targetAssociatedWindowIds: ids })).toBe(false); + expect(isComputerUseSurfaceMessage({ ...update, targetAssociatedWindowIds: [53] })).toBe(false); + expect(isComputerUseSurfaceMessage({ ...native, targetPid: undefined, targetAssociatedWindowIds: [53] })).toBe(false); + expect(isComputerUseSurfaceAction({ type: 'memmy:computer-use-surface:action', surface: 'computer', + sessionKey: 's', channel: 'gui', chatId: 'chat', targetId: 'WPS', action: 'open', targetAssociatedWindowIds: [53] })).toBe(false); + }); + + it('rejects unexpected fields', () => { + expect(isComputerUseSurfaceMessage({ ...update, action: 'execute' })).toBe(false); + }); + + it('accepts a bounded app key on browser and computer updates only', () => { + expect(isComputerUseSurfaceMessage({ ...update, appKey: 'browser:chrome' })).toBe(true); + expect(isComputerUseSurfaceMessage({ ...update, surface: 'computer', appKey: 'browser:chrome' })).toBe(true); + expect(isComputerUseSurfaceMessage({ ...update, type: 'memmy:computer-use-surface:close', appKey: 'browser:chrome' })).toBe(false); + for (const appKey of ['', 'a'.repeat(257), 'app:\nother', 42]) + expect(isComputerUseSurfaceMessage({ ...update, appKey })).toBe(false); + }); + + it('accepts a bounded native turn identity only on computer surfaces', () => { + expect(isComputerUseSurfaceMessage({ ...update, surface: 'computer', turnId: 'turn-1' })).toBe(true); + expect(isComputerUseSurfaceMessage({ ...update, turnId: 'turn-1' })).toBe(false); + expect(isComputerUseSurfaceMessage({ ...update, surface: 'computer', turnId: '' })).toBe(false); + }); + + it('distinguishes closing the PiP from closing the browser session', () => { + const close = { ...update, type: 'memmy:computer-use-surface:close' }; + expect(isComputerUseSurfaceMessage({ ...close, presentationOnly: true })).toBe(true); + expect(isComputerUseSurfaceMessage({ ...update, presentationOnly: true })).toBe(false); + expect(isComputerUseSurfaceMessage({ ...close, surface: 'computer', presentationOnly: true })).toBe(false); + }); +}); + +it('accepts only bounded interactions with the bound surface', () => { + const action = { type: 'memmy:computer-use-surface:action', surface: 'browser', + sessionKey: 's', channel: 'gui', chatId: 'chat', targetId: 'tab', + action: 'click', x: 0.25, y: 0.75 }; + expect(isComputerUseSurfaceAction(action)).toBe(true); + expect(isComputerUseSurfaceAction({ ...action, x: 1.5 })).toBe(false); + expect(isComputerUseSurfaceAction({ ...action, command: 'execute' })).toBe(false); + expect(isComputerUseSurfaceAction({ ...action, surface: 'computer', frameMode: 'live', frameWidth: 1920, frameHeight: 1080 })).toBe(true); + expect(isComputerUseSurfaceAction({ ...action, surface: 'computer', targetWindowId: 42, + frameMode: 'live', frameWidth: 1920, frameHeight: 1080 })).toBe(true); + expect(isComputerUseSurfaceAction({ ...action, targetWindowId: 42 })).toBe(false); + expect(isComputerUseSurfaceAction({ ...action, surface: 'computer', targetWindowId: -1 })).toBe(false); + expect(isComputerUseSurfaceAction({ ...action, surface: 'computer', frameMode: 'live' })).toBe(false); + expect(isComputerUseSurfaceAction({ ...action, surface: 'computer', frameMode: 'live', frameWidth: 0, frameHeight: 1080 })).toBe(false); + expect(isComputerUseSurfaceAction({ ...action, surface: 'computer', frameMode: 'live', frameWidth: 1.5, frameHeight: 1080 })).toBe(false); + expect(isComputerUseSurfaceAction({ ...action, frameMode: 'live' })).toBe(false); + expect(isComputerUseSurfaceAction({ ...action, action: 'open', x: undefined, y: undefined })).toBe(false); + const { x: _x, y: _y, ...withoutCoordinates } = action; + expect(isComputerUseSurfaceAction({ ...withoutCoordinates, action: 'open' })).toBe(true); + expect(isComputerUseSurfaceAction({ ...withoutCoordinates, surface: 'computer', action: 'open' })).toBe(true); + expect(isComputerUseSurfaceAction({ ...withoutCoordinates, surface: 'computer', action: 'interrupt', turnId: 'turn-1' })).toBe(true); + expect(isComputerUseSurfaceAction({ ...withoutCoordinates, surface: 'computer', action: 'interrupt' })).toBe(false); + expect(isComputerUseSurfaceAction({ ...withoutCoordinates, surface: 'browser', action: 'interrupt', turnId: 'turn-1' })).toBe(false); + expect(isComputerUseSurfaceAction({ ...withoutCoordinates, surface: 'computer', action: 'interrupt', turnId: 'turn-1', key: 'Escape' })).toBe(false); + expect(isComputerUseSurfaceAction({ ...withoutCoordinates, surface: 'computer', action: 'interrupt', turnId: 'turn-1', targetWindowId: -1 })).toBe(false); + expect(isComputerUseSurfaceAction({ ...withoutCoordinates, surface: 'computer', action: 'key', key: 'Enter' })).toBe(false); + expect(isComputerUseSurfaceAction({ ...withoutCoordinates, surface: 'computer', action: 'key', key: 'Enter', frameMode: 'live' })).toBe(true); + expect(isComputerUseSurfaceAction({ ...withoutCoordinates, action: 'key', key: 'Enter', frameMode: 'live' })).toBe(false); + expect(isComputerUseSurfaceAction({ ...withoutCoordinates, action: 'back' })).toBe(true); + expect(isComputerUseSurfaceAction({ ...withoutCoordinates, action: 'forward' })).toBe(true); + expect(isComputerUseSurfaceAction({ ...withoutCoordinates, action: 'reload' })).toBe(true); + expect(isComputerUseSurfaceAction({ ...withoutCoordinates, action: 'navigate', url: 'https://example.com/' })).toBe(true); + expect(isComputerUseSurfaceAction({ ...withoutCoordinates, action: 'navigate', url: 'javascript:alert(1)' })).toBe(false); + expect(isComputerUseSurfaceAction({ ...withoutCoordinates, surface: 'computer', action: 'navigate', url: 'https://example.com/' })).toBe(false); + expect(isComputerUseSurfaceAction({ ...action, surface: 'computer', action: 'scroll', deltaY: 120 })).toBe(true); + expect(isComputerUseSurfaceAction({ ...action, surface: 'computer', action: 'scroll', deltaY: 120, + frameMode: 'live', frameWidth: 1920, frameHeight: 1080 })).toBe(true); + expect(isComputerUseSurfaceAction({ ...withoutCoordinates, surface: 'computer', action: 'scroll', deltaY: 120 })).toBe(false); + expect(isComputerUseSurfaceAction({ ...action, action: 'scroll', deltaY: 120 })).toBe(true); + expect(isComputerUseSurfaceAction({ ...action, action: 'scroll', y: undefined, deltaY: 120 })).toBe(false); + expect(isComputerUseSurfaceAction({ ...withoutCoordinates, action: 'scroll', deltaY: 120 })).toBe(true); + expect(isComputerUseSurfaceAction({ ...withoutCoordinates, action: 'scroll', deltaY: 120, frameMode: 'live' })).toBe(false); +}); diff --git a/App/backend/local-api-contracts/src/computer-use-surface.ts b/App/backend/local-api-contracts/src/computer-use-surface.ts new file mode 100644 index 000000000..ae5798fa3 --- /dev/null +++ b/App/backend/local-api-contracts/src/computer-use-surface.ts @@ -0,0 +1,168 @@ +/** Private Agent child -> Desktop main process preview protocol. */ +export const COMPUTER_USE_SURFACE_PREFIX = 'memmy:computer-use-surface:'; +export const COMPUTER_USE_SURFACE_MAX_IMAGE_CHARS = 3_000_000; + +export type ComputerUseSurfaceMessage = { + type: 'memmy:computer-use-surface:update' | 'memmy:computer-use-surface:close'; + surface: 'browser' | 'computer'; + sessionKey: string; + channel: string; + chatId: string; + targetId: string; + /** Active Agent turn that owns a native surface; never supplied by the renderer. */ + turnId?: string; + title: string; + /** Exact native target window for the displayed computer frame, when known. */ + targetWindowId?: number; + /** Trusted owning process for native live capture; never supplied by the renderer. */ + targetPid?: number; + /** Native AX-root association proof; never supplied by a renderer or model argument. */ + targetAssociatedWindowIds?: number[]; + /** Shared by every surface of one user-visible app, such as a Chrome tab and a Chrome window. */ + appKey?: string; + imageDataUrl?: string; + url?: string; + canGoBack?: boolean; + canGoForward?: boolean; + /** Close the turn's PiP while retaining the underlying browser session. */ + presentationOnly?: boolean; + error?: string; +}; + +export type ComputerUseSurfaceAction = { + type: 'memmy:computer-use-surface:action'; + surface: 'browser' | 'computer'; + sessionKey: string; + channel: string; + chatId: string; + targetId: string; + turnId?: string; + targetWindowId?: number; + action: 'click' | 'scroll' | 'key' | 'open' | 'interrupt' | 'navigate' | 'back' | 'forward' | 'reload' | 'fill-credential' | 'fill-contact'; + x?: number; + y?: number; + deltaY?: number; + /** Intrinsic video dimensions, required for live coordinate actions. */ + frameWidth?: number; + frameHeight?: number; + key?: string; + url?: string; + frameMode?: 'live'; + autofill?: { origin: string; username: string; password: string } | { origin: string; name: string; email: string; phone: string; address: string }; +}; + +export function isComputerUseSurfaceAction(value: unknown): value is ComputerUseSurfaceAction { + if (!value || typeof value !== 'object' || Array.isArray(value)) return false; + const action = value as Record; + if (action.type !== `${COMPUTER_USE_SURFACE_PREFIX}action` + || (action.surface !== 'browser' && action.surface !== 'computer')) return false; + if (![action.sessionKey, action.channel, action.chatId, action.targetId] + .every(item => typeof item === 'string' && item.length > 0 && item.length <= 256)) return false; + if (action.turnId !== undefined && (action.surface !== 'computer' + || typeof action.turnId !== 'string' || action.turnId.length === 0 || action.turnId.length > 128)) return false; + const validFrame = action.frameMode === undefined + ? action.frameWidth === undefined && action.frameHeight === undefined + : action.surface === 'computer' && action.frameMode === 'live' + && dimension(action.frameWidth) && dimension(action.frameHeight); + if (action.targetWindowId !== undefined && (action.surface !== 'computer' + || !windowIdentifier(action.targetWindowId))) return false; + if (action.action === 'interrupt') return action.surface === 'computer' && action.turnId !== undefined + && keysOnly(action, ['type', 'surface', 'sessionKey', 'channel', 'chatId', 'targetId', 'turnId', 'targetWindowId', 'action']); + if (action.action === 'click') return normalized(action.x) && normalized(action.y) && validFrame + && keysOnly(action, ['type', 'surface', 'sessionKey', 'channel', 'chatId', 'targetId', 'turnId', 'targetWindowId', 'action', 'x', 'y', 'frameMode', 'frameWidth', 'frameHeight']); + if (action.action === 'scroll') return typeof action.deltaY === 'number' && Number.isFinite(action.deltaY) + && Math.abs(action.deltaY) <= 1000 + && (action.surface === 'browser' + ? (action.x === undefined && action.y === undefined) || (normalized(action.x) && normalized(action.y)) + : normalized(action.x) && normalized(action.y)) + && validFrame + && keysOnly(action, ['type', 'surface', 'sessionKey', 'channel', 'chatId', 'targetId', 'turnId', 'targetWindowId', 'action', 'deltaY', 'x', 'y', 'frameMode', 'frameWidth', 'frameHeight']); + if (action.action === 'key') return typeof action.key === 'string' && action.key.length > 0 && action.key.length <= 64 + && (action.surface === 'computer' ? action.frameMode === 'live' : action.frameMode === undefined) + && keysOnly(action, ['type', 'surface', 'sessionKey', 'channel', 'chatId', 'targetId', 'turnId', 'targetWindowId', 'action', 'key', 'frameMode']); + if (action.action === 'open') return keysOnly(action, + ['type', 'surface', 'sessionKey', 'channel', 'chatId', 'targetId', 'turnId', 'targetWindowId', 'action']); + if (['back', 'forward', 'reload'].includes(String(action.action))) return action.surface === 'browser' + && keysOnly(action, ['type', 'surface', 'sessionKey', 'channel', 'chatId', 'targetId', 'action']); + if (action.action === 'navigate') return action.surface === 'browser' + && typeof action.url === 'string' && action.url.length > 0 && action.url.length <= 4096 + && /^https?:\/\//i.test(action.url) + && keysOnly(action, ['type', 'surface', 'sessionKey', 'channel', 'chatId', 'targetId', 'action', 'url']); + if (action.action === 'fill-credential' || action.action === 'fill-contact') { + if (action.surface !== 'browser' || !keysOnly(action, + ['type', 'surface', 'sessionKey', 'channel', 'chatId', 'targetId', 'action', 'autofill'])) return false; + const autofill = action.autofill; + if (!autofill || typeof autofill !== 'object' || Array.isArray(autofill)) return false; + const fields = autofill as Record; + try { + const origin = new URL(String(fields.origin)); + if (!['http:', 'https:'].includes(origin.protocol) || origin.origin !== fields.origin + || origin.origin.length > 320) return false; + } catch { return false; } + if (action.action === 'fill-credential') return typeof fields.username === 'string' + && fields.username.length <= 256 && typeof fields.password === 'string' + && fields.password.length > 0 && fields.password.length <= 4096 + && keysOnly(fields, ['origin', 'username', 'password']); + return ['name', 'email', 'phone', 'address'].every(key => + typeof fields[key] === 'string' && (fields[key] as string).length <= 1024) + && keysOnly(fields, ['origin', 'name', 'email', 'phone', 'address']); + } + return false; +} + +function normalized(value: unknown): boolean { + return typeof value === 'number' && Number.isFinite(value) && value >= 0 && value <= 1; +} + +function dimension(value: unknown): boolean { + return typeof value === 'number' && Number.isSafeInteger(value) && value > 0 && value <= 16384; +} + +function windowIdentifier(value: unknown): boolean { + return typeof value === 'number' && Number.isSafeInteger(value) && value > 0; +} + +function keysOnly(value: Record, allowed: string[]): boolean { + return Object.keys(value).every(key => allowed.includes(key)); +} + +export function isComputerUseSurfaceMessage(value: unknown): value is ComputerUseSurfaceMessage { + if (!value || typeof value !== 'object' || Array.isArray(value)) return false; + const message = value as Record; + if (message.type !== `${COMPUTER_USE_SURFACE_PREFIX}update` + && message.type !== `${COMPUTER_USE_SURFACE_PREFIX}close`) return false; + if (message.surface !== 'browser' && message.surface !== 'computer') return false; + if (message.targetWindowId !== undefined && (message.surface !== 'computer' + || message.type !== `${COMPUTER_USE_SURFACE_PREFIX}update` + || !windowIdentifier(message.targetWindowId))) return false; + if (message.targetPid !== undefined && (message.surface !== 'computer' + || message.type !== `${COMPUTER_USE_SURFACE_PREFIX}update` + || typeof message.targetPid !== 'number' || !Number.isSafeInteger(message.targetPid) + || message.targetPid < 1 || message.targetPid > 0x7fffffff)) return false; + if (message.targetAssociatedWindowIds !== undefined && (message.surface !== 'computer' + || message.type !== `${COMPUTER_USE_SURFACE_PREFIX}update` + || !windowIdentifier(message.targetWindowId) || message.targetPid === undefined + || !Array.isArray(message.targetAssociatedWindowIds) || message.targetAssociatedWindowIds.length > 64 + || !message.targetAssociatedWindowIds.every(windowIdentifier) + || new Set(message.targetAssociatedWindowIds).size !== message.targetAssociatedWindowIds.length)) return false; + if (![message.sessionKey, message.channel, message.chatId, message.targetId, message.title] + .every(item => typeof item === 'string' && item.length > 0 && item.length <= 256)) return false; + if (message.turnId !== undefined && (message.surface !== 'computer' + || typeof message.turnId !== 'string' || message.turnId.length === 0 || message.turnId.length > 128)) return false; + if (message.appKey !== undefined && (message.type !== `${COMPUTER_USE_SURFACE_PREFIX}update` + || typeof message.appKey !== 'string' || message.appKey.length === 0 || message.appKey.length > 256 + || /[\u0000-\u001f\u007f]/.test(message.appKey))) return false; + if (message.imageDataUrl !== undefined && (typeof message.imageDataUrl !== 'string' + || message.imageDataUrl.length > COMPUTER_USE_SURFACE_MAX_IMAGE_CHARS + || !/^data:image\/(?:png|jpeg|webp);base64,[A-Za-z0-9+/=]+$/.test(message.imageDataUrl))) return false; + if (message.url !== undefined && (message.surface !== 'browser' || typeof message.url !== 'string' + || message.url.length > 4096)) return false; + if (message.canGoBack !== undefined && (message.surface !== 'browser' || typeof message.canGoBack !== 'boolean')) return false; + if (message.canGoForward !== undefined && (message.surface !== 'browser' || typeof message.canGoForward !== 'boolean')) return false; + if (message.presentationOnly !== undefined && (message.type !== `${COMPUTER_USE_SURFACE_PREFIX}close` + || message.surface !== 'browser' || message.presentationOnly !== true)) return false; + if (message.error !== undefined && (message.surface !== 'browser' || typeof message.error !== 'string' + || message.error.length > 256)) return false; + return Object.keys(message).every(key => + ['type', 'surface', 'sessionKey', 'channel', 'chatId', 'targetId', 'turnId', 'targetWindowId', 'targetPid', 'targetAssociatedWindowIds', 'appKey', 'title', 'imageDataUrl', 'url', 'canGoBack', 'canGoForward', 'presentationOnly', 'error'].includes(key)); +} diff --git a/App/backend/local-api-contracts/src/embedded-browser-control.test.ts b/App/backend/local-api-contracts/src/embedded-browser-control.test.ts new file mode 100644 index 000000000..e2fe6db1d --- /dev/null +++ b/App/backend/local-api-contracts/src/embedded-browser-control.test.ts @@ -0,0 +1,38 @@ +import { expect, it } from 'vitest'; +import { EMBEDDED_BROWSER_REQUEST, formatEmbeddedBrowserTabMention, isEmbeddedBrowserRequest, + parseEmbeddedBrowserTabMention } from './embedded-browser-control.js'; + +it('round trips the exact browser, tab, title, and URL snapshot', () => { + const tab = { browserId: 'fb77c359-316a-4221-8cb7-967e230a8d50', tabId: 7, + title: 'A & B', url: 'https://example.com/path?q=one%20two#part' }; + const mention = formatEmbeddedBrowserTabMention(tab); + expect(parseEmbeddedBrowserTabMention(mention)).toEqual(tab); + expect(parseEmbeddedBrowserTabMention(`${mention}&tabId=8`)).toBeNull(); + expect(parseEmbeddedBrowserTabMention(mention.replace('browser@memmy', 'browser@other'))).toBeNull(); +}); + +it.each(['closeTab', 'endSitePermissions'])('requires an exact tab ID for a %s request', command => { + const close = { type: EMBEDDED_BROWSER_REQUEST, requestId: 'test', command, args: {} }; + expect(isEmbeddedBrowserRequest(close)).toBe(false); + expect(isEmbeddedBrowserRequest({ ...close, tabId: 27 })).toBe(true); +}); + +it('accepts a standalone history request without claiming a tab', () => { + expect(isEmbeddedBrowserRequest({ type: EMBEDDED_BROWSER_REQUEST, requestId: 'test', + command: 'queryHistory', args: { from: '2026-09-28T00:00:00Z', to: '2026-09-29T00:00:00Z', + keyword: 'example', limit: 20 } })).toBe(true); +}); + +it('limits external browser metadata and recovery requests to browser identity', () => { + const state = { type: EMBEDDED_BROWSER_REQUEST, requestId: 'test', command: 'externalBrowserState', args: {} }; + expect(isEmbeddedBrowserRequest(state)).toBe(true); + expect(isEmbeddedBrowserRequest({ ...state, tabId: 1 })).toBe(false); + expect(isEmbeddedBrowserRequest({ ...state, args: { path: '/private/profile' } })).toBe(false); + const recover = { ...state, command: 'recoverExternalBrowser', args: { browser: 'chrome' } }; + expect(isEmbeddedBrowserRequest(recover)).toBe(true); + expect(isEmbeddedBrowserRequest({ ...recover, args: { browser: 'edge' } })).toBe(true); + expect(isEmbeddedBrowserRequest({ ...recover, tabId: 1 })).toBe(false); + for (const args of [{}, { browser: 'memmy' }, { browser: 'chrome', url: 'https://example.com' }]) { + expect(isEmbeddedBrowserRequest({ ...recover, args })).toBe(false); + } +}); diff --git a/App/backend/local-api-contracts/src/embedded-browser-control.ts b/App/backend/local-api-contracts/src/embedded-browser-control.ts new file mode 100644 index 000000000..7842f26b4 --- /dev/null +++ b/App/backend/local-api-contracts/src/embedded-browser-control.ts @@ -0,0 +1,131 @@ +/** Chat link the Agent writes to open the Chrome/Edge extension setup in Computer Use settings. */ +export const BROWSER_EXTENSION_SETTINGS_LINK = 'memmy://settings/browser-extension'; +/** Chat link the Agent writes to open the Microsoft Excel connection in Computer Use settings. */ +export const EXCEL_CONNECTION_SETTINGS_LINK = 'memmy://settings/excel'; + +export const EMBEDDED_BROWSER_REQUEST = 'memmy:embedded-browser:request'; +export const EMBEDDED_BROWSER_RESULT = 'memmy:embedded-browser:result'; +export const EMBEDDED_BROWSER_CANCEL = 'memmy:embedded-browser:cancel'; +export type EmbeddedBrowserCancel = { type: typeof EMBEDDED_BROWSER_CANCEL; requestId: string }; +export function isEmbeddedBrowserCancel(value: unknown): value is EmbeddedBrowserCancel { + if (!value || typeof value !== 'object' || Array.isArray(value)) return false; + const input = value as Record; + return input.type === EMBEDDED_BROWSER_CANCEL && typeof input.requestId === 'string' + && /^[a-zA-Z0-9:-]{1,100}$/.test(input.requestId) + && Object.keys(input).every(key => key === 'type' || key === 'requestId'); +} + +export const EMBEDDED_BROWSER_COMMANDS = [ + 'probe', 'listTabs', 'externalBrowserState', 'recoverExternalBrowser', 'queryHistory', 'openTab', 'closeTab', 'allowOrigin', 'snapshot', 'find', 'navigate', 'back', 'forward', 'reload', + 'click', 'type', 'selectOption', 'pressKey', 'scroll', 'waitFor', 'screenshot', + 'consoleMessages', 'networkRequests', 'viewport', 'historyState', 'resize', 'open', + 'fillCredential', 'fillContact', + 'authRequest', 'setSitePermission', 'endSitePermissions', 'setDownloadDestination', 'waitDownloadDestination', + 'upload', + 'cdpCall', 'cdpEvents', 'cdpWrite', + 'playwrightStart', 'playwrightCdp', 'playwrightEnd', +] as const; + +export type ExternalBrowserInstallation = { installed: boolean | null; enabled: boolean | null }; +/** Browser/extension metadata only; no page contents, paths, or process arguments cross child IPC. */ +export type ExternalBrowserState = { + installation: Record<'chrome' | 'edge', ExternalBrowserInstallation>; + runningBrowsers: Array<'chrome' | 'edge'> | null; +}; +export type ExternalBrowserRecovery = 'woken' | 'unavailable' | 'failed'; + +/** Executor CDP events share the result channel, so they stay ordered with replies. */ +export const EMBEDDED_BROWSER_EVENT = 'memmy:embedded-browser:event'; +export type EmbeddedBrowserEvent = { + type: typeof EMBEDDED_BROWSER_EVENT; + tabId: number; + leaseId: string; + method: string; + params: Record; + sessionId?: string; +}; +export function isEmbeddedBrowserEvent(value: unknown): value is EmbeddedBrowserEvent { + if (!value || typeof value !== 'object' || Array.isArray(value)) return false; + const input = value as Partial; + return input.type === EMBEDDED_BROWSER_EVENT + && Number.isSafeInteger(input.tabId) && input.tabId! > 0 + && typeof input.leaseId === 'string' && /^[a-f0-9]{32}$/.test(input.leaseId) + && typeof input.method === 'string' && /^[A-Za-z]+\.[A-Za-z]+$/.test(input.method) + && Boolean(input.params && typeof input.params === 'object' && !Array.isArray(input.params)) + && (input.sessionId === undefined || (typeof input.sessionId === 'string' && input.sessionId.length <= 200)); +} + +export type EmbeddedBrowserTabMention = { + browserId: string; + tabId: number; + title: string; + url: string; +}; + +export function formatEmbeddedBrowserTabMention(tab: EmbeddedBrowserTabMention): string { + const query = new URLSearchParams({ mention: 'tab-v1', browserId: tab.browserId, + tabId: String(tab.tabId), title: tab.title, url: tab.url }); + return `plugin://browser@memmy?${query}`; +} + +export function parseEmbeddedBrowserTabMention(value: unknown): EmbeddedBrowserTabMention | null { + if (typeof value !== 'string' || value.length > 20_000) return null; + let parsed: URL; + try { parsed = new URL(value); } catch { return null; } + if (parsed.protocol !== 'plugin:' || parsed.username !== 'browser' || parsed.password + || parsed.host !== 'memmy' + || parsed.pathname || parsed.hash || parsed.searchParams.size !== 5 + || [...parsed.searchParams.keys()].some(key => parsed.searchParams.getAll(key).length !== 1) + || parsed.searchParams.get('mention') !== 'tab-v1') return null; + const browserId = parsed.searchParams.get('browserId') ?? ''; + const tabIdText = parsed.searchParams.get('tabId') ?? ''; + const title = parsed.searchParams.get('title'); + const url = parsed.searchParams.get('url') ?? ''; + const tabId = Number(tabIdText); + if (!/^[a-f0-9-]{36}$/i.test(browserId) || !/^[1-9]\d*$/.test(tabIdText) + || !Number.isSafeInteger(tabId) || title === null || title.length > 4096 || url.length > 4096) return null; + try { if (!['http:', 'https:'].includes(new URL(url).protocol)) return null; } + catch { return null; } + return { browserId, tabId, title, url }; +} + +export type EmbeddedBrowserCommand = typeof EMBEDDED_BROWSER_COMMANDS[number]; +export type EmbeddedBrowserRequest = { + type: typeof EMBEDDED_BROWSER_REQUEST; + requestId: string; + tabId?: number; + command: EmbeddedBrowserCommand; + args: Record; +}; +export type EmbeddedBrowserResult = { + type: typeof EMBEDDED_BROWSER_RESULT; + requestId: string; + ok: boolean; + result?: unknown; + error?: string; +}; + +export function isEmbeddedBrowserRequest(value: unknown): value is EmbeddedBrowserRequest { + if (!value || typeof value !== 'object' || Array.isArray(value)) return false; + const input = value as Partial; + return input.type === EMBEDDED_BROWSER_REQUEST + && typeof input.requestId === 'string' && /^[a-zA-Z0-9:-]{1,100}$/.test(input.requestId) + && EMBEDDED_BROWSER_COMMANDS.includes(input.command as EmbeddedBrowserCommand) + && (input.tabId === undefined || (Number.isSafeInteger(input.tabId) && input.tabId! > 0)) + && ((input.command !== 'closeTab' && input.command !== 'endSitePermissions') || input.tabId !== undefined) + && (!input.command?.startsWith('playwright') || input.tabId !== undefined) + && Boolean(input.args && typeof input.args === 'object' && !Array.isArray(input.args)) + && (input.command !== 'externalBrowserState' + || (input.tabId === undefined && Object.keys(input.args!).length === 0)) + && (input.command !== 'recoverExternalBrowser' + || (input.tabId === undefined && Object.keys(input.args!).length === 1 + && (input.args!.browser === 'chrome' || input.args!.browser === 'edge'))); +} + +export function isEmbeddedBrowserResult(value: unknown): value is EmbeddedBrowserResult { + if (!value || typeof value !== 'object' || Array.isArray(value)) return false; + const input = value as Partial; + return input.type === EMBEDDED_BROWSER_RESULT + && typeof input.requestId === 'string' && /^[a-zA-Z0-9:-]{1,100}$/.test(input.requestId) + && typeof input.ok === 'boolean'; +} diff --git a/App/backend/local-api-contracts/src/index.ts b/App/backend/local-api-contracts/src/index.ts index 69bf78250..a9bd7fe32 100644 --- a/App/backend/local-api-contracts/src/index.ts +++ b/App/backend/local-api-contracts/src/index.ts @@ -1,5 +1,8 @@ /** Memmy local API contract. */ import { z } from "zod"; +import { normalizeOutputLanguage } from "./output-language.js"; + +export { normalizeOutputLanguage } from "./output-language.js"; export * from "./model-catalog-resolver.js"; @@ -10,6 +13,19 @@ export * from "./memory-l3-world-model.js"; export * from "./endpoints.js"; export * from "./cloud-service.js"; export * from "./desktop-runtime-manifest.js"; +export * from "./computer-use-surface.js"; +export * from "./browser-profile-control.js"; +export * from "./embedded-browser-control.js"; +export * from "./browser-playwright-control.js"; +export * from "./browser-access-approval.js"; +export * from "./native-app-approval.js"; +export * from "./browser-download-approval.js"; +export * from "./browser-capability-approval.js"; +export * from "./browser-global-approval.js"; +export * from "./browser-task-site-permission.js"; +export * from "./browser-use-site-policy.js"; +export * from "./browser-cdp-readonly.js"; +export * from "./browser-cdp-write.js"; export const MANAGED_AGENT_DISCOVERY_PENDING_DATA_PATH = "memmy-agent://history-discovery-pending"; @@ -1165,9 +1181,8 @@ export type AccountChannel = z.infer; export function resolveMemoryLanguage( language: string | undefined, accountChannel?: AccountChannel | string -): "zh-CN" | "en-US" { - if (language === "zh-CN" || language === "en-US") return language; - return accountChannel === "email" ? "en-US" : "zh-CN"; +): string { + return normalizeOutputLanguage(language) ?? (accountChannel === "email" ? "en-US" : "zh-CN"); } export const AccountLocaleSchema = z.enum(["zh", "en"]); diff --git a/App/backend/local-api-contracts/src/memory-runtime.ts b/App/backend/local-api-contracts/src/memory-runtime.ts index 7073866a7..024a5c480 100644 --- a/App/backend/local-api-contracts/src/memory-runtime.ts +++ b/App/backend/local-api-contracts/src/memory-runtime.ts @@ -22,7 +22,7 @@ export const CursorSchema = z.string(); export type Cursor = z.infer; /** Schema for memory kind. */ -export const MemoryKindSchema = z.enum(["user_memory", "trace", "span", "policy", "world_model", "skill", "work_memory"]); +export const MemoryKindSchema = z.enum(["user_memory", "trace", "span", "policy", "world_model", "skill", "work_memory", "observed_activity"]); export type MemoryKind = z.infer; /** Schema for memory layer. */ diff --git a/App/backend/local-api-contracts/src/native-app-approval.ts b/App/backend/local-api-contracts/src/native-app-approval.ts new file mode 100644 index 000000000..8b8809575 --- /dev/null +++ b/App/backend/local-api-contracts/src/native-app-approval.ts @@ -0,0 +1,37 @@ +export const NATIVE_APP_ACCESS_REQUEST = 'memmy:native-app-access:request'; +export const NATIVE_APP_ACCESS_RESULT = 'memmy:native-app-access:result'; + +export type NativeAppAccessDecision = 'allow-once' | 'allow-always' | 'deny'; +export type NativeAppAccessRequest = { + type: typeof NATIVE_APP_ACCESS_REQUEST; + requestId: string; + appId: string; + displayName: string; + platform: 'darwin' | 'win32'; +}; +export type NativeAppAccessResult = { + type: typeof NATIVE_APP_ACCESS_RESULT; + requestId: string; + decision: NativeAppAccessDecision; +}; + +const uuid = /^[a-f\d-]{36}$/i; +const appId = /^[\w.!-]{1,256}$/; + +export function isNativeAppAccessRequest(value: unknown): value is NativeAppAccessRequest { + if (!value || typeof value !== 'object' || Array.isArray(value)) return false; + const item = value as Record; + return item.type === NATIVE_APP_ACCESS_REQUEST && typeof item.requestId === 'string' && uuid.test(item.requestId) + && typeof item.appId === 'string' && appId.test(item.appId) + && typeof item.displayName === 'string' && item.displayName.trim().length > 0 && item.displayName.length <= 256 + && (item.platform === 'darwin' || item.platform === 'win32') + && Object.keys(item).every(key => ['type', 'requestId', 'appId', 'displayName', 'platform'].includes(key)); +} + +export function isNativeAppAccessResult(value: unknown): value is NativeAppAccessResult { + if (!value || typeof value !== 'object' || Array.isArray(value)) return false; + const item = value as Record; + return item.type === NATIVE_APP_ACCESS_RESULT && typeof item.requestId === 'string' && uuid.test(item.requestId) + && (item.decision === 'allow-once' || item.decision === 'allow-always' || item.decision === 'deny') + && Object.keys(item).every(key => ['type', 'requestId', 'decision'].includes(key)); +} diff --git a/App/backend/local-api-contracts/src/output-language.ts b/App/backend/local-api-contracts/src/output-language.ts new file mode 100644 index 000000000..a604218f9 --- /dev/null +++ b/App/backend/local-api-contracts/src/output-language.ts @@ -0,0 +1,23 @@ +/** + * Normalize a concrete language tag for generated content independently of the + * desktop's translated UI catalogs. Adding an App locale must not require a + * second whitelist in the History/Memory/Skill output pipeline. + */ +export function normalizeOutputLanguage(value: unknown): string | undefined { + if (typeof value !== "string" || !value.trim()) return undefined; + if (value.trim().toLowerCase() === "system") return undefined; + try { + const canonical = Intl.getCanonicalLocales(value.trim())[0]; + if (!canonical) return undefined; + const locale = new Intl.Locale(canonical); + // Settings sentinels, unknown/multiple languages, private language codes and + // pseudo-locales are not a concrete language in which to write memories. + if (!/^[a-z]{2,3}$/u.test(locale.language) + || /^(?:und|mul|mis|zxx|qps)$/u.test(locale.language) + || /^q[a-t][a-z]$/u.test(locale.language) + || locale.region === "XA" || locale.region === "XB") return undefined; + return canonical; + } catch { + return undefined; + } +} diff --git a/App/backend/local-api-contracts/tests/language.test.ts b/App/backend/local-api-contracts/tests/language.test.ts index 454b83cde..90eaab031 100644 --- a/App/backend/local-api-contracts/tests/language.test.ts +++ b/App/backend/local-api-contracts/tests/language.test.ts @@ -1,5 +1,22 @@ import { describe, expect, it } from "vitest"; -import { resolveMemoryLanguage } from "../src/index.js"; +import { LanguageSchema, normalizeOutputLanguage, resolveMemoryLanguage } from "../src/index.js"; + +describe("normalizeOutputLanguage", () => { + it("canonicalizes concrete future output languages without enabling untranslated App UI locales", () => { + expect(normalizeOutputLanguage(" ja-jp ")).toBe("ja-JP"); + expect(normalizeOutputLanguage("fr-fr")).toBe("fr-FR"); + expect(normalizeOutputLanguage("pt-br")).toBe("pt-BR"); + expect(normalizeOutputLanguage("zh-hant-tw")).toBe("zh-Hant-TW"); + expect(normalizeOutputLanguage("iw-il")).toBe("he-IL"); + expect(LanguageSchema.safeParse("ja-JP").success).toBe(false); + }); + + it.each([undefined, null, 42, {}, "", " ", "system", "SYSTEM", "default", "auto", + "not a locale", "en_US", "made-up", "und", "mul", "zxx", "qaa", "qps-ploc", "en-XA", "ar-XB"])( + "rejects settings sentinels, invalid tags and pseudo-locales: %s", (value) => { + expect(normalizeOutputLanguage(value)).toBeUndefined(); + }); +}); describe("resolveMemoryLanguage", () => { it("keeps an explicit interface language", () => { @@ -7,6 +24,11 @@ describe("resolveMemoryLanguage", () => { expect(resolveMemoryLanguage("en-US", "phone")).toBe("en-US"); }); + it("preserves future explicit App output languages instead of replacing them with the package default", () => { + expect(resolveMemoryLanguage("ja-JP", "phone")).toBe("ja-JP"); + expect(resolveMemoryLanguage("fr-fr", "email")).toBe("fr-FR"); + }); + it("maps system and unset language to the package channel default", () => { expect(resolveMemoryLanguage("system")).toBe("zh-CN"); expect(resolveMemoryLanguage(undefined, "phone")).toBe("zh-CN"); diff --git a/App/backend/src/adapters/inbound/local-api/routes/agent-runtime/memory.ts b/App/backend/src/adapters/inbound/local-api/routes/agent-runtime/memory.ts index f4165830e..1a57ce958 100644 --- a/App/backend/src/adapters/inbound/local-api/routes/agent-runtime/memory.ts +++ b/App/backend/src/adapters/inbound/local-api/routes/agent-runtime/memory.ts @@ -19,6 +19,11 @@ const RecallParamsSchema = z.object({ queryId: z.string().min(1) }); +const ObservedSkillUpdateSchema = z.object({ + body: z.string().trim().min(1).max(50_000), + expectedVersion: z.number().int().positive() +}).strict(); + export function registerMemoryRoutes(app: FastifyInstance, deps: AgentRuntimeRouteDeps): void { app.get( "/api/v1/memory/recalls/:queryId", @@ -81,6 +86,31 @@ export function registerMemoryRoutes(app: FastifyInstance, deps: AgentRuntimeRou }) ); + app.post( + "/api/v1/evidence/skills/:id/approve", + { preHandler: deps.authenticateRuntimeToken }, + withErrorEnvelope(async (request, reply) => { + const params = MemoryParamsSchema.parse(request.params); + z.object({}).parse(request.body ?? {}); + return reply.send(await deps.services.memoryClient.approveObservedSkill( + params.id, + runtimeContextFromRequest(request, deps.timeZone) + )); + }) + ); + + app.patch( + "/api/v1/evidence/skills/:id", + { preHandler: deps.authenticateRuntimeToken }, + withErrorEnvelope(async (request, reply) => { + const params = MemoryParamsSchema.parse(request.params); + const input = ObservedSkillUpdateSchema.parse(request.body); + return reply.send(await deps.services.memoryClient.updateObservedSkill( + params.id, input, runtimeContextFromRequest(request, deps.timeZone) + )); + }) + ); + app.get( "/api/v1/memory/:id", { preHandler: deps.authenticateRuntimeToken }, diff --git a/App/backend/src/adapters/outbound/agent-source/onboarding-insight-samplers.ts b/App/backend/src/adapters/outbound/agent-source/onboarding-insight-samplers.ts index 488b3c1c2..325bca360 100644 --- a/App/backend/src/adapters/outbound/agent-source/onboarding-insight-samplers.ts +++ b/App/backend/src/adapters/outbound/agent-source/onboarding-insight-samplers.ts @@ -78,6 +78,10 @@ export function createSourceRegistryOnboardingConversationWindowReader( ): OnboardingConversationWindowReader { return { async readConversation(reference, options) { + if (options.signal?.aborted || options.deadlineMs <= 0) { + return null; + } + const deadlineAt = Date.now() + options.deadlineMs; const adapter = sourceRegistry.require(reference.sourceId); const deadlineSignal = AbortSignal.timeout(options.deadlineMs); const signal = options.signal ? AbortSignal.any([options.signal, deadlineSignal]) : deadlineSignal; @@ -88,8 +92,14 @@ export function createSourceRegistryOnboardingConversationWindowReader( for await (const message of adapter.scan({ maxScanTargets: CONVERSATION_SCAN_TARGETS, order: "recent_first", + fullHistory: true, signal })) { + if (signal.aborted || Date.now() >= deadlineAt) { + // The prefix of a long conversation can omit its newest turns. + // Let the caller retain the recent messages it already sampled. + return null; + } if (message.conversationId !== reference.conversationId) { if (foundConversation) { break; @@ -111,11 +121,15 @@ export function createSourceRegistryOnboardingConversationWindowReader( }); } } catch (error) { - if (!deadlineSignal.aborted && !options.signal?.aborted) { + if (!signal.aborted && Date.now() < deadlineAt) { throw error; } + return null; } + if (signal.aborted || Date.now() >= deadlineAt) { + return null; + } const windowMessages = selectConversationWindow(messages); if (windowMessages.length === 0) { return null; diff --git a/App/backend/src/adapters/outbound/agent-source/tests/onboarding-insight-samplers.test.ts b/App/backend/src/adapters/outbound/agent-source/tests/onboarding-insight-samplers.test.ts index 763590426..7edd99102 100644 --- a/App/backend/src/adapters/outbound/agent-source/tests/onboarding-insight-samplers.test.ts +++ b/App/backend/src/adapters/outbound/agent-source/tests/onboarding-insight-samplers.test.ts @@ -135,6 +135,7 @@ describe("onboarding insight samplers", () => { ...message, createdAt: new Date(baseTime + index * 1000).toISOString() })); + const scanStarted = vi.fn(); const sourceRegistry = createSourceRegistry([{ descriptor: { sourceId: "codex", @@ -145,7 +146,8 @@ describe("onboarding insight samplers", () => { async detect() { return true; }, - async *scan() { + async *scan(options) { + scanStarted(options); yield* messages; } }]); @@ -163,12 +165,87 @@ describe("onboarding insight samplers", () => { }); const userIds = window?.messages.filter((message) => message.role === "user").map((message) => message.messageId); + expect(scanStarted).toHaveBeenCalledWith(expect.objectContaining({ + fullHistory: true, + order: "recent_first", + maxScanTargets: 6 + })); expect(userIds).toEqual(["user-0", "user-1", ...Array.from({ length: 12 }, (_, index) => `user-${index + 3}`)]); expect(window?.messages.filter((message) => message.role === "assistant")).toHaveLength(14); expect(window?.messages.filter((message) => message.role === "tool")).toHaveLength(14); expect(window?.messages.find((message) => message.role === "tool")?.text.length).toBeLessThanOrEqual(405); expect(window?.messages.find((message) => message.role === "tool")?.text).toContain("Status: success"); }); + + it("does not start a conversation scan after its signal has expired", async () => { + const scanStarted = vi.fn(); + const sourceRegistry = createSourceRegistry([{ + descriptor: { sourceId: "codex", displayName: "Codex", builtin: true, dataPath: "/tmp/codex" }, + async detect() { + return true; + }, + async *scan() { + scanStarted(); + yield conversationMessage(0, "user-0", "user", "latest question", "latest-conversation"); + } + }]); + const controller = new AbortController(); + controller.abort(); + const reader = createSourceRegistryOnboardingConversationWindowReader(sourceRegistry); + + await expect(reader.readConversation({ + sourceId: "codex", + displayName: "Codex", + conversationId: "latest-conversation", + latestActivityAt: "2026-07-20T10:00:00.000Z", + workspacePath: "/tmp/project" + }, { maxQueryChars: 600, deadlineMs: 5_000, signal: controller.signal })).resolves.toBeNull(); + expect(scanStarted).not.toHaveBeenCalled(); + }); + + it.each(["latest-conversation", "unrelated-conversation"])( + "closes a microtask-only scan at the wall-clock deadline while reading %s", + async (conversationId) => { + let wallClock = 1_000; + vi.spyOn(Date, "now").mockImplementation(() => wallClock); + let yieldedMessages = 0; + let iteratorClosed = false; + let scanSignal: AbortSignal | undefined; + const sourceRegistry = createSourceRegistry([{ + descriptor: { sourceId: "codex", displayName: "Codex", builtin: true, dataPath: "/tmp/codex" }, + async detect() { + return true; + }, + async *scan(options) { + scanSignal = options.signal; + try { + for (let index = 0; index < 10_000; index += 1) { + wallClock += 1_250; + yieldedMessages += 1; + yield conversationMessage(index, `user-${index}`, "user", `question ${index}`, conversationId); + } + } finally { + iteratorClosed = true; + } + } + }]); + const reader = createSourceRegistryOnboardingConversationWindowReader(sourceRegistry); + + const window = await reader.readConversation({ + sourceId: "codex", + displayName: "Codex", + conversationId: "latest-conversation", + latestActivityAt: "2026-07-20T10:00:00.000Z", + workspacePath: "/tmp/project" + }, { maxQueryChars: 600, deadlineMs: 5_000 }); + + expect(yieldedMessages).toBe(4); + expect(iteratorClosed).toBe(true); + expect(scanSignal?.aborted).toBe(false); + // An incomplete prefix must not replace the shallow sample's latest turns. + expect(window).toBeNull(); + } + ); }); function conversationMessage( diff --git a/App/backend/src/adapters/outbound/memory-client/http-memory-client.ts b/App/backend/src/adapters/outbound/memory-client/http-memory-client.ts index 228a4cae7..f3c1582f6 100644 --- a/App/backend/src/adapters/outbound/memory-client/http-memory-client.ts +++ b/App/backend/src/adapters/outbound/memory-client/http-memory-client.ts @@ -50,6 +50,15 @@ export interface CreateHttpMemoryClientOptions { type PathKey = keyof typeof MEMORY_LAYER_PATHS; +const ObservedSkillApprovalOutputSchema = z.object({ + id: z.string().min(1), + status: z.literal("activated"), + serverTime: z.string().datetime() +}); +const ObservedSkillUpdateOutputSchema = ObservedSkillApprovalOutputSchema.extend({ + version: z.number().int().positive() +}); + /** Creates create http memory client. */ export function createHttpMemoryClient( config: MemoryLayerConfig, @@ -58,7 +67,7 @@ export function createHttpMemoryClient( const fetchImpl = options.fetchImpl ?? globalThis.fetch; async function request( - method: "GET" | "POST" | "DELETE", + method: "GET" | "POST" | "PATCH" | "DELETE", pathKey: PathKey, responseSchema: ZodType, requestOptions: { @@ -200,6 +209,18 @@ export function createHttpMemoryClient( }); }, + async approveObservedSkill(memoryId, context) { + return request("POST", "approveObservedSkill", ObservedSkillApprovalOutputSchema, { + params: { id: memoryId }, body: {}, context, maxRetries: 0 + }); + }, + + async updateObservedSkill(memoryId, input, context) { + return request("PATCH", "updateObservedSkill", ObservedSkillUpdateOutputSchema, { + params: { id: memoryId }, body: input, context, maxRetries: 0 + }); + }, + async recallEvidence(queryId, context) { return request("GET", "recallEvidence", RecallEvidenceOutputSchema, { params: { queryId }, diff --git a/App/backend/src/adapters/outbound/memory-client/memory-layer-endpoints.ts b/App/backend/src/adapters/outbound/memory-client/memory-layer-endpoints.ts index f359bd34e..60276ec7e 100644 --- a/App/backend/src/adapters/outbound/memory-client/memory-layer-endpoints.ts +++ b/App/backend/src/adapters/outbound/memory-client/memory-layer-endpoints.ts @@ -15,6 +15,8 @@ export const MEMORY_LAYER_PATHS = Object.freeze({ addMemory: "/api/v1/memory/add", getMemory: "/api/v1/memory/:id", deleteMemory: "/api/v1/memory/:id", + approveObservedSkill: "/api/v1/evidence/skills/:id/approve", + updateObservedSkill: "/api/v1/evidence/skills/:id", recallEvidence: "/api/v1/memory/recalls/:queryId", runWorker: "/api/v1/worker/run", enqueueImportSummaries: "/api/v1/worker/import-summaries/enqueue", diff --git a/App/backend/src/adapters/outbound/memory-client/tests/http-memory-client.test.ts b/App/backend/src/adapters/outbound/memory-client/tests/http-memory-client.test.ts index 742b62e5f..cf90484e2 100644 --- a/App/backend/src/adapters/outbound/memory-client/tests/http-memory-client.test.ts +++ b/App/backend/src/adapters/outbound/memory-client/tests/http-memory-client.test.ts @@ -46,6 +46,8 @@ describe("HttpMemoryClient", () => { "/api/v1/memory/add", "/api/v1/memory/:id", "/api/v1/memory/:id", + "/api/v1/evidence/skills/:id/approve", + "/api/v1/evidence/skills/:id", "/api/v1/memory/recalls/:queryId", "/api/v1/worker/run", "/api/v1/worker/import-summaries/enqueue", @@ -66,6 +68,46 @@ describe("HttpMemoryClient", () => { expect(() => buildMemoryLayerUrl("http://127.0.0.1:8765", "closeSession")).toThrow("Missing path param"); }); + it("forwards observed Skill approval without inventing a namespace and validates the result", async () => { + const calls: Array<{ url: string; init?: RequestInit }> = []; + const client = createHttpMemoryClient({ baseUrl: "http://memory.test", token: "fixture-token", timeoutMs: 500, maxRetries: 0 }, { + fetchImpl: (async (url, init) => { + calls.push({ url: String(url), init }); + return new Response(JSON.stringify({ id: "skill 1", status: "activated", serverTime: now() }), { + status: 200, headers: { "content-type": "application/json" } + }); + }) as typeof fetch + }); + expect(await client.approveObservedSkill("skill 1")).toEqual({ id: "skill 1", status: "activated", serverTime: now() }); + expect(calls[0]?.url).toBe("http://memory.test/api/v1/evidence/skills/skill%201/approve"); + expect(calls[0]?.init?.method).toBe("POST"); + expect(JSON.parse(String(calls[0]?.init?.body))).toEqual({}); + expect(new Headers(calls[0]?.init?.headers).get("authorization")).toBe("Bearer fixture-token"); + expect(new Headers(calls[0]?.init?.headers).has("x-memmy-user-id")).toBe(false); + }); + + it("saves an observed Skill with its version and does not retry an ambiguous edit", async () => { + const calls: Array<{ url: string; init?: RequestInit }> = []; + let failed = false; + const client = createHttpMemoryClient({ baseUrl: "http://memory.test", timeoutMs: 500, maxRetries: 2 }, { + fetchImpl: (async (url, init) => { + calls.push({ url: String(url), init }); + if (failed) throw new Error("connection lost after sending edit"); + return new Response(JSON.stringify({ id: "skill 1", status: "activated", version: 4, serverTime: now() }), { + status: 200, headers: { "content-type": "application/json" } + }); + }) as typeof fetch + }); + const input = { body: "# Release review\n\nCheck the actual result.", expectedVersion: 3 }; + expect(await client.updateObservedSkill("skill 1", input)).toMatchObject({ version: 4, status: "activated" }); + expect(calls[0]?.url).toBe("http://memory.test/api/v1/evidence/skills/skill%201"); + expect(calls[0]?.init?.method).toBe("PATCH"); + expect(JSON.parse(String(calls[0]?.init?.body))).toEqual(input); + failed = true; + await expect(client.updateObservedSkill("skill 1", input)).rejects.toThrow(); + expect(calls).toHaveLength(2); + }); + it("calls every final runtime path and parses schema-valid responses", async () => { const requests: Array<{ method: string; diff --git a/App/backend/src/adapters/outbound/memory-client/tests/mock-memory-client.test.ts b/App/backend/src/adapters/outbound/memory-client/tests/mock-memory-client.test.ts index 7d219bb89..e68074f33 100644 --- a/App/backend/src/adapters/outbound/memory-client/tests/mock-memory-client.test.ts +++ b/App/backend/src/adapters/outbound/memory-client/tests/mock-memory-client.test.ts @@ -35,6 +35,10 @@ describe("createMockMemoryClient", () => { expect(AddMemoryOutputSchema.parse(await client.addMemory(addMemoryInput())).id).toBeTruthy(); expect(GetMemoryOutputSchema.parse(await client.getMemory({ memoryId: "memory-1" })).item.id).toBe("memory-1"); expect(DeleteMemoryOutputSchema.parse(await client.deleteMemory({ memoryId: "memory-1" })).status).toBe("deleted"); + expect(await client.approveObservedSkill("skill-1")).toEqual({ id: "skill-1", status: "activated", serverTime: NOW }); + expect(await client.updateObservedSkill("skill-1", { body: "Updated guide", expectedVersion: 1 })).toEqual({ + id: "skill-1", status: "activated", version: 2, serverTime: NOW + }); expect(MemoryApiLogsOutputSchema.parse(await client.memoryApiLogs({ limit: 20, offset: 0 })).logs).toEqual([]); expect(PanelOverviewOutputSchema.parse(await client.panelOverview()).counts.memories).toBe(0); expect(PanelAnalysisOutputSchema.parse(await client.panelAnalysis()).metrics.avgRecallScore).toBe(0); @@ -54,6 +58,8 @@ describe("createMockMemoryClient", () => { () => client.addMemory(addMemoryInput()), () => client.getMemory({ memoryId: "memory-1" }), () => client.deleteMemory({ memoryId: "memory-1" }), + () => client.approveObservedSkill("skill-1"), + () => client.updateObservedSkill("skill-1", { body: "Updated guide", expectedVersion: 1 }), () => client.memoryApiLogs({ limit: 20, offset: 0 }), () => client.panelOverview(), () => client.panelAnalysis(), @@ -74,6 +80,7 @@ describe("createMockMemoryClient", () => { expect(Object.keys(client).sort()).toEqual([ "addMemory", + "approveObservedSkill", "closeSession", "completeTurn", "completeSourceTurn", @@ -92,7 +99,8 @@ describe("createMockMemoryClient", () => { "retryMemoryProcessing", "runWorker", "search", - "startTurn" + "startTurn", + "updateObservedSkill" ].sort()); }); }); diff --git a/App/backend/src/adapters/outbound/memory-client/types.ts b/App/backend/src/adapters/outbound/memory-client/types.ts index cd2011f09..ffebdd32d 100644 --- a/App/backend/src/adapters/outbound/memory-client/types.ts +++ b/App/backend/src/adapters/outbound/memory-client/types.ts @@ -43,6 +43,21 @@ export interface MemoryRequestContext { userId?: string; } +export interface ApproveObservedSkillOutput { + id: string; + status: "activated"; + serverTime: string; +} + +export interface UpdateObservedSkillInput { + body: string; + expectedVersion: number; +} + +export interface UpdateObservedSkillOutput extends ApproveObservedSkillOutput { + version: number; +} + export interface MemoryClient { health(): Promise; reloadConfig(input?: MemoryReloadConfigInput): Promise; @@ -62,6 +77,8 @@ export interface MemoryClient { addMemory(input: AddMemoryInput, context?: MemoryRequestContext): Promise; getMemory(input: { memoryId: string }, context?: MemoryRequestContext): Promise; deleteMemory(input: DeleteMemoryInput & { memoryId: string }, context?: MemoryRequestContext): Promise; + approveObservedSkill(memoryId: string, context?: MemoryRequestContext): Promise; + updateObservedSkill(memoryId: string, input: UpdateObservedSkillInput, context?: MemoryRequestContext): Promise; recallEvidence(queryId: string, context?: MemoryRequestContext): Promise; enqueueImportSummaries(memoryIds?: string[]): Promise; diff --git a/App/backend/src/index.ts b/App/backend/src/index.ts index d03358925..d441d389f 100644 --- a/App/backend/src/index.ts +++ b/App/backend/src/index.ts @@ -63,6 +63,8 @@ export interface CreateLocalBackendOptions { accountChannel?: AccountChannel; /** Running Agent Gateway client; when present, refreshes MCP after startup config writes. */ memmyAgentAdminClient?: MemmyAgentAdminClient; + /** Skip only the startup MCP reload when the caller will start the Agent after bridge config is written. */ + skipAgentMcpReload?: boolean; } export interface LocalBackend { @@ -183,13 +185,17 @@ export async function createLocalBackend(options: CreateLocalBackendOptions): Pr headers: { "x-memmy-mcp-token": composioMcpToken }, toolTimeout: 60 }); - if (options.memmyAgentAdminClient) { - try { - const result = await options.memmyAgentAdminClient.reloadMcpConfig(); - if (!result.ok) console.warn(`Agent MCP reload did not complete: ${result.message}`); - } catch (error) { - console.warn(`Agent MCP reload unavailable during backend startup: ${error instanceof Error ? error.message : String(error)}`); - } + const memmyAgentAdminClient = options.memmyAgentAdminClient; + if (memmyAgentAdminClient && !options.skipAgentMcpReload) { + // The bridge config must be written first, but MCP connections must not delay the first window. + void Promise.resolve() + .then(() => memmyAgentAdminClient.reloadMcpConfig()) + .then((result) => { + if (!result.ok) console.warn(`Agent MCP reload did not complete: ${result.message}`); + }) + .catch((error: unknown) => { + console.warn(`Agent MCP reload unavailable during backend startup: ${error instanceof Error ? error.message : String(error)}`); + }); } const runtimeConfig = RuntimeConfigSchema.parse({ diff --git a/App/backend/src/infrastructure/memmy-config/index.ts b/App/backend/src/infrastructure/memmy-config/index.ts index 050475886..6c0937c1c 100644 --- a/App/backend/src/infrastructure/memmy-config/index.ts +++ b/App/backend/src/infrastructure/memmy-config/index.ts @@ -11,7 +11,6 @@ import { type ModelConfigInput, type ModelConfigView, type AccountChannel, - type Language, resolveMemoryLanguage, type ModelProvider, type ModelSelectionResolution, @@ -124,7 +123,7 @@ export interface MemmyConfigWriter { writeUserMode?(mode: UserMode): Promise; /** Publish the interface language so Memory can write memories in it. */ - writeMemoryLanguage?(language: Language): Promise; + writeMemoryLanguage?(language: string): Promise; /** Publish custom-key memory pipeline token caps so the Memory worker can pause. */ writeMemoryTokenBudget?(budget: { dailyLimitM: number; totalLimitM: number }): Promise; diff --git a/App/backend/src/infrastructure/memmy-config/model-config-catalog.ts b/App/backend/src/infrastructure/memmy-config/model-config-catalog.ts index c1d25dc37..5d6a8fea0 100644 --- a/App/backend/src/infrastructure/memmy-config/model-config-catalog.ts +++ b/App/backend/src/infrastructure/memmy-config/model-config-catalog.ts @@ -186,6 +186,22 @@ function projectMemoryConfig( const previousModeAssignment = previousAssignments[mode]; const previousRouting = record(record(previousConfig.memmyMemory).roleRouting); + if (mode === "byok") { + // Catalog saves contain the entire assignment object. When a parent model + // changes, unchanged child fields still carry the old inherited ID. Keep + // the previously selected follow route rather than turning it into fixed. + if (previousRouting.evolution === "follow" + && assignment.memoryEvolution === previousModeAssignment.memoryEvolution) { + assignment.memoryEvolution = inheritedAssignment("memory_evolution", assignment.agent.default); + } + if (previousRouting.summary === "follow" + && assignment.memorySummary === previousModeAssignment.memorySummary) { + assignment.memorySummary = inheritedAssignment( + "memory_summary", + assignment.memoryEvolution ?? assignment.agent.default + ); + } + } projectMemoryRole( config, memory, @@ -215,6 +231,12 @@ function projectMemoryConfig( ); config.memmyMemory = memory; + function inheritedAssignment(capability: "memory_summary" | "memory_evolution", presetId: string | null): string | null { + if (!presetId) return null; + const preset = record(record(config.modelPresets)[presetId]); + return arrayValue(preset.capabilities).includes(capability) ? presetId : null; + } + function projectMemoryRole( root: ConfigRecord, target: ConfigRecord, diff --git a/App/backend/src/infrastructure/memmy-config/tests/index.test.ts b/App/backend/src/infrastructure/memmy-config/tests/index.test.ts index 427f3b69c..b0ba92ea0 100644 --- a/App/backend/src/infrastructure/memmy-config/tests/index.test.ts +++ b/App/backend/src/infrastructure/memmy-config/tests/index.test.ts @@ -165,6 +165,16 @@ describe("memmy runtime config current contract", () => { expect(YAML.parse(await import("node:fs/promises").then(({ readFile }) => readFile(explicitTarget, "utf8"))).memmyMemory.language).toBe("zh-CN"); }); + it.each(["ja-jp", "fr-fr"])("publishes the explicit future output language %s without changing unrelated runtime configuration", async (language) => { + const target = file({ memmyMemory: { domain: "keep", futureOption: true }, app: { futureAppField: "keep" } }); + await createMemmyConfigWriter({ configPath: target, accountChannel: "email" }).writeMemoryLanguage?.(language); + const saved = YAML.parse(await import("node:fs/promises").then(({ readFile }) => readFile(target, "utf8"))); + expect(saved).toMatchObject({ + memmyMemory: { domain: "keep", futureOption: true, language: Intl.getCanonicalLocales(language)[0] }, + app: { futureAppField: "keep" } + }); + }); + it("exposes the same shared writers through createMemmyConfigWriter", async () => { const target = file({ futureSection: { keepMe: true }, diff --git a/App/backend/src/infrastructure/memmy-config/tests/model-config-catalog.test.ts b/App/backend/src/infrastructure/memmy-config/tests/model-config-catalog.test.ts index 4515c5ab6..2c535c0fb 100644 --- a/App/backend/src/infrastructure/memmy-config/tests/model-config-catalog.test.ts +++ b/App/backend/src/infrastructure/memmy-config/tests/model-config-catalog.test.ts @@ -491,8 +491,15 @@ describe("model config catalog", () => { embeddingMode: "custom" }); + const evolutionOnlyInput = structuredClone(assigned); + evolutionOnlyInput.configRevision = saved.configRevision; + evolutionOnlyInput.modelAssignments.byok.memoryEvolution = agentId; + const evolutionOnly = await writeModelConfigCatalog(file, evolutionOnlyInput); + expect(evolutionOnly.modelAssignments.byok.memorySummary).toBe(agentId); + expect(evolutionOnly.memorySettings?.roleRouting).toEqual({ summary: "follow", evolution: "follow" }); + const followInput = structuredClone(assigned); - followInput.configRevision = saved.configRevision; + followInput.configRevision = evolutionOnly.configRevision; followInput.modelAssignments.byok.memorySummary = agentId; followInput.modelAssignments.byok.memoryEvolution = agentId; const followed = await writeModelConfigCatalog(file, followInput); diff --git a/App/backend/src/project-version.ts b/App/backend/src/project-version.ts index 915a5e82a..11f9a99d3 100644 --- a/App/backend/src/project-version.ts +++ b/App/backend/src/project-version.ts @@ -1,2 +1,2 @@ /** Generated from the root package.json by scripts/sync-project-version.mjs. */ -export const MEMMY_VERSION = "1.2.2"; +export const MEMMY_VERSION = "1.2.3"; diff --git a/App/backend/src/services/agent-source-service.ts b/App/backend/src/services/agent-source-service.ts index cbecc0dfe..a879722cd 100644 --- a/App/backend/src/services/agent-source-service.ts +++ b/App/backend/src/services/agent-source-service.ts @@ -53,6 +53,7 @@ import { sourceTurnFromMessages, sourceTurnFailureReason, sourceTurnSkipBlocksWatermark, + sourceTurnSkipWatermarkEffect, buildSourceTurnRequest, renderTurnClipped, stableTurnIdentity, @@ -736,7 +737,7 @@ async function ingestPersistentStagedSource( options.agentSourceRepository.setLastScannedAt(sourceId, scannedAt); const skillResult = await ingestSourceSkills(options, sourceId, scanOptions, store, mode); if (stage.errors.length === 0 && ingestion.errors.length === 0 && skillResult.errorCount === 0 && !ingestion.hasUncommittedSkips) { - updatePersistentWatermark(options, sourceId, mode, ingestion.latestSeenAt, scannedAt, since); + updatePersistentWatermark(options, sourceId, mode, ingestion.latestSeenAt, scannedAt, since, ingestion.retryFrom); } const allErrors = [...stage.errors, ...ingestion.errors, ...skillResult.errors]; const errorCount = stage.scanErrorCount + ingestion.errorCount + skillResult.errorCount; @@ -842,7 +843,7 @@ async function ingestPersistentSource( sourceId: string, scanOptions: AgentSourceScanOptions, initialErrors: readonly { conversationId: string; reason: string }[] -): Promise<{ memoryIds: string[]; memoryIdCount: number; deduped: number; errorCount: number; errors: Array<{ conversationId: string; reason: string }>; latestSeenAt: string | null; hasUncommittedSkips: boolean }> { +): Promise<{ memoryIds: string[]; memoryIdCount: number; deduped: number; errorCount: number; errors: Array<{ conversationId: string; reason: string }>; latestSeenAt: string | null; retryFrom: string | null; hasUncommittedSkips: boolean }> { const memoryIds: string[] = []; let memoryIdCount = 0; const pendingIds: string[] = []; @@ -850,12 +851,21 @@ async function ingestPersistentSource( let errorCount = initialErrors.length; let deduped = 0; let latestSeenAt: string | null = null; + let retryFrom: string | null = null; let activeConversationId: string | null = null; let activeConversationFailed = false; let hasUncommittedSkips = false; let processed = 0; - const noteUncommittedSkip = (reason: string) => { + const noteUncommittedSkip = (reason: string, latestActivityAt: string | null) => { if (!sourceTurnSkipBlocksWatermark(reason)) return; + activeConversationFailed = true; + const effect = sourceTurnSkipWatermarkEffect(reason, latestActivityAt); + if (effect === "rescan") hasUncommittedSkips = true; + else if (effect === "retry" && latestActivityAt && (!retryFrom || Date.parse(latestActivityAt) < Date.parse(retryFrom))) { + retryFrom = latestActivityAt; + } + }; + const noteCaptureFailure = () => { activeConversationFailed = true; hasUncommittedSkips = true; }; @@ -925,7 +935,7 @@ async function ingestPersistentSource( if (!sourceTurn) { const reason = sourceTurnFailureReason(turn.messages); skipPersistentTurn(store, sourceId, turn.conversationId, reason); - noteUncommittedSkip(reason); + noteUncommittedSkip(reason, turnLatest); processed += turn.messages.length; emitAddProgress("Capturing conversation turns"); continue; @@ -946,7 +956,7 @@ async function ingestPersistentSource( if (result.status === "pending" || result.status === "conflict") { const reason = result.reason ?? result.status; skipPersistentTurn(store, sourceId, turn.conversationId, reason); - noteUncommittedSkip(reason); + noteUncommittedSkip(reason, turnLatest); processed += turn.messages.length; emitAddProgress("Capturing conversation turns"); continue; @@ -971,7 +981,7 @@ async function ingestPersistentSource( } catch (error) { const reason = error instanceof Error ? error.message : "native turn ingestion failed"; skipPersistentTurn(store, sourceId, turn.conversationId, reason); - noteUncommittedSkip(reason); + noteCaptureFailure(); if (addAnalyticsBase && !reportedAdd) { options.memoryAddAnalytics?.trackAddStarted(addAnalyticsBase); options.memoryAddAnalytics?.trackAddFailed({ @@ -1062,7 +1072,7 @@ async function ingestPersistentSource( for (const failure of failures) recordScanItemSkip(store, sourceId, failure.memoryId, failure.reason); } commitConversation(); - return { memoryIds, memoryIdCount, deduped, errorCount, errors, latestSeenAt, hasUncommittedSkips }; + return { memoryIds, memoryIdCount, deduped, errorCount, errors, latestSeenAt, retryFrom, hasUncommittedSkips }; } function readScanPage(store: AppAgentSourceScanStore, sourceId: string, cursor?: { conversationId: string; createdAt: string; messageId: string; ordinal: number }): ConversationMessage[] { @@ -1090,9 +1100,11 @@ function firstTurnLine(messages: readonly ConversationMessage[]): string | undef return line ? (line.length <= 120 ? line : `${line.slice(0, 117)}...`) : undefined; } -function updatePersistentWatermark(options: CreateAgentSourceServiceOptions, sourceId: string, mode: AgentSourceScanMode, latestSeenAt: string | null, scannedAt: string, since?: string): void { +function updatePersistentWatermark(options: CreateAgentSourceServiceOptions, sourceId: string, mode: AgentSourceScanMode, latestSeenAt: string | null, scannedAt: string, since?: string, retryFrom?: string | null): void { const existing = options.agentSourceRepository.getScanWatermark(sourceId); - options.agentSourceRepository.upsertScanWatermark({ sourceId, mode, baselineAt: existing?.baselineAt ?? since ?? scannedAt, latestSeenCreatedAt: maxIso(existing?.latestSeenCreatedAt ?? null, latestSeenAt), updatedAt: scannedAt }); + // The next cursor is the later of both fields; neither may pass an unfinished recent turn. + const baselineAt = existing?.baselineAt ?? since ?? (retryFrom && Date.parse(retryFrom) < Date.parse(scannedAt) ? retryFrom : scannedAt); + options.agentSourceRepository.upsertScanWatermark({ sourceId, mode, baselineAt, latestSeenCreatedAt: maxIso(existing?.latestSeenCreatedAt ?? null, retryFrom ?? latestSeenAt), updatedAt: scannedAt }); } export interface CollectedSourceScan { diff --git a/App/backend/src/services/byok-token-usage-service.ts b/App/backend/src/services/byok-token-usage-service.ts index 92d2df18a..1399b8ff5 100644 --- a/App/backend/src/services/byok-token-usage-service.ts +++ b/App/backend/src/services/byok-token-usage-service.ts @@ -70,7 +70,8 @@ export function createByokTokenUsageService( lifetimeUsed: snapshot.lifetimeUsed, paused: snapshot.paused, trigger: snapshot.trigger, - nextLocalMidnightAt: snapshot.nextLocalMidnightAt || new Date(nextLocalMidnightMs(current)).toISOString() + nextLocalMidnightAt: snapshot.nextLocalMidnightAt || new Date(nextLocalMidnightMs(current)).toISOString(), + ...(snapshot.stale === undefined ? {} : { stale: snapshot.stale }) }); } @@ -80,7 +81,7 @@ export function createByokTokenUsageService( } const today = localCalendarDate(current); if (lastMemoryBudget.dailyDate === today) { - return budgetFromMemorySnapshot(current, lastMemoryBudget.snapshot); + return { ...budgetFromMemorySnapshot(current, lastMemoryBudget.snapshot), stale: true }; } const events = eventsUsage(current); const rolled = evaluateMemoryTokenBudget({ @@ -138,7 +139,27 @@ export function createByokTokenUsageService( async getMemoryBudget() { const current = now(); - return await readMemoryBudget(current) ?? budgetFromUsage(current, eventsUsage(current)); + const memoryBudget = await readMemoryBudget(current); + const usage = eventsUsage(current); + if (!memoryBudget) { + const fallback = budgetFromUsage(current, usage); + return options.memoryClient ? { ...fallback, stale: true } : fallback; + } + // History reports summary usage directly to App. Memory's in-process ledger + // may not have reconciled those events yet; keep either side's larger total + // without counting the same persisted Memory events twice. + const merged = evaluateMemoryTokenBudget({ + dailyLimitM: memoryBudget.dailyLimitM, + totalLimitM: memoryBudget.totalLimitM, + dailyUsed: Math.max(memoryBudget.dailyUsed, usage.dailyUsed), + lifetimeUsed: Math.max(memoryBudget.lifetimeUsed, usage.lifetimeUsed) + }); + return MemoryTokenBudgetDtoSchema.parse({ + ...memoryBudget, + ...merged, + paused: memoryBudget.paused || merged.paused, + trigger: merged.trigger ?? (memoryBudget.paused ? memoryBudget.trigger : null) + }); } }; } diff --git a/App/backend/src/services/error-envelope.ts b/App/backend/src/services/error-envelope.ts index 737963d06..f4f61084b 100644 --- a/App/backend/src/services/error-envelope.ts +++ b/App/backend/src/services/error-envelope.ts @@ -1,6 +1,6 @@ /** Error envelope module. */ import { ZodError } from "zod"; -import { MemoryLayerError } from "../adapters/outbound/memory-client/index.js"; +import { MemoryLayerError, MemoryLayerNetworkError } from "../adapters/outbound/memory-client/index.js"; import type { FastifyReply, FastifyRequest, RouteHandlerMethod } from "fastify"; export const API_ERROR_CODES = [ @@ -99,6 +99,14 @@ function toErrorEnvelope(error: unknown): ErrorEnvelope { }; } + if (error instanceof MemoryLayerNetworkError) { + return { + code: "memory_layer_unavailable", + status: HTTP_STATUS_BY_CODE.memory_layer_unavailable, + message: error.message + }; + } + if (error instanceof Error && hasApiErrorCode(error)) { return { code: error.code, diff --git a/App/backend/src/services/onboarding-insight-service.ts b/App/backend/src/services/onboarding-insight-service.ts index 1f9bf3224..99d902a1c 100644 --- a/App/backend/src/services/onboarding-insight-service.ts +++ b/App/backend/src/services/onboarding-insight-service.ts @@ -45,6 +45,10 @@ const GENERATED_REPORT_OPEN_MARKERS = [GENERATED_REPORT_OPEN, GENERATED_REPORT_A const GENERATED_REPORT_CLOSE_MARKERS = [GENERATED_REPORT_CLOSE, GENERATED_REPORT_ALIAS_CLOSE] as const; const GENERATED_NAKED_JSON_OPEN = "\n{"; const GENERATED_JSON_FENCE_OPEN = "\n```json"; +const FIRST_ENCOUNTER_USER_MESSAGE = { + "zh-CN": "嗨,第一次见面,你有什么可以为我做的吗?", + "en-US": "Hi, it's our first time meeting. What can you do for me?" +} as const; const TOPIC_PATTERNS: ReadonlyArray<{ keyword: string; pattern: RegExp }> = [ { keyword: "TypeScript", pattern: /\btypescript\b|\bts\b/i }, @@ -68,6 +72,7 @@ const TOPIC_PATTERNS: ReadonlyArray<{ keyword: string; pattern: RegExp }> = [ const PROBLEM_PATTERN = /\berror\b|\bfail(?:ed|ing)?\b|\bbug\b|\bfix\b|报错|失败|修复|问题|构建|编译/i; const DECISION_PATTERN = /方案|设计|取舍|决策|PRD|\bplan\b|\bdesign\b|\bdecision\b/i; const ACTION_PATTERN = /实现|修改|重启|测试|验证|push|排查|检查|补充|更新|落地|继续|接续|整理|整合|rewrite|refactor|verify|restart/i; +const CONCRETE_REQUEST_PATTERN = /请|帮我|帮忙|如何|怎么|设计|写|生成|制作|整理|规划|计划|分析|翻译|推荐|列出|总结|修复|实现|检查|解决|搜索|查找|比较|安排|准备|继续|讨论|please|help|write|create|make|plan|analy[sz]e|translate|recommend|summari[sz]e|fix|check|compare|schedule|continue/i; const HIGH_SIGNAL_PATTERN = /不要|不能|必须|应该|先|后续|可落地|细节|完整|快速|轻量|token|耗时|并行|水位|清除|假数据|隐私|权限|重启|测试|验证|push|don't|must|should|first|fast|lightweight/i; const LOW_VALUE_TASK_PATTERN = /\/tmp\/|memos_missing_demo|请先尝试读取|失败后不要放弃|read\s+\/tmp|smoke|fixture|mock/i; const GENERIC_ACCOUNT_NAMES = [ @@ -554,8 +559,9 @@ async function sampleRecentQueries( const discovered = results.filter((result): result is OnboardingSampleResult => Boolean(result)); const queries = selectBalancedQueries(discovered, MAX_BALANCED_QUERIES); const latestReference = resolveLatestConversationReference(discovered); + const remainingDeadlineMs = Math.max(0, FIRST_LOGIN_SCAN_DEADLINE_MS - (now() - startedAt)); const latestConversation = latestReference - ? await loadLatestConversation(latestReference, discovered, conversationWindowReader, sampleSignal) + ? await loadLatestConversation(latestReference, discovered, conversationWindowReader, sampleSignal, remainingDeadlineMs) : null; return { @@ -628,15 +634,16 @@ async function loadLatestConversation( reference: OnboardingConversationReference, results: readonly OnboardingSampleResult[], reader: OnboardingConversationWindowReader | null | undefined, - signal: AbortSignal + signal: AbortSignal, + deadlineMs: number ): Promise { if (reader) { try { - const loaded = await reader.readConversation(reference, { + const loaded = await readWithinScanDeadline(() => reader.readConversation(reference, { maxQueryChars: MAX_REPORT_QUERY_CHARS, - deadlineMs: FIRST_LOGIN_SCAN_DEADLINE_MS, + deadlineMs, signal - }); + }), signal, deadlineMs); if (loaded?.messages.length) { return loaded; } @@ -656,24 +663,35 @@ async function sampleSamplerWithinDeadline( sampler: OnboardingInsightSampler, signal: AbortSignal ): Promise { - if (signal.aborted) { + return readWithinScanDeadline(() => sampleSampler(sampler, signal), signal, FIRST_LOGIN_SCAN_DEADLINE_MS); +} + +async function readWithinScanDeadline( + read: () => Promise, + signal: AbortSignal, + deadlineMs: number +): Promise { + if (signal.aborted || deadlineMs <= 0) { return null; } + const deadlineAt = Date.now() + deadlineMs; let timeoutId: ReturnType | undefined; let abortHandler: (() => void) | undefined; const timeout = new Promise((resolve) => { const finish = () => resolve(null); abortHandler = finish; - timeoutId = setTimeout(finish, FIRST_LOGIN_SCAN_DEADLINE_MS); + timeoutId = setTimeout(finish, deadlineMs); signal.addEventListener("abort", finish, { once: true }); }); try { - return await Promise.race([ - sampleSampler(sampler, signal), + const result = await Promise.race([ + read(), timeout ]); + // A synchronous read or a long microtask chain can delay the timer itself. + return signal.aborted || Date.now() >= deadlineAt ? null : result; } finally { if (timeoutId) { clearTimeout(timeoutId); @@ -773,7 +791,7 @@ async function buildReportResponse(input: { if (input.sample.queries.length === 0) { return { status: "ready", - reportMarkdown: renderEmptyHistoryReport(input.locale, input.sample), + reportMarkdown: renderEmptyHistoryReport(input.locale), diagnostics: diagnostics(input.sample, false, Math.max(0, input.now() - input.startedAt), input.locale) }; } @@ -818,7 +836,7 @@ async function* streamReportResponse(input: { type: "done", response: { status: "ready", - reportMarkdown: renderEmptyHistoryReport(input.locale, input.sample), + reportMarkdown: renderEmptyHistoryReport(input.locale), diagnostics: diagnostics(input.sample, false, input.elapsedMs, input.locale) } }; @@ -888,25 +906,21 @@ function renderFallbackReport( return locale === "en-US" ? renderEnglishReport(profile, sample) : renderChineseReport(profile, sample); } -function renderEmptyHistoryReport(locale: "zh-CN" | "en-US", sample: SampleBundle): string { - const agentNames = sample.discovered.map((agent) => agent.displayName); - if (agentNames.length > 0) { - const names = agentNames.join(", "); - return locale === "en-US" ? [ - `Memmy found ${names} on this device, but the quick first scan did not return readable conversation history.`, - "Once you use Memmy with a real task, it will preserve the useful background, decisions, and next step for future conversations and other Agents." - ].join("\n\n") : [ - `Memmy 已识别到这台设备上的 ${names},但首次轻量扫描暂时没有读到可用的对话历史。`, - "之后用 Memmy 处理真实任务时,它会记住有用的背景、决策和下一步,方便新对话或其他 Agent 继续。" - ].join("\n\n"); - } +function hasConcreteRecentRequest( + profile: Pick, + conversation: { messages: ReadonlyArray<{ role: "user" | "assistant" | "tool"; text: string }> } | null +): boolean { + return profile.taskCandidates.some((task) => task.score >= 3) || + Boolean(conversation?.messages.some((message) => message.role === "user" && CONCRETE_REQUEST_PATTERN.test(message.text))); +} +function renderEmptyHistoryReport(locale: "zh-CN" | "en-US"): string { return locale === "en-US" ? [ - "There is no readable Agent history on this device yet, so there is nothing useful to pretend I already know.", - "Tell Memmy about one real task. It will preserve the useful background, decisions, and next step so a new conversation—or another Agent such as Cursor or Codex—can continue without making you explain it again." + "Hi, I'm Memmy. I can help you think through questions, organize information and plans, and keep useful context so you can pick up a conversation later or with another AI.", + "I couldn't find enough from this scan to get a sense of what you've been up to. We can start with what you're working on, or just chat and get to know each other." ].join("\n\n") : [ - "这台设备上还没有可读取的 Agent 历史,所以我不会假装已经了解你。", - "先告诉 Memmy 一件你正在做的真实任务。它会记住有用的背景、决策和下一步;之后新开对话,或换到 Cursor、Codex,也不用再从头解释。" + "嗨,我是 Memmy。我可以陪你讨论问题、整理信息和计划,也能记住有用的背景,方便之后或换个 AI 时接着聊。", + "我没能通过扫描获得足够的线索来了解你的近况。你可以从正在做的事说起,也可以随便聊聊,我们慢慢熟悉。" ].join("\n\n"); } @@ -1140,6 +1154,20 @@ function buildFallbackTaskContext(input: OnboardingInsightGenerationInput): Onbo const conversation = input.sample.latestConversation; const messages = conversation?.messages ?? []; const userMessages = messages.filter((message) => message.role === "user"); + if (!hasConcreteRecentRequest(input.profile, conversation)) { + return { + topic: "", + userGoal: "", + latestRequest: summarizeContextMessage(userMessages.at(-1)?.text ?? "", 240), + status: "uncertain", + currentState: "", + agentActions: [], + verifiedResults: [], + unresolvedItems: [], + continuationPoint: "", + trajectorySummary: "" + }; + } const assistantMessages = messages.filter((message) => message.role === "assistant"); const toolMessages = messages.filter((message) => message.role === "tool"); const firstUser = userMessages[0]?.text ?? ""; @@ -1388,61 +1416,65 @@ function isGeneratedMarkerPrefix(value: string, markers: readonly string[]): boo } function renderChineseReport(profile: OnboardingInsightProfileSignals, sample: SampleBundle): string { - const lines: string[] = []; - const nameLine = renderChineseNameLine(profile.nameHints); - if (nameLine) { - lines.push(nameLine); - } + const lines = [`${renderChineseNameLine(profile.nameHints)}我是 Memmy。我可以陪你讨论问题、整理信息和计划,也能把有用的对话背景留给下次或其他 AI 接着用。`]; const preferenceLines = [ renderContextLanguagePreference(profile, "zh-CN"), ...profile.userInsights.slice(0, 4).map((insight) => insight.textZh) ].filter((line): line is string => Boolean(line)); - lines.push(`## 你的偏好\n${preferenceLines.length > 0 ? preferenceLines.map((line) => `- ${line}`).join("\n") : "目前只有少量用户表达,我还不会替你下偏好结论。"}`); + if (preferenceLines.length > 0) { + lines.push(`## 你的偏好\n${preferenceLines.map((line) => `- ${line}`).join("\n")}`); + } const conversation = sample.latestConversation; + if (!hasConcreteRecentRequest(profile, conversation) || !conversation) { + lines.push("你可以直接从眼前想聊的事开始,我会跟着你的节奏来。"); + return lines.join("\n\n"); + } const context = buildFallbackTaskContext({ locale: "zh-CN", profile, sample: toSampleSummary(sample) }); const memoryLines = [ - conversation ? `最近一次会话来自 ${conversation.displayName}${conversation.workspacePath ? `,项目路径是 ${conversation.workspacePath}` : ""}。` : null, - context.userGoal ? `用户目标:${context.userGoal}` : null, - context.currentState ? `当前状态:${context.currentState}` : null, - context.agentActions.length > 0 ? `Agent 已做:${context.agentActions.join(";")}` : null, - context.verifiedResults.length > 0 ? `已验证结果:${context.verifiedResults.join(";")}` : "目前没有明确的验证结果。", + `最近一次会话来自 ${conversation.displayName}${conversation.workspacePath ? `,相关路径是 ${conversation.workspacePath}` : ""}。`, + context.userGoal ? `你当时想做的是:${context.userGoal}` : null, + context.currentState ? `对话里最近提到:${context.currentState}` : null, context.unresolvedItems.length > 0 ? `仍待处理:${context.unresolvedItems.join(";")}` : null ].filter((line): line is string => Boolean(line)); - lines.push(`## 最近项目记忆\n${memoryLines.join("\n\n")}`); + lines.push(`## 最近聊到的事\n${memoryLines.join("\n\n")}`); - lines.push(`## 接下来可以做\n${context.continuationPoint ? `1. ${context.continuationPoint}` : "当前记录中没有明确的未完成待办。"}`); + if (context.continuationPoint) { + lines.push(`## 如果想接着做\n${context.continuationPoint}`); + } return lines.join("\n\n"); } function renderEnglishReport(profile: OnboardingInsightProfileSignals, sample: SampleBundle): string { - const lines: string[] = []; - const nameLine = renderEnglishNameLine(profile.nameHints); - if (nameLine) { - lines.push(nameLine); - } + const lines = [`${renderEnglishNameLine(profile.nameHints)}I'm Memmy. I can help you think through questions, organize information and plans, and carry useful context into another conversation or AI tool.`]; const preferenceLines = [ renderContextLanguagePreference(profile, "en-US"), ...profile.userInsights.slice(0, 4).map((insight) => insight.textEn) ].filter((line): line is string => Boolean(line)); - lines.push(`## Your preferences\n${preferenceLines.length > 0 ? preferenceLines.map((line) => `- ${line}`).join("\n") : "I only have a few user-authored signals, so I will not overstate your preferences yet."}`); + if (preferenceLines.length > 0) { + lines.push(`## Your preferences\n${preferenceLines.map((line) => `- ${line}`).join("\n")}`); + } const conversation = sample.latestConversation; + if (!hasConcreteRecentRequest(profile, conversation) || !conversation) { + lines.push("We can start with whatever is on your mind, and take it from there."); + return lines.join("\n\n"); + } const context = buildFallbackTaskContext({ locale: "en-US", profile, sample: toSampleSummary(sample) }); const memoryLines = [ - conversation ? `Your newest conversation is from ${conversation.displayName}${conversation.workspacePath ? `, at ${conversation.workspacePath}` : ""}.` : null, - context.userGoal ? `User goal: ${context.userGoal}` : null, - context.currentState ? `Current state: ${context.currentState}` : null, - context.agentActions.length > 0 ? `Agent actions: ${context.agentActions.join("; ")}` : null, - context.verifiedResults.length > 0 ? `Verified results: ${context.verifiedResults.join("; ")}` : "There is no explicit verified result yet.", + `Your latest conversation was in ${conversation.displayName}${conversation.workspacePath ? `, with context at ${conversation.workspacePath}` : ""}.`, + context.userGoal ? `You were working on: ${context.userGoal}` : null, + context.currentState ? `The conversation last mentioned: ${context.currentState}` : null, context.unresolvedItems.length > 0 ? `Still unresolved: ${context.unresolvedItems.join("; ")}` : null ].filter((line): line is string => Boolean(line)); - lines.push(`## Latest project memory\n${memoryLines.join("\n\n")}`); + lines.push(`## What we recently discussed\n${memoryLines.join("\n\n")}`); - lines.push(`## What to do next\n${context.continuationPoint ? `1. ${context.continuationPoint}` : "There is no explicit unfinished action in the current record."}`); + if (context.continuationPoint) { + lines.push(`## If you want to continue\n${context.continuationPoint}`); + } return lines.join("\n\n"); } @@ -1450,62 +1482,30 @@ function renderEnglishReport(profile: OnboardingInsightProfileSignals, sample: S function renderChineseNameLine(hints: NameHints): string | null { const name = selectFallbackNameSignal(hints); if (!name) { - return "Hi,我还没看到你明确提过名字,所以先不乱称呼你。"; + return "嗨,"; } const displayName = formatNameForGreeting(name.value); - if (name.kind === "self_declared") { - return `Hi ${displayName},我从对话里看到你这样介绍过自己。`; - } - return `Hi ${displayName},我先按本机线索这样称呼你;如果不对,告诉我就好。`; + return `嗨 ${displayName},`; } function renderEnglishNameLine(hints: NameHints): string | null { const name = selectFallbackNameSignal(hints); if (!name) { - return "Hi, I have not seen a clear name from you yet, so I will not guess one."; + return "Hi, "; } const displayName = formatNameForGreeting(name.value); - if (name.kind === "self_declared") { - return `Hi ${displayName}, I saw you introduce yourself this way in the conversation.`; - } - return `Hi ${displayName}, I am using the local account hint for now; tell me if I should call you something else.`; + return `Hi ${displayName}, `; } function selectFallbackNameSignal(hints: NameHints): NameSignal | null { - if (hints.homePathName && !isGenericAccountName(hints.homePathName)) { - return { - value: hints.homePathName, - source: hints.homeAndComputerMatch ? "~ 路径与电脑用户名一致" : "~ 路径", - kind: "local_account" - }; - } - const selfDeclaredName = hints.selfDeclaredNames.find((name) => !isGenericAccountName(name)) ?? hints.selfDeclaredNames[0]; if (selfDeclaredName) { return { value: selfDeclaredName, source: "query 自称", kind: "self_declared" }; } - - if (hints.computerUserName && !isGenericAccountName(hints.computerUserName)) { - return { value: hints.computerUserName, source: "电脑用户名", kind: "local_account" }; - } - return null; } -function buildNameDecisionRequirement(profile: OnboardingInsightProfileSignals, locale: "zh-CN" | "en-US") { - return { - mustInferDisplayName: true, - mustIncludeDisplayNameInFirstSentence: true, - defaultPriority: "homePathName", - genericAccountNames: profile.nameHints.genericAccountNames, - locale, - openingPattern: locale === "en-US" - ? "Hi , ..." - : "Hi ,..." - }; -} - function formatNameForGreeting(value: string): string { const trimmed = value.trim(); if (/^[a-z][a-z0-9_.-]*$/i.test(trimmed) && !/\p{Script=Han}/u.test(trimmed)) { @@ -1916,34 +1916,30 @@ function buildLlmMessages(input: OnboardingInsightGenerationInput): Array<{ role { role: "system", content: [ - "你是 Memmy 首次登录初见报告撰写者。报告要像一位刚接手工作的可靠搭档:私人化、具体、克制,不是技术日志,也不是营销文案。", + "你是 Memmy。正在回答用户的第一句话,先自然介绍你能帮忙做什么,再根据已有线索展示你能如何帮这位用户。语气像刚认识的可靠搭档,不写技术日志或营销文案。", + "你可以陪用户讨论问题、整理信息、总结近期对话和待办,并保留有用的背景供新对话或其他 Agent 接续。不要把能力限定为编程或修 Bug。", "只依据输入里的明确证据,不要编造项目、完成情况、错误原因或用户偏好。证据不足时直接说明尚不能确认。", "不要把 diagnostics 写给用户,不要出现“轻量样本、采样、query 数、discoveredAgentCount”等实现细节。", - "你必须根据 profile.nameHints 综合判断用户可能希望被怎么称呼。nameHints.selfDeclaredNames 来自扫描到的用户自称,homePathName 是 home 路径最后一段,computerUserName 是电脑用户名,homeAndComputerMatch 表示 home 路径名和电脑用户名一致。", - "名字判断默认优先使用 homePathName,因为用户一定有 home 路径;不要因为 selfDeclaredNames 为空就省略称呼。只有当 homePathName 是 admin、administrator、root、ubuntu、user、test、guest、default、runner、ec2-user 这类泛化账号名,或明显不是可称呼名字时,才降低它的优先级。", - "selfDeclaredNames 和 computerUserName 是辅助判断线索:如果 selfDeclaredNames 有明确人名,可以结合它修正称呼;如果 homePathName 与 computerUserName 一致,说明本机线索更可信。", - "第一句必须包含你判断出的具体称呼:中文报告以“Hi <称呼>,”开头,英文报告以“Hi , ”开头。不得省略名字,不得把名字替换成“这个线索”“这个称呼”“X”等占位词。", - "严禁出现“本机账号显示为”“本机用户名/路径名显示为”“local username/path shows”“我检测到你的用户名”这类工程口径。", - "不要向用户暴露 nameHints、homePathName、computerUserName 这些字段名或来源;如果本机线索只是临时称呼,要用柔和语气表达“如果不对,告诉我就好”。中英文混合名不要使用。", - "输出中文或英文由 locale 决定。profile.preferredResponseLanguage 来自近期用户请求的主语言统计;有值时要自然说明用户最近更常用中文还是英文。", + "只有 profile.nameHints.selfDeclaredNames 中存在可信的用户自称时才使用名字;否则直接打招呼,不猜本机用户名或路径名。中英文混合或明显不适合称呼的名字也不要使用。", + "输出中文或英文由 locale 决定。profile.preferredResponseLanguage 来自近期用户请求的主语言统计;有充分证据时可自然提及语言习惯,不必专门报告统计结果。", "偏好结论的唯一原始证据是 preferenceEvidence 中由用户本人发送的消息,profile 里的偏好字段也只是这些用户消息的结构化归纳。仅总结用户明确表达或在多个请求中稳定体现的偏好;不要把旧任务内容本身当成偏好,也严禁使用 assistant、tool 或 latestConversation 推断偏好。", - "latestConversation 是所有已扫描 Agent 中时间最新的一个会话,只允许依据这个会话总结最近项目、任务、Bug 或关键词及其当前进度。", + "latestConversation 是所有已扫描 Agent 中时间最新的一个会话。它可能是项目任务,也可能只是普通交流;只允许依据这个会话说明最近聊到的事及其进度。", "latestConversation.messages 已按首 2 个和尾 12 个对话轮次截取。user 是用户请求,assistant 是 Agent 回复,tool 是脱敏后的简短工具执行信息。", "区分三类进度证据:用户要求做什么、Agent 表示做了什么、工具结果实际验证了什么。只有明确成功的 tool 结果才能写成已验证;只有 assistant 自述时应写成“Agent 表示/对话中提到”,不能当成确定事实。", - "把 latestConversation 看作一条随时间演进的任务轨迹:合并重复要求,保留关键转折,并让较新的决定、修复和验证覆盖较早的猜测、失败或阻塞。不要逐条复述消息,不要照抄工具日志。", - "正文必须包含三个 Markdown 小节:『你的偏好』『最近项目记忆』『接下来可以做』。可以使用短段落和列表,不要使用表格或代码块。", - "『你的偏好』只总结用户本人有证据支持的语言、沟通方式、输出形式、方案取舍、实现约束或验证要求,最多 3-5 条;不要混入项目进度、Agent 行为、工具结果或空泛性格标签。", - "『最近项目记忆』说明最新会话来自哪个 Agent、用户目标、已做事项、已验证结果、当前状态、仍待处理内容。workspacePath 有值时必须写清项目具体路径。只写当前有效结论,不展开冗长历史。", - "『接下来可以做』只列证据支持且尚未完成的 0-3 条待办,按执行顺序排列。第一条应是当前最小且可立即执行的下一步;任务已完成或没有明确待办时,直接说明暂时没有明确待办,不要补通用建议。", - "正文长度要求:中文 300-500 字,英文 180-300 words。重点是准确提炼最近一个项目现场,不要扩展成跨项目年度总结。", + "如果 latestConversation 是连续事项,把它看作随时间演进的轨迹:合并重复要求,保留关键转折,并让较新的决定、修复和验证覆盖较早的猜测、失败或阻塞。不要逐条复述消息,不要照抄工具日志。", + "如果有明确的近期事项,可以沿用『你的偏好』『最近聊到的事』『接下来可以做』等简短小节,梳理背景、进度和待办;没有证据的偏好或未完成事项就省略对应小节,不要为了报告格式硬凑。", + "『你的偏好』最多 3 条,只写用户本人有证据支持的语言、沟通方式或工作要求;不要混入项目进度、Agent 行为、工具结果或空泛性格标签。", + "『最近聊到的事』说明最新会话来自哪个 Agent、用户目标、当前有效进度。workspacePath 有值且确实关联该事项时可写路径;不要假定每个人都有项目或 Bug。", + "『接下来可以做』只列证据支持且尚未完成的 0-3 条待办,按执行顺序排列;任务已完成或只是普通交流时,不生成待办。", + "正文中文一般 120-300 字,英文一般 80-180 words;有具体任务可稍长,普通交流保持简短。先回答能帮什么,再呈现有依据的线索。", "报告正文只允许使用 Markdown,不得包含任何原始 HTML 标签或样式。不要输出思考过程、执行计划、要求确认、Prompt 复述或起草说明。", "你必须一次输出两个区块,严格使用以下顺序和标签;标签前后不要添加其他文字:", `${GENERATED_REPORT_OPEN}\n这里放给用户看的 Markdown 报告正文\n${GENERATED_REPORT_CLOSE}`, `${GENERATED_TASK_CONTEXT_OPEN}\n这里放一个合法 JSON 对象\n${GENERATED_TASK_CONTEXT_CLOSE}`, "任务上下文 JSON 必须包含且只需包含:topic、userGoal、latestRequest、status、currentState、agentActions、verifiedResults、unresolvedItems、continuationPoint、trajectorySummary。status 只能是 pending、active、waiting、completed、uncertain;后三个集合字段必须是字符串数组。", - "任务上下文使用 locale 对应语言,面向任意类型任务,不要使用仅适合 Coding 的固定分类。它只总结最新任务,不得包含用户偏好,也不得复制原始 query、assistant 回复或工具流水。agentActions 写 Agent 已采取的动作,verifiedResults 只写有结果证据支持的结论,unresolvedItems 只写仍然有效的问题,continuationPoint 写其他 Agent 接手时应从哪里继续。", - "trajectorySummary 用一个紧凑段落总结:用户目标如何演进、Agent 做了什么、得到什么结果、现在停在哪里。最终状态优先;已经被后续解决的问题不能继续写成当前阻塞。", - "任务上下文要短:JSON 必须单行输出、不要缩进、不要代码块;每个普通字段最多一句,三个数组各最多 3 项,trajectorySummary 中文 80-160 字或英文 60-100 words;不要为了填满字段而重复同一事实。", + "任务上下文使用 locale 对应语言,面向任意类型事项,不要使用仅适合 Coding 的固定分类。它只总结最新会话,不得包含用户偏好,也不得复制原始 query、assistant 回复或工具流水。普通交流或缺少任务证据时,status 用 uncertain,任务字段和数组可为空。agentActions 写 Agent 已采取的动作,verifiedResults 只写有结果证据支持的结论,unresolvedItems 只写仍然有效的问题,continuationPoint 只写有证据支持的接续位置。", + "有明确事项时,trajectorySummary 用一个紧凑段落总结目标如何演进、Agent 做了什么、得到什么结果、现在停在哪里;最终状态优先。普通交流时可留空,不要编造任务轨迹。", + "任务上下文要短:JSON 必须单行输出、不要缩进、不要代码块;每个普通字段最多一句,三个数组各最多 3 项,不要为了填满字段而重复同一事实。", "报告正文不要生成按钮、行动卡片、CTA、JSON、Markdown 代码块或表格;任务上下文区块只放 JSON 对象。不要暴露任何密钥。" ].join("\n") }, @@ -1951,22 +1947,21 @@ function buildLlmMessages(input: OnboardingInsightGenerationInput): Array<{ role role: "user", content: JSON.stringify({ locale: input.locale, + openingUserMessage: FIRST_ENCOUNTER_USER_MESSAGE[input.locale], reportGoal: { - primary: "user_preferences_latest_project_memory_and_actionable_todos", + primary: "answer_the_opening_question_with_capabilities_and_evidence_based_context", lengthConstraint: input.locale === "zh-CN" - ? "300-500 Chinese characters" - : "180-300 English words", + ? "usually 120-300 Chinese characters" + : "usually 80-180 English words", requiredSections: [ - "opening_with_name_or_safe_greeting", - "user_preferences", - "latest_project_memory", - "ordered_actionable_todos" + "friendly_greeting_and_capabilities", + "relevant_observations_when_supported", + "recent_context_and_open_todos_when_supported" ], focus: [ - "用户有哪些有证据支持的稳定偏好", - "全局最新会话对应什么项目、任务、Bug 或关键词", - "用户要求、Agent 自述和工具验证分别说明了什么进度", - "接下来尚未完成且最可行的 0-3 个待办是什么" + "直接回答 openingUserMessage:Memmy 能为用户做什么", + "有证据时说明对这位用户已了解到什么", + "有明确近期事项时梳理进度与未完成待办;否则自然打招呼" ], outputEnvelope: { reportTag: GENERATED_REPORT_OPEN, @@ -1986,7 +1981,6 @@ function buildLlmMessages(input: OnboardingInsightGenerationInput): Array<{ role } }, profile: toLlmProfile(input.profile, input.sample.activeAgents), - nameDecisionRequirement: buildNameDecisionRequirement(input.profile, input.locale), activeAgents: input.sample.activeAgents, preferenceEvidence: input.sample.queries, latestConversation: input.sample.latestConversation @@ -2002,6 +1996,7 @@ function toLlmProfile( const agentNames = new Map(activeAgents.map((agent) => [agent.sourceId, agent.displayName])); return { ...profile, + nameHints: { selfDeclaredNames: profile.nameHints.selfDeclaredNames }, taskCandidates: profile.taskCandidates.map((task) => ({ title: task.title, summary: task.summary, diff --git a/App/backend/src/services/tests/byok-agent-token-usage-service.test.ts b/App/backend/src/services/tests/byok-agent-token-usage-service.test.ts index cd536b683..2daa251c8 100644 --- a/App/backend/src/services/tests/byok-agent-token-usage-service.test.ts +++ b/App/backend/src/services/tests/byok-agent-token-usage-service.test.ts @@ -1,5 +1,5 @@ import { describe, expect, it, vi } from "vitest"; -import type { ByokTokenUsageEvent } from "@memmy/local-api-contracts"; +import type { ByokTokenUsageEvent, MemoryTokenBudgetDto } from "@memmy/local-api-contracts"; import { createByokTokenUsageService } from "../byok-token-usage-service.js"; describe("ByokTokenUsageService", () => { @@ -117,7 +117,7 @@ describe("ByokTokenUsageService", () => { }); }); - it("prefers Memory usage for the UI budget and keeps the last snapshot if Memory drops", async () => { + it("keeps Memory's larger pending usage when App has not persisted it and Memory drops", async () => { const repository = { recordEvent: vi.fn(), getSummary: vi.fn(), @@ -156,9 +156,10 @@ describe("ByokTokenUsageService", () => { dailyUsed: 10_001_000, lifetimeUsed: 10_001_000, paused: true, - trigger: "daily" + trigger: "daily", + stale: true }); - expect(repository.getMemoryPipelineUsage).not.toHaveBeenCalled(); + expect(repository.getMemoryPipelineUsage).toHaveBeenCalledTimes(2); }); it("keeps Memory runtime limits and pause when App settings have already changed", async () => { @@ -227,7 +228,7 @@ describe("ByokTokenUsageService", () => { }); await expect(service.getMemoryBudget()).resolves.toMatchObject({ - dailyUsed: 10_000_000, + dailyUsed: 11_000_000, lifetimeUsed: 23_000_000, paused: true, trigger: "daily" @@ -260,7 +261,8 @@ describe("ByokTokenUsageService", () => { now: () => new Date(2026, 8, 18, 12) }); - await expect(service.getMemoryBudget()).resolves.toMatchObject({ + const budget = await service.getMemoryBudget(); + expect(budget).toMatchObject({ dailyLimitM: 10, totalLimitM: 500, dailyUsed: 10_000_000, @@ -268,10 +270,211 @@ describe("ByokTokenUsageService", () => { paused: true, trigger: "daily" }); + expect(budget.stale).toBeUndefined(); expect(repository.getMemoryPipelineUsage).toHaveBeenCalledWith(new Date(2026, 8, 18).toISOString()); }); + + it("includes a newly recorded History summary event before Memory reconciles it", async () => { + const usage = { dailyUsed: 9_999_970, lifetimeUsed: 25_000_000 }; + const repository = { + recordEvent: vi.fn((event: ByokTokenUsageEvent) => { + usage.dailyUsed += event.totalTokens; + usage.lifetimeUsed += event.totalTokens; + }), + getSummary: vi.fn(), + getMemoryPipelineUsage: vi.fn(() => ({ ...usage })) + }; + const service = createByokTokenUsageService({ + repository, + bootstrapRepository: { getAppSettings: () => ({ memoryByokDailyLimitM: 10, memoryByokTotalLimitM: 500 } as never) }, + memoryClient: { getMemoryTokenBudget: vi.fn(async () => memoryBudgetFixture({ + dailyUsed: 9_999_970, + lifetimeUsed: 25_000_000 + })) }, + now: () => new Date(2026, 8, 18, 12) + }); + + await expect(service.getMemoryBudget()).resolves.toMatchObject({ paused: false, trigger: null }); + await service.recordEvent({ + ...eventFixture(), + kind: "memory_summary", + capability: "memory_summary", + operationId: "history-segment-1", + metadata: { operation: "history.summary" }, + createdAt: new Date(2026, 8, 18, 12).toISOString() + }); + await expect(service.getMemoryBudget()).resolves.toMatchObject({ + dailyUsed: 10_000_000, + lifetimeUsed: 25_000_030, + paused: true, + trigger: "daily" + }); + }); + + it.each([ + { dailyUsed: 10_000_000, lifetimeUsed: 20_000_000, trigger: "daily" }, + { dailyUsed: 100, lifetimeUsed: 500_000_000, trigger: "total" }, + { dailyUsed: 10_000_000, lifetimeUsed: 500_000_000, trigger: "total" } + ])("re-evaluates the merged budget when App events reach $trigger limits", async ({ dailyUsed, lifetimeUsed, trigger }) => { + const service = createByokTokenUsageService({ + repository: { + recordEvent: vi.fn(), getSummary: vi.fn(), + getMemoryPipelineUsage: vi.fn(() => ({ dailyUsed, lifetimeUsed })) + }, + bootstrapRepository: { getAppSettings: () => ({ memoryByokDailyLimitM: 10, memoryByokTotalLimitM: 500 } as never) }, + memoryClient: { getMemoryTokenBudget: vi.fn(async () => memoryBudgetFixture()) }, + now: () => new Date(2026, 8, 18, 12) + }); + + await expect(service.getMemoryBudget()).resolves.toMatchObject({ dailyUsed, lifetimeUsed, paused: true, trigger }); + }); + + it("uses the latest runtime limits and permits unlimited zero limits after merging usage", async () => { + const service = createByokTokenUsageService({ + repository: { + recordEvent: vi.fn(), getSummary: vi.fn(), + getMemoryPipelineUsage: vi.fn(() => ({ dailyUsed: 12_000_000, lifetimeUsed: 600_000_000 })) + }, + bootstrapRepository: { getAppSettings: () => ({ memoryByokDailyLimitM: 10, memoryByokTotalLimitM: 500 } as never) }, + memoryClient: { getMemoryTokenBudget: vi.fn() + .mockResolvedValueOnce(memoryBudgetFixture()) + .mockResolvedValueOnce(memoryBudgetFixture({ dailyLimitM: 0, totalLimitM: 0 })) }, + now: () => new Date(2026, 8, 18, 12) + }); + + await expect(service.getMemoryBudget()).resolves.toMatchObject({ paused: true, trigger: "total" }); + await expect(service.getMemoryBudget()).resolves.toMatchObject({ + dailyLimitM: 0, totalLimitM: 0, dailyUsed: 12_000_000, lifetimeUsed: 600_000_000, + paused: false, trigger: null + }); + }); + + it("does not add overlapping persisted totals to Memory's larger pending ledger", async () => { + const service = createByokTokenUsageService({ + repository: { + recordEvent: vi.fn(), getSummary: vi.fn(), + getMemoryPipelineUsage: vi.fn(() => ({ dailyUsed: 6_000_000, lifetimeUsed: 300_000_000 })) + }, + bootstrapRepository: { getAppSettings: () => ({ memoryByokDailyLimitM: 10, memoryByokTotalLimitM: 500 } as never) }, + memoryClient: { getMemoryTokenBudget: vi.fn(async () => memoryBudgetFixture({ + dailyUsed: 7_000_000, lifetimeUsed: 400_000_000 + })) }, + now: () => new Date(2026, 8, 18, 12) + }); + + await expect(service.getMemoryBudget()).resolves.toMatchObject({ + dailyUsed: 7_000_000, lifetimeUsed: 400_000_000, paused: false, trigger: null + }); + }); + + it("merges new App events into a cached same-day snapshot while Memory is unavailable", async () => { + let dailyUsed = 500; + const service = createByokTokenUsageService({ + repository: { + recordEvent: vi.fn(), getSummary: vi.fn(), + getMemoryPipelineUsage: vi.fn(() => ({ dailyUsed, lifetimeUsed: dailyUsed })) + }, + bootstrapRepository: { getAppSettings: () => ({ memoryByokDailyLimitM: 10, memoryByokTotalLimitM: 500 } as never) }, + memoryClient: { getMemoryTokenBudget: vi.fn() + .mockResolvedValueOnce(memoryBudgetFixture({ dailyUsed: 500, lifetimeUsed: 500 })) + .mockRejectedValueOnce(new Error("memory unavailable")) }, + now: () => new Date(2026, 8, 18, 12) + }); + + await expect(service.getMemoryBudget()).resolves.toMatchObject({ paused: false }); + dailyUsed = 10_000_000; + await expect(service.getMemoryBudget()).resolves.toMatchObject({ + dailyUsed, lifetimeUsed: dailyUsed, paused: true, trigger: "daily", stale: true + }); + }); + + it("marks an initial Memory failure stale and clears it after Memory recovers", async () => { + const service = createByokTokenUsageService({ + repository: { + recordEvent: vi.fn(), getSummary: vi.fn(), + getMemoryPipelineUsage: vi.fn(() => ({ dailyUsed: 500, lifetimeUsed: 5_000 })) + }, + bootstrapRepository: { getAppSettings: () => ({ memoryByokDailyLimitM: 20, memoryByokTotalLimitM: 0 } as never) }, + memoryClient: { getMemoryTokenBudget: vi.fn() + .mockRejectedValueOnce(new Error("memory unavailable")) + .mockResolvedValueOnce(memoryBudgetFixture({ dailyUsed: 1_000, lifetimeUsed: 10_000 })) }, + now: () => new Date(2026, 8, 18, 12) + }); + + await expect(service.getMemoryBudget()).resolves.toMatchObject({ + dailyLimitM: 20, totalLimitM: 0, dailyUsed: 500, lifetimeUsed: 5_000, + paused: false, trigger: null, stale: true + }); + const recovered = await service.getMemoryBudget(); + expect(recovered).toMatchObject({ + dailyLimitM: 10, totalLimitM: 500, dailyUsed: 1_000, lifetimeUsed: 10_000, + paused: false, trigger: null + }); + expect(recovered.stale).toBeUndefined(); + }); + + it("retains Memory's active pause and stale marker when merging smaller App usage", async () => { + const service = createByokTokenUsageService({ + repository: { + recordEvent: vi.fn(), getSummary: vi.fn(), + getMemoryPipelineUsage: vi.fn(() => ({ dailyUsed: 100, lifetimeUsed: 100 })) + }, + bootstrapRepository: { getAppSettings: () => ({ memoryByokDailyLimitM: 10, memoryByokTotalLimitM: 500 } as never) }, + memoryClient: { getMemoryTokenBudget: vi.fn(async () => memoryBudgetFixture({ + dailyUsed: 100, lifetimeUsed: 100, paused: true, trigger: "daily", stale: true + })) }, + now: () => new Date(2026, 8, 18, 12) + }); + + await expect(service.getMemoryBudget()).resolves.toMatchObject({ + dailyUsed: 100, lifetimeUsed: 100, paused: true, trigger: "daily", stale: true + }); + }); + + it.each([ + { totalLimitM: 500, paused: false, trigger: null }, + { totalLimitM: 20, paused: true, trigger: "total" } + ])("rolls cached daily usage at local midnight while retaining lifetime usage ($trigger)", async ({ totalLimitM, paused, trigger }) => { + let current = new Date(2026, 8, 18, 23, 59); + const repository = { + recordEvent: vi.fn(), getSummary: vi.fn(), + getMemoryPipelineUsage: vi.fn(() => ({ dailyUsed: 0, lifetimeUsed: 21_000_000 })) + }; + const service = createByokTokenUsageService({ + repository, + bootstrapRepository: { getAppSettings: () => ({ memoryByokDailyLimitM: 10, memoryByokTotalLimitM: 500 } as never) }, + memoryClient: { getMemoryTokenBudget: vi.fn() + .mockResolvedValueOnce(memoryBudgetFixture({ + totalLimitM, dailyUsed: 10_000_000, lifetimeUsed: 23_000_000, paused: true, + trigger: totalLimitM === 20 ? "total" : "daily" + })) + .mockRejectedValueOnce(new Error("memory unavailable")) }, + now: () => current + }); + + await service.getMemoryBudget(); + current = new Date(2026, 8, 19, 0, 1); + await expect(service.getMemoryBudget()).resolves.toMatchObject({ + dailyUsed: 0, lifetimeUsed: 23_000_000, paused, trigger, stale: true, + nextLocalMidnightAt: new Date(2026, 8, 20).toISOString() + }); + expect(repository.getMemoryPipelineUsage).toHaveBeenLastCalledWith(new Date(2026, 8, 19).toISOString()); + }); }); +function memoryBudgetFixture(overrides: Partial = {}): MemoryTokenBudgetDto { + return { + dailyLimitM: 10, + totalLimitM: 500, + dailyUsed: 0, + lifetimeUsed: 0, + paused: false, + trigger: null, + nextLocalMidnightAt: new Date(2026, 8, 19).toISOString(), + ...overrides + }; +} + function eventFixture(): ByokTokenUsageEvent { return { id: "event-1", diff --git a/App/backend/src/services/tests/error-envelope.test.ts b/App/backend/src/services/tests/error-envelope.test.ts index bfde1a9d8..7a3f9fd6f 100644 --- a/App/backend/src/services/tests/error-envelope.test.ts +++ b/App/backend/src/services/tests/error-envelope.test.ts @@ -1,7 +1,7 @@ /** Error envelope tests. */ import Fastify from "fastify"; import { afterEach, describe, expect, it } from "vitest"; -import { MemoryLayerError } from "../../adapters/outbound/memory-client/index.js"; +import { MemoryLayerError, MemoryLayerNetworkError } from "../../adapters/outbound/memory-client/index.js"; import { API_ERROR_CODES, HTTP_STATUS_BY_CODE, withErrorEnvelope } from "../error-envelope.js"; import type { FastifyInstance } from "fastify"; @@ -13,6 +13,17 @@ afterEach(async () => { }); describe("withErrorEnvelope", () => { + it("reports a failed Memory connection as recoverable service unavailability", async () => { + app = Fastify({ logger: false }); + app.get("/health", withErrorEnvelope(async () => { + throw new MemoryLayerNetworkError(new Error("connection reset")); + })); + const response = await app.inject({ method: "GET", url: "/health" }); + expect(response.statusCode).toBe(503); + expect(response.json()).toMatchObject({ error: { + code: "memory_layer_unavailable", message: "memory layer network error" + } }); + }); it("maps every ApiErrorCode to the configured HTTP status", async () => { for (const code of API_ERROR_CODES) { app = Fastify({ logger: false }); diff --git a/App/backend/src/services/tests/native-persistent-scan.test.ts b/App/backend/src/services/tests/native-persistent-scan.test.ts index 6802eb245..09450a08c 100644 --- a/App/backend/src/services/tests/native-persistent-scan.test.ts +++ b/App/backend/src/services/tests/native-persistent-scan.test.ts @@ -90,7 +90,85 @@ describe("persistent Codex scan", () => { })); expect(repository.getConversationCheckpoint("codex", "complete-session")).not.toBeNull(); expect(repository.getConversationCheckpoint("codex", "incomplete-session")).toBeNull(); - expect(repository.getScanWatermark("codex")).toBeNull(); + expect(repository.getScanWatermark("codex")).toMatchObject({ latestSeenCreatedAt: at }); + }); + + it("starts the first watermark at a recent unfinished turn and captures it after it completes in place", async () => { + const root = mkdtempSync(join(tmpdir(), "native-backend-scan-")); roots.push(root); + const store = createAppStateStore({ databasePath: join(root, "app.sqlite") }); stores.push(store); + const repository = store.repositories.agentSources; + const heldAt = new Date(Date.now() - 60 * 60 * 1000).toISOString(); + const doneAt = new Date(Date.now() - 30 * 60 * 1000).toISOString(); + const event = (timestamp: string, type: string, payload: Record) => ({ type, timestamp, payload }); + const write = (file: string, events: unknown[]) => writeFileSync(join(root, file), events.map((value) => JSON.stringify(value)).join("\n") + "\n"); + const heldTurn = (complete: boolean) => [ + event(heldAt, "session_meta", { id: "held-session" }), + event(heldAt, "event_msg", { type: "task_started", turn_id: "held-turn" }), + event(heldAt, "response_item", { type: "message", role: "user", content: [{ text: "Keep working on the release check." }] }), + ...(complete ? [ + event(heldAt, "response_item", { type: "message", role: "assistant", content: [{ text: "The release check passed." }] }), + event(heldAt, "event_msg", { type: "task_complete", turn_id: "held-turn" }) + ] : []) + ]; + write("rollout-done.jsonl", [ + event(doneAt, "session_meta", { id: "done-session" }), + event(doneAt, "event_msg", { type: "task_started", turn_id: "done-turn" }), + event(doneAt, "response_item", { type: "message", role: "user", content: [{ text: "Summarize the finished task." }] }), + event(doneAt, "response_item", { type: "message", role: "assistant", content: [{ text: "Done." }] }), + event(doneAt, "event_msg", { type: "task_complete", turn_id: "done-turn" }) + ]); + write("rollout-held.jsonl", heldTurn(false)); + const client = createMockMemoryClient(); + const completeSourceTurn = vi.spyOn(client, "completeSourceTurn"); + const service = createAgentSourceService({ + sourceRegistry: createSourceRegistry([createCodexSourceAdapter({ sessionsRoot: root })]), + memoryClient: client, + agentSourceRepository: repository, + ingestionService: createIngestionService({ memoryClient: client, agentSourceRepository: repository }), + skillDistributionService: { install: async () => undefined, uninstall: async () => undefined, installPlugin: async () => undefined, uninstallPlugin: async () => undefined }, + scanStoreDirectory: join(root, "scans") + }); + expect((await service.scanOne("codex", { scanJobId: "first-job" })).errors).toEqual([]); + expect(repository.getScanWatermark("codex")).toMatchObject({ baselineAt: heldAt, latestSeenCreatedAt: heldAt }); + write("rollout-held.jsonl", heldTurn(true)); + expect((await service.scanOne("codex", { scanJobId: "second-job" })).errors).toEqual([]); + expect(completeSourceTurn).toHaveBeenLastCalledWith(expect.objectContaining({ + sourceTurn: expect.objectContaining({ conversationId: "held-session", turnId: "held-turn" }) + })); + expect(repository.getConversationCheckpoint("codex", "held-session")).not.toBeNull(); + }); + + it("advances the watermark past an unfinished turn abandoned long ago", async () => { + const root = mkdtempSync(join(tmpdir(), "native-backend-scan-")); roots.push(root); + const store = createAppStateStore({ databasePath: join(root, "app.sqlite") }); stores.push(store); + const repository = store.repositories.agentSources; + const at = "2099-09-09T10:00:00.000Z"; + const abandonedAt = "2020-01-01T00:00:00.000Z"; + const event = (timestamp: string, type: string, payload: Record) => ({ type, timestamp, payload }); + writeFileSync(join(root, "rollout-complete.jsonl"), [ + event(at, "session_meta", { id: "complete-session" }), + event(at, "event_msg", { type: "task_started", turn_id: "complete-turn" }), + event(at, "response_item", { type: "message", role: "user", content: [{ text: "Finish the complete turn." }] }), + event(at, "response_item", { type: "message", role: "assistant", content: [{ text: "Done." }] }), + event(at, "event_msg", { type: "task_complete", turn_id: "complete-turn" }) + ].map((value) => JSON.stringify(value)).join("\n") + "\n"); + writeFileSync(join(root, "rollout-abandoned.jsonl"), [ + event(abandonedAt, "session_meta", { id: "abandoned-session" }), + event(abandonedAt, "event_msg", { type: "task_started", turn_id: "abandoned-turn" }), + event(abandonedAt, "response_item", { type: "message", role: "user", content: [{ text: "This turn never finished." }] }) + ].map((value) => JSON.stringify(value)).join("\n") + "\n"); + const client = createMockMemoryClient(); + const service = createAgentSourceService({ + sourceRegistry: createSourceRegistry([createCodexSourceAdapter({ sessionsRoot: root })]), + memoryClient: client, + agentSourceRepository: repository, + ingestionService: createIngestionService({ memoryClient: client, agentSourceRepository: repository }), + skillDistributionService: { install: async () => undefined, uninstall: async () => undefined, installPlugin: async () => undefined, uninstallPlugin: async () => undefined }, + scanStoreDirectory: join(root, "scans") + }); + expect((await service.scanOne("codex", { scanJobId: "abandoned-job", mode: "full" })).errors).toEqual([]); + expect(repository.getScanWatermark("codex")).toMatchObject({ latestSeenCreatedAt: at }); + expect(repository.getConversationCheckpoint("codex", "abandoned-session")).toBeNull(); }); it("counts skipped cancelled messages in add progress and still writes the watermark", async () => { diff --git a/App/backend/src/services/tests/onboarding-insight-service.test.ts b/App/backend/src/services/tests/onboarding-insight-service.test.ts index df7d25c9e..5ae710065 100644 --- a/App/backend/src/services/tests/onboarding-insight-service.test.ts +++ b/App/backend/src/services/tests/onboarding-insight-service.test.ts @@ -33,7 +33,7 @@ describe("onboarding insight service", () => { const report = await service.generateReport({ locale: "zh-CN" }); expect(report.status).toBe("ready"); - expect(report.reportMarkdown).toContain("Hi"); + expect(report.reportMarkdown).toContain("我是 Memmy"); expect(report.reportMarkdown).toContain("先把方案"); expect(report.reportMarkdown).not.toContain("轻量样本"); expect(report.reportMarkdown).not.toContain("用户 query"); @@ -61,7 +61,7 @@ describe("onboarding insight service", () => { const report = await service.generateReport({ locale: "zh-CN" }); expect(report.reportMarkdown.split("\n")[0]).not.toContain("Grace江"); - expect(report.reportMarkdown).toContain("Hi"); + expect(report.reportMarkdown).toContain("我是 Memmy"); }); it("does not treat ordinary Chinese task phrases after 我是 as a name", async () => { @@ -78,7 +78,39 @@ describe("onboarding insight service", () => { const report = await service.generateReport({ locale: "zh-CN" }); expect(report.reportMarkdown.split("\n")[0]).not.toContain("部署在云服务器上使用的"); - expect(report.reportMarkdown).toContain("Hi"); + expect(report.reportMarkdown).toContain("我是 Memmy"); + }); + + it("answers the first greeting without inventing a project or todo from casual history", async () => { + const service = createOnboardingInsightService({ + samplers: [sampler("cursor", "Cursor", [query("cursor", "1", "今天心情不错,想随便聊聊")])], + reportGenerator: null, + now: () => 100 + }); + + const report = await service.generateReport({ locale: "zh-CN" }); + + expect(report.reportMarkdown).toContain("我是 Memmy。我可以陪你讨论问题、整理信息和计划"); + expect(report.reportMarkdown).toContain("你可以直接从眼前想聊的事开始"); + expect(report.reportMarkdown).not.toContain("## 最近聊到的事"); + expect(report.reportMarkdown).not.toContain("## 如果想接着做"); + }); + + it("keeps a non-coding request available for a useful first follow-up", async () => { + const service = createOnboardingInsightService({ + samplers: [sampler("cursor", "Cursor", [{ + ...query("cursor", "1", "帮我写一封生日祝福信"), + workspacePath: null + }])], + reportGenerator: null, + now: () => 100 + }); + + const report = await service.generateReport({ locale: "zh-CN" }); + + expect(report.reportMarkdown).toContain("我是 Memmy"); + expect(report.reportMarkdown).toContain("帮我写一封生日祝福信"); + expect(report.reportMarkdown).toContain("## 如果想接着做"); }); it("acknowledges detected agents when they have no sampled memory", async () => { @@ -96,10 +128,9 @@ describe("onboarding insight service", () => { const report = await service.generateReport({ locale: "zh-CN" }); expect(report.status).toBe("ready"); - expect(report.reportMarkdown).toBe([ - "Memmy 已识别到这台设备上的 Codex,但首次轻量扫描暂时没有读到可用的对话历史。", - "之后用 Memmy 处理真实任务时,它会记住有用的背景、决策和下一步,方便新对话或其他 Agent 继续。" - ].join("\n\n")); + expect(report.reportMarkdown).toContain("嗨,我是 Memmy。我可以陪你讨论问题、整理信息和计划"); + expect(report.reportMarkdown).toContain("我没能通过扫描获得足够的线索来了解你的近况"); + expect(report.reportMarkdown).not.toContain("Codex"); expect(report.reportMarkdown).not.toContain("not enough recent user messages"); expect(report.diagnostics).toMatchObject({ discoveredAgentCount: 1, @@ -129,10 +160,8 @@ describe("onboarding insight service", () => { const events = await collectStreamEvents(service.streamReport({ locale: "en-US" })); expect(report.status).toBe("ready"); - expect(report.reportMarkdown).toBe([ - "There is no readable Agent history on this device yet, so there is nothing useful to pretend I already know.", - "Tell Memmy about one real task. It will preserve the useful background, decisions, and next step so a new conversation—or another Agent such as Cursor or Codex—can continue without making you explain it again." - ].join("\n\n")); + expect(report.reportMarkdown).toContain("Hi, I'm Memmy. I can help you think through questions"); + expect(report.reportMarkdown).toContain("I couldn't find enough from this scan to get a sense of what you've been up to"); expect(report.reportMarkdown).not.toContain("我没有在本机扫描到"); expect(events).toEqual([ { @@ -151,7 +180,7 @@ describe("onboarding insight service", () => { type: "done", response: expect.objectContaining({ status: "ready", - reportMarkdown: expect.stringContaining("There is no readable Agent history on this device yet"), + reportMarkdown: expect.stringContaining("Hi, I'm Memmy."), diagnostics: expect.objectContaining({ discoveredAgentCount: 0, sampledQueryCount: 0, @@ -973,6 +1002,110 @@ describe("onboarding insight service", () => { }); }); + it("uses sampled conversation context when a reader ignores the scan deadline", async () => { + vi.useFakeTimers(); + const readConversation = vi.fn(() => new Promise(() => undefined)); + const generateReport = vi.fn(async (_input: OnboardingInsightGenerationInput) => "生成的初见报告"); + const service = createOnboardingInsightService({ + samplers: [samplerWithRecentMessages("cursor", "Cursor", "2026-06-02T10:00:00.000Z")], + conversationWindowReader: { readConversation }, + reportGenerator: { generateReport } + }); + + const eventsPromise = collectStreamEvents(service.streamReport({ locale: "zh-CN" })); + await vi.advanceTimersByTimeAsync(3_000); + const events = await eventsPromise; + + expect(readConversation).toHaveBeenCalledOnce(); + expect(events[0]).toMatchObject({ type: "sampled", diagnostics: { elapsedMs: 3_000 } }); + expect(events.at(-1)).toMatchObject({ type: "done", response: { status: "ready" } }); + expect(generateReport.mock.calls[0]?.[0].sample.latestConversation?.messages.map((message) => message.role)) + .toEqual(["user", "assistant"]); + }); + + it("gives conversation enrichment only the remaining shared scan budget and discards late results", async () => { + vi.useFakeTimers(); + const baseSampler = samplerWithRecentMessages("cursor", "Cursor", "2026-06-02T10:00:00.000Z"); + const readConversation = vi.fn(async (reference) => { + await new Promise((resolve) => setTimeout(resolve, 500)); + return { ...reference, messages: [{ ...query("cursor", "late", "迟到的补读"), role: "user" as const }] }; + }); + const generateReport = vi.fn(async (_input: OnboardingInsightGenerationInput) => "生成的初见报告"); + const service = createOnboardingInsightService({ + samplers: [{ + ...baseSampler, + async sampleRecentUserQueries(options) { + await new Promise((resolve) => setTimeout(resolve, 2_800)); + return baseSampler.sampleRecentUserQueries(options); + } + }], + conversationWindowReader: { readConversation }, + reportGenerator: { generateReport } + }); + + const reportPromise = service.generateReport({ locale: "zh-CN" }); + await vi.advanceTimersByTimeAsync(3_000); + const report = await reportPromise; + + expect(readConversation).toHaveBeenCalledWith(expect.anything(), expect.objectContaining({ deadlineMs: 200 })); + expect(report.diagnostics.elapsedMs).toBe(3_000); + expect(generateReport.mock.calls[0]?.[0].sample.latestConversation?.messages[0]?.text) + .toBe("请继续 cursor 的最近任务"); + await vi.advanceTimersByTimeAsync(500); + expect(generateReport).toHaveBeenCalledOnce(); + expect(generateReport.mock.calls[0]?.[0].sample.latestConversation?.messages[0]?.text) + .toBe("请继续 cursor 的最近任务"); + }); + + it("does not start conversation enrichment once shallow sampling has used the budget", async () => { + vi.useFakeTimers(); + const readConversation = vi.fn(async () => null); + const service = createOnboardingInsightService({ + samplers: [ + samplerWithRecentMessages("cursor", "Cursor", "2026-06-02T10:00:00.000Z"), + { sourceId: "slow", displayName: "Slow", detect: async () => true, + sampleRecentUserQueries: () => new Promise(() => undefined) } + ], + conversationWindowReader: { readConversation }, + reportGenerator: null + }); + + const reportPromise = service.generateReport({ locale: "zh-CN" }); + await vi.advanceTimersByTimeAsync(3_000); + const report = await reportPromise; + + expect(readConversation).not.toHaveBeenCalled(); + expect(report.status).toBe("ready"); + expect(report.diagnostics.sampledQueryCount).toBe(1); + expect(report.reportMarkdown).toContain("请继续 cursor 的最近任务"); + }); + + it("discards a reader result past the wall-clock budget even before timers run", async () => { + let wallClock = 1_000; + const clock = vi.spyOn(Date, "now").mockImplementation(() => wallClock); + try { + const generateReport = vi.fn(async (_input: OnboardingInsightGenerationInput) => "生成的初见报告"); + const service = createOnboardingInsightService({ + samplers: [samplerWithRecentMessages("cursor", "Cursor", "2026-06-02T10:00:00.000Z")], + conversationWindowReader: { + async readConversation(reference) { + wallClock += 3_001; + return { ...reference, messages: [{ ...query("cursor", "late", "迟到的旧会话前缀"), role: "user" as const }] }; + } + }, + reportGenerator: { generateReport } + }); + + const report = await service.generateReport({ locale: "zh-CN" }); + + expect(report.status).toBe("ready"); + expect(generateReport.mock.calls[0]?.[0].sample.latestConversation?.messages[0]?.text) + .toBe("请继续 cursor 的最近任务"); + } finally { + clock.mockRestore(); + } + }); + it("requests OpenAI-compatible report generation with stream enabled and parses deltas", async () => { const encoder = new TextEncoder(); const fetchImpl = vi.fn(async () => new Response( @@ -1000,7 +1133,7 @@ describe("onboarding insight service", () => { const body = JSON.parse(String(fetchImpl.mock.calls[0]?.[1]?.body)); expect(body.stream).toBe(true); - expect(body.messages[0].content).toContain("最近项目记忆"); + expect(body.messages[0].content).toContain("最近聊到的事"); expect(body.messages[0].content).toContain("接下来可以做"); expect(body.messages[1].content).toContain('"role": "tool"'); expect(body.messages[1].content).toContain("npm test: success"); @@ -1069,36 +1202,26 @@ describe("onboarding insight service", () => { expect(body.messages[0].content).toContain("偏好结论的唯一原始证据是 preferenceEvidence 中由用户本人发送的消息"); expect(body.messages[0].content).toContain("严禁使用 assistant、tool 或 latestConversation 推断偏好"); expect(body.messages[0].content).not.toContain("我对你的工作偏好"); - expect(body.messages[0].content).toContain("根据 profile.nameHints 综合判断"); - expect(body.messages[0].content).toContain("默认优先使用 homePathName"); - expect(body.messages[0].content).toContain("admin、administrator、root、ubuntu"); - expect(body.messages[0].content).toContain("不得把名字替换成“这个线索”"); - expect(body.messages[0].content).toContain("有值时要自然说明用户最近更常用中文还是英文"); + expect(body.messages[0].content).toContain("只有 profile.nameHints.selfDeclaredNames 中存在可信的用户自称时才使用名字"); + expect(body.messages[0].content).toContain("不猜本机用户名或路径名"); + expect(body.messages[0].content).toContain("不要假定每个人都有项目或 Bug"); expect(body.messages[0].content).toContain("不要生成按钮、行动卡片、CTA"); expect(body.messages[0].content).toContain("不得包含任何原始 HTML 标签或样式"); expect(body.messages[0].content).toContain("不要输出思考过程、执行计划、要求确认、Prompt 复述或起草说明"); expect(body.messages[0].content).not.toContain("[MEMMY_ACTIONS_JSON]"); const userPayload = JSON.parse(String(body.messages[1].content)); - expect(userPayload.reportGoal.primary).toBe("user_preferences_latest_project_memory_and_actionable_todos"); - expect(userPayload.reportGoal.lengthConstraint).toContain("300-500 Chinese characters"); - expect(userPayload.reportGoal.requiredSections).toContain("latest_project_memory"); - expect(userPayload.reportGoal.requiredSections).toContain("user_preferences"); + expect(userPayload.openingUserMessage).toBe("嗨,第一次见面,你有什么可以为我做的吗?"); + expect(userPayload.reportGoal.primary).toBe("answer_the_opening_question_with_capabilities_and_evidence_based_context"); + expect(userPayload.reportGoal.lengthConstraint).toContain("120-300 Chinese characters"); + expect(userPayload.reportGoal.requiredSections).toContain("friendly_greeting_and_capabilities"); + expect(userPayload.reportGoal.requiredSections).toContain("recent_context_and_open_todos_when_supported"); expect(userPayload.reportGoal.outputEnvelope.taskContextFields).toContain("trajectorySummary"); - expect(userPayload.profile.nameHints).toMatchObject({ - selfDeclaredNames: ["Grace"], - homePathName: "jiang", - computerUserName: "jiang", - homeAndComputerMatch: true - }); + expect(userPayload.profile.nameHints).toEqual({ selfDeclaredNames: ["Grace"] }); expect(JSON.stringify(userPayload.profile)).not.toContain("conversationId"); expect(JSON.stringify(userPayload.profile)).not.toContain("messageId"); expect(JSON.stringify(userPayload.profile)).not.toContain("internal-conversation-id"); expect(JSON.stringify(userPayload.profile)).not.toContain("internal-message-id"); - expect(userPayload.nameDecisionRequirement).toMatchObject({ - mustInferDisplayName: true, - mustIncludeDisplayNameInFirstSentence: true, - defaultPriority: "homePathName" - }); + expect(userPayload).not.toHaveProperty("nameDecisionRequirement"); expect(userPayload.preferenceEvidence).toEqual([expect.objectContaining({ agentSource: "Codex", text: "Continue the first report." diff --git a/App/backend/src/tests/index.test.ts b/App/backend/src/tests/index.test.ts index 67b73d0a5..cb99cf0e6 100644 --- a/App/backend/src/tests/index.test.ts +++ b/App/backend/src/tests/index.test.ts @@ -8,6 +8,7 @@ import YAML from "yaml"; import { afterEach, describe, expect, it, vi } from "vitest"; import type { CloudClient } from "../adapters/outbound/cloud-client/index.js"; import type { MemoryClient } from "../adapters/outbound/memory-client/index.js"; +import type { MemmyAgentAdminClient } from "../adapters/outbound/memmy-agent-admin-client/index.js"; import { createLocalBackend, readMemoryLayerConfig, type LocalBackend } from "../index.js"; import { createAppStateStore } from "../infrastructure/app-state-store/index.js"; import { createMockCloudClient } from "./support/mock-cloud-client.js"; @@ -17,6 +18,20 @@ let tempDir: string | undefined; let backend: LocalBackend | undefined; let integrationServer: ReturnType | undefined; +function createMockMemmyAgentAdminClient(reloadMcpConfig: MemmyAgentAdminClient["reloadMcpConfig"]): MemmyAgentAdminClient { + return { + getChannelDefinitions: async () => ({ channels: [] }), + getChannelConnections: async () => ({ connections: [] }), + configureChannel: async () => ({ status: "connected", running: true }), + stopChannel: async () => ({ status: "disabled", running: false }), + startWeixinLogin: async () => ({ status: "pendingQr" }), + pollWeixinLogin: async () => ({ status: "connected" }), + startFeishuLogin: async () => ({ status: "pendingQr" }), + pollFeishuLogin: async () => ({ status: "connected" }), + reloadMcpConfig + }; +} + afterEach(async () => { await backend?.close(); backend = undefined; @@ -132,46 +147,154 @@ describe("local api", () => { expect(YAML.parse(readFileSync(memmyConfigPath, "utf8")).memmyMemory.language).toBe("en-US"); }); - it("reloads Agent MCP only after writing the current Composio bridge config", async () => { + it("starts the local API while Agent MCP reload is pending, after writing the current Composio bridge config", async () => { tempDir = mkdtempSync(join(tmpdir(), "memmy-backend-mcp-startup-reload-")); const memmyConfigPath = join(tempDir, "config.yaml"); const snapshots: unknown[] = []; + let finishReload!: (result: Awaited>) => void; + const reloadComplete = new Promise>>((resolve) => { + finishReload = resolve; + }); - backend = await createLocalBackend({ - databasePath: join(tempDir, "app.sqlite"), - runtimeConfigPath: join(tempDir, "runtime.json"), - localToken: "test-token", - memoryClient: createMockMemoryClient(), - cloudClient: createMockCloudClient(), - memmyConfigPath, - memmyAgentAdminClient: { - getChannelDefinitions: async () => ({ channels: [] }), - getChannelConnections: async () => ({ connections: [] }), - configureChannel: async () => ({ status: "connected", running: true }), - stopChannel: async () => ({ status: "disabled", running: false }), - startWeixinLogin: async () => ({ status: "pendingQr" }), - pollWeixinLogin: async () => ({ status: "connected" }), - startFeishuLogin: async () => ({ status: "pendingQr" }), - pollFeishuLogin: async () => ({ status: "connected" }), - async reloadMcpConfig() { + try { + backend = await createLocalBackend({ + databasePath: join(tempDir, "app.sqlite"), + runtimeConfigPath: join(tempDir, "runtime.json"), + localToken: "test-token", + memoryClient: createMockMemoryClient(), + cloudClient: createMockCloudClient(), + memmyConfigPath, + memmyAgentAdminClient: createMockMemmyAgentAdminClient(async () => { snapshots.push(YAML.parse(readFileSync(memmyConfigPath, "utf8"))); - return { ok: true, message: "reloaded", requires_restart: false }; + return reloadComplete; + }) + }); + + expect(snapshots).toHaveLength(1); + expect(snapshots[0]).toMatchObject({ + tools: { + mcpServers: { + composio: { + type: "streamableHttp", + url: `${backend.runtimeConfig.baseUrl}/mcp/composio`, + headers: { "x-memmy-mcp-token": expect.stringMatching(/^mmt_/) } + } + } } - } - }); + }); + const response = await fetch(`${backend.runtimeConfig.baseUrl}/api/app/bootstrap`, { + headers: { "x-memmy-local-token": "test-token" } + }); + expect(response.status).toBe(200); + expect(AppBootstrapResponseSchema.safeParse(await response.json()).success).toBe(true); + } finally { + finishReload({ ok: true, message: "reloaded", requires_restart: false }); + } + }); - expect(snapshots).toHaveLength(1); - expect(snapshots[0]).toMatchObject({ + it("can prepare the current bridge before Agent startup without requesting a premature MCP reload", async () => { + tempDir = mkdtempSync(join(tmpdir(), "memmy-backend-mcp-prepare-")); + const memmyConfigPath = join(tempDir, "config.yaml"); + const legacyCustom = { type: "stdio", command: "legacy-tool", args: ["--existing"] }; + writeFileSync(memmyConfigPath, YAML.stringify({ tools: { mcpServers: { + legacyCustom, composio: { type: "streamableHttp", - url: `${backend.runtimeConfig.baseUrl}/mcp/composio`, - headers: { "x-memmy-mcp-token": expect.stringMatching(/^mmt_/) } + url: "http://127.0.0.1:1/old-bridge", + headers: { "x-memmy-mcp-token": "old-token" } } } } + }), "utf8"); + const reload = vi.fn(async () => ({ ok: true, message: "reloaded", requires_restart: false })); + backend = await createLocalBackend({ + databasePath: join(tempDir, "app.sqlite"), + runtimeConfigPath: join(tempDir, "runtime.json"), + localToken: "test-token", + memoryClient: createMockMemoryClient(), + cloudClient: createMockCloudClient(), + memmyConfigPath, + memmyAgentAdminClient: createMockMemmyAgentAdminClient(reload), + skipAgentMcpReload: true + }); + + const config = YAML.parse(readFileSync(memmyConfigPath, "utf8")); + const bridge = config.tools.mcpServers.composio; + expect(config.tools.mcpServers.legacyCustom).toEqual(legacyCustom); + expect(bridge).toMatchObject({ + type: "streamableHttp", + url: `${backend.runtimeConfig.baseUrl}/mcp/composio`, + headers: { "x-memmy-mcp-token": expect.stringMatching(/^mmt_/) }, + toolTimeout: 60 + }); + expect(bridge.headers["x-memmy-mcp-token"]).not.toBe("old-token"); + const bootstrap = await fetch(`${backend.runtimeConfig.baseUrl}/api/app/bootstrap`, { + headers: { "x-memmy-local-token": "test-token" } + }); + expect(bootstrap.status).toBe(200); + expect(AppBootstrapResponseSchema.safeParse(await bootstrap.json()).success).toBe(true); + const initialize = await fetch(bridge.url, { + method: "POST", + headers: { + "content-type": "application/json", + accept: "application/json, text/event-stream", + ...bridge.headers + }, + body: JSON.stringify({ + jsonrpc: "2.0", + id: 1, + method: "initialize", + params: { + protocolVersion: "2024-11-05", + capabilities: {}, + clientInfo: { name: "startup-prepare-test", version: "1.0" } + } + }) }); + expect(initialize.status).toBe(200); + await initialize.body?.cancel(); + expect(reload).not.toHaveBeenCalled(); + }); + + it.each(["not-ok", "rejected", "thrown"] as const)("logs %s Agent MCP reload failures without rejecting startup or leaving an unhandled rejection", async (failure) => { + tempDir = mkdtempSync(join(tmpdir(), "memmy-backend-mcp-startup-failure-")); + const warn = vi.spyOn(console, "warn").mockImplementation(() => undefined); + const unhandledRejections: unknown[] = []; + const onUnhandledRejection = (error: unknown) => unhandledRejections.push(error); + let rejectReload: ((error: Error) => void) | undefined; + process.on("unhandledRejection", onUnhandledRejection); + + try { + backend = await createLocalBackend({ + databasePath: join(tempDir, "app.sqlite"), + runtimeConfigPath: join(tempDir, "runtime.json"), + localToken: "test-token", + memoryClient: createMockMemoryClient(), + cloudClient: createMockCloudClient(), + memmyConfigPath: join(tempDir, "config.yaml"), + memmyAgentAdminClient: createMockMemmyAgentAdminClient(() => { + if (failure === "not-ok") { + return Promise.resolve({ ok: false, message: "reload failed", requires_restart: true }); + } + if (failure === "thrown") throw new Error("reload failed"); + return new Promise((_resolve, reject) => { rejectReload = reject; }); + }) + }); + + rejectReload?.(new Error("reload failed")); + await vi.waitFor(() => expect(warn).toHaveBeenCalledWith( + failure === "not-ok" + ? "Agent MCP reload did not complete: reload failed" + : "Agent MCP reload unavailable during backend startup: reload failed" + )); + await new Promise((resolve) => setImmediate(resolve)); + expect(unhandledRejections).toEqual([]); + } finally { + process.off("unhandledRejection", onUnhandledRejection); + warn.mockRestore(); + } }); it("does not block local API startup while managed Memory is still initializing", async () => { diff --git a/App/backend/src/tests/observed-skill-approve-bridge.test.ts b/App/backend/src/tests/observed-skill-approve-bridge.test.ts new file mode 100644 index 000000000..4bdd7a220 --- /dev/null +++ b/App/backend/src/tests/observed-skill-approve-bridge.test.ts @@ -0,0 +1,232 @@ +import Fastify from "fastify"; +import { afterEach, describe, expect, it } from "vitest"; +import { createMemoryHttpServer, DEFAULT_MEMMY_CONFIG, type LlmClient, type LlmMessage, type RuntimeNamespace } from "../../../../Memory/src/index.js"; +import { Repositories } from "../../../../Memory/src/storage/repositories.js"; +import { createMemoryServiceFixture } from "../../../../Memory/tests/fixtures/memory-service-fixture.js"; +import { createHttpMemoryClient } from "../adapters/outbound/memory-client/http-memory-client.js"; +import { registerMemoryRoutes } from "../adapters/inbound/local-api/routes/agent-runtime/memory.js"; +import type { AgentRuntimeRouteDeps } from "../adapters/inbound/local-api/routes/agent-runtime/index.js"; + +const { cleanup, createTestService } = createMemoryServiceFixture(); +afterEach(cleanup); +const namespace = { source: "memmy", profileId: "default", userId: "original-history-user" }; +const draft = { create: true, name: "review_release", summary: "Review the release and record its blocker", + trigger: "When reviewing a release", selectedActionIds: ["a1", "a2", "a3", "a4"] }; + +function model(): LlmClient { + return { + config: DEFAULT_MEMMY_CONFIG.summary, isConfigured: () => true, + async complete() { return ""; }, + async completeJson>(messages: LlmMessage[], options: { operation: string }) { + if (options.operation === "skill.external_evidence.suggest") return draft as unknown as T; + const input = JSON.parse(messages.find((message) => message.role === "user")!.content) as { + sources: Array<{ id: string; content: string }>; + }; + return { keep: true, title: "Release review blocker", summary: "The release is awaiting approval.", + items: [{ category: "blocker", text: "The release is awaiting approval.", + source_memory_id: input.sources[0]!.id, evidence_quote: "The release is awaiting approval." }] } as unknown as T; + }, + status: () => ({ provider: "test", configured: true, remote: false }) + }; +} + +function evidence(scope: RuntimeNamespace, index = 0) { + return { action: "upsert" as const, source: "computer_history", namespace: scope, + sourceRecordId: `2026-09-27T10-${index === 0 ? "00" : "10"}-00Z-10min-summary`, + revision: "2026-09-27T10:20:00.000Z", title: "Release review", summary: "The release is awaiting approval.", + content: "Open the draft. Review the changes. Record the blocker. Save the review. The release is awaiting approval.", + startedAt: "2026-09-27T10:00:00.000Z", endedAt: "2026-09-27T10:20:00.000Z" }; +} + +async function candidate(service: ReturnType["service"], scope: RuntimeNamespace, source = "manual_capture") { + const sources = [evidence(scope, 0), evidence(scope, 1)].map((item) => ({ ...item, source })); + sources.forEach((source) => service.syncExternalEvidence(source)); + await service.runWorkerOnce(100); + await service.runWorkerOnce(100); + return service.suggestExternalEvidenceSkill({ source, namespace: scope, + groupId: "2026-09-27T06-00-00Z-6h-summary", revision: "2026-09-27T12:00:00.000Z", + sourceRecordIds: sources.map((source) => source.sourceRecordId), actions: [ + { sourceRecordId: sources[0]!.sourceRecordId, text: "Open the draft" }, + { sourceRecordId: sources[0]!.sourceRecordId, text: "Review the changes" }, + { sourceRecordId: sources[1]!.sourceRecordId, text: "Record the blocker" }, + { sourceRecordId: sources[1]!.sourceRecordId, text: "Save the review" } + ] }); +} + +async function withMemoryServer( + service: ReturnType["service"], + run: (baseUrl: string) => Promise +) { + const scopeA = { ...namespace, tenantId: "tenant-a" }; + const server = createMemoryHttpServer({ service, startAgentSourceAutomation: false, auth: { + mode: "local", localServiceToken: "local-write-token", + scopedApiKeys: { + "tenant-a-token": { namespace: scopeA, scopes: ["memory:write", "memory:read"] }, + "read-token": { namespace: scopeA, scopes: ["memory:read"] } + }, cloudAccessTokens: { "tenant-a-cloud-token": scopeA } + } }); + await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("expected TCP port"); + try { await run(`http://127.0.0.1:${address.port}`); } + finally { await new Promise((resolve) => server.close(() => resolve())); } +} + +function post(baseUrl: string, path: string, body: unknown, token = "local-write-token", headers: Record = {}) { + return fetch(`${baseUrl}/api/v1/${path}`, { method: "POST", headers: { + authorization: `Bearer ${token}`, "content-type": "application/json", ...headers + }, body: JSON.stringify(body) }); +} + +describe("observed Skill editing and legacy approval through the desktop bridge", () => { + it("approves a shared local candidate using the real empty-body HTTP request", async () => { + const { service, db } = createTestService({ llm: model(), skillLlm: model() }); + const selected = await candidate(service, namespace); + const repos = new Repositories(db.db); + await withMemoryServer(service, async (baseUrl) => { + const response = await post(baseUrl, `evidence/skills/${selected.id}/approve`, {}); + expect(response.status).toBe(200); + expect(await response.json()).toMatchObject({ id: selected.id, status: "activated" }); + expect(repos.memories.get(selected.id!)?.status).toBe("activated"); + expect(db.db.prepare("SELECT COUNT(*) AS n FROM evolution_jobs WHERE job_type = 'embedding' AND target_memory_id = ?") + .get(selected.id)).toEqual({ n: 1 }); + }); + }); + + it("rejects explicit wrong tenant, missing write scope and forged scoped or cloud headers", async () => { + const { service, db } = createTestService({ llm: model(), skillLlm: model() }); + const selected = await candidate(service, { ...namespace, tenantId: "tenant-b" }); + const repos = new Repositories(db.db); + await withMemoryServer(service, async (baseUrl) => { + const path = `evidence/skills/${selected.id}/approve`; + expect((await post(baseUrl, path, { namespace: { ...namespace, tenantId: "tenant-a" } })).status).toBe(404); + expect((await post(baseUrl, path, {}, "read-token")).status).toBe(403); + for (const token of ["tenant-a-token", "tenant-a-cloud-token"]) { + expect((await post(baseUrl, path, {}, token, { + "x-memmy-tenant-id": "tenant-b", "x-memmy-user-id": namespace.userId + })).status).toBe(404); + expect((await post(baseUrl, path, { namespace: { ...namespace, tenantId: "tenant-b" } }, token)).status).toBe(403); + } + expect(repos.memories.get(selected.id!)?.status).toBe("resolving"); + }); + }); + + it("keeps scoped sync and delete in the token tenant despite forged headers", async () => { + const { service, db } = createTestService({ llm: model(), skillLlm: model() }); + const repos = new Repositories(db.db); + const sourceA = service.syncExternalEvidence(evidence({ ...namespace, tenantId: "tenant-a" })); + const sourceB = service.syncExternalEvidence(evidence({ ...namespace, tenantId: "tenant-b" })); + await withMemoryServer(service, async (baseUrl) => { + const { namespace: _scope, ...body } = evidence(namespace); + const headers = { "x-memmy-tenant-id": "tenant-b" }; + const updated = await post(baseUrl, "evidence/sync", { ...body, revision: "2026-09-27T10:21:00.000Z", + content: `${body.content} Still pending.` }, "tenant-a-token", headers); + expect(updated.status).toBe(200); + expect(await updated.json()).toMatchObject({ id: sourceA.id }); + expect(repos.memories.get(sourceB.id)?.properties.internal_info.external_evidence).toMatchObject({ source_content: body.content }); + const deletion = await post(baseUrl, "evidence/sync", { action: "delete", source: body.source, + sourceRecordId: body.sourceRecordId, revision: body.revision }, "tenant-a-token", headers); + expect(await deletion.json()).toMatchObject({ id: sourceA.id, status: "deleted" }); + expect(repos.memories.get(sourceB.id)?.status).toBe("activated"); + }); + }); + + it("forwards the desktop button route into Memory and activates the persisted candidate", async () => { + const { service, db } = createTestService({ llm: model(), skillLlm: model() }); + const selected = await candidate(service, namespace); + const repos = new Repositories(db.db); + await withMemoryServer(service, async (baseUrl) => { + const memoryClient = createHttpMemoryClient({ baseUrl, token: "local-write-token", timeoutMs: 2_000, maxRetries: 0 }); + const desktop = Fastify(); + registerMemoryRoutes(desktop, { + services: { memoryClient } as AgentRuntimeRouteDeps["services"], + async authenticateRuntimeToken(request, reply) { + if (request.headers.authorization !== "Bearer desktop-runtime-token") return reply.code(401).send({ error: "unauthorized" }); + } + }); + try { + const url = `/api/v1/evidence/skills/${selected.id}/approve`; + expect((await desktop.inject({ method: "POST", url, payload: {} })).statusCode).toBe(401); + const response = await desktop.inject({ method: "POST", url, payload: {}, + headers: { authorization: "Bearer desktop-runtime-token" } }); + expect(response.statusCode).toBe(200); + expect(response.json()).toMatchObject({ id: selected.id, status: "activated" }); + expect(repos.memories.get(selected.id!)?.status).toBe("activated"); + } finally { await desktop.close(); } + }); + }); + + it("automatically enables History Skills, then persists a user edit through desktop HTTP with a version guard", async () => { + const { service, db } = createTestService({ llm: model(), skillLlm: model() }); + const selected = await candidate(service, namespace, "computer_history"); + const repos = new Repositories(db.db); + expect(selected.status).toBe("active"); + expect(repos.memories.get(selected.id!)?.status).toBe("activated"); + await withMemoryServer(service, async (baseUrl) => { + const memoryClient = createHttpMemoryClient({ baseUrl, token: "local-write-token", timeoutMs: 2_000, maxRetries: 0 }); + const desktop = Fastify(); + registerMemoryRoutes(desktop, { + services: { memoryClient } as AgentRuntimeRouteDeps["services"], + async authenticateRuntimeToken(request, reply) { + if (request.headers.authorization !== "Bearer desktop-runtime-token") return reply.code(401).send({ error: "unauthorized" }); + } + }); + try { + const url = `/api/v1/evidence/skills/${selected.id}`; + const before = repos.memories.get(selected.id!)!; + const payload = { expectedVersion: before.version, + body: "# release_review\n\nReview the current draft. Record the blocker, then verify the saved result." }; + expect((await desktop.inject({ method: "PATCH", url, payload })).statusCode).toBe(401); + const response = await desktop.inject({ method: "PATCH", url, payload, + headers: { authorization: "Bearer desktop-runtime-token" } }); + expect(response.statusCode).toBe(200); + expect(response.json()).toMatchObject({ id: selected.id, status: "activated", version: before.version + 1 }); + const after = repos.memories.get(selected.id!)!; + expect(after.memoryValue).toBe(payload.body); + expect(after.properties.internal_info.skill).toMatchObject({ invocation_guide: payload.body }); + expect(after.properties.internal_info.external_evidence_candidate).toMatchObject({ sourceMemoryIds: + (before.properties.internal_info.external_evidence_candidate as { sourceMemoryIds: string[] }).sourceMemoryIds }); + const conflicting = await desktop.inject({ method: "PATCH", url, payload: { ...payload, body: "Stale overwrite" }, + headers: { authorization: "Bearer desktop-runtime-token" } }); + expect(conflicting.statusCode).toBe(409); + expect(repos.memories.get(selected.id!)?.memoryValue).toBe(payload.body); + expect(db.db.prepare("SELECT COUNT(*) AS n FROM evolution_jobs WHERE job_type = 'embedding' AND target_memory_id = ?") + .get(selected.id)).toEqual({ n: 2 }); + } finally { await desktop.close(); } + }); + }); + + it("keeps Skill edits in the authenticated tenant despite forged headers", async () => { + const { service, db } = createTestService({ llm: model(), skillLlm: model() }); + const selected = await candidate(service, { ...namespace, tenantId: "tenant-b" }, "computer_history"); + const repos = new Repositories(db.db); + const before = repos.memories.get(selected.id!)!; + await withMemoryServer(service, async (baseUrl) => { + for (const token of ["tenant-a-token", "tenant-a-cloud-token", "read-token"]) { + const response = await fetch(`${baseUrl}/api/v1/evidence/skills/${selected.id}`, { method: "PATCH", headers: { + authorization: `Bearer ${token}`, "content-type": "application/json", "x-memmy-tenant-id": "tenant-b" + }, body: JSON.stringify({ body: "Unauthorized overwrite", expectedVersion: before.version }) }); + expect(response.status).toBe(token === "read-token" ? 403 : 404); + } + expect(repos.memories.get(selected.id!)?.memoryValue).toBe(before.memoryValue); + const empty = await fetch(`${baseUrl}/api/v1/evidence/skills/${selected.id}`, { method: "PATCH", headers: { + authorization: "Bearer local-write-token", "content-type": "application/json" + }, body: JSON.stringify({ body: " ", expectedVersion: before.version }) }); + expect(empty.status).toBe(400); + }); + }); + + it("does not let scoped credentials edit another History user's Skill within the same tenant", async () => { + const { service, db } = createTestService({ llm: model(), skillLlm: model() }); + const selected = await candidate(service, { ...namespace, tenantId: "tenant-a", userId: "another-history-user" }, "computer_history"); + const repos = new Repositories(db.db); + const before = repos.memories.get(selected.id!)!; + await withMemoryServer(service, async (baseUrl) => { + const response = await fetch(`${baseUrl}/api/v1/evidence/skills/${selected.id}`, { method: "PATCH", headers: { + authorization: "Bearer tenant-a-token", "content-type": "application/json", "x-memmy-user-id": "another-history-user" + }, body: JSON.stringify({ body: "Unauthorized same-tenant overwrite", expectedVersion: before.version }) }); + expect(response.status).toBe(404); + expect(repos.memories.get(selected.id!)?.memoryValue).toBe(before.memoryValue); + }); + }); +}); diff --git a/App/backend/src/tests/project-version.test.ts b/App/backend/src/tests/project-version.test.ts index 1b1634caa..1034d5d24 100644 --- a/App/backend/src/tests/project-version.test.ts +++ b/App/backend/src/tests/project-version.test.ts @@ -11,7 +11,6 @@ describe("project version", () => { ); expect(MEMMY_VERSION).toBe(rootManifest.version); - expect(MEMMY_VERSION).toBe("1.2.2"); expect(MEMMY_VERSION).toMatch(/^\d+\.\d+\.\d+(?:[-+][0-9A-Za-z.-]+)?$/); expect(readFileSync(resolve(repoRoot, "App/backend/src/project-version.ts"), "utf8")) .toContain(`MEMMY_VERSION = ${JSON.stringify(rootManifest.version)}`); diff --git a/App/backend/src/tests/support/mock-memory-client.ts b/App/backend/src/tests/support/mock-memory-client.ts index 47a424095..2b24f9841 100644 --- a/App/backend/src/tests/support/mock-memory-client.ts +++ b/App/backend/src/tests/support/mock-memory-client.ts @@ -239,6 +239,16 @@ export function createMockMemoryClient(options: CreateMockMemoryClientOptions = }; }, + async approveObservedSkill(memoryId) { + failIfNeeded(); + return { id: memoryId, status: "activated", serverTime: now() }; + }, + + async updateObservedSkill(memoryId, input) { + failIfNeeded(); + return { id: memoryId, status: "activated", version: input.expectedVersion + 1, serverTime: now() }; + }, + async enqueueImportSummaries() { failIfNeeded(); return { diff --git a/App/frontend/desktop/eslint.config.mjs b/App/frontend/desktop/eslint.config.mjs index c4b6d7af3..938273b70 100644 --- a/App/frontend/desktop/eslint.config.mjs +++ b/App/frontend/desktop/eslint.config.mjs @@ -18,6 +18,8 @@ export default tseslint.config( "no-undef": "off", "@typescript-eslint/no-explicit-any": "off", "@typescript-eslint/no-empty-object-type": "off", + "no-control-regex": "off", + "no-unexpected-multiline": "off", "@typescript-eslint/no-unused-vars": [ "warn", { vars: "all", varsIgnorePattern: "^_", args: "after-used", argsIgnorePattern: "^_" }, diff --git a/App/frontend/desktop/package.json b/App/frontend/desktop/package.json index ff757f6f2..d976a93c9 100644 --- a/App/frontend/desktop/package.json +++ b/App/frontend/desktop/package.json @@ -11,6 +11,7 @@ }, "dependencies": { "@memmy/desktop-interface": "0.0.0", + "@memmy/knowledge": "0.0.0", "@memmy/local-api-contracts": "0.0.0", "@types/react-syntax-highlighter": "^15.5.13", "@vitejs/plugin-react": "^6.0.2", @@ -18,6 +19,7 @@ "electron-log": "^5.4.4", "katex": "^0.16.47", "lucide-react": "^1.16.0", + "monaco-editor": "^0.57.0", "react": "^19.2.6", "react-dom": "^19.2.6", "react-markdown": "^10.1.0", @@ -27,8 +29,7 @@ "remark-gfm": "^4.0.1", "remark-math": "^6.0.0", "vite": "^8.0.14", - "zod": "^4.4.3", - "@memmy/knowledge": "0.0.0" + "zod": "^4.4.3" }, "devDependencies": { "@types/react": "^19.2.15", diff --git a/App/frontend/desktop/src/api/computer-history-contract.ts b/App/frontend/desktop/src/api/computer-history-contract.ts index 3ee3e7a81..7526e2a4b 100644 --- a/App/frontend/desktop/src/api/computer-history-contract.ts +++ b/App/frontend/desktop/src/api/computer-history-contract.ts @@ -8,6 +8,44 @@ export const ComputerHistoryPermissionsSchema = z.object({ export type ComputerHistoryPermissions = z.infer; export type ComputerHistoryPermission = "accessibility" | "inputMonitoring"; +export const ComputerHistoryWeChatReaderPreparationSchema = z.object({ + ready: z.boolean(), + appPath: z.string().nullable(), + error: z.string().nullable(), +}).strict(); +export type ComputerHistoryWeChatReaderPreparation = z.infer; + +export const ComputerHistoryCaptureHealthSchema = z.object({ + status: z.enum(["starting", "healthy", "degraded", "reconnecting", "inactive"]), + lastEventAt: z.string().nullable(), + lastHeartbeatAt: z.string().nullable(), + issues: z.array(z.object({ code: z.string(), count: z.number().int().nonnegative() }).strict()), + limitations: z.array(z.string()), +}).strict(); +export type ComputerHistoryCaptureHealth = z.infer; + +const ObservationBehaviorSchema = z.enum(["observe", "do_not_observe"]); +export const ComputerHistoryObservationSettingsSchema = z.object({ + memory: z.object({ syncEnabled: z.boolean() }).strict().optional(), + observation: z.object({ + defaultApplicationBehavior: ObservationBehaviorSchema, + defaultURLBehavior: ObservationBehaviorSchema, + rules: z.array(z.discriminatedUnion("scope", [ + z.object({ scope: z.literal("app"), bundleID: z.string().min(1), behavior: ObservationBehaviorSchema }).strict(), + z.object({ scope: z.literal("url"), urlDomain: z.string().min(1), behavior: ObservationBehaviorSchema }).strict(), + ])), + }).strict(), +}).strict(); +export type ComputerHistoryObservationSettings = z.infer; + +export const ComputerHistoryApplicationsSchema = z.object({ + applications: z.array(z.object({ bundleId: z.string(), name: z.string() }).strict()), +}).strict(); + +export const ComputerHistoryObservationStatusSchema = z.object({ + state: z.enum(["running", "paused", "stopped", "stopping", "failed"]), +}).strict(); + // The Computer History snapshot contract. // // This lives on its own, free of any browser dependency, so the agent that @@ -24,9 +62,13 @@ export const ComputerHistoryEntrySchema = z.object({ summaryWindow: z.enum(["10min", "6h"]).nullable(), // Keep segments visible when neither metadata nor legacy citations prove coverage. coveredHistoryIds: z.array(z.string()).default([]), + skillMemoryIds: z.array(z.string()).optional(), + // Linked Memory observation. Absent until optional sync succeeds. + memoryId: z.string().optional(), pinned: z.boolean(), eventStreamPath: z.string().nullable(), sourceType: z.enum(["captured", "rollup", "imported", "demo_fixture"]), + summaryStatus: z.enum(["ready", "pending"]).optional(), createdAt: z.string(), // Left out of what the desktop client receives: the timeline never renders a // summary's body, and it was most of every response. The agent reads bodies @@ -55,6 +97,13 @@ export const ComputerHistoryWorkflowSchema = z.object({ // this client enforces. The schema is strict, so a field added on one side and // not the other breaks the page rather than being ignored. export const ComputerHistorySnapshotSchema = z.object({ + memorySync: z.object({ + lastSyncedAt: z.string().nullable(), + error: z.string().nullable(), + // Older agents omit the Skill status; new agents always return it. + skillError: z.string().nullable().optional(), + pendingDeletionCount: z.number().int().nonnegative().optional(), + }).strict().optional(), observation: z.object({ state: z.enum(["running", "paused", "stopped", "stopping", "failed"]), startedAt: z.string().nullable(), @@ -65,9 +114,16 @@ export const ComputerHistorySnapshotSchema = z.object({ narrationErrorCategory: z.literal("quota_exhausted").nullable().optional(), modelSource: z.enum(["account", "byok"]).nullable().optional(), permissions: ComputerHistoryPermissionsSchema.optional(), + captureHealth: ComputerHistoryCaptureHealthSchema.optional(), }).strict(), histories: z.array(ComputerHistoryEntrySchema), workflows: z.array(ComputerHistoryWorkflowSchema), + wechat: z.object({ + enabled: z.boolean(), + connection: z.enum(["disabled", "unavailable", "needs_setup", "connecting", "connected", "error"]), + phase: z.string().nullable(), + error: z.string().nullable(), + }).strict().optional(), privacy: z.object({ screenshots: z.literal(false), audio: z.literal(false), diff --git a/App/frontend/desktop/src/api/memmy-agent-client.ts b/App/frontend/desktop/src/api/memmy-agent-client.ts index 97ccbd236..597007a32 100644 --- a/App/frontend/desktop/src/api/memmy-agent-client.ts +++ b/App/frontend/desktop/src/api/memmy-agent-client.ts @@ -10,8 +10,15 @@ import { ApplicationIconSchema, ComputerHistorySnapshotSchema, ComputerHistoryPermissionsSchema, + ComputerHistoryObservationSettingsSchema, + ComputerHistoryApplicationsSchema, + ComputerHistoryObservationStatusSchema, + ComputerHistoryWeChatReaderPreparationSchema, + type ComputerHistoryObservationSettings, type ComputerHistoryPermission, type ComputerHistoryPermissions, + type ComputerHistoryWeChatReaderPreparation, + type ComputerHistoryCaptureHealth, } from "./computer-history-contract.js"; export { ComputerHistorySnapshotSchema }; @@ -67,11 +74,16 @@ export type ComputerHistoryEntry = { applications: string[]; summaryWindow: "10min" | "6h" | null; coveredHistoryIds: string[]; + /** IDs of Memory Skills linked by the optional History adapter. */ + skillMemoryIds?: string[]; + /** Linked Memory observation, available after optional sync succeeds. */ + memoryId?: string; pinned: boolean; eventStreamPath: string | null; id: string; title: string; sourceType: ComputerHistorySourceType; + summaryStatus?: "ready" | "pending"; createdAt: string; /** Not sent to the client: the timeline shows the description, never the body. */ markdown?: string; @@ -89,6 +101,7 @@ export type ComputerHistoryWorkflow = { }; export type ComputerHistorySnapshot = { + memorySync?: { lastSyncedAt: string | null; error: string | null; skillError?: string | null }; observation: { state: "running" | "paused" | "stopped" | "stopping" | "failed"; startedAt: string | null; @@ -99,9 +112,16 @@ export type ComputerHistorySnapshot = { narrationErrorCategory?: "quota_exhausted" | null; modelSource?: "account" | "byok" | null; permissions?: ComputerHistoryPermissions; + captureHealth?: ComputerHistoryCaptureHealth; }; histories: ComputerHistoryEntry[]; workflows: ComputerHistoryWorkflow[]; + wechat?: { + enabled: boolean; + connection: "disabled" | "unavailable" | "needs_setup" | "connecting" | "connected" | "error"; + phase: string | null; + error: string | null; + }; privacy: { screenshots: false; audio: false; @@ -658,6 +678,7 @@ export type MemmyAgentModelError = { export type MemmyAgentWsEvent = { event: string; + replayed?: boolean; connection_generation?: number; chat_id?: string; client_id?: string; @@ -700,18 +721,72 @@ export type MemmyAgentWsEvent = { [key: string]: unknown; }; +/** Scheduled deliveries are independent of the conversation's active turn. */ +export function isCronProactiveMessage(event: MemmyAgentWsEvent): boolean { + return event.event === "message" + && event.metadata?.proactiveDelivery === "cron" + && event.kind !== "progress" + && event.kind !== "tool_hint" + && event.kind !== "reasoning"; +} + export type MemmyAgentRunLifecycleEvent = MemmyAgentWsEvent & { event: "run_status" | "turn_end" | "stop_result" | "run_status_snapshot"; chat_id: string; }; +export type NativeAppApproval = { + platform: 'darwin' | 'win32'; + appId: string; + displayName: string; + allowedAt: string; +}; + +const NativeAppApprovalsSchema = z.object({ apps: z.array(z.object({ + platform: z.enum(['darwin', 'win32']), appId: z.string(), displayName: z.string(), + allowedAt: z.string(), +})) }); + +const ExcelControlStatusSchema = z.object({ + configured: z.boolean(), connected: z.boolean(), connectionCount: z.number().int(), + setupCompleted: z.boolean(), manifestUrl: z.string().url().nullable(), + manualInstallPath: z.string().min(1).nullable().optional(), + lockedUse: z.object({ supported: z.boolean(), reason: z.string() }), + scope: z.enum(["application", "workbook"]).optional(), + connectionMode: z.enum(["native", "officejs"]).optional(), + available: z.boolean().optional(), enabled: z.boolean().optional(), + workbooks: z.array(z.object({ id: z.string(), name: z.string() })).optional(), + activeWorkbookId: z.string().nullable().optional(), + permissionRequired: z.boolean().optional(), + error: z.string().nullable().optional(), +}); + +export type ExcelControlStatus = z.infer; + export interface MemmyAgentClient { bootstrap(options?: { force?: boolean }): Promise; getSettings(): Promise; + getComputerUseSetting(): Promise<{ enabled: boolean; available: boolean; binaryAvailable: boolean }>; + setComputerUseEnabled(enabled: boolean): Promise<{ enabled: boolean; available: boolean; binaryAvailable: boolean; restartRequired?: boolean }>; + getNativeAppApprovals(): Promise; + revokeNativeAppApproval(platform: 'darwin' | 'win32', appId: string): Promise; + getExternalBrowserStatus(): Promise<{ connected: Array<"chrome" | "edge">; + claims: Array<{ browser: "chrome" | "edge"; title: string; url: string }> }>; + getExcelAddinStatus(): Promise; + enableExcelAddin(): Promise; + setExcelAddinEnabled(enabled: boolean): Promise; + revealExcelAddin(): Promise; getComputerHistory(): Promise; - setComputerHistoryModel(preset: string | null): Promise; + getComputerHistoryObservationStatus(): Promise<{ state: ComputerHistorySnapshot["observation"]["state"] }>; + getComputerHistorySettings(): Promise; + updateComputerHistorySettings(settings: ComputerHistoryObservationSettings): Promise; + listComputerHistoryApplications(): Promise>; + refreshComputerHistoryModel(): Promise; checkComputerHistoryPermissions(): Promise; openComputerHistoryPermission(permission: ComputerHistoryPermission, mode?: "request" | "settings"): Promise; + setComputerHistoryWeChatAccess(enabled: boolean): Promise; + prepareComputerHistoryWeChatReader(): Promise; + connectComputerHistoryWeChat(): Promise; deleteComputerHistory(historyId: string): Promise; clearComputerHistories(scope: "today" | "all"): Promise; pinComputerHistory(historyId: string, pinned: boolean): Promise; @@ -815,6 +890,7 @@ export interface MemmyAgentWebSocketConnection { restart(chatId: string): void; status(chatId: string): void; historyDag(chatId: string): void; + acknowledgeCronDelivery?(deliveryId: string): void; onChat(chatId: string, handler: (event: MemmyAgentWsEvent) => void): MemmyAgentUnsubscribe; onStatusResult(handler: (chatId: string, content: string) => void): MemmyAgentUnsubscribe; onHistoryDagResult(handler: (chatId: string, content: string, payload: HistoryDagPayload) => void): MemmyAgentUnsubscribe; @@ -1061,9 +1137,64 @@ class HttpMemmyAgentClient implements MemmyAgentClient { return this.request("/api/settings", AgentSettingsSchema); } - async setComputerHistoryModel(preset: string | null): Promise { + async getComputerUseSetting(): Promise<{ enabled: boolean; available: boolean; binaryAvailable: boolean }> { + const payload = await this.request("/api/settings/mcp-presets", z.object({ + presets: z.array(z.object({ + name: z.string(), installed: z.boolean(), available: z.boolean(), binary_available: z.boolean().optional() + }).passthrough()) + }).passthrough()); + const preset = payload.presets.find((entry) => entry.name === "memmy_computer_use"); + return { enabled: preset?.installed === true, available: preset?.available === true, binaryAvailable: preset?.binary_available === true }; + } + + async setComputerUseEnabled(enabled: boolean): Promise<{ enabled: boolean; available: boolean; binaryAvailable: boolean; restartRequired?: boolean }> { + const changed = await this.request( + `/api/settings/mcp-presets/${enabled ? "enable" : "remove"}?name=memmy_computer_use`, + z.object({ presets: z.array(z.unknown()), requires_restart: z.boolean().optional() }).passthrough(), + { method: "POST" } + ); + return { ...await this.getComputerUseSetting(), restartRequired: changed.requires_restart === true }; + } + + async getNativeAppApprovals(): Promise { + return (await this.request('/api/settings/computer-use/apps', NativeAppApprovalsSchema)).apps; + } + + async revokeNativeAppApproval(platform: 'darwin' | 'win32', appId: string): Promise { + const params = new URLSearchParams({ platform, app_id: appId }); + return (await this.request(`/api/settings/computer-use/apps?${params}`, NativeAppApprovalsSchema, + { method: 'POST' })).apps; + } + + async getExternalBrowserStatus(): Promise<{ connected: Array<"chrome" | "edge">; + claims: Array<{ browser: "chrome" | "edge"; title: string; url: string }> }> { + return this.request("/api/settings/computer-use/browsers", z.object({ + connected: z.array(z.enum(["chrome", "edge"])), + claims: z.array(z.object({ + browser: z.enum(["chrome", "edge"]), title: z.string(), url: z.string().url(), + })), + })); + } + + async getExcelAddinStatus(): Promise { + return this.request("/api/settings/computer-use/excel", ExcelControlStatusSchema); + } + + async enableExcelAddin(): ReturnType { + return this.request("/api/settings/computer-use/excel", ExcelControlStatusSchema, { method: "POST" }); + } + + async setExcelAddinEnabled(enabled: boolean): ReturnType { + return this.request("/api/settings/computer-use/excel", ExcelControlStatusSchema, { method: "POST", body: { enabled } }); + } + + async revealExcelAddin(): ReturnType { + return this.request("/api/settings/computer-use/excel", ExcelControlStatusSchema, { method: "POST", body: { reveal: true } }); + } + + async refreshComputerHistoryModel(): Promise { return this.request("/api/computer-history/model", ComputerHistorySnapshotSchema, { - method: "POST", body: { model_preset: preset }, + method: "POST", body: {}, }); } @@ -1071,6 +1202,43 @@ class HttpMemmyAgentClient implements MemmyAgentClient { return this.request("/api/computer-history", ComputerHistorySnapshotSchema); } + async getComputerHistoryObservationStatus(): Promise<{ state: ComputerHistorySnapshot["observation"]["state"] }> { + return this.request("/api/computer-history/observation/status", ComputerHistoryObservationStatusSchema); + } + + async setComputerHistoryWeChatAccess(enabled: boolean): Promise { + return this.request("/api/computer-history/wechat/consent", ComputerHistorySnapshotSchema, { + method: "POST", body: { enabled }, + }); + } + + async connectComputerHistoryWeChat(): Promise { + return this.request("/api/computer-history/wechat/connect", ComputerHistorySnapshotSchema, { + method: "POST", body: {}, + }); + } + + async prepareComputerHistoryWeChatReader(): Promise { + return this.request("/api/computer-history/wechat/prepare", ComputerHistoryWeChatReaderPreparationSchema, { + method: "POST", body: {}, + }); + } + + async getComputerHistorySettings(): Promise { + return this.request("/api/computer-history/settings", ComputerHistoryObservationSettingsSchema); + } + + async updateComputerHistorySettings(settings: ComputerHistoryObservationSettings): Promise { + return this.request("/api/computer-history/settings", ComputerHistoryObservationSettingsSchema, { + method: "POST", body: { settings }, + }); + } + + async listComputerHistoryApplications(): Promise> { + const result = await this.request("/api/computer-history/apps", ComputerHistoryApplicationsSchema); + return result.applications; + } + async deleteComputerHistory(historyId: string): Promise { return this.request("/api/computer-history/delete", ComputerHistorySnapshotSchema, { method: "POST", @@ -1591,10 +1759,21 @@ class MemmyAgentWebSocketSession implements MemmyAgentWebSocketConnection { resolve, reject, timer: setTimeout(() => { - if (this.pendingNewChat === pending) { - this.pendingNewChat = null; + if (this.pendingNewChat !== pending) return; + this.pendingNewChat = null; + reject(new Error("request_timeout")); + // A connected transport can still be stuck behind a gateway operation + // (for example, a tool that is slow to stop). Rebuild it so retries do + // not keep sending new_chat to the same unresponsive connection. + const socket = this.socket; + if (socket && this.isCurrentSocket(socket, expectedGeneration)) { + this.handleClose(socket, expectedGeneration); + try { + socket.close(4000, "new chat timeout"); + } catch { + // The old transport is already detached; recovery still proceeds. + } } - reject(new Error("newChat timed out")); }, timeoutMs) }; this.pendingNewChat = pending; @@ -1906,6 +2085,12 @@ class MemmyAgentWebSocketSession implements MemmyAgentWebSocketConnection { this.queueControl({ type: "history_dag", chat_id: chatId }); } + acknowledgeCronDelivery(deliveryId: string): void { + const normalized = deliveryId.trim(); + if (!normalized) return; + this.queueControl({ type: "cron_delivery_ack", delivery_id: normalized }); + } + onChat(chatId: string, handler: (event: MemmyAgentWsEvent) => void): MemmyAgentUnsubscribe { let handlers = this.chatHandlers.get(chatId); if (!handlers) { @@ -2509,6 +2694,10 @@ class MemmyAgentWebSocketSession implements MemmyAgentWebSocketConnection { return; } + // Proactive results are handled by the global listener immediately. Do not + // replay them a second time when the user later opens this conversation. + if (isCronProactiveMessage(event)) return; + const queue = this.pendingInboundByChat.get(chatId) ?? []; queue.push(event); const overflow = queue.length - PENDING_INBOUND_MAX; diff --git a/App/frontend/desktop/src/api/memory-runtime-client.ts b/App/frontend/desktop/src/api/memory-runtime-client.ts index 573ee2a6a..d85e88c1d 100644 --- a/App/frontend/desktop/src/api/memory-runtime-client.ts +++ b/App/frontend/desktop/src/api/memory-runtime-client.ts @@ -61,6 +61,10 @@ import { type RuntimeConfig } from "@memmy/local-api-contracts"; import { ApiRequestError, requestJson } from "./http.js"; +import { z } from "zod"; + +const ApproveObservedSkillOutputSchema = z.object({ id: z.string(), status: z.literal("activated"), serverTime: z.string() }); +const UpdateObservedSkillOutputSchema = ApproveObservedSkillOutputSchema.extend({ version: z.number().int().positive() }); export const MEMORY_RUNTIME_ENDPOINTS = [ "GET /api/v1/health", @@ -71,6 +75,8 @@ export const MEMORY_RUNTIME_ENDPOINTS = [ "POST /api/v1/turns/:turnId/complete", "POST /api/v1/memory/search", "POST /api/v1/memory/add", + "POST /api/v1/evidence/skills/:id/approve", + "PATCH /api/v1/evidence/skills/:id", "POST /api/v1/memory/processing/status", "POST /api/v1/memory/:id/processing/retry", "GET /api/v1/memory/:id", @@ -85,7 +91,7 @@ export const MEMORY_RUNTIME_ENDPOINTS = [ ] as const; export interface MemoryRuntimeClient { - health(): Promise; + health(signal?: AbortSignal): Promise; reloadConfig(input?: MemoryReloadConfigInput): Promise; openSession(input: OpenSessionInput): Promise; closeSession(sessionId: string, input: CloseSessionInput): Promise; @@ -95,6 +101,8 @@ export interface MemoryRuntimeClient { addMemory(input: AddMemoryInput): Promise; getMemory(id: string): Promise; deleteMemory(id: string): Promise; + approveObservedSkill?(id: string): Promise<{ id: string; status: "activated"; serverTime: string }>; + updateObservedSkill?(id: string, input: { body: string; expectedVersion: number }): Promise<{ id: string; status: "activated"; version: number; serverTime: string }>; recallEvidence(queryId: string): Promise; getMemoryProcessingStatus(memoryIds: string[]): Promise; retryMemoryProcessing(id: string): Promise; @@ -108,8 +116,8 @@ export interface MemoryRuntimeClient { export function createHttpMemoryRuntimeClient(config: RuntimeConfig): MemoryRuntimeClient { return { - async health() { - return requestJson({ config, path: "/api/v1/health", schema: MemoryHealthSnapshotSchema }); + async health(signal) { + return requestJson({ config, path: "/api/v1/health", schema: MemoryHealthSnapshotSchema, init: { signal } }); }, async reloadConfig(input = {}) { @@ -173,6 +181,16 @@ export function createHttpMemoryRuntimeClient(config: RuntimeConfig): MemoryRunt }); }, + async approveObservedSkill(id) { + return requestJson({ config, path: `/api/v1/evidence/skills/${encodeURIComponent(id)}/approve`, + schema: ApproveObservedSkillOutputSchema, body: {} }); + }, + + async updateObservedSkill(id, input) { + return requestJson({ config, path: `/api/v1/evidence/skills/${encodeURIComponent(id)}`, + schema: UpdateObservedSkillOutputSchema, body: input, init: { method: "PATCH" } }); + }, + async recallEvidence(queryId) { return requestJson({ config, diff --git a/App/frontend/desktop/src/api/tests/memmy-agent-client.test.ts b/App/frontend/desktop/src/api/tests/memmy-agent-client.test.ts index 63f775c70..45997bfcd 100644 --- a/App/frontend/desktop/src/api/tests/memmy-agent-client.test.ts +++ b/App/frontend/desktop/src/api/tests/memmy-agent-client.test.ts @@ -81,6 +81,74 @@ afterEach(() => { }); describe("memmy-agent client", () => { + it("preserves application-wide Excel status on the compatible Excel routes", async () => { + let enabled = false; + const writes: unknown[] = []; + const status = () => ({ scope: "application", connectionMode: "native", available: true, enabled, + configured: enabled, connected: enabled, connectionCount: enabled ? 1 : 0, setupCompleted: enabled, + manifestUrl: null, manualInstallPath: null, + workbooks: enabled ? [{ id: "book1", name: "Budget.xlsx" }] : [], activeWorkbookId: enabled ? "book1" : null, + permissionRequired: false, error: null, lockedUse: { supported: false, reason: "nativeLockGuardianUnavailable" } }); + const fetchMock = vi.fn(async (input: RequestInfo | URL, init?: RequestInit) => { + const url = new URL(String(input)); + if (url.pathname === "/webui/bootstrap") return json(bootstrap); + expect(url.pathname).toBe("/api/settings/computer-use/excel"); + expect((init?.headers as Record).Authorization).toBe("Bearer agent-token"); + if (init?.method === "POST") { + const body = init.body ? JSON.parse(String(init.body)) : null; + writes.push(body); + if (body?.enabled !== undefined) enabled = body.enabled; + else if (!body?.reveal) enabled = true; + } + return json(status()); + }); + const client = createMemmyAgentClient({ baseUrl: "http://127.0.0.1:18980", fetchFn: fetchMock as typeof fetch }); + await expect(client.getExcelAddinStatus()).resolves.toEqual(status()); + await expect(client.setExcelAddinEnabled(true)).resolves.toMatchObject({ scope: "application", connectionMode: "native", + enabled: true, workbooks: [{ id: "book1", name: "Budget.xlsx" }], activeWorkbookId: "book1", permissionRequired: false }); + const disabled = await client.setExcelAddinEnabled(false); + expect(disabled).toEqual(status()); + await expect(client.enableExcelAddin()).resolves.toMatchObject({ scope: "application", enabled: true }); + await expect(client.revealExcelAddin()).resolves.toMatchObject({ scope: "application", enabled: true }); + expect(writes).toEqual([{ enabled: true }, { enabled: false }, null, { reveal: true }]); + }); + + it("still accepts the older Excel add-in status without application fields", async () => { + const legacy = { configured: true, connected: false, connectionCount: 0, setupCompleted: true, + manifestUrl: "https://localhost:32177/manifest.xml", manualInstallPath: "/tmp/Memmy-Excel.xml", + lockedUse: { supported: false, reason: "nativeLockGuardianUnavailable" } }; + const fetchMock = vi.fn(async (input: RequestInfo | URL) => + json(new URL(String(input)).pathname === "/webui/bootstrap" ? bootstrap : legacy)); + const client = createMemmyAgentClient({ baseUrl: "http://127.0.0.1:18980", fetchFn: fetchMock as typeof fetch }); + await expect(client.getExcelAddinStatus()).resolves.toEqual(legacy); + }); + + it("reads and changes the computer-use MCP setting through authenticated gateway routes", async () => { + let enabled = false; + const paths: string[] = []; + const fetchMock = vi.fn(async (input: RequestInfo | URL, init?: RequestInit) => { + const url = new URL(String(input)); + if (url.pathname === "/webui/bootstrap") return json(bootstrap); + paths.push(url.pathname + url.search); + expect((init?.headers as Record).Authorization).toBe("Bearer agent-token"); + if (url.pathname.endsWith("/enable")) { + expect(init?.method).toBe("POST"); + enabled = true; + } + if (url.pathname.endsWith("/remove")) { + expect(init?.method).toBe("POST"); + enabled = false; + } + return json({ presets: [{ name: "memmy_computer_use", installed: enabled, available: enabled, binary_available: true }] }); + }); + const client = createMemmyAgentClient({ baseUrl: "http://127.0.0.1:18980", fetchFn: fetchMock as typeof fetch }); + await expect(client.getComputerUseSetting()).resolves.toMatchObject({ enabled: false, binaryAvailable: true }); + await expect(client.setComputerUseEnabled(true)).resolves.toMatchObject({ enabled: true }); + await expect(client.setComputerUseEnabled(false)).resolves.toMatchObject({ enabled: false }); + expect(paths).toContain("/api/settings/mcp-presets/enable?name=memmy_computer_use"); + expect(paths).toContain("/api/settings/mcp-presets/remove?name=memmy_computer_use"); + }); + it("syncs the history model and retains its actual model source in the response", async () => { const snapshot = { observation: { state: "running", startedAt: null, segmentId: null, segmentStartedAt: null, error: null, narrationError: null, modelSource: "byok" }, @@ -92,11 +160,11 @@ describe("memmy-agent client", () => { if (url.pathname === "/webui/bootstrap") return json(bootstrap); expect(url.pathname).toBe("/api/computer-history/model"); expect(init?.method).toBe("POST"); - expect(JSON.parse(String(init?.body))).toEqual({ model_preset: "my-api-model" }); + expect(JSON.parse(String(init?.body))).toEqual({}); return json(snapshot); }); const client = createMemmyAgentClient({ baseUrl: "http://127.0.0.1:18980", fetchFn: fetchMock as typeof fetch }); - await expect(client.setComputerHistoryModel("my-api-model")).resolves.toEqual(snapshot); + await expect(client.refreshComputerHistoryModel()).resolves.toEqual(snapshot); }); it.each(["today", "all"] as const)("clears %s Computer History with the server-side clear endpoint", async (scope) => { @@ -1012,9 +1080,10 @@ describe("memmy-agent client", () => { }); }); - it("newChat rejects on timeout and clears pending state", async () => { + it("newChat invalidates a timed-out socket and recovers on a fresh ready generation", async () => { vi.useFakeTimers(); const sockets: FakeSocket[] = []; + const events: MemmyAgentWsEvent[] = []; const client = createMemmyAgentClient({ baseUrl: "https://agent.local:18980", clientId: "frontend-test", @@ -1026,15 +1095,47 @@ describe("memmy-agent client", () => { } }); - const connection = await connectReady(client, sockets); + const connection = await connectReady(client, sockets, (event) => events.push(event)); const pending = connection.newChat(1, 100); - const rejection = expect(pending).rejects.toThrow("newChat timed out"); + const rejection = expect(pending).rejects.toThrow("request_timeout"); await vi.advanceTimersByTimeAsync(100); await rejection; - const second = connection.newChat(1); - const request = JSON.parse(sockets[0]!.sent.at(-1)!); + expect(sockets[0]?.closeCalls).toEqual([{ code: 4000, reason: "new chat timeout" }]); + expect(connection.getReadyGeneration()).toBeNull(); + expect(events.at(-1)).toMatchObject({ event: "connection_closed", connection_generation: 1 }); + await expect(connection.newChat(1)).rejects.toThrow("Agent gateway is not ready"); + + await vi.advanceTimersByTimeAsync(499); + expect(sockets).toHaveLength(1); + await vi.advanceTimersByTimeAsync(1); + expect(sockets).toHaveLength(2); + expect(sockets[1]?.sent).toEqual([]); + expect(connection.getReadyGeneration()).toBeNull(); + + sockets[1]?.emit({ event: "ready", chat_id: "ready-2" }); + expect(connection.getReadyGeneration()).toBe(2); + let secondSettled = false; + const second = connection.newChat(2).then((result) => { + secondSettled = true; + return result; + }); + const request = JSON.parse(sockets[1]!.sent.at(-1)!); + const eventCount = events.length; sockets[0]?.emit({ + event: "attached", + chat_id: "stale-server-chat", + client_request_id: request.client_request_id, + model_preset: "desktop-openai-gpt-5", + model_selection: modelSelectionWire + }); + sockets[0]?.emitClose(); + await Promise.resolve(); + expect(secondSettled).toBe(false); + expect(events).toHaveLength(eventCount); + expect(connection.getReadyGeneration()).toBe(2); + + sockets[1]?.emit({ event: "attached", chat_id: "server-chat", client_request_id: request.client_request_id, @@ -1046,6 +1147,39 @@ describe("memmy-agent client", () => { modelPreset: "desktop-openai-gpt-5", modelSelection }); + expect(sockets[0]?.sent.map((item) => JSON.parse(item).type)).toEqual(["new_chat"]); + expect(sockets[1]?.sent.map((item) => JSON.parse(item))).toEqual([ + { type: "attach", chat_id: "ready-chat" }, + request + ]); + connection.close(); + }); + + it("explicit close cancels newChat and its timeout without reconnecting", async () => { + vi.useFakeTimers(); + const sockets: FakeSocket[] = []; + const client = createMemmyAgentClient({ + baseUrl: "https://agent.local:18980", + clientId: "frontend-test", + fetchFn: vi.fn(async () => json(bootstrap)) as typeof fetch, + webSocketFactory: (url) => { + const socket = new FakeSocket(url); + sockets.push(socket); + return socket; + } + }); + + const connection = await connectReady(client, sockets); + const pending = connection.newChat(1, 100); + const rejection = expect(pending).rejects.toThrow("newChat cancelled"); + connection.close(); + + await rejection; + sockets[0]?.emitClose(); + await vi.advanceTimersByTimeAsync(1_000); + expect(connection.getReadyGeneration()).toBeNull(); + expect(sockets).toHaveLength(1); + expect(sockets[0]?.closeCalls).toEqual([{ code: undefined, reason: undefined }]); }); it("newChat rejects on socket close and clears pending state", async () => { @@ -1213,6 +1347,26 @@ describe("memmy-agent client", () => { unsubscribe(); }); + it("delivers a never-opened chat's cron result globally without retaining a pending copy", async () => { + const sockets: FakeSocket[] = []; + const client = createMemmyAgentClient({ + baseUrl: "https://agent.local:18980", clientId: "frontend-test", + fetchFn: vi.fn(async () => json(bootstrap)) as typeof fetch, + webSocketFactory: (url) => { const socket = new FakeSocket(url); sockets.push(socket); return socket; }, + }); + const globalEvents: MemmyAgentWsEvent[] = []; + const connection = await connectReady(client, sockets, (event) => globalEvents.push(event)); + globalEvents.length = 0; + sockets[0]?.emit({ event: "message", chat_id: "never-opened", text: "暂无微信新消息", + metadata: { proactiveDelivery: "cron", deliveryId: "delivery-1" } }); + expect(globalEvents).toHaveLength(1); + const chatEvents: MemmyAgentWsEvent[] = []; + const unsubscribe = connection.onChat("never-opened", (event) => chatEvents.push(event)); + expect(chatEvents).toEqual([]); + unsubscribe(); + connection.close(); + }); + it("routes status_result events only through the status result handler", async () => { const sockets: FakeSocket[] = []; const client = createMemmyAgentClient({ @@ -2122,6 +2276,29 @@ describe("memmy-agent client", () => { ]); }); + it("sends cron consumption acknowledgements through the ready control queue", async () => { + vi.useFakeTimers(); + const sockets: FakeSocket[] = []; + const client = createMemmyAgentClient({ baseUrl: "https://agent.local:18980", clientId: "frontend-test", + fetchFn: vi.fn(async () => json(bootstrap)) as typeof fetch, + webSocketFactory: (url) => { const socket = new FakeSocket(url); sockets.push(socket); return socket; }, + }); + const connection = await connectReady(client, sockets); + connection.acknowledgeCronDelivery?.(" live-1 "); + connection.acknowledgeCronDelivery?.(" "); + expect(sockets[0]?.sent.map((raw) => JSON.parse(raw))).toEqual([{ type: "cron_delivery_ack", delivery_id: "live-1" }]); + sockets[0]?.emitClose(); + connection.acknowledgeCronDelivery?.("after-disconnect"); + await vi.advanceTimersByTimeAsync(500); + expect(sockets[1]?.sent).toEqual([]); + sockets[1]?.emit({ event: "ready", chat_id: "ready-2" }); + expect(sockets[1]?.sent.map((raw) => JSON.parse(raw))).toEqual([ + { type: "attach", chat_id: "ready-chat" }, + { type: "cron_delivery_ack", delivery_id: "after-disconnect" }, + ]); + connection.close(); + }); + it("ignores all callbacks from an old socket after a newer generation starts", async () => { vi.useFakeTimers(); const sockets: FakeSocket[] = []; diff --git a/App/frontend/desktop/src/api/tests/memory-runtime-client.test.ts b/App/frontend/desktop/src/api/tests/memory-runtime-client.test.ts index e8d80fede..f6f825111 100644 --- a/App/frontend/desktop/src/api/tests/memory-runtime-client.test.ts +++ b/App/frontend/desktop/src/api/tests/memory-runtime-client.test.ts @@ -12,7 +12,7 @@ describe("memory runtime client", () => { }); it("declares the memory runtime endpoints exposed under /api/v1", () => { - expect(MEMORY_RUNTIME_ENDPOINTS).toHaveLength(19); + expect(MEMORY_RUNTIME_ENDPOINTS).toHaveLength(21); expect(MEMORY_RUNTIME_ENDPOINTS).toEqual([ "GET /api/v1/health", "POST /api/v1/admin/reload-config", @@ -22,6 +22,8 @@ describe("memory runtime client", () => { "POST /api/v1/turns/:turnId/complete", "POST /api/v1/memory/search", "POST /api/v1/memory/add", + "POST /api/v1/evidence/skills/:id/approve", + "PATCH /api/v1/evidence/skills/:id", "POST /api/v1/memory/processing/status", "POST /api/v1/memory/:id/processing/retry", "GET /api/v1/memory/:id", @@ -36,6 +38,17 @@ describe("memory runtime client", () => { ]); }); + it("saves observed Skill content with the original version through PATCH", async () => { + const fetchMock = vi.fn(async () => new Response(JSON.stringify({ id: "skill/1", status: "activated", + version: 2, serverTime: "2026-10-09T00:00:00Z" }), { status: 200 })); + vi.stubGlobal("fetch", fetchMock); + const client = createHttpMemoryRuntimeClient(runtimeConfig); + const input = { body: "# Revised skill\n\n1. Check current state.", expectedVersion: 1 }; + await expect(client.updateObservedSkill!("skill/1", input)).resolves.toMatchObject({ version: 2, status: "activated" }); + expect(fetchMock).toHaveBeenCalledWith(new URL("/api/v1/evidence/skills/skill%2F1", runtimeConfig.baseUrl), + expect.objectContaining({ method: "PATCH", body: JSON.stringify(input) })); + }); + it("calls memory health with runtime token through requestJson", async () => { const fetchMock = vi.fn(async () => { return new Response( @@ -79,6 +92,23 @@ describe("memory runtime client", () => { ); }); + it("cancels the actual health fetch through the supplied AbortSignal", async () => { + const controller = new AbortController(); + const fetchMock = vi.fn((_url: URL | RequestInfo, init?: RequestInit) => new Promise((_resolve, reject) => { + init?.signal?.addEventListener("abort", () => reject(init.signal?.reason), { once: true }); + })); + vi.stubGlobal("fetch", fetchMock); + const client = createHttpMemoryRuntimeClient(runtimeConfig); + const failure = new Error("Health probe cancelled"); + const cancelled = expect(client.health(controller.signal)).rejects.toBe(failure); + controller.abort(failure); + await cancelled; + expect(fetchMock).toHaveBeenCalledWith( + new URL("/api/v1/health", runtimeConfig.baseUrl), + expect.objectContaining({ signal: controller.signal }) + ); + }); + it("reloads memory config through the local API admin route", async () => { const fetchMock = vi.fn(async () => { return new Response( diff --git a/App/frontend/desktop/src/app.tsx b/App/frontend/desktop/src/app.tsx index a9ff525c2..f62233178 100644 --- a/App/frontend/desktop/src/app.tsx +++ b/App/frontend/desktop/src/app.tsx @@ -19,6 +19,7 @@ import { AppRouter } from "./app/router.js"; import { UpdateCoordinatorProvider } from "./app/update-coordinator.js"; import { CampaignPromptHost } from "./components/campaign-prompt-host.js"; import { GithubStarPromptHost } from "./components/github-star-prompt-host.js"; +import { HistoryLaunchPromptHost } from "./components/history-launch-prompt-host.js"; import { InviteResultToast } from "./components/invite-result-toast.js"; import { NotificationCenterProvider } from "./components/notification-center.js"; import { TokenCreditToastHost } from "./components/token-credit-toast-host.js"; @@ -42,6 +43,7 @@ import { createEventsConnection } from "./api/events.js"; import { MemmyAgentRequestError, type MemmyAgentClient } from "./api/memmy-agent-client.js"; import { getRuntimeConfig } from "./api/runtime-config.js"; import { readHistoryPermissionSetup } from "./pages/memory/computer-history-permission-state.js"; +import { shouldReturnToWeChatSettings } from "./pages/memory/computer-history-wechat-permission-state.js"; import { clearMemoryPanelCache } from "./pages/memory/memory-panel-cache.js"; import { readLocalNickname } from "./app/nickname.js"; import { @@ -58,6 +60,7 @@ import { } from "./state/app-actions.js"; import type { AgentState } from "./state/agent-chat-slice.js"; import { useAppState } from "./state/app-state.js"; +import { useComputerHistoryTrayIndicator } from "./app/computer-history-tray-indicator.js"; /** Handles app. */ export function App() { @@ -72,12 +75,10 @@ export function App() { function RuntimeApp() { const { state, dispatch } = useAppState(); const { clients, setClients } = useApiClients(); - const historyModelScope = state.agent.currentChatId ?? "draft-new-task"; + useComputerHistoryTrayIndicator(clients?.memmyAgent ?? null); useComputerHistoryModelSync({ client: clients?.memmyAgent ?? null, enabled: Boolean(state.bootstrap && state.modelConfig), - preset: state.agent.pendingPresetByScope[historyModelScope] - ?? state.agent.committedModelSelectionByScope[historyModelScope]?.presetId ?? null, revision: JSON.stringify([state.bootstrap?.app.userMode, state.account.userId, state.modelConfig?.configRevision]), }); const { track } = useAnalytics(); @@ -230,10 +231,11 @@ function RuntimeApp() { guidanceCompleted, modelConfig }); + const returnToWeChatSettings = isComputerHistorySupported() && await shouldReturnToWeChatSettings(); const initialPath = resolveLaunchInitialView({ defaultPath: defaultInitialPath, currentRoute, - launchRouteOverride: launchRouteOverride ?? (isComputerHistorySupported() && readHistoryPermissionSetup() && launchModeOverride !== "pet" ? "/memory" : null), + launchRouteOverride: launchRouteOverride ?? (isComputerHistorySupported() && (readHistoryPermissionSetup() || returnToWeChatSettings) && launchModeOverride !== "pet" ? "/memory" : null), launchModeOverride, petIntent }); @@ -360,6 +362,7 @@ function RuntimeApp() { + diff --git a/App/frontend/desktop/src/app/account-channel.ts b/App/frontend/desktop/src/app/account-channel.ts index e85e0eef4..f72e825c8 100644 --- a/App/frontend/desktop/src/app/account-channel.ts +++ b/App/frontend/desktop/src/app/account-channel.ts @@ -13,5 +13,9 @@ export function resolveDesktopDisplayLanguage( configuredLanguage: string | undefined, rawChannel = import.meta.env.MEMMY_ACCOUNT_CHANNEL ): DesktopDisplayLanguage { - return resolveMemoryLanguage(configuredLanguage, resolveDesktopAccountChannel(rawChannel)); + // Output language tags may grow independently; display language must have an + // App message catalog before it can be selected as the current UI language. + if (configuredLanguage === "zh-CN" || configuredLanguage === "en-US") return configuredLanguage; + const resolvedDefault = resolveMemoryLanguage(undefined, resolveDesktopAccountChannel(rawChannel)); + return resolvedDefault === "en-US" ? "en-US" : "zh-CN"; } diff --git a/App/frontend/desktop/src/app/agent-runtime-bridge.tsx b/App/frontend/desktop/src/app/agent-runtime-bridge.tsx index 893cf27e2..430b5ae44 100644 --- a/App/frontend/desktop/src/app/agent-runtime-bridge.tsx +++ b/App/frontend/desktop/src/app/agent-runtime-bridge.tsx @@ -2,6 +2,7 @@ import { createContext, useCallback, useContext, useEffect, useRef, useState, type ReactNode } from "react"; import { MemmyAgentRequestError, + isCronProactiveMessage, type MemmyAgentClient, type MemmyAgentProject, type MemmyAgentRunStatusSnapshot, @@ -14,6 +15,9 @@ import { agentActions, appActions, createAgentOperationError, type AppAction } f import { updateSidebarStateForTask, type AgentState } from "../state/agent-chat-slice.js"; import { useAppState } from "../state/app-state.js"; import { useApiClients } from "./providers.js"; +import { useTranslation } from "../i18n/use-translation.js"; +import { decideTaskDoneNotification } from "../state/task-done-notification.js"; +import { useTaskBus } from "../lib/task-bus.js"; import type { AppRoutePath } from "./routes.js"; export interface AgentRuntimeBridgeValue { @@ -578,12 +582,33 @@ export function requestDesyncedAgentQueueSnapshots( /** Checks is agent runtime bridge route. */ export function isAgentRuntimeBridgeRoute(path: AppRoutePath): boolean { return path === "/main" + || path === "/pet" || path === "/tools" || path === "/settings" || path === "/memory" || path === "/memory-sources"; } +/** Confirm reducer consumption, independently of whether a notification is shown. */ +export function acknowledgeProcessedCronDeliveries( + connection: Pick, + generation: number, + liveDeliveryIdsByChatId: Record, + acknowledgedKeys: Set, +): void { + if (!connection.acknowledgeCronDelivery || generation <= 0 || connection.getReadyGeneration() !== generation) return; + const prefix = `${generation}\0`; + for (const key of acknowledgedKeys) { + if (!key.startsWith(prefix)) acknowledgedKeys.delete(key); + } + for (const deliveryId of new Set(Object.values(liveDeliveryIdsByChatId).flat())) { + const key = `${prefix}${deliveryId}`; + if (acknowledgedKeys.has(key)) continue; + connection.acknowledgeCronDelivery(deliveryId); + acknowledgedKeys.add(key); + } +} + /** Handles agent runtime bridge. */ export function AgentRuntimeBridge(props: { children: ReactNode; @@ -591,6 +616,8 @@ export function AgentRuntimeBridge(props: { }) { const { clients } = useApiClients(); const { state, dispatch } = useAppState(); + const { t } = useTranslation(); + const { syncAgentTaskStatuses } = useTaskBus(); const enabled = isAgentRuntimeBridgeRoute(state.navigation.currentPath); const connectionRef = useRef(null); const modelCatalogRefreshInFlightRef = useRef(false); @@ -606,7 +633,71 @@ export function AgentRuntimeBridge(props: { const operationErrorTimerRef = useRef | null>(null); const queueSnapshotRequestKeysRef = useRef(new Set()); const agentStateRef = useRef(state.agent); + const notifiedProactiveDeliveryIds = useRef(new Set()); + const acknowledgedCronDeliveryKeys = useRef(new Set()); + const acknowledgingConnection = useRef(null); + useEffect(() => { + if (!connection) return; + if (acknowledgingConnection.current !== connection) { + acknowledgingConnection.current = connection; + acknowledgedCronDeliveryKeys.current.clear(); + } + acknowledgeProcessedCronDeliveries(connection, state.agent.connectionGeneration, + state.agent.proactiveLiveDeliveryIdsByChatId, acknowledgedCronDeliveryKeys.current); + }, [connection, state.agent.connectionGeneration, state.agent.proactiveLiveDeliveryIdsByChatId]); + useEffect(() => { + syncAgentTaskStatuses({ tasks: state.agent.tasks.map((task) => ({ + sessionIds: [task.chatId, task.sessionKey], + isRunning: task.runStartedAt != null || state.agent.goalStatesByChatId[task.chatId]?.status === "active", + completedUnseen: task.completedUnseen, + proactiveUnseenAt: state.agent.proactiveDeliveredAtByChatId[task.chatId], + })) }); + }, [state.agent.goalStatesByChatId, state.agent.tasks, state.agent.proactiveDeliveredAtByChatId, syncAgentTaskStatuses]); agentStateRef.current = state.agent; + useEffect(() => { + const updateFocus = () => dispatch({ + type: "agent/chatWindowFocusChanged", + focused: document.hasFocus() && document.visibilityState !== "hidden", + }); + updateFocus(); + window.addEventListener("focus", updateFocus); + window.addEventListener("blur", updateFocus); + document.addEventListener("visibilitychange", updateFocus); + return () => { + window.removeEventListener("focus", updateFocus); + window.removeEventListener("blur", updateFocus); + document.removeEventListener("visibilitychange", updateFocus); + }; + }, [dispatch]); + useEffect(() => { + const pending = state.agent.proactiveNotifications; + if (!pending.length) return; + for (const delivery of pending) { + if (notifiedProactiveDeliveryIds.current.has(delivery.deliveryId)) continue; + notifiedProactiveDeliveryIds.current.add(delivery.deliveryId); + const chatVisible = document.hasFocus() && document.visibilityState !== "hidden" + && state.agent.chatViewVisible && state.agent.currentChatId === delivery.chatId + && !state.agent.blankDraftActive; + const plan = decideTaskDoneNotification({ + enabled: state.bootstrap?.app?.taskDoneNotificationEnabled ?? true, + soundEnabled: state.bootstrap?.app?.notificationSoundEnabled ?? true, + windowFocused: chatVisible, + }); + if (plan && window.memmy?.notifyTaskDone) { + void window.memmy.notifyTaskDone({ + title: t("notification.scheduled.title"), + body: delivery.content.replace(/\s+/gu, " ").trim().slice(0, 240) || t("notification.scheduled.body"), + silent: plan.silent, + deliveryId: delivery.deliveryId, + }).catch(() => undefined); + } + } + // Remember only recent IDs here; the reducer and native host also dedupe. + if (notifiedProactiveDeliveryIds.current.size > 1024) { + notifiedProactiveDeliveryIds.current = new Set([...notifiedProactiveDeliveryIds.current].slice(-1024)); + } + dispatch({ type: "agent/proactiveNotificationsAcknowledged", deliveryIds: pending.map((item) => item.deliveryId) }); + }, [dispatch, state.agent.proactiveNotifications, state.agent.chatViewVisible, state.agent.currentChatId, state.agent.blankDraftActive, state.bootstrap, t]); const ownedTaskStateCoordinatorRef = useRef<{ client: MemmyAgentClient; coordinator: AgentTaskStateCoordinator; @@ -797,6 +888,11 @@ export function AgentRuntimeBridge(props: { if (isAgentConnectionEvent(event)) { dispatch(agentActions.wsEventReceived(event)); } + if (isCronProactiveMessage(event)) { + // Background chats have no onChat reducer subscription. Deliver + // reminders globally; deliveryId makes current-chat copies idempotent. + dispatch(agentActions.wsEventReceived(event)); + } if ( isAgentQueueProjectionEvent(event) && event.chat_id !== subscribedChatRef.current diff --git a/App/frontend/desktop/src/app/computer-history-launch-intent.ts b/App/frontend/desktop/src/app/computer-history-launch-intent.ts new file mode 100644 index 000000000..f55b731ce --- /dev/null +++ b/App/frontend/desktop/src/app/computer-history-launch-intent.ts @@ -0,0 +1,29 @@ +/** One-shot handoff from the release announcement to History's existing switch. */ +const STORAGE_KEY = "memmy.computerHistory.launchEnable.v1"; +const REQUEST_EVENT = "memmy:computer-history-launch-enable"; +const MAX_AGE_MS = 60_000; + +export function requestComputerHistoryLaunchEnable(storage: Storage | undefined, now = Date.now()): void { + storage?.setItem(STORAGE_KEY, String(now)); + if (typeof window !== "undefined") window.dispatchEvent(new Event(REQUEST_EVENT)); +} + +/** Consumes at most one fresh request; stale navigation cannot open a later session's dialog. */ +export function consumeComputerHistoryLaunchEnable(storage: Storage | undefined, now = Date.now()): boolean { + const raw = storage?.getItem(STORAGE_KEY); + storage?.removeItem(STORAGE_KEY); + if (!raw) return false; + const at = Number(raw); + return Number.isFinite(at) && at <= now && now - at <= MAX_AGE_MS; +} + +export function hasComputerHistoryLaunchEnable(storage: Storage | undefined, now = Date.now()): boolean { + const at = Number(storage?.getItem(STORAGE_KEY)); + return Number.isFinite(at) && at > 0 && at <= now && now - at <= MAX_AGE_MS; +} + +export function onComputerHistoryLaunchEnable(listener: () => void): () => void { + if (typeof window === "undefined") return () => undefined; + window.addEventListener(REQUEST_EVENT, listener); + return () => window.removeEventListener(REQUEST_EVENT, listener); +} diff --git a/App/frontend/desktop/src/app/computer-history-model-sync.ts b/App/frontend/desktop/src/app/computer-history-model-sync.ts index 1ef0fdae5..045763d87 100644 --- a/App/frontend/desktop/src/app/computer-history-model-sync.ts +++ b/App/frontend/desktop/src/app/computer-history-model-sync.ts @@ -2,17 +2,15 @@ import { useEffect, useRef } from "react"; import type { MemmyAgentClient } from "../api/memmy-agent-client.js"; import { isComputerHistorySupported } from "./computer-history-platform.js"; -/** Keep background summaries on the selected task model, even outside History. */ +/** Refresh the saved Memory summary assignment after account/config changes. */ export function useComputerHistoryModelSync(input: { - client: Pick | null; + client: Pick | null; enabled: boolean; - preset: string | null; revision: string; }): void { - // Serialize across selection changes so a slower old request cannot become - // the final selection. Intermediate selections that never started are skipped. + // Serialize configuration refreshes and skip superseded revisions. const queue = useRef(Promise.resolve()); - const { client, preset, revision } = input; + const { client, revision } = input; const enabled = input.enabled && isComputerHistorySupported(); useEffect(() => { if (!client || !enabled) return; @@ -21,8 +19,8 @@ export function useComputerHistoryModelSync(input: { let timer: ReturnType | undefined; const sync = () => { if (disposed || queued) return; - // Full and pet windows can coexist. Only the focused window owns the - // current selection; an inactive window must not restore its stale model. + // Full and pet windows coexist. Avoid duplicate foreground refreshes; + // the gateway also refreshes this global assignment in the background. if (!document.hasFocus()) { clearTimeout(timer); timer = setTimeout(sync, 30_000); @@ -34,9 +32,9 @@ export function useComputerHistoryModelSync(input: { if (disposed) return; let delay = 30_000; try { - await client.setComputerHistoryModel(preset); + await client.refreshComputerHistoryModel(); } catch { - // Reconnect/retry without changing the user's chat selection. + // Retry without changing the saved summary assignment. delay = 5_000; } finally { queued = false; @@ -51,5 +49,5 @@ export function useComputerHistoryModelSync(input: { clearTimeout(timer); window.removeEventListener("focus", sync); }; - }, [client, enabled, preset, revision]); + }, [client, enabled, revision]); } diff --git a/App/frontend/desktop/src/app/computer-history-platform.ts b/App/frontend/desktop/src/app/computer-history-platform.ts index 24e96ba93..5e82abab1 100644 --- a/App/frontend/desktop/src/app/computer-history-platform.ts +++ b/App/frontend/desktop/src/app/computer-history-platform.ts @@ -1,6 +1,9 @@ -/** History records macOS activity. Require the desktop host's platform, not the browser's OS. */ +/** Require the desktop host's platform, not the browser's OS. */ export function isComputerHistorySupported( platform = typeof window === "undefined" ? undefined : window.memmy?.platform, + release = typeof window === "undefined" ? undefined : window.memmy?.osRelease, ): boolean { - return platform === "darwin"; + return platform === "darwin" || (platform === "win32" + && /^10\.0\.(\d+)/u.test(release ?? "") + && Number(/^10\.0\.(\d+)/u.exec(release ?? "")?.[1]) >= 22000); } diff --git a/App/frontend/desktop/src/app/computer-history-tray-indicator.ts b/App/frontend/desktop/src/app/computer-history-tray-indicator.ts new file mode 100644 index 000000000..cdcbf2935 --- /dev/null +++ b/App/frontend/desktop/src/app/computer-history-tray-indicator.ts @@ -0,0 +1,64 @@ +import { useEffect } from "react"; +import type { MemmyAgentClient } from "../api/memmy-agent-client.js"; + +const STORAGE_KEY = "memmy.computerHistory.trayIndicatorEnabled"; +const CHANGE_EVENT = "memmy:computer-history-tray-indicator-changed"; + +export function readComputerHistoryTrayIndicator(): boolean { + try { + return window.localStorage.getItem(STORAGE_KEY) !== "false"; + } catch { + return true; + } +} + +export function writeComputerHistoryTrayIndicator(enabled: boolean): void { + try { + window.localStorage.setItem(STORAGE_KEY, String(enabled)); + } catch { + // The current session can still update the indicator when storage is unavailable. + } + window.dispatchEvent(new Event(CHANGE_EVENT)); +} + +/** Keeps the native menu bar indicator current after navigating away from History. */ +export function useComputerHistoryTrayIndicator(client: MemmyAgentClient | null): void { + useEffect(() => { + if (window.memmy?.platform !== "darwin" || !window.memmy.setComputerHistoryTrayIndicator) return; + let active = true; + let inFlight = false; + const publish = async () => { + if (!active) return; + const enabled = readComputerHistoryTrayIndicator(); + if (!enabled || !client) { + window.memmy?.setComputerHistoryTrayIndicator?.({ enabled, recording: false }); + return; + } + if (inFlight) return; + inFlight = true; + try { + const status = await client.getComputerHistoryObservationStatus(); + if (active && readComputerHistoryTrayIndicator()) window.memmy?.setComputerHistoryTrayIndicator?.({ + enabled: true, + recording: status.state === "running", + }); + } catch { + // A missing recorder is not a reliable signal that it is still recording. + if (active) window.memmy?.setComputerHistoryTrayIndicator?.({ + enabled: readComputerHistoryTrayIndicator(), recording: false, + }); + } finally { + inFlight = false; + } + }; + void publish(); + const timer = window.setInterval(() => void publish(), 10_000); + window.addEventListener(CHANGE_EVENT, publish); + return () => { + active = false; + window.clearInterval(timer); + window.removeEventListener(CHANGE_EVENT, publish); + window.memmy?.setComputerHistoryTrayIndicator?.({ enabled: false, recording: false }); + }; + }, [client]); +} diff --git a/App/frontend/desktop/src/app/desktop-prompt-preview.ts b/App/frontend/desktop/src/app/desktop-prompt-preview.ts index 52da39982..7508c5b03 100644 --- a/App/frontend/desktop/src/app/desktop-prompt-preview.ts +++ b/App/frontend/desktop/src/app/desktop-prompt-preview.ts @@ -1,6 +1,6 @@ -export type DesktopPromptPreviewKind = "campaign" | "tokenCredit"; +export type DesktopPromptPreviewKind = "campaign" | "tokenCredit" | "history"; -/** Dev-only query `?previewPrompt=campaign|tokenCredit|all` to force-show desktop prompts. */ +/** Dev-only query `?previewPrompt=campaign|tokenCredit|history|all` to force-show desktop prompts. */ export function isDesktopPromptPreview(kind: DesktopPromptPreviewKind): boolean { if (typeof window === "undefined" || !import.meta.env.DEV) { return false; diff --git a/App/frontend/desktop/src/app/history-launch-prompt-state.ts b/App/frontend/desktop/src/app/history-launch-prompt-state.ts new file mode 100644 index 000000000..9c96aee4c --- /dev/null +++ b/App/frontend/desktop/src/app/history-launch-prompt-state.ts @@ -0,0 +1,83 @@ +/** The History launch announcement stays up through 2026-10-31 in China, on any app version. */ +export const HISTORY_LAUNCH_PROMPT_EXPIRES_AT = "2026-11-01T00:00:00+08:00"; +export const HISTORY_LAUNCH_PROMPT_STORAGE_KEY = "memmy.historyLaunchPrompt.v1"; +export const HISTORY_LAUNCH_VISIBILITY_EVENT = "memmy:history-launch-prompt-visibility"; +const HISTORY_LAUNCH_NAVIGATION_SESSION_KEY = "memmy.historyLaunchPrompt.navigating"; + +interface HistoryLaunchPromptState { + shown: boolean; + dismissed: boolean; + actioned: boolean; +} + +const EMPTY_STATE: HistoryLaunchPromptState = { shown: false, dismissed: false, actioned: false }; +let promptOpen = false; + +export function isHistoryLaunchPromptOpen(): boolean { + return promptOpen; +} + +export function setHistoryLaunchPromptOpen(open: boolean): void { + if (promptOpen === open) return; + promptOpen = open; + if (typeof window !== "undefined") window.dispatchEvent(new Event(HISTORY_LAUNCH_VISIBILITY_EVENT)); +} + +export function readHistoryLaunchPromptState(storage: Storage | undefined): HistoryLaunchPromptState { + try { + const raw = storage?.getItem(HISTORY_LAUNCH_PROMPT_STORAGE_KEY); + if (!raw) return { ...EMPTY_STATE }; + const parsed = JSON.parse(raw) as Partial; + return { + shown: parsed.shown === true, + dismissed: parsed.dismissed === true, + actioned: parsed.actioned === true, + }; + } catch { + return { ...EMPTY_STATE }; + } +} + +function update(storage: Storage | undefined, patch: Partial): void { + if (!storage) return; + storage.setItem(HISTORY_LAUNCH_PROMPT_STORAGE_KEY, JSON.stringify({ + ...readHistoryLaunchPromptState(storage), ...patch, + })); +} + +export function isHistoryLaunchPromptWithinWindow(nowMs: number = Date.now()): boolean { + return nowMs < Date.parse(HISTORY_LAUNCH_PROMPT_EXPIRES_AT); +} + +export function shouldOfferHistoryLaunchPrompt(input: { + state: HistoryLaunchPromptState; + supported: boolean; + nowMs?: number; +}): boolean { + return input.supported + && isHistoryLaunchPromptWithinWindow(input.nowMs ?? Date.now()) + && !input.state.shown + && !input.state.dismissed + && !input.state.actioned; +} + +export function markHistoryLaunchPromptShown(storage: Storage | undefined): void { + update(storage, { shown: true }); +} + +export function markHistoryLaunchPromptDismissed(storage: Storage | undefined): void { + update(storage, { shown: true, dismissed: true }); +} + +export function markHistoryLaunchPromptActioned(storage: Storage | undefined): void { + update(storage, { shown: true, actioned: true }); +} + +/** The campaign waits until a later app open while History's consent flow is underway. */ +export function deferOtherPromptsForHistoryLaunch(storage: Storage | undefined): void { + storage?.setItem(HISTORY_LAUNCH_NAVIGATION_SESSION_KEY, "1"); +} + +export function areOtherPromptsDeferredForHistoryLaunch(storage: Storage | undefined): boolean { + return storage?.getItem(HISTORY_LAUNCH_NAVIGATION_SESSION_KEY) === "1"; +} diff --git a/App/frontend/desktop/src/app/product-tour.tsx b/App/frontend/desktop/src/app/product-tour.tsx index 0a5b87678..95ba6e532 100644 --- a/App/frontend/desktop/src/app/product-tour.tsx +++ b/App/frontend/desktop/src/app/product-tour.tsx @@ -276,9 +276,22 @@ export function ProductTourGuide(props: ProductTourGuideProps) { const anchorIds = [...new Set([current.highlight.anchorId, ...extraIds, current.bubblePlacement.anchorId])]; let frame = 0; let resizeObserver: ResizeObserver | undefined; + let scrolledHighlight: HTMLElement | null = null; /** Definition for measure layout. */ const measureLayout = () => { + const highlightElement = document.querySelector(`[data-tour-anchor="${current.highlight.anchorId}"]`); + if (highlightElement && highlightElement !== scrolledHighlight) { + // A lazy route may mount after this effect. Scroll once when its anchor actually appears, + // then measure the final position. Repeated DOM/scroll observations must not reset it. + scrolledHighlight = highlightElement; + const scrollMode = current.tab === "agentsScan" + ? "page-end" + : current.tab === "tools" + ? "nearest" + : "page-start"; + scrollProductTourHighlightIntoView(highlightElement, scrollMode); + } setLayout(resolveProductTourStepLayout(current.highlight, current.bubblePlacement, lookup, current.extraHighlights)); if (typeof ResizeObserver === "undefined") { return; @@ -308,20 +321,6 @@ export function ProductTourGuide(props: ProductTourGuideProps) { subtree: true }); - const highlightElement = document.querySelector(`[data-tour-anchor="${current.highlight.anchorId}"]`); - if (highlightElement) { - // Tall page-top anchors (agents list / overview cards / logs) must not be - // centered — that hides the section title. Scan prefs sit near the page - // bottom: scroll the pane to its end so Auto sync is fully on-screen before - // spotlight/bubble measurement. Tools only needs nearest. - const scrollMode = current.tab === "agentsScan" - ? "page-end" - : current.tab === "tools" - ? "nearest" - : "page-start"; - scrollProductTourHighlightIntoView(highlightElement, scrollMode); - } - setLayout(null); // Measure after the (instant) scroll so highlight/bubble use final geometry. scheduleMeasurement(); diff --git a/App/frontend/desktop/src/app/router.tsx b/App/frontend/desktop/src/app/router.tsx index ecabe10ca..38c0627d3 100644 --- a/App/frontend/desktop/src/app/router.tsx +++ b/App/frontend/desktop/src/app/router.tsx @@ -1,5 +1,5 @@ /** Router module. */ -import { useCallback, useEffect, useRef, useState } from "react"; +import { lazy, Suspense, useCallback, useEffect, useRef, useState } from "react"; import { PetGuideModal, readCloseMainWindowAction, @@ -16,6 +16,8 @@ import { ProductTourGuide, productTourIncludesLogs, productTourMemorySubPage, + productTourStartMemorySubPage, + productTourStartRoute, productTourTabRoute, type ProductTourDismissResult, type ProductTourStepInfo, @@ -26,7 +28,6 @@ import { clearDeferredGuidanceStep, clearProductTourStep, readCurrentRoute, - readDeferredGuidanceStep, readLaunchModeOverride, writeCurrentRoute, writeDeferredGuidanceStep, @@ -35,6 +36,8 @@ import { type DeferredGuidanceStep } from "./routes.js"; import { persistNickname } from "./nickname.js"; +import { isCampaignPromptSessionReady, isCampaignPromptSurfaceReady } from "./campaign-prompt-state.js"; +import { useGuidanceProgress } from "./use-guidance-progress.js"; import { useOptionalApiClients } from "./providers.js"; import { useAppState } from "../state/app-state.js"; import { useAnalytics } from "../analytics/use-analytics.js"; @@ -48,23 +51,31 @@ import { appActions } from "../state/app-actions.js"; import { NicknameModal } from "../components/nickname-modal.js"; import { randomNickname } from "../lib/nickname.js"; import { useTranslation } from "../i18n/use-translation.js"; +import { + resolveWindowDragExclusionSelector, + WindowDragRegion +} from "./window-drag-region.js"; import { ApiKeyPage } from "../pages/api-key-page.js"; import { ApiKeyOptionalPage } from "../pages/api-key-optional-page.js"; import { ModelPage } from "../pages/model-page.js"; import { HomePage } from "../pages/home-page.js"; +import { hasMountedBrowserPanel, MEMMY_BROWSER_MOUNTED_EVENT, requestBrowserNewTab } from "../pages/browser-panel.js"; import { LoginPage } from "../pages/login-page.js"; -import { MemoryPage, writeMemorySubPage } from "../pages/memory-page.js"; -import { KnowledgePage } from "../pages/knowledge-page.js"; +import { writeMemorySubPage } from "../pages/memory-sub-page-storage.js"; import { OnboardingPage } from "../pages/onboarding-page.js"; import { PetPage } from "../pages/pet-page.js"; -import { SettingsPage } from "../pages/settings-page.js"; +import { PetStartupScreen } from "../pages/pet-startup-screen.js"; import { StartupScreen } from "../pages/startup-screen.js"; import { TokenDetailPage } from "../pages/token-detail-page.js"; -import { ToolsPage } from "../pages/tools-page.js"; import { WelcomePage } from "../pages/welcome-page.js"; -function readWorkspaceGuidanceOverlay(storage: Storage | undefined): Extract | null { - const step = readDeferredGuidanceStep(storage); +// These pages are not needed to show the startup, sign-in, or chat screens. +const MemoryPage = lazy(() => import("../pages/memory-page.js").then(({ MemoryPage }) => ({ default: MemoryPage }))); +const KnowledgePage = lazy(() => import("../pages/knowledge-page.js").then(({ KnowledgePage }) => ({ default: KnowledgePage }))); +const SettingsPage = lazy(() => import("../pages/settings-page.js").then(({ SettingsPage }) => ({ default: SettingsPage }))); +const ToolsPage = lazy(() => import("../pages/tools-page.js").then(({ ToolsPage }) => ({ default: ToolsPage }))); + +function readWorkspaceGuidanceOverlay(step: DeferredGuidanceStep | null): Extract | null { return step === "product_tour" || step === "nickname" ? step : null; } @@ -75,13 +86,26 @@ export function AppRouter(props: { onRetry: () => void }) { const { track, ready: analyticsReady } = useAnalytics(); const { language } = useTranslation(); const prevPathRef = useRef(null); - const [workspaceGuidanceStep, setWorkspaceGuidanceStep] = useState(() => - readWorkspaceGuidanceOverlay(typeof window === "undefined" ? undefined : window.sessionStorage) - ); + const { deferredGuidanceStep } = useGuidanceProgress(); + const guidanceWorkspaceReady = state.navigation.windowVisible !== false && isCampaignPromptSurfaceReady({ + startupStatus: state.startup.status, + currentPath: state.navigation.currentPath, + }) && isCampaignPromptSessionReady({ + userMode: state.bootstrap?.app.userMode, + accountUserId: state.account.userId, + byokConfigured: Boolean(state.modelConfig?.catalog?.modelAssignments.byok.agent.candidates.length), + guidanceDone: true, + }); + const workspaceGuidanceStep = guidanceWorkspaceReady ? readWorkspaceGuidanceOverlay(deferredGuidanceStep) : null; const [deferredNickname, setDeferredNickname] = useState(""); const [petGuideRequest, setPetGuideRequest] = useState(null); + const [preserveHomeBrowser, setPreserveHomeBrowser] = useState(false); const isPetWindowContext = isPetWindow(state.navigation.currentPath); - const windowDragRegion = !isPetWindowContext ? : null; + const windowDragRegion = !isPetWindowContext ? ( + + ) : null; const completeMainWindowAction = useCallback( (request: MainWindowActionRequest, resolution: MainWindowActionResolution) => { @@ -146,13 +170,40 @@ export function AppRouter(props: { onRetry: () => void }) { const currentPath = state.navigation.currentPath; useEffect(() => { - // Memory/Tools pages are not always wrapped by AppFrame; keep the product tour mounted at router level. - const step = readWorkspaceGuidanceOverlay(typeof window === "undefined" ? undefined : window.sessionStorage); - setWorkspaceGuidanceStep(step); - if (step === "nickname") { + const preserve = () => setPreserveHomeBrowser(true); + window.addEventListener(MEMMY_BROWSER_MOUNTED_EVENT, preserve); + if (hasMountedBrowserPanel()) preserve(); + return () => window.removeEventListener(MEMMY_BROWSER_MOUNTED_EVENT, preserve); + }, []); + useEffect(() => window.memmy?.onEmbeddedBrowserOpen?.((url, owner) => { + requestBrowserNewTab(url, owner); + setPreserveHomeBrowser(true); + }), []); + useEffect(() => { + if (resolveMainWindowActionRoute(currentPath) === "auth") setPreserveHomeBrowser(false); + }, [currentPath]); + useEffect(() => { + if (workspaceGuidanceStep === "nickname") { setDeferredNickname((current) => current || randomNickname(language)); } - }, [currentPath, language, state.startup.status]); + }, [workspaceGuidanceStep, language]); + + useEffect(() => { + if (!guidanceWorkspaceReady || deferredGuidanceStep !== "improvement") return; + if (state.bootstrap?.app.userMode !== "byok" && state.bootstrap?.onboarding.improvementProgram === "unset") { + // Memory has its own frame; the consent choice lives in the main workspace frame. + if (currentPath !== "/main") dispatch(appActions.navigate("/main")); + return; + } + // A resumed guide may belong to a previous mode or already have a consent decision. + const storage = typeof window === "undefined" ? undefined : window.sessionStorage; + const includeLogs = productTourIncludesLogs(state.bootstrap?.onboarding.scanPermission); + clearProductTourStep(storage); + writeMemorySubPage(storage, productTourStartMemorySubPage(includeLogs)); + writeDeferredGuidanceStep(storage, "product_tour"); + dispatch(appActions.navigate(productTourStartRoute(includeLogs))); + }, [currentPath, deferredGuidanceStep, dispatch, guidanceWorkspaceReady, state.bootstrap?.app.userMode, + state.bootstrap?.onboarding.improvementProgram, state.bootstrap?.onboarding.scanPermission]); useEffect(() => { if (!analyticsReady) return; @@ -186,7 +237,6 @@ export function AppRouter(props: { onRetry: () => void }) { // re-read stale `product_tour` and remount the tour on /tools. writeDeferredGuidanceStep(storage, "nickname"); setDeferredNickname(randomNickname(language)); - setWorkspaceGuidanceStep("nickname"); dispatch(appActions.navigate("/main")); } @@ -208,10 +258,10 @@ export function AppRouter(props: { onRetry: () => void }) { track(buildOnboardingCompletedEvent(scanPermission)); writeGuidanceCompleted(typeof window === "undefined" ? undefined : window.localStorage); clearDeferredGuidanceStep(typeof window === "undefined" ? undefined : window.sessionStorage); - setWorkspaceGuidanceStep(null); } if (state.startup.status === "loading" || state.startup.status === "idle") { + if (isPetWindowContext) return ; return ( <> @@ -221,6 +271,7 @@ export function AppRouter(props: { onRetry: () => void }) { } if (state.startup.status === "error") { + if (isPetWindowContext) return ; return ( <> @@ -231,7 +282,21 @@ export function AppRouter(props: { onRetry: () => void }) { return ( <> - {renderRoute(state.navigation.currentPath)} + {(currentPath === "/main" || (preserveHomeBrowser + && resolveMainWindowActionRoute(currentPath) === "workspace")) && ( +
+ +
+ )} + {currentPath === "/main" ? null : ( + }> + {renderRoute(currentPath)} + + )} {windowDragRegion} {workspaceGuidanceStep === "product_tour" && ( void }) { ); } -function WindowDragRegion() { - return ( - <> -