diff --git a/.claude/skills/update-account-server/SKILL.md b/.claude/skills/update-account-server/SKILL.md index e63d5021..5e7031a0 100644 --- a/.claude/skills/update-account-server/SKILL.md +++ b/.claude/skills/update-account-server/SKILL.md @@ -17,7 +17,10 @@ own: `.github/workflows/cloud.yml`, dispatched from ANY branch, runs the The folder's `data/` is the service's whole state and its secret (signing keys, token hashes) and `.env` holds the SMTP/Turnstile/OAuth credentials — -never read them out, copy them off the host, or overwrite them. +never read them out, copy them off the host, or overwrite them. The same +goes for `share.env` (the share host's client secret) and `share-data/` +(its published pages); the `share` service is a Gamma image pinned by tag +in `compose.yml` and is updated only by changing that tag. ## Publish from the branch diff --git a/CLAUDE.md b/CLAUDE.md index 187b90a5..d4d0ac2f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -19,7 +19,8 @@ Topic docs live in `docs/dev/` — **read the relevant one before working in tha - [docs/dev/frontend-refactor.md](docs/dev/frontend-refactor.md) — file organization is implemented; App.jsx state decomposition, migration order, and validation remain planned. - [docs/dev/block_centric.md](docs/dev/block_centric.md) — the block-centric product direction: target model (page = root block, PDF = attachment), inventory of remaining PDF-centric assumptions, staged roadmap. Read before generalizing any PDF-shaped feature. - [docs/dev/latex_editing.md](docs/dev/latex_editing.md) — LaTeX input in the block editor: scalable-delimiter pairing (`editor/latexInput.js`), the snippet caret rules, the caret-anchored preview, and the standalone browser regression. -- [docs/dev/mcp.md](docs/dev/mcp.md) — the read-only MCP endpoint (`/mcp`, `gamma/mcp_server.py`) over the agent's tool registry: manual tokens (`gamma/integrations.py`), browser sign-in with OAuth/PKCE and the workspace consent screen (`gamma/mcp_oauth.py`), the paper picker (`gamma/mcp_picker.py`), the admin-confirmed public URL and host allowlist (`gamma/server_settings.py`), the Codex / Claude Code plugin (`plugins/gamma/`, `tools/*codex*`) and the DeepSeek Harness bundle in the same directory (`package.json` + `cordis.patch.yml` + `dsh-skill.js`, token-based). Read before touching any of those modules. +- [docs/dev/hotkeys.md](docs/dev/hotkeys.md) — keyboard shortcuts: the one command catalog (`app/appCommands.js` + `editor/blockCommands.js`, joined by `app/commands.js`), chords and the dispatcher (`shared/lib/hotkeys.js`), the block commands' plumbing (hop, move, duplicate, delete line), the command palette (Ctrl+Shift+P, `>` in Ctrl+P), the account's `keybindings` preference and Settings → Keyboard. Read before adding, moving or handling any shortcut — never a loose key check. +- [docs/dev/mcp.md](docs/dev/mcp.md) — the read-only MCP endpoint (`/mcp`, `gamma/mcp_server.py`) over the agent's tool registry: manual tokens (`gamma/integrations.py`), browser sign-in with OAuth/PKCE and the workspace consent screen (`gamma/mcp_oauth.py`), link resolution inside the connected workspace (`gamma/mcp_links.py`), the admin-confirmed public URL and host allowlist (`gamma/server_settings.py`), the Codex / Claude Code plugin (`plugins/gamma/`, `tools/*codex*`) and the DeepSeek Harness bundle in the same directory (`package.json` + `cordis.patch.yml` + `dsh-skill.js`, token-based). Read before touching any of those modules. - [docs/dev/handwriting.md](docs/dev/handwriting.md) — stylus/mouse handwriting on PDF pages: ink groups as blocks (`ink_url` + `pdf_position`), the `gamma-ink` stroke file and its codec (mirrored in `gamma/ink.py` and `src/ink/ink.js`), the viewer's ink layer and input rules, the draft → upload → block PATCH commit path, `/Ink` export/import. Read before touching ink blocks or the stroke format. - [docs/dev/pdf_loading.md](docs/dev/pdf_loading.md) — how a PDF reaches the screen: the per-document manifest (`gamma/pdf_meta.py`, `/api/pdf-info`), the skeleton laid out before pdf.js parses, transport by size (one GET vs pdf.js range requests + IndexedDB backfill, `src/pdf/pdfSource.js`), the parsed-document cache, the load phases and their timing marks, the e2e timing probe and its numbers. Read before touching the viewer's open path or anything that stores a PDF. - [docs/dev/api.md](docs/dev/api.md) — every `/api/*` endpoint, grouped, plus the auth model. @@ -39,7 +40,8 @@ Topic docs live in `docs/dev/` — **read the relevant one before working in tha - [sites/README.md](sites/README.md) — the gammapdf.com website (`sites/`): the page sources, the build that copies README artwork and demos and renders `PRIVACY.md`, the Cloudflare Worker deploy (`wrangler.jsonc`, `site.yml`), the short links in `_redirects`. Copy mirrors the README and the Store listing; keep them in step. - [docs/dev/desktop.md](docs/dev/desktop.md) — the Windows/macOS/Linux desktop app (`desktop/`, Electron): servers (local sidecars + remote URLs) and Gamma's workspaces in one switcher, launcher, sidecar lifecycle, PyInstaller freeze, the release workflow. - [docs/dev/ipad.md](docs/dev/ipad.md) — Gamma on the iPad: the web app installed to the home screen (the manifest and icons under `/media/`, the status-bar colour following the theme, standalone-mode CSS), why there is no native client, what only the device can verify. -- [docs/dev/onboarding.md](docs/dev/onboarding.md) — the first-run guide (`src/guide/`: engine + first tour built, started from `?guide=`; welcome page, synced pref, checklist and hints still design): `data-guide` anchors and their registry (`guide/anchors.js` — the guide never selects by class), declarative tours (`guide/tours/*.js`) checked off by named events (`guide/events.js`), the tests that keep tours from rotting. Read before adding, moving or removing a control the guide points at. +- [docs/dev/onboarding.md](docs/dev/onboarding.md) — the guide (`src/guide/`: engine, manual tours from Account › Tours, triggered tours and hints offered once right after a feature is first used — `guide/triggers.js`, the "Suggest tours" preference; welcome page, synced pref and checklist still design): `data-guide` anchors and their registry (`guide/anchors.js` — the guide never selects by class), declarative tours (`guide/tours/*.js`) checked off and triggered by named events (`guide/events.js`), the tests that keep tours from rotting. Read before adding, moving or removing a control the guide points at. +- [docs/dev/i18n.md](docs/dev/i18n.md) — the interface language: `t()` / `tn()` / `T()` with the English sentence as the key (`src/shared/i18n/`), one JSON catalog per language, the `language` preference and its Settings row, `npm run i18n -- --sync` + `tests/i18n.test.mjs` keeping every catalog complete, the Chinese glossary. Read before adding or changing any user-visible string. - [docs/dev/settings.md](docs/dev/settings.md) — where every setting is stored (localStorage / synced prefs / server), the Settings dialog's pane and file layout, storage limits. - [docs/dev/ui-design.md](docs/dev/ui-design.md) — the unified control classes, settings primitives, theme system, layout rules, frontend file map. - [docs/dev/debugging.md](docs/dev/debugging.md) — run/test/debug: commands, test suite, log surfaces, the in-app problem report (`src/support/`, the GitHub issue forms), common gotchas. @@ -66,14 +68,14 @@ python -m pytest tests/test_mcp_oauth.py -q # select files relevant to the chan # Frontend tests — from frontend/ node --test tests/settings.test.mjs # select relevant pure-module tests -npm run build && npm run e2e -- --only settings # for changes to this UI flow +npm run build && npm run e2e -- --changed # the browser groups these changes select (--group / --only to narrow) # Account server (Gamma Cloud) — from cloud/, the backend venv has its deps python manage.py setup && uvicorn app:app --port 9002 --reload python -m pytest tests -q ``` -Test affected modules and their direct consumers by default; do not run full suites after every edit. Prefer whole backend test files (some tests within a file share state), and omit `-n auto` for small selections. UI behavior changes need the relevant browser flow; build once after the final frontend edit before running it. Documentation-only changes need no application tests. Broaden coverage for shared contracts, auth, storage, migrations, or uncertain impact. Full suites remain in PR CI; run them locally for broad changes, explicit requests, or unresolved regression concerns. Report the checks run and relevant gaps. See [the test selection policy](docs/dev/debugging.md#local-changes-test-the-affected-modules). +During development, test only what the change touches: the affected modules and their direct consumers, never full suites after an edit. Prefer whole backend test files (some tests within a file share state), and omit `-n auto` for small selections. UI behavior changes need the browser groups they select (`npm run e2e -- --changed`); build once after the final frontend edit before running it. Documentation-only changes need no application tests. Broaden coverage for shared contracts, auth, storage, migrations, or uncertain impact. The full suites are the merge gate: the PR `check` runs all of them and the `merge` skill never merges over red; run them locally only for broad changes, explicit requests, or unresolved regression concerns. Report the checks run and relevant gaps. See [the test selection policy](docs/dev/debugging.md#local-changes-test-the-affected-modules). Frontend has no linter. UI changes are verified by relevant flows in the browser suite (`frontend/tests/e2e/`, asserts no failed API call / console error) — add coverage for new flows; see [docs/dev/debugging.md](docs/dev/debugging.md). Docker image bundles both (multi-stage build, FastAPI serves `dist/`). @@ -85,13 +87,13 @@ Frontend has no linter. UI changes are verified by relevant flows in the browser - Schema changes are numbered migration steps (`gamma/migrations.py`, run at startup with a snapshot first; `db.py` always creates the current shape — never add a lazy `ALTER TABLE` on connect). Rules: [docs/dev/migrations.md](docs/dev/migrations.md). - Auth: `session` cookie → middleware resolves `request.state.user`; guest data is wiped daily. A cloud sign-in (`gamma/cloud_auth.py`) ends in the same session row — never add a second identity path; an account with an empty `password_hash` is cloud-only and the password login refuses it. The WORKSPACE a request works in is a separate decision: `require_ws(request)` (any member, or anyone signed in for a public workspace) / `require_ws(request, write=True)` (editor or owner) from `?ws=` / the `X-Gamma-Workspace` header / the account's default. Server admins pass the workspace API's checks for every workspace without membership, but read pages only as members. Share tokens are per PAGE of a workspace, Notion-style: invited people each with view/edit, plus general access (anyone / signed-in users / invite only) with its own role; members keep their workspace role on top. Shared-view endpoints resolve the workspace via `resolve_ws` + `share_scope_page`; only the block writers accept an edit share through `require_ws_writer`, still scoped to the page. Data helpers take the workspace id; `auth.actor_of(request)` is the actor — the account, or `link:` for a visitor editing through an anyone-with-the-link edit share (the `X-Gamma-Name` header / the socket's `?name=`; such visitors are the only writers rate limited per IP). Keep those distinctions when touching endpoints (details: [docs/dev/api.md](docs/dev/api.md)). - Route order matters for `/api/blocks/*`: static-prefix routes (`by-doc`, `children`, `subtree`) must be registered before `/{block_id}`. -- AI: providers are per-user GUI entries — there are NO env API keys; AI endpoints must build config through `ai_runtime(user)` (`gamma/ai_settings.py`), never module-level constants. Chat speaks Anthropic Messages, OpenAI Chat Completions, and the ChatGPT-OAuth Responses wire; the library agent's tool registry lives in `gamma/ai_tools.py`. All wiring: [docs/dev/ai.md](docs/dev/ai.md); long-paper context: [docs/dev/ai_context.md](docs/dev/ai_context.md). +- AI: providers are per-user GUI entries — there are NO env API keys; AI endpoints must build config through `ai_runtime(user)` (`gamma/ai_settings.py`), never module-level constants. Chat speaks Anthropic Messages, OpenAI Chat Completions, and the ChatGPT-OAuth Responses wire, each an adapter in `gamma/ai_protocols/` — provider differences go on the adapter, never as a protocol branch in a route; model facts (listings, context windows) are asked live, never tabled in code; the library agent's tool registry lives in `gamma/ai_tools.py`. All wiring: [docs/dev/ai.md](docs/dev/ai.md); long-paper context: [docs/dev/ai_context.md](docs/dev/ai_context.md). - Paper metadata + PDF resolution (arXiv/DOI/Unpaywall chains, AI extraction fallback, BibTeX/citation caching): [docs/dev/paper_metadata.md](docs/dev/paper_metadata.md). - Import/export (embedded-annotation import + strip, Zotero/Logseq imports, Markdown-zip imports incl. Obsidian vaults and Notion exports, Markdown and Obsidian-vault export, the annotated-PDF writer with its font/vector-text/image machinery): [docs/dev/import_export.md](docs/dev/import_export.md). - Endpoints doing slow work (downloads, AI calls, PyPDF2) are deliberately sync `def` — FastAPI runs them in its threadpool so they don't block the event loop. Don't convert them back to `async def` while they hold blocking calls. - Search: `/api/search` is the unified library search — `block_fts` (every page's notes, rebuilt lazily per dirty page, `gamma/block_index.py`) plus `pdf_fts` (pypdfium2-extracted PDF text), hits tagged `source: notes|pdf`; the older `/api/pdf-search` (PDF only) and `/api/block-search` (Python fuzzy/regex scan, what the Ctrl+F panel still uses for notes) remain. All share the same normalization (`gamma/textnorm.py` — ligatures, hyphenated line breaks, digit separators; bump `textnorm.INDEX_VERSION` when extraction/normalization changes). The index stores no positions — the frontend re-finds matches with pdf.js on open. The normalization rules are mirrored in `frontend/src/shared/lib/textnorm.js` (used by `search/SearchPanel.jsx` and `pdf/PdfViewer.jsx`); the cases in `tests/shared/textnorm.json` pin both sides — add one there when a rule changes. - Accounts and admin: first-run admin seeding, `manage.py` CLI, the Settings → Users and Server GUIs (Server opens with a dashboard: build from `GAMMA_VERSION`/`GAMMA_COMMIT` via `gamma/version.py`, uptime, warning/error counts, the GitHub release check), storage limits (`check_upload_allowed`/`can_store`: an account's quota covers its personal workspace only, a shared workspace has its own optional cap), and the scrubbed in-memory server log (`logbuf.log` — use it, not `print()`; `log.warning` for what an admin should notice): [docs/dev/user_db.md](docs/dev/user_db.md). -- Package layout: `gamma/config.py` (env config), `gamma/db.py` (schemas/paths/prefs, `SCHEMA_VERSION`), `gamma/migrations.py` (the versioned upgrade runner + steps), `gamma/normalize.py` (content normalization of a workspace's files), `gamma/workspaces.py` (workspace + membership model), `gamma/auth.py` (middleware + workspace/share resolution), `gamma/seed.py` (workspace file creation, guest welcome page, first admin), `gamma/blocks_store.py` (tree CTE helpers), `gamma/storage.py` (uploads + content-hash PDF store), `gamma/foldertags.py` (folder-label path rules, mirrored in `frontend/src/library/libraryUtils.js`), `gamma/textnorm.py` (search normalization + fuzzy matching), `gamma/block_index.py` + `gamma/pdf_index.py` (the notes / PDF FTS indexes: schemas, shared queries, `purge_page_data`), `gamma/pdf_meta.py` (the viewer's per-document manifest, derived at store time), `gamma/pdf_text.py` (every pdfium walk, behind one lock), `gamma/logseq_import.py` + `gamma/zotero_import.py` (import parsers), `gamma/ai_client.py` (provider wire protocols), `gamma/ai_context.py` (PDF/chat context assembly), `gamma/ai_tools.py` (agent tool registry), `gamma/ai_web.py` (the agent's web reach: Crossref/arXiv paper search + reading a document by DOI/arXiv id/URL, in-memory only), `gamma/mcp_server.py` + `gamma/mcp_oauth.py` + `gamma/mcp_picker.py` + `gamma/integrations.py` (the MCP endpoint, its OAuth flow, the paper picker and the integration tokens), `gamma/publisher_sessions.py` (the Connector-imported publisher cookie sessions), `gamma/server_settings.py` (storage limits, the public URL and the MCP host allowlist), `gamma/sync_tree.py` + `gamma/sync_engine.py` (a mirror's tree diff and its sync rounds, [docs/dev/mirror.md](docs/dev/mirror.md)), `gamma/cloud_auth.py` (the Gamma Cloud sign-in client and its access tokens) + `gamma/cloud_sync.py` (the hourly grant check, the preference-profile sync and this server's entry on the account's server list), `gamma/publish.py` (publishing a page to the Gamma Cloud share host: the share host's token exchange and the publishing server's filtered mirror), `gamma/backup_schedule.py` (periodic backup tasks), `gamma/import_staging.py` + `gamma/import_review.py` (the staged import review), `gamma/routers/*` (one module per API area, `routers/workspaces.py` the workspace API, `routers/sync.py` the change feed, `routers/mirrors.py` the mirror API), `gamma/app.py` (assembly + SPA serving). +- Package layout: `gamma/config.py` (env config), `gamma/db.py` (schemas/paths/prefs, `SCHEMA_VERSION`), `gamma/migrations.py` (the versioned upgrade runner + steps), `gamma/normalize.py` (content normalization of a workspace's files), `gamma/workspaces.py` (workspace + membership model), `gamma/auth.py` (middleware + workspace/share resolution), `gamma/seed.py` (workspace file creation, guest welcome page, first admin), `gamma/blocks_store.py` (tree CTE helpers), `gamma/storage.py` (uploads + content-hash PDF store), `gamma/foldertags.py` (folder-label path rules, mirrored in `frontend/src/library/libraryUtils.js`), `gamma/textnorm.py` (search normalization + fuzzy matching), `gamma/block_index.py` + `gamma/pdf_index.py` (the notes / PDF FTS indexes: schemas, shared queries, `purge_page_data`), `gamma/pdf_meta.py` (the viewer's per-document manifest, derived at store time), `gamma/pdf_text.py` (every pdfium walk, behind one lock), `gamma/logseq_import.py` + `gamma/zotero_import.py` (import parsers), `gamma/ai_protocols/` (one adapter per wire protocol: request, stream, usage, model listing, quota — nothing outside it branches on a protocol id), `gamma/ai_client.py` (the provider-agnostic transport), `gamma/ai_catalog.py` (live model listings and context windows), `gamma/ai_context.py` (PDF/chat context assembly), `gamma/ai_tools.py` (agent tool registry), `gamma/translate_engines.py` (the PDF translated view's machine-translation services — Microsoft's free Edge endpoint, Google Cloud Translation, Youdao — and their per-account keys), `gamma/ai_web.py` (the agent's web reach: Crossref/arXiv paper search + reading a document by DOI/arXiv id/URL, in-memory only), `gamma/mcp_server.py` + `gamma/mcp_oauth.py` + `gamma/mcp_links.py` + `gamma/integrations.py` (the MCP endpoint, its OAuth flow, its link resolution and the integration tokens), `gamma/publisher_sessions.py` (the Connector-imported publisher cookie sessions), `gamma/server_settings.py` (storage limits, the public URL and the MCP host allowlist), `gamma/sync_tree.py` + `gamma/sync_engine.py` (a mirror's tree diff and its sync rounds, [docs/dev/mirror.md](docs/dev/mirror.md)), `gamma/cloud_auth.py` (the Gamma Cloud sign-in client and its access tokens) + `gamma/cloud_sync.py` (the hourly grant check, the preference-profile sync and this server's entry on the account's server list), `gamma/publish.py` (publishing a page to the Gamma Cloud share host: the share host's token exchange and the publishing server's filtered mirror), `gamma/backup_schedule.py` (periodic backup tasks), `gamma/import_staging.py` + `gamma/import_review.py` (the staged import review), `gamma/routers/*` (one module per API area, `routers/workspaces.py` the workspace API, `routers/sync.py` the change feed, `routers/mirrors.py` the mirror API), `gamma/app.py` (assembly + SPA serving). ### Frontend (`frontend/`) @@ -99,13 +101,14 @@ Frontend has no linter. UI changes are verified by relevant flows in the browser - Undo: ONE history for the page (`src/editor/blockHistory.js`), derived from the block tree's own state transitions. Every committed change that isn't a page load or an editMode/collapse toggle pushes the previous tree; no call site declares itself undoable. Quick successive edits of one block merge into a chunk, like an editor's typing groups. Entries carry the editor's pre-change caret: Ctrl+Z inside an open editor restores the text in place with the cursor where the change was (the block editor has no CodeMirror `history()` of its own); Ctrl+Z with no editor focused restores without opening one. The stack clears on page switch; cross-page moves aren't undoable. - `src/app/prefDefs.js` + `src/app/prefs.js` — every localStorage-backed user preference, declared once in `PREFS` (key, default, codec, scope `account` or `browser`) and turned into state by one `useAppPrefs()` hook. Add new preferences there, not as loose `usePersistedState` calls in App.jsx. The account-scoped ones (appearance, reading and editing, library display, chat behaviour, budgets, prompts) sync as ONE account-wide `profile` object through `/api/prefs/profile` (`useProfileSync`: server wins, localStorage is the instant-paint cache, pushes debounced); browser-scoped ones (interface size, handwriting, model picks naming this server's provider entries) stay local. Open tabs, the recents queue, pinned folders and reading positions additionally sync through `/api/prefs/*` per account AND workspace (their localStorage caches are keyed `user@workspace`); the active AI key syncs account-wide under its own key. Storage-layer overview: [docs/dev/settings.md](docs/dev/settings.md). - Workspaces in the frontend: `app/App.jsx` picks the tab's workspace once the session resolves (`chooseWorkspace`/`applyWorkspace`: explicit URL `?ws=` or an unavailable-workspace screen; without `ws`, the page a deep link names, the last used, else the default), sets it on `utils.setCurrentWorkspace` (the fetch wrapper adds `X-Gamma-Workspace` to API calls; `withWorkspace` puts `ws` in URLs, copied links and the socket URL), gates the deep-link boot and data effects on `wsReady`, and makes the tree `readOnly` for a viewer. Saved reading state is scoped to account and workspace. Switching (account menu, or Settings → Workspaces — `src/settings/SettingsWorkspace.jsx`) is a reload on `/?ws=`. Details: [docs/dev/workspaces.md](docs/dev/workspaces.md). +- Keyboard shortcuts are commands, declared once ([docs/dev/hotkeys.md](docs/dev/hotkeys.md)): `app/appCommands.js` (App's window listener) and `editor/blockCommands.js` (a block row's keydown, before the outliner's Enter/Tab/Backspace) each hold `{id, label, group, keys, when, edits, run}`; `shared/lib/hotkeys.js` parses chords (`"Mod-Shift-k"`, physical keys via `e.code`) and dispatches; the command palette (`library/QuickOpen.jsx`, Ctrl+Shift+P or `>`) and Settings → Keyboard (`settings/SettingsKeyboard.jsx`, the `keybindings` account pref: rebind / unbind / reset, conflicts flagged) read the same catalog, and `tests/commands.test.mjs` checks every default chord is in the user guide's cheat sheet. The block editor's CodeMirror carries only `standardKeymap`; the formatting runners live in `editor/markCommands.js`. - `src/search/SearchPanel.jsx` — the workspace search (Ctrl+F): `SearchPanel` popover, results grouped titles → this paper's notes → this PDF → other notes → links → library-wide PDF content; collapsible into a compact find bar (default per place via Settings → Reading → Search). `buildSearchRegex` mirrors the backend's fuzzy rules. Opening a library hit "pins" the search: after the paper renders, the query is re-found via pdf.js and highlighted. There is no replace UI. - `src/pdf/PdfViewer.jsx` — the custom pdf.js viewer (`PdfViewer`/`PdfPage`/`PlainTip`; palette in `src/shared/model/highlightColors.js`): lazy memoized pages, capped DPR, cancelable render tasks, highlight/link overlays; its `searchRef` searches normalized per-page text with a char-level map back to exact rects. The open path ([docs/dev/pdf_loading.md](docs/dev/pdf_loading.md)): a parsed-document cache across tab switches, a skeleton laid out from the server manifest before pdf.js has a document, and `src/pdf/pdfSource.js` deciding one GET vs range requests by file size. - `src/shared/model/blockModel.js` — pure block-tree operations (insert/indent/outdent/flatten/cycle-check). - Files vs documents ([docs/dev/block_centric.md](docs/dev/block_centric.md)): a **file** is content — any non-executable upload referenced from a block as `[name](/api/uploads/.)`, rendered by `src/transfers/FileChip.jsx` (a small card, identical for every type; the right-click menu holds download and, for a PDF or markdown file, "Add to library" — a PDF becomes the page that carries it via `POST /blocks/by-doc/`, no re-upload; a markdown file is imported as a note page via `POST /pages/from-file`, a copy — or "Open page" once one exists, when the chip also shows an open-page button; `POST /pages/by-docs` answers which files have pages, one call per render). A **document** is the one PDF a page carries (`doc_id`; viewer, highlights, metadata), attached only from the header paperclip. Drops: on a block row → into that block; on the page body → new blocks at the end (`appendFileBlocks`); on the home library → PDFs/markdown import as pages. Pasting files into the editor (a screenshot, a PDF copied in the file manager) inserts them the same way at the caret; every such upload is a background-tasks row with a percentage (`fileChip.postFile` is an XHR reporting to App's `setUploadReporter` hook). The page header stays title + labels; metadata lives only in its popover. - `src/editor/BlockCmEditor.jsx` — the block editor (CodeMirror 6 behind a textarea-compatible facade). Decorations are a `StateField` (a ViewPlugin may not replace line breaks). Opening a rendered block puts the caret on the clicked character, found by text rather than coordinates (`editor/clickToSource.js` — the raw source lays out differently), and scrolls so that line stays under the pointer; a drag that started on the rendered text keeps selecting in the editor, and that selection (like a Ctrl+drag across rendered text, mapped back to its source range) rides with the next chat message as `note_selections`, which the agent's `edit_block` mode `"selection"` rewrites without touching the rest of the block ([docs/dev/ai.md](docs/dev/ai.md)); a hover line in the gap between two rendered blocks opens the editor on a new line between them. A blur while the window itself is unfocused (Alt+Tab, file dialogs) does NOT end the editing session — the element stays `document.activeElement` and the browser refocuses it on return; only an in-page blur closes the editor. - Live rendering, Obsidian-style (the construct the caret touches stays raw): closed `$…$`/`$$…$$` math, ``` ``` ``` code fences (highlight.js via `src/editor/codeHighlight.js`, shared with the rendered view), `[[ref]]`/`![[embed]]` chips, `![alt](url)` images (the picture itself; `scanImageSyntax` in `src/editor/mdMarks.js` is the one image scanner, shared with `MdTools`), and markdown (headings, quotes + `> [!type]` callout boxes — `[!type]-`/`+` fold flags render as a native `
` in the rendered view — bold/italic/code/strike/`==highlight==`, colored runs as Obsidian-compatible inline HTML ``/`background:…` with the tags hidden (`scanColorSpans` in `mdMarks.js`), links + bare URLs, clickable todos, bullets, `---` rules). - - Raw math: VSCode-style bracket-pair colorization plus a highlight on the `\command` under the caret. `$` auto-pairs: `$`→`$|$`, a second `$` upgrades the fresh pair to `$$|$$` (tracked per view — textually ambiguous against typing over the closers of `$$…$$`), typing a closer types over, Backspace inside an empty pair deletes both; `\$` and `$` in code fences stay literal. Inside math spans `(`/`[`/`{` auto-pair the same way (wrap selection, closer type-over, empty-pair Backspace, `\{`→`\}`); prose and fence brackets stay plain. + - Raw math: VSCode-style bracket-pair colorization plus a highlight on the `\command` under the caret, and a wavy underline where KaTeX rejects the formula (`editor/latexLint.js`; the range the caret touches waits). `$` auto-pairs: `$`→`$|$`, a second `$` upgrades the fresh pair to `$$|$$` (tracked per view — textually ambiguous against typing over the closers of `$$…$$`), typing a closer types over, Backspace inside an empty pair deletes both; `\$` and `$` in code fences stay literal. Inside math spans `(`/`[`/`{` auto-pair the same way (wrap selection, closer type-over, empty-pair Backspace, `\{`→`\}`); prose and fence brackets stay plain. - Formatting hotkeys (Obsidian's; `src/editor/mdMarks.js` holds the pure toggle and the inline-mark table the live renderer also reads): Ctrl/Cmd+B `**`, +I `*`, +E `` ` ``, +Shift+X `~~`, +Shift+H `==` toggle the mark on the selection (surrounding whitespace excluded, inner text stays selected; caret-only inserts an empty pair `**|**` and a second press removes it; caret inside a span unwraps it; multi-line selections wrap per line). Ctrl+K makes `[sel](|)` and fills the slot from a URL on the clipboard. Inside math, fences or inline code the keys are swallowed. Marks nest (`**a *b* c**`, `*a **b** c*`; `***x***` is one bold+italic span whose layers Ctrl+B / Ctrl+I peel off separately); nothing nests inside inline code. - `src/editor/SlashMenu.jsx` holds the "/" command catalog + popup (incl. the `hidden` color commands — "red text", "blue background", … — that only appear once the query matches); `src/editor/callouts.js` is the callout remark plugin. - `editor/BlockTree.jsx` owns trigger detection and key handling: markdown list continuation on the line-break Enter; Enter/Tab staying inside ``` fences and Enter inside `$$` display math; snippet-style Tab in raw math — hop into the next `{}` argument group / out past closers / out of the span, Shift+Tab back, `\begin{…}`/`\end{…}` name groups skipped, falling through to block indent only when there's nowhere to go (`mathTabJump` in `editor/latexCompletion.js`, the pure catalog + matching module `editor/LatexEditor.jsx` re-exports next to the docked live preview and the popup: prefix → abbreviation → fuzzy-subsequence completion tiers, typing `\begin{` completes environment names, environment snippets insert multi-line inside `$$` — [docs/dev/latex_editing.md](docs/dev/latex_editing.md)). @@ -113,9 +116,10 @@ Frontend has no linter. UI changes are verified by relevant flows in the browser - The ⋮⋮ handle's click menu: copy link; copy as markdown (subtree as an indented list); add to chat (attaches the block as a chip for the next AI message; Ctrl+click on a row does the same); duplicate (fresh ids, highlight anchors stripped); move to page (server-side re-parent via `/blocks/{id}/reorder` + `parent_id`); delete. - Paste: the Notion-style "Paste as" chooser on URL paste (a gamma `?block=` link → mention `[[ref]]` / synced `![[embed]]` / URL; a gamma page or citation link → Citation / Page link — a card, stored host- and workspace-free — / URL; other URLs → URL / titled link) and on structured text (strict TSV → Table / Text / Blocks; other multi-line text → Text / Blocks, where Blocks parses via `POST /api/markdown-blocks` — the `.md` importer's parser — and inserts the tree as sibling blocks). A clipboard that is a single html table (Excel/Sheets) pastes as a pretty-printed markdown table. - The rendered view's `![[embed]]` cards are a full editing surface for the SOURCE block at the synced position: checkboxes, image hover tools and table editing on the card write through (same mdTools transforms); clicking the text edits the raw source in place with the block editor's math live preview + `\command` autocomplete (shared `useMathUi` hook) and image/table paste. Same-page sources route through onChangeText/autosave, cross-page sources `PUT /blocks/{id}` + ref-cache merge. The footer jumps to the source. Notion-style external-link chips show favicon + fetched title via `/api/link-preview`; GitHub URLs are parsed locally to `owner/repo #N`. -- `src/editor/MdTools.jsx` — in-place tools on the RENDERED notes (Notion-style hover affordances, markdown-native storage): `MdImage` (hover toolbar of flat `ctlBtn` icons — zoom lightbox, caption stored as the alt text, download, delete — plus a drag grip on each side of the centred picture (`shared/ui/ResizeGrip.jsx`, shared with Mermaid diagrams, whose size lands in the fence info string as `width=N`) writing the image size Obsidian-style, `![alt|300](url)`; the legacy Logseq `{:width N}` suffix still renders and is normalized to the pipe form on any edit) and `MdTableWrap` (every note table's scroll wrapper; when editable, hover "+" strips add a row/column, column/row handle pills move their column/row by drag (drop-line indicator, one moveCol/moveRow op on release) and a plain click opens a compact menu — insert/delete plus, for columns, a settingsKit `Segmented` alignment chooser, and clicking a cell edits it IN PLACE — tables are deliberately never edited as raw markdown, unlike equations: the cell mousedown stops the block row's edit-on-mousedown, Tab/Shift-Tab hop cells across the commit remount via a module-level session map keyed by block id + table index, Enter commits, Esc cancels). All edits are pure source transforms: `scanImages`/`scanTables` locate the nth rendered construct with the same span-exclusion rules as `mdPreprocess` (tables inside blockquotes stay index-aligned but uneditable), `applyImageEdit`/`applyTableEdit` rewrite it (tables re-serialized pretty-printed — so every table edit auto-formats; `formatTables` additionally runs when a block's raw editor closes, hooked on the `editMode` transition in `BlockRow`, not editor blur), and a failed scan is a no-op — never a guess. `BlockMarkdown` counts rendered images/tables in document order to hand each its index (same idiom as task checkboxes). +- `src/editor/MdTools.jsx` — in-place tools on the RENDERED notes (Notion-style hover affordances, markdown-native storage): `MdImage` (hover toolbar of flat `ctlBtn` icons — zoom lightbox, caption stored as the alt text, download, delete — plus a drag grip on each side of the centred picture (`shared/ui/ResizeGrip.jsx`, shared with Mermaid diagrams, whose size lands in the fence info string as `width=N`) writing the image size Obsidian-style, `![alt|300](url)`; the legacy Logseq `{:width N}` suffix still renders and is normalized to the pipe form on any edit) and `MdTableWrap` (every note table's scroll wrapper; when editable, hover "+" strips add a row/column, column/row handle pills move their column/row by drag (drop-line indicator, one moveCol/moveRow op on release) and a plain click opens a compact menu — insert/delete plus, for columns, a settingsKit `Segmented` alignment chooser; the corner handle selects the whole table (outlined) with Copy table / Delete table, and Delete or Backspace removes the selected table; and clicking a cell edits it IN PLACE — tables are deliberately never edited as raw markdown, unlike equations: the cell mousedown stops the block row's edit-on-mousedown, Tab/Shift-Tab hop cells across the commit remount via a module-level session map keyed by block id + table index, Enter commits, Esc cancels). All edits are pure source transforms: `scanImages`/`scanTables` locate the nth rendered construct with the same span-exclusion rules as `mdPreprocess` (tables inside blockquotes stay index-aligned but uneditable), `applyImageEdit`/`applyTableEdit` rewrite it (tables re-serialized pretty-printed — so every table edit auto-formats; `formatTables` additionally runs when a block's raw editor closes, hooked on the `editMode` transition in `BlockRow`, not editor blur), and a failed scan is a no-op — never a guess. `BlockMarkdown` counts rendered images/tables in document order to hand each its index (same idiom as task checkboxes). Every rendered image, table and Mermaid diagram sits in an object frame (`src/editor/MdObject.jsx`). A press on its body selects it (a picture zooms on double-click); a press on the frame's margin, or in the blank beside a centred picture, opens the editor with the caret at the object's near end, where it stays rendered. Right-click (or the table's corner handle) opens one menu: Edit markdown source, Move to (new block above / below, another page), Copy as markdown, Delete. The frame is a drag source: dropped on a row's edge it becomes a new block, dropped in a row's middle it lands in the gap nearest the pointer, a line showing the spot first. Moves are pure source-range edits (`src/editor/mdObjects.js`; `moveObjectInTree` is one tree transition, so one undo step) and the stored text stays plain markdown. The block editor treats the same objects as objects: a picture or table stays rendered while the caret steps past it or rests at either end, and only a selection reaching inside shows the source. A click puts the caret after it, right-click reveals the source, and the widget drags to another block while its own block is being edited (`onObjectDrag` → the row's object action). A drop into an open editor lands at the nearest line boundary through the editor's own capture handlers (`onObjectDragOver` / `onObjectDrop`), never CodeMirror's paste-at-caret. The scanners all three share are `src/editor/mdScan.js`. - Settings dialog: `src/settings/SettingsDialog.jsx` (one sidebar in three groups — Preferences / AI / Manage — and most panes) + `src/settings/SettingsKit.jsx` (pane primitives and shared controls) + `src/settings/SettingsAi.jsx` + `src/settings/SettingsUsers.jsx`. Panes are built ONLY from the settingsKit primitives; a choice is a `Segmented` (two or three words), `IconChoices` (a small exclusive set as icon tiles) or a `MenuSelect`/`ActionMenu` dropdown (`shared/ui/Menus.jsx`); new controls belong in the shared set — `settings/settings.css` is layout only. Panes carry no explanatory subtitles: a row's short hint plus its hover `title`. Pane map and design rules: [docs/dev/settings.md](docs/dev/settings.md), [docs/dev/ui-design.md](docs/dev/ui-design.md). -- Notices (`src/app/notices.js` + `useNotices.js`, `gamma/notices.py`, `/api/notices`): the red dot on the account button for what wants a look once (a newer release, log errors); each notice names the Settings pane that resolves it, showing the pane records its fingerprint per account, and "Settings…" opens on the strongest one. New sources are `@source` functions in `gamma/notices.py`, cheap reads only. Details: [docs/dev/settings.md](docs/dev/settings.md). +- Notices (`src/app/notices.js` + `useNotices.js`, `gamma/notices.py`, `/api/notices`): the red dot on the account button for what wants a look once (a newer release and log errors for admins; a failed backup task, open clone conflicts, a failed cloud sync, storage nearly full for everyone); each notice names the Settings pane that resolves it, showing the pane records its fingerprint per account, and "Settings…" opens on the strongest one. New sources are `@source` functions in `gamma/notices.py`, cheap reads only. Details: [docs/dev/settings.md](docs/dev/settings.md). +- Interface language ([docs/dev/i18n.md](docs/dev/i18n.md)): every user-visible string in the frontend goes through `t("English sentence")` from `src/shared/i18n/i18n.js` (`tn` for counts, `T` to mark a string in a static table); the English text is the key, `locales/zh.json` holds the Chinese. A new string needs its catalog line in the same change: `npm run i18n -- --sync` adds the empty key, `node --test tests/i18n.test.mjs` fails until it is filled; `npm run i18n:audit` lists text that never reached `t()`. `t()` works in module-level constants (the catalog loads before `App`; a language change reloads the page). Never compare against translated text. The browser suite pins `en-US` on every context. - Theme: Settings → Appearance — System plus the seven pinned themes in `THEMES` (`app/prefs.js`; `gamma-theme` in localStorage, an inline script in `index.html` applies a pinned theme before first paint) plus display-only "Flip page colors" (`gamma-pdf-dark`). - Sharing a page: `src/sharing/SharePopover.jsx` (the header link button, a popover under it like the account menu) — link + Copy + Stop sharing, access as three audience tiles (anyone / signed in / invited only) plus a View / Edit toggle, invited people with their own access, the citation section; built from the settings kit like the workspace Manage dialog. There is no "reset link" — stop and share again. Data functions stay in App.jsx (`loadShareSettings`, `updateShareSettings`, …). - View modes are derived from the URL: `/` home, `/?page=` page (with PDF if it has `source_url`), `/?share=` the share view (`shareMode`: no library/chat/prefs; `readOnly` is state — false once the link resolves with edit rights, and `utils.withShare` puts the token on every API call), `/?block=` jump-to-block; every non-share URL also carries `ws=`. diff --git a/README.md b/README.md index 425906c5..209e7859 100644 --- a/README.md +++ b/README.md @@ -36,8 +36,9 @@ Open a paper by pasting any link — arXiv, DOI, or a publisher page; Gamma find - **Highlight** — select text or drag a box around a figure, pick a color, add a comment. Each highlight becomes a block in your notes. Highlights already saved in the file by Acrobat, Preview or SumatraPDF come in as blocks too. - **Draw** — with a stylus or the mouse: circle a claim, sketch an arrow, highlight freely. Lasso strokes to move, resize, rotate or recolor them; erase whole strokes or part of one. Ink is a note block linked to its place on the page. - **Follow citations** — references in the PDF are clickable; a global **← Back** unwinds jumps across documents, and a cited arXiv/DOI paper is one click from your library. +- **Translate** — redraw a page in your language in place, figures untouched, or translate just a selected sentence. Microsoft's free service works with no setup; a chat model, Google or Youdao are one setting away. -→ Guide: [Reading and highlighting](./docs/user_guide.md#reading-and-highlighting) · [Draw with a pen](./docs/user_guide.md#draw-with-a-pen) · [Links inside the PDF](./docs/user_guide.md#links-inside-the-pdf) +→ Guide: [Reading and highlighting](./docs/user_guide.md#reading-and-highlighting) · [Draw with a pen](./docs/user_guide.md#draw-with-a-pen) · [Links inside the PDF](./docs/user_guide.md#links-inside-the-pdf) · [Translate a paper](./docs/user_guide.md#translate-a-paper) ## Take notes diff --git a/backend/gamma/README.md b/backend/gamma/README.md index 1565b1e6..729318e4 100644 --- a/backend/gamma/README.md +++ b/backend/gamma/README.md @@ -13,7 +13,9 @@ auth.py session middleware → request.state.user; request → worksp seed.py workspace file creation, guest welcome page, first admin blocks_store.py recursive-CTE tree helpers storage.py uploads (content-addressed) + orphan cleanup -ai_client.py provider HTTP protocols + streaming response parsing +ai_protocols/ one adapter per AI wire protocol (request, stream, usage, models, quota) +ai_client.py provider-agnostic AI transport (open, read, stream, errors) +ai_catalog.py live model listings + context windows (provider, then models.dev) ai_context.py PDF attachments, extraction, and chat context assembly logseq_import.py EDN / Markdown importers app.py assembly + SPA serving diff --git a/backend/gamma/ai_catalog.py b/backend/gamma/ai_catalog.py new file mode 100644 index 00000000..03800691 --- /dev/null +++ b/backend/gamma/ai_catalog.py @@ -0,0 +1,132 @@ +"""What a provider entry offers, asked live: its model listing and each +model's context window. The protocol adapters (gamma/ai_protocols) build the +requests and read the answers; this module fetches and caches them. Nothing +here is a table of model names — model facts come from the provider, or +from the public models.dev catalog when the provider's listing carries no +context window (OpenAI's and DeepSeek's don't). + +``fetch_json`` is the one fetch every listing, quota and credential check +goes through (a short, UI-friendly timeout).""" + +import json +import re +import threading +import time +from urllib.request import Request as URLRequest, urlopen + +from . import ai_protocols +from .logbuf import log + +FETCH_TIMEOUT = 5 +MODELS_DEV_URL = "https://models.dev/api.json" +MODELS_DEV_TIMEOUT = 15 +# Like the Codex version: a good answer is kept for hours; a failed lookup +# is retried after minutes, the last good answer served meanwhile. +WINDOW_TTL = 6 * 3600 +WINDOW_RETRY = 600 + +_listings = {} # "|" -> {"windows": {model: n}, "until": t} +_listings_lock = threading.Lock() +_models_dev = {"windows": None, "until": 0.0} # windows: model id -> [(provider key, n)] +_models_dev_lock = threading.Lock() + + +def fetch_json(req: URLRequest): + with urlopen(req, timeout=FETCH_TIMEOUT) as resp: + return json.loads(resp.read()) + + +def list_models(conf: dict) -> list: + """The entry's chat models as ``[{id, context_window}]`` (0 = the + listing names none), in the order to offer them. Raises what the fetch + raises (an HTTPError carries the provider's status).""" + proto = ai_protocols.of(conf) + return proto.models(fetch_json(proto.models_request(conf)), conf) + + +def _listed_windows(provider_id: str, conf: dict) -> dict: + """{model: window} from the entry's own listing, cached.""" + key = f"{provider_id}|{conf['base_url']}" + with _listings_lock: + now = time.time() + cached = _listings.get(key) + if cached and now < cached["until"]: + return cached["windows"] + try: + windows = {m["id"]: m["context_window"] for m in list_models(conf) if m["context_window"]} + _listings[key] = {"windows": windows, "until": now + WINDOW_TTL} + except Exception as e: + log.warning(f"[ai] model listing for context windows failed ({conf.get('name')}): {e}") + _listings[key] = {"windows": cached["windows"] if cached else {}, "until": now + WINDOW_RETRY} + return _listings[key]["windows"] + + +def _models_dev_windows() -> dict: + """models.dev's catalog as {lowercased model id: [(provider key, window)]}, + cached; also indexed by the part after a "vendor/" prefix.""" + with _models_dev_lock: + now = time.time() + if now < _models_dev["until"]: + return _models_dev["windows"] or {} + try: + with urlopen(URLRequest(MODELS_DEV_URL, headers={"Accept": "application/json", + "User-Agent": "Gamma/model-catalog"}), + timeout=MODELS_DEV_TIMEOUT) as resp: + data = json.loads(resp.read()) + windows = {} + for pkey, provider in (data.items() if isinstance(data, dict) else []): + models = provider.get("models") if isinstance(provider, dict) else None + for mid, m in (models.items() if isinstance(models, dict) else []): + n = ((m.get("limit") or {}).get("context")) if isinstance(m, dict) else None + if not isinstance(n, int) or n <= 0: + continue + name = str(m.get("id") or mid).lower() + for alias in {name, name.rsplit("/", 1)[-1]}: + windows.setdefault(alias, []).append((str(pkey).lower(), n)) + if not windows: + raise ValueError("empty catalog") + _models_dev.update(windows=windows, until=now + WINDOW_TTL) + except Exception as e: + log.warning(f"[ai] models.dev catalog lookup failed: {e}") + _models_dev["until"] = now + WINDOW_RETRY + return _models_dev["windows"] or {} + + +def _alnum(text: str) -> str: + return re.sub(r"[^a-z0-9]", "", text.lower()) + + +def _catalog_window(model: str, conf: dict) -> int: + """The model's window per models.dev. Several providers may list one + model, often with their own caps: the provider this entry talks to wins + (``Protocol.catalog_hints`` — by default the endpoint's host), else the + value most of them agree on.""" + windows = _models_dev_windows() + name = model.lower() + found = windows.get(name) or windows.get(name.rsplit("/", 1)[-1]) or [] + if not found: + return 0 + names = set() + for hint in ai_protocols.of(conf).catalog_hints(conf): + # Whole host labels and runs of them: "api.moonshot.ai" names + # moonshot, moonshotai, … — never a substring like "a". + labels = [_alnum(label) for label in (hint or "").split(".")] + names |= {"".join(labels[i:j]) for i in range(len(labels)) for j in range(i + 1, len(labels) + 1)} + for pkey, n in found: + if _alnum(pkey) in names: + return n + counts = {} + for _, n in found: + counts[n] = counts.get(n, 0) + 1 + return max(counts, key=lambda n: (counts[n], n)) + + +def context_window(provider_id: str, conf: dict, model: str) -> tuple: + """``(window, source)`` for one of the entry's models: source + ``"provider"`` (its own listing) or ``"models.dev"``; ``(0, "")`` when + neither knows the model — callers show nothing rather than a guess.""" + n = _listed_windows(provider_id, conf).get(model) + if n: + return n, "provider" + n = _catalog_window(model, conf) + return (n, "models.dev") if n else (0, "") diff --git a/backend/gamma/ai_client.py b/backend/gamma/ai_client.py index 9e359983..6e149211 100644 --- a/backend/gamma/ai_client.py +++ b/backend/gamma/ai_client.py @@ -1,408 +1,27 @@ -"""Provider-agnostic AI transport and provider wire-format adapters. - -This module owns the HTTP request/response details for Anthropic Messages, -OpenAI Chat Completions, and ChatGPT's Responses API. Route handlers should -deal in the common ``messages`` representation and call :func:`call_ai` or -:func:`open_ai`; provider-specific shapes stay here. -""" +"""Provider-agnostic AI transport: open a call, read or stream its reply, +count its tokens. Everything that differs between providers lives on the +protocol adapters (gamma/ai_protocols); route handlers deal in the common +``messages`` representation and call :func:`call_ai` or :func:`open_ai`.""" import json import re import urllib.error import urllib.request -import uuid +from . import ai_protocols from .logbuf import log -# Tools are declared once in a common shape ({name, description, parameters}) -# — see gamma/ai_tools.py — and translated per wire protocol here. Messages may -# carry two agentic extensions beyond {role, content-str}: an assistant message -# with `tool_calls` ([{id, name, arguments-dict}]) and a {"role": "tool", -# "call_id", "content"} result entry; each builder maps them to its wire shape. -# A tool result may also carry `images` ([(media_type, base64)] — a rendered -# PDF page): Anthropic takes image blocks inside the tool_result; the OpenAI -# wires only accept text there, so the pictures follow the round's results -# as one user turn (_TOOL_IMAGES_NOTE) the model reads in call order. - -_TOOL_IMAGES_NOTE = "Pictures returned by the tool calls above, in call order:" - - -def _tool_image_turns(messages, make_turn): - """The common turn list with every run of tool results followed by one - user turn carrying their pictures — ``make_turn(images)`` builds it in - the wire's shape. Yields (message, is_image_turn).""" - pending = [] - for m in messages: - if m["role"] != "tool" and pending: - yield make_turn(pending), True - pending = [] - yield m, False - if m["role"] == "tool": - pending.extend(m.get("images") or []) - if pending: - yield make_turn(pending), True - - -def _attach_index(messages) -> int: - """Index of the message attachments ride on: the last plain user turn - (tool-result entries can follow it in agent rounds).""" - for index in range(len(messages) - 1, -1, -1): - if messages[index]["role"] == "user": - return index - return len(messages) - 1 - - -def anthropic_request( - conf, messages, system, model, pdf_b64s=None, effort="", - max_tokens=8192, images=None, stream=False, tools=None, -): - """Build an Anthropic Messages API request.""" - messages = [dict(m) for m in messages] # attachment injection must not mutate the caller's turn list - if pdf_b64s or images: - last = messages[_attach_index(messages)] - last["content"] = [ - *[ - { - "type": "document", - "source": { - "type": "base64", - "media_type": "application/pdf", - "data": data, - }, - } - for data in (pdf_b64s or []) - ], - *[ - { - "type": "image", - "source": {"type": "base64", "media_type": media_type, "data": data}, - } - for media_type, data in (images or []) - ], - {"type": "text", "text": last["content"]}, - ] - body = { - "model": model, - "max_tokens": max_tokens, - "system": system, - "messages": _anthropic_messages(messages), - } - if tools: - body["tools"] = [{"name": t["name"], "description": t["description"], - "input_schema": t["parameters"]} for t in tools] - if effort: - # "minimal" is OpenAI's lowest level; Anthropic's is "low". - body["output_config"] = {"effort": "low" if effort == "minimal" else effort} - if stream: - body["stream"] = True - return urllib.request.Request( - f"{conf['base_url']}/v1/messages", - data=json.dumps(body).encode(), - headers={ - "x-api-key": conf["api_key"], - "anthropic-version": "2023-06-01", - "Content-Type": "application/json", - }, - ) - - -def _anthropic_messages(messages) -> list: - """Map the common turn list to Anthropic content blocks: tool results are - tool_result blocks in a user turn (consecutive ones coalesced — they must - directly follow the assistant's tool_use turn), tool calls become tool_use - blocks after the assistant's text.""" - out = [] - for m in messages: - if m["role"] == "tool": - block = {"type": "tool_result", "tool_use_id": m["call_id"], "content": m["content"]} - if m.get("images"): - block["content"] = [{"type": "text", "text": m["content"]}] + [ - {"type": "image", "source": {"type": "base64", "media_type": media_type, "data": data}} - for media_type, data in m["images"]] - prev = out[-1] if out else None - if (prev and prev["role"] == "user" and isinstance(prev["content"], list) - and prev["content"] and prev["content"][0].get("type") == "tool_result"): - prev["content"].append(block) - else: - out.append({"role": "user", "content": [block]}) - elif m["role"] == "assistant" and m.get("tool_calls"): - content = [{"type": "text", "text": m["content"]}] if (m.get("content") or "").strip() else [] - content += [{"type": "tool_use", "id": c["id"], "name": c["name"], "input": c["arguments"]} - for c in m["tool_calls"]] - out.append({"role": "assistant", "content": content}) - else: - prev = out[-1] if out else None - if (m["role"] == "user" and prev and prev["role"] == "user" - and isinstance(prev["content"], list) - and prev["content"] and prev["content"][0].get("type") == "tool_result"): - # A tool-only assistant reply leaves its results as the last - # user turn; fold the next real user message into it so roles - # keep alternating. Attachment turns already carry block lists. - prev["content"].extend( - m["content"] if isinstance(m["content"], list) - else [{"type": "text", "text": m["content"]}]) - else: - out.append({"role": m["role"], "content": m["content"]}) - return out - - -def _anthropic_extract(data) -> str: - text = "".join( - item.get("text", "") - for item in data.get("content", []) - if item.get("type") == "text" - ) - if not text.strip(): - raise RuntimeError(f"empty response (stop_reason={data.get('stop_reason', 'unknown')})") - return text - - -def openai_request( - conf, messages, system, model, pdf_b64s=None, effort="", - max_tokens=8192, images=None, stream=False, tools=None, -): - """Build an OpenAI Chat Completions API request.""" - messages = [dict(m) for m in messages] - if pdf_b64s or images: - last = messages[_attach_index(messages)] - last["content"] = [ - *[ - { - "type": "file", - "file": { - "filename": f"document-{index + 1}.pdf", - "file_data": f"data:application/pdf;base64,{data}", - }, - } - for index, data in enumerate(pdf_b64s or []) - ], - *[ - { - "type": "image_url", - "image_url": {"url": f"data:{media_type};base64,{data}"}, - } - for media_type, data in (images or []) - ], - {"type": "text", "text": last["content"]}, - ] - wire = [{"role": "system", "content": system}] if system else [] - image_turn = lambda imgs: {"role": "user", "content": [ # noqa: E731 - {"type": "text", "text": _TOOL_IMAGES_NOTE}, - *[{"type": "image_url", "image_url": {"url": f"data:{media_type};base64,{data}"}} - for media_type, data in imgs]]} - for m, is_image_turn in _tool_image_turns(messages, image_turn): - if is_image_turn: - wire.append(m) - elif m["role"] == "tool": - wire.append({"role": "tool", "tool_call_id": m["call_id"], "content": m["content"]}) - elif m["role"] == "assistant" and m.get("tool_calls"): - wire.append({"role": "assistant", "content": m.get("content") or None, - "tool_calls": [{"id": c["id"], "type": "function", - "function": {"name": c["name"], - "arguments": json.dumps(c["arguments"])}} - for c in m["tool_calls"]]}) - else: - wire.append({"role": m["role"], "content": m["content"]}) - body = { - "model": model, - # Current OpenAI models take max_completion_tokens (the cap includes - # hidden reasoning tokens, so leave a generous default); compatible - # servers (DeepSeek, vLLM, Ollama, …) take the classic max_tokens. - ("max_completion_tokens" if is_openai_platform(conf["base_url"]) else "max_tokens"): max_tokens, - "messages": wire, - } - if tools: - body["tools"] = [{"type": "function", - "function": {"name": t["name"], "description": t["description"], - "parameters": t["parameters"]}} for t in tools] - if effort: - body["reasoning_effort"] = effort - if stream: - body["stream"] = True - # The final chunk then carries the token counts (OpenAI and the - # common compatible servers: vLLM, Ollama, llama.cpp, LiteLLM). - body["stream_options"] = {"include_usage": True} - return urllib.request.Request( - f"{conf['base_url']}/v1/chat/completions", - data=json.dumps(body).encode(), - headers={ - "Authorization": f"Bearer {conf['api_key']}", - "Content-Type": "application/json", - }, - ) - - -def _openai_extract(data) -> str: - choices = data.get("choices") or [{}] - text = (choices[0].get("message") or {}).get("content") or "" - if not text.strip(): - reason = choices[0].get("finish_reason", "unknown") - raise RuntimeError( - f"empty response (finish_reason={reason} — a reasoning model may have spent " - "the whole token budget thinking; try effort: low or a shorter request)" - ) - return text - - -def _responses_input(messages, pdf_b64s=None, images=None) -> list: - """Map the common turn list to Responses API input items (shared by the - ChatGPT/codex backend and OpenAI's platform /v1/responses).""" - items = [] - image_turn = lambda imgs: {"type": "message", "role": "user", "content": [ # noqa: E731 - {"type": "input_text", "text": _TOOL_IMAGES_NOTE}, - *[{"type": "input_image", "image_url": f"data:{media_type};base64,{data}"} - for media_type, data in imgs]]} - image_turns = [] # never the turn the user's own attachments ride on - for message, is_image_turn in _tool_image_turns(messages, image_turn): - if is_image_turn: - items.append(message) - image_turns.append(message) - elif message["role"] == "tool": - items.append({"type": "function_call_output", "call_id": message["call_id"], - "output": message["content"]}) - elif message["role"] == "assistant": - if message.get("content") or not message.get("tool_calls"): - content = [{"type": "output_text", "text": message["content"]}] - items.append({"type": "message", "role": "assistant", "content": content}) - for call in message.get("tool_calls") or []: - items.append({"type": "function_call", "call_id": call["id"], - "name": call["name"], "arguments": json.dumps(call["arguments"])}) - else: - content = [{"type": "input_text", "text": message["content"]}] - items.append({"type": "message", "role": "user", "content": content}) - if pdf_b64s or images: - last = next((item for item in reversed(items) - if item.get("type") == "message" and item.get("role") == "user" - and not any(item is turn for turn in image_turns)), items[-1]) - last["content"] = [ - *[ - { - "type": "input_file", - "filename": f"document-{index + 1}.pdf", - "file_data": f"data:application/pdf;base64,{data}", - } - for index, data in enumerate(pdf_b64s or []) - ], - *[ - {"type": "input_image", "image_url": f"data:{media_type};base64,{data}"} - for media_type, data in (images or []) - ], - *last["content"], - ] - return items - - -def _responses_tools(tools) -> list: - # Responses API uses a flattened function-tool shape (no "function" nesting). - return [{"type": "function", "name": t["name"], "description": t["description"], - "parameters": t["parameters"], "strict": False} for t in (tools or [])] - - -def chatgpt_request( - conf, messages, system, model, pdf_b64s=None, effort="", - max_tokens=8192, images=None, stream=False, tools=None, -): - """Build a ChatGPT subscription Responses API request. - - The backend only streams SSE, including for callers that want a complete - reply. :func:`read_reply` joins those deltas for non-stream callers. - """ - body = { - "model": model, - "instructions": system or "You are a helpful research assistant.", - "input": _responses_input(messages, pdf_b64s, images), - "tools": _responses_tools(tools), - "tool_choice": "auto", - # Batched calls (e.g. renaming a whole folder in one round) — a call - # per round-trip would eat the tool-round budget one page at a time. - "parallel_tool_calls": bool(tools), - "store": False, - "stream": True, - "include": [], - } - if effort: - body["reasoning"] = {"effort": effort} - return urllib.request.Request( - f"{conf['base_url']}/responses", - data=json.dumps(body).encode(), - headers={ - "Authorization": f"Bearer {conf['api_key']}", - "chatgpt-account-id": conf.get("account_id", ""), - "OpenAI-Beta": "responses=experimental", - "originator": "codex_cli_rs", - "session_id": str(uuid.uuid4()), - "Accept": "text/event-stream", - "Content-Type": "application/json", - }, - ) - - -def openai_responses_request( - conf, messages, system, model, pdf_b64s=None, effort="", - max_tokens=8192, images=None, stream=False, tools=None, -): - """Build an OpenAI platform /v1/responses request. - - Used instead of Chat Completions when a call carries function tools: - reasoning models (gpt-5.x) reject tools + reasoning_effort on - /v1/chat/completions and OpenAI's guidance is to use the Responses API. - Always streamed — the tool loop consumes SSE on every protocol. - """ - body = { - "model": model, - "input": _responses_input(messages, pdf_b64s, images), - "tools": _responses_tools(tools), - "tool_choice": "auto", - "parallel_tool_calls": bool(tools), - "store": False, - "stream": True, - "max_output_tokens": max_tokens, - } - if system: - body["instructions"] = system - if effort: - body["reasoning"] = {"effort": effort} - return urllib.request.Request( - f"{conf['base_url']}/v1/responses", - data=json.dumps(body).encode(), - headers={ - "Authorization": f"Bearer {conf['api_key']}", - "Accept": "text/event-stream", - "Content-Type": "application/json", - }, - ) - - -_WIRE = { - "anthropic": (anthropic_request, _anthropic_extract), - "openai": (openai_request, _openai_extract), - "openai-responses": (openai_responses_request, None), - "chatgpt": (chatgpt_request, None), -} - - def protocol(runtime, entry) -> str: - """Return the wire protocol for a model registry entry.""" + """The protocol id of a model registry entry's provider.""" return runtime["providers"][entry["provider"]]["protocol"] -def is_openai_platform(base_url: str) -> bool: - """Whether an openai-protocol entry talks to OpenAI itself rather than a - compatible server (DeepSeek, a gateway, a local model).""" - return base_url.startswith("https://api.openai.com") - - def wire_protocol(runtime, entry, tools=None) -> str: - """The wire dialect a call actually uses. OpenAI-protocol calls that carry - function tools go over the platform Responses API (reasoning models reject - tools on chat completions) — but only against the official endpoint: - OpenAI-compatible gateways behind a custom base URL may not implement - /v1/responses, and chat-completions tools still work there.""" + """The wire a call actually goes over (an OpenAI entry's tool calls go + over OpenAI's Responses API — OpenAIChat.wire).""" conf = runtime["providers"][entry["provider"]] - if tools and conf["protocol"] == "openai" and is_openai_platform(conf["base_url"]): - return "openai-responses" - return conf["protocol"] + return ai_protocols.of(conf).wire(conf, tools).id class UpstreamError(RuntimeError): @@ -449,11 +68,9 @@ def open_ai( ): """Open a provider call without consuming response bytes.""" conf = runtime["providers"][entry["provider"]] - build_request = _WIRE[wire_protocol(runtime, entry, tools)][0] - request = build_request( - conf, messages, system, entry["model"], pdf_b64s, - effort, max_tokens, images, stream, tools, - ) + wire = ai_protocols.of(conf).wire(conf, tools) + request = wire.request(conf, messages, system, entry["model"], pdf_b64s, + effort, max_tokens, images, stream, tools) try: return urllib.request.urlopen(request, timeout=timeout) except urllib.error.HTTPError as error: @@ -462,38 +79,10 @@ def open_ai( raise UpstreamError(error.code, detail) -def _int(value) -> int: - try: - return max(0, int(value or 0)) - except (TypeError, ValueError): - return 0 - - def normalize_usage(raw, provider_protocol) -> dict | None: - """One shape for every provider's token report: ``{input, output, - cache_read, cache_write}``. ``input`` is the whole prompt as the - provider counted it (Anthropic reports the cached and freshly written - parts beside the uncached ones — they are summed here, the way OpenAI's - ``prompt_tokens`` already includes ``cached_tokens``); ``cache_read`` / - ``cache_write`` are the parts of it that came from / went to the prompt - cache. Returns None when the object carries no counts.""" - if not isinstance(raw, dict): - return None - if provider_protocol == "anthropic": - cache_read = _int(raw.get("cache_read_input_tokens")) - cache_write = _int(raw.get("cache_creation_input_tokens")) - usage = {"input": _int(raw.get("input_tokens")) + cache_read + cache_write, - "output": _int(raw.get("output_tokens")), - "cache_read": cache_read, "cache_write": cache_write} - elif provider_protocol in ("chatgpt", "openai-responses"): - usage = {"input": _int(raw.get("input_tokens")), "output": _int(raw.get("output_tokens")), - "cache_read": _int((raw.get("input_tokens_details") or {}).get("cached_tokens")), - "cache_write": 0} - else: - usage = {"input": _int(raw.get("prompt_tokens")), "output": _int(raw.get("completion_tokens")), - "cache_read": _int((raw.get("prompt_tokens_details") or {}).get("cached_tokens")), - "cache_write": 0} - return usage if (usage["input"] or usage["output"]) else None + """One shape for every provider's token report (Protocol.usage): + ``{input, output, cache_read, cache_write}``, None without counts.""" + return ai_protocols.get(provider_protocol).usage(raw) def add_usage(total: dict | None, usage: dict | None) -> dict | None: @@ -509,13 +98,7 @@ def read_reply(response, provider_protocol, on_usage=None) -> str: """Read the full reply text from an open provider response. ``on_usage`` (a callable taking the normalized usage dict) hears the token counts when the provider reports them.""" - if provider_protocol in ("chatgpt", "openai-responses"): - return "".join(sse_deltas(response, provider_protocol, on_usage)) - data = json.loads(response.read()) - usage = normalize_usage(data.get("usage"), provider_protocol) - if usage and on_usage: - on_usage(usage) - return _WIRE[provider_protocol][1](data) + return ai_protocols.get(provider_protocol).read_reply(response, on_usage) def call_ai( @@ -530,6 +113,13 @@ def call_ai( return read_reply(response, protocol(runtime, entry), on_usage) +def sse_events(response, provider_protocol): + """The events of a streamed reply on one wire (Protocol.events): + ``("text", delta)``, ``("tool_delta", {id, name, json})``, ``("tool", + {id, name, arguments})`` and a last ``("usage", {...})``.""" + return ai_protocols.get(provider_protocol).events(response) + + def sse_deltas(response, provider_protocol, on_usage=None): """Yield text deltas from a provider's SSE response; ``on_usage`` hears the stream's token counts.""" @@ -540,14 +130,6 @@ def sse_deltas(response, provider_protocol, on_usage=None): on_usage(data) -def _parse_tool_args(raw) -> dict: - try: - parsed = json.loads(raw or "{}") - except ValueError: - parsed = None - return parsed if isinstance(parsed, dict) else {} - - def _partial_json_string(raw: str): """Decode the *unterminated* JSON string body ``raw`` (everything after its opening quote) as far as it goes: a trailing lone backslash or a @@ -700,142 +282,3 @@ def partial_json_strings(raw: str) -> list: out.append(value) break return out - - -def sse_events(response, provider_protocol): - """Yield ``("text", delta)``, ``("tool", {id, name, arguments})`` and - ``("tool_delta", {id, name, json})`` events from a provider's SSE - response. A ``tool_delta`` carries the tool call's arguments as streamed - SO FAR (raw, possibly truncated JSON — see ``partial_json_object``) so a - consumer can preview a long argument while the model is still writing - it; the ``tool`` event with the parsed arguments always follows. A last - ``("usage", {input, output, cache_read, cache_write})`` event reports - the turn's token counts when the provider sent them - (``normalize_usage``). Raises on a fully empty response (neither text - nor tool calls) with the stop reason attached.""" - got = False - stop = "" - usage = None # the provider's token report, normalized - tool = None # anthropic: {id, name, json} tool_use block being accumulated - pending = {} # openai: index -> {id, name, args} accumulated across deltas - items = {} # responses: item id -> {id (call_id), name, args} being streamed - for raw in response: - line = raw.decode("utf-8", "replace").strip() - if not line.startswith("data:"): - continue - data = line[5:].strip() - if data == "[DONE]": - break - try: - event = json.loads(data) - except ValueError: - continue - if provider_protocol == "anthropic": - kind = event.get("type") - if kind == "message_start": - # Input counts arrive up front; the output count comes with - # the final message_delta (cumulative, so the last one wins). - usage = normalize_usage((event.get("message") or {}).get("usage"), "anthropic") - elif kind == "content_block_start": - block = event.get("content_block") or {} - if block.get("type") == "tool_use": - tool = {"id": block.get("id") or "", "name": block.get("name") or "", "json": ""} - elif kind == "content_block_delta": - delta = event.get("delta") or {} - if delta.get("type") == "input_json_delta" and tool is not None: - tool["json"] += delta.get("partial_json") or "" - yield ("tool_delta", dict(tool)) - else: - text = delta.get("text") or "" - if text: - got = True - yield ("text", text) - elif kind == "content_block_stop": - if tool is not None: - got = True - yield ("tool", {"id": tool["id"], "name": tool["name"], - "arguments": _parse_tool_args(tool["json"])}) - tool = None - elif kind == "message_delta": - stop = (event.get("delta") or {}).get("stop_reason") or stop - delta_usage = normalize_usage(event.get("usage"), "anthropic") - if delta_usage: - usage = usage or {"input": 0, "output": 0, "cache_read": 0, "cache_write": 0} - usage["output"] = delta_usage["output"] - if delta_usage["input"] and not usage["input"]: - usage.update(input=delta_usage["input"], cache_read=delta_usage["cache_read"], - cache_write=delta_usage["cache_write"]) - elif kind == "error": - raise RuntimeError((event.get("error") or {}).get("message") or "stream error") - elif provider_protocol in ("chatgpt", "openai-responses"): - kind = event.get("type") or "" - if kind == "response.output_text.delta": - text = event.get("delta") or "" - if text: - got = True - yield ("text", text) - elif kind == "response.output_item.added": - item = event.get("item") or {} - if item.get("type") == "function_call": - items[item.get("id") or ""] = { - "id": item.get("call_id") or item.get("id") or "", - "name": item.get("name") or "", "json": ""} - elif kind == "response.function_call_arguments.delta": - slot = items.get(event.get("item_id") or "") - if slot is not None: - slot["json"] += event.get("delta") or "" - yield ("tool_delta", dict(slot)) - elif kind == "response.output_item.done": - item = event.get("item") or {} - if item.get("type") == "function_call": - got = True - items.pop(item.get("id") or "", None) - yield ("tool", {"id": item.get("call_id") or item.get("id") or "", - "name": item.get("name") or "", - "arguments": _parse_tool_args(item.get("arguments"))}) - elif kind == "response.completed": - stop = (event.get("response") or {}).get("status") or "completed" - usage = normalize_usage((event.get("response") or {}).get("usage"), - provider_protocol) or usage - elif kind in ("response.failed", "error"): - error = ( - (event.get("response") or {}).get("error") or {} - if kind == "response.failed" - else event - ) - raise RuntimeError(error.get("message") or "stream error") - else: - if event.get("error"): - raise RuntimeError((event["error"] or {}).get("message") or "stream error") - if event.get("usage"): - usage = normalize_usage(event["usage"], "openai") or usage - choice = (event.get("choices") or [{}])[0] - delta = choice.get("delta") or {} - text = delta.get("content") or "" - if text: - got = True - yield ("text", text) - for tc in delta.get("tool_calls") or []: - slot = pending.setdefault(tc.get("index", 0), {"id": "", "name": "", "args": ""}) - if tc.get("id"): - slot["id"] = tc["id"] - fn = tc.get("function") or {} - if fn.get("name"): - slot["name"] = fn["name"] - if fn.get("arguments"): - slot["args"] += fn["arguments"] - yield ("tool_delta", {"id": slot["id"], "name": slot["name"], - "json": slot["args"]}) - stop = choice.get("finish_reason") or stop - # OpenAI announces tool calls piecewise; emit them once the stream ends. - for _, slot in sorted(pending.items()): - got = True - yield ("tool", {"id": slot["id"], "name": slot["name"], - "arguments": _parse_tool_args(slot["args"])}) - if usage: - yield ("usage", usage) - if not got: - raise RuntimeError( - f"empty response (stop reason={stop or 'unknown'} — a reasoning model may have spent " - "the whole token budget thinking; try effort: low or a shorter request)" - ) diff --git a/backend/gamma/ai_protocols/__init__.py b/backend/gamma/ai_protocols/__init__.py new file mode 100644 index 00000000..1a2dbfa1 --- /dev/null +++ b/backend/gamma/ai_protocols/__init__.py @@ -0,0 +1,44 @@ +"""The AI wire protocols, one adapter each (base.Protocol): everything that +differs between providers lives on its adapter, never as a protocol branch +in a route. Adding a provider that speaks a new wire is one module here and +one line in ``WIRES``; a new service on an existing wire is a ``SERVICES`` +preset. + +- ``anthropic`` — Anthropic Messages API (Anthropic, Kimi, GLM, …) +- ``openai`` — OpenAI Chat Completions (OpenAI, DeepSeek, OpenRouter, local + servers …); switches to ``openai-responses`` for OpenAI's own tool calls +- ``chatgpt`` — ChatGPT subscription sign-in (the Codex Responses backend) +""" + +from .anthropic import Anthropic +from .base import Protocol +from .chatgpt import ChatGPT +from .openai import OpenAIChat, is_openai_platform +from .responses import OPENAI_RESPONSES + +# Every wire a call may go over, by id; the ones an entry may name come +# first, in the order the settings form offers them. +WIRES = {p.id: p for p in (Anthropic(), OpenAIChat(), ChatGPT(), OPENAI_RESPONSES)} + +# The protocols a provider entry may name. +PROTOCOLS = {pid: p for pid, p in WIRES.items() if p.entry} + +# Named services the settings form offers next to the raw protocols: a +# protocol plus that service's endpoint. An entry made from one is just +# protocol + base URL; the preset only names it (form, provider label). +SERVICES = [ + {"id": "deepseek", "label": "DeepSeek", "protocol": "openai", "base_url": "https://api.deepseek.com"}, +] + + +def get(protocol_id) -> Protocol | None: + """The adapter of a wire protocol id, None for an unknown one.""" + return WIRES.get(protocol_id) + + +def of(conf: dict) -> Protocol: + """The adapter of a resolved provider entry (ai_runtime's ``conf``).""" + return WIRES[conf["protocol"]] + + +__all__ = ["PROTOCOLS", "SERVICES", "WIRES", "Protocol", "get", "is_openai_platform", "of"] diff --git a/backend/gamma/ai_protocols/anthropic.py b/backend/gamma/ai_protocols/anthropic.py new file mode 100644 index 00000000..593b69f4 --- /dev/null +++ b/backend/gamma/ai_protocols/anthropic.py @@ -0,0 +1,145 @@ +"""Anthropic Messages API — Anthropic itself, and the services that speak it +(Kimi, GLM, … behind their own base URL).""" + +import json +from urllib.request import Request as URLRequest + +from .base import Protocol, as_int, attach_index, parse_tool_args + +API_VERSION = "2023-06-01" + + +def _messages(messages) -> list: + """Map the common turn list to Anthropic content blocks: tool results are + tool_result blocks in a user turn (consecutive ones coalesced — they must + directly follow the assistant's tool_use turn), tool calls become tool_use + blocks after the assistant's text.""" + out = [] + for m in messages: + if m["role"] == "tool": + block = {"type": "tool_result", "tool_use_id": m["call_id"], "content": m["content"]} + if m.get("images"): + block["content"] = [{"type": "text", "text": m["content"]}] + [ + {"type": "image", "source": {"type": "base64", "media_type": media_type, "data": data}} + for media_type, data in m["images"]] + prev = out[-1] if out else None + if (prev and prev["role"] == "user" and isinstance(prev["content"], list) + and prev["content"] and prev["content"][0].get("type") == "tool_result"): + prev["content"].append(block) + else: + out.append({"role": "user", "content": [block]}) + elif m["role"] == "assistant" and m.get("tool_calls"): + content = [{"type": "text", "text": m["content"]}] if (m.get("content") or "").strip() else [] + content += [{"type": "tool_use", "id": c["id"], "name": c["name"], "input": c["arguments"]} + for c in m["tool_calls"]] + out.append({"role": "assistant", "content": content}) + else: + prev = out[-1] if out else None + if (m["role"] == "user" and prev and prev["role"] == "user" + and isinstance(prev["content"], list) + and prev["content"] and prev["content"][0].get("type") == "tool_result"): + # A tool-only assistant reply leaves its results as the last + # user turn; fold the next real user message into it so roles + # keep alternating. Attachment turns already carry block lists. + prev["content"].extend( + m["content"] if isinstance(m["content"], list) + else [{"type": "text", "text": m["content"]}]) + else: + out.append({"role": m["role"], "content": m["content"]}) + return out + + +class Anthropic(Protocol): + id = "anthropic" + label = "Anthropic Messages API" + + def request(self, conf, messages, system, model, pdf_b64s=None, effort="", + max_tokens=8192, images=None, stream=False, tools=None): + messages = [dict(m) for m in messages] # attachment injection must not mutate the caller's turn list + if pdf_b64s or images: + last = messages[attach_index(messages)] + last["content"] = [ + *[{"type": "document", + "source": {"type": "base64", "media_type": "application/pdf", "data": data}} + for data in (pdf_b64s or [])], + *[{"type": "image", "source": {"type": "base64", "media_type": media_type, "data": data}} + for media_type, data in (images or [])], + {"type": "text", "text": last["content"]}, + ] + body = {"model": model, "max_tokens": max_tokens, "system": system, + "messages": _messages(messages)} + if tools: + body["tools"] = [{"name": t["name"], "description": t["description"], + "input_schema": t["parameters"]} for t in tools] + if effort: + # "minimal" is OpenAI's lowest level; Anthropic's is "low". + body["output_config"] = {"effort": "low" if effort == "minimal" else effort} + if stream: + body["stream"] = True + return URLRequest(f"{conf['base_url']}/v1/messages", data=json.dumps(body).encode(), headers={ + "x-api-key": conf["api_key"], + "anthropic-version": API_VERSION, + "Content-Type": "application/json", + }) + + def reply_text(self, data): + text = "".join(item.get("text", "") for item in data.get("content", []) if item.get("type") == "text") + if not text.strip(): + raise RuntimeError(f"empty response (stop_reason={data.get('stop_reason', 'unknown')})") + return text + + def usage(self, raw): + # Cached and freshly written prompt parts are reported beside the + # uncached ones; summed, the way OpenAI's prompt_tokens includes them. + if not isinstance(raw, dict): + return None + cache_read = as_int(raw.get("cache_read_input_tokens")) + cache_write = as_int(raw.get("cache_creation_input_tokens")) + usage = {"input": as_int(raw.get("input_tokens")) + cache_read + cache_write, + "output": as_int(raw.get("output_tokens")), + "cache_read": cache_read, "cache_write": cache_write} + return usage if (usage["input"] or usage["output"]) else None + + def stream_event(self, event, state): + kind = event.get("type") + if kind == "message_start": + # Input counts arrive up front; the output count comes with the + # final message_delta (cumulative, so the last one wins). + state["usage"] = self.usage((event.get("message") or {}).get("usage")) + elif kind == "content_block_start": + block = event.get("content_block") or {} + if block.get("type") == "tool_use": + state["tool"] = {"id": block.get("id") or "", "name": block.get("name") or "", "json": ""} + elif kind == "content_block_delta": + delta = event.get("delta") or {} + tool = state.get("tool") + if delta.get("type") == "input_json_delta" and tool is not None: + tool["json"] += delta.get("partial_json") or "" + yield ("tool_delta", dict(tool)) + elif delta.get("text"): + yield ("text", delta["text"]) + elif kind == "content_block_stop": + tool = state.pop("tool", None) + if tool is not None: + yield ("tool", {"id": tool["id"], "name": tool["name"], + "arguments": parse_tool_args(tool["json"])}) + elif kind == "message_delta": + state["stop"] = (event.get("delta") or {}).get("stop_reason") or state["stop"] + delta_usage = self.usage(event.get("usage")) + if delta_usage: + usage = state["usage"] or {"input": 0, "output": 0, "cache_read": 0, "cache_write": 0} + usage["output"] = delta_usage["output"] + if delta_usage["input"] and not usage["input"]: + usage.update(input=delta_usage["input"], cache_read=delta_usage["cache_read"], + cache_write=delta_usage["cache_write"]) + state["usage"] = usage + elif kind == "error": + raise RuntimeError((event.get("error") or {}).get("message") or "stream error") + + def models_request(self, conf): + return URLRequest(f"{conf['base_url']}/v1/models?limit=100", headers={ + "x-api-key": conf["api_key"], + "anthropic-version": API_VERSION, + "Accept": "application/json", + "User-Agent": "Gamma/model-catalog", + }) diff --git a/backend/gamma/ai_protocols/base.py b/backend/gamma/ai_protocols/base.py new file mode 100644 index 00000000..a450d6a2 --- /dev/null +++ b/backend/gamma/ai_protocols/base.py @@ -0,0 +1,263 @@ +"""The protocol adapter interface, and the helpers every wire shares. + +A ``Protocol`` owns everything that differs between providers — the chat +request, the reply and its stream, the token report, the model listing, +the credential check, the account's quota, what the provider accepts — so +the routes and the chat loop never branch on a protocol id. ``conf`` is one +resolved provider entry (``ai_settings.ai_runtime``): ``{protocol, +base_url, api_key, name}`` plus protocol extras (``account_id``). + +Messages come in one common shape: ``{role, content}`` turns, an assistant +turn may carry ``tool_calls`` ([{id, name, arguments-dict}]), and a +``{"role": "tool", "call_id", "content"}`` entry is a tool result, which may +carry ``images`` ([(media_type, base64)] — a rendered PDF page). Tools are +declared once as ``{name, description, parameters}`` (gamma/ai_tools.py); +each adapter maps both to its wire. +""" + +import json +import secrets +import urllib.parse +from urllib.request import Request as URLRequest + +from ..config import AI_BASE_URLS + +# The OpenAI-style wires only accept text in a tool result, so the pictures +# follow the round's results as one user turn the model reads in call order. +TOOL_IMAGES_NOTE = "Pictures returned by the tool calls above, in call order:" + +EMPTY_REPLY_HINT = ("a reasoning model may have spent the whole token budget thinking; " + "try effort: low or a shorter request") + +# The keys a model listing may carry a context window under: Anthropic's +# max_input_tokens, the Codex backend's context_window, OpenRouter's / +# Together's / Fireworks' context_length, Mistral's max_context_length, +# vLLM's max_model_len. +WINDOW_KEYS = ("max_input_tokens", "context_window", "context_length", "max_context_length", "max_model_len") + + +def tool_image_turns(messages, make_turn): + """The common turn list with every run of tool results followed by one + user turn carrying their pictures — ``make_turn(images)`` builds it in + the wire's shape. Yields (message, is_image_turn).""" + pending = [] + for m in messages: + if m["role"] != "tool" and pending: + yield make_turn(pending), True + pending = [] + yield m, False + if m["role"] == "tool": + pending.extend(m.get("images") or []) + if pending: + yield make_turn(pending), True + + +def attach_index(messages) -> int: + """Index of the message attachments ride on: the last plain user turn + (tool-result entries can follow it in agent rounds).""" + for index in range(len(messages) - 1, -1, -1): + if messages[index]["role"] == "user": + return index + return len(messages) - 1 + + +def parse_tool_args(raw) -> dict: + try: + parsed = json.loads(raw or "{}") + except ValueError: + parsed = None + return parsed if isinstance(parsed, dict) else {} + + +def as_int(value) -> int: + try: + return max(0, int(value or 0)) + except (TypeError, ValueError): + return 0 + + +def listed_window(row) -> int: + """A model listing row's context window, 0 when it names none.""" + if not isinstance(row, dict): + return 0 + for key in WINDOW_KEYS: + value = row.get(key) + if isinstance(value, int) and not isinstance(value, bool) and value > 0: + return value + return 0 + + +def sse_json(response): + """The JSON events of a server-sent-events response, up to ``[DONE]``.""" + for raw in response: + line = raw.decode("utf-8", "replace").strip() + if not line.startswith("data:"): + continue + data = line[5:].strip() + if data == "[DONE]": + return + try: + yield json.loads(data) + except ValueError: + continue + + +def bearer_json_request(url: str, key: str, **headers) -> URLRequest: + return URLRequest(url, headers={"Authorization": f"Bearer {key}", "Accept": "application/json", + "User-Agent": "Gamma/model-catalog", **headers}) + + +class Protocol: + """One wire protocol. Subclasses set the class attributes and override + what their provider does differently; the defaults are the common + OpenAI-shaped behaviour (a bearer key, GET {base}/v1/models).""" + + id = "" + label = "" + auth = "key" # "oauth": the entry holds sign-in tokens, refreshed by ``oauth`` + oauth = None # the sign-in module (needs_refresh / refresh) of an "oauth" protocol + entry = True # an entry may name it (False: a variant another protocol switches to) + native_pdf = True # the provider takes the PDF file itself, not only extracted text + streams_only = False # the reply always arrives as SSE, even for a caller that wants it whole + + @property + def base_url(self) -> str: + """The default endpoint (env-overridable, config.AI_BASE_URLS).""" + return AI_BASE_URLS.get(self.id, "") + + # --- chat -------------------------------------------------------------- + + def wire(self, conf, tools=None) -> "Protocol": + """The adapter a call actually goes over (a protocol may switch to a + sibling wire for some calls).""" + return self + + def request(self, conf, messages, system, model, pdf_b64s=None, effort="", + max_tokens=8192, images=None, stream=False, tools=None) -> URLRequest: + raise NotImplementedError + + def reply_text(self, data) -> str: + """The reply text of a non-streamed response body.""" + raise NotImplementedError + + def usage(self, raw) -> dict | None: + """The provider's token report as ``{input, output, cache_read, + cache_write}``: ``input`` is the whole prompt as the provider counted + it, ``cache_read`` / ``cache_write`` the parts of it that came from / + went to the prompt cache. None when the object carries no counts.""" + raise NotImplementedError + + def read_reply(self, response, on_usage=None) -> str: + """The full reply text of an open response; ``on_usage`` hears the + token counts when the provider reports them.""" + if self.streams_only: + parts = [] + for kind, data in self.events(response): + if kind == "text": + parts.append(data) + elif kind == "usage" and on_usage: + on_usage(data) + return "".join(parts) + data = json.loads(response.read()) + usage = self.usage(data.get("usage")) + if usage and on_usage: + on_usage(usage) + return self.reply_text(data) + + def events(self, response): + """Yield ``("text", delta)``, ``("tool", {id, name, arguments})`` and + ``("tool_delta", {id, name, json})`` events from a streamed reply. A + ``tool_delta`` carries the tool call's arguments as streamed SO FAR + (raw, possibly truncated JSON — ai_client.partial_json_object) so a + consumer can preview a long argument while the model is still + writing it; the ``tool`` event with the parsed arguments always + follows. A last ``("usage", {...})`` event reports the turn's token + counts when the provider sent them. Raises on a fully empty response + (neither text nor tool calls) with the stop reason attached.""" + state = {"got": False, "stop": "", "usage": None} + for event in sse_json(response): + for out in self.stream_event(event, state): + state["got"] = state["got"] or out[0] in ("text", "tool") + yield out + for out in self.stream_end(state): + state["got"] = state["got"] or out[0] == "tool" + yield out + if state["usage"]: + yield ("usage", state["usage"]) + if not state["got"]: + raise RuntimeError(f"empty response (stop reason={state['stop'] or 'unknown'} — {EMPTY_REPLY_HINT})") + + def stream_event(self, event, state): + """The events one parsed SSE event yields; ``state`` is the stream's + scratch dict (``stop`` and ``usage`` are read at the end).""" + raise NotImplementedError + + def stream_end(self, state): + """Events owed once the stream ends (tool calls announced piecewise).""" + return () + + # --- models ------------------------------------------------------------ + + def models_request(self, conf) -> URLRequest: + """The provider's model listing — also the free way to check a + credential (it 401s on a dead key without spending tokens).""" + return bearer_json_request(f"{conf['base_url']}/v1/models", conf["api_key"]) + + def models(self, data, conf) -> list: + """The chat models of a listing body as ``[{id, context_window}]`` + (0 = the listing names no window), in the order to offer them.""" + rows = [r for r in (data.get("data") or []) if isinstance(r, dict) and r.get("id")] + found = {} + for row in rows: + found.setdefault(str(row["id"]), listed_window(row)) + return [{"id": mid, "context_window": found[mid]} for mid in sorted(found)] + + def ping_request(self, conf) -> URLRequest: + """The free credential check behind the login connection check.""" + return self.models_request(conf) + + def catalog_hints(self, conf) -> list: + """Names that pick this entry's provider among models.dev's listings + of one model: by default the endpoint's host.""" + return [urllib.parse.urlparse(conf.get("base_url") or "").hostname or ""] + + # --- account and extras -------------------------------------------------- + + # A subscription quota to show. API-key protocols have none that is + # portable: they bill per token, behind each vendor's own billing API. + has_account_usage = False + + def account_usage_request(self, conf) -> URLRequest: + raise NotImplementedError + + def account_usage(self, data) -> dict: + """``{plan_type, windows: [{name, used_percent, remaining_percent, + window_seconds, reset_at}], credits}`` from the quota response.""" + raise NotImplementedError + + def transcription(self, conf) -> int: + """Speech-to-text through this entry: 0 no, 1 maybe (a compatible + server may implement it), 2 yes.""" + return 0 + + def transcription_request(self, conf, model, language, filename, content_type, audio) -> URLRequest: + """The speech-to-text upload (``language`` "" = auto-detect).""" + raise NotImplementedError + + def transcript(self, data) -> str: + """The text of a transcription response.""" + raise NotImplementedError + + +def multipart_body(fields: dict, filename: str, content_type: str, data: bytes): + """Encode fields + one file as multipart/form-data (urllib has no helper).""" + boundary = secrets.token_hex(16) + parts = [] + for name, value in fields.items(): + parts.append(f'--{boundary}\r\nContent-Disposition: form-data; name="{name}"\r\n\r\n{value}\r\n'.encode()) + parts.append( + f'--{boundary}\r\nContent-Disposition: form-data; name="file"; filename="{filename}"\r\n' + f"Content-Type: {content_type}\r\n\r\n".encode() + data + b"\r\n" + ) + parts.append(f"--{boundary}--\r\n".encode()) + return b"".join(parts), f"multipart/form-data; boundary={boundary}" diff --git a/backend/gamma/ai_protocols/chatgpt.py b/backend/gamma/ai_protocols/chatgpt.py new file mode 100644 index 00000000..43db36d7 --- /dev/null +++ b/backend/gamma/ai_protocols/chatgpt.py @@ -0,0 +1,160 @@ +"""ChatGPT subscription sign-in: the Codex Responses backend, reached with +the OAuth tokens of Codex CLI's flow (gamma/chatgpt_oauth.py) — usage is +billed to the subscription. The backend speaks the Responses API but lists +models, checks quota and gates on the client version the way Codex CLI's +own client does; those contracts are provider-specific and may need +maintenance when upstream changes.""" + +import json +import re +import threading +import time +import uuid +from urllib.request import Request as URLRequest, urlopen + +from .. import chatgpt_oauth +from ..logbuf import log +from .base import listed_window +from .responses import ResponsesWire, responses_body + +# GET {base}/models gates its answer on the caller's version, so the listing +# claims the newest Codex CLI release (npm's `latest` tag), looked up live and +# cached. The floor is only for when npm can't be reached. +CODEX_VERSION_URL = "https://registry.npmjs.org/@openai/codex/latest" +CODEX_VERSION_FLOOR = "0.156.1" +CODEX_VERSION_TTL = 6 * 3600 # a good answer +CODEX_VERSION_RETRY = 600 # after a failed lookup +LOOKUP_TIMEOUT = 5 +_codex_version = {"value": "", "until": 0.0} +_codex_version_lock = threading.Lock() + + +def codex_client_version() -> str: + """The newest Codex CLI version, cached; the last good one (else the + floor) while npm is unreachable.""" + with _codex_version_lock: + now = time.time() + if now < _codex_version["until"]: + return _codex_version["value"] or CODEX_VERSION_FLOOR + try: + with urlopen(URLRequest(CODEX_VERSION_URL, headers={"Accept": "application/json"}), + timeout=LOOKUP_TIMEOUT) as resp: + version = str(json.loads(resp.read()).get("version") or "").strip() + if not re.fullmatch(r"\d+\.\d+\.\d+", version): + raise ValueError(f"unexpected version {version!r}") + _codex_version.update(value=version, until=now + CODEX_VERSION_TTL) + except Exception as e: + log.warning(f"[ai] codex version lookup failed, using " + f"{_codex_version['value'] or CODEX_VERSION_FLOOR}: {e}") + _codex_version["until"] = now + CODEX_VERSION_RETRY + return _codex_version["value"] or CODEX_VERSION_FLOOR + + +def _usage_window(raw, name: str = "") -> dict | None: + if not isinstance(raw, dict): + return None + try: + used = max(0.0, min(100.0, float(raw.get("used_percent", 0)))) + except (TypeError, ValueError): + return None + try: + seconds = max(0, int(raw.get("limit_window_seconds") or 0)) + except (TypeError, ValueError): + seconds = 0 + try: + reset_at = int(raw.get("reset_at") or 0) + except (TypeError, ValueError): + reset_at = 0 + if not name: + if 4 * 3600 <= seconds <= 6 * 3600: + name = "5-hour" + elif 6 * 86400 <= seconds <= 8 * 86400: + name = "Weekly" + elif seconds: + name = f"{max(1, round(seconds / 3600))}-hour" + else: + name = "Usage" + return {"name": name, "used_percent": used, "remaining_percent": max(0.0, 100.0 - used), + "window_seconds": seconds, "reset_at": reset_at} + + +class ChatGPT(ResponsesWire): + id = "chatgpt" + label = "ChatGPT (subscription sign-in)" + auth = "oauth" + oauth = chatgpt_oauth + native_pdf = False # the Codex backend refuses input_file parts; the PDF goes as text + has_account_usage = True + + def request(self, conf, messages, system, model, pdf_b64s=None, effort="", + max_tokens=8192, images=None, stream=False, tools=None): + body = {**responses_body(messages, model, pdf_b64s, images, tools, effort), + "instructions": system or "You are a helpful research assistant.", + "include": []} + return URLRequest(f"{conf['base_url']}/responses", data=json.dumps(body).encode(), headers={ + "Authorization": f"Bearer {conf['api_key']}", + "chatgpt-account-id": conf.get("account_id", ""), + "OpenAI-Beta": "responses=experimental", + "originator": "codex_cli_rs", + "session_id": str(uuid.uuid4()), + "Accept": "text/event-stream", + "Content-Type": "application/json", + }) + + def models_request(self, conf): + # Codex CLI's own listing call. + return URLRequest(f"{conf['base_url']}/models?client_version={codex_client_version()}", headers={ + "Authorization": f"Bearer {conf['api_key']}", + "chatgpt-account-id": conf.get("account_id", ""), + "originator": "codex_cli_rs", + }) + + def models(self, data, conf): + listed, hidden = {}, {} + for m in data.get("models") or []: + if not isinstance(m, dict): + continue + slug = str(m.get("slug") or "").strip() + visibility = m.get("visibility") or "list" + if not slug or visibility == "none": # "none" = not usable by this account + continue + # "hide" marks picker-hidden but usable slugs — offered after the + # listed ones rather than dropped. + (hidden if visibility == "hide" else listed).setdefault(slug, listed_window(m)) + found = {**listed, **{k: v for k, v in hidden.items() if k not in listed}} + return [{"id": slug, "context_window": window} for slug, window in found.items()] + + def account_usage_request(self, conf): + # Codex's account client's .../backend-api/wham/usage, sibling of the + # .../codex model endpoint. Always the administrator-controlled + # protocol endpoint, never a saved entry value: OAuth entries cannot + # redirect their bearer token. + base = self.base_url.rstrip("/") + account_base = base[:-len("/codex")] if base.endswith("/codex") else base + headers = {"Authorization": f"Bearer {conf['api_key']}", "Accept": "application/json", + "User-Agent": "codex-cli"} + if conf.get("account_id"): + headers["ChatGPT-Account-Id"] = conf["account_id"] + return URLRequest(f"{account_base}/wham/usage", headers=headers, method="GET") + + def account_usage(self, data): + rate = data.get("rate_limit") if isinstance(data.get("rate_limit"), dict) else {} + windows = [w for w in (_usage_window(rate.get("primary_window")), + _usage_window(rate.get("secondary_window"))) if w] + for extra in data.get("additional_rate_limits") or []: + if not isinstance(extra, dict): + continue + extra_rate = extra.get("rate_limit") if isinstance(extra.get("rate_limit"), dict) else {} + window = _usage_window(extra_rate.get("primary_window"), + str(extra.get("limit_name") or "Additional limit")) + if window: + windows.append(window) + return {"plan_type": str(data.get("plan_type") or ""), "windows": windows, + "credits": data.get("credits") if isinstance(data.get("credits"), dict) else None} + + def ping_request(self, conf): + # The quota endpoint answers 401 on a dead sign-in, for free. + return self.account_usage_request(conf) + + def catalog_hints(self, conf): + return ["openai"] # models.dev lists these models under OpenAI diff --git a/backend/gamma/ai_protocols/openai.py b/backend/gamma/ai_protocols/openai.py new file mode 100644 index 00000000..9fea6ca5 --- /dev/null +++ b/backend/gamma/ai_protocols/openai.py @@ -0,0 +1,154 @@ +"""OpenAI Chat Completions — OpenAI itself and every compatible server +(DeepSeek, OpenRouter, vLLM, Ollama, llama.cpp, LiteLLM, …). The wire +follows the endpoint: only OpenAI gets max_completion_tokens, the Responses +API for tool calls, and the gpt-/o-family filter on its model listing.""" + +import json +import re +from urllib.request import Request as URLRequest + +from .base import (EMPTY_REPLY_HINT, TOOL_IMAGES_NOTE, Protocol, as_int, attach_index, multipart_body, + parse_tool_args, tool_image_turns) +from .responses import OPENAI_RESPONSES + +# Listings include models the chat endpoint can't use. +_NOT_CHAT = re.compile(r"embed|whisper|tts|audio|image|dall-e|moderation|transcribe|realtime|search") +_OPENAI_CHAT_FAMILIES = re.compile(r"^(gpt-|o\d|chatgpt-)") + + +def is_openai_platform(base_url: str) -> bool: + """Whether an entry talks to OpenAI itself rather than a compatible + server (DeepSeek, a gateway, a local model).""" + return (base_url or "").startswith("https://api.openai.com") + + +class OpenAIChat(Protocol): + id = "openai" + label = "OpenAI Chat Completions API" + + def wire(self, conf, tools=None): + # Tool calls to OpenAI itself go over the Responses API (reasoning + # models reject tools on chat completions); a compatible gateway may + # not implement /v1/responses, and chat-completions tools work there. + return OPENAI_RESPONSES if tools and is_openai_platform(conf["base_url"]) else self + + def request(self, conf, messages, system, model, pdf_b64s=None, effort="", + max_tokens=8192, images=None, stream=False, tools=None): + messages = [dict(m) for m in messages] + if pdf_b64s or images: + last = messages[attach_index(messages)] + last["content"] = [ + *[{"type": "file", "file": {"filename": f"document-{index + 1}.pdf", + "file_data": f"data:application/pdf;base64,{data}"}} + for index, data in enumerate(pdf_b64s or [])], + *[{"type": "image_url", "image_url": {"url": f"data:{media_type};base64,{data}"}} + for media_type, data in (images or [])], + {"type": "text", "text": last["content"]}, + ] + wire = [{"role": "system", "content": system}] if system else [] + image_turn = lambda imgs: {"role": "user", "content": [ # noqa: E731 + {"type": "text", "text": TOOL_IMAGES_NOTE}, + *[{"type": "image_url", "image_url": {"url": f"data:{media_type};base64,{data}"}} + for media_type, data in imgs]]} + for m, is_image_turn in tool_image_turns(messages, image_turn): + if is_image_turn: + wire.append(m) + elif m["role"] == "tool": + wire.append({"role": "tool", "tool_call_id": m["call_id"], "content": m["content"]}) + elif m["role"] == "assistant" and m.get("tool_calls"): + wire.append({"role": "assistant", "content": m.get("content") or None, + "tool_calls": [{"id": c["id"], "type": "function", + "function": {"name": c["name"], + "arguments": json.dumps(c["arguments"])}} + for c in m["tool_calls"]]}) + else: + wire.append({"role": m["role"], "content": m["content"]}) + body = { + "model": model, + # Current OpenAI models take max_completion_tokens (the cap includes + # hidden reasoning tokens, so leave a generous default); compatible + # servers take the classic max_tokens. + ("max_completion_tokens" if is_openai_platform(conf["base_url"]) else "max_tokens"): max_tokens, + "messages": wire, + } + if tools: + body["tools"] = [{"type": "function", + "function": {"name": t["name"], "description": t["description"], + "parameters": t["parameters"]}} for t in tools] + if effort: + body["reasoning_effort"] = effort + if stream: + body["stream"] = True + # The final chunk then carries the token counts (OpenAI and the + # common compatible servers: vLLM, Ollama, llama.cpp, LiteLLM). + body["stream_options"] = {"include_usage": True} + return URLRequest(f"{conf['base_url']}/v1/chat/completions", data=json.dumps(body).encode(), headers={ + "Authorization": f"Bearer {conf['api_key']}", + "Content-Type": "application/json", + }) + + def reply_text(self, data): + choices = data.get("choices") or [{}] + text = (choices[0].get("message") or {}).get("content") or "" + if not text.strip(): + reason = choices[0].get("finish_reason", "unknown") + raise RuntimeError(f"empty response (finish_reason={reason} — {EMPTY_REPLY_HINT})") + return text + + def usage(self, raw): + if not isinstance(raw, dict): + return None + usage = {"input": as_int(raw.get("prompt_tokens")), "output": as_int(raw.get("completion_tokens")), + "cache_read": as_int((raw.get("prompt_tokens_details") or {}).get("cached_tokens")), + "cache_write": 0} + return usage if (usage["input"] or usage["output"]) else None + + def stream_event(self, event, state): + if event.get("error"): + raise RuntimeError((event["error"] or {}).get("message") or "stream error") + if event.get("usage"): + state["usage"] = self.usage(event["usage"]) or state["usage"] + choice = (event.get("choices") or [{}])[0] + delta = choice.get("delta") or {} + if delta.get("content"): + yield ("text", delta["content"]) + pending = state.setdefault("pending", {}) # index -> {id, name, args} across deltas + for tc in delta.get("tool_calls") or []: + slot = pending.setdefault(tc.get("index", 0), {"id": "", "name": "", "args": ""}) + if tc.get("id"): + slot["id"] = tc["id"] + fn = tc.get("function") or {} + if fn.get("name"): + slot["name"] = fn["name"] + if fn.get("arguments"): + slot["args"] += fn["arguments"] + yield ("tool_delta", {"id": slot["id"], "name": slot["name"], "json": slot["args"]}) + state["stop"] = choice.get("finish_reason") or state["stop"] + + def stream_end(self, state): + # Tool calls are announced piecewise; emit them once the stream ends. + for _, slot in sorted(state.get("pending", {}).items()): + yield ("tool", {"id": slot["id"], "name": slot["name"], + "arguments": parse_tool_args(slot["args"])}) + + def models(self, data, conf): + platform = is_openai_platform(conf["base_url"]) + # OpenAI's own listing is narrowed to its conversational families; a + # compatible server names its models however it likes. + return [m for m in super().models(data, conf) + if not _NOT_CHAT.search(m["id"]) and (not platform or _OPENAI_CHAT_FAMILIES.match(m["id"]))] + + def transcription(self, conf): + # OpenAI surely transcribes; a compatible server (DeepSeek) may not. + return 2 if is_openai_platform(conf["base_url"]) else 1 + + def transcription_request(self, conf, model, language, filename, content_type, audio): + fields = {"model": model, **({"language": language} if language else {})} + body, multipart_type = multipart_body(fields, filename, content_type, audio) + return URLRequest(f"{conf['base_url']}/v1/audio/transcriptions", data=body, headers={ + "Authorization": f"Bearer {conf['api_key']}", + "Content-Type": multipart_type, + }) + + def transcript(self, data): + return (data.get("text") or "").strip() diff --git a/backend/gamma/ai_protocols/responses.py b/backend/gamma/ai_protocols/responses.py new file mode 100644 index 00000000..88d3dd94 --- /dev/null +++ b/backend/gamma/ai_protocols/responses.py @@ -0,0 +1,143 @@ +"""The Responses API — shared by OpenAI's platform /v1/responses (the wire +an OpenAI entry switches to for tool calls) and the ChatGPT subscription +backend (chatgpt.py).""" + +import json +from urllib.request import Request as URLRequest + +from .base import TOOL_IMAGES_NOTE, Protocol, as_int, parse_tool_args, tool_image_turns + + +def responses_input(messages, pdf_b64s=None, images=None) -> list: + """Map the common turn list to Responses API input items.""" + items = [] + image_turn = lambda imgs: {"type": "message", "role": "user", "content": [ # noqa: E731 + {"type": "input_text", "text": TOOL_IMAGES_NOTE}, + *[{"type": "input_image", "image_url": f"data:{media_type};base64,{data}"} + for media_type, data in imgs]]} + image_turns = [] # never the turn the user's own attachments ride on + for message, is_image_turn in tool_image_turns(messages, image_turn): + if is_image_turn: + items.append(message) + image_turns.append(message) + elif message["role"] == "tool": + items.append({"type": "function_call_output", "call_id": message["call_id"], + "output": message["content"]}) + elif message["role"] == "assistant": + if message.get("content") or not message.get("tool_calls"): + content = [{"type": "output_text", "text": message["content"]}] + items.append({"type": "message", "role": "assistant", "content": content}) + for call in message.get("tool_calls") or []: + items.append({"type": "function_call", "call_id": call["id"], + "name": call["name"], "arguments": json.dumps(call["arguments"])}) + else: + content = [{"type": "input_text", "text": message["content"]}] + items.append({"type": "message", "role": "user", "content": content}) + if pdf_b64s or images: + last = next((item for item in reversed(items) + if item.get("type") == "message" and item.get("role") == "user" + and not any(item is turn for turn in image_turns)), items[-1]) + last["content"] = [ + *[{"type": "input_file", "filename": f"document-{index + 1}.pdf", + "file_data": f"data:application/pdf;base64,{data}"} + for index, data in enumerate(pdf_b64s or [])], + *[{"type": "input_image", "image_url": f"data:{media_type};base64,{data}"} + for media_type, data in (images or [])], + *last["content"], + ] + return items + + +def responses_tools(tools) -> list: + # Responses API uses a flattened function-tool shape (no "function" nesting). + return [{"type": "function", "name": t["name"], "description": t["description"], + "parameters": t["parameters"], "strict": False} for t in (tools or [])] + + +def responses_body(messages, model, pdf_b64s, images, tools, effort) -> dict: + """The request body both Responses backends share.""" + body = { + "model": model, + "input": responses_input(messages, pdf_b64s, images), + "tools": responses_tools(tools), + "tool_choice": "auto", + # Batched calls (e.g. renaming a whole folder in one round) — a call + # per round-trip would eat the tool-round budget one page at a time. + "parallel_tool_calls": bool(tools), + "store": False, + "stream": True, + } + if effort: + body["reasoning"] = {"effort": effort} + return body + + +class ResponsesWire(Protocol): + """The Responses stream and token report; a backend adds its request.""" + + streams_only = True # always SSE — read_reply joins the deltas + + def usage(self, raw): + if not isinstance(raw, dict): + return None + usage = {"input": as_int(raw.get("input_tokens")), "output": as_int(raw.get("output_tokens")), + "cache_read": as_int((raw.get("input_tokens_details") or {}).get("cached_tokens")), + "cache_write": 0} + return usage if (usage["input"] or usage["output"]) else None + + def stream_event(self, event, state): + kind = event.get("type") or "" + items = state.setdefault("items", {}) # item id -> {id (call_id), name, json} being streamed + if kind == "response.output_text.delta": + if event.get("delta"): + yield ("text", event["delta"]) + elif kind == "response.output_item.added": + item = event.get("item") or {} + if item.get("type") == "function_call": + items[item.get("id") or ""] = {"id": item.get("call_id") or item.get("id") or "", + "name": item.get("name") or "", "json": ""} + elif kind == "response.function_call_arguments.delta": + slot = items.get(event.get("item_id") or "") + if slot is not None: + slot["json"] += event.get("delta") or "" + yield ("tool_delta", dict(slot)) + elif kind == "response.output_item.done": + item = event.get("item") or {} + if item.get("type") == "function_call": + items.pop(item.get("id") or "", None) + yield ("tool", {"id": item.get("call_id") or item.get("id") or "", + "name": item.get("name") or "", + "arguments": parse_tool_args(item.get("arguments"))}) + elif kind == "response.completed": + response = event.get("response") or {} + state["stop"] = response.get("status") or "completed" + state["usage"] = self.usage(response.get("usage")) or state["usage"] + elif kind in ("response.failed", "error"): + error = (event.get("response") or {}).get("error") or {} if kind == "response.failed" else event + raise RuntimeError(error.get("message") or "stream error") + + +class OpenAIResponses(ResponsesWire): + """OpenAI's platform /v1/responses, used instead of Chat Completions when + a call carries function tools: reasoning models (gpt-5.x) reject tools + + reasoning_effort on /v1/chat/completions, and OpenAI's guidance is the + Responses API. Only against OpenAI itself — see OpenAIChat.wire.""" + + id = "openai-responses" + label = "OpenAI Responses API" + entry = False + + def request(self, conf, messages, system, model, pdf_b64s=None, effort="", + max_tokens=8192, images=None, stream=False, tools=None): + body = {**responses_body(messages, model, pdf_b64s, images, tools, effort), + "max_output_tokens": max_tokens} + if system: + body["instructions"] = system + return URLRequest(f"{conf['base_url']}/v1/responses", data=json.dumps(body).encode(), headers={ + "Authorization": f"Bearer {conf['api_key']}", + "Accept": "text/event-stream", + "Content-Type": "application/json", + }) + + +OPENAI_RESPONSES = OpenAIResponses() diff --git a/backend/gamma/ai_settings.py b/backend/gamma/ai_settings.py index 8894243f..fc5b8a28 100644 --- a/backend/gamma/ai_settings.py +++ b/backend/gamma/ai_settings.py @@ -2,7 +2,7 @@ server's shared ones. Users manage a LIST of provider entries (Settings → AI → Connections), each: - {"id", "name", "protocol": a key of config.AI_PROTOCOLS, "api_key" (or + {"id", "name", "protocol": a key of ai_protocols.PROTOCOLS, "api_key" (or "oauth" tokens for a sign-in protocol), "base_url": "" = protocol default, "models": "a, b" = comma list ("" = none offered yet), "test_model", "created_at"} @@ -32,8 +32,7 @@ from cryptography.fernet import InvalidToken from fastapi import HTTPException -from . import chatgpt_oauth -from .config import AI_PROTOCOLS, AI_SERVICES +from . import ai_protocols from .db import connect_users_db, get_pref, page_now, set_pref from .logbuf import log from .publisher_sessions import cipher @@ -66,7 +65,8 @@ def new_provider_id() -> str: def is_oauth_protocol(protocol) -> bool: """Sign-in protocols (ChatGPT) hold OAuth tokens instead of an API key.""" - return AI_PROTOCOLS.get(protocol, {}).get("auth") == "oauth" + proto = ai_protocols.PROTOCOLS.get(protocol) + return bool(proto) and proto.auth == "oauth" def apply_provider_fields(entry: dict, fields) -> None: @@ -104,7 +104,7 @@ def apply_provider_fields(entry: dict, fields) -> None: def new_key_entry(fields, entry_id: str) -> dict: """A new API-key entry from an add request (``fields.protocol`` plus the editable fields); 400 on a sign-in or unknown protocol or a missing key.""" - if fields.protocol not in AI_PROTOCOLS or is_oauth_protocol(fields.protocol): + if fields.protocol not in ai_protocols.PROTOCOLS or is_oauth_protocol(fields.protocol): raise HTTPException(status_code=400, detail="unknown protocol") if not (fields.api_key or "").strip(): raise HTTPException(status_code=400, detail="API key required") @@ -121,7 +121,7 @@ def update_entry(entry: dict, fields) -> None: ignored, so the other fields of such a request (reset for the new service) don't land on the old entry.""" if fields.protocol and fields.protocol != entry.get("protocol"): - if fields.protocol not in AI_PROTOCOLS: + if fields.protocol not in ai_protocols.PROTOCOLS: raise HTTPException(status_code=400, detail="unknown protocol") if is_oauth_protocol(fields.protocol) != is_oauth_protocol(entry.get("protocol")): raise HTTPException(status_code=400, @@ -159,12 +159,11 @@ def protocol_choices(key_only: bool = False) -> dict: """What the settings form offers: the protocols (auth "oauth" = sign-in entries, no API key field) and the named services. ``key_only`` drops the sign-in protocols.""" - protocols = [{"id": pid, "label": conf["label"], "default_base_url": conf["base_url"], - "auth": conf.get("auth", "key")} - for pid, conf in AI_PROTOCOLS.items() - if not (key_only and is_oauth_protocol(pid))] + protocols = [{"id": pid, "label": proto.label, "default_base_url": proto.base_url, "auth": proto.auth} + for pid, proto in ai_protocols.PROTOCOLS.items() + if not (key_only and proto.auth == "oauth")] ids = {p["id"] for p in protocols} - return {"protocols": protocols, "services": [s for s in AI_SERVICES if s["protocol"] in ids]} + return {"protocols": protocols, "services": [s for s in ai_protocols.SERVICES if s["protocol"] in ids]} # --- the server's shared entries ---------------------------------------------- @@ -252,11 +251,12 @@ def provider_label(entry: dict) -> str: return name protocol = entry.get("protocol") base = (entry.get("base_url") or "").strip().rstrip("/") - service = next((s for s in AI_SERVICES + service = next((s for s in ai_protocols.SERVICES if s["protocol"] == protocol and s["base_url"] == base), None) if service: return service["label"] - return AI_PROTOCOLS.get(protocol, {}).get("label") or protocol or "" + proto = ai_protocols.PROTOCOLS.get(protocol) + return proto.label if proto else protocol or "" def entry_models(entry: dict) -> list: @@ -266,9 +266,9 @@ def entry_models(entry: dict) -> list: return [m.strip() for m in (entry.get("models") or "").split(",") if m.strip()] -# A failed ChatGPT token refresh isn't retried for this long: ai_runtime runs +# A failed sign-in token refresh isn't retried for this long: ai_runtime runs # on every AI request, and retrying a dead grant each time would add a full -# auth.openai.com round trip to chat/metadata/model calls. +# round trip to the identity provider to chat/metadata/model calls. REFRESH_BACKOFF_S = 300 # One refresh at a time per account. OpenAI rotates refresh tokens, so of two @@ -283,10 +283,11 @@ def _refresh_lock(user: str) -> threading.Lock: return _refresh_locks.setdefault(user, threading.Lock()) -def _refreshed_oauth(user: str, provider_id: str) -> dict | None: - """Refresh one ChatGPT entry's tokens under the account's lock, reading - the entries fresh so a refresh another request just did is reused, not - repeated. Returns the entry's current oauth dict.""" +def _refreshed_oauth(user: str, provider_id: str, flow) -> dict | None: + """Refresh one sign-in entry's tokens through its protocol's OAuth + ``flow`` under the account's lock, reading the entries fresh so a + refresh another request just did is reused, not repeated. Returns the + entry's current oauth dict.""" with _refresh_lock(user): entries = load_provider_entries(user) e = next((x for x in entries if x.get("id") == provider_id), None) @@ -294,9 +295,9 @@ def _refreshed_oauth(user: str, provider_id: str) -> dict | None: if not oauth or not oauth.get("access_token"): return None failed_at = oauth.get("refresh_failed_at") or 0 - if not chatgpt_oauth.needs_refresh(oauth) or time.time() - failed_at <= REFRESH_BACKOFF_S: + if not flow.needs_refresh(oauth) or time.time() - failed_at <= REFRESH_BACKOFF_S: return oauth - refreshed = chatgpt_oauth.refresh(oauth) + refreshed = flow.refresh(oauth) if refreshed: e["oauth"] = oauth = refreshed else: @@ -320,24 +321,25 @@ def ai_runtime(user: str) -> dict: providers, models = {}, [] for e in own + shared: protocol = e.get("protocol") + proto = ai_protocols.PROTOCOLS.get(protocol) pid = str(e.get("id") or "") - if protocol not in AI_PROTOCOLS or not pid or pid in providers: + if not proto or not pid or pid in providers: continue name = provider_label(e) conf = { - "base_url": ((e.get("base_url") or "").strip() or AI_PROTOCOLS[protocol]["base_url"]).rstrip("/"), + "base_url": ((e.get("base_url") or "").strip() or proto.base_url).rstrip("/"), "protocol": protocol, "name": name, } - if protocol == "chatgpt": - # OAuth entry: the bearer token comes from the ChatGPT sign-in and - # is refreshed lazily here (persisted so other requests reuse it). + if proto.auth == "oauth": + # Sign-in entry: the bearer token comes from the sign-in and is + # refreshed lazily here (persisted so other requests reuse it). oauth = e.get("oauth") if isinstance(e.get("oauth"), dict) else None if not oauth or not oauth.get("access_token"): continue failed_at = oauth.get("refresh_failed_at") or 0 - if chatgpt_oauth.needs_refresh(oauth) and time.time() - failed_at > REFRESH_BACKOFF_S: - oauth = _refreshed_oauth(user, pid) or oauth + if proto.oauth.needs_refresh(oauth) and time.time() - failed_at > REFRESH_BACKOFF_S: + oauth = _refreshed_oauth(user, pid, proto.oauth) or oauth conf["api_key"] = oauth["access_token"] conf["account_id"] = oauth.get("account_id") or "" else: @@ -351,10 +353,9 @@ def ai_runtime(user: str) -> dict: if mid not in [m["id"] for m in models]: models.append({"id": mid, "provider": pid, "provider_name": name, "model": model, # Whether the provider takes the PDF file itself - # (native document part). The ChatGPT sign-in wire - # is the Codex backend, which refuses input_file - # parts — the chat falls back to extracted text. - "native_pdf": protocol != "chatgpt", + # (native document part); if not, the chat sends + # extracted text. + "native_pdf": proto.native_pdf, "shared": is_server_id(pid)}) return { "user": user, # whose config this is — the usage recorder's key diff --git a/backend/gamma/app.py b/backend/gamma/app.py index d15d95f2..21c0c21b 100644 --- a/backend/gamma/app.py +++ b/backend/gamma/app.py @@ -10,6 +10,7 @@ from . import backup_schedule, cloud_sync, config, migrations from . import sync_engine, version +from .publish import check_config as check_publish_config from .auth import session_middleware from .db import connect_data_db, connect_pages_db, connect_users_db from .logbuf import log, setup_logging @@ -76,6 +77,11 @@ def _startup_maintenance(): then per workspace: prune orphaned uploads and apply the per-file schema statements (a restored backup gains page_ops, WAL, ...).""" log.info(f"[startup] Gamma {version.label()}") + try: + check_publish_config() + except ValueError as e: + log.error(f"[startup] {e}") + raise SystemExit(1) try: done = migrations.ensure_current() except migrations.MigrationError as e: diff --git a/backend/gamma/backup_schedule.py b/backend/gamma/backup_schedule.py index dd97671e..5dbb0683 100644 --- a/backend/gamma/backup_schedule.py +++ b/backend/gamma/backup_schedule.py @@ -4,6 +4,7 @@ import json import os import re +import time import uuid from contextlib import asynccontextmanager, contextmanager from datetime import datetime, timedelta, timezone @@ -102,7 +103,16 @@ def _write(task): path.parent.mkdir(parents=True, exist_ok=True) temp = path.with_suffix('.tmp') temp.write_text(json.dumps(task), encoding='utf-8') - temp.replace(path) + # Windows refuses to replace a file another thread is reading (the + # Settings table's list_tasks); a reader holds it for one read_text. + for attempt in range(50): + try: + temp.replace(path) + return + except PermissionError: + if attempt == 49: + raise + time.sleep(0.02) @contextmanager @@ -219,7 +229,9 @@ def mutate(owner, task_id, action): targets(task) task.update(requested=True, state='queued') _write(task) - return task + if action != 'delete': + _wake() # "Run now" starts now, not at the next round + return task def _prune(task, ws, at): @@ -272,24 +284,40 @@ def run_due(at=None): log.exception('[backups] Could not process task %s', path.stem) +_wakers = set() # one per running scheduler loop (tests open several app lifespans) + + +def _wake(): + """Start the scheduler's next round now.""" + for waker in list(_wakers): + waker() + + @asynccontextmanager async def lifespan(): - stop = asyncio.Event() + stop, wake = asyncio.Event(), asyncio.Event() + running = asyncio.get_running_loop() async def loop(): while not stop.is_set(): + wake.clear() try: await asyncio.to_thread(run_due) except Exception: log.exception('[backups] Scheduler round failed') - try: - await asyncio.wait_for(stop.wait(), timeout=30) - except asyncio.TimeoutError: - pass - + waiters = [asyncio.ensure_future(stop.wait()), asyncio.ensure_future(wake.wait())] + await asyncio.wait(waiters, timeout=30, return_when=asyncio.FIRST_COMPLETED) + for w in waiters: + w.cancel() + + # mutate() runs in the threadpool, off the event loop. + def waker(): + running.call_soon_threadsafe(wake.set) + _wakers.add(waker) task = asyncio.create_task(loop()) try: yield finally: + _wakers.discard(waker) stop.set() await task diff --git a/backend/gamma/cloud_auth.py b/backend/gamma/cloud_auth.py index 5f88f56a..b239cd75 100644 --- a/backend/gamma/cloud_auth.py +++ b/backend/gamma/cloud_auth.py @@ -239,7 +239,8 @@ def share_host_url() -> str: if not raw: return "" parts = urlsplit(raw) - if parts.scheme not in ("http", "https") or not parts.netloc or parts.query or parts.fragment or parts.path not in ("", "/") or any(c.isspace() for c in raw): + if (parts.scheme not in ("http", "https") or not parts.netloc or parts.query or parts.fragment + or parts.path not in ("", "/") or any(c.isspace() for c in raw)): raise CloudAuthError("the account server names a share host that is not a server address") return raw @@ -493,8 +494,7 @@ def refresh_grant(refresh_token: str) -> dict: if secret: form["client_secret"] = secret return _http(doc["token_endpoint"], data=urllib.parse.urlencode(form).encode(), - headers={"Content-Type": "application/x-www-form-urlencoded", - "User-Agent": _user_agent(server_url() or "no address")}) + headers={"Content-Type": "application/x-www-form-urlencoded"}) def access_token_for(username: str, *, fresh: bool = False) -> str | None: diff --git a/backend/gamma/cloud_sync.py b/backend/gamma/cloud_sync.py index 7e0a2498..815418c1 100644 --- a/backend/gamma/cloud_sync.py +++ b/backend/gamma/cloud_sync.py @@ -11,13 +11,19 @@ is tried again an hour later and does nothing else, so a laptop without network stays signed in. - **The preference profile** (``sync_profile``): the account-wide - ``profile`` pref against the account server's ``profile`` key, - last-writer-wins by ``updated_at`` compared to the millisecond (the - account server's precision). Pulled on a cloud sign-in, before the - browser loads, and on every check; pushed a few seconds after - ``set_pref`` stores a change made here (``profile_changed``). A push the - account server refuses as older (409) takes its value. A failed push is - retried by the next check. The AI provider entries and the active entry + ``profile`` pref against the account server's ``profile`` key, merged + preference by preference against the copy both sides last agreed on + (``profile-base``): a preference changed on one side only takes that + side's value, one changed on both takes the newer profile's. The first + sync of an account whose two copies differ has no base, and waits for + the person's choice (state "choose"): merge (the defaults as the base), + keep the cloud's, or keep this server's — the same three as Settings' + Sync now / Fetch from cloud / Push to cloud. Synced on a cloud sign-in, + before the browser loads; when a browser reads the profile, at most once + a minute (``sync_if_stale``); on every check; and a few seconds after a + change made here (``profile_changed``). A push the account server + refuses as older (409) reads both sides again. A failed push is retried + by the next check. The AI provider entries and the active entry (``ai-settings``, ``ai-provider``) never sync. The last outcome per account is kept in memory for Settings (``profile_status``). - **The server list** (``register_server``): this server's address under @@ -31,18 +37,22 @@ import asyncio import json import threading +import time from contextlib import asynccontextmanager -from datetime import datetime, timezone +from datetime import datetime, timedelta, timezone from . import cloud_auth from .cloud_auth import PROVIDER, CloudAuthError -from .db import PROFILE_PREF_KEY, connect_users_db, get_pref, page_now, restamp_pref, set_profile +from .db import (PROFILE_BASE_PREF_KEY, PROFILE_PREF_KEY, connect_users_db, get_pref, page_now, replace_profile_if, + restamp_pref, set_pref) from .logbuf import log CHECK_INTERVAL = 3600 # seconds between grant checks PUSH_DELAY = 5.0 # seconds a profile change settles before it is pushed -SIGN_IN_TIMEOUT = 5 # the pull a sign-in waits for +SIGN_IN_TIMEOUT = 5 # the pull a sign-in, or a browser reading the profile, waits for +READ_SYNC_EVERY = 60 # seconds: a browser reading the profile syncs it at most this often PROFILE_PATH = "/api/me/prefs/" + PROFILE_PREF_KEY +RESOLUTIONS = ("merge", "fetch", "push") def _background(fn) -> None: @@ -89,8 +99,10 @@ def _local_form(t: datetime) -> str: # The last profile sync outcome per account, in memory only (Settings reads # it through /api/auth/cloud/sync-status): "synced" (the last pull or push # agreed), "pending" (a push is scheduled, or failed and waits for the next -# check; ``error`` then says why), "error" (the last attempt failed). -# "off" is never stored: ``profile_status`` works it out on every read. +# check; ``error`` then says why), "error" (the last attempt failed), +# "choose" (the first sync found two different copies and waits for the +# person to merge or keep one). "off" is never stored: ``profile_status`` +# works it out on every read. _status: dict[str, dict] = {} _status_lock = threading.Lock() @@ -115,7 +127,7 @@ def profile_status(username: str) -> dict: cloud sign-in or without an identity holding a token; an account that syncs but has no outcome yet (a restart, before the first check) is "pending". No network.""" - if not _syncs(username): + if not syncs(username): return {"state": "off", "at": "", "error": ""} with _status_lock: known = _status.get(username) @@ -124,16 +136,93 @@ def profile_status(username: str) -> dict: # --- the preference profile ----------------------------------------------------- -def sync_profile(username: str, token: str | None = None, *, timeout: float = cloud_auth.HTTP_TIMEOUT) -> str: - """Reconcile the account's profile with the account server's: "pulled", - "pushed", "same", or "" when nothing could be done (no token, a failure - — logged).""" +class NothingToFetch(Exception): + """Fetch from cloud while Gamma Cloud holds no profile.""" + + +_MISSING = object() + + +def _same(a, b) -> bool: + """Equal JSON values (``True`` is not ``1``); ``_MISSING`` only equals itself.""" + if a is _MISSING or b is _MISSING: + return a is b + return json.dumps(a, sort_keys=True) == json.dumps(b, sort_keys=True) + + +def merge_profiles(base: dict, local: dict, remote: dict, *, local_newer: bool) -> dict: + """Three-way merge of two profiles against the copy they both started + from, one preference at a time: changed on one side only → that side's + value (gone when that side dropped it); changed on both to different + values → the newer profile's.""" + out = {} + names = [*local, *(k for k in remote if k not in local), *(k for k in base if k not in local and k not in remote)] + for name in names: + was, here, there = (side.get(name, _MISSING) for side in (base, local, remote)) + if _same(here, there) or _same(there, was): + value = here + elif _same(here, was): + value = there + else: + value = here if local_newer else there + if value is not _MISSING: + out[name] = value + return out + + +_locks: dict[str, threading.Lock] = {} +_locks_guard = threading.Lock() +_tried: dict[str, float] = {} # username -> time.monotonic() of the last sync attempt + + +def _lock_of(username: str) -> threading.Lock: + """One sync per account at a time: the timer, the check and a browser's read may meet.""" + with _locks_guard: + return _locks.setdefault(username, threading.Lock()) + + +def _stamp_past(t: datetime | None) -> datetime: + """A push time the account server takes as newer than ``t``: now, or a + millisecond after ``t`` when that clock runs ahead of this one.""" + now = datetime.now(timezone.utc) + now = now.replace(microsecond=now.microsecond // 1000 * 1000) + return max(now, t + timedelta(milliseconds=1)) if t else now + + +def sync_profile(username: str, token: str | None = None, *, timeout: float = cloud_auth.HTTP_TIMEOUT, + resolve: str = "", defaults: dict | None = None) -> str: + """Reconcile the account's profile with the account server's. + + ``resolve`` "" is the automatic sync: a merge against the last agreed + copy, or "choose" when there is none and the two copies differ. The + person's answers: "merge" (against the last agreed copy, else + ``defaults`` — the web app's default profile), "fetch" (the cloud's + copy replaces this one; ``NothingToFetch`` when there is none), "push" + (this copy replaces the cloud's). + + Returns "pulled" (this copy changed), "pushed" (the cloud's changed), + "merged" (both), "same", "choose", or "" when nothing could be done (no + token, a failure — logged and noted).""" token = token or cloud_auth.access_token_for(username) if not token: - if _syncs(username): # offline, or the refresh failed (logged) + if syncs(username): # offline, or the refresh failed (logged) _note_failure(username, UNREACHABLE) return "" + with _lock_of(username): + _tried[username] = time.monotonic() + for _ in range(3): # the cloud or this copy moved on meanwhile: read both again + outcome = _reconcile(username, token, timeout, resolve, defaults or {}) + if outcome is not None: + return outcome + log.warning(f"cloud: the preference profile of {username} kept changing while it synced (tried again at the next check)") + _note_failure(username, "The settings kept changing while they synced.") + return "" + + +def _reconcile(username: str, token: str, timeout: float, resolve: str, defaults: dict) -> str | None: + """One round of ``sync_profile``; None = read both sides again.""" local, local_at = get_pref(username, PROFILE_PREF_KEY) + local = local if isinstance(local, dict) else None try: remote = _call("GET", PROFILE_PATH, token, timeout=timeout) except CloudAuthError as e: @@ -142,58 +231,98 @@ def sync_profile(username: str, token: str | None = None, *, timeout: float = cl _note_failure(username, e) return "" remote = {} - remote_at = _ms(remote.get("updated_at")) if isinstance(remote.get("value"), dict) else None - here_at = _ms(local_at) if isinstance(local, dict) else None - if remote_at and (not here_at or remote_at > here_at): - set_profile(username, remote["value"], updated_at=_local_form(remote_at)) - _note(username, "synced") - return "pulled" - if not here_at or here_at == remote_at: - _note(username, "synced") - return "same" - return _push(username, token, local, local_at, timeout=timeout) - - -def _push(username: str, token: str, value: dict, updated_at: str, *, timeout: float = cloud_auth.HTTP_TIMEOUT) -> str: + cloud = remote.get("value") if isinstance(remote.get("value"), dict) else None + cloud_at = _ms(remote.get("updated_at")) if cloud is not None else None + base = _base_of(username) + + if resolve == "fetch": + if cloud is None: + raise NothingToFetch() + target = cloud + elif resolve == "push": + target = local if local is not None else {} + elif cloud is None or local is None: + target = local if local is not None else cloud + if target is None: + _note(username, "synced") + return "same" + elif base is None and resolve != "merge" and not _same(local, cloud): + _note(username, "choose") + return "choose" + else: + here_at = _ms(local_at) + newer = bool(here_at and (not cloud_at or here_at > cloud_at)) + if base is not None: + target = merge_profiles(base, local, cloud, local_newer=newer) + else: # a first merge: an entry a side lacks is at its default there, not dropped + target = merge_profiles(defaults, {**defaults, **local}, {**defaults, **cloud}, local_newer=newer) + changes_here = local is None or not _same(target, local) + + if resolve != "push" and cloud is not None and _same(target, cloud): + # the cloud's copy as it stands: nothing to send + if changes_here and not replace_profile_if(username, target, local_at, _local_form(cloud_at)): + return None + _agreed(username, target) + return "pulled" if changes_here else "same" + try: - answer = _call("PUT", PROFILE_PATH, token, {"value": value, "updated_at": updated_at}, timeout=timeout) + answer = _call("PUT", PROFILE_PATH, token, {"value": target, "updated_at": _local_form(_stamp_past(cloud_at))}, + timeout=timeout) except CloudAuthError as e: - stored_at = _ms(e.body.get("updated_at")) - if e.status == 409 and isinstance(e.body.get("value"), dict) and stored_at: - # the account server holds a newer profile: take it - set_profile(username, e.body["value"], updated_at=_local_form(stored_at)) - _note(username, "synced") - return "pulled" + if e.status == 409: # another server pushed meanwhile + return None _failed(username, "push the preference profile", e) _note(username, "pending", str(e) or UNREACHABLE) return "" - stored_at = _ms(answer.get("updated_at")) - if stored_at and stored_at != _ms(updated_at): - # stored under the account server's time (clamped, or cut to the - # millisecond): keep that version here too, so the next check agrees - restamp_pref(username, PROFILE_PREF_KEY, updated_at, _local_form(stored_at)) + # keep the account server's time here too (clamped, or cut to the + # millisecond), so both copies carry one version + stored_at = _local_form(_ms(answer.get("updated_at")) or _stamp_past(cloud_at)) + if changes_here: + if not replace_profile_if(username, target, local_at, stored_at): + return None # a change landed here meanwhile: merge it against the old base + elif local_at: + restamp_pref(username, PROFILE_PREF_KEY, local_at, stored_at) + _agreed(username, target) + return "merged" if changes_here else "pushed" + + +def _agreed(username: str, value: dict) -> None: + """Both copies now hold ``value``: the base of the next merge, kept + with the cloud account it was agreed with.""" + set_pref(username, PROFILE_BASE_PREF_KEY, {"subject": cloud_auth.grant_of(username)[0], "profile": value}) _note(username, "synced") - return "pushed" -def push_profile(username: str) -> str: - """Push the stored profile now (the debounced push).""" - value, updated_at = get_pref(username, PROFILE_PREF_KEY) - if not isinstance(value, dict) or not updated_at: - return "" - token = cloud_auth.access_token_for(username) - if not token: - if _syncs(username): - _note_failure(username, UNREACHABLE) - return "" - return _push(username, token, value, updated_at) +def _base_of(username: str) -> dict | None: + """The last agreed profile, or None: never synced, or agreed with + another cloud account than the one linked now (an unlink, then a link + to someone else's).""" + stored, _ = get_pref(username, PROFILE_BASE_PREF_KEY) + if not isinstance(stored, dict) or not isinstance(stored.get("profile"), dict): + return None + return stored["profile"] if stored.get("subject") == cloud_auth.grant_of(username)[0] else None + + +def sync_if_stale(username: str) -> None: + """A browser is reading the profile (a tab opened or refocused): sync + first when the last attempt is more than ``READ_SYNC_EVERY`` seconds + old, so a change made on another server shows up. Never raises.""" + try: + last = _tried.get(username) + if (last is not None and time.monotonic() - last < READ_SYNC_EVERY) or not syncs(username): + return + _tried[username] = time.monotonic() + sync_profile(username, timeout=SIGN_IN_TIMEOUT) + except Exception as e: + log.exception(f"cloud: syncing the preference profile of {username} on read failed") + _note_failure(username, e) _timers: dict[str, threading.Timer] = {} _timers_lock = threading.Lock() -def _syncs(username: str) -> bool: +def syncs(username: str) -> bool: """Whether the account's profile follows it through Gamma Cloud: cloud sign-in on and an identity holding a token. No network.""" if not cloud_auth.settings()["enabled"]: @@ -203,16 +332,20 @@ def _syncs(username: str) -> bool: def profile_changed(username: str) -> None: - """``set_pref``'s hook for a profile change made here: push it once - changes have settled for ``PUSH_DELAY`` seconds, on a timer thread. - Never raises into the request that stored the change.""" + """The hook for a profile change made here (``set_pref``, + ``patch_profile``): sync it once changes have settled for + ``PUSH_DELAY`` seconds, on a timer thread. Never raises into the + request that stored the change.""" try: - if not _syncs(username): + if not syncs(username): return except Exception: log.exception("cloud: could not check whether a profile change syncs") return - _note(username, "pending") + with _status_lock: + choosing = _status.get(username, {}).get("state") == "choose" + if not choosing: # still waiting for the person's choice: nothing will be sent + _note(username, "pending") with _timers_lock: old = _timers.get(username) if old: @@ -228,7 +361,7 @@ def _push_settled(username: str) -> None: if _timers.get(username) is threading.current_thread(): del _timers[username] try: - push_profile(username) + sync_profile(username) except Exception as e: log.exception(f"cloud: pushing the preference profile of {username} failed") _note_failure(username, e) @@ -306,7 +439,7 @@ def check(username: str) -> str: (offline — nothing happens — or revoked, handled by cloud_auth).""" token = cloud_auth.access_token_for(username, fresh=True) if not token: - if _syncs(username): # offline: the grant stays, the profile waits + if syncs(username): # offline: the grant stays, the profile waits _note_failure(username, UNREACHABLE) return "" sync_profile(username, token) diff --git a/backend/gamma/config.py b/backend/gamma/config.py index eaa7be14..9269730a 100644 --- a/backend/gamma/config.py +++ b/backend/gamma/config.py @@ -57,6 +57,31 @@ def cloud_env() -> dict: "share_host": os.environ.get("GAMMA_CLOUD_SHARE_HOST", "").strip().lower() in ("1", "true", "yes", "on")} +# The share host's published-page cap per Gamma Cloud plan (gamma/publish.py +# page_cap): a plan missing here is unlimited. GAMMA_FREE_PAGE_LIMIT +# overrides the free plan's number (0 lifts the cap). +PLAN_PAGE_LIMITS = {"free": 5} + + +def plan_page_limits() -> dict: + limits = dict(PLAN_PAGE_LIMITS) + raw = os.environ.get("GAMMA_FREE_PAGE_LIMIT", "").strip() + if raw: + n = int(raw) # checked at startup (publish.check_config) + if n > 0: + limits["free"] = n + else: + limits.pop("free", None) + return limits + + +def page_host_pattern() -> str: + """``GAMMA_PAGE_HOST``: the share host's per-account page hostname with + a ``{username}`` placeholder, e.g. ``{username}-pages.gammapdf.com`` + ("" = no pretty addresses, token links only; gamma/publish.py).""" + return os.environ.get("GAMMA_PAGE_HOST", "").strip().lower() + + def sync_interval_s() -> int: """Seconds between mirror sync rounds (gamma/sync_engine.py); 0 turns the background loop off (the API's "sync now" still works).""" @@ -70,49 +95,20 @@ def sync_interval_s() -> int: # --- AI chat ----------------------------------------------------------------- # AI configuration is per-user, not env: each user adds provider entries in the -# GUI (Settings → AI → Connections) — a protocol + credential + optional label, -# base URL, and model list — stored server-side in users.db and resolved per -# request by gamma/ai_settings.ai_runtime(). Three protocols exist: -# "anthropic" — Anthropic Messages API (Anthropic, Kimi, GLM, ...) -# "openai" — OpenAI Chat Completions API (OpenAI, DeepSeek and compatible) -# "chatgpt" — ChatGPT subscription sign-in (the Codex Responses backend) -# No model names live here: an entry offers the models picked for it from the -# provider's live listing. The env can only override each protocol's default -# base URL (shown as the placeholder in the GUI and used when an entry leaves -# it blank): GAMMA_AI_ANTHROPIC_BASE_URL / GAMMA_AI_OPENAI_BASE_URL / -# GAMMA_AI_CHATGPT_BASE_URL (legacy GAMMA_AI_BASE_URL / ANTHROPIC_BASE_URL -# alias the anthropic slot). +# GUI (Settings → AI → Connections), resolved per request by +# gamma/ai_settings.ai_runtime(); the protocols themselves are the adapters in +# gamma/ai_protocols/. The env can only override each protocol's default base +# URL (shown as the placeholder in the GUI and used when an entry leaves it +# blank). Legacy GAMMA_AI_BASE_URL / ANTHROPIC_BASE_URL alias the anthropic +# slot. _legacy_url = os.environ.get("GAMMA_AI_BASE_URL", "") or os.environ.get("ANTHROPIC_BASE_URL", "") -AI_PROTOCOLS = { - "anthropic": { - "label": "Anthropic Messages API", - "base_url": (os.environ.get("GAMMA_AI_ANTHROPIC_BASE_URL", "") or _legacy_url - or "https://api.anthropic.com").rstrip("/"), - }, - "openai": { - "label": "OpenAI Chat Completions API", - "base_url": (os.environ.get("GAMMA_AI_OPENAI_BASE_URL", "") - or "https://api.openai.com").rstrip("/"), - }, - # No API key: the entry holds OAuth tokens from signing in with a ChatGPT - # account (Codex CLI's flow) — usage is billed to the subscription. The - # base URL is the Codex Responses endpoint on the ChatGPT backend. - # auth "oauth" marks sign-in protocols for the settings form and the - # provider CRUD guards (default is "key"). - "chatgpt": { - "label": "ChatGPT (subscription sign-in)", - "base_url": (os.environ.get("GAMMA_AI_CHATGPT_BASE_URL", "") - or "https://chatgpt.com/backend-api/codex").rstrip("/"), - "auth": "oauth", - }, +AI_BASE_URLS = { + "anthropic": (os.environ.get("GAMMA_AI_ANTHROPIC_BASE_URL", "") or _legacy_url + or "https://api.anthropic.com").rstrip("/"), + "openai": (os.environ.get("GAMMA_AI_OPENAI_BASE_URL", "") + or "https://api.openai.com").rstrip("/"), + "chatgpt": (os.environ.get("GAMMA_AI_CHATGPT_BASE_URL", "") + or "https://chatgpt.com/backend-api/codex").rstrip("/"), } - -# Named services the settings form offers next to the raw protocols: one of -# the protocols above plus that service's endpoint. An entry made from one is -# just protocol + base URL; the preset only names it (form, provider label). -AI_SERVICES = [ - {"id": "deepseek", "label": "DeepSeek", "protocol": "openai", - "base_url": "https://api.deepseek.com"}, -] diff --git a/backend/gamma/db.py b/backend/gamma/db.py index 93b4e34d..bc10e2b5 100644 --- a/backend/gamma/db.py +++ b/backend/gamma/db.py @@ -376,13 +376,18 @@ def connect_users_db() -> sqlite3.Connection: # Prefs that follow the account regardless of workspace (stored with -# workspace_id ''): the AI provider entries, the active entry, and the -# preference profile. Everything else is per account + workspace, because the +# workspace_id ''): the AI provider entries, the active entry, the +# translation engine keys (gamma/translate_engines.py), and the preference +# profile. Everything else is per account + workspace, because the # value names that workspace's pages (open tabs, recents, pinned folders, # reading positions). PROFILE_PREF_KEY = "profile" +# gamma/cloud_sync.py: the profile as this server and Gamma Cloud last agreed +# on it, the base of the next three-way merge. Never served by /api/prefs. +PROFILE_BASE_PREF_KEY = "profile-base" NOTICES_SEEN_PREF_KEY = "notices-seen" # gamma/notices.py: {notice id: fingerprint seen} -USER_PREF_KEYS = frozenset({"ai-settings", "ai-provider", PROFILE_PREF_KEY, NOTICES_SEEN_PREF_KEY}) +USER_PREF_KEYS = frozenset({"ai-settings", "ai-provider", "translate-engines", PROFILE_PREF_KEY, + PROFILE_BASE_PREF_KEY, NOTICES_SEEN_PREF_KEY}) def pref_scope(key: str, ws: str) -> str: @@ -484,6 +489,41 @@ def set_profile(username: str, value: dict, *, updated_at: str | None = None) -> return set_pref(username, PROFILE_PREF_KEY, value, updated_at=updated_at) +def replace_profile_if(username: str, value: dict, old: str, new: str) -> bool: + """Store ``value`` under ``new`` only while the profile is still at + ``old`` ("" = none stored yet): the cloud sync's write, which loses to a + change made meanwhile. Never pushes back.""" + with connect_users_db() as db: + if old: + cur = db.execute("UPDATE user_prefs SET value = ?, updated_at = ? WHERE username = ? AND workspace_id = '' " + "AND key = ? AND updated_at = ?", (json.dumps(value), new, username, PROFILE_PREF_KEY, old)) + else: + cur = db.execute("INSERT OR IGNORE INTO user_prefs (username, workspace_id, key, value, updated_at) " + "VALUES (?, '', ?, ?, ?)", (username, PROFILE_PREF_KEY, json.dumps(value), new)) + db.commit() + return bool(cur.rowcount) + + +def patch_profile(username: str, changes: dict) -> tuple[dict, str]: + """Set the preferences in ``changes`` and keep every other entry as + stored: how a browser saves, so its stale copy of a preference it did not + touch never undoes one synced from elsewhere. A change made here (pushed + like set_profile's). Returns (profile, updated_at).""" + for _ in range(5): # another write landed between the read and this one: read again + value, at = get_profile(username) + merged = {**value, **changes} + stamp = page_now() + if at and at >= stamp: + stamp = _stamp_after(at) + if replace_profile_if(username, merged, at, stamp): + break + else: # an unreadable stored row: replace it + return merged, set_profile(username, merged) + from . import cloud_sync # local: cloud_sync imports this module + cloud_sync.profile_changed(username) + return merged, stamp + + # --- workspace files --------------------------------------------------------- def ws_dir(ws: str) -> Path: diff --git a/backend/gamma/migrations.py b/backend/gamma/migrations.py index c726edc3..6c18976d 100644 --- a/backend/gamma/migrations.py +++ b/backend/gamma/migrations.py @@ -528,7 +528,6 @@ def _v17_profile(conn: sqlite3.Connection) -> None: conn.commit() - def _v18_cloud_grant(conn: sqlite3.Connection) -> None: """``sessions`` gains ``via`` ('' a password or the guest, 'cloud' a Gamma Cloud sign-in) and ``identities`` gains ``revoked_at``: the grant diff --git a/backend/gamma/notices.py b/backend/gamma/notices.py index ec4f90ef..1678b2b7 100644 --- a/backend/gamma/notices.py +++ b/backend/gamma/notices.py @@ -1,31 +1,38 @@ """What wants a look: the notices behind the red dot on the account button. A notice is one thing an account should see once — a newer Gamma release, -errors in the server log — and it points at the Settings pane that shows -it. Each carries a *fingerprint* naming what changed (the release version, -the seq of the newest error); "resolved" means the account has seen that -fingerprint, recorded in the account-wide ``notices-seen`` pref as -``{id: fingerprint}``. Visiting the pane records it (the frontend's -``useNotices``); a new release or a fresh error changes the fingerprint -and the notice is back on its own. Nothing is ever dismissed for good. - -Sources are plain functions registered with ``@source``; each returns a -Notice or None and must be cheap — a cached or in-memory read — because -``for_user`` runs on every poll of ``GET /api/notices``. Admin-only -sources are skipped for everyone else, so a member's poll does no work -beyond that. The one network call, the release check, sits behind -``version.latest_release``'s six-hour cache. +errors in the server log, a failed backup task — and it points at the +Settings pane that shows it. Each carries a *fingerprint* naming what +changed (the release version, the seq of the newest error, the failed +task's run time); "resolved" means the account has seen that fingerprint, +recorded in the account-wide ``notices-seen`` pref as ``{id: fingerprint}``. +Visiting the pane records it (the frontend's ``useNotices``); a new release +or a fresh error changes the fingerprint and the notice is back on its own. +Nothing is ever dismissed for good. + +Sources are plain functions ``fn(username) -> Notice | None`` registered +with ``@source``; each must be cheap — a cached, in-memory or small +database read — because ``for_user`` runs them on every poll of +``GET /api/notices``. Admin-only sources are skipped for everyone else. +The one network call, the release check, sits behind +``version.latest_release``'s six-hour cache; the one directory walk, the +storage usage, runs only for an account under a quota and is remembered +for a while. """ +import hashlib import re +import threading +import time from dataclasses import asdict, dataclass -from . import logbuf, version +from . import backup_schedule, cloud_sync, logbuf, server_settings, sync_engine, translate_engines, version from .db import NOTICES_SEEN_PREF_KEY, get_pref, set_pref TONES = ("info", "warn", "error") _ID_RE = re.compile(r"^[a-z][a-z0-9-]{0,31}$") _MAX_SEEN = 64 +MB = 1024 * 1024 @dataclass(frozen=True) @@ -41,15 +48,19 @@ class Notice: def source(*, admin_only=False): - """Register a notice source: ``fn() -> Notice | None``.""" + """Register a notice source: ``fn(username) -> Notice | None``.""" def wrap(fn): _SOURCES.append((fn, admin_only)) return fn return wrap +def _plural(n: int, word: str) -> str: + return f"{n} {word}" + ("" if n == 1 else "s") + + @source(admin_only=True) -def update_available(): +def update_available(_username): """A newer GitHub release than this build (nothing for a checkout, an air-gapped server or an unreachable GitHub).""" release, _error = version.latest_release() @@ -62,7 +73,7 @@ def update_available(): @source(admin_only=True) -def log_errors(): +def log_errors(_username): """Errors logged since the account last looked at the server log. The fingerprint is the start time plus the newest error's seq: a restart resets both, so an old ack never covers a new error.""" @@ -73,6 +84,142 @@ def log_errors(): return Notice("log-errors", f"{started}:{seq}", "error", "server", "New errors in the server log") +@source() +def backup_failed(username): + """The account's backup tasks whose last run failed (Settings → + Backups shows the error). Another failed run, of any of them, is a new + fingerprint.""" + failed = [t for t in backup_schedule.list_tasks(username) if t.get("state") == "failed"] + if not failed: + return None + mark = ",".join(f"{t['id'][:12]}:{t.get('last_run') or ''}" for t in sorted(failed, key=lambda t: t["id"])) + title = (f'The backup task "{failed[0]["name"]}" failed' if len(failed) == 1 + else f"{_plural(len(failed), 'backup task')} failed") + return Notice("backup-failed", mark, "error", "backups", title) + + +def _conflict_marks(username, publications): + """Per clone (or per publication), the open conflict count and newest + conflict id, folded into one short digest (a fingerprint is capped at + 200 characters, which a dozen clones with conflicts would pass); a + mirror with a page filter is a publication.""" + marks, total = [], 0 + for mirror in sync_engine.list_mirrors(username): + if (mirror.get("page_filter") is not None) != publications: + continue + count, newest = sync_engine.open_conflict_mark(mirror["workspace_id"]) + if count: + marks.append(f"{mirror['workspace_id']}:{count}:{newest}") + total += count + return hashlib.sha1(",".join(marks).encode("utf-8")).hexdigest()[:16], total + + +@source() +def mirror_conflicts(username): + """Open conflicts in the clones the account owns (Settings → Account & sync → + Clones). Fingerprint: per clone, the count and the newest + conflict — a new one brings the notice back, resolving old ones does + not.""" + mark, total = _conflict_marks(username, publications=False) + if not total: + return None + return Notice("mirror-conflicts", mark, "warn", "account", + f"{_plural(total, 'sync conflict')} to look at in your clones") + + +@source() +def publish_conflicts(username): + """Open conflicts in the pages the account publishes to Gamma Cloud + (Settings → Account & sync → Publishing), fingerprinted like the clones'.""" + mark, total = _conflict_marks(username, publications=True) + if not total: + return None + return Notice("publish-conflicts", mark, "warn", "account", + f"{_plural(total, 'sync conflict')} to look at in your published pages") + + +@source() +def cloud_sync_failed(username): + """The account's Gamma Cloud sync in its error state (the Account + pane's cloud row says why).""" + status = cloud_sync.profile_status(username) + if status.get("state") != "error": + return None + error = (status.get("error") or "").strip().rstrip(".") + return Notice("cloud-sync", status.get("at") or "", "warn", "account", + f"Gamma Cloud sync failed: {error}" if error else "Gamma Cloud sync failed") + + +@source() +def cloud_sync_choice(username): + """The first settings sync with Gamma Cloud found two different copies + and waits for the person to merge them or keep one (the Account pane).""" + if cloud_sync.profile_status(username).get("state") != "choose": + return None + return Notice("cloud-sync-choice", "choose", "warn", "account", + "Your settings here and on Gamma Cloud differ: choose which to keep") + + +@source() +def free_translate_failing(username): + """Microsoft's free translation endpoint keeps failing for this account + (translate_engines' in-memory streak); gone after one success.""" + failing = translate_engines.free_failing(username) + if not failing: + return None + return Notice("free-translate", failing["since"], "warn", "reading", + "Microsoft's free translation keeps failing — set up Google or Youdao") + + +# The storage walk is the one source that is not a free read: usage is +# remembered per account for a few minutes, and only computed at all when +# the account is under a quota. +_USAGE_TTL = 10 * 60 +_usage: dict[str, tuple[float, int]] = {} +_usage_lock = threading.Lock() + + +def _usage_bytes(username: str) -> int: + now = time.monotonic() + with _usage_lock: + known = _usage.get(username) + if known and now - known[0] < _USAGE_TTL: + return known[1] + used = server_settings.usage_bytes(username) + with _usage_lock: + _usage[username] = (now, used) + return used + + +def forget_usage(username: str | None = None) -> None: + """Drop the remembered usage (tests; a caller that just changed it).""" + with _usage_lock: + if username is None: + _usage.clear() + else: + _usage.pop(username, None) + + +@source() +def storage_nearly_full(username): + """The account's personal storage past nine tenths of its quota (warn) + or full (error). Fingerprint: the threshold crossed, so each fires once + until the pane is seen — and again after the usage drops and climbs + back.""" + quota_mb = server_settings.user_limits(username).get("quota_mb") or 0 + if not quota_mb: + return None + used = _usage_bytes(username) + share = used / (quota_mb * MB) + if share >= 1: + return Notice("storage", "full", "error", "account", + f"Your storage is full ({used // MB} of {quota_mb} MB used)") + if share >= 0.9: + return Notice("storage", "90", "warn", "account", + f"Your storage is nearly full ({used // MB} of {quota_mb} MB used)") + return None + + def seen_map(username: str) -> dict: value, _ = get_pref(username, NOTICES_SEEN_PREF_KEY) return value if isinstance(value, dict) else {} @@ -81,7 +228,7 @@ def seen_map(username: str) -> dict: def for_user(username: str, is_admin: bool) -> list[dict]: """The unresolved notices of an account, strongest tone first.""" found = [notice for fn, admin_only in _SOURCES if is_admin or not admin_only - if (notice := fn()) is not None] + if (notice := fn(username)) is not None] if not found: return [] seen = seen_map(username) diff --git a/backend/gamma/publish.py b/backend/gamma/publish.py index a21b07bc..833da740 100644 --- a/backend/gamma/publish.py +++ b/backend/gamma/publish.py @@ -18,13 +18,20 @@ A workspace that already mirrors another server (a lab NAS) cannot publish: one remote per copy, and the page's home is that other server. + +The share host also caps how many pages a plan may publish (``page_cap``, +``config.PLAN_PAGE_LIMITS``) and, with ``GAMMA_PAGE_HOST`` set, gives every +published page a pretty address on a hostname per account +(``https://-pages.gammapdf.com/-``, resolved by +``resolve_public``; the slug is decoration, the trailing id routes). """ import json import re -from urllib.parse import urlsplit +import unicodedata +from urllib.parse import unquote, urlsplit -from . import cloud_auth, integrations, ratelimit, sync_engine, workspaces +from . import cloud_auth, config, integrations, ratelimit, sync_engine, workspaces from .auth import SHARE_AUDIENCES, SHARE_ROLES from .cloud_auth import CloudAuthError from .db import connect_pages_db, connect_users_db @@ -38,12 +45,14 @@ class PublishError(Exception): - """A refusal with the HTTP status the router answers.""" + """A refusal with the HTTP status the router answers (``extra``: more + fields for the answer's body, next to ``detail``).""" - def __init__(self, status: int, message: str): + def __init__(self, status: int, message: str, extra: dict | None = None): super().__init__(message) self.status = status self.message = message + self.extra = extra or {} def this_is_share_host() -> bool: @@ -56,6 +65,145 @@ def publishing_blocked() -> bool: return this_is_share_host() +# --- slugs and page hosts --------------------------------------------------------- + +SLUG_MAX = 60 +PLACEHOLDER = "{username}" +_LABEL_RE = re.compile(r"^[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?$") + + +def slug(title: str) -> str: + """The decorative part of a page's public path: the title ASCII-folded + (NFKD, marks dropped), lowercased, every run of anything but ``[a-z0-9]`` + one ``-``, trimmed, at most ``SLUG_MAX`` characters. "" when nothing is + left (a CJK title). Mirrored in frontend/src/shared/lib/slug.js; + tests/shared/slug.json pins both.""" + text = unicodedata.normalize("NFKD", title or "") + text = "".join(c for c in text if not unicodedata.category(c).startswith("M")).lower() + text = re.sub(r"[^a-z0-9]+", "-", text).strip("-") + return text[:SLUG_MAX].rstrip("-") + + +def _valid_host(host: str) -> bool: + labels = host.split(".") + return len(host) <= 253 and len(labels) >= 2 and all(_LABEL_RE.match(label) for label in labels) + + +def check_config() -> None: + """Startup: ``GAMMA_PAGE_HOST`` names ``{username}`` exactly once and is + a hostname otherwise; ``GAMMA_FREE_PAGE_LIMIT`` is a whole number. + ValueError with the reason.""" + try: + config.plan_page_limits() + except ValueError: + raise ValueError("GAMMA_FREE_PAGE_LIMIT must be a whole number (0 lifts the cap).") from None + pattern = config.page_host_pattern() + if not pattern: + return + if pattern.count(PLACEHOLDER) != 1: + raise ValueError("GAMMA_PAGE_HOST must contain {username} exactly once, e.g. {username}-pages.example.org.") + if not _valid_host(pattern.replace(PLACEHOLDER, "x")): + raise ValueError("GAMMA_PAGE_HOST must be a hostname with {username} in it (no scheme, port or path), " + "e.g. {username}-pages.example.org.") + + +def page_host_user(pattern: str, host: str) -> str: + """The username a page hostname names under ``pattern`` ("" when + ``host`` — a Host header, port allowed — does not match it).""" + if not pattern or PLACEHOLDER not in pattern: + return "" + head, tail = pattern.split(PLACEHOLDER, 1) + host = re.sub(r":\d+$", "", (host or "").strip().lower()) + m = re.fullmatch(re.escape(head) + r"([a-z0-9](?:[a-z0-9-]*[a-z0-9])?)" + re.escape(tail), host) + return m.group(1) if m and _valid_host(host) else "" + + +def public_url(base_url: str, pattern: str, username: str, title: str, page_id: str) -> str: + """A published page's pretty address for the share host at ``base_url`` + (its scheme and port): ``:///-``, + just ``/`` without a slug. "" when there is no pattern or the + username cannot be a hostname label.""" + host = pattern.replace(PLACEHOLDER, (username or "").lower()) if pattern and username else "" + if not host or not _valid_host(host): + return "" + parts = urlsplit(base_url) + port = f":{parts.port}" if parts.port else "" + name = slug(title) + return f"{parts.scheme or 'https'}://{host}{port}/{name + '-' if name else ''}{page_id}" + + +def resolve_public(host: str, path: str) -> dict: + """``{share, page_id}`` for a page host's path: ``host`` names the + account (``page_host_user``), ``path`` is ``/-`` or ``/``, + and only the trailing id counts — a root page of that account's default + personal workspace that has a share. PublishError(404) otherwise. A page + id may hold a ``-`` itself, so every tail of the path after a ``-`` is a + candidate, the longest shared page winning.""" + missing = PublishError(404, "page not found") + username = page_host_user(config.page_host_pattern(), host) + segment = unquote(path or "").strip("/") + if not username or not segment or "/" in segment or len(segment) > 200: + raise missing + with connect_users_db() as conn: + rows = conn.execute("SELECT default_workspace FROM users WHERE LOWER(username) = ? AND is_guest = 0", + (username,)).fetchall() + ws = rows[0][0] if len(rows) == 1 else "" + if not ws or (workspaces.get(ws) or {}).get("kind") != "personal": + raise missing + candidates = [segment] + [segment[i + 1:] for i, c in enumerate(segment) if c == "-" and segment[i + 1:]] + marks = ",".join("?" * len(candidates)) + with connect_users_db() as conn: + shares = dict(conn.execute(f"SELECT page_id, token FROM shares WHERE workspace_id = ? AND page_id IN ({marks})", + (ws, *candidates)).fetchall()) + if not shares: + raise missing + with connect_pages_db(ws) as conn: + roots = {r[0] for r in conn.execute( + f"SELECT id FROM unified_blocks WHERE parent_id = 'root' AND id IN ({marks})", candidates)} + for page_id in candidates: + if page_id in roots and page_id in shares: + return {"share": shares[page_id], "page_id": page_id} + raise missing + + +# --- the plan's page cap (the share host) ----------------------------------------- + +def cap_message(plan: str, limit: int) -> str: + return (f"{(plan or 'Your').capitalize()} plan: up to {limit} published pages. " + "Unpublish one, or upgrade your Gamma Cloud plan.") + + +def page_cap(ws: str) -> dict: + """``{used, max, plan}`` of a workspace here: its root pages, and the + cap its owner's plan (the linked identity's last ``plan`` claim, stored + at every exchange and sign-in) puts on it. ``max`` is None when there is + none: on a server that is not a share host, for a workspace that is not + its owner's default personal one (the one the exchange publishes into), + for an owner without a cloud identity, and for a plan + ``config.PLAN_PAGE_LIMITS`` does not name.""" + owner = workspaces.personal_owner(ws) + plan = str((cloud_auth.status_of(owner) or {}).get("plan") or "") if owner else "" + cap = None + if owner and this_is_share_host() and workspaces.default_workspace(owner) == ws: + cap = config.plan_page_limits().get(plan) + with connect_pages_db(ws) as conn: + used = conn.execute("SELECT COUNT(*) FROM unified_blocks WHERE parent_id = 'root'").fetchone()[0] + return {"used": used, "max": cap, "plan": plan} + + +def cap_refusal(ws: str) -> dict | None: + """The 402 body (``{detail, limit, used, plan}``) when one more root page + in ``ws`` would pass its plan's cap, else None. Reads nothing more unless + this server is a share host.""" + if not this_is_share_host(): + return None + cap = page_cap(ws) + if cap["max"] is None or cap["used"] < cap["max"]: + return None + return {"detail": cap_message(cap["plan"], cap["max"]), "limit": cap["max"], "used": cap["used"], + "plan": cap["plan"]} + + # --- the share host --------------------------------------------------------------- def token_name(caller: str) -> str: @@ -220,6 +368,76 @@ def _json(remote: Remote, method: str, path: str, payload: dict | None = None) - return json.loads(data) if data else {} +def _read_limit(remote: Remote) -> dict | None: + """``{used, max, plan}`` of the person's workspace on the share host + (``GET /api/publish/limit`` under the mirror's token), None when it + cannot be read. Nothing is cached.""" + try: + out = _json(remote, "GET", "/api/publish/limit") + except (RemoteError, ValueError): + return None + if not isinstance(out, dict) or not isinstance(out.get("used"), int): + return None + cap = out.get("max") + return {"used": out["used"], "max": cap if isinstance(cap, int) else None, "plan": str(out.get("plan") or "")} + + +def _page_host(host: str) -> str: + """The share host's page-host pattern from its ``/api/server-config`` + ("" when it has none or cannot be read).""" + try: + cfg = Remote(host, "", "").get("/api/server-config") or {} + except (RemoteError, ValueError): + return "" + return str(cfg.get("page_host") or "").strip().lower() if isinstance(cfg, dict) else "" + + +def _title(ws: str, page_id: str) -> str: + with connect_pages_db(ws) as conn: + row = conn.execute("SELECT content FROM unified_blocks WHERE id = ?", (page_id,)).fetchone() + return row[0] if row else "" + + +def _addresses(mirror: dict, ws: str, page_id: str, token: str) -> dict: + """The answer's ``url`` (the token link, which always works) and + ``public_url`` (the pretty address when the share host has page hosts, + else the token link).""" + url = f"{mirror['remote_url']}/?share={token}" + pretty = public_url(mirror["remote_url"], _page_host(mirror["remote_url"]), + mirror["status"].get("remote_user") or "", _title(ws, page_id), page_id) + return {"url": url, "public_url": pretty or url} + + +def _cap_error(limit: dict, detail: str = "") -> PublishError: + return PublishError(409, detail or cap_message(limit["plan"], limit["max"]), {"limit": limit}) + + +def _check_cap(user: str, ws: str, page_id: str, host: str, server_name: str) -> None: + """Before a page goes to the share host for the first time: refuse it + (409, the cap message, ``limit``) when the person's workspace there is + full, taking it back out of the filter. The share host's plan is the + one its last exchange saw, so a full workspace is exchanged once more + first: an upgrade counts at once.""" + mirror = sync_engine.get_mirror(ws, with_token=True) + if _synced(ws, page_id) or mirror["page_filter"] is None: + return + limit = _read_limit(_remote(mirror)) + if limit and limit["max"] is not None and limit["used"] >= limit["max"]: + try: + _exchange(user, host, server_name) # hands every publishing mirror of the account the new token + limit = _read_limit(_remote(sync_engine.get_mirror(ws, with_token=True))) or limit + except PublishError: + pass + if limit and limit["max"] is not None and limit["used"] >= limit["max"]: + _drop(ws, page_id) + raise _cap_error(limit) + + +def _drop(ws: str, page_id: str) -> None: + with sync_engine.round_lock(ws): + sync_engine.filter_remove(ws, [page_id]) + + def _synced(ws: str, page_id: str) -> bool: with connect_pages_db(ws) as conn: return bool(conn.execute("SELECT 1 FROM sync_pages WHERE page_id = ?", (page_id,)).fetchone()) @@ -228,6 +446,7 @@ def _synced(ws: str, page_id: str) -> bool: def _mirror_view(mirror: dict) -> dict: ws = mirror["workspace_id"] return {"ws": ws, "status": mirror["status"], "page_filter": mirror["page_filter"], + "mode": mirror["mode"], "detached": mirror["mode"] == "off", "conflicts_open": sync_engine.open_conflicts(ws), "pending_local": mirror["mode"] == "two-way" and sync_engine.has_local_changes(ws)} @@ -237,7 +456,8 @@ def publish(user: str, ws: str, page_id: str, *, audience: str | None = None, ro """Publish ``page_id``: into the filtered mirror (made when missing), one round now, then the share on the share host (default anyone / view; ``audience`` / ``role`` set it, on a new link or an existing one). - ``{url, share, mirror: {ws, status, ...}}``.""" + ``{url, public_url, share, mirror: {ws, status, ...}}``. A page new to + the share host must fit the person's plan there (``_check_cap``).""" if audience is not None and audience not in SHARE_AUDIENCES: raise PublishError(400, "audience must be anyone, users or list") if role is not None and role not in SHARE_ROLES: @@ -249,9 +469,19 @@ def publish(user: str, ws: str, page_id: str, *, audience: str | None = None, ro raise PublishError(409, SIGN_IN) host = share_host() _mirror_for(user, ws, page_id, host, server_name) + _check_cap(user, ws, page_id, host, server_name) status = sync_engine.sync_workspace(ws) if not _synced(ws, page_id): - raise PublishError(502, f"The page did not reach the share host: {status.get('last_error') or 'try again'}") + error = status.get("last_error") or "" + if error.startswith(f"{page_id}: 402: "): + # the cap reached between the check and the round: the share host's own words + _drop(ws, page_id) + mirror = sync_engine.get_mirror(ws, with_token=True) + if mirror["page_filter"]: + sync_engine.sync_workspace(ws) # a clean round, so the refusal does not stay the mirror's error + limit = _read_limit(_remote(mirror)) or {"used": 0, "max": None, "plan": ""} + raise _cap_error(limit, error[len(f"{page_id}: 402: "):]) + raise PublishError(502, f"The page did not reach the share host: {error or 'try again'}") mirror = sync_engine.get_mirror(ws, with_token=True) remote = _remote(mirror) wanted = {k: v for k, v in (("audience", audience), ("role", role)) if v is not None} @@ -261,7 +491,7 @@ def publish(user: str, ws: str, page_id: str, *, audience: str | None = None, ro share = _json(remote, "PUT", f"/api/share-settings/{page_id}", wanted) except RemoteError as e: raise PublishError(502, f"The share host refused the share: {e.detail or e}") from e - return {"url": f"{mirror['remote_url']}/?share={share['token']}", "share": share, "mirror": _mirror_view(mirror)} + return {**_addresses(mirror, ws, page_id, share["token"]), "share": share, "mirror": _mirror_view(mirror)} def unpublish(user: str, ws: str, page_id: str) -> dict: @@ -288,11 +518,13 @@ def unpublish(user: str, ws: str, page_id: str) -> dict: def state(user: str, ws: str, page_id: str) -> dict: - """``{published, can_publish, reason?, url?, share?, status?, mirror?, - error?}``: whether the page is published (in the filter of the - workspace's mirror of the share host), its live share there, and the - mirror's raw status. ``can_publish`` / ``reason`` say whether the - Publish action would be refused before it is tried.""" + """``{published, can_publish, reason?, url?, public_url?, share?, + status?, mirror?, limit?, error?}``: whether the page is published (in + the filter of the workspace's mirror of the share host), its live share + there and addresses, the mirror's raw status, and the person's page cap + there (``limit: {used, max, plan}``, read whenever a publishing token + exists). ``can_publish`` / ``reason`` say whether the Publish action + would be refused before it is tried.""" _require_page(ws, page_id) out: dict = {"published": False, "can_publish": True} reason, host = "", "" @@ -310,6 +542,12 @@ def state(user: str, ws: str, page_id: str) -> dict: reason = e.message if reason: out.update(can_publish=False, reason=reason) + else: + # the cap there, through this workspace's publishing token or another of the account's + lender = mirror if mirror and mirror["page_filter"] is not None else next(iter(_publish_mirrors(user, host)), None) + limit = _read_limit(_remote(lender)) if lender else None + if limit: + out["limit"] = limit # only publishing makes a filtered mirror; a full one counts when it follows the share host if not mirror or (mirror["page_filter"] is None and not (host and _same(mirror["remote_url"], host))): return out @@ -327,5 +565,5 @@ def state(user: str, ws: str, page_id: str) -> dict: if mirror["page_filter"] is not None: out["published"] = True if share.get("token"): - out.update(published=True, share=share, url=f"{mirror['remote_url']}/?share={share['token']}") + out.update(published=True, share=share, **_addresses(mirror, ws, page_id, share["token"])) return out diff --git a/backend/gamma/routers/ai.py b/backend/gamma/routers/ai.py index 442a5220..7b28d5d2 100644 --- a/backend/gamma/routers/ai.py +++ b/backend/gamma/routers/ai.py @@ -4,25 +4,23 @@ import json import queue import re -import secrets import sqlite3 import threading import time import urllib.error from collections import OrderedDict from concurrent.futures import ThreadPoolExecutor -from urllib.request import Request as URLRequest, urlopen +from urllib.request import urlopen from fastapi import APIRouter, File, Form, HTTPException, Request, UploadFile from fastapi.responses import StreamingResponse from pydantic import BaseModel, Field, field_validator -from .. import ai_usage, chatgpt_oauth +from .. import ai_catalog, ai_protocols, ai_usage, chatgpt_oauth, translate_engines from ..ai_client import ( UpstreamError, add_usage as _add_usage, call_ai as _call_ai, - is_openai_platform, open_ai as _open_ai, partial_json_object as _partial_json_object, partial_json_strings as _partial_json_strings, @@ -76,11 +74,11 @@ update_entry, ) from ..auth import require_user, require_ws, ws_role -from ..config import AI_PROTOCOLS from ..db import page_now, ws_db_path from ..logbuf import log from ..pdf_text import extract_text from ..textnorm import INDEX_VERSION +from ..translate_engines import TRANSLATE_LANGS # Note editors whose in-flight arguments the chat streams as "progress" # events: the notes panel types the markdown into the block as the model @@ -283,7 +281,6 @@ def pdf_text_status(doc_id: str, request: Request, preview: int = 0): ) - # Sync def: ai_runtime may refresh a ChatGPT token (a network round trip). @router.get("/ai/models") def ai_models(request: Request): @@ -293,6 +290,9 @@ def ai_models(request: Request): "enabled": rt["enabled"], "models": rt["models"], # [{id: ":", provider, provider_name, model}, ...] "default": rt["default"]["id"] if rt["default"] else "", + # Set-up machine-translation engines [{id: "engine:", label}] — + # the translation picker offers them next to the models. + "translate_engines": translate_engines.configured(user), "efforts": ["low", "medium", "high"], # offered in the UI; omitted unless picked "default_prompt": _SYSTEM_PROMPT, # shown in the prompt editor "metadata_prompt": METADATA_PROMPT, # AI metadata-extraction fallback @@ -344,7 +344,7 @@ def _require_editor(request: Request) -> str: class AIProviderRequest(BaseModel): - protocol: str = "" # an API-key key of AI_PROTOCOLS (required on add) + protocol: str = "" # an API-key key of ai_protocols.PROTOCOLS (required on add) name: str | None = None # display label; "" = service / protocol label api_key: str | None = None # required on add; omitted/empty on edit = keep base_url: str | None = None # "" = protocol default @@ -411,16 +411,12 @@ async def ai_provider_delete(provider_id: str, request: Request): return _masked_settings(request) -def _is_oauth_entry(entry: dict) -> bool: - return _is_oauth_protocol(entry.get("protocol")) - - def _no_credential(entry: dict) -> dict: """The in-body failure for an entry ``ai_runtime`` dropped: no key, or a ChatGPT sign-in whose refresh failed.""" return {"ok": False, "auth": True, "error": "ChatGPT sign-in expired or disconnected — sign in again" - if _is_oauth_entry(entry) + if _is_oauth_protocol(entry.get("protocol")) else "entry has no usable credential — set an API key or sign in again"} @@ -482,60 +478,10 @@ def ai_provider_test(provider_id: str, request: Request, payload: AIProviderTest return _probe_entry(user, entry, payload.model if payload else "") -def _usage_window(raw: dict | None, name: str = "") -> dict | None: - if not isinstance(raw, dict): - return None - try: - used = max(0.0, min(100.0, float(raw.get("used_percent", 0)))) - except (TypeError, ValueError): - return None - try: - seconds = max(0, int(raw.get("limit_window_seconds") or 0)) - except (TypeError, ValueError): - seconds = 0 - try: - reset_at = int(raw.get("reset_at") or 0) - except (TypeError, ValueError): - reset_at = 0 - if not name: - if 4 * 3600 <= seconds <= 6 * 3600: - name = "5-hour" - elif 6 * 86400 <= seconds <= 8 * 86400: - name = "Weekly" - elif seconds: - name = f"{max(1, round(seconds / 3600))}-hour" - else: - name = "Usage" - return { - "name": name, - "used_percent": used, - "remaining_percent": max(0.0, 100.0 - used), - "window_seconds": seconds, - "reset_at": reset_at, - } - - -def _chatgpt_usage_request(conf: dict) -> URLRequest: - """The ChatGPT subscription-usage request (Codex's account client's - .../backend-api/wham/usage, sibling of the .../codex model endpoint). - Always the administrator-controlled protocol endpoint, never a saved - entry value: OAuth entries cannot redirect their bearer token.""" - base = str(AI_PROTOCOLS["chatgpt"]["base_url"]).rstrip("/") - account_base = base[:-len("/codex")] if base.endswith("/codex") else base - headers = { - "Authorization": f"Bearer {conf['api_key']}", - "Accept": "application/json", - "User-Agent": "codex-cli", - } - if conf.get("account_id"): - headers["ChatGPT-Account-Id"] = conf["account_id"] - return URLRequest(f"{account_base}/wham/usage", headers=headers, method="GET") - - # Sync def: this read-only account call runs in FastAPI's threadpool. @router.post("/ai/providers/{provider_id}/usage") def ai_provider_usage(provider_id: str, request: Request): - """Return subscription allowance for a ChatGPT OAuth provider. + """The subscription allowance of a sign-in entry (Protocol.account_usage). API-key protocols have no portable quota endpoint: OpenAI-compatible gateways and Anthropic-style services all expose different billing/admin @@ -545,147 +491,45 @@ def ai_provider_usage(provider_id: str, request: Request): entry = next((e for e in load_provider_entries(user) if e.get("id") == provider_id), None) if not entry: raise HTTPException(status_code=404, detail="provider not found") - if entry.get("protocol") != "chatgpt": + proto = ai_protocols.PROTOCOLS.get(entry.get("protocol")) + if not proto or not proto.has_account_usage: return {"available": False, "reason": "This API-key provider does not expose a standard remaining-usage percentage."} clear_refresh_backoff(user, provider_id) - rt = ai_runtime(user) - conf = rt["providers"].get(provider_id) + conf = ai_runtime(user)["providers"].get(provider_id) if not conf: - return {"available": False, "auth": True, - "reason": "Sign in with ChatGPT again to query usage."} + return {"available": False, "auth": True, "reason": "Sign in again to query usage."} try: - data = _model_catalog_json(_chatgpt_usage_request(conf)) + data = ai_catalog.fetch_json(proto.account_usage_request(conf)) except urllib.error.HTTPError as e: if e.code in (401, 403): # Expired/revoked sign-in: an expected state, not a server error — # report it in-body so the UI can say "reconnect". return {"available": False, "auth": True, - "reason": "ChatGPT sign-in expired — sign in again in this entry's edit form."} + "reason": "The sign-in expired — sign in again in this entry's edit form."} raise HTTPException(status_code=502, detail=f"usage inquiry failed: {_upstream_detail(e, 200)}") except Exception as e: raise HTTPException(status_code=502, detail=f"usage inquiry failed: {e}") if not isinstance(data, dict): raise HTTPException(status_code=502, detail="usage inquiry returned invalid data") + usage = proto.account_usage(data) + return {"available": bool(usage["windows"]), **usage, + "reason": "" if usage["windows"] else "The provider returned no usage windows."} - rate = data.get("rate_limit") - rate = rate if isinstance(rate, dict) else {} - windows = [w for w in ( - _usage_window(rate.get("primary_window")), - _usage_window(rate.get("secondary_window")), - ) if w] - for extra in data.get("additional_rate_limits") or []: - if not isinstance(extra, dict): - continue - extra_rate = extra.get("rate_limit") if isinstance(extra.get("rate_limit"), dict) else {} - window = _usage_window(extra_rate.get("primary_window"), - str(extra.get("limit_name") or "Additional limit")) - if window: - windows.append(window) - return { - "available": bool(windows), - "plan_type": str(data.get("plan_type") or ""), - "windows": windows, - "credits": data.get("credits") if isinstance(data.get("credits"), dict) else None, - "reason": "" if windows else "The provider returned no usage windows.", - } - -# GET {base}/models gates its answer on the caller's version, so the listing -# claims the newest Codex CLI release (npm's `latest` tag), looked up live and -# cached. The floor is only for when npm can't be reached. -_CODEX_VERSION_URL = "https://registry.npmjs.org/@openai/codex/latest" -_CODEX_VERSION_FLOOR = "0.156.1" -_CODEX_VERSION_TTL = 6 * 3600 # a good answer -_CODEX_VERSION_RETRY = 600 # after a failed lookup -_codex_version = {"value": "", "until": 0.0} -_codex_version_lock = threading.Lock() -_MODEL_CATALOG_TIMEOUT = 5 - - -def _codex_client_version() -> str: - """The newest Codex CLI version, cached; the last good one (else the - floor) while npm is unreachable.""" - with _codex_version_lock: - now = time.time() - if now < _codex_version["until"]: - return _codex_version["value"] or _CODEX_VERSION_FLOOR - try: - with urlopen(URLRequest(_CODEX_VERSION_URL, headers={"Accept": "application/json"}), - timeout=_MODEL_CATALOG_TIMEOUT) as resp: - version = str(json.loads(resp.read()).get("version") or "").strip() - if not re.fullmatch(r"\d+\.\d+\.\d+", version): - raise ValueError(f"unexpected version {version!r}") - _codex_version.update(value=version, until=now + _CODEX_VERSION_TTL) - except Exception as e: - log.warning(f"[ai] codex version lookup failed, using " - f"{_codex_version['value'] or _CODEX_VERSION_FLOOR}: {e}") - _codex_version["until"] = now + _CODEX_VERSION_RETRY - return _codex_version["value"] or _CODEX_VERSION_FLOOR - - -def _model_catalog_json(req: URLRequest) -> dict: - """Fetch one model catalog with a short, UI-friendly timeout.""" - with urlopen(req, timeout=_MODEL_CATALOG_TIMEOUT) as resp: - return json.loads(resp.read()) - - -def _models_list_request(protocol: str, key: str, base: str) -> URLRequest: - """GET /v1/models for an API-key protocol — the free way to check a - credential (it 401s on a dead key without spending tokens).""" - if protocol == "anthropic": - return URLRequest(f"{base}/v1/models?limit=100", - headers={ - "x-api-key": key, - "anthropic-version": "2023-06-01", - "Accept": "application/json", - "User-Agent": "Gamma/model-catalog", - }) - return URLRequest(f"{base}/v1/models", headers={ - "Authorization": f"Bearer {key}", - "Accept": "application/json", - "User-Agent": "Gamma/model-catalog", - }) - - -def _chatgpt_model_catalog(user: str, provider_id: str = "") -> list: - """Live model list from the ChatGPT (codex) backend, Codex CLI's own - listing call: GET {base}/models?client_version=… with the OAuth bearer. - Needs a connected entry — the list is account-gated, so there is no - hardcoded fallback: a failed fetch is an error.""" +def _signed_in_conf(user: str, provider_id: str, proto) -> dict: + """A sign-in protocol's model list is account-gated: the named connected + entry's, else (the pre-connect "Add" form has no entry yet) any + connected entry's of that protocol. There is no fallback list.""" providers = ai_runtime(user)["providers"] conf = providers.get(provider_id) - if not conf or conf.get("protocol") != "chatgpt": - # Pre-connect "Add key" form has no entry yet — any connected one will do. - conf = next((c for c in providers.values() if c.get("protocol") == "chatgpt"), None) + if not conf or conf.get("protocol") != proto.id: + conf = next((c for c in providers.values() if c.get("protocol") == proto.id), None) if not conf: - raise HTTPException(status_code=400, - detail="sign in with ChatGPT first — the model list comes from your account") - try: - req = URLRequest( - f"{conf['base_url']}/models?client_version={_codex_client_version()}", - headers={ - "Authorization": f"Bearer {conf['api_key']}", - "chatgpt-account-id": conf.get("account_id", ""), - "originator": "codex_cli_rs", - }) - data = _model_catalog_json(req) - listed, hidden = [], [] - for m in data.get("models") or []: - slug = str(m.get("slug") or "").strip() - vis = m.get("visibility") or "list" - if not slug or vis == "none": # "none" = not usable by this account - continue - (hidden if vis == "hide" else listed).append(slug) - # `hide` marks picker-hidden but usable slugs — offer them after the - # listed ones rather than dropping them. - return list(dict.fromkeys(listed + hidden)) - except urllib.error.HTTPError as e: - raise HTTPException(status_code=502, detail=f"model list failed: {_upstream_detail(e, 200)}") - except Exception as e: - raise HTTPException(status_code=502, detail=f"model list failed: {e}") + raise HTTPException(status_code=400, detail="sign in first — the model list comes from your account") + return conf class ModelCatalogRequest(BaseModel): @@ -695,45 +539,58 @@ class ModelCatalogRequest(BaseModel): base_url: str | None = None -# Sync def: the upstream /v1/models fetch runs in the threadpool. +# Sync def: the upstream listing fetch runs in the threadpool. @router.post("/ai/model-catalog") def ai_model_catalog(payload: ModelCatalogRequest, request: Request): - """Model names offered by a provider, for the settings form's model picker. - API protocols are asked live (GET /v1/models with the entry's key); the - ChatGPT backend is asked via Codex CLI's listing call with the OAuth - token (an error until an entry is connected). An admin editing a shared - entry names it by its ``server:``.""" + """Model names offered by a provider, for the settings form's model + picker, asked live (ai_catalog.list_models): an API-key protocol with the + typed key or the saved entry's, a sign-in protocol with a connected + entry's token. An admin editing a shared entry names it by its + ``server:``.""" user = _require_editor(request) entry = {} protocol = payload.protocol if payload.provider_id: entry = _saved_entry(request, user, payload.provider_id) or {} protocol = protocol or entry.get("protocol") - if protocol == "chatgpt": - return {"models": _chatgpt_model_catalog(user, payload.provider_id)} - if protocol not in AI_PROTOCOLS: + proto = ai_protocols.PROTOCOLS.get(protocol) + if not proto: raise HTTPException(status_code=400, detail="unknown protocol") - key = (payload.api_key or "").strip() or (entry.get("api_key") or "").strip() - if not key: - raise HTTPException(status_code=400, detail="enter the API key first, then load the model list") - base = ((payload.base_url if payload.base_url is not None else entry.get("base_url") or "").strip() - or AI_PROTOCOLS[protocol]["base_url"]).rstrip("/") + if proto.auth == "oauth": + conf = _signed_in_conf(user, payload.provider_id, proto) + else: + key = (payload.api_key or "").strip() or (entry.get("api_key") or "").strip() + if not key: + raise HTTPException(status_code=400, detail="enter the API key first, then load the model list") + base = ((payload.base_url if payload.base_url is not None else entry.get("base_url") or "").strip() + or proto.base_url).rstrip("/") + conf = {"protocol": proto.id, "api_key": key, "base_url": base, "name": ""} + # A key the provider refuses is the form's problem (400); a sign-in's + # listing failing is the upstream's (502). + status = 502 if proto.auth == "oauth" else 400 try: - data = _model_catalog_json(_models_list_request(protocol, key, base)) + models = ai_catalog.list_models(conf) except urllib.error.HTTPError as e: - raise HTTPException(status_code=400, detail=f"model list failed: {_upstream_detail(e, 200)}") + raise HTTPException(status_code=status, detail=f"model list failed: {_upstream_detail(e, 200)}") except Exception as e: - raise HTTPException(status_code=400, detail=f"model list failed: {e}") - ids = [str(m.get("id") or "") for m in (data.get("data") or []) if m.get("id")] - if protocol == "openai": - # Listings include embeddings/audio/image models the chat endpoint - # can't use — drop them. OpenAI's own listing is additionally narrowed - # to its conversational families; a compatible server (DeepSeek, …) - # names its models however it likes. - ids = [i for i in ids - if not re.search(r"embed|whisper|tts|audio|image|dall-e|moderation|transcribe|realtime|search", i) - and (not is_openai_platform(base) or re.match(r"^(gpt-|o\d|chatgpt-)", i))] - return {"models": sorted(set(ids))} + raise HTTPException(status_code=status, detail=f"model list failed: {e}") + return {"models": [m["id"] for m in models]} + + +# Sync def: the listing / catalog fetches run in the threadpool. +@router.get("/ai/context-window") +def ai_context_window(request: Request, model: str = ""): + """The context window of a chat model (":"; "" = the + default one), for the chat's context ring: {model, context_window, + source: "provider" | "models.dev"} — ai_catalog.context_window; + context_window null when neither source knows the model.""" + rt = ai_runtime(require_user(request)) + m = next((x for x in rt["models"] if x["id"] == model), None) or rt["default"] + conf = rt["providers"].get(m["provider"]) if m else None + if not conf: + return {"model": "", "context_window": None, "source": ""} + window, source = ai_catalog.context_window(m["provider"], conf, m["model"]) + return {"model": m["model"], "context_window": window or None, "source": source} class AIHealthRequest(BaseModel): @@ -747,8 +604,8 @@ class AIHealthRequest(BaseModel): def ai_health(payload: AIHealthRequest, request: Request): """Startup connection check for one provider entry, so a broken credential surfaces at login instead of as a failed chat later. "ping" spends no - tokens: OAuth entries ask the subscription usage endpoint, API keys list - /v1/models — both 401 on a dead credential. "test" runs the same tiny + tokens (Protocol.ping_request: API keys list the models, sign-ins ask the + quota endpoint) — it 401s on a dead credential. "test" runs the same tiny completion as the Test button (through the entry's test model). Always answers in-body: {configured, ok, auth?, error?, ...}. The entries are the ones the account can use: its own, then the server's shared ones.""" @@ -767,10 +624,7 @@ def ai_health(payload: AIHealthRequest, request: Request): if not conf: return {**result, **_no_credential(entry)} try: - if conf["protocol"] == "chatgpt": - _model_catalog_json(_chatgpt_usage_request(conf)) - else: - _model_catalog_json(_models_list_request(conf["protocol"], conf["api_key"], conf["base_url"])) + ai_catalog.fetch_json(ai_protocols.of(conf).ping_request(conf)) except urllib.error.HTTPError as e: if e.code in (401, 403): return {**result, "ok": False, "auth": True, "error": _upstream_detail(e, 200)} @@ -859,13 +713,8 @@ def pump(): # persisted to disk, the cache just makes retries, re-shows and halted-job # resumes free until the server restarts. -# Allowlisted target languages (code → name spliced into the prompt). Mirrored -# in frontend/src/app/prefs.js TRANSLATE_LANGS — keep the two in sync. -TRANSLATE_LANGS = { - "en": "English", "zh-CN": "Simplified Chinese", "zh-TW": "Traditional Chinese", - "ja": "Japanese", "ko": "Korean", "de": "German", "fr": "French", - "es": "Spanish", "pt": "Portuguese", "it": "Italian", "ru": "Russian", -} +# The allowlisted target languages are translate_engines.TRANSLATE_LANGS +# (code → the name spliced into the prompt), shared with the engines. _TRANSLATE_PROMPT = ( "You translate paragraphs extracted from an academic paper into {lang}. " @@ -913,7 +762,7 @@ def _cache_put(key: str, text: str): class AITranslateRequest(BaseModel): texts: list = Field(default_factory=list) # source paragraphs, viewer order lang: str = "zh-CN" # target language code (TRANSLATE_LANGS key) - model: str = "" # model registry id; "" = the user's default + model: str = "" # model registry id, or "engine:" (a translation service); "" = the user's default effort: str = "" # reasoning effort; "" = provider default (param omitted) # NDJSON stream: {"i": [indices], "text": partial} lines as the model # writes each paragraph (the viewer types them into the page), then the @@ -957,9 +806,16 @@ def ai_translate(payload: AITranslateRequest, request: Request): if sum(len(t) for t in texts) > _TRANSLATE_MAX_CHARS: raise HTTPException(status_code=413, detail="too much text in one request") - rt = require_ai_runtime(user) - entry = _resolve_model(rt, payload.model) - model_name = entry["model"] + # A machine-translation engine ("engine:", Settings → Reading) needs + # no AI provider; anything else resolves to a chat model. + engine = translate_engines.engine_of(payload.model) + if engine: + engine_conf = translate_engines.credentials(user, engine) + model_id = model_name = translate_engines.MODEL_PREFIX + engine + else: + rt = require_ai_runtime(user) + entry = _resolve_model(rt, payload.model) + model_id, model_name = entry["id"], entry["model"] keys = [_translate_key(user, lang, model_name, t) for t in texts] # hits: key → translation, for every paragraph that won't need the model. @@ -968,22 +824,47 @@ def ai_translate(payload: AITranslateRequest, request: Request): # verbatim. hits = _cache_get(keys) - # Whitespace-only paragraphs never reach the model; every other cache miss - # goes upstream in ONE call (duplicates collapsed), as a JSON array both ways. + # Whitespace-only paragraphs never go upstream; every other cache miss + # does, once (duplicates collapsed): one model call, or a service's + # batches. miss, queued = [], set() for i, t in enumerate(texts): if keys[i] not in hits and keys[i] not in queued and t.strip(): queued.add(keys[i]) miss.append(i) - if not miss: - out = [hits.get(k, texts[i]) for i, k in enumerate(keys)] - final = {"translations": out, "model": entry["id"], "cached": True} + + def reply_with(final): + # The whole answer at once; a streaming client reads it as the + # final NDJSON line. if not payload.stream: return final - return StreamingResponse(iter([json.dumps(final) + "\n"]), + return StreamingResponse(iter([json.dumps(final, ensure_ascii=False) + "\n"]), media_type="application/x-ndjson") + def settle(translated, cached): + """Record the misses' translations (cache + hits) and build the final + response object.""" + for i, t in zip(miss, translated): + hits[keys[i]] = t + if t and t != texts[i]: # identity fallbacks stay uncached so a retry can improve them + _cache_put(keys[i], t) + out = [hits.get(k, texts[i]) for i, k in enumerate(keys)] + return {"translations": out, "model": model_id, "cached": cached} + + if not miss: + return reply_with(settle([], True)) + miss_texts = [texts[i] for i in miss] + if engine: + # One engine call per batch limit, no streaming: the reply is aligned + # by the API, so there is nothing to salvage either. + try: + translated = translate_engines.translate(engine, engine_conf, miss_texts, lang, user) + except translate_engines.EngineError as e: + log.warning(f"[ai_translate] {e}") + raise HTTPException(status_code=502, detail=f"translation failed: {e}") + return reply_with(settle(translated, False)) + system = _TRANSLATE_PROMPT.format(lang=TRANSLATE_LANGS[lang]) effort = _resolve_effort(payload.effort) @@ -1040,12 +921,7 @@ def salvage(t): with ThreadPoolExecutor(max_workers=min(4, len(miss_texts))) as pool: translated = list(pool.map(salvage, miss_texts)) - for i, t in zip(miss, translated): - hits[keys[i]] = t - if t and t != texts[i]: # identity fallbacks stay uncached so a retry can improve them - _cache_put(keys[i], t) - out = [hits.get(k, texts[i]) for i, k in enumerate(keys)] - return {"translations": out, "model": entry["id"], "cached": False} + return settle(translated, False) if not payload.stream: try: @@ -1090,6 +966,55 @@ def ndjson(): media_type="application/x-ndjson") +# --- Machine-translation engine credentials (Settings → Reading) -------------- +# Write-only like the AI keys: GET masks the secrets. Guests can't store keys +# (the guest account is shared by every visitor). + +class TranslateEngineRequest(BaseModel): + fields: dict = Field(default_factory=dict) # {field id: value}; empty secret = keep + + +class TranslateEngineTestRequest(BaseModel): + lang: str = "zh-CN" + + +@router.get("/translate/engines") +def translate_engines_get(request: Request): + user = require_user(request) + return translate_engines.masked(user, can_edit=not request.state.is_guest) + + +@router.put("/translate/engines/{engine}") +def translate_engine_save(engine: str, payload: TranslateEngineRequest, request: Request): + user = _require_editor(request) + translate_engines.save(user, engine, payload.fields) + return translate_engines.masked(user, can_edit=True) + + +@router.delete("/translate/engines/{engine}") +def translate_engine_remove(engine: str, request: Request): + user = _require_editor(request) + translate_engines.remove(user, engine) + return translate_engines.masked(user, can_edit=True) + + +# Sync def: the engine call runs in the threadpool. +@router.post("/translate/engines/{engine}/test") +def translate_engine_test(engine: str, payload: TranslateEngineTestRequest, request: Request): + """Translate one short sentence with the stored credentials: {ok, text} + or {ok: false, error} (in the body, like the AI provider test).""" + user = _require_editor(request) + conf = translate_engines.credentials(user, engine) + lang = payload.lang if payload.lang in TRANSLATE_LANGS else "zh-CN" + sample = ("Le vif renard brun saute par-dessus le chien paresseux." if lang == "en" + else "The quick brown fox jumps over the lazy dog.") + try: + text = translate_engines.translate(engine, conf, [sample], lang, user)[0] + except translate_engines.EngineError as e: + return {"ok": False, "error": str(e)} + return {"ok": True, "text": text} + + # --- Voice dictation ---------------------------------------------------------- # Default = ChatGPT's dictation model (user-overridable per request); whisper-1 @@ -1100,20 +1025,6 @@ def ndjson(): _TRANSCRIBE_MAX_BYTES = 25 * 1024 * 1024 # OpenAI's audio upload limit -def _multipart_body(fields: dict, filename: str, content_type: str, data: bytes): - """Encode fields + one file as multipart/form-data (urllib has no helper).""" - boundary = secrets.token_hex(16) - parts = [] - for name, value in fields.items(): - parts.append(f'--{boundary}\r\nContent-Disposition: form-data; name="{name}"\r\n\r\n{value}\r\n'.encode()) - parts.append( - f'--{boundary}\r\nContent-Disposition: form-data; name="file"; filename="{filename}"\r\n' - f"Content-Type: {content_type}\r\n\r\n".encode() + data + b"\r\n" - ) - parts.append(f"--{boundary}--\r\n".encode()) - return b"".join(parts), f"multipart/form-data; boundary={boundary}" - - # Sync def: the provider upload runs in the threadpool. @router.post("/ai/transcribe") def ai_transcribe(request: Request, file: UploadFile = File(...), @@ -1127,17 +1038,16 @@ def ai_transcribe(request: Request, file: UploadFile = File(...), # Needs an OpenAI credential, not chat models: an entry with none picked # still transcribes. rt = ai_runtime(user) - # Only openai-protocol entries can transcribe, and of those OpenAI itself - # surely can while a compatible server (DeepSeek) may not: the chat's own - # entry if it is OpenAI, else any OpenAI entry, else the chat's compatible - # entry, else any compatible one. + # The entry that surely transcribes (Protocol.transcription: OpenAI + # itself) before one that may (a compatible server), the chat's own entry + # first within each. # (A shared entry's ids have a colon of their own: "server::".) hinted = rt["providers"].get(next((m["provider"] for m in rt["models"] if m["id"] == model_hint), (model_hint or "").split(":", 1)[0])) - speakers = [c for c in ([hinted] if hinted else []) + list(rt["providers"].values()) - if c["protocol"] == "openai"] - conf = next((c for c in speakers if is_openai_platform(c["base_url"])), - speakers[0] if speakers else None) + candidates = [(ai_protocols.of(c).transcription(c), c) + for c in ([hinted] if hinted else []) + list(rt["providers"].values())] + best = max((rank for rank, _ in candidates), default=0) + conf = next((c for rank, c in candidates if rank == best), None) if best else None if not conf: raise HTTPException(status_code=503, detail="Voice input needs an OpenAI API key (Settings → AI → Connections) — " @@ -1158,18 +1068,14 @@ def ai_transcribe(request: Request, file: UploadFile = File(...), if _TRANSCRIBE_FALLBACK not in candidates: candidates.append(_TRANSCRIBE_FALLBACK) detail = "" + proto = ai_protocols.of(conf) for model in candidates: - fields = {"model": model, **({"language": language} if language else {})} - body, content_type = _multipart_body( - fields, filename, file.content_type or "application/octet-stream", audio) - req = URLRequest(f"{conf['base_url']}/v1/audio/transcriptions", data=body, headers={ - "Authorization": f"Bearer {conf['api_key']}", - "Content-Type": content_type, - }) + req = proto.transcription_request(conf, model, language, filename, + file.content_type or "application/octet-stream", audio) try: with urlopen(req, timeout=120) as resp: data = json.loads(resp.read()) - return {"text": (data.get("text") or "").strip(), "model": model} + return {"text": proto.transcript(data), "model": model} except urllib.error.HTTPError as error: detail = _upstream_detail(error) log.warning(f"[transcribe] {model}: {detail}") @@ -1262,7 +1168,7 @@ def chatgpt_auth_complete(payload: ChatGPTAuthComplete, request: Request): # back through ai_runtime. save_provider_entries(user, entries) try: - live = _chatgpt_model_catalog(user, entry["id"]) + live = [m["id"] for m in ai_catalog.list_models(ai_runtime(user)["providers"][entry["id"]])] except Exception: live = [] entry["models"] = ", ".join(live[:2])[:MAX_MODELS_LEN] @@ -1270,8 +1176,6 @@ def chatgpt_auth_complete(payload: ChatGPTAuthComplete, request: Request): return _masked_settings(request) - - # Providers whose backend refused native input_file parts — skip the wasted # upload on later requests. In-memory: a restart retries native once. _NATIVE_PDF_REJECTED: set = set() diff --git a/backend/gamma/routers/cloud_auth.py b/backend/gamma/routers/cloud_auth.py index 670a5350..1fa73cec 100644 --- a/backend/gamma/routers/cloud_auth.py +++ b/backend/gamma/routers/cloud_auth.py @@ -1,7 +1,9 @@ """Sign in with Gamma Cloud — the wire around ``gamma/cloud_auth.py``: - ``GET /api/server-config`` (public): what the login page needs — whether - cloud sign-in is on and the account server's address; + cloud sign-in is on and the account server's address — and ``page_host``, + the per-account page hostname pattern (``GAMMA_PAGE_HOST``, "" = none), + by which the app knows it was opened on a page host (gamma/publish.py); - ``GET /api/auth/cloud/start?next=&link=1`` → redirect to the account server (``link=1`` with a session attaches the identity to that account); - ``GET /api/auth/cloud/callback?code=&state=`` → session cookie + redirect @@ -13,14 +15,18 @@ this server off the person's server list and revokes the grant. - ``GET /api/auth/cloud/sync-status``: the signed-in account's own preference profile sync state (Settings' section tags), from memory. +- ``POST /api/auth/cloud/sync``: Settings → Account's Sync now / Fetch + from cloud / Push to cloud, and the answer to a first sync's choice. """ +from typing import Literal from urllib.parse import urlencode from fastapi import APIRouter, HTTPException, Request from fastapi.responses import RedirectResponse +from pydantic import BaseModel -from .. import cloud_auth, cloud_sync, ratelimit +from .. import cloud_auth, cloud_sync, config, ratelimit from ..auth import require_personal_user, require_user, set_session_cookie from ..cloud_auth import CloudAuthError from ..db import connect_users_db @@ -34,7 +40,8 @@ async def server_config(): cfg = cloud_auth.settings() return {"cloud": {"enabled": cfg["enabled"], "issuer": cfg["issuer"] if cfg["enabled"] else ""}, - "password_login": True, "registration": False, "guest": not cfg["share_host"]} + "password_login": True, "registration": False, "guest": not cfg["share_host"], + "page_host": config.page_host_pattern()} @router.get("/api/auth/cloud/start") @@ -98,6 +105,30 @@ def cloud_sync_status(request: Request): return {"profile": cloud_sync.profile_status(user), "identity": linked} +class SyncRequest(BaseModel): + action: Literal["sync", "merge", "fetch", "push"] = "sync" + defaults: dict = {} # "merge": the web app's default profile, the base of a first merge + + +@router.post("/api/auth/cloud/sync") +def cloud_sync_now(payload: SyncRequest, request: Request): + """Sync the caller's profile with Gamma Cloud now: "sync" merges as the + automatic sync does, "merge" / "fetch" / "push" also settle a first + sync's choice. Answers the outcome and the new sync state.""" + user = require_personal_user(request, "The guest account keeps its settings in the browser.") + if not cloud_sync.syncs(user): + raise HTTPException(400, "Link a Gamma Cloud account first.") + resolve = "" if payload.action == "sync" else payload.action + try: + outcome = cloud_sync.sync_profile(user, resolve=resolve, defaults=payload.defaults) + except cloud_sync.NothingToFetch: + raise HTTPException(409, "Gamma Cloud holds no settings yet.") + status = cloud_sync.profile_status(user) + if not outcome: + raise HTTPException(502, status.get("error") or cloud_sync.UNREACHABLE) + return {"outcome": outcome, "profile": status} + + @router.post("/api/auth/cloud/unlink") async def cloud_unlink(request: Request): """Detach the cloud identity. An account the cloud provisioned has no diff --git a/backend/gamma/routers/mirrors.py b/backend/gamma/routers/mirrors.py index 5f83b7c1..8f655a7e 100644 --- a/backend/gamma/routers/mirrors.py +++ b/backend/gamma/routers/mirrors.py @@ -7,6 +7,7 @@ from .. import config, sync_engine, workspaces from ..auth import require_personal_user +from ..ops import OpError router = APIRouter(prefix="/api/mirrors", tags=["mirrors"]) @@ -54,8 +55,9 @@ def _mine(request: Request, ws: str) -> dict: def _info(mirror: dict) -> dict: info = workspaces.get(mirror["workspace_id"]) + count, newest = sync_engine.open_conflict_mark(mirror["workspace_id"]) return {**mirror, "name": info["name"] if info else "", - "conflicts_open": sync_engine.open_conflicts(mirror["workspace_id"]), + "conflicts_open": count, "conflicts_newest": newest, "pending_local": mirror["mode"] == "two-way" and sync_engine.has_local_changes(mirror["workspace_id"]), "interval_s": config.sync_interval_s(), "detached": mirror["mode"] == "off"} @@ -178,8 +180,14 @@ def list_conflicts(ws: str, request: Request, resolved: int = 0, page: str = "") @router.post("/{ws}/conflicts/{conflict_id}") def resolve_conflict(ws: str, conflict_id: int, payload: Resolution, request: Request): + """``{choice: keep | mine | theirs}``: the text is written first (into + the block's page as it is now), then the conflict is marked resolved; + a write the block refuses (409) leaves the conflict open.""" _mine(request, ws) - out = sync_engine.resolve_conflict(ws, conflict_id, payload.choice) + try: + out = sync_engine.resolve_conflict(ws, conflict_id, payload.choice) + except OpError as e: + raise HTTPException(status_code=409, detail=f"the text could not be written: {e}") if not out: raise HTTPException(status_code=404, detail="no such conflict") return out diff --git a/backend/gamma/routers/pages.py b/backend/gamma/routers/pages.py index a53e4294..958f8c3e 100644 --- a/backend/gamma/routers/pages.py +++ b/backend/gamma/routers/pages.py @@ -16,7 +16,7 @@ from fastapi.responses import JSONResponse from pydantic import BaseModel -from .. import block_index +from .. import block_index, publish from ..auth import require_ws from ..blocks_store import ( BLOCK_COLUMNS, @@ -68,7 +68,10 @@ def _load_page(conn, page_id: str): async def create_page_endpoint(payload: PageCreate, request: Request): """A new text-only page: ``{title?, folder?}`` → the page's block dict. Title defaults to "Untitled"; ``folder`` (a path like ``a/b``) becomes - ``properties.folder``.""" + ``properties.folder``. On a share host, 402 with ``{detail, limit, + used, plan}`` when the owner's plan allows no more pages in the + workspace (the path a publishing mirror creates its pages by; + gamma/publish.py page_cap).""" ws = require_ws(request, write=True) props = dict(payload.properties or {}) folder = clean_path(payload.folder or "") @@ -79,6 +82,9 @@ async def create_page_endpoint(payload: PageCreate, request: Request): with connect_pages_db(ws) as conn: if payload.id and conn.execute("SELECT 1 FROM unified_blocks WHERE id = ?", (payload.id,)).fetchone(): raise HTTPException(status_code=409, detail="a block with that id exists") + refusal = publish.cap_refusal(ws) # only a new page counts + if refusal: + return JSONResponse(status_code=402, content=refusal) return create_page(conn, payload.title, props, block_id=payload.id) diff --git a/backend/gamma/routers/prefs.py b/backend/gamma/routers/prefs.py index 03896b2e..ea6e170f 100644 --- a/backend/gamma/routers/prefs.py +++ b/backend/gamma/routers/prefs.py @@ -11,11 +11,18 @@ `profile` holds every account-scoped setting of the web app as one object keyed by preference name (``db.get_profile`` / ``db.set_profile``); the -server does not look inside it beyond requiring an object. +server does not look inside it beyond requiring an object. The web app +saves it with ``PATCH /prefs/profile`` (only the preferences it changed, +``db.patch_profile``), so a tab's stale copy of the others never undoes a +change synced from Gamma Cloud; reading it syncs with Gamma Cloud first +when the last sync is over a minute old (``cloud_sync.sync_if_stale``) and +says whether a first sync waits for the person's choice (``cloud_choice``). +``profile-base`` (the cloud sync's merge base) is never served here. The `ai-settings` key holds the user's AI provider API keys and is reserved: it is only reachable through /api/ai/settings, which masks the keys — these -generic endpoints must never serve it raw. +generic endpoints must never serve it raw. The same goes for +`translate-engines` (the machine-translation keys, /api/translate/engines). Also here: /api/page-snaps — the recents-card cover thumbnails (small JPEG data URLs the client captures from the rendered viewer). Same "UI state that @@ -29,16 +36,22 @@ from typing import Any from fastapi import APIRouter, HTTPException, Request +from fastapi.concurrency import run_in_threadpool from pydantic import BaseModel +from .. import cloud_sync from ..ai_settings import AI_SETTINGS_PREF_KEY +from ..translate_engines import ENGINES_PREF_KEY from ..auth import require_user, require_ws from ..db import ( + PROFILE_BASE_PREF_KEY, PROFILE_PREF_KEY, USER_PREF_KEYS, delete_page_snap, get_page_snaps, get_pref, + get_profile, + patch_profile, safe_doc_id, set_page_snap, set_pref, @@ -54,7 +67,7 @@ def _check_key(key: str): - if not _KEY_RE.match(key or "") or key == AI_SETTINGS_PREF_KEY: + if not _KEY_RE.match(key or "") or key in (AI_SETTINGS_PREF_KEY, ENGINES_PREF_KEY, PROFILE_BASE_PREF_KEY): raise HTTPException(status_code=400, detail="invalid pref key") @@ -62,12 +75,29 @@ class PrefWriteRequest(BaseModel): value: Any = None # any JSON value +class ProfilePatchRequest(BaseModel): + set: dict[str, Any] # preference name -> its new value; the others stay as stored + + +@router.patch("/prefs/profile") +def write_profile_entries(payload: ProfilePatchRequest, request: Request): + user = require_user(request) + if len(json.dumps({**get_profile(user)[0], **payload.set})) > MAX_VALUE_BYTES: + raise HTTPException(status_code=413, detail="pref value too large") + value, updated_at = patch_profile(user, payload.set) + return {"key": PROFILE_PREF_KEY, "value": value, "updated_at": updated_at} + + @router.get("/prefs/{key}") async def read_pref(key: str, request: Request): user = require_user(request) _check_key(key) + out = {"key": key} + if key == PROFILE_PREF_KEY: + await run_in_threadpool(cloud_sync.sync_if_stale, user) + out["cloud_choice"] = cloud_sync.profile_status(user)["state"] == "choose" value, updated_at = get_pref(user, key, "" if key in USER_PREF_KEYS else require_ws(request)) - return {"key": key, "value": value, "updated_at": updated_at} + return {**out, "value": value, "updated_at": updated_at} @router.put("/prefs/{key}") diff --git a/backend/gamma/routers/publish.py b/backend/gamma/routers/publish.py index 48d3231f..4d0cdc48 100644 --- a/backend/gamma/routers/publish.py +++ b/backend/gamma/routers/publish.py @@ -5,6 +5,10 @@ Bearer `` and ``{server?}`` → ``{token, workspace_id, username, url}``, a write token on the person's workspace there; +- on the share host, ``GET /api/publish/limit`` (the mirror's token) → + ``{used, max, plan}``, the plan's page cap on the person's workspace + there, and ``GET /api/pages/resolve-public?host=&path=`` (no auth) → + ``{share, page_id}`` for a page host's pretty address; - on the publishing server, ``POST / DELETE / GET /api/pages/{id}/publish`` for a page of the request's workspace. @@ -14,10 +18,11 @@ import socket from fastapi import APIRouter, HTTPException, Request +from fastapi.responses import JSONResponse from pydantic import BaseModel, Field from .. import cloud_auth, publish, ratelimit -from ..auth import require_personal_user, require_ws +from ..auth import note_share_miss, require_personal_user, require_ws router = APIRouter(tags=["publish"]) @@ -31,8 +36,8 @@ class PublishBody(BaseModel): role: str | None = Field(default=None, pattern="^(view|edit)$") -def _refused(e: publish.PublishError) -> HTTPException: - return HTTPException(e.status, e.message) +def _refused(e: publish.PublishError) -> JSONResponse: + return JSONResponse({"detail": e.message, **e.extra}, status_code=e.status) @router.post("/api/auth/cloud/exchange") @@ -51,7 +56,7 @@ def cloud_exchange(request: Request, payload: ExchangeBody | None = None): try: return publish.exchange(token, (payload or ExchangeBody()).server, cloud_auth.callback_base(request)) except publish.PublishError as e: - raise _refused(e) + return _refused(e) def _caller(request: Request, write: bool) -> tuple[str, str]: @@ -69,14 +74,17 @@ def publish_page(page_id: str, request: Request, payload: PublishBody | None = N there; default anyone / view) → ``{url, share, mirror: {ws, status, page_filter, conflicts_open, pending_local}}``. 409 with a message when publishing is not possible here (no Gamma Cloud identity, no share host, - a workspace that is a copy of another server, a share host itself).""" + a workspace that is a copy of another server, a share host itself); + 409 with ``limit: {used, max, plan}`` too when the person's plan allows + no more published pages there. The answer also carries ``public_url``, + the pretty address when the share host has page hosts (else ``url``).""" user, ws = _caller(request, write=True) payload = payload or PublishBody() try: return publish.publish(user, ws, page_id, audience=payload.audience, role=payload.role, server_name=_server_name(request)) except publish.PublishError as e: - raise _refused(e) + return _refused(e) @router.delete("/api/pages/{page_id}/publish") @@ -87,15 +95,40 @@ def unpublish_page(page_id: str, request: Request): try: return publish.unpublish(user, ws, page_id) except publish.PublishError as e: - raise _refused(e) + return _refused(e) @router.get("/api/pages/{page_id}/publish") def publication(page_id: str, request: Request): - """``{published, can_publish, reason?, url?, share?, status?, mirror?, - error?}`` — any member of the workspace.""" + """``{published, can_publish, reason?, url?, public_url?, share?, + status?, mirror?, limit?, error?}`` — any member of the workspace.""" user, ws = _caller(request, write=False) try: return publish.state(user, ws, page_id) except publish.PublishError as e: - raise _refused(e) + return _refused(e) + + +@router.get("/api/publish/limit") +def publish_limit(request: Request): + """The share host's half: ``{used, max, plan}`` — the root pages of the + request's workspace (a publishing mirror's token names it) and the cap + its owner's plan puts on them (``max`` null = none).""" + if not publish.this_is_share_host(): + raise HTTPException(404, "This server does not accept published pages.") + return publish.page_cap(require_ws(request)) + + +@router.get("/api/pages/resolve-public") +def resolve_public(request: Request, host: str = "", path: str = ""): + """A page host's pretty address (``host`` the hostname the browser + shows, ``path`` its ``/-``) → ``{share, page_id}``: the share + token the share view then opens with, audience and role its own. 404 + for anything else. No auth; per IP, and misses count as unknown share + links do.""" + ratelimit.check(f"resolve-public:ip:{ratelimit.client_ip(request)}", 120, 300) + try: + return publish.resolve_public(host[:300], path[:300]) + except publish.PublishError as e: + note_share_miss(request) + return _refused(e) diff --git a/backend/gamma/sync_engine.py b/backend/gamma/sync_engine.py index e97d5bf2..b7a79723 100644 --- a/backend/gamma/sync_engine.py +++ b/backend/gamma/sync_engine.py @@ -67,6 +67,7 @@ ADOPT = ("theirs", "mine") # whose version a never-reconciled page takes (a linked workspace, a force) DEBOUNCE_S = 1.0 # a local edit → a round once things have been quiet this long (the loop wakes for it) TICK_S = 1 # the loop's clock +WHOAMI_TTL_S = 900 # how long a round trusts the remote's last whoami (a failed round asks again) STREAM_CHUNK = 256 * 1024 default_fetch = None # the tests point this at an in-process TestClient; None = urllib UPLOAD_NAME_RE = re.compile(r"^[0-9a-f]{8,64}\.[a-z0-9]{1,8}$") @@ -275,12 +276,53 @@ def _save(ws: str, **fields) -> None: conn.commit() +_status_guard = threading.Lock() # every read-modify-write of a mirror's status JSON (and its cursors) + + +def _patch_status(ws: str, patch, *, drop=()) -> dict: + """Change some keys of the mirror's stored status and leave the rest as + they are now — one read-modify-write under a lock, so a round and the + actions taken while it runs (detach, a force, the filter, a direction + change) never overwrite each other's keys. ``patch`` is a dict laid over + the status, or a function of the current status returning the new one; + ``drop`` names keys to remove first. Returns the status as saved.""" + with _status_guard: + mirror = get_mirror(ws) + status = {k: v for k, v in (mirror["status"] if mirror else {}).items() if k not in drop} + status = patch(status) if callable(patch) else {**status, **patch} + _save(ws, status=status) + return status + + +def _stored_mode(ws: str) -> str: + with connect_users_db() as conn: + row = conn.execute("SELECT mode FROM mirrors WHERE workspace_id = ?", (ws,)).fetchone() + return row[0] if row else "off" + + def whoami(remote: Remote) -> dict: """The remote's view of the token: ``{user, workspace: {id, name}, role, scope}`` (``GET /api/sync/whoami``).""" return remote.get("/api/sync/whoami") +_whoami_seen: dict[str, tuple[tuple, float, dict]] = {} # ws -> ((url, remote ws, token), when, answer) + + +def _round_whoami(ws: str, mirror: dict, remote: Remote) -> dict: + """``whoami`` for a round: the answer an earlier round got for the same + link while it is younger than ``WHOAMI_TTL_S``, else a fresh one. A + round that ends with any error forgets it (``_round``), so a revoked + token or a lowered role is seen by the next round.""" + key = (mirror["remote_url"], mirror["remote_ws"], mirror["token"]) + seen = _whoami_seen.get(ws) + if seen and seen[0] == key and time.monotonic() - seen[1] < WHOAMI_TTL_S: + return seen[2] + me = whoami(remote) + _whoami_seen[ws] = (key, time.monotonic(), me) + return me + + def _check_remote(remote_url: str, token: str, mode: str, fetch) -> tuple[str, dict, str]: """Validate the address and the token against the remote's ``whoami``: ``(remote_url, me, mode)`` — the mode dropped to ``pull`` when the token @@ -376,15 +418,12 @@ def filter_add(ws: str, page_id: str, *, adopt: str = "") -> dict: mirror = get_mirror(ws) if not mirror: raise ValueError("not a mirror") - fields = {} + if adopt and adopt not in ADOPT: + raise ValueError("adopt must be theirs or mine") if mirror["page_filter"] is not None and page_id not in mirror["page_filter"]: - fields["page_filter"] = json.dumps(mirror["page_filter"] + [page_id]) + _save(ws, page_filter=json.dumps(mirror["page_filter"] + [page_id])) if adopt: - if adopt not in ADOPT: - raise ValueError("adopt must be theirs or mine") - fields["status"] = {**mirror["status"], "adopt": adopt} - if fields: - _save(ws, **fields) + _patch_status(ws, {"adopt": adopt}) return get_mirror(ws) @@ -397,10 +436,8 @@ def filter_remove(ws: str, page_ids) -> dict | None: if not mirror or mirror["page_filter"] is None: return mirror drop = set(page_ids) - status = mirror["status"] - retry = {k: v for k, v in (status.get("retry") or {}).items() if k not in drop} - _save(ws, page_filter=json.dumps([p for p in mirror["page_filter"] if p not in drop]), - status={**status, "retry": retry}) + _save(ws, page_filter=json.dumps([p for p in mirror["page_filter"] if p not in drop])) + _patch_status(ws, lambda s: {**s, "retry": {k: v for k, v in (s.get("retry") or {}).items() if k not in drop}}) with connect_pages_db(ws) as conn: for page_id in drop: _drop_state(conn, page_id) @@ -417,18 +454,21 @@ def set_cadence(ws: str, *, poll_s: int | None = None, on_change: bool | None = fields["poll_s"] = max(0, min(int(poll_s), 86400)) if on_change is not None: fields["on_change"] = 1 if on_change else 0 - if mode is not None: - if mode not in ("two-way", "pull"): - raise ValueError("mode must be two-way or pull") - fields["mode"] = mode - # a receive-only round moves the local cursor past edits it did not - # push: back in two-way, the next round looks at every page changed - # here since the beginning (one tree compare each) and pushes them - current = get_mirror(ws) - if current and current["mode"] == "pull" and mode == "two-way": - fields["local_cursor"] = "" - if fields: - _save(ws, **fields) + with _status_guard: # the cursor reset must not race a round's cursor save + if mode is not None: + if mode not in ("two-way", "pull"): + raise ValueError("mode must be two-way or pull") + current = get_mirror(ws) + if current and current["mode"] == "off": + raise ValueError("the copy is detached — reattach it first") + fields["mode"] = mode + # a receive-only round leaves the local cursor where it is, so the + # first two-way round pushes what it kept; copies from before that + # rule moved it, so back in two-way the cursor starts over anyway + if current and current["mode"] == "pull" and mode == "two-way": + fields["local_cursor"] = "" + if fields: + _save(ws, **fields) return get_mirror(ws) @@ -440,9 +480,12 @@ def detach_mirror(ws: str) -> dict: mirror = get_mirror(ws) if not mirror: raise ValueError("not a mirror") - status = {**mirror["status"], "running": False, "detached_at": page_now(), "detached_mode": mirror["mode"]} - status.pop("progress", None) - _save(ws, mode="off", status=status) + if mirror["mode"] == "off": + return mirror + # a round in flight stops at its next page (``_round`` checks the stored mode) and + # brings ``running`` down itself; a force asked for but not yet run is forgotten + _save(ws, mode="off") + _patch_status(ws, {"detached_at": page_now(), "detached_mode": mirror["mode"]}, drop=("force",)) return get_mirror(ws) @@ -462,16 +505,17 @@ def relink_mirror(ws: str, *, token: str = "", remote_url: str = "", adopt: str remote_url, me, mode = _check_remote(remote_url or mirror["remote_url"], token or mirror["token"], wanted, fetch) token = (token or mirror["token"]).strip() remote_ws, remote_name = me["workspace"]["id"], me["workspace"].get("name") or "Workspace" - status = {k: v for k, v in mirror["status"].items() if k not in ("detached_at", "detached_mode", "last_error")} - status.update(remote_user=me.get("user"), remote_role=me.get("role")) + patch = {"remote_user": me.get("user"), "remote_role": me.get("role")} fields = {"mode": mode, "remote_url": remote_url, "remote_ws": remote_ws, "remote_name": remote_name, "token": _seal(token)} if remote_url != mirror["remote_url"] or remote_ws != mirror["remote_ws"]: # a different original: the saved bases mean nothing, its pages are adopted _clear_bases(ws) fields.update(remote_cursor="", local_cursor="") - status["adopt"] = adopt - _save(ws, status=status, **fields) + patch["adopt"] = adopt + with _status_guard: + _save(ws, **fields) + _patch_status(ws, patch, drop=("detached_at", "detached_mode", "last_error")) return get_mirror(ws) @@ -481,7 +525,10 @@ def force_sync(ws: str, direction: str) -> None: texts they had are kept in ``diverged`` conflicts), ``push`` replaces the original with this copy. Every page is reconciled from scratch under the adopt policy and pages the losing side alone has are deleted there. The - round runs in the background.""" + round runs in the background: the force is noted on the status + (``force``) and the next round, under the round lock, starts from it — + clearing the bases and cursors while a round is in flight would leave + that round's bookkeeping and the force's fighting over them.""" if direction not in ("pull", "push"): raise ValueError("direction must be pull or push") mirror = get_mirror(ws) @@ -491,12 +538,24 @@ def force_sync(ws: str, direction: str) -> None: raise ValueError("the copy is detached — link it again first") if direction == "push" and mirror["mode"] != "two-way": raise ValueError("a read-only copy cannot replace the original") - _clear_bases(ws) - status = {**mirror["status"], "adopt": "theirs" if direction == "pull" else "mine", "prune": True} - _save(ws, remote_cursor="", local_cursor="", status=status) + _patch_status(ws, {"force": direction}) sync_in_background(ws) +def _start_force(ws: str, mirror: dict) -> dict: + """The first thing a round does under its lock: a force asked for + (``status.force``) becomes the round's policy — every base and both + cursors cleared, ``adopt`` and ``prune`` set. Returns the mirror to run.""" + direction = mirror["status"].get("force") + if direction not in ("pull", "push"): + return mirror + _clear_bases(ws) + with _status_guard: + _save(ws, remote_cursor="", local_cursor="") + status = _patch_status(ws, {"adopt": "theirs" if direction == "pull" else "mine", "prune": True}, drop=("force",)) + return {**mirror, "remote_cursor": "", "local_cursor": "", "status": status} + + def remove_mirror(ws: str) -> None: """Forget the link: the workspace stays as an ordinary local one; its sync state is dropped (a detached copy keeps it — see ``detach_mirror``).""" @@ -647,6 +706,16 @@ def open_conflicts(ws: str) -> int: return conn.execute("SELECT COUNT(*) FROM sync_conflicts WHERE resolved = 0").fetchone()[0] +def open_conflict_mark(ws: str) -> tuple[int, int]: + """``(count, newest id)`` of the open conflicts — the notice's + fingerprint (gamma/notices.py): a new conflict changes it, resolving + some of the old ones does not bring the notice back.""" + with connect_pages_db(ws) as conn: + count, newest = conn.execute( + "SELECT COUNT(*), COALESCE(MAX(id), 0) FROM sync_conflicts WHERE resolved = 0").fetchone() + return int(count), int(newest) + + def list_conflicts(ws: str, *, resolved: bool = False, page_id: str = "") -> list[dict]: """The decisions to look at (or the looked-at ones), newest first, one page's only when ``page_id`` is given.""" @@ -671,16 +740,18 @@ def resolve_conflict(ws: str, conflict_id: int, choice: str) -> dict | None: if not row: return None page_id, block_id, kind, mine, theirs, result = row - conn.execute("UPDATE sync_conflicts SET resolved = 1 WHERE id = ?", (conflict_id,)) - conn.commit() - exists = conn.execute("SELECT 1 FROM unified_blocks WHERE id = ?", (block_id,)).fetchone() - if choice != "keep" and kind in ("merged", "diverged") and exists: + # the block's page now: it may have moved to another page since the conflict was recorded + home = page_root_id(conn, block_id) if choice != "keep" and kind in ("merged", "diverged") else None + if home: # written as an edit from the text the conflict recorded: whatever was # typed into the block since is merged over the chosen version, not lost op = {"op": "set", "id": block_id, "content": mine if choice == "mine" else theirs} if result: op["base"] = result - commit_ops(ws, page_id, [op], actor=ACTOR) + commit_ops(ws, home, [op], actor=ACTOR) # an OpError leaves the conflict open + with connect_pages_db(ws) as conn: + conn.execute("UPDATE sync_conflicts SET resolved = 1 WHERE id = ?", (conflict_id,)) + conn.commit() return {"id": conflict_id, "resolved": True} @@ -839,7 +910,11 @@ def _push(remote: Remote, page_id: str, ops: list[dict]) -> None: def _reconcile_remote_ops(conn, page_id: str, base: dict, local: dict, remote: dict) -> list[dict]: """The remote's diff from base, adjusted so an edit beats a delete: remote deletes of subtrees edited here are dropped, and subtrees - deleted here that the remote edited inside come back whole.""" + deleted here that the remote edited inside come back whole. A block the + remote moved *out* of a subtree deleted here is not part of that + deletion any more: it comes back whole where the remote put it (the + subtree it left stays deleted unless something still inside it was + touched there).""" remote_ops = diff(base, remote, page_id) local_ops = diff(base, local, page_id) local_edited = {op["id"] for op in local_ops if op["op"] in ("set", "move", "insert")} @@ -852,33 +927,50 @@ def _reconcile_remote_ops(conn, page_id: str, base: dict, local: dict, remote: d remote_touched = {op["id"] for op in remote_ops if op["op"] in ("set", "move", "insert")} remote_touched |= {op["parent"] for op in remote_ops if op["op"] in ("insert", "move")} - out, restored, restored_tops = [], set(), [] + out, restored, restored_tops, escaped = [], set(), [], set() for top in local_deleted: if top not in remote: continue # the remote let it go too gone = subtree_ids(base, top) + still = subtree_ids(remote, top) # what the remote keeps inside it now + left = {bid for bid in gone - still if bid in remote} + # the top-most blocks that left the subtree there: each comes back with its remote subtree + escaped |= {bid for bid in left if not any(a in left for a in ancestors(remote, bid))} + if any(bid in still for bid in remote_touched): + restored |= still + restored_tops.append(top) + inserted = set() - def inside(bid): - return bid in gone or any(a in gone for a in ancestors(remote, bid)) + def insert_remote(bid): + r = remote[bid] + known = r["parent"] in local or r["parent"] in restored or r["parent"] in inserted + inserted.add(bid) + out.append({"op": "insert", "id": bid, "parent": r["parent"] if known else page_id, + "position": r["position"], "content": r["content"], "props": dict(r["props"])}) - if any(inside(bid) for bid in remote_touched): - restored |= subtree_ids(remote, top) - restored_tops.append(top) if restored: # re-insert the remote's version of each restored subtree, in tree order + # (a block that exists here — moved in there — is moved by its own op below) for bid in tree_order(remote, page_id): - if bid in restored: - r = remote[bid] - parent = r["parent"] if (r["parent"] in local or r["parent"] in restored) else page_id - out.append({"op": "insert", "id": bid, "parent": parent, "position": r["position"], - "content": r["content"], "props": dict(r["props"])}) + if bid in restored and bid not in local: + insert_remote(bid) for top in restored_tops: _conflict(conn, page_id, top, "restored_remote_edit", theirs=remote[top]["content"], result="kept the other side's version of a subtree deleted here") for op in remote_ops: bid = op["id"] - if bid in restored: + if bid in inserted: continue # already re-inserted whole + if op["op"] == "move" and bid in escaped: + # it left a subtree deleted here: back whole, where the remote moved it, with its own + # subtree (its later set/move ops are covered by the insert; blocks that exist here + # and were moved under it there keep their own move ops) + for eid in [bid] + [d for d in tree_order(remote, page_id) if d != bid and bid in ancestors(remote, d)]: + if eid not in local: + insert_remote(eid) + _conflict(conn, page_id, bid, "restored_remote_edit", theirs=remote[bid]["content"], + result="kept a block the other side moved out of a subtree deleted here") + continue if op["op"] == "delete" and (bid in touched_here or any(x in touched_here for x in subtree_ids(local, bid))): _conflict(conn, page_id, bid, "kept_local_edit", mine=local.get(bid, {}).get("content", ""), result="kept a subtree edited here that the other side deleted") @@ -887,8 +979,10 @@ def inside(bid): continue # gone here, not restored: the other side's change to it is dropped (a delete of # a block already gone — deleted on both sides, or moved to another page here — is done) if op["op"] in ("insert", "move") and op["parent"] not in local and op["parent"] not in restored \ - and not any(o["op"] == "insert" and o["id"] == op["parent"] for o in out): + and op["parent"] not in inserted: op = {**op, "parent": page_id} # its parent is gone here: land at the page's top level + if op["op"] == "insert": + inserted.add(bid) out.append(op) return out @@ -1161,18 +1255,21 @@ def _local_feed_all(ws: str, cursor: str) -> tuple[set, set, str]: def sync_workspace(ws: str, *, fetch=None) -> dict: """One round for the mirror ``ws``. Returns the status saved on the mirror (``{last_sync, last_error, pages_pulled, pages_pushed, ...}``). - Rounds for one workspace never overlap; a second caller waits.""" - mirror = get_mirror(ws, with_token=True) - if not mirror: - raise ValueError("not a mirror") + Rounds for one workspace never overlap; a second caller waits — and + reads the mirror only once it holds the lock, so what changed while it + waited (a page unpublished, a detach, a force) is what it runs with.""" lock = _lock(ws) with lock: + mirror = get_mirror(ws, with_token=True) + if not mirror: + raise ValueError("not a mirror") return _round(ws, mirror, fetch) def _round(ws: str, mirror: dict, fetch) -> dict: if mirror["mode"] == "off": return mirror["status"] # detached: nothing runs until it is linked again + mirror = _start_force(ws, mirror) if mirror["page_filter"] is not None: mirror = {**mirror, "page_filter": _prune_filter(ws, mirror["page_filter"])} if not mirror["page_filter"]: @@ -1180,13 +1277,14 @@ def _round(ws: str, mirror: dict, fetch) -> dict: remote = Remote(mirror["remote_url"], mirror["remote_ws"], mirror["token"], fetch) first = not mirror["status"].get("last_sync") # the first fill (or one that never completed) started = time.monotonic() # local writes up to here are this round's to push - status = {**mirror["status"], "running": True, "started_at": page_now(), "progress": None} - status.pop("interrupted", None) - _save(ws, status=status) + # the status is only ever patched: what a detach, a force or the filter write + # to it while the round runs stays (``_patch_status``) + _patch_status(ws, {"running": True, "started_at": page_now(), "progress": None}, drop=("interrupted",)) report = {"pages_pulled": 0, "pages_pushed": 0, "pages_deleted": 0, "files_pulled": 0, "files_pushed": 0, "blocks_added": 0, "blocks_removed": 0, "blocks_changed": 0, "errors": [], "adopt": mirror["status"].get("adopt"), "prune": bool(mirror["status"].get("prune"))} + progress: dict = {} last_file_save = [0.0] def file_progress(name, done, total, direction): @@ -1195,22 +1293,25 @@ def file_progress(name, done, total, direction): if done < total and now - last_file_save[0] < 0.3: return last_file_save[0] = now - status["progress"] = {**(status.get("progress") or {}), - "file": {"name": name, "done": done, "total": total, "dir": direction}} - _save(ws, status=status) + progress["file"] = {"name": name, "done": done, "total": total, "dir": direction} + _patch_status(ws, {"progress": dict(progress)}) report["progress"] = file_progress mode = mirror["mode"] try: - me = whoami(remote) + me = _round_whoami(ws, mirror, remote) role = me.get("role") if me else None if mode == "two-way" and (me.get("scope") != "write" or role == "viewer"): report["errors"].append("the token or your role on the remote is read-only: pulling only") mode = "pull" remote_pages, remote_deleted, remote_cursor = _feed_all(remote, mirror["remote_cursor"]) - local_pages, local_deleted, local_cursor = _local_feed_all(ws, mirror["local_cursor"]) - if mode != "two-way": - local_pages, local_deleted = set(), set() + if mode == "two-way" or (report["prune"] and (report["adopt"] or "theirs") == "theirs"): + # (a force pull reads the local feed whatever the mode: pages only this copy has go) + local_pages, local_deleted, local_cursor = _local_feed_all(ws, mirror["local_cursor"]) + else: + # receive only, or read-only on the remote for now: the local feed is not walked and + # its cursor stays put, so the first round that may push finds every edit made here + local_pages, local_deleted, local_cursor = set(), set(), mirror["local_cursor"] todo = {} for page_id in set(remote_pages) | set(remote_deleted) | local_pages | local_deleted: todo[page_id] = {"seq": remote_pages.get(page_id), @@ -1223,12 +1324,17 @@ def file_progress(name, done, total, direction): if mirror["page_filter"] is not None: todo = _filtered(ws, mirror["page_filter"], todo, force=report["prune"]) failed = {} - for n, page_id in enumerate(sorted(todo)): + order = sorted(todo) + for n, page_id in enumerate(order): flags = todo[page_id] + if _stored_mode(ws) == "off": + # detached while running: the rest waits for a reattach (the cursors move past it) + failed.update({p: todo[p] for p in order[n:]}) + break # the pill and the popover read this while the round runs: "21 of 79 pages" - status["progress"] = {"done": n, "total": len(todo), "page": _title_of(ws, page_id), - "first": first, "at": page_now()} - _save(ws, status=status) + progress.clear() + progress.update(done=n, total=len(todo), page=_title_of(ws, page_id), first=first, at=page_now()) + _patch_status(ws, {"progress": dict(progress)}) try: _sync_page(ws, remote, page_id, remote_seq_hint=flags["seq"], remote_gone=flags["remote_gone"], local_gone=flags["local_gone"], mode=mode, report=report) @@ -1242,24 +1348,37 @@ def file_progress(name, done, total, direction): # files the copy's pages reference but its uploads folder lacks (an # interrupted round, a file lost on disk): fetched again every round _pull_files(ws, remote, missing_uploads(ws, mirror["page_filter"]), report) - # cursors move only when the round could talk to the remote at all - _save(ws, remote_cursor=remote_cursor, local_cursor=local_cursor) + # cursors move only when the round could talk to the remote at all, and only + # when nothing reset them meanwhile (a direction change; a force waits its turn) + with _status_guard: + now = get_mirror(ws) + if now and now["remote_cursor"] == mirror["remote_cursor"] and now["local_cursor"] == mirror["local_cursor"]: + _save(ws, remote_cursor=remote_cursor, local_cursor=local_cursor) report = {k: v for k, v in report.items() if k not in ("adopt", "prune", "progress")} - status = {**status, **report, "running": False, "last_sync": page_now(), "mode": mode, - "remote_role": role, "remote_user": me.get("user") if me else None, - "last_error": report["errors"][0] if report["errors"] else "", "retry": failed} - if not failed: - # a link's or a force's policy is spent once every page went through - status.pop("adopt", None) - status.pop("prune", None) - if not report["errors"] and _dirty.get(ws, float("inf")) <= started: + errors = report.pop("errors") + patch = {**report, "running": False, "last_sync": page_now(), "mode": mode, + "remote_role": role, "remote_user": me.get("user") if me else None, + "last_error": errors[0] if errors else "", "retry": failed} + + def finish(current): + current = {**current, **patch} + if not failed: + # a link's or a force's policy is spent once every page went through — unless + # a newer one was asked for while the round ran + for key in ("adopt", "prune"): + if current.get(key) == mirror["status"].get(key): + current.pop(key, None) + return current + + status = _patch_status(ws, finish, drop=("progress",)) + if not errors and _dirty.get(ws, float("inf")) <= started: _dirty.pop(ws, None) # everything written before the round started went out with it except Exception as e: # noqa: BLE001 — whatever happens, the running flag comes down - status = {**status, "running": False, "last_error": str(e), "last_attempt": page_now()} + status = _patch_status(ws, {"running": False, "last_error": str(e), "last_attempt": page_now()}, + drop=("progress",)) log.warning(f"[mirror] {ws}: {e}") - status.pop("errors", None) - status.pop("progress", None) - _save(ws, status=status) + if status.get("last_error"): + _whoami_seen.pop(ws, None) return status diff --git a/backend/gamma/translate_engines.py b/backend/gamma/translate_engines.py new file mode 100644 index 00000000..59bfb078 --- /dev/null +++ b/backend/gamma/translate_engines.py @@ -0,0 +1,357 @@ +"""Machine-translation engines for the PDF translated view: Microsoft (the +Edge browser's free endpoint, no key), Google Cloud Translation and Youdao, +next to the LLM path in routers/ai.py. + +An engine is picked like a model: the viewer sends ``model: "engine:"`` +to /api/ai/translate, which then calls ``translate()`` here instead of a chat +model. No AI provider is needed for that path. + +Credentials are per account (Settings → Reading → Translation), stored in +users.db ``user_prefs`` under the reserved account-wide ``translate-engines`` +key as {engine id: {field: value, "updated_at"}}. Like ``ai-settings`` the +generic /api/prefs endpoints refuse the key; the only read path is the +masked GET /api/translate/engines. +""" + +import hashlib +import json +import threading +import time +import uuid +from urllib.error import HTTPError, URLError +from urllib.parse import urlencode +from urllib.request import Request, urlopen + +from fastapi import HTTPException + +from .db import get_pref, page_now, set_pref +from .logbuf import log + +ENGINES_PREF_KEY = "translate-engines" +MODEL_PREFIX = "engine:" +MAX_FIELD_LEN = 512 +TIMEOUT = 60 + +# Microsoft: the endpoint the Edge browser's own page translation calls — +# unauthenticated, unofficial and undocumented (it replaced the +# /translate/auth token flow, which Microsoft retired in July 2026), so it +# may change or throttle without notice. Same reply shape as Translator v3. +MICROSOFT_URL = "https://edge.microsoft.com/translate/translatetext" +GOOGLE_URL = "https://translation.googleapis.com/language/translate/v2" +YOUDAO_URL = "https://openapi.youdao.com/v2/api" + +# The viewer's allowlisted target languages, for both translation paths: +# code → the name the LLM prompt (routers/ai.py) splices in. Mirrored in +# frontend/src/app/prefDefs.js TRANSLATE_LANGS — keep the two in sync. +TRANSLATE_LANGS = { + "en": "English", "zh-CN": "Simplified Chinese", "zh-TW": "Traditional Chinese", + "ja": "Japanese", "ko": "Korean", "de": "German", "fr": "French", + "es": "Spanish", "pt": "Portuguese", "it": "Italian", "ru": "Russian", +} + +# Per engine: its credential fields (secret ones are masked when listed; an +# engine without fields needs no setup and is always offered) and its codes +# for TRANSLATE_LANGS. +ENGINES = { + "microsoft": { + "label": "Microsoft (free)", + "fields": [], + "langs": {"en": "en", "zh-CN": "zh-Hans", "zh-TW": "zh-Hant", "ja": "ja", "ko": "ko", + "de": "de", "fr": "fr", "es": "es", "pt": "pt", "it": "it", "ru": "ru"}, + # Refuses requests past ~50k characters (measured); stay well under. + "batch": (100, 20000), + }, + "google": { + "label": "Google Cloud Translation", + "fields": [{"id": "api_key", "secret": True}], + "langs": {code: code for code in TRANSLATE_LANGS}, + # Google's documented cap is 128 segments per request; the character + # cap keeps each request well under its payload limit. + "batch": (100, 25000), + }, + "youdao": { + "label": "Youdao", + "fields": [{"id": "app_key", "secret": False}, {"id": "app_secret", "secret": True}], + "langs": {"en": "en", "zh-CN": "zh-CHS", "zh-TW": "zh-CHT", "ja": "ja", "ko": "ko", + "de": "de", "fr": "fr", "es": "es", "pt": "pt", "it": "it", "ru": "ru"}, + # A single query may be up to 5000 characters. + "batch": (50, 4500), + }, +} + + +class EngineError(Exception): + """The engine refused or failed the request; the message is shown.""" + + +# --- the free service's health ------------------------------------------------ +# Microsoft's endpoint is unofficial, so it can stop answering any day. Its +# consecutive failures are counted in memory (a restart starts fresh). From +# FREE_ALERT_AFTER on, the server log gets one warning per streak, and each +# account that met the failures gets a notice (notices.free_translate_failing) +# pointing at Settings → Reading, whose Microsoft row names the error. One +# success ends the streak. +FREE_ENGINE = "microsoft" +FREE_ALERT_AFTER = 3 +_health_lock = threading.Lock() +_health = {"failures": 0, "since": "", "error": "", "users": set(), "warned": False} + + +def _note_outcome(engine: str, user: str, error: str | None) -> None: + if engine != FREE_ENGINE: + return + with _health_lock: + if error is None: + _health.update(failures=0, since="", error="", users=set(), warned=False) + return + if not _health["failures"]: + _health["since"] = page_now() + _health["failures"] += 1 + _health["error"] = error + _health["users"].add(user) + warn = _health["failures"] >= FREE_ALERT_AFTER and not _health["warned"] + _health["warned"] = _health["warned"] or warn + if warn: + log.warning(f"[translate] {ENGINES[engine]['label']} failed {FREE_ALERT_AFTER} times in a row " + f"({error}); its endpoint may have changed. Google or Youdao keys are the fallback.") + + +def free_failing(user: str) -> dict | None: + """The free service's current failure streak ({since, error}) once it has + reached FREE_ALERT_AFTER, for an account that met it; else None.""" + with _health_lock: + if _health["failures"] < FREE_ALERT_AFTER or user not in _health["users"]: + return None + return {"since": _health["since"], "error": _health["error"]} + + +# --- stored credentials ------------------------------------------------------- + +def load(user: str) -> dict: + value, _ = get_pref(user, ENGINES_PREF_KEY) + if not isinstance(value, dict): + return {} + return {k: v for k, v in value.items() if k in ENGINES and isinstance(v, dict)} + + +def _complete(engine: str, conf: dict | None) -> bool: + if not ENGINES[engine]["fields"]: + return True # nothing to set up + return bool(conf) and all((conf.get(f["id"]) or "").strip() for f in ENGINES[engine]["fields"]) + + +def configured(user: str) -> list: + """[{id: "engine:", label}] for every engine with all its fields set — + what the translation picker offers.""" + saved = load(user) + return [{"id": MODEL_PREFIX + eid, "label": e["label"]} + for eid, e in ENGINES.items() if _complete(eid, saved.get(eid))] + + +def masked(user: str, can_edit: bool) -> dict: + """The settings view: every engine, its fields (secrets as a last-4 hint), + and whether it is ready to use.""" + saved = load(user) + rows = [] + for eid, e in ENGINES.items(): + conf = saved.get(eid) or {} + fields = {} + for f in e["fields"]: + value = (conf.get(f["id"]) or "").strip() + fields[f["id"]] = ("…" + value[-4:] if len(value) > 8 else "set") if f["secret"] and value else value + rows.append({"id": eid, "label": e["label"], "configured": _complete(eid, conf), + "needs_key": bool(e["fields"]), "fields": fields, + "updated_at": conf.get("updated_at", ""), + "failing": free_failing(user) if eid == FREE_ENGINE else None}) + return {"engines": rows, "can_edit": can_edit} + + +def save(user: str, engine: str, fields: dict) -> None: + """Set an engine's credentials. A secret field left empty keeps the stored + value (the form never sees it); a plain field is taken as given.""" + if engine not in ENGINES: + raise HTTPException(status_code=404, detail="unknown translation engine") + if not ENGINES[engine]["fields"]: + raise HTTPException(status_code=400, detail=f"{ENGINES[engine]['label']} needs no key") + saved = load(user) + old = saved.get(engine) or {} + conf = {} + for f in ENGINES[engine]["fields"]: + value = fields.get(f["id"]) + value = value.strip() if isinstance(value, str) else "" + if len(value) > MAX_FIELD_LEN: + raise HTTPException(status_code=400, detail=f"{f['id']} is too long") + if not value and f["secret"]: + value = old.get(f["id"]) or "" + if not value: + raise HTTPException(status_code=400, detail=f"{f['id']} is required") + conf[f["id"]] = value + conf["updated_at"] = page_now() + saved[engine] = conf + set_pref(user, ENGINES_PREF_KEY, saved) + + +def remove(user: str, engine: str) -> None: + saved = load(user) + if saved.pop(engine, None) is not None: + set_pref(user, ENGINES_PREF_KEY, saved) + + +def engine_of(model: str) -> str: + """The engine id a translate request names (``engine:``), else "".""" + if isinstance(model, str) and model.startswith(MODEL_PREFIX): + return model[len(MODEL_PREFIX):] + return "" + + +def credentials(user: str, engine: str) -> dict: + """The stored credentials for a request; 404 for an unknown engine and + 503 (like a missing AI provider) when it isn't set up.""" + if engine not in ENGINES: + raise HTTPException(status_code=404, detail="unknown translation engine") + conf = load(user).get(engine) + if not _complete(engine, conf): + raise HTTPException(status_code=503, + detail=f"{ENGINES[engine]['label']} is not set up — add its key in Settings → Reading") + return conf or {} + + +# --- translation -------------------------------------------------------------- + +def translate(engine: str, conf: dict, texts: list, lang: str, user: str) -> list: + """``texts`` translated into ``lang`` (a TRANSLATE_LANGS code), same length + and order, for ``user`` (the free service's health counts per account). + Split into the engine's batch limits; raises EngineError.""" + e = ENGINES[engine] + target = e["langs"].get(lang) + if not target: + raise EngineError(f"{e['label']} does not translate into {lang}") + call = {"microsoft": _microsoft, "google": _google, "youdao": _youdao}[engine] + out = [] + try: + for batch in _batches(texts, *e["batch"]): + out.extend(call(conf, batch, target)) + except EngineError as err: + _note_outcome(engine, user, str(err)) + raise + _note_outcome(engine, user, None) + return out + + +def _batches(texts: list, max_n: int, max_chars: int): + batch, size = [], 0 + for t in texts: + if batch and (len(batch) >= max_n or size + len(t) > max_chars): + yield batch + batch, size = [], 0 + batch.append(t) + size += len(t) + if batch: + yield batch + + +def _send(req: Request, name: str) -> dict: + try: + with urlopen(req, timeout=TIMEOUT) as resp: + return json.loads(resp.read().decode("utf-8")) + except HTTPError as err: + # A JSON {"error": {"message"}} body (Google, Microsoft), else the + # start of a plain-text one. + detail = "" + try: + raw = err.read().decode("utf-8", "replace") + try: + body = json.loads(raw) + detail = (body.get("error") or {}).get("message", "") if isinstance(body, dict) else "" + except ValueError: + detail = raw.strip()[:200] + except Exception: + pass + raise EngineError(f"{name}: HTTP {err.code}{' — ' + detail if detail else ''}") from None + except (URLError, TimeoutError, OSError) as err: + raise EngineError(f"{name}: {getattr(err, 'reason', err)}") from None + except ValueError: + raise EngineError(f"{name}: unreadable reply") from None + + +def _microsoft(conf: dict, texts: list, target: str) -> list: + # No "from": the service detects the source language per text. + query = urlencode({"to": target, "isEnterpriseClient": "false"}) + req = Request(f"{MICROSOFT_URL}?{query}", data=json.dumps(texts).encode("utf-8"), method="POST", + headers={"Content-Type": "application/json"}) + data = _send(req, "Microsoft") + if not isinstance(data, list) or len(data) != len(texts): + raise EngineError("Microsoft: unexpected reply") + out = [] + for item, t in zip(data, texts): + tr = (item.get("translations") or [{}])[0] if isinstance(item, dict) else {} + text = tr.get("text") if isinstance(tr, dict) else None + out.append(str(text) if text else t) + return out + + +def _google(conf: dict, texts: list, target: str) -> list: + # Basic (v2) API with an API key, sent as a header so it never lands in + # a URL. format "text": the reply is plain text, not HTML-escaped. + body = json.dumps({"q": texts, "target": target, "format": "text"}).encode("utf-8") + req = Request(GOOGLE_URL, data=body, method="POST", headers={ + "Content-Type": "application/json", "X-Goog-Api-Key": conf["api_key"]}) + data = _send(req, "Google") + items = (data.get("data") or {}).get("translations") if isinstance(data, dict) else None + if not isinstance(items, list) or len(items) != len(texts): + raise EngineError("Google: unexpected reply") + return [str(it.get("translatedText") or t) for it, t in zip(items, texts)] + + +# Youdao error codes worth naming; the rest show as the bare number. +_YOUDAO_ERRORS = { + "101": "missing parameter", "108": "invalid app key", "110": "no service bound to this app", + "202": "signature check failed — check the app secret", "206": "clock skew", + "401": "account balance exhausted", "411": "too many requests", "412": "too many long requests", +} + + +def youdao_sign(app_key: str, app_secret: str, texts: list, salt: str, curtime: str) -> str: + """v3 signature: sha256(appKey + input + salt + curtime + appSecret), where + input is the concatenated queries, shortened to first 10 chars + length + + last 10 chars past 20 characters.""" + q = "".join(texts) + size = len(q) + short = q if size <= 20 else q[:10] + str(size) + q[size - 10:] + return hashlib.sha256((app_key + short + salt + curtime + app_secret).encode("utf-8")).hexdigest() + + +def _youdao(conf: dict, texts: list, target: str) -> list: + salt, curtime = str(uuid.uuid4()), str(int(time.time())) + form = [("q", t) for t in texts] + [ + ("from", "auto"), ("to", target), ("appKey", conf["app_key"]), ("salt", salt), + ("sign", youdao_sign(conf["app_key"], conf["app_secret"], texts, salt, curtime)), + ("signType", "v3"), ("curtime", curtime), + ] + req = Request(YOUDAO_URL, data=urlencode(form).encode("utf-8"), method="POST", + headers={"Content-Type": "application/x-www-form-urlencoded"}) + data = _send(req, "Youdao") + code = str(data.get("errorCode", "")) if isinstance(data, dict) else "" + if code != "0": + raise EngineError(f"Youdao: error {code}" + (f" — {_YOUDAO_ERRORS[code]}" if code in _YOUDAO_ERRORS else "")) + results = data.get("translateResults") + if not isinstance(results, list): + raise EngineError("Youdao: unexpected reply") + # Queries that failed are listed in errorIndex and may be missing from + # the results; they come back verbatim (the viewer shows the original). + failed = {int(i) for i in data.get("errorIndex") or [] if str(i).isdigit()} + ok = [i for i in range(len(texts)) if i not in failed] + if len(results) == len(texts): + pairs = zip(range(len(texts)), results) + elif len(results) == len(ok): + pairs = zip(ok, results) + else: + raise EngineError("Youdao: unexpected reply") + out = list(texts) + for i, r in pairs: + tr = r.get("translation") if isinstance(r, dict) else None + if isinstance(tr, list): + tr = "\n".join(str(x) for x in tr) + if i not in failed and tr: + out[i] = str(tr) + return out diff --git a/backend/gamma/workspaces.py b/backend/gamma/workspaces.py index 358ae5e4..036ad7e5 100644 --- a/backend/gamma/workspaces.py +++ b/backend/gamma/workspaces.py @@ -152,8 +152,10 @@ def list_for_user(username: str) -> list[dict]: the top), then by name. ``members`` counts explicit members; ``mirror_of`` names the remote workspace a mirror follows ("" otherwise); ``publishing`` is true for a workspace that publishes pages to the share - host (a filtered mirror, gamma/publish.py), which is not a clone: - ``mirror_of`` stays "" for it.""" + host (a filtered mirror with at least one page in its filter, + gamma/publish.py), which is not a clone: ``mirror_of`` stays "" for it. + A publication whose last page was unpublished keeps its mirror row (the + token, for the next publish) but is invisible: nothing is published.""" with connect_users_db() as conn: me = conn.execute( "SELECT default_workspace, is_guest FROM users WHERE username = ?", (username,)).fetchone() @@ -164,7 +166,7 @@ def list_for_user(username: str) -> list[dict]: "(SELECT remote_name FROM mirrors mi WHERE mi.workspace_id = w.id AND mi.mode != 'off' " "AND mi.page_filter IS NULL), " "EXISTS (SELECT 1 FROM mirrors mp WHERE mp.workspace_id = w.id AND mp.mode != 'off' " - "AND mp.page_filter IS NOT NULL) " + "AND mp.page_filter IS NOT NULL AND mp.page_filter != '[]') " "FROM workspaces w WHERE w.access = 'public' " "OR EXISTS (SELECT 1 FROM workspace_members m WHERE m.workspace_id = w.id AND m.username = ?)", (username, username)).fetchall() @@ -469,10 +471,7 @@ def cloud_lookup_username(name: str, by: str = "") -> dict | None: cfg = cloud_auth.settings() if not cfg["enabled"]: raise CloudLookupError("Gamma Cloud sign-in is not set up on this server.") - token = _cloud_access_token(by) - if not token: - raise CloudLookupError("This server has no Gamma Cloud access token to look usernames up with.") - return lookup_with_token(cfg["issuer"], token, name) + return lookup_with_token(cfg["issuer"], _cloud_access_token(by), name) def lookup_cloud_username(name, by: str = "") -> dict | None: diff --git a/backend/tests/test_ai_translate.py b/backend/tests/test_ai_translate.py index a4eba811..c86cfdb0 100644 --- a/backend/tests/test_ai_translate.py +++ b/backend/tests/test_ai_translate.py @@ -234,3 +234,268 @@ def boom(*a, **kw): assert r.status_code == 200 lines = [json.loads(l) for l in r.text.splitlines() if l.strip()] assert lines == [{"error": "translation failed: provider down"}] + + +# --- machine-translation engines (Google Cloud Translation, Youdao) ---------- + +class _FakeResp: + def __init__(self, body): + self.body = json.dumps(body).encode() + + def read(self): + return self.body + + def __enter__(self): + return self + + def __exit__(self, *a): + return False + + +def _fake_urlopen(reply): + """A urlopen stand-in for gamma.translate_engines: records each request, + answers `reply(req)` as the JSON body.""" + calls = [] + + def fake(req, timeout=None): + calls.append(req) + return _FakeResp(reply(req)) + + return fake, calls + + +@pytest.fixture(scope="module") +def dave(client): + """An account with NO AI provider — the engine path must not need one.""" + from gamma.app import app + from gamma.db import connect_users_db, page_now + from gamma import workspaces + + with connect_users_db() as conn: + if not conn.execute("SELECT 1 FROM users WHERE username = 'translate_dave'").fetchone(): + conn.execute( + "INSERT INTO users (username, password_hash, is_guest, created_at) VALUES (?, ?, 0, ?)", + ("translate_dave", bcrypt.hashpw(b"pw", bcrypt.gensalt()).decode(), page_now()), + ) + conn.commit() + workspaces.ensure_personal("translate_dave") + c = TestClient(app) + assert c.post("/api/login", json={"username": "translate_dave", "password": "pw"}).status_code == 200 + return c + + +def test_engine_settings_are_masked_and_reserved(dave): + body = dave.get("/api/translate/engines").json() + assert body["can_edit"] is True + # Microsoft needs no key: always ready, nothing to store. + assert {e["id"]: e["configured"] for e in body["engines"]} == {"microsoft": True, "google": False, "youdao": False} + assert [e["id"] for e in body["engines"] if not e["needs_key"]] == ["microsoft"] + assert dave.put("/api/translate/engines/microsoft", json={"fields": {}}).status_code == 400 + + assert dave.put("/api/translate/engines/google", json={"fields": {}}).status_code == 400 + assert dave.put("/api/translate/engines/bing", json={"fields": {"api_key": "x"}}).status_code == 404 + r = dave.put("/api/translate/engines/google", json={"fields": {"api_key": "AIza-secret-key-9876"}}) + assert r.status_code == 200, r.text + google = next(e for e in r.json()["engines"] if e["id"] == "google") + assert google["configured"] is True + assert google["fields"]["api_key"] == "…9876" + assert "AIza-secret-key-9876" not in r.text + + # An empty secret on edit keeps the stored one; the plain app key shows. + r = dave.put("/api/translate/engines/youdao", json={"fields": {"app_key": "app-1", "app_secret": "s3cret-value-1234"}}) + assert r.status_code == 200 + r = dave.put("/api/translate/engines/youdao", json={"fields": {"app_key": "app-2", "app_secret": ""}}) + youdao = next(e for e in r.json()["engines"] if e["id"] == "youdao") + assert youdao["configured"] and youdao["fields"] == {"app_key": "app-2", "app_secret": "…1234"} + + # The raw key never leaves through the generic prefs endpoints. + assert dave.get("/api/prefs/translate-engines").status_code == 400 + assert dave.put("/api/prefs/translate-engines", json={"value": {}}).status_code == 400 + + models = dave.get("/api/ai/models").json() + assert [e["id"] for e in models["translate_engines"]] == ["engine:microsoft", "engine:google", "engine:youdao"] + + r = dave.delete("/api/translate/engines/youdao") + assert [e["id"] for e in r.json()["engines"] if e["configured"]] == ["microsoft", "google"] + + +def test_guest_cannot_store_engine_keys(): + from gamma.app import app + + c = TestClient(app) + assert c.post("/api/login-guest").status_code == 200 + assert c.get("/api/translate/engines").json()["can_edit"] is False + assert c.put("/api/translate/engines/google", json={"fields": {"api_key": "k" * 20}}).status_code == 403 + + +def test_translate_with_google(dave, monkeypatch): + dave.put("/api/translate/engines/google", json={"fields": {"api_key": "AIza-secret-key-9876"}}) + monkeypatch.setattr("gamma.routers.ai._call_ai", lambda *a, **k: pytest.fail("no LLM on the engine path")) + fake, calls = _fake_urlopen(lambda req: {"data": {"translations": [ + {"translatedText": f"G:{t}"} for t in json.loads(req.data)["q"]]}}) + monkeypatch.setattr("gamma.translate_engines.urlopen", fake) + + texts = ["Google one.", " ", "Google two.", "Google one."] + r = dave.post("/api/ai/translate", json={"texts": texts, "lang": "zh-TW", "model": "engine:google"}) + assert r.status_code == 200, r.text + assert r.json() == {"translations": ["G:Google one.", " ", "G:Google two.", "G:Google one."], + "model": "engine:google", "cached": False} + (req,) = calls + assert req.get_header("X-goog-api-key") == "AIza-secret-key-9876" + assert "key=" not in req.full_url + assert json.loads(req.data) == {"q": ["Google one.", "Google two."], "target": "zh-TW", "format": "text"} + + # Cached per engine; the streamed form answers with the final line. + r = dave.post("/api/ai/translate", json={"texts": ["Google two."], "lang": "zh-TW", + "model": "engine:google", "stream": True}) + assert [json.loads(line) for line in r.text.splitlines() if line.strip()] == [ + {"translations": ["G:Google two."], "model": "engine:google", "cached": True}] + assert len(calls) == 1 + + +def test_translate_with_engine_errors(dave, monkeypatch): + import io + from urllib.error import HTTPError + + # Not set up: 503, like a missing AI provider. + dave.delete("/api/translate/engines/youdao") + r = dave.post("/api/ai/translate", json={"texts": ["x y"], "lang": "de", "model": "engine:youdao"}) + assert r.status_code == 503 + + dave.put("/api/translate/engines/google", json={"fields": {"api_key": "AIza-secret-key-9876"}}) + + def denied(req, timeout=None): + raise HTTPError(req.full_url, 403, "Forbidden", {}, + io.BytesIO(json.dumps({"error": {"message": "API key not valid"}}).encode())) + + monkeypatch.setattr("gamma.translate_engines.urlopen", denied) + r = dave.post("/api/ai/translate", json={"texts": ["An uncached line."], "lang": "de", "model": "engine:google"}) + assert r.status_code == 502 + assert "API key not valid" in r.json()["detail"] + r = dave.post("/api/translate/engines/google/test", json={"lang": "de"}) + assert r.json() == {"ok": False, "error": "Google: HTTP 403 — API key not valid"} + + +def test_translate_with_youdao(dave, monkeypatch): + from urllib.parse import parse_qs + from gamma import translate_engines + + dave.put("/api/translate/engines/youdao", json={"fields": {"app_key": "app-1", "app_secret": "sec-1"}}) + + def reply(req): + form = parse_qs(req.data.decode()) + qs = form["q"] + assert form["to"] == ["zh-CHS"] and form["from"] == ["auto"] and form["signType"] == ["v3"] + assert form["sign"] == [translate_engines.youdao_sign( + "app-1", "sec-1", qs, form["salt"][0], form["curtime"][0])] + # The second query failed upstream: listed in errorIndex, absent from the results. + return {"errorCode": "0", "errorIndex": [1], + "translateResults": [{"query": q, "translation": f"Y:{q}"} for j, q in enumerate(qs) if j != 1]} + + fake, calls = _fake_urlopen(reply) + monkeypatch.setattr("gamma.translate_engines.urlopen", fake) + r = dave.post("/api/ai/translate", json={"texts": ["Youdao a.", "Youdao b.", "Youdao c."], + "lang": "zh-CN", "model": "engine:youdao"}) + assert r.status_code == 200, r.text + # The failed one comes back verbatim (and stays uncached for a retry). + assert r.json()["translations"] == ["Y:Youdao a.", "Youdao b.", "Y:Youdao c."] + + monkeypatch.setattr("gamma.translate_engines.urlopen", + _fake_urlopen(lambda req: {"errorCode": "202"})[0]) + r = dave.post("/api/translate/engines/youdao/test", json={"lang": "zh-CN"}) + assert r.json()["ok"] is False and "signature" in r.json()["error"] + + +def test_translate_with_microsoft(dave, monkeypatch): + from urllib.parse import parse_qs, urlsplit + + monkeypatch.setattr("gamma.routers.ai._call_ai", lambda *a, **k: pytest.fail("no LLM on the engine path")) + + def reply(req): + texts = json.loads(req.data) + # Translator v3's shape; the second text comes back without a translation. + return [{"detectedLanguage": {"language": "en", "score": 1.0}, + "translations": [{"text": f"M:{t}", "to": "zh-Hant"}] if j != 1 else []} + for j, t in enumerate(texts)] + + fake, calls = _fake_urlopen(reply) + monkeypatch.setattr("gamma.translate_engines.urlopen", fake) + r = dave.post("/api/ai/translate", json={"texts": ["Edge one.", "Edge two.", "Edge three."], + "lang": "zh-TW", "model": "engine:microsoft"}) + assert r.status_code == 200, r.text + # No setup, no key; a text without a translation stays as it was. + assert r.json()["translations"] == ["M:Edge one.", "Edge two.", "M:Edge three."] + (req,) = calls + url = urlsplit(req.full_url) + assert url.netloc == "edge.microsoft.com" and url.path == "/translate/translatetext" + assert parse_qs(url.query) == {"to": ["zh-Hant"], "isEnterpriseClient": ["false"]} + assert req.get_header("Authorization") is None + assert json.loads(req.data) == ["Edge one.", "Edge two.", "Edge three."] + + # A plain-text error body is surfaced as it is. + from urllib.error import HTTPError + import io + + def too_big(req, timeout=None): + raise HTTPError(req.full_url, 400, "Bad Request", {}, + io.BytesIO(b"Request exceeds the maximum allowed translation size.")) + + monkeypatch.setattr("gamma.translate_engines.urlopen", too_big) + r = dave.post("/api/translate/engines/microsoft/test", json={"lang": "ja"}) + assert r.json() == {"ok": False, + "error": "Microsoft: HTTP 400 \u2014 Request exceeds the maximum allowed translation size."} + + +def test_free_service_failures_warn_and_notify_until_it_answers(dave, carol, monkeypatch): + from urllib.error import URLError + from gamma import logbuf, translate_engines + + monkeypatch.setattr(translate_engines, "_health", + {"failures": 0, "since": "", "error": "", "users": set(), "warned": False}) + notices = lambda c: [n for n in c.get("/api/notices").json()["notices"] if n["id"] == "free-translate"] + + def down(req, timeout=None): + raise URLError("connection refused") + + monkeypatch.setattr("gamma.translate_engines.urlopen", down) + start = logbuf.last_seq("warning") + streak_warnings = lambda: sum("times in a row" in e["msg"] for e in logbuf.tail(start)) + for i in range(translate_engines.FREE_ALERT_AFTER): + assert notices(dave) == [] # not before the streak is long enough + r = dave.post("/api/ai/translate", json={"texts": [f"down {i}"], "lang": "de", "model": "engine:microsoft"}) + assert r.status_code == 502 + (notice,) = notices(dave) + assert notice["pane"] == "reading" and notice["tone"] == "warn" + assert notices(carol) == [] # an account that never met the failures isn't told + row = next(e for e in dave.get("/api/translate/engines").json()["engines"] if e["id"] == "microsoft") + assert row["failing"] == {"since": notice["fingerprint"], "error": "Microsoft: connection refused"} + # One streak warning in the log, however long the streak gets. + assert streak_warnings() == 1 + dave.post("/api/ai/translate", json={"texts": ["down again"], "lang": "de", "model": "engine:microsoft"}) + assert streak_warnings() == 1 + + # One answer ends the streak: the notice and the row's error are gone. + monkeypatch.setattr("gamma.translate_engines.urlopen", _fake_urlopen( + lambda req: [{"translations": [{"text": "wieder da"}]} for _ in json.loads(req.data)])[0]) + assert dave.post("/api/ai/translate", json={"texts": ["back up"], "lang": "de", + "model": "engine:microsoft"}).status_code == 200 + assert notices(dave) == [] + row = next(e for e in dave.get("/api/translate/engines").json()["engines"] if e["id"] == "microsoft") + assert row["failing"] is None + + +def test_youdao_sign_shortens_long_input(): + import hashlib + from gamma.translate_engines import youdao_sign + + assert youdao_sign("k", "s", ["hello"], "salt", "1") == hashlib.sha256(b"khellosalt1s").hexdigest() + texts = ["abcdefghijKLMN", "OPQRSTUVWXyz0123456789"] # 36 chars joined + assert youdao_sign("k", "s", texts, "salt", "1") == hashlib.sha256(b"kabcdefghij360123456789salt1s").hexdigest() + + +def test_engine_batches_split_on_limits(): + from gamma.translate_engines import _batches + + assert list(_batches(["a" * 3, "b" * 3, "c" * 3], 10, 6)) == [["aaa", "bbb"], ["ccc"]] + assert list(_batches(["x"] * 5, 2, 100)) == [["x", "x"], ["x", "x"], ["x"]] + assert list(_batches(["y" * 50], 2, 10)) == [["y" * 50]] # an oversize text goes alone diff --git a/backend/tests/test_ai_usage.py b/backend/tests/test_ai_usage.py index 338bf8b9..f0a97694 100644 --- a/backend/tests/test_ai_usage.py +++ b/backend/tests/test_ai_usage.py @@ -7,7 +7,10 @@ import pytest from ai_fixtures import CONF, FakeResp, ai_provider, org, sse # noqa: F401 (fixtures) -from gamma.ai_client import add_usage, normalize_usage, openai_request, sse_events +from gamma.ai_client import add_usage, normalize_usage, sse_events +from gamma.ai_protocols import WIRES + +openai_request = WIRES["openai"].request @pytest.fixture(scope="module", autouse=True) diff --git a/backend/tests/test_ai_wire.py b/backend/tests/test_ai_wire.py index 4a526529..721e5c54 100644 --- a/backend/tests/test_ai_wire.py +++ b/backend/tests/test_ai_wire.py @@ -5,18 +5,17 @@ import json -from gamma.ai_client import ( - anthropic_request, - chatgpt_request, - openai_request, - openai_responses_request, - sse_events, - wire_protocol, -) +from gamma.ai_client import sse_events, wire_protocol +from gamma.ai_protocols import WIRES from gamma.ai_context import TOOL_REPLAY_BUDGET, build_messages from ai_fixtures import ALL_TOOLS, CONF, TURNS, payload, sse +anthropic_request = WIRES["anthropic"].request +openai_request = WIRES["openai"].request +openai_responses_request = WIRES["openai-responses"].request +chatgpt_request = WIRES["chatgpt"].request + def test_anthropic_wire_tools_and_results(): req = anthropic_request(CONF, [dict(m) for m in TURNS], "sys", "m", tools=ALL_TOOLS) diff --git a/backend/tests/test_backup_schedule.py b/backend/tests/test_backup_schedule.py index 795bcb42..0d947058 100644 --- a/backend/tests/test_backup_schedule.py +++ b/backend/tests/test_backup_schedule.py @@ -1,3 +1,5 @@ +import pathlib +import time from datetime import datetime, timedelta, timezone from types import SimpleNamespace @@ -11,6 +13,8 @@ @pytest.fixture def workspace(tmp_path, monkeypatch, client): monkeypatch.setattr(tasks.config, 'BACKUPS_DIR', tmp_path / 'backups') + # These tests call run_due themselves; "Run now" must not also wake the app's live loop. + monkeypatch.setattr(tasks, '_wake', lambda: None) return make_user('scheduled_owner', 'schedulepass1') @@ -95,6 +99,34 @@ def test_task_retention_catchup_and_manual_isolation(workspace, monkeypatch): assert ws_backup.list_backups(workspace) == before +def test_run_now_wakes_the_running_scheduler(tmp_path, monkeypatch, client): + # The app's own loop (started by `client`), not a run_due call: Run now + # must not wait for the next 30 s round. + monkeypatch.setattr(tasks.config, 'BACKUPS_DIR', tmp_path / 'backups') + task = create(make_user('scheduled_owner', 'schedulepass1'), enabled=False) + tasks.mutate('scheduled_owner', task['id'], 'run') + deadline = time.monotonic() + 10 + while tasks.read(task['id'])['state'] != 'finished' and time.monotonic() < deadline: + time.sleep(0.05) + assert tasks.read(task['id'])['state'] == 'finished' + + +def test_write_waits_out_a_reader_holding_the_file(workspace, monkeypatch): + # Windows: replacing a task file fails while list_tasks is reading it. + task = create(workspace, enabled=False) + real, refusals = pathlib.Path.replace, [] + + def replace(self, target): + if len(refusals) < 2: + refusals.append(target) + raise PermissionError(5, 'Access is denied') + return real(self, target) + + monkeypatch.setattr(pathlib.Path, 'replace', replace) + tasks.mutate('scheduled_owner', task['id'], 'run') + assert len(refusals) == 2 and tasks.read(task['id'])['state'] == 'queued' + + def test_run_now_paused_and_regular_schedule_preserved(workspace, monkeypatch): at = datetime(2026, 9, 21, 4, tzinfo=timezone.utc) monkeypatch.setattr(tasks, 'now', lambda: at) diff --git a/backend/tests/test_chatgpt_oauth.py b/backend/tests/test_chatgpt_oauth.py index 2d291c77..3f61748d 100644 --- a/backend/tests/test_chatgpt_oauth.py +++ b/backend/tests/test_chatgpt_oauth.py @@ -13,9 +13,12 @@ from fastapi.testclient import TestClient import gamma.chatgpt_oauth as co -from gamma.ai_client import chatgpt_request as _chatgpt_request +from gamma import ai_catalog +from gamma.ai_protocols import WIRES, chatgpt as chatgpt_proto from gamma.routers.ai import _sse_deltas +_chatgpt_request = WIRES["chatgpt"].request + def _fake_jwt(claims: dict) -> str: def seg(d): @@ -104,11 +107,10 @@ def test_model_catalog_needs_signin_before_connect(erin): def test_connect_flow_creates_masked_entry_and_models(erin, monkeypatch): - import gamma.routers.ai as ai_mod - monkeypatch.setattr(co, "_token_request", lambda form: _fake_tokens()) + monkeypatch.setattr(chatgpt_proto, "codex_client_version", lambda: "9.9.9") # A fresh connect seeds its model list live from the account (first two). - monkeypatch.setattr(ai_mod, "urlopen", lambda req, timeout=0: _FakeResp({"models": [ + monkeypatch.setattr(ai_catalog, "urlopen", lambda req, timeout=0: _FakeResp({"models": [ {"slug": "gpt-6-sol", "visibility": "list"}, {"slug": "gpt-6-terra", "visibility": "list"}, {"slug": "gpt-6-luna", "visibility": "list"}, @@ -238,7 +240,6 @@ def __exit__(self, *a): def test_chatgpt_provider_usage_reports_remaining_windows(erin, monkeypatch): - import gamma.routers.ai as ai_mod from gamma.ai_settings import load_provider_entries entry = next(e for e in load_provider_entries("erin") if e.get("protocol") == "chatgpt") @@ -273,7 +274,7 @@ def fake_open(req, timeout=0): "credits": {"has_credits": False, "balance": "0"}, }) - monkeypatch.setattr(ai_mod, "urlopen", fake_open) + monkeypatch.setattr(ai_catalog, "urlopen", fake_open) r = erin.post(f"/api/ai/providers/{entry['id']}/usage") assert r.status_code == 200, r.text body = r.json() @@ -286,13 +287,11 @@ def fake_open(req, timeout=0): def test_model_catalog_asks_chatgpt_backend_live(erin, monkeypatch): - import gamma.routers.ai as ai_mod - entry = next(p for p in erin.get("/api/ai/settings").json()["providers"] if p["protocol"] == "chatgpt") seen = {} - monkeypatch.setattr(ai_mod, "_codex_client_version", lambda: "9.9.9") + monkeypatch.setattr(chatgpt_proto, "codex_client_version", lambda: "9.9.9") def fake_urlopen(req, timeout=0): seen["url"] = req.full_url @@ -305,7 +304,7 @@ def fake_urlopen(req, timeout=0): {"slug": "gpt-4-gone", "visibility": "none"}, # dropped ]}) - monkeypatch.setattr(ai_mod, "urlopen", fake_urlopen) + monkeypatch.setattr(ai_catalog, "urlopen", fake_urlopen) r = erin.post("/api/ai/model-catalog", json={"provider_id": entry["id"]}) assert r.status_code == 200 assert r.json()["models"] == ["gpt-6-codex", "gpt-5.1-codex", "gpt-5-codex-mini"] @@ -319,44 +318,41 @@ def fake_urlopen(req, timeout=0): def test_codex_client_version_is_looked_up_and_cached(monkeypatch): - import gamma.routers.ai as ai_mod - - monkeypatch.setattr(ai_mod, "_codex_version", {"value": "", "until": 0.0}) + monkeypatch.setattr(chatgpt_proto, "_codex_version", {"value": "", "until": 0.0}) calls = [] def npm(req, timeout=0): calls.append(req.full_url) return _FakeResp({"name": "@openai/codex", "version": "1.2.3"}) - monkeypatch.setattr(ai_mod, "urlopen", npm) - assert ai_mod._codex_client_version() == "1.2.3" - assert ai_mod._codex_client_version() == "1.2.3" - assert calls == [ai_mod._CODEX_VERSION_URL] # second call served from cache + monkeypatch.setattr(chatgpt_proto, "urlopen", npm) + assert chatgpt_proto.codex_client_version() == "1.2.3" + assert chatgpt_proto.codex_client_version() == "1.2.3" + assert calls == [chatgpt_proto.CODEX_VERSION_URL] # second call served from cache # Expired + npm down: keep the last good version, and don't retry at once. def down(req, timeout=0): calls.append(req.full_url) raise OSError("offline") - ai_mod._codex_version["until"] = 0.0 - monkeypatch.setattr(ai_mod, "urlopen", down) - assert ai_mod._codex_client_version() == "1.2.3" - assert ai_mod._codex_client_version() == "1.2.3" + chatgpt_proto._codex_version["until"] = 0.0 + monkeypatch.setattr(chatgpt_proto, "urlopen", down) + assert chatgpt_proto.codex_client_version() == "1.2.3" + assert chatgpt_proto.codex_client_version() == "1.2.3" assert len(calls) == 2 # Never looked up successfully: the floor. - monkeypatch.setattr(ai_mod, "_codex_version", {"value": "", "until": 0.0}) - assert ai_mod._codex_client_version() == ai_mod._CODEX_VERSION_FLOOR + monkeypatch.setattr(chatgpt_proto, "_codex_version", {"value": "", "until": 0.0}) + assert chatgpt_proto.codex_client_version() == chatgpt_proto.CODEX_VERSION_FLOOR def test_model_catalog_errors_when_listing_fails(erin, monkeypatch): - import gamma.routers.ai as ai_mod - def boom(req, timeout=0): raise OSError("no route to host") # No hardcoded model list to fall back on — the picker shows the error. - monkeypatch.setattr(ai_mod, "urlopen", boom) + monkeypatch.setattr(chatgpt_proto, "codex_client_version", lambda: "9.9.9") + monkeypatch.setattr(ai_catalog, "urlopen", boom) r = erin.post("/api/ai/model-catalog", json={"protocol": "chatgpt"}) assert r.status_code == 502 assert "no route to host" in r.json()["detail"] @@ -368,15 +364,13 @@ def boom(req, timeout=0): def test_api_model_catalog_uses_one_short_attempt(erin, monkeypatch): - import gamma.routers.ai as ai_mod - calls = [] def timed_out(req, timeout=0): calls.append((req, timeout)) raise TimeoutError("timed out") - monkeypatch.setattr(ai_mod, "urlopen", timed_out) + monkeypatch.setattr(ai_catalog, "urlopen", timed_out) r = erin.post("/api/ai/model-catalog", json={ "protocol": "openai", "api_key": "sk-test", @@ -390,8 +384,6 @@ def timed_out(req, timeout=0): def test_api_model_catalog_does_not_retry_auth_error(erin, monkeypatch): - import gamma.routers.ai as ai_mod - calls = 0 def unauthorized(req, timeout=0): @@ -405,7 +397,7 @@ def unauthorized(req, timeout=0): io.BytesIO(b'{"error":{"message":"bad key"}}'), ) - monkeypatch.setattr(ai_mod, "urlopen", unauthorized) + monkeypatch.setattr(ai_catalog, "urlopen", unauthorized) r = erin.post("/api/ai/model-catalog", json={ "protocol": "openai", "api_key": "sk-bad", @@ -424,7 +416,7 @@ def test_catalog_uses_edited_endpoint_and_protocol(erin, monkeypatch): "base_url": "https://old.example", }]) calls = [] - monkeypatch.setattr(ai_mod, "_model_catalog_json", lambda req: calls.append(req) or {"data": []}) + monkeypatch.setattr(ai_catalog, "fetch_json", lambda req: calls.append(req) or {"data": []}) for fields in ({}, {"base_url": ""}, {"protocol": "anthropic", "base_url": "https://new.example"}): response = erin.post("/api/ai/model-catalog", json={"provider_id": "saved", **fields}) assert response.status_code == 200 @@ -468,3 +460,69 @@ def test_chatgpt_sse_deltas_join_and_fail(): empty = [b'data: {"type":"response.completed","response":{"status":"completed"}}\n'] with pytest.raises(RuntimeError, match="empty response"): list(_sse_deltas(empty, "chatgpt")) + + +def test_context_window_comes_from_the_listing_then_models_dev(erin, monkeypatch): + monkeypatch.setattr(ai_catalog, "_listings", {}) + monkeypatch.setattr(ai_catalog, "_models_dev", {"windows": None, "until": 0.0}) + monkeypatch.setattr(chatgpt_proto, "codex_client_version", lambda: "9.9.9") + entry = next(p for p in erin.get("/api/ai/settings").json()["providers"] + if p["protocol"] == "chatgpt") + model = next(m for m in erin.get("/api/ai/models").json()["models"] if m["provider"] == entry["id"]) + calls = [] + listing = {"models": [{"slug": model["model"], "context_window": 272_000}]} + catalog = { + "openai": {"models": {model["model"]: {"id": model["model"], "limit": {"context": 400_000}}}}, + "gateway": {"models": {f"openai/{model['model']}": {"limit": {"context": 128_000}}}}, + } + + def fake_urlopen(req, timeout=0): + calls.append(req.full_url) + return _FakeResp(catalog if "models.dev" in req.full_url else listing) + + monkeypatch.setattr(ai_catalog, "urlopen", fake_urlopen) + ask = lambda: erin.get("/api/ai/context-window", params={"model": model["id"]}).json() + + # The provider's own listing says it; asked once, then cached. + assert ask() == {"model": model["model"], "context_window": 272_000, "source": "provider"} + assert ask()["context_window"] == 272_000 + assert len(calls) == 1 and "/models?client_version=9.9.9" in calls[0] + + # A listing without sizes: models.dev, the vendor behind the protocol winning. + listing = {"models": [{"slug": model["model"]}]} + ai_catalog._listings.clear() + assert ask() == {"model": model["model"], "context_window": 400_000, "source": "models.dev"} + + # Nobody knows it: null, never a guess. + catalog = {"openai": {"models": {}}} + ai_catalog._listings.clear() + ai_catalog._models_dev.update(windows=None, until=0.0) + assert ask() == {"model": model["model"], "context_window": None, "source": ""} + + +def test_context_window_lookups_keep_the_last_good_answer(monkeypatch): + monkeypatch.setattr(ai_catalog, "_listings", {}) + monkeypatch.setattr(ai_catalog, "_models_dev", {"windows": None, "until": 0.0}) + conf = {"protocol": "openai", "api_key": "k", "base_url": "https://api.deepseek.com", "name": "DeepSeek"} + monkeypatch.setattr(ai_catalog, "urlopen", lambda req, timeout=0: _FakeResp( + {"data": [{"id": "llama", "max_model_len": 32_768}, {"id": "bare"}]} if "/v1/models" in req.full_url else { + "deepseek": {"models": {"deepseek-chat": {"limit": {"context": 131_072}}}}, + "a": {"models": {"deepseek-chat": {"limit": {"context": 64_000}}}}, + "b": {"models": {"deepseek-chat": {"limit": {"context": 64_000}}}}, + })) + assert ai_catalog._listed_windows("p", conf) == {"llama": 32_768} + # The provider named in the entry's host wins over the majority... + assert ai_catalog._catalog_window("deepseek-chat", conf) == 131_072 + # ...and without one, the value most providers agree on. + assert ai_catalog._catalog_window("deepseek-chat", {**conf, "base_url": "https://example.org"}) == 64_000 + + # Expired + offline: the last good answers stay, retried only later. + def down(req, timeout=0): + raise OSError("offline") + + monkeypatch.setattr(ai_catalog, "urlopen", down) + for cached in ai_catalog._listings.values(): + cached["until"] = 0.0 + ai_catalog._models_dev["until"] = 0.0 + assert ai_catalog._listed_windows("p", conf) == {"llama": 32_768} + assert ai_catalog._catalog_window("deepseek-chat", conf) == 131_072 diff --git a/backend/tests/test_cloud_auth.py b/backend/tests/test_cloud_auth.py index 0c37b232..6ad237b7 100644 --- a/backend/tests/test_cloud_auth.py +++ b/backend/tests/test_cloud_auth.py @@ -529,12 +529,20 @@ def unavailable(url, *a, **kw): assert cloud_auth.refresh_token_of("ca_lee") == "rt-1+" -def test_profile_pull_push_and_conflict(cloud, monkeypatch): - make_user("ca_mia", "pw-ca_mia-123") - set_profile("ca_mia", {"theme": "light"}) # older than the cloud's; no identity yet, so nothing is pushed - time.sleep(0.01) - cloud.prefs["sub-ca_mia"] = {"profile": {"value": {"theme": "dark"}, "updated_at": _ms_now()}} - # the sign-in pulls the newer profile before the browser loads +def _elsewhere(cloud, name, value): + """Another server of the person behind ``name`` pushes ``value``.""" + time.sleep(0.003) + cloud.prefs.setdefault(f"sub-{name}", {})["profile"] = {"value": value, "updated_at": _ms_now()} + time.sleep(0.003) + + +def _cloud_copy(cloud, name): + return cloud.prefs[f"sub-{name}"]["profile"]["value"] + + +def test_profile_pull_push_and_debounce(cloud, monkeypatch): + # a server with no profile yet takes the cloud's at sign-in, before the browser loads + _elsewhere(cloud, "ca_mia", {"theme": "dark"}) c = link_account(cloud, "ca_mia") value, at = get_profile("ca_mia") assert value == {"theme": "dark"} @@ -546,34 +554,153 @@ def test_profile_pull_push_and_conflict(cloud, monkeypatch): monkeypatch.setattr(cloud_sync, "PUSH_DELAY", 0.2) puts = lambda: [x for x in cloud.calls if x == ("PUT", "/api/me/prefs/profile")] # noqa: E731 for theme in ("sepia", "gray", "night"): - assert c.put("/api/prefs/profile", json={"value": {"theme": theme}}).status_code == 200 - assert until(lambda: cloud.prefs["sub-ca_mia"]["profile"]["value"] == {"theme": "night"}) + assert c.patch("/api/prefs/profile", json={"set": {"theme": theme}}).status_code == 200 + assert until(lambda: _cloud_copy(cloud, "ca_mia") == {"theme": "night"}) time.sleep(0.3) assert len(puts()) == 1 and not cloud_sync._timers assert cloud_sync.sync_profile("ca_mia") == "same" - # the hourly check: a newer copy elsewhere is pulled, a newer one here is pushed - time.sleep(0.01) - cloud.prefs["sub-ca_mia"]["profile"] = {"value": {"theme": "dark", "enterNewNote": True}, "updated_at": _ms_now()} - assert cloud_sync.check("ca_mia") == "ok" and get_profile("ca_mia")[0] == {"theme": "dark", "enterNewNote": True} - monkeypatch.setattr(cloud_sync, "profile_changed", lambda username: None) # no timer: pushed by hand below - set_profile("ca_mia", {"theme": "gray"}) - assert cloud_sync.sync_profile("ca_mia") == "pushed" - assert cloud.prefs["sub-ca_mia"]["profile"]["value"] == {"theme": "gray"} + # a change elsewhere is pulled by the hourly check + _elsewhere(cloud, "ca_mia", {"theme": "night", "enterNewNote": True}) + assert cloud_sync.check("ca_mia") == "ok" and get_profile("ca_mia")[0] == {"theme": "night", "enterNewNote": True} - # a push the account server refuses as older takes its value (409) - set_profile("ca_mia", {"theme": "stale"}) - time.sleep(0.01) - cloud.prefs["sub-ca_mia"]["profile"] = {"value": {"theme": "sepia"}, "updated_at": _ms_now()} - assert cloud_sync.push_profile("ca_mia") == "pulled" - assert get_profile("ca_mia")[0] == {"theme": "sepia"} - - # the cloud has none: the local one goes up + # the cloud lost its copy: this one goes up again cloud.prefs["sub-ca_mia"].clear() assert cloud_sync.sync_profile("ca_mia") == "pushed" - assert cloud.prefs["sub-ca_mia"]["profile"]["value"] == {"theme": "sepia"} + assert _cloud_copy(cloud, "ca_mia") == {"theme": "night", "enterNewNote": True} # only the profile travels: never the provider entries or the active entry assert {path for _, path in cloud.calls if path.startswith("/api/me/prefs")} == {"/api/me/prefs/profile"} + # the merge base is never served by the generic prefs endpoints + assert c.get("/api/prefs/profile-base").status_code == 400 + assert c.put("/api/prefs/profile-base", json={"value": {}}).status_code == 400 + + +def test_first_sync_asks_when_the_copies_differ(cloud, monkeypatch): + monkeypatch.setattr(cloud_sync, "profile_changed", lambda username: None) + make_user("ca_ria", "pw-ca_ria-123") + set_profile("ca_ria", {"theme": "sepia", "enterNewNote": False}) + _elsewhere(cloud, "ca_ria", {"theme": "light", "enterNewNote": True}) + c = link_account(cloud, "ca_ria") + # neither copy replaced the other: the person chooses + assert get_profile("ca_ria")[0] == {"theme": "sepia", "enterNewNote": False} + assert _cloud_copy(cloud, "ca_ria") == {"theme": "light", "enterNewNote": True} + assert c.get("/api/auth/cloud/sync-status").json()["profile"]["state"] == "choose" + assert c.get("/api/prefs/profile").json()["cloud_choice"] is True + assert "cloud-sync-choice" in [n["id"] for n in c.get("/api/notices").json()["notices"]] + assert cloud_sync.check("ca_ria") == "ok" and cloud_sync.profile_status("ca_ria")["state"] == "choose" + assert c.post("/api/auth/cloud/sync", json={"action": "sync"}).json()["outcome"] == "choose" + assert ("PUT", "/api/me/prefs/profile") not in cloud.calls + + # merge: the defaults are the base, so each side keeps what it changed from them; an entry + # neither side holds yet is at its default + r = c.post("/api/auth/cloud/sync", json={"action": "merge", + "defaults": {"theme": "light", "enterNewNote": False, "pdfDarkPage": False}}) + assert r.status_code == 200 and r.json()["outcome"] == "merged" and r.json()["profile"]["state"] == "synced" + both = {"theme": "sepia", "enterNewNote": True, "pdfDarkPage": False} + assert get_profile("ca_ria")[0] == both and _cloud_copy(cloud, "ca_ria") == both + assert c.get("/api/prefs/profile").json()["cloud_choice"] is False + assert "cloud-sync-choice" not in [n["id"] for n in c.get("/api/notices").json()["notices"]] + # a base agreed with another cloud account (unlinked, then linked to someone else's) is no base + assert cloud_sync._base_of("ca_ria") == both + monkeypatch.setattr(cloud_auth, "grant_of", lambda username: ("sub-someone-else", "rt")) + assert cloud_sync._base_of("ca_ria") is None + + +def test_fetch_and_push_replace_one_copy(cloud, monkeypatch): + monkeypatch.setattr(cloud_sync, "profile_changed", lambda username: None) + make_user("ca_sia", "pw-ca_sia-123") + set_profile("ca_sia", {"theme": "sepia"}) + _elsewhere(cloud, "ca_sia", {"theme": "light", "language": "zh"}) + c = link_account(cloud, "ca_sia") + assert cloud_sync.profile_status("ca_sia")["state"] == "choose" + # push: this server's copy replaces the cloud's, even though the cloud's is newer + r = c.post("/api/auth/cloud/sync", json={"action": "push"}) + assert r.json() == {"outcome": "pushed", "profile": r.json()["profile"]} and r.json()["profile"]["state"] == "synced" + assert _cloud_copy(cloud, "ca_sia") == {"theme": "sepia"} and get_profile("ca_sia")[0] == {"theme": "sepia"} + # fetch: the cloud's copy replaces this one, a newer change here included + _elsewhere(cloud, "ca_sia", {"theme": "night"}) + set_profile("ca_sia", {"theme": "gray", "enterNewNote": True}) + assert c.post("/api/auth/cloud/sync", json={"action": "fetch"}).json()["outcome"] == "pulled" + assert get_profile("ca_sia")[0] == {"theme": "night"} + assert c.post("/api/auth/cloud/sync", json={"action": "sync"}).json()["outcome"] == "same" + # nothing to fetch + cloud.prefs["sub-ca_sia"].clear() + r = c.post("/api/auth/cloud/sync", json={"action": "fetch"}) + assert r.status_code == 409 and "no settings" in r.json()["detail"] + # an account without a linked identity has nothing to sync with + make_user("ca_tess", "pw-ca_tess-123") + assert login("ca_tess", "pw-ca_tess-123").post("/api/auth/cloud/sync", json={"action": "sync"}).status_code == 400 + + +def test_changes_on_two_servers_merge_per_preference(cloud, monkeypatch): + monkeypatch.setattr(cloud_sync, "profile_changed", lambda username: None) + c = link_account(cloud, "ca_tom") + edit = lambda **entries: c.patch("/api/prefs/profile", json={"set": entries}).json() # noqa: E731 + edit(theme="light", language="en", enterNewNote=False) + assert cloud_sync.sync_profile("ca_tom") == "pushed" + # here the theme, elsewhere the language, before either synced: both kept + edit(theme="dark") + _elsewhere(cloud, "ca_tom", {"theme": "light", "language": "zh", "enterNewNote": False}) + assert cloud_sync.sync_profile("ca_tom") == "merged" + both = {"theme": "dark", "language": "zh", "enterNewNote": False} + assert get_profile("ca_tom")[0] == both and _cloud_copy(cloud, "ca_tom") == both + # one preference changed on both sides: the newer profile's value + edit(theme="sepia") + _elsewhere(cloud, "ca_tom", {**both, "theme": "night"}) + assert cloud_sync.sync_profile("ca_tom") == "pulled" and get_profile("ca_tom")[0]["theme"] == "night" + _elsewhere(cloud, "ca_tom", {**both, "theme": "gray"}) + edit(theme="solarized") + assert cloud_sync.sync_profile("ca_tom") == "pushed" and _cloud_copy(cloud, "ca_tom")["theme"] == "solarized" + + +def test_a_stale_tab_does_not_undo_a_synced_change(cloud, monkeypatch): + monkeypatch.setattr(cloud_sync, "profile_changed", lambda username: None) + c = link_account(cloud, "ca_ula") + c.patch("/api/prefs/profile", json={"set": {"theme": "light", "language": "en"}}) + assert cloud_sync.sync_profile("ca_ula") == "pushed" + # the other server changes the theme and this server pulls it ... + _elsewhere(cloud, "ca_ula", {"theme": "dark", "language": "en"}) + assert cloud_sync.sync_profile("ca_ula") == "pulled" + # ... while a tab loaded before that still shows "light": it saves only what it changed + r = c.patch("/api/prefs/profile", json={"set": {"language": "zh"}}) + assert r.json()["value"] == {"theme": "dark", "language": "zh"} + assert cloud_sync.sync_profile("ca_ula") == "pushed" + assert _cloud_copy(cloud, "ca_ula") == {"theme": "dark", "language": "zh"} + + +def test_reading_the_profile_syncs_at_most_once_a_minute(cloud, monkeypatch): + monkeypatch.setattr(cloud_sync, "profile_changed", lambda username: None) + c = link_account(cloud, "ca_val") + _elsewhere(cloud, "ca_val", {"theme": "dark"}) + gets = lambda: sum(1 for x in cloud.calls if x == ("GET", "/api/me/prefs/profile")) # noqa: E731 + before = gets() + # the sign-in synced a moment ago: a read answers from here + assert c.get("/api/prefs/profile").json()["value"] is None and gets() == before + # a minute on, the read syncs first and answers with the cloud's change + monkeypatch.setattr(cloud_sync, "READ_SYNC_EVERY", 0) + assert c.get("/api/prefs/profile").json()["value"] == {"theme": "dark"} and gets() == before + 1 + # offline: the read still answers, from here + cloud.offline = True + assert c.get("/api/prefs/profile").json()["value"] == {"theme": "dark"} + + +def test_a_push_that_loses_a_race_merges_again(cloud, monkeypatch): + monkeypatch.setattr(cloud_sync, "profile_changed", lambda username: None) + c = link_account(cloud, "ca_wes") + c.patch("/api/prefs/profile", json={"set": {"theme": "light", "language": "en"}}) + assert cloud_sync.sync_profile("ca_wes") == "pushed" + c.patch("/api/prefs/profile", json={"set": {"theme": "dark"}}) + raced = [] + + def racing(url, data=None, headers=None, method=None, timeout=None): + if method == "PUT" and not raced: # another server's push lands between this one's read and its push + raced.append(1) + _elsewhere(cloud, "ca_wes", {"theme": "light", "language": "zh"}) + return cloud.http(url, data=data, headers=headers, method=method, timeout=timeout) + monkeypatch.setattr(cloud_auth, "_http", racing) + assert cloud_sync.sync_profile("ca_wes") == "merged" + assert raced and _cloud_copy(cloud, "ca_wes") == {"theme": "dark", "language": "zh"} + assert get_profile("ca_wes")[0] == {"theme": "dark", "language": "zh"} def test_edit_right_after_a_pull_counts_as_newer(cloud, monkeypatch): @@ -588,7 +715,7 @@ def test_edit_right_after_a_pull_counts_as_newer(cloud, monkeypatch): assert set_profile("ca_pia", {"theme": "old"}, updated_at="2020-01-01T00:00:00.000000Z") == "2999-01-01T00:00:00.001000Z" assert get_profile("ca_pia")[0] == {"theme": "sepia"} # pushed with that time, the account server clamps it to its now, and this server keeps the clamped time - assert cloud_sync.push_profile("ca_pia") == "pushed" + assert cloud_sync.sync_profile("ca_pia") == "pushed" stored = cloud.prefs["sub-ca_pia"]["profile"]["updated_at"] assert stored < "2999" and get_profile("ca_pia")[1] == stored[:-1] + "000Z" assert cloud_sync.sync_profile("ca_pia") == "same" diff --git a/backend/tests/test_mirror.py b/backend/tests/test_mirror.py index 4ec9e1ff..674dc433 100644 --- a/backend/tests/test_mirror.py +++ b/backend/tests/test_mirror.py @@ -251,6 +251,36 @@ def test_a_file_missing_from_the_copy_is_fetched_again(): assert sync_engine.missing_uploads(local.ws) == set() +def test_rounds_reuse_whoami_until_one_fails(monkeypatch): + """A round trusts the last round's whoami (WHOAMI_TTL_S); a round that + fails forgets it, so the next one asks again.""" + remote, local, _ = _pair() + calls = [] + inner = sync_engine.default_fetch + fail = [False] + + def fetch(method, path, body, headers): + calls.append(path.split("?")[0]) + if fail[0] and path.startswith("/api/sync/changes"): + fail[0] = False + return 503, b'{"detail": "down for a moment"}' + return inner(method, path, body, headers) + + monkeypatch.setattr(sync_engine, "default_fetch", fetch) + _sync(local) + _sync(local) + assert calls.count("/api/sync/whoami") == 1 and calls.count("/api/sync/changes") == 2 + fail[0] = True + r = local.client.post(f"/api/mirrors/{local.ws}/sync?wait=1") + assert "down for a moment" in r.json()["status"]["last_error"] + calls.clear() + _sync(local) + assert calls.count("/api/sync/whoami") == 1 + # the pill's conflict fingerprint rides on the mirror's answer + info = local.client.get(f"/api/mirrors/{local.ws}").json() + assert info["conflicts_open"] == 0 and info["conflicts_newest"] == 0 + + def test_pull_only_with_a_read_token(): remote, local, mirror = _pair(scope="read") assert mirror["mode"] == "pull" @@ -550,3 +580,118 @@ def test_switching_back_to_two_way_pushes_what_receive_only_kept(): _sync(local) # two-way again: the page is looked at once more and the edit goes out assert remote.texts(page["id"])["kh1"] == "text (local)" assert local.client.get(f"/api/mirrors/{local.ws}").json()["pending_local"] is False + + +def _mid_round(monkeypatch, action): + """Run ``action`` once, from inside the next round (just before its + first page is reconciled), as a person would from the UI meanwhile.""" + real = sync_engine._sync_page + done = [False] + + def hooked(*args, **kwargs): + if not done[0]: + done[0] = True + action() + return real(*args, **kwargs) + + monkeypatch.setattr(sync_engine, "_sync_page", hooked) + + +def test_a_detach_during_a_round_is_kept_with_its_direction(monkeypatch): + remote, local, _ = _pair() + page = remote.page("Busy detach") + remote.insert(page["id"], "bd1", "one") + _sync(local) + local.client.patch(f"/api/mirrors/{local.ws}", json={"mode": "pull"}).raise_for_status() + remote.insert(page["id"], "bd2", "two") + other = remote.page("Second page") + _mid_round(monkeypatch, lambda: local.client.post(f"/api/mirrors/{local.ws}/detach").raise_for_status()) + st = local.client.post(f"/api/mirrors/{local.ws}/sync?wait=1").json()["status"] + info = local.client.get(f"/api/mirrors/{local.ws}").json() + assert info["mode"] == "off" and info["status"]["detached_mode"] == "pull" and not st.get("running") + assert set(st.get("retry") or {}) & {page["id"], other["id"]}, "the pages left over wait for the reattach" + # reattached: the direction it had, and the rest of that round's work + assert local.client.post(f"/api/mirrors/{local.ws}/relink", json={}).json()["mode"] == "pull" + _sync(local) + assert local.texts(page["id"]) == {"bd1": "one", "bd2": "two"} and other["id"] in local.pages() + + +def test_a_force_asked_for_during_a_round_runs_as_asked(monkeypatch): + """A force push pressed while a round runs is not turned into a pull by + that round's bookkeeping: it waits for the next round and replaces the + original with the copy.""" + remote, local, _ = _pair() + page = remote.page("Busy force") + remote.insert(page["id"], "bf1", "original") + _sync(local) + local.ops(page["id"], [{"op": "set", "id": "bf1", "content": "the copy's text"}]) + remote.insert(page["id"], "bf2", "added there") + _mid_round(monkeypatch, lambda: local.client.post(f"/api/mirrors/{local.ws}/force", json={"direction": "push"}).raise_for_status()) + st = local.client.post(f"/api/mirrors/{local.ws}/sync?wait=1").json()["status"] + assert st.get("force") == "push", "the force waits for a round of its own" + _sync(local) + assert remote.texts(page["id"])["bf1"] == "the copy's text" == local.texts(page["id"])["bf1"] + st = local.client.get(f"/api/mirrors/{local.ws}").json()["status"] + assert "force" not in st and "adopt" not in st and "prune" not in st + + +def test_edits_made_while_the_remote_was_read_only_are_pushed_later(monkeypatch): + """The remote demotes the account for a while (a viewer): rounds pull + only, and the edits made here meanwhile go out once writing is allowed + again — the pill says they are pending until then.""" + remote, local, _ = _pair() + page = remote.page("Demoted") + remote.insert(page["id"], "dm1", "text") + quiet = remote.page("Untouched afterwards") + remote.insert(quiet["id"], "dm2", "text") + _sync(local) + real = sync_engine.whoami + monkeypatch.setattr(sync_engine, "whoami", lambda r: {**real(r), "role": "viewer"}) + sync_engine._whoami_seen.clear() # a round trusts the last answer for a while; the demotion is seen now + local.ops(quiet["id"], [{"op": "set", "id": "dm2", "content": "text (edited while a viewer)"}]) + st = local.client.post(f"/api/mirrors/{local.ws}/sync?wait=1").json()["status"] + assert "read-only" in st["last_error"] and st["mode"] == "pull" + assert remote.texts(quiet["id"])["dm2"] == "text" + assert local.client.get(f"/api/mirrors/{local.ws}").json()["pending_local"] is True + monkeypatch.setattr(sync_engine, "whoami", real) + st = _sync(local) + assert st["mode"] == "two-way" and remote.texts(quiet["id"])["dm2"] == "text (edited while a viewer)" + assert local.client.get(f"/api/mirrors/{local.ws}").json()["pending_local"] is False + + +def test_force_pull_on_a_receive_only_clone_removes_its_own_pages(monkeypatch): + remote, local, _ = _pair(mode="pull") + page = remote.page("Origin's") + _sync(local) + extra = local.page("Only in the clone") + monkeypatch.setattr(sync_engine, "sync_in_background", lambda ws: sync_engine.sync_workspace(ws)) + assert local.client.post(f"/api/mirrors/{local.ws}/force", json={"direction": "pull"}).status_code == 200 + assert set(local.pages()) == {page["id"]} and extra["id"] not in remote.pages() + + +def test_the_direction_of_a_detached_clone_cannot_be_changed(): + remote, local, _ = _pair() + local.client.post(f"/api/mirrors/{local.ws}/detach").raise_for_status() + r = local.client.patch(f"/api/mirrors/{local.ws}", json={"mode": "pull"}) + assert r.status_code == 400 and "detached" in r.json()["detail"] + assert local.client.get(f"/api/mirrors/{local.ws}").json()["mode"] == "off" + assert local.client.patch(f"/api/mirrors/{local.ws}", json={"poll_s": 5}).status_code == 200 + + +def test_resolving_a_conflict_writes_into_the_page_the_block_is_in_now(): + remote, local, _ = _pair() + page = remote.page("Conflict page") + other = remote.page("Other page") + remote.insert(page["id"], "rc1", "alpha beta") + _sync(local) + local.ops(page["id"], [{"op": "set", "id": "rc1", "content": "ALPHA beta"}]) + remote.ops(page["id"], [{"op": "set", "id": "rc1", "content": "alpha BETA"}]) + _sync(local) + c = local.client.get(f"/api/mirrors/{local.ws}/conflicts").json()["conflicts"] + assert [x["kind"] for x in c] == ["merged"] + # the block moves to another page here before the conflict is looked at + assert local.client.post("/api/blocks/rc1/reorder", json={"parent_id": other["id"]}).status_code == 200 + r = local.client.post(f"/api/mirrors/{local.ws}/conflicts/{c[0]['id']}", json={"choice": "mine"}) + assert r.status_code == 200, r.text + assert local.texts(other["id"])["rc1"] == "ALPHA beta" + assert local.client.get(f"/api/mirrors/{local.ws}").json()["conflicts_open"] == 0 diff --git a/backend/tests/test_mirror_edges.py b/backend/tests/test_mirror_edges.py index aa633680..e9b01f01 100644 --- a/backend/tests/test_mirror_edges.py +++ b/backend/tests/test_mirror_edges.py @@ -263,3 +263,28 @@ def down(method, path, body, headers): _sync(local) assert remote.texts(page["id"])["o1"] == "text (typed on the train)" assert local.client.get(f"/api/mirrors/{local.ws}").json()["pending_local"] is False + + +def test_a_block_moved_out_of_a_subtree_deleted_here_survives(): + """The remote moves a block (with its own child) out of a section, edits + it; this copy deletes the section meanwhile. The moved block is no part + of that deletion any more: it comes back here where the remote put it, + the section stays deleted, and nothing is deleted on the remote.""" + remote, local, _ = _pair() + page = remote.page("Moved out") + remote.insert(page["id"], "mo_x", "section X", position="a0") + remote.insert(page["id"], "mo_y", "section Y", position="a1") + remote.insert(page["id"], "mo_c", "child of X", parent="mo_x") + remote.insert(page["id"], "mo_g", "grandchild", parent="mo_c") + _sync(local) + local.ops(page["id"], [{"op": "delete", "id": "mo_x"}]) + remote.ops(page["id"], [{"op": "move", "id": "mo_c", "parent": "mo_y", "position": "a0"}, + {"op": "set", "id": "mo_c", "content": "child of X, edited there"}]) + _sync(local) + t = same_tree(remote, local, page["id"]) + assert "mo_x" not in t, "the section deleted here stays deleted" + assert t["mo_c"] == {"parent": "mo_y", "content": "child of X, edited there", "props": {}} + assert t["mo_g"]["parent"] == "mo_c" + assert [(c["kind"], c["block_id"]) for c in conflicts(local)] == [("restored_remote_edit", "mo_c")] + _sync(local) + same_tree(remote, local, page["id"]) diff --git a/backend/tests/test_notices.py b/backend/tests/test_notices.py index 91e1b430..a1a019af 100644 --- a/backend/tests/test_notices.py +++ b/backend/tests/test_notices.py @@ -94,3 +94,103 @@ def test_strongest_first_and_bad_acks(nadmin, monkeypatch): nadmin.post(f"/api/notices/{n['id']}/seen", json={"fingerprint": n["fingerprint"]}) assert _ids(nadmin) == [] assert set(notices.seen_map("nadmin")) == {"update", "log-errors"} + + +# --- the account sources (their helpers stubbed: each is a plain read) ------ + +def _only(client, notice_id): + found = [n for n in client.get("/api/notices").json()["notices"] if n["id"] == notice_id] + return found[0] if found else None + + +def test_backup_failed_until_seen_and_again_on_the_next_failure(nuser, monkeypatch): + tasks = [{"id": "a" * 32, "name": "Nightly", "state": "finished", "last_run": "2026-09-20T01:00:00"}] + monkeypatch.setattr(notices.backup_schedule, "list_tasks", lambda owner: tasks if owner == "nuser" else []) + assert _only(nuser, "backup-failed") is None + tasks[0].update(state="failed", last_run="2026-09-21T01:00:00", last_error="disk full") + notice = _only(nuser, "backup-failed") + assert notice["tone"] == "error" and notice["pane"] == "backups" and 'task "Nightly" failed' in notice["title"] + nuser.post("/api/notices/backup-failed/seen", json={"fingerprint": notice["fingerprint"]}) + assert _only(nuser, "backup-failed") is None + tasks[0]["last_run"] = "2026-09-22T01:00:00" # failed again + assert _only(nuser, "backup-failed")["fingerprint"] != notice["fingerprint"] + tasks.append({"id": "b" * 32, "name": "Weekly", "state": "failed", "last_run": "2026-09-22T02:00:00"}) + assert _only(nuser, "backup-failed")["title"] == "2 backup tasks failed" + + +def test_mirror_conflicts_count_new_ones_only(nuser, monkeypatch): + marks = {"ws-clone": (0, 0)} + monkeypatch.setattr(notices.sync_engine, "list_mirrors", lambda owner: [{"workspace_id": "ws-clone"}] if owner == "nuser" else []) + monkeypatch.setattr(notices.sync_engine, "open_conflict_mark", lambda ws: marks[ws]) + assert _only(nuser, "mirror-conflicts") is None + marks["ws-clone"] = (3, 7) + notice = _only(nuser, "mirror-conflicts") + assert notice["pane"] == "account" and notice["tone"] == "warn" and notice["title"].startswith("3 sync conflicts") + nuser.post("/api/notices/mirror-conflicts/seen", json={"fingerprint": notice["fingerprint"]}) + marks["ws-clone"] = (3, 7) + assert _only(nuser, "mirror-conflicts") is None + marks["ws-clone"] = (2, 8) # one resolved, one new + assert _only(nuser, "mirror-conflicts")["title"].startswith("2 sync conflicts") + + +def test_publication_conflicts_point_at_the_sync_pane(nuser, monkeypatch): + mirrors = [{"workspace_id": "ws-clone"}, {"workspace_id": "ws-pub", "page_filter": ["p1"]}] + marks = {"ws-clone": (0, 0), "ws-pub": (2, 5)} + monkeypatch.setattr(notices.sync_engine, "list_mirrors", lambda owner: mirrors if owner == "nuser" else []) + monkeypatch.setattr(notices.sync_engine, "open_conflict_mark", lambda ws: marks[ws]) + assert _only(nuser, "mirror-conflicts") is None + notice = _only(nuser, "publish-conflicts") + assert notice["pane"] == "account" and notice["title"].startswith("2 sync conflicts") + marks["ws-clone"] = (1, 9) + assert _only(nuser, "mirror-conflicts")["title"].startswith("1 sync conflict ") + + +def test_cloud_sync_error_names_the_reason(nuser, monkeypatch): + status = {"state": "off", "at": "", "error": ""} + monkeypatch.setattr(notices.cloud_sync, "profile_status", lambda username: status) + assert _only(nuser, "cloud-sync") is None + status.update(state="error", at="2026-09-24T10:00:00Z", error="Gamma Cloud could not be reached.") + notice = _only(nuser, "cloud-sync") + assert notice["pane"] == "account" and notice["title"] == "Gamma Cloud sync failed: Gamma Cloud could not be reached" + assert notice["fingerprint"] == "2026-09-24T10:00:00Z" + status.update(state="synced") + assert _only(nuser, "cloud-sync") is None + + +def test_storage_thresholds_and_the_remembered_walk(nuser, monkeypatch): + limits = {"quota_mb": 0} + walks = [] + + def usage(username): + walks.append(username) + return used[0] + used = [0] + monkeypatch.setattr(notices.server_settings, "user_limits", lambda username: dict(limits)) + monkeypatch.setattr(notices.server_settings, "usage_bytes", usage) + notices.forget_usage() + assert _only(nuser, "storage") is None and walks == [] # no quota: no walk at all + limits["quota_mb"] = 100 + used[0] = 50 * notices.MB + assert _only(nuser, "storage") is None and walks == ["nuser"] + nuser.get("/api/notices") + assert walks == ["nuser"] # remembered + notices.forget_usage("nuser") + used[0] = 95 * notices.MB + notice = _only(nuser, "storage") + assert notice["tone"] == "warn" and notice["fingerprint"] == "90" and "95 of 100 MB" in notice["title"] + nuser.post("/api/notices/storage/seen", json={"fingerprint": "90"}) + assert _only(nuser, "storage") is None + notices.forget_usage() + used[0] = 100 * notices.MB + notice = _only(nuser, "storage") + assert notice["tone"] == "error" and notice["fingerprint"] == "full" + + +def test_many_clones_with_conflicts_still_fit_one_fingerprint(nuser, monkeypatch): + mirrors = [{"workspace_id": f"ws-clone-{i:02d}"} for i in range(15)] + monkeypatch.setattr(notices.sync_engine, "list_mirrors", lambda owner: mirrors if owner == "nuser" else []) + monkeypatch.setattr(notices.sync_engine, "open_conflict_mark", lambda ws: (12, 345)) + notice = _only(nuser, "mirror-conflicts") + assert notice["title"].startswith("180 sync conflicts") and len(notice["fingerprint"]) <= 32 + assert nuser.post("/api/notices/mirror-conflicts/seen", json={"fingerprint": notice["fingerprint"]}).status_code == 200 + assert _only(nuser, "mirror-conflicts") is None diff --git a/backend/tests/test_prefs_ai_settings.py b/backend/tests/test_prefs_ai_settings.py index 397512e9..93da15ee 100644 --- a/backend/tests/test_prefs_ai_settings.py +++ b/backend/tests/test_prefs_ai_settings.py @@ -70,11 +70,28 @@ def test_profile_is_one_account_wide_object(alice): # set_profile goes through set_pref (last write wins, a newer updated_at) later = db.set_profile("prefs_alice", {"theme": "gray"}) assert later > body["updated_at"] - assert alice.get("/api/prefs/profile").json() == {"key": "profile", "value": {"theme": "gray"}, "updated_at": later} + assert alice.get("/api/prefs/profile").json() == {"key": "profile", "cloud_choice": False, "value": {"theme": "gray"}, + "updated_at": later} with pytest.raises(ValueError): db.set_profile("prefs_alice", ["not", "an", "object"]) +def test_profile_patch_sets_only_the_named_entries(alice): + # the web app's save: the entries it changed, every other one kept as stored + from gamma import db + db.set_profile("prefs_alice", {"theme": "dark", "language": "en"}) + before = db.get_profile("prefs_alice")[1] + r = alice.patch("/api/prefs/profile", json={"set": {"language": "zh", "enterNewNote": True}}) + assert r.status_code == 200 + assert r.json()["value"] == {"theme": "dark", "language": "zh", "enterNewNote": True} + assert r.json()["updated_at"] > before + assert db.get_profile("prefs_alice") == (r.json()["value"], r.json()["updated_at"]) + assert alice.patch("/api/prefs/profile", json={"set": {"chatSystem": "x" * (70 * 1024)}}).status_code == 413 + assert alice.patch("/api/prefs/profile", json={"set": ["theme"]}).status_code == 422 + # the cloud sync's merge base is not a pref the generic endpoints serve + assert alice.get("/api/prefs/profile-base").status_code == 400 + + def test_profile_must_be_an_object_within_the_size_cap(alice): from gamma import db assert db.get_profile("prefs_nobody") == ({}, "") @@ -166,7 +183,7 @@ def test_deepseek_service_preset(alice, monkeypatch): # DeepSeek is offered as a named service: the openai protocol at its # endpoint. Such an entry is labelled DeepSeek, and its live model list is # not narrowed to OpenAI's gpt-/o-families. - import gamma.routers.ai as ai_mod + from gamma import ai_catalog g = alice.get("/api/ai/settings").json() svc = next(s for s in g["services"] if s["id"] == "deepseek") @@ -183,7 +200,7 @@ def test_deepseek_service_preset(alice, monkeypatch): assert any(m["model"] == "deepseek-flash" and m["provider_name"] == "DeepSeek" for m in models) seen = {} - monkeypatch.setattr(ai_mod, "_model_catalog_json", lambda req: seen.update(url=req.full_url) or { + monkeypatch.setattr(ai_catalog, "fetch_json", lambda req: seen.update(url=req.full_url) or { "data": [{"id": "deepseek-flash"}, {"id": "deepseek-v4-pro"}, {"id": "deepseek-embed"}]}) r = alice.post("/api/ai/model-catalog", json={"provider_id": entry["id"]}) assert r.status_code == 200, r.text @@ -322,25 +339,25 @@ def test_ai_health_ping_checks_credential_for_free(alice, monkeypatch): # "ping" mode never runs a completion: API keys are checked via the # provider's model listing; 401 comes back as a broken-credential flag, # 404 (gateway without /v1/models) as ok-but-unverified. - import gamma.routers.ai as ai_mod + from gamma import ai_catalog pid = alice.get("/api/ai/settings").json()["providers"][0]["id"] seen = {} - monkeypatch.setattr(ai_mod, "_model_catalog_json", lambda req: seen.update(url=req.full_url) or {}) + monkeypatch.setattr(ai_catalog, "fetch_json", lambda req: seen.update(url=req.full_url) or {}) body = alice.post("/api/ai/health", json={"provider_id": pid, "mode": "ping"}).json() assert body["configured"] and body["ok"] is True assert "/v1/models" in seen["url"] def dead_key(req): raise _http_error(401, '{"error": {"message": "invalid x-api-key"}}') - monkeypatch.setattr(ai_mod, "_model_catalog_json", dead_key) + monkeypatch.setattr(ai_catalog, "fetch_json", dead_key) body = alice.post("/api/ai/health", json={"provider_id": pid, "mode": "ping"}).json() assert body["ok"] is False and body["auth"] is True assert "invalid x-api-key" in body["error"] def no_listing(req): raise _http_error(404, "") - monkeypatch.setattr(ai_mod, "_model_catalog_json", no_listing) + monkeypatch.setattr(ai_catalog, "fetch_json", no_listing) body = alice.post("/api/ai/health", json={"provider_id": pid, "mode": "ping"}).json() assert body["ok"] is True and body["unverified"] is True @@ -491,6 +508,7 @@ def fake_call(messages, system, entry, rt, **kw): def test_admin_tests_and_lists_models_of_a_shared_entry(admin, shared, monkeypatch): import gamma.routers.ai as ai_mod + from gamma import ai_catalog sid = shared["id"] seen = {} monkeypatch.setattr(ai_mod, "_call_ai", lambda m, s, entry, rt, **kw: seen.update(entry) or "ok") @@ -500,7 +518,7 @@ def test_admin_tests_and_lists_models_of_a_shared_entry(admin, shared, monkeypat def listing(req): seen.update(url=req.full_url, auth=req.headers.get("Authorization")) return {"data": [{"id": "lab-model"}, {"id": "lab-new"}]} - monkeypatch.setattr(ai_mod, "_model_catalog_json", listing) + monkeypatch.setattr(ai_catalog, "fetch_json", listing) r = admin.post("/api/ai/model-catalog", json={"provider_id": sid}) assert r.status_code == 200 and r.json()["models"] == ["lab-model", "lab-new"] assert seen["url"] == "https://llm.example.org/v1/models" and seen["auth"] == f"Bearer {SHARED_KEY}" diff --git a/backend/tests/test_publish.py b/backend/tests/test_publish.py index 0351df28..bf3fe31a 100644 --- a/backend/tests/test_publish.py +++ b/backend/tests/test_publish.py @@ -6,6 +6,9 @@ test_cloud_auth.py grown a /userinfo and a share host address.""" import io +import threading +import json +from pathlib import Path import pytest from fastapi.testclient import TestClient @@ -335,6 +338,8 @@ def offline(*_): r = local.delete(f"/api/pages/{page['id']}/publish") assert r.status_code == 200, r.text assert r.json()["published"] is False and r.json()["mirror"]["page_filter"] == [] + # nothing published: the workspace no longer reads as publishing (no header pill) + assert next(w for w in local.get("/api/workspaces/mine").json()["workspaces"] if w["id"] == local_ws)["publishing"] is False assert anonymous().get(f"/api/share/{token}").status_code == 404 assert page["id"] not in page_ids(host) assert texts(local, page["id"]) == {"pb_n1": "my note, edited there"} @@ -384,3 +389,231 @@ def test_publish_needs_a_cloud_identity_and_a_home_of_its_own(publishing, monkey insert(local, page["id"], "pb_blk", "x") monkeypatch.setattr(publish, "publishing_blocked", lambda: False) assert local.post("/api/pages/pb_blk/publish").status_code == 400 + + +# --- the plan's page cap (the share host) ---------------------------------------------- + +CAP_DETAIL = "Free plan: up to 5 published pages. Unpublish one, or upgrade your Gamma Cloud plan." + + +def test_the_share_host_caps_published_pages_by_plan(publishing, monkeypatch): + local, local_ws = linked(publishing, "pbcap") + host_ws = workspaces.default_workspace("pbcap") + host = bound(login("pbcap", "pw"), host_ws) + pages = [local.post("/api/pages", json={"title": f"Paper {n}"}).json() for n in range(7)] + for page in pages[:5]: + r = local.post(f"/api/pages/{page['id']}/publish") + assert r.status_code == 200, r.text + assert len(page_ids(host)) == 5 + # the local workspace is not the one the exchange publishes into: no cap there + assert len(page_ids(local)) == 7 + + # the state reads the count there + state = local.get(f"/api/pages/{pages[5]['id']}/publish").json() + assert state["limit"] == {"used": 5, "max": 5, "plan": "free"} and state["can_publish"] is True + assert anonymous().get("/api/publish/limit").status_code == 401 + assert host.get("/api/publish/limit").json() == {"used": 5, "max": 5, "plan": "free"} + + # the sixth: 409 here with the share host's words and the count; nothing moved, nothing left in the filter + r = local.post(f"/api/pages/{pages[5]['id']}/publish") + assert r.status_code == 409, r.text + assert r.json() == {"detail": CAP_DETAIL, "limit": {"used": 5, "max": 5, "plan": "free"}} + assert pages[5]["id"] not in sync_engine.get_mirror(local_ws)["page_filter"] + assert pages[5]["id"] not in page_ids(host) + # the count unreadable beforehand (a share host that fills up meanwhile): the round's 402 says the same + read_limit = publish._read_limit + monkeypatch.setattr(publish, "_read_limit", lambda remote: None) + r = local.post(f"/api/pages/{pages[5]['id']}/publish") + assert r.status_code == 409 and r.json()["detail"] == CAP_DETAIL + assert pages[5]["id"] not in sync_engine.get_mirror(local_ws)["page_filter"] + assert not sync_engine.get_mirror(local_ws)["status"]["last_error"] + monkeypatch.setattr(publish, "_read_limit", read_limit) + # ...and 402 on the share host itself, for any new page + r = host.post("/api/pages", json={"title": "One more"}) + assert r.status_code == 402 + assert r.json() == {"detail": CAP_DETAIL, "limit": 5, "used": 5, "plan": "free"} + # a page that is already there answers as before (a round re-creating it tolerates the 409) + assert host.post("/api/pages", json={"id": pages[0]["id"], "title": "x"}).status_code == 409 + # the cap never stops a round of a page already published + insert(local, pages[0]["id"], "pbcap_n1", "an edit at the cap") + sync(local_ws) + assert texts(host, pages[0]["id"]) == {"pbcap_n1": "an edit at the cap"} + + # unpublishing one frees a slot + assert local.delete(f"/api/pages/{pages[1]['id']}/publish").status_code == 200 + r = local.post(f"/api/pages/{pages[5]['id']}/publish") + assert r.status_code == 200, r.text + assert len(page_ids(host)) == 5 + + # an upgrade counts at the next publish: the full workspace is exchanged again, the plan read afresh + publishing.people["sub-pbcap"]["plan"] = "plus" + r = local.post(f"/api/pages/{pages[6]['id']}/publish") + assert r.status_code == 200, r.text + assert len(page_ids(host)) == 6 + assert host.get("/api/publish/limit").json() == {"used": 6, "max": None, "plan": "plus"} + assert host.post("/api/pages", json={"title": "Seventh"}).status_code == 200 + + # GAMMA_FREE_PAGE_LIMIT sets the free plan's number + publishing.people["sub-pbcap"]["plan"] = "free" + with connect_users_db() as conn: + cloud_auth.link(conn, "pbcap", publishing.people["sub-pbcap"]) + conn.commit() + monkeypatch.setenv("GAMMA_FREE_PAGE_LIMIT", "10") + assert host.get("/api/publish/limit").json() == {"used": 7, "max": 10, "plan": "free"} + monkeypatch.setenv("GAMMA_FREE_PAGE_LIMIT", "0") + assert host.get("/api/publish/limit").json()["max"] is None + monkeypatch.delenv("GAMMA_FREE_PAGE_LIMIT") + + # a server that is not a share host ignores the table + monkeypatch.setenv("GAMMA_CLOUD_SHARE_HOST", "0") + assert host.post("/api/pages", json={"title": "Not a share host"}).status_code == 200 + assert host.get("/api/publish/limit").status_code == 404 + + +# --- public addresses: slugs and page hosts ------------------------------------------------ + +SLUGS = json.loads((Path(__file__).resolve().parents[2] / "tests" / "shared" / "slug.json").read_text(encoding="utf-8")) + + +@pytest.mark.parametrize("case", SLUGS["slug"], ids=[c["note"] for c in SLUGS["slug"]]) +def test_slug(case): + assert publish.slug(case["input"]) == case["output"] + + +def test_the_page_host_pattern(monkeypatch): + pattern = "{username}-pages.gammapdf.com" + assert publish.page_host_user(pattern, "tim-pages.gammapdf.com") == "tim" + assert publish.page_host_user(pattern, "Tim-Pages.GammaPDF.com:8443") == "tim" + assert publish.page_host_user(pattern, "a-pages-pages.gammapdf.com") == "a-pages" + for other in ("api.gammapdf.com", "x.tim-pages.gammapdf.com", "-pages.gammapdf.com", "tim-pages.gammapdf.com.evil"): + assert publish.page_host_user(pattern, other) == "", other + assert publish.public_url("https://share.gammapdf.com", pattern, "Tim", "Élan vital", "p-1") \ + == "https://tim-pages.gammapdf.com/elan-vital-p-1" + assert publish.public_url("http://127.0.0.1:9001", pattern, "tim", "量子", "abc") \ + == "http://tim-pages.gammapdf.com:9001/abc" + assert publish.public_url("https://share.gammapdf.com", pattern, "tim_x", "t", "abc") == "" + assert publish.public_url("https://share.gammapdf.com", "", "tim", "t", "abc") == "" + # checked at startup + for good in ("", "{username}-pages.gammapdf.com", "{username}.pages.example.org"): + monkeypatch.setenv("GAMMA_PAGE_HOST", good) + publish.check_config() + for bad in ("pages.gammapdf.com", "{username}-{username}.x.org", "https://{username}.x.org", "{username}.x.org:8443", + "{username}_pages.x.org", "{username}"): + monkeypatch.setenv("GAMMA_PAGE_HOST", bad) + with pytest.raises(ValueError): + publish.check_config() + monkeypatch.delenv("GAMMA_PAGE_HOST") + monkeypatch.setenv("GAMMA_FREE_PAGE_LIMIT", "five") + with pytest.raises(ValueError): + publish.check_config() + + +def resolve(host, path): + return anonymous().get("/api/pages/resolve-public", params={"host": host, "path": path}) + + +def test_the_resolver_opens_a_shared_page_by_its_pretty_address(monkeypatch): + monkeypatch.setenv("GAMMA_PAGE_HOST", "{username}-pages.example.org") + ws = make_user("pbresolve", "pw") + make_user("pbresolve2", "pw") + owner = bound(login("pbresolve", "pw"), ws) + page = owner.post("/api/pages", json={"title": "My paper"}).json() + dashed = owner.post("/api/pages", json={"id": "ab-cd-ef", "title": "Dashed id"}).json() + unshared = owner.post("/api/pages", json={"title": "Private"}).json() + insert(owner, page["id"], "pbresolve_b1", "a block") + token = owner.post(f"/api/share/{page['id']}").json()["token"] + dashed_token = owner.post(f"/api/share/{dashed['id']}").json()["token"] + assert anonymous().get("/api/server-config").json()["page_host"] == "{username}-pages.example.org" + + want = {"share": token, "page_id": page["id"]} + host = "pbresolve-pages.example.org" + assert resolve(host, f"/my-paper-{page['id']}").json() == want + assert resolve(host, f"/{page['id']}").json() == want # no slug + assert resolve(host, f"/renamed-since-{page['id']}/").json() == want # the slug is decoration + assert resolve("PBResolve-Pages.example.org:8443", f"/x-{page['id']}").json() == want + assert resolve(host, "/dashed-id-ab-cd-ef").json() == {"share": dashed_token, "page_id": "ab-cd-ef"} + assert resolve(host, "/ab-cd-ef").json()["page_id"] == "ab-cd-ef" + for miss_host, path in ( + ("pbresolve2-pages.example.org", f"/my-paper-{page['id']}"), # another account's host + ("nobody-pages.example.org", f"/{page['id']}"), # no such account + ("pbresolve.example.org", f"/{page['id']}"), # not a page host + (host, f"/private-{unshared['id']}"), # not shared + (host, "/pbresolve_b1"), # not a page + (host, "/"), (host, f"/a/{page['id']}"), + ): + assert resolve(miss_host, path).status_code == 404, (miss_host, path) + # only the account's default personal workspace is served + other_ws = workspaces.create("Second", "pbresolve")["id"] + second = bound(login("pbresolve", "pw"), other_ws) + other = second.post("/api/pages", json={"title": "Elsewhere"}).json() + second.post(f"/api/share/{other['id']}") + assert resolve(host, f"/elsewhere-{other['id']}").status_code == 404 + + # the share's own audience still rules: signed-in only is the share view's 401 + owner.put(f"/api/share-settings/{page['id']}", json={"audience": "users"}) + assert resolve(host, f"/my-paper-{page['id']}").json() == want + assert anonymous().get(f"/api/share/{token}").status_code == 401 + # the page host off: nothing resolves + monkeypatch.delenv("GAMMA_PAGE_HOST") + assert resolve(host, f"/my-paper-{page['id']}").status_code == 404 + assert anonymous().get("/api/server-config").json()["page_host"] == "" + + +def test_publish_answers_carry_the_public_address(publishing, monkeypatch): + monkeypatch.setenv("GAMMA_PAGE_HOST", "{username}-pages.example.org") + local, local_ws = linked(publishing, "pbpretty") + page = local.post("/api/pages", json={"title": "Café Notes: Part 1"}).json() + out = local.post(f"/api/pages/{page['id']}/publish").json() + token = out["share"]["token"] + assert out["url"] == f"{HOST}/?share={token}" + assert out["public_url"] == f"http://pbpretty-pages.example.org/cafe-notes-part-1-{page['id']}" + state = local.get(f"/api/pages/{page['id']}/publish").json() + assert state["public_url"] == out["public_url"] and state["url"] == out["url"] + assert resolve("pbpretty-pages.example.org", f"/cafe-notes-part-1-{page['id']}").json() \ + == {"share": token, "page_id": page["id"]} + # without page hosts the public address is the token link + monkeypatch.delenv("GAMMA_PAGE_HOST") + assert local.get(f"/api/pages/{page['id']}/publish").json()["public_url"] == out["url"] + + +def test_a_round_queued_behind_an_unpublish_sees_the_page_gone(publishing, monkeypatch): + """A round waiting for the round lock while a page is unpublished reads + the mirror once it holds the lock: the page is out of the filter, so it + is not pushed back to the share host as an unshared copy.""" + local, local_ws = linked(publishing, "pb_queue") + host = bound(login("pb_queue", "pw"), workspaces.default_workspace("pb_queue")) + page = local.post("/api/pages", json={"title": "Queued"}).json() + assert local.post(f"/api/pages/{page['id']}/publish").status_code == 200 + assert page["id"] in page_ids(host) + # the queued round: it takes the lock right after the unpublish did its work + lock = threading.RLock() + monkeypatch.setitem(sync_engine._locks, local_ws, lock) + first = [True] + + class Queued: + def __enter__(self): + lock.acquire() + if first[0]: + first[0] = False + publish.unpublish("pb_queue", local_ws, page["id"]) # in this thread: the lock is re-entrant here + + def __exit__(self, *exc): + lock.release() + + real = sync_engine._lock + monkeypatch.setattr(sync_engine, "_lock", lambda ws: Queued() if ws == local_ws else real(ws)) + st = sync_engine.sync_workspace(local_ws) + assert not st.get("last_error"), st + assert page["id"] not in page_ids(host), "the copy there was not re-created by the queued round" + assert sync_engine.get_mirror(local_ws)["page_filter"] == [] + assert local.get(f"/api/pages/{page['id']}/publish").json()["published"] is False + + +def test_the_publish_state_names_a_detached_publication(publishing): + local, local_ws = linked(publishing, "pb_detach") + page = local.post("/api/pages", json={"title": "Detached pub"}).json() + assert local.post(f"/api/pages/{page['id']}/publish").status_code == 200 + assert local.post(f"/api/mirrors/{local_ws}/detach").status_code == 200 + state = local.get(f"/api/pages/{page['id']}/publish").json() + assert state["published"] is True and state["mirror"]["detached"] is True and state["mirror"]["mode"] == "off" + assert state["can_publish"] is False and "detached" in state["reason"] diff --git a/cloud/deploy/.env.example b/cloud/deploy/.env.example index 9d70d644..3f1e34d1 100644 --- a/cloud/deploy/.env.example +++ b/cloud/deploy/.env.example @@ -32,6 +32,11 @@ GAMMA_CLOUD_GOOGLE_ONE_TAP=1 GAMMA_CLOUD_GITHUB_CLIENT_ID= GAMMA_CLOUD_GITHUB_CLIENT_SECRET= +# The free share host (the compose service `share`, settings in share.env): +# its address, handed to every Gamma server so "Publish" knows where to go. +# Empty = no share host, Publish is off everywhere. +GAMMA_CLOUD_SHARE_HOST_URL=https://share.gammapdf.com + # The built-in public OIDC client every local Gamma is (default gamma-desktop # from config.py; normally unchanged). GAMMA_CLOUD_DESKTOP_CLIENT_ID= diff --git a/cloud/deploy/Caddyfile b/cloud/deploy/Caddyfile index 2c613312..908be322 100644 --- a/cloud/deploy/Caddyfile +++ b/cloud/deploy/Caddyfile @@ -14,3 +14,29 @@ -Server } } + +# The share host and every account's page host (share.gammapdf.com and +# -pages.gammapdf.com) — one wildcard site. Named records such as +# account.gammapdf.com win over the wildcard, and the internal certificate +# covers *.gammapdf.com, so no DNS challenge is needed (Cloudflare "Full"). +# Anything else under the wildcard is a 404. +*.gammapdf.com { + tls internal + encode gzip + @pages header_regexp Host ^[a-z0-9-]+-pages\.gammapdf\.com$ + @share host share.gammapdf.com + handle @share { + reverse_proxy share:9001 + } + handle @pages { + reverse_proxy share:9001 + } + handle { + respond 404 + } + header { + Strict-Transport-Security "max-age=31536000" + X-Content-Type-Options nosniff + -Server + } +} diff --git a/cloud/deploy/README.md b/cloud/deploy/README.md index fe3e7741..6ad110e7 100644 --- a/cloud/deploy/README.md +++ b/cloud/deploy/README.md @@ -173,6 +173,39 @@ Restart the container after editing `.env` (`docker compose up -d`). page under the username), which makes that person its admin on first sign-in. +## The free share host + +The compose file also runs `share`: one Gamma in cloud mode +(`ghcr.io/tim4431/gamma`, pinned to an image tag) that holds every free +account's published pages and answers `share.gammapdf.com` and the page +hosts `-pages.gammapdf.com` ([docs/dev/cloud_accounts.md](../../docs/dev/cloud_accounts.md) +"The share host"). Setting it up once: + +1. A wildcard DNS record at Cloudflare: `*` → this host's address, proxied. + Named records (`account`) keep precedence; the universal certificate + covers the first-level wildcard, and the Caddyfile's `*.gammapdf.com` + site serves the internal certificate behind it (SSL mode "Full"). +2. The share host's client on the account server: + `docker compose exec account python manage.py create-client "Share host" share-host https://share.gammapdf.com/api/auth/cloud/callback` + (the secret is shown once). +3. `share.env` from `share.env.example`: the client id and secret, and + `GAMMA_CLOUD_ADMIN_SUBJECT` = your cloud account id, so your first sign-in + there makes you its admin. The image also seeds an `admin` account with a + random password printed once to the container's log while no account + exists; delete it or set its password from Settings → Users afterwards. +4. `GAMMA_CLOUD_SHARE_HOST_URL=https://share.gammapdf.com` in `.env`, then + `docker compose up -d` (the account server restarts with the new + variable, `share` starts) and `docker compose exec caddy caddy reload + --config /etc/caddy/Caddyfile` for the new site. +5. Check: `curl https://share.gammapdf.com/api/health` answers ok, + `https://account.gammapdf.com/.well-known/openid-configuration` shows + `gamma_share_host`, and from a linked desktop the share popover's + Gamma Cloud section offers Publish. + +The default storage quota per account on the share host is its Settings → +Server storage default; set it small. `share-data/` is the share host's +state (published pages and files) — back it up like `data/`. + ## Updating ```bash diff --git a/cloud/deploy/compose.yml b/cloud/deploy/compose.yml index 7cb156d2..5ac8b2b5 100644 --- a/cloud/deploy/compose.yml +++ b/cloud/deploy/compose.yml @@ -5,6 +5,11 @@ # directory on ./data — that # directory IS the secret (signing keys, token hashes): back it up # encrypted, never share it. +# share the free share host: a Gamma (ghcr.io/tim4431/gamma) in cloud +# mode holding every free account's published pages, its data on +# ./share-data and its settings in share.env (share.env.example). +# Reached only through caddy (no host port). Pinned to an image +# tag: bump it with the desktop release. # caddy TLS on the host's own 80/443 with its internal certificate # (`tls internal`; Cloudflare holds the public one), for a host # with a public address (a VPS). The DNS record for the hostname @@ -24,6 +29,15 @@ services: ports: - "127.0.0.1:9002:9002" + share: + image: ghcr.io/tim4431/gamma:sha-a0d31c6 + restart: unless-stopped + env_file: share.env + volumes: + - ./share-data:/data + depends_on: + - account + caddy: image: caddy:2-alpine restart: unless-stopped @@ -38,6 +52,7 @@ services: - caddy-config:/config depends_on: - account + - share volumes: caddy-data: diff --git a/cloud/deploy/share.env.example b/cloud/deploy/share.env.example new file mode 100644 index 00000000..c1a0799f --- /dev/null +++ b/cloud/deploy/share.env.example @@ -0,0 +1,16 @@ +# Copy to share.env next to compose.yml: the free share host's settings, a +# Gamma in cloud mode (docs/dev/cloud_accounts.md "The share host"). The +# client id and secret come from `manage.py create-client "Share host" +# share-host https://share.gammapdf.com/api/auth/cloud/callback` on the +# account server; the admin subject is the cloud account id (the `sub`) of +# the person who administers the share host. + +GAMMA_PUBLIC_URL=https://share.gammapdf.com +GAMMA_CLOUD_ISSUER=https://account.gammapdf.com +GAMMA_CLOUD_CLIENT_ID= +GAMMA_CLOUD_CLIENT_SECRET= +GAMMA_CLOUD_POLICY=provision +GAMMA_CLOUD_SHARE_HOST=1 +GAMMA_PAGE_HOST={username}-pages.gammapdf.com +GAMMA_CLOUD_ADMIN_SUBJECT= +# GAMMA_FREE_PAGE_LIMIT=5 diff --git a/cloud/gammacloud/accounts.py b/cloud/gammacloud/accounts.py index fd145dcd..afcf2392 100644 --- a/cloud/gammacloud/accounts.py +++ b/cloud/gammacloud/accounts.py @@ -310,15 +310,34 @@ def delete(conn, account_id: str, actor: str = "") -> None: audit(conn, "account.delete", account_id, actor or account_id) +def by_id_deleted(conn, account_id: str): + """A soft-deleted account still in its grace period, or None.""" + return conn.execute("SELECT * FROM accounts WHERE id = ? AND deleted_at IS NOT NULL", (account_id,)).fetchone() + + +def restore(conn, account_id: str, actor: str) -> None: + """Undo a soft delete within the grace period. The username and e-mail + were reserved, so nothing can have taken them. What the delete dropped + stays dropped: the person signs back in through a password reset, or + through Google/GitHub on the same e-mail, which links again.""" + conn.execute("UPDATE accounts SET deleted_at = NULL WHERE id = ?", (account_id,)) + audit(conn, "account.restore", account_id, actor) + + +def purge(conn, account_id: str, actor: str = "system") -> None: + """Remove an account and every row that references it (the audit log + keeps its history); its username and e-mail are free again.""" + for table in ("identities", "portal_sessions", "email_tokens", "grants", "access_tokens", "prefs", + "servers_linked"): + conn.execute(f"DELETE FROM {table} WHERE account_id = ?", (account_id,)) + conn.execute("DELETE FROM accounts WHERE id = ?", (account_id,)) + audit(conn, "account.purge", account_id, actor) + + def purge_deleted(conn, older_than_days: int) -> int: - """Remove accounts deleted more than N days ago and every row that - references them. Returns the count.""" + """Purge accounts deleted more than N days ago. Returns the count.""" cutoff = after(-older_than_days * 86400) rows = conn.execute("SELECT id FROM accounts WHERE deleted_at IS NOT NULL AND deleted_at <= ?", (cutoff,)).fetchall() for row in rows: - for table in ("identities", "portal_sessions", "email_tokens", "grants", "access_tokens", "prefs", - "servers_linked"): - conn.execute(f"DELETE FROM {table} WHERE account_id = ?", (row["id"],)) - conn.execute("DELETE FROM accounts WHERE id = ?", (row["id"],)) - audit(conn, "account.purge", row["id"], "system") + purge(conn, row["id"]) return len(rows) diff --git a/cloud/gammacloud/pages.py b/cloud/gammacloud/pages.py index fcb85235..ce48a133 100644 --- a/cloud/gammacloud/pages.py +++ b/cloud/gammacloud/pages.py @@ -670,12 +670,13 @@ def admin_page(account: dict) -> str: const PLANS = %s; let offset = 0, query = ''; document.querySelectorAll('.tabs button').forEach(b => b.onclick = () => { document.querySelectorAll('.tabs button').forEach(x => x.classList.toggle('on', x === b)); for (const t of ['accounts','invites','clients','audit']) document.getElementById('tab-' + t).hidden = t !== b.dataset.tab; if (b.dataset.tab !== 'accounts') load(b.dataset.tab); }); -function planSelect(a){ return ''; } +function planSelect(a){ return ''; } function accountRow(a){ const status = (a.deleted_at ? 'deleted ' : '') + (a.email_verified ? 'verified' : 'unverified') + (a.is_admin ? ' admin' : ''); return '' + esc(a.username) + '
' + esc(a.id) + '' + esc(a.email) + '' + planSelect(a) + '' + status + '' + esc(a.created_at.slice(0,10)) + '' - + ''; + + ''; } async function loadAccounts(reset){ if (reset) { offset = 0; document.getElementById('accounts').innerHTML = ''; } @@ -694,6 +695,8 @@ def admin_page(account: dict) -> str: else if (v === 'admin' || v === 'unadmin') await api('/api/admin/accounts/' + id, {is_admin: v === 'admin'}, 'PATCH'); else if (v === 'rename') { const u = prompt('New username (lowercase letters, digits, hyphens):'); if (!u) return; await api('/api/admin/accounts/' + id, {username: u}, 'PATCH'); } else if (v === 'delete') { if (!confirm('Delete this account? It is signed out everywhere and purged after the grace period.')) return; await api('/api/admin/accounts/' + id + '/delete', {}); } + else if (v === 'restore') { await api('/api/admin/accounts/' + id + '/restore', {}); alert('Restored. They sign back in with a password reset, or Google/GitHub on the same e-mail.'); } + else if (v === 'purge') { if (!confirm('Purge this account now? Its username and e-mail become free for a new account. This cannot be undone.')) return; await api('/api/admin/accounts/' + id + '/purge', {}); } else return; loadAccounts(true); } catch (e) { alert(e.message); } diff --git a/cloud/gammacloud/routers/admin.py b/cloud/gammacloud/routers/admin.py index 4850ae65..63f42ebd 100644 --- a/cloud/gammacloud/routers/admin.py +++ b/cloud/gammacloud/routers/admin.py @@ -110,6 +110,36 @@ def admin_delete(account_id: str, request: Request): return {"ok": True} +def _deleted(conn, account_id: str): + if accounts.by_id_deleted(conn, account_id): + return + if accounts.by_id(conn, account_id): + raise Problem(409, "The account is not deleted.") + raise HTTPException(404, "no such account") + + +@router.post("/accounts/{account_id}/restore") +def admin_restore(account_id: str, request: Request): + with closing(db.connect()) as conn: + admin = require_admin(conn, request) + _deleted(conn, account_id) + accounts.restore(conn, account_id, admin["id"]) + conn.commit() + return {"account": _row(accounts.by_id(conn, account_id))} + + +@router.post("/accounts/{account_id}/purge") +def admin_purge(account_id: str, request: Request): + """Only a deleted account: a live one is deleted first, so the purge + never skips the delete's sign-out.""" + with closing(db.connect()) as conn: + admin = require_admin(conn, request) + _deleted(conn, account_id) + accounts.purge(conn, account_id, admin["id"]) + conn.commit() + return {"ok": True} + + # --- invites ------------------------------------------------------------------ class InviteBody(BaseModel): diff --git a/cloud/gammacloud/servers.py b/cloud/gammacloud/servers.py index 98204b40..7658e45a 100644 --- a/cloud/gammacloud/servers.py +++ b/cloud/gammacloud/servers.py @@ -21,8 +21,8 @@ from .accounts import Problem from .db import now +from .oidc import LOOPBACK_HOSTS -LOOPBACK_HOSTS = frozenset({"localhost", "127.0.0.1", "::1"}) MAX_NAME = 80 MAX_SERVERS = 50 diff --git a/cloud/manage.py b/cloud/manage.py index 6de0d62f..89e356c1 100644 --- a/cloud/manage.py +++ b/cloud/manage.py @@ -10,6 +10,8 @@ python manage.py set-plan python manage.py verify mark the e-mail confirmed python manage.py delete-account + python manage.py restore-account undo a delete within the grace period + python manage.py purge-account remove a deleted account now python manage.py purge-deleted [--days 30] python manage.py invite [--uses 1] [--plan free] [--note ...] python manage.py invites @@ -128,6 +130,27 @@ def cmd_delete(args): print(f"{args.username}: deleted (purged after the grace period by purge-deleted)") +def _deleted_account(conn, username: str): + row = conn.execute("SELECT * FROM accounts WHERE username = ? AND deleted_at IS NOT NULL", (username,)).fetchone() + if not row: + sys.exit(f"no deleted account with username {username!r}") + return row + + +def cmd_restore(args): + with closing(db.connect()) as conn: + accounts.restore(conn, _deleted_account(conn, args.username)["id"], "cli") + conn.commit() + print(f"{args.username}: restored (signs back in with a password reset or Google/GitHub)") + + +def cmd_purge_account(args): + with closing(db.connect()) as conn: + accounts.purge(conn, _deleted_account(conn, args.username)["id"], "cli") + conn.commit() + print(f"{args.username}: purged") + + def cmd_purge(args): with closing(db.connect()) as conn: n = accounts.purge_deleted(conn, args.days) @@ -195,6 +218,8 @@ def main(argv=None): pl = sub.add_parser("set-plan"); pl.add_argument("username"); pl.add_argument("plan", choices=config.PLANS); pl.set_defaults(fn=cmd_set_plan) v = sub.add_parser("verify"); v.add_argument("username"); v.set_defaults(fn=cmd_verify) d = sub.add_parser("delete-account"); d.add_argument("username"); d.set_defaults(fn=cmd_delete) + r = sub.add_parser("restore-account"); r.add_argument("username"); r.set_defaults(fn=cmd_restore) + pa = sub.add_parser("purge-account"); pa.add_argument("username"); pa.set_defaults(fn=cmd_purge_account) pu = sub.add_parser("purge-deleted"); pu.add_argument("--days", type=int, default=30); pu.set_defaults(fn=cmd_purge) i = sub.add_parser("invite"); i.add_argument("--uses", type=int, default=1); i.add_argument("--plan", default="free", choices=config.PLANS) i.add_argument("--note"); i.set_defaults(fn=cmd_invite) diff --git a/cloud/tests/test_admin.py b/cloud/tests/test_admin.py index cd765952..1e4e799b 100644 --- a/cloud/tests/test_admin.py +++ b/cloud/tests/test_admin.py @@ -1,7 +1,9 @@ +from contextlib import closing + from conftest import invite, make_admin, register from fastapi.testclient import TestClient -from gammacloud import mail +from gammacloud import accounts, db, mail def test_admin_only(client): @@ -34,6 +36,16 @@ def test_accounts_admin_flow(client): assert client.post(f"/api/admin/accounts/{me['id']}/delete").status_code == 400 assert client.post(f"/api/admin/accounts/{bob['id']}/delete").status_code == 200 assert client.get(f"/api/admin/accounts/{bob['id']}").json()["account"]["deleted_at"] + # restore brings the row back; purge only takes a deleted account and frees the name and address + assert client.post(f"/api/admin/accounts/{bob['id']}/restore").json()["account"]["username"] == "robert" + assert client.post(f"/api/admin/accounts/{bob['id']}/restore").status_code == 409 + assert client.post(f"/api/admin/accounts/{bob['id']}/purge").status_code == 409 + assert client.post(f"/api/admin/accounts/{bob['id']}/delete").status_code == 200 + assert client.post(f"/api/admin/accounts/{bob['id']}/purge").status_code == 200 + assert client.get(f"/api/admin/accounts/{bob['id']}").status_code == 404 + assert client.post("/api/admin/accounts/nope/restore").status_code == 404 + with closing(db.connect()) as conn: + accounts.create(conn, email="bob@example.org", username="robert", password=None) def test_invites_and_clients(client): diff --git a/cloud/tests/test_devices.py b/cloud/tests/test_devices.py index 6c74a2fd..81775ca3 100644 --- a/cloud/tests/test_devices.py +++ b/cloud/tests/test_devices.py @@ -2,7 +2,6 @@ a revoke, code replay, sign-out-everywhere, the refresh retry window and reuse detection, one grant per device, last activity, and the page.""" -import re import sqlite3 import threading import time @@ -57,6 +56,29 @@ def in_thread(fn): return t, box +def _open_quote(line): + """The quote of a JS string still open at the end of ``line``, else None. + Skips escapes, ``//`` comments and regex literals outside strings.""" + quote, in_class, i = None, False, 0 + while i < len(line): + c = line[i] + if quote: + if c == "\\": + i += 1 + elif quote == "/" and c in "[]": + in_class = c == "[" + elif c == quote and not in_class: + quote = None + elif c in "'\"`": + quote = c + elif line.startswith("//", i): + break + elif c == "/" and line[:i].rstrip()[-1:] in "(,=:[!&|?{};": + quote = "/" + i += 1 + return quote + + # --- races -------------------------------------------------------------------- def test_revoke_during_a_refresh_leaves_no_live_token(client): @@ -198,9 +220,8 @@ def test_the_devices_page(client): # the inline script must parse: an apostrophe inside a single-quoted JS # string once broke every button on this page script = page[page.rindex("")] - for quoted in re.findall(r"'((?:[^'\ -]|\.)*)'", script): - assert "'" not in quoted + for line in script.splitlines(): + assert _open_quote(line) is None, line assert 'confirm("Sign out every Gamma app' in script diff --git a/cloud/tests/test_manage.py b/cloud/tests/test_manage.py index 0de18b9f..2f1c5159 100644 --- a/cloud/tests/test_manage.py +++ b/cloud/tests/test_manage.py @@ -41,6 +41,21 @@ def test_purge_deleted(client): conn.commit() +def test_restore_and_purge_account(capsys): + manage.main(["create-account", "dave@example.org", "dave", "--password", "correct horse battery"]) + with pytest.raises(SystemExit): + manage.main(["restore-account", "dave"]) # not deleted + manage.main(["delete-account", "dave"]) + manage.main(["restore-account", "dave"]) + with closing(db.connect()) as conn: + assert accounts.by_username(conn, "dave") + manage.main(["delete-account", "dave"]) + manage.main(["purge-account", "dave"]) + assert "dave: purged" in capsys.readouterr().out + with closing(db.connect()) as conn: + assert not conn.execute("SELECT 1 FROM accounts WHERE username = 'dave'").fetchone() + + def test_newer_db_refused(): with closing(db.connect()) as conn: conn.execute(f"PRAGMA user_version = {db.SCHEMA_VERSION + 1}") diff --git a/design/brand/generated.json b/design/brand/generated.json index a5006c8b..7bb36ef2 100644 --- a/design/brand/generated.json +++ b/design/brand/generated.json @@ -7,8 +7,9 @@ "design/brand/tokens.json": "8778598768aefb172e4dc17f43e060fff18f87717ffa00977b64494e341c397f", "frontend/package-lock.json": "abe9fbcd79def8f77486d434b0699da29b27142ce726b32717172439e08cf1e2", "frontend/src/shared/illustrations/brands/LICENSE.md": "9046848b63a5c92bff14e4accca80bd987e0623b74adf9226ce5198d312b79d5", - "frontend/src/shared/illustrations/brands/README.md": "ca093f73aa381e7d039c82076cff0f83166edd525102e904640815b08418a4c5", + "frontend/src/shared/illustrations/brands/README.md": "e03bdd31f59c2c1f152f3316539084a18d1c8f3925e98a7530573102fd1e2b24", "frontend/src/shared/illustrations/brands/claude.svg": "2d6fda79eb18ddccca35b799eeb3cece0dfabc22520ce3b10abd25668df9fa93", + "frontend/src/shared/illustrations/brands/deepseek.svg": "7a55a0a7391d116eba7d32807d6838478f9209f6034612941e74fbb14934e2ef", "frontend/src/shared/illustrations/brands/logseq.svg": "5888e86163ec7064fa77d4eb7825b1782ae26600888397f7fce25c7f26d6df1c", "frontend/src/shared/illustrations/brands/notion.svg": "b17d2a2b592a06252efef522d5205f0c7a958f748d40df1011ed081417e42f85", "frontend/src/shared/illustrations/brands/obsidian.svg": "ceee016a3e7afcfa931e19b3dfa9eb26685fc8a9b4f177ff921e358d628abb33", @@ -16,7 +17,7 @@ "frontend/src/shared/illustrations/brands/zotero.png": "30c76a7a6441884df75567287db08ee40be6b763af7309f452a434800e069cd4", "tools/branding/branding.py": "85382fce3e48b5935c66c9df18c56e892e2f4f98463b24a1ac3aa2a29b44ace3", "tools/branding/build-anywhere.py": "ff525cc5769d79a0489c98ff052f7b431dc7cf0d5f59086c9e5d25d31687f8f8", - "tools/branding/build-connections.py": "dd736751e1d26f53361aa3ce9aacca82c32705b1bd73c879b142ae657bf550cb", + "tools/branding/build-connections.py": "a86fef54e560430ce91df2bd18e25d96ef45b5e0e567c5cdfcca2c5f0c958bc4", "tools/branding/build-demos.py": "11ee38fd99d2c30bafbde6991f1d29d20deddce15b7583e825dc4e9ef8c24c91", "tools/branding/build-library.py": "a3a65d14cbb9763674632f3e05a6506017c3267857b3f96323d634c5a3818673", "tools/branding/build-workspaces.py": "36f319915fac8ebd446c6a8a6c37013320e1754beb5a472befa4d2750e85af9e", @@ -83,7 +84,7 @@ 1920, 1080 ], - "sha256": "5ed331696e6c0e2513bd5f735b5e949c8f51f32a0c7dcf382e4e80f79cb292e5" + "sha256": "23efcd5de455e5cee23b67dbd919697a902e62f2a8972fc9f12d5965c29c81c3" }, "docs/assets/branding/gamma-demo-annotate-light.svg": { "source": "tools/branding/build-demos.py", diff --git a/docs/assets/branding/gamma-connections-light.svg b/docs/assets/branding/gamma-connections-light.svg index b927fd31..a687dd4d 100644 --- a/docs/assets/branding/gamma-connections-light.svg +++ b/docs/assets/branding/gamma-connections-light.svg @@ -1,6 +1,6 @@ Gamma PDF: your research, connected - Gamma in the middle of three connections. Left: an assistant prompt in Codex or Claude Code that mentions @Gamma and a paper card and asks how the blockade radius is measured; Gamma answers with papers and notes. Right: Obsidian, Notion and Zotero, with an Export arrow above and an Import arrow below. Bottom: the Gamma Connector browser extension saving a paper from a journal page, with the publisher sign-in saved per journal so the server can fetch its PDFs later. + Gamma in the middle of three connections. Left: an assistant prompt in Codex, Claude Code or DeepSeek Harness that mentions @Gamma and a paper card and asks how the blockade radius is measured; Gamma answers with papers and notes. Right: Obsidian, Notion and Zotero, with an Export arrow above and an Import arrow below. Bottom: the Gamma Connector browser extension saving a paper from a journal page, with the publisher sign-in saved per journal so the server can fetch its PDFs later. @@ -66,9 +66,9 @@ Papers + notes - + ASSISTANTS - + @Gamma in @@ -78,12 +78,13 @@ how is the blockade radius measured? - - - Codex - - Claude Code - Any MCP client · one workspace you approve + + Codex + + Claude Code + + DeepSeek Harness + Any MCP client · one workspace you approve diff --git a/docs/dev/ai.md b/docs/dev/ai.md index 301e18be..b9ddb3c5 100644 --- a/docs/dev/ai.md +++ b/docs/dev/ai.md @@ -5,13 +5,14 @@ request/stream shape, and the library agent: what it can reach, how the tool loop runs, and what the user controls. The tools themselves are catalogued in [ai_tools.md](ai_tools.md); how long papers reach the model is [ai_context.md](ai_context.md). Code: `gamma/ai_settings.py`, -`gamma/ai_client.py`, `gamma/ai_context.py`, `gamma/ai_tools.py`, +`gamma/ai_protocols/` (one adapter per wire), `gamma/ai_client.py`, +`gamma/ai_catalog.py`, `gamma/ai_context.py`, `gamma/ai_tools.py`, `gamma/chatgpt_oauth.py`, `gamma/routers/ai.py` + the chat-history router. ## Provider and models There are NO env API keys; providers are GUI entries: each account's own -(Settings → Provider and models), plus the server's shared ones an admin adds +(Settings → AI › Connections), plus the server's shared ones an admin adds (below). An account's entries are stored under the reserved account-wide `ai-settings` pref in `users.db` — a LIST of `{id, name, protocol, api_key, base_url, models}` managed via `POST/PUT/DELETE /api/ai/providers[/{id}]`. An entry offers exactly the @@ -35,8 +36,8 @@ details are summarized before display everywhere (`upstream_detail` in (a proxy's 502 page) to their ``. `POST /api/ai/health` ({provider_id, mode}; "" = first entry) is the login -connection check (Settings → Provider and models → "Check at login", -localStorage `gamma-ai-login-check`, default on): mode `"ping"` verifies the +connection check (Settings → AI › Connections → "Check at login", +account pref `gamma-ai-login-check`, default `ping`): mode `"ping"` verifies the credential for free — OAuth entries hit the usage endpoint, API keys list `/v1/models`, both 401 on a dead credential (404/405 = gateway without a listing → ok-but-unverified, no false alarm) — and `"test"` runs the same tiny @@ -54,13 +55,13 @@ module-level config constants for credentials or model routing. Env vars set each protocol's administrator-controlled default base URL, including `GAMMA_AI_CHATGPT_BASE_URL`. -Named services (`AI_SERVICES` in `gamma/config.py`, sent as `services` with the +Named services (`SERVICES` in `gamma/ai_protocols/__init__.py`, sent as `services` with the settings) are form presets: a protocol plus a fixed endpoint, listed in the form's service menu between the protocols and "Custom endpoint". DeepSeek is the `openai` protocol at `https://api.deepseek.com`. An entry made from one stores only protocol + base URL; `provider_label` recognizes the pair and names the entry after the service when it has no name of its own. The -`openai` wire follows the endpoint (`is_openai_platform` in `ai_client.py`): +`openai` wire follows the endpoint (`is_openai_platform` in `ai_protocols/openai.py`): only OpenAI itself gets `max_completion_tokens`, the Responses API for tool calls, and the gpt-/o-family filter on its model listing. Compatible servers get `max_tokens`, Chat Completions tools and their full listing (minus @@ -69,7 +70,7 @@ absent: Anthropic's terms forbid third-party apps from routing requests through Free/Pro/Max plan credentials, so Claude is reached with a Console API key. -The Provider and models pane also exposes `POST /api/ai/providers/{id}/usage`. For a +The Connections pane also exposes `POST /api/ai/providers/{id}/usage`. For a ChatGPT OAuth entry it reads normalized subscription rate-limit windows (`used_percent`, `remaining_percent`, and reset time) without exposing the bearer token. Opening the pane queries OAuth usage automatically (the Usage @@ -87,6 +88,42 @@ or base URL; this prevents a settings request from redirecting a bearer token. The ChatGPT account endpoint is provider-specific and may require maintenance if its upstream contract changes. +### Protocol adapters + +Everything that differs between providers lives on one adapter per wire in +`gamma/ai_protocols/` (`base.Protocol`), and nothing outside that package +branches on a protocol id. Routes, the chat loop, `ai_settings` and +`ai_client` ask the entry's adapter (`ai_protocols.of(conf)`): + +| Concern | Adapter member | +|---|---| +| what the form offers | `label`, `base_url` (env default, `config.AI_BASE_URLS`), `auth` (`"key"` / `"oauth"` + the `oauth` module that refreshes tokens), `entry` | +| the chat call | `wire(conf, tools)` (a sibling wire for some calls), `request(...)`, `reply_text`, `read_reply`, `streams_only` | +| the stream | `events` (one loop in the base) over `stream_event` / `stream_end` | +| token counts | `usage(raw)` → `{input, output, cache_read, cache_write}` | +| models | `models_request`, `models(data, conf)` → `[{id, context_window}]`, `catalog_hints` | +| credential check | `ping_request` (default: the model listing) | +| quota | `has_account_usage`, `account_usage_request`, `account_usage` | +| attachments, dictation | `native_pdf`, `transcription` (a rank), `transcription_request`, `transcript` | + +The wires: `anthropic.py` (Messages API), `openai.py` (Chat Completions for +OpenAI and every compatible server), `responses.py` (the Responses API that +OpenAI's platform and the ChatGPT backend both speak; `openai-responses` is +the variant an OpenAI entry switches to for tool calls, never an entry's own +protocol), `chatgpt.py` (the Codex backend: its listing, quota and client +version). `ai_client.py` is the transport (open, read, stream, errors); +`ai_catalog.py` fetches and caches listings and context windows +(`fetch_json` is the one fetch every listing, quota and ping goes through). +The sign-in flow itself (`chatgpt_oauth.py`, `/api/ai/oauth/chatgpt/*`) is +provider-specific by nature. + +A new service on an existing wire (a gateway, a hosted model) is a +`SERVICES` preset or just a custom base URL. A new wire is one module +subclassing `Protocol` (or `ResponsesWire`) that overrides what differs from +the OpenAI-shaped defaults, plus one line in `WIRES` and its default URL in +`config.AI_BASE_URLS`; `tests/test_ai_wire.py` pins each wire's request and +stream shapes. + ### Shared provider entries An admin can add provider entries for the whole server (Settings → Server → @@ -129,13 +166,13 @@ deltas), and PDF attachments go as native `input_file` parts with an automatic retry as extracted text if the backend rejects them. That retry applies to any provider that answers a native-PDF request with a 4xx other than 401/403/429 (compatible servers may refuse `file` parts too). Anthropic has -no `minimal` effort; `anthropic_request` sends `low` for it. +no `minimal` effort; its adapter sends `low` for it. Its model list (`POST /api/ai/model-catalog`) is Codex CLI's own listing call, `GET {base}/models?client_version=…`, made with the entry's token. The backend hides models newer than the client version it is told, so Gamma claims the newest Codex CLI release: npm's `latest` for `@openai/codex`, cached for 6 h -(`_codex_client_version`; on a failed lookup the last good version, else a +(`codex_client_version` in `ai_protocols/chatgpt.py`; on a failed lookup the last good version, else a floor constant, with a retry after 10 min). No model names are hardcoded. A failed listing is a 502 the picker shows, and a fresh connect whose listing fails starts with no models. The sign-in `state` belongs to the account that @@ -417,11 +454,11 @@ Rounds and the ≤200-mutation ceiling are runaway guards, not workload caps. ### The tool loop The router runs a loop (`agent_events`) over `ai_client.sse_events`, which -parses tool calls from all three protocols' SSE (tool defs + -`tool_calls`/`role:"tool"` message extensions are translated per wire in the -request builders; the Responses builders enable `parallel_tool_calls` when -tools ride along, so bulk renames batch per round): the model calls tools → -the server executes them → results go back → repeat until it answers. +parses tool calls from every wire's SSE (`Protocol.events`): the model calls +tools → the server executes them → results go back → repeat until it +answers. Each adapter's `request` maps the tool defs and the +`tool_calls`/`role:"tool"` turns to its wire. The Responses body enables +`parallel_tool_calls` when tools ride along, so bulk renames batch per round. Every tool call streams back as an `{"action": {kind, summary, tool, args, result}}` NDJSON line (kinds @@ -486,7 +523,7 @@ call again before quoting or editing. The base prompt says the same, so a request to read, show or check something is answered from a fresh call, not last turn's outline (the agent's own edits change what `read_block` returns). Results share `TOOL_REPLAY_BUDGET` chars newest-first -(older ones elided), and `_anthropic_messages` folds a plain user turn into a +(older ones elided), and `_messages` in `ai_protocols/anthropic.py` folds a plain user turn into a preceding tool_result turn to keep roles alternating. Plain chats never replay (providers reject tool blocks without tool defs). Renamed tools replay under their current name (`ai_context.DEPRECATED_TOOLS`, e.g. the saved @@ -497,7 +534,7 @@ dispatch, and the resulting action chip carries the current name. Old names are never offered as tools. OpenAI-protocol calls that carry tools are rerouted to the platform -`/v1/responses` (`wire_protocol`) — gpt-5.x rejects function tools on chat +`/v1/responses` (`OpenAIChat.wire`) — gpt-5.x rejects function tools on chat completions — but only against the official api.openai.com base URL; custom gateways keep chat-completions tools. @@ -505,15 +542,21 @@ gateways keep chat-completions tools. ## PDF translation `POST /api/ai/translate` backs the viewer's translated view. ONE 文A button -in the PDF zoom column does everything by state: click translates the -current page when nothing is translated yet, toggles show/hide for ALL pages -once translations exist under the current language+model (switching either -in Settings makes the button translate afresh; hidden = slashed icon; -holding Alt peeks), and -halts a running job; right-click (long-press on touch) opens the option -menu — Translate this page / Translate whole document / Show -original·translation (Stop translating while running). A whole-document job -queues pages nearest the current page first (forward before backward at +in the PDF zoom column does everything by state: + +- On the page being read, a click translates it unless it is already fully + translated under the current language and model. On such a page a click + toggles show/hide for ALL pages (the viewer reports `current` next to + `pages` in its state). +- A page a halted job left half-done counts as untranslated, so a click + finishes it from the cache. Switching language or model in Settings makes + the button translate afresh. +- Hidden = slashed icon; holding Alt peeks. A click during a job halts it. +- Right-click (long-press on touch) opens the option menu: Translate this + page / Translate whole document / Show original·translation (Stop + translating while running). + +A whole-document job queues pages nearest the current page first (forward before backward at equal distance), so the page being read paints immediately. The queue lives in `pdf/PdfViewer.jsx` (`translateCtl`), producer/consumer style: the producer segments queued pages in order and feeds one flat list of ~6-paragraph / @@ -557,16 +600,84 @@ cloned background — so figures a paragraph brushes against are never painted over, and the layout never moves. Translated text is selectable/copyable; while shown, the invisible original text layer stands down. -Targets are the allowlisted `TRANSLATE_LANGS` codes (mirrored in -`frontend/src/app/prefDefs.js`); model and reasoning `effort` come from Settings → -Reading (model follows the chat model by default; effort omitted unless -picked — Low/Minimal is the speed lever for reasoning models); the whole -Translation section can be switched off there too. The server keeps an -**in-memory only** LRU (~5k entries, lock-guarded — requests run in the -threadpool) per (user, language, bare model name, source text) — -deliberately nothing on disk; it makes halts/retries/re-shows free until a -restart. Duplicate paragraphs within a request go upstream once. Caps: 200 -texts / 60k chars per request. +Targets are the allowlisted `TRANSLATE_LANGS` codes +(`gamma/translate_engines.py`, shared by both translation paths; mirrored in +`frontend/src/app/prefDefs.js`). What translates is Settings → Reading › +"Translate with" (`translateModel`, a browser pref; "" is the default). +`translateModelFor` (`app/prefDefs.js`) turns the pick into what is sent: +the pick while it is still offered, the free Microsoft service when there +is no chat model at all, else the chat model. Reasoning `effort` and +parallel requests sit in the same Translation section. Effort is omitted +unless picked; Low/Minimal is the speed lever for reasoning models. The +"Translation button" switch hides the viewer's button; the selection +translator has its own switch. + +The server keeps an **in-memory only** LRU (~5k entries, lock-guarded +because requests run in the threadpool) per (user, language, bare model +name, source text). Nothing goes to disk; the cache makes +halts/retries/re-shows free until a restart. Duplicate paragraphs within a +request go upstream once. Caps: 200 texts / 60k chars per request. + +**Selection translation.** The text-selection popup (`PlainTip` in +`pdf/PdfViewer.jsx`, the highlight colors + link) carries a 文A button +when Settings → Reading › "Translate a selection" is on (`selTranslate`, +account pref, default on). + +- It sends the selection as ONE text through the page translator's request + (`translateChunk`: same model or service, language, server cache, + streamed partials). Lines are rejoined by `selectionParagraphs` + (`pdf/pdfTranslate.js`, the page blocks' hyphen/CJK rules), capped at + 5000 characters. +- The result shows under the colors as a fold-out panel: a header with the + language, spinner and copy button, then selectable text. The button + refolds it. +- "Translate on select" (`selTranslateAuto`, default off) starts it as soon + as the popup opens. +- The popup is keyed by the selection, so a new selection starts over and + aborts the previous request. A click or selection inside the popup keeps + it open (the viewer's selection sync ignores a selection anchored in + `.plainTip`). +- `TipFrame` flips the popup above the selection when it would run past the + window's bottom. Like the popup itself, it needs edit rights on the page. + +**Machine-translation services.** "Translate with" also offers the services +in `gamma/translate_engines.py`. The viewer then sends `model: +"engine:"`, and `/api/ai/translate` hands the misses to +`translate_engines.translate` instead of a chat model. + +- **Microsoft (free)** needs no setup. It is the endpoint Edge's own page + translation calls: `POST + edge.microsoft.com/translate/translatetext?to=&isEnterpriseClient=false` + with a JSON array of strings and no key or token. The reply has + Translator v3's shape (`[{detectedLanguage, translations: [{text, to}]}]`); + the source language is detected per text. +- The endpoint is unofficial and undocumented, so it can change or throttle + without notice; Google and Youdao are the fallback. The older + `/translate/auth` token flow answers 404 since July 2026. +- Microsoft refuses requests past about 50k characters (measured), so + batches stay at 100 texts / 20k characters. +- Its consecutive failures are counted in memory. From `FREE_ALERT_AFTER` + (3) on, the server log gets one warning per streak and each account that + met them gets the `free-translate` notice ([settings.md](settings.md) + "Notices"); the Settings row shows the error. One success ends the streak. +- **Google Cloud Translation**: v2 basic, the API key sent as + `X-Goog-Api-Key`, `format: "text"`. +- **Youdao**: the `openapi.youdao.com/v2/api` batch, with a v3 SHA-256 + signature over the concatenated queries. A query listed in `errorIndex` + comes back verbatim and uncached. +- Each service maps the target codes to its own (Microsoft `zh-Hans`, + Youdao `zh-CHS`) and splits a request by its batch limits. +- The service path needs no AI provider. It has no effort, no streamed + partials (the stream is just the final line), no token usage and nothing + to salvage, since the APIs answer aligned lists. +- Cache, validation and dedup are the LLM path's, keyed on `engine:`. + +Credentials are per account under the reserved `translate-engines` pref. +Like `ai-settings`, `/api/prefs` refuses it and the only read path is the +masked `GET /api/translate/engines`; guests can't store any. Unlike the LLM +prompt, nothing tells these services to leave math, `[12]` citation markers +or URLs alone. PDF text carries no LaTeX and math-heavy paragraphs are +skipped client-side, so the risk is small. ## Token usage @@ -601,6 +712,27 @@ show what a week cost. Code: `gamma/ai_usage.py`, `ai_client.normalize_usage`, (`estimateTokens`: characters received / 4, text deltas and previewed tool arguments alike; reset when that round's report lands). Replies saved before this carry no counts and show nothing. +- **Context ring.** Left of the chat header's ⚙, Claude Code style: how full + the model's window is. The figure is the latest reply's LAST round alone + (its input + output, which the next message carries as history), saved on + the reply as `context_tokens` — the summed `usage` would overcount an + agent reply. A reply saved before that field counts only when it had no + tool rounds. The window is looked up live, never tabled in the code + (`ai_catalog.context_window`, for every protocol alike): + `GET /api/ai/context-window?model=:` reads the entry's own + model listing first (`Protocol.models` — Anthropic's `max_input_tokens`, + the Codex backend's `context_window`, the `context_length` / + `max_model_len` of OpenRouter, vLLM, Groq, …), then the public models.dev + catalog for listings that carry no size (OpenAI's, DeepSeek's) — there the + provider this entry talks to wins (`Protocol.catalog_hints`: the + endpoint's host labels, OpenAI for the ChatGPT backend), else the value + most providers agree on. Both are cached like the + Codex version (6 h; a failed lookup retried after 10 min, the last good + answer kept). A model neither knows gets `null`: no ring, and the popover + shows the token count alone. The client asks once per model per page load + (`useContextWindow` in `ChatDock.jsx`). The ring turns red past 80%; + clicking it opens the chat-settings popover, whose Tokens section spells + the figure out (`contextUsed`). - **Stored.** `ai_usage.record` writes one row per call to `ai_usage` in `users.db` (account, time, kind, provider id + name, model, the four counts); `ai_usage.recorder(kind, entry, rt)` is the `on_usage` callback the diff --git a/docs/dev/ai_tools.md b/docs/dev/ai_tools.md index 9cd97d2f..18ecdf62 100644 --- a/docs/dev/ai_tools.md +++ b/docs/dev/ai_tools.md @@ -121,8 +121,8 @@ picture reaches the model once and is never saved into the chat or replayed Anthropic `tool_result` carries the image blocks after the text; the OpenAI chat-completions and Responses wires only take text in a tool result, so their pictures follow the round's results as one user turn ("Pictures -returned by the tool calls above, in call order"; `ai_client.py` -`_tool_image_turns`) — never the turn the user's own attachments ride on. +returned by the tool calls above, in call order"; `ai_protocols/base.py` +`tool_image_turns`) — never the turn the user's own attachments ride on. The armed prompt tells the model when a picture is worth its tokens (missing or garbled extracted text, a figure, handwriting) and to say when an answer was read from one. A page without a PDF, a page number past the diff --git a/docs/dev/api.md b/docs/dev/api.md index 14dd0927..8ad7443e 100644 --- a/docs/dev/api.md +++ b/docs/dev/api.md @@ -112,11 +112,12 @@ else; in dev, Vite proxies `/api` → `127.0.0.1:9001`. | Method | Path | Purpose | |---|---|---| | POST | `/login`, `/login-guest`, `/logout` | session management (`/login` refuses an account with an empty password hash — one only its cloud identity signs in; `/login-guest` is 403 on a share host) | -| GET | `/server-config` | public: what the login page offers besides a password — `{cloud: {enabled, issuer}, password_login, registration, guest}` (`guest` false on a share host) ([cloud_accounts.md](cloud_accounts.md)) | +| GET | `/server-config` | public: what the login page offers besides a password — `{cloud: {enabled, issuer}, password_login, registration, guest, page_host}` (`guest` false on a share host; `page_host` the per-account page hostname pattern from `GAMMA_PAGE_HOST`, e.g. `{username}-pages.gammapdf.com`, "" when unset — how the app knows it was opened on a page host) ([cloud_accounts.md](cloud_accounts.md)) | | POST | `/auth/cloud/exchange` | the share host's half of publishing ([mirror.md](mirror.md) "Publishing"): `Authorization: Bearer `, body `{server?}` (the calling server's name) → `{token, workspace_id, username, url}` — a write-scope integration token (365 days, named "Published pages from ", replacing the live one of that name) on the person's default personal workspace here, the account resolved under the sign-in policy like a first sign-in (provisioned under `provision`, pending invitations claimed), and this server's address. 403 unless the server accepts published pages (`cloud_share_host`), for an unconfirmed cloud e-mail, or when the policy refuses the account; 401 for a token the account server does not know; 503 when it cannot be asked; 429 past 20 per IP or 10 per cloud account in 10 minutes | | GET | `/auth/cloud/start?next=&link=1` | Sign in with Gamma Cloud: stores the pending PKCE sign-in and redirects to the account server; `link=1` needs a session and attaches the cloud identity to that account | | GET | `/auth/cloud/callback?code=&state=` | the account server's return: verifies the ID token, resolves or creates the local account per the policy (`gamma/cloud_auth.py`), pulls the preference profile, registers this server on the person's server list (`gamma/cloud_sync.py`), mints a session and redirects to `next`; a refusal goes back to `/?cloud_error=` | -| GET | `/auth/cloud/sync-status` | the signed-in account's own preference profile sync (session only; guests and integration tokens get 403): `{profile: {state, at, error}, identity: {linked, username?}}`. `state` is `off` (cloud sign-in off, or no identity holding a token), `pending` (a push is scheduled, or failed and waits for the next check; `error` then says why), `synced` (the last pull or push agreed, at `at`) or `error` (the last attempt failed). Read from memory (`cloud_sync.profile_status`), no network; the Settings dialog polls it while open | +| GET | `/auth/cloud/sync-status` | the signed-in account's own preference profile sync (session only; guests and integration tokens get 403): `{profile: {state, at, error}, identity: {linked, username?}}`. `state` is `off` (cloud sign-in off, or no identity holding a token), `pending` (a push is scheduled, or failed and waits for the next check; `error` then says why), `synced` (the last pull or push agreed, at `at`), `error` (the last attempt failed) or `choose` (the first sync found two different copies and waits for the person's choice). Read from memory (`cloud_sync.profile_status`), no network; the Settings dialog polls it while open | +| POST | `/auth/cloud/sync` | sync the caller's profile with Gamma Cloud now (session only): `{action, defaults?}` with `action` `sync` (the automatic merge; answers `choose` while a first sync waits), `merge` (also settles the choice; `defaults` — the web app's default profile — is the base of a first merge), `fetch` (the cloud's copy replaces this server's; 409 when the cloud has none) or `push` (this server's replaces the cloud's). Answers `{outcome: pulled / pushed / merged / same / choose, profile}`; 400 without a linked identity holding a token, 502 when the account server could not be reached | | GET / POST | `/auth/cloud/status`, `/auth/cloud/unlink` | the signed-in account's own cloud identity (username, plan, e-mail, linked at, `offline` — a refresh token is held — and `revoked_at`) plus `enabled` and the `issuer` (the account server's address, which the Account pane's "Open account" button opens); unlink is refused for an account without a password, and takes this server off the person's server list before revoking the grant | | GET | `/session` | who am I, plus `workspaces: [{id, name, kind, role, access, public_role, personal, default, members}]` (memberships + every public workspace) and `default_workspace` (quota lives in `/quota`); `build` (`version`, `commit`, `label`, `frozen`) is what a problem report names the server by, sent to the login page too | | GET | `/accounts[?q=]` | the account directory for the invite / owner pickers: `{accounts: [{username, is_admin}]}`, non-guest accounts only (signed-in non-guest callers). On a share host only admins get the list; anyone else gets the one account named exactly `q`, or none | @@ -128,7 +129,7 @@ else; in dev, Vite proxies `/api` → `127.0.0.1:9001`. | Method | Path | Purpose | |---|---|---| | POST | `/workspaces` | create a personal one (`{name}`; guests 403); admins may add `kind: "shared"`, `owner`, `access`, `public_role`, `quota_mb` | -| GET | `/workspaces/mine` | Settings → Workspaces: every workspace I can open with its `used_bytes` (and `mirror_of`, the remote workspace's name when it is an offline copy, `publishing` true when it publishes pages to Gamma Cloud, where `mirror_of` stays empty — [mirror.md](mirror.md)), plus `account` (my limits and the usage of all my personal workspaces) | +| GET | `/workspaces/mine` | Settings → Workspaces: every workspace I can open with its `used_bytes` (and `mirror_of`, the remote workspace's name when it is an offline copy, `publishing` true while at least one of its pages is published to Gamma Cloud, where `mirror_of` stays empty — [mirror.md](mirror.md)), plus `account` (my limits and the usage of all my personal workspaces) | | GET/PUT/DELETE | `/workspaces/{id}` | kind + members (pending invitations last, `pending: true` + `subject`) + quota + `personal_of` + `default` (any member; admins) / rename `{name}` (owner), `default: true` (a personal workspace's owner), kind, access + public role, workspace quota (admin) / delete (owner; not an account's last personal one) | | GET/POST | `/workspaces/{id}/backups` | the workspace's server-kept snapshots (any member) / take one now `{label?, uploads?}` (owner; at most `ws_backup.MAX_PER_WORKSPACE`) | | GET | `/workspaces/{id}/backups/{name}/download` | the snapshot as a zip — the same zip `/export` gives (any member) | @@ -180,7 +181,7 @@ the tree reflects (the live session catches up from it). ### Pages (`pages.py`) — page first, PDF as an action on it | Method | Path | Purpose | |---|---|---| -| POST | `/pages` | create a text-only root page: body `{title?, folder?, id?, properties?}` (title defaults to `Untitled`, `folder` → `properties.folder`; `id` keeps a page's id when a mirror brings it over — 400 when malformed, 409 when taken; `properties` seeds the page's own) → the block dict | +| POST | `/pages` | create a text-only root page: body `{title?, folder?, id?, properties?}` (title defaults to `Untitled`, `folder` → `properties.folder`; `id` keeps a page's id when a mirror brings it over — 400 when malformed, 409 when taken; `properties` seeds the page's own) → the block dict. On a share host, 402 `{detail, limit, used, plan}` when the owner's plan allows no more pages in their default personal workspace ([mirror.md](mirror.md) "Publishing") | | POST | `/pages/by-docs` | which pages these stored files became: body `{doc_ids: [, ...]}` (≤500) → `{pages: {hash: {id, title}}}` — a hash matches the page carrying it as its PDF (`doc_id`) or the note page imported from it (a markdown upload's `markdown_import`); hashes with no page absent; any member. The file chips ask once per page render for their "open page" button and the menu's "Open page" / "Add to library" | | POST | `/pages/from-file` | "Add to library" on a markdown file chip: body `{filename: ".md", original?, folder?}` → `{page, created, imported?}` — the stored upload becomes a note page through the `/import/markdown` importer (title from front matter, else `original` minus its extension), filed in `folder`; idempotent (a page whose `markdown_import` is the hash is returned with `created: false`); the file is untouched, the page is a copy. 400 for anything but a stored markdown name, 404 when the file is not in the workspace; workspace editors | | POST | `/pages/{page_id}/attachment` | attach a PDF to a page that has none: body `{doc_id?, source_url?, original_filename?}` (at least one of `doc_id`/`source_url`; `doc_id` is shape-validated only — a URL-opened PDF's id is the URL hash and the proxy fetches it lazily, like `by-doc`; `source_url` defaults to `/api/uploads/.pdf`). While the title is still automatic (`Untitled`/empty) it becomes the file name / URL tail and is marked `auto_title`. → the updated block. 400 bad input / not a root page, 404 unknown page, 409 `{"detail": "page already has an attachment"}`, 409 `{"detail": "attachment belongs to another page", "page_id"}` | @@ -263,7 +264,7 @@ the request's workspace — the extension names none, so its personal one. | POST | `/metadata/fetch` | resolve a paper or book (arXiv → DOI → ISBN via Open Library/Google Books → Crossref search → AI extraction, verified against Crossref / the book registries), cache meta + BibTeX + the slide citation on the page. Body also takes `cite_prompt`/`cite_model`; returns `meta` (with `unverified`), `bibtex`, `ppt_cite` (`""` when AI is off or that call failed), `source`, `cached`, `page_title` (the page's title after the write — always, since a concurrent lookup may have renamed it) and `title_updated` (this call replaced the automatic title) | | POST | `/metadata/update` | save hand-edited fields incl. `publisher`/`isbn` (rebuilds BibTeX, keeps the document kind, drops the cached citation) | | POST | `/metadata/cite` | BibTeX → PPT-style citation via AI (regenerate / fallback; the fetch already produces one) | -| GET | `/metadata/status` | library-wide health table (feeds Settings → Library): every page with a PDF attachment plus pages carrying `properties.meta` without one (`has_file: false`); per paper `meta_source`, `meta_kind`, `meta_unverified` (null for pre-flag records) | +| GET | `/metadata/status` | library-wide health table (feeds Settings → Library maintenance): every page with a PDF attachment plus pages carrying `properties.meta` without one (`has_file: false`); per paper `meta_source`, `meta_kind`, `meta_unverified` (null for pre-flag records) | ### AI (`ai.py`) — all config is GUI entries (each account's own plus the server's shared ones), no env API keys | Method | Path | Purpose | @@ -278,9 +279,13 @@ the request's workspace — the extension names none, so its personal one. | DELETE | `/ai/usage` | forget the account's usage rows | | POST | `/ai/health` | login connection check of one entry the account can use, its own or shared (`{provider_id, mode}`; `""` = the first): `mode: "ping"` is the free credential check (OAuth → usage endpoint, API key → `/v1/models`), `"test"` the tiny live completion; always answers in-body `{configured, ok, auth?, error?}` | | POST | `/ai/model-catalog` | list models available to a credential: the typed key, or a saved entry's (`provider_id`; admins may name a shared `server:`) | +| GET | `/ai/context-window?model=:` | the chat model's context window for the context ring: `{model, context_window, source: "provider" \| "models.dev"}`, from the entry's own model listing, else the models.dev catalog; `context_window` null when neither knows it (`""` = the default model) | | POST | `/ai/oauth/chatgpt/start`, `/complete` | ChatGPT OAuth (PKCE, pasted callback URL) | | POST | `/ai/transcribe` | voice dictation | -| POST | `/ai/translate` | translate paragraph texts for the viewer's translated view (`{texts, lang, model, effort, stream}` → `{translations}`; with `stream: true` an NDJSON stream of `{i: [indices], text}` partials as each paragraph is written, then the same final object; in-memory per-paragraph cache) | +| POST | `/ai/translate` | translate paragraph texts for the viewer's translated view (`{texts, lang, model, effort, stream}` → `{translations}`; with `stream: true` an NDJSON stream of `{i: [indices], text}` partials as each paragraph is written, then the same final object; in-memory per-paragraph cache). `model: "engine:google"` / `"engine:youdao"` translates with that machine-translation service instead — no AI provider needed, 503 when it isn't set up, never streams partials | +| GET | `/translate/engines` | the account's machine-translation services (`{engines: [{id, label, configured, needs_key, fields, updated_at, failing}], can_edit}`; secret fields as a `…last4` hint; `microsoft` needs no key and is always configured, and its `failing` is `{since, error}` during a failure streak, else null) | +| PUT / DELETE | `/translate/engines/{id}` | set (`{fields: {…}}`, an empty secret keeps the stored one) or remove a service's credentials (400 for a service that needs no key); guests 403; answers the GET shape | +| POST | `/translate/engines/{id}/test` | translate one sentence into `{lang}` with the stored credentials; in-body `{ok, text}` / `{ok: false, error}` | | GET | `/pdf-text-status` | whether a doc has extractable text | ### Chats (`chats.py`, prefix `/api/chats`) @@ -322,7 +327,8 @@ archived conversation browsing remains session-only. ### Prefs (`prefs.py`) | Method | Path | Purpose | |---|---|---| -| GET/PUT | `/prefs/{key}` | small synced JSON KV per account: `open-tabs`, `recent-views`, `pinned-folders`, `read-pos` are stored per workspace (the request's), `profile` / `ai-provider` account-wide (`db.USER_PREF_KEYS`); `profile` is the web app's account-scoped settings as one object keyed by preference name (400 unless an object; `db.get_profile` / `db.set_profile`, [settings.md](settings.md)); values over 64 KB get 413; refuses the reserved `ai-settings` key | +| GET/PUT | `/prefs/{key}` | small synced JSON KV per account: `open-tabs`, `recent-views`, `pinned-folders`, `read-pos` are stored per workspace (the request's), `profile` / `ai-provider` account-wide (`db.USER_PREF_KEYS`); `profile` is the web app's account-scoped settings as one object keyed by preference name (400 unless an object; `db.get_profile` / `db.set_profile`, [settings.md](settings.md)); reading `profile` first syncs it with Gamma Cloud when the last sync is over a minute old, and its answer carries `cloud_choice` (a first sync waits for the person's choice); values over 64 KB get 413; refuses the reserved `ai-settings`, `translate-engines` and `profile-base` keys | +| PATCH | `/prefs/profile` | `{set: {name: value}}`: sets those entries of the profile and keeps every other one as stored (`db.patch_profile`) — how the web app saves, so a tab's stale copy of an entry it did not touch never undoes one synced from elsewhere; answers `{key, value, updated_at}` with the whole profile; 413 over 64 KB | | GET | `/page-snaps` | all recents-card cover thumbnails `{snaps: {pageId: {img, at}}}`; `?after=` returns only newer ones (the focus-pull delta) | | PUT | `/page-snaps/{page_id}` | store a cover (JPEG data URL body `{img, at}`; per-page newest-`at` wins, count-capped server-side) | | DELETE | `/page-snaps/{page_id}` | drop a cover (the recents card's ×) | @@ -330,7 +336,7 @@ archived conversation browsing remains session-only. ### Notices (`notices.py`, `gamma/notices.py`) — see [settings.md](settings.md) "Notices" | Method | Path | Purpose | |---|---|---| -| GET | `/notices` | `{notices: [{id, fingerprint, tone, pane, title}]}` the account has not looked at yet, strongest `tone` (`info` / `warn` / `error`) first; `pane` is the Settings pane that resolves it. Admin-only sources (`update`: a newer GitHub release; `log-errors`: errors logged since the last look) are skipped for members; guests and integration tokens get `[]`. Sync: the release check may hit the network when its cache is stale | +| GET | `/notices` | `{notices: [{id, fingerprint, tone, pane, title}]}` the account has not looked at yet, strongest `tone` (`info` / `warn` / `error`) first; `pane` is the Settings pane that resolves it. Sources: `update` and `log-errors` (admins: a newer GitHub release, errors logged since the last look), `backup-failed`, `mirror-conflicts`, `cloud-sync`, `storage` (everyone: a failed backup task, open conflicts in an owned clone, a failed Gamma Cloud sync, personal storage past 90 % or full) — the table in [settings.md](settings.md); guests and integration tokens get `[]`. Sync: the release check may hit the network when its cache is stale | | POST | `/notices/{id}/seen` | `{fingerprint}` — the account has seen this version of the notice (kept in the account-wide `notices-seen` pref); it stays quiet until the fingerprint changes. 403 for guests and tokens, 400 for a malformed id or fingerprint | ### Integrations and MCP (`routers/integrations.py`, `mcp_oauth.py`, `mcp_server.py`) — see [mcp.md](mcp.md) @@ -348,16 +354,16 @@ A manual token (`gamma_…`, not an OAuth one) is also accepted on every `/api/* |---|---|---| | GET | `/mirrors` | the caller's offline copies with their sync status | | POST | `/mirrors` | `{remote_url, token, name?, mode?: two-way \| pull, workspace_id?, adopt?: theirs \| mine}` → the mirror: a new personal workspace that follows the remote workspace the token belongs to, or with `workspace_id` an existing personal workspace of the caller's whose pages adopt one side's version (validated against the remote's `/sync/whoami` first; a read token or a viewer's role gives `pull`); the first fill runs in the background | -| GET | `/mirrors/{ws}` | one mirror, with `conflicts_open` (unresolved merges), `pending_local` (a local write no round has pushed yet; two-way copies only), `poll_s` / `on_change` (its cadence), `detached`, `interval_s` (0 = the loop is off), `page_filter` (null = every page travels, else the ids of the only pages that do — a publication); `status.progress` `{done, total, page, first, file?}` while a round runs | -| PATCH | `/mirrors/{ws}` | `{poll_s?, on_change?, mode?}` | +| GET | `/mirrors/{ws}` | one mirror, with `conflicts_open` (unresolved merges) and `conflicts_newest` (the newest open one's id), `pending_local` (a local write no round has pushed yet; two-way copies only), `poll_s` / `on_change` (its cadence), `detached`, `interval_s` (0 = the loop is off), `page_filter` (null = every page travels, else the ids of the only pages that do — a publication); `status.progress` `{done, total, page, first, file?}` while a round runs | +| PATCH | `/mirrors/{ws}` | `{poll_s?, on_change?, mode?}` (`mode` on a detached copy: 400 — reattach first) | | POST | `/mirrors/{ws}/detach` | detach, the link kept | | POST | `/mirrors/{ws}/relink` | `{token?, remote_url?, adopt?}` — link again | -| POST | `/mirrors/{ws}/force` | `{direction: pull \| push}` — replace one side with the other | +| POST | `/mirrors/{ws}/force` | `{direction: pull \| push}` — replace one side with the other; noted as `status.force` and applied by the next round | | POST | `/mirrors/{ws}/sync[?wait=1]` | a sync round now (`wait=1` answers with the round's status) | | DELETE | `/mirrors/{ws}` | stop mirroring; the workspace stays | | GET | `/mirrors/{ws}/log?limit=` | what the last rounds did, page by page, newest first: `{changes: [{id, at, page_id, title, action, stats, changes, exists}]}` — `stats` the git-style block counts `{add, del, mod}` (`{}` on rows from before they were kept), `changes` what each edit did block by block (`[{k: add \| del \| mod \| props \| move, id, text, old?}]`) | | GET | `/mirrors/{ws}/conflicts[?resolved=1][&page=]` | the merges the engine decided on its own (kinds `merged`, `diverged`, `kept_local_edit`, `restored_remote_edit`, `page_restored`, `page_restored_from_remote`) | -| POST | `/mirrors/{ws}/conflicts/{id}` | `{choice: keep \| mine \| theirs}` | +| POST | `/mirrors/{ws}/conflicts/{id}` | `{choice: keep \| mine \| theirs}` — the text is written into the block's page as it is now, then the conflict is marked resolved; 409 (the conflict stays open) when the block refuses the write | Session only, the mirror's owner, never a guest. @@ -365,9 +371,11 @@ Session only, the mirror's owner, never a guest. | method | path | what | |---|---|---| -| POST | `/pages/{id}/publish` | publish the page to the share host Gamma Cloud names: body `{audience?: anyone \| users \| list, role?: view \| edit}` (optional; the share there, default anyone / view, applied to a new or an existing link) → `{url: "/?share=", share: {token, page_id, audience, role, users, created_by}, mirror: {ws, status, page_filter, conflicts_open, pending_local}}`. Adds the page to the workspace's filtered mirror of the share host (made on the first publication through `/auth/cloud/exchange` there), runs one round and makes the share. Workspace editors, session only, never a guest. 409 with a message when it cannot: no linked Gamma Cloud identity with a token ("Sign in with Gamma Cloud to publish."), no share host named, a workspace that is a copy of another server, a mirror owned by someone else, detached or receive-only, or this server is itself a share host; 400 for a block that is not a page; 502 when the share host refused or the page did not reach it; 503 when the account server cannot be read | +| POST | `/pages/{id}/publish` | publish the page to the share host Gamma Cloud names: body `{audience?: anyone \| users \| list, role?: view \| edit}` (optional; the share there, default anyone / view, applied to a new or an existing link) → `{url: "/?share=", public_url, share: {token, page_id, audience, role, users, created_by}, mirror: {ws, status, page_filter, mode, detached, conflicts_open, pending_local}}` — `public_url` the page's pretty address (`https://-pages.gammapdf.com/-`) when the share host reports a `page_host`, else the same as `url`. Adds the page to the workspace's filtered mirror of the share host (made on the first publication through `/auth/cloud/exchange` there), runs one round and makes the share. Workspace editors, session only, never a guest. 409 with a message when it cannot: no linked Gamma Cloud identity with a token ("Sign in with Gamma Cloud to publish."), no share host named, a workspace that is a copy of another server, a mirror owned by someone else, detached or receive-only, or this server is itself a share host, or the plan's cap there (the share host's words — "Free plan: up to 5 published pages. Unpublish one, or upgrade your Gamma Cloud plan." — plus `limit: {used, max, plan}`); 400 for a block that is not a page; 502 when the share host refused or the page did not reach it; 503 when the account server cannot be read | | DELETE | `/pages/{id}/publish` | unpublish: the share there stops, the copy there is deleted, the page leaves the filter; the page here is untouched → `{published: false, mirror}`. 409 when the page is not published; 502 (nothing changed) when the share host cannot be reached | -| GET | `/pages/{id}/publish` | `{published, can_publish, reason?, url?, share?, status?, mirror?, error?}` — `can_publish` / `reason` say whether publishing would be refused and why (the same messages as POST), `status` is the mirror's raw status (the pill's reading is the frontend's), `share` the live share settings there (`url` with them), `error` when the share host could not be read. Any member | +| GET | `/pages/{id}/publish` | `{published, can_publish, reason?, url?, public_url?, share?, status?, mirror?, limit?, error?}` — `can_publish` / `reason` say whether publishing would be refused and why (the same messages as POST), `status` is the mirror's raw status (the pill's reading is the frontend's), `share` the live share settings there (`url` and `public_url` with them), `limit` `{used, max, plan}` read from the share host whenever the account holds a publishing token (`max` null = no cap), `error` when the share host could not be read. Any member | +| GET | `/publish/limit` | the share host's half: `{used, max, plan}` — the root pages of the request's workspace (a publishing mirror's token names it) and the cap its owner's plan puts on them (`max` null = none). 404 on a server that is not a share host. Nothing cached | +| GET | `/pages/resolve-public?host=&path=` | no auth: a page host's pretty address → `{share, page_id}`, the share token the share view opens with (audience and role its own). `host` must match `GAMMA_PAGE_HOST` (the username read out of it), `path` is `/-` or `/`; only the trailing id counts, a root page with a share in that account's default personal workspace. 404 otherwise (counted like an unknown share token); 429 past 120 per IP in 5 minutes | | GET | `/integrations/oauth/request?request_id=` | the pending consent (client name, the account's workspaces) for the consent screen | | POST | `/integrations/oauth/consent` | approve or deny a pending sign-in for one workspace | | GET | `/.well-known/oauth-authorization-server`, `/.well-known/oauth-protected-resource` | OAuth discovery for MCP clients (no `/api` prefix) | diff --git a/docs/dev/cloud_accounts.md b/docs/dev/cloud_accounts.md index 5d52e48f..29e01ecc 100644 --- a/docs/dev/cloud_accounts.md +++ b/docs/dev/cloud_accounts.md @@ -7,10 +7,11 @@ their plan, billing and hosted container. It holds no pages, files or notes, and a Gamma server never calls it on a data request: an ID token is verified locally with the published keys. Code: `cloud/` (package `gammacloud`, entry `cloud/app.py`, CLI `cloud/manage.py`), tests -`cloud/tests/`. It imports nothing from `backend/`; the two small pieces it -shares with Gamma (the fixed-window rate limiter, the PKCE rules) are -copies, so the account server can move to its own repository without a -change. The rate limiter has drifted from `backend/`'s: it drops +`cloud/tests/`. It imports nothing from `backend/`. The three small pieces +it shares with Gamma are copies, so the account server can move to its own +repository without a change: the fixed-window rate limiter, the PKCE rules +and the public-URL rules (`servers.norm_url`, from +`server_settings.validate_public_url`). The rate limiter has drifted from `backend/`'s: it drops `on_first_exceed` and prunes stale keys. Status: **v0**, plus the preference profile, the linked-server list and @@ -135,15 +136,15 @@ page) and the **app** shell (a sidebar and a content column): A *Get started* checklist (account created, e-mail confirmed, signed in from a Gamma app — `app_signed_in_at`, so signing everything out does not undo it) with a progress bar, hidden once all three are done. Then the - signed-in Gamma apps and, under them, the Gamma servers the account is - linked on (`servers.of_account`): each name links to the server's + signed-in Gamma apps, and under them the Gamma servers the account is + linked on (`servers.of_account`). Each server's name links to its address, with the last time it checked in; a loopback address (the - desktop sidecar) is *This computer*, not a link. Beside them a plan card - (a placeholder pointing at self-hosting until hosted servers exist) and - an account summary: - username, e-mail state, member since, and the account id with a copy - button — what Gamma servers key on; it never changes. `?mail=failed` - (registration could not send the mail) changes the verify notice. + desktop sidecar) is *This computer*, not a link. Beside them sit a plan + card (a placeholder pointing at self-hosting until hosted servers exist) + and an account summary: username, e-mail state, member since, and the + account id with a copy button. The id is what Gamma servers key on; it + never changes. `?mail=failed` (registration could not send the mail) + changes the verify notice. - **Devices** (`/devices`): two lists. - *Gamma apps*: every live grant, titled by the machine's name when the app sent one (`device_name`), else the client's name; then the client, @@ -159,10 +160,10 @@ page) and the **app** shell (a sidebar and a content column): A row signed out leaves in place. *Sign out everywhere else* is `accounts.revoke_everything` behind a confirm. The page says what signing an app out does: its key stops working, and its Gamma server - ends the sessions it opened with it at its next hourly grant check (see - "The Gamma side" below). Dates are UTC on the server and shown in the viewer's time - zone by the page's script (`