chore(deps): bump pyjwt from 2.12.1 to 2.13.0 - #1444
Open
dependabot[bot] wants to merge 872 commits into
Open
dependabot[bot] wants to merge 872 commits into
dependabot[bot] wants to merge 872 commits into
Conversation
QBO POST creates were not idempotent. The 401 refresh path inside QuickBooksOnlineService._request re-POSTs the same body after refreshing the access token; if QBO ever partial-commits before returning 401, or if any future transport-level retry fires, the user gets a duplicate entity. Intuit's docs explicitly recommend the requestid query param for this case. Pass a UUID4 requestid on create_entity, update_entity, and send_entity_email. The same requestid is reused on the 401 retry because params is built once per logical call and threaded through _request unchanged. New tests assert: - requestid is set on every mutating POST and absent on GET /query - the 401 retry reuses the same requestid (so QBO collapses the pair) - distinct logical calls get distinct requestids - input validation in send_entity_email still short-circuits before any network attempt Investigated as part of issue #1035 (estimate duplication on first real-user onboarding). Railway logs and DB confirmed only one POST for that incident, so the duplicate the user reported was the receipt-rendering issue already fixed in #1055. This PR closes the latent idempotency gap so a future retry can never produce a real duplicate entity. Co-authored-by: njbrake <njbrake@users.noreply.github.com>
* fix(quickbooks): use application/octet-stream for /send endpoint QBO's /send endpoint requires Content-Type: application/octet-stream, not application/json. We hardcoded JSON for every QBO request, so sending an Estimate or Invoice via email reliably 500'd. Confirmed against Intuit's official PHP SDK (DataService::SENDEMAIL uses CONTENTTYPE_OCTETSTREAM). The 500 surfaces with no error envelope because Intuit's email pipeline crashes parsing JSON content-type with an empty body, so existing error handling couldn't recover anything useful. Hit during real-user usage: estimate creation succeeded, but every attempt to email it to the client returned 500 from QBO. Two retries, same error. Add a content_type kwarg to _request defaulting to application/json (every other endpoint stays unchanged) and override it from send_entity_email. Body remains empty so the new content type isn't a lie. Also adopt the exc.response.json() error-body extraction pattern from qb_create in qb_send so future failures (which will have valid JSON envelopes now that the content-type is right) surface Intuit's Fault.Error[] message instead of the bare HTTPStatusError string. Tests assert on captured httpx.Request headers via MockTransport, not the FakeQBService shortcut that bypasses _request entirely: that shortcut is why the bug never showed up in CI. Fixes #1063 * refactor(quickbooks): use isinstance for httpx error inspection CLAUDE.md prefers isinstance over hasattr/getattr at typed boundaries. The httpx.HTTPStatusError type guarantees a populated .response, so the hasattr guard was both weaker than necessary and required a # type: ignore[union-attr] comment to silence the type checker. Convert all four QB tool error handlers (qb_query, qb_create, qb_update, qb_send) to the same isinstance(exc, httpx.HTTPStatusError) pattern. Same runtime behavior, drops the type-ignore. Addresses /review-pr feedback on #1064. --------- Co-authored-by: njbrake <njbrake@users.noreply.github.com>
…1070) * fix(onboarding): tolerate flat-format USER.md, trust timezone column The onboarding heuristic at backend/app/agent/onboarding.py required bullet-format - Name: and - Timezone: in user_text. The LLM is free to rewrite user_text and frequently picks a flat heading-style format without dashes. The strict regex meant the heuristic returned False for users with a clearly populated profile, leaving onboarding stuck on pending and gating heartbeats. The timezone check also regex-grepped user_text for the timezone, even though users.timezone is a populated DB column that's the actual source of truth (set via the dashboard PUT /user/profile endpoint). The text-grep was a brittle proxy for what the DB already knew. Two changes: 1. _has_user_timezone now checks bool(user.timezone) directly. Drop the regex. 2. _has_real_user_profile loosens the regex to accept both bulleted (- Name: X) and flat (Name: X) formats while keeping the same-line constraint that prevents matching across newlines. First real user observed today with this exact pattern: 48 inbound messages, complete profile in flat format, populated timezone column, soul customized -- but onboarding_complete was still false because the heuristic regex didn't match the LLM-chosen format. Just below the 50-message hard ceiling that would have force-completed. Existing tests updated: timezone is now set on the user object explicitly (mirroring how production sets it), and a new flat-format regression test asserts the heuristic fires on the real-world shape. Fixes #1068 * docs: fix _has_user_timezone docstring to reference real write path The previous docstring claimed timezone is set by a "dedicated timezone-update tool". No such tool exists. The column is populated via PUT /user/profile (dashboard / browser onboarding flow). Updated to reference the actual mechanism. Addresses /review-pr feedback on #1070. --------- Co-authored-by: njbrake <njbrake@users.noreply.github.com>
… links (#1072) * fix(companycam): drop URL from upload_photo content to stop duplicate links The success-path ToolResult for companycam_upload_photo embedded the photo URL in its content string while also attaching the same URL via ToolReceipt. The LLM saw the URL in the tool result and copied it into prose, then the receipts appender added the canonical receipt with the URL, so users saw each photo URL twice in the reply. The receipt is the canonical channel for surfacing deep links on plain-text channels. When a tool sets ToolReceipt(url=...), the content string should describe the action without inlining the URL. Also adds a generic invariant test that exercises every CompanyCam tool returning a receipt-with-URL and asserts the URL never appears in the content string, plus a regression test for the two-photo upload case. Fixes #1069 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(quickbooks): extend no-URL-duplication invariant to qb_create/update/send Per review feedback, parallel the CompanyCam invariant test with one for QuickBooks. Adds FakeQBOServiceWithURL (subclasses QuickBooksOnlineService so _build_qbo_url returns a real deep link) and asserts that for every QB tool returning a ToolReceipt with a URL, ToolResult.content does not contain that URL. The QB tools are clean today, but this guards against a future regression that would inline a deep link in the content string. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: njbrake <njbrake@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(quickbooks): accept JSON-string data in qb_create/qb_update The LLM occasionally over-quotes deeply nested QBO payloads and emits the `data` argument as a JSON-encoded string instead of an object. The strict `dict[str, Any]` Pydantic field rejected these, forcing a retry on the next round and wasting an LLM round-trip plus output tokens. Add a `field_validator(mode="before")` that parses JSON strings into dicts on the way in. Behavior is strictly more permissive: existing dict callers are unaffected, and unparseable / non-object strings still surface as `ValidationError` so the agent's existing retry hint path runs instead of blowing up as a ServerError later. Fixes #1066 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(quickbooks): drop redundant parenthetical from data field descriptions The validator already accepts both shapes; the prompt-side hedge is overkill once the runtime is permissive. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: njbrake <njbrake@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…TZ (#1075) * fix(agent): stop promising timed reminders, anchor time math on user TZ The agent claimed it could set reminders at specific clock times, but actually stored them as text in heartbeat_text. The heartbeat system runs every 30 minutes and surfaces items in any window, not at an exact time. The user's "set reminders for noon, 3pm, 7:30am" requests therefore silently became approximate periodic check-ins. Two changes, no new infrastructure: 1. End the lie. Tool descriptions, usage hints, and the system prompt now say plainly that heartbeat is not a scheduler. For one-shot timed reminders, the agent routes to calendar_create_event when Google Calendar is enabled, or tells the user honestly that it cannot fire at exact times and offers to connect calendar or set the reminder on their phone. 2. Fix the compounding time-math bug. build_time_user_context now appends the IANA timezone to the rendered time string (e.g. "(America/Denver)"). Without this anchor, the LLM has been observed treating local times as UTC, producing wrong deltas (e.g. saying 7:30am is "minutes from now" when local was 6:27am). No new tables, scheduler, tools, or migrations. Calendar integration already exists; this PR just teaches the agent to use it for the "remind me at 2pm" use case. Fixes #1067 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(calendar): add reminder_minutes_before so timed reminders fire on time The first commit told the agent to call calendar_create_event "with a short pre-event alert," but the tool API did not actually accept a reminder parameter. Events created via the tool inherited the user's Google Calendar default reminders (typically 30 min before start), so "remind me at 2pm" would fire at 1:30pm. Promise was unfulfillable. Plumb a new optional `reminder_minutes_before: int | None` field through: - CalendarCreateEventParams (the agent-facing tool schema) - CalendarEventCreate (the service DTO) - _build_event_body (maps to Google's `reminders.overrides`) Semantics chosen to preserve existing callers: - None (default): no `reminders` field on the request body, Google applies the user's default reminders. Existing behavior preserved. - 0: popup fires at the exact event start. This is the path for "remind me at 2pm". - N > 0: popup fires N minutes before start. Capped at 40320 (4 weeks) per the Google Calendar API spec for `reminders.overrides[].minutes`. Updates the agent prompts (instructions.md, proactive.md, and the heartbeat_tools description) to point at the new parameter explicitly: "call calendar_create_event with reminder_minutes_before=0 so the popup fires at the exact moment." Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(agents): treat user-chosen assistant names as soft PII Each user can rename their Clawbolt assistant via SOUL.md to anything they like. When that custom name shows up in a session transcript and gets quoted into commits, PR bodies, or GitHub comments, it ties the content back to a specific user. Add a paragraph to the Privacy & PII section telling future contributors (human and AI) to generalize to "Clawbolt" or "the agent" in their own copy, while still allowing verbatim quotes of upstream messages. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(agents): tighten soft-PII note Cut the soft-PII paragraph from ~100 words to one sentence. The rule itself is small; the long version paid the token cost on every load without adding clarity. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(agent): drop hardcoded 30-min heartbeat interval from prompts User.heartbeat_frequency is per-user configurable, but my earlier copy in instructions.md, proactive.md, and the update_heartbeat tool description claimed the system "runs every 30 minutes." For users who have changed their cadence, that statement is wrong and could lead the agent to give the user inaccurate guidance about when items will fire. Reword to "checks on the user's configured interval and surfaces items within a window, not at an exact clock time." The "not at an exact clock time" framing is the load-bearing part for issue #1067 and is preserved. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(agent): generalize proactive.md timed-reminder routing The proactive.md framing should describe the rule of thumb without binding the agent's general behavior to a specific integration name. The concrete tool guidance still lives in instructions.md (which names calendar_create_event and reminder_minutes_before) where the operational detail belongs. proactive.md now just states the rule: the heartbeat system cannot fire at exact times, so route via a timed-reminder integration if one is connected, otherwise be honest. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(agent): tighten Timed reminders section in instructions.md Trim ~110 words to ~60. Drop redundant "you cannot fire reminders at exact clock times yourself" (already covered by "not a scheduler"), explanatory "Google's notification system delivers the popup" (the agent doesn't need to explain the implementation), and the "end can be a few minutes after start" detail (the agent can pick a sensible duration). Also drop the "Google Calendar" framing in favor of the more general "calendar tool / calendar integration" so the copy survives a future Apple Calendar or other provider integration. HEARTBEAT.md bullet trimmed similarly: cross-reference the Timed reminders section instead of restating the rule. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(agents): generalize soft-PII rule Drop the user-chosen-assistant-names example. The principle is "use judgment for content that ties to a specific user even if it's not on the hard list"; enumerating examples invites the wrong behavior of treating the list as exhaustive. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(agents): drop verbatim-quote carve-out from soft-PII rule The carve-out undermined the rule it lived under. The point of treating soft PII carefully is to not propagate it into new persistent artifacts; a verbatim quote in a new PR body is a new artifact carrying the soft PII forward. If a verbatim quote is ever unavoidable, judgment can decide on the spot. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: njbrake <njbrake@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…#1071) * feat: envelope-encrypt credential columns with pluggable KEK provider Replaces the single-Fernet-key EncryptedString with envelope encryption: each row gets a fresh DEK that's wrapped by a configurable KEKProvider and stored inline alongside the ciphertext. OSS ships LocalKEKProvider backed by ENCRYPTION_KEY; premium plugins override via the auth loader to plug in KMS-backed per-tenant wrapping. The driver isn't urgent live-prod compliance: it's that the migration is trivial today (one user, a handful of rows) and gets exponentially harder once the premium deployment onboards more tenants. Same reasoning applies to the structured credential.read log line added at OAuthService read sites: a small forensic down-payment that's cheap now and useful later. - Inline envelope format (clw1.<kek_id>.<b64-wrapped>.<fernet>) means any EncryptedString column gets envelope encryption with zero schema change. - Migration 018 re-keys existing rows; idempotent re-runs are safe. No automatic downgrade -- restore from backup if needed. - EncryptedString fails loudly on a non-envelope read so a missed migration is visible at startup instead of silently corrupting decrypted tokens. - LocalKEKProvider with no ENCRYPTION_KEY generates an ephemeral process-local key and warns. Stored credentials become unreadable after restart -- the loudest signal that the operator forgot to configure a key. Fixes #1065 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: remove dead ENVELOPE_VERSION constant The version is already encoded in the ENVELOPE_PREFIX (``clw1``); having both invites them to disagree. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: cover legacy-Fernet decrypt path in migration 018 The previous migration test only exercised the plaintext-passthrough branch (legacy=None). The branch most production rows actually take -- decrypt with the pre-envelope HKDF/Fernet derivation, then re-encrypt under the new envelope -- was untested. A bug there could silently re-encrypt Fernet ciphertext as if it were plaintext, leaving the row syntactically valid but pointing to garbage. Adds two tests: - Legacy Fernet round-trip: encrypt under the old derivation, run _rekey with the legacy Fernet, assert the new envelope decrypts back to the original plaintext. - InvalidToken fallback: pass a non-Fernet value with a legacy Fernet configured, assert it's treated as plaintext (documenting the rotated-key risk that backup-restore is the recovery path for). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: allow premium plugin's get_kek_provider() to return None (#1074) * feat: allow premium plugin's get_kek_provider() to return None The KEK provider loader treated the plugin's hook as authoritative: if the plugin module exposed get_kek_provider(), its return value was cached and used unconditionally. That coupled the plugin code merge to the infra rollout: shipping a premium provider class meant the env vars (KMS ARN, AWS creds) had to be wired the same day or the plugin would crash on first credential read. This change lets the plugin signal "not configured yet" by returning None. The loader then falls back to LocalKEKProvider as it would without a plugin at all. Premium can ship the KMS provider dormant, and platform engineering activates it later by setting env vars and restarting -- no code change required. Stacks on #1071 (introduces get_kek_provider). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: hoist sys/types imports + extract fake-plugin helper CLAUDE.md requires all imports at the top of the file. The two new tests had inline ``import sys`` / ``import types`` inside their bodies. Moves both to the module-level import block. While there, extracts the fake-plugin install pattern into a small helper so the two tests don't duplicate it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: njbrake <njbrake@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: njbrake <njbrake@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The compaction loop called any-llm `amessages` directly without writing to `llm_usage_logs`, so trim-based and session-end consolidation calls were invisible in the per-user usage breakdown. As we lean on the "infinite session" model, compaction is a non-trivial fraction of spend per user, and undercounting it breaks the cost picture. Log usage right after the successful `amessages` call (inside the try/except so failures stay no-ops). Use the resolved `model` so the `compaction_model` override is reflected in the breakdown. Fixes #1076 Co-authored-by: njbrake <njbrake@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ues) (#1079) * feat(onboarding): bundle UX fixes from first-user observation Fixes 12 issues from the 2026-04-28 first-user observation session. Onboarding flow / copy / branching: - #1050 onboarding too long: rewrite bootstrap.md to require only name + timezone, then delete BOOTSTRAP.md immediately. Drop the capabilities tour and the personality interrogation. - #1044 remove "How do you like to talk?" step: personality question is gone from bootstrap. The default soul is loaded at provision time and stays unless the user customizes. - #1043 photo access copy: bootstrap.md and the get-started page both reassure once that Clawbolt only sees photos the user sends. - #1040 hide Telegram from the default flow: getVisibleChannels drops Telegram entirely when telegram_bot_token_set is false. - #1029 hide unavailable channels (don't grey out): drop the isDisabled codepath in the wizard radio item; visibleChannels is the gate. - #1037 don't re-ask about already-connected tools: move the "call manage_integration(action='status') first" rule from bootstrap.md (which gets deleted on completion) to the durable manage_integration usage_hint. UI polish: - #1041 OAuth callback CTAs: replace tertiary <Link> on both success and error states with a primary <Button> (full width on mobile, auto on desktop). - #1039 mobile QR is the wrong UX (same-device problem): add a prominent "Open Messages" sms: deep-link button above the QR on mobile, plus tap-to-copy on the phone number. Desktop still shows the QR for cross-device pairing. - #1038 phone number entry: new normalizeUsPhone helper auto- prepends +1 to bare US digits, strips formatting, accepts a leading + verbatim, and shows an inline E.164 validation error on submit. Applied to PremiumChannelLinkForm, OssLinqForm, and BlueBubblesForm. Mobile reframe: GetStartedPage detects the mobile breakpoint (new useIsMobile hook) and renders a single-screen flow on phones -- phone input + "Text Clawbolt" button that calls setLinqLink / setBlueBubblesLink and then opens sms: -- replacing the 4-step wizard. Desktop keeps the wizard. Agent behavior: - #1046 tone down initial-response affirmations: bootstrap.md and instructions.md now ban "Great question!", "Absolutely!", "I'd love to help!", "Happy to!" with positive examples of the desired direct tone. - #1047 vision only on explicit ask: tighten analyze_photo description with concrete positive triggers ("what is this", "estimate from this") and explicit negative cases ("DO NOT call when routing to CompanyCam, attaching to a job"). Also tighten the system_prompt.py:131-141 vision-routing sentence so the description change is not shadowed. - #1045 typing indicator already fires immediately on inbound via ingestion._send_early_typing_indicator. Continuous-refresh stretch deferred (cancellation complexity not worth it for this PR). Carved out as follow-ups: - #1048 comfort-level permission setting: reframe to "audit per- tool ASK defaults and flip to ALLOW where approval is not load-bearing" before adding a per-user toggle. - #1051 just-in-time feature explainers: real multi-week feature. - mobile-pairing-setup-code: auto-grab user phone from first inbound via one-shot setup code (eliminates phone-number entry on mobile entirely). Tests: - New: phone.test.ts (9 cases for normalize + validate), channel-utils.test.ts (6 cases including empty-config and Telegram-hidden semantics). - Backend regression test for #1037 in test_integration_tools.py asserting the manage_integration usage_hint instructs the agent to call status before offering connects. - Updated existing GetStartedPage / ChannelsPage / prompts / session-endpoints tests for the new behavior. DoD: - pytest 1975 passing - ruff check + format clean - ty check clean - frontend tsc + knip clean - frontend tests for touched files all pass Visual Playwright verification not run -- the OSS app validates LLM credentials at startup and the sandbox has no key. The 56 frontend tests covering the changed components plus the structural nature of the changes (mobile/desktop layout split, copy edits, phone-input parsing) make the regression risk low, but visual sign-off should happen on njbrake's side before merge. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(onboarding): rethink bootstrap purpose; generalize vision guidance Two review-feedback fixes. 1. Bootstrap was incoherent as written. The "delete BOOTSTRAP.md immediately on name+tz capture" rule meant onboarding could end in 2 turns, before the dictation hint, photo-access reassurance, or any tone-shaping had a chance to land. Once BOOTSTRAP.md is deleted, those guidance bits are gone from the system prompt forever, so a 2-turn exit defeats the point of having a bootstrap at all. Replace the exit rule with a softer two-condition gate: name + tz captured AND the conversation has texture beyond data capture (answered something useful, or a "things worth weaving in" moment has come and gone). Restore the "things worth weaving in" section so the LLM has something substantive to do during the onboarding window. Trust the LLM to judge "feels established" rather than forcing a transactional exit. 2. analyze_photo description and the system_prompt media-handling section were enumerating specific user phrases ("what is this", "estimate from this", "save to the Acme job") and named integrations (CompanyCam) as positive/negative triggers. That encodes our specific integrations and presumes user intent. Drop the examples; replace with the general rule: don't analyze unless the user asked or there's a clear need to see the contents. DoD: 1975 tests pass, ruff check + format + ty all clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(onboarding): system-driven exit + BlueBubbles email + OSS persist Five review-feedback fixes across the onboarding fixup PR. 1. System-driven bootstrap exit (the big one). Bootstrap.md no longer tells the LLM when to call delete_file; the system removes BOOTSTRAP.md itself once name + timezone are captured AND the user has sent at least MIN_USER_MESSAGES_FOR_AUTO_EXIT (=4) messages. The message floor ensures the conversation has texture even when a maximally-cooperative user supplies name + tz in turn 1, so the "things worth weaving in" content (dictation hint, photo policy) has a chance to land before the bootstrap context disappears. Five completion paths in OnboardingSubscriber, evaluated in order: (1) defense-in-depth (BOOTSTRAP.md missing for any reason); (2) auto-exit, primary path (name + tz + >=4 messages); (3) strict heuristic backstop for mid-flight users from the pre-2026-04 bootstrap who customized SOUL.md; (4) hard ceiling at 50 messages; (5) pre-populated user. Bootstrap.md gets a "You don't decide when onboarding ends" section telling the LLM not to call delete_file or announce a transition. Adds a priority signal for the things-worth-weaving-in section (dictation is the most universally useful). Em dash removed (CLAUDE.md compliance). New tests: auto_exit fires on name+tz+>=MIN_TURNS; auto_exit does not fire below the message floor; auto_exit does not fire without timezone. 2. BlueBubbles email-address sms: link. When the BlueBubbles backend is configured with an iCloud email, ``sms:user@icloud.com`` is malformed and most OS handlers silently reject it. Detect the email shape (``includes('@')``) in MobileGetStarted and TextAssistantCard; render a copy-the-address UX with explicit "send a note from your iCloud-connected device" copy. Hide the QR on desktop when address is an email (QR encoding the bad sms: URI was useless anyway). 3. OSS phone persistence in mobile flow. Previously the mobile "Text Clawbolt" button only persisted the phone for premium users; OSS users typed their number, the button cleared, and their first inbound was silently rejected by the backend's allowed_numbers gate. Now OSS users persist via updateChannelConfig({linq_allowed_numbers: ...}) or the bluebubbles equivalent. 4. normalizeUsPhone 10-digit edge case. A bare 10-digit input like ``1234567890`` previously stripped the leading 1 and produced ``+1234567890`` (9-digit body) which passed the loose E.164 regex. Now drop the leading 1 only when the digit string has exactly 11 chars; 10-digit input stays as-is so the malformed ``+11234567890`` is caught by the validator instead of silently shifting the user's intent. 5. Mobile-flow test coverage. New describe block in GetStartedPage.test.tsx mocks window.matchMedia for the mobile breakpoint and asserts: single-screen layout renders (not wizard); phone validation error on bad input; OSS persists via linq_allowed_numbers; OSS bluebubbles persists via bluebubbles_allowed_numbers; empty-state copy when no iMessage backend is configured. DoD: 1978 backend tests pass (+3), ruff/format/ty clean, frontend typecheck/knip clean, 62 frontend tests for touched files pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: untrack PLAN_onboarding_fixup.md (working scratch, not landed code) --------- Co-authored-by: njbrake <njbrake@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…gured (#1080) Bug from dev testing: a user had Telegram configured (no iMessage backend) and got the "ask your admin to enable iMessage" empty state on the mobile get-started page. The mobile flow was hard-gated on ``imessageBackend !== null``; Telegram was a complete blind spot. Fix: 1. ``MobileGetStarted`` now branches by available channels. New ``MobileImessageFlow`` and ``MobileTelegramFlow`` siblings replace the inlined iMessage logic. When both backends are configured, a tab toggle (``MobileChannelToggle``) lets the user pick; default is iMessage (most clawbolt users on mobile reach for Messages first). When only one is configured, the toggle is hidden and the flow renders directly. 2. ``MobileTelegramFlow``: numeric Telegram user ID input with the "send /start to @userinfobot" help text; "Open Telegram" button that persists the ID (``setTelegramLink`` for premium, ``updateChannelConfig({telegram_allowed_chat_id})`` for OSS) and then opens the bot via the deep link from ``api.getTelegramBotInfo()``. Linked card mirrors the iMessage pattern with a re-open Button and copy of the bot username. 3. Empty-state copy is now channel-agnostic: "No messaging channels are configured on the server yet. Use web chat for now, or ask your admin to enable iMessage or Telegram." 4. ``activeChannel`` is derived rather than seeded from props at mount. The previous ``useState`` initializer ran once when ``channelConfig`` was still undefined and locked the wrong default. New pattern: store only the user's explicit toggle choice; fall back to ``imessageAvailable ? 'imessage' : 'telegram'`` which re-evaluates as ``channelConfig`` loads. Tests: 4 new mobile-flow tests covering Telegram-only render, toggle when both available with iMessage default, OSS persists Telegram via ``updateChannelConfig``, non-numeric ID rejected inline. Existing 19 tests still pass. DoD: 1978 backend tests, ruff/format/ty clean, frontend typecheck/knip clean, 23 frontend tests on GetStartedPage. Co-authored-by: njbrake <njbrake@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add a no-DB-touching ``/api/health/live`` endpoint as a safer target
for the deployment platform's healthcheck path. Today's ``/api/health``
opens a sync DB session and runs ``SELECT 1`` from inside an async
handler. When the event loop is blocked (sync DB call backed up under
load, slow third-party HTTP call, etc.), the healthcheck call piles up
on the same blocked worker and starts returning 502 to Railway's edge,
which then refuses to roll traffic to a fresh container. Worse: the
existing /api/health is the path Railway actually probes during deploy.
The new ``/api/health/live`` returns ``{"status":"ok",
"database":"not_checked"}`` instantly without acquiring any DB
connection. Premium will switch ``railway.toml``'s healthcheckPath to
this in a follow-up PR. ``/api/health`` stays for richer "is the
system actually working" checks (ops dashboards, monitoring).
Co-authored-by: njbrake <njbrake@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The /autoplan skill writes PLAN_<topic>.md files at repo root as ephemeral working state. PLAN_onboarding_fixup.md from a previous session was sitting untracked at the root. Add the pattern so future planning docs do not surface in git status, and remove the stale one. Co-authored-by: njbrake <njbrake@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ded retries (#1083, #1088) (#1089) * fix(channels): bluebubbles reliability - typing fire-and-forget + bounded retries Two BlueBubbles fixes that combine to turn "BlueBubbles flakiness drops user replies" into "BlueBubbles flakiness adds a few seconds of delay". #1083: typing indicators block reply delivery The outbound dispatcher in ChannelManager._run_outbound_dispatcher was awaiting typing-indicator calls inline. Because the dispatcher is a single asyncio task, every queued reply waited behind the typing call. Production observation: an agent reply sat ~27s before being delivered, behind a 30s typing-indicator timeout. Fix: typing/stop_typing are now spawned as background tasks via a new _spawn_typing_task() helper. Tasks are tracked on self._typing_tasks and cancelled in stop_all(). BlueBubbles typing calls also get an explicit 3s timeout so orphan tasks self-terminate quickly. #1088: zero retry on transient send failures The dispatcher catches Exception from send_text/send_media and silently drops the reply. Self-hosted BlueBubbles servers running on consumer hardware (the maintainer's basement Mac) hit transient failures regularly: Mac waking from sleep, brief WiFi blip, BlueBubbles process restart. Without retry the user never sees the reply. Fix: new _post_with_retry helper. 4 attempts (1 initial + 3 retries), 1s/2s/4s backoff, 10s per-attempt timeout. Retries on httpx.ConnectError, ReadTimeout, WriteTimeout, PoolTimeout, RemoteProtocolError and 5xx responses. Never retries 4xx (caller error). tempGuid is generated once before the retry loop so BlueBubbles client-side dedupe stops a partial-success retry from creating a duplicate iMessage. Typing indicators are intentionally not retried. Worst-case timing: if BlueBubbles is fully unreachable, send_text spends up to ~47s exhausting retries before giving up. Acceptable because most basement-Mac flakes are sub-10s and clear on the first retry. Closes #1083, #1088 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(openapi): regenerate frontend types for /api/health/live PR #1081 added the /api/health/live liveness endpoint but did not regenerate frontend/openapi.json or frontend/src/generated/api.d.ts, which broke the frontend-lint check on main. Including the regeneration here so this PR can pass CI; the change is unrelated to the BlueBubbles reliability work but blocks merge. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: njbrake <njbrake@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…se (#1091) Closes mozilla-ai/clawbolt-premium#332. In premium prod, tenants are routed via tenant-specific bots and the global TELEGRAM_BOT_TOKEN env var is empty, so the OSS /api/channels/telegram/bot-info endpoint returns 404 by design. The mobile get-started page and admin Channels page still called it on every load, producing four 404s per session. useChannelStates fired useTelegramBotInfo(isPremium) without checking whether a bot token was configured. Gate the query on channelConfig.telegram_bot_token_set so the call only fires when the endpoint can actually return useful data. Adds a regression test in ChannelsPage.test.tsx that mocks telegram_bot_token_set=false and asserts api.getTelegramBotInfo is not called. Co-authored-by: njbrake <njbrake@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
#1092) * perf(oauth): cache load_token reads for 30s to dedupe per-turn DB hits Production logs showed 6+ credential.read action=load entries per inbound message. A single agent turn reads each integration's token multiple times: once during specialist auth_check at registry build, again inside factory.create() when the OAuth client is constructed, and again per actual tool invocation. Each read was a fresh DB roundtrip. Add a per-process TTL cache (30s) keyed on (user_id, integration). Both positive and negative results are cached so unconnected integrations also stop spamming the DB. Cache invalidation: - save_token drops the entry so a refresh's new value is observed. - delete_token drops the entry so the next load returns None. - refresh_token explicitly drops the entry before its post-advisory- lock reload. The whole point of that reload is to detect a peer worker's just-persisted refresh; a stale cache would mask it and cause a redundant HTTP refresh that may overwrite the rotated refresh_token. Cross-worker staleness is bounded by the 30s TTL. Within that window a worker may return a token whose access_token was rotated by a peer, but the cached access_token is still valid until its own expires_at, so callers behave correctly. Tests: - load_token_cached_within_ttl - save_token_invalidates_cache - delete_token_invalidates_cache - load_token_caches_negative_lookup - refresh_token_bypasses_cache_for_post_lock_reload (cross-worker race) Closes #1085 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(oauth): address review feedback on token cache Three changes from the self-review of PR #1092: 1. Bug: build_on_refresh_callback._persist also reloads under the advisory lock to detect a peer worker's just-persisted token. Without bypassing the cache there, the same race the refresh_token bypass was meant to prevent could still occur via the on-refresh callback path. Two providers (QuickBooks, Google Calendar) refresh via this callback when a tool call hits 401, so this is a real prod hazard, not theoretical. 2. Refactor: extract _load_token_uncached() helper. Both post-lock sites now call it, so future post-lock callers will not silently regress. 3. UX: shorter TTL for negative cache entries (_NEGATIVE_TOKEN_CACHE_TTL_SECONDS = 5.0). Cross-worker OAuth completion no longer leaves a 30s window where one worker reports the user as still unconnected. Positive entries keep the 30s TTL since they are the dominant per-turn dedup case. Also expands the OAuthService docstring to document cache lifetime, invalidation, and the post-lock bypass requirement so future readers understand the cross-worker semantics. Adds regression test test_build_on_refresh_callback_bypasses_cache_for_post_lock_reload. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: njbrake <njbrake@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… expiry (#1093) * feat(oauth): background scheduler proactively refreshes tokens before expiry Closes #1087. When a user message arrived during the 5 minute pre-expiry window, get_valid_token would refresh inline on the user-facing path, adding ~150ms to the reply latency. With three integrations per user (CompanyCam, QuickBooks, Google Calendar) and tokens that typically last an hour, every active user hit this path several times an hour. Add OAuthRefreshScheduler that sweeps every 2 minutes, finds tokens expiring within the next 6 minutes (one minute past the 5 minute inline threshold so the sweep always wins the race), and refreshes them in the background. Inline refresh in get_valid_token remains as a safety net for tokens the sweep missed (cold start, sweep paused, etc.). The sweep: - skips tokens with expires_at <= 0 (non-expiring tokens never need refresh) - skips tokens without a refresh_token (cannot be refreshed) - continues past individual failures so one user's revoked grant does not block all other users' background refreshes - reuses the existing refresh_token() advisory-lock path so a concurrent inline refresh in another worker does not double-refresh Wired into the OSS lifespan in backend/app/main.py. Premium will pick up the same scheduler in a follow-up once OSS_REF is bumped. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(oauth): address review feedback on refresh scheduler Three changes from the self-review of PR #1093: 1. Hoist oauth_refresh_scheduler import to the top of main.py per the project rule that all imports live at module top (no inline or deferred imports inside functions, except TYPE_CHECKING). The two inline imports inside lifespan() are now a single top-level import. 2. Add random jitter (+/- 15s) to the inter-sweep sleep so schedulers running in N uvicorn workers do not synchronize and stampede the DB and advisory locks at the same instant. Effective interval stays in the [105s, 135s] range. 3. Add test_scheduler_start_is_idempotent covering the lifecycle contract that calling start() twice does not spawn a second sweep task. Locks in the existing guard so a future regression that removes it gets caught. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(oauth): gate background refresh on recent user activity Address security concern raised during review: refresh tokens for some providers (notably QuickBooks, 100 day inactivity expiry) are intentionally designed to expire when a user stops using the integration so re-consent is required on return. Background refresh as originally proposed would silently keep dormant accounts' refresh tokens alive forever, bypassing that signal. Gate the sweep on recent activity: only refresh tokens for users whose most recent channel_routes.last_inbound_at is within the _REFRESH_ACTIVITY_WINDOW_DAYS (14 day) window. Inactive users still get the inline refresh path on their next interaction; if their refresh token has naturally expired, the inline path detects it and prompts re-auth as the provider intended. Activity signal is "received an inbound message via any channel route" -- captures iMessage, Telegram, webchat, etc. Two new regression tests: - test_refresh_sweep_skips_inactive_users - test_refresh_sweep_skips_users_with_no_channel_route Existing sweep tests updated to mark their test_user as recently active so they continue to exercise the happy path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: njbrake <njbrake@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes mozilla-ai/clawbolt-premium#333. Production logs from a short user session showed 4-6 GET /api/user/profile 401 responses paired with POST /api/auth/refresh on every page load. The existing reactive flow does the right thing functionally (401 -> refresh -> retry) but produces noisy 401s in network panels and adds a round trip per page load near token expiry. Decode the access token's `exp` claim when it is set, and have the openapi-fetch middleware refresh proactively when the current time is within 30 seconds of expiry. The reactive 401 path remains as a safety net for tokens whose exp we could not decode (opaque tokens, JWT parsing failure) and for any race where the token expires between the proactive check and the request hitting the server. The proactive-refresh decision is extracted to a small _shouldProactivelyRefresh helper for direct testing; the JWT decode is similarly exposed. Tests: - _decodeJwtExp: valid/invalid/missing-exp/unparseable - _shouldProactivelyRefresh: inside leeway, outside, null exp - setAccessToken: tracks/clears exp, handles opaque tokens - tryRefresh: dedup, 401 clears tokens, no-refresh-token early-return Co-authored-by: njbrake <njbrake@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…1095) Closes #1084. Production logs showed the agent's first turn after a user idle gap always logged hit_ratio=0.00 with cache_create > 0 and cache_read=0. The next turn within the same session correctly hit cache. The cause is not a non-deterministic prompt; the stable prefix is byte-stable per user. The cause is the default Anthropic ephemeral cache TTL of 5 minutes: any user with a >5min gap between messages misses on their first turn back. Switch to the 1-hour extended TTL via cache_control.ttl="1h": - 1h cache_create costs 1.5x normal (vs 1.25x for 5min ephemeral). - Cache reads are unchanged at 0.1x normal. - For a user whose median inter-message gap is >5min and <1h, hit rate goes from ~0% to ~100% on those returning turns; net per- message cost drops by roughly 35%. - For users actively messaging within 5min: identical behavior. Adds settings.llm_cache_extended_ttl (default True) as an escape hatch in case a non-Anthropic provider rejects the ttl field. Anthropic itself silently ignores unknown cache_control keys per their docs. Tests cover both TTL modes and confirm the cache_boundary split still applies cache_control to only the stable prefix. Co-authored-by: njbrake <njbrake@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Compaction is a routine operation, not edge case. For an active user
at ~15k tokens/day, the 400k token trim threshold lands every ~27
days; for a power user at 5x that rate, every 5-6 days. Across a
multi-tenant deployment, compactions fire frequently enough that
their cost and quality matter.
Add a single space-delimited key=value log line per run so log
aggregators (Railway, Loki) can group / filter without JSON parsing:
compaction.summary user=<id> trimmed_count=N trimmed_chars=N
input_tokens=N output_tokens=N duration_ms=N
memory_updated=bool user_updated=bool soul_updated=bool
summary_len=N
Fields support three audit angles:
- Frequency: count occurrences per user per week.
- Cost: sum input/output tokens; multiply by current model price.
- Quality: rate of all-False *_updated flags. Persistent emptiness
signals an upstream issue (agent isn't surfacing facts in
conversation, or the prompt isn't extracting them).
Two regression tests pin the field set so future changes are
intentional.
Co-authored-by: njbrake <njbrake@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to #1095 which added the setting but missed the env example and configuration docs. test_env_example failed on main with: AssertionError: Settings fields missing from .env.example: ['LLM_CACHE_EXTENDED_TTL'] AssertionError: Settings fields missing from docs/self-host/configuration.md: ['LLM_CACHE_EXTENDED_TTL'] Adds both entries with the same description: 1-hour Anthropic cache TTL by default, set to false on non-Anthropic providers that reject the ttl field. Co-authored-by: njbrake <njbrake@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(agent): execute parallel tool calls within a turn Run all approved tool calls from a single LLM turn concurrently. Tools that share a non-None ``concurrency_group`` serialize within that group in submission order; tools with different keys (or ``None``) fan out via ``asyncio.gather``. The model is responsible for sequencing dependent calls across turns; within a turn we only fan out what the model asked for. Why now: the agent often emits 5-10 read-only tool calls in a single turn (multiple ``calculate``, ``read_file``, calendar reads). Running them serially adds up to seconds of latency for work that has no real dependency between calls. Why ``concurrency_group``: workspace document writes (USER.md, SOUL.md, MEMORY.md, HEARTBEAT.md), heartbeat updates, and reply senders touch shared state another tool could touch in the same turn. A static serialization key per Tool keeps fan-out correct without trying to infer races at runtime. Existing groups: ``workspace_writes`` for workspace mutations, ``user_outbound`` to preserve reply order. Other changes: - typing indicator now fires once per tool round instead of once per tool, since concurrent tools would otherwise emit a burst of identical signals. - result and record append order continues to follow ``approved_entries`` position, not completion order, so the inspection UI and any downstream consumer see tools in the order the model emitted them. - registry-wide test asserts every ``MODIFIES_PROFILE`` or ``SENDS_REPLY`` tool declares a concurrency_group so future contributors cannot silently regress this. - AGENTS.md (CLAUDE.md is a symlink) gains a checklist step covering when to set ``concurrency_group``. Approval flow is intentionally untouched: the sequential approval pass stays as-is and is a follow-up. Fixes #1098 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(agent): address self-review of parallel tool execution Followups from the PR #1099 review. 1. Allow ``Tool.concurrency_group`` to be a callable on validated args. Mirrors ``ApprovalPolicy.resource_extractor`` and lets one Tool route distinct calls to distinct buckets. Workspace writers now key by file path (``workspace_path:USER.md``), so two writes to different files parallelize while two writes to the same file still serialize. Heartbeat updates share the ``workspace_path:HEARTBEAT.md`` key with the workspace tool, so a heartbeat update and a workspace write to HEARTBEAT.md cannot race. 2. Extract ``_execute_single_tool`` from the inner closure into a private method on ``ClawboltAgent``, and lift ``_ToolEntry`` plus the bucketing logic to module level as ``_resolve_concurrency_group`` and ``_bucket_by_concurrency_group``. The scheduler is now a pure function exercised directly by ``test_bucket_by_concurrency_group_pure_function``. 3. Tag ``manage_integration`` with ``concurrency_group="user_integrations"``. It mutates the per-user ``tool_configs`` row and the OAuth token store; two enable/disable calls in a single turn must serialize. External-API mutators (Calendar, QuickBooks, CompanyCam) intentionally stay untagged. Different entities do not race against each other; if the LLM batches duplicate-id mutations the upstream API surfaces its own conflict, which is the correct corrective signal. Tests: - ``test_callable_concurrency_group_keys_by_args``: same-key calls serialize, different-key calls overlap. - ``test_resolve_concurrency_group_handles_string_callable_and_none``: resolver contract pinned for static, callable, and None. - ``test_bucket_by_concurrency_group_pure_function``: bucketing invariants exercised without an agent. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: njbrake <njbrake@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
) * feat: add data_sharing_consent + dedicated setter endpoint Privacy gate for user-research / admin-content visibility, ahead of the clawbolt-premium /admin/shared-data router (item 3 of premium issue #325). OSS-side change here: column + migration + endpoint + tests + regenerated OpenAPI / TS types. Premium consumes the column in a follow-up PR after this lands. What changed: - ``User.data_sharing_consent: bool`` (NOT NULL, default False) and ``User.data_sharing_consent_at: datetime | None`` (nullable). The defaults are intentionally opt-out — existing users see no behavior change until they explicitly toggle on. ``server_default='false'`` on the migration so the backfill runs at the database level. - Migration ``019_add_data_sharing_consent`` (down_revision: ``018``). - ``GET /api/user/data-sharing-consent`` returns current state. ``PUT /api/user/data-sharing-consent`` toggles the bool AND always stamps ``data_sharing_consent_at`` with the current UTC time — whether the value changed or not, whether opting in or opting out. This makes the column track "last toggled at" rather than "first opted in at" — the cheaper guarantee to keep correct: a one-shot accidental double-PUT can't drift the meaning. - ``UserProfileResponse`` exposes both fields read-only. ``UserProfileUpdate`` deliberately does NOT accept consent — routing it through the generic patch endpoint would lose the timestamp guarantee. The dedicated route is the only writer. Tests: - Default state on a new user: ``False`` + ``None`` (regression for the migration's nullable=False boolean). - Opt-in stamps timestamp; opt-out also stamps timestamp (the symmetric guarantee). - ``PUT /user/profile`` silently strips ``data_sharing_consent`` even when bundled with other fields, so the backdoor is closed. Why this lives in OSS rather than premium-only: ``User`` is the canonical row and has to grow the column for premium to read it. Premium pins ``OSS_REF`` to a clawbolt commit in CI; once this PR merges, premium bumps the ref and ships the gated admin router that reads (and PII-redacts) message content for consenting users. 🤖 Generated with [Claude Code](https://claude.com/claude-code) * fix: address review feedback on #1100 Three changes from review pass: 1. Drop em dashes from comments and docstrings to match the OSS CLAUDE.md style rule. Eight instances across migration 019, models.py, schemas.py, user_profile.py, test_profile_endpoints.py. 2. Pin extra='ignore' on UserProfileUpdate.model_config so the silent-strip contract for ``data_sharing_consent`` survives a future pydantic global-default change. The dedicated-endpoint test relies on this contract; without the explicit pin, a pydantic upgrade that flipped the default to "forbid" would start returning 422 instead of silently dropping the field. 3. Extract a ``_data_sharing_consent_now`` helper so tests can pin the clock to a known instant. New test asserts the helper is the only source of "now" the route consults, by monkeypatching it to a fixed datetime and verifying the response's timestamp matches. Regenerated frontend openapi spec and TS types to reflect the new ``model_config`` declaration.
…1101) * feat: envelope-encrypt messages.body and processed_context at rest Item 4 (envelope encryption) of the clawbolt-premium privacy redesign, issue #325. Switches the two user-content columns on the ``Message`` table from plaintext ``Text`` to ``EncryptedString``. Per-row DEKs are wrapped by ``LocalKEKProvider`` in OSS and by KMS-backed providers in premium — the same pattern PR #1071 introduced for OAuth tokens. What's encrypted - ``Message.body`` — the raw text the user / channel sent. - ``Message.processed_context`` — the same content after media transcription / OCR / preprocessing. Encrypting body without this would be theater: ``processed_context`` is what the agent reads on every turn and contains the full content surface. What's NOT encrypted (and why) - ``tool_interactions_json`` — structured tool args/results. Often contains user content, but encrypting it complicates future tool- replay debugging and the premium audit log already redacts it before surfacing to admins. Tracked for a follow-up. - ``external_message_id`` — channel-side ID; needs to be searchable in cleartext for inbound webhook idempotency. - ``media_urls_json`` — pointers, not bytes. Migration 020 - Stacks behind ``019`` (the data_sharing_consent column from a separate in-flight PR). If ``019`` doesn't land, this rebases to depend on ``018`` directly. - Streams rows in 1000-row batches with ``WHERE id > :last`` so a multi-million-message database doesn't load the whole result set into memory. - Idempotent on re-run: rows already in envelope format are returned by identity from ``_rekey``, the loop detects "nothing to do" via ``is`` comparison, and no UPDATE fires. - No automatic downgrade: rolling back would require unwrapping every envelope at a moment in time, which fails if the KEK rotates between upgrade and downgrade. Restore from backup is the recovery path. Deployment ordering: run ``alembic upgrade head`` BEFORE rolling out the new application code. ``EncryptedString`` raises on a non-envelope read, so a code-deployed / migration-not-run gap fails the very first request rather than silently corrupting data. Tests - ``test_message_body_round_trip_through_orm`` writes plaintext via the ORM, reads it back as plaintext, and confirms the underlying PostgreSQL column holds an envelope (the at-rest guarantee). This is the regression that catches "developer accidentally bypassed the type decorator." - ``test_message_body_empty_string_passes_through`` ensures empty bodies don't trigger wrap calls on the KEK provider — measurable on high-throughput deployments where half of outbound messages are tool-call-only and have empty bodies. - ``test_migration_020_rekey_helper_envelopes_plaintext`` covers the happy path, idempotent re-runs, and empty/None passthrough. - ``test_migration_020_processed_context_uses_distinct_column_context`` confirms each column gets its own ``EncryptionContext`` tag for future per-column key rotation. 🤖 Generated with [Claude Code](https://claude.com/claude-code) * fix(migration): point 020 at 018, not 019 The e2e-playwright CI job runs `alembic upgrade head` to bring up a real app instance for browser tests. With down_revision='019', that fails on this branch standalone because revision 019 (data_sharing _consent in clawbolt#1100) only exists on a separate in-flight branch. Migrations 019 and 020 touch different tables (users vs messages) so their order is functionally independent. Whichever PR merges second will see a two-heads conflict from alembic and needs a one-line down_revision bump. Pinning to 018 here lets each PR pass CI on its own without cross-branch dependencies, and the rebase cost falls on whichever merges second instead of blocking both. * fix: address review feedback on #1101 Four review changes: 1. Drop em dashes from comments and docstrings to match the OSS CLAUDE.md style rule. Ten instances across the migration, models.py message docstring, and test_encryption.py. 2. Hoist ``from sqlalchemy import text`` to the top of test_encryption (CLAUDE.md: "All imports at the top of the file. No inline or deferred imports inside functions."). Imported as ``_sa_text`` to avoid colliding with anything that wants ``text`` as a local name in a test fixture later. Added a top-level ``_alembic_op`` import alongside it for the new e2e migration test. 3. Operator preflight in migration 020: refuse to run on a non-empty messages table when ``ENCRYPTION_KEY`` is unset. ``LocalKEKProvider()`` with no configured key falls back to an ephemeral process-local KEK, which would leave message bodies unrecoverable on the next process restart. The 018 (oauth_tokens) precedent didn't bother with this check because OAuth tokens are rare and re-issuable; message bodies are not. Empty messages tables pass through (nothing to lose). Migration docstring spells out the rationale. 4. End-to-end migration test against the real test database (``test_migration_020_full_upgrade_loop_against_real_db``). Inserts plaintext rows via raw SQL (bypassing the type decorator to simulate the pre-migration state), runs ``upgrade()`` against the test connection via monkeypatched ``op.get_bind``, then asserts envelopes on disk for non-empty rows + empty-string passthrough on the all-empty row. A second ``upgrade()`` invocation confirms the idempotent fast-path terminates instead of looping forever (regression test for the cursor reach-around at the end of the batch). Also sharpened the migration's cursor comment to spell out exactly when the in-loop ``last_id = row_id`` fires and why the end-of-batch reach-around is load-bearing for already-migrated databases. Added a one-line note in the docstring that alembic wraps ``upgrade()`` in a single transaction so operators on million-row deployments should plan a maintenance window. * test: seed ENCRYPTION_KEY in the migration 020 e2e test The preflight check added in the previous commit fired during the e2e test because the test database has no ENCRYPTION_KEY configured by default. Monkeypatch ``settings.encryption_key`` to a synthetic 32-byte value before running ``upgrade()``, which is the operator workflow the preflight is documenting (set the key, then migrate). * test: capture chat_session_id as plain int to avoid DetachedInstanceError The e2e migration test creates ``cs`` (ChatSession) inside one session context, closes the session, then references ``cs.id`` later in verification raw-SQL params. SQLAlchemy 2.0 raises ``DetachedInstanceError`` when an attribute access on a detached instance triggers a refresh; the trip wire here was the ``cs.id`` read after ``db.close()``. Capture the id into a plain ``int`` while the session is still open so the verification queries don't re-touch the ORM object. * fix(migration): chain 020 onto 019 now that #1100 has merged
…flagging (#1102) * feat: add /report inline command for user-initiated conversation flagging Item 5 of clawbolt-premium issue #325. Users on iMessage / Telegram / SMS-only deployments need a way to flag a conversation for admin review without ever opening the web app. Solution: intercept the literal text ``/report [reason]`` in the inbound pipeline, persist a ``ReportedConversation`` row, and send the user a fixed acknowledgement. The premium ``/admin/reported-conversations`` router (separate PR) will consume the rows. What ships in this PR - New ``ReportedConversation`` model with ``user_id`` / ``session_id`` FKs (CASCADE delete on user/session removal), ``anchor_seq`` (the inbound message's seq at report time so the admin UI can highlight the surrounding window), ``reason`` (the optional free-text after ``/report``), ``created_at``, ``dismissed_at``, and ``reviewed_admin_user_id`` (FK with SET NULL on admin deletion). Indices on user_id, session_id, created_at. - Migration 021 creates the table with ``server_default=func.now()`` on ``created_at`` and ``server_default=''`` on ``reason``. - ``_parse_report_command(text)`` recognizes ``/report``, ``/report reason``, ``/Report`` (autocap), ``/REPORT``, with leading/trailing whitespace stripped. Rejects ``/reportbot`` (different word) and ``please /report this`` (mid-sentence) so users typing /report in normal conversation aren't surprised by a flag. - ``_handle_report_command`` resolves (or creates) the user's session, captures the latest message seq as ``anchor_seq``, persists the row, and publishes the canonical ack to the message bus. Failures inside the persistence path log + roll back but STILL send the ack so the user never sees a mysterious silence for ``/report``. - ``process_inbound_from_bus`` calls the report intercept BEFORE the approval-response intercept on purpose: a user mid-approval who decides to file a report should still be able to. The /report ack short-circuits the agent pipeline; the approval gate stays pending and resolves on the next message. Tests - Eight ``_parse_report_command`` parser tests: bare /report, with reason, case-insensitive, whitespace stripping, ``/reportbot`` rejection, mid-sentence rejection, empty input, plain prose. - Three end-to-end intercept tests: persists row + sends ack + doesn't call agent pipeline; ``anchor_seq`` points at the latest message in the session; non-/report messages reach the pipeline normally and don't write rows. Why an inline command rather than a web-app form The clawbolt premium issue #325 left this open. Most clawbolt users never open the web app (they live on iMessage / Telegram), so a web-only report flow misses the population that most needs it. An inline command is verifiable (the text comes through the user's own authenticated channel route) and zero-friction (no auth surface to build, no email integration to maintain). 🤖 Generated with [Claude Code](https://claude.com/claude-code) * fix: address review feedback on #1102 Four review changes: 1. Hoist ``ChatSession`` and ``Message`` to the module-level import line (CLAUDE.md: "All imports at the top of the file"). The bus import inside ``_handle_report_command`` is left lazy on purpose and now carries a comment explaining why: ``backend.app.bus`` transitively imports ingestion via the channel manager, so a top-level bus import would form a cycle. Other functions in this file already do the same. 2. Add ``User.reported_conversations`` ORM relationship + the matching ``ReportedConversation.user`` reverse side. Disambiguated with ``foreign_keys=`` because the table has two FKs to ``users.id`` (``user_id`` for the reporter, ``reviewed_admin_user_id`` for the admin who closed the report). We don't expose the admin-side relationship; the audit log already answers "what did this admin do" without an ORM round-trip. 3. Restructure ``_handle_report_command`` so the ack-send is in a single ``finally`` block. Exactly one publish_outbound call regardless of which failure path the persistence takes (session resolution failure, missing ChatSession row, persistence exception, or success). The previous shape was correct but structurally fragile. 4. Cap the persisted ``reason`` text at 4096 chars. Defends against accidentally storing a 100KB rant or pasted transcript when a user goes long after ``/report``. Truncation is silent so the report still files; the user just sees the same canned ack. Implemented inside ``_parse_report_command`` so the cap applies uniformly. New parser test confirms the slice happens.
Extends the at-rest encryption coverage from migration 020 (Message.body, processed_context) to five more user-content columns. Same pattern: per-row DEKs wrapped by ``LocalKEKProvider`` (or the premium KMS-backed provider). The ``EncryptedString`` type decorator on each ORM column transparently decrypts on read, so existing application code keeps working unchanged. What's now encrypted - ``HeartbeatLog.message_text``: the proactive message text the heartbeat scheduler sent (or would have sent on a skip). - ``HeartbeatLog.reasoning``: the LLM's free-text rationale, often paraphrasing user content back. - ``HeartbeatLog.tasks``: serialized task state with user-authored task descriptions. - ``MemoryDocument.memory_text``: the user's working memory file (notes, reminders, recent context). One of the most sensitive content surfaces in the database. - ``MemoryDocument.history_text``: compacted older sessions. What's still plaintext on these tables - ``HeartbeatLog.action_type`` and ``channel``: short enums needed for filtering / aggregation, no PII. - ``HeartbeatLog.created_at``: timestamp. Migration 022 - Operator preflight (mirrors 020): refuses to run on a non-empty target table when ``ENCRYPTION_KEY`` is unset, since ``LocalKEKProvider()`` would fall back to an ephemeral key and leave content unrecoverable on the next restart. - Streams in 1000-row batches with ``WHERE id > :last`` cursor advancement. ``heartbeat_logs`` can grow large on chatty deployments; ``memory_documents`` is at most one row per user but uses the same loop for code uniformity. - Idempotent: rows already in envelope format are returned by identity from ``_rekey`` and the loop skips the UPDATE. The end-of-batch cursor reach-around guarantees an already-migrated DB doesn't loop forever on the same SELECT. - No automatic downgrade. Restore from backup if needed. Tests - ``test_migration_022_rekey_helper_envelopes_plaintext_per_table_column``: helper threads the (table, column) tuple into the envelope context for all five (table, column) targets so the on-disk envelope matches what ``EncryptedString.process_result_value`` will pass on read. Idempotent re-run + empty/None passthrough also covered. - ``test_migration_022_full_upgrade_loop_against_real_db``: inserts plaintext rows directly via raw SQL (bypassing the type decorator) into both target tables, runs ``upgrade()``, asserts envelopes on disk for non-empty rows + empty-string passthrough on the all-empty rows. Re-runs ``upgrade()`` to confirm idempotent termination. - ``test_migration_022_refuses_when_encryption_key_unset_and_data_exists``: the operator preflight raises rather than silently encrypting under an ephemeral key. Deployment ordering: same as migration 020. Run ``alembic upgrade head`` BEFORE rolling out the new application code. ``EncryptedString`` raises on a non-envelope read, so a code-deployed / migration-not-run gap fails the very first request. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
…it (#1104) * fix(bus): keep response future after resolve so SSE handler can read it The webchat SSE flow has three actors that race: 1. The chat router calls register_response_future(request_id) and returns a request_id to the client. 2. The bus consumer + handler runs and produces an OutboundMessage, the dispatcher calls resolve_response(request_id, msg). 3. The client opens the SSE stream and the handler calls get_response_future(request_id) to read the result. Order (1) -> (2) -> (3) is normal for fast short-circuit replies like /report or approval acks. Pre-fix, resolve_response popped the future from the registry, so step (3) saw None, registered a fresh future, and the spinner hung until the SSE timeout. Fix: resolve_response now leaves the future in the registry. The TTL cleanup task scheduled in register_response_future evicts the entry 300s later regardless of whether it was resolved, so memory stays bounded. Verified end-to-end: POST /api/user/chat with /report now delivers the SSE reply within ~1s instead of hanging. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: ruff format Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds an end-user surface for ``data_sharing_consent``. Until now the column was settable only via PUT /api/user/data-sharing-consent, with no UI consumer outside the admin "Shared" tab; users had no way to opt in even if they wanted to. Two surfaces share one component (``DataSharingConsentSection``): - Onboarding (Get Started page): a non-numbered "Help improve Clawbolt" card after Step 4. Visible at first run; not blocking. - Settings (About You page): a "Privacy" section beneath the user bio so people can change their mind any time. Default OFF in both places: opt-in only. The setter endpoint already stamps ``data_sharing_consent_at`` on every flip, so opt-out moments are auditable. Admin reads continue to re-check consent at fetch time, so toggling off has immediate effect. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
#1106) * refactor(privacy): move data-sharing consent to a Privacy settings tab Users were looking on the Settings page for the privacy toggle, not on About You where it was first wired up. Move the section to a new "Privacy" tab in Settings so it sits next to Model / Storage / Heartbeat / Telegram, where people expect config knobs to live. The same ``DataSharingConsentSection`` component is reused; the onboarding card on Get Started is unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(settings): add 'privacy' to showOssSettingsTabs default list Without this, the new Privacy tab is filtered out by .filter((t) => visibleOssKeys.includes(t.key)) and never renders. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ttings tab (#1107) * refactor(nav): rename Advanced -> Personalize, fold Approvals into Settings tab Cleans up two adjacencies that drifted: 1. The "Advanced" sidebar group mixed content surfaces people actively edit (Knowledge / Personality / About You / Priorities) with a config knob (Approvals). Folks were looking for Approvals on the Settings page, not under a chevron labeled "Advanced". 2. PR #1106 added a Privacy tab to Settings but forgot to add the key to ``showOssSettingsTabs``, so the tab silently filtered out for every visibility tier. Same fix lands here for both ``approvals`` and ``privacy``. Changes: - Sidebar group renamed to "Personalize"; ``NAV_PERSONALIZE`` now holds Knowledge, Priorities, Personality, About You. Approvals removed. - ``SettingsPage`` gains an "Approvals" tab that renders the existing ``PermissionsPage`` component (its h2 dropped so the Settings page heading dominates). Privacy + Approvals added to ``showOssSettingsTabs``. - ``/app/permissions`` now redirects to ``/app/settings/approvals`` so any existing bookmarks still land in the right place. - ``AppShell.test.tsx`` updated to assert "Personalize" and that Approvals is no longer a sidebar item. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(e2e): rename Advanced -> Personalize in OSS smoke nav assertion --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* build: add html-to-image dependency for snippet image export Premium's admin activity feed gains a Share snippet dialog that can export a PNG of the selected conversation (mozilla-ai/clawbolt-premium#607). The DOM-to-image rasterization uses html-to-image; premium builds from this frontend's node_modules, so the dependency lives here and is imported from the premium overlay. * build: tell knip html-to-image is used by the premium overlay knip runs against OSS source only, where nothing imports html-to-image (the premium share-snippet overlay does), so it flags the dependency as unused. Add it to ignoreDependencies alongside the other premium-overlay deps.
* feat: connect ServiceTitan and AppFolio in the web app, not chat
Connecting ServiceTitan (Tenant ID, Client ID, Client Secret) and AppFolio
Vendor Portal (single-use magic link) used to happen through chat tools
(connect_servicetitan, appfolio_connect). That left those secrets in the
user's message thread, where they cannot be cleared.
Move secret entry to the authenticated web app:
- Remove the connect_servicetitan and appfolio_connect chat tools, their
auth factories, names, and the now-unused hidden-core-factory pairings.
- Extract the validate-and-persist orchestration into reusable
servicetitan.auth.connect_credentials and
appfolio_vendor.service.connect_via_magic_link.
- Add POST/DELETE /api/integrations/{servicetitan,appfolio_vendor} endpoints
that collect the secrets over an authenticated session.
- Surface a connect_form key on the tool config so the Settings page renders
a credential form (Connect/Disconnect) instead of an OAuth button.
- Route the agent to the web app: data-factory auth_check reasons,
manage_integration connect, SKILL.md Connecting sections, and the
not-connected/expired tool hints no longer accept secrets over chat.
Fixes #1337
* fix: address review feedback on web-app integration connect
- Route users to the Integrations page (not "Settings"): the connect form's
sidebar label is "Integrations". Updated auth_check reasons, SKILL.md,
manage_integration, and the not-connected/expired tool hints.
- Map upstream outages to HTTP 502, not 400: add ServiceTitanUnavailableError
and AppFolioUnavailableError for network/5xx failures so a vendor outage no
longer reads as bad user input. Credential rejection stays 400.
- Mask secrets in schemas: client_secret and magic_link are SecretStr now
(write-only in OpenAPI, masked in logs/reprs); endpoints read
get_secret_value().
- Gate ServiceTitan on SERVICETITAN_APP_KEY: status, the live status feed,
the chat connect action, and the web connect_form all report/treat it as
"not configured by admin" when the operator has not set the App Key, instead
of advertising a connect flow that would hard-fail.
- Frontend: trim modal inputs before submit; invalidate the tools query on
disconnect onSettled (so a 404 still refreshes the badge); hoist mid-file
imports in ToolsPage to the module header.
Added tests for the 502 upstream paths and the ServiceTitan not-configured
states. Regenerated OpenAPI types.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…story cache (#1424) * fix: emit dynamic system content after history to keep the message-history cache The agent placed dynamic system content (memory, tool guidelines, integration status, cross-session context) in the `system` param, ahead of the conversation history in the prompt-cache prefix. Anthropic builds its cache prefix in tools -> system -> messages order, so any change to that dynamic block invalidated the cache for everything after it, including the entire message history. Every memory write (a routine event for an active user) re-billed the full history at uncached input price on the next turn. Move the dynamic content out of `system` and inject it into the current user turn (after time context, before the user's message), and add an explicit cache_control breakpoint on the prior-history message so the history is independently cacheable instead of relying on automatic prefix caching the dynamic suffix used to break. - SystemPromptBuilder.build_parts() splits sections into stable/dynamic halves via the existing dynamic flag; CACHE_BOUNDARY is removed. - apply_history_cache_breakpoint() stamps the message before the current inbound turn (the last user-role string-content message); the marker advances forward as the conversation grows. - The dynamic half ships on the user turn; the stable half is the only thing in the cacheable system param. Regression tests assert the dynamic block lands on the user turn (not system) and the breakpoint sits on the prior-history message, not the volatile current turn. Fixes #1420 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: hoist inline imports to module scope in cache tests Move the function-local imports added in the prior commit (apply_history_cache_breakpoint, _cache_control, build_agent_system_prompt_parts, prepare_system_with_caching) to the top of the test modules, per the repo rule that all imports live at module scope. Flagged by CodeRabbit on #1424. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…oken hysteresis (#1425) Each user has one ever-growing conversation, so the trim window is the permanent definition of how much raw history the assistant sees. The window was governed by a turn cap (target 80, trigger ~96) that bound far below the 400K token budget, so an active user permanently worked from only the last few active days of raw history. Make the token budget the primary governor and add the token-side hysteresis the path never had: - context_trim_target_tokens 400K -> 120K (drop-to budget). - New context_trim_trigger_tokens (150K): trim fires when the prompt exceeds the trigger and drops to the target, mirroring the existing turn hysteresis. Without it, the resting state would sit at the cap and re-fire compaction every message once tokens are the binding constraint (the per-message-compaction failure #1193 fixed for the turn path). - context_trim_target_turns 80 -> 600: demote the turn cap to a backstop. Tokens bind first in normal use (~330 turns for a token-light user); the turn cap only catches pathological message-count bloat. - Startup validation extended to the invariant target_tokens < trigger_tokens <= max_input_tokens. This widens raw recall from ~3 active days to ~2 weeks for a typical messaging user and, as a side effect, fires compaction less often. It depends on the cache-ordering fix (#1420, landed in #1424): without it a larger window would re-send up to 150K tokens at full input price after every memory write. This is a recall-for-cost trade, not a cost reduction: the full window is sent every turn, so per-turn history cost scales with window size (~3-4x, order +$15/month for the measured user). The target is the dial. Fixes #1421 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bumps [vitest](https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest) from 4.0.18 to 4.1.0. - [Release notes](https://github.com/vitest-dev/vitest/releases) - [Changelog](https://github.com/vitest-dev/vitest/blob/main/docs/releases.md) - [Commits](https://github.com/vitest-dev/vitest/commits/v4.1.0/packages/vitest) --- updated-dependencies: - dependency-name: vitest dependency-version: 4.1.0 dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [react-router](https://github.com/remix-run/react-router/tree/HEAD/packages/react-router) to 7.17.0 and updates ancestor dependency [react-router-dom](https://github.com/remix-run/react-router/tree/HEAD/packages/react-router-dom). These dependencies need to be updated together. Updates `react-router` from 7.13.1 to 7.17.0 - [Release notes](https://github.com/remix-run/react-router/releases) - [Changelog](https://github.com/remix-run/react-router/blob/main/packages/react-router/CHANGELOG.md) - [Commits](https://github.com/remix-run/react-router/commits/react-router@7.17.0/packages/react-router) Updates `react-router-dom` from 7.13.1 to 7.17.0 - [Release notes](https://github.com/remix-run/react-router/releases) - [Changelog](https://github.com/remix-run/react-router/blob/main/packages/react-router-dom/CHANGELOG.md) - [Commits](https://github.com/remix-run/react-router/commits/react-router-dom@7.17.0/packages/react-router-dom) --- updated-dependencies: - dependency-name: react-router dependency-version: 7.16.0 dependency-type: indirect - dependency-name: react-router-dom dependency-version: 7.16.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [starlette](https://github.com/Kludex/starlette) from 0.52.1 to 1.0.1. - [Release notes](https://github.com/Kludex/starlette/releases) - [Changelog](https://github.com/Kludex/starlette/blob/main/docs/release-notes.md) - [Commits](Kludex/starlette@0.52.1...1.0.1) --- updated-dependencies: - dependency-name: starlette dependency-version: 1.0.1 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ion time (#1426) * fix: anchor compaction history breadcrumbs to event time, not compaction time Compaction replaced the [TIMESTAMP] placeholder in every HISTORY.md breadcrumb with the moment the compaction ran, so once older messages were trimmed the agent lost when events actually happened. It would then read a timeless or wrongly-stamped breadcrumb and place a past action "today", e.g. claiming a calendar event added the previous evening was scheduled today. The conversation handed to the compactor already carries the per-message time markers added for history (the [Weekday, YYYY-MM-DD HH:MM] markers), so the model can anchor each breadcrumb to the event's own time. This: - Tells the compaction prompt the markers exist and to use them. - Mandates one timestamp format ([YYYY-MM-DD HH:MM], or date-only when the time is unknown) instead of the mix of weekday/AM-PM/range shapes seen in production history. - Forbids propagating the batch's opening marker time onto every event, and forbids relative-time words ("today", "tomorrow") that drift once the breadcrumb is read days later. - Keeps the now-time substitution as a fallback only for events with no visible marker. The code change is just a rename plus a comment: .replace("[TIMESTAMP]", now) already leaves real timestamps untouched, so it degrades cleanly into the no-marker fallback. Regression test covers a multi-line summary mixing a concrete event time (preserved verbatim) and a placeholder (filled with the fallback). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: tighten compaction Step 4 prompt, cut duplicated marker-cadence fact Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: route history-window overflow through compaction Token-light conversations could grow past the conversation_history_limit row cap (500) while staying under the 150k token trim trigger forever. Rows beyond the window were silently excluded from LLM context without ever passing through trim_messages, so the trim-driven compaction path never saw them: no MEMORY.md update, no HISTORY.md breadcrumb, no watermark advance. The turn backstop could not catch this either, since its default (600 turns, trigger 616) was unreachable behind a window that holds at most ~250 user turns. Three changes: 1. load_conversation_history now routes rows that fall outside the window through trigger_compaction_for_dropped, plus a 32-row headroom batch so the path does not re-fire on every subsequent message. The watermark advance plus batching is what makes this safe; PR #843 removed the old loader-driven compaction because it re-fired per message with no watermark. When every overflowed row is filtered (approval prompts, blank placeholders) the watermark still advances so the branch cannot re-detect the same rows forever. 2. context_trim_target_turns default drops from 600 to 200 so the turn backstop is reachable inside the loader window (2 * 217 = 434 <= 500 rows). The trigger buffer constant moves to config.py so the config validator can compute the effective trigger without importing agent code. 3. log_config_warnings warns when 2 * (effective trigger + 1) exceeds conversation_history_limit, the configuration that reproduces the original bug. Fixes #1427 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: re-check live watermark before compacting dropped rows The window-overflow path in load_conversation_history runs earlier in the same turn than the trim path in process_message, and both can fire when a conversation crosses the row cap and the token budget together. The trim path computes its dropped list from a session snapshot taken before the overflow path advanced the watermark, so the overlap would be compacted twice: double LLM cost and a duplicate audit row for the same seq range. trigger_compaction_for_dropped now reads the live watermark inside the phase-1 transaction and filters out rows already covered, skipping entirely when nothing remains. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: cap window-overflow compaction batch at 200 rows per turn A legacy session can arrive with a multi-thousand-row backlog above the watermark. Compacting it all at once would feed the whole backlog into a single compaction LLM call and blow its context. The batch is always a contiguous prefix starting at the oldest row above the watermark, so capping it is safe: the watermark advances to the end of the capped batch and the next message picks up the next chunk. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
get_or_create_session_async loaded every message the user had ever produced, including rows at or below the trim watermark that load_conversation_history filters out in Python a moment later. Messages are never deleted, so the per-message DB load grew with the user's lifetime message count: O(lifetime) rows materialized into ORM objects and DTOs (including multi-KB tool_interactions_json blobs) per inbound message, forever. The watermark filter now runs SQL-side via a dedicated builder (seq > last_trim_seq, indexed by the uq_message_seq constraint), so the hot path loads only the visible window. Every consumer of this method either applies the same filter anyway (load_conversation_history, admin_compact_visible_messages) or never reads messages at all (heartbeat persistence, inbound recovery refresh). Full-transcript surfaces (webchat history endpoint, admin views) go through load_session_async / list_sessions_async, which are unchanged. The issue also proposed a tail LIMIT. Deliberately not included: a naive LIMIT would hide the oldest visible rows from the window-overflow compaction path (issue #1427), which must see a contiguous prefix starting at the oldest row above the watermark. The row count above the watermark is bounded by that path instead. Fixes #1428 Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
) compact_session reads MEMORY.md / USER.md / SOUL.md, runs an LLM call that takes tens of seconds, then writes full rewrites. Compaction runs as a background task, so the conversation keeps going during the call: the agent's workspace tools can write a new fact in that window, and a second compaction for the same user can land first (the code already acknowledges concurrent compact_session tasks). The blind overwrite then clobbers the newer value, silently, including explicit user saves. HISTORY.md was already protected (advisory lock + FOR UPDATE in append_history); the other three files were last-writer-wins. write_memory_async / write_user_async / write_soul_async now accept an optional expected_current and perform the compare inside the write transaction under FOR UPDATE (plus the per-user advisory lock for the no-row-yet MEMORY.md branch, mirroring append_history). On drift the write is skipped and False is returned; compaction logs the skip and records memory_updated=False on the audit row. Losing one batch's extraction is recoverable (the conversation sent to the LLM is kept in the event row's prompt_text audit column); clobbering a durable file is not. Fixes #1429 Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The only message-side cache breakpoint sat before the current inbound user turn and never moved during the tool loop. Every round N > 0 re-sent the current turn (carrying the full dynamic context: memory up to 25KB, integrations, cross-session) plus all prior rounds' tool calls and results as uncached input. With max_tool_rounds=10 and large tool results the per-turn cost grows quadratically with round count. apply_in_turn_cache_breakpoint stamps a cache_control marker on the trailing tool_result block when the request ends in tool results, so round N reads the current turn plus rounds 0..N-1 from cache and pays cache-write only on the newest round. Message dicts are re-serialized from typed messages every round, so the marker advances with the loop instead of accumulating: at most four breakpoints per request (system, tools, prior-history tail, in-turn), which is Anthropic's limit. Round 0 ends in the current user turn (string content) and is a no-op, same as today. Fixes #1430 Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix: retry compaction events stuck in pending on startup trigger_compaction_for_dropped advances the trim watermark synchronously, then runs the compaction LLM call as a fire-and-forget background task. When the process dies mid-call (deploy restart, OOM) or the call fails (provider outage), the CompactionEvent row stays 'pending' forever: the watermark is already advanced, so the seq range never reaches the LLM again and its facts are never extracted into MEMORY.md. The row records everything needed to recover (the seq range) and messages are never deleted, but nothing acted on it. recover_pending_compactions sweeps stale pending rows on app startup, mirroring the inbound_recovery pattern: per-process pg_try_advisory_lock on a dedicated AsyncConnection, lookback window (new compaction_retry_lookback_minutes setting, default 7 days, 0 disables), best-effort semantics. Each row is claimed by incrementing the new retry_count column (migration 039) before the LLM call so a crash mid-retry still consumes the attempt; rows at the 3-attempt cap stay 'pending' for admin visibility but stop being selected, so a poisoned range cannot retry forever. A 10-minute grace floor keeps the sweep from racing a compaction call that is legitimately still in flight. Ranges whose messages were deleted are exhausted immediately. Fixes #1431 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: document COMPACTION_RETRY_LOOKBACK_MINUTES Covers the .env.example and configuration.md completeness checks in test_env_example.py for the setting added by the compaction retry sweep. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…te (#1439) trigger_compaction_for_dropped advances the trim watermark synchronously, so dropped rows vanish from LLM context on the very next message, but the compaction LLM call that extracts their facts into MEMORY.md is async and may still be in flight (or may have failed). In that window the agent has amnesia for the dropped range, immediately after the trim, when the dropped content is most likely still topical. The deterministic trim summary the trim turn saw was never persisted, so it existed for one turn only. While the user has a recent (under 60 minutes) 'pending' compaction event, a terse deterministic summary of the covered rows is rebuilt from the durable message rows (same summarize_dropped_messages shape) and injected as a dynamic system-prompt section. Once the event flips to 'completed', the note disappears and MEMORY.md carries the facts: the context now contains either the compacted facts or the note, never neither. Design choices: the summary is recomputed per turn instead of persisted (inputs are durable, the summarizer is cheap and deterministic, no migration needed); the note rides the dynamic half of the prompt so it never busts the history cache; the 60-minute window keeps a permanently stuck event from pinning a stale note forever (that case belongs to the retry sweep, issue #1431). Watermark semantics are untouched. Fixes #1432 Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…ristic, erosion signal) (#1440) Four small fixes from the conversation-design review, bundled per issue #1433: 1. Same-row trim atomicity. History rebuild expands one outbound DB row into a tool-call AssistantMessage, its ToolResultMessages, and a final-reply AssistantMessage, all sharing the row's seq. trim_messages treated the reply as a separate block, so it could drop the tool-call half while keeping the reply; the watermark then advanced over the shared seq and silently filtered the kept reply from the next turn. The reply now trims atomically with its block (live-loop messages carry seq=None and are unaffected). 2. Remove the dead cross-session context section. Migration 026 collapsed sessions to one per user, so the query excluding the current session always returned empty; every message paid a wasted DB query. Removed build_cross_session_context, the get_other_session_messages_async store method, and the current_session_id plumbing through the prompt assemblers. 3. Accurate proactive trim. A fresh ClawboltAgent is built per message, so the trim decision always used the chars/4 + flat-overhead heuristic, which ignores tool schemas and real system prompt size. A bounded process-local LRU now carries the last API-reported input_tokens per user across agent instances; the heuristic remains the cold-start fallback. 4. Memory erosion signal. Compaction full-rewrites MEMORY.md and the compliance audit encourages deletion, so a valid line can vanish on any cycle with no signal. The compaction.summary log line now carries memory_lines_added / memory_lines_removed (multiset diff, reorder-insensitive) so an aggregator can alert on large unexplained removals. Persisting the counts on compaction_events can follow once migration 039 (PR #1438) lands, avoiding an Alembic head collision. Fixes #1433 Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…es (#1441) Four fixes for failure modes observed in production sessions: 1. Numeric tool args for string fields: LLMs emit numeric-looking values (work order numbers, street numbers, numeric event titles) as JSON numbers; Pydantic v2 rejects int/float for str fields, failing the whole tool call. The agent now stringifies the offending values and revalidates once before surfacing the error. 2. Text-only inbound messages no longer wrapped in "[Text message]:": the wrapper only earns its keep next to media parts; alone it is formatting noise the model can parrot back as its own reply. 3. AppFolio: appfolio_get_work_order retries with the canonical customer id on a scope 401 (mirrors the invoice tool), and a no-match search now explains why a number may be missing (closed, archived, or a different property manager) instead of dead-ending. 4. QuickBooks SKILL.md: amounts, balances, and statuses must be quoted from rows a query returned this turn, not from conversation memory of an earlier pull. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
… reports (#1443) AppFolio successes were only logged as a byte count, so a 200 OK with a surprising shape left nothing to debug from after the fact. The concrete gap: a work-order number search whose raw response carries only an internal id and no user-facing number. The formatter falls back to the id, the agent reads a false mismatch and can refuse to act, and we keep no record of what AppFolio actually returned. We can't reach AppFolio ourselves, so the only way to debug a user's report later is to have captured the raw response at the time. Add an INFO-level raw-response log in the one request funnel (covers list, search, get, and writes): - short scalar fields (id, numberForDisplay, customer_id, status) pass through intact, which is what's needed to reconstruct the display-number-to-internal-id mapping; - base64/long strings collapsed (reuses the request summarizer) so photo blobs don't bury the structure; - credential keys (jwt, access_token, refresh_token, ...) and PII keys (names, addresses, email, phone) redacted, so neither secrets nor tenant data reach the logs; - a one-line shape descriptor matching log_unexpected_response_shape; - truncated at 8000 chars; non-JSON bodies log size + content-type only. Adds unit and wire tests for the summarizer, secret and PII redaction, and the success-path log line. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
dependabot
Bot
added
dependencies
…arden log PII (#1445) The raw-response logging (shipped previously) revealed the actual work_order_search response: it is a universal-search hit schema, not the maintenance work-order schema. The user-facing number lives in ``result_text`` ("<number> - <unit>") and the customer id is a list under ``customer_ids``. The formatter reads ``numberForDisplay`` and ``customer_id``, so it matched neither, fell back to the bare internal ``id``, and the agent reported a false "that number doesn't match" and had no customer_id (invoices then 404'd until a 105-row list call backfilled it). - Add ``_normalize_search_hit`` to remap a search hit into the work-order shape (result_text -> numberForDisplay, customer_ids[0] -> customer_id) before formatting. No extra API call: the data was already in the search response. Kills the false mismatch and the invoice 404. PII hardening for the raw-response log (separate, prompted by the same captures): the exact-key redaction missed several PII-bearing keys (company_email, business names like portfolio_name, *_last_four, company_primary_address_1, and free-text description/instructions that carry tenant name + phone). Add those to the set plus a narrow PII substring check (email/phone/address/name/last_four/ssn/tax_id) for schema variants, while keeping the identifier fields the debugging needs (id, numberForDisplay, result_text, customer_id, customer_ids, status). Adds tests for the search remap, the search-tool output, and the expanded redaction. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nd not partially apply (#1450) * fix(approval): re-prompt on ambiguous reply instead of aborting the batch When the agent batches several tool calls behind sequential approval prompts (for example rescheduling a multi-day job stored as one calendar event per day) and the user replies with short filler ("lol", "haha", "ok wait"), the classifier returned no decision and ingestion resolved the gate as INTERRUPTED. That aborted the remaining items, so a multi-event operation was left partially applied and the agent reported that it "skipped" the rest because they "weren't approved" -- confusing for a casual reply, and requiring a second confirmation to finish. Filler is neither a yes/no nor a genuine change of subject, so treat it as such: classify_approval_response now returns a dedicated "ambiguous" result, and ingestion re-sends the approval prompt ("Sorry, I didn't catch that as a yes or no.") while leaving the gate pending, so the blocked agent loop keeps waiting for a real decision. A clear yes/no then completes the whole batch in one step. Re-prompts are capped (the approval timeout remains the outer bound) so persistent filler still falls back to INTERRUPTED. Genuine new requests still classify as unrelated and interrupt as before. Fixes #1449 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(calendar): scope update/delete approvals per action, not per event calendar_update_event and calendar_delete_event extracted an approval resource of f"update {event_id}" / f"delete {event_id}", scoping the stored permission to a single event. Rescheduling a multi-day job is one update per day, each with a distinct event_id, so "always allow" on the first day never matched the others: the within-turn approval cache (keyed on (tool_name, resource)) missed, the persisted ALWAYS override missed, and the user was re-prompted on every day even after choosing "always allow" (observed in production: always_allow, always_allow, approved, then interrupted across four prompts for one reschedule). Use a coarse constant resource ("update event" / "delete event"), matching calendar_create_event's existing "create event". Now one approval (yes or always) covers every same-type mutation in the batch, and "always allow" persists tool-wide as the user expects. Fixes #1449 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ls (#1453) * feat(approval): add blanket "always all" approval for resource-scoped tools Resource-scoped tools (like qb_send) remember an "always allow" decision per resource: the user had to approve every new invoice recipient individually. Add a distinct ALWAYS_ALLOW_ALL decision that persists a tool-level permission (resource=None), so one "always all" reply covers every recipient. The store and resolution layers already supported tool-level entries and the resolution order (resource match, then tool match, then default), so this only adds a way to record the blanket decision: - New ApprovalDecision.ALWAYS_ALLOW_ALL, fast-path keywords ("always all", "allow all", "always everyone", etc.), and an LLM-classifier label. - format_approval_message gains an opt-in "always all" line, inserted before the "never" trailer so context.py can still identify stored approval prompts in history. - ApprovalPolicy.resource_noun drives the wording; only tools that set it (qb_send: "recipients") offer the blanket option. The per-resource "always" still works, and a resource-level NEVER still overrides the blanket ALWAYS. Fixes #1451 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(approval): scope ALWAYS_ALLOW_ALL to opt-in tools only The "always all" / "allow all" keywords resolve to ALWAYS_ALLOW_ALL globally, but only tools that declare a resource_noun show the option. As written, the blanket decision persisted a tool-level grant for any tool, so a typed "allow all" on a resource-scoped tool that never offered it (e.g. web_fetch, scoped by domain) would grant a broader permission than was advertised. Gate the tool-level write on the tool having opted in (resource_noun set); otherwise scope the grant to the resource actually shown. Add a regression test covering the opt-out case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style(approval): address review nitpicks - Replace "--" separator with a period in an approval.py comment, per the no-em-dash/no-double-hyphen copy rule. - Move the _is_approval_prompt import in test_approval.py to the top of the file, per the all-imports-at-top rule. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…1456) With prompt caching, usage.input_tokens counts only the uncached slice of the prompt; the cached slice is reported in cache_read_input_tokens and cache_creation_input_tokens. The agent loop recorded only usage.input_tokens as _last_input_tokens, so the token-side trim trigger read a mostly-cached 165k-token context as ~7k and never fired. Every trim-driven compaction in production has come from the 216-user-turn backstop instead (visible as uniform ~34-seq compaction_events ranges). Worse, the ContextLengthExceededError retry path trims with the same undercounted figure, so once a conversation reaches the model's hard context limit the retry trim is a no-op and the agent fails on every message until the turn backstop happens to fire. Record the full prompt size (input + cache creation + cache read) for the trim signal. Billing/usage accounting fields are unchanged. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…#1455) * fix: bump genai-prices to price claude-opus-4-8 (fixes $0 usage rows) genai-prices 0.0.57 predates claude-opus-4-8, so calc_price raised LookupError for the model deployed in prod and compute_cost fell through to Decimal('0.000000'). Every llm_usage_logs row for anthropic/claude-opus-4-8 was written with cost=0 while token counts logged correctly. Bump the lock to 0.0.69 (first version with opus-4-8 pricing is 0.0.63); this pulls in genai-prices' httpx2 stack (pydantic-org, same maintainer). The test suite covered claude-opus-4-7 and claude-sonnet-4-6 but never the actually-deployed claude-opus-4-8, so CI stayed green while prod zeroed out cost. Add claude-opus-4-8 to both the is_known_model parametrize list and the nonzero-cost smoke test so a future model launch that outruns the pricing data fails CI instead of silently zeroing revenue data. Note: this only fixes rows written after deploy. Existing cost=0 rows for claude-opus-4-8 need a separate backfill from stored token counts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: add backfill script for $0 llm_usage_logs rows Pairs with the genai-prices bump: that fixes rows written after deploy, this repairs the historical claude-opus-4-8 rows already persisted with cost=0. Recomputes cost from the token counts stored on each row (input, output, and both cache buckets) via services.llm_pricing.compute_cost and updates in place. Safety: dry-run by default (--apply to commit); only touches cost=0 rows with non-zero tokens so settled costs are never re-priced; only writes when the recomputed cost is > 0 so a still-unknown model is a no-op; idempotent. The --model guard refuses to run when genai-prices does not know the model, catching "I forgot to bump genai-prices first". Keyset paginated by id to bound memory on large tables. Tests cover reprice-on-apply, dry-run writes nothing, unpriceable rows stay zero, already-costed rows are untouched, and zero-token rows are skipped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bumps [pyjwt](https://github.com/jpadilla/pyjwt) from 2.12.1 to 2.13.0. - [Release notes](https://github.com/jpadilla/pyjwt/releases) - [Changelog](https://github.com/jpadilla/pyjwt/blob/master/CHANGELOG.rst) - [Commits](jpadilla/pyjwt@2.12.1...2.13.0) --- updated-dependencies: - dependency-name: pyjwt dependency-version: 2.13.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
dependabot
Bot
force-pushed
the
dependabot/uv/pyjwt-2.13.0
branch
from
July 14, 2026 17:05
e4c3956 to
850477a
Compare
njbrake
force-pushed
the
main
branch
2 times, most recently
from
August 14, 2026 18:01
82563d8 to
c3aa550
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bumps pyjwt from 2.12.1 to 2.13.0.
Release notes
Sourced from pyjwt's releases.
Changelog
Sourced from pyjwt's changelog.
Commits
7144e45Apply ruff formatd2f4becRestorecast()calls with cross-versiontype: ignoreforprepare_key22f478cRemove redundant casts inRSAAlgorithm.prepare_keyand `ECAlgorithm.prepare...95791b1Bundle security fixes and hardening into 2.13.0dcc27a9[pre-commit.ci] pre-commit autoupdate (#1155)9d08a9a[pre-commit.ci] pre-commit autoupdate (#1146)b87c100Bump codecov/codecov-action from 5 to 6 (#1154)40e3147Migrate development extras to dependency groups (#1152)