diff --git a/docs/context.md b/docs/context.md index aa22ef501..1d91aefcb 100644 --- a/docs/context.md +++ b/docs/context.md @@ -207,7 +207,8 @@ cancel it. A goal continues through scheduled turns until the agent emits `[BOITE_GOAL_COMPLETE]` on its own line. The prompt requests that marker only after verification. `[BOITE_GOAL_BLOCKED]`, an error or Escape pauses it. With the Reviewed goals experiment, `/goal2` turns that marker into a claim -that a fresh reviewer conversation must pass ([experiments](experiments.md#reviewed-goals)). +that a fresh reviewer conversation must pass, and keeps a checklist of the +user's requests that each turn hears ([experiments](experiments.md#the-checklist)). A blocked goal is marked `blocked`: the bar asks for an answer, and the user's next message, other than a native `/command`, resumes the goal once that reply's turn ends. Escape also pauses a loop between runs. The activity bar has pause, resume, diff --git a/docs/experiments.md b/docs/experiments.md index b21bff928..d24093afa 100644 --- a/docs/experiments.md +++ b/docs/experiments.md @@ -118,6 +118,14 @@ Reviewing during a review, and draws one dot per verdict plus a pulsing dot for the running review. A dot opens its reviewer conversation. The details list each verdict with its findings. +Before a reviewer or a tracker conversation starts on a model its agent has +not listed since the last account change, the core reads that agent's models +on the account the child will use, as `boite delegate spawn` does, once per +review; the start then goes through the scheduler, so a goal replaced or paused +meanwhile is not reviewed on the old read. An account update forgets Claude's list, and a +review that started without reading it again paused the goal with "the agent +has not listed this model". + Reviewers are not subagents. Like workflow steps, they need no enabled team, a paused team holds them, and their answer goes to the goal, never to the parent as mail. They are not listed in the Subagents tab. Pausing, removing or @@ -125,9 +133,69 @@ completing the goal by hand stops the running reviewer, and resuming a goal paused during its review starts a new review. A core restart keeps the verdicts and drops the interrupted review; the resume starts a fresh one. +### The checklist + +A reviewed goal also keeps the checklist of what the user asked for, so the +agent works on the user's requests rather than on its own plan +(`goal-tracker.ts`, `goal-tracker-prompt.ts`). A tracker, one more child +conversation of the thread with none of its history, runs short passes, one at +a time, in the order they were asked for: + +1. When the goal starts, it writes the list from the objective: one item per + distinct request, in the user's language, at most 12 words, a verb and a + visible result, with the user's own words it came from. Corrections ("or + rather", "but", `mot*`) keep their final meaning. +2. Each later message of the user, a new prompt or a steer, adds, drops or + changes items. The tracker answers with additions, cancellations and + changes; the core applies them, so ids never move and a dropped item stays, + struck through. +3. After each finished turn of the conversation, the agent's or the user's, it + reads a digest of that turn (one line per tool call, the failed ones marked, + and the agent's last answer) and marks the open items the turn did. An + item the turn never touched stays open, whatever the answer claims. The + goal's next turn or review waits for that check. + +The next goal turn hears what is still open, as JSON data, in place of the +usual line about the agent's own task list. A completion claim with an open +item goes back to the agent before any review runs, and says so; each item +holds a claim back twice at most, then the review decides alone with the whole +list, statuses included, in its prompt. + +The tracker runs on `Settings.goalTracker`, picked under the switch in +Settings, Experiments: a provider, a model and its reasoning level, stored by +the core of that machine. Automatic (`null`) takes the newest Opus at medium on +Claude or the newest Sol at high on Codex (`GOAL_TRACKER_DEFAULTS`), the +conversation's own agent first, else the conversation's own model. A small +model is never chosen by itself: on nine real prompts of 900 to 4000 +characters (2026-10-18), Haiku 5.5 at xhigh dropped or inverted about two +requests per long prompt, the same ones on every run, where Opus 5.5 at medium +dropped none. On Claude the tracker starts without the user's guides, servers +and hooks and keeps only `Read`; on every agent its rules tell it to answer +from the text alone. + +`ThreadActivity.goal.checklist` holds the tracker conversation, its model, the +phase (`writing`, `checking` or `idle`), the items with their status (`open`, +`done`, `cancelled`), whether a later message `added` or `changed` them, how +often each `bounced` a claim, and the last pass's `error`. The goal bar keeps +one line: the first open item and `done/total`, dropped items left out. On the +owner's app that line opens the side panel's Tasks tab. There the checklist +sits under the goal it breaks down, in the same section: a progress bar with +`done/total`, then the items in the order asked, done ones ticked and struck, +the next open one marked. Dropped items wait folded under a "1 dropped" line, +and an icon beside it opens the tracker conversation. The Reviews section lists +each review, newest first, with what it found. A paired device, which has no +Tasks tab, unfolds the same list in the bar instead. A press on an item shows +its source. An item that bounced twice says the final review decides it. The +screen leaves out the `added` and `changed` flags. A pass +that fails, answers without its JSON block or runs past five minutes leaves the +list as it was with the reason, and the goal goes on; a first list is tried +three times at most. A core restart keeps the list and the tracker; work no +pass had read yet is lost. + This version does not run checks itself before the review: the reviewer runs them and spends tokens doing so. It has no token budget and no detection of -the same finding repeated: the five-review stop bounds a goal instead. +the same finding repeated: the five-review stop bounds a goal instead. The +tracker adds one pass per user message and per finished turn. ## Edge scrollbars @@ -204,9 +272,16 @@ No live provider login is required. `goal-review.test.ts` drives reviewed goals through the real core with the echo driver: a failed review then a pass, the five-review stop and the reply -that allows one more, pause and resume during a review, and a restart. UI unit -tests cover the `/goal2` switch, the goal bar's verdicts and the fake core's -reviewer. +that allows one more, pause and resume during a review, and a restart. +`goal-tracker.test.ts` drives the checklist the same way: the first list, the +checks and what the next turn hears, a claim sent back twice then reviewed, +the user's additions and cancellations, the model Settings names, a tracker +answer without JSON, a tracker stuck on a pass, a goal finished by hand during +a pass, models read again after an account change, and a restart. The opt-in +`goal-tracker.live.test.ts` (`BOITE_E2E_CLAUDE=1`) runs one real goal on Haiku +with the checklist on Automatic. UI unit tests cover the `/goal2` switch, +the checklist model row, the goal bar's verdicts and checklist, and the fake +core's reviewer and tracker. `whip.test.ts` covers the toggle, persistence, desktop and phone layout, drop and rethrow, whole interface movement, rope drawing and release, cancellation and reduced motion. `shell.test.ts` verifies diff --git a/packages/contracts/src/goal-tracker.ts b/packages/contracts/src/goal-tracker.ts new file mode 100644 index 000000000..6e77895cd --- /dev/null +++ b/packages/contracts/src/goal-tracker.ts @@ -0,0 +1,45 @@ +import type { ModelInfo, Protocol } from './index.ts'; +import { newestModel } from './title-models.ts'; + +/** + * The models a `/goal2` checklist is written and checked with when Settings + * names none: the newest Opus at medium on Claude, else the newest Sol at high + * on Codex. The conversation's own agent goes first when it is one of them; + * with neither signed in, the conversation's own model keeps the list. A small + * model is never picked by itself: on long prompts it dropped requests and + * turned some into their opposite. + */ +export interface GoalTrackerRule { + protocol: Protocol; + /** Matched against model ids; the newest listed one wins. */ + family: RegExp; + effort: string; +} + +export const GOAL_TRACKER_DEFAULTS: readonly GoalTrackerRule[] = [ + { protocol: 'claude-sdk', family: /^claude-opus-\d/i, effort: 'medium' }, + { protocol: 'codex-appserver', family: /^gpt-[\d.]+-sol$/i, effort: 'high' }, +]; + +/** The default rule's model and effort among one agent's models, or null when it lists none of them. */ +export function defaultTrackerModel(protocol: Protocol, models: readonly Pick[]): { model: string; effort: string } | null { + for (const rule of GOAL_TRACKER_DEFAULTS) { + if (rule.protocol !== protocol) continue; + const model = newestModel(models, rule.family); + if (model !== null) return { model, effort: rule.effort }; + } + return null; +} + +/** The effort a picked model starts on: the default rule's for its family, else the model's own default. */ +export function trackerEffort(model: Pick): string | null { + const levels = model.effort?.levels.map((level) => level.id) ?? []; + const rule = GOAL_TRACKER_DEFAULTS.find((entry) => entry.family.test(model.id)); + if (rule && levels.includes(rule.effort)) return rule.effort; + return model.effort?.default ?? null; +} + +/** Items a checklist keeps; a longer list is cut, its last items dropped. */ +export const GOAL_CHECKLIST_MAX_ITEMS = 80; +export const GOAL_CHECKLIST_TEXT_MAX = 160; +export const GOAL_CHECKLIST_SOURCE_MAX = 300; diff --git a/packages/contracts/src/index.ts b/packages/contracts/src/index.ts index f3d62e26f..4f1a7e626 100644 --- a/packages/contracts/src/index.ts +++ b/packages/contracts/src/index.ts @@ -178,7 +178,8 @@ export type RpcEventName = keyof RpcEvents; export { attachmentError, answerAttachmentError } from './attachment-validation.ts'; export { AUTO_COMPACT_MOMENTS, AUTO_COMPACT_TOKENS, BROWSER_ORIGINS_MAX, CACHE_KEEPALIVE_HOURS, MODEL_FAVORITES_MAX, checkSettingsPatch, compareModelFavorites, keepLaterFavorites, type AutoCompact, type AutoCompactMoment, type CacheKeepalive, type SettingsPatchCheck } from './settings-validation.ts'; -export { TITLE_MODEL_DEFAULTS, defaultTitleModel } from './title-models.ts'; +export { TITLE_MODEL_DEFAULTS, defaultTitleModel, newestModel } from './title-models.ts'; +export { GOAL_TRACKER_DEFAULTS, GOAL_CHECKLIST_MAX_ITEMS, GOAL_CHECKLIST_SOURCE_MAX, GOAL_CHECKLIST_TEXT_MAX, defaultTrackerModel, trackerEffort, type GoalTrackerRule } from './goal-tracker.ts'; export { DEVICE_METHODS, DEVICE_EVENTS, AGENT_EVENTS } from './access.ts'; export { APP_LANGUAGES, CLIENT_APP_PATTERN, CLIENT_LABELS_MAX, CLIENT_LABEL_CHARS, EXPERIMENT_IDS, REQUESTABLE_SETTINGS, REQUESTABLE_SETTING_KEYS, SETTING_REASON_MAX, SETTINGS_PAGES, SETTINGS_SECTION_PATTERN, diff --git a/packages/contracts/src/settings-validation.ts b/packages/contracts/src/settings-validation.ts index 5e74babe5..245659b28 100644 --- a/packages/contracts/src/settings-validation.ts +++ b/packages/contracts/src/settings-validation.ts @@ -1,4 +1,4 @@ -import type { ModelFavorite, ModelFavorites, Settings, TitleModel } from './index.ts'; +import type { GoalTrackerModel, ModelFavorite, ModelFavorites, Settings, TitleModel } from './index.ts'; import { BROWSER_PROFILE_NAME_MAX, BROWSER_PROFILES_MAX, browserProfileIdError, DEFAULT_BROWSER_PROFILE, PRIVATE_BROWSER_PROFILE, type BrowserProfile } from './browser'; // Both supported hosts expose URL; contracts otherwise need no DOM or Node types. @@ -274,6 +274,15 @@ export function checkSettingsPatch(patch: Partial): SettingsPatchCheck } next.titleModel = { providerId: text(providerId), model: text(model) }; } + if (patch.goalTracker !== undefined && patch.goalTracker !== null) { + const { providerId, model, effort } = (typeof patch.goalTracker === 'object' ? patch.goalTracker : {}) as Partial; + const text = (value: unknown) => (typeof value === 'string' ? value.trim() : ''); + const level = effort === null || effort === undefined ? null : text(effort); + if (text(providerId).length === 0 || text(model).length === 0 || text(model).length > TITLE_MODEL_MAX || level === '' || (level?.length ?? 0) > 40) { + return { ok: false, field: 'goalTracker', message: `goalTracker must be null or { providerId, model, effort }: two non-empty strings, the model at most ${TITLE_MODEL_MAX} characters, and the effort a level name or null` }; + } + next.goalTracker = { providerId: text(providerId), model: text(model), effort: level }; + } if (patch.autoCompact !== undefined && patch.autoCompact !== null) { const { tokens, moments } = (typeof patch.autoCompact === 'object' ? patch.autoCompact : {}) as Partial; const TOKENS = AUTO_COMPACT_TOKENS, MOMENTS: readonly AutoCompactMoment[] = AUTO_COMPACT_MOMENTS; diff --git a/packages/contracts/src/settings.ts b/packages/contracts/src/settings.ts index 64fdb35ed..ced8db0b7 100644 --- a/packages/contracts/src/settings.ts +++ b/packages/contracts/src/settings.ts @@ -120,6 +120,11 @@ export interface Settings { * longer write one falls back to that too. */ titleModel?: TitleModel | null; + /** + * The model that writes and checks every `/goal2` checklist. Null or + * missing: `GOAL_TRACKER_DEFAULTS`, else the conversation's own model. + */ + goalTracker?: GoalTrackerModel | null; /** * When the core compacts a conversation by itself, between two turns. Null * or missing: never, each agent keeps its own behaviour. Missing on older cores. @@ -174,6 +179,14 @@ export interface TitleModel { model: string; } +/** One provider's model and reasoning level, picked in Settings, Experiments to keep `/goal2` checklists. */ +export interface GoalTrackerModel { + providerId: ProviderId; + model: string; + /** One of the model's levels; null for a model without them. */ + effort: string | null; +} + export interface SettingsRpcMethods { /** * A settings card on the agent's thread (`boite settings request|open`), diff --git a/packages/contracts/src/threads.ts b/packages/contracts/src/threads.ts index 82209403f..880f5f68a 100644 --- a/packages/contracts/src/threads.ts +++ b/packages/contracts/src/threads.ts @@ -378,12 +378,56 @@ export interface GoalReview { verdicts: GoalVerdict[]; } +/** One thing the user asked for in a `/goal2`, as its tracker wrote it. */ +export interface GoalChecklistItem { + /** From 1, in the order the items were written; never reused. */ + id: number; + /** A verb and a visible result in the user's language, at most `GOAL_CHECKLIST_TEXT_MAX` characters. */ + text: string; + /** The words of the user's message it comes from, at most `GOAL_CHECKLIST_SOURCE_MAX` characters. */ + source: string; + /** `done`: a check saw it done. `cancelled`: a later message of the user dropped it. */ + status: 'open' | 'done' | 'cancelled'; + /** Added by a message after the objective. */ + added?: boolean; + /** Changed by a message after the objective. */ + changed?: boolean; + /** Completion claims sent back to the agent because this item was still open; at 2 it no longer holds one. */ + bounced?: number; +} + +/** + * A `/goal2`'s checklist. A tracker conversation, a child of the goal's + * thread with none of its history, writes it from the objective, updates it + * on every later message of the user, and after each turn of the agent marks + * what the turn did. What stays open goes to the agent's next goal turn; a + * completion claim with an open item goes back to the agent before any review. + * The tracker reads only what Boite hands it: it runs nothing and edits nothing. + */ +export interface GoalChecklist { + /** The tracker conversation; null before it starts or when it could not. */ + threadId: ThreadId | null; + /** `writing`: reading the objective or a new message. `checking`: reading the agent's last turn. */ + phase: 'idle' | 'writing' | 'checking'; + items: GoalChecklistItem[]; + /** The tracker's provider, model and effort, as `claude/claude-opus-5-5`, for the panel. */ + model: string | null; + effort: string | null; + /** Why the last pass gave nothing; the goal goes on without it. */ + error: string | null; +} + /** What an automated goal or loop turn carries on its user message, for the driver prompt. */ export interface ActivityTurn { kind: 'goal' | 'loop'; iteration: number; /** A reviewed goal's first turn after a failed review: what the reviewer reported. */ review?: { round: number; feedback: string }; + /** + * A `/goal2` turn after its tracker's check: the checklist items still open. + * `bounced`: the agent claimed completion with them open, so no review ran. + */ + checklist?: { open: string[]; bounced: boolean }; } export interface ThreadActivity { @@ -391,7 +435,7 @@ export interface ThreadActivity { * `blocked`: the agent ended a goal turn with `[BOITE_GOAL_BLOCKED]` and * waits for the user. The goal is paused; the user's next message resumes it. */ - goal: { objective: string; status: 'active' | 'paused' | 'complete'; iterations: number; error: string | null; dismissed?: boolean; blocked?: boolean; review?: GoalReview } | null; + goal: { objective: string; status: 'active' | 'paused' | 'complete'; iterations: number; error: string | null; dismissed?: boolean; blocked?: boolean; review?: GoalReview; checklist?: GoalChecklist } | null; loop: { prompt: string; intervalMs: number; maxIterations?: number | null; status: 'active' | 'paused' | 'complete'; iterations: number; nextRunAt: number | null; error: string | null; history?: ActivityIteration[] } | null; tasks: AgentTask[]; tasksDismissed?: boolean; diff --git a/packages/contracts/src/title-models.ts b/packages/contracts/src/title-models.ts index 3a087836c..8cd5123dd 100644 --- a/packages/contracts/src/title-models.ts +++ b/packages/contracts/src/title-models.ts @@ -55,9 +55,15 @@ export function defaultTitleModel( const rule = TITLE_MODEL_DEFAULTS[provider.id] ?? TITLE_MODEL_DEFAULTS[BY_PROTOCOL[provider.protocol] ?? '']; if (rule === undefined) return null; for (const family of rule.families) { - const listed = models.filter((model) => family.test(model.id) && !model.id.includes('[')); - if (listed.length === 0) continue; - return listed.reduce((best, model) => (newer(version(model.id), version(best.id)) > 0 ? model : best)).id; + const newest = newestModel(models, family); + if (newest !== null) return newest; } return rule.fallback; } + +/** The newest listed model of a family, a variant such as `[1m]` left out; null when none is listed. */ +export function newestModel(models: readonly Pick[], family: RegExp): string | null { + const listed = models.filter((model) => family.test(model.id) && !model.id.includes('[')); + if (listed.length === 0) return null; + return listed.reduce((best, model) => (newer(version(model.id), version(best.id)) > 0 ? model : best)).id; +} diff --git a/packages/core/src/activity-prompt.ts b/packages/core/src/activity-prompt.ts index 9d3f9cc53..1878d6266 100644 --- a/packages/core/src/activity-prompt.ts +++ b/packages/core/src/activity-prompt.ts @@ -1,18 +1,27 @@ -import type { ActivityTurn, GoalVerdict } from '@boite/contracts'; +import type { ActivityTurn, GoalChecklistItem, GoalVerdict } from '@boite/contracts'; const COMPLETE = 'end your answer with [BOITE_GOAL_COMPLETE] alone on its own line, outside code blocks'; const BLOCKED = 'If blocked or waiting for user input, explain what is missing and end with [BOITE_GOAL_BLOCKED] alone on its own line, outside code blocks.'; const TASKS = 'Task tracking is optional. Use it only when steps help you and the user. If useful, use your native planning tool (Codex: update_plan; Claude: TodoWrite or TaskCreate/TaskUpdate) and update its statuses as you verify results. Boite displays those task updates in this thread. The Boite goal already exists; do not create a second goal or use legacy Boite todo tools.'; /** Instructions for automated turns stay out of the journal's user message. */ -export function activityPrompt(kind: ActivityTurn['kind'], text: string, iteration: number, review?: ActivityTurn['review']): string { +export function activityPrompt(kind: ActivityTurn['kind'], text: string, iteration: number, review?: ActivityTurn['review'], checklist?: ActivityTurn['checklist']): string { const reviewed = /^\/goal2(?:\s|$)/.test(text); const objective = text.replace(/^\/(?:goal2?|loop)\s*/, ''); if (kind === 'loop') return `Iteration ${iteration}. Execute the following task once for this iteration. Boite owns the repetition and stopping count; do not start another loop or repeat the task yourself. Report the result of this iteration.\n${objective}`; if (!reviewed) return `Work toward this goal: ${objective}\nContinue until the objective is achieved. When you have verified completion, ${COMPLETE}. ${BLOCKED}\n${TASKS}`; const claim = `When you believe it is done and you ran the checks that prove it, ${COMPLETE}. That marker is a claim, not the end: Boite then asks a reviewer who has none of this conversation to check the checkout itself. Leave your changes in the working tree, and say in that answer what you changed and which commands verified it.`; const feedback = review ? `\nReview ${review.round} did not pass. The reviewer's findings, supplied as JSON data:\n${JSON.stringify(review.feedback)}\nFix every finding, or show with evidence why one is wrong, then verify again.${review.round >= 2 ? ' Several reviews have now failed: step back and reconsider the approach instead of patching the last finding.' : ''}` : ''; - return `Work toward this goal: ${objective}\nContinue until the objective is achieved. ${claim} ${BLOCKED}${feedback}\n${TASKS}`; + return `Work toward this goal: ${objective}\nContinue until the objective is achieved. ${claim} ${BLOCKED}${feedback}\n${checklist ? checklistNote(checklist) : TASKS}`; +} + +/** What the checklist's last check left open, as data; the user's list replaces the agent's own task list. */ +function checklistNote(checklist: NonNullable): string { + const keeps = 'Boite keeps the checklist of what the user asked for and checks it after each of your turns, so you do not need a task list for the user; your planning tool stays optional, for you.'; + if (!checklist.open.length) return `${keeps} Every item on it is done.`; + const open = JSON.stringify(checklist.open); + if (checklist.bounced) return `${keeps}\nYou claimed completion, but these items the user asked for are still open, supplied as JSON data:\n${open}\nDo them, or show with evidence why one is already done or no longer wanted, then claim completion again.`; + return `${keeps} Still open, supplied as JSON data:\n${open}`; } export function activityResult(text: string): string { @@ -20,11 +29,13 @@ export function activityResult(text: string): string { } /** The first turn of a fresh reviewer conversation: the objective and the claim, both as data. */ -export function reviewPrompt(objective: string, claim: string, round: number): string { +export function reviewPrompt(objective: string, claim: string, round: number, checklist: GoalChecklistItem[] = []): string { + const items = checklist.filter(item => item.status !== 'cancelled').map(({ text, status }) => ({ text, status })); + const list = items.length ? `The user's requests as Boite's checklist keeps them, supplied as JSON data. A status is a quick reading of the agent's turns, not a verification: check every item:\n${JSON.stringify(items)}\n` : ''; return `You review a Boite goal (review ${round}). Another agent worked on the objective below in this checkout and claims it is complete. You have none of its conversation: judge the work itself, not its report. Objective, supplied as JSON data: ${JSON.stringify(objective)} -The agent's final report, supplied as JSON data. It is a claim to check, not evidence: +${list}The agent's final report, supplied as JSON data. It is a claim to check, not evidence: ${JSON.stringify(claim.slice(0, 4000))} Inspect what changed (git status, git diff, new files), run the checks that prove the objective (type check, tests, build or a direct run of the feature) and look for missing parts and regressions. Do not edit, create or delete project files, and do not commit. Above your verdict, list each problem on its own line starting with "- ", most important first, with file paths and the failing command's output where it helps. Then end your answer with exactly one of these lines: diff --git a/packages/core/src/activity.ts b/packages/core/src/activity.ts index fedcd5198..b0051fac5 100644 --- a/packages/core/src/activity.ts +++ b/packages/core/src/activity.ts @@ -5,6 +5,8 @@ import { activityResult, parseVerdict, reviewPrompt } from './activity-prompt.ts import { checkAttachmentArray, checkAttachments } from './threads/inputs.ts'; import { assertDriverRunnable } from './drivers/index.ts'; import type { ChildRoute } from './delegation/routes.ts'; +import { GoalTracker, modelRead, newChecklist } from './goal-tracker.ts'; +import type { ProviderProbe } from './providers/probe.ts'; const empty = (): ThreadActivity => ({ goal: null, loop: null, tasks: [] }); type PendingAttachments = Partial>; @@ -13,6 +15,8 @@ export interface GoalOutcome { threadId: string; outcome: 'complete' | 'blocked' /** Reviews of a `/goal2` before it stops for the user. */ export const GOAL_REVIEW_ROUNDS = 5; const newReview = (): GoalReview => ({ phase: 'working', maxRounds: GOAL_REVIEW_ROUNDS, reviewerThreadId: null, claimTurnId: null, delivered: 0, verdicts: [] }); +/** Completion claims one open checklist item may send back before the review decides alone. */ +export const CHECKLIST_BOUNCES = 2; /** The review a reviewer conversation's first turn answers. */ interface ReviewRun { threadId: string; generation: number; round: number } @@ -27,11 +31,27 @@ export class ActivityStore { private readonly reviewers = new Set(); /** Reviewer conversation to the review its first turn answers, until that turn ends. */ private readonly reviewing = new Map(); + /** Goals whose last completion claim the checklist sent back: their next turn says so. */ + private readonly bounced = new Set(); + /** Keeps each `/goal2`'s checklist ([goal-tracker](./goal-tracker.ts)). */ + readonly tracker: GoalTracker; + /** Reads an agent's models before a reviewer starts on one; set by `modules.ts`. */ + private probe: ProviderProbe | null = null; + /** Goals whose reviewer waits for its agent's models to be read. */ + private readonly readingModels = new Set(); + /** The goal generation and round whose reviewer already had its models read: one read per review. */ + private readonly modelsRead = new Map(); private closed = false; /** `threadJsonChecked`: the core validated visible thread JSON just before, with nothing written since. */ constructor(private readonly core: Core, threadJsonChecked = false) { if (!threadJsonChecked) core.journal.validateVisibleThreadJson(); + this.tracker = new GoalTracker(core, { + goal: threadId => this.states.get(threadId)?.goal ?? null, + generation: threadId => this.generations.get(`${threadId}:goal`) ?? 0, + save: threadId => this.save(threadId), + checked: threadId => this.checklistChecked(threadId), + }); const savedThreads = core.journal.db.query(`SELECT threads.id FROM threads JOIN settings ON settings.key = 'activity:' || threads.id WHERE threads.id NOT IN (SELECT thread_id FROM thread_deletions) ORDER BY threads.rowid`).all() as { id: string }[]; for (const thread of savedThreads) { @@ -42,6 +62,7 @@ export class ActivityStore { for (const verdict of review?.verdicts ?? []) this.reviewers.add(verdict.threadId); // The interrupted review is not adopted: resuming starts a fresh one. if (review?.reviewerThreadId) { this.reviewers.add(review.reviewerThreadId); review.reviewerThreadId = null; changed = true; } + if (saved.goal?.checklist && this.tracker.adopt(thread.id, saved.goal.checklist)) changed = true; for (const run of saved.loop?.history ?? []) { if (run.status === 'running') { run.status = 'error'; run.summary = 'Core restarted before this iteration finished.'; run.finishedAt = Date.now(); changed = true; } } @@ -65,6 +86,10 @@ export class ActivityStore { if (review) this.reviewEnded(review, threadId, 'The reviewer conversation was removed. Resume to review again.'); for (const [reviewer, run] of this.reviewing) if (run.threadId === threadId) this.reviewing.delete(reviewer); this.reviewers.delete(threadId); + this.tracker.removed(threadId); + this.bounced.delete(threadId); + this.readingModels.delete(threadId); + this.modelsRead.delete(threadId); this.clearTimer(threadId); this.states.delete(threadId); core.journal.deleteSetting(`activity:${threadId}`); @@ -73,6 +98,9 @@ export class ActivityStore { }); } + /** How reviewers and checklist trackers read an agent's models before starting on one. */ + useProbe(probe: ProviderProbe): void { this.probe = probe; this.tracker.useProbe(probe); } + /** Called after the goal's new state is saved. Returns the unsubscribe. */ onGoal(listener: (event: GoalOutcome) => void): () => void { this.goalListeners.add(listener); @@ -97,6 +125,15 @@ export class ActivityStore { /** A `/goal2` reviewer: its answers go to the goal, never to the parent as mail. */ ownsReviewer(threadId: string): boolean { return this.reviewers.has(threadId); } + /** A `/goal2` checklist's tracker: its answers go to the checklist, never to the parent as mail. */ + ownsTracker(threadId: string): boolean { return this.tracker.owns(threadId); } + + /** A message of the user on a thread whose `/goal2` runs: its checklist may change. */ + userMessage(threadId: string, text: string): void { + const goal = this.states.get(threadId)?.goal; + if (goal?.checklist && goal.status !== 'complete') this.tracker.message(threadId, text); + } + set(params: RpcParams<'threads.activity.set'>): ThreadActivity { const thread = this.core.journal.getThread(params.threadId); if (thread?.agentSessionId) throw refused('persistent agent sessions are scheduled through Agents, not thread goals or loops'); @@ -111,7 +148,8 @@ export class ActivityStore { if (params.goal && params.goal.review !== undefined && typeof params.goal.review !== 'boolean') throw invalidParams('goal.review must be a boolean'); if (params.goal?.review && thread.parentThreadId) throw refused('a reviewed goal starts reviewer conversations; start it in a top-level thread'); this.stopReview(params.threadId); - state.goal = params.goal === null ? null : { objective: params.goal.objective.trim(), status: 'active', iterations: 0, error: null, ...(params.goal.review ? { review: newReview() } : {}) }; + this.bounced.delete(params.threadId); + state.goal = params.goal === null ? null : { objective: params.goal.objective.trim(), status: 'active', iterations: 0, error: null, ...(params.goal.review ? { review: newReview(), checklist: newChecklist() } : {}) }; } if (params.loop !== undefined) { if (params.loop !== null) validateLoop(params.loop); @@ -127,6 +165,10 @@ export class ActivityStore { this.saveAttachments(params.threadId, pending); this.states.set(params.threadId, state); this.save(params.threadId); + if (params.goal !== undefined) { + if (state.goal?.checklist) this.tracker.begin(params.threadId); + else this.tracker.stop(params.threadId); + } this.schedule(params.threadId, 0); return this.get(params.threadId); } @@ -161,6 +203,8 @@ export class ActivityStore { if (state.loop && state.loop.status !== 'active') state.loop.nextRunAt = null; if (params.kind === 'loop' && params.action === 'resume' && state.loop) state.loop.nextRunAt = Date.now(); this.states.set(params.threadId, state); + // A paused goal's checklist still follows the conversation; a removed or finished one stops, on the state just stored. + if (params.kind === 'goal' && (params.action === 'remove' || params.action === 'complete')) { this.tracker.stop(params.threadId); this.bounced.delete(params.threadId); } this.save(params.threadId); this.schedule(params.threadId, 0); return this.get(params.threadId); @@ -228,9 +272,12 @@ export class ActivityStore { private finished(turn: Turn): void { const review = this.reviewing.get(turn.threadId); if (review) { this.reviewFinished(turn, review); return; } + if (this.tracker.finished(turn)) return; const owned = this.ownTurns.get(turn.id); this.ownTurns.delete(turn.id); const state = this.states.get(turn.threadId); + // Every finished turn of the conversation, the user's included: the goal's next turn waits for this check. + if (turn.status === 'done' && !turn.execution?.operation && state?.goal?.checklist && state.goal.status !== 'complete') this.tracker.turnEnded(turn.threadId, turn.id); const current = owned && owned.generation === (this.generations.get(`${turn.threadId}:${owned.kind}`) ?? 0); if (current && owned.kind === 'loop' && state?.loop) { const run = state.loop.history?.find(run => run.turnId === turn.id); @@ -290,15 +337,19 @@ export class ActivityStore { return; } const review = kind === 'goal' ? state.goal!.review : undefined; + // The checklist's check of the last turn decides what the next turn hears, or whether a claim goes back. + if (kind === 'goal' && this.tracker.holds(threadId)) return; if (review?.phase === 'reviewing') { this.startReview(threadId, thread); return; } const last = review?.verdicts.at(-1); // Only the first turn after a review carries its findings; later ones have them in the history. const feedback = review && last && last.verdict !== 'pass' && review.delivered < review.verdicts.length ? { round: last.round, feedback: last.summary } : undefined; + const list = kind === 'goal' ? state.goal!.checklist : undefined; + const checklist = list?.items.length ? { open: list.items.filter(item => item.status === 'open').map(item => item.text), bounced: this.bounced.delete(threadId) } : undefined; const prompt = `/${review ? 'goal2' : kind} ${kind === 'goal' ? state.goal!.objective : state.loop!.prompt}`; const iteration = state[kind]!.iterations + 1; try { const pending = this.pendingAttachments(threadId); - const turnActivity: ActivityTurn = { kind, iteration, ...(feedback ? { review: feedback } : {}) }; + const turnActivity: ActivityTurn = { kind, iteration, ...(feedback ? { review: feedback } : {}), ...(checklist ? { checklist } : {}) }; const turn = this.core.threads.startTurn(threadId, prompt, pending[kind] ?? [], undefined, undefined, turnActivity); delete pending[kind]; this.saveAttachments(threadId, pending); @@ -349,13 +400,23 @@ export class ActivityStore { private startReview(threadId: string, parent: ThreadSummary): void { const goal = this.states.get(threadId)?.goal; const review = goal?.review; - if (!goal || !review || goal.status !== 'active' || review.phase !== 'reviewing') return; + if (!goal || !review || goal.status !== 'active' || review.phase !== 'reviewing' || this.readingModels.has(threadId)) return; for (const run of this.reviewing.values()) if (run.threadId === threadId) return; const round = review.verdicts.length + 1; const generation = this.generations.get(`${threadId}:goal`) ?? 0; let reviewer: string | null = null; try { const route = this.reviewerRoute(parent); + // An account change since the last read forgets its agent's models: read them, as a spawn does, then + // start through the scheduler, which finds the goal as it is then: replaced, paused or still reviewing. + const attempt = `${generation}:${round}`; + const read = this.modelsRead.get(threadId) === attempt ? null : modelRead(this.core, this.probe, route); + if (read) { + this.modelsRead.set(threadId, attempt); + this.readingModels.add(threadId); + void read.catch(() => undefined).then(() => { this.readingModels.delete(threadId); this.schedule(threadId, 0); }); + return; + } const provider = this.core.providers.require(route.providerId); assertDriverRunnable(provider.protocol, this.core.providers.summary(provider.id), this.core.accounts.require(route.accountId), () => this.core.providers.launcherScriptOnly(provider.id)); const claim = review.claimTurnId ? this.answer({ threadId, id: review.claimTurnId }) : ''; @@ -363,7 +424,7 @@ export class ActivityStore { this.reviewing.set(reviewer, { threadId, generation, round }); review.reviewerThreadId = reviewer; this.save(threadId); - this.core.threads.startTurn(reviewer, reviewPrompt(goal.objective, claim, round), [], undefined, 'delegation', undefined, undefined, `Review the goal: ${goal.objective}`.slice(0, 500)); + this.core.threads.startTurn(reviewer, reviewPrompt(goal.objective, claim, round, goal.checklist?.items ?? []), [], undefined, 'delegation', undefined, undefined, `Review the goal: ${goal.objective}`.slice(0, 500)); } catch (error) { if (reviewer) this.reviewing.delete(reviewer); review.reviewerThreadId = null; @@ -407,6 +468,26 @@ export class ActivityStore { else this.save(run.threadId); } + /** + * The checklist read the agent's last turn. A completion claim with an item + * still open goes back to the agent instead of to a reviewer, at most + * `CHECKLIST_BOUNCES` times per item; after that the review judges alone. + */ + private checklistChecked(threadId: string): void { + const goal = this.states.get(threadId)?.goal; + const review = goal?.review; + if (goal?.status === 'active' && review?.phase === 'reviewing' && !review.reviewerThreadId && goal.checklist) { + const open = goal.checklist.items.filter(item => item.status === 'open' && (item.bounced ?? 0) < CHECKLIST_BOUNCES); + if (open.length) { + for (const item of open) item.bounced = (item.bounced ?? 0) + 1; + Object.assign(review, { phase: 'working', claimTurnId: null }); + this.bounced.add(threadId); + this.save(threadId); + } + } + this.schedule(threadId, 0); + } + private stopReview(threadId: string): void { for (const [reviewer, run] of this.reviewing) { if (run.threadId !== threadId) continue; @@ -422,7 +503,7 @@ export class ActivityStore { this.core.bus.emit('thread.activity', { threadId, activity }); } - close(): void { this.closed = true; for (const threadId of this.states.keys()) this.pauseAll(threadId); } + close(): void { this.closed = true; this.tracker.close(); for (const threadId of this.states.keys()) this.pauseAll(threadId); } } export function validateLoop(loop: NonNullable['loop']>): void { diff --git a/packages/core/src/delegation.ts b/packages/core/src/delegation.ts index e5834c3cd..01fe421f2 100644 --- a/packages/core/src/delegation.ts +++ b/packages/core/src/delegation.ts @@ -424,11 +424,11 @@ export class Delegation { } /** * A child the core runs for a plan rather than for the agent: a workflow - * step, a `/goal2` reviewer or an agent's memory pass. It needs no enabled team, only an unpaused + * step, a `/goal2` reviewer or checklist tracker, or an agent's memory pass. It needs no enabled team, only an unpaused * one, and its answer never comes back as mail. */ private stepOwned(threadId: string): boolean { - return this.core.workflows.owns(threadId) || this.core.activity.ownsReviewer(threadId) || this.core.workforce.memory.owns(threadId); + return this.core.workflows.owns(threadId) || this.core.activity.ownsReviewer(threadId) || this.core.activity.ownsTracker(threadId) || this.core.workforce.memory.owns(threadId); } /** * Whether a child's team lets its turn start. The scheduler passes the parent it @@ -822,8 +822,8 @@ export class Delegation { instructions(threadId: string, request?: string, fresh = true): string { const thread = this.core.threads.require(threadId), root = this.root(threadId), config = this.config(root.id); if (thread.parentThreadId && this.core.workflows.owns(threadId)) return this.core.workflows.instructions(threadId); - // A memory pass's prompt says everything it needs; it neither delegates nor writes mail. - if (thread.parentThreadId && this.core.workforce.memory.owns(threadId)) return ''; + // A memory pass's or a `/goal2` checklist pass's prompt says everything it needs; it neither delegates nor writes mail. + if (thread.parentThreadId && (this.core.workforce.memory.owns(threadId) || this.core.activity.ownsTracker(threadId))) return ''; if (thread.parentThreadId && this.core.activity.ownsReviewer(threadId)) return '\nYou are a Boite goal reviewer. Your final answer and its VERDICT line go to the goal automatically; do not message the parent, start subagents or edit project files.\n'; if (thread.parentThreadId) { const own = this.worktrees.get(threadId); diff --git a/packages/core/src/goal-tracker-prompt.ts b/packages/core/src/goal-tracker-prompt.ts new file mode 100644 index 000000000..0425160bb --- /dev/null +++ b/packages/core/src/goal-tracker-prompt.ts @@ -0,0 +1,208 @@ +import { GOAL_CHECKLIST_MAX_ITEMS, GOAL_CHECKLIST_SOURCE_MAX, GOAL_CHECKLIST_TEXT_MAX } from '@boite/contracts'; +import type { GoalChecklistItem, Message } from '@boite/contracts'; +import { activityResult } from './activity-prompt.ts'; + +/* + * The prompts of a `/goal2` tracker and the reading of its answers. A tracker + * pass answers with one JSON block; anything else in it is ignored. + */ + +const HEAD = 'You keep the checklist of what the user asked for in a Boite goal. Another agent does the work; you only keep the list. Use no tool: answer from the text below.'; + +const RULES = `Rules for items: +- One item per distinct thing the user asks for. A sentence that asks for two things gives two items. Keep every request, even a small, vague or passing one. +- Read the user's corrections as meant: "or rather", "actually", "but", "att" and a word followed by * replace what came before. Keep the final meaning, never its opposite. +- A constraint on the result ("do not change the save format") is an item. How the agent should organise its work (subagents, worktrees, commits) is not. +- Never invent work the user did not ask for, and do not split a request into implementation steps. +- text: in the user's language, at most 12 words, starting with a verb, the visible result in plain words. No file paths or code names: someone who forgot the conversation must understand it. +- When the request points at an attached image, end the text with "(image N)". +- source: the exact words of the user's message the item comes from, at most 200 characters.`; + +const END = 'End your answer with one fenced json block and nothing after it:'; + +/** The first pass: the objective becomes the list. */ +export function writePrompt(objective: string): string { + return `${HEAD} + +${RULES} + +The user's objective, supplied as JSON data: +${JSON.stringify(objective)} + +${END} +\`\`\`json +{"items":[{"text":"...","source":"..."}]} +\`\`\``; +} + +/** A later message of the user: what it adds, drops or changes. */ +export function updatePrompt(items: GoalChecklistItem[], messages: string[]): string { + const current = items.map(({ id, text, status }) => ({ id, text, status })); + return `${HEAD} + +The user sent new messages while the agent works. Update the checklist: +- A new request becomes a new item. +- A message that drops an item cancels it. +- A message that changes an item changes it: give its new text. +- A message that only answers a question, thanks the agent or comments on the work changes nothing. + +${RULES} + +The current checklist, supplied as JSON data: +${JSON.stringify(current)} + +The new messages, oldest first, supplied as JSON data: +${JSON.stringify(messages)} + +${END} (empty lists when nothing changes) +\`\`\`json +{"add":[{"text":"...","source":"..."}],"cancel":[3],"change":[{"id":4,"text":"...","source":"..."}]} +\`\`\``; +} + +/** After the agent's turn: which open items it did. */ +export function checkPrompt(items: GoalChecklistItem[], digest: TurnDigest): string { + const open = items.filter(item => item.status === 'open').map(({ id, text }) => ({ id, text })); + return `You keep the checklist of what the user asked for in a Boite goal. The agent working on it just finished a turn. Say which open items that turn did. Use no tool: judge only from the report below. +You skim, you do not review. An item is done when the turn's actions clearly did it and the agent's answer agrees. An item the actions never touched stays open, even when the agent claims it. When unsure, leave it open. + +The open items, supplied as JSON data: +${JSON.stringify(open)} + +What the agent did in that turn, supplied as JSON data. The answer is a claim, not evidence: +${JSON.stringify(digest)} + +${END} +\`\`\`json +{"done":[1,4]} +\`\`\``; +} + +/** The rules every tracker conversation starts with, on every agent. */ +export const TRACKER_RULES = 'You keep a Boite goal\'s checklist. Answer only from the text each message gives you. Do not run commands, read or edit files, start subagents or message anyone.'; + +/** What a turn did, as a tracker reads it: its tool calls in one line each and the agent's last answer. */ +export interface TurnDigest { + actions: string[]; + answer: string; +} + +const ACTIONS_MAX = 120; +const ACTION_CHARS = 200; +const ANSWER_CHARS = 4000; + +/** One line per tool call, the failed ones marked, then the last assistant text without goal markers. */ +export function turnDigest(messages: Iterable): TurnDigest { + const actions: string[] = []; + let answer = ''; + for (const message of messages) { + if (message.role !== 'assistant') continue; + const texts: string[] = []; + for (const part of message.parts) { + if (part.type === 'text') texts.push(part.text); + if (part.type !== 'tool') continue; + const failed = part.status === 'error' || (typeof part.exitCode === 'number' && part.exitCode !== 0); + actions.push(`${toolLine(part.name, part.input)}${failed ? ' (failed)' : ''}`.slice(0, ACTION_CHARS)); + } + const text = activityResult(texts.join('\n')); + if (text) answer = text; + } + return { actions: actions.slice(-ACTIONS_MAX), answer: answer.slice(-ANSWER_CHARS) }; +} + +/** Several turns read as one: their actions in order and the last answer. */ +export function joinDigests(digests: TurnDigest[]): TurnDigest { + return { actions: digests.flatMap(digest => digest.actions).slice(-ACTIONS_MAX), answer: digests.findLast(digest => digest.answer)?.answer ?? '' }; +} + +function toolLine(name: string, input: unknown): string { + const fields = input && typeof input === 'object' ? input as Record : {}; + const path = [fields.file_path, fields.path, fields.filePath, fields.notebook_path].find(value => typeof value === 'string'); + if (typeof path === 'string') return `${name} ${path}`; + const command = fields.command ?? fields.cmd; + if (typeof command === 'string') return `${name}: ${command}`; + if (Array.isArray(command)) return `${name}: ${command.filter(word => typeof word === 'string').join(' ')}`; + const patch = [fields.patch, fields.input].find(value => typeof value === 'string'); + if (typeof patch === 'string') { + const files = [...patch.matchAll(/^\*\*\* (?:Add|Update|Delete) File: (.+)$/gm)].map(match => match[1]!.trim()); + if (files.length) return `${name} ${files.join(', ')}`; + } + return name; +} + +/** The last JSON object of an answer: a fenced block first, else the last brace that parses. */ +export function lastJson(text: string): Record | null { + const blocks = [...text.matchAll(/```[a-zA-Z]*[ \t]*\r?\n([\s\S]*?)```/g)].map(match => match[1]!); + for (const block of blocks.reverse()) { + const value = parseObject(block); + if (value) return value; + } + const end = text.lastIndexOf('}'); + if (end < 0) return null; + for (let start = text.lastIndexOf('{', end), tries = 0; start >= 0 && tries < 200; start = text.lastIndexOf('{', start - 1), tries++) { + const value = parseObject(text.slice(start, end + 1)); + if (value) return value; + } + return null; +} + +function parseObject(text: string): Record | null { + try { + const value: unknown = JSON.parse(text.trim()); + return value && typeof value === 'object' && !Array.isArray(value) ? value as Record : null; + } catch { return null; } +} + +const clip = (value: unknown, max: number): string => typeof value === 'string' ? value.replace(/\s+/g, ' ').trim().slice(0, max) : ''; +const ids = (value: unknown): number[] => Array.isArray(value) ? value.filter((id): id is number => Number.isInteger(id)) : []; + +/** A written item without its id: what a write or an add gives. */ +type Draft = Pick; + +function drafts(value: unknown): Draft[] { + if (!Array.isArray(value)) return []; + return value.flatMap(entry => { + const row = entry && typeof entry === 'object' ? entry as Record : {}; + const text = clip(row.text, GOAL_CHECKLIST_TEXT_MAX); + return text ? [{ text, source: clip(row.source, GOAL_CHECKLIST_SOURCE_MAX) }] : []; + }); +} + +/** The list a first pass wrote, or the reason it is unusable. */ +export function readWrite(answer: string): GoalChecklistItem[] | string { + const json = lastJson(answer); + if (!json || !Array.isArray(json.items)) return 'The tracker answered without a {"items": [...]} JSON block.'; + const items = drafts(json.items).slice(0, GOAL_CHECKLIST_MAX_ITEMS); + if (!items.length) return 'The tracker wrote an empty checklist.'; + return items.map((item, index) => ({ id: index + 1, ...item, status: 'open' })); +} + +/** The list after an update's additions, cancellations and changes, or the reason the answer is unusable. */ +export function readUpdate(answer: string, items: GoalChecklistItem[]): GoalChecklistItem[] | string { + const json = lastJson(answer); + if (!json || !['add', 'cancel', 'change'].some(key => key in json)) return 'The tracker answered without an {"add", "cancel", "change"} JSON block.'; + const cancelled = new Set(ids(json.cancel)); + const changes = new Map(); + for (const entry of Array.isArray(json.change) ? json.change : []) { + const row = entry && typeof entry === 'object' ? entry as Record : {}; + const text = clip(row.text, GOAL_CHECKLIST_TEXT_MAX); + if (Number.isInteger(row.id) && text) changes.set(row.id as number, { text, source: clip(row.source, GOAL_CHECKLIST_SOURCE_MAX) }); + } + const next = items.map((item): GoalChecklistItem => { + if (cancelled.has(item.id)) return { ...item, status: 'cancelled' }; + const change = changes.get(item.id); + // A changed request is work again, whatever was done for the old one. + return change && change.text !== item.text ? { ...item, ...change, source: change.source || item.source, status: 'open', changed: true, bounced: 0 } : item; + }); + let id = Math.max(0, ...items.map(item => item.id)); + for (const draft of drafts(json.add)) next.push({ id: ++id, ...draft, status: 'open', added: true }); + return next.slice(0, GOAL_CHECKLIST_MAX_ITEMS); +} + +/** The open items a check saw done, or the reason its answer is unusable. */ +export function readCheck(answer: string, items: GoalChecklistItem[]): GoalChecklistItem[] | string { + const json = lastJson(answer); + if (!json || !Array.isArray(json.done)) return 'The tracker answered without a {"done": [...]} JSON block.'; + const done = new Set(ids(json.done)); + return items.map(item => item.status === 'open' && done.has(item.id) ? { ...item, status: 'done' } : item); +} diff --git a/packages/core/src/goal-tracker.ts b/packages/core/src/goal-tracker.ts new file mode 100644 index 000000000..a60888f5d --- /dev/null +++ b/packages/core/src/goal-tracker.ts @@ -0,0 +1,361 @@ +import { CONTEXT_TRIMMABLE_PROTOCOLS, GOAL_TRACKER_DEFAULTS, defaultTrackerModel } from '@boite/contracts'; +import type { GoalChecklist, ThreadActivity, ThreadSummary, Turn } from '@boite/contracts'; +import type { Core } from './core.ts'; +import { messageOf, refused } from './errors.ts'; +import { accountFor, conversationRoute, type ChildRoute } from './delegation/routes.ts'; +import { checkEffort, modelsFor, PROBED_PROTOCOLS } from './threads/selection.ts'; +import { assertDriverRunnable, probedModelsOf } from './drivers/index.ts'; +import type { ProviderProbe } from './providers/probe.ts'; +import { checkPrompt, joinDigests, readCheck, readUpdate, readWrite, TRACKER_RULES, turnDigest, updatePrompt, writePrompt } from './goal-tracker-prompt.ts'; + +/** + * The tracker of a `/goal2`: one child conversation per goal that writes the + * checklist from the objective, updates it on the user's later messages and, + * after each turn of the agent, marks what that turn did. Its passes run one + * at a time, in the order they were asked for; a check holds the goal's next + * turn until it answers, so the agent hears what is still open. + */ + +/** A pass that has not answered by then is stopped; the goal goes on without it. Mutable for tests. */ +export const TRACKER_PASS = { timeoutMs: 5 * 60_000, retryMs: 250 }; +/** First passes tried before a goal goes on without a checklist. */ +const WRITE_TRIES = 3; +/** Polls of a tracker conversation that is still settling its last turn. */ +const BUSY_TRIES = 40; + +type Goal = NonNullable; +type Job = { kind: 'write' } | { kind: 'update'; messages: string[] } | { kind: 'check'; turns: string[] }; + +/** What the goal's activity store lends the tracker. */ +export interface TrackerHost { + /** The live goal: the tracker changes its checklist, then saves. */ + goal(threadId: string): Goal | null; + generation(threadId: string): number; + save(threadId: string): void; + /** A check answered, failed or was dropped: the goal may go on. */ + checked(threadId: string): void; +} + +interface Tracking { + generation: number; + written: boolean; + writes: number; + /** The user's messages no pass has read yet. */ + messages: string[]; + /** The agent's turns no check has read yet. */ + turns: string[]; + /** The pass in flight, on the tracker conversation `threadId`. */ + running: { job: Job; threadId: string; turnId: string; timer: ReturnType } | null; + /** Its agent's models are being read before the tracker conversation starts; once per try. */ + reading: boolean; + read: boolean; + busy: number; + timer: ReturnType | null; +} + +export const newChecklist = (): GoalChecklist => ({ threadId: null, phase: 'idle', items: [], model: null, effort: null, error: null }); + +/** A read of an agent's models that has not answered by then gives up; the child's creation then says why. */ +const MODEL_READ_MS = 30_000; + +/** + * A child about to start on a model its agent has not listed since the last + * account change forgot the list: that route's own account reads it first, as + * a spawn does. Null when the model is listed, the route takes the agent's + * default, the agent lists its models without a read, the list was read + * already (reading again would change nothing), or the core has no probe. + */ +export function modelRead(core: Core, probe: ProviderProbe | null, route: ChildRoute): Promise | null { + if (!probe || route.model === null) return null; + const provider = core.providers.require(route.providerId); + if (!PROBED_PROTOCOLS.includes(provider.protocol) || probedModelsOf(provider.protocol, provider.id, route.accountId) !== null) return null; + if (modelsFor(provider, route.accountId).some(model => model.id === route.model)) return null; + let timer: ReturnType | undefined; + const late = new Promise((_, reject) => { timer = setTimeout(() => reject(new Error('timed out reading its models')), MODEL_READ_MS); }); + return Promise.race([probe({ providerId: provider.id, accountId: route.accountId }), late]).then(() => undefined).finally(() => clearTimeout(timer)); +} + +export class GoalTracker { + private readonly goals = new Map(); + /** Every tracker conversation, to its goal's thread. */ + private readonly trackers = new Map(); + private closed = false; + private probe: ProviderProbe | null = null; + + constructor(private readonly core: Core, private readonly host: TrackerHost) {} + + /** How the tracker reads an agent's models before its conversation starts on one. */ + useProbe(probe: ProviderProbe): void { this.probe = probe; } + + /** A tracker conversation: delegation runs it as a step it does not own, and its answers never become mail. */ + owns(threadId: string): boolean { return this.trackers.has(threadId); } + + /** A saved goal's tracker after a restart: known again, its unread work lost. */ + adopt(goalThreadId: string, list: GoalChecklist): boolean { + if (list.threadId) this.trackers.set(list.threadId, goalThreadId); + if (list.phase === 'idle') return false; + list.phase = 'idle'; + return true; + } + + /** A new `/goal2`: its first pass writes the list. */ + begin(threadId: string): void { + this.stop(threadId); + this.goals.set(threadId, this.fresh(threadId, false)); + this.pump(threadId); + } + + /** A message of the user while the goal runs: the list may grow, shrink or change. */ + message(threadId: string, text: string): void { + const tracking = this.tracking(threadId); + if (!tracking || !text.trim()) return; + tracking.messages.push(text.trim()); + this.pump(threadId); + } + + /** A finished turn of the agent: the next check reads it, and the goal waits for that check. */ + turnEnded(threadId: string, turnId: string): void { + const tracking = this.tracking(threadId); + if (!tracking) return; + tracking.turns.push(turnId); + this.pump(threadId); + } + + /** True while a turn waits for its check: the goal's next turn or review waits too. */ + holds(threadId: string): boolean { + const tracking = this.goals.get(threadId); + return !!tracking && (tracking.turns.length > 0 || tracking.running?.job.kind === 'check'); + } + + /** The goal ended or was replaced: its unread work is dropped and a running pass stopped. */ + stop(threadId: string): void { + const tracking = this.goals.get(threadId); + if (!tracking) return; + this.goals.delete(threadId); + if (tracking.timer) clearTimeout(tracking.timer); + if (!tracking.running) return; + const { threadId: tracker, timer } = tracking.running; + clearTimeout(timer); + try { this.core.threads.stopTurn(tracker); } catch (error) { this.core.log('warn', `goal checklist ${tracker}: ${messageOf(error)}`); } + // A replaced goal's list is the new one's: only the list this pass worked on goes idle. + const list = this.host.goal(threadId)?.checklist; + if (list?.threadId === tracker) { list.phase = 'idle'; this.host.save(threadId); } + } + + /** A conversation was deleted: a goal's tracking or a tracker's record goes with it. */ + removed(threadId: string): void { + this.stop(threadId); + this.trackers.delete(threadId); + for (const [tracker, goal] of this.trackers) if (goal === threadId) this.trackers.delete(tracker); + } + + /** A tracker conversation's turn ended. False when the turn is not a tracker's. */ + finished(turn: Turn): boolean { + const threadId = this.trackers.get(turn.threadId); + if (threadId === undefined) return false; + const tracking = this.goals.get(threadId); + const run = tracking?.running; + if (!tracking || !run || run.turnId !== turn.id) return true; + clearTimeout(run.timer); + tracking.running = null; + const list = this.host.goal(threadId)?.checklist; + if (list && tracking.generation === this.host.generation(threadId)) { + if (turn.status !== 'done') list.error = `The checklist pass ${turn.status === 'stopped' ? 'was stopped' : `failed: ${turn.error ?? 'no error given'}`}.`; + else this.apply(tracking, list, run.job, this.answer(turn)); + list.phase = 'idle'; + this.host.save(threadId); + } + if (run.job.kind === 'check') this.host.checked(threadId); + this.later(threadId, TRACKER_PASS.retryMs); + return true; + } + + close(): void { + this.closed = true; + for (const threadId of [...this.goals.keys()]) this.stop(threadId); + } + + private apply(tracking: Tracking, list: GoalChecklist, job: Job, answer: string): void { + const result = job.kind === 'write' ? readWrite(answer) : job.kind === 'update' ? readUpdate(answer, list.items) : readCheck(answer, list.items); + if (typeof result === 'string') { list.error = result; return; } + list.items = result; + list.error = null; + if (job.kind === 'write') tracking.written = true; + } + + /** The goal's tracking, rebuilt after a restart from what its list already holds. */ + private tracking(threadId: string): Tracking | undefined { + const goal = this.host.goal(threadId); + if (this.closed || !goal?.checklist || goal.status === 'complete') return undefined; + let tracking = this.goals.get(threadId); + if (!tracking) this.goals.set(threadId, tracking = this.fresh(threadId, goal.checklist.items.length > 0)); + return tracking; + } + + private fresh(threadId: string, written: boolean): Tracking { + return { generation: this.host.generation(threadId), written, writes: written ? 1 : 0, messages: [], turns: [], running: null, reading: false, read: false, busy: 0, timer: null }; + } + + private later(threadId: string, delay: number): void { + const tracking = this.goals.get(threadId); + if (!tracking || this.closed) return; + if (tracking.timer) clearTimeout(tracking.timer); + tracking.timer = setTimeout(() => { tracking.timer = null; this.pump(threadId); }, delay); + } + + /** Start the next pass: the first list, then the user's messages, then the agent's turns. */ + private pump(threadId: string): void { + const tracking = this.goals.get(threadId); + const goal = this.host.goal(threadId); + const list = goal?.checklist; + if (this.closed || !tracking || tracking.running || tracking.reading || !goal || !list) return; + if (goal.status === 'complete' || tracking.generation !== this.host.generation(threadId)) { this.release(threadId, tracking); return; } + const waiting = tracking.messages.length > 0 || tracking.turns.length > 0; + if (!tracking.written) { + if (tracking.writes === 0 || (waiting && tracking.writes < WRITE_TRIES)) { this.pass(threadId, tracking, goal, { kind: 'write' }); return; } + // Without a list there is nothing to update or check. + if (tracking.writes >= WRITE_TRIES) { tracking.messages = []; this.release(threadId, tracking); } + return; + } + if (tracking.messages.length) { this.pass(threadId, tracking, goal, { kind: 'update', messages: tracking.messages.splice(0) }); return; } + if (!tracking.turns.length) return; + const turns = tracking.turns.splice(0); + // Nothing left open: no check to pay for. + if (!list.items.some(item => item.status === 'open')) { this.host.checked(threadId); return; } + this.pass(threadId, tracking, goal, { kind: 'check', turns }); + } + + /** Drop the turns waiting for a check and let the goal go on. */ + private release(threadId: string, tracking: Tracking): void { + if (!tracking.turns.length) return; + tracking.turns = []; + this.host.checked(threadId); + } + + private pass(threadId: string, tracking: Tracking, goal: Goal, job: Job): void { + const list = goal.checklist!; + const parent = this.core.journal.getThread(threadId); + // A first pass counts as tried even when it cannot start: the goal stops waiting for a list after three. + if (job.kind === 'write') tracking.writes++; + try { + if (!parent || parent.archived) throw refused('the goal\'s conversation is gone or archived'); + let tracker = list.threadId ? this.core.journal.getThread(list.threadId) : null; + if (tracker && (['queued', 'running', 'waiting'].includes(tracker.status) || this.core.threads.runner.handles.has(tracker.id))) { + // The last pass is still settling: try again in a moment. + if (++tracking.busy < BUSY_TRIES) { + if (job.kind === 'write') tracking.writes--; + this.putBack(tracking, job); + this.later(threadId, TRACKER_PASS.retryMs); + return; + } + // The next pass waits again in full: this one fails, counted, and lets the goal go on. + tracking.busy = 0; + throw refused('the checklist conversation is still busy with an earlier pass'); + } + tracking.busy = 0; + if (!tracker || tracker.archived) { + const route = this.route(parent); + const read = tracking.read ? null : modelRead(this.core, this.probe, route); + if (read) { + // Read once, then try again: a model still missing is refused by the conversation's creation. + if (job.kind === 'write') tracking.writes--; + this.putBack(tracking, job); + Object.assign(tracking, { reading: true, read: true }); + void read.catch(() => undefined).then(() => { tracking.reading = false; this.later(threadId, 0); }); + return; + } + tracker = this.core.threads.require(this.create(parent, goal, list, route)); + } + const provider = this.core.providers.require(tracker.providerId); + assertDriverRunnable(provider.protocol, this.core.providers.summary(provider.id), this.core.accounts.require(tracker.accountId), () => this.core.providers.launcherScriptOnly(provider.id)); + const prompt = job.kind === 'write' ? writePrompt(goal.objective) + : job.kind === 'update' ? updatePrompt(list.items, job.messages) + : checkPrompt(list.items, joinDigests(job.turns.map(id => turnDigest(this.core.journal.walkTurnMessages(threadId, id, { blobs: false }))))); + const label = job.kind === 'write' ? 'Write the checklist' : job.kind === 'update' ? 'Update the checklist from your message' : 'Check what the agent did'; + const turn = this.core.threads.startTurn(tracker.id, prompt, [], undefined, 'delegation', undefined, undefined, label); + const timer = setTimeout(() => this.timeout(threadId, turn.id), TRACKER_PASS.timeoutMs); + tracking.running = { job, threadId: tracker.id, turnId: turn.id, timer }; + list.phase = job.kind === 'check' ? 'checking' : 'writing'; + this.host.save(threadId); + } catch (error) { + list.error = `The checklist could not ${job.kind === 'write' ? 'be written' : job.kind === 'update' ? 'be updated' : 'be checked'}: ${messageOf(error)}`; + // The next try reads the agent's models again: an account change may have forgotten them meanwhile. + tracking.read = false; + list.phase = 'idle'; + this.host.save(threadId); + if (job.kind === 'check') this.host.checked(threadId); + // A first pass is tried again on the next message or turn, then the goal goes on without a list. + this.later(threadId, TRACKER_PASS.retryMs); + } + } + + private putBack(tracking: Tracking, job: Job): void { + if (job.kind === 'update') tracking.messages.unshift(...job.messages); + if (job.kind === 'check') tracking.turns.unshift(...job.turns); + } + + private create(parent: ThreadSummary, goal: Goal, list: GoalChecklist, route: ChildRoute): string { + const provider = this.core.providers.require(route.providerId); + // No guides, servers or hooks where the agent can leave them out, and one harmless tool: a pass reads only its prompt. + const context = CONTEXT_TRIMMABLE_PROTOCOLS.includes(provider.protocol) ? { preset: 'neutral', tools: ['Read'], rules: TRACKER_RULES } : { drop: ['guides'], rules: TRACKER_RULES }; + const id = this.core.delegation.createChild(parent, route, `Checklist · ${goal.objective}`.replace(/\s+/g, ' ').slice(0, 120), child => { this.trackers.set(child, parent.id); }, context); + Object.assign(list, { threadId: id, model: `${route.providerId}/${route.model ?? 'default'}`, effort: route.effort ?? null }); + return id; + } + + /** + * The model Settings names, else the default rules on the conversation's own + * agent, then on any signed-in one, else the conversation's own model. + */ + route(parent: ThreadSummary): ChildRoute { + const chosen = this.core.settings.get().goalTracker; + if (chosen) { + const provider = this.core.providers.get(chosen.providerId); + if (!provider) throw refused(`the checklist model ${chosen.providerId}/${chosen.model} is not installed on this machine; pick another in Settings, Experiments`, { field: 'goalTracker' }); + const account = accountFor(this.core, parent, provider.id); + if (!account || account.status === 'unauthenticated') throw refused(`${provider.name} has no signed-in account to keep the checklist; sign in or pick another model in Settings, Experiments`, { field: 'goalTracker' }); + const listed = modelsFor(provider, account.id).some(model => model.id === chosen.model); + const effort = listed ? checkEffort(provider, account.id, chosen.model, chosen.effort) : chosen.effort; + return { id: 'goal-tracker', name: chosen.model, providerId: provider.id, accountId: account.id, model: chosen.model, effort }; + } + const candidates = this.core.providers.available().flatMap(summary => { + const provider = this.core.providers.require(summary.id); + const account = accountFor(this.core, parent, summary.id); + if (!account || account.status === 'unauthenticated') return []; + const pick = defaultTrackerModel(provider.protocol, modelsFor(provider, account.id)); + return pick ? [{ provider, account, ...pick }] : []; + }); + const own = candidates.find(entry => entry.provider.id === parent.providerId); + const best = own ?? GOAL_TRACKER_DEFAULTS.flatMap(rule => candidates.filter(entry => entry.provider.protocol === rule.protocol))[0]; + if (!best) return conversationRoute(parent); + let effort: string | null = best.effort; + try { effort = checkEffort(best.provider, best.account.id, best.model, best.effort); } catch { effort = null; } + return { id: 'goal-tracker', name: best.model, providerId: best.provider.id, accountId: best.account.id, model: best.model, effort }; + } + + private timeout(threadId: string, turnId: string): void { + const tracking = this.goals.get(threadId); + if (!tracking?.running || tracking.running.turnId !== turnId) return; + const { job, threadId: tracker } = tracking.running; + tracking.running = null; + try { this.core.threads.stopTurn(tracker); } catch { /* it ended meanwhile */ } + const list = this.host.goal(threadId)?.checklist; + if (list) { + list.error = `The checklist pass took over ${Math.round(TRACKER_PASS.timeoutMs / 60_000)} minutes and was stopped.`; + list.phase = 'idle'; + this.host.save(threadId); + } + if (job.kind === 'check') this.host.checked(threadId); + this.later(threadId, TRACKER_PASS.retryMs); + } + + /** Every text the pass wrote, its JSON block included. */ + private answer(turn: Turn): string { + const texts: string[] = []; + for (const message of this.core.journal.walkTurnMessages(turn.threadId, turn.id, { blobs: false })) { + if (message.role !== 'assistant') continue; + for (const part of message.parts) if (part.type === 'text') texts.push(part.text); + } + return texts.join('\n'); + } +} diff --git a/packages/core/src/modules.ts b/packages/core/src/modules.ts index 70fc172a6..985f56fc9 100644 --- a/packages/core/src/modules.ts +++ b/packages/core/src/modules.ts @@ -86,6 +86,7 @@ export function registerModules(core: Core): void { registerProjectMethods(core); registerProviderMethods(core); const probe = registerProbeMethods(core); + core.activity.useProbe(probe); core.router.register('delegation.result', params => core.delegation.resultPage(params)); core.router.register('delegation.wait', (params, ctx) => core.delegation.wait(params, ctx.connection)); core.router.register('delegation.get', params => core.delegation.get(params.threadId)); diff --git a/packages/core/src/settings.ts b/packages/core/src/settings.ts index 6d370d009..ce9f2b1aa 100644 --- a/packages/core/src/settings.ts +++ b/packages/core/src/settings.ts @@ -62,6 +62,10 @@ export class SettingsStore { }); } } + const tracker = checked.patch.goalTracker; + if (tracker !== undefined && tracker !== null && this.core.providers.get(tracker.providerId) === undefined) { + throw invalidParams(`goalTracker names ${tracker.providerId}, which is not a loaded provider`, { field: 'goalTracker', expected: this.core.providers.available().map(provider => provider.id) }); + } return checked.patch; } diff --git a/packages/core/src/threads.ts b/packages/core/src/threads.ts index 22838f5d1..83e053d48 100644 --- a/packages/core/src/threads.ts +++ b/packages/core/src/threads.ts @@ -699,6 +699,8 @@ export class ThreadStore { // His own message, a resume or a parent's message to its subagent puts the agent back to work, // so a pause the core left on Stop, a failure or a restart ends with it; the owner's stays. if (!activity && !startedBy && (!operation || operation === 'resume' || (operation === 'delegation' && thread.parentThreadId))) this.core.coordination.resumeForUser(threadId); + // A `/goal2`'s checklist reads what the user adds, drops or changes. + if (!activity && !operation && !startedBy && !nativeCommandPrompt(prompt)) this.core.activity.userMessage(threadId, prompt); }); this.autoCompact.cancel(threadId); this.cacheKeepalive.cancel(threadId); diff --git a/packages/core/src/threads/turn-context.ts b/packages/core/src/threads/turn-context.ts index 1e2864877..ffbf0edde 100644 --- a/packages/core/src/threads/turn-context.ts +++ b/packages/core/src/threads/turn-context.ts @@ -504,7 +504,7 @@ export class TurnContexts { const note = moved?.type === 'text' ? moved.moved?.note ?? '' : ''; const sent = message.role === 'user' ? message.parts.find((part) => part.type === 'text' && part.sentFrom !== undefined) : undefined; return { - prompt: note + message.parts.map((part) => part.type === 'text' ? part.activity ? activityPrompt(part.activity.kind, part.text, part.activity.iteration, part.activity.review) : part.text : '').join(''), + prompt: note + message.parts.map((part) => part.type === 'text' ? part.activity ? activityPrompt(part.activity.kind, part.text, part.activity.iteration, part.activity.review, part.activity.checklist) : part.text : '').join(''), attachments, ...(note ? { moved: true as const } : {}), ...(sent?.type === 'text' && sent.sentFrom ? { sentFrom: { from: sent.sentFrom, messageId: message.id } } : {}), diff --git a/packages/core/src/threads/user-steering.ts b/packages/core/src/threads/user-steering.ts index a02e13669..2afeebdcb 100644 --- a/packages/core/src/threads/user-steering.ts +++ b/packages/core/src/threads/user-steering.ts @@ -74,6 +74,8 @@ export async function steerUser(core: Core, threads: ThreadStore, params: RpcPar runner.answerAfter.set(turnId, message.createdAt); saveThread(core, threads.require(threadId), 'thread.userInput'); })()); + // A `/goal2`'s checklist reads what the user adds, drops or changes. + core.activity.userMessage(threadId, prompt); return { accepted: true }; } catch (error) { throw refused(`delivery of this message is unconfirmed: ${messageOf(error)}`, { reason: 'delivery-uncertain', threadId }); diff --git a/packages/core/test/goal-review.test.ts b/packages/core/test/goal-review.test.ts index fbac64c68..d6c698f50 100644 --- a/packages/core/test/goal-review.test.ts +++ b/packages/core/test/goal-review.test.ts @@ -12,6 +12,9 @@ beforeEach(async () => { h = await startTestCore(); }); afterEach(async () => { restore?.(); restore = undefined; await h.stop(); }); const isReview = (ctx: TurnContext) => ctx.prompt.includes('You review a Boite goal'); +/** A checklist pass: here it lists the objective as one item and sees it done after the first turn. */ +const tracked = (ctx: TurnContext) => h.core.activity.ownsTracker(ctx.thread.id); +const checklist = (ctx: TurnContext) => ctx.prompt.includes('Say which open items') ? '```json\n{"done":[1]}\n```' : '```json\n{"items":[{"text":"Do the objective","source":"objective"}]}\n```'; /** An echo agent that answers with `answer(ctx)` and finishes at once. */ function agent(answer: (ctx: TurnContext) => string): () => void { @@ -19,7 +22,7 @@ function agent(answer: (ctx: TurnContext) => string): () => void { protocol: 'echo', startTurn(ctx) { const id = ctx.emit.startMessage('assistant'); - ctx.emit.part(id, 0, { type: 'text', text: answer(ctx) }); + ctx.emit.part(id, 0, { type: 'text', text: tracked(ctx) ? checklist(ctx) : answer(ctx) }); ctx.emit.complete(id, 'complete'); return { stop() {}, done: Promise.resolve({ status: 'done', sessionId: null, usage: null }) }; }, @@ -72,10 +75,11 @@ test('a reviewed goal sends its claim to a fresh reviewer and works on the findi expect((h.core.journal.db.query('SELECT count(*) AS n FROM delegation_messages').get() as { n: number }).n).toBe(0); const thread = await client.call('threads.get', { threadId }); const prompts = thread.messages.filter(m => m.role === 'user').map(m => m.parts[0]); - expect(prompts).toEqual([ - { type: 'text', text: '/goal2 Fix the parser', activity: { kind: 'goal', iteration: 1 } }, - { type: 'text', text: '/goal2 Fix the parser', activity: { kind: 'goal', iteration: 2, review: { round: 1, feedback: '- The parser test fails: expected 2, got 3' } } }, - ]); + // The first turn may start before or after the checklist is written; the second always follows its check. + expect(prompts).toHaveLength(2); + expect(prompts[0]).toMatchObject({ type: 'text', text: '/goal2 Fix the parser', activity: { kind: 'goal', iteration: 1 } }); + expect(prompts[0]).not.toHaveProperty('activity.review'); + expect(prompts[1]).toEqual({ type: 'text', text: '/goal2 Fix the parser', activity: { kind: 'goal', iteration: 2, review: { round: 1, feedback: '- The parser test fails: expected 2, got 3' }, checklist: { open: [], bounced: false } } }); }); test('a reviewed goal stops for the user after five failed reviews, and each reply allows one more', async () => { @@ -116,7 +120,7 @@ test('pausing a reviewed goal stops its reviewer, and resuming starts a fresh re startTurn(ctx) { const id = ctx.emit.startMessage('assistant'); if (!isReview(ctx)) { - ctx.emit.part(id, 0, { type: 'text', text: 'Done.\n[BOITE_GOAL_COMPLETE]' }); + ctx.emit.part(id, 0, { type: 'text', text: tracked(ctx) ? checklist(ctx) : 'Done.\n[BOITE_GOAL_COMPLETE]' }); ctx.emit.complete(id, 'complete'); return { stop() {}, done: Promise.resolve({ status: 'done', sessionId: null, usage: null }) }; } diff --git a/packages/core/test/goal-tracker.live.test.ts b/packages/core/test/goal-tracker.live.test.ts new file mode 100644 index 000000000..91c4273cc --- /dev/null +++ b/packages/core/test/goal-tracker.live.test.ts @@ -0,0 +1,62 @@ +import { existsSync, mkdtempSync, readFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { afterEach, beforeEach, describe, expect, test } from 'bun:test'; +import { removeDir, startTestCore, waitFor } from './harness.ts'; +import { modelsFor } from '../src/threads/selection.ts'; +import { probedModelsOf } from '../src/drivers/index.ts'; +import type { TestCore } from './harness.ts'; + +/** + * A real `/goal2` on the user's own Claude login: Haiku does the work and + * reviews it, the checklist runs on Automatic (Opus at medium). Both children + * start after the account check that forgets Claude's model list, so each + * reads it again first. Opt-in, it spends a few + * cents of the subscription. + */ +const live = process.env['BOITE_E2E_CLAUDE'] === '1' ? describe : describe.skip; + +live('goal checklist, live', () => { + let harness: TestCore; + let projectDir: string; + + beforeEach(async () => { + harness = await startTestCore(); + projectDir = mkdtempSync(join(tmpdir(), 'boite-live-goal-')); + }); + + afterEach(async () => { + await harness.stop(); + await removeDir(projectDir); + }); + + test('Opus writes the checklist with the correction kept, and checks both items done', async () => { + const client = await harness.connect(); + const project = await client.call('projects.add', { path: projectDir, name: 'live goal' }); + const account = (await client.call('accounts.list', {})).find((entry) => entry.providerId === 'claude' && entry.label === 'Default'); + if (account === undefined) throw new Error('no default claude account on this machine'); + const thread = await client.call('threads.create', { projectId: project.id, providerId: 'claude', accountId: account.id, model: 'claude-haiku-5-5', cwd: projectDir, permissionMode: 'bypassPermissions', title: 'live goal' }); + + await client.call('threads.activity.set', { threadId: thread.id, goal: { objective: 'crée hello.txt avec hi dedans. et aussi bye.txt avec bye, ou plutot au revoir dedans', review: true } }); + const report = () => { + const goal = harness.core.activity.get(thread.id).goal; + const family = [thread.id, ...harness.core.journal.listThreads().filter((row) => row.parentThreadId === thread.id).map((row) => row.id)]; + console.log(JSON.stringify({ goal: { ...goal, objective: undefined }, threads: family.map((id) => ({ id, model: harness.core.threads.get(id).model, effort: harness.core.threads.get(id).effort, status: harness.core.threads.get(id).status, turns: harness.core.journal.listTurns(id).map((turn) => [turn.status, turn.error]) })), permissions: harness.core.threads.listPermissions().map((request) => [request.threadId, request.toolName]), listed: modelsFor(harness.core.providers.require('claude'), account.id).map((model) => model.id), probed: probedModelsOf('claude-sdk', 'claude', account.id)?.map((model) => model.id) ?? null }, null, 1)); + }; + try { + await waitFor(() => { const goal = harness.core.activity.get(thread.id).goal; return goal?.status === 'complete' || goal?.status === 'paused'; }, 240_000); + } finally { report(); } + + const goal = harness.core.activity.get(thread.id).goal!; + expect(goal.status).toBe('complete'); + const list = goal.checklist!; + expect(list).toMatchObject({ model: 'claude/claude-opus-5-5', effort: 'medium', error: null, phase: 'idle' }); + expect(harness.core.threads.get(list.threadId!).contextMode).toMatchObject({ tools: ['Read'], guides: false }); + expect(list.items.length).toBeGreaterThanOrEqual(2); + expect(list.items.every((item) => item.status === 'done')).toBe(true); + // The correction keeps its final meaning. + expect(list.items.some((item) => /au revoir/i.test(item.text))).toBe(true); + expect(readFileSync(join(projectDir, 'bye.txt'), 'utf8')).toMatch(/au revoir/i); + expect(existsSync(join(projectDir, 'hello.txt'))).toBe(true); + }, 300_000); +}); diff --git a/packages/core/test/goal-tracker.test.ts b/packages/core/test/goal-tracker.test.ts new file mode 100644 index 000000000..d066d350e --- /dev/null +++ b/packages/core/test/goal-tracker.test.ts @@ -0,0 +1,396 @@ +import { afterEach, beforeEach, expect, test } from 'bun:test'; +import { DEFAULT_DELEGATION_CONFIG, defaultTrackerModel, type Message, type ThreadActivity } from '@boite/contracts'; +import { ActivityStore } from '../src/activity.ts'; +import { lastJson, readUpdate, TRACKER_RULES, turnDigest } from '../src/goal-tracker-prompt.ts'; +import { TRACKER_PASS } from '../src/goal-tracker.ts'; +import { setDriver } from '../src/drivers/index.ts'; +import { echoDriver } from '../src/drivers/echo.ts'; +import { ModelProbes } from '../src/drivers/model-probes.ts'; +import type { TurnContext } from '../src/drivers/types.ts'; +import { echoThread, startTestCore, waitFor, type TestCore } from './harness.ts'; + +let h: TestCore; +let restore: (() => void) | undefined; +beforeEach(async () => { h = await startTestCore(); }); +afterEach(async () => { restore?.(); restore = undefined; await h.stop(); }); + +type Pass = 'write' | 'update' | 'check'; +const passOf = (prompt: string): Pass => prompt.includes('Say which open items') ? 'check' : prompt.includes('The new messages, oldest first') ? 'update' : 'write'; +const json = (value: unknown) => `Here is the list.\n\`\`\`json\n${JSON.stringify(value)}\n\`\`\``; + +/** + * An echo core with three roles: the goal's agent, its checklist tracker and + * its reviewer, each answering from the test's script. The agent may report a + * tool call before its text. + */ +function roles(script: { + agent: (ctx: TurnContext) => { text: string; tool?: { name: string; input: unknown } }; + tracker: (pass: Pass, ctx: TurnContext) => string; + reviewer?: (ctx: TurnContext) => string; +}): () => void { + return setDriver('echo', { + protocol: 'echo', + startTurn(ctx) { + const id = ctx.emit.startMessage('assistant'); + if (h.core.activity.ownsTracker(ctx.thread.id)) ctx.emit.part(id, 0, { type: 'text', text: script.tracker(passOf(ctx.prompt), ctx) }); + else if (h.core.activity.ownsReviewer(ctx.thread.id)) ctx.emit.part(id, 0, { type: 'text', text: script.reviewer?.(ctx) ?? 'Checks pass.\nVERDICT: PASS' }); + else { + const { text, tool } = script.agent(ctx); + if (tool) ctx.emit.part(id, 0, { type: 'tool', toolId: 't1', name: tool.name, input: tool.input, output: 'ok', status: 'done' }); + ctx.emit.part(id, tool ? 1 : 0, { type: 'text', text }); + } + ctx.emit.complete(id, 'complete'); + return { stop() {}, done: Promise.resolve({ status: 'done', sessionId: null, usage: null }) }; + }, + }); +} + +const goalOf = (threadId: string) => h.core.activity.get(threadId).goal!; +const userParts = async (client: Awaited>, threadId: string) => + (await client.call('threads.get', { threadId })).messages.filter(m => m.role === 'user').map(m => m.parts[0]!); + +test('a reviewed goal\'s tracker writes the checklist, checks each turn and tells the next turn what is still open', async () => { + const client = await h.connect(); + const { threadId } = await echoThread(h, client); + const agent: string[] = []; + const passes: { pass: Pass; prompt: string; threadId: string }[] = []; + const reviews: string[] = []; + restore = roles({ + agent: ctx => { + agent.push(ctx.prompt); + return agent.length === 1 ? { text: 'Added the dark theme.', tool: { name: 'Edit', input: { file_path: 'src/theme.css' } } } : { text: 'Added the settings page.\n[BOITE_GOAL_COMPLETE]', tool: { name: 'Bash', input: { command: 'bun test' } } }; + }, + tracker: (pass, ctx) => { + passes.push({ pass, prompt: ctx.prompt, threadId: ctx.thread.id }); + if (pass === 'write') return json({ items: [{ text: 'Ajouter un thème sombre', source: 'un thème sombre' }, { text: 'Ajouter une page de réglages', source: 'une page de réglages' }] }); + return json({ done: ctx.prompt.includes('Added the dark theme') ? [1] : [2] }); + }, + reviewer: ctx => { reviews.push(ctx.prompt); return 'All there.\nVERDICT: PASS'; }, + }); + await client.call('threads.activity.set', { threadId, goal: { objective: 'fait un thème sombre et une page de réglages', review: true } }); + await waitFor(() => goalOf(threadId).status === 'complete'); + + const list = goalOf(threadId).checklist!; + expect(list.items.map(item => [item.id, item.text, item.status])).toEqual([[1, 'Ajouter un thème sombre', 'done'], [2, 'Ajouter une page de réglages', 'done']]); + expect(list).toMatchObject({ phase: 'idle', error: null, model: 'echo/echo' }); + // One tracker conversation for the whole goal, a child of its thread, with no guides and its own rules. + expect(new Set(passes.map(p => p.threadId))).toEqual(new Set([list.threadId!])); + expect(passes.map(p => p.pass)).toEqual(['write', 'check', 'check']); + const tracker = h.core.threads.get(list.threadId!); + expect(tracker.parentThreadId).toBe(threadId); + expect(tracker.contextMode).toMatchObject({ label: 'custom', instructions: false, guides: false, tools: ['Read'], rules: TRACKER_RULES }); + // A check reads the turn's tool calls and its answer; the objective goes to the first pass as data. + expect(passes[0]!.prompt).toContain(JSON.stringify('fait un thème sombre et une page de réglages')); + expect(passes[1]!.prompt).toContain('Edit src/theme.css'); + expect(passes[2]!.prompt).toContain('Bash: bun test'); + // The second turn hears what is still open; the agent's own task list is no longer asked for. + expect(agent).toHaveLength(2); + expect(agent[1]).toContain('Still open, supplied as JSON data:\n["Ajouter une page de réglages"]'); + expect(agent[1]).not.toContain('use your native planning tool'); + const parts = await userParts(client, threadId); + expect(parts.at(-1)).toMatchObject({ activity: { kind: 'goal', iteration: 2, checklist: { open: ['Ajouter une page de réglages'], bounced: false } } }); + // The reviewer gets the list too, and nothing went to the parent as mail. + expect(reviews).toHaveLength(1); + expect(reviews[0]).toContain('{"text":"Ajouter une page de réglages","status":"done"}'); + expect((h.core.journal.db.query('SELECT count(*) AS n FROM delegation_messages').get() as { n: number }).n).toBe(0); +}); + +test('a completion claim with an open item goes back to the agent, twice at most, before the review judges', async () => { + const client = await h.connect(); + const { threadId } = await echoThread(h, client); + const agent: string[] = []; + const reviews: string[] = []; + restore = roles({ + agent: ctx => { agent.push(ctx.prompt); return { text: 'All done.\n[BOITE_GOAL_COMPLETE]' }; }, + tracker: pass => pass === 'write' ? json({ items: [{ text: 'Fix the login', source: 'login' }, { text: 'Add an export button', source: 'export' }] }) : json({ done: [1] }), + reviewer: ctx => { reviews.push(ctx.prompt); return '- The export button is missing\nVERDICT: PASS'; }, + }); + await client.call('threads.activity.set', { threadId, goal: { objective: 'fix the login and add an export button', review: true } }); + await waitFor(() => goalOf(threadId).status === 'complete'); + + // Two claims went back without a review; the third went to the reviewer with the item still open. + expect(agent).toHaveLength(3); + expect(reviews).toHaveLength(1); + expect(agent[1]).toContain('You claimed completion, but these items the user asked for are still open, supplied as JSON data:\n["Add an export button"]'); + expect(agent[2]).toContain('You claimed completion'); + expect(reviews[0]).toContain('{"text":"Add an export button","status":"open"}'); + const goal = goalOf(threadId); + expect(goal.checklist!.items.map(item => [item.status, item.bounced ?? 0])).toEqual([['done', 0], ['open', 2]]); + expect(goal.review!.verdicts.map(v => v.round)).toEqual([1]); + const parts = await userParts(client, threadId); + expect(parts.slice(1).map(part => part.type === 'text' && part.activity?.checklist?.bounced)).toEqual([true, true]); +}); + +test('the user\'s later messages add, drop and change items, and his own turns are checked too', async () => { + const client = await h.connect(); + const { threadId } = await echoThread(h, client); + const updates: string[] = []; + let answered = false; + restore = roles({ + agent: ctx => { + if (ctx.prompt.startsWith('Blue.')) { answered = true; return { text: 'Blue it is.', tool: { name: 'Write', input: { path: 'logout.ts' } } }; } + return answered ? { text: 'Done.\n[BOITE_GOAL_COMPLETE]' } : { text: 'Which colour?\n[BOITE_GOAL_BLOCKED]' }; + }, + tracker: (pass, ctx) => { + if (pass === 'write') return json({ items: [{ text: 'Paint the header', source: 'header' }, { text: 'Add a settings page', source: 'settings' }] }); + if (pass === 'update') { updates.push(ctx.prompt); return json({ add: [{ text: 'Add a logout button', source: 'add a logout button' }], cancel: [2], change: [{ id: 1, text: 'Paint the header blue', source: 'Blue.' }] }); } + return json({ done: ctx.prompt.includes('logout.ts') ? [1, 3] : [] }); + }, + }); + await client.call('threads.activity.set', { threadId, goal: { objective: 'paint the header and add a settings page', review: true } }); + await waitFor(() => goalOf(threadId).blocked === true); + await waitFor(() => goalOf(threadId).checklist!.phase === 'idle'); + + h.core.threads.startTurn(threadId, 'Blue. Also drop the settings page and add a logout button'); + await waitFor(() => goalOf(threadId).status === 'complete'); + + expect(updates).toHaveLength(1); + expect(updates[0]).toContain(JSON.stringify(['Blue. Also drop the settings page and add a logout button'])); + expect(updates[0]).toContain('{"id":2,"text":"Add a settings page","status":"open"}'); + expect(goalOf(threadId).checklist!.items.map(({ id, text, status, added, changed }) => ({ id, text, status, added, changed }))).toEqual([ + { id: 1, text: 'Paint the header blue', status: 'done', added: undefined, changed: true }, + { id: 2, text: 'Add a settings page', status: 'cancelled', added: undefined, changed: undefined }, + { id: 3, text: 'Add a logout button', status: 'done', added: true, changed: undefined }, + ]); +}); + +test('the checklist runs on the model Settings names, and an unknown provider is refused', async () => { + const client = await h.connect(); + const { threadId } = await echoThread(h, client); + await expect(client.call('settings.set', { goalTracker: { providerId: 'nobody', model: 'x', effort: null } })).rejects.toThrow('goalTracker names nobody, which is not a loaded provider'); + await expect(client.call('settings.set', { goalTracker: { providerId: 'echo', model: '', effort: null } })).rejects.toThrow('goalTracker must be null'); + const set = await client.call('settings.set', { goalTracker: { providerId: ' echo ', model: 'echo', effort: 'low' } }); + expect(set.goalTracker).toEqual({ providerId: 'echo', model: 'echo', effort: 'low' }); + restore = roles({ agent: () => ({ text: 'Done.\n[BOITE_GOAL_COMPLETE]' }), tracker: pass => pass === 'write' ? json({ items: [{ text: 'Do it', source: 'it' }] }) : json({ done: [1] }) }); + await client.call('threads.activity.set', { threadId, goal: { objective: 'do it', review: true } }); + await waitFor(() => goalOf(threadId).status === 'complete'); + const list = goalOf(threadId).checklist!; + expect(list).toMatchObject({ model: 'echo/echo', effort: 'low' }); + expect(h.core.threads.get(list.threadId!)).toMatchObject({ model: 'echo', effort: 'low' }); + + // A level the model does not offer stops the list, never the goal. + await client.call('settings.set', { goalTracker: { providerId: 'echo', model: 'echo', effort: 'medium' } }); + await client.call('threads.activity.set', { threadId, goal: { objective: 'do it again', review: true } }); + await waitFor(() => goalOf(threadId).status === 'complete'); + expect(goalOf(threadId).checklist).toMatchObject({ threadId: null, items: [], error: expect.stringContaining('reasoning effort') }); +}); + +test('a tracker that answers without its JSON leaves the goal running without a list', async () => { + const client = await h.connect(); + const { threadId } = await echoThread(h, client); + let writes = 0; + restore = roles({ agent: () => ({ text: 'Done.\n[BOITE_GOAL_COMPLETE]' }), tracker: () => { writes++; return 'I could not decide.'; } }); + await client.call('threads.activity.set', { threadId, goal: { objective: 'ship it', review: true } }); + await waitFor(() => goalOf(threadId).status === 'complete'); + const list = goalOf(threadId).checklist!; + expect(list).toMatchObject({ items: [], error: 'The tracker answered without a {"items": [...]} JSON block.' }); + // The first pass and at most two more when the turn waits; then the goal goes on alone. + expect(writes).toBeGreaterThanOrEqual(1); + expect(writes).toBeLessThanOrEqual(3); +}); + +test('a restarted core knows its trackers again and leaves no pass marked as running', async () => { + const client = await h.connect(); + const { threadId } = await echoThread(h, client); + const saved: ThreadActivity = { tasks: [], loop: null, goal: { objective: 'Ship it', status: 'active', iterations: 1, error: null, + review: { phase: 'working', maxRounds: 5, reviewerThreadId: null, claimTurnId: null, delivered: 0, verdicts: [] }, + checklist: { threadId: 'thr_tracker', phase: 'checking', items: [{ id: 1, text: 'Ship it', source: 'Ship it', status: 'open' }], model: 'echo/echo', effort: null, error: null } } }; + h.core.journal.setSetting(`activity:${threadId}`, saved); + const loaded = new ActivityStore(h.core); + try { + expect(loaded.ownsTracker('thr_tracker')).toBe(true); + expect(loaded.get(threadId).goal).toMatchObject({ status: 'paused', checklist: { phase: 'idle', items: [{ status: 'open' }] } }); + } finally { loaded.close(); } +}); + +test('an update keeps ids, never reuses one, and ignores ids it does not know', () => { + const items = [{ id: 1, text: 'A', source: 'a', status: 'done' as const }, { id: 2, text: 'B', source: 'b', status: 'open' as const }]; + expect(readUpdate(json({ add: [{ text: 'C', source: 'c' }, { text: ' ' }], cancel: [2, 9], change: [{ id: 1, text: 'A', source: '' }] }), items)).toEqual([ + { id: 1, text: 'A', source: 'a', status: 'done' }, + { id: 2, text: 'B', source: 'b', status: 'cancelled' }, + { id: 3, text: 'C', source: 'c', status: 'open', added: true }, + ]); + expect(readUpdate('no json', items)).toBe('The tracker answered without an {"add", "cancel", "change"} JSON block.'); + // The last block wins; a bare object at the end is read too. + expect(lastJson('```json\n{"done":[1]}\n```\nthen\n```json\n{"done":[2]}\n```')).toEqual({ done: [2] }); + expect(lastJson('Result: {"done": [3]} as asked')).toEqual({ done: [3] }); +}); + +test('a turn digest lists tool calls, marks failed ones and keeps the last answer without goal markers', () => { + const message = (parts: Message['parts']): Message => ({ id: 'm', threadId: 't', turnId: 'u', role: 'assistant', parts, state: 'complete', createdAt: 0 }); + const digest = turnDigest([ + message([{ type: 'tool', toolId: '1', name: 'apply_patch', input: { input: '*** Begin Patch\n*** Update File: a.ts\n*** Add File: b.ts\n' }, output: null, status: 'done' }]), + message([{ type: 'tool', toolId: '2', name: 'exec_command', input: { cmd: ['bun', 'test'] }, output: null, status: 'done', exitCode: 1 }, { type: 'text', text: 'Tests fail.\n[BOITE_GOAL_BLOCKED]' }]), + ]); + expect(digest).toEqual({ actions: ['apply_patch a.ts, b.ts', 'exec_command: bun test (failed)'], answer: 'Tests fail.' }); +}); + +test('the default checklist model is the newest Opus at medium on Claude and the newest Sol at high on Codex, never a small model', () => { + expect(defaultTrackerModel('claude-sdk', [{ id: 'claude-haiku-5-5' }, { id: 'claude-opus-4-8' }, { id: 'claude-opus-5-5' }, { id: 'claude-opus-5-5[1m]' }])).toEqual({ model: 'claude-opus-5-5', effort: 'medium' }); + expect(defaultTrackerModel('codex-appserver', [{ id: 'gpt-6-luna' }, { id: 'gpt-6-sol' }, { id: 'gpt-6.1-sol' }])).toEqual({ model: 'gpt-6.1-sol', effort: 'high' }); + expect(defaultTrackerModel('claude-sdk', [{ id: 'claude-haiku-5-5' }])).toBeNull(); + expect(defaultTrackerModel('echo', [{ id: 'echo' }])).toBeNull(); +}); + +/** The tracker's passes hang until `finish`; `stops` says whether a stop ends one. */ +function hangingTracker(stops: boolean) { + const pending: (() => void)[] = []; + const reset = setDriver('echo', { + protocol: 'echo', + startTurn(ctx) { + const id = ctx.emit.startMessage('assistant'); + const answer = (text: string) => { ctx.emit.part(id, 0, { type: 'text', text }); ctx.emit.complete(id, 'complete'); }; + if (h.core.activity.ownsTracker(ctx.thread.id)) { + let end!: () => void; + const done = new Promise<{ status: 'stopped'; sessionId: null; usage: null }>(resolve => { end = () => resolve({ status: 'stopped', sessionId: null, usage: null }); }); + pending.push(end); + return { stop() { if (stops) end(); }, done }; + } + answer(h.core.activity.ownsReviewer(ctx.thread.id) ? 'VERDICT: PASS' : 'Done.\n[BOITE_GOAL_COMPLETE]'); + return { stop() {}, done: Promise.resolve({ status: 'done', sessionId: null, usage: null }) }; + }, + }); + return { finish: () => { for (const end of pending.splice(0)) end(); }, reset }; +} + +test('a tracker stuck on a pass it will not stop gives the goal back after three tries', async () => { + const client = await h.connect(); + const { threadId } = await echoThread(h, client); + const saved = { ...TRACKER_PASS }; + Object.assign(TRACKER_PASS, { timeoutMs: 100, retryMs: 5 }); + const stuck = hangingTracker(false); + try { + await client.call('threads.activity.set', { threadId, goal: { objective: 'ship it', review: true } }); + await waitFor(() => goalOf(threadId).status === 'complete', 10_000); + expect(goalOf(threadId).checklist).toMatchObject({ items: [], phase: 'idle', error: 'The checklist could not be written: the checklist conversation is still busy with an earlier pass' }); + } finally { stuck.finish(); stuck.reset(); Object.assign(TRACKER_PASS, saved); } +}); + +test('finishing a goal by hand during a checklist pass leaves its list idle', async () => { + const client = await h.connect(); + const { threadId } = await echoThread(h, client); + const slow = hangingTracker(true); + try { + await client.call('threads.activity.set', { threadId, goal: { objective: 'ship it', review: true } }); + expect(goalOf(threadId).checklist!.phase).toBe('writing'); + const after = await client.call('threads.activity.control', { threadId, kind: 'goal', action: 'complete' }); + expect(after.goal!.checklist!.phase).toBe('idle'); + expect(goalOf(threadId).checklist!.phase).toBe('idle'); + } finally { slow.finish(); slow.reset(); } +}); + +test('a reviewer and a tracker read their agent\'s models again when an account change forgot them', async () => { + const client = await h.connect(); + const { threadId } = await echoThread(h, client); + // An agent whose models are known only from a read, as Claude's are. + const descriptor = h.core.providers.require('echo'); + descriptor.protocol = 'acp'; + let reads = 0; + const cache = new ModelProbes(async () => { reads++; return [{ id: 'discovered', name: 'Discovered' }]; }); + const reset = setDriver('acp', { + ...echoDriver, protocol: 'acp', + probe: ctx => cache.probe(ctx), + probedModels: (providerId, accountId) => cache.models(providerId, accountId), + forgetProbes: filter => cache.forget(filter), + startTurn(ctx) { + const id = ctx.emit.startMessage('assistant'); + const answer = (text: string) => { ctx.emit.part(id, 0, { type: 'text', text }); ctx.emit.complete(id, 'complete'); return { status: 'done' as const, sessionId: null, usage: null }; }; + if (h.core.activity.ownsTracker(ctx.thread.id)) return { stop() {}, done: Promise.resolve(answer(passOf(ctx.prompt) === 'write' ? json({ items: [{ text: 'Ship it', source: 'ship it' }] }) : json({ done: [1] }))) }; + if (h.core.activity.ownsReviewer(ctx.thread.id)) return { stop() {}, done: Promise.resolve(answer('VERDICT: PASS')) }; + // What an account update does in the middle of the goal, once its tracker is there. + const done = waitFor(() => !!goalOf(threadId).checklist?.threadId).then(() => { cache.forget({}); return answer('Shipped.\n[BOITE_GOAL_COMPLETE]'); }); + return { stop() {}, done }; + }, + }); + try { + await client.call('threads.update', { threadId, model: 'discovered' }); + expect(reads).toBe(1); + cache.forget({}); + await client.call('threads.activity.set', { threadId, goal: { objective: 'ship it', review: true } }); + await waitFor(() => { const goal = goalOf(threadId); return goal.status === 'complete' || goal.status === 'paused'; }); + const goal = goalOf(threadId); + expect(goal).toMatchObject({ status: 'complete', error: null, checklist: { error: null, items: [{ status: 'done' }] } }); + expect(h.core.threads.get(goal.checklist!.threadId!).model).toBe('discovered'); + expect(h.core.threads.get(goal.review!.verdicts[0]!.threadId).model).toBe('discovered'); + // One read for the thread, one before the tracker, one before the reviewer. + expect(reads).toBe(3); + } finally { reset(); descriptor.protocol = 'echo'; } +}); + +/** + * An agent whose models are known only from a read, per account, with one + * account's next read held until `release`. Its goal turns run `onClaim` + * (an account update, say) before claiming completion. + */ +function readAgent(onClaim: (prompt: string) => void) { + const descriptor = h.core.providers.require('echo'); + descriptor.protocol = 'acp'; + const reads: string[] = []; + let held: string | null = null; + const gate = Promise.withResolvers(); + const cache = new ModelProbes(async ctx => { + reads.push(ctx.accountId); + if (ctx.accountId === held) { held = null; await gate.promise; } + return [{ id: 'discovered', name: 'Discovered' }]; + }); + const reset = setDriver('acp', { + ...echoDriver, protocol: 'acp', + probe: ctx => cache.probe(ctx), + probedModels: (providerId, accountId) => cache.models(providerId, accountId), + forgetProbes: filter => cache.forget(filter), + startTurn(ctx) { + const id = ctx.emit.startMessage('assistant'); + let text = 'VERDICT: PASS'; + if (h.core.activity.ownsTracker(ctx.thread.id)) text = passOf(ctx.prompt) === 'write' ? json({ items: [{ text: 'Ship it', source: 'ship it' }] }) : json({ done: [1] }); + else if (!h.core.activity.ownsReviewer(ctx.thread.id)) { onClaim(ctx.prompt); text = 'Shipped.\n[BOITE_GOAL_COMPLETE]'; } + ctx.emit.part(id, 0, { type: 'text', text }); + ctx.emit.complete(id, 'complete'); + return { stop() {}, done: Promise.resolve({ status: 'done', sessionId: null, usage: null }) }; + }, + }); + return { cache, reads, hold: (accountId: string) => { held = accountId; }, release: () => gate.resolve(), reset: () => { gate.resolve(); reset(); descriptor.protocol = 'echo'; } }; +} + +const reviewerOn = (accountId: string) => ({ ...DEFAULT_DELEGATION_CONFIG, profiles: [{ id: 'reviewer', name: 'reviewer', providerId: 'echo', accountId, model: 'discovered', effort: null }] }); + +test('a reviewer on another account reads that account\'s models, not the conversation\'s', async () => { + const client = await h.connect(); + const { threadId, accountId } = await echoThread(h, client); + h.core.journal.putAccount({ ...h.core.accounts.require(accountId), id: 'second-account', label: 'Second' }); + const agent = readAgent(() => agent.cache.forget({ accountId: 'second-account' })); + try { + for (const account of [accountId, 'second-account']) await client.call('providers.probe', { providerId: 'echo', accountId: account }); + await client.call('delegation.configure', { threadId, config: reviewerOn('second-account') }); + await client.call('threads.activity.set', { threadId, goal: { objective: 'ship it', review: true } }); + await waitFor(() => { const goal = goalOf(threadId); return goal.status === 'complete' || goal.status === 'paused'; }); + const goal = goalOf(threadId); + expect(goal).toMatchObject({ status: 'complete', error: null }); + expect(h.core.threads.get(goal.review!.verdicts[0]!.threadId)).toMatchObject({ accountId: 'second-account', model: 'discovered' }); + // The conversation's own list was still there; the reviewer's account was read again. + expect(agent.reads).toEqual([accountId, 'second-account', 'second-account']); + } finally { agent.reset(); } +}); + +test('a read that ends after its goal was replaced lets the new goal read its own reviewer\'s models', async () => { + const client = await h.connect(); + const { threadId, accountId } = await echoThread(h, client); + for (const id of ['second-account', 'third-account']) h.core.journal.putAccount({ ...h.core.accounts.require(accountId), id, label: id }); + const agent = readAgent(prompt => agent.cache.forget({ accountId: prompt.includes('first') ? 'second-account' : 'third-account' })); + try { + for (const account of [accountId, 'second-account', 'third-account']) await client.call('providers.probe', { providerId: 'echo', accountId: account }); + await client.call('delegation.configure', { threadId, config: reviewerOn('second-account') }); + agent.hold('second-account'); + await client.call('threads.activity.set', { threadId, goal: { objective: 'first goal', review: true } }); + await waitFor(() => agent.reads.length === 4); + // While the first reviewer's read hangs, a new goal with a reviewer on another account claims completion. + await client.call('delegation.configure', { threadId, config: reviewerOn('third-account') }); + await client.call('threads.activity.set', { threadId, goal: { objective: 'second goal', review: true } }); + await waitFor(() => { const goal = goalOf(threadId); return goal.objective === 'second goal' && goal.review!.phase === 'reviewing' && h.core.threads.get(threadId).status === 'idle'; }); + agent.release(); + await waitFor(() => { const goal = goalOf(threadId); return goal.status === 'complete' || goal.status === 'paused'; }); + const goal = goalOf(threadId); + expect(goal).toMatchObject({ objective: 'second goal', status: 'complete', error: null }); + expect(h.core.threads.get(goal.review!.verdicts[0]!.threadId)).toMatchObject({ accountId: 'third-account', model: 'discovered' }); + expect(agent.reads.at(-1)).toBe('third-account'); + } finally { agent.reset(); } +}); diff --git a/packages/ui/src/components/ExperimentsPage.svelte b/packages/ui/src/components/ExperimentsPage.svelte index 1a6dc1642..c802b3a03 100644 --- a/packages/ui/src/components/ExperimentsPage.svelte +++ b/packages/ui/src/components/ExperimentsPage.svelte @@ -1,6 +1,7 @@ + +{#snippet row(item: GoalChecklistItem)} + {@const state = item.status === 'cancelled' ? 'cancelled' : item.status === 'done' ? 'done' : held(item) ? 'held' : item.id === next ? 'next' : 'open'} + {@const label = state === 'done' ? strings.activity.checklistDone : state === 'cancelled' ? strings.activity.checklistDropped : state === 'held' ? strings.activity.checklistHeld : state === 'next' ? strings.activity.checklistNext : strings.activity.checklistOpen} +
  • + + {#if shown === item.id && item.source} +

    {fill(strings.activity.checklistSource, { source: `"${item.source}"` })}

    + {/if} +
    • +{/snippet} + +
      + {#each listed as item (item.id)}{@render row(item)}{/each} +
    +{#if checklist.error}

    {checklist.error}

    {/if} +{#if dropped.length || checklist.threadId} + +
    + {#if !checklist.items.length && checklist.phase === 'writing'} + {strings.activity.checklistWriting} + {:else if dropped.length} + + {/if} + {#if checklist.threadId} + {@const tracker = checklist.threadId} + + {/if} +
    +{/if} +{#if dropped.length && showDropped} +
      + {#each dropped as item (item.id)}{@render row(item)}{/each} +
    +{/if} + + diff --git a/packages/ui/src/components/GoalChecklistLoader.svelte b/packages/ui/src/components/GoalChecklistLoader.svelte new file mode 100644 index 000000000..0cbe3cc53 --- /dev/null +++ b/packages/ui/src/components/GoalChecklistLoader.svelte @@ -0,0 +1,26 @@ + + +{#await loading then module} + +{:catch} +
    + {strings.activity.checklistLoadFailed} + +
    +{/await} + + diff --git a/packages/ui/src/components/GoalTrackerSetting.svelte b/packages/ui/src/components/GoalTrackerSetting.svelte new file mode 100644 index 000000000..81407899b --- /dev/null +++ b/packages/ui/src/components/GoalTrackerSetting.svelte @@ -0,0 +1,104 @@ + + +
    + + {strings.experiments.goal2.trackerModel} + + + + {current} + + {#if efforts.length > 0} + + {effort} + + {/if} + +
    + + diff --git a/packages/ui/src/components/GoalTrackerSetting.svelte.test.ts b/packages/ui/src/components/GoalTrackerSetting.svelte.test.ts new file mode 100644 index 000000000..79c295c7b --- /dev/null +++ b/packages/ui/src/components/GoalTrackerSetting.svelte.test.ts @@ -0,0 +1,100 @@ +import { afterEach, expect, test, vi } from 'vitest'; +import { flushSync, mount, unmount } from 'svelte'; +import type { GoalTrackerModel, ModelInfo, Settings } from '@boite/contracts'; +import type { Store } from '../lib/store.svelte'; +import GoalTrackerSetting from './GoalTrackerSetting.svelte'; + +let mounted: ReturnType | undefined; +afterEach(async () => { + if (mounted) await unmount(mounted); + mounted = undefined; + document.body.innerHTML = ''; +}); + +const settle = async () => { for (let i = 0; i < 20; i++) { await Promise.resolve(); flushSync(); } }; +const levels = (...ids: string[]) => ids.map((id) => ({ id, label: id[0]!.toUpperCase() + id.slice(1) })); +const provider = (id: string, protocol: string, shortName: string, available = true) => ({ id, shortName, name: shortName, protocol, available, models: [] }); +const MODELS: Record = { + claude: [ + { id: 'default', name: 'Default' }, + { id: 'claude-haiku-5-5', name: 'Haiku 5.5', effort: { levels: levels('low', 'medium', 'high'), default: 'high' } }, + { id: 'claude-opus-5-5', name: 'Opus 5.5', effort: { levels: levels('low', 'medium', 'high', 'max'), default: 'high' } }, + ], + codex: [ + { id: 'gpt-6-luna', name: 'GPT-6-Luna', effort: { levels: levels('low', 'medium'), default: 'medium' } }, + { id: 'gpt-6.1-sol', name: 'GPT-6.1-Sol', effort: { levels: levels('low', 'medium', 'high'), default: 'low' } }, + ], + pi: [{ id: 'pi-model', name: 'Pi model' }], +}; + +function render(goalTracker: GoalTrackerModel | null) { + const store = $state({ + settings: { goalTracker } as Partial, + providers: [provider('claude', 'claude-sdk', 'Claude'), provider('codex', 'codex-appserver', 'Codex'), provider('pi', 'pi-rpc', 'Pi', false)], + accounts: [{ id: 'claude-main', providerId: 'claude', status: 'ok' }, { id: 'codex-main', providerId: 'codex', status: 'ok' }, { id: 'pi-main', providerId: 'pi', status: 'ok' }], + accountsOf: (providerId: string) => store.accounts.filter((account) => account.providerId === providerId), + modelsOf: (providerId: string) => MODELS[providerId] ?? [], + saveSettings: vi.fn(async (patch: Partial) => { + store.settings = { ...store.settings, ...patch }; + return true; + }), + }); + mounted = mount(GoalTrackerSetting, { target: document.body, props: { store: store as unknown as Store } }); + return store; +} + +const trigger = (testid: string) => document.querySelector(`[data-testid="${testid}"]`); +const rows = () => [...document.querySelectorAll('[data-testid="goal2-tracker-model-menu"] > *')].map((row) => + row.classList.contains('separator') ? '---' : `${row.querySelector('.label')!.textContent!.trim()} | ${row.querySelector('.hint')?.textContent ?? ''}`); + +test('the checklist model starts on Automatic, which suggests Opus and Sol and never a small model', async () => { + const store = render(null); + await settle(); + expect(trigger('goal2-tracker-model')!.textContent).toContain('Automatic'); + // Automatic carries its own level: no reasoning menu until a model is picked. + expect(trigger('goal2-tracker-effort')).toBeNull(); + + trigger('goal2-tracker-model')!.click(); + await settle(); + // Pi is not installed; Haiku and Luna are offered, never suggested. + expect(rows()).toEqual([ + 'Automatic | Opus medium or Sol high', + '---', + 'Opus 5.5 | Claude · Automatic', + 'Haiku 5.5 | Claude', + '---', + 'GPT-6.1-Sol | Codex · Automatic', + 'GPT-6-Luna | Codex', + ]); + + document.querySelector('[data-value=\'["codex","gpt-6.1-sol"]\']')!.click(); + await settle(); + // A model of a suggested family starts on that family's level, not on the model's own default. + expect(store.saveSettings).toHaveBeenLastCalledWith({ goalTracker: { providerId: 'codex', model: 'gpt-6.1-sol', effort: 'high' } }); + expect(trigger('goal2-tracker-model')!.textContent).toContain('GPT-6.1-Sol'); + expect(trigger('goal2-tracker-effort')!.textContent).toContain('High'); +}); + +test('a small model is the user\'s own pick, on its own default level, and its level can change', async () => { + const store = render(null); + await settle(); + trigger('goal2-tracker-model')!.click(); + await settle(); + document.querySelector('[data-value=\'["claude","claude-haiku-5-5"]\']')!.click(); + await settle(); + expect(store.saveSettings).toHaveBeenLastCalledWith({ goalTracker: { providerId: 'claude', model: 'claude-haiku-5-5', effort: 'high' } }); + + trigger('goal2-tracker-effort')!.click(); + await settle(); + expect([...document.querySelectorAll('[data-testid="goal2-tracker-effort-menu"] .label')].map((label) => label.textContent?.trim())).toEqual(['Low', 'Medium', 'High']); + document.querySelector('[data-testid="goal2-tracker-effort-menu"] [data-value="medium"]')!.click(); + await settle(); + expect(store.saveSettings).toHaveBeenLastCalledWith({ goalTracker: { providerId: 'claude', model: 'claude-haiku-5-5', effort: 'medium' } }); + + trigger('goal2-tracker-model')!.click(); + await settle(); + document.querySelector('[data-value="auto"]')!.click(); + await settle(); + expect(store.saveSettings).toHaveBeenLastCalledWith({ goalTracker: null }); + expect(trigger('goal2-tracker-model')!.textContent).toContain('Automatic'); +}); diff --git a/packages/ui/src/components/TasksSurface.svelte b/packages/ui/src/components/TasksSurface.svelte index 7baf04f36..5ca5c3b52 100644 --- a/packages/ui/src/components/TasksSurface.svelte +++ b/packages/ui/src/components/TasksSurface.svelte @@ -3,6 +3,7 @@ import type { Todo } from '@boite/contracts'; import { watchStatus } from '../lib/pull-request-watch'; import InfoTip from './InfoTip.svelte'; + import GoalChecklist from './GoalChecklistLoader.svelte'; import { time } from '../lib/format'; import { fill, strings } from '../lib/strings'; import type { Store } from '../lib/store.svelte'; @@ -11,12 +12,22 @@ const sectionId = $props.id(); let draft = $state(''); - let open = $state({ activity: true, watches: true, tasks: true, todos: true }); + let open = $state({ activity: true, reviews: true, watches: true, tasks: true, todos: true }); + /** A long objective shows three lines until pressed, so the checklist under it stays in view. */ + let wholeObjective = $state(false); let thread = $derived(store.openThread); let activity = $derived(thread?.activity ?? null); let tasks = $derived(activity?.tasks ?? []); let watches = $derived(thread?.pullRequestWatches ?? []); + /** A `/goal2`'s checklist, from the first list its tracker writes, as the goal bar's button that leads here. Dropped items count for nothing. */ + let checklist = $derived(activity?.goal?.checklist && (activity.goal.checklist.items.length > 0 || activity.goal.checklist.phase !== 'idle' || activity.goal.checklist.error) ? activity.goal.checklist : null); + let listed = $derived(checklist?.items.filter((item) => item.status !== 'cancelled') ?? []); + let ticked = $derived(listed.filter((item) => item.status === 'done').length); + /** The section says what it holds: a goal, a loop, or both. */ + let activityLabel = $derived(activity?.goal && !activity.loop ? strings.activity.goal : activity?.loop && !activity.goal ? strings.activity.loop : strings.tasks.goalSection); + /** A `/goal2`'s reviews, newest first, with what each reviewer found: the bar keeps only their dots. */ + let verdicts = $derived([...(activity?.goal?.review?.verdicts ?? [])].reverse()); let todos = $derived(thread?.projectId ? (store.todos[thread.projectId] ?? []) : []); // The project's list is the same for every thread of it, so it is read once @@ -53,32 +64,55 @@ onclick={() => (open.activity = !open.activity)} > - {strings.tasks.goalSection} + {activityLabel}
    {#if activity?.goal} -
    + {@const goal = activity.goal} +
    - {activity.goal.objective} - {fill(strings.tasks.iterations, { count: String(activity.goal.iterations) })} - {#if activity.goal.status === 'paused' && activity.goal.blocked} - {strings.activity.blocked} - {:else} - {strings.activity[activity.goal.status]} - {/if} +
    + + + {#if goal.status === 'paused' && goal.blocked} + {strings.activity.blocked} + {:else} + {strings.activity[goal.status]} + {/if} + {fill(strings.tasks.iterations, { count: String(goal.iterations) })} + +
    + {#if checklist} + +
    + {#if listed.length > 0} +
    + + {fill(strings.activity.checklistCount, { done: String(ticked), total: String(listed.length) })} +
    + {/if} + +
    + {/if} {/if} {#if activity?.loop} -
    +
    - {activity.loop.prompt} - {#if activity.loop.nextRunAt} - {fill(strings.tasks.nextRun, { time: time(activity.loop.nextRunAt) })} - {:else} - {fill(strings.tasks.iterations, { count: String(activity.loop.iterations) })} - {/if} - {strings.activity[activity.loop.status]} +
    + {activity.loop.prompt} + + {strings.activity[activity.loop.status]} + {#if activity.loop.nextRunAt} + {fill(strings.tasks.nextRun, { time: time(activity.loop.nextRunAt) })} + {:else} + {fill(strings.tasks.iterations, { count: String(activity.loop.iterations) })} + {/if} + +
    {/if} {#if !activity?.goal && !activity?.loop} @@ -88,6 +122,37 @@
    + {#if verdicts.length > 0} +
    + +
    +
    + {#each verdicts as verdict (verdict.threadId)} +
    +
    + {fill(strings.activity.reviewRound, { round: String(verdict.round) })} + {strings.activity[verdict.verdict]} + +
    + {#if verdict.summary}

    {verdict.summary}

    {/if} +
    + {/each} +
    +
    +
    + {/if} + {#if watches.length > 0}
    @@ -129,6 +194,7 @@
    {/if} + {#if tasks.length > 0}
    @@ -165,12 +229,11 @@ {task.text}
    - {:else} -

    {strings.tasks.noTasks}

    {/each}
    + {/if}