Repository navigation
fix(scout): a first reply answers what was asked in two or three sentences, then one next step #3317
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
fix(scout): a first reply answers what was asked in two or three sentences, then one next step #3317
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
33 changes: 33 additions & 0 deletions
33
backend/__tests__/unit/config/nativeAgents.scoutReplyLength.test.js
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| /** | ||
| * Scout's first replies stay short (F60, staging walk 2026-11-02). | ||
| * | ||
| * Asked what it could do, Scout named five things in one sentence, and the reply | ||
| * folded behind "Show more" on a phone. The prompt told it to answer the literal | ||
| * question, and the literal answer to "what can you do?" is the whole catalogue. | ||
| * Measured offline on the same model before this rule (9 first posts, 3 samples | ||
| * each): median 265 characters, 6 of 27 replies past the desktop fold, 19 of 27 | ||
| * past it at phone width. With it: 181, 0 of 27, 4 of 27. | ||
| */ | ||
| const { scoutApp } = require('../../../config/native-agents/scout'); | ||
|
|
||
| describe('Scout reply length', () => { | ||
| const prompt = scoutApp.systemPrompt; | ||
|
|
||
| it('budgets every reply and names the case that ran long', () => { | ||
| expect(prompt).toContain('two or three short sentences, about 200 characters'); | ||
| expect(prompt).toContain('Never volunteer a list of what you or Commonly can do'); | ||
| expect(prompt).toContain('name the one thing most useful to them now'); | ||
| }); | ||
|
|
||
| it('exempts what the person asked for and every caveat the rules require', () => { | ||
| expect(prompt).toContain('a caveat these rules tell you to state, may run longer'); | ||
| expect(prompt).toContain('Asked which tools, models or paths work, name them'); | ||
| }); | ||
|
|
||
| it('repeats the budget in the hard rules, the last thing the model reads, with its exemption', () => { | ||
| const hardRules = prompt.slice(prompt.lastIndexOf('ROLE-SPECIFIC HARD RULES:')); | ||
| expect(hardRules).toContain('A first impression is also short'); | ||
| expect(hardRules).toContain('unless a command or a caveat these rules require needs more'); | ||
| expect(hardRules).toContain('never a tour of what you can do'); | ||
| }); | ||
| }); | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.