Repository navigation
feat(agents): deterministic connect_local_agent card on connect intent (TASK-417) #2848
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
f6c854e
feat(agents): deterministic connect_local_agent card on connect inten…
lilyshen0722 0f21115
fix(agents): connect floor stays out of a turn that already replied (…
lilyshen0722 47f4238
fix(agents): tighten the connect-intent rule to tool names (TASK-417)
lilyshen0722 dd4fff3
fix(agents): drop bare claude from the connect-intent rule (TASK-417)
lilyshen0722 7bb9511
fix(agents): connect-intent rule needs a determiner before the tool n…
lilyshen0722 c894b38
fix(agents): the connect floor takes the offer the model got (TASK-417)
lilyshen0722 13646fd
test(agents): drive the connect floor through runAgent (TASK-417)
lilyshen0722 a3aace4
fix(native): keep the connect floor out of guardrail-blocked turns
lilyshen0722 fd63172
fix(agents): the connect floor honours a broker-tainted bridged refus…
lilyshen0722 ceacb8f
fix(agents): read the bridge for a broker call with no web tool (TASK…
lilyshen0722 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
209 changes: 209 additions & 0 deletions
209
backend/__tests__/unit/services/nativeRuntimeService.connectFloor.runAgent.test.js
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,209 @@ | ||
| /** | ||
| * TASK-417 — the connect-intent floor, driven through runAgent. The unit file | ||
| * pins the rule and the helper; this one pins the turn: the floor fires only | ||
| * when the model replied with nothing and the install offers propose_action. | ||
| */ | ||
| jest.mock('axios', () => ({ | ||
| __esModule: true, | ||
| default: { post: jest.fn() }, | ||
| })); | ||
|
|
||
| jest.mock('../../../models/Pod', () => ({ | ||
| findById: jest.fn(() => ({ | ||
| lean: jest.fn().mockResolvedValue({ name: 'Launch Room' }), | ||
| select: () => ({ lean: async () => ({ members: ['seat-1'] }) }), | ||
| })), | ||
| })); | ||
|
|
||
| jest.mock('../../../models/AgentRun', () => ({ create: jest.fn() })); | ||
|
|
||
| jest.mock('../../../models/User', () => ({ | ||
| findById: jest.fn(() => ({ select: () => ({ lean: async () => ({ isBot: false }) }) })), | ||
| findOne: jest.fn(() => ({ select: () => ({ lean: async () => ({ _id: 'seat-1' }) }) })), | ||
| })); | ||
|
|
||
| jest.mock('../../../services/grantBrokerProjectionService', () => ({ | ||
| hostedBrokerToolsForRun: jest.fn(), | ||
| dispatchHostedBrokerTool: jest.fn(), | ||
| })); | ||
|
|
||
| jest.mock('../../../models/ApprovalAction', () => ({ exists: jest.fn() })); | ||
|
|
||
| jest.mock('../../../services/approvalActionService', () => ({ | ||
| proposeAction: jest.fn(), | ||
| suggestLocalAgentName: jest.fn(), | ||
| })); | ||
|
|
||
| jest.mock('../../../services/nativeOutsiderGrantPolicy', () => ({ | ||
| podHasOutsiderWritableGrant: jest.fn(), | ||
| })); | ||
|
|
||
| jest.mock('../../../services/agentTypingService', () => ({ | ||
| emitAgentTypingStart: jest.fn(), | ||
| emitAgentTypingStop: jest.fn(), | ||
| })); | ||
|
|
||
| const mockBridged = jest.fn(); | ||
| jest.mock('../../../services/outsiderGrantPodEligibility', () => ({ | ||
| podIsBridged: (...args) => mockBridged(...args), | ||
| })); | ||
|
|
||
| const mockSearchWeb = jest.fn(); | ||
| jest.mock('../../../services/webSearchService', () => ({ | ||
| isSearchConfigured: () => true, | ||
| searchWeb: (...args) => mockSearchWeb(...args), | ||
| })); | ||
|
|
||
| jest.mock('../../../services/messageClaimService', () => ({ | ||
| claim: jest.fn().mockResolvedValue({ claimed: true }), | ||
| release: jest.fn().mockResolvedValue(undefined), | ||
| })); | ||
|
|
||
| jest.mock('../../../services/agentMessageService', () => ({ | ||
| postMessage: jest.fn().mockResolvedValue({ success: true }), | ||
| })); | ||
|
|
||
| const axios = require('axios').default; | ||
| const AgentRun = require('../../../models/AgentRun'); | ||
| const ApprovalAction = require('../../../models/ApprovalAction'); | ||
| const AgentMessageService = require('../../../services/agentMessageService'); | ||
| const MessageClaimService = require('../../../services/messageClaimService'); | ||
| const { proposeAction, suggestLocalAgentName } = require('../../../services/approvalActionService'); | ||
| const { podHasOutsiderWritableGrant } = require('../../../services/nativeOutsiderGrantPolicy'); | ||
| const brokerProjection = require('../../../services/grantBrokerProjectionService'); | ||
| const { runAgent } = require('../../../services/nativeRuntimeService'); | ||
|
|
||
| const SCOUT = (tools) => ({ | ||
| podId: '507f1f77bcf86cd799439011', | ||
| agentName: 'scout', | ||
| instanceId: 'default', | ||
| displayName: 'Scout', | ||
| config: { runtime: { runtimeType: 'native' }, ...(tools ? { tools } : {}) }, | ||
| }); | ||
|
|
||
| const call = (id, name, args) => ({ id, type: 'function', function: { name, arguments: JSON.stringify(args) } }); | ||
| const toolCalls = (...calls) => ({ | ||
| status: 200, | ||
| headers: {}, | ||
| data: { | ||
| choices: [{ message: { role: 'assistant', content: null, tool_calls: calls } }], | ||
| usage: { prompt_tokens: 10, completion_tokens: 8, total_tokens: 18 }, | ||
| }, | ||
| }); | ||
|
|
||
| const CONNECT_MESSAGE = { | ||
| type: 'message.posted', | ||
| eventId: '507f1f77bcf86cd799439012', | ||
| payload: { messageId: 'm1', userId: 'u1', content: 'connect my Claude Code to this pod' }, | ||
| }; | ||
|
|
||
| // The model's reply: a text answer, or no content and no tool call at all. | ||
| const modelSays = (content) => ({ | ||
| status: 200, | ||
| data: { | ||
| choices: [{ message: { role: 'assistant', content } }], | ||
| usage: { prompt_tokens: 10, completion_tokens: 4, total_tokens: 14 }, | ||
| }, | ||
| headers: {}, | ||
| }); | ||
|
|
||
| describe('connect-intent floor through runAgent', () => { | ||
| beforeEach(() => { | ||
| jest.clearAllMocks(); | ||
| process.env.LITELLM_BASE_URL = 'http://litellm.test'; | ||
| process.env.LITELLM_MASTER_KEY = 'test-key'; | ||
| AgentRun.create.mockImplementation(async (doc) => ({ | ||
| _id: 'run-1', ...doc, save: jest.fn().mockResolvedValue(undefined), | ||
| })); | ||
| ApprovalAction.exists.mockResolvedValue(null); | ||
| suggestLocalAgentName.mockResolvedValue('sam-claude'); | ||
| proposeAction.mockResolvedValue({ ok: true }); | ||
| podHasOutsiderWritableGrant.mockResolvedValue(false); | ||
| brokerProjection.hostedBrokerToolsForRun.mockResolvedValue({ tools: [], dispatch: new Map() }); | ||
| brokerProjection.dispatchHostedBrokerTool.mockResolvedValue({ callId: 'b1', outcome: 'ok' }); | ||
| }); | ||
|
|
||
| test('proposes the card when the model replies with nothing', async () => { | ||
| axios.post.mockResolvedValue(modelSays('')); | ||
|
|
||
| await runAgent(SCOUT(['commonly_propose_action']), CONNECT_MESSAGE); | ||
|
|
||
| expect(proposeAction).toHaveBeenCalledTimes(1); | ||
| expect(proposeAction.mock.calls[0][0]).toEqual(expect.objectContaining({ | ||
| actionType: 'connect_local_agent', | ||
| params: { name: 'sam-claude', adapter: 'claude' }, | ||
| })); | ||
| }); | ||
|
|
||
| test('stays out when the model already replied in text', async () => { | ||
| axios.post.mockResolvedValue(modelSays('Open the connect page to link it.')); | ||
|
|
||
| await runAgent(SCOUT(['commonly_propose_action']), CONNECT_MESSAGE); | ||
|
|
||
| expect(proposeAction).not.toHaveBeenCalled(); | ||
| }); | ||
|
|
||
| test('stays out when the install does not declare propose_action', async () => { | ||
| axios.post.mockResolvedValue(modelSays('')); | ||
|
|
||
| await runAgent(SCOUT(['commonly_post_message']), CONNECT_MESSAGE); | ||
|
|
||
| expect(proposeAction).not.toHaveBeenCalled(); | ||
| }); | ||
|
|
||
| test('a guardrail-blocked turn gets no connect card', async () => { | ||
| mockSearchWeb.mockResolvedValue({ provider: 'exa', results: [{ title: 'x', url: 'https://example.com', text: 'x' }] }); | ||
| mockBridged.mockResolvedValueOnce(false).mockResolvedValueOnce(false).mockResolvedValue(true); | ||
| axios.post | ||
| .mockResolvedValueOnce(toolCalls(call('s1', 'commonly_web_search', { query: 'connect' }))) | ||
| .mockResolvedValueOnce(modelSays('Open the connect page to link it.')); | ||
|
|
||
| await runAgent(SCOUT(['commonly_propose_action', 'commonly_web_search']), CONNECT_MESSAGE); | ||
|
|
||
| expect(AgentMessageService.postMessage).not.toHaveBeenCalled(); | ||
| expect(proposeAction).not.toHaveBeenCalled(); | ||
| }); | ||
|
|
||
| test('a broker-tainted turn in a bridged pod gets no connect card', async () => { | ||
| // A granted broker tool ran this turn and the pod is bridged. dispatchTool | ||
| // refuses commonly_propose_action here, so the floor must not post it. | ||
| mockBridged.mockResolvedValue(true); | ||
| axios.post | ||
| .mockResolvedValueOnce(toolCalls(call('b1', 'commonly_brokered_lookup', {}))) | ||
| .mockResolvedValueOnce(modelSays('')); | ||
|
|
||
| // No web tool declared: the floor must read the bridge because of the broker call alone. | ||
| await runAgent(SCOUT(['commonly_propose_action']), CONNECT_MESSAGE); | ||
|
|
||
| expect(brokerProjection.dispatchHostedBrokerTool).toHaveBeenCalled(); | ||
| expect(proposeAction).not.toHaveBeenCalled(); | ||
| expect(AgentMessageService.postMessage).not.toHaveBeenCalled(); | ||
| }); | ||
|
|
||
| test('a human connect turn that posts the card completes its claim', async () => { | ||
| const humanConnect = { | ||
| ...CONNECT_MESSAGE, | ||
| payload: { ...CONNECT_MESSAGE.payload, senderIsHuman: true }, | ||
| }; | ||
| axios.post.mockResolvedValue(modelSays('')); | ||
|
|
||
| const result = await runAgent(SCOUT(['commonly_propose_action']), humanConnect); | ||
|
|
||
| expect(result.status).toBe('succeeded'); | ||
| expect(proposeAction).toHaveBeenCalledTimes(1); | ||
| expect(MessageClaimService.release).toHaveBeenCalledWith(expect.objectContaining({ | ||
| messageId: 'm1', | ||
| outcome: 'completed', | ||
| })); | ||
| }); | ||
|
|
||
| test('stays out in an outsider-granted pod, where only in-pod tools are offered', async () => { | ||
| axios.post.mockResolvedValue(modelSays('')); | ||
| podHasOutsiderWritableGrant.mockResolvedValue(true); | ||
|
|
||
| await runAgent(SCOUT(['commonly_propose_action']), CONNECT_MESSAGE); | ||
|
|
||
| expect(proposeAction).not.toHaveBeenCalled(); | ||
| expect(AgentMessageService.postMessage).not.toHaveBeenCalled(); | ||
| }); | ||
| }); |
172 changes: 172 additions & 0 deletions
172
backend/__tests__/unit/services/nativeRuntimeService.connectFloor.test.js
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,172 @@ | ||
| /** | ||
| * TASK-417 / ADR-020 — the connect-intent floor. A connect-intent message from | ||
| * a person gets a connect_local_agent card even when Scout's model proposes | ||
| * nothing that turn. The floor only proposes: the owner still approves. | ||
| */ | ||
| jest.mock('../../../models/User', () => ({ findById: jest.fn() })); | ||
| jest.mock('../../../models/ApprovalAction', () => ({ exists: jest.fn() })); | ||
| jest.mock('../../../services/approvalActionService', () => ({ | ||
| proposeAction: jest.fn(), | ||
| suggestLocalAgentName: jest.fn(), | ||
| })); | ||
|
|
||
| const User = require('../../../models/User'); | ||
| const ApprovalAction = require('../../../models/ApprovalAction'); | ||
| const { proposeAction, suggestLocalAgentName } = require('../../../services/approvalActionService'); | ||
| const { | ||
| isConnectIntent, connectIntentFloor, connectRequestAnswered, toolsForConfig, | ||
| } = require('../../../services/nativeRuntimeService'); | ||
|
|
||
| const humanSender = (isBot = false) => User.findById.mockImplementation(() => ({ | ||
| select: () => ({ lean: async () => ({ isBot }) }), | ||
| })); | ||
|
|
||
| const trigger = (content, overrides = {}) => ({ | ||
| type: 'message.posted', | ||
| payload: { | ||
| messageId: 'm1', | ||
| userId: 'u1', | ||
| content, | ||
| ...overrides, | ||
| }, | ||
| }); | ||
|
|
||
| const floorInput = (content, overrides = {}) => ({ | ||
| trigger: trigger(content), | ||
| podId: 'pod1', | ||
| agentName: 'scout', | ||
| instanceId: 'default', | ||
| displayName: 'Scout', | ||
| runId: 'run1', | ||
| installationConfig: {}, | ||
| noOutboundRelay: false, | ||
| ...overrides, | ||
| }); | ||
|
|
||
| beforeEach(() => { | ||
| jest.clearAllMocks(); | ||
| humanSender(false); | ||
| ApprovalAction.exists.mockResolvedValue(null); | ||
| suggestLocalAgentName.mockResolvedValue('sam-claude'); | ||
| proposeAction.mockResolvedValue({ ok: true }); | ||
| }); | ||
|
|
||
| describe('isConnectIntent', () => { | ||
| it.each([ | ||
| 'connect my Claude Code to Commonly', | ||
| 'can you link my laptop so my codex runs here?', | ||
| 'set up a local agent for me', | ||
| 'connect a codex to this pod', | ||
| ])('matches a connect request: %s', (text) => { | ||
| expect(isConnectIntent(text)).toBe(true); | ||
| }); | ||
|
|
||
| it.each([ | ||
| 'what can you do?', | ||
| 'create a pod for the team', | ||
| 'connect with the team on Friday', // no machine/tool named | ||
| 'attach a photo from my laptop', | ||
| 'link: https://claude.ai/chat/abc', | ||
| 'connect to claude.ai for research', | ||
| 'connect to the Claude API', | ||
| 'link to the Claude docs', | ||
| 'connect to the Codex API', | ||
| 'link to the codex repo', | ||
| ])('does not match: %s', (text) => { | ||
| expect(isConnectIntent(text)).toBe(false); | ||
| }); | ||
| }); | ||
|
|
||
| describe('connectIntentFloor', () => { | ||
| it('proposes a connect_local_agent card for a human connect request', async () => { | ||
| const posted = await connectIntentFloor(floorInput('connect my Claude Code to Commonly')); | ||
|
|
||
| expect(posted).toBe(true); | ||
| expect(proposeAction).toHaveBeenCalledTimes(1); | ||
| expect(proposeAction.mock.calls[0][0]).toEqual(expect.objectContaining({ | ||
| actionType: 'connect_local_agent', | ||
| params: { name: 'sam-claude', adapter: 'claude' }, | ||
| podId: 'pod1', | ||
| })); | ||
| }); | ||
|
|
||
| it('picks codex when the person names it', async () => { | ||
| await connectIntentFloor(floorInput('link my codex to this pod')); | ||
|
|
||
| expect(suggestLocalAgentName).toHaveBeenCalledWith('codex', 'u1'); | ||
| expect(proposeAction.mock.calls[0][0].params.adapter).toBe('codex'); | ||
| }); | ||
|
|
||
| it('does nothing when another agent sent the message', async () => { | ||
| humanSender(true); | ||
|
|
||
| expect(await connectIntentFloor(floorInput('connect my Claude Code'))).toBe(false); | ||
| expect(proposeAction).not.toHaveBeenCalled(); | ||
| }); | ||
|
|
||
| it('does nothing for a non-Scout agent', async () => { | ||
| expect(await connectIntentFloor(floorInput('connect my Claude Code', { agentName: 'sage' }))) | ||
| .toBe(false); | ||
| expect(proposeAction).not.toHaveBeenCalled(); | ||
| }); | ||
|
|
||
| it('does nothing when the message has no connect intent', async () => { | ||
| expect(await connectIntentFloor(floorInput('what can you do?'))).toBe(false); | ||
| expect(proposeAction).not.toHaveBeenCalled(); | ||
| }); | ||
|
|
||
| it('does not stack a second card while one is pending in the pod', async () => { | ||
| ApprovalAction.exists.mockResolvedValue({ _id: 'existing' }); | ||
|
|
||
| expect(await connectIntentFloor(floorInput('connect my Claude Code'))).toBe(false); | ||
| expect(proposeAction).not.toHaveBeenCalled(); | ||
| }); | ||
|
|
||
| it('reports a refused proposal as not posted', async () => { | ||
| proposeAction.mockResolvedValue({ ok: false, error: 'no owner' }); | ||
|
|
||
| expect(await connectIntentFloor(floorInput('connect my Claude Code'))).toBe(false); | ||
| }); | ||
| }); | ||
|
|
||
| describe('connectRequestAnswered', () => { | ||
| it('is answered when Scout posted a reply without proposing a card', () => { | ||
| expect(connectRequestAnswered(true, 'Here is how to connect your Claude Code.')).toBe(true); | ||
| }); | ||
|
|
||
| it('is answered by a proposed card alone', () => { | ||
| expect(connectRequestAnswered(true, undefined)).toBe(true); | ||
| }); | ||
|
|
||
| it('is answered by the text fallback alone', () => { | ||
| expect(connectRequestAnswered(false, 'Sure, open the connect page.')).toBe(true); | ||
| }); | ||
|
|
||
| it('is open only when the turn posted nothing', () => { | ||
| expect(connectRequestAnswered(false, undefined)).toBe(false); | ||
| }); | ||
| }); | ||
|
|
||
| describe('the floor takes the offer the model got', () => { | ||
| // The floor runs the same proposal the model could make, so it is gated on | ||
| // the same offer: declaration, outsider grant and web taint. | ||
| const declaresPropose = { tools: ['commonly_propose_action'] }; | ||
| const offers = (cfg, opts = {}) => toolsForConfig(cfg, opts.inPodOnly, opts) | ||
| .some((tool) => tool.function.name === 'commonly_propose_action'); | ||
|
|
||
| it('is offered when the install declares propose and nothing narrows it', () => { | ||
| expect(offers(declaresPropose)).toBe(true); | ||
| }); | ||
|
|
||
| it('is not offered when the install does not declare it', () => { | ||
| expect(offers({ tools: ['commonly_post_message'] })).toBe(false); | ||
| }); | ||
|
|
||
| it('is not offered in an outsider-granted pod', () => { | ||
| expect(offers(declaresPropose, { inPodOnly: true })).toBe(false); | ||
| }); | ||
|
|
||
| it('is not offered after a web result taints the turn', () => { | ||
| expect(offers(declaresPropose, { webTainted: true })).toBe(false); | ||
| }); | ||
| }); | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Script executed:
Repository: Team-Commonly/commonly
Length of output: 40407
Script executed:
Repository: Team-Commonly/commonly
Length of output: 9246
Script executed:
Repository: Team-Commonly/commonly
Length of output: 9645
Test the floor through
runAgent.nativeRuntimeService.connectFloor.test.jsonly callsconnectIntentFloorandconnectRequestAnswereddirectly. Add arunAgenttest with a humanmessage.postedconnect request and a silent model turn. Assert the proposed card and the downstream claim outcome. Add a contrasting replied or already-proposed turn that asserts no second card and the corresponding claim outcome.