Skip to content
20 changes: 20 additions & 0 deletions backend/__tests__/unit/services/litellmClient.models.test.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
const LiteLLMClient = require('../../../../external/commonly-agent-services/shared/litellm-client');

// The client reads hosted gateway availability. It accepts no installations;
// there is currently no backend caller that rewrites local model selections.
describe('LiteLLM model listing', () => {
const originalFetch = global.fetch;
afterEach(() => { global.fetch = originalFetch; });

it.each([200, 401, 503])('returns models or rejects gateway status %s', async (status) => {
global.fetch = jest.fn().mockResolvedValue({
ok: status === 200,
status,
json: async () => ({ data: [{ id: 'deepseek-v4-flash' }] }),
});
const client = new LiteLLMClient({ baseUrl: 'http://fake-litellm', apiKey: 'test-key' });
if (status === 200) await expect(client.listModels()).resolves.toEqual([{ id: 'deepseek-v4-flash' }]);
else await expect(client.listModels()).rejects.toThrow(`Failed to list models: ${status}`);
expect(global.fetch).toHaveBeenCalledWith('http://fake-litellm/v1/models', expect.any(Object));
});
});
4 changes: 0 additions & 4 deletions backend/services/gatewayProvisionerServiceK8s.ts
Original file line number Diff line number Diff line change
Expand Up @@ -305,10 +305,6 @@ const createOrUpdateDeployment = async ({
name: 'GEMINI_API_KEY',
valueFrom: { secretKeyRef: { name: 'api-keys', key: 'gemini-api-key' } },
},
{
name: 'ANTHROPIC_API_KEY',
valueFrom: { secretKeyRef: { name: 'api-keys', key: 'anthropic-api-key', optional: true } },
},
{
name: 'OPENAI_API_KEY',
valueFrom: { secretKeyRef: { name: 'api-keys', key: 'openai-api-key', optional: true } },
Expand Down
48 changes: 48 additions & 0 deletions docs/SUMMARIZER_AND_AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,54 @@ static async runSummarizer() {

---

### Commonly-bot model and failure behavior

The Helm deployment sets `AGENT_MODEL=deepseek-v4-flash`, matching the backend's
`LITELLM_CHAT_MODEL`. The standalone bot uses the same default;
`COMMONLY_SOCIAL_REPHRASE_MODEL` overrides it for both pod summaries and social
rewrites. Both use the shared LiteLLM client. The chart's LiteLLM config must
route the selected group; `commonly-bot/index.test.js` exercises the actual
request body and checks that the default is a routed, non-Claude group.

On a failed pod-summary model call, the bot logs `feature=pod-summary` and the
model, then builds a heuristic recap from recent messages. A silent summary
request persists that recap before acknowledging its event; normal delivery
posts the recap before acknowledgement. This preserves an output but loses the
model-generated synthesis. A failed optional social rewrite logs
`feature=social-rephrase` and the model, and uses heuristic feed-post text.
These logs omit provider error bodies and chat content.

A LiteLLM 401 alone does not identify the feature that called it. Correlate the
caller and runtime model configuration before attributing lost user output;
requests from proxy loopback can also be operator probes.

---

### Direct Anthropic removal (D76)

Direct `anthropic/*` LiteLLM routes and cluster-key consumers are removed.
Claude on BYO agents still runs on the owner's machine. Hosted Claude routing
through OpenRouter is separate work; existing model choices must not be
rewritten while that decision is pending.

[SRE's read-only census, relayed in TASK-001 on 2026-10-25](https://github.com/Team-Commonly/commonly/pull/2704#issuecomment-6396818571),
found no hosted/native Claude selections; the 47 Claude selections were BYO seats. There is no migration script and no apply
path. Removing a hosted route must not rewrite BYO or unknown-tier models.
There is currently no backend caller or installation-rewrite path for the
shared client's model listing. `litellmClient.models.test.js` checks only the
listing response and gateway errors. It does not prove BYO persistence safety;
a future availability-driven rewrite requires its own runtime-tier regression.
This change includes no installation writes or migration code.

Deployment sequencing: merge and deploy the runtime/chart removal first.
SRE then removes the existing Secret Manager value and ESO mapping from
`templates/secrets/api-keys.yaml`, verifies ESO reconciliation, and checks
remaining runtime/PVC credential copies. The ExternalSecret entry is retained
until that owned cleanup so it is not confused with removal already performed
on production. No secret value is handled by these tests.

---

### **2. commonly-bot Agent Identity**

**Location**: Created via `AgentIdentityService.getOrCreateAgentUser()`
Expand Down
14 changes: 11 additions & 3 deletions external/commonly-agent-services/commonly-bot/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ const ENABLE_SOCIAL_REPHRASE = process.env.COMMONLY_SOCIAL_REPHRASE_ENABLED !==
const ENABLE_SOCIAL_FEED_POST = process.env.COMMONLY_SOCIAL_POST_TO_FEED === '1';
const ENABLE_SOCIAL_IMAGE = process.env.COMMONLY_SOCIAL_IMAGE_ENABLED === '1';
const ENABLE_SOCIAL_EXTERNAL_PUBLISH = process.env.COMMONLY_SOCIAL_PUBLISH_EXTERNAL === '1';
const SOCIAL_REPHRASE_MODEL = process.env.COMMONLY_SOCIAL_REPHRASE_MODEL || process.env.AGENT_MODEL || 'gemini-2.5-flash';
const SOCIAL_REPHRASE_MODEL = process.env.COMMONLY_SOCIAL_REPHRASE_MODEL || process.env.AGENT_MODEL || 'deepseek-v4-flash';
const SOCIAL_IMAGE_MODEL = process.env.COMMONLY_SOCIAL_IMAGE_MODEL || 'gemini-2.5-flash-image';
const llmClient = new LiteLLMClient({
model: SOCIAL_REPHRASE_MODEL,
Expand Down Expand Up @@ -232,16 +232,23 @@ const rephraseSocialPost = async ({ post, podName }) => {
{ model: SOCIAL_REPHRASE_MODEL, temperature: 0.5, maxTokens: 300 },
);
const parsed = parseJsonBlock(raw);
if (!parsed?.summary) return null;
if (typeof parsed?.summary !== 'string' || !parsed.summary.trim()) {
console.warn(`[commonly-bot] feature=social-rephrase model=${SOCIAL_REPHRASE_MODEL} rejected=missing-summary; using heuristic feed post`);
return null;
}
const overlap = lexicalOverlap(sourceText, parsed.summary);
if (overlap > 0.7) return null;
if (overlap > 0.7) {
console.warn(`[commonly-bot] feature=social-rephrase model=${SOCIAL_REPHRASE_MODEL} rejected=source-overlap; using heuristic feed post`);
return null;
}
return {
summary: String(parsed.summary).trim(),
why: String(parsed.why || '').trim(),
tags: Array.isArray(parsed.tags) ? parsed.tags.map((t) => String(t || '').trim()).filter(Boolean).slice(0, 2) : [],
overlap,
};
} catch {
console.warn(`[commonly-bot] feature=social-rephrase model=${SOCIAL_REPHRASE_MODEL} failed; using heuristic feed post`);
return null;
}
};
Expand Down Expand Up @@ -511,6 +518,7 @@ const buildLlmPodSummary = async ({ messages = [], podName = 'this pod', client
// summary takes over instead of a brace reaching the reader.
return podSummaryFromLlmReply(raw, { messageCount: meaningful.length });
} catch {
console.warn(`[commonly-bot] feature=pod-summary model=${SOCIAL_REPHRASE_MODEL} failed; using heuristic summary`);
return null;
}
};
Expand Down
126 changes: 125 additions & 1 deletion external/commonly-agent-services/commonly-bot/index.test.js
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
const test = require('node:test');
const assert = require('node:assert/strict');
const fs = require('node:fs');
const path = require('node:path');
const { execFileSync } = require('node:child_process');
const { buildLlmPodSummary, handleEvent } = require('./index');

const messages = [
Expand Down Expand Up @@ -81,10 +84,131 @@ test('summary.request stores recovered prose when the model reply ends mid-envel
});

test('summary.request stores heuristic prose and acknowledges after an LLM error', async () => {
const writes = await runSummaryRequest(new Error('LLM unavailable'));
const warnings = [];
const originalWarn = console.warn;
let writes;
console.warn = (line) => warnings.push(line);
try {
writes = await runSummaryRequest(new Error('LLM unavailable'));
} finally {
console.warn = originalWarn;
}
assert.equal(warnings.length, 1);
assert.match(warnings[0], /feature=pod-summary model=\S+ failed; using heuristic summary/);
assert.equal(writes.length, 2);
assert.ok(writes[0].url.endsWith('/summaries'));
assert.match(writes[0].body.summary, /^A quick update from Sharpen:/);
assert.match(writes[0].body.summary, /The deploy fix landed\./);
assert.ok(writes[1].url.endsWith('/events/event-1/ack'));
});


test('deployed and standalone pod summaries request the routed non-Claude default', () => {
const chart = path.resolve(__dirname, '../../../k8s/helm/commonly/templates');
const deployment = fs.readFileSync(path.join(chart, 'agents/commonly-bot-deployment.yaml'), 'utf8');
const model = deployment.match(/- name: AGENT_MODEL\n\s+value: "([^"]+)"/)[1];
const backend = fs.readFileSync(path.join(chart, 'core/backend-deployment.yaml'), 'utf8');
const backendModel = backend.match(/- name: LITELLM_CHAT_MODEL[\s\S]*?value: "([^"]+)"/)[1];
assert.equal(model, backendModel);
const router = fs.readFileSync(path.join(chart, 'configmaps/litellm-config.yaml'), 'utf8');
const groups = [...router.matchAll(/- model_name: ([^\n]+)\n\s+litellm_params:\n\s+model: ([^\n]+)/g)];
const route = groups.find((group) => group[1] === model);
assert.ok(route, `No route for ${model}`);
assert.doesNotMatch(`${model} ${route[2]}`, /claude|anthropic/i);
for (const deployed of [false, true]) {
const env = { ...process.env, LITELLM_BASE_URL: 'http://fake-llm' };
delete env.COMMONLY_SOCIAL_REPHRASE_MODEL;
delete env.AGENT_MODEL;
if (deployed) env.AGENT_MODEL = model;
// Fresh process exercises actual client + request body, without singleton
// env leakage or a paid model call.
const result = execFileSync(process.execPath, ['-e', `
const { buildLlmPodSummary } = require('./index');
global.fetch = async (_url, options) => {
process.stdout.write(JSON.parse(options.body).model);
return { ok: true, json: async () => ({ choices: [{ message: { content: '{"summary":"Done"}' } }] }) };
};
buildLlmPodSummary({ messages: [{ username: 'Sam', content: 'Done' }] }).catch(() => process.exit(1));
`], { cwd: __dirname, env, encoding: 'utf8' });
assert.equal(result, model);
}
});


for (const [mode, reason] of [
['http-error', 'failed'],
['missing', 'rejected=missing-summary'],
['blank', 'rejected=missing-summary'],
['copied', 'rejected=source-overlap'],
]) {
test(`${mode} social rewrite names its feature/model and still posts heuristic content`, () => {
const env = {
...process.env,
COMMONLY_TEST_REWRITE_MODE: mode,
COMMONLY_TEST_REWRITE_REASON: reason,
COMMONLY_SOCIAL_POST_TO_FEED: '1',
COMMONLY_SOCIAL_REPHRASE_ENABLED: '1',
COMMONLY_SOCIAL_REPHRASE_MODEL: 'deepseek-v4-flash',
COMMONLY_SOCIAL_IMAGE_ENABLED: '0',
COMMONLY_SOCIAL_PUBLISH_EXTERNAL: '0',
LITELLM_BASE_URL: 'http://fake-llm',
OPENROUTER_API_KEY: '',
GEMINI_API_KEY: '',
};
const result = execFileSync(process.execPath, ['-e', `
const assert = require('node:assert/strict');
const { handleEvent } = require('./index');
const calls = [];
const warnings = [];
console.warn = (line) => warnings.push(line);
global.fetch = async (url, options = {}) => {
calls.push({ url, body: options.body && JSON.parse(options.body) });
let data = {};
if (url.endsWith('/context')) data = { pod: { name: 'Testing' } };
else if (url.endsWith('/api/posts?category=Social')) data = [{ content: 'Testing deploy finished', source: { provider: 'x' } }];
else if (url.endsWith('/social-policy')) data = { policy: { socialMode: 'rewrite' } };
else if (url.endsWith('/v1/chat/completions')) {
if (process.env.COMMONLY_TEST_REWRITE_MODE === 'http-error') return { ok: false, status: 401, text: async () => 'bad key' };
const reply = process.env.COMMONLY_TEST_REWRITE_MODE === 'missing' ? { why: 'No summary' }
: { summary: process.env.COMMONLY_TEST_REWRITE_MODE === 'blank' ? ' ' : 'Testing deploy finished' };
data = { choices: [{ message: { content: JSON.stringify(reply) } }] };
}
else if (url.includes('category=Social')) data = [];
return { ok: true, json: async () => data };
};
handleEvent({ runtimeToken: 'test', userToken: 'test' }, { _id: 'curate-1', type: 'curate', podId: 'pod-1' }).then(() => {
assert.deepEqual(warnings, ['[commonly-bot] feature=social-rephrase model=deepseek-v4-flash ' + process.env.COMMONLY_TEST_REWRITE_REASON + '; using heuristic feed post']);
const feed = calls.find((call) => call.url.endsWith('/api/posts'));
assert.match(feed.body.content, /A notable update/);
assert.ok(calls.at(-1).url.endsWith('/curate-1/ack'));
}).catch((error) => { console.error(error); process.exit(1); });
`], { cwd: __dirname, env, encoding: 'utf8' });
assert.equal(result, '');
});

}

test('chart and dynamic gateways no longer consume a direct Anthropic key or route', () => {
const root = path.resolve(__dirname, '../../..');
const chart = path.join(root, 'k8s/helm/commonly');
const routes = fs.readFileSync(path.join(chart, 'templates/configmaps/litellm-config.yaml'), 'utf8');
const directRoute = /model_name:\s*claude-|model:\s*anthropic\//i;
const directKeyOrFallback = /ANTHROPIC_API_KEY|["']claude-[^"']*["']/;
assert.match(' - model_name: claude-haiku-4-5', directRoute);
assert.match(' model: anthropic/claude-haiku-4-5-20251001', directRoute);
assert.match(' api_key: os.environ/ANTHROPIC_API_KEY', directKeyOrFallback);
assert.match(' - {"claude-haiku-4-5": ["deepseek-v4-flash"]}', directKeyOrFallback);
assert.doesNotMatch(routes, directRoute);
assert.doesNotMatch(routes, directKeyOrFallback);
for (const file of [
'templates/core/backend-deployment.yaml',
'templates/ops/ops-runner-deployment.yaml',
'templates/agents/litellm-deployment.yaml',
'templates/agents/clawdbot-deployment.yaml',
'templates/secrets/local-secrets.yaml',
'values-local.yaml',
]) {
assert.doesNotMatch(fs.readFileSync(path.join(chart, file), 'utf8'), /ANTHROPIC_API_KEY|anthropic-api-key|anthropicApiKey/);
}
assert.doesNotMatch(fs.readFileSync(path.join(root, 'backend/services/gatewayProvisionerServiceK8s.ts'), 'utf8'), /ANTHROPIC_API_KEY|anthropic-api-key/);
});
16 changes: 0 additions & 16 deletions k8s/helm/commonly/templates/agents/clawdbot-deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,6 @@ spec:
const geminiKey = (process.env.GEMINI_API_KEY || '').trim();
const geminiKey2 = (process.env.GEMINI_API_KEY_2 || '').trim();
const openrouterKey = (process.env.OPENROUTER_API_KEY || '').trim();
const anthropicKey = (process.env.ANTHROPIC_API_KEY || '').trim();
const openaiKey = (process.env.OPENAI_API_KEY || '').trim();
const codexAccess = (process.env.OPENAI_CODEX_ACCESS_TOKEN || '').trim();
const codexRefresh = (process.env.OPENAI_CODEX_REFRESH_TOKEN || '').trim();
Expand Down Expand Up @@ -144,9 +143,6 @@ spec:
if (openrouterKey && !store.profiles['openrouter:default']) {
store.profiles['openrouter:default'] = { type: 'api_key', provider: 'openrouter', key: openrouterKey };
}
if (anthropicKey && !store.profiles['anthropic:default']) {
store.profiles['anthropic:default'] = { type: 'api_key', provider: 'anthropic', key: anthropicKey };
}
if (openaiKey && !store.profiles['openai:default']) {
store.profiles['openai:default'] = { type: 'api_key', provider: 'openai', key: openaiKey };
}
Expand Down Expand Up @@ -338,12 +334,6 @@ spec:
name: api-keys
key: gemini-api-key-2
optional: true
- name: ANTHROPIC_API_KEY
valueFrom:
secretKeyRef:
name: api-keys
key: anthropic-api-key
optional: true
- name: OPENROUTER_API_KEY
valueFrom:
secretKeyRef:
Expand Down Expand Up @@ -638,12 +628,6 @@ spec:
secretKeyRef:
name: api-keys
key: gemini-api-key
- name: ANTHROPIC_API_KEY
valueFrom:
secretKeyRef:
name: api-keys
key: anthropic-api-key
optional: true
- name: OPENAI_API_KEY
valueFrom:
secretKeyRef:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,9 @@ spec:
value: /app/config/runtime.json
- name: COMMONLY_AGENT_POLL_MS
value: {{ .Values.agents.commonlyBot.env.pollIntervalMs | quote }}
# Match the backend default; avoid a dead Gemini primary before fallback.
- name: AGENT_MODEL
value: "deepseek-v4-flash"
- name: LITELLM_BASE_URL
value: http://litellm:4000
- name: LITELLM_MASTER_KEY
Expand Down
8 changes: 1 addition & 7 deletions k8s/helm/commonly/templates/agents/litellm-deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -179,7 +179,7 @@ spec:
# The source secret was fixed, but a malformed key from ANY source must never be
# able to break routing again — so sanitize here too. litellm inherits this env
# via exec below.
for _k in OPENROUTER_API_KEY OPENAI_API_KEY ANTHROPIC_API_KEY GEMINI_API_KEY; do
for _k in OPENROUTER_API_KEY OPENAI_API_KEY GEMINI_API_KEY; do
eval "_orig=\${$_k:-}"
_clean=$(printf '%s' "$_orig" | tr -d '\r\n')
[ "$_orig" != "$_clean" ] && echo "sanitized $_k (stripped CR/LF)"
Expand Down Expand Up @@ -243,12 +243,6 @@ spec:
name: api-keys
key: openai-api-key
optional: true
- name: ANTHROPIC_API_KEY
valueFrom:
secretKeyRef:
name: api-keys
key: anthropic-api-key
optional: true
# chatgpt/ provider reads auth.json from CHATGPT_TOKEN_DIR (written by the
# codex-auth-rotator sidecar; source files seeded by operator via the codex-cli
# sidecar). See ADR-014.
Expand Down
Loading
Loading