Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 58 additions & 0 deletions backend/__tests__/unit/services/skillsCatalogPath.test.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
const fs = require('fs');
const path = require('path');
const {
backendRootFrom,
BUNDLED_CATALOG_PATH,
resolveSkillsCatalogPath,
} = require('../../../services/skillsCatalogPath');

const BACKEND_ROOT = path.resolve(__dirname, '../../..');
const REPO_ROOT = path.resolve(BACKEND_ROOT, '..');

describe('skillsCatalogPath', () => {
let savedPath;
let savedDir;

beforeEach(() => {
savedPath = process.env.SKILLS_CATALOG_PATH;
savedDir = process.env.SKILLS_CATALOG_DIR;
delete process.env.SKILLS_CATALOG_PATH;
delete process.env.SKILLS_CATALOG_DIR;
});

afterEach(() => {
if (savedPath === undefined) delete process.env.SKILLS_CATALOG_PATH;
else process.env.SKILLS_CATALOG_PATH = savedPath;
if (savedDir === undefined) delete process.env.SKILLS_CATALOG_DIR;
else process.env.SKILLS_CATALOG_DIR = savedDir;
});

it('resolves the backend root from both the source and the built layout', () => {
expect(backendRootFrom('/repo/backend/services')).toBe('/repo/backend');
expect(backendRootFrom('/app/dist/services')).toBe('/app');
});

it('defaults to the one checked-in copy under backend/docs/skills', () => {
expect(BUNDLED_CATALOG_PATH).toBe(
path.join(BACKEND_ROOT, 'docs', 'skills', 'awesome-agent-skills-index.json'),
);
expect(resolveSkillsCatalogPath()).toBe(BUNDLED_CATALOG_PATH);
expect(fs.existsSync(BUNDLED_CATALOG_PATH)).toBe(true);
});

it('keeps a single copy of the catalog in the repo', () => {
// The root and Helm copies were deleted in TASK-029. Nothing reads them,
// so a reappearance would be drift, not a fallback.
expect(fs.existsSync(path.join(REPO_ROOT, 'docs/skills/awesome-agent-skills-index.json'))).toBe(false);
expect(
fs.existsSync(path.join(REPO_ROOT, 'k8s/helm/commonly/configs/awesome-agent-skills-index.json')),
).toBe(false);
});

it('prefers SKILLS_CATALOG_PATH, then SKILLS_CATALOG_DIR', () => {
process.env.SKILLS_CATALOG_DIR = '/catalog-dir';
expect(resolveSkillsCatalogPath()).toBe('/catalog-dir/awesome-agent-skills-index.json');
process.env.SKILLS_CATALOG_PATH = '/explicit/index.json';
expect(resolveSkillsCatalogPath()).toBe('/explicit/index.json');
});
});
4 changes: 2 additions & 2 deletions backend/commonly-bundled-skills/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
Skills shipped with this repo that aren't (yet) in the upstream
`VoltAgent/awesome-agent-skills` catalog. The provisioner's preset auto-import
helper reads `commonly-bundled-skills/<skillId>/SKILL.md` first; if missing,
falls back to the catalog index at `docs/skills/awesome-agent-skills-index.json`.
falls back to the catalog index at `backend/docs/skills/awesome-agent-skills-index.json`.

Each skill lives in its own subdirectory:

Expand All @@ -17,7 +17,7 @@ commonly-bundled-skills/

## Why bundle locally instead of catalog-only

The upstream catalog (`docs/skills/awesome-agent-skills-index.json`) is synced
The upstream catalog (`backend/docs/skills/awesome-agent-skills-index.json`) is synced
periodically via `scripts/sync-awesome-agent-skills.sh`. New upstream skills land
in the catalog only after a sync. For skills we want to ship before the next
sync, or skills the upstream maintainer hasn't accepted (or that we maintain
Expand Down
12 changes: 4 additions & 8 deletions backend/services/presetSkillsAutoImport.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,8 @@
* Resolution order for each skillId:
* 1. Local bundle: `commonly-bundled-skills/<skillId>/SKILL.md`
* Used for skills not yet in the upstream catalog (today: officecli).
* 2. Catalog index: `docs/skills/awesome-agent-skills-index.json` entry's
* 2. Catalog index (`resolveSkillsCatalogPath`, bundled copy at
* `backend/docs/skills/awesome-agent-skills-index.json`): the entry's
* `sourceUrl` field — content fetched via `fetchSkillContentFromSource`.
* 3. Skip with a warning if neither is found.
*
Expand All @@ -29,15 +30,10 @@ import {
fetchSkillBundleFromClawHub,
parseLegacyOpenclawSkillsUrl,
} from './skillsCatalogService';
import { resolveSkillsCatalogPath } from './skillsCatalogPath';

const REPO_ROOT = path.resolve(__dirname, '..', '..');
const BUNDLED_SKILLS_DIR = path.join(REPO_ROOT, 'commonly-bundled-skills');
const CATALOG_INDEX_PATH = path.join(
REPO_ROOT,
'docs',
'skills',
'awesome-agent-skills-index.json',
);

interface CatalogEntry {
id: string;
Expand All @@ -61,7 +57,7 @@ const warnedClawhubFailFor = new Set<string>();
const loadCatalog = async (): Promise<Map<string, CatalogEntry[]>> => {
if (catalogCache) return catalogCache;
try {
const raw = await fs.readFile(CATALOG_INDEX_PATH, 'utf8');
const raw = await fs.readFile(resolveSkillsCatalogPath(), 'utf8');
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
const parsed = JSON.parse(raw);
const items: CatalogEntry[] = Array.isArray(parsed?.items) ? parsed.items : [];
const grouped = new Map<string, CatalogEntry[]>();
Expand Down
35 changes: 35 additions & 0 deletions backend/services/skillsCatalogPath.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
import path from 'path';

// The one checked-in copy of the skills catalog lives at
// backend/docs/skills/, the only location inside the backend Docker build
// context. The image bakes it at /app/docs/skills/, and the Helm init
// container copies it from there when SKILLS_CATALOG_URL is unset or fails.

const CATALOG_FILE = 'awesome-agent-skills-index.json';

// This module runs from backend/services/ under ts-node and jest, and from
// backend/dist/services/ in the built image. Either way the answer is the
// backend package root.
export const backendRootFrom = (dir: string): string => {
const parent = path.resolve(dir, '..');
return path.basename(parent) === 'dist' ? path.resolve(parent, '..') : parent;
};

export const BUNDLED_CATALOG_PATH = path.join(
backendRootFrom(__dirname),
'docs',
'skills',
CATALOG_FILE,
);

// SKILLS_CATALOG_PATH wins, then SKILLS_CATALOG_DIR, then the bundled copy.
// Every reader and the refresh writer go through here so they agree.
export const resolveSkillsCatalogPath = (): string => {
if (process.env.SKILLS_CATALOG_PATH) {
return process.env.SKILLS_CATALOG_PATH;
}
if (process.env.SKILLS_CATALOG_DIR) {
return path.join(process.env.SKILLS_CATALOG_DIR, CATALOG_FILE);
}
return BUNDLED_CATALOG_PATH;
};
10 changes: 2 additions & 8 deletions backend/services/skillsCatalogService.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import fs from 'fs';
import path from 'path';

import { fetchBytesFromAllowedHost, fetchTextFromAllowedHost, isOutboundUrlRefused } from './outboundUrlGuard';
import { resolveSkillsCatalogPath } from './skillsCatalogPath';

const DEFAULT_SOURCE = 'awesome';

Expand Down Expand Up @@ -94,13 +94,7 @@ interface GitHubSourceInfo {

const resolveCatalogPath = (source: string): string | null => {
if (source === DEFAULT_SOURCE) {
if (process.env.SKILLS_CATALOG_PATH) {
return process.env.SKILLS_CATALOG_PATH;
}
if (process.env.SKILLS_CATALOG_DIR) {
return path.join(process.env.SKILLS_CATALOG_DIR, 'awesome-agent-skills-index.json');
}
return path.resolve(__dirname, '../../docs/skills/awesome-agent-skills-index.json');
return resolveSkillsCatalogPath();
}
return null;
};
Expand Down
15 changes: 3 additions & 12 deletions backend/services/skillsRefreshService.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import fs from 'fs';
import path from 'path';
import GitHubAppService from './githubAppService';
import { resolveSkillsCatalogPath } from './skillsCatalogPath';

/**
* skillsRefreshService
Expand Down Expand Up @@ -47,16 +48,6 @@ interface RefreshResult {
// via `getLastRefreshedAt()` and the catalog endpoint.
let lastRefreshAt: Date | null = null;

const resolveLocalCatalogPath = (): string => {
if (process.env.SKILLS_CATALOG_PATH) {
return process.env.SKILLS_CATALOG_PATH;
}
if (process.env.SKILLS_CATALOG_DIR) {
return path.join(process.env.SKILLS_CATALOG_DIR, 'awesome-agent-skills-index.json');
}
// Matches the read path in skillsCatalogService.ts#resolveCatalogPath.
return path.resolve(__dirname, '../../docs/skills/awesome-agent-skills-index.json');
};

// One credential per refresh run. The App installation token is the default
// (an hour is longer than a run); GitHubAppService falls back to GITHUB_PAT only
Expand Down Expand Up @@ -187,7 +178,7 @@ export const refreshSkillsIndex = async (): Promise<RefreshResult> => {
// Fall back: read the local file and still run the star-count refresh on
// its entries. This is the common dev case when the upstream URL is wrong
// or the mirror has moved.
const localPath = resolveLocalCatalogPath();
const localPath = resolveSkillsCatalogPath();
const localCatalog = readLocalCatalog(localPath);
if (!localCatalog) {
console.warn('[skills-refresh] No local catalog available; nothing to refresh.');
Expand Down Expand Up @@ -302,7 +293,7 @@ export const refreshSkillsIndex = async (): Promise<RefreshResult> => {
items,
};

const targetPath = resolveLocalCatalogPath();
const targetPath = resolveSkillsCatalogPath();
try {
fs.mkdirSync(path.dirname(targetPath), { recursive: true });
fs.writeFileSync(targetPath, `${JSON.stringify(output, null, 2)}\n`, 'utf8');
Expand Down
2 changes: 1 addition & 1 deletion docker-compose.dev.yml
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@ services:
# Mount marketplace manifest from repo root
- ./packages/commonly-marketplace:/app/packages/commonly-marketplace:ro
# Mount skills catalog index
- ./docs/skills:/app/docs/skills:ro
- ./backend/docs/skills:/app/docs/skills:ro
# Mount agent service manifests for auto-registration
- ./external/commonly-agent-services:/app/external/commonly-agent-services:ro
# Mount agent runtime config state for provisioning
Expand Down
2 changes: 1 addition & 1 deletion docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ services:
- ./packages/commonly-marketplace:/app/packages/commonly-marketplace:ro
- ./external/commonly-agent-services:/app/external/commonly-agent-services:ro
# Mount skills catalog index
- ./docs/skills:/app/docs/skills:ro
- ./backend/docs/skills:/app/docs/skills:ro
# Mount agent runtime config state for provisioning
- ./external/clawdbot-state/config:/app/external/clawdbot-state/config
- ./external/commonly-bot-state:/app/external/commonly-bot-state
Expand Down
6 changes: 3 additions & 3 deletions docs/adr/ADR-013-agent-file-production-and-skill-bundles.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ So the question "how do agents make files" surfaces a deeper question: **what is

| Surface | Status | Reference |
|---|---|---|
| Skill catalog (~1,659 community-authored skills as of 2026-02-05 sync) | Synced from `VoltAgent/awesome-agent-skills` | `external/awesome-openclaw-skills/`, `docs/skills/awesome-agent-skills-index.json` |
| Skill catalog (~1,659 community-authored skills as of 2026-02-05 sync) | Synced from `VoltAgent/awesome-agent-skills` | `external/awesome-openclaw-skills/`, `backend/docs/skills/awesome-agent-skills-index.json` |
| Skill install API | Shipped | `backend/routes/skills.ts` — `GET /catalog`, `POST /import`, `GET /requirements`, `GET/DELETE /pods/:podId/imported`, ratings |
| Skill sync to gateway PVC | Shipped | `syncOpenClawSkills` in `agentProvisionerServiceK8s.ts`; called from provision + reprovision + `/api/skills/import` |
| `defaultSkills` in preset definitions | Shipped, mostly unused | `backend/routes/registry/presets.ts` |
Expand All @@ -50,7 +50,7 @@ Excluded: `tiangong-wps-*` (Windows COM only — won't run in our Linux gateway)

**Observation:** the catalog has solid PDF coverage and decent parse-direction tooling, but **no Linux-runnable PPTX or XLSX generation skills**. That's a gap we may fill upstream later (§Phase 4); for now we ship without it.

> **Caveat on catalog-derived numbers throughout this ADR.** The skill count (~1,659), star counts (e.g. `github` skill 603 stars, used below), category counts, and license fields are all read from `docs/skills/awesome-agent-skills-index.json`, which is a **frozen snapshot from `scripts/sync-awesome-agent-skills.sh` last run on 2026-02-05** (nearly 3 months ago at draft time). Upstream `VoltAgent/awesome-agent-skills` is actively maintained — every figure cited from the index drifts daily. Numbers are point-in-time evidence that the skill / count / popularity *exists*, not live counts. Re-sync before publishing or making sales-pitchy claims; treat all sync-derived figures as ≥ that value, never == it. Numbers fetched live via GitHub API in this ADR (notably OfficeCLI's 2,768 stars and v1.0.70 release date) are timestamped explicitly and are real-time as of 2026-05-03.
> **Caveat on catalog-derived numbers throughout this ADR.** The skill count (~1,659), star counts (e.g. `github` skill 603 stars, used below), category counts, and license fields are all read from `backend/docs/skills/awesome-agent-skills-index.json`, which is a **frozen snapshot from `scripts/sync-awesome-agent-skills.sh` last run on 2026-02-05** (nearly 3 months ago at draft time). Upstream `VoltAgent/awesome-agent-skills` is actively maintained — every figure cited from the index drifts daily. Numbers are point-in-time evidence that the skill / count / popularity *exists*, not live counts. Re-sync before publishing or making sales-pitchy claims; treat all sync-derived figures as ≥ that value, never == it. Numbers fetched live via GitHub API in this ADR (notably OfficeCLI's 2,768 stars and v1.0.70 release date) are timestamped explicitly and are real-time as of 2026-05-03.

### Why this ADR now

Expand Down Expand Up @@ -771,7 +771,7 @@ ADR-013 makes the `acpx_run` deprecation easier in two ways:
- 2026-05-03 — v3: closed open question #2 (`github` skill is real — `steipete/github`, MIT, 603 stars, requires `gh` already in image). Added §"Relationship to Installable taxonomy (ADR-001)" mapping each piece to the Installable model + the migration story when Phase 3 unpauses. Added new open questions on OfficeCLI version-pin cadence and Phase-3 acceleration vs. deferral.
- 2026-05-03 — v4: expanded Part 4 from "inspector Skills tab only" to a four-section design covering (4a) top-level V2 Marketplace page at `/v2/marketplace`, (4b) inspector Skills tab, (4c) inspector Agents tab, (4d) shared `<V2MarketplaceList>` component pattern. Title broadened. Phasing expanded from 4 phases to 8 active + 2 deferred. This ADR now also lands the active "Marketplace frontend" track from ADR-011 alongside the file-production work.
- 2026-05-03 — v5: **major correction after verifying actual backend/frontend state.** (a) `/v2/marketplace` already exists (renders v1 `AppsMarketplacePage`, 771 lines, with `Discover | Installed` sub-tabs). `/v2/agents/browse` and `/v2/skills` also exist. v4's "design new top-level page" was wrong; right move is to extend `AppsMarketplacePage` in place with Apps · Agents · Skills · Integrations kind tabs. (b) `/api/marketplace/*` (Randy, PR #215+#230) is **already Installable-canonical** via dual-write — not legacy AR as v3/v4 framed. ADR-013's Phase 3 wires the agents tab against the Installable side from day one and *is* the marketplace-frontend track from ADR-011 by definition. Phasing collapsed from 8 active to 5 active (extend page, build inspector tabs, deprecate v1 pages). §"Relationship to Installable taxonomy" rewritten to reflect the per-kind split (agents already on Installable; skills still on AR). Open questions #7 + #8 closed; #9 narrowed (Randy's `/api/marketplace/mine` likely covers fork ownership, just verify response shape).
- 2026-05-03 — v5.1: added §Caveat on catalog-derived numbers in the §Context section. Catalog index `docs/skills/awesome-agent-skills-index.json` was last synced **2026-02-05** (nearly 3 months ago); all derived numbers (`~1,659` skills, `github` skill `603 stars`, etc.) are point-in-time snapshots, not live counts. Annotated each citation in-place. Numbers fetched live via GitHub API today (notably OfficeCLI's 2,768 stars + v1.0.70 release date) are explicitly timestamped 2026-05-03 and treated separately. Reviewers should re-sync before any sales-pitchy or external use of the figures.
- 2026-05-03 — v5.1: added §Caveat on catalog-derived numbers in the §Context section. Catalog index `backend/docs/skills/awesome-agent-skills-index.json` was last synced **2026-02-05** (nearly 3 months ago); all derived numbers (`~1,659` skills, `github` skill `603 stars`, etc.) are point-in-time snapshots, not live counts. Annotated each citation in-place. Numbers fetched live via GitHub API today (notably OfficeCLI's 2,768 stars + v1.0.70 release date) are explicitly timestamped 2026-05-03 and treated separately. Reviewers should re-sync before any sales-pitchy or external use of the figures.
- 2026-05-03 — v5.2: ASCII mockups had **invented numbers that looked authoritative** (`Apps (12)`, `Agents (78)`, `★ 4.8 · 240 installs`, `★ 4.6 · 88 installs`, `★ 4.5 · 175 installs`). All replaced with `—` placeholders so reviewers don't read them as projections or commitments. Added a mockup-note clarifying live values come from the marketplace API at runtime.
- 2026-05-03 — v5.3: corrected the v5.2 over-redaction. **GitHub stars should render** (real upstream signal from catalog/live API) — restored on cards as `★ <gh-stars>` with explicit data source. **Commonly review-style ratings (`★ 4.8`) should NOT render** — they don't exist as a system; would need their own ADR. **Install counts** (real Commonly metric) render where the query is cheap, omitted otherwise. Added §4d **per-agent skill management drawer** — the "friendly UX" surface the user flagged was missing. Configure drawer opens from any agent click (Members, Agents tab, Marketplace) and shows that agent's skills as a single-column list with inline `[×]` remove + preset-aware "Recommended for ___" gaps. Added open question #11 for verifying per-agent install scoping in `POST /api/skills/import`. Existing 4d (shared card-list subcomponent) renumbered to 4e. Phase 4 verification expanded to cover the new drawer.
- 2026-05-03 — v5.5: added §"Relationship to ADR-005 (Local CLI wrapper driver — `acpx_run` deprecation)" parallel to the ADR-001 relationship section. Calls out that `acpx_run` is being deprecated because Commonly's own primitives (CAP + agent-rooms + ADR-005's A2A-via-DM wrapper-agent pattern) already cover the use case. ADR-013's `commonly_attach_file` actually *makes the deprecation easier* by adding a kernel-level file-delivery verb that's runtime-agnostic — wrapper agents (`sam-local-codex` etc.) get file-attach for free without depending on `acpx_run`. Phase 0b's `acpx_run` smoke test reframed as a pragmatic bootstrap, not a commitment. Updated companion ADRs in the header to include ADR-005. Added a corresponding bullet in §"What this unlocks."
Expand Down
Loading
Loading