diff --git a/.env.example b/.env.example index 5f3f300c5..3a564a075 100644 --- a/.env.example +++ b/.env.example @@ -101,6 +101,14 @@ INTERNAL_API_SECRET= # admin — if your env points at production APIs, that is production admin. # NUXI_CLI_ADMIN=1 +# Optional: a GitHub personal access token. Nuxi's coder subagent uploads its +# screenshots and recordings with it as GitHub attachments, to show them in +# pull requests. GitHub refuses that upload from the token of a GitHub App, so +# the `nuxi-code` connector can't do it. The token's account needs write access +# to the repositories Nuxi works on, and a classic token needs the `repo` +# scope. Use a token of a bot account: every attachment belongs to its account. +# NUXI_GITHUB_ATTACHMENTS_PAT= + # Canonical site URL for MCP + internal API callbacks (also used for OG images # when pre-rendering the docs) NUXT_PUBLIC_SITE_URL= diff --git a/layers/nuxi/README.md b/layers/nuxi/README.md index e821e4518..ac9cd3eed 100644 --- a/layers/nuxi/README.md +++ b/layers/nuxi/README.md @@ -87,14 +87,45 @@ Dispatch is restricted to channels listed under `discord.channels.admin`, `.publ When someone on Slack or in an admin Discord channel gives Nuxi a coding task on nuxt.com, such as a fix, a feature, a content change or a question about the code, Nuxi hands it to the `coder` subagent (`agent/subagents/coder/`). `coder` is a coding agent on Claude Opus 5.5 that delivers its changes as pull requests. It mounts [`@github-tools/eve-extension`](https://github-tools.com/integrations/eve) with the `pr-author` preset, trimmed to what opening a pull request needs. It makes the change on a new `nuxi/*` branch of `nuxt/nuxt.com` with `editFiles`, opens a pull request when it's done, and replies with the link. It never merges. - **Who:** `coder`'s `agent.ts` returns `null` outside Slack, `discord.channels.admin` channels, and `pnpm nuxi` with `NUXI_CLI_ADMIN=1`, so the web chat, public Discord channels and schedules never see it or its tools. `githubToolsApproval` in `agent/lib/github.ts`, the root agent's `approval` policy, also covers subagents and denies every `github__*` call anywhere else. -- **Where:** `guardrails` refuse any call outside `nuxt/nuxt.com` and any write outside `nuxi/*` branches. -- **Approval:** creating the branch, committing and opening the pull request run without asking; a person reviews and merges on GitHub. `updatePullRequest` still asks the person who requested it, with **Approve** and **Deny** buttons in the thread. On Discord, the buttons need the Interactions Endpoint URL from the [Discord setup](#discord-channel). -- **Tools:** `coder/tools/` disables eve's sandbox tools (`bash` and the file tools), so it changes code through the GitHub API only and never needs a sandbox. It keeps `web_search` and `web_fetch` to read docs, issues and pages linked in the request. `web_fetch` stays off on the root agent, which serves public chats. +- **Where:** `GITHUB_REPOSITORIES` in `agent/lib/github.ts` lists the repositories `coder` works on, which is `nuxt/nuxt.com` for now. `guardrails` refuse any call outside that list and any write outside `nuxi/*` branches. See [Work on another repository](#work-on-another-repository). +- **Approval:** creating the branch, committing and opening the pull request run without asking; a person reviews and merges on GitHub. `updatePullRequest` still asks the person who requested it, with **Approve** and **Deny** buttons in the thread. So does a commit to a branch that already has an open pull request (`githubToolsApproval` in `agent/lib/github.ts`): `coder` can't add to a pull request from another task, or to its own on a follow-up, without a click. The same policy denies a commit that names no branch, or a repository outside the list. It compares repository names without their case, as GitHub does. On Discord, the buttons need the Interactions Endpoint URL from the [Discord setup](#discord-channel). +- **Tools:** `coder` changes code through the GitHub API only. Its sandbox holds a browser and no checkout, see [Browser and captures](#browser-and-captures). `coder/tools/` disables `edit_file` and `write_file`, and keeps `bash` and `read_file` to work on files in the sandbox, such as a capture or a logo. It keeps `web_search` and `web_fetch` to read docs, issues and pages linked in the request. `web_fetch` stays off on the root agent, which serves public chats. On Slack, Nuxi first moves the request into a [code channel](https://docs.slack.dev/ai/slack-code) with `tools/open_code_channel.ts`, offered to the same people as `coder`. The channel starts with the thread so far and invites the person who asked. Every message in it reaches Nuxi without a mention, the whole channel is one session, and anyone in it can stop the agent from Slack. This needs **Code Channels** turned on in the `slack/nuxi` and `slack/nuxi-preview` connector settings, which adds the `code_channels:manage` scope and the `message.session` and `agent_session_stopped` events, plus the `groups:read` and `mpim:read` bot scopes. Discord has no equivalent and keeps using threads. Auth uses the `nuxi-code` GitHub App on the `nuxt` org through the Vercel Connect connector `github/nuxi-code`, connected to every environment of the project. `coder` runs with `actAs: 'caller'`: every call uses the GitHub token of the person who asked, so the branch, commits and pull request are theirs, and GitHub enforces their own access to `nuxt/nuxt.com`. The extension's `attribution` still credits the App: the pull request body ends with "Generated by nuxi-code", and the commits list the App's bot as co-author. The first time someone asks, Nuxi replies with a sign-in button to authorize the App on their GitHub account; the task continues once they have. A person never falls back to the App's own token. Only `pnpm nuxi` with `NUXI_CLI_ADMIN=1`, which has no GitHub user, runs as the App, so local tests act with the App's permissions. `searchCode` is excluded because its free-text query can name another repository, which the `repositories` guardrail can't check. The App needs **Contents** and **Pull requests** (read and write) and **Metadata** (read). +### Browser and captures + +`coder` has its own sandbox (`coder/sandbox.ts`) with Chromium, [agent-browser](https://agent-browser.dev) and ffmpeg. It uses the sandbox to look at pages, and to produce the images a task needs instead of leaving them to a person. For example, a showcase pull request arrives with its screenshot and logo. A task that only asks for a capture, such as a clip of a page for a Slack reply, opens no pull request. + +- **Browser:** `coder/extensions/browser/` mounts [`@agent-browser/eve`](https://www.npmjs.com/package/@agent-browser/eve) as `browser__*`, which opens and operates any public page. The sandbox holds no credentials and no checkout, so `coder` can't run the site or capture its own branch. On Vercel, the sandbox's firewall refuses every connection to a private address (`networkPolicy` in `coder/sandbox.ts`). A local Docker sandbox has no such rule: there, the capture tools only refuse a URL that names a private address. The capture tools drive the same browser, so their calls run one at a time, even when the model asks for several together. +- **`capture_screenshot`:** saves a 2560×1440 WebP, the format of `public/assets/websites/`, and shows the frame to the model. Its `cookies`, `hideElements`, `removeElements`, `scripts` and `delay` options match `screenshotOptions` in `content/showcase.yml`. +- **`record_video`:** plays a list of steps with a visible cursor, saves an MP4 without sound, and shows six frames spread over the clip to the model. GitHub plays a video inline only from its own attachments, which is where `publish_media` uploads it. +- **`publish_media`:** uploads a capture to GitHub as an attachment of the repository, the upload behind [`gh --attach`](https://docs.github.com/en/github-cli/github-cli/attaching-files-with-github-cli), and returns its link and its Markdown once GitHub serves it. This is how a capture reaches a pull request body, and nothing is written to the repository. An image embeds as `![alt](url)`, and the URL of a clip alone on a line renders as a video player. GitHub serves the link to everyone only once a pull request, an issue or a comment of a public repository embeds it, so the link of a capture that no pull request embeds, such as one made for a Slack reply, can answer 404. GitHub has no way to delete an attachment. +- **`commit_assets`:** commits images to a `nuxi/*` branch, because `editFiles` writes text only. It accepts images of 1 MB at most whose bytes match their extension, so `capture_screenshot` takes a lower `quality` for a page that passes the limit. It uses the same GitHub token as the GitHub tools. Unlike `editFiles`, its commits don't list the App's bot as co-author. + +Every one of these tools calls `assertCanUseGithubTools`, so they are available to the same people as the GitHub tools. Two skills in `coder/skills/` hold the procedures: `showcase-entry` for a complete showcase entry, and `visual-evidence` for the captures of a pull request. + +Setup: + +- **Deploys:** `eve build` installs the browser into a sandbox snapshot once per build, which takes about a minute. Sessions start from that snapshot. +- **Local:** the browser needs Docker running before `pnpm nuxi`. The first browser call builds the sandbox image, which takes about two minutes. Without Docker, eve falls back to just-bash, which runs no native binaries, so the browser and capture tools fail. On Apple Silicon, the Docker sandbox gets no browser, because Chrome for Testing has no Linux arm64 build. +- **Attachment token:** `publish_media` needs `NUXI_GITHUB_ATTACHMENTS_PAT` on the eve service, a GitHub personal access token. GitHub accepts the attachment upload from a personal access token only: it answers 404 to the `nuxi-code` installation token and to the token of a person signed in through the App ([cli/cli#14309](https://github.com/cli/cli/issues/14309)). So this tool never uses the token of the person who asked, and every attachment belongs to the token's account. That account needs write access to each repository in `GITHUB_REPOSITORIES`, and a classic token needs the `repo` scope. Use a token of a bot account instead of a person's. Without the variable, the tool reports that it has no token, and the other tools still work. +- **Local GitHub access:** the GitHub tools and `commit_assets` need a linked Vercel project (`vercel link`), because Vercel Connect authenticates with the project's OIDC token. Without it, they fail with `VercelOidcTokenError`. +- **Local publishing:** set `NUXI_GITHUB_ATTACHMENTS_PAT` in `.env`. It needs no Vercel project. +- **`--no-default-extensions`:** the `dev` script passes this flag to turn off eve's bundled self-modification subagent. That subagent uses a just-bash sandbox, and eve prepares every sandbox when `coder` first opens its own. Preparing it runs `pnpm add just-bash` in `layers/nuxi`, which fails on the ignored build scripts and leaves a stray lockfile. + +### Work on another repository + +Nothing in `coder`'s tools is specific to `nuxt/nuxt.com`. To let it work on another repository: + +1. Add the repository to `GITHUB_REPOSITORIES` in `agent/lib/github.ts`. The first entry stays the default. +2. Give the `nuxi-code` GitHub App access to the repository, and give the account of `NUXI_GITHUB_ATTACHMENTS_PAT` write access to it. +3. Add a section for it to `coder/instructions.md`, like the `nuxt/nuxt.com` one. Say where its site is deployed, where its images live, and how its pull requests preview. + +`commit_assets` and `publish_media` take the repository as an input, and the `github__*` tools take `owner` and `repo`. + ## Caller identity `agent/lib/identity/context.ts` resolves `Context.person` (id, name, bot flag) for every surface, and `agent/lib/identity/caller-instructions.ts` renders it into the always-on prompt so "who am I?" has an answer. Discord's display name (`global_name`) comes for free from the inbound message payload; the `@`-mention handle (`username`) is still kept on the auth context for future use, it's just not what the model sees. Slack's `app_mention`/`message` events only carry a user id — `channels/slack.ts` resolves the real name via `users.info` (`agent/lib/slack/api.ts`, cached 1h per user), which needs the **`users:read`** bot scope on top of the ones above; without it, Slack callers fall back to their user id. diff --git a/layers/nuxi/agent/lib/github-files.ts b/layers/nuxi/agent/lib/github-files.ts new file mode 100644 index 000000000..207be7257 --- /dev/null +++ b/layers/nuxi/agent/lib/github-files.ts @@ -0,0 +1,116 @@ +import { checkedMediaType } from './media/media-type.js' + +const API = 'https://api.github.com' +/** Same rule as the `branches` guardrail of the GitHub tools. */ +const BRANCH = /^nuxi\/[a-z0-9][a-z0-9._/-]*$/ +const FILE_PATH = /^[\w.-]+(?:\/[\w.-]+)*$/ + +/** Showcase screenshots are 80 to 350 KB; an image near this is a mistake, not an asset. */ +export const MAX_COMMIT_BYTES = 1024 * 1024 + +export class GithubRequestError extends Error { + constructor(message: string, readonly status: number) { + super(message) + this.name = 'GithubRequestError' + } +} + +/** Where a call goes, and as whom. */ +interface Github { + /** `owner/name` */ + readonly repository: string + readonly token: string + readonly fetchImpl?: typeof fetch +} + +/** GET `url`, or POST `body` to it: bytes as they are, anything else as JSON. */ +export async function request({ token, fetchImpl = fetch }: Pick, url: string, body?: object): Promise { + const binary = body instanceof Uint8Array + const response = await fetchImpl(url, { + method: body ? 'POST' : 'GET', + headers: { + accept: 'application/vnd.github+json', + authorization: `Bearer ${token}`, + ...(body && { 'content-type': binary ? 'application/octet-stream' : 'application/json' }) + }, + body: binary ? Buffer.from(body) : body && JSON.stringify(body), + signal: AbortSignal.timeout(60_000) + }) + const data = await response.json().catch(() => ({})) as T & { message?: string } + if (!response.ok) { + throw new GithubRequestError(`GitHub answered HTTP ${response.status} on ${new URL(url).pathname}: ${data.message ?? 'no message'}`, response.status) + } + return data +} + +/** The number of the open pull request whose head is `branch`, if it has one. */ +export async function openPullRequestOf({ branch, ...github }: Github & { readonly branch: string }): Promise { + const head = `${github.repository.split('/')[0]}:${branch}` + const pulls = await request>(github, `${API}/repos/${github.repository}/pulls?state=open&per_page=1&head=${encodeURIComponent(head)}`) + return pulls[0]?.number +} + +interface CommitInput extends Github { + readonly branch: string + /** A conventional commit: the first line is the headline, the rest the body. */ + readonly message: string + /** `path` is the repository path, such as `public/assets/websites/world-triathlon.webp`. */ + readonly files: ReadonlyArray<{ readonly path: string, readonly bytes: Uint8Array }> +} + +const COMMIT_MUTATION = 'mutation ($input: CreateCommitOnBranchInput!) { createCommitOnBranch(input: $input) { commit { oid url } } }' + +/** + * Commit images to a `nuxi/*` branch in one commit, creating the branch from the default branch + * when it doesn't exist. `github__editFiles` takes text only, so an image can't go through it. + * GitHub signs a `createCommitOnBranch` commit and records the token's owner as its author. + */ +export async function commitAssets({ branch, message, files, ...github }: CommitInput) { + if (!BRANCH.test(branch) || branch.includes('..')) throw new Error(`Images are committed to a \`nuxi/*\` branch only, not "${branch}".`) + if (!files.length) throw new Error('No files to commit.') + for (const { path, bytes } of files) { + if (!FILE_PATH.test(path) || path.includes('..')) throw new Error(`"${path}" is not a path in the repository.`) + if (checkedMediaType(path, bytes, MAX_COMMIT_BYTES) === 'video/mp4') throw new Error(`"${path}" is a video: only images are committed.`) + } + + const repository = `${API}/repos/${github.repository}` + async function headOf(name: string) { + try { + return (await request<{ object: { sha: string } }>(github, `${repository}/git/ref/heads/${name}`)).object.sha + } catch (error) { + if (error instanceof GithubRequestError && error.status === 404) return undefined + throw error + } + } + + let head = await headOf(branch) + const createdBranch = !head + if (!head) { + const { default_branch } = await request<{ default_branch: string }>(github, repository) + head = await headOf(default_branch) + if (!head) throw new Error(`Could not read the head of ${default_branch}.`) + await request(github, `${repository}/git/refs`, { ref: `refs/heads/${branch}`, sha: head }) + } + + const [headline = '', ...rest] = message.trim().split('\n') + const body = rest.join('\n').trim() + const { data, errors } = await request<{ + data?: { createCommitOnBranch?: { commit?: { oid: string, url: string } } } + errors?: Array<{ message?: string }> + }>(github, `${API}/graphql`, { + query: COMMIT_MUTATION, + variables: { + input: { + branch: { repositoryNameWithOwner: github.repository, branchName: branch }, + expectedHeadOid: head, + message: body ? { headline, body } : { headline }, + fileChanges: { additions: files.map(({ path, bytes }) => ({ path, contents: Buffer.from(bytes).toString('base64') })) } + } + } + }) + + const commit = data?.createCommitOnBranch?.commit + if (!commit) throw new Error(`GitHub refused the commit: ${errors?.map(error => error.message).join('; ') || 'no reason given'}`) + + return { repository: github.repository, branch, createdBranch, commitSha: commit.oid, commitUrl: commit.url, paths: files.map(file => file.path) } +} diff --git a/layers/nuxi/agent/lib/github-token.ts b/layers/nuxi/agent/lib/github-token.ts new file mode 100644 index 000000000..d68e92885 --- /dev/null +++ b/layers/nuxi/agent/lib/github-token.ts @@ -0,0 +1,23 @@ +import { connect } from '@vercel/connect/eve' +import type { ToolContext } from 'eve/tools' +import { appToken, GITHUB_CONNECTOR } from './github.js' +import { GithubRequestError } from './github-files.js' + +const userAuth = connect(GITHUB_CONNECTOR) + +/** + * Run `call` with a GitHub token for `repository`, chosen like the GitHub tools' `actAs: 'caller'`: + * the token of the person who asked, or the App's when no person drives the turn. A person whose + * token GitHub rejects is asked to sign in again. + */ +export async function withGithubToken(ctx: ToolContext, repository: string, call: (token: string) => Promise): Promise { + if (ctx.session.auth.current?.principalType !== 'user') return call(await appToken(repository)) + + const { token } = await ctx.getToken(userAuth) + try { + return await call(token) + } catch (error) { + if (error instanceof GithubRequestError && error.status === 401) ctx.requireAuth(userAuth) + throw error + } +} diff --git a/layers/nuxi/agent/lib/github.ts b/layers/nuxi/agent/lib/github.ts index f35a66f37..55f6b07aa 100644 --- a/layers/nuxi/agent/lib/github.ts +++ b/layers/nuxi/agent/lib/github.ts @@ -1,10 +1,28 @@ +import { getToken } from '@vercel/connect' import type { ApprovalPolicy } from 'eve/tools/approval' +import { openPullRequestOf } from './github-files.js' import { isAdminMode } from './identity/admin-mode.js' import { resolveContext, type AuthContext, type Surface } from './identity/context.js' /** The `nuxi-code` GitHub App on the nuxt org, connected to every environment of the project. */ export const GITHUB_CONNECTOR = 'github/nuxi-code' +/** + * The repositories the coder works on, as `owner/name`; the first is the default. To let Nuxi work + * on another one, add it here, give the `nuxi-code` App access to it, and describe it in + * `subagents/coder/instructions.md`. + */ +export const GITHUB_REPOSITORIES = ['nuxt/nuxt.com'] as const + +/** + * The entry of `GITHUB_REPOSITORIES` that `repository` names, if any. GitHub ignores the case of + * owners and repository names, and so does the `repositories` guardrail of the GitHub tools. + */ +function allowedRepository(repository: unknown) { + if (typeof repository !== 'string') return undefined + return GITHUB_REPOSITORIES.find(allowed => allowed.toLowerCase() === repository.toLowerCase()) +} + const GITHUB_SURFACES = new Set(['slack', 'discord', 'cli']) /** @@ -15,13 +33,69 @@ export async function canUseGithubTools(auth: AuthContext | null | undefined): P return GITHUB_SURFACES.has(resolveContext(auth).surface) && isAdminMode(auth) } +/** For the `execute` of the coder's own tools: a session can continue with a different caller than the one it started with. */ +export async function assertCanUseGithubTools(auth: AuthContext | null | undefined): Promise { + if (!(await canUseGithubTools(auth))) { + throw new Error('This tool is only available to the Nuxt team on Slack and in admin Discord channels.') + } +} + +/** A token of the App itself for `repository`, for a call no person's token is at hand for. */ +export function appToken(repository: string): Promise { + const [org = '', name = ''] = repository.split('/') + return getToken(GITHUB_CONNECTOR, { + subject: { type: 'app' }, + authorizationDetails: [{ type: 'github_app_installation', org, repositories: [name] }] + }) +} + +const [DEFAULT_OWNER, DEFAULT_REPO] = GITHUB_REPOSITORIES[0].split('/') + +/** + * The coder's tools that commit to a branch, with the repository each one commits to, read from + * the inputs that tool reads and with its defaults. Both name the branch `branch`. + */ +const COMMIT_TOOLS = new Map>) => unknown>([ + ['github__editFiles', ({ owner = DEFAULT_OWNER, repo = DEFAULT_REPO }) => typeof owner === 'string' && typeof repo === 'string' ? `${owner}/${repo}` : undefined], + ['commit_assets', ({ repository = GITHUB_REPOSITORIES[0] }) => repository] +]) + +/** + * Whether `branch` has an open pull request: work someone may be reviewing, which the coder may + * not have opened. Not knowing counts as yes. + */ +async function hasOpenPullRequest(repository: string, branch: string): Promise { + try { + return await openPullRequestOf({ repository, branch, token: await appToken(repository) }) !== undefined + } catch (error) { + console.warn(`[nuxi] could not look up the pull request of ${repository}@${branch}, so the commit asks for approval`, error) + return true + } +} + /** - * Agent-wide, so it also covers the coder subagent, which only these sessions are offered. A - * second line of defense: anywhere else, every `github__*` call, read or write, is denied before - * it reaches GitHub. + * Agent-wide, so it also covers the coder subagent, which only these sessions are offered. + * + * - A second line of defense: anywhere else, every `github__*` call, read or write, is denied + * before it reaches GitHub. + * - A commit to a branch that already has an open pull request asks the person first, like + * `updatePullRequest`. The coder's branches are new, so this only stops a commit to a pull + * request that is already open: its own on a follow-up, or one from another task. + * - A commit this policy can't place, with no branch or a repository outside the list, is denied. + * Letting it through would rely on the tool to refuse it, and the two once read the same input + * differently: `Nuxt/nuxt.com` was unknown here and allowed there. */ -export const githubToolsApproval: ApprovalPolicy = async ({ toolName, session }) => { - if (!toolName.startsWith('github__')) return undefined - if (await canUseGithubTools(session.auth.current)) return undefined - return { type: 'denied', reason: 'GitHub tools are only available to the Nuxt team on Slack and in admin Discord channels.' } +export const githubToolsApproval: ApprovalPolicy = async ({ toolName, toolInput, session }) => { + if (toolName.startsWith('github__') && !(await canUseGithubTools(session.auth.current))) { + return { type: 'denied', reason: 'GitHub tools are only available to the Nuxt team on Slack and in admin Discord channels.' } + } + const repositoryOf = COMMIT_TOOLS.get(toolName) + if (!repositoryOf) return undefined + + const repository = allowedRepository(repositoryOf(toolInput ?? {})) + const branch = toolInput?.branch + if (!repository || typeof branch !== 'string' || !branch) { + return { type: 'denied', reason: `A commit names a branch of one of these repositories: ${GITHUB_REPOSITORIES.join(', ')}.` } + } + return await hasOpenPullRequest(repository, branch) ? 'user-approval' : undefined } diff --git a/layers/nuxi/agent/lib/media/capture.ts b/layers/nuxi/agent/lib/media/capture.ts new file mode 100644 index 000000000..30f5d1110 --- /dev/null +++ b/layers/nuxi/agent/lib/media/capture.ts @@ -0,0 +1,304 @@ +import { BlockList, isIPv4 } from 'node:net' +import { quoteShellArg as quote, runAgentBrowser, type EveToolContext } from '@agent-browser/eve/sandbox' +import { z } from 'zod' + +/** One browser per sandbox: the `browser__*` tools and the capture tools drive the same page. */ +export const BROWSER_SESSION = 'nuxi' + +/** + * Launch flags of the sandbox's Chromium, which `subagents/coder/sandbox.ts` writes to agent-browser's + * user config. Headless Chromium reports no mouse, so `(hover: hover)` doesn't match and a site built + * with Tailwind v4 never shows its `hover:` styles in a capture. These Blink settings make it report + * one. The setting is per browser, so the `tablet` and `mobile` viewports report a mouse too. + */ +export const BROWSER_LAUNCH_ARGS = '--blink-settings=primaryHoverType=2,availableHoverTypes=2,primaryPointerType=4,availablePointerTypes=4' + +/** Where captures land in the sandbox, as `.`. */ +export const CAPTURES_DIR = '/workspace/captures' + +const VIEWPORTS = { + desktop: { width: 1280, height: 720 }, + tablet: { width: 768, height: 1024 }, + mobile: { width: 390, height: 844 } +} + +/** Between two recorded steps, so the clip reads as a walkthrough. */ +const STEP_PAUSE_MS = 500 +/** Frames per second of a clip. The recorder captures at this rate too: at its default of 30, its encoder falls behind in a slow sandbox and the recording fails. */ +const CLIP_FPS = 12 +/** Frames in a clip's preview, three per row. */ +const PREVIEW_FRAMES = 6 +/** How long stopping the recorder may take. agent-browser sets no limit, and a stop that hangs would hold the turn. */ +const STOP_TIMEOUT_MS = 60_000 + +const selectors = z.array(z.string().min(1).max(200)).max(20).optional() + +/** What both capture tools take: the page, and what to take out of the frame. */ +const page = { + name: z.string().regex(/^[a-z0-9][a-z0-9-]{0,62}$/, 'Lowercase letters, digits and hyphens only').describe('File name without extension, such as `world-triathlon`. A capture with the same name is replaced.'), + url: z.string().optional().describe('Public http(s) URL to open. Omit to use the page the `browser__*` tools left open, in the state they left it.'), + viewport: z.enum(['desktop', 'tablet', 'mobile']).optional().describe('desktop 1280x720 (default), tablet 768x1024 or mobile 390x844'), + colorScheme: z.enum(['light', 'dark']).optional().describe('Emulated `prefers-color-scheme`. Omit for the browser\'s default, which is light.'), + cookies: z.array(z.string().min(3).max(2000)).max(10).optional().describe('Cookies set before the page opens, as `name=value`, such as a consent cookie. Needs `url`.'), + hideElements: selectors.describe('CSS selectors hidden with `visibility: hidden`, such as `#onetrust-consent-sdk`. The element keeps its place in the layout.'), + removeElements: selectors.describe('CSS selectors removed with `display: none`. Use it for a widget that `hideElements` leaves in the frame, or a bar whose empty space should close up.') +} + +/** `cookies`, `hideElements`, `removeElements`, `scripts` and `delay` match `screenshotOptions` in the content files, so what works here can be saved there. */ +export const screenshotRequest = z.object({ + ...page, + scripts: z.array(z.string().min(1).max(2000)).max(5).optional().describe('JavaScript run in the page before the frame, such as a click that dismisses a dialog'), + delay: z.number().min(0).max(15).optional().describe('Seconds to wait before the frame, for fonts, images and entrance animations. Default 2.'), + selector: z.string().min(1).max(200).optional().describe('CSS selector or `@ref` scrolled into view before the frame'), + quality: z.number().int().min(1).max(100).optional().describe('WebP quality. Default 80. Lower it when the file is too large for `commit_assets`.') +}) + +const target = z.string().min(1).max(200).describe('CSS selector or `@ref` from `browser__snapshot`') + +const recordingStep = z.discriminatedUnion('action', [ + z.object({ action: z.literal('wait'), ms: z.number().int().min(100).max(5000) }), + z.object({ action: z.literal('scroll'), direction: z.enum(['up', 'down']), px: z.number().int().min(50).max(3000) }), + z.object({ action: z.literal('scroll_to'), selector: target }), + z.object({ action: z.literal('hover'), selector: target }), + z.object({ action: z.literal('click'), selector: target }), + z.object({ action: z.literal('press'), key: z.string().min(1).max(40).describe('Such as `Enter`, `Escape` or `Control+k`') }), + z.object({ action: z.literal('type'), selector: target, text: z.string().max(500) }) +]) + +export const recordingRequest = z.object({ + ...page, + steps: z.array(recordingStep).min(1).max(20).describe('Played in order while recording') +}) + +type PageRequest = z.infer | z.infer + +/** The sandbox methods a capture needs; eve's `SandboxSession` satisfies it. */ +export interface MediaSandbox { + run(options: { command: string, signal?: AbortSignal }): PromiseLike<{ exitCode?: number, stdout?: string, stderr?: string }> + readBinaryFile(options: { path: string }): PromiseLike +} + +/** + * The IPv4 ranges that are not the public internet: 100.64/10 holds a cloud metadata address, and + * 224/3 is multicast and above. On Vercel, the sandbox's firewall refuses every connection to them + * (`subagents/coder/sandbox.ts`). + */ +export const PRIVATE_SUBNETS = ['10.0.0.0/8', '100.64.0.0/10', '169.254.0.0/16', '172.16.0.0/12', '192.0.0.0/24', '192.168.0.0/16', '198.18.0.0/15', '224.0.0.0/3'] + +const privateAddresses = new BlockList() +// 0/8 and loopback never leave the sandbox, so only the URL check knows them. +for (const subnet of [...PRIVATE_SUBNETS, '0.0.0.0/8', '127.0.0.0/8']) { + const [network = '', prefix] = subnet.split('/') + privateAddresses.addSubnet(network, Number(prefix)) +} +const LOCAL_NAME = /^(?:localhost|.*\.(?:localhost|local|internal))$/ + +/** + * Why `raw` can't be captured, or null: public http(s) pages only. It reads the URL and resolves + * nothing, so it stops a mistake with a clear message. A hostname that points to a private address, + * a redirect to one and the `browser__*` tools are the firewall's to refuse. + */ +export function captureUrlError(raw: string): string | null { + let url: URL + try { + url = new URL(raw) + } catch { + return `"${raw}" is not an absolute URL.` + } + if (url.protocol !== 'https:' && url.protocol !== 'http:') return `"${raw}" must use http(s).` + if (url.username || url.password) return 'URLs with credentials are not captured.' + // `localhost.` resolves like `localhost`. + const host = url.hostname.toLowerCase().replace(/\.$/, '') + // An IPv6 literal is refused whatever it is. `URL` writes every IPv4 form as dotted decimal. + if (host.startsWith('[') || LOCAL_NAME.test(host) || (isIPv4(host) && privateAddresses.check(host))) return `"${host}" is a local or private address.` + return null +} + +/** `name=value` from a cookie string as `screenshotOptions.cookies` writes them, such as `consent=1;`. */ +export function parseCookie(cookie: string): { name: string, value: string } | null { + const pair = cookie.split(';')[0]!.trim() + const separator = pair.indexOf('=') + if (separator <= 0) return null + return { name: pair.slice(0, separator).trim(), value: pair.slice(separator + 1).trim() } +} + +/** + * JavaScript that adds a stylesheet to the page. A stylesheet, not a one-off query, so a banner + * that mounts after the page loads is covered too. `JSON.stringify` leaves `<`, `>`, U+2028 and + * U+2029 as they are, and they can end a script or a line, so they are escaped. + */ +function styleExpression(css: string): string { + const literal = JSON.stringify(css).replace(/[<>\u2028\u2029]/g, char => `\\u${char.charCodeAt(0).toString(16).padStart(4, '0')}`) + return `(() => { const style = document.createElement('style'); style.textContent = ${literal}; document.head.append(style); return true })()` +} + +/** Like capture-website's `hideElements`: the element keeps its place in the layout. */ +export function hideElementsExpression(selectors: readonly string[]): string { + return styleExpression(`${selectors.join(', ')} { visibility: hidden !important }`) +} + +/** Like capture-website's `removeElements`. A child can override its parent's `visibility` and stay in the frame; nothing overrides a parent that isn't displayed. */ +export function removeElementsExpression(selectors: readonly string[]): string { + return styleExpression(`${selectors.join(', ')} { display: none !important }`) +} + +/** The agent-browser command for one recorded step. */ +export function stepCommand(step: z.infer): string[] { + switch (step.action) { + case 'wait': return ['wait', String(step.ms)] + case 'scroll': return ['scroll', step.direction, String(step.px)] + case 'scroll_to': return ['scrollintoview', step.selector] + case 'hover': return ['hover', step.selector] + case 'click': return ['click', step.selector] + case 'press': return ['press', step.key] + case 'type': return ['type', step.selector, step.text] + } +} + +type Runner = ReturnType + +function runner(sandbox: MediaSandbox, signal?: AbortSignal) { + // agent-browser passes `abortSignal`; eve's sandbox calls it `signal`. + const ctx: EveToolContext = { getSandbox: () => Promise.resolve({ run: ({ command }) => sandbox.run({ command, signal }) }) } + return { + async browser>(...args: string[]): Promise { + const result = await runAgentBrowser<{ data?: T }>(ctx, args, { session: BROWSER_SESSION, abortSignal: signal }) + return result.json?.data + }, + async shell(command: string): Promise { + const result = await sandbox.run({ command, signal }) + if (result.exitCode) { + throw new Error(`\`${command.split(' ')[0]}\` exited ${result.exitCode}: ${(result.stderr || result.stdout || '').trim().slice(-600)}`) + } + return String(result.stdout ?? '') + } + } +} + +/** Size the viewport, open `url` when given, and take what is in the way out of the frame. */ +async function preparePage(run: Runner, request: PageRequest, scale: 1 | 2) { + const viewport = VIEWPORTS[request.viewport ?? 'desktop'] + + await run.shell(`mkdir -p ${CAPTURES_DIR}`) + await run.browser('set', 'viewport', String(viewport.width), String(viewport.height), String(scale)) + // Set on every capture: the browser keeps the scheme, and without a value it goes back to its default. + await run.browser('set', 'media', ...(request.colorScheme ? [request.colorScheme] : [])) + + if (request.url) { + const reason = captureUrlError(request.url) + if (reason) throw new Error(reason) + for (const cookie of request.cookies ?? []) { + const parsed = parseCookie(cookie) + if (!parsed) throw new Error(`Cookie "${cookie}" is not a name=value pair.`) + await run.browser('cookies', 'set', parsed.name, parsed.value, '--url', request.url) + } + await run.browser('open', request.url) + } else if (request.cookies?.length) { + throw new Error('`cookies` are set before a page opens: pass `url` with them.') + } + + if (request.hideElements?.length) await run.browser('eval', hideElementsExpression(request.hideElements)) + if (request.removeElements?.length) await run.browser('eval', removeElementsExpression(request.removeElements)) + return viewport +} + +/** The URL of the page the browser shows, refused when it isn't public: the `browser__*` tools, a redirect or a click may have led anywhere. */ +async function shownUrl(run: Runner) { + const url = (await run.browser<{ url?: string }>('get', 'url'))?.url ?? '' + const reason = captureUrlError(url) + if (reason) throw new Error(`The browser shows a page that can't be captured: ${reason}`) + return url +} + +/** A capture's file size and its preview as base64: an image small enough to show the model. */ +async function captured(run: Runner, sandbox: MediaSandbox, path: string, previewPath: string) { + const preview = await sandbox.readBinaryFile({ path: previewPath }) + if (!preview) throw new Error(`The capture left no preview at ${previewPath}.`) + return { + path, + bytes: Number((await run.shell(`stat -c %s ${quote(path)}`)).trim()), + preview: Buffer.from(preview).toString('base64') + } +} + +/** + * Captures run one at a time. They drive the same browser, and eve runs the tool calls of one model + * step together: a second capture would move the page under the first, or find the recorder busy. + */ +let lastCapture: Promise = Promise.resolve() +function oneAtATime(capture: () => Promise): Promise { + const result = lastCapture.then(capture) + lastCapture = result.catch(() => {}) + return result +} + +/** One frame of a page, saved as `.webp` at 2x: a 1280x720 viewport gives the 2560x1440 of the showcase and template images. */ +export const captureScreenshot = (...args: Parameters) => oneAtATime(() => screenshot(...args)) + +async function screenshot(sandbox: MediaSandbox, request: z.infer, signal?: AbortSignal) { + const run = runner(sandbox, signal) + const base = `${CAPTURES_DIR}/${request.name}` + + const { width, height } = await preparePage(run, request, 2) + for (const script of request.scripts ?? []) await run.browser('eval', script) + await run.browser('wait', String(Math.round((request.delay ?? 2) * 1000))) + if (request.selector) { + await run.browser('scrollintoview', request.selector) + await run.browser('wait', '300') + } + + const url = await shownUrl(run) + const title = (await run.browser<{ title?: string }>('get', 'title'))?.title ?? '' + await run.browser('screenshot', `${base}.png`) + await run.shell(`ffmpeg -y -loglevel error -i ${quote(`${base}.png`)} -c:v libwebp -quality ${request.quality ?? 80} ${quote(`${base}.webp`)}`) + await run.shell(`ffmpeg -y -loglevel error -i ${quote(`${base}.png`)} -vf scale=iw/2:-2 -c:v libwebp -quality 60 ${quote(`${base}.preview.webp`)}`) + await run.shell(`rm -f ${quote(`${base}.png`)}`) + + return { url, title, width: width * 2, height: height * 2, ...await captured(run, sandbox, `${base}.webp`, `${base}.preview.webp`) } +} + +/** + * A short clip of the steps, played with a visible cursor and saved as `.mp4`: GitHub plays + * a video inline from its own attachments only, which is where `publish_media` uploads it. Its + * preview is one image of frames spread evenly over the clip. ffmpeg builds it once the recording + * is done: the recorder's own contact sheet is analysed while it records, and fails the recording + * when a slow sandbox can't keep up. + */ +export const recordVideo = (...args: Parameters) => oneAtATime(() => recording(...args)) + +async function recording(sandbox: MediaSandbox, request: z.infer, signal?: AbortSignal) { + const run = runner(sandbox, signal) + const base = `${CAPTURES_DIR}/${request.name}` + + // 1x: the recorder captures frames as screenshots, and 2x frames quadruple the clip for no gain at video sizes. + await preparePage(run, request, 1) + await run.browser('wait', '1000') + const url = await shownUrl(run) + + // The recorder outlives this call, so it is stopped without the turn's signal: a cancelled turn must not leave it running. + const stop = () => runner(sandbox, AbortSignal.timeout(STOP_TIMEOUT_MS)).browser('record', 'stop') + try { + await run.browser('record', 'start', `${base}.webm`, '--cursor', '--fps', String(CLIP_FPS)) + await run.browser('wait', String(STEP_PAUSE_MS)) + for (const step of request.steps) { + await run.browser(...stepCommand(step)) + if (step.action !== 'wait') await run.browser('wait', String(STEP_PAUSE_MS)) + // A step can navigate. Checked after the pause, once a navigation has had time to land. + await shownUrl(run) + } + } catch (error) { + // Nothing recorded so far is kept: it may show a page that isn't public. + await stop().catch(() => {}) + await runner(sandbox).shell(`rm -f ${quote(`${base}.webm`)}`).catch(() => {}) + throw error + } + await stop() + + const seconds = Number(await run.shell(`ffprobe -v error -show_entries format=duration -of csv=p=0 ${quote(`${base}.webm`)}`)) || 0 + // H.264 in yuv420p plays in every browser; `faststart` lets it play before it has loaded in full. + await run.shell(`ffmpeg -y -loglevel error -i ${quote(`${base}.webm`)} -c:v libx264 -pix_fmt yuv420p -movflags +faststart -an ${quote(`${base}.mp4`)}`) + await run.shell(`ffmpeg -y -loglevel error -i ${quote(`${base}.webm`)} -vf "fps=${PREVIEW_FRAMES}/${Math.max(seconds, 1)},scale=540:-2,tile=3x2" -frames:v 1 -c:v libwebp -quality 60 ${quote(`${base}.preview.webp`)}`) + await run.shell(`rm -f ${quote(`${base}.webm`)}`) + + return { url, seconds: Math.round(seconds * 10) / 10, ...await captured(run, sandbox, `${base}.mp4`, `${base}.preview.webp`) } +} diff --git a/layers/nuxi/agent/lib/media/media-type.ts b/layers/nuxi/agent/lib/media/media-type.ts new file mode 100644 index 000000000..acdc0b623 --- /dev/null +++ b/layers/nuxi/agent/lib/media/media-type.ts @@ -0,0 +1,51 @@ +import { extname } from 'node:path/posix' + +export type MediaType = 'image/png' | 'image/jpeg' | 'image/gif' | 'image/webp' | 'video/mp4' + +const EXTENSIONS = new Map([ + ['png', 'image/png'], + ['jpg', 'image/jpeg'], + ['jpeg', 'image/jpeg'], + ['gif', 'image/gif'], + ['webp', 'image/webp'], + ['mp4', 'video/mp4'] +]) + +/** Media type from the file extension, or null for anything that is not an image or a clip we handle. */ +export function mediaTypeFromPath(path: string): MediaType | null { + return EXTENSIONS.get(extname(path).slice(1).toLowerCase()) ?? null +} + +/** The major brands of an MP4 video. ffmpeg writes `isom`. */ +const MP4_BRANDS = new Set(['isom', 'iso2', 'iso4', 'iso5', 'iso6', 'mp41', 'mp42', 'avc1']) + +function hasBytes(bytes: Uint8Array, offset: number, signature: readonly number[]): boolean { + return bytes.byteLength >= offset + signature.length && signature.every((byte, index) => bytes[offset + index] === byte) +} + +/** + * Media type from the file's first bytes, or null. A sandbox file is published or committed only + * when this agrees with its extension, so the extension alone never decides what the bytes are. + */ +export function sniffMediaType(bytes: Uint8Array): MediaType | null { + if (hasBytes(bytes, 0, [0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A])) return 'image/png' + if (hasBytes(bytes, 0, [0xFF, 0xD8, 0xFF])) return 'image/jpeg' + if (hasBytes(bytes, 0, [0x47, 0x49, 0x46, 0x38])) return 'image/gif' + // RIFF....WEBP + if (hasBytes(bytes, 0, [0x52, 0x49, 0x46, 0x46]) && hasBytes(bytes, 8, [0x57, 0x45, 0x42, 0x50])) return 'image/webp' + // ....ftyp, then the major brand. AVIF and HEIC images start with the same box, so the brand decides. + if (hasBytes(bytes, 4, [0x66, 0x74, 0x79, 0x70]) && MP4_BRANDS.has(String.fromCharCode(...bytes.subarray(8, 12)))) return 'video/mp4' + return null +} + +/** + * The media type of a sandbox file about to leave the sandbox. Its name and its first bytes have + * to agree, so nothing but a real image or clip leaves under a media name. + */ +export function checkedMediaType(name: string, bytes: Uint8Array, maxBytes: number): MediaType { + const mediaType = mediaTypeFromPath(name) + if (!mediaType) throw new Error(`"${name}" is not a png, jpg, gif, webp or mp4 file.`) + if (bytes.byteLength > maxBytes) throw new Error(`"${name}" is ${bytes.byteLength} bytes; the limit is ${maxBytes}.`) + if (sniffMediaType(bytes) !== mediaType) throw new Error(`The content of "${name}" does not match its extension.`) + return mediaType +} diff --git a/layers/nuxi/agent/lib/media/publish.ts b/layers/nuxi/agent/lib/media/publish.ts new file mode 100644 index 000000000..7085cf084 --- /dev/null +++ b/layers/nuxi/agent/lib/media/publish.ts @@ -0,0 +1,57 @@ +import { GithubRequestError, request } from '../github-files.js' +import { checkedMediaType } from './media-type.js' + +/** The variable that holds the personal access token captures are attached with. */ +export const ATTACHMENTS_TOKEN_ENV = 'NUXI_GITHUB_ATTACHMENTS_PAT' + +/** GitHub's limit for an attached image, and for a video in a repository on a free plan. */ +const MAX_ATTACHMENT_BYTES = 10 * 1024 * 1024 + +interface PublishInput { + /** `owner/name`: an attachment belongs to a repository. */ + readonly repository: string + /** File name with its extension. */ + readonly name: string + readonly bytes: Uint8Array + readonly fetchImpl?: typeof fetch +} + +/** + * Attach an image or a clip to `repository`, the way a file dropped into a pull request + * description is, and return its URL once GitHub serves it. It is the request `gh --attach` makes. + * + * GitHub accepts that upload from a personal access token only. It answers 404 to every token the + * `nuxi-code` App can give, its own and the one of a person signed in through it, so the token of + * the person who asked is never used here. The token is read from `NUXI_GITHUB_ATTACHMENTS_PAT`, + * and its account, which needs write access to the repository, owns the attachment. + * + * Until the URL is in a pull request, an issue or a comment of a public repository, GitHub answers + * 404 to a reader who is not signed in. From then on anyone can open it. GitHub has no way to + * delete an attachment. + */ +export async function publishMedia({ repository, name, bytes, fetchImpl = fetch }: PublishInput) { + const mediaType = checkedMediaType(name, bytes, MAX_ATTACHMENT_BYTES) + const token = process.env[ATTACHMENTS_TOKEN_ENV] + if (!token) throw new Error(`${ATTACHMENTS_TOKEN_ENV} is not set, so there is no token to attach the capture with.`) + const github = { token, fetchImpl } + + // An attachment names its repository by numeric id. + const { id } = await request<{ id: number }>(github, `https://api.github.com/repos/${repository}`) + const query = new URLSearchParams({ name, content_type: mediaType, repository_id: String(id) }) + const { url } = await request<{ url?: string }>(github, `https://uploads.github.com/user-attachments/assets?${query}`, bytes).catch((error: unknown) => { + // The upload answers 404, not 403, to a token it doesn't accept. + if (error instanceof GithubRequestError && error.status === 404) { + throw new GithubRequestError(`GitHub refused the attachment (HTTP 404). ${ATTACHMENTS_TOKEN_ENV} must be a personal access token whose account can write to ${repository}: a classic one needs the \`repo\` scope.`, 404) + } + throw error + }) + if (!url) throw new Error('GitHub returned no URL for the attachment.') + + // A URL is a claim until it loads: nothing unverified goes into a pull request. Read with the + // token, since nothing references the attachment yet. + const served = await fetchImpl(url, { headers: { authorization: `Bearer ${token}` }, signal: AbortSignal.timeout(30_000) }) + await served.body?.cancel() + if (!served.ok) throw new Error(`Uploaded, but ${url} answers HTTP ${served.status}.`) + + return { url, mediaType, bytes: bytes.byteLength } +} diff --git a/layers/nuxi/agent/subagents/coder/agent.ts b/layers/nuxi/agent/subagents/coder/agent.ts index b0af74ca2..73ac4ccfc 100644 --- a/layers/nuxi/agent/subagents/coder/agent.ts +++ b/layers/nuxi/agent/subagents/coder/agent.ts @@ -1,18 +1,19 @@ import { defineAgent, defineDynamic } from 'eve' -import { canUseGithubTools } from '../../lib/github.js' +import { canUseGithubTools, GITHUB_REPOSITORIES } from '../../lib/github.js' import { nuxiGatewayOptions, nuxiGatewayTags } from '../../lib/gateway-attribution.js' // Offered only where the GitHub tools are allowed, so web chat and public Discord channels never // see it. eve runs this before every parent model call, hence the per-turn memo: on Discord, // `canUseGithubTools` reads Global Config. The root agent's `approval` and `limits.sessionUsage` -// apply here too, and `tools/` turns off eve's sandbox tools: it changes code through the GitHub API -// only, and keeps `web_search` and `web_fetch` to read docs and issues. +// apply here too. `sandbox.ts` gives it a browser, driven by `extensions/browser/` and the capture +// tools in `tools/`, which also turns off eve's file-writing tools: the sandbox holds no checkout, so +// code changes through the GitHub API only, on the repositories in `GITHUB_REPOSITORIES`. It keeps `web_search` and `web_fetch` to read docs and issues. export default defineDynamic(async (ctx) => { const auth = ctx.session.auth.current if (!(await ctx.memo(ctx.turn?.id ?? null, () => canUseGithubTools(auth)))) return null return defineAgent({ - description: 'Nuxi\'s coding agent for nuxt.com, the nuxt/nuxt.com repository. Hand it any coding task there: a fix, a feature, a content or docs change, or a question about how the code works. It reads and searches the repository and the web, makes the change on a new `nuxi/*` branch, and opens a pull request when it is done. It does not see this conversation, so the message must hold the full task, links or excerpts from the thread, and the name of the person asking.', + description: `Nuxi's coding agent for these GitHub repositories: ${GITHUB_REPOSITORIES.join(', ')}. Hand it any coding task there: a fix, a feature, a content or docs change, a new showcase site or template with its screenshot and logo, or a question about how the code works. It reads and searches the repository and the web. When the task changes files, it makes the change on a new \`nuxi/*\` branch and opens a pull request, with the screenshots and short clips that show the change. Hand it any capture task too: it has the only browser, so only it can open a public page and say what it sees, take a screenshot (\`capture_screenshot\`) or record a short clip (\`record_video\`), and attach each to the repository, which gives it a link (\`publish_media\`). A task that changes no file opens no pull request, and a capture's link opens for everyone only once a pull request embeds it. It does not see this conversation, so the message must hold the full task, links or excerpts from the thread, and the name of the person asking.`, model: 'anthropic/claude-opus-5.5', modelOptions: { providerOptions: { diff --git a/layers/nuxi/agent/subagents/coder/extensions/browser/extension.ts b/layers/nuxi/agent/subagents/coder/extensions/browser/extension.ts new file mode 100644 index 000000000..05fe18e84 --- /dev/null +++ b/layers/nuxi/agent/subagents/coder/extensions/browser/extension.ts @@ -0,0 +1,17 @@ +import browser from '@agent-browser/eve' +import { BROWSER_SESSION } from '../../../../lib/media/capture.js' + +/** + * `browser__*`: navigate, read, click, fill and inspect pages in the sandbox's Chromium. Any public + * site, since a showcase candidate or a linked issue can be anywhere; the sandbox holds no + * credentials and the extension exposes no cookie or storage reads. The session name is fixed so + * `capture_screenshot` and `record_video` act on the page these tools left open. + */ +export default browser({ + session: BROWSER_SESSION, + // The sandbox template installs the matching version; a session never pays for the install. + autoInstall: false, + contentBoundaries: true, + maxOutputChars: 50_000, + inlineScreenshots: false +}) diff --git a/layers/nuxi/agent/subagents/coder/extensions/browser/tools/screenshot.ts b/layers/nuxi/agent/subagents/coder/extensions/browser/tools/screenshot.ts new file mode 100644 index 000000000..6f395c224 --- /dev/null +++ b/layers/nuxi/agent/subagents/coder/extensions/browser/tools/screenshot.ts @@ -0,0 +1,5 @@ +import { disableTool } from 'eve/tools' + +// `capture_screenshot` replaces it: it returns the frame to the model, which this tool doesn't, and +// saves the WebP the other tools publish or commit. +export default disableTool() diff --git a/layers/nuxi/agent/subagents/coder/tools/bash.ts b/layers/nuxi/agent/subagents/coder/extensions/browser/tools/upload.ts similarity index 54% rename from layers/nuxi/agent/subagents/coder/tools/bash.ts rename to layers/nuxi/agent/subagents/coder/extensions/browser/tools/upload.ts index 2156acb9c..d5654ba89 100644 --- a/layers/nuxi/agent/subagents/coder/tools/bash.ts +++ b/layers/nuxi/agent/subagents/coder/extensions/browser/tools/upload.ts @@ -1,3 +1,4 @@ import { disableTool } from 'eve/tools' +// Nothing here needs to send a sandbox file to a website. export default disableTool() diff --git a/layers/nuxi/agent/subagents/coder/extensions/github.ts b/layers/nuxi/agent/subagents/coder/extensions/github.ts index 93296d8f9..2fe9712df 100644 --- a/layers/nuxi/agent/subagents/coder/extensions/github.ts +++ b/layers/nuxi/agent/subagents/coder/extensions/github.ts @@ -1,5 +1,7 @@ import githubExtension from '@github-tools/eve-extension' -import { GITHUB_CONNECTOR } from '../../../lib/github.js' +import { GITHUB_CONNECTOR, GITHUB_REPOSITORIES } from '../../../lib/github.js' + +const [owner, repo] = GITHUB_REPOSITORIES[0].split('/') // Mounted under the coder subagent only, so the root agent and its public sessions never list // these tools. `githubToolsApproval` in the root `agent.ts` still gates every call. @@ -18,17 +20,18 @@ export default githubExtension({ 'resolveReviewThread', 'getCommit', 'compareCommits', - 'searchCode' // a free-text query: `repositories` can't hold a `repo:` qualifier to nuxt/nuxt.com + 'searchCode' // a free-text query: `repositories` can't hold it to the allowed repositories ], - context: { owner: 'nuxt', repo: 'nuxt.com' }, - guardrails: { repositories: ['nuxt/nuxt.com'], branches: ['nuxi/*'] }, + // Calls default to the first repository and are refused outside the list. + context: { owner, repo }, + guardrails: { repositories: [...GITHUB_REPOSITORIES], branches: ['nuxi/*'] }, // Each call runs as the person who asked, so the pull request and its commits are theirs on // GitHub. Someone who hasn't connected GitHub yet gets a sign-in prompt instead of the App. Only // `pnpm nuxi` with `NUXI_CLI_ADMIN=1`, which has no GitHub user, runs as the App, for local testing. // `attribution` (on by default) credits the App with a PR footer and a commit co-author. actAs: 'caller', - // Opening a pull request needs no click: the guardrails keep it on a new `nuxi/*` branch of - // nuxt/nuxt.com, and a person reviews and merges it on GitHub. Editing an existing pull request + // Opening a pull request needs no click: the guardrails keep it on a new `nuxi/*` branch of an + // allowed repository, and a person reviews and merges it on GitHub. Editing an existing pull request // (`updatePullRequest`) still asks. requireApproval: { createBranch: 'never', diff --git a/layers/nuxi/agent/subagents/coder/instructions.md b/layers/nuxi/agent/subagents/coder/instructions.md index aa00cfb0c..e650ff304 100644 --- a/layers/nuxi/agent/subagents/coder/instructions.md +++ b/layers/nuxi/agent/subagents/coder/instructions.md @@ -1,17 +1,35 @@ -You are Nuxi's coding agent for nuxt.com, whose source is the `nuxt/nuxt.com` repository on GitHub. You work through the `github__*` tools, plus `web_search` and `web_fetch`. You never see the conversation that asked for the work: work from the message you receive. +You are Nuxi's coding agent. You work on the repositories the `github__*` tools accept: `nuxt/nuxt.com`, unless the task names another one they accept. You never see the conversation that asked for the work: work from the message you receive. + +## What you work with + +- The repository, through the `github__*` tools and `commit_assets`. There is no checkout: you can't run the project, its dev server or its tests. +- A browser in a sandbox. The `browser__*` tools open and operate any public page, `capture_screenshot` and `record_video` capture it and show you the result, and `publish_media` attaches a capture to the repository, which gives it a link for a pull request. +- `web_search` and `web_fetch`, to read pages the task links to, such as a GitHub issue or a docs page, or to check an API you're unsure of. ## How you work - Understand the code before you change it. Find the right files with `github__getRepositoryTree` instead of guessing paths, and read every file you change with `github__getFileContent`. -- Use `web_search` and `web_fetch` to read pages the task links to, such as a GitHub issue or a docs page, or to check an API you're unsure of. +- When the task is about how a page looks or behaves, open the page and look at it before you read the code. - Keep the change to what was asked, and match the style of the surrounding code and content. +- Finish the task. When it needs a file you can produce, such as a screenshot of a site or a logo the site publishes, produce it and commit it instead of listing it as a step for a person. - When the task is a question about the code, answer it with file paths and line references. Change nothing unless asked. - When the task is unclear, or the change would be large or risky, don't guess: reply with what you found and the question to ask. +- When a GitHub tool fails on authentication, the connection is down for this session and no other credential exists. Finish what needs no GitHub, then reply with the error and what you prepared. ## Delivering a change -- Work on `nuxt/nuxt.com` only, on a new branch named `nuxi/` created from `main`. Never commit to a branch you did not create for this task. -- Commit with `github__editFiles`, in as few commits as the change needs. +- Work on one repository per task, on a new branch named `nuxi/` created from its default branch. Never commit to a branch you did not create for this task. When a branch with the name you want already exists, it belongs to another task: pick another name. A commit to a branch that has an open pull request asks the person to approve it first. +- Commit text files with `github__editFiles` and images with `commit_assets`, in as few commits as the change needs. - When the change is done, open a pull request. Title it and its commits with a conventional commit, such as `fix(docs): correct the useFetch example`. In the body, say what changed and why, and credit the person who asked by name. +- Open a draft pull request only when a part of the task is left that you tried and could not do. Name that part and what a person needs to do. - Never merge, and never edit a pull request you did not open for this task. -- Reply with the pull request link and a one-line summary of the change. +- Reply with the pull request link, a one-line summary of the change, and the URL of every capture you published. + +## nuxt/nuxt.com + +The source of nuxt.com. + +- `https://nuxt.com` serves `main`. +- The images the site serves live under `public/assets/`. Screenshots of showcase sites (`websites/.webp`), templates (`templates/.webp`) and video courses (`video-courses/.webp`) are all what `capture_screenshot` saves at its default viewport. +- The Vercel preview of a pull request redirects to a Vercel sign-in, so you can't open it. +- A pull request that only changes files under `content/` gets no Vercel build. It previews at `https://nuxt.com/pull/nuxt.com/` once a maintainer adds the `preview:enabled` label. Open that page only when the request says the label is on. diff --git a/layers/nuxi/agent/subagents/coder/sandbox.ts b/layers/nuxi/agent/subagents/coder/sandbox.ts new file mode 100644 index 000000000..e366cfc1c --- /dev/null +++ b/layers/nuxi/agent/subagents/coder/sandbox.ts @@ -0,0 +1,53 @@ +import { installAgentBrowser, quoteShellArg } from '@agent-browser/eve/sandbox' +import { DefaultSandbox, defineSandbox } from 'eve/sandbox' +import { BROWSER_LAUNCH_ARGS, PRIVATE_SUBNETS } from '../../lib/media/capture.js' + +type PrepareSandbox = NonNullable[0]>['prepare']> + +/** + * Chromium, agent-browser and ffmpeg, installed once per template: every coder session starts with + * a browser that can screenshot and record. It holds no checkout and no credentials. Code still + * changes through the GitHub API, and the tools that publish or commit a capture run in the app + * runtime and read the file out of the sandbox. + */ +const prepare: PrepareSandbox = async (sandbox) => { + // just-bash (`eve dev` without Docker) runs no native binaries: leave it empty, and the capture + // tools fail there with the browser's own "not installed" error. + const native = await sandbox.run({ command: 'command -v apt-get' }) + if (native.exitCode !== 0) return + + // Chrome for Testing has no Linux arm64 build, so a Docker sandbox on Apple Silicon gets the + // tooling without a browser. + const { stdout: architecture } = await sandbox.run({ command: 'uname -m' }) + const installBrowser = !['arm64', 'aarch64'].includes(String(architecture).trim()) + + // agent-browser passes `abortSignal`; eve's sandbox calls it `signal`. + await installAgentBrowser( + { run: ({ command, abortSignal }) => sandbox.run({ command, signal: abortSignal }) }, + { installBrowser, installSystemDependencies: true } + ) + + // agent-browser reads its user config at every launch, so the `browser__*` tools and the capture + // tools start the same Chromium whichever runs first. `args` is one string: it refuses an array. + const config = quoteShellArg(JSON.stringify({ args: BROWSER_LAUNCH_ARGS })) + const configured = await sandbox.run({ command: `mkdir -p "$HOME/.agent-browser" && printf '%s' ${config} > "$HOME/.agent-browser/config.json"` }) + if (configured.exitCode !== 0) throw new Error(`Could not write the agent-browser config: ${configured.stderr || configured.stdout}`) + + const ffmpeg = await sandbox.run({ command: 'sudo apt-get install -y --no-install-recommends ffmpeg' }) + if (ffmpeg.exitCode !== 0) throw new Error(`Could not install ffmpeg: ${ffmpeg.stderr || ffmpeg.stdout}`) +} + +export const environment = DefaultSandbox.environment({ + prepare, + vercel: { + // Chromium, a recording and an ffmpeg encode at once. + resources: { vcpus: 4 }, + // The browser opens any public page and nothing else: the firewall resets a connection to a + // private address, whatever name, redirect or tool led to it. Docker has no such rule. + networkPolicy: { allow: ['*'], subnets: { deny: PRIVATE_SUBNETS } } + }, + // Don't add `just-bash` to the app's dependencies for a sandbox that can't run a browser anyway. + justBash: { autoInstall: false } +}) + +export default defineSandbox(() => environment.open()) diff --git a/layers/nuxi/agent/subagents/coder/skills/showcase-entry/SKILL.md b/layers/nuxi/agent/subagents/coder/skills/showcase-entry/SKILL.md new file mode 100644 index 000000000..690118379 --- /dev/null +++ b/layers/nuxi/agent/subagents/coder/skills/showcase-entry/SKILL.md @@ -0,0 +1,67 @@ +--- +description: Add or update a website on the nuxt.com showcase (/showcase) as a complete entry, with its screenshot and logo. Load it before touching `content/showcase.yml` or anything under `public/assets/websites/`. +--- + +A showcase tile needs three things, and a pull request that has all three is ready to merge. Produce all three yourself. + +1. An entry in `websites` in `content/showcase.yml`. +2. A screenshot at `public/assets/websites/.webp`, shown when the tile is hovered. +3. A logo at `public/assets/websites/logos/.svg`, shown on the tile. + +`` comes from `name`: drop every character that is not a letter or a digit, then kebab-case what is left, which splits on capitals. `World Triathlon` gives `world-triathlon`, `GitLab` gives `git-lab`, `McDonald's France` gives `mc-donalds-france`. The site finds both files by this slug only, so a wrong slug shows an empty tile. Compare yours with the file names already in `public/assets/websites/`. + +## Before you start + +- Read `content/showcase.yml`. If the site is already listed, update its entry instead of adding one. +- Confirm the site runs Nuxt: open it with `browser__open` and check for `__NUXT_DATA__`, `window.__NUXT__` or `#__nuxt` with `browser__eval`, or use the nuxt.fyi link from the request. If it does not, stop and reply with what you found. + +## The entry + +```yaml + - name: World Triathlon + url: https://triathlon.org/ + industry: Travel & Leisure + logo: + width: 232 + height: 24 +``` + +- Add it at the end of `websites`, and leave the other entries untouched. +- `industry` is one of the values under `wall.filters`. Pick the one that sites of the same kind already use, and name the runner-up in the pull request when it is close. +- `logo.width` and `logo.height` are the two last numbers of the logo's `viewBox`. +- Add `screenshotOptions` only with the options the screenshot needed (see below). + +## The screenshot + +Call `capture_screenshot` with `name: `, the site's `url` and the default viewport. It returns the frame: look at it. + +The frame is good when it shows the site's own homepage, above the fold, with nothing on top of it. Fix what is in the way and capture again: + +- A cookie banner, a chat bubble, an accessibility widget, a newsletter popup: pass their selectors in `hideElements`. Find them with `browser__eval`, for example by listing the fixed-position elements of the page. A consent cookie in `cookies` also works when you can find its name. When a widget is still in the next frame, its children override `visibility`: pass the same selector in `removeElements`. +- A carousel or a video that has not loaded, or an entrance animation caught halfway: raise `delay`. +- A splash screen or a country selector that needs a click: pass the click in `scripts`. +- A bot challenge or an error page instead of the site: you can't capture it. Don't commit that frame. + +When the frame is clean, write the options that got you there into the entry's `screenshotOptions`, with the same names (`cookies`, `hideElements`, `removeElements`, `scripts`, `delay`), so a person can regenerate the image later. + +Commit it with `commit_assets`, from `/workspace/captures/.webp` to `public/assets/websites/.webp`. It takes 1 MB at most: when the capture's `bytes` is higher, capture again with a lower `quality`, such as 60. + +## The logo + +The tile uses the logo as a mask and paints it in one colour. Only its shape counts: its own colours are ignored, and anything opaque shows as a solid block. + +- Take the logo the site publishes. On most sites it is an inline `` in the header: get its markup with `browser__eval`, such as `document.querySelector('header svg').outerHTML`. When the header uses ``, read that file with `web_fetch`. A press or brand page is the next place to look. +- Never draw a logo, trace one from a raster image, or take one from a third-party logo site. +- Clean the markup: keep `xmlns` and `viewBox` on the root, and remove `width`, `height`, scripts and comments. Remove an `id` only when nothing in the file references it, such as `href="#mark"` or `url(#mark)`. Remove a background rectangle. If it has no `viewBox`, build it from its `width` and `height`. +- Remove `class` and `style` last, and first write what they do to the shape as attributes on the same element: `fill="none"`, `stroke`, `stroke-width`, `stroke-linecap`, `stroke-linejoin`, `fill-rule` and `clip-rule`. Without them, a logo drawn as an outline becomes a filled block. Delete an element they hide with `display: none`, `visibility: hidden` or `opacity: 0`. For an inline ``, a class gets its rules from the site's stylesheet: read the values with `getComputedStyle` in `browser__eval`. +- It can't work as a mask when it has a raster ``, references another file, or relies on two colours to be readable, such as white text on a coloured block. Look for a one-colour version on the site. + +Look at the cleaned logo before you commit it. With `browser__eval` on the site's page, cover the page with a dark `position: fixed` box that holds one element with a white background and `mask: url("data:image/svg+xml,…") center/contain no-repeat`, where the URL is the cleaned markup through `encodeURIComponent`. The tile draws the logo the same way. Call `capture_screenshot` without `url` and compare the shape with the logo in the site's header. A solid block, an empty box or a missing part means the cleaning broke the logo: fix the markup and look again. When the box is empty for the site's original markup too, the page refuses `data:` images: do the same on another public page, such as `https://example.com/`. + +Commit the SVG with `github__editFiles`; it is text. + +## The pull request + +- One branch, `nuxi/showcase-`, and a title such as `feat(showcase): add World Triathlon`. +- In the body: what the site is in one sentence, the Nuxt evidence with its link, why you picked the `industry`, and the screenshot. Show the screenshot by publishing the same capture with `publish_media` and pasting its Markdown. +- Open it ready for review when the three parts are in. Open it as a draft only when you tried and could not produce one of them, and say in the body which one, what you tried, and what a person needs to add. A draft is not the default for a task you did not finish trying. diff --git a/layers/nuxi/agent/subagents/coder/skills/visual-evidence/SKILL.md b/layers/nuxi/agent/subagents/coder/skills/visual-evidence/SKILL.md new file mode 100644 index 000000000..fa06462cc --- /dev/null +++ b/layers/nuxi/agent/subagents/coder/skills/visual-evidence/SKILL.md @@ -0,0 +1,30 @@ +--- +description: Show a change or a page with screenshots and short clips. Load it before opening a pull request that changes how a page looks or behaves, and when someone asks for a screenshot, a recording, a video, a before/after, or to look at a page. +--- + +A reviewer should see what you saw. A sentence that describes a page is not evidence; a capture is. + +## What you can and can't capture + +- **Production**, the deployed default branch of the repository, such as `https://nuxt.com`. Capture it freely: it is the "before" of every change. +- **Your branch**, only when your instructions name a preview of the repository that you can open. The sandbox has no checkout to run, so without such a preview you have no "after" for a code change. Say so in the pull request in one line, and never present a production capture as the result of your change. +- **Any other public page**: a site for the showcase, a template's demo, a page linked in an issue. +- Capture public pages only. Never a dashboard, an admin page, or a page that shows a person's data. + +## Still or clip + +- A still shows a state. Use `capture_screenshot`, with `selector` to bring the part that matters into the frame. +- A clip shows what a still can't: a hover, a menu opening, an animation, a scroll, a flow of several steps. Use `record_video` and keep it under ten seconds. +- To capture a state that needs interaction first, such as an open menu, reach it with the `browser__*` tools, then call `capture_screenshot` without `url`. +- Use the same viewport and the same `selector` for two captures that will be compared. Add a `mobile` capture when the change is responsive. + +Look at every capture you get back before you use it, and check that it shows the thing you mean. A frame of the top of the page when the change is in the footer is not evidence. Capture again instead of explaining around it. + +## Putting it where people see it + +- Call `publish_media` and paste the Markdown it returns into the pull request body, under a short label that says what the capture shows and where it was taken, such as "Before, production, desktop". For a clip, the Markdown is its URL: keep it alone on its line, with a blank line before and after, and it renders as a video player. +- Put the URL of every published capture in your reply too. A capture opens for everyone only once the pull request embeds it, so open the pull request before you reply. +- When the task has no pull request, such as a request to look at a page, say what the capture shows in your reply, give its URL, and say that the link may not open for someone else: nothing on GitHub embeds it. +- Never commit a capture that is evidence to the branch. Only images the site serves go into the repository, with `commit_assets`. +- When `publish_media` fails, say in the pull request and in your reply that the capture exists but could not be published. Never write "see the screenshot" for an image that is not there. +- A change that only adds something has no "before". Don't pair it with an unrelated capture to fill a table. diff --git a/layers/nuxi/agent/subagents/coder/tools/capture_screenshot.ts b/layers/nuxi/agent/subagents/coder/tools/capture_screenshot.ts new file mode 100644 index 000000000..8b27371ea --- /dev/null +++ b/layers/nuxi/agent/subagents/coder/tools/capture_screenshot.ts @@ -0,0 +1,26 @@ +import { defineTool, toolOutput, toolOutputPart } from 'eve/tools' +import { assertCanUseGithubTools } from '../../../lib/github.js' +import { captureScreenshot, screenshotRequest } from '../../../lib/media/capture.js' + +export default defineTool({ + description: [ + 'Screenshot a web page in the sandbox browser and look at it: the frame comes back as an image.', + 'Saves a 2x WebP at /workspace/captures/.webp: 2560x1440 for the default desktop viewport.', + 'Use `cookies`, `hideElements`, `removeElements`, `scripts` and `delay` to get a cookie banner, a chat widget or a splash screen out of the frame, then look at the result and call again until the frame is clean.', + 'Pass the returned `path` to `commit_assets` to add the image to a repository, or to `publish_media` to get its link, for evidence in a pull request or your reply.' + ].join(' '), + inputSchema: screenshotRequest, + label: { + start: ({ url, name }) => url ? `Screenshot ${url}` : `Screenshot the current page as ${name}` + }, + async execute(input, ctx) { + await assertCanUseGithubTools(ctx.session.auth.current) + return captureScreenshot(await ctx.getSandbox(), input, ctx.abortSignal) + }, + toModelOutput({ preview, ...screenshot }) { + return toolOutput.content([ + toolOutputPart.text(`${JSON.stringify(screenshot)}\nThe frame at half size:`), + toolOutputPart.file(preview, { mediaType: 'image/webp' }) + ]) + } +}) diff --git a/layers/nuxi/agent/subagents/coder/tools/commit_assets.ts b/layers/nuxi/agent/subagents/coder/tools/commit_assets.ts new file mode 100644 index 000000000..45c4ea237 --- /dev/null +++ b/layers/nuxi/agent/subagents/coder/tools/commit_assets.ts @@ -0,0 +1,40 @@ +import { defineTool } from 'eve/tools' +import { z } from 'zod' +import { assertCanUseGithubTools, GITHUB_REPOSITORIES } from '../../../lib/github.js' +import { commitAssets } from '../../../lib/github-files.js' +import { withGithubToken } from '../../../lib/github-token.js' + +export default defineTool({ + description: [ + 'Commit images from the sandbox to a `nuxi/*` branch of a repository, in one commit: a showcase screenshot, a template preview, an illustration for a post.', + '`github__editFiles` writes text only, so this is how a png, jpg, gif or webp reaches a repository, 1 MB each at most. Commit an SVG, which is text, with `github__editFiles`.', + 'The branch is created from the default branch when it does not exist, so this can be the first or a later commit of the task; open the pull request with the GitHub tools.', + 'Never use it for evidence of a change (before and after captures): those go through `publish_media`, not onto the branch.' + ].join(' '), + inputSchema: z.object({ + repository: z.enum(GITHUB_REPOSITORIES).default(GITHUB_REPOSITORIES[0]).describe('Repository to commit to, as `owner/name`'), + branch: z.string().min(6).max(120).describe('The task\'s branch, such as `nuxi/showcase-world-triathlon`'), + message: z.string().trim().min(8).max(1000).describe('A conventional commit message, such as `feat(showcase): add World Triathlon screenshot`'), + files: z.array(z.object({ + from: z.string().min(1).max(300).describe('Sandbox path, such as /workspace/captures/world-triathlon.webp'), + to: z.string().min(1).max(300).describe('Repository path, such as public/assets/websites/world-triathlon.webp') + })).min(1).max(10) + }), + async execute({ repository, branch, message, files }, ctx) { + await assertCanUseGithubTools(ctx.session.auth.current) + + const sandbox = await ctx.getSandbox() + const images = await Promise.all(files.map(async ({ from, to }) => { + const bytes = await sandbox.readBinaryFile({ path: from }) + if (!bytes) throw new Error(`No file at "${from}" in the sandbox.`) + return { path: to, bytes } + })) + + return withGithubToken(ctx, repository, token => commitAssets({ repository, token, branch, message, files: images })) + }, + // After `execute`, so TypeScript infers `output`. + label: { + start: ({ branch, files }) => `Commit ${files.length === 1 ? files[0]!.to : `${files.length} images`} to ${branch}`, + complete: (_input, output) => `Committed ${output.paths.join(', ')} to ${output.branch}` + } +}) diff --git a/layers/nuxi/agent/subagents/coder/tools/publish_media.ts b/layers/nuxi/agent/subagents/coder/tools/publish_media.ts new file mode 100644 index 000000000..a3cf2d156 --- /dev/null +++ b/layers/nuxi/agent/subagents/coder/tools/publish_media.ts @@ -0,0 +1,34 @@ +import { basename } from 'node:path/posix' +import { defineTool } from 'eve/tools' +import { z } from 'zod' +import { assertCanUseGithubTools, GITHUB_REPOSITORIES } from '../../../lib/github.js' +import { CAPTURES_DIR } from '../../../lib/media/capture.js' +import { publishMedia } from '../../../lib/media/publish.js' + +export default defineTool({ + description: [ + `Upload a capture from ${CAPTURES_DIR} to GitHub as an attachment of a repository, like a file dropped into a pull request description: a screenshot from \`capture_screenshot\` or a clip from \`record_video\`. Returns its \`url\` and the \`markdown\` that embeds it.`, + 'Use it for evidence: paste the `markdown` into the body of a pull request, an issue or a comment of that repository. For an image it is `![alt](url)`. For a clip it is the URL, which renders as a video player when it stands alone on a line with a blank line before and after.', + 'GitHub is checked to serve the URL before it is returned. The URL opens for everyone only once a pull request, an issue or a comment of the repository embeds it. Until then, a person who opens it can get a 404.', + 'An attachment can\'t be deleted, so publish captures of public pages only, and only the ones you will use.', + 'Not for images the site serves: those are committed to the branch with `commit_assets`.' + ].join(' '), + inputSchema: z.object({ + repository: z.enum(GITHUB_REPOSITORIES).default(GITHUB_REPOSITORIES[0]).describe('Repository whose pull request, issue or comment will embed the capture, as `owner/name`'), + path: z.string().min(1).max(300).describe('A `path` that `capture_screenshot` or `record_video` returned, such as /workspace/captures/showcase-after.webp'), + alt: z.string().trim().min(1).max(200).describe('What the capture shows, such as `World Triathlon tile on /showcase, hovered`. A clip has no alt text on GitHub, so say it in the label above the clip too.') + }), + label: { + start: ({ path }) => `Publish ${basename(path)}` + }, + async execute({ repository, path, alt }, ctx) { + await assertCanUseGithubTools(ctx.session.auth.current) + if (!path.startsWith(`${CAPTURES_DIR}/`) || path.includes('..')) throw new Error(`Only files under ${CAPTURES_DIR} are published.`) + + const bytes = await (await ctx.getSandbox()).readBinaryFile({ path }) + if (!bytes) throw new Error(`No file at "${path}".`) + const published = await publishMedia({ repository, name: basename(path), bytes }) + + return { ...published, markdown: published.mediaType === 'video/mp4' ? published.url : `![${alt.replace(/[[\]\n]/g, ' ')}](${published.url})` } + } +}) diff --git a/layers/nuxi/agent/subagents/coder/tools/read_file.ts b/layers/nuxi/agent/subagents/coder/tools/read_file.ts deleted file mode 100644 index 2156acb9c..000000000 --- a/layers/nuxi/agent/subagents/coder/tools/read_file.ts +++ /dev/null @@ -1,3 +0,0 @@ -import { disableTool } from 'eve/tools' - -export default disableTool() diff --git a/layers/nuxi/agent/subagents/coder/tools/record_video.ts b/layers/nuxi/agent/subagents/coder/tools/record_video.ts new file mode 100644 index 000000000..d5f74747d --- /dev/null +++ b/layers/nuxi/agent/subagents/coder/tools/record_video.ts @@ -0,0 +1,26 @@ +import { defineTool, toolOutput, toolOutputPart } from 'eve/tools' +import { assertCanUseGithubTools } from '../../../lib/github.js' +import { recordingRequest, recordVideo } from '../../../lib/media/capture.js' + +export default defineTool({ + description: [ + 'Record a short clip of an interaction in the sandbox browser: `steps` play with a visible cursor while recording.', + 'Use it when a still can\'t show the change: a hover state, a menu, an animation, a scroll, a multi-step flow.', + 'A half-second pause follows every step, so the clip reads as a walkthrough; add `wait` steps where something needs longer. Keep it under ten seconds.', + 'Saves /workspace/captures/.mp4, a video without sound. You get six frames spread over the clip back as one image, read left to right, then down: check that they show the interaction, then pass the returned `path` to `publish_media` to attach the clip to a pull request.' + ].join(' '), + inputSchema: recordingRequest, + label: { + start: ({ url, name }) => url ? `Record ${url}` : `Record the current page as ${name}` + }, + async execute(input, ctx) { + await assertCanUseGithubTools(ctx.session.auth.current) + return recordVideo(await ctx.getSandbox(), input, ctx.abortSignal) + }, + toModelOutput({ preview, ...recording }) { + return toolOutput.content([ + toolOutputPart.text(`${JSON.stringify(recording)}\nSix frames spread evenly over the clip, left to right, then down:`), + toolOutputPart.file(preview, { mediaType: 'image/webp' }) + ]) + } +}) diff --git a/layers/nuxi/package.json b/layers/nuxi/package.json index f6d2276de..96643d01c 100644 --- a/layers/nuxi/package.json +++ b/layers/nuxi/package.json @@ -4,10 +4,11 @@ "type": "module", "scripts": { "build": "eve build", - "dev": "eve dev", + "dev": "eve dev --no-default-extensions", "start": "eve start" }, "dependencies": { + "@agent-browser/eve": "^0.43.0", "@ai-sdk/gateway": "^4.0.138", "@chat-adapter/discord": "^4.44.0", "@chat-adapter/state-memory": "^4.44.0", diff --git a/package.json b/package.json index 6db327fa6..804d8817d 100644 --- a/package.json +++ b/package.json @@ -35,6 +35,7 @@ "templates:check": "nuxt task run templates:check" }, "dependencies": { + "@agent-browser/eve": "^0.43.0", "@ai-sdk/mcp": "^2.0.93", "@chat-adapter/discord": "^4.44.0", "@chat-adapter/state-memory": "^4.44.0", diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index ba6b3f7ce..af49626a2 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -175,6 +175,9 @@ importers: .: dependencies: + '@agent-browser/eve': + specifier: ^0.43.0 + version: 0.43.0(@vercel/sandbox@3.9.1)(eve@1.0.1(e8610aed677a9637da7c9b90397e6745)) '@ai-sdk/mcp': specifier: ^2.0.93 version: 2.0.93(zod@4.6.7) @@ -501,6 +504,20 @@ packages: bcrypt: optional: true + '@agent-browser/eve@0.43.0': + resolution: {integrity: sha512-BgtOD/JmrgKLt4HO0UVL3BIRweph+fkQrDgmAFI8y3hCGh+wJtcswSijtV72L72wnfB++Ko0W6fNupnKYJwwMQ==} + engines: {node: '>=24.0.0'} + peerDependencies: + eve: '>=0.84.0' + + '@agent-browser/sandbox@0.43.0': + resolution: {integrity: sha512-je4YRrWUpNh7sPljcWltT/iAw+yzZLIgIS14YYRj014SiorSS+Xu07JkUnEt8+hrB913fgLIXgHEpW4zhcU41g==} + peerDependencies: + '@vercel/sandbox': '>=1.0.0' + peerDependenciesMeta: + '@vercel/sandbox': + optional: true + '@ai-sdk/gateway@4.0.139': resolution: {integrity: sha512-2YEIPm4n59nm4PBhMGjFWH0g+ztPgfybqoeMIjfJYNiB8zLHftUrONgb8P3Ho3Yy5cdoLCjiDE/Hv2Crol0JBw==} engines: {node: '>=22'} @@ -10969,6 +10986,9 @@ packages: zod@4.1.11: resolution: {integrity: sha512-WPsqwxITS2tzx1bzhIKsEs19ABD5vmCVa4xBo2tq/SrV4RNZtfws1EnCWQXM6yh8bD08a1idvkB5MZSBiZsjwg==} + zod@4.4.3: + resolution: {integrity: sha512-ytENFjIJFl2UwYglde2jchW2Hwm4GJFLDiSXWdTrJQBIN9Fcyp7n4DhxJEiWNAJMV1/BqWfW/kkg71UDcHJyTQ==} + zod@4.5.4: resolution: {integrity: sha512-sC95tT5iHHH9gtpj6A81kh+NEaRAUFN+qlUPDUbRfOMvNf5QCBqsb3WgvnpVtK5Y+4UfA6KqufotuTvMGiTlsA==} @@ -10985,6 +11005,18 @@ snapshots: '@phc/format': 1.0.0 '@poppinss/utils': 6.10.1 + '@agent-browser/eve@0.43.0(@vercel/sandbox@3.9.1)(eve@1.0.1(e8610aed677a9637da7c9b90397e6745))': + dependencies: + '@agent-browser/sandbox': 0.43.0(@vercel/sandbox@3.9.1) + eve: 1.0.1(e8610aed677a9637da7c9b90397e6745) + zod: 4.4.3 + transitivePeerDependencies: + - '@vercel/sandbox' + + '@agent-browser/sandbox@0.43.0(@vercel/sandbox@3.9.1)': + optionalDependencies: + '@vercel/sandbox': 3.9.1 + '@ai-sdk/gateway@4.0.139(zod@4.6.7)': dependencies: '@ai-sdk/provider': 4.0.37 @@ -22802,6 +22834,8 @@ snapshots: zod@4.1.11: {} + zod@4.4.3: {} + zod@4.5.4: {} zod@4.6.7: {} diff --git a/test/unit/admin-mode.spec.ts b/test/unit/admin-mode.spec.ts index ee4e959f4..ef9bb10a8 100644 --- a/test/unit/admin-mode.spec.ts +++ b/test/unit/admin-mode.spec.ts @@ -2,6 +2,8 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import { isAdminMode } from '../../layers/nuxi/agent/lib/identity/admin-mode' import { githubToolsApproval } from '../../layers/nuxi/agent/lib/github' +vi.mock('@vercel/connect', () => ({ getToken: vi.fn(async () => 'app-token') })) + vi.mock('@vercel/global-config', () => ({ getAll: vi.fn(async () => ({ discord: { channels: { admin: ['C_ADMIN'], public: ['C_PUBLIC'] } } })) })) @@ -124,15 +126,87 @@ describe('githubToolsApproval', () => { afterEach(() => { vi.unstubAllEnvs() + vi.unstubAllGlobals() + vi.restoreAllMocks() }) const slackUser = { issuer: 'slack:T123', principalId: 'slack:T123:U1', principalType: 'user', attributes: { user_id: 'U1', author_type: 'user' } } const denied = { type: 'denied', reason: 'GitHub tools are only available to the Nuxt team on Slack and in admin Discord channels.' } - function decide(toolName: string, auth: Record | null) { - return githubToolsApproval({ toolName, session: { auth: { current: auth } } } as unknown as Parameters[0]) + function decide(toolName: string, auth: Record | null, toolInput?: Record) { + return githubToolsApproval({ toolName, toolInput, session: { auth: { current: auth } } } as unknown as Parameters[0]) + } + + /** GitHub with one open pull request, from `nuxi/showcase-world-triathlon`. Returns the `head` of every lookup. */ + function githubWithOnePullRequest() { + const heads: Array = [] + vi.stubGlobal('fetch', async (input: string | URL) => { + const url = new URL(String(input)) + const head = url.searchParams.get('head') + heads.push(head) + return Response.json(url.pathname === '/repos/nuxt/nuxt.com/pulls' && head === 'nuxt:nuxi/showcase-world-triathlon' ? [{ number: 2679 }] : []) + }) + return heads } + it('asks before a commit to a branch that has an open pull request', async () => { + githubWithOnePullRequest() + + expect(await decide('github__editFiles', slackUser, { branch: 'nuxi/showcase-world-triathlon', message: 'feat(showcase): add World Triathlon' })).toBe('user-approval') + expect(await decide('commit_assets', slackUser, { repository: 'nuxt/nuxt.com', branch: 'nuxi/showcase-world-triathlon' })).toBe('user-approval') + }) + + it('lets a commit to a branch without a pull request through, and looks up nothing for other tools', async () => { + const heads = githubWithOnePullRequest() + + expect(await decide('github__editFiles', slackUser, { owner: 'nuxt', repo: 'nuxt.com', branch: 'nuxi/showcase-world-triathlon-assets' })).toBeUndefined() + expect(await decide('commit_assets', slackUser, { branch: 'nuxi/showcase-world-triathlon-assets' })).toBeUndefined() + expect(await decide('github__createBranch', slackUser, { branch: 'nuxi/showcase-world-triathlon' })).toBeUndefined() + expect(await decide('capture_screenshot', slackUser, { name: 'world-triathlon' })).toBeUndefined() + expect(heads).toEqual(['nuxt:nuxi/showcase-world-triathlon-assets', 'nuxt:nuxi/showcase-world-triathlon-assets']) + }) + + it('asks whatever the case of the repository, which GitHub and the guardrails ignore', async () => { + const heads = githubWithOnePullRequest() + + expect(await decide('github__editFiles', slackUser, { owner: 'Nuxt', repo: 'nuxt.com', branch: 'nuxi/showcase-world-triathlon' })).toBe('user-approval') + expect(await decide('github__editFiles', slackUser, { repo: 'NUXT.COM', branch: 'nuxi/showcase-world-triathlon' })).toBe('user-approval') + expect(await decide('commit_assets', slackUser, { repository: 'Nuxt/Nuxt.com', branch: 'nuxi/showcase-world-triathlon' })).toBe('user-approval') + expect(heads).toEqual(Array(3).fill('nuxt:nuxi/showcase-world-triathlon')) + }) + + it('reads the repository where the tool does, so an input the tool ignores can\'t move the lookup', async () => { + githubWithOnePullRequest() + + expect(await decide('github__editFiles', slackUser, { repository: 'nuxt/ui', branch: 'nuxi/showcase-world-triathlon' })).toBe('user-approval') + expect(await decide('commit_assets', slackUser, { owner: 'nuxt', repo: 'ui', branch: 'nuxi/showcase-world-triathlon' })).toBe('user-approval') + }) + + it('denies a commit it can\'t place, instead of letting it through unasked', async () => { + const heads = githubWithOnePullRequest() + const unplaced = { type: 'denied', reason: 'A commit names a branch of one of these repositories: nuxt/nuxt.com.' } + + expect(await decide('github__editFiles', slackUser, { owner: 'nuxt', repo: 'ui', branch: 'nuxi/showcase-world-triathlon' })).toEqual(unplaced) + expect(await decide('github__editFiles', slackUser, { owner: null, branch: 'nuxi/showcase-world-triathlon' })).toEqual(unplaced) + expect(await decide('commit_assets', slackUser, { repository: 'nuxt/ui', branch: 'nuxi/showcase-world-triathlon' })).toEqual(unplaced) + expect(await decide('github__editFiles', slackUser, { message: 'docs: fix typo' })).toEqual(unplaced) + expect(await decide('commit_assets', slackUser)).toEqual(unplaced) + expect(heads).toEqual([]) + }) + + it('asks when GitHub can\'t say whether the branch has a pull request', async () => { + vi.stubGlobal('fetch', async () => Response.json({ message: 'Bad credentials' }, { status: 401 })) + vi.spyOn(console, 'warn').mockImplementation(() => {}) + + expect(await decide('github__editFiles', slackUser, { branch: 'nuxi/fix-typo' })).toBe('user-approval') + }) + + it('still denies a commit where GitHub tools are not offered', async () => { + githubWithOnePullRequest() + + expect(await decide('github__editFiles', { issuer: 'discord', principalId: 'discord:U1', principalType: 'user', attributes: { channel_id: 'C_PUBLIC' } }, { branch: 'nuxi/showcase-world-triathlon' })).toEqual(denied) + }) + it('leaves GitHub tools to their own policy for a human on Slack and in an admin Discord channel', async () => { expect(await decide('github__createPullRequest', slackUser)).toBeUndefined() expect(await decide('github__createPullRequest', { issuer: 'discord', principalId: 'discord:U1', principalType: 'user', attributes: { channel_id: 'C_ADMIN' } })).toBeUndefined() diff --git a/test/unit/nuxi-media.spec.ts b/test/unit/nuxi-media.spec.ts new file mode 100644 index 000000000..23f0bf9e5 --- /dev/null +++ b/test/unit/nuxi-media.spec.ts @@ -0,0 +1,395 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { captureScreenshot, captureUrlError, hideElementsExpression, parseCookie, recordVideo, removeElementsExpression, stepCommand } from '../../layers/nuxi/agent/lib/media/capture' +import { mediaTypeFromPath, sniffMediaType } from '../../layers/nuxi/agent/lib/media/media-type' +import { publishMedia } from '../../layers/nuxi/agent/lib/media/publish' +import { commitAssets, MAX_COMMIT_BYTES, openPullRequestOf } from '../../layers/nuxi/agent/lib/github-files' + +const ascii = (text: string) => [...text].map(char => char.charCodeAt(0)) +const WEBP = new Uint8Array([...ascii('RIFF'), 0x24, 0, 0, 0, ...ascii('WEBPVP8 '), 1, 2, 3]) +const PNG = new Uint8Array([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A, 1, 2, 3]) +const MP4 = new Uint8Array([0, 0, 0, 0x20, ...ascii('ftypisom'), 1, 2, 3]) +const AVIF = new Uint8Array([0, 0, 0, 0x20, ...ascii('ftypavif'), 1, 2, 3]) +const TEXT = new Uint8Array(ascii('TOKEN=secret')) + +/** A `fetch` that answers from `routes`, keyed by `METHOD url-without-query`, 404 for the rest, and records every request. */ +function fakeGithub(routes: Record) { + const requests: Array<{ call: string, url: URL, headers: Headers, body: unknown }> = [] + const fetchImpl = (async (input: string | URL | Request, init?: RequestInit) => { + const url = new URL(String(input)) + const call = `${init?.method ?? 'GET'} ${url.origin}${url.pathname}` + requests.push({ call, url, headers: new Headers(init?.headers), body: init?.body }) + return call in routes ? Response.json(routes[call]) : Response.json({ message: 'Not Found' }, { status: 404 }) + }) as typeof fetch + return { fetchImpl, requests } +} + +describe('captureUrlError', () => { + it('accepts a public page', () => { + expect(captureUrlError('https://triathlon.org/')).toBeNull() + // Each one is the address just outside a reserved range. + expect(captureUrlError('http://172.15.0.1/')).toBeNull() + expect(captureUrlError('http://100.63.255.255/')).toBeNull() + expect(captureUrlError('http://100.128.0.1/')).toBeNull() + expect(captureUrlError('http://198.20.0.1/')).toBeNull() + expect(captureUrlError('http://192.0.1.1/')).toBeNull() + expect(captureUrlError('http://223.255.255.255/')).toBeNull() + expect(captureUrlError('https://internal.example.com/')).toBeNull() + }) + + it.each([ + 'triathlon.org', + 'about:blank', + 'file:///etc/passwd', + 'javascript:alert(1)', + 'https://user:pass@example.com/', + 'http://localhost:3000/', + 'http://localhost./', + 'http://app.localhost/', + 'http://db.internal/', + 'http://127.0.0.1/', + // The URL parser normalises these to 127.0.0.1. + 'http://2130706433/', + 'http://0x7f.1/', + 'http://10.1.2.3/', + 'http://172.16.0.1/', + 'http://172.31.255.255/', + 'http://192.168.1.1/', + 'http://169.254.169.254/latest/meta-data/', + 'http://100.64.0.1/', + 'http://100.100.100.200/latest/meta-data/', + 'http://100.127.255.255/', + 'http://198.18.0.1/', + 'http://198.19.255.255/', + 'http://192.0.0.8/', + 'http://224.0.0.1/', + 'http://255.255.255.255/', + 'http://[::1]/' + ])('rejects %s', (url) => { + expect(captureUrlError(url)).toEqual(expect.any(String)) + }) +}) + +describe('parseCookie', () => { + it('reads cookies as `screenshotOptions.cookies` writes them', () => { + expect(parseCookie('consent=1;')).toEqual({ name: 'consent', value: '1' }) + expect(parseCookie('cc_cookie={"categories":["a=b"]}; Path=/')).toEqual({ name: 'cc_cookie', value: '{"categories":["a=b"]}' }) + }) + + it('rejects a string without a name', () => { + expect(parseCookie('consent')).toBeNull() + expect(parseCookie('=1')).toBeNull() + }) +}) + +describe('hideElementsExpression and removeElementsExpression', () => { + /** Runs the expression against a fake document and returns the stylesheets it appended. */ + function stylesFrom(expression: string) { + const appended: Array<{ textContent: string }> = [] + const document = { + createElement: () => ({ textContent: '' }), + head: { append: (style: { textContent: string }) => appended.push(style) } + } + expect(new Function('document', `return ${expression}`)(document)).toBe(true) + return appended.map(style => style.textContent) + } + + it('hides every selector, whatever quotes it contains', () => { + expect(stylesFrom(hideElementsExpression(['#cookie-banner', '[aria-label="It\'s a \\"banner\\""]']))) + .toEqual(['#cookie-banner, [aria-label="It\'s a \\"banner\\""] { visibility: hidden !important }']) + }) + + it('keeps characters that could end a script or a line out of the expression', () => { + const selectors = ['nav > a', '[data-x="