Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions .github/workflows/isolated.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
name: Isolated

# Runs each Worker test case in its own process, catching a test that passes
# only on module-level state another test left behind. Too slow for every pull
# request, so it runs daily on main.
on:
schedule:
- cron: "23 6 * * *"
workflow_dispatch:

permissions:
contents: read

concurrency:
group: isolated-${{ github.ref }}
cancel-in-progress: true

jobs:
worker:
runs-on: ubuntu-latest
timeout-minutes: 120
steps:
- uses: DeterminateSystems/nix-installer-action@1d87d45818068401a10cf16bdc5f00b24994a83f
with:
determinate: false
# The build snapshots finished quests from the full history, as the Check job's does.
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
fetch-depth: 0
persist-credentials: false
- name: Install dependencies
run: nix develop --command just install
- name: Run each Worker test case alone
run: nix develop --command just worker-test-isolated
16 changes: 14 additions & 2 deletions cloudflare/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -388,7 +388,8 @@ projects through the Worker's own routes: ready, claimed, and blocked quests,
issues (one known only from a webhook, one with a long title), pull requests
with checks and reviews (one clean to merge), and a paused second project. It then adds runners and
jobs that are queued, running, failed, or waiting on an agent's questions, a
tool permission, or an account limit.
tool permission, or an account limit, and raw output behind their cards: one
long enough to page, a short one, and one that reached the size cap.

`dev/github.ts` fakes GitHub for every outbound request, so nothing leaves
localhost and no real credential is used. It models what publication needs:
Expand All @@ -397,7 +398,9 @@ request creation, so a runner's job publishes end to end. Like
GitHub, it scopes each installation token to the repositories and permissions
it was minted for. Notifications are off, since a push
would reach a real push service. Sign in with GitHub completes as the
seeded maintainer. From a shell, sign in with a cookie jar:
seeded maintainer. Browsers share localhost's cookies across ports, so each
seeded server names its cookies after its port (`seeded-8788.…`), and servers
on different ports keep their own sessions. From a shell, sign in with a cookie jar:

```sh
curl -sL -c jar -b jar -H 'Origin: http://localhost:8787' -d provider=github http://localhost:8787/sign-in
Expand All @@ -407,6 +410,8 @@ Check phone and desktop widths with the browser's device emulation. The Worker
tests run the same seed (`test/seed.test.ts`), so it follows schema and route
changes, and later end-to-end tests can start from it. The publication tests
(`test/publish.test.ts`) use the same fake with a smaller world of their own.
`just worker-test` also starts this server with `--check` on a free port, which
checks what only it wires up: the built static assets and its cookie names.

## Checks

Expand Down Expand Up @@ -439,6 +444,13 @@ tests encrypt RFC 8291's example, verify VAPID signatures, decrypt what a
subscribed device receives, and check that only decisions push, once each. No
OAuth credentials or Cloudflare account are required for checks.

The Worker tests share one runtime and reset only storage between tests, so a
test could still lean on module-level state another left behind. A daily CI job
(`.github/workflows/isolated.yml`) runs each test case in its own process
instead, which is too slow for every pull request. Run it locally with
`just worker-test-isolated`, optionally with `--jobs N` and file filters such as
`inbox`.

After changing bindings, regenerate types:

```sh
Expand Down
155 changes: 95 additions & 60 deletions cloudflare/dev/github.ts

Large diffs are not rendered by default.

48 changes: 43 additions & 5 deletions cloudflare/dev/seed.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,9 @@
// stubbed end-to-end tests. Sign-in, enrollment, observations, and webhooks go
// through the Worker's own routes against the fake GitHub, so they follow the
// current code; only runners and jobs, which need a live runner, are written
// to D1 directly.
// to D1 directly, and their raw output through the store a runner's uploads use.
import { append } from "../src/jobs/output";
import type { Runner } from "../src/jobs/store";
import {
type Comment,
type Pull,
Expand Down Expand Up @@ -250,7 +252,15 @@ const form = (fields: Record<string, string>) => ({
* paused), observe them, and add jobs in every state the inbox shows.
* Returns the signed-in user and session cookie.
*/
export async function seed(options: { fetch: Fetch; db: D1Database; world: World; origin: string; secret: string }) {
export async function seed(options: {
fetch: Fetch;
db: D1Database;
/** The Worker's `OUTPUT` bucket, for the jobs' raw output. */
output: R2Bucket;
world: World;
origin: string;
secret: string;
}) {
const { db, world, origin } = options;
const post = (path: string, fields: Record<string, string>, cookie: string) => {
const body = form(fields);
Expand All @@ -272,9 +282,10 @@ export async function seed(options: { fetch: Fetch; db: D1Database; world: World
302,
"OAuth callback",
);
// The seeded server names its cookies after its port, so match the session by suffix.
const cookie = cookies(callback)
.split("; ")
.filter((c) => c.startsWith("better-auth.session_token="))
.filter((c) => c.split("=")[0]!.endsWith("better-auth.session_token"))
.join("; ");
if (!cookie) throw new Error("Seed sign-in returned no session");
const me = world.users[0]!;
Expand Down Expand Up @@ -321,12 +332,13 @@ export async function seed(options: { fetch: Fetch; db: D1Database; world: World
"issue webhook",
);

await jobs(db, user, rocket);
await jobs({ DB: db, OUTPUT: options.output }, user, rocket);
return { user, cookie };
}

// Runners and jobs a live runner would have reported, in each state the inbox shows.
async function jobs(db: D1Database, user: string, repo: Repo) {
async function jobs(storage: { DB: D1Database; OUTPUT: R2Bucket }, user: string, repo: Repo) {
const db = storage.DB;
const now = Date.now();
const commit = commitOf(repo);
const head = (n: number) =>
Expand Down Expand Up @@ -628,4 +640,30 @@ async function jobs(db: D1Database, user: string, repo: Repo) {
];
}),
]);

// Raw output behind the agent cards' disclosure: a transcript long enough to
// page, a short one, and one that reached the size cap.
const output: Record<string, string[]> = {
"seed-plan-settings": [
"$ quest guide\nRead the guide.\n",
"$ git log --oneline -3 -- src/settings\n4b1c2d0 Save settings on blur\n9e8f7a6 Add the settings page\n",
"Opening issue #7 and its five comments.\nThe reports all mention a background tab on a phone.\n",
"$ just test settings\n✓ saves on blur (12 ms)\n✓ restores a draft (8 ms)\n",
"Simulating an expired session while the tab is hidden...\nThe save request returns 401 and the draft is dropped.\n",
"Reproduced. Writing up the cause before planning the fix.\n",
],
"seed-plan-config": [
"$ rg -n 'config.toml' src\nsrc/config.rs:14: let text = fs::read_to_string(path)?;\n",
"An empty file parses as an error, not as defaults. Asking how it should behave.\n",
],
"seed-iterate-cache": ["$ nix build .#cache\nbuilding '/nix/store/…-cache.drv'...\n"],
};
for (const [job, chunks] of Object.entries(output)) {
const id = list.find((j) => j.id === job)!.runner!;
const [, name, platform, vm] = runners.find(([runner]) => runner === id)!;
const runner: Runner = { id, owner: user, name, platform, vm };
for (const [seq, text] of chunks.entries()) await append(storage, runner, { job, attempt: 1, seq, text }, now);
}
// The cap's note, without uploading 64 MiB.
await db.prepare("UPDATE job_output_attempts SET truncated=1 WHERE job='seed-iterate-cache'").run();
}
87 changes: 74 additions & 13 deletions cloudflare/dev/serve.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,10 @@
// local sign-in, for checking UI changes in a browser. It runs the bundle that
// `wrangler deploy --dry-run` writes (see the `seeded` package script) on
// Miniflare directly, since `wrangler dev`'s proxy hangs in some sandboxes.
// Storage is in memory, so every run starts from the same seed.
// Storage is in memory, so every run starts from the same seed. With --check,
// it checks what only this server wires up, on a free port, and exits.
import { readFileSync, readdirSync } from "node:fs";
import { type AddressInfo, createServer } from "node:net";
import { join } from "node:path";
import { parseArgs } from "node:util";
import {
Expand All @@ -18,8 +20,21 @@ import { fakeGitHub } from "./github";
import { seed, world } from "./seed";

const root = join(import.meta.dirname, "..");
const { values } = parseArgs({ options: { port: { type: "string", default: "8787" } } });
const port = Number(values.port);
const { values } = parseArgs({
options: { port: { type: "string", default: "8787" }, check: { type: "boolean", default: false } },
});

/** A port nothing listens on, so checks run beside other seeded servers. */
async function freePort() {
const server = createServer();
await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve));
const { port } = server.address() as AddressInfo;
await new Promise((resolve) => server.close(resolve));
return port;
}

const port = values.check ? await freePort() : Number(values.port);
const prefix = `seeded-${port}.`;
const origin = `http://localhost:${port}`;
const secret = "dev-webhook-secret";
const data = world();
Expand All @@ -33,20 +48,44 @@ const { workerOptions } = unstable_getMiniflareWorkerOptions(join(root, "wrangle
// The bundle is a single ES module, and Miniflare 5 cannot convert module rules.
const { modulesRules: _, ...app } = workerOptions;

// GitHub's authorize page would leave localhost, so the entry Worker sends that
// redirect straight to the callback, which the fake GitHub completes as the
// seeded maintainer. Everything else, WebSockets included, passes through.
// Browsers share localhost's cookies across ports, so seeded servers on two
// ports would sign each other out. The entry Worker gives this server's cookies
// their own names: it prefixes each cookie the app sets with COOKIES, and
// passes the app only the cookies so prefixed, without it.
//
// GitHub's authorize page would leave localhost, so the entry Worker also sends
// that redirect straight to the callback, which the fake GitHub completes as
// the seeded maintainer. Everything else, WebSockets included, passes through.
const entry = `export default {
async fetch(request, env) {
const response = await env.APP.fetch(request);
const sent = (request.headers.get("Cookie") ?? "")
.split(/;\\s*/)
.filter((cookie) => cookie.startsWith(env.COOKIES))
.map((cookie) => cookie.slice(env.COOKIES.length));
const forwarded = new Headers(request.headers);
if (sent.length) forwarded.set("Cookie", sent.join("; "));
else forwarded.delete("Cookie");
const response = await env.APP.fetch(new Request(request, { headers: forwarded }));

const cookies = response.headers.getSetCookie();
const location = response.headers.get("Location");
if (!location?.startsWith("https://github.com/login/oauth/authorize")) return response;
const callback = new URL("/api/auth/callback/github", request.url);
callback.searchParams.set("code", "fake");
callback.searchParams.set("state", new URL(location).searchParams.get("state") ?? "");
const authorize = location?.startsWith("https://github.com/login/oauth/authorize");
if (!cookies.length && !authorize) return response;
const headers = new Headers(response.headers);
headers.set("Location", callback.href);
return new Response(null, { status: response.status, headers });
headers.delete("Set-Cookie");
for (const cookie of cookies) headers.append("Set-Cookie", env.COOKIES + cookie);
if (authorize) {
const callback = new URL("/api/auth/callback/github", request.url);
callback.searchParams.set("code", "fake");
callback.searchParams.set("state", new URL(location).searchParams.get("state") ?? "");
headers.set("Location", callback.href);
}
return new Response(response.body, {
status: response.status,
statusText: response.statusText,
headers,
webSocket: response.webSocket,
});
},
};`;

Expand All @@ -60,6 +99,7 @@ const mf = new Miniflare(
modules: true,
script: entry,
compatibilityDate: app.compatibilityDate,
bindings: { COOKIES: prefix },
serviceBindings: { APP: "app" },
},
{
Expand Down Expand Up @@ -123,10 +163,31 @@ for (const file of readdirSync(migrations)
const { cookie } = await seed({
fetch: (path, init) => mf.dispatchFetch(`${origin}${path}`, init as never) as unknown as Promise<Response>,
db,
output: (await mf.getR2Bucket("OUTPUT", "app")) as unknown as R2Bucket,
world: data,
origin,
secret,
});

if (values.check) {
// Over the listening socket, as a browser would reach it.
const problems: string[] = [];
const css = await fetch(`${origin}/site.css`);
if (!css.ok || !css.headers.get("Content-Type")?.startsWith("text/css"))
problems.push(`/site.css returned ${css.status} ${css.headers.get("Content-Type")}, not the built stylesheet`);
if (!cookie.startsWith(prefix)) problems.push(`the session cookie is not named after the port: ${cookie}`);
const inbox = async (sent: string) =>
(await fetch(`${origin}/inbox`, { headers: { Cookie: sent }, redirect: "manual" })).status;
const own = await inbox(cookie);
if (own !== 200) problems.push(`/inbox returned ${own} with the session cookie`);
// Another server's cookie, as a browser sends every localhost port's, is not this server's session.
const foreign = await inbox(cookie.slice(prefix.length));
if (foreign === 200) problems.push("/inbox accepted the session under another server's cookie name");
await mf.dispose();
for (const problem of problems) console.error(`Seeded server check failed: ${problem}`);
process.exit(problems.length ? 1 : 0);
}

const [login] = data.users.map((user) => user.login);
console.log(`
Seeded kix at ${origin}/inbox, signed in as ${login}. Ctrl-C stops it.
Expand Down
2 changes: 1 addition & 1 deletion cloudflare/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
"seeded": "wrangler deploy --dry-run --outdir .wrangler/seeded && bun dev/serve.ts",
"build": "bun scripts/build.ts",
"check": "bun run build && wrangler types --check && tsc --noEmit && prettier --check . && wrangler deploy --dry-run",
"test": "bun run build && vitest run",
"test": "bun run build && vitest run && bun run seeded --check",
"fix": "prettier --write .",
"types": "wrangler types",
"migrate": "wrangler d1 migrations apply DB --local",
Expand Down
83 changes: 83 additions & 0 deletions cloudflare/scripts/isolated.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
// Run each Worker test case in its own vitest process. The suite shares one
// runtime and resets only storage between tests (vitest.config.ts), so a test
// can still lean on module-level state another left behind. Each case alone is
// too slow for every PR (seconds of startup apiece), so a scheduled CI job runs
// this instead. Cases declared on one line, such as a loop's, run together.
//
// bun scripts/isolated.ts [--jobs N] [filter...]
//
// Filters keep the cases whose file path contains one of them. Run it after
// `bun run build`, as `vitest run` is.
import { spawn } from "node:child_process";
import { mkdtempSync, readFileSync, rmSync } from "node:fs";
import { availableParallelism, tmpdir } from "node:os";
import { join, relative } from "node:path";
import { parseArgs } from "node:util";

const root = join(import.meta.dirname, "..");
const { values, positionals: filters } = parseArgs({
options: { jobs: { type: "string", default: String(Math.max(1, Math.floor(availableParallelism() / 2))) } },
allowPositionals: true,
});
const jobs = Number(values.jobs);
if (!Number.isInteger(jobs) || jobs < 1) throw new Error(`--jobs must be a positive integer, not ${values.jobs}`);

/** Run vitest with `args` in the Worker's directory, returning its exit code and combined output. */
function vitest(args: string[]) {
return new Promise<{ code: number; output: string }>((resolve, reject) => {
const child = spawn("bun", ["x", "vitest", ...args], { cwd: root, stdio: ["ignore", "pipe", "pipe"] });
let output = "";
child.stdout.on("data", (chunk) => (output += chunk));
child.stderr.on("data", (chunk) => (output += chunk));
child.on("error", reject);
child.on("close", (code, signal) => resolve({ code: code ?? (signal ? 1 : 0), output }));
});
}

// Every case's location, from vitest's own collection.
const scratch = mkdtempSync(join(tmpdir(), "quest-isolated-"));
let cases: string[];
try {
const file = join(scratch, "cases.json");
const listed = await vitest(["list", `--json=${file}`]);
if (listed.code !== 0) {
process.stderr.write(listed.output);
throw new Error("vitest list failed");
}
const tests = JSON.parse(readFileSync(file, "utf8")) as { file: string; location?: { line: number } }[];
cases = [
...new Set(
tests.map((test) => {
if (!test.location) throw new Error(`vitest listed ${test.file} without a location`);
return `${relative(root, test.file)}:${test.location.line}`;
}),
),
].filter((location) => !filters.length || filters.some((filter) => location.includes(filter)));
} finally {
rmSync(scratch, { recursive: true, force: true });
}
if (!cases.length) throw new Error("No test cases matched");

console.log(`Running ${cases.length} test cases, each in its own process, ${jobs} at a time.`);
const failed: string[] = [];
let done = 0;
const queue = [...cases];
await Promise.all(
Array.from({ length: Math.min(jobs, queue.length) }, async () => {
for (let location = queue.shift(); location; location = queue.shift()) {
// A location matching no test fails too, so a stale listing cannot pass.
const { code, output } = await vitest(["run", location]);
done++;
if (code === 0) continue;
failed.push(location);
console.log(`\n✗ ${location} (${done}/${cases.length})\n${output}`);
}
}),
);

if (failed.length) {
console.log(`\n${failed.length} of ${cases.length} test cases failed on their own:`);
for (const location of failed.sort()) console.log(` ${location}`);
process.exit(1);
}
console.log(`All ${cases.length} test cases passed on their own.`);
Loading