Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 4 additions & 5 deletions docs/design/artboards/d215/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,23 +6,22 @@ The spec for D215 (artboard v2). Open `index.html` in a browser. The toggle at t
- A second block under agent settings draws the states of that page (policy and machine, identity edit, connector grants, runtime with the model and effort selects, own API key) at 1440 and 390. The runtime and key frames follow V2RuntimeControls and V2AgentProviderKey, including the stored-model refusal and the key-save error. Those frames carry `data-state`, so `measure_after.js` skips them and `targets.json` does not move.
- Two state-only surfaces: `task-record` (the task record page, loaded and not-available, under the reply trace) and `handoff` (the OAuth and Discord return pages, `data-state` working, done, error, on the auth card). Every frame carries `data-state`, so the scorer skips them and `targets.json` does not move.
- A block under Settings draws the free-plan Plan row and the administrator-only Administration group (`data-state="admin-free"`, 1440 and 390, also skipped by the scorer).
- Under Activity and under Artifacts, a block redraws each page in the picked language (`#s-activity-signal`, `#s-artifacts-signal`): busy and day zero at 1440 and 390, built only from the seven picked surfaces' blocks. Activity's build scores its four frames (no `data-state`) and retired the traced `#s-activity` section. Artifacts' frames still carry `data-state="signal"`, so the scorer skips them until its build lands.
- Under Activity and under Artifacts, a block redraws each page in the picked language (`#s-activity-signal`, `#s-artifacts-signal`): busy and day zero at 1440 and 390, built only from the seven picked surfaces' blocks. Sam approved both on 11-03. Each page's build dropped the `data-state`, so the eight frames (`activity-busy`, `activity-dayzero`, `artifacts-busy`, `artifacts-dayzero`) are the scored ones, and retired the traced `#s-activity` and `#s-artifacts` sections and their CSS.
- The theme switch (sun / moon, D224) sits in the rail above the avatar at 1440. At 390 there is no rail, so the same switch rides the Pods top bar, left of Search. It is a two-item segment: the item for the theme in effect is the white key.
- `before/` holds prod captures from 2026-10-31, blurred so they show layout and density only.
- `targets.json` holds the helper-word and scroll targets per frame. `cap/measure_after.js` writes it from `index.html`.
- `cap/` holds the measuring scripts. See the header of `measure_after.js` for its env.
- Every frame without `data-state` must be listed in `measure_after.js`, either in `SURFACES` (scored) or in `SKIPPED` (not scored, with a reason). Activity, Artifacts and the two trace "before" frames are skipped. `node --test cap/measure_after.test.js` fails on any frame that is in neither list.
- Every frame without `data-state` must be listed in `measure_after.js`, either in `SURFACES` (scored) or in `SKIPPED` (not scored, with a reason). The two trace "before" frames, `run-trace-before` and `chat-trace-before`, draw the old pages and are skipped: they are never a target. On prod, `activity-busy` is measured at `/v2/activity` and `artifacts-busy` at `/v2/artifacts`; the two day-zero frames need an account with nothing in it, so they are artboard-only. The Artifacts prod check refuses the error line, the spinner and any list shorter than the frame's six rows. It then measures the first six rows, because prod returns up to 50 and the frame draws six. `node --test cap/measure_after.test.js` fails on any frame that is in neither list.

## Where each frame came from

Passing the prod check means a page matches its frame. It means the page matches the style Sam picked only when the frame came from his pick. Every frame is in one of four groups. A group's frames are scored only where the table says so, and prod scores are reported per group, never as one total.
Passing the prod check means a page matches its frame. It means the page matches the style Sam picked only when the frame came from his pick. Every frame is in one of three groups. A group's frames are scored only where the table says so, and prod scores are reported per group, never as one total.

| Group | Frames | Source |
|---|---|---|
| **From Sam's pick** | sidebar, chat, board, agent-memory, agent-settings, settings, connectors | The seven surfaces picked on 10-31 (#3059). Later edits: the theme switch (#3175) and the open thread (#3178). #3288 (the 390 agent-settings and settings frames, brought up to the shipped pages) and #3289 (memory notes drawn as the API holds them) moved those frames toward the live page, so read them with that in mind. |
| **Drawn after the pick** | dm, byo, byo-handoff, connect, team, firstrun-\*, signin, signup, signup-sent, verify-email, forgot-password, reset-password, cli-authorize, invite, invite-required, profile, agent-profile, run-trace-after, chat-trace, chat-agents-away, and the agent-settings and settings state blocks | New drawings in the picked language, from 11-01 on. Sam has not reviewed them as a set. |
| **Approved by Sam, 11-03** | activity-busy, activity-dayzero (`#s-activity-signal`, scored since the Activity build); artifacts-\* with `data-state="signal"`; task-record-\*; handoff-\* | Sam saw each one live-beside-proposed and answered "Build as drawn" for all four pages. Activity and Artifacts are built only from the picked surfaces' blocks. The scorer still skips these frames (they carry `data-state`). Each page's build PR makes its approved frame the scored one and retires the frame it replaces. |
| **Drawn from live, being retired** | artifacts-\* without `data-state` | Traced from the live page with small fixes (#3318). The scorer skips them (`SKIPPED` in `cap/measure_after.js`); scoring a page against them would only show it has not drifted from itself. The Artifacts build PR removes them; the Activity build removed its traced frames. The two before frames, `run-trace-before` and `chat-trace-before`, draw the old pages and are never a target. |
| **Approved by Sam, 11-03** | activity-busy, activity-dayzero (`#s-activity-signal`); artifacts-busy, artifacts-dayzero (`#s-artifacts-signal`); task-record-\*; handoff-\* | Sam saw each one live-beside-proposed and answered "Build as drawn" for all four pages. Activity and Artifacts are built only from the picked surfaces' blocks, and both are scored since their builds. Task record and handoff still carry `data-state`, so the scorer skips them. |

## Rules for D215 PRs

Expand Down
38 changes: 29 additions & 9 deletions docs/design/artboards/d215/cap/measure_after.js
Original file line number Diff line number Diff line change
Expand Up @@ -29,15 +29,12 @@ const SURFACES = [
'settings', 'agent-settings', 'agent-memory', 'board', 'connectors', 'chat', 'sidebar', 'byo', 'byo-handoff',
'firstrun-welcome', 'firstrun-waiting', 'firstrun-connected', 'firstrun-error', 'connect', 'team',
'chat-agents-away', 'chat-trace', 'run-trace-after', 'profile', 'firstrun-welcome-nohost',
// Activity: the frames Sam approved 11-03 ("Build as drawn"), built in V2ActivityPage.
'activity-busy', 'activity-dayzero',
// Activity and Artifacts: the frames Sam approved 11-03 ("Build as drawn"), built in
// V2ActivityPage and V2ArtifactsPage.
'activity-busy', 'activity-dayzero', 'artifacts-busy', 'artifacts-dayzero',
];
// Base frames (no data-state) that are drawn but deliberately not scored. Each needs a reason.
const SKIPPED = {
// Artifacts is being redesigned in another session; targets taken now would pin frames that
// are about to change. Score them once that redesign lands.
'artifacts-busy': 'redesign in progress elsewhere',
'artifacts-dayzero': 'redesign in progress elsewhere',
// Drawings of the old page beside their "after" frames (390 only). They are what the fix moves
// away from, so they are never a target.
'chat-trace-before': 'before shot, not a target',
Expand All @@ -47,8 +44,9 @@ const SKIPPED = {
// a real key. The first-run dialog is a latch-gated overlay, and the Scout welcome only exists in a
// brand-new pod. chat-agents-away and chat-trace need a pod in that exact state, run-trace-after a
// run id and profile a person id, none of which prod() takes yet, so they are artboard-only for now.
// activity-dayzero needs an account with no asks and no agents, so only the busy page is checked.
const PROD_SURFACES = ['settings', 'agent-settings', 'agent-memory', 'board', 'connectors', 'chat', 'sidebar', 'byo', 'connect', 'team', 'activity-busy'];
// activity-dayzero needs an account with no asks and no agents, and artifacts-dayzero one with no
// shared file, so only the two busy pages are checked on prod.
const PROD_SURFACES = ['settings', 'agent-settings', 'agent-memory', 'board', 'connectors', 'chat', 'sidebar', 'byo', 'connect', 'team', 'activity-busy', 'artifacts-busy'];
const VIEWPORTS = ['1440', '390'];
const EXPECTED = SURFACES.flatMap(s => VIEWPORTS.map(v => `${s}-${v}`));

Expand Down Expand Up @@ -192,6 +190,16 @@ async function checkSurface(p, [name, route, opt = {}], vp, targets) {
if (!(await p.locator(opt.list390.list).first().isVisible())) throw new Error(`${name}-${vp}: pods list ${opt.list390.list} is not visible`);
}
assertLanded(url, p.url());
if (opt.busy) {
const B = opt.busy;
if (await p.locator(B.refuse).first().isVisible()) throw new Error(`${name}-${vp}: the list did not load (${B.refuse} on screen in place of rows)`);
const n = await p.locator(B.rows).count();
if (n < B.keep) throw new Error(`${name}-${vp}: ${n} ${B.rows} rows on the page; the busy frame draws ${B.keep}, so this is not the busy state`);
await p.evaluate(([rows, group, keep]) => {
document.querySelectorAll(rows).forEach((el, i) => { if (i >= keep) el.remove(); });
document.querySelectorAll(group).forEach(g => { if (!g.querySelector(rows)) g.remove(); });
}, [B.rows, B.group, B.keep]);
}
const m = await p.evaluate(METRICS);
const key = `${name}-${vp}`, t = targets[key];
return { key, m, t, ok: m.helperWords <= t.helperWords && m.innerRatio <= t.innerRatio };
Expand All @@ -206,6 +214,17 @@ const SIDEBAR_390 = {
failed: '.v2-shell--list .v2-pods__empty:not(.v2-pods__list .v2-pods__empty)',
};

// Artifacts' busy frame draws six file rows; prod returns a page of up to 50. The scorer refuses
// the error line and the spinner, refuses a list shorter than the frame (an empty account or the
// day-zero page), and then measures the first six rows only, the way chat's scroll target is
// measured with its fold applied. Groups left with no rows go too.
const ARTIFACTS_BUSY = {
rows: '.v2a-file',
group: '.v2a-group',
refuse: '.v2a-state, .v2a-loading',
keep: 6,
};

async function prod(only) {
const { open } = require('./lib');
const pod = process.env.D215_POD, agent = process.env.D215_AGENT;
Expand All @@ -222,6 +241,7 @@ async function prod(only) {
['connect', () => '/v2/connect'],
['team', () => '/v2/agents'],
['activity-busy', () => '/v2/activity'],
['artifacts-busy', () => '/v2/artifacts', { busy: ARTIFACTS_BUSY }],
['chat', () => `/v2/pods/${pod}`],
['sidebar', vp => (vp === '1440' ? `/v2/pods/${side}` : '/v2'), { list390: SIDEBAR_390 }],
];
Expand All @@ -241,7 +261,7 @@ async function prod(only) {
process.exit(miss ? 1 : 0);
}

module.exports = { parseArgs, validateTargets, unknownFrameError, keyFrames, checkSurface, EXPECTED, SURFACES, SKIPPED, PROD_SURFACES, SIDEBAR_390 };
module.exports = { ARTIFACTS_BUSY, parseArgs, validateTargets, unknownFrameError, keyFrames, checkSurface, EXPECTED, SURFACES, SKIPPED, PROD_SURFACES, SIDEBAR_390 };

if (require.main === module) {
let args;
Expand Down
45 changes: 42 additions & 3 deletions docs/design/artboards/d215/cap/measure_after.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ const test = require('node:test');
const assert = require('node:assert');
const fs = require('fs');
const path = require('path');
const { parseArgs, validateTargets, unknownFrameError, keyFrames, checkSurface, EXPECTED, SURFACES, SKIPPED, PROD_SURFACES, SIDEBAR_390 } = require('./measure_after');
const { ARTIFACTS_BUSY, parseArgs, validateTargets, unknownFrameError, keyFrames, checkSurface, EXPECTED, SURFACES, SKIPPED, PROD_SURFACES, SIDEBAR_390 } = require('./measure_after');
const { assertLanded, BASE } = require('./lib');

const argv = (...a) => ['node', 'measure_after.js', ...a];
Expand All @@ -29,7 +29,7 @@ test('--only with an unknown surface throws', () => {

test('the committed targets.json passes its own validation', () => {
const t = JSON.parse(fs.readFileSync(path.join(__dirname, '..', 'targets.json'), 'utf8'));
assert.strictEqual(Object.keys(t).length, 44);
assert.strictEqual(Object.keys(t).length, 48);
validateTargets(t);
});

Expand All @@ -39,7 +39,7 @@ test('an empty or partial measurement is refused', () => {
const partial = { ...t }; delete partial['board-390'];
assert.throws(() => validateTargets(partial), /missing board-390/);
assert.throws(() => validateTargets({ ...t, 'chat-1440': { helperWords: NaN, innerRatio: 2 } }), /invalid chat-1440/);
assert.strictEqual(EXPECTED.length, 44);
assert.strictEqual(EXPECTED.length, 48);
});

test('a frame the scorer does not know is named, with the way to fix it', () => {
Expand Down Expand Up @@ -193,3 +193,42 @@ test('only the /v2 pod auto-select counts as landing', () => {
assert.throws(() => assertLanded('/v2/agents/a/default', BASE + '/v2'), /redirected to \/v2$/);
assert.throws(() => assertLanded('/v2/settings', BASE + '/v2/pods/abc123'), /redirected/);
});

// The Artifacts page on its route, with `rows` file rows and optionally the error line or the
// spinner on screen. `trimmed` records the trim the scorer asked for before measuring.
const fakeArtifacts = ({ rows, refused = false }) => {
let url = 'about:blank';
const page = {
trimmed: null,
measured: 0,
goto: async () => { url = BASE + '/v2/artifacts'; return { ok: () => true, status: () => 200 }; },
waitForTimeout: async () => {},
url: () => url,
locator: sel => {
if (sel === ARTIFACTS_BUSY.refuse) return { first: () => ({ isVisible: async () => refused }) };
if (sel === ARTIFACTS_BUSY.rows) return { count: async () => rows };
throw new Error(`unexpected locator ${sel}`);
},
evaluate: async (fn, arg) => {
if (arg) { page.trimmed = arg; return undefined; }
page.measured++;
return { helperWords: 0, innerRatio: 1 };
},
};
return page;
};

test('artifacts-busy scores only a loaded list as long as the frame, measured at the frame\'s six rows', async () => {
const t = JSON.parse(fs.readFileSync(path.join(__dirname, '..', 'targets.json'), 'utf8'));
const surf = ['artifacts-busy', () => '/v2/artifacts', { busy: ARTIFACTS_BUSY }];
const failed = fakeArtifacts({ rows: 0, refused: true });
await assert.rejects(checkSurface(failed, surf, '1440', t), /did not load/);
assert.strictEqual(failed.measured, 0);
const short = fakeArtifacts({ rows: 2 });
await assert.rejects(checkSurface(short, surf, '390', t), /2 \.v2a-file rows .* not the busy state/);
assert.strictEqual(short.measured, 0);
const busy = fakeArtifacts({ rows: 50 });
assert.strictEqual((await checkSurface(busy, surf, '1440', t)).ok, true);
assert.deepStrictEqual(busy.trimmed, ['.v2a-file', '.v2a-group', 6]);
assert.strictEqual(busy.measured, 1);
});
Loading
Loading