TASK-706: daily first-user walk on prod (six of seven steps)

docs/runbooks/first-user-walk.md

@@ -0,0 +1,65 @@
+# First-user walk (daily, prod)
+
+`scripts/first-user-walk.mjs` (TASK-706) runs the path a stranger takes on
+commonly.me: sign up, get a hosted agent reply in the first pod, and invite a
+person whose link opens. Each step prints PASS or FAIL, its duration, and the
+first error text.
+
+## What a run creates
+
+- One non-admin account named `walk-<base36 timestamp>`, with the email
+  `walk-<…>@example.invalid`.
+- One team pod named `walk-<…> workspace`.
+- One hosted agent named `walk-<…>-agent`, with a message in that pod.
+- One invite link for that pod.
+
+The walk prefix `walk-` is how Growth's numbers exclude these accounts. Do not
+rename it without changing the exclusion list too.
+
+## Run it
+
+```bash
+COMMONLY_WALK_PASSWORD="$(openssl rand -base64 24)" \
+  node scripts/first-user-walk.mjs --base https://api.commonly.me
+```
+
+- Generate a fresh password per run. The script reads it from the environment
+  and never writes it anywhere. Do not put a credential in the command you
+  paste into a pod, a PR or a task row.
+- Exit codes: `0` every step passed, `1` a step failed or the account survived
+  cleanup, `2` the walk passed but a step is not yet covered (today, the
+  Bring-your-own connect step).
+
+## Cleanup is part of the run
+
+The walk deletes its own account in a `finally` block, sending the password and
+`confirmation: "DELETE"` to `DELETE /api/users/me`. If that delete fails, the
+run exits `1` and prints the error. A surviving account is a failed run, not a
+warning.
+
+The walk's token stops the hosted runtime, removes the hosted seat, every pod the
+account created (matched by `createdBy`, never a pod it did not make), and the
+account. If signup created the account but login failed, cleanup signs in once
+more; if that also fails, the account is reported as a survivor. Every request
+has a 20-second timeout, cleanup included. The cleanup then
+checks each one is gone: a pod still listed, or an account whose password still
+logs in, is a survivor. A run with survivors exits 1 and names them.
+
+Each run writes a manifest, ids only, to `~/.commonly/state/walk-706/<stamp>.json`
+(override with `COMMONLY_WALK_STATE_DIR`). It holds no password, token or email:
+the walk's user id, its agent name, its pod ids, and the `botIds` of every bot
+member of those pods, read before each pod is deleted (the hosted agent's User
+and the per-run Scout bot). SRE purges what the walk's token cannot reach from
+that manifest, by id, never by name prefix.
+
+Signup returns no user id, so the manifest records the run's exact username as
+soon as signup starts. If signup created the account and login then failed,
+cleanup reports `account <username>` as a survivor and the manifest is
+`needs-admin-purge`: SRE looks the account up by that exact name. The manifest
+never holds the email. Only a run that dies before its manifest write (before
+signup starts) leaves nothing to find, and that case needs no purge.
+
+## Daily schedule
+
+The CTO schedules the run on the host (launchd) and posts the result to the
+pod. Run it twice in a row before it counts as passing.

scripts/first-user-walk-cleanup.mjs

@@ -0,0 +1,98 @@
+// Cleanup for one first-user walk run (TASK-706). It runs with the walk
+// account's own token, because that is the only authority a run has, and it
+// removes what that token can reach: the hosted runtime, the seat, each pod the
+// account created, then the account. Anything it cannot remove or cannot
+// confirm is reported by class in `survivors`, so the caller fails loudly
+// instead of leaving it silent.
+//
+// Left for the admin purge, by design: Scout's per-user bot and the hosted
+// agent's own User. Account deletion keeps independent agent users, so the
+// run records their ids in the manifest and SRE purges them from there.
+
+const ids = (value) => (Array.isArray(value) ? value : value?.pods || []);
+
+// GET /api/pods returns createdBy either as an id or as a populated
+// { _id, ... } object, depending on the route. Accept both.
+export const ownerOf = (pod) => String(pod.createdBy?._id ?? pod.createdBy ?? '');
+
+// Only the bots this run created go in the manifest: the hosted agent (its
+// username carries this run's agent name) and the per-user Scout (`scout-u…`).
+// The shared default bot is a member of every new pod and is never recorded,
+// because SRE's purge would delete an identity every pod uses.
+export const isRunBot = (member, agentName) => {
+  const name = String(member?.username || '');
+  return member?.isBot === true && (name === agentName || name.startsWith(`${agentName}-`) || name.startsWith('scout-u'));
+};
+
+// `api` is the walk's request helper: (method, path, { token, body }) => { status, data }.
+// `podIds` are the pods the walk recorded as created in its manifest. They are
+// removed and verified even when the owner match misses them, so a filter
+// miss cannot turn into a silent survivor. `onBotIds(podId, botIds)` is called
+// before each pod is deleted, so a throw later in the loop cannot lose the ids.
+export const cleanupRun = async ({ api, token, password, email, userId, agentName, podIds = [], onBotIds = () => {} }) => {
+  const survivors = [];
+  const botIds = new Set();
+
+  const listOwnPods = async () => {
+    const list = await api('GET', '/api/pods', { token });
+    if (list.status !== 200) return null;
+    return ids(list.data).filter((pod) => ownerOf(pod) === String(userId));
+  };
+
+  // Stop the runtime before the seat goes: uninstalling the seat does not stop
+  // the hosted worker. 404 means there was no hosted install to stop.
+  const stop = await api('POST', '/api/hosted/deprovision', { token, body: { agentName } });
+  if (stop.status >= 300 && stop.status !== 404) survivors.push(`hosted:${agentName} (deprovision ${stop.status})`);
+
+  const owned = await listOwnPods();
+  if (owned === null) survivors.push('pods (list failed)');
+  const known = new Set(podIds.map(String));
+  for (const pod of owned || []) known.add(String(pod._id));
+
+  // Pods only: the walk never joins a pod it did not create, so every pod
+  // whose createdBy is this account is the run's own footprint. Bot members
+  // are read before the delete, because they outlive the account and the
+  // admin purge needs their ids. A lookup that fails is a survivor, and the
+  // pod is left alone: an empty member list must never read as "no bots".
+  for (const id of known) {
+    const detail = await api('GET', `/api/pods/${id}`, { token });
+    if (detail.status === 404) continue;
+    if (detail.status !== 200) {
+      survivors.push(`pod:${id} (lookup ${detail.status})`);
+      continue;
+    }
+    const bots = (detail.data?.members || []).filter((m) => isRunBot(m, agentName)).map((m) => String(m._id));
+    bots.forEach((bot) => botIds.add(bot));
+    onBotIds(id, bots);
+
+    const seat = await api('DELETE', `/api/registry/agents/${encodeURIComponent(agentName)}/pods/${id}?instanceId=default`, { token });
+    if (seat.status >= 300 && seat.status !== 404) survivors.push(`seat:${id} (uninstall ${seat.status})`);
+    const removed = await api('DELETE', `/api/pods/${id}`, { token });
+    if (removed.status >= 300 && removed.status !== 404) survivors.push(`pod:${id} (delete ${removed.status})`);
+  }
+
+  // Verify rather than trust the delete responses: a pod that is still
+  // listed under this account, or still readable by id, is a survivor
+  // whatever the delete said. A list that fails here cannot prove the pods
+  // are gone, so it is a survivor too.
+  const after = await listOwnPods();
+  if (after === null) survivors.push('pods (list failed on verify)');
+  for (const pod of after || []) survivors.push(`pod:${String(pod._id)} (still listed)`);
+  for (const id of known) {
+    const check = await api('GET', `/api/pods/${id}`, { token });
+    if (check.status === 200) survivors.push(`pod:${id} (still readable)`);
+    else if (check.status !== 404) survivors.push(`pod:${id} (verify ${check.status})`);
+  }
+
+  const del = await api('DELETE', '/api/users/me', { token, body: { password, confirmation: 'DELETE' } });
+  if (del.status >= 300) {
+    survivors.push(`account (delete ${del.status})`);
+  } else {
+    // The delete reply is not proof: the account is gone only if its own
+    // password no longer logs in.
+    const login = await api('POST', '/api/auth/login', { body: { email, password } });
+    if (login.status === 200) survivors.push('account (still logs in)');
+  }
+
+  return { ok: survivors.length === 0, survivors, botIds: [...botIds] };
+};

scripts/first-user-walk-cleanup.test.mjs

@@ -0,0 +1,196 @@
+// Run: node --test scripts/first-user-walk-cleanup.test.mjs
+import test from 'node:test';
+import assert from 'node:assert/strict';
+import { cleanupRun } from './first-user-walk-cleanup.mjs';
+
+const USER = 'user-walk';
+const OTHER = 'user-other';
+
+// A fake backend holding pods and one account. Options:
+//   stuck            pods whose delete returns 200 but leaves them listed
+//   accountFails     the account delete returns 500
+//   deleteIgnored    the account delete returns 200 but the account still logs in
+//   lookupFails      pods whose GET /api/pods/:id returns 500
+//   listFails        GET /api/pods returns 500
+//   seatFails        the registry seat uninstall returns 500
+//   deprovisionFails POST /api/hosted/deprovision returns 500
+//   throwOn          a DELETE to this pod path throws (network error)
+// Calls are recorded so the test can check the order.
+const fakeBackend = ({
+  pods, stuck = new Set(), accountFails = false, deleteIgnored = false,
+  lookupFails = new Set(), listFails = false, seatFails = false, deprovisionFails = false, throwOn = null,
+}) => {
+  const state = { pods: [...pods], accountGone: false, calls: [] };
+  const api = async (method, path) => {
+    const bare = path.split('?')[0];
+    state.calls.push(`${method} ${bare}`);
+    if (method === 'GET' && path === '/api/pods') {
+      if (listFails) return { status: 500, data: {} };
+      return { status: 200, data: state.pods.map((p) => ({ ...p })) };
+    }
+    const podMatch = path.match(/^\/api\/pods\/([^/?]+)$/);
+    if (method === 'GET' && podMatch) {
+      if (lookupFails.has(podMatch[1])) return { status: 500, data: { error: 'boom' } };
+      const pod = state.pods.find((p) => String(p._id) === podMatch[1]);
+      return pod ? { status: 200, data: pod } : { status: 404, data: {} };
+    }
+    if (method === 'DELETE' && podMatch) {
+      const id = podMatch[1];
+      if (throwOn === `/api/pods/${id}`) throw new Error('socket hang up');
+      if (!stuck.has(id)) state.pods = state.pods.filter((p) => String(p._id) !== id);
+      return { status: 200, data: {} };
+    }
+    if (method === 'DELETE' && path.startsWith('/api/registry/agents/')) {
+      if (seatFails) return { status: 500, data: {} };
+      return { status: 200, data: {} };
+    }
+    if (method === 'POST' && bare === '/api/hosted/deprovision') {
+      if (deprovisionFails) return { status: 500, data: {} };
+      return { status: 200, data: { deprovisioned: true } };
+    }
+    if (method === 'DELETE' && path === '/api/users/me') {
+      if (accountFails) return { status: 500, data: { error: 'boom' } };
+      if (!deleteIgnored) state.accountGone = true;
+      return { status: 200, data: {} };
+    }
+    if (method === 'POST' && path === '/api/auth/login') {
+      return state.accountGone ? { status: 401, data: {} } : { status: 200, data: { token: 't' } };
+    }
+    return { status: 404, data: {} };
+  };
+  return { api, state };
+};
+
+const run = (backend, opts = {}) => cleanupRun({
+  api: backend.api, token: 'tok', password: 'pw', email: 'walk@example.invalid', userId: USER, agentName: 'walk-x-agent', ...opts,
+});
+
+test('reports the bot members of each pod before the pod goes', async () => {
+  const members = [{ _id: 'bot-agent', isBot: true, username: 'walk-x-agent' }, { _id: USER, isBot: false, username: 'walk' }];
+  const backend = fakeBackend({ pods: [{ _id: 'p1', createdBy: USER, members }] });
+  const result = await run(backend);
+  assert.equal(result.ok, true);
+  assert.deepEqual(result.botIds, ['bot-agent']);
+});
+
+test('fails when the account still logs in after a 200 delete', async () => {
+  const backend = fakeBackend({ pods: [], deleteIgnored: true });
+  const result = await run(backend);
+  assert.equal(result.ok, false);
+  assert.ok(result.survivors.includes('account (still logs in)'), `survivors: ${result.survivors}`);
+});
+
+test('a clean run removes every pod the account created, then the account', async () => {
+  const backend = fakeBackend({ pods: [{ _id: 'p1', createdBy: USER }, { _id: 'p2', createdBy: USER }] });
+  const result = await run(backend);
+  assert.equal(result.ok, true);
+  assert.deepEqual(result.survivors, []);
+  assert.equal(backend.state.pods.length, 0);
+  assert.equal(backend.state.accountGone, true);
+});
+
+test('a pod the account did not create is never touched', async () => {
+  const backend = fakeBackend({ pods: [{ _id: 'p1', createdBy: USER }, { _id: 'pub', createdBy: OTHER }] });
+  const result = await run(backend);
+  assert.equal(result.ok, true);
+  assert.deepEqual(backend.state.pods.map((p) => p._id), ['pub']);
+  assert.ok(!backend.state.calls.includes('DELETE /api/pods/pub'));
+});
+
+test('fails and names the pod when a delete leaves it listed', async () => {
+  const backend = fakeBackend({ pods: [{ _id: 'p1', createdBy: USER }], stuck: new Set(['p1']) });
+  const result = await run(backend);
+  assert.equal(result.ok, false);
+  assert.ok(result.survivors.some((s) => s.startsWith('pod:p1')), `survivors: ${result.survivors}`);
+});
+
+test('fails and names the account when its delete is refused', async () => {
+  const backend = fakeBackend({ pods: [{ _id: 'p1', createdBy: USER }], accountFails: true });
+  const result = await run(backend);
+  assert.equal(result.ok, false);
+  assert.ok(result.survivors.includes('account (delete 500)'), `survivors: ${result.survivors}`);
+  assert.equal(backend.state.accountGone, false);
+});
+
+test('matches a populated createdBy object, not only a bare id', async () => {
+  const backend = fakeBackend({ pods: [{ _id: 'p1', createdBy: { _id: USER, username: 'walk' } }] });
+  const result = await run(backend);
+  assert.equal(result.ok, true);
+  assert.equal(backend.state.pods.length, 0);
+});
+
+test('removes a recorded pod the owner match misses, and checks it is gone', async () => {
+  const backend = fakeBackend({ pods: [{ _id: 'p1', createdBy: { _id: OTHER } }] });
+  const result = await run(backend, { podIds: ['p1'] });
+  assert.equal(result.ok, true);
+  assert.equal(backend.state.pods.length, 0);
+});
+
+test('fails and names a recorded pod that is still readable after delete', async () => {
+  const backend = fakeBackend({ pods: [{ _id: 'p1', createdBy: { _id: OTHER } }], stuck: new Set(['p1']) });
+  const result = await run(backend, { podIds: ['p1'] });
+  assert.equal(result.ok, false);
+  assert.ok(result.survivors.includes('pod:p1 (still readable)'), `survivors: ${result.survivors}`);
+});
+
+test('a failed pod lookup is a survivor, and the pod is not deleted', async () => {
+  const backend = fakeBackend({ pods: [{ _id: 'p1', createdBy: USER }], lookupFails: new Set(['p1']) });
+  const result = await run(backend);
+  assert.equal(result.ok, false);
+  assert.ok(result.survivors.includes('pod:p1 (lookup 500)'), `survivors: ${result.survivors}`);
+  assert.ok(!backend.state.calls.includes('DELETE /api/pods/p1'), 'a pod whose members are unknown must not be deleted');
+});
+
+test('a failed pod list is a survivor, not an empty list of pods', async () => {
+  const backend = fakeBackend({ pods: [{ _id: 'p1', createdBy: USER }], listFails: true });
+  const result = await run(backend);
+  assert.equal(result.ok, false);
+  assert.ok(result.survivors.includes('pods (list failed)'), `survivors: ${result.survivors}`);
+});
+
+test('a refused seat uninstall is a survivor', async () => {
+  const backend = fakeBackend({ pods: [{ _id: 'p1', createdBy: USER }], seatFails: true });
+  const result = await run(backend);
+  assert.equal(result.ok, false);
+  assert.ok(result.survivors.some((s) => s.startsWith('seat:p1 (uninstall 500)')), `survivors: ${result.survivors}`);
+});
+
+test('the hosted runtime is stopped before the seat and pods go', async () => {
+  const backend = fakeBackend({ pods: [{ _id: 'p1', createdBy: USER }] });
+  await run(backend);
+  const calls = backend.state.calls;
+  assert.ok(calls.indexOf('POST /api/hosted/deprovision') < calls.indexOf('DELETE /api/pods/p1'));
+});
+
+test('a refused deprovision is a survivor', async () => {
+  const backend = fakeBackend({ pods: [], deprovisionFails: true });
+  const result = await run(backend);
+  assert.equal(result.ok, false);
+  assert.ok(result.survivors.includes('hosted:walk-x-agent (deprovision 500)'), `survivors: ${result.survivors}`);
+});
+
+test('bot ids are reported before each pod is deleted, so a later throw keeps them', async () => {
+  const members = [{ _id: 'bot-scout', isBot: true, username: 'scout-uabc123' }];
+  const backend = fakeBackend({
+    pods: [{ _id: 'p1', createdBy: USER, members }, { _id: 'p2', createdBy: USER, members }],
+    throwOn: '/api/pods/p2',
+  });
+  const reported = [];
+  await assert.rejects(run(backend, { onBotIds: (podId, bots) => reported.push([podId, bots]) }), /socket hang up/);
+  assert.deepEqual(reported[0], ['p1', ['bot-scout']]);
+  assert.ok(backend.state.calls.indexOf('DELETE /api/pods/p1') > backend.state.calls.indexOf('GET /api/pods/p1'));
+});
+
+test('the shared default bot is never recorded, even though it is a pod member', async () => {
+  const members = [
+    { _id: 'shared-bot', isBot: true, username: 'commonly-bot' },
+    { _id: 'scout', isBot: true, username: 'scout-uabc123' },
+    { _id: 'agent', isBot: true, username: 'walk-x-agent' },
+    { _id: USER, isBot: false, username: 'walk' },
+  ];
+  const backend = fakeBackend({ pods: [{ _id: 'p1', createdBy: USER, members }] });
+  const result = await run(backend);
+  assert.equal(result.ok, true);
+  assert.deepEqual(result.botIds, ['scout', 'agent']);
+  assert.ok(!result.botIds.includes('shared-bot'));
+});