From f5d69cbb5873bc6c3ee8a5cf5f6204dd867353a2 Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Fri, 30 Oct 2026 23:29:36 -0700 Subject: [PATCH 1/3] docs(runbooks): a missing onboarding mark is no mark recorded, not proof Follow-up to #3039, from CodeRabbit's review. markPodOpened is fire-and-forget, so a failed write leaves the mark unset after a real visit. The marks are also recorded only for a pod the person created and only on accounts created after they shipped. And podOpenedAt records the first authorized messages fetch, set before the messages are read, not a render. The table row and the two bullets that read the marks now say so. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01N4VWLC9MiWprFWXFNjJSnr --- docs/runbooks/new-signup-stall.md | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/docs/runbooks/new-signup-stall.md b/docs/runbooks/new-signup-stall.md index 5bbd3c7270..5b0b3cfb42 100644 --- a/docs/runbooks/new-signup-stall.md +++ b/docs/runbooks/new-signup-stall.md @@ -7,23 +7,31 @@ on it?** The account already records the answer. ## Read the onboarding marks first `User.onboarding` carries write-once first-session marks, each set -server-side (`services/onboardingMarks.ts`). Accounts created before a mark -shipped don't have it; nothing is backfilled. +server-side (`services/onboardingMarks.ts`). + +**A missing mark means "no mark recorded", not proof the step didn't happen.** +The writes are fire-and-forget: `markPodOpened` calls `User.updateOne(...)` +without awaiting it, and a failed write leaves the mark unset after a real +visit. The marks are recorded only for a pod the person created, and only on +accounts created after the mark shipped; nothing is backfilled. Treat a gap as +a lead to confirm, for example against `lastActive` or the person's messages. | mark | set when | tells you | |---|---|---| -| `podOpenedAt` | the person first loads the messages of a pod they created, such as My Workspace | they reached the workspace and its welcome | +| `podOpenedAt` | the first authorized fetch of the messages of a pod they created, such as My Workspace. It is set before the messages are read, so it records the request, not a render | their client asked for the workspace's messages, so the welcome was very likely on its way to the screen | | `firstStartAt` + `firstStartId` | their first message in a pod they created matches a start's label, or the server resolves a click on their own "Run an agent here" | they acted, and which choice | | `byoOpenedAt` | they first load Bring your own (`GET /api/hosted/availability`) | they looked at setup | Read them together with the person's first own message in their workspace. -- **No `podOpenedAt`**: they never reached the screen. Look at the signup and - redirect path. +- **No `podOpenedAt`**: no messages fetch was recorded for their own pod. They + may never have reached the workspace, or the write may have failed. Check + `lastActive` first, then look at the signup and redirect path. - **`podOpenedAt` but no `firstStartAt`, no `byoOpenedAt` and no message**: - they saw the welcome and did nothing. That is a copy or comprehension - question, not a render bug. Compare against people who did act: how soon - after `podOpenedAt` did they click? + most likely they saw the welcome and did nothing. If a staging walk shows the + render is clean, treat it as a copy or comprehension question, not a render + bug. Compare against people who did act: how soon after `podOpenedAt` did + they click? - **`firstStartAt` but nothing after it**: the click is the place to look, above all `run-hosted`. It sends no message, and the mark is set when the server resolves the click, before the install and provision run. So a From 90d93b569941c43267aced633a81a7888393c8b7 Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Fri, 30 Oct 2026 23:35:35 -0700 Subject: [PATCH 2/3] docs(runbooks): no createdAt check on the marks; podOpenedAt doesn't prove the welcome showed From CodeRabbit's review of #3040. markPodOpened never checks when the account was created, so an older account gets the mark on its first eligible request after the deploy, and the timestamp can postdate the real first visit. podOpenedAt is written before the messages are listed, so rule out a failed fetch or render before calling a stall a copy question. And reading "the person's messages" now says whether they sent one, by timestamps and ids, to match the no-message-text rule below it. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01N4VWLC9MiWprFWXFNjJSnr --- docs/runbooks/new-signup-stall.md | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/docs/runbooks/new-signup-stall.md b/docs/runbooks/new-signup-stall.md index 5b0b3cfb42..63ba08f8fc 100644 --- a/docs/runbooks/new-signup-stall.md +++ b/docs/runbooks/new-signup-stall.md @@ -12,9 +12,12 @@ server-side (`services/onboardingMarks.ts`). **A missing mark means "no mark recorded", not proof the step didn't happen.** The writes are fire-and-forget: `markPodOpened` calls `User.updateOne(...)` without awaiting it, and a failed write leaves the mark unset after a real -visit. The marks are recorded only for a pod the person created, and only on -accounts created after the mark shipped; nothing is backfilled. Treat a gap as -a lead to confirm, for example against `lastActive` or the person's messages. +visit. The marks are recorded only for a pod the person created. Nothing is +backfilled, and nothing checks when the account was created: an account older +than a mark gets it on its first eligible request after the deploy, so its +timestamp can be later than the real first visit. Treat a gap as a lead to +confirm, for example against `lastActive` or whether they sent a message +(timestamps and ids). | mark | set when | tells you | |---|---|---| @@ -22,16 +25,19 @@ a lead to confirm, for example against `lastActive` or the person's messages. | `firstStartAt` + `firstStartId` | their first message in a pod they created matches a start's label, or the server resolves a click on their own "Run an agent here" | they acted, and which choice | | `byoOpenedAt` | they first load Bring your own (`GET /api/hosted/availability`) | they looked at setup | -Read them together with the person's first own message in their workspace. +Read them together with whether the person sent a message in their workspace, +and when (timestamps and ids). - **No `podOpenedAt`**: no messages fetch was recorded for their own pod. They may never have reached the workspace, or the write may have failed. Check `lastActive` first, then look at the signup and redirect path. - **`podOpenedAt` but no `firstStartAt`, no `byoOpenedAt` and no message**: - most likely they saw the welcome and did nothing. If a staging walk shows the - render is clean, treat it as a copy or comprehension question, not a render - bug. Compare against people who did act: how soon after `podOpenedAt` did - they click? + the request for the welcome was made, but the mark is written before the + messages are listed, so it doesn't prove the welcome showed. First rule out + a failed fetch or render: walk it on staging, and check the backend error + logs around `podOpenedAt`. Only then treat it as a copy or comprehension + question. Compare against people who did act: how soon after `podOpenedAt` + did they click? - **`firstStartAt` but nothing after it**: the click is the place to look, above all `run-hosted`. It sends no message, and the mark is set when the server resolves the click, before the install and provision run. So a From a16858c3467679a4621befe4d6df753165f86f96 Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Fri, 30 Oct 2026 23:43:55 -0700 Subject: [PATCH 3/3] docs(runbooks): scope each mark exactly; the account records evidence, not the answer From CodeRabbit's second review of #3040. The creator-pod condition applies to podOpenedAt and a typed first start, not to byoOpenedAt, which has no pod check. podOpenedAt is the first FIRST-PAGE fetch (no `before` cursor) of any pod the person created, not necessarily My Workspace. "An account older than a mark" becomes "created before that mark's deploy". And the opener now says the account usually records evidence, which agrees with the failed-write caveat. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01N4VWLC9MiWprFWXFNjJSnr --- docs/runbooks/new-signup-stall.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/runbooks/new-signup-stall.md b/docs/runbooks/new-signup-stall.md index 63ba08f8fc..9b21116288 100644 --- a/docs/runbooks/new-signup-stall.md +++ b/docs/runbooks/new-signup-stall.md @@ -2,7 +2,7 @@ A new person signs up and sends nothing. Before you walk the flow or change copy, settle one question: **did they reach the first screen, and did they act -on it?** The account already records the answer. +on it?** The account usually records evidence of both. ## Read the onboarding marks first @@ -12,16 +12,17 @@ server-side (`services/onboardingMarks.ts`). **A missing mark means "no mark recorded", not proof the step didn't happen.** The writes are fire-and-forget: `markPodOpened` calls `User.updateOne(...)` without awaiting it, and a failed write leaves the mark unset after a real -visit. The marks are recorded only for a pod the person created. Nothing is -backfilled, and nothing checks when the account was created: an account older -than a mark gets it on its first eligible request after the deploy, so its +visit. `podOpenedAt` and a typed `firstStartAt` count only in a pod the +person created; `byoOpenedAt` has no pod check. Nothing is backfilled, and +nothing checks when the account was created: an account created before that +mark's deploy gets it on its first eligible request afterwards, so its timestamp can be later than the real first visit. Treat a gap as a lead to confirm, for example against `lastActive` or whether they sent a message (timestamps and ids). | mark | set when | tells you | |---|---|---| -| `podOpenedAt` | the first authorized fetch of the messages of a pod they created, such as My Workspace. It is set before the messages are read, so it records the request, not a render | their client asked for the workspace's messages, so the welcome was very likely on its way to the screen | +| `podOpenedAt` | the first authorized first-page fetch (no `before` cursor) of the messages of any pod they created; for a new signup that is normally My Workspace. It is set before the messages are read, so it records the request, not a render | their client asked for a pod's messages; for a new signup, very likely the welcome | | `firstStartAt` + `firstStartId` | their first message in a pod they created matches a start's label, or the server resolves a click on their own "Run an agent here" | they acted, and which choice | | `byoOpenedAt` | they first load Bring your own (`GET /api/hosted/availability`) | they looked at setup | @@ -32,7 +33,8 @@ and when (timestamps and ids). may never have reached the workspace, or the write may have failed. Check `lastActive` first, then look at the signup and redirect path. - **`podOpenedAt` but no `firstStartAt`, no `byoOpenedAt` and no message**: - the request for the welcome was made, but the mark is written before the + a first-page fetch of a pod they created was made (for a new signup, + normally My Workspace, where the welcome is). The mark is written before the messages are listed, so it doesn't prove the welcome showed. First rule out a failed fetch or render: walk it on staging, and check the backend error logs around `podOpenedAt`. Only then treat it as a copy or comprehension