Add verified Gmail drafts and agent-guided self-hosted OAuth setup

.gitignore

@@ -8,4 +8,8 @@ coverage/
 !.dev.vars.example
 .env*
 *.log
+client_secret*.json
+credentials.json
+token.json
+gmail-state/
 .DS_Store

README.md

@@ -15,22 +15,29 @@ The name plays on “read pixels”: the past tense of “read” sounds like 
 
 ## Set up with your agent
 
-Choose **one** prompt for an agent with terminal access and permission to use your Cloudflare account. New installations can use Cloudflare's [cf CLI](https://developers.cloudflare.com/cf/get-started/) (currently beta); the Wrangler path remains available. Both use the same Worker code and SQLite storage. Gmail connection is a separate sign-in step if the agent is not connected yet.
+Choose **one** prompt for an agent with terminal access and permission to use your accounts. New installations can use Cloudflare's [cf CLI](https://developers.cloudflare.com/cf/get-started/) (currently beta); the Wrangler path remains available. Both use the same Worker code and SQLite storage. The standard self-hosted Gmail path uses the bundled CLI and **your own Google Cloud project and Desktop OAuth client**. A compatible existing Gmail connector is an [optional alternative](docs/gmail-connectors.md).
 
 ### cf CLI
 
 ```text
 Set up redpixels using cf by following:
 https://raw.githubusercontent.com/arkorlab/redpixels/main/docs/agent-setup-cf.md
 
-Deploy it to my Cloudflare account and connect it to my agent's HTML Gmail
-draft workflow. Reuse existing cf credentials, generate and store secrets
-securely, and preserve unrelated settings. Do the work directly; ask only
-for required sign-in, account selection, or missing information.
-
-You may send exactly one clearly labeled setup test email to my authenticated
-Gmail address. Report the selected CLI, Worker URL, persistent draft integration
-location, test results, and webhook status.
+Deploy it to my Cloudflare account, then set up the local Gmail CLI following:
+https://raw.githubusercontent.com/arkorlab/redpixels/main/docs/agent-setup-gmail.md
+
+Reuse appropriate existing settings or create a dedicated Google Cloud project
+and Desktop OAuth client for my personal use. Enable only Gmail API; do not
+enable billing, start a paid trial, or request increased quotas. Reuse cf
+credentials and preserve unrelated apps and settings. Store secrets securely,
+outside Git and chat. Do the work directly; leave required sign-in, passkeys,
+terms acceptance and OAuth consent to me.
+
+Create one labeled test draft to my authenticated Gmail address and verify
+the saved HTML image. Let me review and press Send. Persist the verified draft
+workflow for future requests. Explain manual arming versus immediate tracking;
+do not claim automatic tracking of ordinary Gmail-composed mail. Report the
+CLI, Worker URL, configuration locations, test evidence and webhook status.
 ```
 
 ### Wrangler
@@ -39,22 +46,33 @@ location, test results, and webhook status.
 Set up redpixels using Wrangler by following:
 https://raw.githubusercontent.com/arkorlab/redpixels/main/docs/agent-setup-wrangler.md
 
-Deploy it to my Cloudflare account and connect it to my agent's HTML Gmail
-draft workflow. Reuse existing Wrangler credentials, generate and store secrets
-securely, and preserve unrelated settings. Do the work directly; ask only
-for required sign-in, account selection, or missing information.
-
-You may send exactly one clearly labeled setup test email to my authenticated
-Gmail address. Report the selected CLI, Worker URL, persistent draft integration
-location, test results, and webhook status.
+Deploy it to my Cloudflare account, then set up the local Gmail CLI following:
+https://raw.githubusercontent.com/arkorlab/redpixels/main/docs/agent-setup-gmail.md
+
+Reuse appropriate existing settings or create a dedicated Google Cloud project
+and Desktop OAuth client for my personal use. Enable only Gmail API; do not
+enable billing, start a paid trial, or request increased quotas. Reuse Wrangler
+credentials and preserve unrelated apps and settings. Store secrets securely,
+outside Git and chat. Do the work directly; leave required sign-in, passkeys,
+terms acceptance and OAuth consent to me.
+
+Create one labeled test draft to my authenticated Gmail address and verify
+the saved HTML image. Let me review and press Send. Persist the verified draft
+workflow for future requests. Explain manual arming versus immediate tracking;
+do not claim automatic tracking of ordinary Gmail-composed mail. Report the
+CLI, Worker URL, configuration locations, test evidence and webhook status.
 ```
 
 If you already have a compatible HTTPS webhook receiver, append its URL and whether you want `first` or `every` notifications. Otherwise the agent can finish history tracking and report webhooks as disabled. Keep API keys in your agent's secret store, not in the prompt.
 
-The [agent setup guide](docs/agent-setup.md) covers deployment, secrets, persistent draft integration, and live verification. Deploying redpixels alone does not make ordinary Gmail-composed emails trackable; the drafting workflow must embed the returned pixel. The manual steps follow below.
+Already deployed? Give your agent the [Gmail-only setup prompt and detailed procedure](docs/agent-setup-gmail.md). It covers project creation, personal versus Workspace accounts, OAuth consent, protected credentials, recovery, costs, and an end-to-end test. The [common agent guide](docs/agent-setup.md) connects this with Cloudflare deployment.
+
+**Current workflow:** the agent creates a pixel-bearing draft; you review and send it in Gmail. Ordinary mail started from Gmail's blank compose screen is not automatically tracked. The CLI defaults to suppressing notifications until a manual post-send arming step; `--armed` starts immediately but can count your previews. Gmail Sent detection is not implemented. Google Cloud setup does not remove these limitations.
 
 ## Deploy in a few minutes
 
+This section deploys the tracking Worker. First-time Gmail OAuth setup is separate and requires additional Google Console steps.
+
 You need a Cloudflare account, Node.js **22.18+ or 24 LTS**, and [pnpm](https://pnpm.io/installation) 10. Both CLIs are pinned locally; no global CLI installation is needed. SQLite Durable Object storage is provisioned by the configuration, without a manual database ID, KV namespace, or external database. Cloudflare usage limits and charges apply; see [Durable Objects pricing](https://developers.cloudflare.com/durable-objects/platform/pricing/).
 
 ```sh
@@ -123,14 +141,50 @@ Use a new `Idempotency-Key` per email. Retrying the same key and body returns th
 
 ## Gmail draft integration
 
+### Standard self-hosted setup: local Gmail CLI
+
+Use the bundled raw MIME CLI with **your own Google Cloud project and Desktop OAuth client**. It registers the message, inserts the pixel, submits `message.raw` directly to Gmail, and verifies the saved HTML MIME part. It creates drafts only. The Worker never receives Google credentials. An agent can follow the [complete Gmail setup instructions](docs/agent-setup-gmail.md); manual instructions are in the [CLI guide](docs/gmail-cli.md). Do not extract private tokens from an agent's existing connection to configure it.
+
+After [one-time Desktop OAuth setup](docs/gmail-cli.md), an agent can run:
+
+```sh
+# REDPIXELS_URL and REDPIXELS_API_TOKEN (or REDPIXELS_SECRETS_FILE) configured locally
+pnpm --silent gmail draft --to alice@example.com --subject 'Hello' \
+  --html examples/email.html --operation hello-001
+```
+
+The JSON result includes the tracking/draft IDs and `state:"verified"`. Drafts default to unarmed: **review and press Send in Gmail**, then run:
+
+```sh
+pnpm --silent gmail arm --operation hello-001 --confirm-sent
+```
+
+Pre-arm fetches remain in history with `before_armed` but do not notify. Sender previews after arming and proxy caching cannot be eliminated. Use a fresh message for each Web/iOS acceptance test. See the [CLI setup, OAuth storage, recovery, limitations and manual test steps](docs/gmail-cli.md). Google's official `google-auth-library` handles OAuth; OS credential storage is the default, with an explicit protected-file option for headless use.
+
+In a live trial on 2026-10-16, this CLI completed Desktop OAuth, stored credentials in macOS Keychain, and verified two saved raw-MIME drafts. After human sending and arming, separate recipient Gmail Web/iOS tests each produced a new recorded fetch and a matching signature-verified webhook (HTTP 204). See the [trial scope and remaining gaps](docs/gmail-cli.md#manual-acceptance-test).
+
+### Optional: reuse an existing Gmail connection
+
+A connected Gmail tool that preserves MIME content and can read back the saved raw message can avoid a separate Google Cloud/OAuth setup. Register with `"armed":false`, insert `pixel_html`, create through the MIME/raw byte path, and verify the actual saved image before reporting success. Keep both IDs, let the human send, then [arm the message](#draft-previews-and-arming). An `htmlBody` field alone is not proof of compatibility: some connectors strip images. See the [capability checks and manual test](docs/gmail-connectors.md).
+
+On 2026-10-16, an existing Codex Gmail connector's `mime_type:"text/html"` and `body.base64_url_content` path created an **unsent** draft whose raw HTML matched the input, including the image. That trial verified saved drafts only; sending and recipient opens were tested separately with the CLI above.
+
+### Google Cloud costs and a future managed edition
+
+For the current local CLI, standard Gmail API use has no additional charge, and **billing/card registration is not required**. No GCP server or Pub/Sub is used. A normal new draft uses 31 Gmail quota units including saved-draft verification; 100 drafts/day would use about 3,100 units before retries or extra checks. Google's published daily threshold is 80 million units/project. Google plans charges for over-standard usage later in 2026 with at least 90 days' notice; this is not a promise of permanent unlimited free use. Cloudflare usage and any existing Workspace or agent subscription are separate. See the [official quota/pricing policy](https://developers.google.com/workspace/gmail/api/reference/quota) and [setup cost details](docs/agent-setup-gmail.md#costs-and-prerequisites-to-explain).
+
+A future managed edition could handle deployment and the operator's OAuth application so users only authorize Gmail. It is not included in this release. Hosting alone does not add automatic pixel insertion to Gmail's compose screen or automatic send detection; those need separate integration work.
+
+### Existing raw-MIME workflows and low-level example
+
 1. Register the email with redpixels.
 2. Insert `pixel_html` into the HTML MIME part.
 3. Base64url-encode the complete MIME message and call Gmail `users.drafts.create`.
 4. Review and send in Gmail. Preserve the pixel whenever replacing a draft.
 
-[Google's draft guide](https://developers.google.com/workspace/gmail/api/guides/drafts) documents MIME and base64url encoding. redpixels does not read your inbox, obtain Gmail OAuth access, or send mail.
+[Google's draft guide](https://developers.google.com/workspace/gmail/api/guides/drafts) documents MIME and base64url encoding. The Worker does not read your inbox or obtain Gmail OAuth access. The local CLI obtains draft access on your machine; neither it nor the Worker sends mail.
 
-The dependency-free [Node example](examples/gmail-draft.mts) creates a draft without sending it:
+The existing dependency-free [low-level example](examples/gmail-draft.mts) remains available for workflows supplying a short-lived access token. It does not manage refresh tokens or verify the saved draft; use the CLI or a verified connector for agent use:
 
 ```sh
 export GMAIL_ACCESS_TOKEN='<OAuth access token with gmail.compose permission>'
@@ -141,11 +195,11 @@ export GMAIL_TEXT='Hello Alice!'
 node --experimental-strip-types examples/gmail-draft.mts examples/email.html
 ```
 
-Replace [examples/email.html](examples/email.html) with your own HTML file when ready. Use your assistant's existing Gmail authorization; if you do not have one, configure Gmail API OAuth first using [Google's scope guide](https://developers.google.com/workspace/gmail/api/auth/scopes). A redpixels deployment does not supply Gmail authorization. Gmail credentials are used only by this local example; never put them in Worker variables. On Gmail failure/timeout, the example retains and prints the redpixels ID for recovery. Re-running can create another Gmail draft; `REDPIXELS_IDEMPOTENCY_KEY` deduplicates redpixels registration only, not Gmail draft creation.
+Replace [examples/email.html](examples/email.html) with your own HTML file when ready. Use a token explicitly supplied through a supported credential flow; never extract private tokens from a connected agent tool. This low-level example needs separate Gmail API authorization as described in [Google's scope guide](https://developers.google.com/workspace/gmail/api/auth/scopes). A redpixels deployment does not supply Gmail authorization. Gmail credentials are used only by this local example; never put them in Worker variables. On Gmail failure/timeout, the example retains and prints the redpixels ID for recovery. Re-running can create another Gmail draft; `REDPIXELS_IDEMPOTENCY_KEY` deduplicates redpixels registration only, not Gmail draft creation.
 
 ### Draft previews and arming
 
-By default tracking starts immediately. A preview can fetch before sending. If your workflow can act after sending, register with `"armed":false`, then call:
+The HTTP API defaults to immediate tracking; the local CLI defaults to unarmed drafts. A preview can fetch before sending. If your API workflow can act after sending, register with `"armed":false`, then call:
 
 ```sh
 curl --fail-with-body -X POST "$REDPIXELS_URL/v1/messages/<id>/arm" \
@@ -325,6 +379,8 @@ Local development uses local storage and does not deploy. Owner cookies require
 
 `pnpm check` runs strict TypeScript checks for the Worker, tests, Node examples, and both CLI configurations. Runtime types are generated by the pinned Wrangler version; the generated file is ignored by Git. `src/types.ts` defines the public open records, flags, API responses, and webhook payload; `src/storage-types.ts` keeps SQLite row types separate. Types do not replace the existing runtime input validation.
 
+Node tests also cover CLI raw-MIME submission/read-back, stripped images, OAuth PKCE/state/refresh with the real Google SDK and mocked HTTP, account checks, private storage and uncertain-write recovery. These do not establish real Gmail OAuth or recipient viewing for the new CLI.
+
 Tests use the Workers runtime and real local SQLite Durable Objects: concurrent counts/first notifications, auth, idempotency, pagination, classification, owner signals, eviction, deletion, signatures and bounded retries. Node tests cover receiver verification and MIME construction. CI builds both CLI paths with deployment dry runs and checks that their Worker identity, SQLite bindings, and runtime bundles match. These do not verify production deployment, Gmail delivery, mobile editing or real MPP accuracy.
 
 See [architecture](docs/architecture.md). This release is single-owner self-hosted OSS. Authentication, storage, classification and delivery have separate boundaries for later tenant/key resolution and metering. Billing and multi-user management are outside this release.

cli/contracts.mts

@@ -0,0 +1,143 @@
+export const GMAIL_SCOPE = "https://www.googleapis.com/auth/gmail.compose";
+export const GMAIL_API = "https://gmail.googleapis.com/gmail/v1/users/me";
+export type Fetcher = (url: string, init?: RequestInit) => Promise<Response>;
+
+export interface OAuthClient {
+  clientId: string;
+  clientSecret: string;
+}
+export interface GmailCredential extends OAuthClient {
+  version: 1;
+  email: string;
+  refreshToken: string;
+  scopes: string[];
+}
+export interface CredentialStore {
+  read(): Promise<GmailCredential | null>;
+  write(value: GmailCredential): Promise<void>;
+  remove(): Promise<void>;
+}
+export interface GmailPart {
+  mimeType?: string;
+  body?: { data?: string };
+  headers?: { name: string; value: string }[];
+  parts?: GmailPart[];
+}
+export interface GmailDraft {
+  id: string;
+  message: { id: string; payload?: GmailPart };
+}
+export type OperationState =
+  | "registered"
+  | "creating"
+  | "created"
+  | "verified"
+  | "verification_failed"
+  | "ambiguous";
+export interface DraftOperation {
+  version: 1;
+  operation: string;
+  fingerprint: string;
+  account: string;
+  trackerOrigin: string;
+  trackingId: string;
+  pixelUrl: string;
+  pixelHtml: string;
+  armed: boolean;
+  state: OperationState;
+  draftId?: string;
+  gmailMessageId?: string;
+  createdAt: string;
+  verifiedAt?: string;
+}
+export interface DraftInput {
+  operation: string;
+  to: string;
+  subject: string;
+  html: string;
+  text: string;
+  armed: boolean;
+}
+export interface TrackerSettings {
+  origin: string;
+  token: string;
+}
+
+export function object(value: unknown): Record<string, unknown> {
+  if (!value || typeof value !== "object" || Array.isArray(value))
+    throw new Error("Invalid JSON object in service response or local state");
+  return value as Record<string, unknown>;
+}
+export function string(value: unknown, label: string): string {
+  if (typeof value !== "string" || !value || value.length > 16384)
+    throw new Error(`Invalid ${label}`);
+  return value;
+}
+export function email(value: unknown): string {
+  const address = string(value, "email address");
+  if (
+    address.length > 254 ||
+    !/^[a-zA-Z0-9.!#$%&'*+/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9.-]*[a-zA-Z0-9])?$/.test(
+      address,
+    )
+  )
+    throw new Error(
+      "Use one bare email address, without display names or lists",
+    );
+  return address;
+}
+export function operationName(value: string): string {
+  if (!/^[a-zA-Z0-9._-]{1,128}$/.test(value))
+    throw new Error(
+      "Operation must be 1–128 letters, digits, dots, underscores or hyphens",
+    );
+  return value;
+}
+export class HttpFailure extends Error {
+  readonly status: number;
+  constructor(status: number, service: string) {
+    super(`${service} returned HTTP ${status}`);
+    this.status = status;
+  }
+}
+export async function requestJson(
+  fetcher: Fetcher,
+  url: string,
+  init: RequestInit,
+  service: string,
+): Promise<Record<string, unknown>> {
+  let response: Response;
+  try {
+    response = await fetcher(url, {
+      ...init,
+      redirect: "error",
+      signal: init.signal ?? AbortSignal.timeout(30000),
+    });
+  } catch {
+    throw new Error(`${service} request failed or timed out`);
+  }
+  if (!response.ok) {
+    await response.body?.cancel();
+    throw new HttpFailure(response.status, service);
+  }
+  // Never include response bodies or credential values in errors.
+  const reader = response.body?.getReader();
+  if (!reader) throw new Error(`${service} returned an empty response`);
+  const chunks: Uint8Array[] = [];
+  let size = 0;
+  for (;;) {
+    const { value, done } = await reader.read();
+    if (done) break;
+    size += value.length;
+    if (size > 12 * 1024 * 1024) {
+      await reader.cancel();
+      throw new Error(`${service} response exceeds 12 MiB`);
+    }
+    chunks.push(value);
+  }
+  try {
+    return object(JSON.parse(Buffer.concat(chunks).toString("utf8")));
+  } catch {
+    throw new Error(`${service} returned invalid JSON`);
+  }
+}