wasimxyz · Oct 16, 2026 · Oct 16, 2026
diff --git a/.cursor/rules/code-comments.mdc b/.cursor/rules/code-comments.mdc
diff --git a/.cursor/rules/run-checks.mdc b/.cursor/rules/run-checks.mdc
diff --git a/AGENTS.md b/AGENTS.md
@@ -1,79 +1,63 @@
 # AGENTS.md
 
-## Commits and pull requests
+Data Hub collects files from lab instrument PCs (`watcher/`), processes them in AWS Lambda (`lambda/`), and serves them through a Next.js web app, REST API, and MCP server (`web/`). Developer docs start at `docs/developer/README.md`.
 
-Do **not** put sensitive information from internal environments (production, staging, or a coworker's machine) into commit messages, PR titles/bodies, review comments, or committed files. That includes:
+## Commits and pull requests
 
-- Credentials and secrets: PATs (`dhub_…`), `AUTH_SECRET`, AWS keys/session tokens, database passwords, webhook secrets
-- Pre-signed or authenticated URLs (S3 download links, anything with `X-Amz-Security-Token` / signature query params)
-- Identifiers from internal environments: instrument IDs, run IDs, file IDs, and real filenames from those runs
-- Raw dumps from internal MCP tools, API responses, or logs that embed the above
-- `.env` contents or other gitignored config copied into the diff
+Keep sensitive information from internal environments (production, staging, or a coworker's machine) out of commit messages, pull request titles and bodies, review comments, and committed files. That includes:
 
-Safe to reference: public docs URLs, redacted error messages, and synthetic fixtures. When a PR needs to describe a production incident, summarize the failure mode — don't paste tokens, signed URLs, IDs, or full internal payloads.
+- Credentials and secrets: personal access tokens (`dhub_…`), `AUTH_SECRET`, AWS keys and session tokens, database passwords, and webhook secrets.
+- Pre-signed or authenticated URLs, such as S3 download links or anything with `X-Amz-Security-Token` or signature query parameters.
+- Identifiers from internal environments: instrument IDs, run IDs, file IDs, and real filenames from those runs.
+- Raw output from internal MCP tools, API responses, or logs that contains any of the above.
+- `.env` contents or other gitignored config.
 
-## Documentation
+Public docs URLs, redacted error messages, and synthetic fixtures are safe. To describe a production incident, summarize the failure mode.
 
-Documentation lives in `docs/`, split by who is reading:
+## Changelog
 
-- `docs/site/` is the public docs site (Fumadocs on Next.js) at https://datahub.arcadiascience.com/docs. It covers installing a watcher, setting up an instrument, managing tokens, security and permissions, and the API, MCP, and watcher CLI reference. Pages are MDX files in `docs/site/content/docs/`. Read `docs/site/AGENTS.md` before editing it. For "how do I use Data Hub" questions, search the published site first and don't rely on training data or guess at UI flows: its `/docs/llms.txt` and `/docs/llms-full.txt` routes (and a `.md` suffix on any page URL) serve clean Markdown that's cheap to fetch.
-- `docs/developer/` covers contributing to and self-hosting Data Hub itself: architecture internals, local dev setup (`getting-started.md`, `local-development.md`), conventions, the step-by-step self-hosting guide for the web app and AWS infrastructure (`first-time-deployment.md`) plus CI/ongoing-deploy reference (`ci-and-deployment.md`), and per-package references (`lambda.md`, `watcher.md`, `shared-library.md`). These are plain Markdown read on GitHub. The docs site does not render them. See `docs/developer/README.md` for the full index.
+- A user-visible change to the web app, file processing, the API, or AI assistant behavior needs an entry in `web/content/changelog/` in the same branch.
+- Name the file `YYYY-MM-DD-short-slug.md` with today's date, and rename it if the pull request merges into `staging` on a later day. The tests for `web/lib/changelog/parse.ts` check the name and frontmatter.
+- If the change needs a new watcher version, the entry names that version.
+- Tests, docs, CI, dependency updates, infrastructure, and watcher-only releases don't get entries.
 
-The docs site deploys from the `production` branch, so a docs change goes live when `staging` is promoted, together with the feature it describes. Change the docs in the same branch as the feature.
+## Generated snapshots
 
-### Watcher version
+- After you change the REST API, MCP tools, prompts, or resources, the watcher CLI or version, a page in `docs/site/content/docs/`, or a dependency in a `package.json` or lockfile, run `make docs-catalogs` and commit every `*.snapshot.json` it changes. Dependency upgrades count because they can change the generated schemas.
+- Never edit a `*.snapshot.json` file by hand. CI regenerates each one and fails when the committed copy differs.
 
-Shipped watcher changes (`watcher/src/`) need a version bump in the same branch. Follow `watcher/AGENTS.md`: bump `[project].version` in `watcher/pyproject.toml` once if it still matches the merge-base, then run `uv lock`, then run `make py-watcher-cli-catalog docs-bundle-generate` and commit the updated `docs/site/src/lib/cli-catalog.snapshot.json`, which records the version, and `web/lib/mcp/docs/docs-bundle.snapshot.json`, which embeds the CLI reference page. Do not tag the release from the feature branch.
+## Docs
 
-### Changelog
+- The docs site deploys from `production`, so change the docs in the same branch as the feature.
+- To learn how Data Hub behaves for users, read `docs/site/content/docs/` first. The published site at https://datahub.arcadiascience.com/docs shows what is live on `production`.
 
-User-visible changes need a dated entry in `web/content/changelog/` in the same branch. That includes the web app, file processing, the API, and AI assistant behavior.
+## Testing
 
-Name the file `YYYY-MM-DD-short-slug.md`. The date is the day the pull request merges into `staging`. The slug is lowercase words separated by hyphens. The file starts with YAML frontmatter (`title`, and `docs` when a docs page explains the change) and a short markdown body. If the change needs a new watcher version, the entry names that version. Watcher versions do not get their own entries.
+Before you say a change is done:
 
-Do not add an entry for tests, docs, CI, dependency updates, or infrastructure.
+- Run `make check`. It rewrites files to fix formatting, so review the diff afterward.
+- Run the unit tests for each package you touched: `make fe-test-unit` for `web/` and `make py-test-unit` for the Python packages.
+- If the change touches API routes, the database schema, or the watcher-to-server protocol, also run `make fe-test-integration` or `make py-test-integration`. Both run `next build` into `web/.next`, so stop `make dev` first.
+- Check UI changes in the running app. `web/AGENTS.md` says how.
+- Make sure a new test fails without your change by breaking the code once.
+- CI runs everything else.
 
-### Reference snapshots (docs site)
+## Before you push
 
-The API, MCP, and [Watcher CLI](https://datahub.arcadiascience.com/docs/cli-reference) pages on the docs site render from three JSON files committed in `docs/site/src/lib/`: `openapi.snapshot.json`, `mcp-catalog.snapshot.json`, and `cli-catalog.snapshot.json`. They are generated from the code and never edited by hand.
+- Once the code is final, reread every comment, docs line, and changelog sentence you wrote and check it against the code.
+- After you rename or move a file, search for the old path in imports, the `Makefile`, `package.json` scripts, `.github/workflows/`, and the docs.
 
-1. Change the REST API (`web/lib/api/openapi/`), the MCP tools, prompts, or resources (`web/lib/mcp/`), or the watcher CLI help or options (`watcher/src/data_hub_watcher/cli.py`).
-2. Run `make docs-catalogs`. To refresh only the watcher CLI file, run `make py-watcher-cli-catalog docs-bundle-generate`, because the [docs bundle for the MCP server](#docs-bundle-for-the-mcp-server) embeds the CLI reference page.
-3. Commit the changed snapshot files in the same branch.
+## Python
 
-CI regenerates each file and fails when the committed copy differs.
+Shipped Python in `lambda/`, `watcher/`, and `packages/shared/` must run on Python 3.12, because the Lambda image and lab PCs use 3.12. Local development uses 3.13 from `.python-version`. Run Python tools through `uv run`.
 
-### Docs bundle for the MCP server
+## Skills
 
-The MCP server answers questions about Data Hub from a bundled copy of the docs pages, `web/lib/mcp/docs/docs-bundle.snapshot.json`. It is generated from `docs/site/content/docs/` (and the watcher CLI snapshot) and never edited by hand. After changing a docs page or the watcher CLI snapshot, including a watcher version bump, run `make docs-catalogs` and commit the result in the same branch. CI regenerates it and fails when the committed copy differs. See `docs/developer/mcp-docs.md`.
+- When an `AGENTS.md` names a skill, use it if it is installed. Otherwise load it for the session with `npx skills use <source> --skill <name>`, or with `npx skills use <pinned URL>` when the `AGENTS.md` gives a pinned URL.
+- Ask before you install a skill that no `AGENTS.md` names, and never pass `-y`.
+- When a skill's advice conflicts with an `AGENTS.md` rule, follow the `AGENTS.md`.
+- `ultracite` (haydenbleasel/ultracite): fixing lint and format errors in `web/` and `docs/site/`.
 
 ## Cursor Cloud specific instructions
 
-Data Hub is a multi-component repo (see `README.md`). The component you can run end-to-end locally with zero external credentials is the **Next.js web app + REST API + PostgreSQL** (`web/`). The `lambda/`, `watcher/`, and `packages/shared/` Python packages are exercised via tests and a local S3 mirror — no real AWS is needed for local work.
-
-Standard commands live in the `Makefile`, `web/package.json`, `docs/developer/getting-started.md`, and `docs/developer/local-development.md`. The notes below are the non-obvious caveats that those docs don't make obvious for a fresh cloud VM (where the update script has already installed deps).
-
-### Starting services (not handled by the update script)
-
-- **PostgreSQL must be started on every fresh VM** — the cluster is installed and the data (roles + databases) persist in the snapshot, but the server process is not running at boot: `sudo pg_ctlcluster 16 main start` (or `sudo service postgresql start`).
-- Postgres is reachable at `postgres://postgres:postgres@127.0.0.1:5432`. Databases `data-hub-local` (dev) and `data_hub_test` (integration tests) already exist. The integration harness (`web/tests/integration/global-setup.ts`) hardcodes these same credentials and creates `data_hub_test` itself if missing.
-- **Web dev server:** `make dev` (Next.js + Turbopack on http://localhost:3000). Sign in at `/login` with the "Sign in (dev)" button using email `alice@example.com` (workspace admin; shared seed password is submitted invisibly).
-
-### Environment file
-
-`web/.env` is gitignored and required for `make dev` / seeding. If it is missing on a fresh VM, recreate it from the "Minimal `.env`" block in `docs/developer/local-development.md` (the key lines are `DATABASE_URL=postgres://postgres:postgres@127.0.0.1:5432/data-hub-local`, a 32+ char `AUTH_SECRET`, `BETTER_AUTH_URL=http://localhost:3000`, dummy `AWS_*` values, and `LOCAL_S3_MIRROR=../lambda/.local-s3`).
-
-### Node / Python toolchain
-
-- Use **Node 24 (npm 11)** — it is the nvm default and is what CI uses. `npm ci` against the committed `web/package-lock.json` **fails under npm 10** ("Missing: esbuild@… from lock file"), so don't downgrade. A clean login shell already selects Node 24 via nvm.
-- Python is managed by `uv` (Python 3.13, pinned in `.python-version`). Run Python tools through `uv run …` (e.g. `uv run pytest`); the Makefile targets already do this.
-
-### Seeding and local file bytes
-
-- `make db-reseed` resets + pushes the Drizzle schema + seeds deterministic data. It prints a personal access token (`dhub_…`) for the dev user — use it for `Authorization: Bearer` API calls.
-- The seed's fixture-processing step is **skipped if the dev server isn't running** (it prints a hint). To populate processed artifacts (gel-doc PNGs, plate-reader CSVs, qPCR metadata), start `make dev` first, then run `npm run db:process-fixtures` from `web/`.
-
-### Testing caveat
-
-- `make fe-test-integration` and `make py-test-integration` run `next build` + `next start`, which writes to `web/.next` and **contends with a running `make dev`** (also using `.next`). Stop the dev server before running integration tests, then restart it afterward.
-- Lint/format/typecheck: `make check` (note: `py-format`/`fe-format`/`docs-format` auto-rewrite files; use `uv run ruff check .`, `npm run lint:check` (Biome formatter + linter, read-only), and `npm run typecheck` for read-only checks). `make check` also covers the docs site, which has its own dependencies: run `npm ci` in `docs/site/` first. `make docs-links` checks the links between the docs, the web app, and the changelog.
+In a Cursor Cloud VM, read `docs/developer/cursor-cloud.md` before you run anything. Postgres may be missing or stopped, and `web/.env` may be missing.
diff --git a/README.md b/README.md
@@ -85,7 +85,7 @@ Docs are split by who is reading:
 
 ## Run it locally
 
-The web app, API, and database run with no AWS or Google credentials, which is enough for work on the UI, REST endpoints, MCP tools, or the schema. You need PostgreSQL 15 or later and Node.js 22 or later:
+The web app, API, and database run with no AWS or Google credentials, which is enough for work on the UI, REST endpoints, MCP tools, or the schema. You need PostgreSQL 15 or later and Node.js 24 with npm 11:
 
 ```sh
 cd web && npm install && cd ..

diff --git a/docs/developer/README.md b/docs/developer/README.md
@@ -11,6 +11,7 @@ the docs site does not render them.
 
 - [Getting started](getting-started.md) — development setup, environment variables, running locally
 - [Local development](local-development.md) — zero-credential dev workflow for the web app + API + database (no watcher / Lambda needed)
+- [Cursor Cloud](cursor-cloud.md) — PostgreSQL, `web/.env`, and toolchain setup on a Cursor Cloud VM, plus the recommended dashboard install script
 - [Architecture](architecture.md) — system overview, data flow, and design decisions
 - [MCP Apps](mcp-apps.md) — run-report View: host render flow, the three app-only tools, the shared data source, `mcp-apps:build`, and the gitignored HTML artifact
 - [Docs and changelog over MCP](mcp-docs.md) — how the `search_docs`, `read_doc`, and `search_changelog` tools get their content, how search works, how changelog entry links work, and what to update when docs change

diff --git a/docs/developer/ci-and-deployment.md b/docs/developer/ci-and-deployment.md
@@ -2,7 +2,7 @@
 
 ## GitHub Actions
 
-Five workflows run on pushes to `staging`/`production` and on pull requests targeting those branches. A sixth (`apply-migrations.yml`) runs on merges to `staging`/`production` that touch migration files, and a seventh (`publish-watcher.yml`) runs only on `watcher-v*` tag pushes and manual dispatch.
+Six workflows run on pushes to `staging`/`production` and on pull requests targeting those branches. `apply-migrations.yml` runs on merges to `staging`/`production` that touch migration files, `deploy-lambda.yml` runs on merges that touch the Lambda or its infrastructure (see [Lambda (AWS)](#lambda-aws)), and `publish-watcher.yml` runs only on `watcher-v*` tag pushes and manual dispatch. `docs-screenshots.yml` runs monthly and on demand (see [Docs screenshots](local-development.md#docs-screenshots)).
 
 ### Python lint and typecheck (`python-lint.yml`)
 
@@ -15,7 +15,7 @@ Five workflows run on pushes to `staging`/`production` and on pull requests targ
 - Starts a Postgres 17 service container.
 - Installs both Node.js 24 and Python packages.
 - `make py-test` — runs all pytest tests (unit and integration). Integration tests build and start a real Next.js production server, seed a test database, and exercise the Lambda and watcher against the live API.
-- `make py-watcher-cli-catalog`, then fails if `docs/site/src/lib/cli-catalog.snapshot.json` changed. The docs site renders its watcher CLI reference from that committed file, so a change to the CLI or a watcher version bump has to commit the regenerated snapshot. The MCP docs bundle embeds the same page, so run `make py-watcher-cli-catalog docs-bundle-generate` and commit both files, or `docs-lint.yml` fails next.
+- `make py-watcher-cli-catalog`, then fails if `docs/site/src/lib/cli-catalog.snapshot.json` changed. The docs site renders its watcher CLI reference from that committed file, so a change to the CLI or a watcher version bump has to commit the regenerated snapshot. The MCP docs bundle embeds the same page, so run `make docs-catalogs` and commit both files, or `docs-lint.yml` fails next.
 
 ### TypeScript lint and typecheck (`typescript-lint.yml`)
 
@@ -40,6 +40,13 @@ Runs when `docs/**`, `web/lib/docs.ts`, `web/lib/mcp/docs/docs-bundle.snapshot.j
 - `make fe-test-unit` — runs `tests/unit/` and `tests/mcp/` (in-memory MCP protocol tests) together; no Postgres, no Next.js server, no global setup. See [Testing](testing.md).
 - `make fe-test-integration` — runs Vitest integration tests that test the API routes and MCP server over HTTP against a real database.
 
+### Infra template validation (`infra-validate.yml`)
+
+Runs when `infra/**`, `deploy-lambda.yml`, `.github/scripts/check_sam_deploy_overrides.py`, or the `Makefile` changes:
+
+1. `make sam-validate` — lints both CloudFormation templates with cfn-lint.
+2. `make sam-check-overrides` — runs the CI and Makefile `sam deploy` commands with dummy credentials and no reachable AWS endpoint, to catch parameter overrides SAM rejects before they reach AWS.
+
 ### Apply database migrations (`apply-migrations.yml`)
 
 Triggered on pushes to `staging`/`production` (i.e. PR merges) that change files under `web/drizzle/`. The single job sets `environment: ${{ github.ref_name }}` so GitHub selects that environment's secrets and protection rules, then runs `npm run db:migrate` (Drizzle) against the environment's PostgreSQL database using the environment's `DATABASE_URL` secret. A per-branch `concurrency` group prevents overlapping migration runs.
@@ -238,4 +245,7 @@ This is equivalent to:
 ```sh
 make py-check    # py-format + py-lint + py-typecheck
 make fe-check    # fe-format + fe-lint + fe-typecheck
+make docs-check  # docs-format + docs-lint + docs-typecheck + docs-links
 ```
+
+The `*-format` steps rewrite files, so review the diff afterward. The docs checks need `npm ci` in `docs/site/` first.