Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file removed .github/images/dashboard.png
Binary file not shown.
Binary file removed .github/images/run-detail.png
Binary file not shown.
Binary file removed .github/images/runs-table.png
Binary file not shown.
117 changes: 117 additions & 0 deletions .github/workflows/docs-screenshots.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
name: Docs screenshots

# Retakes the docs site screenshots and opens a pull request into staging.
# Run it by hand from the Actions tab for an extra refresh. Pull requests made
# with the built-in token don't start other workflows, which is fine because
# staging requires a review and no checks.

on:
schedule:
# 17:00 UTC on the 1st, which is morning in US Pacific time.
- cron: "0 17 1 * *"
workflow_dispatch:

permissions:
contents: write
pull-requests: write

# A second run would force-push over the first one's branch.
concurrency:
group: docs-screenshots
cancel-in-progress: false

env:
BRANCH: automation/docs-screenshots

jobs:
docs-screenshots:
runs-on: ubuntu-latest
timeout-minutes: 60

services:
postgres:
image: postgres:17
env:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
ports:
- 5432:5432
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5

steps:
- uses: actions/checkout@v6
with:
ref: staging

- uses: actions/setup-node@v6
with:
node-version: 24

- uses: astral-sh/setup-uv@v8.1.0

- name: Install Node.js packages
run: npm ci
working-directory: web

# The sample files go through the Python processors, as in local
# development.
- name: Install Python packages
run: uv sync --all-packages

- name: Install the browser
run: npx playwright install --with-deps chromium
working-directory: web

# A screen that fails keeps its old image and is listed in the pull
# request. The run exits non-zero for it, so the next steps must not
# depend on this one passing.
- name: Take the screenshots
id: shots
run: make docs-screenshots
continue-on-error: true

- name: Check for changes
id: changes
run: |
git add docs/site/public/images docs/site/src/lib/screenshots.generated.ts
if git diff --cached --quiet; then
echo "changed=false" >> "$GITHUB_OUTPUT"
else
echo "changed=true" >> "$GITHUB_OUTPUT"
fi

- name: Open or update the pull request
if: steps.changes.outputs.changed == 'true'
env:
GH_TOKEN: ${{ github.token }}
run: |
summary=web/.docs-screenshots/summary.md
if [ ! -f "$summary" ]; then
echo "Automated refresh of the docs screenshots. The run did not write a summary, so check the workflow log." > "$summary"
fi

git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git switch -C "$BRANCH"
git commit -m "Refresh docs screenshots"
git push --force origin "$BRANCH"

open=$(gh pr list --head "$BRANCH" --base staging --state open --json number --jq 'length')
if [ "$open" = "0" ]; then
gh pr create --base staging --head "$BRANCH" \
--title "Refresh docs screenshots" --body-file "$summary"
else
gh pr edit "$BRANCH" --body-file "$summary"
fi

# If nothing changed and the run failed, it never got as far as taking
# screenshots. Failing here makes that visible.
- name: Fail when nothing was captured
if: steps.changes.outputs.changed == 'false' && steps.shots.outcome == 'failure'
run: |
echo "::error::The screenshot run failed before it changed any image. Check the log above."
exit 1
8 changes: 8 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,14 @@ docs-catalogs:
make py-watcher-cli-catalog
make docs-bundle-generate

# Retake the docs site screenshots from a throwaway local database and rewrite
# docs/site/public/images. Needs Postgres, uv, and Playwright's Chromium.
# Usage: make docs-screenshots [ONLY=home,my-runs] [SKIP_SEED=1]. See
# docs/developer/local-development.md.
.PHONY: docs-screenshots
docs-screenshots:
cd web && npm run docs:screenshots -- $(if $(ONLY),--only $(ONLY)) $(if $(SKIP_SEED),--skip-seed)

.PHONY: docs-format
docs-format:
cd docs/site && npm run lint:fix
Expand Down
20 changes: 15 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,21 @@ Nobody signs up for Data Hub. Your team deploys it, so raw instrument data and e

[Documentation](https://datahub.arcadiascience.com/docs) · [Set up an instrument](https://datahub.arcadiascience.com/docs/set-up-an-instrument) · [Deploy it yourself](docs/developer/first-time-deployment.md) · [Architecture](docs/developer/architecture.md)

![The Data Hub dashboard, showing run counts, instrument status, and a table of recent runs](.github/images/dashboard.png)

![The runs table for a SpectraMax iD5 plate reader, filtered by measurement mode, with wavelength and measurement type columns](.github/images/runs-table.png)

![A plate reader run detail page showing measurement metadata, a file list, and a color-coded 96-well plate map](.github/images/run-detail.png)
<!-- These are the docs site's screenshots, retaken by `make docs-screenshots`. -->
<picture>
<source media="(prefers-color-scheme: dark)" srcset="docs/site/public/images/home.dark.webp">
<img alt="The Data Hub dashboard, showing run counts, instrument status, and a table of recent runs" src="docs/site/public/images/home.webp">
</picture>

<picture>
<source media="(prefers-color-scheme: dark)" srcset="docs/site/public/images/plate-reader-runs-table.dark.webp">
<img alt="The runs table for a SpectraMax iD5 plate reader, with wavelength, measurement mode, and measurement type columns" src="docs/site/public/images/plate-reader-runs-table.webp">
</picture>

<picture>
<source media="(prefers-color-scheme: dark)" srcset="docs/site/public/images/plate-reader-run-details-1.dark.webp">
<img alt="A plate reader run detail page showing measurement metadata, a file list, and a color-coded plate map" src="docs/site/public/images/plate-reader-run-details-1.webp">
</picture>

## Web app, REST API, and MCP

Expand Down
38 changes: 38 additions & 0 deletions docs/developer/local-development.md
Original file line number Diff line number Diff line change
Expand Up @@ -227,9 +227,47 @@ A few details worth knowing:
- The route is gated on `NODE_ENV !== "production"` AND `LOCAL_S3_MIRROR` set; either condition unmet returns 404 unconditionally, so a production build can never expose the filesystem.
- The MCP tool at `/mcp/v1` returns a relative `/api/local-s3/...` URL when the mirror is active — browsers resolve it against the current origin, but non-browser MCP clients on localhost may need to prefix with `http://localhost:3000`.

## Docs screenshots

The screenshots on the docs site are generated. `make docs-screenshots` seeds a separate database, starts the web app on its own port, opens each screen in a headless Chromium, and writes a light and a dark WebP file for it into `docs/site/public/images/`. The workflow [.github/workflows/docs-screenshots.yml](../../.github/workflows/docs-screenshots.yml) runs it on the first of every month and opens a pull request into `staging`. You can also start it from the Actions tab with **Run workflow**.

You need Postgres running with the same `postgres` login as above, `uv` (the sample files go through the Python processors), and the browser, which you install once:

```sh
cd web && npx playwright install chromium
```

```sh
make docs-screenshots # every screenshot
make docs-screenshots ONLY=home,my-runs # only these
make docs-screenshots SKIP_SEED=1 # reuse the data from the last run
cd web && npm run docs:screenshots -- --serve # seed and start the app, then wait
```

How it works:

- It uses its own database, `data_hub_screenshots`, and its own file mirror under `web/.docs-screenshots/`, so your local data is untouched and `make dev` can keep running.
- It runs the app with `next dev`, because the local file mirror that serves the gel images, PDFs, and plate maps is turned off in production builds. It restores `web/tsconfig.json`, `web/AGENTS.md`, and `web/next-env.d.ts`, which `next dev` edits when it starts.
- The data is the same as `make db-reseed` ([web/scripts/seed-workspace.ts](../../web/scripts/seed-workspace.ts)), plus a few things only the screenshots need ([web/scripts/docs-screenshots/setup.ts](../../web/scripts/docs-screenshots/setup.ts)): realistic access tokens, a fake Linear workspace from the feedback package's test server, sample feedback reports, and a sample AI client for the sign-in approval screen. Shots that change data, such as the manual-upload run and the deleted run, undo their change when they finish.
- It signs in by writing a session row for Alice, so the dev-only sign-in form isn't needed.
- Dates are real, and every run rewrites every image. Before each shot it moves the seeded watcher heartbeats forward, because a watcher shows as offline after five minutes without one.
- With `--serve`, it writes `web/.docs-screenshots/serve.json` with the app's address and a session cookie, so you can drive the app yourself while you work out the steps for a new shot.

If a screen fails, its old images stay. The run prints the failure, saves what the browser showed to `web/.docs-screenshots/failures/`, and writes `web/.docs-screenshots/summary.md`, which the workflow uses as the pull request description.

To add a screenshot:

1. Add an entry to [web/scripts/docs-screenshots/shots.ts](../../web/scripts/docs-screenshots/shots.ts). It names the page to open, the steps to reach the right state, and whether it is a full window or a close-up of one element.
2. Run `make docs-screenshots ONLY=<name>`.
3. Reference `![Alt text](/images/<name>.webp)` from a docs page. The docs site draws a browser frame around full windows and shows the light or dark image to match the reader's theme.
4. Commit the new images and `docs/site/src/lib/screenshots.generated.ts`, which the script writes.

A screenshot the script can't take, such as the Claude desktop app in `mcp.png`, is listed under `MANUAL_SHOTS` in the same file with the date it was taken. The pull request lists how old each one is.

## Where the seed lives

- [web/lib/db/seed.ts](../../web/lib/db/seed.ts) — shared builder functions (`seedDevUser`, `seedInstruments`, `seedRuns`, etc.) plus a schema-driven `clearAll()`.
- [web/scripts/seed-workspace.ts](../../web/scripts/seed-workspace.ts) — the full seeded workspace, shared by `npm run db:seed` and the docs screenshots.
- [web/scripts/seed-database.ts](../../web/scripts/seed-database.ts) — the entry point that `npm run db:seed` runs.
- [web/scripts/process-fixtures.ts](../../web/scripts/process-fixtures.ts) — shared probe/spawn logic for the post-seed handler step.
- [web/scripts/process-seeded-fixtures.ts](../../web/scripts/process-seeded-fixtures.ts) — `npm run db:process-fixtures` entry point for re-running the handler step on demand.
Expand Down
3 changes: 2 additions & 1 deletion docs/site/biome.json
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,8 @@
"!dist",
"!build",
"!.source",
"!src/lib/*.snapshot.json"
"!src/lib/*.snapshot.json",
"!src/lib/screenshots.generated.ts"
]
},
"formatter": {
Expand Down
26 changes: 17 additions & 9 deletions docs/site/content/docs/browse-runs.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -14,18 +14,20 @@ The sidebar and the home page are the two starting points for daily work.

**Home** lists recent runs across instruments, most recent first, with counts for today and this week at the top. Filter by date range, who ran it, and instrument-specific metadata when it’s available, then page through results. Below the runs, Home shows up to four of today’s comments; use the date picker to see comments from a different range.

![Home dashboard with instrument status and recent runs](/images/home.png)
![Home dashboard with instrument status and recent runs](/images/home.webp)

**Instruments** in the sidebar lists five instruments, and **View all** opens the full list. Each instrument’s page shows the folder its watcher reads from and the file patterns it uploads, then a runs table with column filters that match the instrument type (for example wavelengths on a plate reader).

![Plate reader instrument page with the runs table and column filters](/images/plate-reader-runs-table.png)
![Plate reader instrument page with the runs table and column filters](/images/plate-reader-runs-table.webp)

**My runs** lists runs you’ve claimed, so you can return to your own work without re-filtering the full dashboard.

![My runs page listing runs attributed to the signed-in user](/images/my-runs.png)
![My runs page listing runs attributed to the signed-in user](/images/my-runs.webp)

**Comments** lists every run comment by day. Open it from **View all comments** on Home. Each person’s profile page also shows the comments they wrote and the comments on runs they ran.

![Comments page listing each run comment by day, with filters for instrument and for comments on your runs](/images/comments-page.webp)

### Narrow a long run list [#narrow-a-long-run-list]

Home and the instrument pages share the same four controls:
Expand Down Expand Up @@ -57,7 +59,7 @@ A run with files in several states shows the first status in this table that app

Press **⌘&nbsp;K** (macOS) or **Ctrl&nbsp;K** (Windows/Linux) to open global search from any page. Results are grouped into runs, files, instruments, users, and comments. That means you can also look up a colleague’s work or the text of a lab note. A file hit opens the parent run with the files table already filtered to that file.

![Global search modal with results for runs and files](/images/search-modal.png)
![Global search modal with results for runs and files](/images/search-modal.webp)

## Open a run and inspect results [#open-run-detail]

Expand All @@ -75,11 +77,11 @@ Reports with many images, PDFs, or spectra show one item at a time. Search by fi

Per-type inputs and outputs are in [Instrument data preprocessing](/docs/instrument-preprocessing).

![Gel doc run detail with metadata badges, files table, and report preview](/images/gel-doc-run-details.png)
![Gel doc run detail with metadata badges, files table, and report preview](/images/gel-doc-run-details.webp)

![Plate reader run detail with plate map visualization](/images/plate-reader-run-details-1.png)
![Plate reader run detail with plate map visualization](/images/plate-reader-run-details-1.webp)

![Plate reader run detail with kinetic traces and comments](/images/plate-reader-run-details-2.png)
![Plate reader run detail with kinetic traces and comments](/images/plate-reader-run-details-2.webp)

### Filter the files table [#filter-files]

Expand All @@ -101,6 +103,8 @@ From the files table you can:
- Select files and choose **Download** in the bar that appears above the files table. One file downloads directly. Two or more come as a single zip, which builds in the background like **Download all**
- Choose **Download all** to build a zip of active files in the run, with any active file filters applied

![File preview window showing the first rows of a processed CSV, with a Download button and buttons for the next and previous file](/images/csv-preview.webp)

Large zips build in the background. When one isn’t ready yet, Data Hub tells you how long to wait before trying again.

## Act on several runs at once [#bulk-actions]
Expand All @@ -120,15 +124,15 @@ The selection stays when you move to another page of the same results. It clears

The files table on a run page works the same way: select files to get **Upload**, **Dismiss**, **Reprocess**, and **Download** for just those files. Its selection also clears when you change the files search, a filter, or the sort.

![Run list with several runs selected and the bulk action bar open](/images/bulk-actions.png)
![Run list with several runs selected and the bulk action bar open](/images/bulk-actions.webp)

## Comment on a run [#comment]

The comments thread on run details is for lab notes and follow-ups. Comment activity can notify people who claimed the run or already commented, and clicking a comment notification scrolls to that comment. Set preferences under **Settings → Notifications**; see [Configure notifications](/docs/configure-notifications).

Only the author of a comment can edit it. Authors can delete their own comments, and workspace admins can delete any comment. Comments posted by a script through an API token show the token’s name with a key icon instead of a person.

![Gel doc processed image with a lab comment on the run](/images/gel-doc-run-comments.png)
![Gel doc processed image with a lab comment on the run](/images/gel-doc-run-comments.webp)

## Reprocess or delete a run [#reprocess-and-delete]

Expand All @@ -142,12 +146,16 @@ Reprocessing sends only the original instrument files. Files Data Hub generated,

**Delete** marks the run deleted without removing stored files. To find a deleted run again, turn on **Show deleted runs** under **Add filter**, then open it and choose **Restore**. Anyone signed in can delete or restore a run.

![Deleted run page with a red banner naming who deleted it and when, and a Restore button](/images/deleted-run.webp)

### Request an upload [#request-upload]

A file that the watcher found but hasn’t uploaded shows **Pending**. Choose **Upload** on that file, or select runs and choose **Upload**, to ask the watcher to send it. The watcher picks up the request within a minute. On instruments that upload automatically, this needs watcher 1.2.1 or later; see [Roll out watcher releases](/docs/watcher-releases#what-each-version-adds).

For instruments in [manual upload mode](/docs/concepts#upload-mode), every file waits for this approval. **Dismiss** tells the watcher not to upload a file. See [Manage the watcher fleet](/docs/manage-watchers#approving-manual-uploads).

![Run page for a manual-upload instrument with its pending files selected and a bar offering Upload and Dismiss](/images/manual-upload.webp)

## Use the same data outside the UI [#programmatic-access]

You can also list and fetch the same runs and files over the [REST API](/docs/api) or the [Model Context Protocol (MCP) endpoint](/docs/mcp). REST needs a [personal access token](/docs/concepts#personal-access-token); scopes and creation are covered in [Issue and revoke tokens](/docs/manage-tokens). MCP clients sign in through the browser instead, and clients that support MCP Apps show the same interactive run report you see here.
4 changes: 3 additions & 1 deletion docs/site/content/docs/configure-notifications.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ contentType: How-to

Data Hub can notify you in the app and in Slack when new runs arrive, when someone comments on a run you care about, when a script sends you a message, and when feedback you sent is closed. Open **Settings → Notifications** while signed in. Everything on that page is a personal choice that affects only you.

![Settings Notifications page with in-app preferences and the notification bell](/images/notifications.png)
![Settings Notifications page with in-app preferences and the notification bell](/images/notifications.webp)

Connections that the whole workspace shares, such as the Slack channel for new runs and the Slack app behind direct messages, are under **Settings → Integrations** and only admins can change them. See [Set up integrations](/docs/integrations).

Expand Down Expand Up @@ -37,6 +37,8 @@ The bell icon in the app lists recent notifications:
- Clicking a comment notification opens the run and scrolls to that comment.
- Messages sent by a script through an API token show the token’s name with a key icon.

![Notification list opened from the bell, with new runs grouped by day and instrument](/images/notification-bell.webp)

## Slack direct messages [#slack-direct-messages]

Connect your Slack account on the same settings page to receive matching events as personal direct messages. After connecting, choose which types also go to Slack:
Expand Down
Loading
Loading