Skip to content

Fix review findings from production snapshot (17 issues) - #131

Open
m-murphy wants to merge 89 commits into
feat/container-selectionfrom
integrate/review-fixes
Open

m-murphy wants to merge 89 commits into
feat/container-selectionfrom
integrate/review-fixes

Conversation

@m-murphy

@m-murphy m-murphy commented Nov 1, 2026 •

Copy link
Copy Markdown
Member

Summary

Fixes from a review of this branch against a production snapshot (arwen, 2026-10-31), merged in migration order with conflicts resolved.

migrations (generic: each converts what its rule can, leaves the rest for the integrity report, never aborts on data)
  008  collection dates → YYYY-MM-DD (M/D/YY read as 20YY)          #104
  009  sessions store a SHA-256 token hash; old sessions dropped     #121
  010  repair legacy Grid positions (--A01, barcode-as-position, NA) #125
  011  lookup indexes                                                #106
  012  control batch properties stored once-encoded                  #133
  startup copies the database (VACUUM INTO) before any migration     #132

one-time, site-specific data fixes (not migrations)
  bun run data-fix <file> [--dry-run]   copy first · one transaction · recorded in data_fix_log
  ops/data-fixes/2026-11-01-eppicenter-legacy-cleanup.sql            #124 #126  (our database only)

one rule per concern (packages/contract)
  grid-position.ts       parseGridPosition: every write; unplaced = fails it   #102 #103 #125
  container-quantity.ts  0 ≤ remaining ≤ total, used > 0, no overdraw          #109
  collection-name.ts     newCollectionName for derivations                     #130
Area Change Issues
Collections Plate and box pages list Unplaced containers; delete dialog shows impact and refreshes; collection delete is admin-only; detail pages load papers in two queries #103 #113 #112 #106
Writes Atomic container edit; quantity bounds; ambiguous barcodes refused; case-insensitive barcode uniqueness; Location can't drop "can contain collections" while holding them; derivation dates validated #108 #109 #105 #133
Subjects Merge Subjects searches the server (past 1,000); Create New Subject saves subject and specimen in one transaction and refuses an existing name; names compare ignoring case #135 #126
Moves Grid position rule on every write; partial moves return 200 with what moved; one transaction; duplicate sheet ids moved once #102 #110 #111 #134
Import Collection names from name columns; errors numbered by file row; blank Derivations rows dropped; control-batch names unique, quantity 0 kept, CSV rows numbered as in a spreadsheet #114 #138 #136
Lists and API input Server pagination and sort for specimens; studies search covers lead person and description; malformed ids, pages and dates return 400; pagination settings apply without restart #115 #128 #120 #127
Dates ISO collection dates; UTC server timestamps shown local; study ranges, enrollment and collection timelines in date order and per day #104 #117 #107
UI Typed sorting; honest dashboard tiles; cache invalidation after bulk and reference-data writes; Admin Users errors inside modals and no double submit; Reference Data toggle errors shown, "Add" label #116 #118 #119 #137
Export Tags applied in CSV mode; errors stay visible; stale counts ignored; JSON export rows keep every column (ADR 0005 exception) #122
Integrity report Checks for placeholder and future dates, non-ISO dates, invalid Grid positions, exhausted-with-quantity, case-variant names, duplicate Specimens, disallowed container types, duplicate barcodes across tube types #126
Security and ops Hashed session tokens; capped error-log intake; generic production 500s; units endpoint needs login; specimen-type statistics cache cleared on change #121 #134
qPCR Uploads and delete each in one transaction; skipped amplification rows reported #123 #129

Addresses #102–#138 except #124 (decisions, closes with this) and #131 itself; #107 and #122 rely on #96/#98 from #101.

Evidence

bun run ci:verify on 23d38e1: contract 143, API 1,585, web 1,191 (178 files), 0 failures. Each fix branch added tests that fail on the old code.

The API was run on a fresh copy of the production snapshot (schema v4):

Check Before After
Startup — .pre-v12-….bak written (v4, 118,069 specimens), migrations 5–12 applied
GET /api/specimens, no params 28 MB 12.5 KB, 58 ms
Bag 166 detail slow N+1 35 ms
Plate 103 (id 4) detail 1 of 96 tubes 96, listed as unplaced
Box 790 detail 20 of 73 tubes visible 73 placed
collection_date_to=2000-01-01 18,009 17,307 (placeholders)
Studies search "Greenhouse" 0 22
subject_id=abc, page 10²⁰ 200 / 500 400
Units endpoint without login 200 401
Turn off "can contain collections" on Location 107 (120 collections) 200 409, unchanged
Admin session list cookie values {id, expiresAt} only

Migrations 008/010 made lenient: the production result is byte-identical to the aborting versions (checksums of collection_date and every position column match).

data-fix on the migrated copy:

Integrity check Before After
Placeholder collection dates 17,306 0
Exhausted Containers with quantity left 46 0
Case-variant Subject names 2 0
Container with no subtype (34513) 1 0
Future dates (kept for the lab) 68 68
Duplicate Specimens (kept; cleared dates add undated twins) 113 517

A second run prints "already ran … nothing to do". A fresh empty install starts at version 12 with no data_fix_log table.

Merge Danger

Door: one-way

Migrations 008 and 010 rewrite production rows (1,148 collection dates, 366 Grid positions) and 009 drops every session. Startup now writes a VACUUM INTO copy before migrating, so rolling back means redeploying the previous image and restoring that copy. The data fix is a separate, recorded, one-way step an operator runs by hand.

Blast Radius: wide

  • Everyone logs in again after deploy.
  • API behaviour changes for clients:
    • malformed ids, page numbers and date filters return 400
    • partial best-effort moves return 200
    • member 403s on collection delete-with-contents, control batch delete, Derivation edit and delete
    • the units endpoint requires login
    • JSON exports include null columns
    • /subjects/with-specimens accepts newSubjectOnly
  • Imports and moves with bad positions, overdraws or ambiguous barcodes now fail where they used to succeed. Subject names that differ only by case are refused.
  • Other installs: migrations never abort on their data; leftovers appear in the integrity report. Our ops/data-fixes file must not be run elsewhere (its guards abort if it is).
  • Disk: the pre-migration copy doubles database size on disk until deleted.
  • Migration numbers matter: deploy 008–012 together.

Not covered: the 8 Polyclonal Ab tubes in micronix (Specimen 141173) need a lab decision, and the web app can't yet edit a Specimen's collection date to fix a leftover the report lists.

🤖 Generated with Claude Code

m-murphy and others added 30 commits October 31, 2026 18:29
…abels (#108)

PATCH /containers/:id now checks the Container, unit, barcode or sublabel,
and Tags before any write, and runs the checks and writes in one
withWriteTransaction. Tag ids are de-duplicated; an unknown Tag is a 400.
A failed edit, including a UNIQUE failure on the last write, leaves the
Container unchanged. Errors go through handleRouteError, so they are logged.

The edit modal shows a Sublabel field for Paper and sends it as `sublabel`
instead of `barcode`, which the API rejects. It saves through
containersApi.update and a new useUpdateContainer hook (invalidates every
container query) instead of raw fetch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- DataTable columns declare a sortType (text, number, date) instead of
  sortable; text sorts naturally (2 before 10) via a shared comparator in
  lib/sort-compare.ts, and missing values sort last in both directions.
  Strings are no longer sorted as dates because `new Date()` parses them.
- Studies: compute ascending order for every key and negate only for
  descending, so Title and Lead ascending run A to Z. Date and count
  defaults (newest or largest first) are unchanged.
- Studies loads every study (paging at the API's max page size) instead of
  infinite-scrolling pages of 50, so sorting and filtering cover the whole
  list. Sorting by a count prefetches every study's summary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#123)

Result upload now writes the run, well results, amplification data and the
experiment status in one withWriteTransaction, inserting rows in chunks sized
to stay under bun:sqlite's bound-parameter limit. Plate upload and experiment
delete run in one transaction each. The qPCR templates are serialized with
serializeCsv, so a delimiter in a field is quoted and a missing value is empty.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…olumns (#114)

The import wizard decided a missing Collection identifier was a barcode by
its shape (uppercase, hyphens, longer than five characters), so a
plate_name of PLATE-001 became an empty plate "Collection-PLATE-001" while
the server created a second PLATE-001 holding the Specimens.

Each row now yields one Collection reference whose name comes from
plate_name/box_name/bag_name and whose barcode comes from
collection_barcode. A row giving both makes one Collection, and a
Collection given only a barcode is named after it, as the server's planner
does.

The Papers template example row now puts BOX-003 in box_name, leaves
bag_name empty, and fills sheet_name and sublabel. The required-fields
list shows "box_name or bag_name". Every template is now tested against the
required fields, the contract's CSV row rules, and the bulk-combined
validate schema.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#102)

Add parseGridPosition and GRID_GEOMETRY to @sampledb/contract: a row
letter plus a column number, normalized to A01 form and checked against
the Collection type (micronix plate A01-H12, cryovial box A01-J10).

Moves and Scan moves, Specimen registration (bulk import, specimens,
derivations, control batches) and the derivation CSV preview now reject
any other position with an error naming the row and the value; the
cryovial Scan move flags bad target positions at upload. Lookups keep the
lenient normalizer so legacy malformed rows still resolve and move.

Closes #102

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The micronix plate detail keyed wells by `position || ''`, so tubes without
a Grid position overwrote each other and plate 103 showed all 96 tubes as
Empty. The cryovial box detail dropped tubes with non-grid positions from
the grid and drew only rows that held a tube.

- @sampledb/contract: add collection-grid (plate A-H x 1-12, cryovial box
  A-J x 1-10) and gridPositionKey, which maps a stored position to its cell
  or null.
- API: plate and box detail return placed Containers keyed by cell plus an
  `unplaced` list (missing or non-grid position, labelled with the stored
  value). Plate scan validation counts unplaced tubes as expected: a scanned
  one gets status `unplaced_in_scan`, not `extra_in_scan`, and the result
  lists every unplaced tube.
- Web: plate and box pages render an Unplaced section, and the table view
  and its CSV export append unplaced rows. The cryovial grid always draws
  the full 10x10 box. Scan validation shows the new status and an Unplaced
  tubes table.
- CONTEXT.md gains an "Unplaced container" entry; user guide updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…duction 500s (#121)

Sessions
- The session cookie is now a random 256-bit token stored only as its
  SHA-256 hash (sessions.token_hash, unique index). sessions.id is a
  separate public id that admins list and revoke by; it cannot
  authenticate. Cookie lookups stay one indexed equality on token_hash.
- Session create/lookup/delete moves into lib/auth/auth-service.ts
  (login, logout, switch user, password change and reset, user delete,
  auth middleware all use it).
- Migration 009 (009_session_token_hash.sql) drops and recreates the
  sessions table. Existing sessions stored the cookie value in id and
  SQLite cannot hash it in SQL, so they are invalidated: every user logs
  in again once after this release. Noted in the user-guide release notes.
- CURRENT_SCHEMA_VERSION goes 7 -> 9; 008 is reserved for another
  change and must land before any database reaches 9.

Error-log intake
- POST /api/error-logs takes a 64 KiB body cap (413, PAYLOAD_TOO_LARGE,
  not persisted) and per-field caps (message 2,000, stack 16,000,
  errorCode 200, context 16,000 chars of JSON; 400 VALIDATION_ERROR).
  The caps and schema live in @sampledb/contract (error-report.ts); the
  web error logger cuts its reports to fit before posting.

Error messages
- With NODE_ENV=production a 500 from handleRouteError returns
  {"error":"Internal server error","errorCode":"INTERNAL_ERROR"}; the
  full error goes to stdout (observability) and error_logs.
  containersFetchFailedBody drops the cause's message in production too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Migration 008 rewrites legacy M/D/YY and M/D/YYYY collection dates
  (2-digit year read as 20YY), trims whitespace, turns blank values into
  NULL, and aborts with a message if any other value remains.
- List, statistics and Container selection (export) date filters reject
  a malformed bound with a 400 instead of comparing it as a string.
- Derivation refuses to copy a non-ISO parent date; the derivation CSV
  normalizes parent_collection_date and rejects a value that isn't a date.
- The integrity report gains a "Collection date not ISO" check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ositions

# Conflicts:
#	packages/api/src/routes/__tests__/collections.test.ts
#	packages/contract/src/index.ts
The API stores UTC timestamps as SQLite `YYYY-MM-DD HH:MM:SS` with no zone
marker, and the web parsed them as local time. West of UTC a plate created
just after midnight UTC showed the next day and a negative age.

parseDisplayDate is now the one parser for dates from the API: a zoneless
timestamp (space or T separator) is UTC, a bare YYYY-MM-DD or YYYY-MM is a
local calendar date, and anything with Z or an offset keeps its zone. Every
created, last_updated, last_login, error log and activity timestamp now goes
through it (or the new displayDateTime / formatDateTime helpers). Relative
ages clamp at zero so clock skew reads "Just now". Statistics and dashboard
timeline months no longer shift back a month in the Americas.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#102 and #103 each added a contract module with its own geometry and its
own position regex. Fold collection-grid.ts into grid-position.ts:
GRID_GEOMETRY is the only geometry, parseGridPosition the only parser, and
gridPositionKey is parseGridPosition with failure meaning "unplaced", so a
position a write would reject can never be placed by a read.

- gridAxes(type), gridCellKeys(type) and gridPositionKey(position, type)
  take the Collection type; MICRONIX_PLATE_GRID, CRYOVIAL_BOX_GRID and
  CollectionGrid are gone and every caller is updated.
- Columns take one or two digits (#102's rule). No geometry has more than
  12 columns, and no write path produces "A001"; #103's 1-3 digit read
  pattern would have placed positions that writes reject.
- isStoredGridPosition says whether a stored value is already in the
  form the rule produces, for the integrity report.
- container-move and specimen-registration use the contract's
  GridCollectionType instead of their own copies.
- CONTEXT.md: Grid position defines the rule once; Unplaced container
  refers to it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…te (#113)

The delete dialog now runs the preflight when it opens and lists the
Containers, Specimens and Subjects the delete removes before the user
confirms. Subjects count as removed only when "Also remove study
participants" is on; otherwise the dialog says how many would be left
with no Specimens. Preflight blockers show up front and disable Delete.

The preflight summary gains emptySubjectCount: Subjects whose every
Specimen the delete removes.

The delete goes through useDeleteCollectionWithContents, which
invalidates every query with invalidateAfterBulkWrite so the Collections
list, dashboard, study, Subject and Location data refresh. The deleted
Collection's own queries are marked stale without a refetch, so the
detail page navigates away instead of refetching into a 404.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#111)

- Execute Moves is disabled while a request is in flight, and resolve,
  create, and execute actions drop a re-entrant call, so a second click
  can't send a second move or overwrite the first result.
- A failed lookup is a 'lookup' error that doesn't block Next; Next
  retries the resolve with the uploaded files, and each resolve replaces
  the previous attempt's errors instead of stacking them.
- The relocation check compares positions in stored form via
  parseGridPosition, so an A1 scan matches the A01 well.
- Move errors map the endpoint's request row back to the CSV file and
  row, on failure (including error responses) as well as success.
- Scanner CSVs reject a repeated position or barcode; cryovial files
  reject a repeated target position or source tube.

Closes #111

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A move that moved anything returns 200 with `moved` and `errors`; a move
  that moved nothing still returns 400. Applies to Container and Collection moves.
- POST /collections/move takes a Collection type per move, so one request
  (one transaction) covers plates, boxes and bags and all-or-nothing holds
  across types. The top-level collectionType remains as a default.
  Error rows are now 1-based (row 0 is request-level), matching Container moves.
- Container moves set storage_container.last_updated/updated_by; Collection
  moves now set updated_by alongside last_updated.
- The Collection Move page sends one request, reads results from both 200
  and 400 bodies, refreshes queries whenever anything moved, and names the
  Collection behind each error row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ists

GET /studies and GET /subjects ignored the created-date filters, so the
dashboard's 30-day trend for Studies and Subjects compared the total
with itself. Both lists now filter on `created` through a shared
dateRangeConditions helper (also used by the specimens list), with
created_to covering its whole day.

Refs #118

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Trends compare records created in the last 30 days with the 30 days
  before (UTC calendar dates), for Specimens, Studies and Subjects. No
  percentage is shown when the earlier window is empty, and a decline
  carries a minus sign.
- listTotal rejects on a failed request instead of returning 0. Each
  count is its own query, so a failed count shows an error with Retry
  on its tile and the other tiles still load (ADR 0001).
- The tile's icon moves into the title row and the value no longer
  wraps. Metric groups take two columns from lg and three only from
  2xl, so a six-digit count fits at 1024-1535px next to the sidebar.
- User guide says what the trend compares and how a failed card looks.

Closes #118

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Members could delete a Collection with its contents (and run its
preflight), delete a Control batch with its Specimens, and edit or
delete a Derivation, although the user guide reserves these for
admins. These routes now use admin middleware and return 403 to
members.

The web app hides Delete collection (through a shared
CollectionDeleteButton) and Delete Batch from non-admins. The app has
no Derivation edit or delete UI.

CONTEXT.md now defines the Admin, Member and Viewer roles and lists
each Destructive action, and the user guide says Collection and
Control batch deletes need an admin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Migration 010 brings every stored Grid position into the form the Grid
position rule produces:
- leading dashes are stripped and the rest normalized (--A01, -F02 -> A01, F02)
- a position that is a letter followed by the tube's own barcode is cleared
  (plate 254, missing_samples)
- NA positions are cleared (box 84, NA01-NA05)
Cleared Containers stay in their Collection as Unplaced containers. Like
004-006, the migration aborts and changes nothing if a repair would put two
Containers in one position of a Collection, or if it finds a position that
fits none of these cases.

The integrity report gains "Containers with an invalid Grid position":
tubes or wells whose stored position fails the rule or is not in stored
form. It counts toward the overview's issue total.

On the 2026-10-31 production snapshot: 160 dash positions normalized
(boxes 790, 794, 795), 201 barcode positions and 5 NA positions cleared, 0
collisions, 0 left that fail the rule; box 790 shows all 73 tubes.

Migrations 008 and 009 are on other branches, so this one is numbered 010
and CURRENT_SCHEMA_VERSION jumps to 10; merge 008 and 009 first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…te (#109)

One rule in @sampledb/contract (container-quantity.ts) decides the bounds:
total and remaining are 0 or more, remaining is no more than total, and a
quantity used is greater than 0 and no more than the Parent container's
remaining quantity. Every write path uses it:

- PATCH /containers/:id rejects a negative remaining quantity, or one
  above the Container's total, with a 400 before any write.
- Derivation checks the Child container quantity and the quantity used up
  front and throws a 400 ValidationError. It no longer clamps an overdraw
  to 0, and a negative quantity used can no longer refill the parent.
- The derivations CSV validate step reports these as row errors and adds
  up rows that draw from the same Parent container. Import rejects with
  the row number. The unused validateParentQuantity setting is removed.
- Specimen registration (bulk combined import, POST /specimens, control
  batches, Child containers) checks the resolved quantities, so bulk
  import errors carry their row numbers.

Web: the Container edit modal checks the bounds against the total before
saving and shows the API's message. The derivations bulk import review
step lists the invalid rows from validation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…control batch specimen delete stay member actions (#112)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…105)

Barcode lookups ended in .get() on non-unique columns, so move by barcode,
Derivation parent lookup and the specimen barcode filter acted on whichever
row came first.

- One resolver, findContainersByBarcode in identifier-resolution.ts, returns
  every match. resolveContainerByBarcode throws AmbiguousBarcodeError listing
  the candidates; container-move's own resolver is gone.
- Move by barcode fails the row with that error; the resolve endpoint returns
  it with the candidates, and the Scan move core shows it on the row.
- Derivation parent lookup (CSV validate, import, dry run) rejects an
  ambiguous parent_container_barcode.
- The specimen list barcode filter returns every Specimen with a matching
  Container.
- A new or changed micronix or cryovial barcode must not match any tube's
  barcode, ignoring case and surrounding whitespace (registration, Derivation
  CSV, Container edit). An unchanged barcode is not rechecked, so Containers
  whose barcode is already duplicated still save.
- The integrity report groups duplicate tube barcodes across types, with case
  and whitespace variants.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	packages/api/initial_schema.sql
#	packages/api/src/db/__tests__/schema-evolution.test.ts
#	packages/api/src/db/schema-evolution.ts
# Conflicts:
#	packages/api/initial_schema.sql
#	packages/api/src/db/__tests__/schema-evolution.test.ts
#	packages/api/src/db/schema-evolution.ts
#	packages/api/src/lib/data-audit.ts
#	packages/api/src/routes/__tests__/data-audit.test.ts
#	packages/contract/src/index.ts
#	packages/docs/src/content/docs/guides/advanced/data-integrity.md
#	packages/web/src/lib/api/admin.ts
#	packages/web/src/pages/AdminDataIntegrityOverview.tsx
#	packages/web/src/pages/AdminDataIntegrityReport.tsx
#	packages/web/src/pages/__tests__/AdminDataIntegrityEmptyCollections.test.tsx
#	packages/web/src/pages/__tests__/AdminDataIntegrityOverview.test.tsx
#	packages/web/src/pages/__tests__/AdminDataIntegrityReport.test.tsx
# Conflicts:
#	packages/api/src/lib/container-move.ts
#	packages/api/src/routes/__tests__/collections.test.ts
# Conflicts:
#	packages/api/src/lib/derivations-csv.ts
#	packages/contract/src/index.ts
# Conflicts:
#	packages/api/src/lib/__tests__/derivations-csv.test.ts
#	packages/api/src/lib/container-move.ts
#	packages/api/src/lib/derivations-csv.ts
#	packages/api/src/lib/specimens/specimen-read.ts
#	packages/api/src/routes/__tests__/collections.test.ts
#	packages/docs/src/content/docs/guides/bulk-operations/container-movement.md
#	packages/web/src/pages/__tests__/AdminDataIntegrityReport.test.tsx
m-murphy and others added 26 commits November 1, 2026 07:17
The default study's title and short code were Date.now() plus a random
number below 1000. The specimens pagination test creates 60 in a few
milliseconds, so two often collided on study.short_code (2 of 5 runs
failed locally). Use a module counter instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d derivation dates (#133)

- Control batches store `properties` as a JSON object; batch-create and
  batch-with-specimens no longer JSON.stringify it before the JSON column.
  Migration 012 unwraps rows stored as a JSON string holding an object and
  leaves every other row alone, so it is idempotent.
- PUT /api/locations/:id refuses `canContainCollections: false` with a 409
  naming how many Collections the Location holds.
- A derivationDate must be YYYY-MM-DD, optionally followed by a time
  (`isDateOrTimestamp` in lib/date-filters.ts). POST derive, PATCH
  derivation, the derivation CSV (per-row and shared setting) and
  createDerivation itself reject anything else with a 400.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… rows

Specimen import errors now name the row a spreadsheet shows, counting the
header as row 1. Blank rows no longer shift the count, a combined-import
error on a specimen grouped under an earlier subject names that specimen's
row, and a per-subject import failure names every row of the subject.

Derivations import drops rows whose cells are all blank, as the API does,
so they no longer appear in review or get submitted, and validation, review
and result rows are numbered by their file row.

Both parsers share parseCsvFileRows (web lib/csv-file-rows.ts), a small
helper that keeps each data row's spreadsheet row number.

Closes #138

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ect with its specimen

Merge Subjects loaded one page of up to 1,000 subjects and filtered it in the
browser, so a subject past that could not be picked. Each picker now searches
GET /studies/:id/subjects with a new `search` parameter (name contains the
text, taken literally, or the exact subject ID) and lists the first 50 matches,
with their specimen counts from the same response.

Specimen registration with "Create new subject" created the subject, then the
specimen; a failed specimen (e.g. a duplicate barcode) left the subject behind
and a retry failed on it. The form now sends both in one request to
POST /subjects/with-specimens, which writes them in one transaction, and shows
the API's error. Also corrects the web type of that endpoint's response, whose
specimens are flat.

Fixes #135

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	packages/api/src/__tests__/helpers/factories.ts
#135)

POST /subjects/with-specimens takes `newSubjectOnly`. When set and the
study already has a subject with that name, the transaction is rolled back
and the route returns 409 "Subject name '…' already exists in this study",
instead of adding the specimen to that subject. The specimen form sends it,
so a mistyped or reused name can't silently attach a specimen to another
subject. Bulk import is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e data (#107)

- Study summaries, the study summary route and the study timeline order the
  collection date range by calendar date, reading legacy M/D/YYYY values as
  dates, and report it as YYYY-MM-DD. Study duration uses the same dates.
- The study enrollment timeline buckets Subjects by UTC day of `created`
  instead of by the full timestamp, and both daily timelines count with a
  Map instead of a quadratic find.
- The statistics collection timeline reads a collection date's month from
  the date itself, so `5/1/2024` is May in every server time zone.
- Statistics `storage.byLocation` lists every Location holding Containers,
  grouped by Location id, so the Storage Locations card (its length) is the
  real count instead of at most 20. The wire shape is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… JSON rows keep every column (#122)

- CSV-mode count and export send the selected Tags.
- A failed export leaves the modal open with its error; only a successful
  manual export closes it after one second.
- Each count request takes a sequence number; a response, error or
  loading change from a superseded request is ignored.
- A JSON Container export (GET, single-study POST, multi-study, barcode)
  fills every export column, null when empty, and opts out of
  omit-on-wire with keepNullsOnWire. Other responses still omit nulls.
  ADR 0005 records the exception.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…code export (#122)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…13 (#126)

Migration 013 carries out decisions 1, 3, 4, 5, 6 and 8 from #124:
clears the four placeholder collection dates, sets the reviewed
exhausted Containers to remaining 0, empties NA-style cryovial
barcodes, deletes Container 34513 and empty plate 1760, and merges the
two CTRL case-variant Subject pairs the way mergeSubjects does. It then
adds a unique index on (study_id, name COLLATE NOCASE).

Each step is guarded by a temp trigger that aborts on rows the lab did
not review: counts above the snapshot's, exhausted Containers outside
the reviewed id list, a placeholder micronix barcode, a changed
Container 34513 or plate 1760, a renamed merge pair, or any other
case-variant Subject pair.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Create, bulk create and validate, rename, and Combined import refuse a
Subject name that differs from one the study already has only by case,
with "Subject name 'abc-1' differs only by case from Subject 'ABC-1' in
this study". validateSubjectName compares with COLLATE NOCASE, matching
migration 013's unique index, and batch duplicate checks fold ASCII case
the same way. Rename now goes through validateSubjectName, so a Subject
can still change the case of its own name.

CONTEXT.md records the Subject name rule (#124 decision 6) and Specimen
identity (decision 7), including that fresh installs have
idx_specimen_study_subject_type_date and production does not. The user
guide says names ignore case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…#126)

The integrity report adds seven checks, each with a section on the
report page and a count on the overview:

- duplicate Specimen identity (Subject, Specimen type and collection
  date, dated or undated)
- Subject names in one Study that differ only by case
- Collection names that differ only by case or surrounding spaces
  (sheets within their box or bag)
- Containers in a Container type their Specimen type does not allow
- Containers commented "exhausted" with remaining quantity above 0
- the four known placeholder collection dates
- collection dates after today (#124 decision 2)

The duplicate-barcode check already covers micronix and cryovial tubes
(#105); the overview card no longer calls it micronix-only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rting

Migrations run at startup on every install, so an install with a value
their rule can't handle could not start until someone fixed it in SQL.

- 008 converts M/D/YY and M/D/YYYY collection dates that are real
  calendar dates, trims whitespace and nulls blanks as before, and leaves
  every other value exactly as it was.
- 010 applies a repair only when no other Container holds, or another
  repair would take, the same position in that Collection; every repair
  to a contested position is left, so none is preferred. A position that
  fits no repair case is left as it was.
- The integrity report's existing "Collection date not ISO" and "invalid
  Grid position" checks list what is left; the tests run them on the
  migrated database. Docs and CONTEXT.md say what is left and why.

On the 2026-10-31 production snapshot the result is identical to the
aborting versions: 1,148 dates converted, 160 positions normalized and
206 cleared, nothing left over.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… ranges and sorts

Migration 008 now leaves a collection date it can't read (13/2/2024) as
it was. Compared as text, such a value passed or failed a date bound by
accident and became the earliest or latest date of a range.

- Specimen list, statistics and Container selection date filters also
  require a real YYYY-MM-DD date (isCalendarDate in sql-helpers.ts).
- Sorting the specimen list by collection date puts such a value after
  every real date and before a missing one.
- Study summary, study summaries, study timeline, subject and control
  batch summaries and the monthly collection timeline read only real
  dates; a subject timeline places such a Specimen by its created time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…126)

Migrations ship to every install, and 013 named our production rows and
carried out our lab's #124 decisions, so on another install it would
either stop startup or change data on our lab's say-so. This reverts
d88b559: CURRENT_SCHEMA_VERSION goes back to 12, and initial_schema.sql,
schema.ts and the legacy fixtures go back with it.

There is no case-insensitive unique index on Subject names: it would
stop startup on any install that already holds such names. The write
paths from 30ea366 enforce the rule, and the integrity report lists
existing pairs. CONTEXT.md, the code comments, the integrity report page
and the user guide no longer mention migration 013, our lab's choices
or our snapshot's counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ile (#126)

`bun run data-fix <fix.sql> [--db=…] [--dry-run]` in packages/api runs a
site-specific SQL file against a database by hand. It refuses a database
not at CURRENT_SCHEMA_VERSION, copies the database first with VACUUM INTO
(<file>.pre-data-fix-<name>-<timestamp>.bak, sharing migration-backup.ts),
runs the statements in one BEGIN IMMEDIATE transaction with foreign keys
on, and records name, checksum and time in data_fix_log, a table the
runner creates in the fix's transaction rather than a numbered
migration. A second run does nothing and says when the fix ran; a changed
file under the same name is refused. --dry-run rolls back and prints the
rows each statement would change.

ops/data-fixes/2026-11-01-eppicenter-legacy-cleanup.sql carries the
former migration 013's decisions 1, 3, 4, 5, 6 and 8 from #124 with its
guards, minus the Subject name index, and a header saying it is only for
the EPPIcenter database. Its tests moved to
src/db/__tests__/data-fixes/.

The db README, the deployment and backup guides, and CONTEXT.md (Data
fix) document the runner and say other installs must not run our fixes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	packages/api/src/lib/statistics/specimen-aggregates.ts
#	packages/api/src/lib/studies/study-read.ts
#	packages/api/src/routes/studies.ts
m-murphy and others added 3 commits November 2, 2026 15:33
…cleanup

Specimen type Polyclonal Ab (47) had no allowed Container types, so the
integrity report listed the 8 micronix tubes holding Specimen 141173 as
being in a disallowed type. The lab decided the type allows micronix
tubes. The fix adds that row, guarded on type 47 still being named
Polyclonal Ab; a database without type 47 is left alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each settled keystroke changed the subjects query key, so the picker
dropped to "Searching subjects..." and unmounted every option until the
new page arrived. A click landing in that window hit a detached button and
selected nothing, which is how "searches the API, so a subject past the
first 1,000 can be merged" failed in CI. useStudySubjects takes
keepPreviousResults (placeholderData: keepPreviousData) and the picker
uses it; a new test holds the newer search open and clicks an option.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…finding options

The picker searches 250 ms after typing stops; on a loaded CI runner the
result rendered after findByRole's default 1 s, so the option lookups for
P01050 and P01052 timed out. findOption waits for the search request and
gives the lookup 5 s, as ExportModal.test.tsx does for its slow waits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant