feat(core): seal personal data in scheduled commands and dead letters, cancelled by forget

.changeset/personal-data.md

@@ -8,3 +8,5 @@
 ---
 
 Personal data in events, forgotten by crypto-shredding. An event's `payload` receives `personal` and `person` next to `z`: `personal(schema)` marks a field as personal data, stored encrypted (AES-256-GCM through WebCrypto, on Node and in `workerd`) with a key of the person the event names, `person(schema)` marks the field that names them, and without it the aggregate's id does. `app.forget(personId)`, on every `BoundaClient`, deletes the key and appends `PersonalDataForgotten` to the stream `bounda:<personId>`, in one transaction: from then on every personal field of theirs reads as `null`, typed `T | null` in every handler. A read model that projects an event with personal fields must handle it from `projections/bounda/personal-data-forgotten.ts`, which the generator checks in `.bounda/types.ts`; a policy may react to it from `policies/bounda/`. The adapters keep the keys, with their fingerprints, in a new `personal_keys` table, `Storage` and `StorageTransaction` gain `personalKeys`, and the storage contracts cover it. The global stream shows the fields as stored, encrypted. A command that loaded a person's data before `forget` and appends after it fails with `ConcurrencyError` and is decided again, instead of writing their data back under the new key another command may have given them meanwhile: the state remembers the key each person's data was read under, and `AppendArgs` gains `persons` (`{ person, fingerprint }`) for this, which a store ignores. Snapshots and remembered rejections name the same and are written only while that key is still the person's; in PostgreSQL a GIN index on their `persons` lets `forget` find them without reading every row.
+
+A command's `payload` receives `personal` and `person` too (`CommandPayloadArgs`), for the commands that carry personal data: wherever the command is stored, scheduled with `delay`, retried or dead-lettered, its marked fields are encrypted with the key of the person it names, and its handler still receives them as the schema gives them, never `null`, since a command whose person is forgotten never runs. In the same transaction, `forget` cancels the person's scheduled commands and removes their dead letters, and `PersonalDataForgotten` names them in `cancelledCommands` and `removedDeadLetters`. A run already under way when `forget` comes writes nothing, dead letter included, and one sealed under a key its person no longer holds is dropped unrun. `app.deadLetters` shows a scheduled command's payload opened and refuses to retry one whose person was forgotten. `ScheduledCommand`, `ScheduleArgs` and `DeadLetter` gain `persons`, `Scheduler` and `DeadLetterStore` gain `forget(person)`, both tables a `persons` column (with a GIN index in PostgreSQL), and the storage contracts cover them.

docs/src/content/docs/concepts/forgetting-in-an-immutable-log.md

@@ -130,11 +130,9 @@ have work to do when it happens.
   systems a policy wrote to. The key store is a table like the others: a backup of the whole
   database restores the keys too, so a backup kept past a person's erasure is itself a copy of
   their data.
-- **Scheduled commands and dead letters.** A command scheduled with `delay` is stored with its
-  payload, in the clear, until it runs, and a dead-lettered scheduled command keeps it. Keep
-  personal data out of delayed commands, or carry an id and resolve it when the command runs.
-  A dead letter also keeps the error a handler threw, message and stack: one that quotes a
-  decrypted field keeps it in the clear.
+- **Errors.** A dead letter keeps the error a handler threw, message and stack: one that quotes a
+  decrypted field keeps it in the clear. A command's payload marked `personal` is encrypted where
+  it is stored, scheduled or dead-lettered, and `forget` cancels it; a payload left unmarked is not.
 - **Encrypted data under the GDPR.** Whether a ciphertext without a key still counts as personal
   data is a legal question Verraes raises and this page does not answer.
 

docs/src/content/docs/guides/dead-letters.md

@@ -34,7 +34,7 @@ What a retry runs, and when it marks the letter, depends on the kind:
 | Policy | The handler again, for the stored event, with the event's correlation. The inbox is bypassed on purpose: it already says the handler ran, and you are asking for another run. | In the same transaction as what the run writes |
 | Process | The handler again for the stored event, with the instance's current state, or for a deadline's letter (`deadline:<field>`) the handler of that deadline, with a new `idempotencyKey`; an event that completes the process completes it. Then the events parked behind the failure, in order, and the process is `started` again, its deadlines scheduled at their moments (one already past runs at once); see [a failed process](#a-failed-process). | Once the instance has drained what was parked; a retry cut short is taken up by retrying the same letter |
 | Follow-up of a timed-out process | Its handler and nothing else: the process stays timed out, and a handler a deploy removed lets the event through. | In the same transaction as what the run writes |
-| Scheduled command | The dropped command again, with the payload the letter recorded. A rejection settles the letter too: it is the aggregate's answer, logged as `command rejected`. | In the same transaction as what the run writes |
+| Scheduled command | The dropped command again, with the payload the letter recorded, its [personal fields](/guides/personal-data/#in-a-commands-payload) opened. A rejection settles the letter too: it is the aggregate's answer, logged as `command rejected`. | In the same transaction as what the run writes |
 
 Two retries of one letter, or a retry and a discard, never both settle it, even when they run at
 once: whichever gets there second is refused with `DeadLetterSettledError`
@@ -43,8 +43,8 @@ is refused, so a double click does not store a command's decision twice, though
 have run in both; a process's retry has already handled its events by then. A letter the app as
 it now is cannot retry is refused with `DeadLetterNotRetriableError` (`DEAD_LETTER_NOT_RETRIABLE`)
 without running anything: its policy, process or scheduled command is gone from the registry, its
-policy or process no longer handles the event, or its instance failed on another step whose letter
-comes first.
+policy or process no longer handles the event, its instance failed on another step whose letter
+comes first, or the person whose data its scheduled command holds was forgotten since.
 
 The same operations are on the app as `app.deadLetters` — `list`, `count`, `get`, `retry` and
 `discard` — for a script or an admin route.

docs/src/content/docs/guides/personal-data.md

@@ -45,9 +45,30 @@ export const payload = ({ z, personal, person }: Event.PayloadArgs) =>
   });
 ```
 
-An event names at most one person. Only events have the marks: a command's or a query's
-`PayloadArgs` has no `personal`, since commands are not stored and nothing of theirs has to be
-forgotten.
+An event names at most one person.
+
+## In a command's payload
+
+A command is stored too when it waits: dispatched with `delay`, retried by the scheduler after a
+failure, or dead-lettered once it gives up. A command that carries personal data marks its payload
+the same way, with `personal` and `person` from its own `PayloadArgs`:
+
+```ts
+// app/domain/customer/commands/remind-customer.ts
+import type { Command } from "./+types/remind-customer";
+
+export const payload = ({ z, personal }: Command.PayloadArgs) =>
+  z.object({ customerId: z.string(), email: personal(z.email()) });
+```
+
+Wherever the command is stored, the marked fields are encrypted with the key of the person it
+names, the aggregate's id or the field marked `person`, and [`forget`](#forgetting-a-person)
+cancels it. A command dispatched at once is never stored, so the marks change nothing for it. The
+handler reads `command.payload.email` as a `string`, never `null`: a command whose person is
+forgotten never runs. A query's `PayloadArgs` has no `personal`, since a query is never stored.
+
+A command scheduled before its fields were marked keeps its payload as it was written, in the
+clear, and runs with it; `forget` cannot find it.
 
 ## What the types say
 
@@ -69,9 +90,12 @@ await app.catchUpReadModels({ through: forgotten });
 ```
 
 `forget` is on every `BoundaClient`: the app, the Durable Object through `connect`, the `bounda`
-context of React Router. In one transaction it deletes the person's key and appends
-`PersonalDataForgotten`, with `{ personId }` as payload, to the stream `bounda:<personId>`: the
-runtime's own aggregate, `bounda`, has one stream per person it forgot. It resolves like a
+context of React Router. In one transaction it deletes the person's key, cancels the scheduled
+commands that hold their data, removes the dead letters of those that gave up, and appends
+`PersonalDataForgotten` to the stream `bounda:<personId>`: the runtime's own aggregate, `bounda`,
+has one stream per person it forgot. Its payload is `{ personId, cancelledCommands,
+removedDeadLetters }`, the ids and types of what went, the only record of commands that never
+became events. It resolves like a
 command, with the event's position, so `catchUpReadModels({ through })` waits for the read
 models that handle it, and the Durable Object's `forget` waits for them itself unless
 `consistency` is `"eventual"`. A person without a key is forgotten all the same, since a read
@@ -86,6 +110,13 @@ person's data was read under, and an append that would seal their fields under a
 with [`ConcurrencyError`](/reference/errors/), as on a stream that moved. The retry decides on the
 state without their data.
 
+A scheduled command already running when `forget` comes writes nothing: its claim is gone, so
+neither its events nor a dead letter are stored, though its handler may have run. One that a
+`forget` misses, stored in the instant between reading the key and the key going, is sealed under
+a key no one holds: it is dropped when it comes due, unrun, and the worker logs `scheduled command
+dropped: its person was forgotten`. The [dead letters](/guides/dead-letters/) list shows a
+scheduled command's payload with its fields opened.
+
 A [snapshot](/concepts/long-streams/#snapshots) holds the state in the clear, so the same
 transaction removes every snapshot whose state carried a field of theirs; each one names the
 people it holds and the key it read each one under, and none is written for a person who no
@@ -98,14 +129,14 @@ what it answered, and a rejection's message may quote the state it was decided o
 rejection names the people whose fields were in that state, with their keys as a snapshot does,
 and `forget` removes it too: the same request sent again is decided again, on the state without
 them. What it cannot see is personal
-data that came in the command's payload, which is never marked: keep that out of a rejection's
+data that came in the command's payload, marked or not: keep that out of a rejection's
 message and say what went wrong with its code and the ids involved.
 
-Command payloads are never sealed anywhere, so `forget` does not reach what holds one: a command
-scheduled with `delay` keeps its payload until it runs, a
-[dead letter](/guides/dead-letters/) of a scheduled command keeps it until it is retried or
-discarded, and your logs keep whatever you wrote to them. Pass ids in those payloads, not the
-data itself: the handler reads the email from the state, where it is sealed and forgotten.
+What `forget` does not reach is text your code writes and data left unmarked: your logs keep
+whatever you wrote to them, the message of an error a handler throws stays in its dead letter and,
+for a scheduled command, in the `ScheduledCommandFailed` event on the aggregate's stream, and a
+command's payload left unmarked is stored as it came. Log and throw with ids, not the data itself,
+and mark every field of a command's payload that holds it.
 
 ## Handling it in a read model
 

docs/src/content/docs/reference/errors.md

@@ -79,7 +79,8 @@ dead letter with `errorType: "retriable_exhausted"`.
   retried or discarded after another retry or discard had settled it.
 - **`DEAD_LETTER_NOT_RETRIABLE`** (`DeadLetterNotRetriableError`; an operator's answer). A dead
   letter cannot be retried in the app as it is now: its handler is gone or no longer handles the
-  event, or its process failed on another step first. The letter stays `failed`.
+  event, its process failed on another step first, or the person whose data its scheduled command
+  holds was forgotten since. The letter stays `failed`.
 - **`REBUILD_SUPERSEDED`** (`RebuildSupersededError`; an operator's answer). Another rebuild of the
   same read model took over; this one stopped without writing. `readModel`.
 - **`SNAPSHOT_FAILED`** (`SnapshotError`; only in an app from `createTestApp`). A

docs/src/content/docs/reference/limitations.md

@@ -21,10 +21,6 @@ get today. Each item says why, so nobody discovers it the hard way:
 - **Notifications for scheduled commands.** The worker that runs due commands polls at
   `pollInterval`; only the event dispatcher is woken by `NOTIFY`. See
   [Tuning](/guides/deployment/#tuning).
-- **Personal data outside events.** A field marked `personal` is encrypted in the event store and
-  nowhere else: a command scheduled with `delay` keeps its payload in the clear until it runs,
-  and so does the dead letter of one that gave up. Keep personal data out of delayed commands.
-  See [Forgetting in an immutable log](/concepts/forgetting-in-an-immutable-log/#what-stays-outside).
 
 What a production app does get, and where it is explained:
 [rebuilding a read model](/guides/deployment/#rebuilding-a-read-model) without taking it offline,

packages/cli/src/generate/emit/plus-types.ts

@@ -101,7 +101,7 @@ const commandFiles = (aggregate: AggregateModel, typesPath: string): GeneratedFi
       imports: { generated: true, module: command.path },
       namespace: "Command",
       members: [
-        ["PayloadArgs", "core.PayloadArgs"],
+        ["PayloadArgs", "core.CommandPayloadArgs"],
         [
           "RejectionsArgs",
           generic("core.CommandRejectionsArgs", [

packages/core/src/adapter/deferred-writes.ts

@@ -27,8 +27,9 @@ export interface DeferredWriteTarget {
  * every write in order, and `flush` runs the writes recorded so far against another set of the
  * same stores, a transaction's or the live ones. `add` answers the letter as filed, whatever the
  * store holds under that id already; `get` on the idempotency store answers a record `put`
- * recorded before anything `live` holds. `tryClaim`, `claimDue`, both `renew` and `purge` act at
- * once, on `live`, since their answer decides what happens next.
+ * recorded before anything `live` holds. `tryClaim`, `claimDue`, both `renew`, `purge` and the
+ * `forget` of the scheduler and the dead letter store act at once, on `live`, since their answer
+ * decides what happens next.
  */
 export interface DeferredWrites {
   readonly stores: DeferredStores;
@@ -81,6 +82,7 @@ export const deferWrites: DeferWritesFunction = ({
           target.deadLetterStore.updateStatus(id, status),
         ),
         remove: later((target, id) => target.deadLetterStore.remove(id)),
+        forget: deadLetterStore.forget,
       },
       scheduler: {
         claimDue: scheduler.claimDue,
@@ -92,6 +94,7 @@ export const deferWrites: DeferWritesFunction = ({
         fail: later((target, args) => target.scheduler.fail(args)),
         defer: later((target, args) => target.scheduler.defer(args)),
         renew: scheduler.renew,
+        forget: scheduler.forget,
       },
       idempotencyStore: {
         get: async (args) => records.get(args.key) ?? idempotencyStore.get(args),

packages/core/src/adapter/sql/dead-letters.ts

@@ -0,0 +1,13 @@
+import type { DeadLetter } from "../storage/dead-letter-store.ts";
+import { codePointOrder } from "./order.ts";
+
+export interface ByFirstFailureFunction {
+  (a: DeadLetter, b: DeadLetter): number;
+}
+
+/**
+ * The order of `DeadLetterStore.list` and `DeadLetterStore.forget`: `firstFailedAt`, then `id` by
+ * code point. For a store whose engine hands rows back unordered, as `RETURNING` does.
+ */
+export const byFirstFailure: ByFirstFailureFunction = (a, b) =>
+  codePointOrder(a.firstFailedAt, b.firstFailedAt) || codePointOrder(a.id, b.id);

packages/core/src/adapter/sql/index.ts

@@ -1,4 +1,5 @@
 export type { SqlDatabase, SqlTransaction } from "./database.ts";
+export { byFirstFailure } from "./dead-letters.ts";
 export { postgresqlDialect } from "./dialect.ts";
 export { quoteIdentifier, storageTableNameFor, tableNameFor } from "./identifiers.ts";
 export type { ExistingColumn } from "./read-model-schema.ts";

packages/core/src/adapter/sqlite/dead-letter-store.ts

@@ -5,9 +5,12 @@ import type {
   DeadLetterKind,
   DeadLetterStatus,
   DeadLetterStore,
+  KeyedPerson,
   ListDeadLettersArgs,
 } from "../index.ts";
 import type { SqlDatabase } from "../sql/database.ts";
+import { byFirstFailure } from "../sql/dead-letters.ts";
+import { naming } from "./held-keys.ts";
 
 export interface CreateSqliteDeadLetterStoreArgs {
   readonly db: SqlDatabase;
@@ -19,7 +22,12 @@ export interface CreateSqliteDeadLetterStoreFunction {
 }
 
 const COLUMNS =
-  '"id", "kind", "handler", "event_id", "event_type", "aggregate_type", "aggregate_id", "error_type", "error_message", "error_stack", "attempts", "first_failed_at", "last_failed_at", "status", "payload"';
+  '"id", "kind", "handler", "event_id", "event_type", "aggregate_type", "aggregate_id", "error_type", "error_message", "error_stack", "attempts", "first_failed_at", "last_failed_at", "status", "payload", "persons"';
+
+const personsOf = (row: Record<string, unknown>): { persons?: KeyedPerson[] } => {
+  const persons = JSON.parse(String(row.persons)) as KeyedPerson[];
+  return persons.length === 0 ? {} : { persons };
+};
 
 const toLetter = (row: Record<string, unknown>): DeadLetter => ({
   id: String(row.id),
@@ -41,6 +49,7 @@ const toLetter = (row: Record<string, unknown>): DeadLetter => ({
   ...(row.payload === null || row.payload === undefined
     ? {}
     : { payload: JSON.parse(String(row.payload)) as unknown }),
+  ...personsOf(row),
 });
 
 const filters = (
@@ -70,7 +79,7 @@ export const createSqliteDeadLetterStore: CreateSqliteDeadLetterStoreFunction =
   return {
     add: async (letter) => {
       await db.run(
-        `INSERT INTO ${table} (${COLUMNS}) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'failed', ?) ON CONFLICT ("id") DO NOTHING`,
+        `INSERT INTO ${table} (${COLUMNS}) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'failed', ?, ?) ON CONFLICT ("id") DO NOTHING`,
         [
           letter.id,
           letter.kind,
@@ -86,6 +95,7 @@ export const createSqliteDeadLetterStore: CreateSqliteDeadLetterStoreFunction =
           letter.firstFailedAt,
           letter.lastFailedAt,
           letter.payload === undefined ? null : JSON.stringify(letter.payload),
+          JSON.stringify(letter.persons ?? []),
         ],
       );
       return (await get(letter.id)) as DeadLetter;
@@ -115,5 +125,9 @@ export const createSqliteDeadLetterStore: CreateSqliteDeadLetterStoreFunction =
       if (updated.length === 0) throw new DeadLetterSettledError({ id });
     },
     remove: (id) => db.run(`DELETE FROM ${table} WHERE "id" = ?`, [id]),
+    forget: async (person) =>
+      (await db.all(`DELETE FROM ${table} WHERE ${naming(table)} RETURNING ${COLUMNS}`, [person]))
+        .map(toLetter)
+        .sort(byFirstFailure),
   };
 };