Skip to content

Commit a batch of card writes all-or-nothing, under one index job and one event - #6072

Open
habdelra wants to merge 15 commits into
mainfrom
cs-12792-card-ops-createupdatedelete-executors-and-the-all-or-nothing
Open

Commit a batch of card writes all-or-nothing, under one index job and one event#6072
habdelra wants to merge 15 commits into
mainfrom
cs-12792-card-ops-createupdatedelete-executors-and-the-all-or-nothing

Conversation

@habdelra

@habdelra habdelra commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

What this adds

A batch of card writes that lands all-or-nothing: several changes to several
cards that either all commit or none do, under one index job and one index
event.

Executors (packages/runtime-common/card-operations/executors.ts) — each
base write operation is a pure staging function. It reads the batch's
pre-loaded state, works out the exact bytes every file it touches should end
up holding, and returns them. It writes nothing, enqueues nothing, broadcasts
nothing, and rewrites only its own copies of what it was handed.

  • stageCreate decides the new card's URL and directory, resolves the links
    it declares, and serializes it against its type's definition — from a
    JSON:API document, or from a named create's lowered of + fill template
    with params() / actor() / instance() resolved by substitution.
    included resources become additional creates linked by lid.
  • stageUpdate merges a patch over the stored file under the rules the
    PATCH handler applies: the merge base is the file rather than the index, a
    changed adoptsFrom is refused, type / meta.realmInfo / meta.realmURL
    / meta.screenshots are stripped, arrays replace rather than merge,
    relationships merge, and no change means no write.
  • stageDelete establishes that its target is a card that is there to remove.
    A stored .json that is not a card document — a config, a fixture, anything
    the realm keeps but does not serve as a card — is refused with the 404
    DELETE gives for the same URL. The bytes already read answer that, so it
    costs no read and keeps a card written moments ago deletable, which
    consulting the index would not. A realm's own realm.json is not covered
    by this and is not meant to be: it is itself a card document, and DELETE
    removes it too, so refusing it here would put the batch out of step with the
    endpoint rather than protect anything the endpoint protects.

A create mints a card rather than replacing one, so a destination a stored
card already occupies refuses the batch with 409 — the answer the atomic
endpoint gives an add whose href is taken. A created card's id and directory
come from the caller and are spliced into a path, so both are held to naming
the file the card is stored in: each has to be a plain path segment, and the
path that comes back out of the URL has to be the path that went in. Together
those refuse a .. in any spelling walking out of the type's directory, and a
? or # cutting the stored path short so the id and the file stop naming
each other.

A local id links a card only where a link can be recorded. A collection's
edges are stored one key per member — field.0, field.1 — so a local id
written inside a single data array has no key of its own to carry the link
and is refused rather than staged with the edge silently missing.

The coordinator (card-operations/coordinator.ts) takes the realm's write
lock once, drains indexing already in flight, resolves every lid up front,
reads every target's file, runs every executor in memory, and only then
commits. A lid resolves by path math over the type a card adopts — no read,
no write — so a create's URL is known before anything is written and a later
entry can link to it. An entry that cannot be carried out is therefore found
while the realm is still untouched: nothing to undo, no index job to cancel,
no event a subscriber could have acted on. A duplicate lid and a link to one
no entry declares are each refused with the position of the entry that
produced them.

A batch is a sequence, so two entries may name one card and the second builds
on the first: it merges over the bytes the first staged, and the commit writes
the file once holding the last of them. What composes is a change over a
change. Three things do not, and are refused rather than resolved by
ordering:

  • A write to a path an earlier entry removed. The two entries ask for
    opposite things, and cancelling the removal would report it as completed —
    which a caller cannot tell from a removal that happened.
  • A side-loaded resource landing on a card another entry changed. A side-load
    is serialized whole rather than merged, so it cannot compose over anything;
    letting it land last would drop the earlier change with nothing said while
    that entry still reported success.
  • A removal aimed at a path the realm ignores. An ignored file is never
    visited, so it never gets an index row, and DELETE — which needs one —
    refuses it permanently. Reading its bytes off disk is not the same
    permission.

The corners that do fall out of the sequence: a change after a removal finds
nothing there and refuses as it would outside a batch; a removal after a
change drops that change's write rather than writing bytes the same commit
then unlinks, and both entries report no state; and a card the batch mints is
not folded in, so it is still reached by the lid other entries link to
rather than by a URL they target.

Checks the realm applies per file as it writes are applied to the staged bytes
first, for the same reason. The size ceiling refuses an oversized payload
while the realm is still untouched, carrying the realm's own 413 through under
a payload-too-large code — the status is the remedy, since it tells a caller
to send less rather than to send again. _screenshot/ is refused as a write
destination, as it is for direct writes and for /_atomic, because a realm
file stored there could never be read back; removals stay admitted there, as
they do in both other paths, being the recovery route for anything already
stored. An empty batch takes no lock and announces nothing — an empty index
event would tell every subscriber that something changed.

That guarantee is over the entries. It is not over the file system: the realm
changes a batch's files one at a time with no rollback, as it does for every
multi-file write it serves, so a mid-commit failure leaves the files handled
before it changed. Both the coordinator's header and _commitBatchUnlocked
say where that line falls rather than implying there isn't one.

Realm._commitBatchUnlocked is the batch-write path extended with a
removal leg, so writes and removals commit under one index job and one index
event rather than two. FileWriteResult gains contentHash — the file's
version, and the token a later request passes as baseVersion.

A baseVersion is reported against the version the entry actually merged
over, captured as that entry stages: within a batch the state moves, so the
second change to one card is compared to what the first staged rather than to
what the batch started from. That fingerprint comes from the bytes read inside
the write lock rather than from the file's recorded row — a file changed out
from under the realm is re-indexed without being rewritten, so the row can
name a version the bytes no longer hold, which is the exact case a base
version exists to catch. A file whose bytes are already what the caller staged
records its hash on the row as well as reporting it, which fills the row the
file's metadata resource reads.

RealmIndexUpdater.enqueueChanges is the general form of enqueueUpdate: one
job over a change set that may mix removals with updates, so the invalidation
fan-out for all of them is computed against one snapshot of the realm.
enqueueUpdate is that with a single operation for the whole set, unchanged
for its callers.

Not in scope

No existing handler or route changes. The POST / PATCH / DELETE
handlers keep their own copies of this logic; the facade that dispatches them
through the coordinator removes those copies.

Two deliberate divergences for that facade to settle. A batch create refuses
an occupied destination, where the plain POST handler overwrites it today.
And a batch update writes a card under a _-prefixed path, which POST,
DELETE and /_atomic all admit and only PATCH refuses — the batch sides
with the three rather than the one.

Test plan

packages/runtime-common/tests/card-operations-batch-test.ts — 52 tests
against a stubbed realm, covering what the coordinator decides before it
commits: the paths a create stages at, lid resolution across entries (on the
entry and on the resource), side-loaded creates and side-loads over stored
cards, the patch merge rules down to relationship-map merging and the field
metadata a replaced array drops, two entries composing over one card and the
version each reports, the three conflicts that do not compose, a change after
a removal and a removal after a change, a minted card not being a later
entry's target, a realm config removed at parity with the endpoint,
realm-managed keys never reaching the file, an id that would name a file
outside the type it creates, an occupied destination, a malformed side-load,
an unlinkable local id, a named create missing a declared param or an actor,
bytes over the realm's size ceiling and the 413 they answer with, a write into
the capture subtree, a removal aimed at stored JSON that is not a card, an
empty batch, that indexing is drained before anything is serialized, that the
caller's own document is left alone, every refusal and its entry position, and
— the point of the stub — that a refused batch never calls the commit at all.

packages/realm-server/tests/card-operations-commit-test.ts — 7 tests against
a real realm, covering what only a realm can show: a create-and-link batch's
files and resolved links on disk; a failing second entry leaving the first
unwritten with no incremental-index row in the jobs table and no realm event;
a write and a removal producing exactly one index job and one index event
carrying both URLs; baseMatched true and false; a no-op patch leaving the
file's bytes and modification time alone with nothing queued, and recording on
the row the version it reports; and an update entry writing a file
byte-identical to a PATCH of the same document through the existing handler.

Locally green: 192 tests across the two new files, card-endpoints-test,
atomic-endpoints-test and atomic-batch-indexing-test — all three untouched
— and card-operations-core-test and card-operations-dispatch-test.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EBB6JdRo5XCY5WhUkgBckS

Hassan Abdel-Rahman and others added 4 commits September 10, 2026 13:55
…oordinator

A batch of card writes needs to land together: several changes to several
cards that either all commit or none do, under one index job and one index
event. This adds the pieces that make that possible.

Each base write operation becomes a pure staging function in
card-operations/executors.ts: it reads the batch's pre-loaded state and
returns the exact bytes every file it touches should hold, writing nothing.
`stageCreate` decides a new card's URL and directory, resolves its links and
serializes it — from a JSON:API document, or from a named create's lowered
`of` + `fill` template. `stageUpdate` merges a patch over the stored file
under the same rules the PATCH handler applies. `stageDelete` establishes
that its target is there to remove.

card-operations/coordinator.ts takes the realm's write lock once, resolves
every `lid` up front (path math over the type a card adopts, so a create's
URL is known before anything is written and a later entry can link to it),
runs every executor in memory, and only then commits. A refusal therefore
happens while the realm is still untouched.

Realm gains `_commitBatchUnlocked`, which is the batch-write path extended
with a removal leg so writes and removals commit under one index job and one
index event, and `FileWriteResult.contentHash`, which is the version a caller
passes back as `baseVersion`. RealmIndexUpdater gains `enqueueChanges` for a
change set that mixes removals with updates; `enqueueUpdate` is that with one
operation for the whole set.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EBB6JdRo5XCY5WhUkgBckS
… loop's index arithmetic

The staging decisions — which files a batch would write, where each local id
resolves, whether the batch reaches the commit at all — are checkable without
a realm, and checking them against a stub is what makes "nothing is committed"
an assertion about the property itself rather than a proxy for it. The realm-
driven suite keeps what only exists against a realm: bytes on disk, one index
job, one index event, and byte-for-byte agreement with the PATCH handler.

`stageCreate` handles its primary before its side-loaded resources rather than
walking one list and branching on the index, which drops two non-null
assertions and lets both it and `stageUpdate` share one side-loaded staging
step.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EBB6JdRo5XCY5WhUkgBckS
…dex event

A created card lands under its type's directory, so a root-relative module
reference in its `adoptsFrom` resolves against that directory rather than the
realm — the test was asking for a module that is not there. Say where module
references resolve from, in the one place that decides it, and have the test
name the module the way a caller who does not want to reason about the
directory would.

The realm broadcasts into the Matrix room out of band from the commit, so the
one-event assertion waits for the batch's own event to arrive rather than
reading the room once.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EBB6JdRo5XCY5WhUkgBckS
Five of the tests were refusals the coordinator makes before it commits — a
duplicate local id, a link to one nothing creates, a removal with no target,
two entries on one card, a type change. None of them reach the realm, and all
of them are already checked against the stub, where "nothing is committed" is
the property itself. What is left is what a stub cannot show: bytes on disk,
one index job, one index event, and byte-for-byte agreement with the PATCH
handler.

Each remaining test now works on its own card, so one realm serves the whole
file. Rebuilding it per test bought no isolation and cost a Matrix session
room each time — enough of them in a burst to trip Synapse's room-creation
limit and fail a test on the realm's inability to open a session.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EBB6JdRo5XCY5WhUkgBckS
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 10, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-10T14:43:53.714732Z 45bf428 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

await this.trackOwnWrite(path);
let { lastModified } = await this.#adapter.write(path, content);

P1 Badge Roll back files when a batch commit fails partway

If a later adapter write or removal fails—for example because the disk becomes full or a path is unwritable—earlier iterations have already modified their files, and this method has no rollback or atomic rename strategy before rejecting commitBatch. The advertised all-or-nothing operation can therefore leave a partially applied batch on disk and out of sync with the index; the commit needs transactional staging/rollback across every write and delete.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/runtime-common/card-operations/executors.ts Outdated
Comment thread packages/runtime-common/realm.ts Outdated
A `lid` and a directory come from the caller and are spliced into a path, so
both are checked before a URL is built from them. A `lid` of `../realm` — or
`%2e%2e/realm`, which no character check catches — resolved out of the type's
directory and onto the realm's own config file, and a `?` or `#` cut the
stored path short so the card's id and its file stopped naming each other.

Two checks, because neither covers the other: each has to be a plain path
segment, and the path that comes back out of the URL has to be the path that
went in.

An unchanged file now records the hash it reports as well as returning it. A
file written before the realm recorded hashes carries none on its row, so
returning a token the row does not hold made the next write quoting it as
`baseVersion` report a moved base for a file that had not moved.

Refusals from the local-id pre-pass carry the entry's position, like the ones
executors raise.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EBB6JdRo5XCY5WhUkgBckS
@habdelra
habdelra requested a lite review from Copilot September 10, 2026 15:01
…s not

The guarantee is over the entries: every way an entry can be wrong is found
while the realm is still untouched. It is not over the file system. The realm
changes a batch's files one at a time with no rollback, so a mid-commit
failure leaves the files handled before it changed — as it does for every
multi-file write the realm serves. Reaching past that needs transactional
staging in the write primitive, which is not something the coordinator can do
above it, so the comments say where the line is rather than implying there
isn't one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EBB6JdRo5XCY5WhUkgBckS

Copy link
Copy Markdown
Contributor Author

[Claude Code 🤖]

Answering the third finding from the Codex review — "Roll back files when a batch commit fails partway". It was left as the review's top-level body rather than an inline thread, so it needs a reply here.

The finding is accurate about the behavior. _commitBatchUnlocked changes files one at a time, so a failure partway — a full disk, a path that will not open — leaves the files handled before it changed, and the method rejects with the realm in that state.

It is not introduced here. _batchWriteUnlocked on main writes in the same per-file loop with no rollback and no atomic rename, so every multi-file write the realm serves already behaves this way; /_atomic pushing fifty instances has exactly the same exposure. This change adds a removal leg to that loop, and inherits the property rather than creating it.

What I have not done, and why. The remedy the finding names — transactional staging and rollback across every write and delete — is a change to the realm's write primitive, affecting every caller of it, and it is not something the coordinator can provide from above. Doing it here would widen this change well past the coordinator and would put the realm's whole write path under a rewrite that deserves its own review rather than riding along.

What I have done is stop the PR from claiming more than it delivers, in c87abb25. The guarantee is over the entries — a malformed document, a missing target, a rejected field, a local id naming nothing or naming two cards, all found while the realm is still untouched, which is what a caller composing a batch controls. Both the coordinator's header and _commitBatchUnlocked's comment now say that, and say plainly that the file system underneath is not covered.

Worth its own ticket, and I want to name the sharper half rather than leave it implied: when a write fails partway, the method rejects before enqueueing the index job, so the files that did land are on disk and never indexed. Disk and index then disagree until an unrelated edit or a full reindex. That divergence is the more damaging half of this and is probably cheaper to close than full transactional staging.


Generated by Claude Code

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved critical atomicity and overwrite risks, along with validation gaps, block approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds staged, all-or-nothing card batches with one realm commit, index job, and event.

Changes:

  • Adds create, update, and delete staging with local-ID resolution.
  • Supports mixed writes/removals, content hashes, and unified indexing.
  • Adds coordinator and real-realm coverage for atomicity and indexing.
File summaries
File Reviewed changes and findings
packages/runtime-common/tests/card-operations-batch-test.ts Tests staging, link resolution, validation, and refusal behavior.
packages/runtime-common/realm.ts Commits mixed changes and records hashes. Critical (1 vote): sequential commit operations can partially persist on failure.
packages/runtime-common/realm-index-updater.ts Supports mixed update/removal index change sets.
packages/runtime-common/card-operations/types.ts Defines batch result and version metadata.
packages/runtime-common/card-operations/index.ts Exports batch operations and staging types.
packages/runtime-common/card-operations/executors.ts Stages card mutations and named creates. Findings: critical (1 vote) possible overwrite of existing create destinations; moderate (3 votes) missing named-create parameter validation; moderate (1 vote each) insufficient included validation for creates and updates, mutation of caller-owned resources, and inconsistent lid-less included-resource handling.
packages/runtime-common/card-operations/coordinator.ts Coordinates validation, locking, staging, and commits. Moderate (1 vote each): empty lids bypass validation, and primary document.data.lid values are not indexed.
packages/realm-server/tests/card-operations-commit-test.ts Tests filesystem commits, indexing, events, hashes, and no-op behavior.
packages/realm-server/tests/card-operations-batch-test.ts Registers shared batch-operation coverage.
Review details

Suppressed comments (6)

packages/runtime-common/card-operations/coordinator.ts:253

  • [Claude Code 🤖] The truthiness check treats an explicitly supplied empty string as if no lid was supplied, so create({ lid: '' }) bypasses assertPathSegment, mints a UUID, and returns no lid instead of rejecting the invalid caller id. This disagrees with the included-resource path, which validates any string lid; distinguish undefined from an empty string before indexing and staging (including createIdentity).
      let primary = entry.document?.data;
      if (entry.op === 'create' && entry.lid) {
        claim(
          entry.lid,
          createIdentity(entry, primary, paths, new Map()),

packages/runtime-common/card-operations/coordinator.ts:254

  • [Claude Code 🤖] This indexes only entry.lid, but a raw JSON:API create carries its local id on document.data.lid (the same shape accepted by createCard). A standard payload with data.lid = "owner" therefore gets a generated URL, and any later relationship referring to owner is rejected as an unknown local id. Normalize/validate the primary resource lid into the staged identity before indexing and staging.
      if (entry.op === 'create' && entry.lid) {
        claim(
          entry.lid,
          createIdentity(entry, primary, paths, new Map()),
          `entry ${index}`,

packages/runtime-common/card-operations/executors.ts:217

  • [Claude Code 🤖] This loop only filters lid and foreign-realm resources; it never validates that included is an array or that each side-loaded value is a card resource. A malformed side-load can therefore reach stageSideLoaded/the serializer (or fail as a 500), unlike the existing POST validation in realm.ts:6044-6058. Validate the container and every included resource before staging.
  for (let resource of entry.document?.included ?? []) {
    // A side-loaded resource with no `lid` is not staged: it has no id to be
    // created under and nothing in the batch can link to it, so the client
    // sent a resource the realm has no way to name. One naming another realm
    // is not this batch's to write.
    if (
      typeof resource.lid !== 'string' ||
      namesForeignRealm(resource, ctx.realmURL)
    ) {
      continue;

packages/runtime-common/card-operations/executors.ts:396

  • [Claude Code 🤖] The update path has the same validation gap: malformed included values are accepted past this loop and can be passed to stageSideLoaded/serialization, whereas the existing PATCH handler rejects a non-array or non-card side-load with 400 at realm.ts:6204-6219. Apply the same included-resource validation before staging update side-loads.
    for (let resource of included) {
      if (
        typeof resource.lid !== 'string' ||
        namesForeignRealm(resource, ctx.realmURL)
      ) {
        continue;
      }
      writes.push(await stageSideLoaded(resource, resource.lid, url.href, ctx));

packages/runtime-common/card-operations/executors.ts:245

  • [Claude Code 🤖] stageSideLoaded mutates the caller-owned included resource via promoteStagedLinks and visitModuleDeps. If a later entry fails, the batch has still changed its input, and retrying the same entries can stage already-resolved module/link values; this contradicts the executor's documented pure-staging contract. Clone the resource before these rewrites and serialize the clone.
  let identity = stagedLid(lid, ctx);
  promoteStagedLinks(resource, ctx);
  visitModuleDeps(resource, (moduleId, setModuleId) => {
    setModuleId(ctx.resolveModuleId(moduleId, relativeTo));
  });

packages/runtime-common/card-operations/executors.ts:397

  • [Claude Code 🤖] This silently drops an included resource whose lid is absent, while the existing PATCH handler writes such an in-realm resource using a generated UUID (realm.ts:6392-6405). Consequently the same patch produces different files depending on whether it goes through the batch path, despite the stated PATCH merge parity. Either apply the same generated identity here or reject lid-less included resources consistently.
    for (let resource of included) {
      if (
        typeof resource.lid !== 'string' ||
        namesForeignRealm(resource, ctx.realmURL)
      ) {
        continue;
      }
      writes.push(await stageSideLoaded(resource, resource.lid, url.href, ctx));
    }
  • Files reviewed: 9/9 changed files
  • Comments generated: 3
  • Review effort level: Lite

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread packages/runtime-common/card-operations/executors.ts
Comment thread packages/runtime-common/realm.ts
Comment thread packages/runtime-common/card-operations/executors.ts
…ing gaps

A create mints a card; it does not replace one. A caller choosing its own
local id can choose one a stored card already answers to, and committing over
it destroyed that card with nothing said. Each staged entry now reports the
files it brings into existence, and an occupied destination refuses the batch
inside the same lock the commit runs in — the answer the atomic endpoint
already gives an `add` whose href is taken.

Four gaps alongside it:

- A named create validates every param its declaration asks for before
  substituting any of them. A missing value resolved to `undefined` and the
  field was left off the card, so a caller who forgot one got a card quietly
  missing it rather than being told.
- `included` is held to being a list of card resources. A malformed side-load
  reached the serializer and came back as an internal failure, where the card
  endpoints answer the caller with a 400.
- A side-loaded resource is rewritten on a copy. Staging is what makes a batch
  abandonable, and rewriting in place left the caller's own document carrying
  resolved links after a batch that committed nothing.
- A create's local id is read from the resource as well as the entry, the way
  a POST body carries it. A payload naming its card only on the resource was
  minted under a generated id, and relationships elsewhere in the batch
  pointing at that local id resolved to nothing.

The local-id pre-pass reads both the local id and the side-loads through the
same accessors the executors use, so it claims the identities staging will ask
for and refuses a malformed payload in the same terms.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EBB6JdRo5XCY5WhUkgBckS

Copy link
Copy Markdown
Contributor Author

[Claude Code 🤖]

The Copilot review listed six further findings as suppressed comments in its body, so they have no threads to answer on. Four are real and fixed in d1159675; two do not reproduce. Taking them in the order they were listed.

Fixed

  • included is not validated, on creates and on updates. Right, and the consequence was a status regression: a non-array included, or a side-load that is not a card resource, reached the serializer and came back as an internal failure where the POST and PATCH handlers answer the caller with a 400. Both executors now read side-loads through one accessor that holds them to that shape. Covered by a malformed side-load is refused rather than reaching the serializer.

  • stageSideLoaded mutates caller-owned resources. Right, and it contradicted the executors' documented pure-staging contract — the primary was cloned on the way in, side-loads were not. A batch that committed nothing still left the caller's document carrying resolved links and absolutized modules, which a retry of the same entries would then stage on top of. Rewrites now happen on a copy. Covered by staging leaves the document it was handed alone.

  • Primary document.data.lid is not indexed. Right, and this one is a compatibility break rather than a nicety: a raw JSON:API create carries its local id on the resource, the way a POST body does, and such a payload was minted under a generated id — so every relationship elsewhere in the batch naming that local id was rejected as unknown. Both spellings now resolve through one accessor, used by the pre-pass and the executor alike. Covered by a create names its card by the local id on the resource.

  • Named-create parameters, and the occupied-destination overwrite. Answered on their own threads.

Does not reproduce

  • "Empty lids bypass validation." The claim is that create({ lid: '' }) mints a UUID and returns no lid. It does not: the fallback is entry.lid ?? uuidV4(), and ?? does not fall back on '', so the empty string is carried into assertPathSegment and refused. Probed directly against createIdentity:

    lid=undefined -> Person/eabd82b9-50e2-477e-bafc-2877868a62e3.json
    lid=""        -> REFUSED: invalid-params: id "" is not a single path segment
    lid="ok"      -> Person/ok.json
    

    A 400 either way; the reading that it silently mints one rests on || semantics the code does not use.

  • "Inconsistent lid-less included-resource handling." The claim is that PATCH writes a lid-less in-realm side-load under a generated UUID while the batch path drops it. PATCH drops it too. Its loop opens with

    if ((i > 0 && typeof resource.lid !== 'string') || ) { continue; }

    so a side-load without a string lid never reaches the resource.lid ?? uuidV4() below — that fallback is only live for i === 0, the primary, which is addressed by its own href. The batch path skips exactly the same resources, so the two agree and the stated parity holds.

Verification. 32 stub tests (80 assertions) and all 7 realm-driven tests pass against d1159675, including the create-and-link test, which confirms the new occupied-destination check does not refuse a legitimate create. card-endpoints-test and atomic-endpoints-test are running.


Generated by Claude Code

@github-actions

github-actions Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Host Test Results

    1 files  ±0      1 suites  ±0   2h 26m 2s ⏱️ - 6m 53s
4 739 tests ±0  4 725 ✅ ±0  14 💤 ±0  0 ❌ ±0 
4 754 runs  ±0  4 740 ✅ ±0  14 💤 ±0  0 ❌ ±0 

Results for commit 9de3b7a. ± Comparison against earlier commit 5e0b5b9.

Realm Server Test Results

    1 files  ±0    216 suites  ±0   1h 17m 4s ⏱️ - 2m 35s
2 865 tests ±0  2 865 ✅ +1  0 💤 ±0  0 ❌  - 1 
2 904 runs  ±0  2 904 ✅ +1  0 💤 ±0  0 ❌  - 1 

Results for commit 9de3b7a. ± Comparison against earlier commit 5e0b5b9.

… staging gaps

A batch's promise is that a refusal costs nothing: the realm untouched, no
index job queued, no event out. Several checks the realm applies per file as
it writes were only reaching the batch inside the commit, where a refusal
already costs the files written ahead of it.

- The size ceiling is applied to every staged write up front, so a payload
  the realm will not store is refused while nothing is written.
- Indexing already in flight is drained inside the lock before anything is
  staged. Staging serializes a card against its type's definition, and a
  module written moments earlier may still be indexing.
- An empty batch returns without taking the lock; broadcasting an empty index
  event would tell every subscriber that something changed.

A `baseVersion` is now compared against a fingerprint of the bytes just read
rather than the file's recorded row. A file changed out from under the realm
is re-indexed without being rewritten, so the row can name a version the
bytes no longer hold — the exact case a base version exists to catch.

The staging gaps:

- A create's side-loaded resource keeps the module references it was sent
  with. Resolving them against the primary resolved them against a card one
  directory deep, which is not where the side-load lands.
- An update stages no mints. A side-load addressed by a caller-chosen local
  id may name a card the caller is deliberately rewriting, which is what the
  PATCH handler this mirrors does; treating it as a mint answered 409 where
  the handler succeeds.
- An update carrying no `data` is refused rather than dereferenced.
- A local id inside a `data` array is refused rather than silently dropped.
  A collection's edges are stored one key per member, so a local id written
  there has no key of its own to record the link on.
- A template reading `actor` with none in scope is refused, and an
  `instance(…)` attribute is read as an own property.

The id-to-path round trip refuses a path that resolves outside the realm
rather than throwing out of `RealmPaths.local` while building its message.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EBB6JdRo5XCY5WhUkgBckS
The realm is shared across this file's tests and index broadcasts are
fire-and-forget, so a straggler from an earlier test can land inside the
window and fail an assertion counting events over it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EBB6JdRo5XCY5WhUkgBckS
@habdelra

Copy link
Copy Markdown
Contributor Author

[Claude Code 🤖] Requirements landed this week that this coordinator will grow into; nothing here needs to change in this PR, but four shape choices are cheap now and expensive later:

  • StagedChange will gain an append leg. A new appendLine operation appends bytes to a text file without reading it, so _commitBatchUnlocked will need to carry appends: { path, content }[] next to writes and deletes and call an adapter append primitive. If it is easy to keep the change shape a record of legs (rather than folding everything into writes), the append arrives as one more leg rather than a reshaping.
  • Staging will run concurrently. Batches will be able to contain parallel groups whose members stage at the same time (Promise.all over stageEntry), with the commit still all-or-nothing under the one lock. readStoredFiles already loads everything up front, which is the right precondition; the ask is that stageEntry stays a pure function of (entry, stored files, context) with no shared mutable state, so a later Promise.all is a scheduling change and not a correctness one.
  • Targets will sometimes be resolved before staging. An entry will be able to name its target by a query against the realm's index (or by hopping one link from a queried card), and the coordinator resolves those into ordinary paths before staging. readStoredFiles' "collect the paths from entry.href" step is the natural place for that resolver to plug in — keep path collection a distinct pre-step rather than something stageEntry does for itself.
  • Two parallel writers on one path will be a batch error, not last-wins. assertDestinationsFree already checks minted paths against disk; a sibling check that two staged changes in the same parallel group touch the same path can sit right beside it later.

The card-operations barrel is the entry point a consumer imports directly
rather than through the package barrel, precisely so it costs less than the
package barrel does. Reaching back through `../index.ts` for one function
gives that subpath an eager dependency on the whole package.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EBB6JdRo5XCY5WhUkgBckS
@habdelra
habdelra changed the base branch from cs-12791-card-ops-operation-core-types-dispatch-the-read-executor to main September 10, 2026 19:59
@habdelra

Copy link
Copy Markdown
Contributor Author

[Claude Code 🤖] Treating the actor as a bare identity string here is right, and it is the shape the rest of the stack is converging on: actor() returns only the caller's user id, takes no argument, and is never a card. Two follow-ons, neither blocking:

  • In the fill-template resolver, the marker.key === 'id' allowance can go. The typed reference is losing its argument, so refusing every keyed actor marker is the simpler rule and matches what lowering will record at index time.
  • An actor marker on a field whose declared kind is link should be refused (400) rather than written as an attribute. The lowering will record it as an authoring issue when the module is indexed; the runtime check is the backstop for a stale definition. Writing the user id into a link's attribute slot leaves a card whose link can never resolve.

If this PR ends up assembling a BxlMutationContext for any program, pass the actor string straight through and keep that construction in one place. The context type's actor is changing from { id, ... } to a plain string, and one call site makes that a one-line switch.

Copy link
Copy Markdown
Contributor Author

[Claude Code 🤖]

Agreed on the actor being a bare identity string — that is what this resolver already returns, and there is no code path here that builds an actor record. On the two follow-ons: the first is right about the destination but cannot land ahead of the lowering change, and the second does not reproduce. Both probed against the coordinator rather than read off the source.

Dropping the marker.key === 'id' allowance would refuse what lowering emits today. Lowering narrows a keyless actor() to actor("id") and wraps it as card(actor("id")) when the declaring type's fieldDefs say the target field is a link — that is its own header example (author: actor()author:card(actor("id"))) and the reason its narrowing comment gives. The card(…) case here resolves its inner marker, so the keyed form is exactly what arrives:

card(actor("id")) — what lowering emits for a linksTo field
  OK  attributes=null relationships={"author":{"links":{"self":"@tester:localhost"}}}
bare actor("id")
  OK  attributes={"firstName":"@tester:localhost"} relationships=null
bare actor()
  OK  attributes={"firstName":"@tester:localhost"} relationships=null

So the allowance is load-bearing for the current lowered shape. It should go in the same change that stops emitting the argument, not before it — otherwise a declaration that lowers today stops staging.

An actor marker on a link field already lands as a link, not an attribute. The template resolver routes a value through relationships when either the marker says it is a link or the created type declares that field linksTo / linksToMany, and the second test is independent of which marker produced the value. With author declared linksTo:

author declared linksTo, definition readable:
  attributes:    null
  relationships: {"author":{"links":{"self":"@tester:localhost"}}}

The actor's user id is the link's target, which is the identity a link to them needs. There is no state where a readable definition puts the id in a link's attribute slot, so there is nothing to refuse.

The one path that does write it as an attribute is a type whose entry cannot be read:

author declared linksTo, definition NOT readable:
  attributes:    {"author":"@tester:localhost"}
  relationships: null

A 400 is not available there either — the field's declared kind is precisely what is missing. This is also not actor-specific: with no definition, every marker except card(…) and a linkTo param falls back this way, which is the behavior linkFieldsOf documents, since those two say "this is a link" on their own authority and the rest need the type to say it.

What I am adding rather than changing: nothing currently covers the card(…) marker, so the most common authoring case — actor() on a link field — has no test pinning it. I will add one asserting card(actor("id")) stages a relationship whose target is the actor id. That way the change that drops the argument gets a failing test naming this contract instead of a silent change in where the value lands.

BxlMutationContext does not apply here: no program is assembled in this change, so there is no construction site to keep in one place. When one appears, passing the string straight through is the shape this resolver already hands it.


Generated by Claude Code

Copy link
Copy Markdown
Contributor Author

[Claude Code 🤖]

Correcting my previous comment, and withdrawing the test I said I would add.

I was wrong that an actor marker landing in relationships is the right outcome. I wrote that the user id is "the identity a link to them needs." It is not: no card represents a user, so a relationship whose target is a Matrix user id can never resolve. The mechanism I traced holds — the created type's linksTo declaration routes the value through setLink — but the outcome it produces is a dangling edge, not a correct one. So the second follow-on was right about the defect and my refutation of it was wrong.

Where the refusal belongs is still not the executor, and for the reason the probe showed rather than the one I gave. A link to the acting person is supposed to arrive through params as linkTo(Person), and an actor marker in a link position is an authoring mistake — which is knowable when the module is indexed and its fieldDefs are in hand, and not knowable here when the type's entry cannot be read, since the field's declared kind is exactly what is missing in that case. That makes it a lowering issue recorded against the declaration, not a staging-time 400.

The marker.key === 'id' allowance goes with the narrowing, not before it. That is unchanged from what I said: lowering emits actor("id") today, wrapped as card(actor("id")) for a link field, and the probe confirmed the executor resolves it. Once the builtin, the typed reference and lowering are narrowed together — the keyed form stops being emitted and an actor in a link position is refused at lowering — the allowance here is dead code and can be deleted alongside them, in the change that makes it dead.

Not adding the card(…) marker test. It would have asserted that card(actor("id")) stages a relationship pointing at the actor id, which is the behavior being removed. Writing it here would hand that work a test to delete and would read as this change endorsing an actor in a link position. The card(…) marker is still uncovered, and the right place to cover it is the change that settles what it accepts.

No code change here as a result. The actor reaches this coordinator as a bare actor: string and is substituted as one, which is the shape the other layers are being brought to.


Generated by Claude Code

@habdelra

Copy link
Copy Markdown
Contributor Author

note that the comments above are a bunch of churn resulting from teh fact that I filed CS-12933 to narrow actor() to just return the user id instead of allow the caller to return arbitrary props from the actor payload object.

Three gaps a convergence review turned up, each a divergence from the handler
the batch mirrors:

- An oversized payload was reported as `500 internal-error`. `assertWriteSize`
  throws the realm's own `CardError` carrying 413, and `atEntry` preserved a
  status only for an `OperationFailure`, so the one thing that tells a caller
  to send less was replaced by the one thing that tells it to send again.
  The status is carried across under a new `payload-too-large` code; `POST`,
  `PATCH` and `/_atomic` all answer 413 for the same bytes.
- A removal took any stored `.json` at its href, `realm.json` among them, so
  a batch could take the realm's own configuration with it. `DELETE` answers
  404 there because the index holds no card. The staged bytes answer the same
  question without a read, which keeps a card written moments ago deletable
  where consulting the index would not.
- A write into `_screenshot/` was staged. The subtree is claimed by capture
  serving, so a realm file stored there could never be read back; direct
  writes and `/_atomic` each refuse it, and a batch is the third way in.
  Removals stay admitted, as they are in both other paths, since they are the
  recovery route for anything already there.

Two tests were green for the wrong reason and are now falsifiable:

- The oversized-payload test stubbed `assertWriteSize` with a bare `Error`,
  which cannot carry a status, so the flattening above was invisible to it.
  The stub now throws what the realm throws.
- The `realmURL` assertion could not fail: the realm stamps its own URL over
  whatever the patch carried, so comparing against the client's value held
  whether or not the strip ran. It now pins the stamp, which is the property
  the stored file actually depends on.

The unchanged-file hash is recorded for the file's metadata resource, not for
`baseVersion` — that is computed from the bytes read inside the write lock and
never consults the row. The comment claiming otherwise is corrected, and its
test now asserts the row it was only pretending to exercise.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EBB6JdRo5XCY5WhUkgBckS
@habdelra

Copy link
Copy Markdown
Contributor Author

[Claude Code 🤖] The one-change-per-file refusal in claim() is right for parallel siblings, but a plain serial batch is allowed to touch one card twice, and the two changes are meant to compose: the later entry stages from the earlier entry's staged bytes, and the commit still writes the file once.

Concretely: where readStoredFiles would hand an executor the pre-batch bytes for a path that an earlier entry in this batch already staged, hand it the staged content (and its hash) instead, and let the later entry's staged write replace the earlier one in the commit map. claim() then only has to refuse a second write for a path when the two entries are parallel siblings, which nothing in this PR can express yet, so here it should compose rather than refuse. Two things do not change: a read entry still sees pre-batch state, and a card created in the batch is still not a target for later entries. Every entry that touched the shared path reports the committed version.

A batch is a sequence, so two entries naming one card are meant to build on
each other. They were refused instead, on the reasoning that both would be
computed against the state the batch started from and the second would
silently discard the first. That is a consequence of staging every entry
against one fixed snapshot, not a fact about batches: feed the later entry
the bytes the earlier one staged and they compose, with nothing discarded and
the file written once holding the last of them.

An entry's staged writes are folded into the state the next entry stages
against, and the commit collapses the sequence per file. The refusal is gone;
it belongs to parallel siblings, which nothing here can express yet.

The corners this opens, each falling out of the sequence rather than needing
a rule of its own:

- A removal takes its path back out of the staged state, so a change aimed at
  the same card afterwards finds nothing there and refuses the way it would
  outside a batch.
- A removal after a change drops that change's write entirely rather than
  writing bytes the same commit then unlinks, and both entries report no
  state, which is what a removal reports.
- A create's bytes are not folded in, so a card the batch mints is still
  reached by the local id other entries link to rather than by a URL they
  target. Tying the batch's meaning to path math a caller has to reproduce is
  the coupling the local id exists to avoid.

`baseMatched` now compares against the version the entry actually merged
over, captured as it stages. Reading it back from the stored map at commit
time was correct only while that map stood still; a second entry on one card
would otherwise have been told its base matched when it merged over bytes
that base never named.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EBB6JdRo5XCY5WhUkgBckS

Copy link
Copy Markdown
Contributor Author

[Claude Code 🤖]

Done in 1ccbca15, and the diagnosis was right: the refusal rested on a premise the code created rather than on anything true about batches. Its comment justified itself with "two entries touching one file would each have been computed against the state the batch started from, so the second would silently discard the first" — which only holds because every entry staged against one fixed snapshot. Composing removes the premise, so the refusal goes with it.

Each entry's staged writes are folded into the state the next entry stages against, and the commit collapses the sequence per file.

The corners, each falling out of the sequence rather than getting a rule of its own:

  • A change after a removal finds nothing — the removal takes its path out of the staged state, so stageUpdate refuses with the 404 it would give outside a batch.
  • A removal after a change drops that change's write rather than writing bytes the same commit then unlinks, and both entries report null, which is what a removal reports.
  • A minted card is still not a later entry's target. A create's bytes are not folded in, so it stays reachable by the lid other entries link to. Tying a batch's meaning to path math a caller has to reproduce is the coupling the local id exists to avoid.

One thing the change surfaced that was not in the ask. baseMatched was computed at commit time by reading the stored map back — correct only while that map stood still. With composition it moves, so the second entry on a card would have been told its base matched when it merged over bytes that base never named. The version each entry merged over is now captured as it stages, which also makes the reporting say what it means: a caller quoting the pre-batch version on the second of two changes to one card gets baseMatched: false, because that is not the base its merge was computed against.

Five tests, each verified to fail without the specific mechanism it covers rather than merely passing: two entries composing with the file written once and both reporting the committed version; the base-version distinction above; the two removal corners; and a minted card refusing a later entry that targets its URL.

Locally green: 188 tests, including card-endpoints, atomic-endpoints and atomic-batch-indexing untouched.

Deliberately unchanged: a removal followed by a create at the same path is still refused, since assertDestinationsFree tests the disk rather than the staged state. That is pre-existing rather than something composition introduced, and making it order-aware is a larger change than this one.


Generated by Claude Code

… is safe

Composing two changes to one card opened three ways for a batch to lose data
quietly, and the removal guard shipped alongside it was documented as
protection it does not give.

- A write to a path an earlier entry removed cancelled that removal, and a
  removal reports the same `null` whether it happened or not — so a card
  could be resurrected while the entry that asked to remove it reported
  success. Refused: the two entries are asking for opposite things, which is
  not a sequence to compose.
- A side-loaded resource is serialized whole rather than merged, so it cannot
  compose over an earlier entry's change the way a second patch does. Landing
  it last dropped that change with nothing said while its entry still
  reported success and a matching base version. Refused, naming both entries.
- A removal took a card under an ignored path. Ignored files are never
  visited, so they never get an index row, and `DELETE` — which needs one —
  refuses them forever; reading the bytes off disk is not the same
  permission. The batch would have destroyed a file no other caller can,
  and the realm would go on ignoring the absence.

`realm.json` is not protected by the card-document check and was never going
to be: a realm's config is itself a card document, and `DELETE` removes it
too, so refusing it here would put the batch out of step with the endpoint
rather than protect anything. The comment saying otherwise is corrected, and
the test that appeared to prove it stored a bare settings object no realm
writes — it now stores the config the way a realm does and pins the parity.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EBB6JdRo5XCY5WhUkgBckS
Two files conflicted, both where main's read-your-writes drain — which tags
every incremental index job with the matrix user whose request produced it —
met this branch's refactor of the same call sites.

`realm-index-updater.ts`: main added `initiatedBy` to the inline options type
that this branch had extracted into `IncrementalIndexOptions` and generalized
from `enqueueUpdate` to `enqueueChanges`. The field moves onto the interface,
and onto the `Pick` lists `update` and `updateChanges` expose, so a caller can
still scope a drain to itself.

`realm.ts`: four hunks of the same shape, where main passed `delete: true`
alongside `initiatedBy` and this branch had already moved the removal into the
change set as `operation: 'delete'`. The change-set form stands and carries
`initiatedBy` through each.

The batch commit path now tags its job too. Nothing in the merge required
that — the coordinator is new on this side and main could not have known to
thread it — but a batch would otherwise be the one write path whose index job
no reader can wait on, silently opting out of a guarantee every other path
gives. `CommitBatchOptions.actor` is that user already.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EBB6JdRo5XCY5WhUkgBckS
@habdelra
habdelra requested a review from a team September 11, 2026 14:00
The write-route tagging test shadows the updater to capture each job's
`initiatedBy`, and it shadowed `enqueueUpdate` on the reasoning — its own
comment's — that the awaited path calls through it, so one spy covered both.
That stopped being true when the awaited path became `updateChanges`, which
enqueues a change set directly: the spy sat on a method no write reaches any
more, recorded nothing, and reported every route as untagged.

The tagging itself was never broken. `initiatedBy` travels from the request
context through `updateIndexAndCollectInvalidations` to the deferred the
drain reads, and the suite's other cases — which exercise that drain rather
than the enqueue — passed throughout.

Shadowing `enqueueChanges` restores the coverage and widens it: every
incremental job is enqueued there, `enqueueUpdate` and both awaited forms
delegate to it, so one spy now covers every route whichever form it entered
by. Verified to still fail when the tag is dropped at the shared call site,
rather than merely to pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EBB6JdRo5XCY5WhUkgBckS
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants