Skip to content

feat(events): semantic events for registered named graphs, with a graph filter - #784

Merged
scbrown merged 8 commits into
mainfrom
wu/xdvg2-graph-events-20261024
Oct 24, 2026
Merged

scbrown merged 8 commits into
mainfrom
wu/xdvg2-graph-events-20261024

Conversation

@scbrown

@scbrown scbrown commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator

Step 1 of the event-push design (#783 §4): semantic events for registered committed named graphs, plus a graph filter.

Behaviour

  • A commit to a registered committed named graph emits episode.ingested and entity.added / entity.updated, one per entity per transaction, with "added" judged within that graph. Events are tagged with the graph IRI in a new nullable events.graph column.
  • No edge.* events (the facts are on the change feed) and no type.new / predicate.new (store-wide, ROOT only).
  • Overlays and unregistered graphs emit nothing, as the module docs require (committed_graph_id excludes overlay-class graphs).
  • GET /events, GET /events/stream and the MCP quipu_events tool accept graph=<iri>. An unregistered or overlay graph returns HTTP 400. graph combines with types and group by AND, and there is no all-graphs wildcard.

Unchanged for existing consumers

  • Without graph, a read returns ROOT's events exactly as before: named-graph rows are excluded, and a ROOT event's wire form has no graph key.
  • lag counts only the reader's own graph, so a caught-up ROOT consumer's lag does not climb with board writes it never receives.
  • events_after (used by push subscriptions and tests) stays ROOT-only. events_after_in takes a graph.

Storage

events.graph TEXT holds the IRI, like subject, never a term id (it is listed in the respace map as None). It is nullable with NULL = ROOT, added by an idempotent migration with a partial index (graph, offset) WHERE graph IS NOT NULL, and is not backfilled. A read-only handle on an unmigrated store treats every row as ROOT.

Tests

  • Store tests: tagged entity events, "added" judged within the graph, overlays and unregistered graphs silent, graph isolation and AND-combination, the wire key only on named-graph events, lag scoped to the reader's graph.
  • Server test: an /events/stream filtered to one graph carries only that graph's events.
  • Local: cargo test --release --no-fail-fast passed all 81 result groups. The quipu-server feed tests (--features shacl,onnx,server) passed 11/0. Clippy default and shacl,onnx,server with -D warnings are clean, as are fmt, markdownlint, the repository-knowledge witness test (rest-api.md re-pinned) and tool_schema_fields.

Rollback

Revert. The nullable column is ignored by older code.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

scbrown added a commit that referenced this pull request Oct 24, 2026
Record what step 1 (#784) found: the column is TEXT (the IRI), overlays
are registered graphs too and must stay silent, and lag must be scoped to
the reader's graph.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
@scbrown

scbrown commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator Author

Reworked to the repaired design (#783 at 8461101). New head e2bb5b0; the review scope is that commit.

Named-graph events now go in their own graph_events log, and their cursors in graph_consumers. events and consumers are exactly what any older binary reads; ROOT offsets no longer move with board writes. Reader, lag, consumer resume, POST /events/commit {graph} (and the MCP twin) and pruning are each per log.

Rollback acceptance pair (src/store/events_rollback_tests.rs). The pre-change events_after, consumers resume and prune_events are pinned as their literal SQL and run against a store this code wrote:

  • they see only the ROOT log, at its offsets;
  • the old pruner leaves every board row;
  • re-upgraded, the board reader resumes from its own cursor;
  • control: a graph column on events leaks the board event to the same old reader.

Local checks:

  • full default suite: 3138 passed, 0 failed;
  • clippy on 3 feature sets: clean;
  • fmt and markdownlint: clean;
  • size gate: clean (the named-graph half moved to store/graph_events.rs);
  • repository-knowledge test: clean (one rest-api.md witness hash refreshed; its summary still holds).

@scbrown

scbrown commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator Author

SAME repair/integration review: CHANGES at e2bb5b0.

[P1] Detect old-server graph commit support before mutating a cursor — docs/book/src/reference/rest-api.md, graph-consumer rollback paragraph ("Do not commit a cursor when the commit response does not echo graph").

The separate graph_events/graph_consumers tables fix my original old-reader leakage finding. The new commit-response rule does not yet preserve ROOT consumer positions across rollback: the pre-change POST /events/commit handler accepts JSON with an extra graph key, calls commit_consumer in the ROOT consumers table, and only then returns a response without graph. Checking that response is too late to prevent the write. The new graph_consumers key explicitly permits the same consumer_id in ROOT and graph logs.

Private paired SQLite control used the exact pre-change Store::commit_consumer SQL (e2bb5b0 parent; HTTP call order separately inspected): ROOT cursor5, graph cursor100, pending ROOT events6..9. New graph-only commit advances graph to101 and leaves ROOT5. The same graph-bearing request against old commit semantics overwrites ROOT to102 before the response lacks graph; ROOT resume now skips6..9. Graph cursor101 remains. Command0/HOME157paths+5UIDresourcesCLEAN. This is a private SQL/handler-contract proof, not an actual old-server integration test or a production incident.

Please replace the post-write echo check with a downgrade-safe commit protocol that older servers reject before mutation (for example a distinct graph-commit route), or another explicit contract that prevents ROOT cursor corruption, including valid identical consumer IDs and downgrade between requests. Add the paired old-handler commit case; retain separate-table reader/pruner/re-upgrade controls and all history/cursors.

Fresh exact-head24SUCCESS verified. The dependent design retains this same rollback contract defect; normal strict783 classification refused missing current base0153317d; no current native approval or merge. SAME repair scope only; no second broad design audit. No runtime/hold/production mutation. Rollback for any future admitted landing is the normal scoped source revert with exact-main watch; no event/cursor deletion.

scbrown and others added 5 commits October 24, 2026 06:39
…ph filter

A commit to a registered committed named graph now emits episode.ingested
and entity.added / entity.updated (one per entity per transaction, "added"
judged within that graph), tagged with the graph IRI in a new nullable
events.graph column. No edge.* (the facts are on the change feed) and no
type.new / predicate.new (store-wide, ROOT only). Overlays and unregistered
graphs stay silent, as the module docs require.

Readers name a graph: GET /events and /events/stream accept graph=<iri>
(400 for an unregistered or overlay graph) and combine it with types and
group by AND. Without graph a read returns ROOT's events exactly as before:
named-graph rows are excluded, a ROOT event's wire form has no graph key,
and lag counts only the reader's own graph, so a caught-up ROOT consumer
does not see its lag climb with board writes it never receives.

events.graph is TEXT (the IRI, like subject), nullable with NULL = ROOT,
added by an idempotent migration with a partial index; no backfill. A
read-only handle on an unmigrated store treats every row as ROOT.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
reads.rs crossed the 500-code-line gate with it; it belongs next to
committed_graph_registered in overlays.rs anyway.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
Review of the event-push design (P1): a `graph` column on `events` is not
rollback-safe. quipu opens a store whatever its schema, and the older
`events_after` has no graph predicate, so after a revert every ROOT consumer
would receive board events.

- Named-graph events go in `graph_events` (own offsets), and their
  readers' cursors in `graph_consumers`. `events` and `consumers` are
  exactly what every older binary reads; ROOT offsets no longer move with
  board writes.
- `events_after_in`, lag, consumer resume and `POST /events/commit {graph}`
  (and its MCP twin) read and write the graph's own log. Pruning holds each
  graph's log back by that graph's consumers only.
- respace and share-completeness classify both tables.
- The named-graph half moves to `store/graph_events.rs` (size gate).

The rollback acceptance pair (`events_rollback_tests.rs`) pins the
pre-change reader, resume and pruner as their literal SQL:
- run against this store, they see only the ROOT log, at its offsets, and
  never touch a board row;
- re-upgraded, the board reader resumes from its own cursor;
- control: the rejected column design leaks to the same reader.

Docs: the graph consumer contract for older servers. They ignore `graph`, so
discard events not tagged with your graph, and do not commit when the
response does not echo `graph`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
…ade-safe

Review (P1): the separate tables were not enough. A server that predates
named-graph events ignores an unknown `graph` and, on `/events/commit`,
upserts that consumer_id's ROOT cursor BEFORE answering. A client that
checked the echoed `graph` afterward was already too late; the reviewer's
paired control moved a ROOT cursor 5 -> 102, skipping 6..9.

Named graphs now use routes an older server answers 404 (writing nothing):
- GET /events/graph?graph=<IRI> reads one graph's own log;
- POST /events/graph/commit {consumer_id, graph, offset} commits its cursor.
`/events` and `/events/commit` refuse `graph`, so no client learns the unsafe
form. `/events/stream?graph=` reads the graph's log; an older stream already
refused `graph` on events. MCP twins: quipu_graph_events and
quipu_graph_events_commit, so the manifest is 81 tools by default (83 with
owl). The new write route is classified (http_auth, write_kind).

Tests: each route reads only its own log; the ROOT routes refuse `graph` and
the graph routes require it; a graph commit under the SAME consumer_id never
moves its ROOT cursor (the reviewer's control, new side). The named-graph
event stream reads the graph log.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
…OT tools

The schema-field audit requires every field a handler reads to be declared
in its tool's schema. /events and /events/commit now read `graph` only to
refuse it, and declaring it would advertise the refused form, so the audit
takes an explicit refused_fields list. It self-checks: a refused field must
stay undeclared and must really be read. The two new tools map to
events_graph_get and events_graph_commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
@scbrown
scbrown force-pushed the wu/xdvg2-graph-events-20261024 branch from e2bb5b0 to 7763954 Compare October 24, 2026 10:49
scbrown added a commit that referenced this pull request Oct 24, 2026
Record what step 1 (#784) found: the column is TEXT (the IRI), overlays
are registered graphs too and must stay silent, and lag must be scoped to
the reader's graph.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
@scbrown

scbrown commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator Author

Repair for your rollback-commit finding (comments 6382961435 / 6382961637). Both branches are refreshed onto current main.

The finding holds: an older /events/commit ignores graph and upserts the ROOT cursor before it answers, so an echo check comes too late.

The fix: routes, not a parameter.

  • GET /events/graph reads one graph's log; POST /events/graph/commit commits its cursor.

  • /events and /events/commit refuse graph.

  • The pre-change router (verified at af45220) has only /events, /events/stream and /events/commit, with no fallback, so it answers the new routes 404 and writes nothing.

  • The pre-change /events/stream already refuses graph on event streams.

  • MCP twins quipu_graph_events and quipu_graph_events_commit. The schema audit gains an explicit, self-checking refused_fields list for graph on the ROOT tools.

  • docs(design): event push over one feed read per transaction #783 design: 7b1659f. §4 is now "Routes, not a parameter", and §7 gains your same-consumer-id case as acceptance step 4.

  • feat(events): semantic events for registered named graphs, with a graph filter #784 source: 7763954. src/server/feed_graph_events_tests.rs, new side of your control: a graph commit under the SAME consumer_id leaves its ROOT cursor at 1, and the refused ROOT commit with graph writes nothing.

  • Local: lib 2775 passed / 0 failed, server binary 231 passed / 0 failed, tool_schema_fields 5 passed. Clippy on 3 feature sets, the size gate and the knowledge witness test (rest-api hash refreshed) are clean.

@scbrown

scbrown commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator Author

Repair2 pre-review gate:7837b1659fe currentOPEN/MERGEABLE but incompleteCI;78477639547 currentOPEN/MERGEABLE REALRED. Normal exactjoblogs Clippy(default)120853169869 andExtendedfeature120853169856 both show tests/tool_schema_fields/parity.rs:65 clippy::doc_markdown missingbackticks around quipu_graph_events andquipu_graph_events_commit; both exit101 compilefailure. Raw33281B/71973B retained. Initial literal error search missed ANSI styling; direct rawtail established actualerror, no emptylog inference. AuthorWu notified. No new source re-review, approval, merge, holdchange or runtime action. Original CHANGES remains until actualnewheadcompletegreen and SAMEverification; runtime/recovery unchanged.

clippy::doc_markdown refused the bare identifiers under -D warnings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb
@scbrown

scbrown commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator Author

The red at 7763954 was one lint: clippy::doc_markdown at tests/tool_schema_fields/parity.rs:65. The Clippy (default/shacl), Extended feature surfaces and pre-commit no-default-clippy jobs all failed on it.

Fixed in 1b7f71a: the two tool names are now in backticks. It is a doc comment only, with no code change.

Local runs with a private target:

  • cargo clippy --all-targets -D warnings -A missing-docs is clean with no-default features, default features and --features shacl.
  • cargo test --test tool_schema_fields passes 5/0.

The repair content from 10:50Z is unchanged (routes, not a parameter). The review can self-start once 1b7f71a is fully green.

@scbrown

scbrown commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator Author

Head is now 6c7d799: 1b7f71a (the lint fix) plus two merges of main.

Local checks with a private target:

  • full default suite: 3151 passed, 0 failed;
  • --features shacl,onnx,server --bin quipu-server: 232 passed, 0 failed;
  • clippy is clean on no-default, default and shacl;
  • the repository-knowledge test is OK.

The repair content is unchanged since 10:50Z.

scbrown added a commit that referenced this pull request Oct 24, 2026
* docs(design): event push over one feed read per transaction

Design for the shared feed hub (one store read per transaction for all
live subscribers), a multiplexed WebSocket transport over it, and
semantic events for registered named graphs with a graph filter. Frames,
cursors and authorization are unchanged; delivery is in three
independently revertible steps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* docs(design): pin the events.graph type, committed-only scope and lag

Record what step 1 (#784) found: the column is TEXT (the IRI), overlays
are registered graphs too and must stay silent, and lag must be scoped to
the reader's graph.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* docs(design): named-graph events in their own tables, so any revert is safe

Review of this design (P1): a nullable `graph` column on `events` is not
rollback-safe. quipu opens a store whatever its schema, and the older
`events_after` has no graph predicate, so after a revert every ROOT consumer
would receive board events.

Named-graph events now go in `graph_events` and their cursors in
`graph_consumers`. Older code never reads either, so a revert to ANY older
binary keeps ROOT delivery, offsets, cursors and pruning unchanged, and the
graph rows stay retained for re-upgrade. No floor or deploy order is needed,
because none could be enforced.

Adds the rollback acceptance pair: the pre-change reader and pruner against
the post-change store, with the rejected column design as the failing control.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* docs(design): the graph consumer's contract against an older server

Older code ignores the graph parameter, so it serves ROOT events to a graph
consumer and records its commit as a ROOT cursor. The consumer discards
events not tagged with its graph and commits only on an echoed graph.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

* docs(design): named graphs on their own routes, so older code 404s rather than misreads

Review (P1): older /events/commit ignores `graph` and upserts the ROOT cursor
of that consumer id before answering, so checking the echo afterward is too
late. Named graphs read at GET /events/graph and commit at POST
/events/graph/commit; the ROOT routes refuse `graph`. Older code answers 404
and writes nothing, and already refuses `graph` on an event stream. Adds the
same-consumer-id case to the rollback acceptance pair.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkdpDkAkbTvXvNbNacVpJb

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@scbrown

scbrown commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator Author

SAME repair APPROVE at 6c7d799; native accepted review a73c9ada9364f64c72adc7292faeb5669d297c5b7b73c062adbfae0bea4d2322 (Ian, nonauthor).

The separate tables fix the original old-reader leakage, and the distinct GET /events/graph + POST /events/graph/commit routes fix commit-before-echo: the pinned pre-change router has no matching routes/fallback; the old ROOT handler is never entered. Current ROOT handlers reject graph before mutation. Valid identical consumer IDs remain isolated by log. MCP route/schema/write annotations, stream routing, auth inventory, lint-only repair and inherited main integrations were checked in the repair scope; prior review and CHANGES remain historical, superseded by this verified repair.

Independent native exact-source rollback controls3PASS, event suite24PASS; HOME157paths+5UIDresourcesCLEAN. Primary exact-head hosted Extended feature surfaces job120862618486 independently read: three new feed_graph_events_tests PASS, server233PASS/4ignored, rollback pair PASS. These hosted results are distinct from local tests; no actual old-server HTTP integration test or production recovery claim. Original private oldSQL mutation control remains the negative. Exact78321/21 and78424/24 completedSUCCESS checked before review. Normal native review IDs above are accepted; merge/exact-main/installed acceptance are separate.

Rollback: normal scoped source revert of actual merge, preserving event logs, graph consumers and unrelated main changes; exact revert CI/normal CD smoke. No data/cursor deletion. Archive contains prepared whole-tree inverse and original refusal evidence.

@scbrown
scbrown merged commit 8379cc9 into main Oct 24, 2026
24 checks passed
@scbrown
scbrown deleted the wu/xdvg2-graph-events-20261024 branch October 24, 2026 12:14
@scbrown scbrown mentioned this pull request Oct 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant