[Quick104]

Problem

Closes #2314
Related issue: Silo-Server/silo-plugin-sdk#40
Validation tasks: none

With more than one API server, every event_consumer.v1 plugin received each event once per server, so a plugin that sends a notification for an event sent one per server. A delivery that failed, for example while the plugin restarted, was logged and dropped. On a single server without Redis, events published on the bus never reached plugins at all. This change stores each event once, on the server that produced it, in a Postgres outbox, and any API server delivers it with retries.

Every API server's dispatcher subscribed to the silo:catalog, silo:admin and silo:playback bus channels, which Redis hands to every subscriber. It also read every events.Hub envelope, and each hub relays envelopes to every other server, so plugin-published events were duplicated the same way.

Approach

• Only the producing server stores an event: one plugin_event_deliveries row per subscribing installation. Bus events come from a cache.ObservedEventBus wrapper that sees what the process itself publishes, with or without Redis. Hub envelopes are taken only when their SourceID is the local hub's. Both paths check the subscriber index first, so unsubscribed events cost a map lookup.
• Every API server claims due rows with a MATERIALIZED CTE using FOR UPDATE SKIP LOCKED and a two-minute lease, calls HandleEvent on its own plugin process, and deletes the row on success. A failure is retried after 10 s, 30 s, 1 min, 3 min and 15 min, then dropped with a warning. A row whose server died mid-delivery is claimed again when its lease runs out, and outcome writes are guarded by the attempt count, so a server whose lease lapsed cannot finish a row another server has claimed since.
• Delivery semantics for plugin authors: at least once per subscribing installation for every stored event, with the same event_id on every attempt, and no ordering. HandleEventRequest gains event_id in silo-plugin-sdk v0.25.0 (feat(eventconsumer): give each delivered event a stable event_id silo-plugin-sdk#41), which also documents this contract; go.mod moves to that release. docs/architecture/plugin-event-delivery.md records the invariants and the cases where an event can be lost before it is stored.
• Delivering only from the producing server, without an outbox, would fix the duplicates but could not retry past a restart or survive that server dying mid-delivery.

Validation

• Focused Go tests, with -race: two servers sharing one bus and one outbox deliver each bus event and each hub envelope once. The same test setup on main delivered each event twice. Other tests cover retries that keep the event_id, giving up after the last retry, a disabled installation being retried, a delivery claimed by a dead server being delivered by another after the lease, events published during shutdown still being stored, and unsubscribed events never being queued.
• Postgres contract test, added to scripts/ci/db-contracts.txt: a claim takes at most its limit, concurrent claimers never share a row, only the latest claim can finish a row, a row for an uninstalled installation does not fail the others' insert even when the uninstall races it, and a payload with a \u0000 escape is stored.
• Live, on a disposable dev deployment with a second API server and a probe plugin that logs each delivery. On main, one setting change reached the plugin twice with no event_id, and a failed delivery was never retried. On this branch the change reached it once, and a failed delivery was retried with the same event_id and delivered.
• Gate: Go build, gofmt, go vet, make lint-changed, and the generated-contract, fixture and docs checks pass. make test-go passes except two real-ffmpeg tests in internal/mediasample. Those fail the same way on main with this host's ffmpeg. Web checks were not run because the diff does not touch web/.
• Multi-agent code review of the diff (three finders, a validation agent and a sweep). Its findings are fixed on this branch, apart from the limits the architecture doc lists.

Evidence: https://evidence.siloserver.org/r/silo-server/issue-2314/

Evidence

No user-visible change. The probe plugin's before and after logs are on the evidence page linked above.

Risks

• Migration. Adds one table with a foreign key to plugin_installations. Existing tables are not changed.
• Plugin behavior. Each event now reaches one plugin process rather than one per API server. A plugin that relied on every process seeing an event, to refresh a per-process cache for example, no longer gets that. No first-party plugin implements event_consumer.v1.
• Single server without Redis. Bus events now reach plugins; before, they were dropped.
• Rolling upgrade. Until every API server runs this build, older servers still deliver from their subscriptions, once per server and without event_id.
• Known limits, documented. An event can be lost before it is stored: when its server dies first, when the 1024-event queue is full, or when the outbox write fails. Each server delivers four events at a time across all plugins, so a hung plugin holds a slot for 10 seconds per attempt and delays other plugins' events. After a plugin is installed or enabled on one server, another server stores the events it produces by its old subscriber list until plugins_changed reaches it. That is a Redis round trip, or up to a minute if the message is lost. The announcement now goes out before the slow lifecycle hooks.
• Load. A stored event costs one insert per subscriber, then a claim and a delete. Each API server also checks the outbox every 10 seconds.

Follow-ups:

• Apple and Android need no change. Neither does jellycompat: this is the host-to-plugin contract only.
• Found while reviewing this change and filed to follow it: [bug] The integration-tagged plugin dispatch test does not compile #2335 (the integration-tagged plugin dispatch test does not compile) and [bug] PublishEventToInstallation is in the plugin SDK but the host answers Unimplemented #2336 (the host does not implement PublishEventToInstallation).

User manual: unaffected. It does not describe plugin event delivery.

Checklist

• I read and can explain the complete diff.
• This pull request addresses one concern.
• The Evidence section shows every change a user can see, or says there is none.

AI Disclosure

• Harness: Claude Code in T3 Code

• Tool(s): Claude Code CLI, Claude Code subagents for code review

• Model(s): claude-opus-5-5; claude-sonnet-5-5 for two review subagents

• Involvement: AI-assisted

• Adversarial review: A multi-agent review covered the diff's lines, callers, contract and failure modes, cleanup, and validation impact, followed by a sweep. It found:

  • a WaitGroup race in Stop;
  • events lost when the server stops;
  • a success not recorded when shutdown began mid-call;
  • a plugin launch that could outlast the lease;
  • rows dropped on a peer's stale installation cache;
  • the intake filling with unsubscribed events;
  • jsonb refusing \u0000;
  • one uninstalled installation failing every subscriber's insert;
  • the lost 10-second call timeout.

  All are fixed on this branch. A cross-provider review (Codex, gpt-6.1-sol, reviewed 0df92e9) found that an event already taken from the queue was lost when shutdown cancelled its write; fixed in e8b1229. The Codex connector's finding on peer subscriber staleness is narrowed in 4bd9eba and answered in its thread. Kody's findings on an uninstall racing the insert and on plugin error text reaching logs are fixed in 68c16a0.

🤖 Generated with Claude Code

[silo-kody]

Silo Kody — review complete

Review finished. Check the inline comments for findings and verify each suggestion against the code and tests.

Reviewing changes in Silo

• Include the related issue, expected behavior, and validation steps in the PR description.
• For API changes, describe the effect on Apple and Android clients and Jellyfin compatibility.
• For plugin changes, identify the affected SDK contract, plugin, and catalog entry.
• Follow this repository's AGENTS.md and CONTRIBUTING.md.
• Request another review with @kody start-review in a PR comment.
• React with 👍 or 👎 to give feedback on individual suggestions.

Review settings

Review Options

The following review options are enabled or disabled:

Options	Enabled
Bug	✅
Performance	✅
Security	✅
Business Logic	❌

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-12T02:16:32.490498Z	68c16a0	New commits
🔒 Security Review	✅ Completed 2026-10-12T02:16:57.924116Z	68c16a0	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 0a881901-9004-413e-860a-04a9ebe2b1a6

---

📥 Commits

Reviewing files that changed from the base of the PR and between 1747a12 and 5ecb116.

---

⛔ Files ignored due to path filters (1)

• go.sum is excluded by !**/*.sum

---

📒 Files selected for processing (17)

• cmd/silo/main.go
• docs/architecture/plugin-event-delivery.md
• docs/architecture/redis-event-bus.md
• go.mod
• internal/cache/observed_bus.go
• internal/cache/observed_bus_test.go
• internal/events/hub.go
• internal/plugins/event_delivery_store.go
• internal/plugins/event_delivery_store_db_test.go
• internal/plugins/event_dispatcher.go
• internal/plugins/event_dispatcher_delivery_test.go
• internal/plugins/event_dispatcher_index_test.go
• internal/plugins/event_dispatcher_test.go
• internal/plugins/lifecycle_events.go
• internal/plugins/service.go
• migrations/sql/20261012002113_add_plugin_event_deliveries.sql
• scripts/ci/db-contracts.txt

---

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 0 remain after this review.

---

---

📝 Walkthrough

📝 Walkthrough

Walkthrough

Plugin event delivery now observes local bus publishes and stores subscriber deliveries in a PostgreSQL outbox. The dispatcher claims and delivers queued events asynchronously, retries failed deliveries, and uses persisted event IDs. The change also updates event-bus wiring, lifecycle hook ordering, database migrations, tests, and architecture documentation.

Changes

Plugin event delivery

Layer / File(s)	Summary
Event observation and intake internal/cache/observed_bus.go, internal/cache/observed_bus_test.go, internal/events/hub.go, internal/plugins/lifecycle_events.go, internal/plugins/service.go, internal/plugins/event_dispatcher_index_test.go, cmd/silo/main.go, docs/architecture/redis-event-bus.md	The observed bus invokes a registered observer after publishing. Hub.SourceID() exposes the hub source ID. Lifecycle announcements are inserted before other hooks, and index tests cover invalidation and lazy rebuilding.
Persistent delivery outbox internal/plugins/event_delivery_store.go, migrations/sql/20261012002113_add_plugin_event_deliveries.sql, internal/plugins/event_delivery_store_db_test.go, scripts/ci/db-contracts.txt	The outbox stores per-installation deliveries, claims due rows with leases, and records retry or delete outcomes using the claim attempt count. The migration adds the table and indexes; database tests cover claims, retries, installation changes, and payload storage.
Asynchronous delivery and server wiring internal/plugins/event_dispatcher.go, cmd/silo/main.go, internal/plugins/event_dispatcher_delivery_test.go, internal/plugins/event_dispatcher_test.go, go.mod, docs/architecture/plugin-event-delivery.md	The dispatcher queues eligible bus and locally sourced hub events, stores subscriber deliveries, and calls plugin clients with persisted event IDs. It retries transient failures and handles shutdown and lease recovery. The server supplies the database-backed store, and the plugin SDK dependency is upgraded to v0.25.0.

Priority: ⬆️ High

Estimated code review effort: 4 (Complex) | ~50 minutes

Change: Bug fix · Severity of issue fixed: Medium

Sequence Diagram(s)

sequenceDiagram
  participant EventPublisher
  participant ObservedEventBus
  participant EventDispatcher
  participant EventDeliveryStore
  participant PluginEventConsumer
  EventPublisher->>ObservedEventBus: Publish event
  ObservedEventBus->>EventDispatcher: Notify publish observer
  EventDispatcher->>EventDeliveryStore: Enqueue subscriber deliveries
  EventDispatcher->>EventDeliveryStore: Claim due deliveries
  EventDispatcher->>PluginEventConsumer: HandleEvent with persisted event ID
  EventDispatcher->>EventDeliveryStore: Delete success or schedule retry

Loading

---

Merge Risk: ⚪ Minimal · up to 5ecb1

Plugin events are now stored in a database outbox and delivered once across API servers, with retries. Earlier concerns about event loss during shutdown and a race with uninstall appear to be addressed. No outstanding merge-blocking problem is established.

Pre-merge checks | 4 | 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 56.76% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 37 functions across 12 files. (5 skipped:…	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check		The PR addresses the active coding requirements in #2314. It stores events in a PostgreSQL outbox on the producing server, filters bus and local hub events, and claims deliveries across API servers. I…
Out of Scope Changes check		The changes stay within #2314. The migration, dispatcher and bus changes implement outbox delivery. The SDK dependency enables the required event_id contract. Tests, contract registration, architect…
Title check		The title clearly and concisely identifies the main change: exactly-once plugin event delivery across API servers with retries.
Description check		The description is directly related to the changeset and explains the problem, implementation, validation, risks, and known limits.

---

Full details: Docstring Coverage

Explanation

Docstring coverage is 56.76% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 37 functions across 12 files. (5 skipped: 5 unsupported.)

---

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Commit to this branch
• Create a new PR

---

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

---

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0df92e949d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Refresh subscribers before dropping cross-node events

When an installation is enabled on API server A and server B produces a subscribed event before B's asynchronous plugins_changed handler invalidates its index, this check treats B's stale index as authoritative and drops the event permanently; the later outbox path also uses that same cached index, while A now intentionally ignores B's relayed event. Thus the new single-producer design can lose events during ordinary cross-node lifecycle propagation. Verify current subscribers from the store, or otherwise synchronize the index revision, before excluding a locally produced event.

AGENTS.md reference: AGENTS.md:L13-L18

Useful? React with 👍 / 👎.

[Quick104]

Real, and narrowed in 4bd9eba. Before this PR, server A delivered B's relayed copy, which covered B's stale index. Now only B stores its events, so B's view decides. The window was longer than a Redis round trip: A published plugins_changed only after the resident reconcile and the provider reloads, which can take seconds. PublishLifecycleChanges now puts the announcement first, right after the dispatcher drops its own index, and B's dispatcher drops its index as soon as the message arrives. A test pins that order.

What remains is the Redis round trip, or up to the one-minute index age if the message is lost. docs/architecture/plugin-event-delivery.md lists it under Limits. I didn't add a store read before excluding each event. That would put 1+N store reads back on every event on every server, the cost #1372 removed, to cover a window that only opens when a plugin is installed or enabled. Leaving this open in case a maintainer wants the stronger guarantee.

[Quick104]

Adversarial review (gpt-6.1-sol, reviewed 0df92e9)

One verified finding: an orderly shutdown can discard an event whose outbox write is in flight. Focused dispatcher and observer tests pass with -race; the finding reproduces with a cancellation-aware outbox test double. No local Postgres was available. I also inspected follow-up 4bd9eba; it does not change this storage path.

AI disclosure: Codex in T3 Code, gpt-6.1-sol.

[silo-kody]

Silo Kody — review complete

Review finished. Check the inline comments for findings and verify each suggestion against the code and tests.

Reviewing changes in Silo

• Include the related issue, expected behavior, and validation steps in the PR description.
• For API changes, describe the effect on Apple and Android clients and Jellyfin compatibility.
• For plugin changes, identify the affected SDK contract, plugin, and catalog entry.
• Follow this repository's AGENTS.md and CONTRIBUTING.md.
• Request another review with @kody start-review in a PR comment.
• React with 👍 or 👎 to give feedback on individual suggestions.

Review settings

Review Options

The following review options are enabled or disabled:

Options	Enabled
Bug	✅
Performance	✅
Security	✅
Business Logic	❌