feat: receive Appwrite webhooks and forward MCP events

[ChiragAgg5k]

Stack

feat/events ← #131 ← #132 ← #133 ← #134 ← #135 (events/subscribe / events/unsubscribe)

Merges into feat/events; feat/events → main lands as one feature after appwrite/appwrite#14293. Each PR's diff shows only its own layer, and every layer passes the full checklist on its own.

Summary

Part 4 of the MCP Events work (#127), covering plan step 7 (ingress and delivery). Behind MCP_EVENTS, the hosted server now receives Appwrite webhooks at POST /appwrite/webhooks/{id}. It verifies them, opens the sealed subscription, projects the body down to the event's payload fields, and forwards one Standard-Webhooks-signed event to the subscriber. events/subscribe / events/unsubscribe are still not served; they come in PR 5.

• One failure enum. delivery.Reason duplicated errors.CallbackFailure value for value. delivery.py now uses CallbackFailure, and CallbackError.reason is typed with it, so subscribe can pass it straight to EventsError.callback_endpoint. feat: sign and deliver MCP events safely #133's tests are updated to match.
• events/projection.py. One projector per catalog event. Each reads only the fields in that event's payload_schema, so raw bodies never go out. It checks resource IDs against the sealed arguments, matches X-Appwrite-Webhook-Events against the subscription's patterns, applies the status filter, and normalizes timestamps.
• events/ingress.py. The Starlette endpoint and the Ingress that owns the delivery task group, plus Ingress.from_env().
• http_app.py. The route is mounted only when events_protocol.enabled("http"). ingress.run() joins session_manager.run() in the lifespan through an AsyncExitStack. build_app(ingress=None) takes an injected ingress for tests. With the flag on and no MCP_EVENTS_SEALING_KEYS, startup fails with KeyringError.
• envelope.appwrite_hmac(url, key). The HMAC state behind appwrite_signature, so the ingress can hash the body while it streams in. appwrite_signature now uses it, and its PHP test vector still passes.
• Telemetry. Two counters, mcp.events.ingress (outcome, reason, event) and mcp.events.deliveries (event, outcome, reason). This also removes the # --- … --- section-header comments from telemetry.py.

Flow

1. Credentials. The Basic auth password is the envelope, a JWE from feat: seal MCP event subscriptions into an envelope #132's mcp-subscription+jwe profile. envelope.key_id reads the kid from its protected header. If there is no Basic auth, the password is not JWE Compact with a valid kid, or the id is not a subscription id, the response is 401 credentials.
2. Signature. An envelope naming a key that is not in the ring is 200 retired_key. Otherwise the ingress derives keyring.signing_key(id, k) for every key k in the ring. It rebuilds the registered URL as {MCP_PUBLIC_URL}/appwrite/webhooks/{id} and ignores the inbound Host. It streams the body into HMAC-SHA1(url + body) under each key and keeps up to 2 MiB, then compares each result in constant time with X-Appwrite-Webhook-Signature. No match is 401 signature. The read is bounded because the route is public:

• A request without a signature is refused before its body is read.
• A body over 16 MiB (declared or streamed), or still arriving after 10 s, is not read on and gets 200 dropped unread.
• A process reads at most 32 bodies at once (READS). A request waits for a slot inside its 10 s read deadline; every read ends by its own deadline, so a stalled forger can delay real deliveries but not starve them.

3. Envelope. keyring.open(envelope, Context(server=MCP_PUBLIC_URL, tenant=X-Appwrite-Webhook-Project-Id, resource=id)). The expected context comes only from configuration, the routed id and the project header, never from the envelope. A package EnvelopeError (tampered, not the profile, bound to another server, project or webhook) or a record that does not hash to the id is 401 envelope. The package's ExpiredSubscription is 200 dropped expired.
4. Drop checks. Catalog lookup, body size, JSON object, event-header match, then projection (shape, resource IDs, status filter).
5. Accept. eventId = "evt_" + sha256(canonical_json([X-Appwrite-Webhook-Delivery-Id, subscription id]))[:32], which stays the same across Appwrite retries. timestamp is the event time from the body, or the receive time if the body has none. The ingress takes a backlog slot (at most 1024 deliveries in flight per process and 64 per subscription, retries included). It then calls task_group.start_soon(dispatcher.deliver, …) and returns 200 {"status":"accepted","eventId":…} straight away. With no slot free, it answers 200 dropped backlog_full and nothing is delivered. A delivery that raises is caught, sent to Sentry and counted, so it never cancels the group.

Appwrite-specific mappings (from spike/events-stateless and the Appwrite source):

Event	Rule
Executions	Function from resourceId; resourceType must be functions. Sync runs fire .create already terminal and are delivered. Async .create with waiting is dropped. Async .update with failed is delivered, even with no $createdAt (created_at: null).
Deployments	Only ready / failed. Activation (PATCH /functions/:id/deployment) fires deployments.*.update with a function model, which has no resourceType; it is detected by shape and dropped. A duplicate fires with a deployment that is still waiting and is dropped.
Rows / files	$databaseId / $tableId / bucketId, when present, must equal the sealed arguments
Users	Only user_id and created_at; no email, phone, name, labels or prefs
Timestamps	DB format without a zone (2026-10-09 09:38:26.634, UTC) or response-model format, normalized to 2026-10-09T09:38:26.634Z; missing or invalid gives null

Drop / ACK rules

Appwrite treats any >= 400 as a failure. After 10 failures in a row (_APP_WEBHOOK_MAX_FAILED_ATTEMPTS) it pauses the webhook and emails the org owners (src/Appwrite/Platform/Workers/Webhooks.php). So everything that comes from the subscriber's own webhook gets a 200.

Case	Response	Delivered
Valid, matching event	200 accepted	yes
Subscription expired	200 dropped expired	no
Envelope names a key no longer in the ring	200 dropped retired_key	no
Sealed event name not in the catalog	200 dropped unknown_event	no
X-Appwrite-Webhook-Events has none of the subscription's patterns	200 dropped event_mismatch	no
Body names another function, table or bucket	200 dropped resource_mismatch	no
Body is not the expected model (deployment activation, wrong resourceType, no $id)	200 dropped shape	no
Status filter rejects (waiting, ready vs a failed filter)	200 dropped status	no
Body is not a JSON object	200 dropped malformed	no
Body over 2 MiB, signature valid	200 dropped too_large	no
Body over 16 MiB or slower than 10 s (signature never checked)	200 dropped unread	no
Delivery backlog full (process or subscription)	200 dropped backlog_full	no
No Basic auth, or the password is not an envelope	401 credentials	no
Signature missing or wrong	401 signature	no
Envelope tampered with, or for another webhook or project	401 envelope	no

Why a legitimate webhook never gets a 401 in normal operation:

• During a rotation the old key stays in the ring for at least the max TTL, so envelopes and secrets written with it still open and verify (tested).

• Two replicas can refresh one subscription during a rolling rotation: the old one seals with k1, the new one with k2. Their writes can interleave (old PUT, new PUT, new PATCH /secret, old PATCH /secret) and leave a k2 envelope next to a k1-derived secret. The signature is therefore checked against every ring key, not only the envelope's.

  I picked this over a separate, non-rotating root for the Appwrite secret. It keeps the webhook format and the rotation procedure unchanged, needs no second secret to configure, and costs one extra HMAC per key in a ring that holds one or two keys. A non-rotating root could never be rotated itself.

• Once the old key is removed, only expired subscriptions still name it. Their webhooks are orphans waiting for cleanup, so I deliberately answer them with 200 retired_key instead of 401. This differs from the "bad envelope → 401" rule because it is the one bad-envelope case a real webhook produces in normal operation. Nothing is delivered, and the 200 tells an unauthenticated caller nothing.

• The 401 response body is only {"status":"rejected","reason":…}. Appwrite shows failure bodies in the webhook logs in the Console.

• Changing MCP_PUBLIC_URL breaks every existing subscription's signature until it refreshes. This is documented.

Oversized bodies are hashed to the end, so a valid signature still gets the 200 drop and a forged one still gets 401, and memory stays bounded.

Tests

tests/unit/test_events_ingress.py (41 tests) is replaced by end-to-end flows through the real server in tests/e2e/test_events_ingress.py. The recorded-shape Appwrite fixtures moved to tests/e2e/fixtures/appwrite/, unchanged.

Harness additions (tests/e2e/support.py):

• Appwrite plays the webhooks worker. It sends the X-Appwrite-Webhook-* headers, uses Basic auth only when user and password are both set, and signs X-Appwrite-Webhook-Signature = base64(HMAC-SHA1(url . body, secret)) over the webhook's configured URL. X-Appwrite-Webhook-Events is generated like Event::generateEvents, and the delivery id is md5(event:webhook). Requests are addressed to the server's bound socket while MCP_PUBLIC_URL is https://mcp.e2e.test, like a load balancer in front of the pod.
• Receiver plays ChatGPT's endpoint: a real HTTPS server on localhost with a self-signed certificate. It checks every request with the official standardwebhooks library (new test-only dependency in the e2e group), echoes verification challenges, records SNI, and replays scripted replies (status, delay, redirect).
• Collector is an OTLP/HTTP endpoint. Every server exports its real metrics to it, so counters are asserted on what the server emits. force_flush makes reads immediate.

The server under test is the production ingress. The keyring and public URL come from the environment, and it is injected through build_app(ingress=...) with an egress that allows loopback and trusts the receiver's certificate, a 1 s timeout, and a retry policy of 0.3 s / 0.3 s / 0.6 s. Until events/subscribe exists, each test writes the webhook exactly as PR 5 will (sealed envelope as authPassword, derived secret, the event's patterns).

E2E flows:

• Every catalog event reaches the subscriber: for all 6 events, a signed Appwrite delivery gets 200 accepted and exactly one POST at the receiver. The POST is standardwebhooks-verified, webhook-id = eventId = the ingress's eventId, X-MCP-Subscription-Id is set, the timestamp is fresh, and SNI / Host are localhost. data validates against the payloadSchema the same server publishes on events/list and has exactly its keys. No email, phone, sk_live_ key, build-log secret, card number or prompt-injection text from the fixtures is on the wire. The row event and the users payload (user_id, created_at only) are checked exactly.
• Appwrite quirks: a sync execution .create (already failed) is delivered. An async .create while waiting, and a completed execution, are dropped (status). An async .update (failed) is delivered with DB-format timestamps normalized and created_at: null. A wrong resourceType is dropped (shape). A failed deployment is delivered. Activation (function model) is dropped (shape), as is a duplicate (waiting, status) and a deployment still building (status). The status argument narrows delivery: with status=failed, a ready deployment is dropped and a failed one is delivered. Bad mimeType / size types become null, a +02:00 time converts to UTC, a body without $id is dropped (shape), and an unparseable time becomes null with the event stamped on receipt.
• Stable event ids: replaying one Appwrite delivery id yields the same eventId twice at the receiver. A new delivery, or the same delivery to another subscription, yields a new one.
• Acknowledged drops, nothing delivered: expired, event mismatch (other table, empty header), resource mismatch (row table, execution function), malformed (not json, [1, 2]), unknown event, a signed body over 2 MiB (too_large; the same body with a bad signature gets 401), and a retired key (200 retired_key). A projected event over 256 KiB is accepted but never sent. mcp.events.ingress{outcome=dropped} rises by exactly the expected amount per reason, and mcp.events.deliveries{outcome=too_large} by one.
• Forged deliveries get 401 and nothing is delivered: wrong or empty signature; signed over the inbound URL instead of the registered one; tampered envelope; envelope for another project; envelope copied from another webhook; an envelope sealed with the server's key for this webhook and project but by another deployment (another MCP_PUBLIC_URL); a JWE naming a ring key with an unsupported enc; an envelope sealed with the server's key for this webhook and project but holding another subscription's contents (all 401 envelope); no Basic auth, a Bearer header, bad base64, an empty password, the old v1.k1.… format, too many parts, a non-empty encrypted-key segment, a header that is not base64url JSON, a header without kid, or an invalid kid (all 401 credentials); a path id that is not a subscription id, including Appwrite-valid sub_zzzz… / sub_AAAA…. mcp.events.ingress{outcome=rejected} rises by 23. GET returns 405. A spoofed Host / X-Forwarded-Host is ignored and the delivery succeeds.
• Retries and give-up: receiver 500 → timeout → 302 → 200 gives four attempts with the same webhook-id and newer timestamps, and the redirect target is never requested. 410 and 413 get one attempt each. 503 is abandoned after 4. An untrusted certificate gets no request. Counters: delivered +1, rejected/http_4xx +2, abandoned/http_5xx +1, abandoned/tls_error +1.
• Burst: 8 events to one host all arrive, with at most 4 in flight. With 0.6 s replies and a 1 s timeout, the 4 queued behind the first four time out (the deadline includes the slot wait, feat: sign and deliver MCP events safely #133) and arrive on retry: 12 attempts.
• Backlog: a server with backlogs of 3 per process and 2 per subscription accepts two deliveries for one subscription and answers the third with 200 dropped backlog_full. It accepts one for another subscription, then drops the next because the process is full. Finished deliveries free their slots. backlog_full rises by 2.
• Bounded reads: on a server with a 0.5 s read timeout and one read slot, three bodies are each dropped as 200 unread with nothing delivered: a correctly signed 16 MiB+ body with a declared length, a forged 17 MiB chunked body, and a body that stalls for 1.5 s. While the stalled forgery holds the slot, a real delivery waits over 0.3 s and is then accepted and delivered; afterwards one is accepted in under 0.3 s. unread rises by 3.
• Sealing key rotation across restarts: k1 → restart with k2,k1 (old and new envelopes deliver, and so does a k2 envelope with a k1-derived signing key, the interleaved-refresh state) → restart with k2 (old gets 200 retired_key, new delivers).
• Production egress: a server built purely from the environment refuses a loopback callback (rejected/connection_refused), and the receiver gets nothing.
• Fault isolation: with a dispatcher that raises once, the counter records error and the next delivery still arrives.
• Startup: python -m mcp_server_appwrite --transport http --events 1 exits non-zero with the MCP_EVENTS_SEALING_KEYS is not set message. Short, all-zero, repeated, non-base64, id-less and duplicate-id keys stop startup with KeyringError.
• Flag off (added to feat: advertise MCP events and list the event catalog #131's EventsDisabledFlow): with the flag unset, 0 or false, POST /appwrite/webhooks/{id} returns 404.

Unit tests: none remain for the ingress or projection. test_telemetry.py's two events-counter tests are removed because the flows above assert the same counters through the real OTLP export. The CallbackFailure refactor commit updates #133's remaining delivery unit tests to the shared enum.

Review changes

326d338:

• Hansi, unbounded delivery backlog: deliveries in flight are now capped at 1024 per process and 64 per subscription. When full, Appwrite still gets 200 and the drop is counted as backlog_full. Documented under "Delivery backlog" in docs/events.md.
• Code review, interleaved refreshes across a key rotation: the signature is now verified against every ring key (see above).

2a428f7:

• Hansi re-review, unbounded body reads: a missing signature is refused before the body is read, and bodies over 16 MiB or slower than 10 s are dropped as unread.

Also in 326d338:

• Coverage moved up from the stack's earlier unit tests: status-filter cases (completed execution, building deployment, requested status delivered); truncated, non-base64, invalid key id and wrong-binding envelopes; foreign ids; the egress deadline including the slot wait.

edd3eab (rebase onto #132's JWE envelope):

• The ingress reads the key id from the JWE header (envelope.key_id) and opens with Keyring.open(envelope, Context(...)), binding this server as well as the project and webhook. Expiry now comes from the package's ExpiredSubscription (still 200 dropped expired); the expired drop no longer carries the event label, because an expired envelope is not returned.
• A subscription holds one delivery secret, so the "rotated client secrets" flow is removed, and 9339d89 hands Callback (one secret since feat: sign and deliver MCP events safely #133's 656ac20) the record's secret. The forged-deliveries flow gains an envelope sealed for another server and one with an unsupported header, and its malformed cases are JWE-shaped now.
• docs/events.md gains an Envelope section: the profile, the record, context binding, key requirements (32 random bytes each) and the measured sizes.

91b2fa9 (Hansi re-review after the rebase):

• Bound concurrent unauthenticated body reads: a process reads at most READS = 32 bodies at once; see Flow step 2. The bounded-reads flow covers it and fails without the limiter.

Verification

Run locally on Python 3.12.8, lockfile written with uv 0.11.22 (the CI version):

• uv run --group dev ruff check src tests: pass
• uv run --group dev black --check src tests: pass
• uv run --group dev pyright: 0 errors
• uv run python -m unittest discover -s tests/unit: 309 tests OK
• uv run --group e2e python -m unittest discover -s tests/e2e: 16 tests OK (about 26 s)
• docker build -t appwrite-mcp:jwe .: builds

Left for PR 5

events/subscribe and events/unsubscribe: authorization reads, verification handshake, webhook upsert with url = Ingress.url(id), tls: true, a fixed non-empty authUsername (Appwrite only sends Basic auth when both user and password are set), authPassword = keyring.seal(...), secret = keyring.signing_key(id) and events = event.patterns(arguments), plus cleanup of expired webhooks. Dashboard panels for the two new counters go in the dashboards repo.

[hansi-codes]

🟢 Tier S · Ready to merge

Adds the Appwrite webhook ingress and event projection, then forwards accepted events through the existing signed callback delivery path. The route is enabled with MCP Events, uses sealed subscription envelopes and bounded reads/backlogs, and includes end-to-end coverage plus documentation and telemetry updates.

1 of 26 changed files was too large to include in full.

Verdict	New comments	Fixed	Still open
💬 Commented	0	0	0

Note

Part of the diff was too large to review, so Hansi did not approve.

📂 Walkthrough · 16

File	Change
.env.example	Documents the sealing-key requirement when MCP Events is enabled.
AGENTS.md	Adds the events package and responsibilities to the architecture table.
docs/events.md	Documents ingress, envelope binding, projection, limits, drop behavior, and metrics.
docs/flags.md	Updates MCP Events setup and documents the webhook route.
pyproject.toml	Adds the Standard Webhooks library to the e2e test dependencies.
src/mcp_server_appwrite/events/delivery.py	Uses the shared callback failure enum in delivery results and verification errors.
src/mcp_server_appwrite/events/envelope.py	Factors Appwrite HMAC initialization into a streaming helper.
src/mcp_server_appwrite/events/ingress.py	Adds the bounded Appwrite webhook endpoint and background delivery lifecycle.
src/mcp_server_appwrite/events/projection.py	Adds catalog-specific payload projection, matching, and drop checks.
src/mcp_server_appwrite/http_app.py	Mounts the ingress when events are enabled and manages it in the app lifespan.
src/mcp_server_appwrite/telemetry.py	Adds ingress and callback delivery counters.
tests/e2e/fixtures/appwrite/	Adds recorded-shape webhook fixtures for catalog events and Appwrite edge cases.
tests/e2e/support.py	Adds Appwrite, HTTPS receiver, and OTLP collector test harnesses.
tests/e2e/test_events_ingress.py	Exercises webhook authentication, projection, delivery, limits, rotation, and failure flows end to end.
tests/e2e/test_events_protocol.py	Covers the ingress route behavior with the events flag enabled and disabled.
tests/unit/test_events_delivery.py	Updates delivery tests to assert the shared callback failure enum.

---

_{Reviewed 9339d89 · Details · Comment @hansi-codes review to re-run, or mention @hansi-codes with a question.}

[hansi-codes]

🔵 Tier A · See the inline comments. Summary

[hansi-codes]

🟡 Tier B · 1 blocking finding to address. Summary

[hansi-codes]

🟢 Tier S · Looks good to merge. Summary

The findings that requested changes are resolved. See the summary comment for what is still open.

[hansi-codes]

🟡 Tier B · 1 blocking finding to address. Summary

The findings that requested changes are resolved. See the summary comment for what is still open.

[hansi-codes]

🟢 Tier S · Looks good to merge. Summary