Skip to content

fix: deliver events persisted after a remote join to the joining server - #292

Open
nikzen wants to merge 2 commits into
masterfrom
fix/event-concurrent-with-join-master
Open

nikzen wants to merge 2 commits into
masterfrom
fix/event-concurrent-with-join-master

Conversation

@nikzen

@nikzen nikzen commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Fixed a bug where a server that joins a room over federation sometimes never received an event that was sent while its join was being handled.

What goes wrong

An event can be created on the resident server (with prev_events chosen) before a remote send_join is persisted, but persisted after it. It then becomes a sibling of the join in the DAG, not a child of it. The federation sender sends it only to the hosts that were in the room at its prev_events, and the joining server is not one of them. The joining server never learns about the event. Nothing references it either, until some later event happens to pick it as a prev_event.

element-hq/synapse#19390 (by @FrenchGithubUser) sends a dummy event after send_join when the room has several forward extremities. But it only looks at extremities up to the join's stream ordering, so an event persisted after the join is missed.

Fix

At the end of persist_and_notify_client_events, check each persisted event. If it is a forward extremity next to another extremity sent by a remote server that was not in the room at the event's prev_events, send a dummy event (reusing _send_dummy_event_after_room_join). The dummy event references both extremities, so the joining server fetches the missing event through get_missing_events.

The check is cheap in the common case. It returns early unless the room has at least two forward extremities and the event is one of them.

How we found it

The bug showed up in a bot that joins a room over federation right after it is invited. Now and then, the first message the inviter sent while the bot was joining never arrived. The DB edges on the resident server showed the pattern: the lost message had the invite as its prev_event, but a stream ordering after the remote join. The joining server's DB had no trace of it.

Tests

  • tests/federation/test_federation_server.py::SendJoinFederationTests
    • test_event_sent_between_make_join_and_send_join_is_discoverable
    • test_event_created_before_a_remote_join_and_persisted_after_it_is_discoverable: creates the event, runs send_join, then persists the event. It is red without the fix and green with it.
  • Full trial suite on this branch: Ran 5277 tests, PASSED (skips=493, successes=4784). ruff check, ruff format and mypy are clean on the changed files.
  • End-to-end on a two-homeserver kind cluster: 40 rooms each invite the bot and send a first message staggered 0–3 s after the invite.
    • Unpatched: 2 of 40 messages lost.
    • Patched: 160 of 160 delivered over 4 runs.

🤖 Generated with Claude Code

…nown to the joining server

forward extremity at the join's stream ordering. An event whose prev_events
were chosen before the join was persisted, but which is itself persisted after
the join, slips through: the send_join check runs before it is persisted, and
the event is only sent to the servers in the room before it, so the joining
server never learns about it until another event references it.

After persisting a local event, check whether it is a forward extremity beside
one sent by a server that was not in the room before it, and if so tie them
together with the same dummy event, so the joining server backfills it.

Seen in practice when a user sends the first message into a room right after
inviting a remote bot: the bot's homeserver never got the message.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@nikzen
nikzen requested a review from a team as a code owner October 4, 2026 13:58
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit f9177d0. Configure here.

get_domain_from_id(other.sender) not in hosts
and not self.hs.is_mine_id(other.sender)
for other in others
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Join dummy hides later sibling events

Medium Severity

Issue: _is_unknown_to_a_concurrently_joined_server only treats a late-persisted event as missing when another current extremity has a remote sender outside hosts. If on_send_join_request already sent a dummy (another event persisted during the handshake), that dummy replaces the join as a tip. The late sibling then sits beside a local dummy, the check returns false, and no second dummy is sent. The joining server still never learns about the late event.

Fix: Detect a newly joined remote host from membership (current hosts vs hosts at prev_events), not from the sender of leftover extremities, so a prior dummy does not hide the race.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit f9177d0. Configure here.

@codecov

codecov Bot commented Oct 4, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.09%. Comparing base (08a8de7) to head (f9177d0).

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #292      +/-   ##
==========================================
+ Coverage   81.07%   81.09%   +0.01%     
==========================================
  Files         509      509              
  Lines       73728    73741      +13     
  Branches    11193    11197       +4     
==========================================
+ Hits        59775    59799      +24     
+ Misses      10631    10621      -10     
+ Partials     3322     3321       -1     
Files with missing lines Coverage Δ
synapse/handlers/message.py 80.32% <100.00%> (+0.58%) ⬆️

... and 3 files with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 08a8de7...f9177d0. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant