Skip to content

Fix unread navigation loading entire message history #740

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
witbrock merged 1 commit into from
Sep 17, 2026
Merged

Fix unread navigation loading entire message history #740

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 66 additions & 0 deletions docs/engineering/unread_message_latency_2026_09_17.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# Unread-message navigation latency evidence — 17 September 2026

Evidence record for native task
`#V#task_agent_43536ba9d1dd211b359a0cf4709f8707`.
The task remains the delivery/decision surface.

Source screenshot: `#V#computer_file_copy_ce8757decd8044169fdb4ae3fd5f74d5`,
SHA-256 `3034e3c2c05f2ef86940c8277c5b351af4a45a537960675ee89c9d02a2f7b199`.
The staged original was inspected; it shows the disabled “Loading unread
messages…” action. It provides no measured duration or server trace.

## Diagnosis and matched measurement

At baseline `9fdc8b623`, `jumpToFirstUnread` exhausts every earlier page,
reconciling/rendering the growing message list on each response. It does this
even when an unread message is already visible, because older unread gaps may
exist. This makes client requests, transport and rendering scale with complete
history, rather than the target window.

`tests/browser/messageUnreadLatency.cjs` exercises the production pane/CSS in
Chromium with 1,000 synthetic messages, 50 messages per page and a fixed 50 ms
HTTP response delay. Unread messages are at positions 700 and 980; the catalogue
count deliberately understates them. IntersectionObserver is disabled for this
latency comparison so read acknowledgements cannot change the target.

| Viewport width | Baseline | Candidate | Requests before/after | Rendered messages before/after |
| --- | --- | --- | --- | --- |
| 390 px | 2,083.5 ms | 66.3 ms | 19 / 1 | 1,000 / 50 |
| 1,280 px | 2,616.1 ms | 77.4 ms | 19 / 1 | 1,000 / 50 |

These are matched local fixture observations (final candidate rerun), not production percentiles
or measured Atlas query latency. The candidate asks the existing scoped exchange
service to locate the earliest unread, then returns a bounded chronological
window. Both reads retain actor visibility, exact participants, organisation and
deleted-message filtering. Earlier/later cursors retain history access. A window
with later pages pauses latest-page polling until the reader loads through it or
uses Latest message, avoiding disconnected ranges in the displayed transcript.
No database migration, index change, represented write or model request is used.

## Validation and limitations

- Backend exchange and route tests: scope/group/deletion/recipient filtering,
authenticated actor selection, unread gaps, equal-time ordering, bidirectional
pagination and empty results.
- Frontend exchange tests: one-request seek, stale counts, retry, pending
position preservation, conversation switching, later pagination and return to
latest; existing read acknowledgement and draft coverage.
- `tests/browser/messageUnreadNavigation.cjs`: desktop/phone navigation, draft
preservation, visible-read receipts and catalogue reconciliation, using real
IntersectionObserver in its read-state cases.
- `tests/browser/messageUnreadLatency.cjs`: timed click-to-focused-target with
the production frontend; the candidate asserts the fixture's one-second target.

Local evidence is retained under `.run/unread-latency-baseline`,
`.run/unread-latency-candidate` and `.run/unread-navigation` in the task worktree.
The baseline module copy is `.run/unread-baseline/messagePanel.js`.

There is no prepared authenticated browser profile bound to this task checkout.
The host browser README was inspected; its retained profiles belong to other
checkouts/tasks and were not reused. Private database access and live mutations
are outside this worker request. Thus the production bottleneck's exact share,
actual database query plan, and public under-one-second target remain unverified.
The observed history traversal is independently sufficient to explain delay and
its removal is a bounded improvement. A future authorised live measurement can
use a task-bound, read-only operator-prepared profile without asking Michael to
log in. No deployment is requested by this task.
2 changes: 2 additions & 0 deletions src/backend/server/routes/message_routes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1047,6 +1047,8 @@ def message_exchange():
),
limit=50,
before=payload.get("before"),
after=payload.get("after"),
first_unread=payload.get("first_unread") is True,
)
for message in result["messages"]:
message["_id"] = str(message.get("_id", ""))
Expand Down
92 changes: 72 additions & 20 deletions src/backend/services/message_catalogue_service.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -267,55 +267,107 @@ def list_exchanges(
}


def get_exchange(actor, participants, *, organisation=None, limit=50, before=None):
def get_exchange(
actor,
participants,
*,
organisation=None,
limit=50,
before=None,
after=None,
first_unread=False,
):
if (
not actor
or not isinstance(participants, list)
or actor not in participants
or len(participants) > 50
):
raise PermissionError("Conversation unavailable.")
if sum(bool(value) for value in (before, after, first_unread)) > 1:
raise ValueError("Choose one message navigation direction.")
limit = min(50, max(1, limit))
query = _query(actor, organisation)
query["$expr"] = {"$setEquals": [participant_expression(), participants]}
if before:
collection = get_concepts_collection()
if collection is None:
raise RuntimeError("Messages unavailable.")

def cursor(row):
return {
"created_at": row["created_at"].isoformat(),
"concept_id": row["concept_id"],
}

anchor = before or after
if first_unread:
# Locate the earliest visible unread without transporting/rendering the
# entire exchange. Keep the exact same actor, organisation and group ACL.
unread_query = {
"$and": [
query,
{
f"relationships.{PREDICATE_RECIPIENT}": actor,
"concept_data.read_by": {"$ne": actor},
},
]
}
unread = next(
iter(
collection.find(unread_query, {"created_at": 1, "concept_id": 1})
.sort([("created_at", 1), ("concept_id", 1)])
.limit(1)
),
None,
)
if unread is None:
return {
"messages": [],
"has_more": False,
"before": None,
"after": None,
"current_user_id": actor,
}
anchor = cursor(unread)
if anchor:
from .chat_history_service import _coerce_datetime

timestamp = _coerce_datetime(before.get("created_at"))
if not timestamp or not isinstance(before.get("concept_id"), str):
if not isinstance(anchor, dict):
raise ValueError("Invalid message cursor.")
timestamp = _coerce_datetime(anchor.get("created_at"))
if not timestamp or not isinstance(anchor.get("concept_id"), str):
raise ValueError("Invalid message cursor.")
comparison = "$lt" if before else "$gt"
query.setdefault("$and", []).append(
{
"$or": [
{"created_at": {"$lt": timestamp}},
{"created_at": {comparison: timestamp}},
{
"created_at": timestamp,
"concept_id": {"$lt": before["concept_id"]},
"concept_id": {
"$gte" if first_unread else comparison: anchor["concept_id"]
},
},
]
}
)
collection = get_concepts_collection()
if collection is None:
raise RuntimeError("Messages unavailable.")
forward = bool(after or first_unread)
direction = 1 if forward else -1
rows = list(
collection.find(query)
.sort([("created_at", -1), ("concept_id", -1)])
.sort([("created_at", direction), ("concept_id", direction)])
.limit(limit + 1)
)
more = len(rows) > limit
rows = rows[:limit]
cursor = (
{
"created_at": rows[-1]["created_at"].isoformat(),
"concept_id": rows[-1]["concept_id"],
}
if more and rows
else None
)
if not forward:
rows.reverse()
return {
"messages": list(reversed(rows)),
"messages": rows,
"has_more": more,
"before": cursor,
# A seek window may have earlier history; one empty page is harmless.
"before": cursor(rows[0]) if rows and (forward or more) else None,
"after": cursor(rows[-1]) if rows and forward and more else None,
"current_user_id": actor,
}

Expand Down
72 changes: 48 additions & 24 deletions src/frontend/web/von_interface/static/js/components/messagePanel.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ let _deliveryKeySequence = 0;
let _exchange = null;
let _exchangeGeneration = 0;
let _olderCursor = null;
let _newerCursor = null;
let _unreadJumpGeneration = null;
let _readObserver = null;
const _exchangeDrafts = new Map();
Expand Down Expand Up @@ -113,6 +114,7 @@ export async function openMessageExchange(row) {
_exchange = { ...row, browser_scope: getSessionScopedOrgId() };
resetMessagesViewportPosition();
_olderCursor = null;
_newerCursor = null;
if (!_messagesContainer) initializeMessagePanel();
if (!_messagesContainer?.querySelector('#messageViewContent')) renderMessagesTabContent();
_currentUserId = row.viewer_id;
Expand Down Expand Up @@ -162,7 +164,7 @@ export async function showMessageComposer() {
}

export async function refreshOpenMessageExchange() {
if (!_exchange || document.hidden || !document.getElementById('conversationWorkspace')?.classList.contains('show-message-exchange')) return;
if (!_exchange || _newerCursor || _unreadJumpGeneration !== null || document.hidden || !document.getElementById('conversationWorkspace')?.classList.contains('show-message-exchange')) return;
await loadConversation(_currentConversationUserId, { silent: true });
}

Expand Down Expand Up @@ -199,27 +201,21 @@ async function jumpToFirstUnread() {
const stillCurrent = () => generation === _exchangeGeneration && org === getSessionScopedOrgId();
updateMessageNavigation();
try {
while (stillCurrent()) {
// Read markers can have gaps, and the catalogue count can be stale.
// Exhaust earlier pages before choosing the chronological first unread.
if (!_olderCursor) {
const target = content?.querySelector('.is-unread[data-contribution-id]');
if (target) {
target.scrollIntoView({ block: 'start' });
focusConversationTarget(target);
} else showUnreadNavigationStatus('No unread messages remain in this conversation.');
return;
}
const cursor = _olderCursor;
if (_olderCursor || _newerCursor) {
const loaded = await loadConversation(_currentConversationUserId, {
silent: true, before: cursor, observeReads: false
silent: true, firstUnread: true, observeReads: false
});
if (!stillCurrent()) return;
if (!loaded || _olderCursor === cursor) {
if (!loaded) {
showUnreadNavigationStatus('Could not load earlier unread messages. Try jumping again.');
return;
}
}
const target = content?.querySelector('.is-unread[data-contribution-id]');
if (target) {
target.scrollIntoView({ block: 'start' });
focusConversationTarget(target);
} else showUnreadNavigationStatus('No unread messages remain in this conversation.');
} finally {
if (_unreadJumpGeneration === generation) _unreadJumpGeneration = null;
if (stillCurrent()) {
Expand Down Expand Up @@ -249,7 +245,16 @@ function updateMessageNavigation() {
unread.type = 'button';
unread.className = 'message-jump-unread';
unread.onclick = () => void jumpToFirstUnread();
const latest = createLatestMessageButton(() => {
const latest = createLatestMessageButton(async () => {
if (_isLoading) return;
const generation = _exchangeGeneration;
const org = getSessionScopedOrgId();
const loaded = !_newerCursor || await loadConversation(_currentConversationUserId, { latest: true });
if (generation !== _exchangeGeneration || org !== getSessionScopedOrgId()) return;
if (!loaded) {
showUnreadNavigationStatus('Could not load the latest messages. Try again.');
return;
}
content.scrollTop = content.scrollHeight;
focusConversationTarget(content.querySelector('.message-list')?.lastElementChild || content);
updateMessageNavigation();
Expand All @@ -264,9 +269,9 @@ function updateMessageNavigation() {
unread.textContent = jumping ? 'Loading unread messages…' : 'Jump to first unread';
unread.disabled = jumping || _isLoading;
unread.hidden = !jumping && !content.querySelector('.is-unread')
&& !(_olderCursor && _exchange?.shared_unread_count > 0);
&& !((_olderCursor || _newerCursor) && _exchange?.shared_unread_count > 0);
setNavigationVisible(navigation.querySelector('.chat-scroll-to-end-btn'),
content.scrollHeight - content.scrollTop - content.clientHeight > 60);
Boolean(_newerCursor) || content.scrollHeight - content.scrollTop - content.clientHeight > 60);
}

function observeDisplayedMessages() {
Expand Down Expand Up @@ -865,7 +870,7 @@ function bindMessageStreamMenu(element, getOtherUserId) {
/**
* Load messages for a conversation with a specific user.
*/
async function loadConversation(userId, { silent = false, before = null, observeReads = true } = {}) {
async function loadConversation(userId, { silent = false, before = null, after = null, firstUnread = false, latest = false, observeReads = true } = {}) {
if (_isLoading) return;
_isLoading = true;
const generation = _exchangeGeneration;
Expand All @@ -874,14 +879,24 @@ async function loadConversation(userId, { silent = false, before = null, observe
if (contentEl && !silent && !_currentMessages.length) contentEl.innerHTML = '<div class="loading">Loading conversation…</div>';
try {
const response = _exchange
? await postJson('/api/messages/exchange', { participant_ids: _exchange.participant_ids, organisation_concept_id: _exchange.organisation_concept_id || null, before })
? await postJson('/api/messages/exchange', { participant_ids: _exchange.participant_ids, organisation_concept_id: _exchange.organisation_concept_id || null, before, ...(after ? { after } : {}), ...(firstUnread ? { first_unread: true } : {}) })
: await getJson(`/api/messages/conversation/${encodeURIComponent(userId)}?limit=50`);
if (generation !== _exchangeGeneration || org !== getSessionScopedOrgId() || userId !== _currentConversationUserId) return;
const incoming = response.messages || [];
if (firstUnread && !incoming.length) {
// A successful scoped lookup supersedes stale catalogue/read markers.
_exchange.shared_unread_count = 0;
_currentMessages.forEach(message => {
const data = message.concept_data ||= {};
data.read_by = [...new Set([...(data.read_by || []), _currentUserId])];
});
updateUnreadMarkers();
return true;
}
const compare = (a, b) => new Date(a.created_at) - new Date(b.created_at) || String(a.concept_id).localeCompare(String(b.concept_id));
let nextMessages = incoming;
if (before) nextMessages = [...new Map([..._currentMessages, ...incoming].map(m => [m.concept_id, m])).values()].sort(compare);
else if (silent && incoming.length) {
if (before || after) nextMessages = [...new Map([..._currentMessages, ...incoming].map(m => [m.concept_id, m])).values()].sort(compare);
else if (silent && !firstUnread && !latest && incoming.length) {
// Reconcile the returned range, including deletions and edits, while
// retaining already loaded earlier pages. A reconnect gap remains pageable.
const overlap = incoming.some(m => _currentMessages.some(old => old.concept_id === m.concept_id));
Expand All @@ -890,12 +905,16 @@ async function loadConversation(userId, { silent = false, before = null, observe
}
const sameContent = JSON.stringify(nextMessages.map(messageRenderKey)) === JSON.stringify(_currentMessages.map(messageRenderKey));
_currentMessages = nextMessages;
if (silent && !before && sameContent) {
if (silent && !before && !after && !firstUnread && !latest && sameContent) {
updateUnreadMarkers();
if (observeReads) observeDisplayedMessages();
return;
}
if (before || !silent) _olderCursor = response.before || null;
if (firstUnread || latest || !silent) {
_olderCursor = response.before || null;
_newerCursor = response.after || null;
} else if (before) _olderCursor = response.before || null;
else if (after) _newerCursor = response.after || null;
_currentUserId = response.current_user_id || null;
_readObserver?.disconnect();
_readObserver = null;
Expand All @@ -916,6 +935,11 @@ async function loadConversation(userId, { silent = false, before = null, observe
earlier.onclick = () => void loadConversation(userId, { silent: true, before: _olderCursor });
contentEl.prepend(earlier);
}
if (_newerCursor && contentEl) {
const later = document.createElement('button'); later.type = 'button'; later.textContent = 'Load later messages';
later.onclick = () => void loadConversation(userId, { silent: true, after: _newerCursor });
contentEl.append(later);
}
updateUnreadMarkers();
if (contentEl) {
const restoredAnchor = [...contentEl.querySelectorAll('[data-contribution-id]')]
Expand Down
Loading
Loading