Unify conversation discovery and add scoped participant avatars

docs/design_index.md

@@ -93,6 +93,7 @@ live.
 | Private Otter archive access, conversation images or research-slide reading | [Private Otter archive and conversation images](engineering/otter_archive_and_conversation_images.md) |
 | Reusing external ontology concepts with source identity and attribution | [KnowKat ontology sources and governed adoption](engineering/knowkat_ontology_sources.md) |
 | Importing external human/agent transcripts into conversation carriers | [External conversation import](engineering/external_conversation_import.md) |
+| Unified conversation discovery, message exchanges, read state or participant avatars | [Unified conversations and participant profiles](engineering/unified_conversations_and_participant_profiles.md) |
 | Durable assertions, text/logical assertion typing, propositions, context-sensitive retrieval, hypotheses, publication or promotion, or user-, organisation-, project-, source-, theory-, or time-relative knowledge | [Assertion and propositional-sentence ontology](engineering/assertion_and_propositional_sentence_ontology.md), then [contextual knowledge evolution](engineering/contextual_knowledge_evolution.md); for canonical ontology publication or scope change, the active [ontology publication authority boundary](engineering/ontology_publication_authority.md) |
 | Evaluation, benchmarks, or research-sensitive architecture | [agent evaluation and research uptake](engineering/agent_evaluation_and_research_uptake.md) |
 | Minimal imposition, elicitation, or write policy | [minimal-imposition principle](engineering/minimal_imposition_design_principle.md) |

docs/engineering/unified_conversations_and_participant_profiles.md

@@ -0,0 +1,116 @@
+# Unified conversations and participant profiles
+
+- **Kind:** Capability and implementation boundary
+- **Lifecycle:** Active
+- **Owner:** Von conversation interface
+- **Review trigger:** Changes to participant identity, conversation transport or avatar audience semantics
+- **Native task:** `#V#task_agent_3f9f792044179b8f9c90ebeb121f7784`
+
+## User outcome and authority
+
+Conversations provides one activity-ordered list of ordinary Von conversations and
+person/agent message exchanges. Selecting a row opens its appropriate reader and
+composer. Persisting a visible contribution promotes the same row; viewing,
+renaming, retrying a duplicate append, tool activity and progress heartbeats do not.
+Pins remain a separate user preference. Incoming activity does not change selection
+or discard the draft in another conversation.
+
+This is a shared discovery and presentation layer over the existing canonical
+stores. Ordinary transcripts remain in `chat_history`; direct messages remain
+Vontology message concepts. A message exchange is identified by the exact set of
+participants and its organisation, not by an agent's name. Group exchanges are
+readable but the existing pair-only sending capability is not presented as a group
+reply. Imported actors and transcript provenance remain intact. The old Messages
+tab reference redirects to Conversations; message references retain their source
+and exact retrieval locator.
+
+The simplest adequate baseline is the existing readers and canonical APIs with a
+common metadata projection. No transcript migration, new workflow, universal
+conversation schema or model call is required for list maintenance. Generation is
+an explicit image request, where model judgement is useful.
+
+## Catalogue and read behaviour
+
+`GET /api/messages/conversation-catalogue` merges bounded server-side keyset pages
+from both stores. Its opaque cursor is bound to actor, namespace, organisation and
+mailbox facet. Each source applies the common time/key boundary before limiting;
+an unconsumed candidate is fetched on the next page. A source failure is reported
+and does not advance the common cursor past the unavailable source.
+
+Ordinary conversations use persisted contribution metadata. Legacy dates are
+materialised from the latest visible transcript entry by conditional, batched
+derived updates without changing the transcript or its modification time. Message
+exchange dates come from canonical message creation times. The foreground browser
+polls on a four-second cadence without accumulating overlapping requests; exact
+latency also depends on the service and network.
+
+The default catalogue uses the window's current context. **All organisations**
+adds the viewer's message exchanges across their verified memberships; this does
+not broaden ordinary conversation access. The reply destination explicitly shows
+the original organisation even when it differs from the current window.
+
+Unread acknowledgements require visible contributions in an active document.
+Ordinary conversations use per-viewer monotonic read cursors with a quiet initial
+baseline; precise timestamps and stable turn IDs prevent a rounded HTTP date from
+losing the read acknowledgement. Existing shared-conversation invitation receipts
+remain authoritative. Direct messages retain canonical `read_by` state. Loading a
+background source alone does not mark it read.
+
+The title/participant and Unread filters apply to loaded catalogue rows. **Load
+older conversations** extends the bounded list. When further pages exist the badge
+is an explicit lower bound (`N+`), not a claimed total over unexamined history.
+Existing loaded pins remain visible across refreshes; discovering a pin beyond the
+initial history window still requires loading older conversations. These bounded
+list limitations are not a replacement for global conversation/content search.
+
+## Reusable identity and avatar capability
+
+The same profile editor is available from Settings, a participant's concept view,
+a conversation header and contribution portraits. It resolves display names from
+the existing concept identity, offers concept details, and supports upload,
+private generation preview, explicit publication and removal. Other participants'
+profiles are readable where visible; editing requires the authenticated subject's
+own identity. Legacy identity headers cannot authorise profile changes.
+
+`participant_profile` provides agent parity through the internal MCP catalogue:
+`get`, `generate`, `set_avatar` and `remove_avatar`. The HTTP entry points are
+`/von/api/participants/profile`, `/profiles`, `/avatar` and `/avatar/generate`.
+The image generation provider is requested once with retries disabled; a private
+candidate is not automatically published. Provider image charges are described in
+the editor, and are not claimed as part of an ordinary conversation's model cost.
+
+Avatar variants use Von's existing scope modes:
+
+| Scope | Mode |
+|---|---|
+| User | `user_only_default` |
+| Organisation | `organisation_general` |
+| User and organisation | `user_org_default` |
+| Global | `global_general` |
+
+Canonical file-copy visibility enforces the audience. Matching combined, user,
+organisation and global variants take precedence in that order; a variant from a
+different organisation is not selected as the current organisation's portrait.
+Publication creates a centred 256-pixel PNG derivative with source metadata
+removed. The original upload/generated candidate stays private. Replacing or
+removing a variant retires that publication, allowing another applicable variant
+to become visible. Image reads validate the viewer and window context, use private
+no-store caching, and never expose the private source blob URL in the public
+profile projection.
+
+The generally reusable support added here is participant presentation, explicit
+scope-aware avatar publication, common contribution rendering and copy actions,
+exact exchange locators, activity metadata, and durable per-viewer read state.
+These capabilities preserve source-specific affordances, including Thinking,
+model controls, recommendations, tasks and original reply recipients where they
+actually apply.
+
+## Acceptance and delivery boundary
+
+The release claim is bounded behaviour (Tier 1), with canonical read-back and
+negative authority/scope cases for avatar publication and exchange retrieval
+(Tier 2). Required evidence includes interleaved/equal-time pagination, unread
+precision, draft and selection preservation, exact group/organisation identity,
+profile permission denial, publication read-back, and authenticated desktop and
+narrow-viewport checks. The native task holds dated delivery evidence and the
+current merge decision. Repository publication does not activate a running server.

src/backend/integrations/internal_mcp/catalogue.py

@@ -41797,6 +41797,7 @@ def _build_default_catalogue_knowledge_io_definitions() -> List[MethodDefinition
         *otter_collection_definitions(),
         *knowkat_definitions(),
         *conversation_image_definitions(),
+        *participant_profile_definitions(),
         # Owner-scoped LinkedIn export search via a local external MCP server.
         MethodDefinition(
             name="linkedin_index_status",
@@ -46669,6 +46670,7 @@ def _register_dynamic_catalogue_methods(
 
 
 from .conversation_image_tools import definitions as conversation_image_definitions
+from .participant_profile_tools import definitions as participant_profile_definitions
 from .otter_archive_tools import definitions as otter_archive_definitions
 from .otter_collection_tools import definitions as otter_collection_definitions
 

src/backend/integrations/internal_mcp/participant_profile_tools.py

@@ -0,0 +1,65 @@
+"""Agent parity for the participant profile editor."""
+
+import base64
+
+from .gateway import MethodDefinition
+from .schemas import Schema
+
+
+def participant_profile(**args):
+    from ...services import participant_profile_service as service
+    from .catalogue import make_error_response
+
+    action = args.pop("action", "get")
+    try:
+        if action == "get":
+            return {
+                "success": True,
+                "profile": service.get_profile(args.get("concept_id")),
+            }
+        if action == "generate":
+            return {
+                "success": True,
+                "image": service.generate_avatar(
+                    prompt=args.get("prompt"), concept_id=args.get("concept_id")
+                ),
+            }
+        if action in ("set_avatar", "remove_avatar"):
+            encoded = args.get("image_base64")
+            if encoded and len(encoded) > 12 * 1024 * 1024:
+                raise ValueError("Image too large.")
+            profile = service.set_avatar(
+                concept_id=args.get("concept_id"),
+                image_concept_id=args.get("image_concept_id"),
+                data=base64.b64decode(encoded, validate=True) if encoded else None,
+                remove=action == "remove_avatar",
+                scope=args.get("scope", "global_general"),
+            )
+            return {"success": True, "profile": profile}
+        raise ValueError("Unknown participant profile action.")
+    except (ValueError, PermissionError) as exc:
+        return make_error_response("participant_profile_unavailable", str(exc))
+
+
+def definitions():
+    return [
+        MethodDefinition(
+            name="participant_profile",
+            handler=participant_profile,
+            input_schema=Schema(
+                required={},
+                optional={
+                    "action": str,
+                    "concept_id": str,
+                    "prompt": str,
+                    "scope": str,
+                    "image_concept_id": str,
+                    "image_base64": str,
+                },
+                allow_unknown=False,
+            ),
+            output_schema=Schema(required={}, optional={}, allow_unknown=True),
+            category="write",
+            description="Read a participant profile (action=get), generate a private avatar preview (generate, prompt), publish an avatar (set_avatar with private image_concept_id or PNG/JPEG/WebP image_base64), or remove_avatar. Defaults to the authenticated actor; edits only their own profile. Generation uses one paid image request and does not publish automatically. Scope selects user_only_default, organisation_general, user_org_default (Von combined audience) or global_general. Publishing creates a sanitised 256px derivative with that scope; the private original stays private. Returns canonical profile readback. Use from conversations, concept views or settings.",
+        )
+    ]

src/backend/server/routes/message_routes.py

@@ -361,16 +361,21 @@ def send_message() -> ResponseReturnValue:
             ),
             409,
         )
-    if not isinstance(org_id, str) or not org_id:
-        return jsonify({"error": "No organisation context"}), 400
-
     recipient_ids = _normalise_recipient_ids(data.get("recipient_ids"))
     content = data.get("content", "").strip()
 
     if not recipient_ids:
         return jsonify({"error": "At least one recipient is required"}), 400
     if not content:
         return jsonify({"error": "Message content is required"}), 400
+    if not isinstance(org_id, str) or not org_id:
+        return jsonify(
+            error="Choose a shared organisation for this message",
+            error_code="message_organisation_required",
+            common_organisation_options=_build_common_organisation_options(
+                sender_id=sender_id, recipient_ids=recipient_ids
+            ),
+        ), 409
     if "idempotency_key" in data:
         return (
             jsonify(
@@ -512,6 +517,26 @@ def send_message() -> ResponseReturnValue:
 
         # Return sanitised response
         delivery_status = "reused" if reused else "created"
+        from ...services.message_catalogue_service import exchange_id
+        from ...services.chat_history_service import _coerce_datetime
+
+        participants = sorted(set([sender_id, *recipient_ids]))
+        published_at = _coerce_datetime(message.get("created_at"))
+        conversation = {
+            "session_id": exchange_id(participants, org_id),
+            "source_kind": "message_exchange",
+            "participant_ids": participants,
+            "other_participant_ids": [p for p in participants if p != sender_id],
+            "viewer_id": sender_id,
+            "organisation_concept_id": org_id,
+            "session_name": ", ".join(
+                p.removeprefix("#V#").replace("_", " ") for p in recipient_ids
+            ),
+            "last_message_id": message.get("concept_id"),
+            "last_message_at": published_at.isoformat() if published_at else None,
+            "last_author_id": sender_id,
+            "preview": content[:240],
+        }
         return (
             jsonify(
                 {
@@ -522,6 +547,7 @@ def send_message() -> ResponseReturnValue:
                     "idempotent_replay": reused,
                     "delivery_status": delivery_status,
                     "message_id": message.get("concept_id"),
+                    "conversation": conversation,
                     "sent_at": message.get("concept_data", {}).get("sent_at"),
                     "attribution": message.get("concept_data", {}).get(
                         "attribution",
@@ -920,3 +946,127 @@ def search() -> ResponseReturnValue:
         ),
         200,
     )
+
+
+@message_bp.route("/conversation-catalogue", methods=["GET"])
+def unified_conversation_catalogue():
+    from ...services.conversation_catalogue_service import list_catalogue
+
+    actor = _get_current_user_concept_id()
+    if not actor:
+        return jsonify(error="authentication_required"), 401
+    try:
+        effective = get_effective_context(
+            request.headers.get("X-Von-Window-Session"),
+            dict(session),
+            actor,
+            require_known_window=True,
+        )
+        from ...services.namespace_service import derive_namespace_for_actor
+
+        namespace = effective.get("namespace") or derive_namespace_for_actor(
+            actor, effective.get("organisation_id")
+        )
+        return jsonify(
+            list_catalogue(
+                actor,
+                namespace=namespace,
+                organisation=_get_current_org_concept_id(
+                    actor, require_known_window=True
+                ),
+                limit=request.args.get("limit", 100, type=int),
+                cursor=request.args.get("cursor"),
+                all_contexts=request.args.get("all_contexts") == "true",
+            )
+        )
+    except (ValueError, PermissionError) as exc:
+        return jsonify(error=str(exc)), 400
+
+
+@message_bp.route("/catalogue", methods=["GET"])
+def message_catalogue():
+    from ...services.message_catalogue_service import list_exchanges
+
+    actor = _get_current_user_concept_id()
+    if not actor:
+        return jsonify(error="authentication_required"), 401
+    try:
+        result = list_exchanges(
+            actor,
+            organisation=_get_current_org_concept_id(actor, require_known_window=True),
+            limit=max(1, min(200, request.args.get("limit", 100, type=int))),
+            skip=max(0, request.args.get("skip", 0, type=int)),
+            query_text=request.args.get("q"),
+            all_contexts=request.args.get("all_contexts") == "true",
+        )
+        return jsonify(**result, current_user_id=actor)
+    except Exception:
+        _log.exception("Message catalogue unavailable")
+        return jsonify(
+            error="messages_unavailable", conversations=[], coverage_complete=False
+        ), 503
+
+
+@message_bp.route("/exchange", methods=["POST"])
+def message_exchange():
+    from ...services.message_catalogue_service import get_exchange
+
+    actor = _get_current_user_concept_id()
+    if not actor:
+        return jsonify(error="authentication_required"), 401
+    payload = request.get_json(silent=True) or {}
+    try:
+        result = get_exchange(
+            actor,
+            payload.get("participant_ids"),
+            organisation=(
+                _normalise_concept_id(payload["organisation_concept_id"])
+                if "organisation_concept_id" in payload
+                else _get_current_org_concept_id(actor, require_known_window=True)
+            ),
+            limit=50,
+            before=payload.get("before"),
+        )
+        for message in result["messages"]:
+            message["_id"] = str(message.get("_id", ""))
+        return jsonify(result)
+    except PermissionError:
+        return jsonify(error="conversation_unavailable"), 403
+    except ValueError as exc:
+        return jsonify(error=str(exc)), 400
+    except Exception:
+        _log.exception("Message exchange unavailable")
+        return jsonify(error="messages_unavailable"), 503
+
+
+@message_bp.route("/exchange/preference", methods=["POST"])
+def message_exchange_preference():
+    from ...services.message_catalogue_service import get_exchange, exchange_id
+    from ...services.conversation_management_service import set_conversation_preference
+
+    actor = _get_current_user_concept_id()
+    if not actor:
+        return jsonify(error="authentication_required"), 401
+    payload = request.get_json(silent=True) or {}
+    participants = payload.get("participant_ids")
+    action = payload.get("action")
+    if action not in ("pin", "unpin", "hide", "unhide"):
+        return jsonify(error="unsupported_action"), 400
+    try:
+        org = (
+            _normalise_concept_id(payload["organisation_concept_id"])
+            if "organisation_concept_id" in payload
+            else _get_current_org_concept_id(actor, require_known_window=True)
+        )
+        exchange = get_exchange(actor, participants, organisation=org, limit=1)
+        if not exchange["messages"]:
+            return jsonify(error="conversation_unavailable"), 404
+        field = "pinned" if action in ("pin", "unpin") else "hidden"
+        result = set_conversation_preference(
+            actor_user_id=actor,
+            session_id=exchange_id(participants, org),
+            **{field: action in ("pin", "hide")},
+        )
+        return jsonify(success=True, preference=result)
+    except PermissionError:
+        return jsonify(error="conversation_unavailable"), 403