Skip to content

Add bots, and name the speakers in transcripts - #81

Merged
robo-monk merged 20 commits into
mainfrom
feat/meeting-agents
Nov 4, 2026
Merged

robo-monk merged 20 commits into
mainfrom
feat/meeting-agents

Conversation

@robo-monk

@robo-monk robo-monk commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds speaker names to transcripts and lets a room owner connect a bot to a call: any program, such as captions, a dial-in bridge or an AI agent. Puts moil behind TIDE_MOIL, off by default. Removes the old name klisi. tide now owns the transcribe bundle.

Speaker names. The bundle keeps the model's diarization. Active-speaker data only decides who each diarized speaker is, so two voices on one mic can map to one participant.

sequenceDiagram
    participant LK as LiveKit
    participant Obs as Hidden observer
    participant S3
    participant Job as Transcribe bundle (owner's machine)
    Obs->>LK: joins when recording starts, publishes nothing
    LK-->>Obs: active speakers
    Obs->>S3: speakers.json, before the recording is completed
    S3-->>Job: audio + speakers.json
    Job->>Job: diarize, then align to names: Ada, or Ada (1) and Ada (2)
Loading

A recording with no timeline still gets a transcript, with labels such as Speaker 1.

Bundle. server/internal/transcripts/bundle/ is the source of the transcribe bundle, with its tests. scripts/vendor-moil.sh copies only the moil SDK, so a bundle change needs no moil commit. make check and CI run the bundle tests.

Flag. TIDE_MOIL needs recording. Off, tide serves no moil, machine, pairing or transcript routes, keeps no speaker timeline, and the SPA hides those controls. Bot sessions work either way.

CI. Removes .forgejo/, so GitHub Actions is the only CI.

Bots. Only the room's owner can get a session, and it needs no moil. tide mints the token and shows the bot. It ships no bot.

+POST /api/rooms/{slug}/bot-session    # owner only
+  → { ws_url, token, room, identity, name, owner_identity, expires_at }

 roles (server/internal/livekit/token.go)
   guest, host
+  bot        # publishes like a guest, plus chat; no admin; shown to everyone
+  observer   # hidden, publishes nothing

docs/BOTS.md is the contract, and examples/bots/hello_bot.py is a working client.

Evidence

  • Before: transcripts labelled speakers Speaker 1. tide had no session for a program.
    After:
make check                pass, includes the bundle tests
make bundle-check         37/37
make moil-e2e             pass, against moil 5ad3e78
make media                pass (23, 3, 18), 23b8c4a, before TIDE_MOIL
mocked bot, machine, moil-off, recording-off, anonymous and meeting-chrome specs    pass

Not verified:

  • Name accuracy on a real multi-person meeting. Tests use synthetic speech and the thresholds are untuned.
  • The final bundle with real models. The last real-model run used moil c25d7f7, which has no coverage threshold.
  • That egress StartedAt matches the first audio sample of the file.
  • The observer behind a real NAT or load balancer.
  • A recording with a bot in the room.
  • A bot publishing a camera against the real SFU. The token test covers the grant.
  • A bot reconnecting after 10 minutes, and tide refusing an expired token.
  • chat.spec, lobby.spec and moderation.spec against the real stack. They time out on the signed-in dashboard. chat.spec also times out on the base commit. The other two have not run there.
  • The rest of the non-media Playwright suite. recording-off.spec.ts:31 is flaky, also on the base commit.

Merge Danger

Door: two-way

If this PR is reverted, deleting a recording leaves its speakers.json in S3. That file holds participant names.

Blast Radius: deployments

  • Moil must land first. The vendored SDK COMMIT points at moil 5ad3e78 on chore/service-agnostic.
  • Machine owners must approve the bundle again. Its hash changes.
  • Forgejo's CI jobs are gone. No job runs make build now. Images of main no longer go to the Forgejo registry, and the nightly media fuzz is gone. release.yml still publishes v* tags.
  • TIDE_MOIL replaces TIDE_TRANSCRIPTS. tide ignores the old name, so an install with transcripts must set TIDE_MOIL=true when it upgrades. Otherwise transcripts and pairing turn off until it does. Nothing is lost.
  • The media gate runs with TIDE_MOIL=false, as a public install does, so it doesn't exercise the speaker observer. make moil-e2e does.
  • S3 needs ListBucket. Without it, a missing timeline reads as a 403. The transcript still runs, without names.
  • Bots take meeting content into programs that may run tools. docs/BOTS.md covers prompt injection. A bot must identify its owner by owner_identity, never by a name.

Reviewed with sol-loop: 4 rounds with gpt-6.1-sol on high. Sol verdict: approved

🤖 Generated with Claude Code

robo-monk and others added 20 commits November 4, 2026 13:21
Transcripts are on whenever recording is on, and pairing a machine stays
each host's own opt-in. Config.Transcripts, its env read and its check are
gone, and so is `transcripts` on /api/me; the SPA reads `recording`.

The switch's tests move to tests keyed on recording: without it the moil,
machine and transcript routes answer 404, and rows and pairings survive.

The accepted cost, recorded in ARCHITECTURE §17: every deployment with
recording serves /moil/, including the unauthenticated POST /moil/v1/pair,
with no off switch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MintToken took a host bool, which can't describe the two new participants
meeting agents need. A Role decides every grant in one place.

An agent publishes audio, data and a screen share, but no camera, and has no
admin rights. An observer subscribes, publishes nothing and is hidden. Both are
kind AGENT, so neither holds a room open; the Recorder grant would also do that,
but it marks the room as recording.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
moil c25d7f7 finishes the klisi rename and teaches the transcribe bundle to
name speakers from a speakers.json timeline. The bundle's hash changes on
purpose, so every machine owner approves it once more. The pin's comment now
says which files count: the README and COMMIT beside them don't.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /api/rooms/:slug/agent-session answers a new AgentSession, the
contents of the session.json an agent reads: a token with the agent role,
an agent:<random> identity, and the room's address. It's the owner's alone,
not an administrator's as recording is, because an agent is a new live
listener and §8.1 keeps audio with the owner. It needs no machine, so it
works in anonymous mode too, and it counts against the join limit.

The room's public face gains is_owner, narrower than can_manage, so the
SPA can offer this to the owner alone.

Real-media tests show a room closing with only an agent left, a kicked
agent staying out on the token it still holds, and clients never seeing
tide's hidden observer although it has the same kind as an agent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agents now depend on the chat payload and its topic, so they move from
room.svelte.ts into server/internal/api and are generated like every other
wire type. Nothing changes on the wire.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Anyone in a meeting needs to know a program is listening, and this is the
only thing that tells them. An agent carries the word "Agent" on its tile
and in the participant list, and while one is in the room the hairline has
a green line and the stage's cluster an AGENT chip, as recording has its
red one and REC. The line is its own element rather than a hairline state,
so recording or a reconnect can't hide it: alone it takes the whole width,
beside another state the right half.

The room's owner gets "Copy agent session" in the meeting details, the way
a program joins without moil. A clipboard that refuses leaves the session
on screen, selected, as the link is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs/MEETING-AGENTS.md is the contract a program builds on: the session
file, what its grants allow, hearing each speaker, speaking, sharing a
screen, chat, how it leaves and how long a token matters. It says plainly
that anything said in a meeting reaches an agent, so a model with tools is
open to prompt injection from anyone in the call.

examples/agents/hello_agent.py does all of it in one file, with its
dependencies inline for uv.

The frozen feature rule now names agents and why they're admitted: they
run outside the server, under the room owner's control, in view of
everyone. §4's grant table gains agent and observer, and §7, §9, §10, §15
and §17 follow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A session has no owner_identity, so a program can't match the owner, and
the docs shouldn't tell it to. Hosts are what it can tell apart, by the
metadata in their token. The example now also ignores chat it can't read,
as the meeting page does: anyone in the room can send it anything.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each recording gets a hidden observer: tide itself in the room, subscribed to
the microphones because LiveKit reports speakers only among subscriptions. It
keeps spans per display name, since identities are per connection, and writes
speakers.json beside the recording before the row says completed, on both the
webhook and reconciler paths, because that is what lets a transcript job start.

All of it is best effort: an observer that fails costs the transcript its
names, never the recording. Observers leave their rooms before the media
server stops.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The bundle tells its two inputs apart by name, so the job's params stay as they
are. A recording without a timeline has one input and keeps Speaker N; storage
that can't say whether there is one fails the attempt like any other storage
failure rather than lose the names for good.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agents should take commands from the owner only, and the host role can't say
who that is: administrators have it too. A session now carries owner_identity,
which matches the owner's connections the way isOwnerIdentity does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A recording with no timeline couldn't get a transcript on an S3 key with the
documented permissions: without ListBucket, S3 answers 403 for a missing object,
which read as a storage failure until the job ran out of attempts. Names aren't
worth a transcript, so that now costs only the names, and the docs ask for
ListBucket.

The observer could freeze the end of every recording. The SDK calls
OnTrackUnpublished holding the room's lock, and the callback waited for mu, which
refresh holds while asking for the room's lock. Starting work now needs only a
lock of its own.

Re-vendor moil at 96d676c: a speaker's name must cover a quarter of their speech,
so a shared video can't take the name of someone who talked over it, and a huge
span no longer fails the job. This changes the bundle's hash, pinned on purpose.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The bundle is tide's product, so its source and tests now live in
server/internal/transcripts/bundle/, and a change to it needs no moil
commit first. scripts/vendor-moil.sh copies only the SDK, and make check
and CI run the bundle's tests. The bundle hash doesn't change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
moil is alpha and /moil/ takes unauthenticated pairing requests, so an
operator turns it on. Off, tide serves no moil, machine, pairing or
transcript routes, and /api/me reports moil: false so the SPA hides them.
The speaker observer needs it too, because only transcripts read the
timeline. Agent sessions don't.

TIDE_MOIL replaces TIDE_TRANSCRIPTS and needs recording.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The feature was named for one use. A session is a token for any program
joining a room in view of everyone: captions, a dial-in bridge, a stream
relay or an AI agent. So the route, types, role, UI and docs now say bot,
and docs/MEETING-AGENTS.md becomes docs/BOTS.md. LiveKit's AGENT
participant kind stays, as it's a protocol constant.

A bot publishes the same sources as a guest, so a video relay works.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@robo-monk robo-monk changed the title Add meeting agents, and name the speakers in transcripts Add bots, and name the speakers in transcripts Nov 4, 2026
@robo-monk
robo-monk merged commit 0ba754f into main Nov 4, 2026
5 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant