Repository navigation
Add bots, and name the speakers in transcripts - #81
Merged
Merged
Conversation
Transcripts are on whenever recording is on, and pairing a machine stays each host's own opt-in. Config.Transcripts, its env read and its check are gone, and so is `transcripts` on /api/me; the SPA reads `recording`. The switch's tests move to tests keyed on recording: without it the moil, machine and transcript routes answer 404, and rows and pairings survive. The accepted cost, recorded in ARCHITECTURE §17: every deployment with recording serves /moil/, including the unauthenticated POST /moil/v1/pair, with no off switch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MintToken took a host bool, which can't describe the two new participants meeting agents need. A Role decides every grant in one place. An agent publishes audio, data and a screen share, but no camera, and has no admin rights. An observer subscribes, publishes nothing and is hidden. Both are kind AGENT, so neither holds a room open; the Recorder grant would also do that, but it marks the room as recording. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
moil c25d7f7 finishes the klisi rename and teaches the transcribe bundle to name speakers from a speakers.json timeline. The bundle's hash changes on purpose, so every machine owner approves it once more. The pin's comment now says which files count: the README and COMMIT beside them don't. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /api/rooms/:slug/agent-session answers a new AgentSession, the contents of the session.json an agent reads: a token with the agent role, an agent:<random> identity, and the room's address. It's the owner's alone, not an administrator's as recording is, because an agent is a new live listener and §8.1 keeps audio with the owner. It needs no machine, so it works in anonymous mode too, and it counts against the join limit. The room's public face gains is_owner, narrower than can_manage, so the SPA can offer this to the owner alone. Real-media tests show a room closing with only an agent left, a kicked agent staying out on the token it still holds, and clients never seeing tide's hidden observer although it has the same kind as an agent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agents now depend on the chat payload and its topic, so they move from room.svelte.ts into server/internal/api and are generated like every other wire type. Nothing changes on the wire. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Anyone in a meeting needs to know a program is listening, and this is the only thing that tells them. An agent carries the word "Agent" on its tile and in the participant list, and while one is in the room the hairline has a green line and the stage's cluster an AGENT chip, as recording has its red one and REC. The line is its own element rather than a hairline state, so recording or a reconnect can't hide it: alone it takes the whole width, beside another state the right half. The room's owner gets "Copy agent session" in the meeting details, the way a program joins without moil. A clipboard that refuses leaves the session on screen, selected, as the link is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs/MEETING-AGENTS.md is the contract a program builds on: the session file, what its grants allow, hearing each speaker, speaking, sharing a screen, chat, how it leaves and how long a token matters. It says plainly that anything said in a meeting reaches an agent, so a model with tools is open to prompt injection from anyone in the call. examples/agents/hello_agent.py does all of it in one file, with its dependencies inline for uv. The frozen feature rule now names agents and why they're admitted: they run outside the server, under the room owner's control, in view of everyone. §4's grant table gains agent and observer, and §7, §9, §10, §15 and §17 follow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A session has no owner_identity, so a program can't match the owner, and the docs shouldn't tell it to. Hosts are what it can tell apart, by the metadata in their token. The example now also ignores chat it can't read, as the meeting page does: anyone in the room can send it anything. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each recording gets a hidden observer: tide itself in the room, subscribed to the microphones because LiveKit reports speakers only among subscriptions. It keeps spans per display name, since identities are per connection, and writes speakers.json beside the recording before the row says completed, on both the webhook and reconciler paths, because that is what lets a transcript job start. All of it is best effort: an observer that fails costs the transcript its names, never the recording. Observers leave their rooms before the media server stops. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The bundle tells its two inputs apart by name, so the job's params stay as they are. A recording without a timeline has one input and keeps Speaker N; storage that can't say whether there is one fails the attempt like any other storage failure rather than lose the names for good. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # docs/ARCHITECTURE.md
Agents should take commands from the owner only, and the host role can't say who that is: administrators have it too. A session now carries owner_identity, which matches the owner's connections the way isOwnerIdentity does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A recording with no timeline couldn't get a transcript on an S3 key with the documented permissions: without ListBucket, S3 answers 403 for a missing object, which read as a storage failure until the job ran out of attempts. Names aren't worth a transcript, so that now costs only the names, and the docs ask for ListBucket. The observer could freeze the end of every recording. The SDK calls OnTrackUnpublished holding the room's lock, and the callback waited for mu, which refresh holds while asking for the room's lock. Starting work now needs only a lock of its own. Re-vendor moil at 96d676c: a speaker's name must cover a quarter of their speech, so a shared video can't take the name of someone who talked over it, and a huge span no longer fails the job. This changes the bundle's hash, pinned on purpose. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The bundle is tide's product, so its source and tests now live in server/internal/transcripts/bundle/, and a change to it needs no moil commit first. scripts/vendor-moil.sh copies only the SDK, and make check and CI run the bundle's tests. The bundle hash doesn't change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
moil is alpha and /moil/ takes unauthenticated pairing requests, so an operator turns it on. Off, tide serves no moil, machine, pairing or transcript routes, and /api/me reports moil: false so the SPA hides them. The speaker observer needs it too, because only transcripts read the timeline. Agent sessions don't. TIDE_MOIL replaces TIDE_TRANSCRIPTS and needs recording. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The feature was named for one use. A session is a token for any program joining a room in view of everyone: captions, a dial-in bridge, a stream relay or an AI agent. So the route, types, role, UI and docs now say bot, and docs/MEETING-AGENTS.md becomes docs/BOTS.md. LiveKit's AGENT participant kind stays, as it's a protocol constant. A bot publishes the same sources as a guest, so a video relay works. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds speaker names to transcripts and lets a room owner connect a bot to a call: any program, such as captions, a dial-in bridge or an AI agent. Puts moil behind
TIDE_MOIL, off by default. Removes the old nameklisi. tide now owns the transcribe bundle.Speaker names. The bundle keeps the model's diarization. Active-speaker data only decides who each diarized speaker is, so two voices on one mic can map to one participant.
sequenceDiagram participant LK as LiveKit participant Obs as Hidden observer participant S3 participant Job as Transcribe bundle (owner's machine) Obs->>LK: joins when recording starts, publishes nothing LK-->>Obs: active speakers Obs->>S3: speakers.json, before the recording is completed S3-->>Job: audio + speakers.json Job->>Job: diarize, then align to names: Ada, or Ada (1) and Ada (2)A recording with no timeline still gets a transcript, with labels such as
Speaker 1.Bundle.
server/internal/transcripts/bundle/is the source of the transcribe bundle, with its tests.scripts/vendor-moil.shcopies only the moil SDK, so a bundle change needs no moil commit.make checkand CI run the bundle tests.Flag.
TIDE_MOILneeds recording. Off, tide serves no moil, machine, pairing or transcript routes, keeps no speaker timeline, and the SPA hides those controls. Bot sessions work either way.CI. Removes
.forgejo/, so GitHub Actions is the only CI.Bots. Only the room's owner can get a session, and it needs no moil. tide mints the token and shows the bot. It ships no bot.
docs/BOTS.mdis the contract, andexamples/bots/hello_bot.pyis a working client.Evidence
Speaker 1. tide had no session for a program.After:
Not verified:
c25d7f7, which has no coverage threshold.StartedAtmatches the first audio sample of the file.chat.spec,lobby.specandmoderation.specagainst the real stack. They time out on the signed-in dashboard.chat.specalso times out on the base commit. The other two have not run there.recording-off.spec.ts:31is flaky, also on the base commit.Merge Danger
Door: two-way
If this PR is reverted, deleting a recording leaves its
speakers.jsonin S3. That file holds participant names.Blast Radius: deployments
COMMITpoints at moil5ad3e78onchore/service-agnostic.make buildnow. Images ofmainno longer go to the Forgejo registry, and the nightly media fuzz is gone.release.ymlstill publishesv*tags.TIDE_MOILreplacesTIDE_TRANSCRIPTS. tide ignores the old name, so an install with transcripts must setTIDE_MOIL=truewhen it upgrades. Otherwise transcripts and pairing turn off until it does. Nothing is lost.TIDE_MOIL=false, as a public install does, so it doesn't exercise the speaker observer.make moil-e2edoes.ListBucket. Without it, a missing timeline reads as a 403. The transcript still runs, without names.docs/BOTS.mdcovers prompt injection. A bot must identify its owner byowner_identity, never by a name.Reviewed with sol-loop: 4 rounds with gpt-6.1-sol on high. Sol verdict: approved
🤖 Generated with Claude Code