Fix Cohere long-form transcription - #763
Conversation
Greptile SummaryReplaces Cohere’s Core ML backend with the existing transcribe.cpp integration and adds bounded long-form decoding with overlapping seam merging.
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 529f9b6cc1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| var bestRightCount = 0 | ||
| var bestMatchLength = 0 | ||
| for leftCount in 2...maximum { |
There was a problem hiding this comment.
When either nonempty chunk has only one whitespace-delimited token and the exact-overlap pass does not accept it, maximum is 1 and constructing the closed range 2...maximum traps with Range requires lowerBound <= upperBound. Thus sparse long-form audio such as two separated one-word chunks can crash the app during seam merging; return zero overlap before these phrase loops when maximum < 2.
Useful? React with 👍 / 👎.
| case .cohereTranscribeSixBit: | ||
| return self.getExternalCoreMLProvider() | ||
| return self.getWhisperProvider() |
There was a problem hiding this comment.
Routing Cohere through WhisperProvider also applies that provider's availableMemoryGB < requiredMemoryGB guard, while Cohere's requirement is 8 GB and availableMemoryGB() counts only free, inactive, and purgeable pages. An 8-GB Apple Silicon Mac can never report the full 8 GB as available after macOS is running, so users on an otherwise supported configuration now download the GGUF and then always receive an insufficient-memory error; use total physical memory or a realistic free-memory threshold for this model.
Useful? React with 👍 / 👎.
| let ranges = CohereTranscribeCppLongFormProcessor.ranges( | ||
| sampleCount: samples.count, | ||
| sampleRate: 16_000 | ||
| ) |
There was a problem hiding this comment.
During timer-based Cohere streaming, ASRService passes the complete growing recording prefix to the provider, and the default transcribeStreaming now reaches this long-form loop. After 30 seconds every preview therefore re-decodes all windows from the beginning rather than the bounded latest window used by the previous Cohere provider, making work grow with recording length and causing stop to wait for an increasingly expensive in-flight preview; override the streaming path to decode only a bounded suffix or disable previews for this model.
Useful? React with 👍 / 👎.
FluidVoice PR build readyDownload FluidVoice-PR-763-18f536f2c631 The artifact contains the ad-hoc-signed app ZIP, Xcode archive, build manifest, and installation instructions. It expires 5 days after the build. Install the app
This build has its own app identity, so its permissions are separate from the release version of FluidVoice. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1c337be8a0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| return left + String(String.UnicodeScalarView(rightScalars.dropFirst(endIndex))) | ||
| } | ||
| } | ||
| return left + right |
There was a problem hiding this comment.
When two adjacent chunks contain CJK characters but share no recognized overlap, this fallback concatenates them without any separator. This corrupts Korean, which is included in containsCJK and normally uses spaces—for example, merging "한국어 테스트" and "새로운 문장" produces "한국어 테스트새로운 문장"; mixed CJK/English boundaries can be joined similarly. Preserve an appropriate boundary separator when no overlap was found instead of unconditionally using left + right.
Useful? React with 👍 / 👎.
| if similarCount >= max(2, count - 1) { | ||
| return count |
There was a problem hiding this comment.
When an overlapped window actually begins a new phrase, two unrelated short words can each pass the one-edit tolerance and cause content to be discarded. For example, left text ending in "turn on" and right text beginning "burn in slowly" is treated as a two-word overlap (turn/burn and on/in are each one edit apart), producing "turn on slowly" and losing "burn in". Require stronger evidence, such as sufficient token length or aggregate matched characters, before accepting this fuzzy overlap.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ab571b8f88
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| return self.getParakeetRealtimeProvider() | ||
| case .cohereTranscribeSixBit: | ||
| return self.getExternalCoreMLProvider() | ||
| return self.getWhisperProvider() |
There was a problem hiding this comment.
When Cohere is used to transcribe a meeting file longer than 20 minutes, routing it to WhisperProvider makes it inherit prefersNativeFileTranscription == false. The fallback in MeetingTranscriptionService.transcribeFile divides the file into disjoint 20-minute chunks and directly joins their transcripts, so the new 30-second overlap processor only handles seams within each outer chunk and cannot recover words split across a 20-minute boundary. The previous Cohere provider opted into whole-file transcription; preserve that behavior or add overlap and seam merging to the outer chunks.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a58b21b1bd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let transcript = try await session.run( | ||
| samples, | ||
| options: RunOptions(timestamps: .segment) | ||
| let isCohere = self.selectedModel == .cohereTranscribeSixBit |
There was a problem hiding this comment.
If the user changes speech models while a meeting transcription is running, this provider remains retained by MeetingTranscriptionService, but selectedModel starts returning the new global setting even though session still contains the previously loaded model. On the next outer chunk, a Cohere session can therefore take the non-Cohere branch and receive the entire roughly 20-minute buffer with Whisper timestamp options instead of bounded 30-second chunks, causing incorrect output or excessive resource use. Store the model identity alongside the installed session and use that identity for transcription behavior.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b934e32e52
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let remaining = rightTokens.dropFirst(overlap) | ||
| guard !remaining.isEmpty else { continue } | ||
| result += " " + remaining.joined(separator: " ") |
There was a problem hiding this comment.
When the earlier window ends before sentence punctuation but the overlapping window recognizes it, dropping all overlapped right tokens also drops that punctuation. For example, merging "The end" with "The end. Next" produces "The end Next", and merging with "The end." produces "The end"; this removes sentence boundaries from Cohere long-form output. Reconcile punctuation on matched boundary tokens before discarding the overlap.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 18f536f2c6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| var areSpeechModelActionsBlocked: Bool { | ||
| self.asr.isRunning | ||
| self.asr.isAnyTranscriptionRunning |
There was a problem hiding this comment.
When a Cohere meeting transcription runs longer than the outer 20-minute chunk, this gate disables model actions but does not disable speechModelLanguagePicker, whose menu still writes selectedCohereLanguage. WhisperProvider.transcribe reads that global setting again for every outer chunk, so changing the language mid-job makes later chunks use a different decoder language and can corrupt the combined transcript. Disable language controls during file transcription or capture and reuse the language selected when the job begins.
Useful? React with 👍 / 👎.
| try FileManager.default.removeItem(at: self.modelDirectory) | ||
| } | ||
| } | ||
| self.removeLegacyCohereCachesIfNeeded(for: targetModel) |
There was a problem hiding this comment.
When a Cohere GGUF download is cancelled before installation, ASRService.downloadModel observes modelsExistOnDisk() == false and invokes clearCache() as cancellation cleanup. This unconditional legacy cleanup then deletes both existing Core ML cache directories even though the replacement model never loaded, silently discarding the previous model data on a routine cancellation. Keep legacy-cache migration out of incomplete-download cleanup, or customize Cohere's cancellation cleanup behavior.
Useful? React with 👍 / 👎.
1 participant
Description
Replaces Cohere's Core ML backend with the existing transcribe.cpp integration and the validated Q4_K_M GGUF. Cohere audio is decoded in bounded 30-second windows with 2-second overlap and multilingual seam merging, preventing silent long-form truncation while leaving other speech models unchanged.
Type of Change
Related Issue or Discussion
Closes #736
Testing
swiftlint --strict --config .swiftlint.yml Sourcesswiftformat --config .swiftformat SourcesScreenshots / Video
Notes
The Q4_K_M artifact is 1.45 GiB and reports comparable LibriSpeech WER to BF16 (1.25% versus 1.26%). Existing Cohere Core ML artifacts are removed only after the GGUF loads successfully.
Cohere uses total physical RAM for its 8 GB compatibility gate. Live previews decode only the latest 30 seconds, keeping preview work bounded while final transcription still processes the complete recording.
Voice-model actions and microphone dictation are disabled during meeting transcription, and the job remains pinned to the model loaded when it started.