diff --git a/.gitignore b/.gitignore index 67fa165..4bf7840 100644 --- a/.gitignore +++ b/.gitignore @@ -52,4 +52,6 @@ logs/ # Temporary files *.tmp -*.temp \ No newline at end of file +*.temp +# Playwright saved login state for the local test user +frontend/playwright/.auth/ diff --git a/README.md b/README.md index 6f330a4..5eba7a3 100644 --- a/README.md +++ b/README.md @@ -334,6 +334,10 @@ class WhisperTranscriptionService: ### Backend (pytest-django) - 25+ tests covering auth, voice notes, serializers, API contracts - Runs against SQLite in CI, PostgreSQL in Docker +- Language handling is pinned by `tests/test_language_detection.py` (recorded provider + responses) and `tests/live/` (real audio fixtures through real Deepgram, run with + `pytest -m live`; excluded from the default run). Fixtures live in + `tests/fixtures/audio/`; see `docs/adr/004-language-detection-strategy.md`. ### Frontend (Playwright E2E) - Auth setup: register + login with storage state persistence diff --git a/backend/config/settings.py b/backend/config/settings.py index 5966791..07b748e 100644 --- a/backend/config/settings.py +++ b/backend/config/settings.py @@ -191,6 +191,7 @@ 'user-agent', 'x-csrftoken', 'x-requested-with', + 'x-request-id', # browser-originated trace id (see RequestIDMiddleware); without it cross-origin note creation fails preflight 'range', # Important for audio/video streaming ] diff --git a/backend/pytest.ini b/backend/pytest.ini index 1f466da..70fd0a6 100644 --- a/backend/pytest.ini +++ b/backend/pytest.ini @@ -3,4 +3,6 @@ DJANGO_SETTINGS_MODULE = config.settings python_files = test_*.py python_classes = Test* python_functions = test_* -addopts = --tb=short -q +markers = + live: sends real audio to the real transcription provider (needs DEEPGRAM_API_KEY); run with -m live +addopts = --tb=short -q -m "not live" diff --git a/backend/tests/fixtures/audio/en_us.m4a b/backend/tests/fixtures/audio/en_us.m4a new file mode 100644 index 0000000..d2d6b2e Binary files /dev/null and b/backend/tests/fixtures/audio/en_us.m4a differ diff --git a/backend/tests/fixtures/audio/es_es.m4a b/backend/tests/fixtures/audio/es_es.m4a new file mode 100644 index 0000000..df9e83b Binary files /dev/null and b/backend/tests/fixtures/audio/es_es.m4a differ diff --git a/backend/tests/fixtures/audio/es_mx.m4a b/backend/tests/fixtures/audio/es_mx.m4a new file mode 100644 index 0000000..1883137 Binary files /dev/null and b/backend/tests/fixtures/audio/es_mx.m4a differ diff --git a/backend/tests/fixtures/audio/ko_kr.m4a b/backend/tests/fixtures/audio/ko_kr.m4a new file mode 100644 index 0000000..bf7a941 Binary files /dev/null and b/backend/tests/fixtures/audio/ko_kr.m4a differ diff --git a/backend/tests/fixtures/audio/mixed_en_es.m4a b/backend/tests/fixtures/audio/mixed_en_es.m4a new file mode 100644 index 0000000..2bd121d Binary files /dev/null and b/backend/tests/fixtures/audio/mixed_en_es.m4a differ diff --git a/backend/tests/fixtures/audio/zh_cn.m4a b/backend/tests/fixtures/audio/zh_cn.m4a new file mode 100644 index 0000000..5eb7ead Binary files /dev/null and b/backend/tests/fixtures/audio/zh_cn.m4a differ diff --git a/backend/tests/live/__init__.py b/backend/tests/live/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/backend/tests/live/test_deepgram_live.py b/backend/tests/live/test_deepgram_live.py new file mode 100644 index 0000000..2543a7b --- /dev/null +++ b/backend/tests/live/test_deepgram_live.py @@ -0,0 +1,59 @@ +"""Live provider check: real audio through the real Deepgram service. + +Excluded from the default run (pytest.ini adds ``-m "not live"``). Run before a +deploy with:: + + DEEPGRAM_API_KEY=... pytest -m live + +Each fixture in tests/fixtures/audio/ is a short synthetic clip (macOS speech +voices) with a known language and a keyword the transcript must contain. +This is the MODULE-tier evidence that the auto path keeps the spoken language; +the unit tests only prove how a recorded response is handled. +""" +from pathlib import Path + +import pytest +from django.conf import settings +from django.core.files.uploadedfile import SimpleUploadedFile + +from apps.core.services import DeepgramTranscriptionService + +FIXTURES = Path(__file__).resolve().parent.parent / 'fixtures' / 'audio' + +# file, expected language, keyword that must appear in the transcript +CASES = [ + ('es_mx.m4a', 'es', 'hermana'), + ('es_es.m4a', 'es', 'informe'), + ('en_us.m4a', 'en', 'dentist'), + ('ko_kr.m4a', 'ko', '병원'), + ('zh_cn.m4a', 'zh', '医生'), +] + +pytestmark = pytest.mark.live + + +@pytest.fixture(scope='module') +def service(): + if not settings.DEEPGRAM_API_KEY: + pytest.skip('DEEPGRAM_API_KEY not set; live provider check skipped') + return DeepgramTranscriptionService() + + +@pytest.mark.parametrize('file_name, expected_language, keyword', CASES) +def test_auto_keeps_spoken_language(service, file_name, expected_language, keyword): + audio = SimpleUploadedFile(file_name, (FIXTURES / file_name).read_bytes(), content_type='audio/mp4') + result = service.transcribe_audio(audio, language='auto') + assert result['success'] is True, result.get('error') + assert result['language'] == expected_language + assert keyword in result['text'].lower() + + +def test_mixed_english_spanish_keeps_both_languages(service): + audio = SimpleUploadedFile( + 'mixed_en_es.m4a', (FIXTURES / 'mixed_en_es.m4a').read_bytes(), content_type='audio/mp4' + ) + result = service.transcribe_audio(audio, language='auto') + assert result['success'] is True, result.get('error') + text = result['text'].lower() + assert 'sister' in text # English half survives + assert 'hermana' in text # Spanish half survives, not translated diff --git a/backend/tests/test_cors_preflight.py b/backend/tests/test_cors_preflight.py new file mode 100644 index 0000000..b9088ca --- /dev/null +++ b/backend/tests/test_cors_preflight.py @@ -0,0 +1,23 @@ +"""CORS preflight contract for the headers the frontend actually sends. + +The browser sends X-Request-ID on note creation so the trace starts client-side +(frontend/src/services/api.ts). If the header is missing from +CORS_ALLOW_HEADERS the preflight succeeds but the browser drops the real POST, +so uploads silently fail on any cross-origin deployment (the dev stack, CI e2e). +""" +import pytest + + +@pytest.mark.django_db +def test_preflight_allows_x_request_id(api_client): + resp = api_client.options( + '/api/notes/', + HTTP_ORIGIN='http://localhost:3011', + HTTP_ACCESS_CONTROL_REQUEST_METHOD='POST', + HTTP_ACCESS_CONTROL_REQUEST_HEADERS='content-type,x-request-id', + ) + assert resp.status_code == 200 + allowed = {h.strip().lower() for h in resp['Access-Control-Allow-Headers'].split(',')} + assert 'x-request-id' in allowed + assert 'content-type' in allowed + assert resp['Access-Control-Allow-Origin'] == 'http://localhost:3011' diff --git a/backend/tests/test_language_detection.py b/backend/tests/test_language_detection.py new file mode 100644 index 0000000..f7ec918 --- /dev/null +++ b/backend/tests/test_language_detection.py @@ -0,0 +1,108 @@ +"""Spoken-language handling for the Deepgram transcription path. + +Regression guard for the report "Spanish audio comes back as English". +These tests pin the contract that the auto path asks Deepgram to detect the +language and that whatever it detects is what the note stores, for every +language the app offers. Provider responses are recorded shapes (see +docs/adr/004-language-detection-strategy.md); the live provider check lives +in tests/live/. +""" +from typing import Optional +from unittest import mock + +import pytest + +from apps.core.services import DeepgramTranscriptionService +from apps.voice_notes.models import VoiceNote + + +def _payload(transcript: str, detected: Optional[str], confidence: float = 0.99) -> dict: + channel = { + 'alternatives': [{'transcript': transcript, 'confidence': 0.97, 'words': []}], + } + if detected is not None: + channel['detected_language'] = detected + channel['language_confidence'] = confidence + return { + 'metadata': {'duration': 7.5}, + 'results': { + 'channels': [channel], + 'utterances': [ + {'start': 0.0, 'end': 7.5, 'transcript': transcript, 'speaker': 0, 'confidence': 0.97}, + ], + }, + } + + +def _mock_response(payload: dict) -> mock.Mock: + response = mock.Mock(status_code=200) + response.json.return_value = payload + response.raise_for_status.return_value = None + return response + + +@pytest.fixture +def deepgram(settings): + settings.DEEPGRAM_API_KEY = 'dg-test-key' + return DeepgramTranscriptionService() + + +OFFERED_LANGUAGES = [code for code, _ in VoiceNote.LANGUAGE_CHOICES if code != 'auto'] + + +class TestAutoLanguagePath: + def test_auto_requests_detection_not_a_fixed_language(self, deepgram, audio_file): + with mock.patch( + 'apps.core.services.requests.post', return_value=_mock_response(_payload('Hola', 'es')) + ) as post: + deepgram.transcribe_audio(audio_file, language='auto') + params = post.call_args.kwargs['params'] + assert params['detect_language'] == 'true' + assert 'language' not in params + + @pytest.mark.parametrize('detected', OFFERED_LANGUAGES) + def test_detected_language_is_stored_verbatim(self, deepgram, audio_file, detected): + with mock.patch( + 'apps.core.services.requests.post', + return_value=_mock_response(_payload('texto', detected)), + ): + result = deepgram.transcribe_audio(audio_file, language='auto') + assert result['success'] is True + assert result['language'] == detected + + def test_spanish_text_is_kept_not_translated(self, deepgram, audio_file): + spanish = 'Hola, soy tu hermana. Mañana tengo cita con el médico.' + with mock.patch( + 'apps.core.services.requests.post', return_value=_mock_response(_payload(spanish, 'es')) + ): + result = deepgram.transcribe_audio(audio_file, language='auto') + assert result['text'] == spanish + assert result['segments'][0].text == spanish + assert result['language'] == 'es' + + def test_regional_code_is_normalised_to_base_language(self, deepgram, audio_file): + with mock.patch( + 'apps.core.services.requests.post', return_value=_mock_response(_payload('Olá', 'pt-BR')) + ): + result = deepgram.transcribe_audio(audio_file, language='auto') + assert result['language'] == 'pt' + + def test_missing_detection_falls_back_to_auto(self, deepgram, audio_file): + with mock.patch( + 'apps.core.services.requests.post', return_value=_mock_response(_payload('...', None)) + ): + result = deepgram.transcribe_audio(audio_file, language='auto') + assert result['language'] == 'auto' + + +class TestExplicitLanguagePath: + @pytest.mark.parametrize('requested', ['es', 'ko', 'zh']) + def test_explicit_language_is_sent_and_stored(self, deepgram, audio_file, requested): + with mock.patch( + 'apps.core.services.requests.post', return_value=_mock_response(_payload('texto', None)) + ) as post: + result = deepgram.transcribe_audio(audio_file, language=requested) + params = post.call_args.kwargs['params'] + assert params['language'] == requested + assert 'detect_language' not in params + assert result['language'] == requested diff --git a/docs/adr/004-language-detection-strategy.md b/docs/adr/004-language-detection-strategy.md new file mode 100644 index 0000000..8daca5b --- /dev/null +++ b/docs/adr/004-language-detection-strategy.md @@ -0,0 +1,47 @@ +# ADR-004: Spoken-language detection strategy + +## Status +Accepted (2026-09-01) + +## Context +A user reported that Spanish audio was transcribed as English. The report predates the +switch to Deepgram (PR #38, deployed 2026-06-29); the earlier path was a self-hosted +`faster-whisper-small` model, which is weak on short non-English clips. + +The auto path today sends `detect_language=true` to Deepgram nova-3. Deepgram also offers +`language=multi` (code-switching mode, per-word language tags). Before choosing, both were +run against synthetic clips (macOS speech voices, now committed under +`backend/tests/fixtures/audio/`): + +| clip | `detect_language` | `language=multi` | +|---|---|---| +| Spanish (MX, ES) | correct Spanish, detected `es` at 0.99 | correct Spanish | +| English then Spanish | both kept; minor errors ("dentista", "Than") | cleanest text; words tagged en/es | +| Korean | correct, detected `ko` at 1.00 | garbage, tagged `ja` | +| Chinese | correct, detected `zh` at 1.00 | garbage, tagged `ja` | + +`multi` returns no `detected_language`; the language would have to be derived from word +tags. Its supported language set does not include Korean or Chinese, both of which the app +offers. + +## Decision +Keep `detect_language=true` as the auto path. Do not switch to `multi`. + +Add the evidence instead of a code change: committed audio fixtures, unit tests that pin +"whatever Deepgram detects is what the note stores" for every offered language, a live +provider test (`pytest -m live`) that runs the fixtures through real Deepgram, and a +Playwright upload flow that asserts Spanish text and a Spanish badge on the note page. + +## Alternatives considered +- **`language=multi` for auto.** Rejected: breaks Korean and Chinese (table above). +- **Two-pass: detect, then re-run with `multi` when `language_confidence` is low.** The + mixed clip scored 0.56 against 0.99+ for single-language clips, so a threshold would + separate them. Rejected for now: doubles provider cost on ambiguous clips, and the + second pass still needs a guard for languages `multi` cannot handle. Revisit only if + code-switched notes show real accuracy problems. + +## Consequences +- Mixed-language recordings keep each language but may carry small errors at the switch + points. +- The live test is the pre-deploy gate for provider behaviour changes; it is not part of CI. +- Translation is a separate capability and is not addressed here. diff --git a/frontend/e2e/tests/multilingual-upload.spec.js b/frontend/e2e/tests/multilingual-upload.spec.js new file mode 100644 index 0000000..4fce652 --- /dev/null +++ b/frontend/e2e/tests/multilingual-upload.spec.js @@ -0,0 +1,55 @@ +// Spoken-language proof through the real UI: upload a Spanish clip, wait for the +// worker to transcribe it, and check the note shows Spanish text with a Spanish +// badge. Needs the dev stack running with a real provider key. Opt in with +// CLIO_LIVE_E2E=1 so CI (which has no provider key) is unaffected. +import path from 'path'; +import { test, expect } from '@playwright/test'; + +const FIXTURES = path.resolve(__dirname, '../../../backend/tests/fixtures/audio'); + +test.use({ storageState: 'playwright/.auth/user.json' }); + +const uploadAndWaitForTranscript = async (page, fileName, title) => { + await page.goto('/record'); + await page.waitForLoadState('networkidle'); + + await page.getByRole('tab', { name: 'Upload' }).click(); + await page.getByLabel('Choose an audio file to upload').setInputFiles(path.join(FIXTURES, fileName)); + await expect(page.getByText('Ready to transcribe')).toBeVisible(); + + await page.fill('input#note-title', title); + await page.getByRole('button', { name: 'Save & Transcribe' }).click(); + + await page.waitForURL(/\/notes\/\d+/, { timeout: 15000 }); + // The page polls while the worker runs. Wait for a positive completion signal (the + // language badge only renders once transcription finished) rather than for the + // processing indicator to disappear, which is not yet rendered on first paint. + await expect(page.getByTestId('note-language')).toBeVisible({ timeout: 90000 }); + return page; +}; + +test.describe('Multilingual transcription', () => { + test.skip(!process.env.CLIO_LIVE_E2E, 'set CLIO_LIVE_E2E=1 against a dev stack that has a real provider key'); + + test('Spanish upload comes back as Spanish text with a Spanish badge', async ({ page }) => { + await uploadAndWaitForTranscript(page, 'es_mx.m4a', 'e2e spanish'); + + const badge = page.getByTestId('note-language'); + await expect(badge).toHaveText('Spanish'); + await expect(badge).toHaveAttribute('data-language', 'es'); + + const body = (await page.textContent('main, body')).toLowerCase(); + expect(body).toContain('hermana'); + expect(body).not.toContain('my sister'); // not translated + await page.screenshot({ path: 'test-results/multilingual-spanish.png', fullPage: true }); + }); + + test('English then Spanish upload keeps both languages', async ({ page }) => { + await uploadAndWaitForTranscript(page, 'mixed_en_es.m4a', 'e2e mixed'); + + const body = (await page.textContent('main, body')).toLowerCase(); + expect(body).toContain('sister'); + expect(body).toContain('hermana'); + await page.screenshot({ path: 'test-results/multilingual-mixed.png', fullPage: true }); + }); +}); diff --git a/frontend/src/pages/NoteDetailPage.tsx b/frontend/src/pages/NoteDetailPage.tsx index a9a91ba..66148b5 100644 --- a/frontend/src/pages/NoteDetailPage.tsx +++ b/frontend/src/pages/NoteDetailPage.tsx @@ -18,6 +18,25 @@ interface SpeakerTurn { segments: TranscriptionSegment[]; } +// Languages the app offers for transcription. Mirrors VoiceNote.LANGUAGE_CHOICES +// on the backend; the badge and the re-transcribe dialog both read from here. +const LANGUAGE_OPTIONS = [ + { value: 'auto', label: 'Auto-detect' }, + { value: 'en', label: 'English' }, + { value: 'es', label: 'Spanish' }, + { value: 'fr', label: 'French' }, + { value: 'de', label: 'German' }, + { value: 'it', label: 'Italian' }, + { value: 'pt', label: 'Portuguese' }, + { value: 'ja', label: 'Japanese' }, + { value: 'ko', label: 'Korean' }, + { value: 'zh', label: 'Chinese' }, +]; + +// Human label for a detected language code; unknown codes show as-is. +const languageLabel = (code: string): string => + LANGUAGE_OPTIONS.find((o) => o.value === code)?.label ?? code; + // Collapse consecutive same-speaker segments into a single turn for display. const groupBySpeaker = (segments: TranscriptionSegment[]): SpeakerTurn[] => { const turns: SpeakerTurn[] = []; @@ -196,18 +215,7 @@ const NoteDetailPage: React.FC = () => { } ); - const languageOptions = [ - { value: 'auto', label: 'Auto-detect' }, - { value: 'en', label: 'English' }, - { value: 'es', label: 'Spanish' }, - { value: 'fr', label: 'French' }, - { value: 'de', label: 'German' }, - { value: 'it', label: 'Italian' }, - { value: 'pt', label: 'Portuguese' }, - { value: 'ja', label: 'Japanese' }, - { value: 'ko', label: 'Korean' }, - { value: 'zh', label: 'Chinese' }, - ]; + const languageOptions = LANGUAGE_OPTIONS; const handleRetranscribe = () => { retranscribeMutation.mutate({ language: selectedLanguage }); @@ -272,7 +280,11 @@ const NoteDetailPage: React.FC = () => { {note.language_detected && note.language_detected !== 'auto' && ( <> · - {note.language_detected} + {/* The label itself is English, so no `lang` here; the detected code is + exposed as data for tests and tooling. */} + + {languageLabel(note.language_detected)} + )}