From 52db5e6f6de767634ec1ee26e4da454f85d71a3e Mon Sep 17 00:00:00 2001 From: SaribAzim Date: Mon, 3 Aug 2026 14:21:11 +0500 Subject: [PATCH] update README to document EMA smoothing, impulse response model, baseline calibration, and acoustic feature extraction --- README.md | 311 ++++++++++++++++++++++++++++++------------------------ 1 file changed, 176 insertions(+), 135 deletions(-) diff --git a/README.md b/README.md index 6e3f027..d7bd3d1 100644 --- a/README.md +++ b/README.md @@ -1,138 +1,133 @@ # VoiceMonitor -![Python](https://img.shields.io/badge/python-3.10%2B-blue.svg) -![PyPI](https://img.shields.io/pypi/v/voiceMonitor) -![License](https://img.shields.io/badge/license-MIT-green) -![Downloads](https://img.shields.io/pypi/dm/voiceMonitor) -![Status](https://img.shields.io/badge/status-research--grade-success) +Real Time Vocal Fatigue Monitoring for Continuous Speech Analytics -**Real-Time Vocal Fatigue Monitoring for Continuous Speech Analytics** +VoiceMonitor is a Python library for real time vocal fatigue monitoring built on top of the auralis_vfs vocal fatigue scoring framework. It enables continuous microphone monitoring, fatigue scoring using sliding audio windows, personal baseline calibration, a vocal load impulse response model for recovery time estimation, and session level analytics for voice health monitoring. -VoiceMonitor is a Python library for **real-time vocal fatigue monitoring** built on top of the **auralis_vfs** vocal fatigue scoring framework. It enables continuous microphone monitoring, fatigue scoring using sliding audio windows, and session-level analytics for voice health monitoring. - -The system is designed for **researchers, speech engineers, and voice professionals** who require automated analysis of vocal strain during prolonged speech activity. - -VoiceMonitor processes live microphone input, extracts standardized audio segments, computes fatigue scores, and produces real-time alerts and session analytics. - ---- +The system is designed for researchers, speech engineers, and voice professionals who require automated analysis of vocal strain during prolonged speech activity. ## Overview -Prolonged speaking can lead to **vocal fatigue**, a condition characterized by strain, reduced vocal efficiency, and potential long-term damage to vocal health. - -VoiceMonitor provides a **real-time monitoring pipeline** that: - -* captures microphone audio streams -* processes sliding audio windows -* computes vocal fatigue scores -* tracks fatigue progression over time -* generates session analytics and warnings +Prolonged speaking can lead to vocal fatigue, a condition characterized by strain, reduced vocal efficiency, and potential long term damage to vocal health. -The system leverages the **ECAPA-TDNN-VHE vocal fatigue estimation model** and the `auralis_vfs` scoring framework developed as part of ongoing research in speech health monitoring. +VoiceMonitor provides a real time monitoring pipeline that: ---- +- captures microphone audio streams +- processes sliding audio windows +- computes vocal fatigue scores using the ECAPA-TDNN-VHE model via auralis_vfs +- smooths the fatigue signal with an exponential moving average +- models accumulating vocal load with a two component impulse response model, estimating a recovery time +- calibrates a personal baseline for the current speaker +- extracts auxiliary acoustic features (jitter, shimmer, harmonics to noise ratio, CPPS) +- generates session analytics, per window records, and warnings ## Key Features -* Real-time microphone audio monitoring -* Continuous vocal fatigue scoring -* Sliding window fatigue analysis -* Fatigue threshold warnings -* Session-level analytics and reports -* Chunk-based audio processing pipeline -* JSON session export for downstream analysis -* Lightweight CLI interface for quick experiments - ---- +- Real time microphone audio monitoring +- Continuous vocal fatigue scoring +- Sliding window fatigue analysis +- Exponential moving average smoothing of the raw fatigue signal +- Vocal load impulse response model with acute and chronic load components and recovery time estimation +- Personal baseline calibration during the first portion of a session +- Auxiliary acoustic feature extraction (jitter, shimmer, HNR, CPPS) +- Fatigue threshold warnings +- Session level analytics and reports, including per window detail +- Chunk based audio processing pipeline +- JSON session export for downstream analysis +- Lightweight CLI interface for quick experiments ## Architecture -VoiceMonitor uses a **sliding window inference pipeline** for continuous analysis. +VoiceMonitor uses a sliding window inference pipeline for continuous analysis. ``` Microphone Input - │ - ▼ + | + v Audio Stream Buffer - │ - ▼ + | + v Sliding Window Segmentation (5s) - │ - ▼ + | + v auralis_vfs Preprocessing - │ - ▼ -Vocal Fatigue Scoring - │ - ▼ + | + v +Vocal Fatigue Scoring (ECAPA-TDNN-VHE) + | + v +Acoustic Feature Extraction (auxiliary) + | + v +Personal Baseline Calibration + | + v +EMA Smoothing + Impulse Response Model (acute/chronic load, recovery ETA) + | + v Session Analytics Engine - │ - ▼ + | + v Fatigue Alerts + Reports ``` -Each processed window produces a **fatigue score**, enabling real-time tracking of vocal strain progression during speech sessions. - ---- +Each processed window produces a raw fatigue score, a smoothed score, acute and chronic load values, an estimated recovery time, baseline calibration state, and auxiliary acoustic features, all recorded per window and summarized at the session level. ## Installation ### Requirements -* Python ≥ 3.10 -* FFmpeg installed on system -* Microphone access +- Python 3.10 or newer +- FFmpeg installed on system +- Microphone access ### Install from PyPI -```bash +``` pip install voicemonitor ``` ### Install from source -```bash -git clone https://github.com/Khubaib8281/voiceMonitor.git -cd voicemonitor +``` +git clone https://github.com/Inference-LAB/voiceMonitor.git +cd voiceMonitor pip install -e . ``` ---- - -## Dependencies +### Dependencies VoiceMonitor relies on the following core libraries: -* `auralis_vfs` -* `numpy` -* `sounddevice` -* `soundfile` -* `pydub` -* `tqdm` +- auralis_vfs +- numpy +- sounddevice +- soundfile +- pydub +- tqdm +- praat-parselmouth (for acoustic feature extraction) FFmpeg must be installed for audio processing. ---- - ## Quick Start ### CLI Usage -Start real-time vocal fatigue monitoring: +Start real time vocal fatigue monitoring: -```bash +``` voicemonitor ``` Monitor for a fixed duration: -```bash +``` voicemonitor --duration 120 ``` Set a custom fatigue warning threshold: -```bash +``` voicemonitor --threshold 65 ``` @@ -153,14 +148,10 @@ After the session completes, a report is generated: session_report.json ``` ---- - -## Python API - -VoiceMonitor can also be used directly in Python applications. +### Python API ```python -from voicemonitor import VoiceMonitor +from voiceMonitor import VoiceMonitor monitor = VoiceMonitor(threshold=70) @@ -169,17 +160,29 @@ session = monitor.start(duration_sec=120) session.export_json("session_report.json") ``` ---- - ## Session Analytics -Each monitoring session records: +Each monitoring session records, per window: + +- raw fatigue score +- exponentially smoothed fatigue score +- acute load (short term strain, fast decay) +- chronic load (strain accumulating across the session, slow decay) +- readiness_experimental, an unvalidated exploratory metric (chronic minus acute) +- recovery_eta_sec, estimated seconds until acute load returns to a safe level +- baseline, the speaker's calibrated fatigue baseline +- baseline_adjusted_score, the raw score minus the calibrated baseline +- baseline_is_provisional, true while the baseline is still being calibrated +- auxiliary acoustic features (jitter, shimmer, HNR, CPPS), where available +- timestamp and chunk file path -* average fatigue score -* maximum fatigue score -* timestamps of processed windows -* processed audio chunk paths -* fatigue warning events +At the session level, the summary includes: + +- average and maximum raw fatigue score +- number of readings +- most recent smoothed fatigue score +- most recent acute load, chronic load, and readiness_experimental +- most recent recovery_eta_sec Example report: @@ -188,68 +191,101 @@ Example report: "summary": { "average_fatigue": 38.2, "max_fatigue": 74.1, - "readings": 25 + "readings": 25, + "smoothed_fatigue": 41.7, + "acute_load": 58.3, + "chronic_load": 44.1, + "readiness_experimental": -14.2, + "recovery_eta_sec": 33.9 }, "records": [ { "timestamp": "20260312_182001", "chunk": "chunks/20260312_182001.wav", - "score": 22.51 + "score": 22.51, + "features": { + "jitter_local": 0.012, + "shimmer_local": 0.041, + "hnr": 14.2, + "cpps": 7.8 + }, + "smoothed_score": 22.51, + "acute_load": 22.51, + "chronic_load": 22.51, + "readiness_experimental": 0.0, + "recovery_eta_sec": 0.0, + "baseline": 30, + "baseline_adjusted_score": 0, + "baseline_is_provisional": true } ] } ``` ---- +Note that auxiliary acoustic features and baseline adjusted scores are collected for inspection and future analysis. They do not currently influence the raw fatigue score or the warning threshold logic. ## Configuration -- Audio overlap is of `1 sec` -- default threshold is 70 +`voicemonitor/config.py` exposes the following tunable values: ---- +| Setting | Default | Description | +|---|---|---| +| SAMPLE_RATE | 16000 | Audio capture sample rate | +| WINDOW_SEC | 5 | Sliding window size in seconds | +| STEP_SEC | 4 | Step size between windows in seconds | +| DEFAULT_THRESHOLD | 70 | Default fatigue warning threshold (0 to 100 scale) | +| SAVE_CHUNKS | True | Whether to persist raw audio chunks to disk | +| CHUNK_DIR | "chunks" | Directory for saved audio chunks | +| EMA_ALPHA | 0.3 | Smoothing weight given to the newest window (0 to 1) | +| TAU_FAST | 90 | Acute component decay constant, in seconds | +| TAU_SLOW | 1500 | Chronic component decay constant, in seconds (25 minutes) | +| SAFE_RECOVERY_LEVEL | 40 | Acute load level considered recovered | +| EXTRACT_ACOUSTIC_FEATURES | True | Whether to extract auxiliary acoustic features per window | +| BASELINE_CALIBRATION_SEC | 45 | Length of the personal baseline calibration window, in seconds | +| DEFAULT_POPULATION_BASELINE | 30 | Fallback baseline used before calibration completes | -## Research Background +## Vocal Load Impulse Response Model -VoiceMonitor is built upon the **auralis_vfs vocal fatigue scoring framework**, which was developed as part of research on automated vocal fatigue detection. +Vocal load is modeled using two parallel leaky integrators, adapted from the Banister impulse response framework used in sports science to model athletic training load and recovery. Each fatigue score is treated as an impulse applied to a fast decaying acute component and a slow decaying chronic component. Readiness (chronic minus acute) mirrors how the athletic model computes freshness as fitness minus fatigue, though this reinterpretation is exploratory and has not yet been empirically validated for vocal fatigue, hence the `readiness_experimental` field name. Acute and chronic load are exposed separately so they can be used independently of the readiness metric. -The underlying fatigue estimation model is based on an **ECAPA-TDNN architecture adapted for vocal health estimation**. +Because voice has no direct positive equivalent to athletic fitness, both components here represent strain at different timescales rather than fitness and fatigue. Elapsed time for decay is derived from actual audio samples processed, not a wall clock, so it stays consistent with the session's own timestamps regardless of processing latency between windows. -Research paper: +## Personal Baseline Calibration -> [Modeling Vocal Fatigue as Embedding-Space Deviation Using Contrastively Trained ECAPA-TDNNs](https://doi.org/10.5281/zenodo.18366305) +During the first `BASELINE_CALIBRATION_SEC` seconds of a session, a personal fatigue baseline is estimated from the scores seen so far, with readings flagged as provisional. After the calibration window closes, the baseline locks and stays fixed for the rest of the session. If too little data is available, a conservative population default is used instead. The baseline adjusted score is stored per window but does not currently drive the fatigue warning threshold. -Model repository: +## Research Background -> [huggingface.co/Khubaib01/ECAPA-TDNN-VHE](https://huggingface.co/Khubaib01/ECAPA-TDNN-VHE) +VoiceMonitor is built upon the auralis_vfs vocal fatigue scoring framework, which was developed as part of research on automated vocal fatigue detection. The underlying fatigue estimation model is the ECAPA-TDNN-VHE architecture. -The model estimates vocal fatigue levels from short speech segments and provides a continuous fatigue score representing vocal strain. +Research paper: + +Modeling Vocal Fatigue as Embedding-Space Deviation Using Contrastively Trained ECAPA-TDNNs -VoiceMonitor extends this work by enabling **real-time fatigue monitoring and session analytics**. +Model repository: ---- +huggingface.co/Khubaib01/ECAPA-TDNN-VHE ## Applications -VoiceMonitor can be used in a variety of speech-intensive environments: - -* speech research -* voice health monitoring -* call center voice analytics -* teacher vocal load monitoring -* podcast and streaming voice tracking -* speech therapy experiments -* human-computer interaction studies +VoiceMonitor can be used in a variety of speech intensive environments: ---- +- speech research +- voice health monitoring +- call center voice analytics +- teacher vocal load monitoring +- podcast and streaming voice tracking +- speech therapy experiments +- human computer interaction studies ## Project Structure ``` -voicemonitor/ -├── voicemonitor/ +voiceMonitor/ +├── voiceMonitor/ │ ├── audio_stream.py │ ├── analytics.py +│ ├── baseline.py │ ├── session.py │ ├── utils.py │ ├── config.py @@ -259,48 +295,53 @@ voicemonitor/ │ └── live.py │ ├── tests/ -│ └── test_session.py +│ ├── test_analytics_impulse.py +│ ├── test_acoustic_features.py +│ ├── test_audio_stream.py +│ ├── test_baseline.py +│ ├── test_cli.py +│ ├── test_session.py +│ ├── test_session_report.py +│ └── test_utils.py │ -├── LISENCE +├── .github/ +│ └── workflows/ +│ └── ci.yml +│ +├── LICENSE ├── setup.cfg ├── requirements.txt ├── pyproject.toml └── README.md ``` ---- - ## Future Development Planned enhancements include: -* real-time visualization dashboard -* web API for remote monitoring -* desktop GUI interface -* voice activity detection integration -* fatigue trend prediction models -* speaker-aware monitoring - ---- +- integrating baseline adjusted scores and acoustic features into the fatigue score and warning logic, once validated +- empirical validation of the readiness metric before promoting it out of experimental status +- real time visualization dashboard +- web API for remote monitoring +- desktop GUI interface +- voice activity detection integration +- speaker aware monitoring for multi speaker sessions ## Citation If you use VoiceMonitor in research, please cite the underlying work: -```bibtex Ahmad, M. K. (2026). Modeling Vocal Fatigue as Embedding-Space Deviation Using Contrastively Trained ECAPA-TDNNs. Zenodo. https://doi.org/10.5281/zenodo.18366305 -``` - ---- ## License This project is released under the MIT License. ---- - ## Author Muhammad Khubaib Ahmad -AI / ML Engineer -Speech Intelligence and Audio AI Systems +AI / ML Engineer, Speech Intelligence and Audio AI Systems + +## Contributors + +Sarib Azim, Core Engineer (Research, Implementation and Evaluation), INFERENCE Lab Engineering Fellowship Cohort 01 \ No newline at end of file