diff --git a/README.md b/README.md index 172d5fc..ca8a7c2 100644 --- a/README.md +++ b/README.md @@ -67,6 +67,7 @@ Additional docs: - [Tested vacuums](docs/tested_vacuums.md) - [Home Assistant](docs/home_assistant.md) for the add-on install path and integration setup - [Mobile App Options](docs/roborock_app.md) +- [Voice packs](docs/voice_packs.md) for changing the robot's voice without internet access - [Custom MQTT](docs/custom_mqtt.md) - [Custom certificate management](docs/custom_cert_management.md) diff --git a/config.example.toml b/config.example.toml index 0230287..585513b 100644 --- a/config.example.toml +++ b/config.example.toml @@ -74,3 +74,10 @@ username = "" password = "" realm = "api-roborock.example.com" ttl = 86400 + +[voice_packs] +# Voice packs for robots without internet access (see docs/voice_packs.md). mirror fetches +# Roborock's list and each pack the first time a robot or app asks, then serves the copies kept +# under data_dir/voice_packs. offline serves only those copies and never contacts Roborock. +# disabled turns the /voice-pkg routes off. +mode = "mirror" # mirror | offline | disabled diff --git a/docs/index.md b/docs/index.md index 0d508a6..1be48a4 100644 --- a/docs/index.md +++ b/docs/index.md @@ -21,6 +21,7 @@ If you want to support this project, next time you buy a Roborock, use one of my - [Known limitations](known_limitations.md) - [Home Assistant](home_assistant.md) - [Mobile App Options](roborock_app.md) +- [Voice Packs](voice_packs.md) - [Updating](updating.md) - [Tested vacuums](tested_vacuums.md) - [Reverse proxy](reverse_proxy.md) diff --git a/docs/voice_packs.md b/docs/voice_packs.md new file mode 100644 index 0000000..3cd531f --- /dev/null +++ b/docs/voice_packs.md @@ -0,0 +1,65 @@ +# Voice Packs + +A robot changes its voice when an app sends it `dnld_install_sound` with a URL to download the pack +from. The official Roborock app takes that URL from Roborock's own list, so the robot is told to fetch +the pack from Roborock's servers. A robot whose internet access is blocked, as it usually is in a +local setup, cannot do that and the voice change fails. + +This server keeps its own copy of the list and the packs and serves them from your stack, so a client +can hand the robot a URL on your server instead. + +## How it works + +- `GET https:///voice-pkg//info` returns Roborock's list for a product, with every + pack, picture and preview URL rewritten to point at this server. +- `GET https:///voice-pkg//{package,pic,wav}/` serves a file. The first + request fetches it from Roborock, checks it against the checksum Roborock's storage sends with it (when there is one), and keeps it under + `/voice_packs//`; later requests are served from that copy. +- The first time a bucket's list is asked for, the server starts downloading every pack in it in the + background, so later voice changes need no internet at all. +- The list is refreshed from Roborock at most once a day. When Roborock can't be reached, the last + stored list is used. + +The routes need no login, as the robot downloads without credentials (Roborock's own copies are +public too). Only bucket names and plain file names from the list are accepted. + +The upstream host follows `network.region` (`eu`, `us` or `cn`; other regions use `eu`). + +## Buckets + +Roborock keeps each product's voice packs in a folder named after the product's code name in its +device plugin: + +| Bucket | Models | +|---|---| +| `vida` | Saros 20 Neo (`roborock.vacuum.a295`, `roborock.vacuum.a296`) | + +Other products' buckets can be found in their device plugin (the `bucket` value next to the +voice-pack page). Add them here when you confirm one. + +## Clients + +The official Roborock app reads the list from Roborock directly, so it still sends Roborock's URLs +and voice changes from it keep failing on a robot without internet access. A client that wants to +use this mirror reads `/voice-pkg//info` and sends the robot: + +```json +{"method": "dnld_install_sound", "params": {"sid": , "url": "", "md5": "", "sver": }} +``` + +with the values of one `voice_list` entry, then follows progress with `get_sound_progress`. +`voice_pkg_md5` is passed to the robot exactly as listed (it is not the MD5 of the file). + +## Configuration + +```toml +[voice_packs] +mode = "mirror" # mirror | offline | disabled +``` + +- `mirror` (default): fetch from Roborock on first use and keep copies, as described above. +- `offline`: serve only the copies already stored and never contact Roborock. Use this after the + packs have been mirrored once, or after copying a `voice_packs` folder into `` yourself. +- `disabled`: the `/voice-pkg` routes answer 404. + +In the Home Assistant add-on the same choice is the `voice_packs_mode` option. diff --git a/mkdocs.yml b/mkdocs.yml index 9bcc256..8f6f404 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -57,6 +57,7 @@ nav: - Integrations: - Mobile App Options: roborock_app.md - Camera (Live Video): camera.md + - Voice Packs: voice_packs.md - Reference: - Log Retention: logging.md - Known Limitations: known_limitations.md diff --git a/roborock_local_server_addon/DOCS.md b/roborock_local_server_addon/DOCS.md index e48789e..35f0f66 100644 --- a/roborock_local_server_addon/DOCS.md +++ b/roborock_local_server_addon/DOCS.md @@ -21,6 +21,8 @@ It publishes two TLS ports directly: For camera live view, set `turn_mode` to `provided` to start coturn inside the add-on. It uses `stack_fqdn` as the advertised host and generates a persistent password when `turn_password` is blank. Set `turn_mode` to `external` with `turn_host`, `turn_port`, `turn_username`, `turn_password`, and optionally `turn_realm` to advertise your existing relay. `disabled` leaves TURN off. In provided mode, allow UDP 3478 and 49160-49179 between the robot, viewer, and Home Assistant host. +`voice_packs_mode` controls the voice-pack mirror for robots without internet access: `mirror` (default) fetches Roborock's packs on first use and keeps copies, `offline` serves only stored copies, `disabled` turns it off. See the Voice Packs page of the documentation. + Before choosing the TLS mode, check the tested-vacuum certificate guidance in `docs/tested_vacuums.md`. Different models may need `zerossl`, `actalis`, or your own certificate chain. For most users, prefer `zerossl`. Use `actalis` mainly for older vacuums or when your model is already known to need it. The add-on always runs the embedded MQTT broker and keeps the topic bridge enabled. diff --git a/roborock_local_server_addon/config.yaml b/roborock_local_server_addon/config.yaml index a5fcda8..bacedc0 100644 --- a/roborock_local_server_addon/config.yaml +++ b/roborock_local_server_addon/config.yaml @@ -91,6 +91,7 @@ options: turn_password: "" turn_realm: "" turn_ttl: 86400 + voice_packs_mode: "mirror" schema: stack_fqdn: str https_port: port @@ -119,3 +120,4 @@ schema: turn_password: password? turn_realm: str? turn_ttl: int? + voice_packs_mode: list(mirror|offline|disabled)? diff --git a/src/roborock_local_server/config.py b/src/roborock_local_server/config.py index 1469631..2426990 100644 --- a/src/roborock_local_server/config.py +++ b/src/roborock_local_server/config.py @@ -104,6 +104,20 @@ class TurnConfig: ttl: int = 86400 +VOICE_PACK_MODES = ("mirror", "offline", "disabled") + + +@dataclass(frozen=True) +class VoicePacksConfig: + """Voice packs served to robots that cannot reach Roborock (see docs/voice_packs.md). + + ``mirror`` fetches Roborock's list and packs on first use and keeps copies; ``offline`` serves + only the copies already stored and never contacts Roborock; ``disabled`` turns the routes off. + """ + + mode: str = "mirror" + + @dataclass(frozen=True) class AppConfig: network: NetworkConfig @@ -113,6 +127,7 @@ class AppConfig: admin: AdminConfig turn: TurnConfig = TurnConfig() logging: LoggingConfig = LoggingConfig() + voice_packs: VoicePacksConfig = VoicePacksConfig() @dataclass(frozen=True) @@ -259,6 +274,7 @@ def load_config(path: str | Path) -> AppConfig: logging = _get_section(parsed, "logging") tls = _get_section(parsed, "tls") admin = _get_section(parsed, "admin") + voice_packs = _get_section(parsed, "voice_packs") turn = _get_section(parsed, "turn") turn_mode = str(turn.get("mode", "external" if _as_bool(turn.get("enabled"), False) else "disabled")).strip().lower() broker_mode = str(broker.get("mode", "embedded")).strip().lower() @@ -368,6 +384,7 @@ def load_config(path: str | Path) -> AppConfig: realm=str(turn.get("realm", "") or turn.get("host", "")).strip(), ttl=_as_int(turn.get("ttl"), "turn.ttl", 86400), ), + voice_packs=VoicePacksConfig(mode=str(voice_packs.get("mode", "mirror")).strip().lower()), ) if len(config.admin.session_secret) < 24: @@ -378,6 +395,8 @@ def load_config(path: str | Path) -> AppConfig: if config.broker.mode == "external": _require_non_empty(config.broker.host, "broker.host") + if config.voice_packs.mode not in VOICE_PACK_MODES: + raise ValueError("voice_packs.mode must be 'mirror', 'offline', or 'disabled'") if config.turn.mode not in {"disabled", "provided", "external"}: raise ValueError("turn.mode must be 'disabled', 'provided', or 'external'") if config.turn.mode != "disabled": diff --git a/src/roborock_local_server/ha_addon.py b/src/roborock_local_server/ha_addon.py index 7d8ccf9..ee57d26 100644 --- a/src/roborock_local_server/ha_addon.py +++ b/src/roborock_local_server/ha_addon.py @@ -43,6 +43,7 @@ "turn_password": "", "turn_realm": "", "turn_ttl": 86400, + "voice_packs_mode": "mirror", } _HOST_RE = re.compile(r"^[a-z0-9.-]+$") @@ -258,6 +259,9 @@ def _render_config_toml( acme_eab_kid_file = str(acme_eab_kid_path) if acme_server == "actalis" else "" acme_eab_hmac_key_file = str(acme_eab_hmac_key_path) if acme_server == "actalis" else "" + voice_packs_mode = str(merged.get("voice_packs_mode") or "mirror").strip().lower() + if voice_packs_mode not in {"mirror", "offline", "disabled"}: + raise ValueError("voice_packs_mode must be mirror, offline, or disabled") turn_mode = str(merged.get("turn_mode") or "disabled").strip().lower() if turn_mode not in {"disabled", "provided", "external"}: raise ValueError("turn_mode must be disabled, provided, or external") @@ -373,6 +377,9 @@ def _render_config_toml( f"realm = {_toml_string(turn_realm)}", f"ttl = {turn_ttl}", "", + "[voice_packs]", + f"mode = {_toml_string(voice_packs_mode)}", + "", ] ) secrets_to_write: dict[Path, str] = {} diff --git a/src/roborock_local_server/server.py b/src/roborock_local_server/server.py index 105a2aa..cfe8b58 100644 --- a/src/roborock_local_server/server.py +++ b/src/roborock_local_server/server.py @@ -18,7 +18,7 @@ from urllib.parse import parse_qs from fastapi import FastAPI, HTTPException, Request -from fastapi.responses import JSONResponse, Response +from fastapi.responses import FileResponse, JSONResponse, Response from python_multipart.exceptions import MultipartParseError import uvicorn from uvicorn.middleware.proxy_headers import ProxyHeadersMiddleware @@ -32,6 +32,7 @@ from .config import AppConfig, AppPaths, load_config, resolve_paths from .log_retention import configure_jsonl_logging from .turn_relay import EmbeddedTurnRelay +from .voice_packs import VoicePackError, VoicePackMirror, rewrite_list as rewrite_voice_list from .standalone_admin import register_standalone_admin_routes from .backend import ( MqttTlsProxy, @@ -348,6 +349,9 @@ def __init__( self.config = config self.paths = paths self.enable_standalone_admin = bool(enable_standalone_admin) + self.voice_packs = VoicePackMirror(paths.data_dir, config.network.region, offline=config.voice_packs.mode == "offline") + self._voice_prefetch_started: set[str] = set() + self._voice_prefetch_tasks: set[asyncio.Task] = set() self.root_logger = logging.getLogger("roborock_local_server") self.root_logger.setLevel(logging.INFO) @@ -1842,6 +1846,33 @@ async def ui_vacuums(request: Request) -> JSONResponse: self._require_admin(request) return JSONResponse(self._ui_vacuums_payload()) + # Voice packs, mirrored from Roborock for robots that can't reach it (see voice_packs.py). + # Public like Roborock's own copies: the robot downloads them without credentials. + @app.get("/voice-pkg/{bucket}/info") + async def voice_pack_list(bucket: str) -> Response: + if self.config.voice_packs.mode == "disabled": + return JSONResponse({"error": "Not Found"}, status_code=404) + try: + info = await self.voice_packs.get_list(bucket) + except VoicePackError as exc: + return JSONResponse({"error": str(exc)}, status_code=exc.status) + if not self.voice_packs.offline and bucket not in self._voice_prefetch_started: + # Keep a copy of every pack from the first time the list is asked for. + self._start_voice_prefetch(bucket) + port = self.config.network.advertised_https_port + base_url = f"https://{self.config.network.stack_fqdn}" + ("" if port in (0, 443) else f":{port}") + return JSONResponse(rewrite_voice_list(info, base_url, bucket)) + + @app.get("/voice-pkg/{bucket}/{kind}/{name}") + async def voice_pack_file(bucket: str, kind: str, name: str) -> Response: + if self.config.voice_packs.mode == "disabled": + return JSONResponse({"error": "Not Found"}, status_code=404) + try: + path = await self.voice_packs.get_file(bucket, kind, name) + except VoicePackError as exc: + return JSONResponse({"error": str(exc)}, status_code=exc.status) + return FileResponse(path) + @app.api_route("/", methods=list(ALL_HTTP_METHODS)) async def root_handler(request: Request) -> Response: if self._is_standalone_route_path(request.url.path): @@ -1997,8 +2028,35 @@ async def start(self) -> None: if self._uses_local_tls() and self.config.tls.mode == "cloudflare_acme": self._renew_task = asyncio.create_task(self._renew_loop(), name="tls-renew-loop") + def _start_voice_prefetch(self, bucket: str) -> None: + """Download every pack of [bucket] in the background; a failed run may be started again later.""" + self._voice_prefetch_started.add(bucket) + task = asyncio.create_task(self.voice_packs.prefetch(bucket), name=f"voice-prefetch-{bucket}") + self._voice_prefetch_tasks.add(task) + + def done(t: asyncio.Task) -> None: + self._voice_prefetch_tasks.discard(t) + if t.cancelled(): + self._voice_prefetch_started.discard(bucket) + return + exc = t.exception() + if exc is not None: + self._voice_prefetch_started.discard(bucket) + self.root_logger.warning("Voice packs: prefetch of %s failed: %s", bucket, exc) + return + result = t.result() + if result.get("failed"): + self._voice_prefetch_started.discard(bucket) + self.root_logger.info( + "Voice packs: %s has %s files stored, %s failed", bucket, result.get("stored"), len(result.get("failed") or {}) + ) + + task.add_done_callback(done) + async def stop(self) -> None: self._stop_event.set() + for task in list(self._voice_prefetch_tasks): + task.cancel() self._turn_relay.stop() self._refresh_turn_health() if self._renew_task is not None: diff --git a/src/roborock_local_server/voice_packs.py b/src/roborock_local_server/voice_packs.py new file mode 100644 index 0000000..375288f --- /dev/null +++ b/src/roborock_local_server/voice_packs.py @@ -0,0 +1,308 @@ +"""Voice packs for robots that cannot reach Roborock's servers. + +The official app reads a list of voice packs from Roborock's resource host and then tells the robot, +with ``dnld_install_sound``, to download one from a URL in that list. A robot whose internet access is +blocked (as a local setup usually is) cannot fetch it. This module mirrors the list and the files on +this server: the list is served with every URL pointing here, and each file is fetched from Roborock +once, checked, and kept under ``/voice_packs`` from then on. + +Layout upstream (from the official app's device plugin): ``https://-app.roborock.com/app/ +/voice-pkg/info`` for the list, and ``.../voice-pkg/{package,pic,wav}/`` for the files. +The bucket is per product (for example ``vida`` for the Saros 20 Neo, a295/a296). +""" + +from __future__ import annotations + +import asyncio +import copy +import hashlib +import json +import logging +import re +import time +from pathlib import Path +from typing import Any + +import aiohttp + +UPSTREAM_HOSTS = {"eu": "eu-app.roborock.com", "us": "us-app.roborock.com", "cn": "cn-app.roborock.com"} +KINDS = {"package": ".pkg", "pic": ".png", "wav": ".wav"} +_BUCKET_RE = re.compile(r"^[a-z0-9][a-z0-9_-]{0,31}$") +_FILE_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9_.-]{0,63}$") +LIST_MAX_AGE_SECONDS = 24 * 3600 +LIST_TIMEOUT_SECONDS = 15.0 +# A refused file, or a list whose refresh failed, is not asked for again for this long. +RETRY_AFTER_SECONDS = 600.0 +MAX_RETRY_ENTRIES = 1024 +# Voice packs are a few MB; anything far larger is not one. +MAX_FILE_BYTES = 64 * 1024 * 1024 +MAX_LIST_BYTES = 2 * 1024 * 1024 +# A home has a handful of products; this keeps public requests from mirroring every bucket Roborock has. +MAX_BUCKETS = 8 +_LOGGER = logging.getLogger(__name__) +_URL_KEYS = {"voice_pkg_url": "package", "bg_pic": "pic", "voice_pre_listen": "wav"} + + +class VoicePackError(Exception): + def __init__(self, status: int, message: str) -> None: + super().__init__(message) + self.status = status + + +def valid_bucket(bucket: str) -> bool: + return bool(_BUCKET_RE.match(bucket or "")) + + +def valid_file(kind: str, name: str) -> bool: + ext = KINDS.get(kind) + return bool(ext and _FILE_RE.match(name or "") and name.endswith(ext) and ".." not in name) + + +def rewrite_list(info: dict[str, Any], base_url: str, bucket: str) -> dict[str, Any]: + """The list with every pack, picture and preview URL pointing at this server.""" + out = copy.deepcopy(info) + for voice in out.get("voice_list") or []: + if not isinstance(voice, dict): + continue + for key, kind in _URL_KEYS.items(): + url = voice.get(key) + if isinstance(url, str) and url: + name = url.rsplit("/", 1)[-1].split("?", 1)[0] + if valid_file(kind, name): + voice[key] = f"{base_url.rstrip('/')}/voice-pkg/{bucket}/{kind}/{name}" + return out + + +def listed_entry(info: dict[str, Any] | None, kind: str, name: str) -> dict[str, Any] | None: + """The list entry whose package, picture or preview ([kind]) is the file [name], if any.""" + keys = [key for key, k in _URL_KEYS.items() if k == kind] + for voice in (info or {}).get("voice_list") or []: + if not isinstance(voice, dict): + continue + for key in keys: + if str(voice.get(key) or "").rsplit("/", 1)[-1].split("?", 1)[0] == name: + return voice + return None + + +def _source_tag(entry: dict[str, Any]) -> str: + """What identifies a package's content in the list; a change means Roborock published a new file.""" + return f"{entry.get('voice_pkg_md5', '')}:{entry.get('version', '')}" + + +class VoicePackMirror: + """With ``offline`` set, only copies already under ``/voice_packs`` are served and Roborock is never contacted.""" + + def __init__(self, data_dir: Path, region: str, timeout_seconds: float = 120.0, offline: bool = False) -> None: + self.root = Path(data_dir) / "voice_packs" + self.offline = offline + region = (region or "").lower() + if region not in UPSTREAM_HOSTS: + _LOGGER.warning("Voice packs: no Roborock host known for region %r, using eu", region) + self.upstream_host = UPSTREAM_HOSTS.get(region, UPSTREAM_HOSTS["eu"]) + self.timeout = aiohttp.ClientTimeout(total=timeout_seconds) + self.list_timeout = aiohttp.ClientTimeout(total=min(timeout_seconds, LIST_TIMEOUT_SECONDS)) + # Lists are small and rare: one lock for all of them. File locks are only made for files in a + # stored list, so their number stays bounded. + self._list_lock = asyncio.Lock() + self._file_locks: dict[str, asyncio.Lock] = {} + # Keys Roborock refused, and lists whose refresh failed, with when to ask again. + self._retry_after: dict[str, float] = {} + + def _upstream(self, bucket: str, tail: str) -> str: + return f"https://{self.upstream_host}/app/{bucket}/voice-pkg/{tail}" + + def _list_path(self, bucket: str) -> Path: + return self.root / bucket / "info.json" + + def _stored_buckets(self) -> set[str]: + try: + return {p.parent.name for p in self.root.glob("*/info.json")} + except OSError: + return set() + + def _backing_off(self, key: str) -> bool: + until = self._retry_after.get(key) + if until is None: + return False + if until > time.monotonic(): + return True + del self._retry_after[key] + return False + + def _back_off(self, key: str) -> None: + if len(self._retry_after) >= MAX_RETRY_ENTRIES: + now = time.monotonic() + self._retry_after = {k: v for k, v in self._retry_after.items() if v > now} + if len(self._retry_after) >= MAX_RETRY_ENTRIES: + self._retry_after.clear() + self._retry_after[key] = time.monotonic() + RETRY_AFTER_SECONDS + + async def _download(self, url: str, max_bytes: int = MAX_FILE_BYTES, list_request: bool = False) -> bytes: + """ + The body at [url], read in chunks and refused past [max_bytes], and checked against Roborock's + ETag when that is an MD5 (their storage sets it to the file's MD5). + """ + timeout = self.list_timeout if list_request else self.timeout + async with aiohttp.ClientSession(timeout=timeout) as session: + async with session.get(url) as response: + if response.status != 200: + raise VoicePackError(502 if response.status != 404 else 404, f"Roborock answered {response.status} for {url}") + if (response.content_length or 0) > max_bytes: + raise VoicePackError(502, f"{url} is larger than {max_bytes} bytes") + digest = hashlib.md5() + chunks: list[bytes] = [] + size = 0 + async for chunk in response.content.iter_chunked(64 * 1024): + size += len(chunk) + if size > max_bytes: + raise VoicePackError(502, f"{url} is larger than {max_bytes} bytes") + digest.update(chunk) + chunks.append(chunk) + etag = (response.headers.get("ETag") or "").strip('"').lower() + if re.fullmatch(r"[0-9a-f]{32}", etag) and digest.hexdigest() != etag: + raise VoicePackError(502, f"The download of {url} was incomplete") + return b"".join(chunks) + + def _read_stored_list(self, path: Path) -> dict[str, Any] | None: + """The stored list, or None when there is none or it no longer parses (it is then removed).""" + try: + info = json.loads(path.read_text(encoding="utf-8")) + except FileNotFoundError: + return None + except (OSError, ValueError): + _LOGGER.warning("Voice packs: removing unreadable list %s", path) + path.unlink(missing_ok=True) + return None + return info if isinstance(info, dict) and isinstance(info.get("voice_list"), list) else None + + @staticmethod + def _write_atomic(path: Path, data: bytes) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_name(path.name + ".tmp") + try: + tmp.write_bytes(data) + tmp.replace(path) + except OSError as exc: + tmp.unlink(missing_ok=True) + raise VoicePackError(507, f"Could not store {path.name}: {exc}") from exc + + async def get_list(self, bucket: str) -> dict[str, Any]: + """ + Roborock's list for [bucket], fetched at most once a day; the last copy is used when Roborock + can't be reached, and a failed refresh is not retried for a while. At most [MAX_BUCKETS] + buckets are mirrored. + """ + if not valid_bucket(bucket): + raise VoicePackError(400, "Unknown voice pack bucket") + path = self._list_path(bucket) + if self.offline: + info = self._read_stored_list(path) + if info is None: + raise VoicePackError(404, "No stored voice pack list for this bucket") + return info + async with self._list_lock: + stored = self._read_stored_list(path) + try: + fresh = stored is not None and time.time() - path.stat().st_mtime < LIST_MAX_AGE_SECONDS + except OSError: + fresh = False + key = f"list:{bucket}" + if fresh or self._backing_off(key): + if stored is None: + raise VoicePackError(404, "No voice pack list for this bucket") + return stored + if stored is None and len(self._stored_buckets()) >= MAX_BUCKETS: + raise VoicePackError(404, "This server already mirrors the most voice pack buckets it keeps") + try: + raw = await self._download(self._upstream(bucket, "info"), max_bytes=MAX_LIST_BYTES, list_request=True) + info = json.loads(raw) + if not isinstance(info, dict) or not isinstance(info.get("voice_list"), list): + raise VoicePackError(502, "Roborock's voice pack list has an unexpected shape") + self._write_atomic(path, json.dumps(info, ensure_ascii=False).encode("utf-8")) + return info + except (aiohttp.ClientError, asyncio.TimeoutError, ValueError, VoicePackError) as exc: + self._back_off(key) + if stored is not None: + return stored + if isinstance(exc, VoicePackError): + raise + raise VoicePackError(502, f"Could not fetch the voice pack list: {exc}") from exc + + async def get_file(self, bucket: str, kind: str, name: str) -> Path: + """ + The local copy of a file in the list, fetched from Roborock the first time, and again when the + list shows a new version of a package. The list's `voice_pkg_md5` is not the MD5 of the file (an + a296 `nl.pkg` hashes to its ETag instead), so it is passed on to the robot untouched and the + download is checked against the ETag. + """ + if not valid_bucket(bucket) or not valid_file(kind, name): + raise VoicePackError(404, "Not Found") + path = self.root / bucket / kind / name + if self.offline: + if path.exists(): + return path + raise VoicePackError(404, "Not Found") + entry = listed_entry(await self.get_list(bucket), kind, name) + if entry is None: + raise VoicePackError(404, "This file is not in the voice pack list") + tag_path = path.with_name(path.name + ".src") + tag = _source_tag(entry) if kind == "package" else "" + + def current() -> bool: + if not path.exists(): + return False + if kind != "package": + return True + try: + return tag_path.read_text(encoding="utf-8") == tag + except OSError: + # Copies stored before the tag existed are kept. + return not tag_path.exists() + + if current(): + return path + key = f"{bucket}/{kind}/{name}" + async with self._file_locks.setdefault(key, asyncio.Lock()): + if current(): + return path + if self._backing_off(f"file:{key}"): + if path.exists(): + return path + raise VoicePackError(404, f"Roborock did not have {name}") + try: + data = await self._download(self._upstream(bucket, f"{kind}/{name}")) + except (aiohttp.ClientError, asyncio.TimeoutError) as exc: + if path.exists(): + return path + raise VoicePackError(502, f"Could not fetch {name}: {exc}") from exc + except VoicePackError: + self._back_off(f"file:{key}") + if path.exists(): + return path + raise + self._write_atomic(path, data) + if kind == "package": + self._write_atomic(tag_path, tag.encode("utf-8")) + return path + + async def prefetch(self, bucket: str) -> dict[str, Any]: + """Downloads every package, picture and preview in the list, so later installs need no internet.""" + if self.offline: + return {"bucket": bucket, "stored": 0, "failed": {}} + info = await self.get_list(bucket) + done: list[str] = [] + failed: dict[str, str] = {} + for voice in info.get("voice_list") or []: + if not isinstance(voice, dict): + continue + for key, kind in _URL_KEYS.items(): + name = str(voice.get(key) or "").rsplit("/", 1)[-1].split("?", 1)[0] + if not valid_file(kind, name): + continue + try: + await self.get_file(bucket, kind, name) + done.append(f"{kind}/{name}") + except VoicePackError as exc: + failed[f"{kind}/{name}"] = str(exc) + return {"bucket": bucket, "stored": len(done), "failed": failed} diff --git a/tests/test_config.py b/tests/test_config.py index ba56958..ad50feb 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -557,3 +557,14 @@ def test_load_config_rejects_invalid_trusted_proxy(tmp_path: Path) -> None: with pytest.raises(ValueError, match="network.trusted_proxies entry 'traefik'"): load_config(config_file) + + +def test_voice_packs_mode_defaults_to_mirror_and_is_validated(tmp_path): + path = write_release_config(tmp_path) + base = path.read_text() + assert load_config(path).voice_packs.mode == "mirror" + path.write_text(base + '\n[voice_packs]\nmode="Offline"\n') + assert load_config(path).voice_packs.mode == "offline" + path.write_text(base + '\n[voice_packs]\nmode="sometimes"\n') + with pytest.raises(ValueError, match="voice_packs.mode"): + load_config(path) diff --git a/tests/test_ha_addon.py b/tests/test_ha_addon.py index 8cdbdd4..9a6802b 100644 --- a/tests/test_ha_addon.py +++ b/tests/test_ha_addon.py @@ -556,6 +556,7 @@ def test_write_config_from_home_assistant_options_renders_turn_settings(tmp_path assert parsed["turn"]["password"] == "secretpassword" assert parsed["turn"]["realm"] == "turn.example.com" assert parsed["turn"]["ttl"] == 86400 + assert parsed["voice_packs"]["mode"] == "mirror" def test_provided_turn_generates_stable_credentials_and_uses_stack_host(tmp_path: Path) -> None: diff --git a/tests/test_voice_packs.py b/tests/test_voice_packs.py new file mode 100644 index 0000000..66c2aa6 --- /dev/null +++ b/tests/test_voice_packs.py @@ -0,0 +1,279 @@ +import asyncio +import hashlib +import json + +import pytest + +from roborock_local_server import voice_packs as vp + +INFO = { + "voice_pkg_version": 1, + "voice_list": [ + { + "voice_id": 129, "lang": "nl", "version": 4, + "voice_pkg_url": "https://eu-app.roborock.com/app/vida/voice-pkg/package/nl.pkg", + "bg_pic": "https://eu-app.roborock.com/app/vida/voice-pkg/pic/nl.png", + "voice_pre_listen": "https://eu-app.roborock.com/app/vida/voice-pkg/wav/nl.wav", + "voice_pkg_md5": hashlib.md5(b"dutch").hexdigest(), + } + ], +} + + +def test_list_urls_point_at_this_server(): + out = vp.rewrite_list(INFO, "https://api-roborock.example.com", "vida") + voice = out["voice_list"][0] + assert voice["voice_pkg_url"] == "https://api-roborock.example.com/voice-pkg/vida/package/nl.pkg" + assert voice["bg_pic"] == "https://api-roborock.example.com/voice-pkg/vida/pic/nl.png" + assert voice["voice_pre_listen"] == "https://api-roborock.example.com/voice-pkg/vida/wav/nl.wav" + assert voice["voice_pkg_md5"] == INFO["voice_list"][0]["voice_pkg_md5"] + # The original is left alone. + assert INFO["voice_list"][0]["voice_pkg_url"].startswith("https://eu-app.roborock.com/") + + +@pytest.mark.parametrize("kind,name,ok", [ + ("package", "nl.pkg", True), ("pic", "nl.png", True), ("wav", "es-la.wav", True), + ("package", "../x.pkg", False), ("package", "nl.png", False), ("other", "nl.pkg", False), + ("package", ".pkg", False), ("package", "a/b.pkg", False), +]) +def test_only_plain_file_names_are_served(kind, name, ok): + assert vp.valid_file(kind, name) is ok + + +@pytest.mark.parametrize("bucket,ok", [("vida", True), ("a-b_1", True), ("", False), ("../etc", False), ("Vida", False)]) +def test_bucket_names(bucket, ok): + assert vp.valid_bucket(bucket) is ok + + +def test_files_are_fetched_once_and_checked(tmp_path, monkeypatch): + asyncio.run(_fetched_once(tmp_path, monkeypatch)) + + +async def _fetched_once(tmp_path, monkeypatch): + mirror = vp.VoicePackMirror(tmp_path, "eu") + calls = [] + + async def fake_download(url, **kwargs): + calls.append(url) + if url.endswith("/info"): + return json.dumps(INFO).encode() + return b"dutch" + + monkeypatch.setattr(mirror, "_download", fake_download) + path = await mirror.get_file("vida", "package", "nl.pkg") + assert path.read_bytes() == b"dutch" + assert calls == [ + "https://eu-app.roborock.com/app/vida/voice-pkg/info", + "https://eu-app.roborock.com/app/vida/voice-pkg/package/nl.pkg", + ] + await mirror.get_file("vida", "package", "nl.pkg") + assert len(calls) == 2 + + +def test_a_package_not_in_the_list_is_not_fetched(tmp_path, monkeypatch): + asyncio.run(_not_listed(tmp_path, monkeypatch)) + + +async def _not_listed(tmp_path, monkeypatch): + mirror = vp.VoicePackMirror(tmp_path, "eu") + calls = [] + + async def fake_download(url, **kwargs): + calls.append(url) + return json.dumps(INFO).encode() + + monkeypatch.setattr(mirror, "_download", fake_download) + with pytest.raises(vp.VoicePackError): + await mirror.get_file("vida", "package", "xx.pkg") + assert calls == ["https://eu-app.roborock.com/app/vida/voice-pkg/info"] + + +def test_the_last_list_is_used_when_roborock_is_unreachable(tmp_path, monkeypatch): + asyncio.run(_last_list(tmp_path, monkeypatch)) + + +async def _last_list(tmp_path, monkeypatch): + mirror = vp.VoicePackMirror(tmp_path, "eu") + + async def ok(url, **kwargs): + return json.dumps(INFO).encode() + + monkeypatch.setattr(mirror, "_download", ok) + await mirror.get_list("vida") + monkeypatch.setattr(vp, "LIST_MAX_AGE_SECONDS", -1) + + async def down(url, **kwargs): + raise vp.aiohttp.ClientError("offline") + + monkeypatch.setattr(mirror, "_download", down) + assert (await mirror.get_list("vida"))["voice_list"][0]["voice_id"] == 129 + + +def test_offline_serves_stored_copies_and_never_downloads(tmp_path, monkeypatch): + asyncio.run(_offline(tmp_path, monkeypatch)) + + +async def _offline(tmp_path, monkeypatch): + online = vp.VoicePackMirror(tmp_path, "eu") + + async def fake_download(url, **kwargs): + return json.dumps(INFO).encode() if url.endswith("/info") else b"dutch" + + monkeypatch.setattr(online, "_download", fake_download) + await online.get_file("vida", "package", "nl.pkg") + + offline = vp.VoicePackMirror(tmp_path, "eu", offline=True) + + async def no_download(url, **kwargs): + raise AssertionError(f"offline mode fetched {url}") + + monkeypatch.setattr(offline, "_download", no_download) + assert (await offline.get_list("vida"))["voice_list"][0]["voice_id"] == 129 + assert (await offline.get_file("vida", "package", "nl.pkg")).read_bytes() == b"dutch" + with pytest.raises(vp.VoicePackError) as missing: + await offline.get_file("vida", "wav", "nl.wav") + assert missing.value.status == 404 + with pytest.raises(vp.VoicePackError) as no_list: + await offline.get_list("other") + assert no_list.value.status == 404 + assert (await offline.prefetch("vida"))["stored"] == 0 + + +def _client(tmp_path, monkeypatch, extra: str = "", https_port: int = 443): + from fastapi.testclient import TestClient + from conftest import write_release_config + from roborock_local_server.config import load_config, resolve_paths + from roborock_local_server.server import ReleaseSupervisor + + config_file = write_release_config(tmp_path, https_port=https_port) + if extra: + config_file.write_text(config_file.read_text() + extra) + config = load_config(config_file) + supervisor = ReleaseSupervisor(config=config, paths=resolve_paths(config_file, config)) + + async def fake_download(url, **kwargs): + return json.dumps(INFO).encode() if url.endswith("/info") else b"dutch" + + monkeypatch.setattr(supervisor.voice_packs, "_download", fake_download) + monkeypatch.setattr(supervisor.voice_packs, "prefetch", lambda bucket: asyncio.sleep(0)) + return TestClient(supervisor.app) + + +def test_the_list_route_points_at_the_advertised_https_port(tmp_path, monkeypatch): + client = _client(tmp_path, monkeypatch) + voice = client.get("/voice-pkg/vida/info").json()["voice_list"][0] + assert voice["voice_pkg_url"] == "https://api-roborock.example.com/voice-pkg/vida/package/nl.pkg" + assert client.get("/voice-pkg/vida/package/nl.pkg").content == b"dutch" + + other = _client(tmp_path / "p555", monkeypatch, https_port=555) + voice = other.get("/voice-pkg/vida/info").json()["voice_list"][0] + assert voice["voice_pkg_url"] == "https://api-roborock.example.com:555/voice-pkg/vida/package/nl.pkg" + + +def test_disabled_mode_turns_the_routes_off(tmp_path, monkeypatch): + client = _client(tmp_path, monkeypatch, extra='\n[voice_packs]\nmode="disabled"\n') + assert client.get("/voice-pkg/vida/info").status_code == 404 + assert client.get("/voice-pkg/vida/package/nl.pkg").status_code == 404 + + +def test_a_failed_refresh_is_not_retried_at_once(tmp_path, monkeypatch): + asyncio.run(_backoff(tmp_path, monkeypatch)) + + +async def _backoff(tmp_path, monkeypatch): + mirror = vp.VoicePackMirror(tmp_path, "eu") + calls = [] + + async def fake(url, **kwargs): + calls.append(url) + if len(calls) == 1: + return json.dumps(INFO).encode() + raise vp.aiohttp.ClientError("offline") + + monkeypatch.setattr(mirror, "_download", fake) + await mirror.get_list("vida") + monkeypatch.setattr(vp, "LIST_MAX_AGE_SECONDS", -1) + for _ in range(3): + assert (await mirror.get_list("vida"))["voice_list"][0]["voice_id"] == 129 + assert len(calls) == 2 + + +def test_only_files_in_the_list_are_fetched_and_refusals_are_remembered(tmp_path, monkeypatch): + asyncio.run(_listed_only(tmp_path, monkeypatch)) + + +async def _listed_only(tmp_path, monkeypatch): + mirror = vp.VoicePackMirror(tmp_path, "eu") + calls = [] + + async def fake(url, **kwargs): + calls.append(url) + if url.endswith("/info"): + return json.dumps(INFO).encode() + raise vp.VoicePackError(404, "gone") + + monkeypatch.setattr(mirror, "_download", fake) + with pytest.raises(vp.VoicePackError): + await mirror.get_file("vida", "pic", "other.png") + assert calls == ["https://eu-app.roborock.com/app/vida/voice-pkg/info"] + for _ in range(3): + with pytest.raises(vp.VoicePackError): + await mirror.get_file("vida", "wav", "nl.wav") + assert calls.count("https://eu-app.roborock.com/app/vida/voice-pkg/wav/nl.wav") == 1 + + +def test_a_republished_package_is_fetched_again(tmp_path, monkeypatch): + asyncio.run(_republished(tmp_path, monkeypatch)) + + +async def _republished(tmp_path, monkeypatch): + mirror = vp.VoicePackMirror(tmp_path, "eu") + info = json.loads(json.dumps(INFO)) + body = {"v": b"old"} + + async def fake(url, **kwargs): + return json.dumps(info).encode() if url.endswith("/info") else body["v"] + + monkeypatch.setattr(mirror, "_download", fake) + assert (await mirror.get_file("vida", "package", "nl.pkg")).read_bytes() == b"old" + info["voice_list"][0]["version"] = 5 + body["v"] = b"new" + monkeypatch.setattr(vp, "LIST_MAX_AGE_SECONDS", -1) + assert (await mirror.get_file("vida", "package", "nl.pkg")).read_bytes() == b"new" + + +def test_the_number_of_mirrored_buckets_is_capped(tmp_path, monkeypatch): + asyncio.run(_capped(tmp_path, monkeypatch)) + + +async def _capped(tmp_path, monkeypatch): + mirror = vp.VoicePackMirror(tmp_path, "eu") + + async def fake(url, **kwargs): + return json.dumps(INFO).encode() + + monkeypatch.setattr(mirror, "_download", fake) + monkeypatch.setattr(vp, "MAX_BUCKETS", 2) + await mirror.get_list("a") + await mirror.get_list("b") + with pytest.raises(vp.VoicePackError) as refused: + await mirror.get_list("c") + assert refused.value.status == 404 + assert (await mirror.get_list("a"))["voice_list"] + + +def test_a_corrupt_stored_list_is_replaced(tmp_path, monkeypatch): + asyncio.run(_corrupt(tmp_path, monkeypatch)) + + +async def _corrupt(tmp_path, monkeypatch): + mirror = vp.VoicePackMirror(tmp_path, "eu") + path = tmp_path / "voice_packs" / "vida" / "info.json" + path.parent.mkdir(parents=True) + path.write_text("{not json") + + async def fake(url, **kwargs): + return json.dumps(INFO).encode() + + monkeypatch.setattr(mirror, "_download", fake) + assert (await mirror.get_list("vida"))["voice_list"][0]["voice_id"] == 129