diff --git a/.env.docker.example b/.env.docker.example index cf35afd..4dbb942 100644 --- a/.env.docker.example +++ b/.env.docker.example @@ -19,8 +19,8 @@ COMPOSE_PROFILES=local-postgres # Docker Swarm stack deployments cannot build from source. Build/tag these locally for a # single-node Swarm, or set them to registry images available to every Swarm node. -APP_IMAGE=ghcr.io/cosmicc/trip-tracker-app:1.5.0 -NGINX_IMAGE=ghcr.io/cosmicc/trip-tracker-nginx:1.5.0 +APP_IMAGE=ghcr.io/cosmicc/trip-tracker-app:1.5.1 +NGINX_IMAGE=ghcr.io/cosmicc/trip-tracker-nginx:1.5.1 # Swarm ttapp-container identity. Match these to the owner of HOST_DATA_DIR on shared storage. APP_UID=1000 diff --git a/.vscode/SKILL-api-and-web-routes.md b/.vscode/SKILL-api-and-web-routes.md index a59a909..d03aa9b 100644 --- a/.vscode/SKILL-api-and-web-routes.md +++ b/.vscode/SKILL-api-and-web-routes.md @@ -100,7 +100,10 @@ use the dedicated session factory so PostgreSQL failures can be translated into Authenticate, decrypt, validate, verify Alembic migrations, and commit the raw payload before returning `200 []`. When PostgreSQL or migrations are unavailable, return `503 Service Unavailable` with `Retry-After: 30` and `Cache-Control: no-store`. Exact HTTP retries must reuse the -existing raw event rather than inserting a duplicate. Do not add a server-side buffer or MQTT path. +existing raw event rather than inserting a duplicate. Return immediately after the commit; never +call or wait for automatic trip generation, odometer calculation, retention cleanup, or the trip +processor lock in the HTTP request. The background `AutomaticTripProcessor` consumes committed rows +on its next checkpointed pass. Do not add a server-side buffer or MQTT path. The `/api/health` endpoint is unauthenticated inside the app for container health checks; `/api/owntracks` requires authentication. diff --git a/.vscode/SKILL-trip-processor.md b/.vscode/SKILL-trip-processor.md index 5416d57..3b554e9 100644 --- a/.vscode/SKILL-trip-processor.md +++ b/.vscode/SKILL-trip-processor.md @@ -15,6 +15,9 @@ OwnTracks ingestion is HTTP-only. The endpoint commits raw messages directly to returns `200 []` only after acceptance. When PostgreSQL or migrations are unavailable, it returns a retryable `503` so the OwnTracks mobile app retains and resends its own queue. Exact HTTP retries must not create duplicate raw rows. There is no server-side buffer, replay worker, or MQTT path. +The HTTP request must not invoke or wait for trip processing or `_PROCESSING_LOCK`; the background +`AutomaticTripProcessor` exclusively performs trip generation, odometer updates, backfills, and +retention after ingestion has returned. --- diff --git a/AGENTS.md b/AGENTS.md index 92438ce..4fffe53 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -97,6 +97,9 @@ backward-compatibility handling. PostgreSQL or migrations are unavailable so OwnTracks retains and retries its own HTTP queue. - Returns `200 []` only after PostgreSQL accepts the payload. Exact HTTP retries reuse the existing raw event instead of inserting it twice. +- Returns immediately after the payload commit without waiting for automatic trip generation, + odometer calculation, retention cleanup, or the trip processor lock. The background + `AutomaticTripProcessor` owns all derived processing on its next pass. **[login_failures.py](trip_tracker/services/login_failures.py)** — Web login audit logging - Stores structured PostgreSQL records for successful and failed web UI login attempts and emits @@ -335,6 +338,8 @@ dates. Manual trips are not restricted by those indexes. - After authentication, decryption, and validation, verify Alembic migrations and attempt the PostgreSQL write. Return `200 []` only after the commit succeeds. Return `503`, `Retry-After: 30`, and `Cache-Control: no-store` when PostgreSQL or migrations are unavailable. +- Do not call or wait for `run_automatic_trip_processing()` from the OwnTracks request. Once raw + storage commits, return promptly and let the background processor consume the checkpointed rows. - The OwnTracks mobile app is the only outage queue. Do not add a server-side SQLite queue, replay worker, MQTT subscriber, or queue storage volume. - Preserve retry idempotency: an exact resent HTTP event must not create a second raw event row. diff --git a/CHANGELOG.md b/CHANGELOG.md index 77b22b1..5f8ccb2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,23 @@ # Changelog -## 1.5.0 - Unreleased +## 1.5.1 - 09.25.2026 + +### Added +- Added regression coverage proving encrypted location, waypoint, duplicate retry, and sequential + backlog requests persist successfully without invoking automatic trip processing in the HTTP + request. + +### Changed +- Changed OwnTracks HTTP ingestion to return `200 []` immediately after PostgreSQL commits the + validated payload, leaving trip generation, odometer updates, backfills, and retention cleanup + to the existing background processor. +- Bumped the Trip Tracker package and deployment image version to 1.5.1. + +### Fixed +- Fixed OwnTracks Android requests timing out while waiting for the automatic trip processor or + its global lock, which caused retried messages to block the phone's queued location updates. + +## 1.5.0 - 07.27.2026 ### Added - Added a focused 1.5.0 upgrade guide covering the coordinated database, PostgreSQL role, Docker diff --git a/INSTALL.md b/INSTALL.md index d55c6b2..821492d 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -339,11 +339,11 @@ Swarm does not build images during `docker stack deploy`, does not support Compo does not preserve the normal Compose loopback-only nginx port binding. The Swarm stack therefore uses image tags and overlay networking. The `Build and publish Swarm images` GitHub workflow publishes the app and nginx images to GHCR with the package version, `latest`, and an immutable -commit-SHA tag. For v1.5.0, use: +commit-SHA tag. For v1.5.1, use: ```bash -APP_IMAGE=ghcr.io/cosmicc/trip-tracker-app:1.5.0 -NGINX_IMAGE=ghcr.io/cosmicc/trip-tracker-nginx:1.5.0 +APP_IMAGE=ghcr.io/cosmicc/trip-tracker-app:1.5.1 +NGINX_IMAGE=ghcr.io/cosmicc/trip-tracker-nginx:1.5.1 ``` If the GHCR packages are private, configure GHCR registry credentials in Portainer or authenticate @@ -356,8 +356,8 @@ needed variables in the shell before deploying. Remote PostgreSQL Swarm deployment: ```bash -export APP_IMAGE=ghcr.io/cosmicc/trip-tracker-app:1.5.0 -export NGINX_IMAGE=ghcr.io/cosmicc/trip-tracker-nginx:1.5.0 +export APP_IMAGE=ghcr.io/cosmicc/trip-tracker-app:1.5.1 +export NGINX_IMAGE=ghcr.io/cosmicc/trip-tracker-nginx:1.5.1 export DATABASE_URL=postgresql+psycopg://triptracker:url_encoded_password@central-db-host:5432/trip_tracker docker stack deploy -c docker-stack.yml trip-tracker ``` @@ -365,8 +365,8 @@ docker stack deploy -c docker-stack.yml trip-tracker Bundled PostgreSQL Swarm deployment: ```bash -export APP_IMAGE=ghcr.io/cosmicc/trip-tracker-app:1.5.0 -export NGINX_IMAGE=ghcr.io/cosmicc/trip-tracker-nginx:1.5.0 +export APP_IMAGE=ghcr.io/cosmicc/trip-tracker-app:1.5.1 +export NGINX_IMAGE=ghcr.io/cosmicc/trip-tracker-nginx:1.5.1 export DATABASE_URL=postgresql+psycopg://triptracker:your-db-password@postgres:5432/trip_tracker docker stack deploy -c docker-stack.yml -c docker-stack.local-postgres.yml trip-tracker ``` diff --git a/README.md b/README.md index f651830..cefdcb5 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@ stores them in PostgreSQL, lets you review and edit generated waypoint work trip monthly mileage and expense PDF logs. Upgrading an existing deployment from 1.4.4 or earlier requires coordinated database, storage, -service, and image-name changes. Follow [UPGRADE-1.5.md](UPGRADE-1.5.md) before deploying 1.5.0. +service, and image-name changes. Follow [UPGRADE-1.5.md](UPGRADE-1.5.md) before deploying 1.5.1. ## Current Scope @@ -16,6 +16,8 @@ service, and image-name changes. Follow [UPGRADE-1.5.md](UPGRADE-1.5.md) before - OwnTracks HTTP endpoint at `/api/owntracks` and Recorder-compatible `/api/pub`. - Retry-safe HTTP ingestion that returns `503 Service Unavailable` while PostgreSQL is unavailable so the OwnTracks mobile app retains and resends its own queued messages after recovery. +- Fast OwnTracks acknowledgements that return `200 []` immediately after PostgreSQL commits the + message; automatic trip generation and odometer processing continue in the background. - OwnTracks waypoint transition model used to turn leave/enter events into work trips, with location updates between those events used as the primary trip distance. - Manual current-odometer entry from the Diagnostics page, with the Manual Odometer card showing @@ -141,8 +143,8 @@ Docker Swarm deployments use [docker-stack.yml](docker-stack.yml) instead of `do Swarm cannot build images, use Compose profiles, or keep the normal Compose loopback-only port binding. The `Build and publish Swarm images` GitHub workflow publishes versioned, `latest`, and commit-SHA app and nginx images to GHCR. Set `APP_IMAGE` to -`ghcr.io/cosmicc/trip-tracker-app:1.5.0` and `NGINX_IMAGE` to -`ghcr.io/cosmicc/trip-tracker-nginx:1.5.0` through Portainer or the shell, and deploy the base +`ghcr.io/cosmicc/trip-tracker-app:1.5.1` and `NGINX_IMAGE` to +`ghcr.io/cosmicc/trip-tracker-nginx:1.5.1` through Portainer or the shell, and deploy the base stack for remote PostgreSQL. Add [docker-stack.local-postgres.yml](docker-stack.local-postgres.yml) only when the bundled PostgreSQL service should be part of the Swarm stack. In Swarm, configure the Cloudflare Tunnel diff --git a/VERSION b/VERSION index bc80560..26ca594 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.5.0 +1.5.1 diff --git a/pyproject.toml b/pyproject.toml index d24bb15..4ad5ea2 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "trip-tracker" -version = "1.5.0" +version = "1.5.1" description = "OwnTracks trip tracker with PostgreSQL storage and monthly mileage and expense PDFs." readme = "README.md" requires-python = ">=3.12" diff --git a/tests/test_api_auth.py b/tests/test_api_auth.py index 3069f10..b2e3843 100644 --- a/tests/test_api_auth.py +++ b/tests/test_api_auth.py @@ -16,7 +16,7 @@ from trip_tracker.app import app from trip_tracker.config import Settings from trip_tracker.database import get_db -from trip_tracker.models import Base, OwnTracksLocation +from trip_tracker.models import Base, OwnTracksLocation, Site def _test_client_session() -> tuple[TestClient, sessionmaker[Session]]: @@ -72,6 +72,12 @@ def _encrypted_owntracks_payload(payload: dict, secret: str) -> dict: } +def _fail_if_trip_processing_runs(*args, **kwargs) -> None: + """Fail when the HTTP ingestion path invokes derived trip processing.""" + + raise AssertionError("HTTP ingestion must not invoke automatic trip processing") + + def test_owntracks_endpoint_requires_basic_auth_and_encrypted_payload(monkeypatch) -> None: settings = Settings( database_url="sqlite://", @@ -154,6 +160,11 @@ def test_owntracks_endpoint_uses_dedicated_database_session(monkeypatch) -> None automatic_trip_processing_enabled=False, ) _patch_settings(monkeypatch, settings) + monkeypatch.setattr( + "trip_tracker.services.owntracks._run_trip_processing", + _fail_if_trip_processing_runs, + raising=False, + ) client, session_factory = _test_client_session() payload = { "_type": "location", @@ -172,6 +183,7 @@ def test_owntracks_endpoint_uses_dedicated_database_session(monkeypatch) -> None ) assert response.status_code == 200 + assert response.json() == [] assert response.headers["Cache-Control"] == "no-store" assert "X-Trip-Tracker-OwnTracks-Buffered" not in response.headers with session_factory() as db: @@ -180,6 +192,49 @@ def test_owntracks_endpoint_uses_dedicated_database_session(monkeypatch) -> None app.dependency_overrides.clear() +def test_owntracks_waypoint_endpoint_persists_without_trip_processing(monkeypatch) -> None: + settings = Settings( + database_url="sqlite://", + owntracks_username="owntracks", + owntracks_password="owntracks-password", + owntracks_encryption_key="owntracks-secret", + automatic_trip_processing_enabled=False, + ) + _patch_settings(monkeypatch, settings) + monkeypatch.setattr( + "trip_tracker.services.owntracks._run_trip_processing", + _fail_if_trip_processing_runs, + raising=False, + ) + client, session_factory = _test_client_session() + payload = { + "_type": "waypoint", + "desc": "Client Warehouse", + "lat": 42.3314, + "lon": -83.0458, + "rad": 75, + "rid": "warehouse-1", + "tst": int(datetime(2026, 6, 30, 12, 0, tzinfo=UTC).timestamp()), + } + encrypted_payload = _encrypted_owntracks_payload(payload, settings.owntracks_encryption_key) + try: + response = client.post( + "/api/owntracks", + json=encrypted_payload, + auth=("owntracks", "owntracks-password"), + ) + + assert response.status_code == 200 + assert response.json() == [] + with session_factory() as db: + site = db.scalar(select(Site).where(Site.owntracks_region_id == "warehouse-1")) + assert site is not None + assert site.name == "Client Warehouse" + assert site.radius_m == 75 + finally: + app.dependency_overrides.clear() + + def test_owntracks_endpoint_returns_retryable_503_when_database_is_offline(monkeypatch) -> None: settings = Settings( database_url="sqlite://", @@ -303,6 +358,51 @@ def test_owntracks_exact_http_retry_is_not_stored_twice(monkeypatch) -> None: app.dependency_overrides.clear() +def test_owntracks_sequential_backlog_persists_without_inline_processing(monkeypatch) -> None: + settings = Settings( + database_url="sqlite://", + owntracks_username="owntracks", + owntracks_password="owntracks-password", + owntracks_encryption_key="owntracks-secret", + automatic_trip_processing_enabled=False, + ) + _patch_settings(monkeypatch, settings) + monkeypatch.setattr( + "trip_tracker.services.owntracks._run_trip_processing", + _fail_if_trip_processing_runs, + raising=False, + ) + client, session_factory = _test_client_session() + first_timestamp = int(datetime(2026, 6, 30, 12, 0, tzinfo=UTC).timestamp()) + try: + for offset in range(50): + encrypted_payload = _encrypted_owntracks_payload( + { + "_type": "location", + "lat": 42.3314 + (offset / 100_000), + "lon": -83.0458, + "tst": first_timestamp + offset, + "tid": "IP", + "topic": "owntracks/ian/phone", + }, + settings.owntracks_encryption_key, + ) + + response = client.post( + "/api/owntracks", + json=encrypted_payload, + auth=("owntracks", "owntracks-password"), + ) + + assert response.status_code == 200 + assert response.json() == [] + + with session_factory() as db: + assert db.scalar(select(func.count(OwnTracksLocation.id))) == 50 + finally: + app.dependency_overrides.clear() + + def test_non_owntracks_api_requires_separate_bearer_key(monkeypatch) -> None: settings = Settings(database_url="sqlite://", web_api_key="web-api-secret") _patch_settings(monkeypatch, settings) diff --git a/tests/test_docker_compose.py b/tests/test_docker_compose.py index e47d7f9..49bf631 100644 --- a/tests/test_docker_compose.py +++ b/tests/test_docker_compose.py @@ -219,8 +219,8 @@ def test_swarm_stack_has_optional_local_postgres_overlay() -> None: ) assert "POSTGRES_DATA_VOLUME=trip-tracker-postgres-data" in env_text assert "trip-tracker-internal" in local_postgres_text - assert "APP_IMAGE=ghcr.io/cosmicc/trip-tracker-app:1.5.0" in env_text - assert "NGINX_IMAGE=ghcr.io/cosmicc/trip-tracker-nginx:1.5.0" in env_text + assert "APP_IMAGE=ghcr.io/cosmicc/trip-tracker-app:1.5.1" in env_text + assert "NGINX_IMAGE=ghcr.io/cosmicc/trip-tracker-nginx:1.5.1" in env_text assert "APP_UID=1000" in env_text assert "APP_GID=100" in env_text diff --git a/tests/test_owntracks.py b/tests/test_owntracks.py index e476556..b86cf08 100644 --- a/tests/test_owntracks.py +++ b/tests/test_owntracks.py @@ -7,6 +7,7 @@ from trip_tracker.models import Base, OwnTracksLocation, Site, Trip from trip_tracker.services.owntracks import parse_owntracks_location, process_owntracks_payload +from trip_tracker.services.trip_processor import run_automatic_trip_processing def _session() -> Session: @@ -84,7 +85,7 @@ def test_process_owntracks_location_with_region_does_not_create_waypoint() -> No assert before_receive <= received_at <= after_receive -def test_process_owntracks_payload_automatically_creates_trip() -> None: +def test_process_owntracks_payload_defers_trip_creation_to_background_processor() -> None: db = _session() day = datetime(2030, 1, 1, 13, 0, tzinfo=UTC) db.add_all( @@ -135,6 +136,10 @@ def test_process_owntracks_payload_automatically_creates_trip() -> None: ).encode("utf-8"), ) + assert list(db.scalars(select(Trip))) == [] + + run_automatic_trip_processing(db, now=day + timedelta(minutes=40)) + trips = list(db.scalars(select(Trip).order_by(Trip.started_at.asc()))) client_a = db.scalar(select(Site).where(Site.name == "Client A")) client_b = db.scalar(select(Site).where(Site.name == "Client B")) diff --git a/tests/test_web.py b/tests/test_web.py index a932907..774ee38 100644 --- a/tests/test_web.py +++ b/tests/test_web.py @@ -1364,7 +1364,7 @@ def test_install_assets_stay_available_when_web_login_is_enabled(monkeypatch) -> assert manifest["scope"] == "/" assert manifest_response.headers["cache-control"] == "no-store" assert {icon["purpose"] for icon in manifest["icons"]} == {"any", "maskable"} - assert "/static/icons/trip-tracker-icon-512.png?v=1.5.0-brand1" in { + assert "/static/icons/trip-tracker-icon-512.png?v=1.5.1-brand1" in { icon["src"] for icon in manifest["icons"] } diff --git a/trip_tracker/__init__.py b/trip_tracker/__init__.py index ea9cc93..886fbf4 100644 --- a/trip_tracker/__init__.py +++ b/trip_tracker/__init__.py @@ -1,3 +1,3 @@ __all__ = ["__version__"] -__version__ = "1.5.0" +__version__ = "1.5.1" diff --git a/trip_tracker/services/owntracks.py b/trip_tracker/services/owntracks.py index fc44162..5641db6 100644 --- a/trip_tracker/services/owntracks.py +++ b/trip_tracker/services/owntracks.py @@ -3,7 +3,7 @@ import json import logging from dataclasses import dataclass -from datetime import UTC, date, datetime +from datetime import UTC, datetime from decimal import Decimal from nacl.exceptions import CryptoError @@ -13,8 +13,6 @@ from trip_tracker.config import Settings, get_settings from trip_tracker.models import OwnTracksLocation, Site -from trip_tracker.services.timezone import datetime_to_local_date -from trip_tracker.services.trip_processor import run_automatic_trip_processing logger = logging.getLogger(__name__) @@ -263,23 +261,6 @@ def _site_radius_from_payload(payload: dict) -> int: return settings.owntracks_default_site_radius_m -def _payload_date(payload: dict) -> date: - return datetime_to_local_date(_payload_datetime(payload)) - - -def _run_trip_processing(db: Session, payload: dict) -> None: - touched_date = _payload_date(payload) - finalize_completed_days = touched_date >= datetime_to_local_date(datetime.now(UTC)) - try: - run_automatic_trip_processing( - db, - touched_date=touched_date, - finalize_completed_days=finalize_completed_days, - ) - except Exception: - logger.exception("Automatic trip processing failed after OwnTracks payload") - - def sync_site_from_owntracks_payload( db: Session, payload: dict, @@ -466,7 +447,12 @@ def process_owntracks_payload( user: str | None = None, device: str | None = None, ) -> OwnTracksProcessResult: - """Persist one validated HTTP payload and run best-effort trip processing.""" + """Persist one validated HTTP payload without waiting for derived trip processing. + + Automatic trip generation, odometer updates, and retention are owned by the + background ``AutomaticTripProcessor``. Keeping those operations outside this + function lets the HTTP route acknowledge a committed OwnTracks message promptly. + """ payload = _decode_payload(body) payload_type = payload.get("_type") @@ -477,10 +463,8 @@ def process_owntracks_payload( db.commit() if site is not None: db.refresh(site) - _run_trip_processing(db, payload) return OwnTracksProcessResult(location=None, site=site) message = _location_message_from_payload(payload, topic=topic, user=user, device=device) location = store_owntracks_location(db, message) - _run_trip_processing(db, payload) return OwnTracksProcessResult(location=location, site=None) diff --git a/trip_tracker/web/static/manifest.webmanifest b/trip_tracker/web/static/manifest.webmanifest index efda57c..e293b94 100644 --- a/trip_tracker/web/static/manifest.webmanifest +++ b/trip_tracker/web/static/manifest.webmanifest @@ -21,25 +21,25 @@ ], "icons": [ { - "src": "/static/icons/trip-tracker-icon-192.png?v=1.5.0-brand1", + "src": "/static/icons/trip-tracker-icon-192.png?v=1.5.1-brand1", "sizes": "192x192", "type": "image/png", "purpose": "any" }, { - "src": "/static/icons/trip-tracker-icon-512.png?v=1.5.0-brand1", + "src": "/static/icons/trip-tracker-icon-512.png?v=1.5.1-brand1", "sizes": "512x512", "type": "image/png", "purpose": "any" }, { - "src": "/static/icons/trip-tracker-maskable-192.png?v=1.5.0-brand1", + "src": "/static/icons/trip-tracker-maskable-192.png?v=1.5.1-brand1", "sizes": "192x192", "type": "image/png", "purpose": "maskable" }, { - "src": "/static/icons/trip-tracker-maskable-512.png?v=1.5.0-brand1", + "src": "/static/icons/trip-tracker-maskable-512.png?v=1.5.1-brand1", "sizes": "512x512", "type": "image/png", "purpose": "maskable" @@ -52,7 +52,7 @@ "url": "/trips", "icons": [ { - "src": "/static/icons/trip-tracker-icon-192.png?v=1.5.0-brand1", + "src": "/static/icons/trip-tracker-icon-192.png?v=1.5.1-brand1", "sizes": "192x192", "type": "image/png" } @@ -64,7 +64,7 @@ "url": "/waypoints", "icons": [ { - "src": "/static/icons/trip-tracker-icon-192.png?v=1.5.0-brand1", + "src": "/static/icons/trip-tracker-icon-192.png?v=1.5.1-brand1", "sizes": "192x192", "type": "image/png" } diff --git a/trip_tracker/web/templates/layout.html b/trip_tracker/web/templates/layout.html index bac2f1b..80249ab 100644 --- a/trip_tracker/web/templates/layout.html +++ b/trip_tracker/web/templates/layout.html @@ -15,7 +15,7 @@ {% if not hide_app_chrome %} {% if not public_device_session %}{% endif %} - + {% endif %}