diff --git a/docs/cookie-import.md b/docs/cookie-import.md index ed17d0f..ce19e6a 100644 --- a/docs/cookie-import.md +++ b/docs/cookie-import.md @@ -2,7 +2,19 @@ ArcHive stores cookies encrypted per site for browse and yt-dlp downloads. -## Recommended: Cookie-Editor extension +## Recommended: Log in with browser (desktop + Android) + +1. In **Settings → Cookies**, select the site (or open a cookie-required site in Browse). +2. Press **Log in with browser**. ArcHive opens the site in its own browser: + - **Desktop:** a separate window. Log in, then press **Save cookies** in ArcHive (or just close the window). + - **Android:** a full-screen browser with **Cancel** / **Save cookies** buttons at the top. The system back button navigates back inside the page. +3. The captured cookies, including HttpOnly session cookies that page scripts cannot read, are converted to Netscape format and stored in the vault (on the LAN host when the app is in Remote LAN mode). + +Android does not let one app read another app's cookies (for example Chrome's), so the login has to happen inside ArcHive. Google sign-in may refuse embedded browsers; for YouTube, import cookies with Cookie-Editor instead. + +Implementation: `src-tauri/src/cookie_login.rs` (desktop window + Android bridge) and `src-tauri/android-overlays/CookieBrowserPlugin.kt` (installed by `scripts/patch-android-ytdlp.ps1` / `.sh`). + +## Cookie-Editor extension 1. Install [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) (Chrome, Edge, Brave) or the Firefox equivalent. 2. Log in on the target site (PornHub, xHamster, etc.). @@ -31,6 +43,8 @@ Export or build a Netscape cookie file and paste into the Netscape textarea → | xvideos | xvideos.com | | thisvid | thisvid.com | +Lines exported as `#HttpOnly_` are treated as cookies, not comments. + ## Security - Cookies are encrypted at rest (AES-256-GCM). diff --git a/docs/mobile-android.md b/docs/mobile-android.md index 85182bd..aa07265 100644 --- a/docs/mobile-android.md +++ b/docs/mobile-android.md @@ -177,20 +177,21 @@ Video playback uses `GET /api/scenes/{id}/media` and `GET /api/files/stream` wit ## Troubleshooting -| Issue | Fix | -| -------------------------------------- | ---------------------------------------------------------------------------------------- | -| `FrameInsert open fail` in logcat | Usually MIUI noise — see [troubleshooting-android.md](troubleshooting-android.md) | -| `om.archhive.app` in logs | Truncated `com.archhive.app` — verify with `android:regen` if UI is broken | -| -------------------------------------- | ---------------------------------------------------------------------------------------- | -| `yt-dlp-aarch64-linux-android` missing | Sidecars are desktop-only; run `bun run android:regen` | -| `Unresolved reference: TauriActivity` | Stale `gen/android` after identifier change; run `bun run android:regen` | -| App runs but `invoke` is undefined | Set Engine → **Remote LAN**; ensure desktop LAN is on. Rebuild after capability changes. | -| Read-only filesystem on Android | Mobile uses app data dir for DB; downloads go via Remote LAN desktop host | -| `ERR_CLEARTEXT_NOT_PERMITTED` | Run `scripts/patch-android-lan.ps1` and rebuild | -| Connection refused | Firewall: allow TCP **8787** on PC (API). `:1420` is dev UI only. | -| Used `:1420` in Remote LAN | Remote LAN API is port **8787**, not the Vite dev port 1420 | -| 401 Unauthorized | Verify API token matches desktop LAN token | -| Empty browse results | Ensure desktop app is running and cookies are configured | +| Issue | Fix | +| ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `FrameInsert open fail` in logcat | Usually MIUI noise — see [troubleshooting-android.md](troubleshooting-android.md) | +| `om.archhive.app` in logs | Truncated `com.archhive.app` — verify with `android:regen` if UI is broken | +| -------------------------------------- | ---------------------------------------------------------------------------------------- | +| `yt-dlp-aarch64-linux-android` missing | Sidecars are desktop-only; run `bun run android:regen` | +| `Unresolved reference: TauriActivity` | Stale `gen/android` after identifier change; run `bun run android:regen` | +| App runs but `invoke` is undefined | Set Engine → **Remote LAN**; ensure desktop LAN is on. Rebuild after capability changes. | +| Read-only filesystem on Android | Mobile uses app data dir for DB; downloads go via Remote LAN desktop host | +| `ERR_CLEARTEXT_NOT_PERMITTED` | Run `scripts/patch-android-lan.ps1` and rebuild | +| Connection refused | Firewall: allow TCP **8787** on PC (API). `:1420` is dev UI only. | +| Used `:1420` in Remote LAN | Remote LAN API is port **8787**, not the Vite dev port 1420 | +| 401 Unauthorized | Verify API token matches desktop LAN token | +| Empty browse results | Ensure desktop app is running and cookies are configured | +| `YoutubeDL/FFmpeg not initialized`, no versions in Settings | APK was packaged with compressed native libs (`extractNativeLibs=false`). `patch-android-ytdlp` sets `jniLibs.useLegacyPackaging = true`; run `bun run android:patches` and rebuild | ## Scripts diff --git a/docs/sites.md b/docs/sites.md index c39dd2d..1efca5d 100644 --- a/docs/sites.md +++ b/docs/sites.md @@ -2,21 +2,90 @@ ## Tier A — yt-dlp (browse + download) -| Site ID | Display Name | Browse kinds | Cookies | -| --------- | ------------ | -------------------------------------------- | --------------------------- | -| youtube | YouTube | channel, search, video | Optional | -| tiktok | TikTok | channel, search, video | No | -| twitter | Twitter / X | channel, video | Recommended | -| instagram | Instagram | channel, video | Yes | -| thisvid | ThisVid | tag, search, video | Yes | -| thothub | ThotHub | tag, model, search, video | Optional (`thethothub.com`) | -| pornhub | PornHub | category, tag, search, channel, model, video | Yes | -| youporn | YouPorn | category, model, search, video | Optional | -| xnxx | XNXX | search, tag, channel, video | Optional | -| xhamster | xHamster | tag, search, channel, video | Yes | -| xvideos | XVIDEOS | tag, search, channel, video | Yes | -| reddit | Reddit | channel, search, video | OAuth for subreddits | -| eporner | Eporner | search, tag, category, video | No | +| Site ID | Display Name | Browse kinds | Cookies | +| --------- | ------------ | -------------------------------------------- | -------------------------------------------------------------- | +| youtube | YouTube | channel, search, video | Optional | +| tiktok | TikTok | channel, search, video | No | +| twitter | Twitter / X | channel, video | Recommended | +| instagram | Instagram | channel, video | Yes | +| thisvid | ThisVid | tag, search, video | Yes | +| thothub | ThotHub | tag, model, search, video | Optional (`thethothub.com`) | +| pornhub | PornHub | category, tag, search, channel, model, video | Yes | +| youporn | YouPorn | category, model, search, video | Optional | +| xnxx | XNXX | search, tag, channel, video | Optional | +| xhamster | xHamster | tag, search, channel, video | Yes | +| xvideos | XVIDEOS | tag, search, channel, video | Yes | +| reddit | Reddit | channel, search, video | Login cookies (Reddit hides listings from logged-out visitors) | +| eporner | Eporner | search, tag, category, video | No | + +## Aggregators + +| Site ID | Display Name | Browse kinds | Cookies | +| --------- | ------------ | ----------------------- | ------- | +| xpomelo | xPomelo | search, category, model | No | +| pornwitch | Pornwitch | category, search | No | + +xPomelo hosts nothing: each result links to its source tube (PornHub, +XVIDEOS, xHamster, XNXX, Eporner, SpankBang, …). Items keep the source URL +and, where ArcHive has an adapter, the source site's id, so playback, +downloads and the Info dialog run through that site's own pipeline; other +sources go through plain yt-dlp. Category listings take any number of +required and excluded categories (`/category/a%2Cb%2C-c`). Results from +paysites (Brazzers, FapHouse, Clips4Sale, …) and unsupported sources (Beeg, +SxyPrn, VipTube) are hidden unless the Source filter is set. +Cloudflare blocks HTTP/1.1 clients on this site; the main HTTP/2 client passes. + +Pornwitch also hosts nothing. Its search matches titles plus the source +site's tags, and every word is required (synonyms and plurals count), so a +tag combination `a,b,-c` runs as the query `a b -c`; a Search keeps the +site's own syntax (`"exact phrase"`, `-word`). PornHub, XVIDEOS, xHamster and +Eporner results get their source URL (and adapter) straight from the card; +TXXX, beeg and paysite results keep Pornwitch's `/go?id=` link, which is +followed (one request) only when played or downloaded. Pages after the first +continue from the previous page's last-card cursor. Word suggestions come +from `/suggest` (`GET /api/sites/{id}/suggest?q=`), debounced and cached; +requests to pornwitch.com are spaced 2.5–4.5 s apart. + +### Bot checks (Cloudflare) + +#### Desktop + +When yt-dlp is refused by a bot check, ArcHive retries it with browser +impersonation (`--impersonate`, `sites/clearance.rs`). This needs the +standalone yt-dlp builds that bundle curl_cffi (`yt-dlp.exe`, `yt-dlp_macos`, +`yt-dlp_linux`); `setup-binaries` downloads those. If the site still refuses, +a small verification window opens on the page and closes by itself once the +check passes. Its cookies (`cf_clearance`, session) are saved encrypted per +site and reused, together with the same User-Agent, by yt-dlp and by the KVS +resolver. + +#### Android + +Some sources (SpankBang, KVS tubes such as CrazyPorn, Love4Porn, 4KPorn) +answer the embedded engines with a Cloudflare challenge. When playback or a +download fails that way on Android, ArcHive opens the page in its in-app +browser, where the check passes by itself or after one tap. It then reads +the video URL from the rendered page (`sites/challenge.rs`) and follows +session-bound KVS `get_file` links inside the same browser session. Sites +with dedicated adapters (PornHub, XVIDEOS, …) are excluded. + +### xPomelo sources: tested 2026-10-13 + +From a datacenter connection. Cloudflare-protected sites then show an +interactive check that can't be passed from there; on a home connection the +bot-check fallbacks below are expected to get through. That part is +unverified. + +| Result | Sources | +| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Play + download | XVIDEOS, xHamster, XNXX, Eporner, YouPorn, RedTube, PornZog, XGroovy, PornOne, Hoes, HDtube¹, SexVid¹, XXXBP², TXXX, EroThots, PornHat, YouJizz, Analdin, AnySex, Xozilla, Nuvid, Clips4Sale (trailers) | +| Cloudflare check (bot-check fallback) | SpankBang, CrazyPorn, Love4Porn, 4KPorn, FapCat, TNAFlix, DrTuber, Noodlemagazine (intermittent) | +| Native resolver, CDN unreliable | HQPorner (`mydaddy.cc` player; files often 404 or rate-limited) | +| Not supported | Beeg (API no longer returns streams; yt-dlp broken too), SxyPrn (yt-dlp marks it unsupported), VipTube (empty player config) | +| Stale listings | Some xPomelo results point at removed videos (HTTP 404/410, Tube8 `video-inactive`) | + +¹ yt-dlp can't extract these; playback and downloads use the KVS resolver. +² No yt-dlp extractor; `api.xxxbp.tv/hls`. ## Tier B — Live Cam Sites (scraping + webview bridge) @@ -27,9 +96,9 @@ ## Tier C — gallery-dl -| Site ID | Display Name | Notes | -| ------- | ------------ | ---------------------------- | -| redgifs | RedGifs | Uses gallery-dl for download | +| Site ID | Display Name | Notes | +| ------- | ------------ | -------------------------------------------------------------------- | +| redgifs | RedGifs | Browse via the RedGifs v2 API (temporary token); direct MP4 download | ## Cookie Requirements @@ -79,6 +148,35 @@ desktop host. the StashDB id, merges aliases (deduped, capped), and sets the portrait only when none is stored. +## Sorting, filters and categories + +What each site can strictly combine (tags, categories, search + category, +exclusions) is documented in [tag-combining.md](tag-combining.md). + +Adapters advertise what they support via `SiteAdapter::browse_options()` +(returned in `SiteInfo.browse_options`); the browse page shows matching +Sort / Date / Length / Quality selects and passes the chosen ids as +`BrowseQuery.filters`. Unknown ids are dropped before they reach a URL. +Every page also has on-device filters (words, `-exclude`, length) over +loaded results. Category / tag indexes come from `list_categories()` +(`list_site_categories` command, `GET /api/sites/{id}/categories`). + +| Site | Sort | Date / Length / Quality | Categories | +| ---------- | --------------------------------------- | -------------------------------------- | --------------------- | +| PornHub | newest, views, rating, longest, hottest | period, duration, HD | JSON-embedded catalog | +| XVIDEOS | relevance, date, rating, views, length | period, duration, HD/1080p | `/tags` (top 400) | +| XNXX | views, random | period, duration, HD/Full HD | `/tags` (top 400) | +| xHamster | search: 5 sorts; category: newest/best | duration, 720p/1080p/4K | `/categories` | +| YouPorn | relevance, date, views, rating, length | — | `/categories/` | +| Eporner | 7 API `order` values | — | `/cats/` | +| xPomelo | relevance, date, views, length | period, duration, HD/4K, source | `data-tags` JSON (6k) | +| Pornwitch | hottest, newest, oldest, longest, short | period, minutes range, 720p–4K, source | `/categories` (180) | +| ThotHub | KVS `sort_by` (5) | — | — | +| ThisVid | KVS `sort_by` (5) | — | `/categories/` | +| Reddit | hot, new, top, rising | period (top) | — | +| RedGifs | trending, latest, top | — | — | +| Chaturbate | — | — | top hashtags API | + ## SCrawler Parity Roadmap Future adapters: Threads, Pinterest, OnlyFans (cookie + DRM limitations), Bluesky, LPSG. diff --git a/docs/tag-combining.md b/docs/tag-combining.md new file mode 100644 index 0000000..f20857d --- /dev/null +++ b/docs/tag-combining.md @@ -0,0 +1,203 @@ +# Combining tags, categories and searches + +What each site can **strictly** combine in one listing, i.e. results that must +match _every_ term, versus what only works as a relevance-ranked word search. +Everything marked strict was verified against the live sites (October 2026) +by opening result videos and reading **their own** tag / category lists (not +titles, and not the site-wide menus on video pages). URL schemes change, so +re-check before relying on an entry for automation. + +**Strict** = the site intersects the terms (each result has all of them). +**Word search** = the terms are keywords; results are ranked by relevance and +are not guaranteed to match every term. + +## Summary + +| Site | Tag + tag | Category + category | Search + category | Exclude | +| -------- | --------------------------- | --------------------- | ------------------------------------ | ---------------------- | +| XVIDEOS | ❌ (keyword search) | ❌ | ❌ | ❌ (`-tag` is ignored) | +| PornHub | ❌ (word search) | ✅ strict, exactly 2 | ✅ strict, 1 category | ✅ up to 10 categories | +| Eporner | ❌ (word search) | ✅ strict, any number | ✅ strict (search/tag + categories) | ✅ categories | +| xHamster | ❌ (word search) | ❌ | ❌ | ❌ | +| XNXX | ❌ (word search) | — (no categories) | ❌ | ❌ | +| ThisVid | ❌ (one tag only) | ❌ | ❌ (search + category is not strict) | ❌ | +| YouPorn | ❌ (word search) | ❌ | ❌ | ❌ | +| Others | not supported (word search) | not supported | not supported | not supported | + +ThisVid's one strict combination — **one category + one tag** — does not fit +the columns above; see its section. + +On every site, the in-app **Filter results** box (`word`, `-word`) can further +narrow the _loaded_ results on the device; that is a local text filter on +titles/tags, not a site query. + +## XVIDEOS + +- **No strict tag matching — not even for a single multi-word tag.** Tag pages + are a keyword search over titles and tags: + - `/tags/petite-brunette` lists **73,466** videos — more than `petite` + (46,462) or `brunette` (50,095) alone, which a real tag could not. Of 8 + results opened, **none** carried a `petite-brunette` tag, only 2 had both + `petite` and `brunette` tags, and some matched only through the title. + - `/tags/milf+anal` and `/tags/redhead+big-tits+milf` behave the same: results + were missing one or more of the tags (none of the 5 checked had all three). + - `/tags/milf/anal` just redirects to the `anal` tag; `-tag` exclusion + (`/tags/milf+anal+-lesbian`) is ignored. + - The counts confirm it. Results-page counts: `milf` 48,587, `anal` 59,259, + `milf+anal` **40,951** — that would make 84% of all MILF videos also anal; + a real intersection would be a small fraction of either. `petite-brunette` + (= `petite+brunette`) shows 26,078 against `petite` 32,679 (80%). The + tag-index badges are worse: `milf-anal` 122,495 is larger than `milf` + (73,089) and `anal` (95,337) put together. The search box is OR-like: + `/?k=milf` 82,413 → `/?k=milf+anal` 135,138. + - Single-word tags (`/tags/milf`) are the closest to a real tag listing. +- **Categories** (`/c/Femdom-235`, ~38 of them) cannot be combined with each + other, with tags, or with a search: `?c=`, `?tag=` and `?k=` on a category + page are ignored, `/c/X/tags/Y` is a 404. Category pages do support sort / + length / quality as path segments: `/c/s:uploaddate/d:10-20min/q:hd/Femdom-235`. +- **Search + tag**: not possible (`/?k=…&tag=…` and `/tags/x?k=…` ignore the + extra parameter). + +**In the app:** Combine runs a word search (`/?k=milf+anal`); results are +relevance-ranked, not strict. Use the **Filter results** box to drop loaded +results by word. + +## PornHub + +- **Categories — strict, exactly 2 (straight, gay and trans).** + `/video/incategories/milf/anal` (paging `?page=N`, sort `&o=mv` etc. work). + Verified: 10 of 10 results (from the page's `#incategoryVideos` grid) list + both MILF and Anal in their own categories. + - Three categories (`/incategories/a/b/c`) → 404. + - Gay: `/gay/video/incategories/amateur/bareback` (catalog slugs without + their `-gay` suffix). Verified: 8 of 8 results list both categories. + - Trans has no category index of its own (`/transgender/categories` is a + 404); a trans category is `/video/incategories//transgender`. + Verified: 8 of 8 `milf/transgender` results list both MILF and + Transgender. Most of the 111 straight categories have such a page; a few + pairs (BBW, Casting, Bisexual Male, Scissoring, Described Video) are 404s, + and the app falls back to a word search for those. + - Lesbian "categories" are not categories: every entry on + `/lesbian/categories` links to a search (`/lesbian/video/search?search=anal`), + so lesbian combinations are word searches. +- **Search + 1 category, minus up to 10 categories — strict.** The search + page's own filter (the category include / exclude lists) uses numeric + category ids: + `/video/search?search=redhead&filter_category=29&exclude_category=35-8` + (29 MILF, 35 Anal, 8 Big Tits). + - Verified: 8 of 8 results list MILF and neither Anal nor Big Tits in their + own categories; the same holds on page 2 with a sort (`&o=mv&page=2`). + - Several exclusions are **dash**-separated (`35-8`), at most 10. A comma + (`35,8`) keeps only the first one. + - Only **one** `filter_category`. + - `?search=` on a category page is a 404. +- **Category page minus categories — strict.** `exclude_category` also works + on category listings, `/video?c=29&exclude_category=8` (8 of 8 MILF results + without Big Tits, where the unfiltered page is mostly Big Tits), and on + `/video/incategories/milf/anal?exclude_category=8`. `filter_category` is + ignored on category pages, and `incategories` only takes slugs (ids 404). +- **Tags**: no tag-intersection URL; multiple tags are a word search. + +**In the app:** + +- Two categories: PornHub category picker → **Combine** → pick 2 (Straight or + Gay) → _Browse 2 together_. On Lesbian and Trans, Combine runs a word search. +- The picker loads PornHub's live catalog (111 straight, 53 gay, 73 lesbian + entries). The **Trans** tab lists the straight categories, each opened as + that category ∩ Transgender; _All trans_ is the whole trans section. +- Search + categories: run a search, open the **Categories** panel under the + filters, tap a category once to require it (one at most), and tap others + twice to exclude them (up to 10). +- On a category page the same panel only excludes. + +## Eporner + +- **Search or tag + categories — strict, include and exclude.** + `/tag/redhead/?include=milf,anal&exclude=lesbian` (also on category pages, + `/cat/japanese/?include=milf`). + - Paging and sort: the page number goes **before** the sort segment — + `/tag/redhead/2/longest/?include=milf`. (`/longest/2/` redirects to page 1.) + - Verified: 8 of 8 `?include=milf` results and 5 of 5 + `?include=milf&exclude=anal` results had the right categories on their + own pages. + - Only **categories** (the ~86 on `/cats/`) work in `include`/`exclude`; + tag slugs (`femdom`, `pantyhose`) are silently dropped. +- **Tag + tag**: not strict. Every multi-tag form (`/tag/redhead+milf/`, + `/search/redhead milf/`, `/tag/redhead-milf/`) lands on a keyword-search page + (`/tag/redhead-milf/`). The search narrows with each word (API totals: + `redhead` 100k+, `redhead milf` 52,597, `redhead milf anal` 16,993), so it + behaves AND-like, but it is keyword matching, not tag intersection. +- Tag slugs are case-sensitive: `/tag/czech-ouQ9EM84/` works, lowercase 404s. + +**For a strict Eporner match:** search or open a tag, then use the +**Categories** panel on the results page — tap once to require a category, +twice to exclude it. Required categories are strict; the search term itself is +a keyword match. + +## xHamster + +- **No strict combination of any kind.** + - `/categories/milf/anal` → 404; there is no multi-tag or multi-category URL. + - Multiple terms only work as a word search: `/search/redhead+milf` + (the form xHamster's own "related searches" use). + - A search can only be narrowed by orientation (`?orientations=gay`), plus + sort/length/quality. +- Single tags (`/tags/`) and categories (`/categories/`) are real + listings with sort (`/newest`, `/best`), HD (`/hd`) and paging (`/2`). + +**In the app:** Combine runs a word search; results are relevance-ranked. + +## XNXX + +- **No strict combination.** XNXX has no tag pages: `/tags/milf` redirects to + the search `/search/milf`, and `/tags/milf/anal` redirects to `/search/anal`. +- `/search/big-tits` (hyphenated, as XNXX's own tag links use) ranks + differently from `/search/big+tits` (separate words); neither is a strict + match. The app uses the hyphenated form for a single tag. + +**In the app:** Combine runs a word search. + +## ThisVid + +- **Category + one tag — strict.** `/categories/amateurs/?tag=anal` (or the + other way round, `/tags/anal/?category=amateurs`) returns only videos in that + category **and** with that tag. Checked by opening result pages: every one + listed both (amateurs + anal, asian + massage). Paging works + (`/categories/amateurs/2/?tag=anal`). +- **Only one tag.** `?tag=anal&tag=bdsm` keeps just the last tag; comma or `+` + lists (`?tag=anal,bdsm`) are 404s; `?tag[]=…` is a server error. +- **Tag + tag**: no. `/tags/anal/?tag=bdsm` _replaces_ the tag (results are + bdsm, not anal); `/tags/a,b/`, `/tags/a+b/`, `/tags/a/b/` are 404s. +- **Search + category**: not strict. `/search/?q=wife&category=amateurs` + reorders the results but still returns other categories. + +**In the app:** open a ThisVid category, type a tag in **Narrow by tag** +(autocomplete from ThisVid's A–Z tag index) and tap Apply — a strict +category + tag listing, with sort and paging. Combining several ThisVid tags +in the tag browser still runs a word search. + +## YouPorn + +- **No strict combination.** `/category/milf/anal/` is a 403; `category_id=` and + `tag=` on search, tag or category pages are stripped by a redirect. +- Its "porn tags" are search aliases: `/porntags/redhead-milf/` returns exactly + the same videos as `/search/?query=redhead+milf`, and `/porntags/redhead/` + the same as `/search/?query=redhead`. + +**In the app:** Combine runs a word search. + +## Other sites + +ThotHub, RedGifs, Reddit and the cam sites were **not tested** for combined +queries. In the app, combining tags on them runs a word search (the site's +normal search with all the words). + +## How the app decides + +Each site advertises `BrowseOptions.combine_tags` (how many tags one URL can +intersect exactly: PornHub 2, others 0) and `include_exclude` (Eporner, +PornHub). `max_include` / `max_exclude` cap the panel (PornHub 1 / 10), and +`exclude_only_kinds` lists listings that take only exclusions (PornHub +category pages). A combined tag slug is sent as `a,b,…`; when the count exceeds the +site's exact limit, `AppState::browse` turns it into a word search, and the +results page shows a note that the combination is not exact. diff --git a/public/site-icons/chaturbate.png b/public/site-icons/chaturbate.png new file mode 100644 index 0000000..32054a3 Binary files /dev/null and b/public/site-icons/chaturbate.png differ diff --git a/public/site-icons/eporner.png b/public/site-icons/eporner.png new file mode 100644 index 0000000..ecf9357 Binary files /dev/null and b/public/site-icons/eporner.png differ diff --git a/public/site-icons/instagram.png b/public/site-icons/instagram.png new file mode 100644 index 0000000..2d1ee27 Binary files /dev/null and b/public/site-icons/instagram.png differ diff --git a/public/site-icons/pornhub.png b/public/site-icons/pornhub.png new file mode 100644 index 0000000..7278cc7 Binary files /dev/null and b/public/site-icons/pornhub.png differ diff --git a/public/site-icons/reddit.png b/public/site-icons/reddit.png new file mode 100644 index 0000000..fe0fefd Binary files /dev/null and b/public/site-icons/reddit.png differ diff --git a/public/site-icons/redgifs.png b/public/site-icons/redgifs.png new file mode 100644 index 0000000..09a6cc3 Binary files /dev/null and b/public/site-icons/redgifs.png differ diff --git a/public/site-icons/stripchat.png b/public/site-icons/stripchat.png new file mode 100644 index 0000000..fdb6145 Binary files /dev/null and b/public/site-icons/stripchat.png differ diff --git a/public/site-icons/tiktok.png b/public/site-icons/tiktok.png new file mode 100644 index 0000000..dd836d3 Binary files /dev/null and b/public/site-icons/tiktok.png differ diff --git a/public/site-icons/twitter.png b/public/site-icons/twitter.png new file mode 100644 index 0000000..7ecf1ca Binary files /dev/null and b/public/site-icons/twitter.png differ diff --git a/public/site-icons/xhamster.png b/public/site-icons/xhamster.png new file mode 100644 index 0000000..3e0cd4a Binary files /dev/null and b/public/site-icons/xhamster.png differ diff --git a/public/site-icons/xnxx.png b/public/site-icons/xnxx.png new file mode 100644 index 0000000..6706f7c Binary files /dev/null and b/public/site-icons/xnxx.png differ diff --git a/public/site-icons/xpomelo.png b/public/site-icons/xpomelo.png new file mode 100644 index 0000000..7c69ce5 Binary files /dev/null and b/public/site-icons/xpomelo.png differ diff --git a/public/site-icons/xvideos.png b/public/site-icons/xvideos.png new file mode 100644 index 0000000..951ea46 Binary files /dev/null and b/public/site-icons/xvideos.png differ diff --git a/public/site-icons/youporn.png b/public/site-icons/youporn.png new file mode 100644 index 0000000..adf4049 Binary files /dev/null and b/public/site-icons/youporn.png differ diff --git a/public/site-icons/youtube.png b/public/site-icons/youtube.png new file mode 100644 index 0000000..2dac66c Binary files /dev/null and b/public/site-icons/youtube.png differ diff --git a/scripts/patch-android-lan.ps1 b/scripts/patch-android-lan.ps1 index b22b92b..74966ef 100644 --- a/scripts/patch-android-lan.ps1 +++ b/scripts/patch-android-lan.ps1 @@ -13,10 +13,14 @@ if (-not (Test-Path $GradlePath)) { } $content = Get-Content $GradlePath -Raw -if ($content -notmatch 'manifestPlaceholders\["usesCleartextTraffic"\] = "true"') { - $content = $content -replace 'manifestPlaceholders\["usesCleartextTraffic"\] = "false"', 'manifestPlaceholders\["usesCleartextTraffic"\] = "true"' +# The generated defaultConfig sets "false" and only the debug build type sets +# "true", so test for a remaining "false" (not for any "true"): otherwise +# release APKs keep cleartext off and the WebView cannot reach the on-device +# 127.0.0.1 media server or a Remote LAN host. +if ($content -match 'manifestPlaceholders\["usesCleartextTraffic"\] = "false"') { + $content = $content -replace 'manifestPlaceholders\["usesCleartextTraffic"\] = "false"', 'manifestPlaceholders["usesCleartextTraffic"] = "true"' Set-Content -Path $GradlePath -Value $content -NoNewline - Write-Host "Patched $GradlePath for cleartext HTTP (Remote LAN)." + Write-Host "Patched $GradlePath for cleartext HTTP (loopback media server + Remote LAN)." } else { Write-Host "Android LAN cleartext patch already applied." } @@ -44,6 +48,21 @@ $1 Write-Host "Android storage permissions already present." } + # All files access: lets ArcHive delete library files in shared storage + # (e.g. Download/ArcHive) it did not create in this install. Granted by the + # user in system settings; WRITE_EXTERNAL_STORAGE covers Android 10 and older. + if ($manifest -notmatch 'MANAGE_EXTERNAL_STORAGE') { + $manifest = $manifest -replace '()', @' +$1 + + +'@ + Set-Content -Path $ManifestPath -Value $manifest -NoNewline + Write-Host "Patched $ManifestPath with all-files access permissions." + } else { + Write-Host "Android all-files access permission already present." + } + # Enable Activity PiP so HTML video.requestPictureInPicture can float. if ($manifest -notmatch 'supportsPictureInPicture') { if ($manifest -match 'android:configChanges="[^"]*"') { diff --git a/scripts/patch-android-lan.sh b/scripts/patch-android-lan.sh index 15b326f..f25ef50 100644 --- a/scripts/patch-android-lan.sh +++ b/scripts/patch-android-lan.sh @@ -40,6 +40,18 @@ else echo "Patched $MANIFEST with mDNS and media storage permissions." fi +# All files access: lets ArcHive delete library files in shared storage +# (e.g. Download/ArcHive) it didn't create in this install. Granted by the +# user in system settings; WRITE_EXTERNAL_STORAGE covers Android <= 10. +if grep -q 'MANAGE_EXTERNAL_STORAGE' "$MANIFEST"; then + echo "Android all-files access permission already present." +else + sed -i '//a\ + \ + ' "$MANIFEST" + echo "Patched $MANIFEST with all-files access permissions." +fi + # Enable Activity PiP so HTML video.requestPictureInPicture can float. if grep -q 'supportsPictureInPicture' "$MANIFEST"; then echo "Android Picture-in-Picture already enabled." diff --git a/scripts/patch-android-ytdlp.ps1 b/scripts/patch-android-ytdlp.ps1 index 8730bc5..91eecdc 100644 --- a/scripts/patch-android-ytdlp.ps1 +++ b/scripts/patch-android-ytdlp.ps1 @@ -47,6 +47,19 @@ if (Test-Path $resumeWorkerSrc) { Copy-Item -Path $resumeWorkerSrc -Destination (Join-Path $pluginDestDir "PendingResumeWorker.kt") -Force Write-Host "Installed PendingResumeWorker.kt" } +$cookieBrowserSrc = Join-Path $Overlays "CookieBrowserPlugin.kt" +if (Test-Path $cookieBrowserSrc) { + Copy-Item -Path $cookieBrowserSrc -Destination (Join-Path $pluginDestDir "CookieBrowserPlugin.kt") -Force + Write-Host "Installed CookieBrowserPlugin.kt" +} +# FileProvider roots: shared storage + app-private dirs ("Open with default app"). +$filePathsSrc = Join-Path $Overlays "file_paths.xml" +$filePathsDest = Join-Path $GenAndroid "app\src\main\res\xml\file_paths.xml" +if (Test-Path $filePathsSrc) { + New-Item -ItemType Directory -Force -Path (Split-Path $filePathsDest) | Out-Null + Copy-Item -Path $filePathsSrc -Destination $filePathsDest -Force +} + # Ensure youtubedl-android + WorkManager dependencies $gradle = Get-Content $gradlePath -Raw @@ -166,6 +179,41 @@ if (Test-Path $proguardPath) { Write-Host "Created $proguardPath with YtDlp keep rules" } +# Projects patched before these rules existed: add the library keeps that +# stop R8 from breaking YoutubeDL.init / FFmpeg.init in release builds. +$proguardNow = Get-Content $proguardPath -Raw +if ($proguardNow -notmatch 'org\.apache\.commons\.compress') { + $libKeeps = @" +-keep class org.apache.commons.compress.** { *; } +-keep class org.apache.commons.io.** { *; } +-keep class com.fasterxml.jackson.** { *; } +-keepattributes Signature,*Annotation*,InnerClasses,EnclosingMethod +-dontwarn org.apache.commons.compress.** +-dontwarn org.apache.commons.io.** +-dontwarn com.fasterxml.jackson.** +"@ + $proguardNow = $proguardNow.TrimEnd() + "`r`n" + $libKeeps + Set-Content -Path $proguardPath -Value $proguardNow -NoNewline + Write-Host "Added youtubedl-android library ProGuard keeps." +} + +# Projects patched before the cookie login browser existed: add its keeps. +$proguardNow = Get-Content $proguardPath -Raw +if ($proguardNow -notmatch 'CookieBrowserPlugin') { + $cookieKeeps = "-keep class com.archhive.app.CookieBrowserPlugin { *; }`r`n-keep class com.archhive.app.CookieBrowserPlugin`$OpenArgs { *; }`r`n" + Set-Content -Path $proguardPath -Value ($proguardNow.TrimEnd() + "`r`n" + $cookieKeeps) -NoNewline + Write-Host "Added CookieBrowserPlugin ProGuard keeps." +} + +# Projects patched before the plugin Args keep existed: R8 stripped the +# nested *Args classes, breaking every yt-dlp invoke in release builds. +$proguardNow = Get-Content $proguardPath -Raw +if ($proguardNow -notmatch [regex]::Escape('com.archhive.app.**Args')) { + $argsKeep = "-keep class com.archhive.app.**Args { *; }`r`n" + Set-Content -Path $proguardPath -Value ($proguardNow.TrimEnd() + "`r`n" + $argsKeep) -NoNewline + Write-Host "Added plugin Args ProGuard keeps." +} + Write-Host "Android YtDlp overlay patch complete (youtubedl-android $YoutubeDlAndroidVersion)." # Remove stale Linux BtbN ffmpeg/ffprobe assets left from older tauri.android.conf.json diff --git a/scripts/patch-android-ytdlp.sh b/scripts/patch-android-ytdlp.sh index 7a9ef44..a388f1b 100644 --- a/scripts/patch-android-ytdlp.sh +++ b/scripts/patch-android-ytdlp.sh @@ -45,6 +45,10 @@ if [[ -f "$OVERLAYS/PendingResumeWorker.kt" ]]; then cp "$OVERLAYS/PendingResumeWorker.kt" "$PLUGIN_DEST_DIR/PendingResumeWorker.kt" echo "Installed PendingResumeWorker.kt" fi +if [[ -f "$OVERLAYS/CookieBrowserPlugin.kt" ]]; then + cp "$OVERLAYS/CookieBrowserPlugin.kt" "$PLUGIN_DEST_DIR/CookieBrowserPlugin.kt" + echo "Installed CookieBrowserPlugin.kt" +fi if ! grep -q 'youtubedl-android:library' "$GRADLE"; then python3 - "$GRADLE" "$YOUTUBE_DL_ANDROID_VERSION" "$WORK_RUNTIME_KTX_VERSION" <<'PY' @@ -143,6 +147,28 @@ if [[ ! -f "$PROGUARD_OVERLAY" ]]; then exit 1 fi +# youtubedl-android unpacks libpython.zip.so / libffmpeg.zip.so from the installed +# native library dir; modern AGP leaves .so files inside the APK +# (extractNativeLibs=false) and YoutubeDL.init fails. Use legacy packaging. +if ! grep -q 'useLegacyPackaging' "$GRADLE"; then + python3 - "$GRADLE" <<'PY' +import re, sys +from pathlib import Path +path = Path(sys.argv[1]) +text = path.read_text() +block = ''' packaging { + // youtubedl-android needs extracted native libs (libpython.zip.so etc.) + jniLibs.useLegacyPackaging = true + } +''' +new, n = re.subn(r'(?m)^(android \{)\n', lambda m: m.group(1) + "\n" + block, text, count=1) +if n == 0: + raise SystemExit("Could not find android block for useLegacyPackaging") +path.write_text(new) +print("Enabled jniLibs.useLegacyPackaging (extractNativeLibs) for youtubedl-android") +PY +fi + if [[ -f "$PROGUARD" ]] && grep -q 'YtDlpPlugin' "$PROGUARD"; then echo "ProGuard YtDlp keep rules already present." else @@ -158,6 +184,28 @@ else echo "Wrote ProGuard keep rules to $PROGUARD" fi +# Projects patched before these rules existed: keep the libraries +# youtubedl-android drives reflectively (R8 broke release-build init). +if ! grep -q 'org.apache.commons.compress' "$PROGUARD"; then + cat >> "$PROGUARD" <<'PG' + +-keep class org.apache.commons.compress.** { *; } +-keep class org.apache.commons.io.** { *; } +-keep class com.fasterxml.jackson.** { *; } +-keepattributes Signature,*Annotation*,InnerClasses,EnclosingMethod +-dontwarn org.apache.commons.compress.** +-dontwarn org.apache.commons.io.** +-dontwarn com.fasterxml.jackson.** +PG + echo "Added youtubedl-android library ProGuard keeps." +fi + +# Projects patched before the cookie login browser existed: add its keeps. +if ! grep -q 'CookieBrowserPlugin' "$PROGUARD"; then + printf '\n-keep class com.archhive.app.CookieBrowserPlugin { *; }\n-keep class com.archhive.app.CookieBrowserPlugin$OpenArgs { *; }\n' >> "$PROGUARD" + echo "Added CookieBrowserPlugin ProGuard keeps." +fi + echo "Android YtDlp overlay patch complete (youtubedl-android $YOUTUBE_DL_ANDROID_VERSION)." STALE_BINARIES="$GEN_ANDROID/app/src/main/assets/binaries" diff --git a/scripts/setup-binaries.sh b/scripts/setup-binaries.sh index 60a6861..ac30798 100644 --- a/scripts/setup-binaries.sh +++ b/scripts/setup-binaries.sh @@ -10,7 +10,17 @@ if [[ "$ARCH" == *windows* ]]; then EXT=".exe"; fi YTDLP_OUT="$BIN_DIR/yt-dlp-$ARCH$EXT" echo "Downloading yt-dlp to $YTDLP_OUT..." -curl -fsSL "https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp" -o "$YTDLP_OUT" +# Standalone builds bundle Python and curl_cffi, which `--impersonate` needs +# to get past Cloudflare fingerprint blocks; the plain `yt-dlp` zipapp needs +# a system Python and can't impersonate. +case "$ARCH" in + *apple-darwin*) YTDLP_ASSET="yt-dlp_macos" ;; + x86_64-*linux*) YTDLP_ASSET="yt-dlp_linux" ;; + aarch64-*linux*) YTDLP_ASSET="yt-dlp_linux_aarch64" ;; + *windows*) YTDLP_ASSET="yt-dlp.exe" ;; + *) YTDLP_ASSET="yt-dlp" ;; +esac +curl -fsSL "https://github.com/yt-dlp/yt-dlp/releases/latest/download/$YTDLP_ASSET" -o "$YTDLP_OUT" chmod +x "$YTDLP_OUT" FFMPEG_OUT="$BIN_DIR/ffmpeg-$ARCH$EXT" diff --git a/scripts/validate-android-build.ps1 b/scripts/validate-android-build.ps1 index ea68a5f..5198779 100644 --- a/scripts/validate-android-build.ps1 +++ b/scripts/validate-android-build.ps1 @@ -36,6 +36,7 @@ $sharePlugin = Join-Path $gen "app\src\main\java\com\archhive\app\ShareIntentPlu $mainActivity = Join-Path $gen "app\src\main\java\com\archhive\app\MainActivity.kt" $fgService = Join-Path $gen "app\src\main\java\com\archhive\app\DownloadForegroundService.kt" $resumeWorker = Join-Path $gen "app\src\main\java\com\archhive\app\PendingResumeWorker.kt" +$cookieBrowser = Join-Path $gen "app\src\main\java\com\archhive\app\CookieBrowserPlugin.kt" $gradle = Join-Path $gen "app\build.gradle.kts" $proguard = Join-Path $gen "app\proguard-rules.pro" $manifest = Join-Path $gen "app\src\main\AndroidManifest.xml" @@ -56,6 +57,9 @@ if (-not (Test-Path $fgService)) { if (-not (Test-Path $resumeWorker)) { Add-Fail "Missing PendingResumeWorker.kt" } +if (-not (Test-Path $cookieBrowser)) { + Add-Fail "Missing CookieBrowserPlugin.kt (run scripts/apply-android-patches.ps1)" +} if (-not (Test-Path $gradle)) { Add-Fail "Missing app/build.gradle.kts" @@ -70,8 +74,11 @@ if (-not (Test-Path $gradle)) { if ($gradleText -notmatch 'work-runtime-ktx') { Add-Warn "build.gradle.kts missing work-runtime-ktx (PendingResumeWorker)" } - if ($gradleText -notmatch 'manifestPlaceholders\["usesCleartextTraffic"\]\s*=\s*"true"') { - Add-Fail "build.gradle.kts missing usesCleartextTraffic=true placeholder (LAN HTTP)" + if ($gradleText -notmatch 'useLegacyPackaging\s*=\s*true') { + Add-Fail "build.gradle.kts missing jniLibs.useLegacyPackaging = true (youtubedl-android cannot init without extracted native libs)" + } + if ($gradleText -match 'manifestPlaceholders\["usesCleartextTraffic"\]\s*=\s*"false"') { + Add-Fail "build.gradle.kts still sets usesCleartextTraffic=false (release WebView cannot reach the 127.0.0.1 media server)" } } diff --git a/src-tauri/Cargo.lock b/src-tauri/Cargo.lock index 6f9a70d..323d126 100644 --- a/src-tauri/Cargo.lock +++ b/src-tauri/Cargo.lock @@ -4889,7 +4889,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd" dependencies = [ "fastrand", - "getrandom 0.4.3", + "getrandom 0.3.4", "once_cell", "rustix", "windows-sys 0.61.2", diff --git a/src-tauri/android-overlays/CookieBrowserPlugin.kt b/src-tauri/android-overlays/CookieBrowserPlugin.kt new file mode 100644 index 0000000..29b3520 --- /dev/null +++ b/src-tauri/android-overlays/CookieBrowserPlugin.kt @@ -0,0 +1,837 @@ +package com.archhive.app + +import android.annotation.SuppressLint +import android.app.Activity +import android.app.AlertDialog +import android.app.Dialog +import android.content.Context +import android.os.Message +import android.content.Intent +import android.net.Uri +import android.graphics.Bitmap +import android.graphics.Color +import android.os.Handler +import android.os.Looper +import android.util.Log +import android.view.Gravity +import android.view.ViewGroup +import android.webkit.CookieManager +import android.webkit.WebChromeClient +import android.webkit.WebResourceRequest +import android.webkit.WebResourceResponse +import android.webkit.WebSettings +import android.webkit.WebStorage +import android.webkit.WebView +import android.webkit.WebViewClient +import android.widget.Button +import android.widget.LinearLayout +import android.widget.ProgressBar +import android.widget.TextView +import app.tauri.annotation.Command +import app.tauri.annotation.TauriPlugin +import app.tauri.plugin.Invoke +import app.tauri.plugin.JSArray +import app.tauri.plugin.JSObject +import app.tauri.plugin.Plugin +import androidx.core.content.FileProvider +import java.io.ByteArrayInputStream +import java.io.File +import org.json.JSONObject +import org.json.JSONTokener + +/** + * In-app login browser used to capture site cookies on Android. + * + * Android apps cannot read Chrome's cookie jar, so ArcHive opens its own + * full-screen WebView on the site's login page. When the user taps "Save + * cookies", every cookie the WebView holds for the requested URLs, including + * HttpOnly session cookies, is read from the process-wide [CookieManager] + * and returned to Rust, which stores them in the encrypted cookie vault. + * + * Args: { url: string, cookieUrls: string[], title: string } + * Returns: { cookies: [{ url: string, header: string }] } + * Rejects with "cancelled" when the user backs out. + */ +@TauriPlugin +class CookieBrowserPlugin(private val activity: Activity) : Plugin(activity) { + + companion object { + private const val TAG = "CookieBrowserPlugin" + private const val EASYLIST_URL = "https://easylist.to/easylist/easylist.txt" + } + + class OpenArgs @JvmOverloads constructor( + var url: String = "", + var cookieUrls: Array = emptyArray(), + var title: String = "Log in", + ) + + @Volatile private var dialog: Dialog? = null + + class ResolveArgs @JvmOverloads constructor( + var url: String = "", + var title: String = "Opening video page", + ) + + class FollowArgs @JvmOverloads constructor( + var url: String = "", + var referer: String = "", + ) + + class BrowseArgs @JvmOverloads constructor( + var url: String = "", + var title: String = "", + var adblock: Boolean = true, + ) + + class ShareArgs @JvmOverloads constructor( + var text: String = "", + var path: String = "", + var mime: String = "", + var title: String = "Share", + ) + + /** + * Hosts of ad / popunder networks common on tube sites. Requests to them + * (and their subdomains) get an empty answer in the in-app browser. + */ + private val adHosts = setOf( + "exoclick.com", "exosrv.com", "exdynsrv.com", "realsrv.com", "trafficjunky.com", + "trafficjunky.net", "trafficfactory.biz", "juicyads.com", "adsterra.com", + "adsterratech.com", "highperformanceformat.com", "profitablecpmrate.com", + "popads.net", "popcash.net", "propellerads.com", "propellerclick.com", + "hilltopads.net", "hilltopads.com", "adspyglass.com", "tsyndicate.com", + "trafficstars.com", "magsrv.com", "a-ads.com", "ero-advertising.com", + "plugrush.com", "clickadu.com", "adnium.com", "twinrdsrv.com", + "doubleclick.net", "googlesyndication.com", "google-analytics.com", + "googletagmanager.com", "adservice.google.com", "scorecardresearch.com", + "onclickads.net", "onclkds.com", "richpartners.co", "galaksion.com", + "pemsrv.com", "rmhfrtnd.com", "tsyndicate.com", "dtscout.com", + ) + + /** Site of a host for "is this another site?": its last two labels + * (m.site.com and www.site.com are the same site). */ + private fun siteOf(host: String?): String { + val parts = (host ?: "").lowercase().split('.').filter { it.isNotEmpty() } + return parts.takeLast(2).joinToString(".") + } + + /** Ad-server domains from EasyList (its `||domain^` rules), cached in + * the app's files and refreshed every few days. Empty until loaded. */ + @Volatile private var listHosts: Set = emptySet() + @Volatile private var listLoading = false + + private fun listFile() = File(activity.filesDir, "adblock-easylist-domains.txt") + + /** Load the cached EasyList domains, downloading the list first when + * it is missing or older than 4 days (its own update interval). Runs + * in the background; blocking uses the built-in list until then. */ + private fun ensureFilterList() { + if (listLoading) return + listLoading = true + Thread { + try { + val file = listFile() + val stale = !file.exists() || + System.currentTimeMillis() - file.lastModified() > 4L * 24 * 3600 * 1000 + if (stale) { + try { + val conn = java.net.URL(EASYLIST_URL).openConnection() as java.net.HttpURLConnection + conn.connectTimeout = 15000 + conn.readTimeout = 30000 + val domains = HashSet() + conn.inputStream.bufferedReader().useLines { lines -> + for (line in lines) parseRule(line)?.let { domains.add(it) } + } + if (domains.size > 1000) { + file.writeText(domains.joinToString("\n")) + } + } catch (e: Exception) { + Log.w(TAG, "EasyList download failed; using the cached copy", e) + } + } + if (file.exists() && (listHosts.isEmpty() || stale)) { + listHosts = file.readLines().filter { it.isNotBlank() }.toHashSet() + Log.i(TAG, "ad blocking: ${listHosts.size} EasyList domains") + } + } catch (e: Exception) { + Log.w(TAG, "EasyList load failed", e) + } finally { + listLoading = false + } + }.start() + } + + /** `||ads.example.com^` (optionally with `$options`) → `ads.example.com`. + * Rules limited to some sites (`domain=`), paths, wildcards and + * element-hiding rules are skipped: only whole ad-server domains. */ + private fun parseRule(raw: String): String? { + val line = raw.trim() + if (!line.startsWith("||")) return null + val body = line.substring(2) + val rule = body.substringBefore('$') + if (body.contains('$') && body.substringAfter('$').contains("domain=")) return null + val host = rule.removeSuffix("^").lowercase() + if (host.isEmpty() || !host.contains('.')) return null + if (host.any { !(it.isLetterOrDigit() || it == '.' || it == '-') }) return null + return host + } + + private fun isAdHost(host: String?): Boolean { + if (host.isNullOrEmpty()) return false + val fromList = listHosts + var h: String = host.lowercase() + while (true) { + if (h in adHosts || h in fromList) return true + val dot = h.indexOf('.') + if (dot < 0 || h.indexOf('.', dot + 1) < 0) return false + h = h.substring(dot + 1) + } + } + + /** Expire every cookie this site set (all its domain levels) and its + * local storage; other sites keep theirs. */ + private fun clearSiteData(pageUrl: String) { + val uri = Uri.parse(pageUrl) + val host = uri.host ?: return + val cm = CookieManager.getInstance() + val parts = host.split('.') + val domains = (0 until maxOf(1, parts.size - 1)).map { parts.drop(it).joinToString(".") } + for (d in domains) { + for (scheme in listOf("https", "http")) { + val u = "$scheme://$d/" + val header = cm.getCookie(u) ?: continue + for (pair in header.split(';')) { + val name = pair.substringBefore('=').trim() + if (name.isEmpty()) continue + for (dom in listOf(d, ".$d")) { + cm.setCookie(u, "$name=; Max-Age=0; Path=/; Domain=$dom") + } + cm.setCookie(u, "$name=; Max-Age=0; Path=/") + } + } + WebStorage.getInstance().deleteOrigin("https://$d") + WebStorage.getInstance().deleteOrigin("http://$d") + } + cm.flush() + } + + private fun shareText(text: String, title: String) { + val send = Intent(Intent.ACTION_SEND).apply { + type = "text/plain" + putExtra(Intent.EXTRA_TEXT, text) + } + activity.startActivity(Intent.createChooser(send, title)) + } + + /** + * Share sheet: a URL / text, or a library file (to video players and + * other apps, through the app's FileProvider). + */ + @Command + fun share(invoke: Invoke) { + val args = invoke.parseArgs(ShareArgs::class.java) + activity.runOnUiThread { + try { + if (args.path.isNotEmpty()) { + val file = File(args.path) + if (!file.exists()) { + invoke.reject("File not found: ${args.path}") + return@runOnUiThread + } + val uri = FileProvider.getUriForFile( + activity, "${activity.packageName}.fileprovider", file, + ) + val send = Intent(Intent.ACTION_SEND).apply { + type = args.mime.ifEmpty { "video/*" } + putExtra(Intent.EXTRA_STREAM, uri) + addFlags(Intent.FLAG_GRANT_READ_URI_PERMISSION) + } + val chooser = Intent.createChooser(send, args.title) + chooser.addFlags(Intent.FLAG_GRANT_READ_URI_PERMISSION) + activity.startActivity(chooser) + } else if (args.text.isNotEmpty()) { + shareText(args.text, args.title) + } else { + invoke.reject("Nothing to share") + return@runOnUiThread + } + invoke.resolve() + } catch (e: Exception) { + Log.e(TAG, "share failed", e) + invoke.reject("Share failed: ${e.message}") + } + } + } + + /** + * In-app browser for watching a page the app can't play itself, instead + * of the system browser (and its history). Keeps the WebView's disk cache + * and cookies; blocks ad / popunder networks and popups; the toolbar can + * clear this site's cookies and storage, or share the page. + */ + @SuppressLint("SetJavaScriptEnabled") + @Command + fun browse(invoke: Invoke) { + val args = invoke.parseArgs(BrowseArgs::class.java) + if (!args.url.startsWith("https://") && !args.url.startsWith("http://")) { + invoke.reject("Invalid URL") + return + } + activity.runOnUiThread { + try { + dialog?.dismiss() + val cookieManager = CookieManager.getInstance() + cookieManager.setAcceptCookie(true) + val dlg = Dialog(activity, android.R.style.Theme_Black_NoTitleBar_Fullscreen) + var settled = false + var blocked = 0 + val root = LinearLayout(activity).apply { + orientation = LinearLayout.VERTICAL + setBackgroundColor(Color.parseColor("#111114")) + } + val bar = LinearLayout(activity).apply { + orientation = LinearLayout.HORIZONTAL + gravity = Gravity.CENTER_VERTICAL + setPadding(8, 8, 8, 8) + } + val close = Button(activity).apply { text = "Close" } + val title = TextView(activity).apply { + text = args.title.ifEmpty { args.url } + setTextColor(Color.WHITE) + textSize = 13f + maxLines = 2 + setPadding(12, 0, 12, 0) + } + val clear = Button(activity).apply { text = "Clear site" } + val shareBtn = Button(activity).apply { text = "Share" } + // Ad blocking is opt-in (sites with anti-adblock refuse to + // play when ad requests fail); remembered across sessions. + val prefs = activity.getSharedPreferences("archhive_browser", Context.MODE_PRIVATE) + var adblock = prefs.getBoolean("adblock", false) + if (adblock) ensureFilterList() + val adsBtn = Button(activity) + fun adsLabel() { adsBtn.text = if (adblock) "Ads: off" else "Ads: on" } + adsLabel() + var popups = 0 + bar.addView(close) + bar.addView(title, LinearLayout.LayoutParams(0, ViewGroup.LayoutParams.WRAP_CONTENT, 1f)) + bar.addView(adsBtn) + bar.addView(clear) + bar.addView(shareBtn) + val progress = ProgressBar(activity, null, android.R.attr.progressBarStyleHorizontal).apply { + isIndeterminate = false + max = 100 + } + val webView = WebView(activity).apply { + settings.javaScriptEnabled = true + settings.domStorageEnabled = true + settings.databaseEnabled = true + settings.cacheMode = WebSettings.LOAD_DEFAULT + settings.loadWithOverviewMode = true + settings.useWideViewPort = true + settings.mediaPlaybackRequiresUserGesture = false + // Popups (click-anywhere popunders): with multiple windows + // off, window.open() would load the ad in this page, so + // windows are "supported" and every request is refused + // in onCreateWindow. + settings.setSupportMultipleWindows(true) + settings.javaScriptCanOpenWindowsAutomatically = false + settings.userAgentString = settings.userAgentString.replace("; wv", "") + webViewClient = object : WebViewClient() { + override fun shouldInterceptRequest( + view: WebView, + request: WebResourceRequest, + ): WebResourceResponse? { + if (adblock && isAdHost(request.url.host)) { + blocked++ + return WebResourceResponse( + "text/plain", "utf-8", ByteArrayInputStream(ByteArray(0)), + ) + } + return null + } + + override fun shouldOverrideUrlLoading(view: WebView, request: WebResourceRequest): Boolean { + val scheme = request.url.scheme ?: return true + if (scheme != "http" && scheme != "https") return true + if (!request.isForMainFrame) return false + if (adblock && isAdHost(request.url.host)) return true + // Click hijacks send the page to another site; + // server redirects (e.g. a /go link) and links + // within the site load as usual. + val here = siteOf(Uri.parse(view.url ?: args.url).host) + val there = siteOf(request.url.host) + if (request.isRedirect || here.isEmpty() || here == there) return false + val target = request.url.toString() + AlertDialog.Builder(activity) + .setTitle("Leave for ${request.url.host}?") + .setMessage(target.take(200)) + .setPositiveButton("Open") { _, _ -> view.loadUrl(target) } + .setNegativeButton("Stay", null) + .show() + return true + } + + override fun onPageStarted(view: WebView, url: String?, favicon: Bitmap?) { + title.text = url ?: "" + } + + override fun onPageFinished(view: WebView, url: String?) { + val label = view.title?.takeIf { it.isNotBlank() } ?: url ?: "" + title.text = if (blocked > 0) "$label\n$blocked ads blocked" else label + } + } + webChromeClient = object : WebChromeClient() { + override fun onProgressChanged(view: WebView, newProgress: Int) { + progress.progress = newProgress + progress.visibility = if (newProgress >= 100) android.view.View.GONE else android.view.View.VISIBLE + } + + // Every new window (popunder, target=_blank ad) is refused. + override fun onCreateWindow( + view: WebView, + isDialog: Boolean, + isUserGesture: Boolean, + resultMsg: Message, + ): Boolean { + popups++ + title.text = "${view.title ?: view.url ?: ""}\n$popups popups blocked" + return false + } + } + } + cookieManager.setAcceptThirdPartyCookies(webView, true) + adsBtn.setOnClickListener { + adblock = !adblock + prefs.edit().putBoolean("adblock", adblock).apply() + if (adblock) ensureFilterList() + adsLabel() + webView.reload() + } + root.addView(bar, LinearLayout.LayoutParams(ViewGroup.LayoutParams.MATCH_PARENT, ViewGroup.LayoutParams.WRAP_CONTENT)) + root.addView(progress, LinearLayout.LayoutParams(ViewGroup.LayoutParams.MATCH_PARENT, 6)) + root.addView(webView, LinearLayout.LayoutParams(ViewGroup.LayoutParams.MATCH_PARENT, 0, 1f)) + dlg.setContentView(root) + + fun finish() { + if (settled) return + settled = true + cookieManager.flush() + webView.stopLoading() + webView.destroy() + dlg.dismiss() + dialog = null + invoke.resolve() + } + close.setOnClickListener { finish() } + dlg.setOnCancelListener { finish() } + clear.setOnClickListener { + val current = webView.url ?: args.url + clearSiteData(current) + webView.clearCache(false) + title.text = "Cookies and storage cleared for ${Uri.parse(current).host ?: "this site"}" + webView.reload() + } + shareBtn.setOnClickListener { + try { + shareText(webView.url ?: args.url, "Share page") + } catch (e: Exception) { + Log.w(TAG, "share failed", e) + } + } + dlg.setOnKeyListener { _, keyCode, event -> + if (keyCode == android.view.KeyEvent.KEYCODE_BACK && + event.action == android.view.KeyEvent.ACTION_UP && + webView.canGoBack() + ) { + webView.goBack() + true + } else { + false + } + } + dialog = dlg + dlg.show() + webView.loadUrl(args.url) + } catch (e: Exception) { + Log.e(TAG, "failed to open in-app browser", e) + invoke.reject("Could not open the in-app browser: ${e.message}") + } + } + } + + /** + * Page resolver used when a site shows a Cloudflare / bot check to the + * embedded engines (yt-dlp, the Rust HTTP client). The page is opened in + * a real WebView — the check passes there (or the user taps it once) — + * and Rust gets the rendered HTML, cookies and User-Agent back. The + * dialog stays open so [followRedirect] can resolve session-bound links + * (KVS `get_file`) inside the same browser session; [closeResolver] + * dismisses it. One resolver at a time. + */ + private inner class Resolver( + val dlg: Dialog, + val webView: WebView, + val status: TextView, + ) { + var pending: Invoke? = null + var mode = "page" // page | idle | follow + var followTarget: String? = null + var timeout: Runnable? = null + var poll: Runnable? = null + } + + @Volatile private var resolver: Resolver? = null + private val main = Handler(Looper.getMainLooper()) + + private fun isChallengeTitle(t: String): Boolean = + Regex("just a moment|attention required|checking your browser|ddos-guard|security verification|verify you are human", RegexOption.IGNORE_CASE) + .containsMatchIn(t) + + private fun settle(r: Resolver, ok: JSObject?, error: String?) { + val inv = r.pending ?: return + r.pending = null + r.timeout?.let { main.removeCallbacks(it) } + r.timeout = null + if (ok != null) inv.resolve(ok) else inv.reject(error ?: "failed") + } + + private fun closeResolverNow(error: String?) { + val r = resolver ?: return + resolver = null + r.poll?.let { main.removeCallbacks(it) } + settle(r, null, error ?: "closed") + try { + r.webView.stopLoading() + r.webView.destroy() + } catch (_: Exception) { + } + try { + r.dlg.dismiss() + } catch (_: Exception) { + } + } + + private fun checkPage(r: Resolver) { + if (resolver !== r || r.mode != "page") return + val probe = "(function(){var t=document.title||'';" + + "var cf=!!document.querySelector('#challenge-form,#challenge-running,#cf-challenge-running,.cf-turnstile,[name=cf-turnstile-response],#turnstile-wrapper');" + + "return JSON.stringify({t:t,cf:cf,ready:document.readyState});})()" + r.webView.evaluateJavascript(probe) { raw -> + if (resolver !== r || r.mode != "page") return@evaluateJavascript + val info = try { + JSONObject(JSONTokener(raw).nextValue() as String) + } catch (_: Exception) { + null + } ?: return@evaluateJavascript + val title = info.optString("t") + if (info.optBoolean("cf") || isChallengeTitle(title)) { + r.status.text = "Security check — tap the box if one appears. This closes by itself." + return@evaluateJavascript + } + if (info.optString("ready") != "complete") return@evaluateJavascript + r.mode = "idle" + r.status.text = "Loading video…" + // Give the site's player scripts a moment to fill in the source. + main.postDelayed({ + if (resolver !== r) return@postDelayed + r.webView.evaluateJavascript("document.documentElement.outerHTML") { htmlRaw -> + if (resolver !== r) return@evaluateJavascript + val html = try { + JSONTokener(htmlRaw).nextValue() as? String ?: "" + } catch (_: Exception) { + "" + } + val pageUrl = r.webView.url ?: "" + val ret = JSObject() + ret.put("html", html) + ret.put("url", pageUrl) + ret.put("userAgent", r.webView.settings.userAgentString) + ret.put("cookies", CookieManager.getInstance().getCookie(pageUrl) ?: "") + settle(r, ret, null) + } + }, 1500) + } + } + + private fun finishFollow(r: Resolver, url: String) { + if (r.mode != "follow") return + r.mode = "idle" + try { + r.webView.stopLoading() + } catch (_: Exception) { + } + val ret = JSObject() + ret.put("url", url) + ret.put("cookies", CookieManager.getInstance().getCookie(url) ?: "") + settle(r, ret, null) + } + + @SuppressLint("SetJavaScriptEnabled") + @Command + fun resolvePage(invoke: Invoke) { + val args = invoke.parseArgs(ResolveArgs::class.java) + if (!args.url.startsWith("https://") && !args.url.startsWith("http://")) { + invoke.reject("Invalid page URL") + return + } + activity.runOnUiThread { + try { + closeResolverNow("superseded") + val cookieManager = CookieManager.getInstance() + cookieManager.setAcceptCookie(true) + val dlg = Dialog(activity, android.R.style.Theme_Black_NoTitleBar_Fullscreen) + val root = LinearLayout(activity).apply { + orientation = LinearLayout.VERTICAL + setBackgroundColor(Color.parseColor("#111114")) + } + val bar = LinearLayout(activity).apply { + orientation = LinearLayout.HORIZONTAL + gravity = Gravity.CENTER_VERTICAL + setPadding(16, 12, 16, 12) + } + val cancel = Button(activity).apply { text = "Cancel" } + val status = TextView(activity).apply { + text = "${args.title}\nOpening the page… this closes by itself." + setTextColor(Color.WHITE) + textSize = 14f + maxLines = 3 + setPadding(16, 0, 16, 0) + } + bar.addView(cancel) + bar.addView(status, LinearLayout.LayoutParams(0, ViewGroup.LayoutParams.WRAP_CONTENT, 1f)) + val webView = WebView(activity) + val r = Resolver(dlg, webView, status) + webView.apply { + settings.javaScriptEnabled = true + settings.domStorageEnabled = true + settings.databaseEnabled = true + settings.loadWithOverviewMode = true + settings.useWideViewPort = true + settings.mediaPlaybackRequiresUserGesture = true + settings.setSupportMultipleWindows(false) + settings.userAgentString = settings.userAgentString.replace("; wv", "") + webViewClient = object : WebViewClient() { + override fun shouldOverrideUrlLoading(view: WebView, request: WebResourceRequest): Boolean { + val scheme = request.url.scheme ?: return true + if (scheme != "http" && scheme != "https") return true + if (r.mode == "follow" && request.isForMainFrame && request.isRedirect) { + finishFollow(r, request.url.toString()) + return true + } + return false + } + + override fun onPageStarted(view: WebView, url: String?, favicon: Bitmap?) { + // A followed link that did not redirect (or whose + // redirect skipped shouldOverrideUrlLoading). + if (r.mode == "follow" && url != null) finishFollow(r, url) + } + + override fun onPageFinished(view: WebView, url: String?) { + if (r.mode == "page") main.postDelayed({ checkPage(r) }, 400) + } + } + } + cookieManager.setAcceptThirdPartyCookies(webView, true) + root.addView(bar, LinearLayout.LayoutParams(ViewGroup.LayoutParams.MATCH_PARENT, ViewGroup.LayoutParams.WRAP_CONTENT)) + root.addView(webView, LinearLayout.LayoutParams(ViewGroup.LayoutParams.MATCH_PARENT, 0, 1f)) + dlg.setContentView(root) + cancel.setOnClickListener { closeResolverNow("cancelled") } + dlg.setOnCancelListener { closeResolverNow("cancelled") } + + r.pending = invoke + resolver = r + // Challenge pages often finish via JS without a new page load. + val poll = object : Runnable { + override fun run() { + if (resolver !== r) return + if (r.mode == "page") checkPage(r) + main.postDelayed(this, 2000) + } + } + r.poll = poll + main.postDelayed(poll, 2500) + val timeout = Runnable { closeResolverNow("timed out waiting for the page") } + r.timeout = timeout + main.postDelayed(timeout, 120_000) + dlg.show() + webView.loadUrl(args.url) + } catch (e: Exception) { + Log.e(TAG, "failed to open page resolver", e) + invoke.reject("Could not open the in-app browser: ${e.message}") + } + } + } + + /** Load [FollowArgs.url] in the resolver's session and report where it redirects. */ + @Command + fun followRedirect(invoke: Invoke) { + val args = invoke.parseArgs(FollowArgs::class.java) + activity.runOnUiThread { + val r = resolver + if (r == null) { + invoke.reject("resolver closed") + return@runOnUiThread + } + settle(r, null, "superseded") + r.pending = invoke + r.mode = "follow" + r.followTarget = args.url + r.status.text = "Loading video…" + val timeout = Runnable { finishFollow(r, args.url) } + r.timeout = timeout + main.postDelayed(timeout, 20_000) + val headers = if (args.referer.isNotBlank()) mapOf("Referer" to args.referer) else emptyMap() + r.webView.loadUrl(args.url, headers) + } + } + + @Command + fun closeResolver(invoke: Invoke) { + activity.runOnUiThread { + closeResolverNow("closed") + invoke.resolve() + } + } + + @SuppressLint("SetJavaScriptEnabled") + @Command + fun open(invoke: Invoke) { + val args = invoke.parseArgs(OpenArgs::class.java) + if (!args.url.startsWith("https://") && !args.url.startsWith("http://")) { + invoke.reject("Invalid login URL") + return + } + val cookieUrls = if (args.cookieUrls.isEmpty()) arrayOf(args.url) else args.cookieUrls + + activity.runOnUiThread { + try { + dialog?.dismiss() + val cookieManager = CookieManager.getInstance() + cookieManager.setAcceptCookie(true) + + val dlg = Dialog(activity, android.R.style.Theme_Black_NoTitleBar_Fullscreen) + var settled = false + + val root = LinearLayout(activity).apply { + orientation = LinearLayout.VERTICAL + setBackgroundColor(Color.parseColor("#111114")) + } + val bar = LinearLayout(activity).apply { + orientation = LinearLayout.HORIZONTAL + gravity = Gravity.CENTER_VERTICAL + setPadding(16, 12, 16, 12) + } + val cancel = Button(activity).apply { text = "Cancel" } + val title = TextView(activity).apply { + text = args.title + setTextColor(Color.WHITE) + textSize = 15f + maxLines = 2 + setPadding(16, 0, 16, 0) + } + val save = Button(activity).apply { text = "Save cookies" } + bar.addView(cancel) + bar.addView(title, LinearLayout.LayoutParams(0, ViewGroup.LayoutParams.WRAP_CONTENT, 1f)) + bar.addView(save) + + val progress = ProgressBar(activity, null, android.R.attr.progressBarStyleHorizontal).apply { + isIndeterminate = false + max = 100 + } + + val webView = WebView(activity).apply { + settings.javaScriptEnabled = true + settings.domStorageEnabled = true + settings.databaseEnabled = true + settings.loadWithOverviewMode = true + settings.useWideViewPort = true + settings.setSupportMultipleWindows(false) + // Drop the "; wv" token: some login pages (Google, X) + // refuse embedded WebViews that advertise themselves. + settings.userAgentString = settings.userAgentString.replace("; wv", "") + webViewClient = object : WebViewClient() { + override fun shouldOverrideUrlLoading(view: WebView, request: WebResourceRequest): Boolean { + val scheme = request.url.scheme ?: return true + // Stay inside the WebView for web pages; ignore app deep links. + return scheme != "http" && scheme != "https" + } + + override fun onPageStarted(view: WebView, url: String?, favicon: Bitmap?) { + title.text = "${args.title}\n${url ?: ""}" + } + } + webChromeClient = object : WebChromeClient() { + override fun onProgressChanged(view: WebView, newProgress: Int) { + progress.progress = newProgress + progress.visibility = if (newProgress >= 100) android.view.View.GONE else android.view.View.VISIBLE + } + } + } + cookieManager.setAcceptThirdPartyCookies(webView, true) + + root.addView(bar, LinearLayout.LayoutParams(ViewGroup.LayoutParams.MATCH_PARENT, ViewGroup.LayoutParams.WRAP_CONTENT)) + root.addView(progress, LinearLayout.LayoutParams(ViewGroup.LayoutParams.MATCH_PARENT, 6)) + root.addView(webView, LinearLayout.LayoutParams(ViewGroup.LayoutParams.MATCH_PARENT, 0, 1f)) + dlg.setContentView(root) + + fun finish(saveCookies: Boolean) { + if (settled) return + settled = true + try { + if (saveCookies) { + cookieManager.flush() + val list = JSArray() + for (u in cookieUrls) { + val header = cookieManager.getCookie(u) + if (!header.isNullOrBlank()) { + val entry = JSObject() + entry.put("url", u) + entry.put("header", header) + list.put(entry) + } + } + val ret = JSObject() + ret.put("cookies", list) + invoke.resolve(ret) + } else { + invoke.reject("cancelled") + } + } catch (e: Exception) { + Log.w(TAG, "cookie capture failed", e) + invoke.reject("Cookie capture failed: ${e.message}") + } finally { + webView.stopLoading() + webView.destroy() + dlg.dismiss() + dialog = null + } + } + + save.setOnClickListener { finish(true) } + cancel.setOnClickListener { finish(false) } + dlg.setOnCancelListener { finish(false) } + dlg.setOnKeyListener { _, keyCode, event -> + if (keyCode == android.view.KeyEvent.KEYCODE_BACK && + event.action == android.view.KeyEvent.ACTION_UP && + webView.canGoBack() + ) { + webView.goBack() + true + } else { + false + } + } + + dialog = dlg + dlg.show() + webView.loadUrl(args.url) + } catch (e: Exception) { + Log.e(TAG, "failed to open login browser", e) + invoke.reject("Could not open login browser: ${e.message}") + } + } + } +} diff --git a/src-tauri/android-overlays/YtDlpPlugin.kt b/src-tauri/android-overlays/YtDlpPlugin.kt index 3c099bd..a38d0e9 100644 --- a/src-tauri/android-overlays/YtDlpPlugin.kt +++ b/src-tauri/android-overlays/YtDlpPlugin.kt @@ -12,6 +12,7 @@ import app.tauri.plugin.Plugin import com.yausername.ffmpeg.FFmpeg import com.yausername.youtubedl_android.YoutubeDL import com.yausername.youtubedl_android.YoutubeDLRequest +import kotlinx.coroutines.CoroutineExceptionHandler import kotlinx.coroutines.CoroutineScope import kotlinx.coroutines.Dispatchers import kotlinx.coroutines.launch @@ -24,11 +25,19 @@ import java.util.concurrent.TimeUnit @TauriPlugin class YtDlpPlugin(private val activity: Activity) : Plugin(activity) { - private val scope = CoroutineScope(Dispatchers.IO) + // Last-resort guard: an uncaught Throwable in a plugin coroutine (e.g. a + // LinkageError from youtubedl-android) must not take the whole app down. + private val scope = CoroutineScope( + Dispatchers.IO + CoroutineExceptionHandler { _, e -> + Log.e("YtDlpPlugin", "Uncaught error in plugin coroutine", e) + } + ) @Volatile private var initialized = false @Volatile private var ffmpegPath: String? = null @Volatile private var ffprobePath: String? = null @Volatile private var ffmpegLibDir: String? = null + /** Last init failure, surfaced in "not initialized" responses for diagnosis. */ + @Volatile private var initError: String? = null /** * Lazily initialize YoutubeDL + FFmpeg (idempotent, thread-safe). @@ -44,7 +53,7 @@ class YtDlpPlugin(private val activity: Activity) : Plugin(activity) { YoutubeDL.getInstance().init(activity) try { FFmpeg.getInstance().init(activity) - } catch (e: Exception) { + } catch (e: Throwable) { // FFmpeg module may be absent in some builds; yt-dlp can still run. Log.w("YtDlpPlugin", "FFmpeg.init failed (media tools may be unavailable)", e) } @@ -52,12 +61,18 @@ class YtDlpPlugin(private val activity: Activity) : Plugin(activity) { resolveMediaToolsLocked() initialized = true true - } catch (e: Exception) { + } catch (e: Throwable) { Log.e("YtDlpPlugin", "Failed to init YoutubeDL", e) + initError = "${e.javaClass.simpleName}: ${e.message ?: ""}" false } } + private fun notInitialized(what: String): String { + val cause = initError ?: return "$what not initialized" + return "$what not initialized ($cause)" + } + /** FFmpeg.init is async; wait until libavdevice appears in the unpacked tree. */ private fun waitForFfmpegLibsUnpack() { val libDir = File( @@ -184,7 +199,7 @@ class YtDlpPlugin(private val activity: Activity) : Plugin(activity) { val ret = JSObject() if (!ensureInitialized()) { ret.put("ok", false) - ret.put("message", "YoutubeDL/FFmpeg not initialized") + ret.put("message", notInitialized("YoutubeDL/FFmpeg")) invoke.resolve(ret) return@launch } @@ -253,7 +268,7 @@ class YtDlpPlugin(private val activity: Activity) : Plugin(activity) { if (!ensureInitialized()) { val ret = JSObject() ret.put("stdout", "") - ret.put("stderr", "YoutubeDL/FFmpeg not initialized") + ret.put("stderr", notInitialized("YoutubeDL/FFmpeg")) ret.put("exitCode", -1) invoke.resolve(ret) return@launch @@ -276,7 +291,7 @@ class YtDlpPlugin(private val activity: Activity) : Plugin(activity) { ret.put("stderr", stderr) ret.put("exitCode", code) invoke.resolve(ret) - } catch (e: Exception) { + } catch (e: Throwable) { val ret = JSObject() ret.put("stdout", "") ret.put("stderr", e.message ?: "Unknown error") @@ -298,7 +313,7 @@ class YtDlpPlugin(private val activity: Activity) : Plugin(activity) { if (!ensureInitialized()) { val ret = JSObject() ret.put("stdout", "") - ret.put("stderr", "YoutubeDL not initialized") + ret.put("stderr", notInitialized("YoutubeDL")) ret.put("exitCode", -1) invoke.resolve(ret) return @@ -315,7 +330,7 @@ class YtDlpPlugin(private val activity: Activity) : Plugin(activity) { ret.put("stderr", result.err) ret.put("exitCode", 0) invoke.resolve(ret) - } catch (e: Exception) { + } catch (e: Throwable) { val ret = JSObject() ret.put("stdout", "") ret.put("stderr", e.message ?: "Unknown error") @@ -346,7 +361,7 @@ class YtDlpPlugin(private val activity: Activity) : Plugin(activity) { val ret = JSObject() ret.put("version", ver ?: "") invoke.resolve(ret) - } catch (e: Exception) { + } catch (e: Throwable) { val ret = JSObject() ret.put("version", "") invoke.resolve(ret) @@ -363,7 +378,7 @@ class YtDlpPlugin(private val activity: Activity) : Plugin(activity) { if (!ensureInitialized()) { val ret = JSObject() ret.put("success", false) - ret.put("message", "YoutubeDL not initialized") + ret.put("message", notInitialized("YoutubeDL")) invoke.resolve(ret) return } @@ -377,7 +392,7 @@ class YtDlpPlugin(private val activity: Activity) : Plugin(activity) { ret.put("success", true) ret.put("message", "Updated successfully") invoke.resolve(ret) - } catch (e: Exception) { + } catch (e: Throwable) { val ret = JSObject() ret.put("success", false) ret.put("message", e.message ?: "Update failed") @@ -400,7 +415,7 @@ class YtDlpPlugin(private val activity: Activity) : Plugin(activity) { val ret = JSObject() ret.put("ok", true) invoke.resolve(ret) - } catch (e: Exception) { + } catch (e: Throwable) { val ret = JSObject() ret.put("ok", false) ret.put("message", e.message ?: "startKeepAlive failed") @@ -421,7 +436,7 @@ class YtDlpPlugin(private val activity: Activity) : Plugin(activity) { val ret = JSObject() ret.put("ok", true) invoke.resolve(ret) - } catch (e: Exception) { + } catch (e: Throwable) { val ret = JSObject() ret.put("ok", false) ret.put("message", e.message ?: "updateKeepAlive failed") @@ -436,7 +451,7 @@ class YtDlpPlugin(private val activity: Activity) : Plugin(activity) { val ret = JSObject() ret.put("ok", true) invoke.resolve(ret) - } catch (e: Exception) { + } catch (e: Throwable) { val ret = JSObject() ret.put("ok", false) ret.put("message", e.message ?: "stopKeepAlive failed") @@ -444,6 +459,143 @@ class YtDlpPlugin(private val activity: Activity) : Plugin(activity) { } } + /** + * Open a local file in another app (video player, gallery, …) via + * ACTION_VIEW. Android forbids file:// URIs across apps, so the file is + * shared through the app's FileProvider (`${applicationId}.fileprovider`). + */ + @Command + fun openFile(invoke: Invoke) { + val args = invoke.parseArgs(OpenFileArgs::class.java) + val ret = JSObject() + try { + val file = File(args.path) + if (!file.isFile) { + ret.put("ok", false) + ret.put("message", "File not found: ${args.path}") + invoke.resolve(ret) + return + } + val uri = androidx.core.content.FileProvider.getUriForFile( + activity, + "${activity.packageName}.fileprovider", + file, + ) + val mime = args.mime.ifBlank { + android.webkit.MimeTypeMap.getSingleton() + .getMimeTypeFromExtension(file.extension.lowercase()) ?: "video/*" + } + val intent = android.content.Intent(android.content.Intent.ACTION_VIEW) + .setDataAndType(uri, mime) + .addFlags(android.content.Intent.FLAG_GRANT_READ_URI_PERMISSION) + activity.startActivity(android.content.Intent.createChooser(intent, "Open with")) + ret.put("ok", true) + } catch (e: Throwable) { + Log.w("YtDlpPlugin", "openFile failed for ${args.path}", e) + ret.put("ok", false) + ret.put("message", e.message ?: e.javaClass.simpleName) + } + invoke.resolve(ret) + } + + /** + * Video fullscreen. wry's WebChromeClient cancels HTML5 fullscreen + * (onShowCustomView), so the web player expands itself and asks for an + * immersive window here: system bars hidden (swipe to peek) and, for + * landscape video, a sensor-landscape orientation lock. + */ + /** + * Whether ArcHive may manage all files in shared storage ("All files + * access", Android 11+). Needed to delete library files in e.g. + * Download/ArcHive that this install didn't create. Android 10 and + * older use WRITE_EXTERNAL_STORAGE instead and report granted. + */ + @Command + fun storageAccess(invoke: Invoke) { + val ret = JSObject() + val granted = if (android.os.Build.VERSION.SDK_INT >= android.os.Build.VERSION_CODES.R) { + android.os.Environment.isExternalStorageManager() + } else { + true + } + ret.put("ok", granted) + ret.put("message", if (granted) "granted" else "missing") + invoke.resolve(ret) + } + + /** Open the system "All files access" screen for ArcHive. */ + @Command + fun openStorageAccessSettings(invoke: Invoke) { + activity.runOnUiThread { + val ret = JSObject() + try { + if (android.os.Build.VERSION.SDK_INT >= android.os.Build.VERSION_CODES.R) { + val intent = try { + android.content.Intent( + android.provider.Settings.ACTION_MANAGE_APP_ALL_FILES_ACCESS_PERMISSION, + android.net.Uri.parse("package:" + activity.packageName), + ) + } catch (e: Throwable) { + android.content.Intent( + android.provider.Settings.ACTION_MANAGE_ALL_FILES_ACCESS_PERMISSION, + ) + } + try { + activity.startActivity(intent) + } catch (e: android.content.ActivityNotFoundException) { + activity.startActivity( + android.content.Intent( + android.provider.Settings.ACTION_MANAGE_ALL_FILES_ACCESS_PERMISSION, + ), + ) + } + } + ret.put("ok", true) + } catch (e: Throwable) { + Log.w("YtDlpPlugin", "openStorageAccessSettings failed", e) + ret.put("ok", false) + ret.put("message", e.message ?: e.javaClass.simpleName) + } + invoke.resolve(ret) + } + } + + @Command + fun setImmersive(invoke: Invoke) { + val args = invoke.parseArgs(ImmersiveArgs::class.java) + activity.runOnUiThread { + val ret = JSObject() + try { + val window = activity.window + val controller = androidx.core.view.WindowCompat.getInsetsController( + window, + window.decorView, + ) + if (args.enabled) { + controller.systemBarsBehavior = + androidx.core.view.WindowInsetsControllerCompat + .BEHAVIOR_SHOW_TRANSIENT_BARS_BY_SWIPE + controller.hide(androidx.core.view.WindowInsetsCompat.Type.systemBars()) + activity.requestedOrientation = if (args.landscape) { + android.content.pm.ActivityInfo.SCREEN_ORIENTATION_SENSOR_LANDSCAPE + } else { + android.content.pm.ActivityInfo.SCREEN_ORIENTATION_UNSPECIFIED + } + } else { + controller.show(androidx.core.view.WindowInsetsCompat.Type.systemBars()) + activity.requestedOrientation = + android.content.pm.ActivityInfo.SCREEN_ORIENTATION_UNSPECIFIED + } + ret.put("ok", true) + } catch (e: Throwable) { + Log.w("YtDlpPlugin", "setImmersive failed", e) + ret.put("ok", false) + ret.put("message", e.message ?: e.javaClass.simpleName) + } + invoke.resolve(ret) + } + } + /** Schedule WorkManager to wake the app when network is available. */ @Command fun schedulePendingResume(invoke: Invoke) { @@ -453,7 +605,7 @@ class YtDlpPlugin(private val activity: Activity) : Plugin(activity) { val ret = JSObject() ret.put("ok", true) invoke.resolve(ret) - } catch (e: Exception) { + } catch (e: Throwable) { val ret = JSObject() ret.put("ok", false) ret.put("message", e.message ?: "schedulePendingResume failed") @@ -468,7 +620,7 @@ class YtDlpPlugin(private val activity: Activity) : Plugin(activity) { val ret = JSObject() ret.put("ok", true) invoke.resolve(ret) - } catch (e: Exception) { + } catch (e: Throwable) { val ret = JSObject() ret.put("ok", false) ret.put("message", e.message ?: "cancelPendingResume failed") @@ -533,6 +685,16 @@ class YtDlpPlugin(private val activity: Activity) : Plugin(activity) { var requireUnmetered: Boolean = false, ) + class ImmersiveArgs @JvmOverloads constructor( + var enabled: Boolean = false, + var landscape: Boolean = true, + ) + + class OpenFileArgs @JvmOverloads constructor( + var path: String = "", + var mime: String = "", + ) + class ShareFileArgs @JvmOverloads constructor( var filename: String = "archhive-backup.json", var text: String = "", diff --git a/src-tauri/android-overlays/file_paths.xml b/src-tauri/android-overlays/file_paths.xml new file mode 100644 index 0000000..a99fa9b --- /dev/null +++ b/src-tauri/android-overlays/file_paths.xml @@ -0,0 +1,10 @@ + + + + + + + + diff --git a/src-tauri/android-overlays/proguard-ytdlp.pro b/src-tauri/android-overlays/proguard-ytdlp.pro index 49701ac..02c7780 100644 --- a/src-tauri/android-overlays/proguard-ytdlp.pro +++ b/src-tauri/android-overlays/proguard-ytdlp.pro @@ -8,6 +8,12 @@ -keep class com.archhive.app.** { *; } -keepattributes Signature,InnerClasses,EnclosingMethod +# Plugin argument classes (YtDlpPlugin$Args, $MediaArgs, CookieBrowserPlugin$OpenArgs, ...): +# Tauri builds them reflectively from JSON via Jackson, so R8 must keep their +# no-arg constructors and fields. Without this, release builds fail with +# "Cannot construct instance of YtDlpPlugin$Args (no Creators ...)". +-keep class com.archhive.app.**Args { *; } + # Tauri plugin framework (annotations + Plugin base used by register_android_plugin) -keep class app.tauri.** { *; } -keep @app.tauri.annotation.TauriPlugin class * { *; } @@ -26,3 +32,10 @@ # with "class ...AsiExtraField is not a concrete class" (seen v0.6.0, OnePlus 7T). -keep class org.apache.commons.compress.** { *; } -dontwarn org.apache.commons.compress.** +# ...and the libraries it drives reflectively during YoutubeDL.init / +# FFmpeg.init (zip unpacking, JSON). +-keep class org.apache.commons.io.** { *; } +-keep class com.fasterxml.jackson.** { *; } +-keepattributes Signature,*Annotation*,InnerClasses,EnclosingMethod +-dontwarn org.apache.commons.io.** +-dontwarn com.fasterxml.jackson.** diff --git a/src-tauri/src/commands.rs b/src-tauri/src/commands.rs index 0e0ea08..40791a6 100644 --- a/src-tauri/src/commands.rs +++ b/src-tauri/src/commands.rs @@ -38,10 +38,42 @@ pub async fn browse( slug: String, page: Option, orientation: Option, + filters: Option, ) -> CmdResult { map_err( state - .browse(&site_id, kind, &slug, page.unwrap_or(1), orientation) + .browse( + &site_id, + kind, + &slug, + page.unwrap_or(1), + orientation, + filters.unwrap_or_default(), + ) + .await, + ) +} + +#[tauri::command] +pub async fn site_suggest( + state: State<'_, Arc>, + site_id: String, + text: String, + orientation: Option, +) -> CmdResult> { + map_err(state.site_suggest(&site_id, &text, orientation).await) +} + +#[tauri::command] +pub async fn list_site_categories( + state: State<'_, Arc>, + site_id: String, + group: Option, + refresh: Option, +) -> CmdResult> { + map_err( + state + .list_site_categories(&site_id, group.as_deref(), refresh.unwrap_or(false)) .await, ) } @@ -417,6 +449,73 @@ pub fn save_site_cookies( map_err(state.save_site_cookies(&site_id, &cookies)) } +/// Open the in-app login browser for a site and return the cookies captured +/// once the user finishes (the frontend stores them via `save_site_cookies`, +/// which also covers Remote LAN hosts). Async: reading webview cookies from a +/// synchronous command deadlocks on Windows. +#[tauri::command] +pub async fn cookie_login( + app: AppHandle, + state: State<'_, Arc>, + site_id: String, + url: Option, +) -> CmdResult { + // Per-domain entries (`host-`, any site without an adapter) take + // the site's URL from the caller; it must belong to that entry. + let (display_name, base_url) = if crate::sites::cookie_keys::is_host_key(&site_id) { + let url = url + .filter(|u| u.starts_with("https://") || u.starts_with("http://")) + .ok_or_else(|| "A site URL is required for this cookie entry.".to_string())?; + if crate::sites::cookie_keys::host_key(&url).as_deref() != Some(site_id.as_str()) { + return Err(format!("{url} does not belong to cookie entry {site_id}")); + } + let host = url::Url::parse(&url) + .ok() + .and_then(|u| u.host_str().map(str::to_string)) + .unwrap_or_else(|| site_id.clone()); + (host, url) + } else { + map_err(state.site_login_info(&site_id))? + }; + let target = crate::cookie_login::login_target(&site_id, &base_url); + map_err(crate::cookie_login::capture(&app, &site_id, &display_name, &target).await) +} + +/// Desktop: save (`save = true`) or cancel the open login window. +#[tauri::command] +pub fn cookie_login_finish(app: AppHandle, save: bool) -> CmdResult<()> { + map_err(crate::cookie_login::finish(&app, save)) +} + +/// Android: open a page in the in-app browser (not the system browser). +#[tauri::command] +pub async fn inapp_browse( + app: tauri::AppHandle, + url: String, + title: Option, +) -> CmdResult<()> { + map_err(crate::cookie_login::browse(&app, &url, title.as_deref().unwrap_or("")).await) +} + +/// Android: share a URL, or a library file (`path`, `mime`) to other apps. +#[tauri::command] +pub async fn android_share( + app: tauri::AppHandle, + text: Option, + path: Option, + mime: Option, +) -> CmdResult<()> { + map_err( + crate::cookie_login::share( + &app, + text.as_deref().unwrap_or(""), + path.as_deref().unwrap_or(""), + mime.as_deref().unwrap_or(""), + ) + .await, + ) +} + #[tauri::command] pub fn delete_site_cookies(state: State<'_, Arc>, site_id: String) -> CmdResult<()> { map_err(state.delete_site_cookies(&site_id)) @@ -531,14 +630,23 @@ pub async fn open_scene_with_default( state: State<'_, Arc>, id: String, ) -> CmdResult<()> { - use tauri_plugin_opener::OpenerExt; let scene = map_err(state.get_scene(&id))?; let path = scene .path .ok_or_else(|| "Scene has no file on disk".to_string())?; - app.opener() - .open_path(&path, None::<&str>) - .map_err(|e| e.to_string()) + // The opener plugin's open_path does nothing useful on Android (needs a + // content:// URI); hand the file to other apps via our FileProvider. + #[cfg(target_os = "android")] + { + map_err(crate::mobile::ytdlp_bridge::open_file(&app, &path)) + } + #[cfg(not(target_os = "android"))] + { + use tauri_plugin_opener::OpenerExt; + app.opener() + .open_path(&path, None::<&str>) + .map_err(|e| e.to_string()) + } } #[tauri::command] @@ -943,11 +1051,12 @@ pub fn browse_dirs( map_err(state.browse_dirs(path.as_deref())) } -/// Resolve the default on-device download folder (`/downloads`, -/// creating it if needed). Used by the Settings folder picker "Reset" action. +/// Resolve the default on-device download folder (Android: shared +/// `Download/ArcHive` when writable; else `/downloads`), creating +/// it if needed. Used by the Settings folder picker "Reset" action. #[tauri::command] pub fn default_library_dir(state: State<'_, Arc>) -> CmdResult { - let dir = state.data_dir.join("downloads"); + let dir = crate::default_library_dir(&state.data_dir); std::fs::create_dir_all(&dir).map_err(|e| e.to_string())?; Ok(dir.to_string_lossy().to_string()) } @@ -1186,12 +1295,14 @@ pub async fn query_stashdb_for_scene( scene_id: String, endpoint_id: Option, fingerprint_only: Option, + min_score: Option, ) -> CmdResult> { state .query_stashdb_for_scene( &scene_id, endpoint_id.as_deref(), fingerprint_only.unwrap_or(false), + min_score, ) .await .map_err(|e| e.to_string()) @@ -1211,9 +1322,10 @@ pub async fn search_stashdb_scenes( state: State<'_, Arc>, title: String, endpoint_id: Option, + scene_id: Option, ) -> CmdResult> { state - .search_stashdb_scenes(&title, endpoint_id.as_deref()) + .search_stashdb_scenes(&title, endpoint_id.as_deref(), scene_id.as_deref()) .await .map_err(|e| e.to_string()) } @@ -1289,3 +1401,154 @@ pub async fn load_demo_scene(state: State<'_, Arc>) -> CmdResult CmdResult<()> { + map_err(crate::mobile::ytdlp_bridge::set_immersive( + &app, enabled, landscape, + )) +} + +/// Android "All files access" (deleting library files in shared storage). +/// Always true on other platforms. +#[tauri::command] +pub fn storage_access_status(app: AppHandle) -> CmdResult { + map_err(crate::mobile::ytdlp_bridge::storage_access(&app)) +} + +/// Android: open the system "All files access" screen for ArcHive. +#[tauri::command] +pub fn open_storage_access_settings(app: AppHandle) -> CmdResult<()> { + map_err(crate::mobile::ytdlp_bridge::open_storage_access_settings( + &app, + )) +} + +// ---- Online library (local copy of listing results) ---- + +#[tauri::command] +pub fn query_online_items( + state: State<'_, Arc>, + query: crate::models::OnlineQuery, +) -> CmdResult { + map_err(state.db.query_online_items(&query)) +} + +#[tauri::command] +pub fn online_tag_counts( + state: State<'_, Arc>, + search_id: Option, + limit: Option, +) -> CmdResult> { + map_err( + state + .db + .online_tag_counts(search_id.as_deref(), limit.unwrap_or(400)), + ) +} + +#[tauri::command] +pub fn online_search_info( + state: State<'_, Arc>, +) -> CmdResult> { + map_err(state.db.online_search_info()) +} + +#[tauri::command] +pub async fn update_online_search( + state: State<'_, Arc>, + id: String, + pages: Option, +) -> CmdResult { + state + .update_online_search(&id, pages.unwrap_or(3)) + .await + .map_err(|e| e.to_string()) +} + +#[tauri::command] +pub async fn fetch_online_details( + state: State<'_, Arc>, + keys: Vec, +) -> CmdResult { + state + .fetch_online_details(&keys) + .await + .map_err(|e| e.to_string()) +} + +#[tauri::command] +pub fn clear_online_history(state: State<'_, Arc>) -> CmdResult { + map_err(state.db.clear_online_history()) +} + +#[tauri::command] +pub fn delete_online_items( + state: State<'_, Arc>, + search_id: Option, + keys: Vec, +) -> CmdResult { + map_err(state.db.delete_online_items(search_id.as_deref(), &keys)) +} + +/// Bookmark a browsed video (adds it to the Online library's Bookmarks). +#[tauri::command] +pub fn online_bookmark( + state: State<'_, Arc>, + item: crate::models::MediaItem, +) -> CmdResult { + map_err(state.db.bookmark_online_item(&item)) +} + +#[tauri::command] +pub fn online_unbookmark(state: State<'_, Arc>, url: String) -> CmdResult<()> { + map_err( + state + .db + .unbookmark_online_item(&crate::sites::urls::video_key(&url)), + ) +} + +#[tauri::command] +pub fn online_is_bookmarked(state: State<'_, Arc>, url: String) -> CmdResult { + map_err( + state + .db + .is_online_bookmarked(&crate::sites::urls::video_key(&url)), + ) +} + +#[tauri::command] +pub fn delete_online_matching( + state: State<'_, Arc>, + query: crate::models::OnlineQuery, +) -> CmdResult { + map_err(state.db.delete_online_matching(&query)) +} + +#[tauri::command] +pub fn start_full_fetch( + state: State<'_, Arc>, + id: String, + restart: Option, +) -> CmdResult { + map_err( + state + .inner() + .start_full_fetch(&id, restart.unwrap_or(false)), + ) +} + +#[tauri::command] +pub fn stop_full_fetch(state: State<'_, Arc>, id: String) -> CmdResult<()> { + map_err(state.stop_full_fetch(&id)) +} + +#[tauri::command] +pub fn list_full_fetches( + state: State<'_, Arc>, +) -> CmdResult> { + map_err(state.db.list_online_crawls()) +} diff --git a/src-tauri/src/cookie_login.rs b/src-tauri/src/cookie_login.rs new file mode 100644 index 0000000..7b2ff92 --- /dev/null +++ b/src-tauri/src/cookie_login.rs @@ -0,0 +1,750 @@ +//! In-app browser login that captures a site's cookies. +//! +//! Instead of exporting cookies from an external browser, the user logs in +//! inside a browser ArcHive controls and the resulting cookies (including +//! HttpOnly session cookies that page JavaScript cannot see) are converted to +//! Netscape format for the cookie vault / yt-dlp `--cookies`. +//! +//! * Desktop: a secondary `WebviewWindow` (no IPC capability) is opened on the +//! site; closing it, or pressing "Save cookies" in the app, reads the +//! cookies via `Webview::cookies_for_url`. +//! * Android: the `CookieBrowserPlugin` Kotlin overlay shows a full-screen +//! WebView and returns `CookieManager` cookie headers when the user taps +//! "Save cookies". Other apps' cookie jars (e.g. Chrome's) are not readable +//! on Android, which is why the login happens in-app. + +use crate::error::{AppError, AppResult}; +use serde::Serialize; +use tauri::plugin::{Builder, TauriPlugin}; +use tauri::{AppHandle, Runtime}; + +/// Lifetime given to session / undated cookies so yt-dlp and the vault keep them. +const SESSION_COOKIE_TTL_SECS: i64 = 30 * 24 * 3600; +/// Fallback lifetime for header-only cookies (Android `CookieManager` hides expiry). +#[cfg_attr(not(target_os = "android"), allow(dead_code))] +const HEADER_COOKIE_TTL_SECS: i64 = 365 * 24 * 3600; + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct CapturedCookie { + pub domain: String, + pub include_subdomains: bool, + pub path: String, + pub secure: bool, + pub expires: i64, + pub name: String, + pub value: String, +} + +#[derive(Debug, Clone, Serialize)] +pub struct CookieLoginResult { + pub site_id: String, + /// Netscape cookie file contents, ready for `save_site_cookies`. + pub netscape: String, + pub count: usize, +} + +/// A video page as rendered by the in-app browser after any bot check +/// (Android `CookieBrowserPlugin.resolvePage`). +#[derive(Debug, Clone, serde::Deserialize)] +#[cfg_attr(not(target_os = "android"), allow(dead_code))] +pub struct PageCapture { + pub html: String, + /// Final page URL (after redirects). + pub url: String, + #[serde(rename = "userAgent", default)] + pub user_agent: String, + /// `Cookie:` header for the page URL. + #[serde(default)] + pub cookies: String, +} + +/// Where a link led inside the in-app browser session. +#[derive(Debug, Clone, serde::Deserialize)] +#[cfg_attr(not(target_os = "android"), allow(dead_code))] +pub struct FollowResult { + pub url: String, + #[serde(default)] + pub cookies: String, +} + +/// Open `url` in the in-app browser until no bot check is showing and +/// return the rendered page. The browser stays open for [`follow_in_page`] +/// until [`close_page`]. +#[cfg(target_os = "android")] +pub use platform::{close_page, follow as follow_in_page, open_page}; + +/// Desktop: pass a bot check in a browser window and return its cookies +/// (see `sites::clearance`). +#[cfg(desktop)] +pub use platform::clear_bot_check; + +/// Where the login browser starts and which URLs' cookies to collect. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct LoginTarget { + pub start_url: String, + pub cookie_urls: Vec, +} + +pub fn login_target(site_id: &str, base_url: &str) -> LoginTarget { + let base = format!("{}/", base_url.trim_end_matches('/')); + let (start, extra): (&str, &[&str]) = match site_id { + "reddit" => ( + "https://www.reddit.com/login/", + &[ + "https://www.reddit.com/", + "https://old.reddit.com/", + "https://reddit.com/", + ], + ), + "twitter" => ( + "https://x.com/i/flow/login", + &["https://x.com/", "https://twitter.com/"], + ), + "instagram" => ("https://www.instagram.com/accounts/login/", &[]), + "tiktok" => ("https://www.tiktok.com/login", &[]), + "youtube" => ( + "https://accounts.google.com/ServiceLogin?service=youtube&continue=https%3A%2F%2Fwww.youtube.com%2F", + &["https://www.youtube.com/"], + ), + "redgifs" => ("https://www.redgifs.com/login", &[]), + _ => ("", &[]), + }; + let mut cookie_urls = vec![base.clone()]; + for u in extra { + if !cookie_urls.iter().any(|c| c == u) { + cookie_urls.push((*u).to_string()); + } + } + LoginTarget { + start_url: if start.is_empty() { + base + } else { + start.to_string() + }, + cookie_urls, + } +} + +/// Netscape cookie file text. HttpOnly cookies are written as ordinary lines +/// (the `#HttpOnly_` prefix would be dropped by simple parsers). +pub fn to_netscape(cookies: &[CapturedCookie]) -> String { + let mut out = + String::from("# Netscape HTTP Cookie File\n# Captured by ArcHive login browser\n\n"); + for c in cookies { + out.push_str(&format!( + "{}\t{}\t{}\t{}\t{}\t{}\t{}\n", + c.domain, + if c.include_subdomains { + "TRUE" + } else { + "FALSE" + }, + c.path, + if c.secure { "TRUE" } else { "FALSE" }, + c.expires, + c.name, + c.value + )); + } + out +} + +/// Drop duplicates (same domain/path/name) keeping the first occurrence, and +/// cookies whose name/value would corrupt the tab-separated format. +pub fn dedupe(cookies: Vec) -> Vec { + let mut seen = std::collections::HashSet::new(); + cookies + .into_iter() + .filter(|c| { + !c.name.is_empty() + && !c.name.contains(['\t', '\n', '\r', ';', '=']) + && !c.value.contains(['\t', '\n', '\r']) + }) + .filter(|c| seen.insert((c.domain.clone(), c.path.clone(), c.name.clone()))) + .collect() +} + +/// Cookies from a `Cookie:`-style header (`a=1; b=2`) captured for `url`. +/// The header carries no domain/expiry, so cookies are scoped to the site's +/// registrable-ish host (`www.` stripped, subdomains included). +#[cfg_attr(not(target_os = "android"), allow(dead_code))] +pub fn cookies_from_header(url: &str, header: &str, now: i64) -> Vec { + let Ok(parsed) = url::Url::parse(url) else { + return Vec::new(); + }; + let Some(host) = parsed.host_str() else { + return Vec::new(); + }; + let domain = format!(".{}", host.trim_start_matches("www.")); + let secure = parsed.scheme() == "https"; + header + .split(';') + .filter_map(|pair| { + let (name, value) = pair.split_once('=')?; + let name = name.trim(); + (!name.is_empty()).then(|| CapturedCookie { + domain: domain.clone(), + include_subdomains: true, + path: "/".to_string(), + secure, + expires: now + HEADER_COOKIE_TTL_SECS, + name: name.to_string(), + value: value.trim().to_string(), + }) + }) + .collect() +} + +fn now_unix() -> i64 { + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.as_secs() as i64) + .unwrap_or(0) +} + +/// Open the login browser for `site_id`, wait for the user to finish, and +/// return the captured cookies as a Netscape file. +pub async fn capture( + app: &AppHandle, + site_id: &str, + display_name: &str, + target: &LoginTarget, +) -> AppResult { + let title = format!("Log in to {display_name}, then save cookies"); + let cookies = dedupe(platform::capture(app, &title, target).await?); + if cookies.is_empty() { + return Err(AppError::Site(format!( + "No cookies were captured for {display_name}. Make sure the page finished loading and you are logged in." + ))); + } + Ok(CookieLoginResult { + site_id: site_id.to_string(), + count: cookies.len(), + netscape: to_netscape(&cookies), + }) +} + +/// Desktop: finish (save=true) or cancel the open login window. +pub fn finish(app: &AppHandle, save: bool) -> AppResult<()> { + platform::finish(app, save) +} + +pub fn init() -> TauriPlugin { + Builder::new("cookie-browser") + .setup(|app, api| { + platform::setup(app, api); + Ok(()) + }) + .build() +} + +#[cfg(desktop)] +mod platform { + use super::*; + use std::sync::Mutex; + use tauri::{Manager, WebviewUrl, WebviewWindowBuilder, WindowEvent}; + use tokio::sync::mpsc; + + const LABEL: &str = "cookie-login"; + + #[derive(Default)] + pub struct DesktopLoginState { + signal: Mutex>>, + } + + pub fn setup(app: &AppHandle, _api: tauri::plugin::PluginApi) { + app.manage(DesktopLoginState::default()); + } + + fn send(app: &AppHandle, save: bool) -> bool { + app.try_state::() + .and_then(|s| s.signal.lock().ok().and_then(|g| g.clone())) + .is_some_and(|tx| tx.send(save).is_ok()) + } + + pub fn finish(app: &AppHandle, save: bool) -> AppResult<()> { + if send(app, save) { + Ok(()) + } else { + Err(AppError::Other("No login window is open.".into())) + } + } + + /// Every cookie the window holds for `urls`, HttpOnly ones included. + fn window_cookies(window: &tauri::WebviewWindow, urls: &[String]) -> Vec { + let mut out = Vec::new(); + let now = now_unix(); + for u in urls { + let Ok(parsed) = url::Url::parse(u) else { + continue; + }; + let host = parsed.host_str().unwrap_or_default().to_string(); + // Must run off the main thread (Windows deadlocks otherwise); + // this is an async command task, so that holds. + let cookies = match window.cookies_for_url(parsed) { + Ok(c) => c, + Err(e) => { + tracing::warn!("[cookie-login] cookies_for_url({u}) failed: {e}"); + continue; + } + }; + for c in cookies { + let (domain, include_subdomains) = match c.domain() { + Some(d) if !d.is_empty() => { + let d = if d.starts_with('.') || d == host { + d.to_string() + } else { + format!(".{d}") + }; + let sub = d.starts_with('.'); + (d, sub) + } + _ => (host.clone(), false), + }; + let expires = c + .expires_datetime() + .map(|t| t.unix_timestamp()) + .filter(|t| *t > now) + .unwrap_or(now + SESSION_COOKIE_TTL_SECS); + out.push(CapturedCookie { + domain, + include_subdomains, + path: c + .path() + .filter(|p| !p.is_empty()) + .unwrap_or("/") + .to_string(), + secure: c.secure().unwrap_or(false), + expires, + name: c.name().to_string(), + value: c.value().to_string(), + }); + } + } + out + } + + /// Show `url` in a small window (sending `user_agent`) until no bot + /// check is showing, then return its cookies. Closes by itself; closing + /// it by hand cancels. + pub async fn clear_bot_check( + app: &AppHandle, + url: &str, + host: &str, + user_agent: &str, + ) -> AppResult> { + use std::sync::atomic::{AtomicBool, Ordering}; + use std::sync::Arc; + // A fresh label per window: destroying the previous one is async, so + // reusing a label can fail with "webview already exists". + static SEQ: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); + let label = format!("bot-check-{}", SEQ.fetch_add(1, Ordering::Relaxed)); + let start: url::Url = url + .parse() + .map_err(|e| AppError::InvalidInput(format!("bad URL: {e}")))?; + let title = Arc::new(Mutex::new(String::new())); + let loaded = Arc::new(AtomicBool::new(false)); + let closed = Arc::new(AtomicBool::new(false)); + let (t, l) = (title.clone(), loaded.clone()); + let window = WebviewWindowBuilder::new(app, &label, WebviewUrl::External(start)) + .title(format!("Verifying {host} — closes by itself")) + .inner_size(720.0, 640.0) + .user_agent(user_agent) + .focused(true) + .on_document_title_changed(move |_, new_title| { + if let Ok(mut g) = t.lock() { + *g = new_title; + } + }) + .on_page_load(move |_, payload| { + l.store( + matches!(payload.event(), tauri::webview::PageLoadEvent::Finished), + Ordering::SeqCst, + ); + }) + .build() + .map_err(|e| AppError::Other(format!("could not open the browser window: {e}")))?; + let c = closed.clone(); + window.on_window_event(move |event| { + if let WindowEvent::CloseRequested { api, .. } = event { + api.prevent_close(); + c.store(true, Ordering::SeqCst); + } + }); + + let started = std::time::Instant::now(); + let mut passed = false; + // ARCHHIVE_BOT_CHECK_TIMEOUT_SECS: how long to wait for the check + // to pass (default 150 s). + let limit = std::env::var("ARCHHIVE_BOT_CHECK_TIMEOUT_SECS") + .ok() + .and_then(|v| v.parse::().ok()) + .unwrap_or(150); + while started.elapsed() < std::time::Duration::from_secs(limit) { + tokio::time::sleep(std::time::Duration::from_millis(500)).await; + if closed.load(Ordering::SeqCst) { + break; + } + let current = title.lock().map(|g| g.clone()).unwrap_or_default(); + if loaded.load(Ordering::SeqCst) + && !current.trim().is_empty() + && !crate::sites::challenge::is_challenge_title(¤t) + { + // Let the cleared page settle its cookies. + tokio::time::sleep(std::time::Duration::from_millis(1200)).await; + passed = true; + break; + } + } + let cookies = if passed { + let page = window + .url() + .map(|u| u.to_string()) + .unwrap_or_else(|_| url.to_string()); + let mut urls = vec![page]; + if let Ok(u) = url::Url::parse(url) { + urls.push(format!( + "{}://{}/", + u.scheme(), + u.host_str().unwrap_or_default() + )); + } + window_cookies(&window, &urls) + } else { + Vec::new() + }; + let _ = window.destroy(); + if !passed { + return Err(AppError::Other(if closed.load(Ordering::SeqCst) { + "Verification window closed.".into() + } else { + format!("{host} kept showing its bot check.") + })); + } + Ok(dedupe(cookies)) + } + + pub async fn capture( + app: &AppHandle, + title: &str, + target: &LoginTarget, + ) -> AppResult> { + if let Some(existing) = app.get_webview_window(LABEL) { + send(app, false); + let _ = existing.destroy(); + } + let start: url::Url = target + .start_url + .parse() + .map_err(|e| AppError::InvalidInput(format!("bad login url: {e}")))?; + + let (tx, mut rx) = mpsc::unbounded_channel::(); + let state = app + .try_state::() + .ok_or_else(|| AppError::Other("cookie-browser plugin not initialized".into()))?; + if let Ok(mut slot) = state.signal.lock() { + *slot = Some(tx.clone()); + } + + let window = WebviewWindowBuilder::new(app, LABEL, WebviewUrl::External(start)) + .title(title) + .inner_size(1024.0, 780.0) + .focused(true) + .build() + .map_err(|e| AppError::Other(format!("could not open login window: {e}")))?; + // Closing the window means "done": keep it alive until cookies are read. + window.on_window_event(move |event| { + if let WindowEvent::CloseRequested { api, .. } = event { + api.prevent_close(); + let _ = tx.send(true); + } + }); + + let save = rx.recv().await.unwrap_or(false); + if let Ok(mut slot) = state.signal.lock() { + *slot = None; + } + + let out = if save { + window_cookies(&window, &target.cookie_urls) + } else { + Vec::new() + }; + let _ = window.destroy(); + if !save { + return Err(AppError::Other("Login cancelled.".into())); + } + Ok(out) + } +} + +#[cfg(target_os = "android")] +mod platform { + use super::*; + use serde::Deserialize; + use tauri::plugin::PluginHandle; + use tauri::Manager; + + pub struct AndroidLoginState { + handle: Option>, + } + + #[derive(Deserialize)] + struct Entry { + url: String, + header: String, + } + + #[derive(Deserialize)] + struct Response { + #[serde(default)] + cookies: Vec, + } + + pub fn setup(app: &AppHandle, api: tauri::plugin::PluginApi) { + let handle = match api.register_android_plugin("com.archhive.app", "CookieBrowserPlugin") { + Ok(h) => Some(h), + Err(e) => { + tracing::warn!("CookieBrowserPlugin registration failed: {e}"); + None + } + }; + app.manage(AndroidLoginState { handle }); + } + + pub fn finish(_app: &AppHandle, _save: bool) -> AppResult<()> { + // The native dialog has its own Save / Cancel buttons. + Ok(()) + } + + pub async fn capture( + app: &AppHandle, + title: &str, + target: &LoginTarget, + ) -> AppResult> { + let handle = app + .try_state::>() + .and_then(|s| s.handle.clone()) + .ok_or_else(|| { + AppError::Other( + "Login browser unavailable (CookieBrowserPlugin not registered). Re-run android:patches and rebuild the APK." + .into(), + ) + })?; + let payload = serde_json::json!({ + "url": target.start_url, + "cookieUrls": target.cookie_urls, + "title": title, + }); + // Blocks until the user taps Save / Cancel: keep it off the async workers. + let response: Response = tokio::task::spawn_blocking(move || { + handle.run_mobile_plugin::("open", payload) + }) + .await + .map_err(|e| AppError::Other(format!("login browser task failed: {e}")))? + .map_err(|e| { + let msg = e.to_string(); + if msg.contains("cancelled") { + AppError::Other("Login cancelled.".into()) + } else { + AppError::Other(format!("Login browser failed: {msg}")) + } + })?; + let now = now_unix(); + Ok(response + .cookies + .iter() + .flat_map(|e| cookies_from_header(&e.url, &e.header, now)) + .collect()) + } + + fn plugin(app: &AppHandle) -> AppResult> { + app.try_state::>() + .and_then(|s| s.handle.clone()) + .ok_or_else(|| AppError::Other("In-app browser unavailable on this build.".into())) + } + + async fn call( + app: &AppHandle, + command: &'static str, + payload: serde_json::Value, + ) -> AppResult { + let handle = plugin(app)?; + tokio::task::spawn_blocking(move || handle.run_mobile_plugin::(command, payload)) + .await + .map_err(|e| AppError::Other(format!("in-app browser task failed: {e}")))? + .map_err(|e| { + let msg = e.to_string(); + if msg.contains("cancelled") { + AppError::Other("Cancelled in the in-app browser.".into()) + } else { + AppError::Other(format!("In-app browser: {msg}")) + } + }) + } + + pub async fn open_page(app: &AppHandle, url: &str, title: &str) -> AppResult { + call( + app, + "resolvePage", + serde_json::json!({ "url": url, "title": title }), + ) + .await + } + + pub async fn follow(app: &AppHandle, url: &str, referer: &str) -> AppResult { + call( + app, + "followRedirect", + serde_json::json!({ "url": url, "referer": referer }), + ) + .await + } + + pub async fn close_page(app: &AppHandle) { + let _ = call::(app, "closeResolver", serde_json::json!({})).await; + } + + /// In-app browser for watching a page (ad blocking, per-site clearing, + /// share); returns when the user closes it. + pub async fn browse(app: &AppHandle, url: &str, title: &str) -> AppResult<()> { + call::( + app, + "browse", + serde_json::json!({ "url": url, "title": title, "adblock": true }), + ) + .await + .map(|_| ()) + } + + /// Android share sheet for a URL / text or a local file. + pub async fn share(app: &AppHandle, text: &str, path: &str, mime: &str) -> AppResult<()> { + call::( + app, + "share", + serde_json::json!({ "text": text, "path": path, "mime": mime }), + ) + .await + .map(|_| ()) + } +} + +#[cfg(all(mobile, not(target_os = "android")))] +mod platform { + use super::*; + + pub fn setup(_app: &AppHandle, _api: tauri::plugin::PluginApi) {} + + pub fn finish(_app: &AppHandle, _save: bool) -> AppResult<()> { + Ok(()) + } + + pub async fn capture( + _app: &AppHandle, + _title: &str, + _target: &LoginTarget, + ) -> AppResult> { + Err(AppError::Other( + "The login browser is not available on this platform yet.".into(), + )) + } +} + +/// In-app browser (Android) for a page the app can't play itself. +pub async fn browse(app: &AppHandle, url: &str, title: &str) -> AppResult<()> { + #[cfg(target_os = "android")] + { + platform::browse(app, url, title).await + } + #[cfg(not(target_os = "android"))] + { + let _ = (app, url, title); + Err(AppError::Other( + "The in-app browser is only available on Android.".into(), + )) + } +} + +/// Android share sheet: `text` (a URL) or `path` (a library file, `mime`). +pub async fn share(app: &AppHandle, text: &str, path: &str, mime: &str) -> AppResult<()> { + #[cfg(target_os = "android")] + { + platform::share(app, text, path, mime).await + } + #[cfg(not(target_os = "android"))] + { + let _ = (app, text, path, mime); + Err(AppError::Other( + "Sharing is only available on Android.".into(), + )) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn header_cookies_scope_to_site_domain() { + let c = cookies_from_header( + "https://www.reddit.com/", + "reddit_session=abc; token_v2=x=y", + 100, + ); + assert_eq!(c.len(), 2); + assert_eq!(c[0].domain, ".reddit.com"); + assert!(c[0].include_subdomains && c[0].secure); + assert_eq!(c[0].expires, 100 + HEADER_COOKIE_TTL_SECS); + assert_eq!(c[1].name, "token_v2"); + assert_eq!(c[1].value, "x=y"); + } + + #[test] + fn netscape_output_round_trips_through_vault_format() { + let cookies = dedupe(vec![ + CapturedCookie { + domain: ".example.com".into(), + include_subdomains: true, + path: "/".into(), + secure: true, + expires: 2_000_000_000, + name: "sid".into(), + value: "v1".into(), + }, + CapturedCookie { + domain: ".example.com".into(), + include_subdomains: true, + path: "/".into(), + secure: true, + expires: 2_000_000_000, + name: "sid".into(), + value: "dupe".into(), + }, + CapturedCookie { + domain: "example.com".into(), + include_subdomains: false, + path: "/".into(), + secure: false, + expires: 1, + name: "bad\tname".into(), + value: "x".into(), + }, + ]); + assert_eq!(cookies.len(), 1); + let text = to_netscape(&cookies); + assert!(text.starts_with("# Netscape HTTP Cookie File")); + assert!(text.contains(".example.com\tTRUE\t/\tTRUE\t2000000000\tsid\tv1\n")); + } + + #[test] + fn login_targets() { + let r = login_target("reddit", "https://old.reddit.com"); + assert_eq!(r.start_url, "https://www.reddit.com/login/"); + assert!(r + .cookie_urls + .contains(&"https://old.reddit.com/".to_string())); + let x = login_target("xvideos", "https://www.xvideos.com"); + assert_eq!(x.start_url, "https://www.xvideos.com/"); + assert_eq!(x.cookie_urls, vec!["https://www.xvideos.com/".to_string()]); + } +} diff --git a/src-tauri/src/db/migrations.rs b/src-tauri/src/db/migrations.rs index c8934bb..0f72dbe 100644 --- a/src-tauri/src/db/migrations.rs +++ b/src-tauri/src/db/migrations.rs @@ -1,279 +1,356 @@ -pub const MIGRATION_001: &str = r#" -CREATE TABLE IF NOT EXISTS app_settings ( - key TEXT PRIMARY KEY, - value TEXT NOT NULL -); - -CREATE TABLE IF NOT EXISTS download_jobs ( - id TEXT PRIMARY KEY, - url TEXT NOT NULL, - adapter TEXT NOT NULL, - status TEXT NOT NULL, - progress REAL NOT NULL DEFAULT 0, - output_path TEXT, - error TEXT, - title TEXT, - created_at TEXT NOT NULL -); - -CREATE TABLE IF NOT EXISTS performers ( - id TEXT PRIMARY KEY, - name TEXT NOT NULL UNIQUE, - aliases TEXT NOT NULL DEFAULT '[]', - image TEXT, - favorite INTEGER NOT NULL DEFAULT 0 -); - -CREATE TABLE IF NOT EXISTS tags ( - id TEXT PRIMARY KEY, - name TEXT NOT NULL UNIQUE, - parent_id TEXT -); - -CREATE TABLE IF NOT EXISTS scenes ( - id TEXT PRIMARY KEY, - title TEXT NOT NULL, - path TEXT, - source_url TEXT, - thumb TEXT, - phash TEXT, - oshash TEXT, - duration INTEGER, - studio_id TEXT, - date TEXT, - rating INTEGER, - created_at TEXT NOT NULL -); - -CREATE TABLE IF NOT EXISTS scene_performers ( - scene_id TEXT NOT NULL, - performer_id TEXT NOT NULL, - PRIMARY KEY (scene_id, performer_id) -); - -CREATE TABLE IF NOT EXISTS scene_tags ( - scene_id TEXT NOT NULL, - tag_id TEXT NOT NULL, - PRIMARY KEY (scene_id, tag_id) -); - -CREATE VIRTUAL TABLE IF NOT EXISTS scenes_fts USING fts5( - title, - notes, - content='scenes', - content_rowid='rowid' -); - -CREATE TRIGGER IF NOT EXISTS scenes_ai AFTER INSERT ON scenes BEGIN - INSERT INTO scenes_fts(rowid, title, notes) VALUES (new.rowid, new.title, new.notes); -END; - -CREATE TRIGGER IF NOT EXISTS scenes_ad AFTER DELETE ON scenes BEGIN - INSERT INTO scenes_fts(scenes_fts, rowid, title, notes) VALUES('delete', old.rowid, old.title, old.notes); -END; - -CREATE TRIGGER IF NOT EXISTS scenes_au AFTER UPDATE ON scenes BEGIN - INSERT INTO scenes_fts(scenes_fts, rowid, title, notes) VALUES('delete', old.rowid, old.title, old.notes); - INSERT INTO scenes_fts(rowid, title, notes) VALUES (new.rowid, new.title, new.notes); -END; -"#; - -pub const MIGRATION_002: &str = r#" -CREATE TABLE IF NOT EXISTS site_cookies ( - site_id TEXT PRIMARY KEY, - encrypted_data BLOB NOT NULL, - updated_at TEXT NOT NULL -); -"#; - -pub const MIGRATION_003: &str = r#" -ALTER TABLE download_jobs ADD COLUMN metadata TEXT; -"#; - -pub const MIGRATION_004: &str = r#" -ALTER TABLE scenes ADD COLUMN channel TEXT; -"#; - -pub const MIGRATION_005: &str = r#" -ALTER TABLE scenes ADD COLUMN file_size INTEGER; -"#; - -pub const MIGRATION_006: &str = r#" -ALTER TABLE scenes ADD COLUMN notes TEXT; -"#; - -pub const MIGRATION_007: &str = r#" -ALTER TABLE download_jobs ADD COLUMN retry_count INTEGER NOT NULL DEFAULT 0; -ALTER TABLE download_jobs ADD COLUMN last_retry_at TEXT; -"#; - -pub const MIGRATION_008: &str = r#" -ALTER TABLE scenes ADD COLUMN width INTEGER; -ALTER TABLE scenes ADD COLUMN height INTEGER; -"#; - -pub const MIGRATION_009: &str = r#" -CREATE TABLE IF NOT EXISTS watch_history ( - scene_id TEXT PRIMARY KEY, - position_secs REAL NOT NULL DEFAULT 0, - duration_secs REAL NOT NULL DEFAULT 0, - watched INTEGER NOT NULL DEFAULT 0, - updated_at TEXT NOT NULL -); -"#; - -pub const MIGRATION_010: &str = r#" -CREATE TABLE IF NOT EXISTS saved_searches ( - id TEXT PRIMARY KEY, - name TEXT NOT NULL, - site_id TEXT NOT NULL, - kind TEXT NOT NULL, - slug TEXT NOT NULL, - orientation TEXT, - last_checked_at TEXT, - last_item_keys TEXT NOT NULL DEFAULT '[]', - new_count INTEGER NOT NULL DEFAULT 0, - created_at TEXT NOT NULL -); -"#; - -pub const MIGRATION_011: &str = r#" -ALTER TABLE saved_searches ADD COLUMN auto_queue INTEGER NOT NULL DEFAULT 0; -"#; - -pub const MIGRATION_012: &str = r#" -CREATE TABLE IF NOT EXISTS watchlist_poll_state ( - id INTEGER PRIMARY KEY CHECK (id = 1), - finished_at TEXT NOT NULL, - checked INTEGER NOT NULL DEFAULT 0, - queued INTEGER NOT NULL DEFAULT 0, - errors INTEGER NOT NULL DEFAULT 0 -); -"#; - -pub const MIGRATION_013: &str = r#" -DROP TRIGGER IF EXISTS scenes_ai; -DROP TRIGGER IF EXISTS scenes_ad; -DROP TRIGGER IF EXISTS scenes_au; -DROP TABLE IF EXISTS scenes_fts; - -CREATE VIRTUAL TABLE scenes_fts USING fts5( - title, - notes, - content='scenes', - content_rowid='rowid' -); - -CREATE TRIGGER scenes_ai AFTER INSERT ON scenes BEGIN - INSERT INTO scenes_fts(rowid, title, notes) VALUES (new.rowid, new.title, new.notes); -END; - -CREATE TRIGGER scenes_ad AFTER DELETE ON scenes BEGIN - INSERT INTO scenes_fts(scenes_fts, rowid, title, notes) VALUES('delete', old.rowid, old.title, old.notes); -END; - -CREATE TRIGGER scenes_au AFTER UPDATE ON scenes BEGIN - INSERT INTO scenes_fts(scenes_fts, rowid, title, notes) VALUES('delete', old.rowid, old.title, old.notes); - INSERT INTO scenes_fts(rowid, title, notes) VALUES (new.rowid, new.title, new.notes); -END; - -INSERT INTO scenes_fts(rowid, title, notes) -SELECT rowid, title, notes FROM scenes; -"#; - -pub const MIGRATION_014: &str = r#" -CREATE TABLE IF NOT EXISTS collections ( - id TEXT PRIMARY KEY, - name TEXT NOT NULL, - type TEXT NOT NULL DEFAULT 'collection', - description TEXT, - created_at TEXT NOT NULL, - updated_at TEXT NOT NULL -); - -CREATE TABLE IF NOT EXISTS collection_scenes ( - collection_id TEXT NOT NULL, - scene_id TEXT NOT NULL, - position INTEGER NOT NULL DEFAULT 0, - added_at TEXT NOT NULL, - PRIMARY KEY (collection_id, scene_id), - FOREIGN KEY (collection_id) REFERENCES collections(id) ON DELETE CASCADE, - FOREIGN KEY (scene_id) REFERENCES scenes(id) ON DELETE CASCADE -); - -CREATE INDEX IF NOT EXISTS idx_collection_scenes_scene ON collection_scenes(scene_id); -CREATE INDEX IF NOT EXISTS idx_collection_scenes_order ON collection_scenes(collection_id, position); -"#; - -pub const MIGRATION_015: &str = r#" -ALTER TABLE collections ADD COLUMN filter_json TEXT; - -CREATE INDEX IF NOT EXISTS idx_collections_type ON collections(type); -"#; - -pub const MIGRATION_016: &str = r#" -CREATE TABLE IF NOT EXISTS browse_cache ( - cache_key TEXT PRIMARY KEY, - site_id TEXT NOT NULL, - kind TEXT NOT NULL, - slug TEXT NOT NULL, - page INTEGER NOT NULL, - orientation TEXT, - payload TEXT NOT NULL, - fetched_at TEXT NOT NULL, - ttl_secs INTEGER NOT NULL DEFAULT 86400 -); - -CREATE INDEX IF NOT EXISTS idx_browse_cache_fetched ON browse_cache(fetched_at); -"#; - -/// V2 BodyMatch: performer body attributes (cup size + hair color). -/// Values are validated in Rust (see `normalize_cup_size` / `normalize_hair_color`); -/// `NULL` means "unset". Old DBs get the columns via `column_exists` gating in -/// `Database::new` (bundled SQLite may not support `ADD COLUMN IF NOT EXISTS`). -pub const MIGRATION_017: &str = r#" -ALTER TABLE performers ADD COLUMN cup_size TEXT; -ALTER TABLE performers ADD COLUMN hair_color TEXT; - -CREATE INDEX IF NOT EXISTS idx_performers_cup ON performers(cup_size); -CREATE INDEX IF NOT EXISTS idx_performers_hair ON performers(hair_color); -"#; - -/// StashDB enrichment: remote stash-box ids + local studio cache. -/// `scenes.studio_name` mirrors the matched studio (kept denormalized so -/// library lists don't need a join). `vault_secrets` holds stash-box API -/// keys encrypted (same AES-256-GCM cipher as cookies). -pub const MIGRATION_018: &str = r#" -ALTER TABLE scenes ADD COLUMN stash_id TEXT; -ALTER TABLE scenes ADD COLUMN stashdb_updated_at TEXT; -ALTER TABLE scenes ADD COLUMN studio_name TEXT; -ALTER TABLE performers ADD COLUMN stash_id TEXT; - -CREATE TABLE IF NOT EXISTS studios ( - id TEXT PRIMARY KEY, - name TEXT NOT NULL UNIQUE, - stash_id TEXT, - url TEXT, - image TEXT -); - -CREATE TABLE IF NOT EXISTS vault_secrets ( - key TEXT PRIMARY KEY, - encrypted_data BLOB NOT NULL, - updated_at TEXT NOT NULL -); - -CREATE INDEX IF NOT EXISTS idx_scenes_stash_id ON scenes(stash_id); -CREATE INDEX IF NOT EXISTS idx_performers_stash_id ON performers(stash_id); -CREATE INDEX IF NOT EXISTS idx_studios_stash_id ON studios(stash_id); -"#; - -/// StashDB fingerprint fix: `scenes.md5` (full-file MD5, StashDB's most common -/// fingerprint — previously never computed) plus cleanup of bogus `oshash` -/// rows. The old `compute_oshash` wrote a 64-char SHA-256 digest; real -/// OSHASH values are always exactly 16 hex chars, so anything else is -/// unreachable junk that would break batch Identify counts. -pub const MIGRATION_019: &str = r#" -ALTER TABLE scenes ADD COLUMN md5 TEXT; -UPDATE scenes SET oshash = NULL WHERE oshash IS NOT NULL AND length(oshash) != 16; -"#; +pub const MIGRATION_001: &str = r#" +CREATE TABLE IF NOT EXISTS app_settings ( + key TEXT PRIMARY KEY, + value TEXT NOT NULL +); + +CREATE TABLE IF NOT EXISTS download_jobs ( + id TEXT PRIMARY KEY, + url TEXT NOT NULL, + adapter TEXT NOT NULL, + status TEXT NOT NULL, + progress REAL NOT NULL DEFAULT 0, + output_path TEXT, + error TEXT, + title TEXT, + created_at TEXT NOT NULL +); + +CREATE TABLE IF NOT EXISTS performers ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL UNIQUE, + aliases TEXT NOT NULL DEFAULT '[]', + image TEXT, + favorite INTEGER NOT NULL DEFAULT 0 +); + +CREATE TABLE IF NOT EXISTS tags ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL UNIQUE, + parent_id TEXT +); + +CREATE TABLE IF NOT EXISTS scenes ( + id TEXT PRIMARY KEY, + title TEXT NOT NULL, + path TEXT, + source_url TEXT, + thumb TEXT, + phash TEXT, + oshash TEXT, + duration INTEGER, + studio_id TEXT, + date TEXT, + rating INTEGER, + created_at TEXT NOT NULL +); + +CREATE TABLE IF NOT EXISTS scene_performers ( + scene_id TEXT NOT NULL, + performer_id TEXT NOT NULL, + PRIMARY KEY (scene_id, performer_id) +); + +CREATE TABLE IF NOT EXISTS scene_tags ( + scene_id TEXT NOT NULL, + tag_id TEXT NOT NULL, + PRIMARY KEY (scene_id, tag_id) +); + +CREATE VIRTUAL TABLE IF NOT EXISTS scenes_fts USING fts5( + title, + notes, + content='scenes', + content_rowid='rowid' +); + +CREATE TRIGGER IF NOT EXISTS scenes_ai AFTER INSERT ON scenes BEGIN + INSERT INTO scenes_fts(rowid, title, notes) VALUES (new.rowid, new.title, new.notes); +END; + +CREATE TRIGGER IF NOT EXISTS scenes_ad AFTER DELETE ON scenes BEGIN + INSERT INTO scenes_fts(scenes_fts, rowid, title, notes) VALUES('delete', old.rowid, old.title, old.notes); +END; + +CREATE TRIGGER IF NOT EXISTS scenes_au AFTER UPDATE ON scenes BEGIN + INSERT INTO scenes_fts(scenes_fts, rowid, title, notes) VALUES('delete', old.rowid, old.title, old.notes); + INSERT INTO scenes_fts(rowid, title, notes) VALUES (new.rowid, new.title, new.notes); +END; +"#; + +pub const MIGRATION_002: &str = r#" +CREATE TABLE IF NOT EXISTS site_cookies ( + site_id TEXT PRIMARY KEY, + encrypted_data BLOB NOT NULL, + updated_at TEXT NOT NULL +); +"#; + +pub const MIGRATION_003: &str = r#" +ALTER TABLE download_jobs ADD COLUMN metadata TEXT; +"#; + +pub const MIGRATION_004: &str = r#" +ALTER TABLE scenes ADD COLUMN channel TEXT; +"#; + +pub const MIGRATION_005: &str = r#" +ALTER TABLE scenes ADD COLUMN file_size INTEGER; +"#; + +pub const MIGRATION_006: &str = r#" +ALTER TABLE scenes ADD COLUMN notes TEXT; +"#; + +pub const MIGRATION_007: &str = r#" +ALTER TABLE download_jobs ADD COLUMN retry_count INTEGER NOT NULL DEFAULT 0; +ALTER TABLE download_jobs ADD COLUMN last_retry_at TEXT; +"#; + +pub const MIGRATION_008: &str = r#" +ALTER TABLE scenes ADD COLUMN width INTEGER; +ALTER TABLE scenes ADD COLUMN height INTEGER; +"#; + +pub const MIGRATION_009: &str = r#" +CREATE TABLE IF NOT EXISTS watch_history ( + scene_id TEXT PRIMARY KEY, + position_secs REAL NOT NULL DEFAULT 0, + duration_secs REAL NOT NULL DEFAULT 0, + watched INTEGER NOT NULL DEFAULT 0, + updated_at TEXT NOT NULL +); +"#; + +pub const MIGRATION_010: &str = r#" +CREATE TABLE IF NOT EXISTS saved_searches ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL, + site_id TEXT NOT NULL, + kind TEXT NOT NULL, + slug TEXT NOT NULL, + orientation TEXT, + last_checked_at TEXT, + last_item_keys TEXT NOT NULL DEFAULT '[]', + new_count INTEGER NOT NULL DEFAULT 0, + created_at TEXT NOT NULL +); +"#; + +pub const MIGRATION_011: &str = r#" +ALTER TABLE saved_searches ADD COLUMN auto_queue INTEGER NOT NULL DEFAULT 0; +"#; + +pub const MIGRATION_012: &str = r#" +CREATE TABLE IF NOT EXISTS watchlist_poll_state ( + id INTEGER PRIMARY KEY CHECK (id = 1), + finished_at TEXT NOT NULL, + checked INTEGER NOT NULL DEFAULT 0, + queued INTEGER NOT NULL DEFAULT 0, + errors INTEGER NOT NULL DEFAULT 0 +); +"#; + +pub const MIGRATION_013: &str = r#" +DROP TRIGGER IF EXISTS scenes_ai; +DROP TRIGGER IF EXISTS scenes_ad; +DROP TRIGGER IF EXISTS scenes_au; +DROP TABLE IF EXISTS scenes_fts; + +CREATE VIRTUAL TABLE scenes_fts USING fts5( + title, + notes, + content='scenes', + content_rowid='rowid' +); + +CREATE TRIGGER scenes_ai AFTER INSERT ON scenes BEGIN + INSERT INTO scenes_fts(rowid, title, notes) VALUES (new.rowid, new.title, new.notes); +END; + +CREATE TRIGGER scenes_ad AFTER DELETE ON scenes BEGIN + INSERT INTO scenes_fts(scenes_fts, rowid, title, notes) VALUES('delete', old.rowid, old.title, old.notes); +END; + +CREATE TRIGGER scenes_au AFTER UPDATE ON scenes BEGIN + INSERT INTO scenes_fts(scenes_fts, rowid, title, notes) VALUES('delete', old.rowid, old.title, old.notes); + INSERT INTO scenes_fts(rowid, title, notes) VALUES (new.rowid, new.title, new.notes); +END; + +INSERT INTO scenes_fts(rowid, title, notes) +SELECT rowid, title, notes FROM scenes; +"#; + +pub const MIGRATION_014: &str = r#" +CREATE TABLE IF NOT EXISTS collections ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL, + type TEXT NOT NULL DEFAULT 'collection', + description TEXT, + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL +); + +CREATE TABLE IF NOT EXISTS collection_scenes ( + collection_id TEXT NOT NULL, + scene_id TEXT NOT NULL, + position INTEGER NOT NULL DEFAULT 0, + added_at TEXT NOT NULL, + PRIMARY KEY (collection_id, scene_id), + FOREIGN KEY (collection_id) REFERENCES collections(id) ON DELETE CASCADE, + FOREIGN KEY (scene_id) REFERENCES scenes(id) ON DELETE CASCADE +); + +CREATE INDEX IF NOT EXISTS idx_collection_scenes_scene ON collection_scenes(scene_id); +CREATE INDEX IF NOT EXISTS idx_collection_scenes_order ON collection_scenes(collection_id, position); +"#; + +pub const MIGRATION_015: &str = r#" +ALTER TABLE collections ADD COLUMN filter_json TEXT; + +CREATE INDEX IF NOT EXISTS idx_collections_type ON collections(type); +"#; + +pub const MIGRATION_016: &str = r#" +CREATE TABLE IF NOT EXISTS browse_cache ( + cache_key TEXT PRIMARY KEY, + site_id TEXT NOT NULL, + kind TEXT NOT NULL, + slug TEXT NOT NULL, + page INTEGER NOT NULL, + orientation TEXT, + payload TEXT NOT NULL, + fetched_at TEXT NOT NULL, + ttl_secs INTEGER NOT NULL DEFAULT 86400 +); + +CREATE INDEX IF NOT EXISTS idx_browse_cache_fetched ON browse_cache(fetched_at); +"#; + +/// V2 BodyMatch: performer body attributes (cup size + hair color). +/// Values are validated in Rust (see `normalize_cup_size` / `normalize_hair_color`); +/// `NULL` means "unset". Old DBs get the columns via `column_exists` gating in +/// `Database::new` (bundled SQLite may not support `ADD COLUMN IF NOT EXISTS`). +pub const MIGRATION_017: &str = r#" +ALTER TABLE performers ADD COLUMN cup_size TEXT; +ALTER TABLE performers ADD COLUMN hair_color TEXT; + +CREATE INDEX IF NOT EXISTS idx_performers_cup ON performers(cup_size); +CREATE INDEX IF NOT EXISTS idx_performers_hair ON performers(hair_color); +"#; + +/// StashDB enrichment: remote stash-box ids + local studio cache. +/// `scenes.studio_name` mirrors the matched studio (kept denormalized so +/// library lists don't need a join). `vault_secrets` holds stash-box API +/// keys encrypted (same AES-256-GCM cipher as cookies). +pub const MIGRATION_018: &str = r#" +ALTER TABLE scenes ADD COLUMN stash_id TEXT; +ALTER TABLE scenes ADD COLUMN stashdb_updated_at TEXT; +ALTER TABLE scenes ADD COLUMN studio_name TEXT; +ALTER TABLE performers ADD COLUMN stash_id TEXT; + +CREATE TABLE IF NOT EXISTS studios ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL UNIQUE, + stash_id TEXT, + url TEXT, + image TEXT +); + +CREATE TABLE IF NOT EXISTS vault_secrets ( + key TEXT PRIMARY KEY, + encrypted_data BLOB NOT NULL, + updated_at TEXT NOT NULL +); + +CREATE INDEX IF NOT EXISTS idx_scenes_stash_id ON scenes(stash_id); +CREATE INDEX IF NOT EXISTS idx_performers_stash_id ON performers(stash_id); +CREATE INDEX IF NOT EXISTS idx_studios_stash_id ON studios(stash_id); +"#; + +/// StashDB fingerprint fix: `scenes.md5` (full-file MD5, StashDB's most common +/// fingerprint — previously never computed) plus cleanup of bogus `oshash` +/// rows. The old `compute_oshash` wrote a 64-char SHA-256 digest; real +/// OSHASH values are always exactly 16 hex chars, so anything else is +/// unreachable junk that would break batch Identify counts. +pub const MIGRATION_019: &str = r#" +ALTER TABLE scenes ADD COLUMN md5 TEXT; +UPDATE scenes SET oshash = NULL WHERE oshash IS NOT NULL AND length(oshash) != 16; +"#; + +/// Saved searches remember the listing's sort / filters (JSON `BrowseFilters`), +/// so checks and re-opens run the same listing that was saved. +pub const MIGRATION_020: &str = r#" +ALTER TABLE saved_searches ADD COLUMN filters TEXT; +"#; + +/// Local "Online" library: every listing result seen, which saved search +/// it belongs to, and tag rows for include / exclude lookups. +pub const MIGRATION_021: &str = r#" +CREATE TABLE IF NOT EXISTS online_items ( + key TEXT PRIMARY KEY, + url TEXT NOT NULL, + site_id TEXT NOT NULL, + title TEXT NOT NULL, + thumbnail TEXT, + duration INTEGER, + channel TEXT, + performers TEXT NOT NULL DEFAULT '[]', + tags TEXT NOT NULL DEFAULT '[]', + first_seen TEXT NOT NULL, + last_seen TEXT NOT NULL, + details_at TEXT +); +CREATE INDEX IF NOT EXISTS idx_online_items_last_seen ON online_items(last_seen); +CREATE INDEX IF NOT EXISTS idx_online_items_duration ON online_items(duration); +CREATE TABLE IF NOT EXISTS online_item_tags ( + item_key TEXT NOT NULL, + tag TEXT NOT NULL, + PRIMARY KEY (item_key, tag) +); +CREATE INDEX IF NOT EXISTS idx_online_item_tags_tag ON online_item_tags(tag); +CREATE TABLE IF NOT EXISTS saved_search_items ( + search_id TEXT NOT NULL, + item_key TEXT NOT NULL, + rank INTEGER NOT NULL, + first_seen TEXT NOT NULL, + PRIMARY KEY (search_id, item_key) +); +CREATE INDEX IF NOT EXISTS idx_saved_search_items_key ON saved_search_items(item_key); +CREATE TABLE IF NOT EXISTS online_search_updates ( + search_id TEXT PRIMARY KEY, + updated_at TEXT NOT NULL +); +"#; + +/// Online library entries the user removed: per saved search, or '' +/// for browsing history. Updates and browsing don't bring them back. +pub const MIGRATION_022: &str = r#" +CREATE TABLE IF NOT EXISTS online_dismissed ( + scope TEXT NOT NULL, + item_key TEXT NOT NULL, + PRIMARY KEY (scope, item_key) +); +"#; + +/// Full fetches of saved searches: where each one is, so it can pause on +/// rate limits and resume (also after the app restarts). +pub const MIGRATION_023: &str = r#" +CREATE TABLE IF NOT EXISTS online_crawls ( + search_id TEXT PRIMARY KEY, + next_page INTEGER NOT NULL, + pages_done INTEGER NOT NULL DEFAULT 0, + new_items INTEGER NOT NULL DEFAULT 0, + site_total INTEGER, + state TEXT NOT NULL, + message TEXT, + resume_at TEXT, + updated_at TEXT NOT NULL +); +"#; + +/// Full fetches keep their slice queue (JSON) so big xPomelo searches, +/// split under its 10-page cap, resume where they were. +pub const MIGRATION_024: &str = r#" +ALTER TABLE online_crawls ADD COLUMN plan TEXT; +"#; diff --git a/src-tauri/src/db/mod.rs b/src-tauri/src/db/mod.rs index 08e88a6..b0cf5f0 100644 --- a/src-tauri/src/db/mod.rs +++ b/src-tauri/src/db/mod.rs @@ -1,10 +1,11 @@ mod migrations; +mod online; use crate::db::migrations::{ MIGRATION_001, MIGRATION_002, MIGRATION_003, MIGRATION_004, MIGRATION_005, MIGRATION_006, MIGRATION_007, MIGRATION_008, MIGRATION_009, MIGRATION_010, MIGRATION_011, MIGRATION_012, MIGRATION_013, MIGRATION_014, MIGRATION_015, MIGRATION_016, MIGRATION_017, MIGRATION_018, - MIGRATION_019, + MIGRATION_019, MIGRATION_020, MIGRATION_021, MIGRATION_022, MIGRATION_023, MIGRATION_024, }; use crate::error::{AppError, AppResult}; use crate::models::{ @@ -166,6 +167,15 @@ impl Database { if !column_exists(&conn, "scenes", "md5") { conn.execute_batch(MIGRATION_019)?; } + if !column_exists(&conn, "saved_searches", "filters") { + conn.execute_batch(MIGRATION_020)?; + } + conn.execute_batch(MIGRATION_021)?; + conn.execute_batch(MIGRATION_022)?; + conn.execute_batch(MIGRATION_023)?; + if !column_exists(&conn, "online_crawls", "plan") { + conn.execute_batch(MIGRATION_024)?; + } // Backfill the default StashDB endpoint for DBs created before it // existed (fresh DBs get it via AppSettings::default). { @@ -1069,6 +1079,15 @@ impl Database { BrowseOrientation::Lesbian => "lesbian", BrowseOrientation::Transgender => "transgender", }); + let filters = req.filters.clone().normalized(); + let filters_json = if filters.is_empty() { + None + } else { + Some( + serde_json::to_string(&filters) + .map_err(|e| AppError::Other(format!("filters serialize: {e}")))?, + ) + }; let conn = self .conn .lock() @@ -1076,8 +1095,8 @@ impl Database { let id = uuid::Uuid::new_v4().to_string(); let created_at = chrono::Utc::now().to_rfc3339(); conn.execute( - "INSERT INTO saved_searches (id, name, site_id, kind, slug, orientation, last_checked_at, last_item_keys, new_count, created_at) - VALUES (?1, ?2, ?3, ?4, ?5, ?6, NULL, '[]', 0, ?7)", + "INSERT INTO saved_searches (id, name, site_id, kind, slug, orientation, last_checked_at, last_item_keys, new_count, created_at, filters) + VALUES (?1, ?2, ?3, ?4, ?5, ?6, NULL, '[]', 0, ?7, ?8)", rusqlite::params![ id, name, @@ -1085,7 +1104,8 @@ impl Database { kind_str, req.slug.trim(), orientation_str, - created_at + created_at, + filters_json ], )?; Ok(crate::models::SavedSearch { @@ -1095,6 +1115,7 @@ impl Database { kind: req.kind, slug: req.slug.trim().to_string(), orientation: req.orientation, + filters, last_checked_at: None, new_count: 0, auto_queue: false, @@ -1134,6 +1155,11 @@ impl Database { new_count: row.get::<_, i64>(8).unwrap_or(0).max(0) as u32, auto_queue: row.get::<_, i64>(10).unwrap_or(0) != 0, created_at: row.get(9)?, + // Unreadable filter JSON degrades to the site defaults. + filters: row + .get::<_, Option>(11)? + .and_then(|j| serde_json::from_str(&j).ok()) + .unwrap_or_default(), }) } @@ -1143,7 +1169,7 @@ impl Database { .lock() .map_err(|e| AppError::Other(e.to_string()))?; let mut stmt = conn.prepare( - "SELECT id, name, site_id, kind, slug, orientation, last_checked_at, last_item_keys, new_count, created_at, auto_queue + "SELECT id, name, site_id, kind, slug, orientation, last_checked_at, last_item_keys, new_count, created_at, auto_queue, filters FROM saved_searches ORDER BY created_at DESC", )?; let rows = stmt.query_map([], Self::map_saved_search_row)?; @@ -1157,7 +1183,7 @@ impl Database { .map_err(|e| AppError::Other(e.to_string()))?; let row = conn .query_row( - "SELECT id, name, site_id, kind, slug, orientation, last_checked_at, last_item_keys, new_count, created_at, auto_queue + "SELECT id, name, site_id, kind, slug, orientation, last_checked_at, last_item_keys, new_count, created_at, auto_queue, filters FROM saved_searches WHERE id = ?1", rusqlite::params![id], Self::map_saved_search_row, @@ -1247,6 +1273,20 @@ impl Database { "DELETE FROM saved_searches WHERE id = ?1", rusqlite::params![id], )?; + // Its slice of the Online library goes too (the items stay in the + // general history). + conn.execute( + "DELETE FROM saved_search_items WHERE search_id = ?1", + rusqlite::params![id], + )?; + conn.execute( + "DELETE FROM online_search_updates WHERE search_id = ?1", + rusqlite::params![id], + )?; + conn.execute( + "DELETE FROM online_crawls WHERE search_id = ?1", + rusqlite::params![id], + )?; Ok(changed > 0) } @@ -1411,6 +1451,70 @@ impl Database { )); } + // Query builder: groups ANDed, terms within a group ORed, exclusions, + // and a score (group + boost terms found) that ranks the best first. + // Each scene's words are built once (materialized), not per term. + let mut args: Vec = Vec::new(); + let mut score_parts: Vec = Vec::new(); + let mut score_args: Vec = Vec::new(); + let mut with_sql = String::new(); + if !filter.groups.is_empty() + || !filter.exclude_terms.is_empty() + || !filter.boost_terms.is_empty() + { + let words = online::words_sql( + "scenes.title || ' ' || COALESCE((SELECT group_concat(qt.name, ' ') FROM scene_tags qst \ + JOIN tags qt ON qt.id = qst.tag_id WHERE qst.scene_id = scenes.id), '') || ' ' || \ + COALESCE((SELECT group_concat(qp.name, ' ') FROM scene_performers qsp \ + JOIN performers qp ON qp.id = qsp.performer_id WHERE qsp.scene_id = scenes.id), '')", + ); + with_sql = format!( + "WITH qwords AS MATERIALIZED (SELECT scenes.id AS id, {words} AS w FROM scenes)\n " + ); + joins.push("JOIN qwords ON qwords.id = scenes.id".to_string()); + for group in &filter.groups { + let terms: Vec<(String, Vec)> = group + .iter() + .filter_map(|t| online::term_condition("qwords.w", t)) + .collect(); + if terms.is_empty() { + continue; + } + let any = terms + .iter() + .map(|(c, _)| c.as_str()) + .collect::>() + .join(" OR "); + conditions.push(format!("({any})")); + for (cond, pats) in terms { + args.extend(pats.iter().map(|p| p.clone().into())); + score_parts.push(cond); + score_args.extend(pats.into_iter().map(Into::into)); + } + } + for (cond, pats) in filter + .exclude_terms + .iter() + .filter_map(|t| online::term_condition("qwords.w", t)) + { + conditions.push(format!("NOT {cond}")); + args.extend(pats.into_iter().map(Into::into)); + } + for (cond, pats) in filter + .boost_terms + .iter() + .filter_map(|t| online::term_condition("qwords.w", t)) + { + score_parts.push(cond); + score_args.extend(pats.into_iter().map(Into::into)); + } + } + let score_sql = if score_parts.is_empty() { + String::new() + } else { + format!("({}) DESC, ", score_parts.join(" + ")) + }; + let where_clause = if conditions.is_empty() { String::new() } else { @@ -1445,15 +1549,18 @@ impl Database { let offset_val = offset.unwrap_or(0).min(limit_val * 100); let sql = format!( - "SELECT scenes.id, scenes.title, scenes.path, scenes.thumb, scenes.source_url, scenes.duration, scenes.channel, scenes.file_size, scenes.width, scenes.height, scenes.rating + "{with_sql}SELECT scenes.id, scenes.title, scenes.path, scenes.thumb, scenes.source_url, scenes.duration, scenes.channel, scenes.file_size, scenes.width, scenes.height, scenes.rating FROM scenes{joins_sql} {where_clause}{group_by} - ORDER BY scenes.created_at DESC + ORDER BY {score_sql}scenes.created_at DESC LIMIT ? OFFSET ?" ); + args.extend(score_args); + args.push(limit_val.into()); + args.push(offset_val.into()); let mut stmt = conn.prepare(&sql)?; - let rows = stmt.query_map(params![limit_val, offset_val], |row| { + let rows = stmt.query_map(rusqlite::params_from_iter(args), |row| { Ok(( row.get::<_, String>(0)?, row.get::<_, String>(1)?, @@ -2010,6 +2117,39 @@ impl Database { } /// Load all known scene paths into a HashSet for O(1) lookups during scan. + /// `(id, path)` of scenes whose file path ends with `suffix`. + pub fn scenes_with_path_suffix(&self, suffix: &str) -> AppResult> { + let conn = self + .conn + .lock() + .map_err(|e| AppError::Other(e.to_string()))?; + let mut stmt = conn.prepare("SELECT id, path FROM scenes WHERE path LIKE ?1")?; + // `ends_with` below is the real filter; LIKE only narrows the scan. + let pattern = format!("%{suffix}"); + let rows = stmt.query_map(params![pattern], |row| { + Ok((row.get::<_, String>(0)?, row.get::<_, String>(1)?)) + })?; + let mut out = Vec::new(); + for row in rows { + let (id, path) = row?; + if path.ends_with(suffix) { + out.push((id, path)); + } + } + Ok(out) + } + + /// Ids of scenes whose file path is exactly `path`. + pub fn scene_ids_with_path(&self, path: &str) -> AppResult> { + let conn = self + .conn + .lock() + .map_err(|e| AppError::Other(e.to_string()))?; + let mut stmt = conn.prepare("SELECT id FROM scenes WHERE path = ?1")?; + let rows = stmt.query_map(params![path], |row| row.get::<_, String>(0))?; + rows.collect::, _>>().map_err(AppError::from) + } + pub fn all_scene_paths(&self) -> AppResult> { let conn = self .conn @@ -2733,27 +2873,42 @@ impl Database { let new_notes = notes.unwrap_or(existing.notes.as_deref().unwrap_or("")); let mut new_path = existing.path.clone(); + let mut new_thumb: Option = None; if rename_file { - if let Some(ref old_path) = existing.path { - let old = std::path::Path::new(old_path); - if old.exists() { - let ext = old.extension().and_then(|e| e.to_str()).unwrap_or("mp4"); - let safe: String = new_title - .chars() - .map(|c| { - if c.is_ascii_alphanumeric() || c == '-' || c == '_' || c == ' ' { - c - } else { - '_' - } - }) - .collect(); - let safe = safe.trim().replace(' ', "_"); - let parent = old.parent().unwrap_or_else(|| std::path::Path::new(".")); - let target = parent.join(format!("{safe}.{ext}")); - if target != old { - std::fs::rename(old, &target)?; - new_path = Some(target.to_string_lossy().to_string()); + let old_path = existing + .path + .as_deref() + .ok_or_else(|| AppError::InvalidInput("scene has no file to rename".into()))?; + let old = std::path::Path::new(old_path); + if !old.is_file() { + return Err(AppError::NotFound(format!("file not found: {old_path}"))); + } + let ext = old.extension().and_then(|e| e.to_str()).unwrap_or("mp4"); + let target = old.with_file_name(format!("{}.{ext}", safe_file_stem(new_title))); + if target != old { + // fs::rename silently replaces an existing file; never clobber + // another video that already has this name. + if target.exists() { + return Err(AppError::InvalidInput(format!( + "a file named {} already exists", + target.file_name().and_then(|n| n.to_str()).unwrap_or("") + ))); + } + std::fs::rename(old, &target) + .map_err(|e| AppError::Other(format!("could not rename {old_path}: {e}")))?; + new_path = Some(target.to_string_lossy().to_string()); + // Keep the `.jpg` thumbnail sidecar next to the video. + if let Some(thumb) = existing.thumb.as_deref() { + let thumb = std::path::Path::new(thumb); + if thumb.parent() == old.parent() + && thumb.file_stem() == old.file_stem() + && thumb.is_file() + { + let thumb_ext = thumb.extension().and_then(|e| e.to_str()).unwrap_or("jpg"); + let thumb_target = target.with_extension(thumb_ext); + if !thumb_target.exists() && std::fs::rename(thumb, &thumb_target).is_ok() { + new_thumb = Some(thumb_target.to_string_lossy().to_string()); + } } } } @@ -2764,8 +2919,8 @@ impl Database { .lock() .map_err(|e| AppError::Other(e.to_string()))?; conn.execute( - "UPDATE scenes SET title = ?2, path = COALESCE(?3, path), notes = ?4, rating = COALESCE(?5, rating) WHERE id = ?1", - params![id, new_title, new_path, new_notes, rating], + "UPDATE scenes SET title = ?2, path = COALESCE(?3, path), notes = ?4, rating = COALESCE(?5, rating), thumb = COALESCE(?6, thumb) WHERE id = ?1", + params![id, new_title, new_path, new_notes, rating, new_thumb], )?; drop(conn); @@ -2830,10 +2985,23 @@ impl Database { |row| Ok((row.get(0)?, row.get(1)?)), ) .optional()?; - let Some(row) = row else { - return Err(AppError::NotFound(format!("scene {id}"))); - }; + row.ok_or_else(|| AppError::NotFound(format!("scene {id}")))? + }; + // Delete the video first: if the OS refuses (Android shared storage + // without "All files access"), keep the library entry and say so + // instead of silently leaving the file behind. + if delete_files { + if let Some(p) = path.as_deref() { + remove_media_file(p)?; + } + } + + { + let conn = self + .conn + .lock() + .map_err(|e| AppError::Other(e.to_string()))?; conn.execute( "DELETE FROM scene_performers WHERE scene_id = ?1", params![id], @@ -2841,13 +3009,9 @@ impl Database { conn.execute("DELETE FROM scene_tags WHERE scene_id = ?1", params![id])?; conn.execute("DELETE FROM watch_history WHERE scene_id = ?1", params![id])?; conn.execute("DELETE FROM scenes WHERE id = ?1", params![id])?; - row - }; + } if delete_files { - if let Some(p) = path { - let _ = std::fs::remove_file(p); - } if let Some(t) = thumb { let _ = std::fs::remove_file(t); } @@ -2983,6 +3147,44 @@ fn parse_status(s: &str) -> DownloadStatus { } } +/// File-name stem for "rename file to match title": ASCII letters, digits, +/// `-` and `_` kept, everything else `_`, spaces collapsed to `_`. +fn safe_file_stem(title: &str) -> String { + let safe: String = title + .chars() + .map(|c| { + if c.is_ascii_alphanumeric() || c == '-' || c == '_' || c == ' ' { + c + } else { + '_' + } + }) + .collect(); + let safe = safe.trim().replace(' ', "_"); + if safe.is_empty() { + "untitled".to_string() + } else { + safe + } +} + +/// Delete a library video file. Missing files are fine; a refusal is +/// reported (Android shared storage needs "All files access" for files this +/// install didn't create). +fn remove_media_file(path: &str) -> AppResult<()> { + match std::fs::remove_file(path) { + Ok(()) => Ok(()), + Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()), + Err(e) if e.kind() == std::io::ErrorKind::PermissionDenied => { + Err(AppError::Other(format!( + "ArcHive isn't allowed to delete {path}. On Android, allow \"All files access\" \ + for ArcHive (Settings → Library), then try again." + ))) + } + Err(e) => Err(AppError::Other(format!("Couldn't delete {path}: {e}"))), + } +} + #[cfg(test)] mod tests { use super::*; @@ -3005,6 +3207,64 @@ mod tests { .unwrap() } + #[test] + fn rename_file_moves_thumb_and_never_overwrites() { + let dir = tempdir().unwrap(); + let db = Database::new(dir.path().to_path_buf()).unwrap(); + let video = dir.path().join("abc123.mp4"); + let thumb = dir.path().join("abc123.jpg"); + std::fs::write(&video, b"v").unwrap(); + std::fs::write(&thumb, b"t").unwrap(); + let id = db + .insert_scene( + "abc123", + Some(video.to_str().unwrap()), + None, + &[], + &[], + Some(thumb.to_str().unwrap()), + None, + None, + None, + None, + None, + ) + .unwrap(); + + let scene = db + .update_scene(&id, Some("My Clip!"), None, None, true, None, None) + .unwrap(); + let renamed = dir.path().join("My_Clip_.mp4"); + assert_eq!(scene.path.as_deref(), renamed.to_str()); + assert!(renamed.is_file() && !video.exists()); + assert!(dir.path().join("My_Clip_.jpg").is_file()); + + // A second scene renamed to the same title must not clobber the first. + let other = dir.path().join("other.mp4"); + std::fs::write(&other, b"o").unwrap(); + let id2 = db + .insert_scene( + "other", + Some(other.to_str().unwrap()), + None, + &[], + &[], + None, + None, + None, + None, + None, + None, + ) + .unwrap(); + let err = db + .update_scene(&id2, Some("My Clip!"), None, None, true, None, None) + .unwrap_err(); + assert!(err.to_string().contains("already exists")); + assert_eq!(std::fs::read(&renamed).unwrap(), b"v"); + assert!(other.is_file()); + } + #[test] fn watch_progress_round_trip() { let dir = tempdir().unwrap(); @@ -3083,9 +3343,20 @@ mod tests { kind: crate::models::BrowseKind::Tag, slug: "abc".to_string(), orientation: None, + filters: crate::models::BrowseFilters { + sort: Some("mv".into()), + exclude: Some("35".into()), + period: Some(" ".into()), + ..Default::default() + }, }) .unwrap(); assert_eq!(saved.name, "test"); + // Filters round-trip (blank values dropped). + let stored = db.get_saved_search(&saved.id).unwrap().unwrap(); + assert_eq!(stored.filters.sort.as_deref(), Some("mv")); + assert_eq!(stored.filters.exclude.as_deref(), Some("35")); + assert_eq!(stored.filters.period, None); assert!(!saved.auto_queue); assert_eq!(saved.new_count, 0); @@ -3097,6 +3368,7 @@ mod tests { kind: crate::models::BrowseKind::Tag, slug: "abc".to_string(), orientation: None, + filters: Default::default(), }) .is_err()); @@ -3398,4 +3670,73 @@ mod tests { .unwrap(); assert!(mismatch.is_empty()); } + + #[test] + fn scene_filter_query_builder_groups_exclusions_and_ranking() { + let dir = tempdir().unwrap(); + let db = Database::new(dir.path().to_path_buf()).unwrap(); + let add = |title: &str, performers: &[&str], tags: &[&str]| { + let p: Vec = performers.iter().map(|s| s.to_string()).collect(); + let t: Vec = tags.iter().map(|s| s.to_string()).collect(); + db.insert_scene( + title, None, None, &p, &t, None, None, None, None, None, None, + ) + .unwrap(); + }; + add("Beach day", &["Alice"], &["Blowjobs", "Outdoor"]); + add("Hot blowjob POV", &[], &[]); + add("Solo blowjob", &[], &["Solo"]); + add("Kitchen talk", &["Betty"], &["Outdoor"]); + + let run = |groups: Vec>, exclude: Vec<&str>, boost: Vec<&str>| { + let f = crate::models::SceneFilter { + groups: groups + .into_iter() + .map(|g| g.into_iter().map(String::from).collect()) + .collect(), + exclude_terms: exclude.into_iter().map(String::from).collect(), + boost_terms: boost.into_iter().map(String::from).collect(), + ..Default::default() + }; + let mut titles: Vec = db + .list_scenes_with_filter(&f, None, None) + .unwrap() + .into_iter() + .map(|s| s.title) + .collect(); + if f.boost_terms.is_empty() { + titles.sort(); + } + titles + }; + + // Singular matches the plural tag; titles count too. + assert_eq!( + run(vec![vec!["blowjob"]], vec![], vec![]), + ["Beach day", "Hot blowjob POV", "Solo blowjob"] + ); + // Exclusions, and AND between groups (tags and performers). + assert_eq!( + run(vec![vec!["blowjob"]], vec!["solo"], vec![]), + ["Beach day", "Hot blowjob POV"] + ); + assert_eq!( + run(vec![vec!["outdoor"], vec!["alice", "pov"]], vec![], vec![]), + ["Beach day"] + ); + // Boost terms rank first without being required. + assert_eq!( + run(vec![vec!["blowjob"]], vec!["solo"], vec!["pov"])[0], + "Hot blowjob POV" + ); + // Combines with the tag filter (its JOIN + GROUP BY). + let f = crate::models::SceneFilter { + tag_names: vec!["Outdoor".to_string()], + exclude_terms: vec!["blowjob".to_string()], + ..Default::default() + }; + let r = db.list_scenes_with_filter(&f, None, None).unwrap(); + assert_eq!(r.len(), 1); + assert_eq!(r[0].title, "Kitchen talk"); + } } diff --git a/src-tauri/src/db/online.rs b/src-tauri/src/db/online.rs new file mode 100644 index 0000000..507ea7a --- /dev/null +++ b/src-tauri/src/db/online.rs @@ -0,0 +1,1131 @@ +//! Local "Online" library: listing results kept on the device so saved +//! searches (and everything browsed before) can be filtered, sorted and +//! searched without asking the site again. + +use super::Database; +use crate::error::{AppError, AppResult}; +use crate::models::{ + MediaItem, OnlineCrawl, OnlineItem, OnlinePage, OnlineQuery, OnlineSearchInfo, OnlineTagCount, +}; +use rusqlite::{params, params_from_iter, OptionalExtension}; + +/// Tag as stored for lookups: lowercase, single spaces. +/// Scope (a pseudo saved-search id) holding bookmarked videos; queries, +/// tag counts and removal take it like any saved search id. +pub const BOOKMARKS: &str = "@bookmarks"; + +/// `text` (an SQL expression) as lowercase words separated by single spaces, +/// with a space at each end, so ` word ` matches whole words and phrases +/// (punctuation, JSON quotes and hyphens become spaces). +pub(crate) fn words_sql(text: &str) -> String { + let mut out = format!("lower({text})"); + for c in [ + "\"", ",", "[", "]", "-", ".", "!", "?", ":", "(", ")", "/", "_", + ] { + out = format!("replace({out}, '{c}', ' ')"); + } + format!("(' ' || {out} || ' ')") +} + +/// An online item's title, tags and performers as words (see `words_sql`). +fn online_words() -> String { + words_sql("o.title || ' ' || COALESCE(o.tags, '') || ' ' || COALESCE(o.performers, '')") +} + +/// A builder term as LIKE patterns on a `words_sql` expression, any of which matches: +/// `big tit` → `% big tit %`, `% big tits %`; `pussies` → also `% pussy %` +/// (singular / plural of the last word); `tit*` → `% tit%` (prefix only). +/// Empty for an empty term. +fn term_patterns(term: &str) -> Vec { + let t = term.trim().to_lowercase(); + let (t, prefix) = match t.strip_suffix('*') { + Some(rest) => (rest.to_string(), true), + None => (t, false), + }; + let words: Vec = t + .chars() + .map(|c| { + if c.is_alphanumeric() || c == '\'' { + c + } else { + ' ' + } + }) + .collect::() + .split_whitespace() + .map(str::to_string) + .collect(); + let Some((last, head)) = words.split_last() else { + return Vec::new(); + }; + let head = head.iter().map(|w| format!("{w} ")).collect::(); + if prefix { + return vec![format!("% {head}{last}%")]; + } + let mut forms = vec![last.clone()]; + if last.len() >= 3 { + if let Some(stem) = last.strip_suffix("ies") { + forms.push(format!("{stem}y")); + } else if let Some(stem) = last.strip_suffix("es") { + forms.push(stem.to_string()); + forms.push(format!("{stem}e")); + } else if let Some(stem) = last.strip_suffix('s') { + forms.push(stem.to_string()); + } + if let Some(stem) = last.strip_suffix('y') { + forms.push(format!("{stem}ies")); + } + if !last.ends_with('s') { + forms.push(format!("{last}s")); + forms.push(format!("{last}es")); + } + } + let mut seen = std::collections::HashSet::new(); + forms + .into_iter() + .filter(|f| f.len() >= 2 && seen.insert(f.clone())) + .map(|f| format!("% {head}{f} %")) + .collect() +} + +/// SQL condition for one term (any of its patterns) on the `words` +/// expression (see `words_sql`), and its arguments. +pub(crate) fn term_condition(words: &str, term: &str) -> Option<(String, Vec)> { + let pats = term_patterns(term); + if pats.is_empty() { + return None; + } + let cond = vec![format!("{words} LIKE ?"); pats.len()].join(" OR "); + Some((format!("({cond})"), pats)) +} + +fn norm_tag(tag: &str) -> String { + tag.split_whitespace() + .collect::>() + .join(" ") + .to_lowercase() +} + +fn json_list(v: &[String]) -> String { + serde_json::to_string(v).unwrap_or_else(|_| "[]".into()) +} + +fn parse_list(raw: Option) -> Vec { + raw.and_then(|r| serde_json::from_str(&r).ok()) + .unwrap_or_default() +} + +const ITEM_COLUMNS: &str = "o.key, o.url, o.site_id, o.title, o.thumbnail, o.duration, o.channel, \ + o.performers, o.tags, o.first_seen, o.last_seen, o.details_at"; + +fn map_item(row: &rusqlite::Row<'_>) -> rusqlite::Result { + Ok(OnlineItem { + key: row.get(0)?, + url: row.get(1)?, + site_id: row.get(2)?, + title: row.get(3)?, + // Entries saved before a thumbnail fix still show (urls::repair_thumbnail). + thumbnail: row + .get::<_, Option>(4)? + .map(|t| crate::sites::urls::repair_thumbnail(&t)), + duration: row.get::<_, Option>(5)?.map(|d| d as u32), + channel: row.get(6)?, + performers: parse_list(row.get(7)?), + tags: parse_list(row.get(8)?), + first_seen: row.get(9)?, + last_seen: row.get(10)?, + details_at: row.get(11)?, + }) +} + +fn map_crawl(r: &rusqlite::Row<'_>) -> rusqlite::Result { + Ok(OnlineCrawl { + search_id: r.get(0)?, + next_page: r.get::<_, i64>(1)? as u32, + pages_done: r.get::<_, i64>(2)? as u32, + new_items: r.get::<_, i64>(3)? as u32, + site_total: r.get::<_, Option>(4)?.map(|t| t as u32), + state: r.get(5)?, + message: r.get(6)?, + resume_at: r.get(7)?, + updated_at: r.get(8)?, + ..crawl_plan_summary(r.get(9)?) + }) +} + +/// Slice counts and current slice label from a stored plan. +fn crawl_plan_summary(plan: Option) -> OnlineCrawl { + let slices: Vec = plan + .as_deref() + .and_then(|p| serde_json::from_str(p).ok()) + .unwrap_or_default(); + let done = |s: &serde_json::Value| s.get("done").and_then(|d| d.as_bool()) == Some(true); + OnlineCrawl { + slices_done: slices.iter().filter(|s| done(s)).count() as u32, + slices_total: slices.len() as u32, + slice_label: slices + .iter() + .find(|s| !done(s)) + .and_then(|s| s.get("label").and_then(|l| l.as_str())) + .filter(|l| !l.is_empty()) + .map(str::to_string), + plan, + ..Default::default() + } +} + +fn replace_tags(conn: &rusqlite::Connection, key: &str, tags: &[String]) -> rusqlite::Result<()> { + conn.execute( + "DELETE FROM online_item_tags WHERE item_key = ?1", + params![key], + )?; + let mut stmt = + conn.prepare("INSERT OR IGNORE INTO online_item_tags (item_key, tag) VALUES (?1, ?2)")?; + for t in tags.iter().map(|t| norm_tag(t)).filter(|t| !t.is_empty()) { + stmt.execute(params![key, t])?; + } + Ok(()) +} + +impl Database { + /// Record listing results. With `search`, they also join that saved + /// search's slice, ranked from `base_rank` in listing order. Returns how + /// many were new (to the search when given, otherwise to the library). + pub fn record_online_items( + &self, + items: &[MediaItem], + search: Option<(&str, i64)>, + ) -> AppResult { + let mut conn = self + .conn + .lock() + .map_err(|e| AppError::Other(e.to_string()))?; + let tx = conn.transaction()?; + let now = chrono::Utc::now().to_rfc3339(); + let mut new_count = 0u32; + for (i, item) in items.iter().enumerate() { + if item.url.trim().is_empty() || item.is_live == Some(true) { + continue; + } + let key = crate::sites::urls::video_key(&item.url); + let dismissed = |scope: &str| -> rusqlite::Result { + tx.query_row( + "SELECT 1 FROM online_dismissed WHERE scope = ?1 AND item_key = ?2", + params![scope, key], + |_| Ok(()), + ) + .optional() + .map(|r| r.is_some()) + }; + // Deleted from history: not recorded again. + if dismissed("")? { + continue; + } + let existing: Option> = tx + .query_row( + "SELECT details_at FROM online_items WHERE key = ?1", + params![key], + |r| r.get(0), + ) + .optional()?; + match &existing { + None => { + tx.execute( + "INSERT INTO online_items (key, url, site_id, title, thumbnail, duration, + channel, performers, tags, first_seen, last_seen) + VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?10)", + params![ + key, + item.url, + item.site_id, + item.title, + item.thumbnail, + item.duration, + item.channel, + json_list(&item.performers), + json_list(&item.tags), + now, + ], + )?; + replace_tags(&tx, &key, &item.tags)?; + if search.is_none() { + new_count += 1; + } + } + Some(details_at) => { + tx.execute( + "UPDATE online_items SET url = ?2, site_id = ?3, title = ?4, + thumbnail = COALESCE(?5, thumbnail), + duration = COALESCE(?6, duration), + channel = COALESCE(channel, ?7), + last_seen = ?8 + WHERE key = ?1", + params![ + key, + item.url, + item.site_id, + item.title, + item.thumbnail, + item.duration, + item.channel, + now, + ], + )?; + // Listing tags only until the video page's own were fetched. + if details_at.is_none() && !item.tags.is_empty() { + tx.execute( + "UPDATE online_items SET tags = ?2 WHERE key = ?1", + params![key, json_list(&item.tags)], + )?; + replace_tags(&tx, &key, &item.tags)?; + } + } + } + if let Some((search_id, base_rank)) = search { + // Removed from this saved search: Update doesn't re-add it. + if dismissed(search_id)? { + continue; + } + let added = tx.execute( + "INSERT OR IGNORE INTO saved_search_items (search_id, item_key, rank, first_seen) + VALUES (?1, ?2, ?3, ?4)", + params![search_id, key, base_rank + i as i64, now], + )?; + if added > 0 { + new_count += 1; + } else { + tx.execute( + "UPDATE saved_search_items SET rank = ?3 + WHERE search_id = ?1 AND item_key = ?2", + params![search_id, key, base_rank + i as i64], + )?; + } + } + } + tx.commit()?; + Ok(new_count) + } + + pub fn mark_online_search_updated(&self, search_id: &str) -> AppResult<()> { + let conn = self + .conn + .lock() + .map_err(|e| AppError::Other(e.to_string()))?; + conn.execute( + "INSERT INTO online_search_updates (search_id, updated_at) VALUES (?1, ?2) + ON CONFLICT(search_id) DO UPDATE SET updated_at = excluded.updated_at", + params![search_id, chrono::Utc::now().to_rfc3339()], + )?; + Ok(()) + } + + pub fn online_search_updated_at(&self, search_id: &str) -> AppResult> { + let conn = self + .conn + .lock() + .map_err(|e| AppError::Other(e.to_string()))?; + Ok(conn + .query_row( + "SELECT updated_at FROM online_search_updates WHERE search_id = ?1", + params![search_id], + |r| r.get(0), + ) + .optional()?) + } + + /// Tags / performers / channel from the video's own page. + pub fn set_online_item_details( + &self, + key: &str, + details: &MediaItem, + ) -> AppResult> { + let mut conn = self + .conn + .lock() + .map_err(|e| AppError::Other(e.to_string()))?; + let tx = conn.transaction()?; + tx.execute( + "UPDATE online_items SET + performers = ?2, tags = ?3, + channel = COALESCE(?4, channel), + duration = COALESCE(duration, ?5), + details_at = ?6 + WHERE key = ?1", + params![ + key, + json_list(&details.performers), + json_list(&details.tags), + details.channel, + details.duration, + chrono::Utc::now().to_rfc3339(), + ], + )?; + replace_tags(&tx, key, &details.tags)?; + let item = tx + .query_row( + &format!("SELECT {ITEM_COLUMNS} FROM online_items o WHERE o.key = ?1"), + params![key], + map_item, + ) + .optional()?; + tx.commit()?; + Ok(item) + } + + pub fn get_online_item(&self, key: &str) -> AppResult> { + let conn = self + .conn + .lock() + .map_err(|e| AppError::Other(e.to_string()))?; + Ok(conn + .query_row( + &format!("SELECT {ITEM_COLUMNS} FROM online_items o WHERE o.key = ?1"), + params![key], + map_item, + ) + .optional()?) + } + + pub fn query_online_items(&self, q: &OnlineQuery) -> AppResult { + let conn = self + .conn + .lock() + .map_err(|e| AppError::Other(e.to_string()))?; + let mut from = String::from("FROM online_items o"); + let mut wheres: Vec = Vec::new(); + let mut args: Vec = Vec::new(); + let search_id = q.search_id.as_deref().filter(|s| !s.is_empty()); + if let Some(id) = search_id { + from.push_str(" JOIN saved_search_items s ON s.item_key = o.key AND s.search_id = ?"); + args.push(id.to_string().into()); + } + for word in q + .text + .as_deref() + .unwrap_or("") + .split_whitespace() + .map(str::to_lowercase) + { + wheres.push( + "(lower(o.title) LIKE ? OR lower(COALESCE(o.channel, '')) LIKE ? \ + OR lower(o.performers) LIKE ? OR lower(o.tags) LIKE ?)" + .into(), + ); + let like = format!("%{word}%"); + for _ in 0..4 { + args.push(like.clone().into()); + } + } + // A tag counts when the item has it, or (most items have no tags + // until fetched) when the title contains it. + let tag_match = "(EXISTS (SELECT 1 FROM online_item_tags t \ + WHERE t.item_key = o.key AND t.tag = ?) OR lower(o.title) LIKE ?)"; + for tag in q + .include_tags + .iter() + .map(|t| norm_tag(t)) + .filter(|t| !t.is_empty()) + { + wheres.push(tag_match.into()); + args.push(tag.clone().into()); + args.push(format!("%{tag}%").into()); + } + for tag in q + .exclude_tags + .iter() + .map(|t| norm_tag(t)) + .filter(|t| !t.is_empty()) + { + wheres.push(format!("NOT {tag_match}")); + args.push(tag.clone().into()); + args.push(format!("%{tag}%").into()); + } + if let Some(min) = q.min_duration.filter(|m| *m > 0) { + wheres.push("o.duration >= ?".into()); + args.push((min as i64).into()); + } + if let Some(max) = q.max_duration.filter(|m| *m > 0) { + wheres.push("o.duration < ?".into()); + args.push((max as i64).into()); + } + if q.missing_details { + wheres.push("o.details_at IS NULL".into()); + } + if q.no_tags { + wheres.push("(o.tags IS NULL OR o.tags IN ('', '[]'))".into()); + } + // Query builder: groups ANDed, terms within a group ORed; exclusions; + // a score counting every group / boost term found, for "best". + let words = online_words(); + let mut score_parts: Vec = Vec::new(); + let mut score_args: Vec = Vec::new(); + for group in &q.groups { + let terms: Vec<(String, Vec)> = group + .iter() + .filter_map(|t| term_condition(&words, t)) + .collect(); + if terms.is_empty() { + continue; + } + wheres.push(format!( + "({})", + terms + .iter() + .map(|(c, _)| c.as_str()) + .collect::>() + .join(" OR ") + )); + for (cond, pats) in &terms { + args.extend(pats.iter().map(|p| p.clone().into())); + score_parts.push(cond.clone()); + score_args.extend(pats.iter().map(|p| p.clone().into())); + } + } + for (cond, pats) in q + .exclude_terms + .iter() + .filter_map(|t| term_condition(&words, t)) + { + wheres.push(format!("NOT {cond}")); + args.extend(pats.into_iter().map(Into::into)); + } + for (cond, pats) in q + .boost_terms + .iter() + .filter_map(|t| term_condition(&words, t)) + { + score_parts.push(cond); + score_args.extend(pats.into_iter().map(Into::into)); + } + let where_sql = if wheres.is_empty() { + String::new() + } else { + format!(" WHERE {}", wheres.join(" AND ")) + }; + let best = + !score_parts.is_empty() && matches!(q.sort.as_deref().unwrap_or("best"), "best" | ""); + let score_sql = if best { + format!("({}) DESC, ", score_parts.join(" + ")) + } else { + String::new() + }; + let order = match q.sort.as_deref().unwrap_or("") { + "longest" => "o.duration IS NULL, o.duration DESC", + "shortest" => "o.duration IS NULL, o.duration ASC", + "title" => "lower(o.title) ASC", + "site" if search_id.is_some() => "s.rank ASC", + "newest" if search_id.is_some() => "s.first_seen DESC, s.rank ASC", + _ if search_id.is_some() => "s.rank ASC", + _ => "o.last_seen DESC", + }; + let total: i64 = conn.query_row( + &format!("SELECT COUNT(*) {from}{where_sql}"), + params_from_iter(args.iter()), + |r| r.get(0), + )?; + let limit = if q.limit == 0 { 60 } else { q.limit.min(500) }; + let mut page_args = args.clone(); + if best { + page_args.extend(score_args); + } + page_args.push((limit as i64).into()); + page_args.push((q.offset as i64).into()); + let mut stmt = conn.prepare(&format!( + "SELECT {ITEM_COLUMNS} {from}{where_sql} ORDER BY {score_sql}{order} LIMIT ? OFFSET ?" + ))?; + let items = stmt + .query_map(params_from_iter(page_args.iter()), map_item)? + .collect::, _>>()?; + Ok(OnlinePage { + items, + total: total as u32, + }) + } + + /// Most common tags (of one saved search, or the whole library). + pub fn online_tag_counts( + &self, + search_id: Option<&str>, + limit: u32, + ) -> AppResult> { + let conn = self + .conn + .lock() + .map_err(|e| AppError::Other(e.to_string()))?; + let limit = limit.clamp(1, 2000) as i64; + let rows = match search_id.filter(|s| !s.is_empty()) { + Some(id) => { + let mut stmt = conn.prepare( + "SELECT t.tag, COUNT(*) AS n FROM online_item_tags t + JOIN saved_search_items s ON s.item_key = t.item_key AND s.search_id = ?1 + GROUP BY t.tag ORDER BY n DESC, t.tag LIMIT ?2", + )?; + let r = stmt + .query_map(params![id, limit], |r| { + Ok(OnlineTagCount { + tag: r.get(0)?, + count: r.get::<_, i64>(1)? as u32, + }) + })? + .collect::, _>>()?; + r + } + None => { + let mut stmt = conn.prepare( + "SELECT tag, COUNT(*) AS n FROM online_item_tags + GROUP BY tag ORDER BY n DESC, tag LIMIT ?1", + )?; + let r = stmt + .query_map(params![limit], |r| { + Ok(OnlineTagCount { + tag: r.get(0)?, + count: r.get::<_, i64>(1)? as u32, + }) + })? + .collect::, _>>()?; + r + } + }; + Ok(rows) + } + + /// Item count and last update per saved search. + pub fn online_search_info(&self) -> AppResult> { + let conn = self + .conn + .lock() + .map_err(|e| AppError::Other(e.to_string()))?; + let mut stmt = conn.prepare( + "SELECT ss.id, + (SELECT COUNT(*) FROM saved_search_items i WHERE i.search_id = ss.id), + u.updated_at + FROM saved_searches ss + LEFT JOIN online_search_updates u ON u.search_id = ss.id", + )?; + let mut rows = stmt + .query_map([], |r| { + Ok(OnlineSearchInfo { + search_id: r.get(0)?, + item_count: r.get::<_, i64>(1)? as u32, + updated_at: r.get(2)?, + }) + })? + .collect::, _>>()?; + let bookmarks: i64 = conn.query_row( + "SELECT COUNT(*) FROM saved_search_items WHERE search_id = ?1", + params![BOOKMARKS], + |r| r.get(0), + )?; + rows.push(OnlineSearchInfo { + search_id: BOOKMARKS.to_string(), + item_count: bookmarks as u32, + updated_at: None, + }); + Ok(rows) + } + + /// Remove entries: from one saved search (`search_id`), or from the + /// library altogether. Either way they are remembered as removed so + /// updates / browsing don't bring them back. Returns how many went. + pub fn delete_online_items(&self, search_id: Option<&str>, keys: &[String]) -> AppResult { + let mut conn = self + .conn + .lock() + .map_err(|e| AppError::Other(e.to_string()))?; + let tx = conn.transaction()?; + let scope = search_id.filter(|s| !s.is_empty()).unwrap_or(""); + let mut removed = 0u32; + for key in keys { + let n = if scope.is_empty() { + tx.execute( + "DELETE FROM saved_search_items WHERE item_key = ?1", + params![key], + )?; + tx.execute( + "DELETE FROM online_item_tags WHERE item_key = ?1", + params![key], + )?; + tx.execute("DELETE FROM online_items WHERE key = ?1", params![key])? + } else { + tx.execute( + "DELETE FROM saved_search_items WHERE search_id = ?1 AND item_key = ?2", + params![scope, key], + )? + }; + tx.execute( + "INSERT OR IGNORE INTO online_dismissed (scope, item_key) VALUES (?1, ?2)", + params![scope, key], + )?; + removed += n as u32; + } + tx.commit()?; + Ok(removed) + } + + /// Remove everything the query currently shows (all pages). + pub fn delete_online_matching(&self, q: &OnlineQuery) -> AppResult { + let mut all = q.clone(); + all.offset = 0; + all.limit = 500; + let mut keys = Vec::new(); + loop { + let page = self.query_online_items(&all)?; + let n = page.items.len(); + keys.extend(page.items.into_iter().map(|i| i.key)); + if n < 500 { + break; + } + all.offset += 500; + } + self.delete_online_items(q.search_id.as_deref(), &keys) + } + + pub fn get_online_crawl(&self, search_id: &str) -> AppResult> { + let conn = self + .conn + .lock() + .map_err(|e| AppError::Other(e.to_string()))?; + Ok(conn + .query_row( + "SELECT search_id, next_page, pages_done, new_items, site_total, state, message, + resume_at, updated_at, plan + FROM online_crawls WHERE search_id = ?1", + params![search_id], + map_crawl, + ) + .optional()?) + } + + pub fn list_online_crawls(&self) -> AppResult> { + let conn = self + .conn + .lock() + .map_err(|e| AppError::Other(e.to_string()))?; + let mut stmt = conn.prepare( + "SELECT search_id, next_page, pages_done, new_items, site_total, state, message, + resume_at, updated_at, plan + FROM online_crawls ORDER BY updated_at DESC", + )?; + let rows = stmt + .query_map([], map_crawl)? + .collect::, _>>()?; + Ok(rows) + } + + pub fn put_online_crawl(&self, c: &OnlineCrawl) -> AppResult<()> { + let conn = self + .conn + .lock() + .map_err(|e| AppError::Other(e.to_string()))?; + conn.execute( + "INSERT INTO online_crawls (search_id, next_page, pages_done, new_items, site_total, + state, message, resume_at, updated_at, plan) + VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10) + ON CONFLICT(search_id) DO UPDATE SET next_page = excluded.next_page, + pages_done = excluded.pages_done, new_items = excluded.new_items, + site_total = excluded.site_total, state = excluded.state, + message = excluded.message, resume_at = excluded.resume_at, + updated_at = excluded.updated_at, plan = excluded.plan", + params![ + c.search_id, + c.next_page, + c.pages_done, + c.new_items, + c.site_total, + c.state, + c.message, + c.resume_at, + chrono::Utc::now().to_rfc3339(), + c.plan, + ], + )?; + Ok(()) + } + + /// Forget browsing history (items not in any saved search). + /// Bookmark a video: it joins the Online library (with whatever tags / + /// performers it carries) in the `BOOKMARKS` scope, even if it was + /// removed before. Returns its key. + pub fn bookmark_online_item(&self, item: &MediaItem) -> AppResult { + let key = crate::sites::urls::video_key(&item.url); + { + let conn = self + .conn + .lock() + .map_err(|e| AppError::Other(e.to_string()))?; + conn.execute( + "DELETE FROM online_dismissed WHERE item_key = ?1 AND scope IN ('', ?2)", + params![key, BOOKMARKS], + )?; + } + // Newest bookmark first (the scope lists by rank). + let rank = -chrono::Utc::now().timestamp(); + self.record_online_items(std::slice::from_ref(item), Some((BOOKMARKS, rank)))?; + Ok(key) + } + + /// Take a video out of the bookmarks. It leaves the Online library too, + /// unless a saved search also holds it. + pub fn unbookmark_online_item(&self, key: &str) -> AppResult<()> { + let conn = self + .conn + .lock() + .map_err(|e| AppError::Other(e.to_string()))?; + conn.execute( + "DELETE FROM saved_search_items WHERE search_id = ?1 AND item_key = ?2", + params![BOOKMARKS, key], + )?; + let removed = conn.execute( + "DELETE FROM online_items WHERE key = ?1 + AND key NOT IN (SELECT item_key FROM saved_search_items)", + params![key], + )?; + if removed > 0 { + conn.execute( + "DELETE FROM online_item_tags WHERE item_key = ?1", + params![key], + )?; + } + Ok(()) + } + + pub fn is_online_bookmarked(&self, key: &str) -> AppResult { + let conn = self + .conn + .lock() + .map_err(|e| AppError::Other(e.to_string()))?; + Ok(conn + .query_row( + "SELECT 1 FROM saved_search_items WHERE search_id = ?1 AND item_key = ?2", + params![BOOKMARKS, key], + |_| Ok(()), + ) + .optional()? + .is_some()) + } + + /// Move an entry to its real source URL (a resolved redirect link): + /// new key, URL and site; saved searches / bookmarks follow it. When + /// the new key already exists, the old entry is merged into it. Returns + /// the new key. + pub fn rekey_online_item(&self, old_key: &str, url: &str, site_id: &str) -> AppResult { + let new_key = crate::sites::urls::video_key(url); + let mut conn = self + .conn + .lock() + .map_err(|e| AppError::Other(e.to_string()))?; + let tx = conn.transaction()?; + if new_key == old_key { + tx.execute( + "UPDATE online_items SET url = ?2, site_id = ?3 WHERE key = ?1", + params![old_key, url, site_id], + )?; + } else { + let exists = tx + .query_row( + "SELECT 1 FROM online_items WHERE key = ?1", + params![new_key], + |_| Ok(()), + ) + .optional()? + .is_some(); + tx.execute( + "UPDATE OR IGNORE saved_search_items SET item_key = ?2 WHERE item_key = ?1", + params![old_key, new_key], + )?; + tx.execute( + "DELETE FROM saved_search_items WHERE item_key = ?1", + params![old_key], + )?; + if exists { + tx.execute("DELETE FROM online_items WHERE key = ?1", params![old_key])?; + tx.execute( + "DELETE FROM online_item_tags WHERE item_key = ?1", + params![old_key], + )?; + } else { + tx.execute( + "UPDATE online_items SET key = ?2, url = ?3, site_id = ?4 WHERE key = ?1", + params![old_key, new_key, url, site_id], + )?; + tx.execute( + "UPDATE OR IGNORE online_item_tags SET item_key = ?2 WHERE item_key = ?1", + params![old_key, new_key], + )?; + tx.execute( + "DELETE FROM online_item_tags WHERE item_key = ?1", + params![old_key], + )?; + } + } + tx.commit()?; + Ok(new_key) + } + + pub fn clear_online_history(&self) -> AppResult { + let conn = self + .conn + .lock() + .map_err(|e| AppError::Other(e.to_string()))?; + let removed = conn.execute( + "DELETE FROM online_items + WHERE key NOT IN (SELECT item_key FROM saved_search_items)", + [], + )?; + conn.execute( + "DELETE FROM online_item_tags + WHERE item_key NOT IN (SELECT key FROM online_items)", + [], + )?; + Ok(removed as u32) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn item(url: &str, title: &str, dur: u32, tags: &[&str]) -> MediaItem { + MediaItem { + id: url.into(), + url: url.into(), + title: title.into(), + site_id: "xpomelo".into(), + duration: Some(dur), + tags: tags.iter().map(|t| t.to_string()).collect(), + ..Default::default() + } + } + + #[test] + fn query_builder_groups_exclusions_and_ranking() { + let dir = tempfile::tempdir().unwrap(); + let db = Database::new(dir.path().to_path_buf()).unwrap(); + let page = vec![ + item( + "https://a.com/1", + "Prostate massage and blowjob", + 600, + &["handjob"], + ), + item("https://a.com/2", "Sloppy BJ, prostate milking", 600, &[]), + item("https://a.com/3", "Prostate massage solo", 600, &["solo"]), + item("https://a.com/4", "Classic glasses blowjob", 600, &[]), + item( + "https://a.com/5", + "Prostate-massage, blowjob & handjob", + 600, + &[], + ), + ]; + db.record_online_items(&page, None).unwrap(); + let q = |groups: Vec>, exclude: Vec<&str>, boost: Vec<&str>| OnlineQuery { + groups: groups + .into_iter() + .map(|g| g.into_iter().map(String::from).collect()) + .collect(), + exclude_terms: exclude.into_iter().map(String::from).collect(), + boost_terms: boost.into_iter().map(String::from).collect(), + limit: 50, + ..Default::default() + }; + let titles = |q: OnlineQuery| -> Vec { + db.query_online_items(&q) + .unwrap() + .items + .into_iter() + .map(|i| i.url) + .collect() + }; + // (prostate) AND (blowjob OR bj), no solo; "ass" must not match "glasses". + let r = titles(q( + vec![vec!["prostate*"], vec!["blowjob", "bj"]], + vec!["solo"], + vec!["handjob"], + )); + assert_eq!(r.len(), 3, "{r:?}"); + assert!(!r.contains(&"https://a.com/3".to_string())); + // Boosted / more matches first: #1 and #5 have handjob. + assert!(r[..2].contains(&"https://a.com/1".to_string())); + assert!(r[..2].contains(&"https://a.com/5".to_string())); + assert!(titles(q(vec![vec!["ass"]], vec![], vec![])).is_empty()); + let p = term_patterns("Big-Tit"); + assert!(p.contains(&"% big tit %".to_string()) && p.contains(&"% big tits %".to_string())); + assert!(term_patterns("pussies").contains(&"% pussy %".to_string())); + assert!(term_patterns("blowjob").contains(&"% blowjobs %".to_string())); + assert_eq!(term_patterns("tit*"), vec!["% tit%".to_string()]); + assert!(term_patterns(" , ").is_empty()); + // Plurals: the term "blowjobs" also matches titles saying "blowjob". + assert_eq!(titles(q(vec![vec!["blowjobs"]], vec![], vec![])).len(), 3); + } + + #[test] + fn rekey_moves_redirect_links_to_their_source() { + let dir = tempfile::tempdir().unwrap(); + let db = Database::new(dir.path().to_path_buf()).unwrap(); + let go = item("https://pornwitch.com/go?id=txxx1", "A", 60, &[]); + db.record_online_items(std::slice::from_ref(&go), Some(("s1", 5))) + .unwrap(); + let old = crate::sites::urls::video_key(&go.url); + let new = db + .rekey_online_item(&old, "https://txxx.com/videos/1/a/", "pornwitch") + .unwrap(); + assert_ne!(old, new); + assert!(db.get_online_item(&old).unwrap().is_none()); + let moved = db.get_online_item(&new).unwrap().unwrap(); + assert_eq!(moved.url, "https://txxx.com/videos/1/a/"); + let page = db + .query_online_items(&OnlineQuery { + search_id: Some("s1".into()), + limit: 10, + ..Default::default() + }) + .unwrap(); + assert_eq!(page.total, 1); + } + + #[test] + fn bookmarks_survive_history_clear_and_come_back_after_removal() { + let dir = tempfile::tempdir().unwrap(); + let db = Database::new(dir.path().to_path_buf()).unwrap(); + let a = item("https://www.xvideos.com/video.abc/x", "A", 60, &["solo"]); + let key = db.bookmark_online_item(&a).unwrap(); + assert!(db.is_online_bookmarked(&key).unwrap()); + assert_eq!(db.clear_online_history().unwrap(), 0); + let info = db.online_search_info().unwrap(); + assert_eq!( + info.iter() + .find(|i| i.search_id == BOOKMARKS) + .map(|i| i.item_count), + Some(1) + ); + // Removed everywhere, then bookmarked again: it comes back. + db.delete_online_items(None, std::slice::from_ref(&key)) + .unwrap(); + assert!(!db.is_online_bookmarked(&key).unwrap()); + db.bookmark_online_item(&a).unwrap(); + assert!(db.is_online_bookmarked(&key).unwrap()); + db.unbookmark_online_item(&key).unwrap(); + assert!(!db.is_online_bookmarked(&key).unwrap()); + // Only bookmarked: un-bookmarking takes it out of the library. + assert!(db.get_online_item(&key).unwrap().is_none()); + // Also in a saved search: it stays. + db.bookmark_online_item(&a).unwrap(); + db.record_online_items(std::slice::from_ref(&a), Some(("s1", 0))) + .unwrap(); + db.unbookmark_online_item(&key).unwrap(); + assert!(db.get_online_item(&key).unwrap().is_some()); + } + + #[test] + fn records_filters_and_ranks() { + let dir = tempfile::tempdir().unwrap(); + let db = Database::new(dir.path().to_path_buf()).unwrap(); + let page = vec![ + item( + "https://www.xvideos.com/video.abc/blonde_teacher", + "Blonde teacher", + 1500, + &[], + ), + item( + "https://www.pornhub.com/view_video.php?viewkey=1", + "Trailer", + 90, + &["Anal"], + ), + item( + "https://spankbang.com/x/video/y", + "Long one", + 2400, + &["Big Tits", "blonde"], + ), + ]; + assert_eq!(db.record_online_items(&page, None).unwrap(), 3); + // Same xvideos video under its translated slug: not new. + let again = vec![item( + "https://www.xvideos.com/video.abc/professora_loira", + "Blonde teacher", + 1500, + &[], + )]; + assert_eq!(db.record_online_items(&again, None).unwrap(), 0); + assert_eq!(db.record_online_items(&page, Some(("s1", 0))).unwrap(), 3); + assert_eq!(db.record_online_items(&page, Some(("s1", 0))).unwrap(), 0); + + let q = |f: &dyn Fn(&mut OnlineQuery)| { + let mut q = OnlineQuery { + search_id: Some("s1".into()), + ..Default::default() + }; + f(&mut q); + db.query_online_items(&q).unwrap() + }; + let all = q(&|_| {}); + assert_eq!(all.total, 3); + assert_eq!(all.items[0].title, "Blonde teacher"); + // 10+ minutes drops the trailer. + assert_eq!(q(&|q| q.min_duration = Some(600)).total, 2); + // "blonde": tag on one, title word on the other. + assert_eq!(q(&|q| q.include_tags = vec!["Blonde".into()]).total, 2); + assert_eq!( + q(&|q| { + q.include_tags = vec!["blonde".into()]; + q.exclude_tags = vec!["big tits".into()]; + }) + .total, + 1 + ); + assert_eq!(q(&|q| q.exclude_tags = vec!["anal".into()]).total, 2); + let longest = q(&|q| q.sort = Some("longest".into())); + assert_eq!(longest.items[0].title, "Long one"); + assert_eq!(q(&|q| q.text = Some("long".into())).total, 1); + + let tags = db.online_tag_counts(Some("s1"), 10).unwrap(); + assert!(tags.iter().any(|t| t.tag == "big tits" && t.count == 1)); + + let key = crate::sites::urls::video_key("https://www.xvideos.com/video.abc/x"); + let details = MediaItem { + tags: vec!["Teacher".into()], + performers: vec!["Riley Reid".into()], + ..Default::default() + }; + let updated = db.set_online_item_details(&key, &details).unwrap().unwrap(); + assert_eq!(updated.performers, ["Riley Reid"]); + assert!(updated.details_at.is_some()); + assert_eq!(q(&|q| q.include_tags = vec!["teacher".into()]).total, 1); + assert_eq!(q(&|q| q.missing_details = true).total, 2); + + // Removing from the search: gone there, and Update doesn't re-add it. + let short = q(&|q| q.max_duration = Some(600)); + assert_eq!(short.total, 1); + assert_eq!( + db.delete_online_matching(&OnlineQuery { + search_id: Some("s1".into()), + max_duration: Some(600), + ..Default::default() + }) + .unwrap(), + 1 + ); + assert_eq!(q(&|_| {}).total, 2); + assert_eq!(db.record_online_items(&page, Some(("s1", 0))).unwrap(), 0); + assert_eq!(q(&|_| {}).total, 2); + // Deleting from history removes it everywhere and for good. + let long_key = crate::sites::urls::video_key("https://spankbang.com/x/video/y"); + assert_eq!(db.delete_online_items(None, &[long_key]).unwrap(), 1); + assert_eq!(q(&|_| {}).total, 1); + db.record_online_items(&page, None).unwrap(); + assert_eq!( + db.query_online_items(&OnlineQuery::default()) + .unwrap() + .total, + 2 + ); + + db.mark_online_search_updated("s1").unwrap(); + // History-only items are removed (the trailer, taken out of s1), + // saved-search ones kept. + assert_eq!(db.clear_online_history().unwrap(), 1); + } +} diff --git a/src-tauri/src/downloads/image.rs b/src-tauri/src/downloads/image.rs index cc207e8..b5fbe40 100644 --- a/src-tauri/src/downloads/image.rs +++ b/src-tauri/src/downloads/image.rs @@ -55,7 +55,7 @@ pub async fn download_direct( use tokio::io::AsyncWriteExt; let client = reqwest::Client::builder() - .user_agent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36") + .user_agent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36") .build() .map_err(|e| AppError::Download(e.to_string()))?; diff --git a/src-tauri/src/downloads/manager.rs b/src-tauri/src/downloads/manager.rs index 696449a..63c24bd 100644 --- a/src-tauri/src/downloads/manager.rs +++ b/src-tauri/src/downloads/manager.rs @@ -479,6 +479,160 @@ async fn worker_loop( } } +/// Download a page the embedded engine was refused on: resolve it in the +/// in-app browser (see `sites::challenge`), then fetch the stream directly. +#[cfg(target_os = "android")] +async fn download_via_webview( + app: &AppHandle, + plan: &DownloadPlan, + library_path: &str, + on_progress: impl FnMut(Option), +) -> AppResult { + let media = crate::sites::challenge::resolve_via_webview(app, &plan.url).await?; + download_resolved( + app, + plan, + library_path, + &media.url, + &media.referer, + media.cookie.as_deref(), + on_progress, + ) + .await +} + +/// Desktop: re-run yt-dlp with browser impersonation (+ a saved clearance), +/// else pass the bot check in a browser window and retry with its cookies +/// and User-Agent (see `sites::clearance`). +#[cfg(desktop)] +#[allow(clippy::too_many_arguments)] +async fn download_cleared( + app: &AppHandle, + vault: &CookieVault, + runner: &SidecarRunner, + plan: &DownloadPlan, + library_path: &str, + site_cookies: Option<&Path>, + cancel: Arc, + format_args: &[String], + on_line: impl Fn(&str), +) -> AppResult { + use crate::sites::clearance as cl; + let attempt = |clearance: Option| { + let on_line = &on_line; + let cancel = cancel.clone(); + async move { + let cookies = clearance + .as_ref() + .map(|c| c.cookie_file.clone()) + .or_else(|| site_cookies.map(Path::to_path_buf)); + let ua = clearance.as_ref().map(|c| c.user_agent.clone()); + let mut last = None; + for target in [ + Some(cl::IMPERSONATE_TARGET), + Some(cl::IMPERSONATE_FALLBACK), + None, + ] { + let mut args = format_args.to_vec(); + match target { + Some(t) => args.extend(cl::impersonate_args(t, ua.as_deref())), + None => { + if let Some(ua) = ua.as_deref() { + args.push("--user-agent".into()); + args.push(ua.into()); + } + } + } + let result = runner + .run_yt_dlp( + &plan.url, + library_path, + &plan.output_template, + cookies.as_deref(), + cancel.clone(), + on_line, + &args, + ) + .await; + match result { + Err(e) if cl::impersonation_unavailable(&e.to_string()) => last = Some(e), + other => return other, + } + } + Err(last.unwrap_or_else(|| AppError::Download("yt-dlp failed".into()))) + } + }; + match attempt(cl::stored(vault, &plan.url)).await { + Ok(path) => return Ok(path), + Err(e) if !crate::sites::challenge::is_bot_block(&e.to_string()) => return Err(e), + Err(_) => {} + } + let fresh = cl::obtain(app, vault, &plan.url).await?; + attempt(Some(fresh)).await +} + +/// Fetch an already-resolved stream URL (MP4 directly, HLS via ffmpeg). +async fn download_resolved( + app: &AppHandle, + plan: &DownloadPlan, + library_path: &str, + media_url: &str, + referer: &str, + cookie: Option<&str>, + mut on_progress: impl FnMut(Option), +) -> AppResult { + let is_hls = media_url + .split(['?', '#']) + .next() + .is_some_and(|p| p.to_ascii_lowercase().ends_with(".m3u8")); + if is_hls { + let base = plan + .title + .as_deref() + .map(crate::downloads::image::sanitize_filename) + .filter(|s| !s.is_empty()) + .unwrap_or_else(|| "video".to_string()); + let out_path = crate::downloads::image::unique_path( + Path::new(library_path).join(format!("{base}.mp4")), + ); + let result = FfmpegProcessor::new(app.clone()) + .download_hls( + media_url, + &out_path, + Some(referer), + cookie, + plan.duration, + Some(on_progress), + ) + .await; + if result.is_err() { + let _ = std::fs::remove_file(&out_path); + } + result?; + return Ok(out_path.to_string_lossy().to_string()); + } + let mut last_reported = 0u64; + crate::downloads::image::download_direct( + media_url, + library_path, + plan.title.as_deref(), + Some(referer), + cookie, + Some(|downloaded: u64, total: Option| { + let done = total.is_some_and(|t| downloaded >= t); + if downloaded >= last_reported + 1024 * 1024 || done { + last_reported = downloaded; + on_progress( + total + .filter(|t| *t > 0) + .map(|t| downloaded as f32 / t as f32), + ); + } + }), + ) + .await +} + async fn run_job_with_plan( db: Arc, app: AppHandle, @@ -512,7 +666,10 @@ async fn run_job_with_plan( let db_emit = db.clone(); let app_emit = app.clone(); let job_id_emit = job_id.clone(); - let cookies = vault.cookie_file_for_site(&plan.adapter_id); + // Sites without their own adapter (xPomelo sources, pasted URLs) keep + // cookies per domain; see `sites::cookie_keys`. + let cookie_key = crate::sites::cookie_keys::key_for(&plan.adapter_id, &plan.url); + let cookies = vault.cookie_file_for_site(&cookie_key); let result: AppResult> = match plan.tool { DownloadTool::GalleryDl => { @@ -544,7 +701,7 @@ async fn run_job_with_plan( if !cancel.load(Ordering::Relaxed) { return handle_stopped(&db, &app, &job_id); } - let cookie_header = vault.cookie_header(&plan.adapter_id).ok().flatten(); + let cookie_header = vault.cookie_header(&cookie_key).ok().flatten(); let mut last_reported = 0u64; let path = crate::downloads::image::download_direct( &plan.url, @@ -578,7 +735,7 @@ async fn run_job_with_plan( if !cancel.load(Ordering::Relaxed) { return handle_stopped(&db, &app, &job_id); } - let cookie_header = vault.cookie_header(&plan.adapter_id).ok().flatten(); + let cookie_header = vault.cookie_header(&cookie_key).ok().flatten(); std::fs::create_dir_all(library_path)?; let base = plan .title @@ -656,6 +813,104 @@ async fn run_job_with_plan( if path_result.is_err() && !cancel.load(Ordering::Relaxed) { return handle_stopped(&db, &app, &job_id); } + // Sites without their own adapter: when yt-dlp fails (no + // extractor, a broken one, a session-bound KVS link), try the + // native resolvers (KVS in-session, XXXBP, HQPorner) and fetch + // the stream directly. + let path_result = match path_result { + Err(e) + if crate::sites::cookie_keys::is_host_key(&cookie_key) + && cancel.load(Ordering::Relaxed) => + { + let ctx = app + .try_state::>() + .map(|s| s.site_ctx.clone()); + let native = match ctx.as_ref() { + Some(ctx) => { + crate::sites::native::resolve_stream(ctx, &plan.url, &cookie_key).await + } + None => crate::sites::native::resolve_download(&plan.url).await, + }; + match native { + Some(Ok(media)) => { + tracing::info!("[download] yt-dlp failed; using the native resolver"); + download_resolved( + &app, + plan, + library_path, + &media, + &plan.url, + None, + |p| { + update_progress(&db_emit, &app_emit, &job_id_emit, "", p); + }, + ) + .await + } + Some(Err(ne)) if !crate::sites::challenge::is_bot_block(&e.to_string()) => { + Err(AppError::Download(format!("{ne} ({e})"))) + } + _ => Err(e), + } + } + other => other, + }; + #[cfg(desktop)] + let path_result = match path_result { + Err(e) + if crate::sites::challenge::is_bot_block(&e.to_string()) + && crate::sites::challenge::webview_fallback_applies(&plan.url) + && cancel.load(Ordering::Relaxed) => + { + tracing::info!( + "[download] bot check on {}; impersonation / clearance", + plan.url + ); + download_cleared( + &app, + &vault, + &runner, + plan, + library_path, + cookies.as_deref(), + cancel.clone(), + &format_args, + |line| { + let progress = SidecarRunner::parse_progress(line); + update_progress(&db_emit, &app_emit, &job_id_emit, line, progress); + }, + ) + .await + .map_err(|ce| { + AppError::Download(format!( + "{ce} (The embedded engine was blocked first: {e})" + )) + }) + } + other => other, + }; + #[cfg(target_os = "android")] + let path_result = match path_result { + Err(e) + if crate::sites::challenge::is_bot_block(&e.to_string()) + && crate::sites::challenge::webview_fallback_applies(&plan.url) => + { + tracing::info!( + "[download] bot check on {}; trying the in-app browser", + plan.url + ); + download_via_webview(&app, plan, library_path, |p| { + update_progress(&db_emit, &app_emit, &job_id_emit, "", p); + }) + .await + .map_err(|we| { + AppError::Download(format!( + "{we} (The embedded engine was blocked first: {e})" + )) + }) + } + other => other, + }; Ok(vec![path_result?]) } }; @@ -704,7 +959,7 @@ async fn run_job_with_plan( && enriched_performers.is_empty() && enriched_tags.is_empty() { - let cookies = vault.cookie_file_for_site(&plan.adapter_id); + let cookies = vault.cookie_file_for_site(&cookie_key); if let Ok(json) = runner .resolve_media_json(&plan.url, cookies.as_deref()) .await diff --git a/src-tauri/src/downloads/naming.rs b/src-tauri/src/downloads/naming.rs index 9c781eb..ecf00fa 100644 --- a/src-tauri/src/downloads/naming.rs +++ b/src-tauri/src/downloads/naming.rs @@ -3,10 +3,15 @@ /// Settings use placeholders like `{performer}/{title}.{ext}`. yt-dlp expects /// `%(uploader)s/%(title)s.%(ext)s`. Without conversion, files are written to a /// literal path named `{performer}/{title}.{ext}`. +/// yt-dlp field for `{performer}` / `{uploader}` / `{channel}`: the uploader, +/// else the channel, else the site's domain. Many tube sites report no +/// uploader, and plain `%(uploader)s` then files them under a literal `NA/`. +const UPLOADER: &str = "%(uploader,channel,webpage_url_domain|Unknown)s"; + pub fn to_ytdlp_output_template(template: &str) -> String { let trimmed = template.trim(); if trimmed.is_empty() { - return "%(uploader)s/%(title)s.%(ext)s".to_string(); + return format!("{UPLOADER}/%(title)s.%(ext)s"); } // Already a yt-dlp template. if trimmed.contains("%(") { @@ -15,9 +20,9 @@ pub fn to_ytdlp_output_template(template: &str) -> String { let mut out = trimmed.to_string(); let replacements = [ - ("{performer}", "%(uploader)s"), - ("{channel}", "%(uploader)s"), - ("{uploader}", "%(uploader)s"), + ("{performer}", UPLOADER), + ("{channel}", UPLOADER), + ("{uploader}", UPLOADER), ("{title}", "%(title)s"), ("{ext}", "%(ext)s"), ("{id}", "%(id)s"), @@ -29,14 +34,14 @@ pub fn to_ytdlp_output_template(template: &str) -> String { out = out.replace(from, to); } // Common brace variants users might type. - out = out.replace("{Performer}", "%(uploader)s"); + out = out.replace("{Performer}", UPLOADER); out = out.replace("{Title}", "%(title)s"); out = out.replace("{Ext}", "%(ext)s"); if !out.contains("%(") { // Unknown brace template — fall back to a safe default rather than // writing a literal `{foo}` path again. - return "%(uploader)s/%(title)s.%(ext)s".to_string(); + return format!("{UPLOADER}/%(title)s.%(ext)s"); } out } @@ -49,7 +54,7 @@ mod tests { fn converts_default_archhive_template() { assert_eq!( to_ytdlp_output_template("{performer}/{title}.{ext}"), - "%(uploader)s/%(title)s.%(ext)s" + "%(uploader,channel,webpage_url_domain|Unknown)s/%(title)s.%(ext)s" ); } @@ -63,7 +68,7 @@ mod tests { fn empty_falls_back() { assert_eq!( to_ytdlp_output_template(" "), - "%(uploader)s/%(title)s.%(ext)s" + "%(uploader,channel,webpage_url_domain|Unknown)s/%(title)s.%(ext)s" ); } } diff --git a/src-tauri/src/error.rs b/src-tauri/src/error.rs index afd4672..2c3959f 100644 --- a/src-tauri/src/error.rs +++ b/src-tauri/src/error.rs @@ -1,25 +1,64 @@ -use thiserror::Error; - -#[derive(Debug, Error)] -pub enum AppError { - #[error("database error: {0}")] - Db(#[from] rusqlite::Error), - #[error("http error: {0}")] - Http(#[from] reqwest::Error), - #[error("io error: {0}")] - Io(#[from] std::io::Error), - #[error("image error: {0}")] - Image(#[from] image::ImageError), - #[error("not found: {0}")] - NotFound(String), - #[error("invalid input: {0}")] - InvalidInput(String), - #[error("download error: {0}")] - Download(String), - #[error("site error: {0}")] - Site(String), - #[error("{0}")] - Other(String), -} - -pub type AppResult = Result; +use thiserror::Error; + +#[derive(Debug, Error)] +pub enum AppError { + #[error("database error: {0}")] + Db(#[from] rusqlite::Error), + #[error("http error: {}", reqwest_chain(.0))] + Http(#[from] reqwest::Error), + #[error("io error: {0}")] + Io(#[from] std::io::Error), + #[error("image error: {0}")] + Image(#[from] image::ImageError), + #[error("not found: {0}")] + NotFound(String), + #[error("invalid input: {0}")] + InvalidInput(String), + #[error("download error: {0}")] + Download(String), + #[error("site error: {0}")] + Site(String), + #[error("{0}")] + Other(String), +} + +pub type AppResult = Result; + +/// reqwest's own message ("error sending request for url (…)") hides the +/// cause; append the source chain (DNS / connect / TLS / timeout detail) so +/// field reports say what actually failed. +fn reqwest_chain(e: &reqwest::Error) -> String { + let mut out = e.to_string(); + let mut source = std::error::Error::source(e); + while let Some(cause) = source { + let text = cause.to_string(); + if !out.contains(&text) { + out.push_str(": "); + out.push_str(&text); + } + source = cause.source(); + } + out +} + +#[cfg(test)] +mod tests { + #[tokio::test] + async fn http_errors_include_the_cause() { + // Port 9 on localhost refuses connections: the message must say so. + let err = reqwest::Client::new() + .get("http://127.0.0.1:9/") + .send() + .await + .unwrap_err(); + let msg = super::AppError::from(err).to_string(); + assert!( + msg.starts_with("http error: error sending request"), + "{msg}" + ); + assert!( + msg.len() > "http error: error sending request for url (http://127.0.0.1:9/)".len(), + "{msg}" + ); + } +} diff --git a/src-tauri/src/lib.rs b/src-tauri/src/lib.rs index 79f128e..cbcc502 100644 --- a/src-tauri/src/lib.rs +++ b/src-tauri/src/lib.rs @@ -1,4 +1,5 @@ mod commands; +mod cookie_login; mod db; mod discovery; mod downloads; @@ -82,6 +83,22 @@ fn resolve_lan_static_ui(app: &tauri::AppHandle) -> Option { } /// Mobile-only defaults (compiled into Android/iOS builds via `cfg(mobile)`). +/// Default download folder. Android: the shared `Download/ArcHive` (visible +/// in the Files app and kept on uninstall) when the app can write there, +/// else `/downloads`, which is also the default elsewhere. +pub(crate) fn default_library_dir(data_dir: &std::path::Path) -> std::path::PathBuf { + #[cfg(target_os = "android")] + { + let shared = std::path::Path::new("/storage/emulated/0/Download/ArcHive"); + let probe = shared.join(".archhive-write-test"); + if std::fs::create_dir_all(shared).is_ok() && std::fs::write(&probe, b"ok").is_ok() { + let _ = std::fs::remove_file(&probe); + return shared.to_path_buf(); + } + } + data_dir.join("downloads") +} + #[cfg_attr(not(mobile), allow(dead_code))] fn bootstrap_mobile_settings(db: &Database, data_dir: &std::path::Path) -> Result<(), String> { let mut settings = db.get_settings().unwrap_or_default(); @@ -96,8 +113,14 @@ fn bootstrap_mobile_settings(db: &Database, data_dir: &std::path::Path) -> Resul settings.engine_mode = EngineMode::Local; changed = true; } + // "Standalone" (YouTube + direct links only) is retired: the on-device + // engine runs full yt-dlp, so Local covers everything it did. + if settings.engine_mode == EngineMode::Standalone { + settings.engine_mode = EngineMode::Local; + changed = true; + } if settings.library_path.is_empty() { - let downloads = data_dir.join("downloads"); + let downloads = default_library_dir(data_dir); std::fs::create_dir_all(&downloads).map_err(|e| e.to_string())?; settings.library_path = downloads.to_string_lossy().to_string(); changed = true; @@ -179,6 +202,8 @@ pub fn run() { #[cfg(mobile)] let builder = builder.plugin(mobile::share::init()); + let builder = builder.plugin(cookie_login::init()); + let builder = builder.setup(|app| { let data_dir = app .path() @@ -213,6 +238,33 @@ pub fn run() { }); } + // Full fetches of saved searches interrupted by closing the app. + { + let state = state.clone(); + tauri::async_runtime::spawn(async move { + tokio::time::sleep(std::time::Duration::from_secs(20)).await; + state.resume_full_fetches(); + }); + } + + // One-time repair: downloads that kept a second `.remux.mp4` copy. + { + let db = state.db.clone(); + tauri::async_runtime::spawn(async move { + match tokio::task::spawn_blocking(move || { + crate::library::LibraryScanner::fold_remux_copies(&db) + }) + .await + { + Ok(Ok(n)) if n > 0 => { + tracing::info!("[library] removed {n} duplicate .remux.mp4 copies") + } + Ok(Err(e)) => tracing::warn!("[library] remux cleanup failed: {e}"), + _ => {} + } + }); + } + // #19 watchlist auto-queue poller (desktop only). #[cfg(not(mobile))] state.spawn_watchlist_poller(); @@ -289,6 +341,10 @@ pub fn run() { commands::merge_duplicates, commands::list_cookie_sites, commands::save_site_cookies, + commands::cookie_login, + commands::list_site_categories, + commands::site_suggest, + commands::cookie_login_finish, commands::delete_site_cookies, commands::export_settings_backup, commands::import_settings_backup, @@ -306,6 +362,9 @@ pub fn run() { commands::open_scene_with_default, commands::batch_update_scenes, commands::list_pornhub_categories, + commands::set_player_fullscreen, + commands::storage_access_status, + commands::open_storage_access_settings, commands::probe_scene_metadata, commands::probe_library_durations, commands::ffmpeg_status, @@ -364,6 +423,22 @@ pub fn run() { commands::delete_stashbox_endpoint, commands::test_stashbox_endpoint, commands::query_stashdb_for_scene, + commands::query_online_items, + commands::online_tag_counts, + commands::online_search_info, + commands::update_online_search, + commands::fetch_online_details, + commands::clear_online_history, + commands::delete_online_items, + commands::online_bookmark, + commands::inapp_browse, + commands::android_share, + commands::online_unbookmark, + commands::online_is_bookmarked, + commands::delete_online_matching, + commands::start_full_fetch, + commands::stop_full_fetch, + commands::list_full_fetches, commands::list_unenriched_scenes, commands::search_stashdb_scenes, commands::search_stashdb_performers, diff --git a/src-tauri/src/library/mod.rs b/src-tauri/src/library/mod.rs index 2812bc7..cc8b87e 100644 --- a/src-tauri/src/library/mod.rs +++ b/src-tauri/src/library/mod.rs @@ -312,13 +312,13 @@ impl LibraryScanner { ) -> AppResult<(Option, Option, Option)> { let ffmpeg = FfmpegProcessor::new(app.clone()); let thumb = ffmpeg.extract_thumbnail(video_path).await.ok(); - let remuxed = if video_path.extension().and_then(|e| e.to_str()) == Some("mp4") { - ffmpeg.remux_faststart(video_path).await.ok() - } else { - None - }; - - let final_path: &Path = remuxed.as_deref().unwrap_or(video_path); + // Faststart remux in place (same path); a failure keeps the original. + if video_path.extension().and_then(|e| e.to_str()) == Some("mp4") { + if let Err(e) = ffmpeg.remux_faststart(video_path).await { + tracing::info!("[library] faststart remux skipped for {video_path:?}: {e}"); + } + } + let final_path: &Path = video_path; // Offload blocking hash computations to the blocking thread pool. let oshash = { @@ -360,6 +360,41 @@ impl LibraryScanner { Ok((phash, oshash, thumb_str)) } + /// One-time repair for downloads made while the faststart remux was kept + /// as a second file: the scene points at `.remux.mp4` and the + /// original `.mp4` sits next to it (sometimes imported by a scan + /// as a second scene). Keep the remux under the original name, delete + /// the duplicate file and any scene row that only referenced it. + /// Returns how many scenes were repaired. + pub fn fold_remux_copies(db: &Database) -> AppResult { + let mut fixed = 0; + for (id, path) in db.scenes_with_path_suffix(".remux.mp4")? { + let remux = Path::new(&path); + let Some(original) = path.strip_suffix(".remux.mp4").map(|s| format!("{s}.mp4")) else { + continue; + }; + let original_path = Path::new(&original); + if remux.exists() { + // A scan may have imported the duplicate original as its own + // scene; this download's scene (with metadata) wins. + for other in db.scene_ids_with_path(&original)? { + if other != id { + db.delete_scene(&other, false)?; + } + } + if original_path.exists() { + std::fs::remove_file(original_path)?; + } + std::fs::rename(remux, original_path)?; + } else if !original_path.exists() { + continue; + } + db.update_scene_path(&id, &original, None)?; + fixed += 1; + } + Ok(fixed) + } + /// (Re)compute MD5 + OSHASH for every library file missing them, without /// touching thumbnails. One-shot repair for libraries hashed before the /// fingerprint fix (old rows stored SHA-256 as `oshash`, since purged). @@ -449,3 +484,112 @@ impl LibraryScanner { }) } } + +#[cfg(test)] +mod remux_fold_tests { + use super::*; + + #[test] + fn folds_remux_copy_back_and_drops_duplicate_scene() { + let dir = tempfile::tempdir().unwrap(); + let db = Database::new(dir.path().to_path_buf()).unwrap(); + let original = dir.path().join("clip.mp4"); + let remux = dir.path().join("clip.remux.mp4"); + std::fs::write(&original, b"original").unwrap(); + std::fs::write(&remux, b"remuxed").unwrap(); + let s = |p: &Path| p.to_string_lossy().to_string(); + let download = db + .insert_scene( + "clip", + Some(&s(&remux)), + Some("https://site.test/v/1"), + &[], + &[], + None, + None, + None, + None, + None, + None, + ) + .unwrap(); + // A scan imported the leftover original as its own scene. + let scanned = db + .insert_scene( + "clip", + Some(&s(&original)), + None, + &[], + &[], + None, + None, + None, + None, + None, + None, + ) + .unwrap(); + + assert_eq!(LibraryScanner::fold_remux_copies(&db).unwrap(), 1); + assert!(!remux.exists()); + assert_eq!(std::fs::read(&original).unwrap(), b"remuxed"); + assert_eq!( + db.get_scene(&download).unwrap().path.as_deref(), + Some(s(&original).as_str()) + ); + assert!(db.get_scene(&scanned).is_err()); + // Idempotent. + assert_eq!(LibraryScanner::fold_remux_copies(&db).unwrap(), 0); + } +} + +#[cfg(test)] +mod delete_scene_tests { + use super::*; + + #[test] + fn delete_with_files_removes_video_and_thumb() { + let dir = tempfile::tempdir().unwrap(); + let db = Database::new(dir.path().to_path_buf()).unwrap(); + let video = dir.path().join("v.mp4"); + let thumb = dir.path().join("v.jpg"); + std::fs::write(&video, b"x").unwrap(); + std::fs::write(&thumb, b"y").unwrap(); + let s = |p: &Path| p.to_string_lossy().to_string(); + let id = db + .insert_scene( + "v", + Some(&s(&video)), + None, + &[], + &[], + Some(&s(&thumb)), + None, + None, + None, + None, + None, + ) + .unwrap(); + db.delete_scene(&id, true).unwrap(); + assert!(!video.exists() && !thumb.exists()); + assert!(db.get_scene(&id).is_err()); + // A file that is already gone doesn't block deleting the entry. + let id2 = db + .insert_scene( + "gone", + Some(&s(&dir.path().join("gone.mp4"))), + None, + &[], + &[], + None, + None, + None, + None, + None, + None, + ) + .unwrap(); + db.delete_scene(&id2, true).unwrap(); + } +} diff --git a/src-tauri/src/library/thumbnail.rs b/src-tauri/src/library/thumbnail.rs index a8ed283..3161790 100644 --- a/src-tauri/src/library/thumbnail.rs +++ b/src-tauri/src/library/thumbnail.rs @@ -20,7 +20,7 @@ pub async fn download_remote_thumbnail(thumbnail_url: &str, video_path: &Path) - } let client = reqwest::Client::builder() - .user_agent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36") + .user_agent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36") .timeout(THUMB_TIMEOUT) .redirect(reqwest::redirect::Policy::limited(5)) .build() diff --git a/src-tauri/src/media/ffmpeg.rs b/src-tauri/src/media/ffmpeg.rs index c605dad..54a97df 100644 --- a/src-tauri/src/media/ffmpeg.rs +++ b/src-tauri/src/media/ffmpeg.rs @@ -165,7 +165,38 @@ impl FfmpegProcessor { Ok(()) } + /// Move the MP4 index (`moov`) to the front so playback can start before + /// the whole file is read, replacing the file in place. The remux goes to + /// a temporary `.remux.mp4` that replaces the original once it is + /// complete; on any failure the original is left untouched and the + /// temporary file removed, so only one copy of the video remains. pub async fn remux_faststart(&self, video_path: &Path) -> AppResult { + let tmp = self.remux_faststart_copy(video_path).await; + let tmp = match tmp { + Ok(p) => p, + Err(e) => { + let _ = std::fs::remove_file(video_path.with_extension("remux.mp4")); + return Err(e); + } + }; + let original_len = std::fs::metadata(video_path).map(|m| m.len()).unwrap_or(0); + let remux_len = std::fs::metadata(&tmp).map(|m| m.len()).unwrap_or(0); + // A stream copy keeps the size within a few percent; anything much + // smaller is a truncated remux. + if remux_len == 0 || remux_len < original_len / 10 * 9 { + let _ = std::fs::remove_file(&tmp); + return Err(crate::error::AppError::Other(format!( + "remux produced an incomplete file ({remux_len} of {original_len} bytes)" + ))); + } + if let Err(e) = std::fs::rename(&tmp, video_path) { + let _ = std::fs::remove_file(&tmp); + return Err(e.into()); + } + Ok(video_path.to_path_buf()) + } + + async fn remux_faststart_copy(&self, video_path: &Path) -> AppResult { let out_path = video_path.with_extension("remux.mp4"); let args = vec![ "-y".to_string(), diff --git a/src-tauri/src/metadata/matcher.rs b/src-tauri/src/metadata/matcher.rs new file mode 100644 index 0000000..63e26e6 --- /dev/null +++ b/src-tauri/src/metadata/matcher.rs @@ -0,0 +1,414 @@ +//! Metadata matching for StashDB when fingerprints don't match — the usual +//! case for tube clips, which are re-encoded, cut, or carry a site intro. +//! +//! A few targeted text searches (cleaned title; performers + key title +//! words; studio / channel + key words) collect candidates, and each one is +//! scored against the library scene on title words, performers, studio and +//! duration. + +use crate::models::{Scene, StashSceneMatch}; +use std::collections::HashSet; + +/// Words that say nothing about which scene it is. +const STOP_WORDS: &[&str] = &[ + "a", "an", "and", "the", "with", "her", "his", "him", "she", "he", "gets", "get", "got", "in", + "on", "of", "to", "for", "by", "is", "my", "me", "i", "you", "your", "at", "from", "it", + "this", "that", "who", "while", "then", "into", "up", "out", "so", "but", "or", "as", "be", + "are", "was", "has", "have", "all", "very", "too", "s", "t", "de", "la", "el", "le", "und", +]; + +/// Tube-site noise in titles. +const NOISE_WORDS: &[&str] = &[ + "hd", + "4k", + "uhd", + "1080p", + "720p", + "480p", + "2160p", + "60fps", + "full", + "video", + "videos", + "porn", + "porno", + "xxx", + "sex", + "new", + "premium", + "exclusive", + "free", + "official", + "clip", + "trailer", + "preview", + "pornhub", + "xvideos", + "xhamster", + "xnxx", + "spankbang", + "eporner", + "youporn", + "redtube", + "tube8", + "com", + "net", + "www", + "red", + "onlyfans", + "of", + "fansly", + "watch", + "more", + "part", + "pt", + "vol", +]; + +fn normalize(text: &str) -> String { + text.to_lowercase() + .chars() + .map(|c| if c.is_alphanumeric() { c } else { ' ' }) + .collect::() + .split_whitespace() + .collect::>() + .join(" ") +} + +fn squash(text: &str) -> String { + normalize(text).replace(' ', "") +} + +/// Meaningful words of a title, in order. +fn title_words(text: &str) -> Vec { + let mut seen = HashSet::new(); + normalize(text) + .split(' ') + .filter(|w| !w.is_empty() && !STOP_WORDS.contains(w) && !NOISE_WORDS.contains(w)) + .filter(|w| seen.insert(w.to_string())) + .map(str::to_string) + .collect() +} + +/// Title with tube noise removed, for searching. +pub fn clean_title(title: &str) -> String { + // "Title - Pornhub.com", "Title | SiteName": keep the first part when + // it still has a few words. + let main = title + .split([' ', '\t']) + .collect::>() + .join(" ") + .split(" | ") + .next() + .unwrap_or(title) + .to_string(); + let main = match main.rsplit_once(" - ") { + Some((head, tail)) if title_words(head).len() >= 3 && title_words(tail).len() <= 2 => { + head.to_string() + } + _ => main, + }; + normalize(&main) + .split(' ') + .filter(|w| !w.is_empty() && !NOISE_WORDS.contains(w)) + .take(12) + .collect::>() + .join(" ") +} + +/// Most distinctive title words (longest first), skipping performer names. +fn key_words(scene: &Scene, n: usize) -> Vec { + let names: HashSet = scene + .performers + .iter() + .flat_map(|p| title_words(p)) + .collect(); + let mut words: Vec = title_words(&scene.title) + .into_iter() + .filter(|w| !names.contains(w) && w.len() >= 4 && !w.chars().all(|c| c.is_numeric())) + .collect(); + words.sort_by_key(|w| std::cmp::Reverse(w.len())); + words.truncate(n); + words +} + +/// Up to three searches for one scene, most specific first. +pub fn search_terms(scene: &Scene) -> Vec { + let mut terms = Vec::new(); + let title = clean_title(&scene.title); + if !title.is_empty() { + terms.push(title); + } + let words = key_words(scene, 3).join(" "); + if !scene.performers.is_empty() { + let names = scene + .performers + .iter() + .take(2) + .map(String::as_str) + .collect::>() + .join(" "); + terms.push(format!("{names} {words}").trim().to_string()); + } + let studio = scene + .studio_name + .as_deref() + .or(scene.channel.as_deref()) + .map(str::trim) + .filter(|s| !s.is_empty()); + if let Some(studio) = studio { + if !words.is_empty() { + terms.push(format!("{studio} {words}")); + } + } + let mut seen = HashSet::new(); + terms.retain(|t| !t.is_empty() && seen.insert(normalize(t))); + terms.truncate(3); + terms +} + +/// How well `candidate` fits `scene`: 0.0–1.0 plus the reasons shown in the UI. +pub fn score(scene: &Scene, candidate: &StashSceneMatch) -> (f32, Vec) { + let mut total = 0.0f32; + let mut weight = 0.0f32; + let mut reasons = Vec::new(); + let local_text = normalize(&format!( + "{} {} {}", + scene.title, + scene.performers.join(" "), + scene.channel.as_deref().unwrap_or("") + )); + let local_squashed = local_text.replace(' ', ""); + + // Title words: how much of the official title the local one contains. + let cand: HashSet = title_words(&candidate.title).into_iter().collect(); + let local: HashSet = title_words(&local_text).into_iter().collect(); + if !cand.is_empty() { + let common = cand.intersection(&local).count() as f32; + let s = 0.75 * common / cand.len() as f32 + 0.25 * common / local.len().max(1) as f32; + total += 0.45 * s; + weight += 0.45; + if s >= 0.3 { + reasons.push(format!("title {}%", (s * 100.0).round())); + } + } + + // Performers: named in the local performers or in the title itself. + let local_names: HashSet = scene.performers.iter().map(|p| squash(p)).collect(); + if !candidate.performers.is_empty() { + let matched: Vec<&str> = candidate + .performers + .iter() + .filter(|p| { + let n = squash(&p.name); + n.len() >= 4 && (local_names.contains(&n) || local_squashed.contains(&n)) + }) + .map(|p| p.name.as_str()) + .collect(); + let s = matched.len() as f32 / candidate.performers.len().min(3) as f32; + total += 0.30 * s.min(1.0); + weight += 0.30; + if !matched.is_empty() { + reasons.push(matched.join(", ")); + } + } + + // Studio / channel: only when the library scene has one to compare + // (a studio named in the title still counts as a hit). + if let Some(studio) = candidate.studio.as_ref().map(|s| squash(&s.name)) { + let local_studio = squash( + scene + .studio_name + .as_deref() + .or(scene.channel.as_deref()) + .unwrap_or(""), + ); + let hit = studio.len() >= 3 && (local_studio == studio || local_squashed.contains(&studio)); + if hit || !local_studio.is_empty() { + total += if hit { 0.10 } else { 0.0 }; + weight += 0.10; + } + if hit { + reasons.push( + candidate + .studio + .as_ref() + .map(|s| s.name.clone()) + .unwrap_or_default(), + ); + } + } + + // Duration: tube uploads add intros / outros or are cut down. + if let (Some(local), Some(theirs)) = (scene.duration, candidate.duration) { + if local > 0 && theirs > 0 { + let diff = local.abs_diff(theirs); + let s = if diff <= 20 { + reasons.push("same length".into()); + 1.0 + } else if diff <= 90 { + reasons.push(format!("length ±{diff}s")); + 0.7 + } else if local < theirs { + // A clip of a longer scene. + 0.3 + } else { + 0.0 + }; + total += 0.15 * s; + weight += 0.15; + } + } + + if weight == 0.0 { + return (0.0, reasons); + } + ((total / weight).clamp(0.0, 1.0), reasons) +} + +/// Score, sort best first and drop duplicates. +pub fn rank(scene: &Scene, candidates: Vec) -> Vec { + let mut seen = HashSet::new(); + let mut out: Vec = candidates + .into_iter() + .filter(|c| seen.insert(c.stash_id.clone())) + .map(|mut c| { + let (s, reasons) = score(scene, &c); + c.score = Some(s); + c.match_reasons = reasons; + c + }) + .collect(); + out.sort_by(|a, b| { + b.score + .unwrap_or(0.0) + .partial_cmp(&a.score.unwrap_or(0.0)) + .unwrap_or(std::cmp::Ordering::Equal) + }); + out +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::models::{StashPerformerRef, StashStudioRef}; + + fn scene(title: &str, performers: &[&str], channel: Option<&str>, dur: u32) -> Scene { + Scene { + title: title.into(), + performers: performers.iter().map(|p| p.to_string()).collect(), + channel: channel.map(str::to_string), + duration: Some(dur), + ..Default::default() + } + } + + fn cand(id: &str, title: &str, performers: &[&str], studio: &str, dur: u32) -> StashSceneMatch { + StashSceneMatch { + stash_id: id.into(), + title: title.into(), + performers: performers + .iter() + .map(|p| StashPerformerRef { + name: p.to_string(), + ..Default::default() + }) + .collect(), + studio: Some(StashStudioRef { + name: studio.into(), + ..Default::default() + }), + duration: Some(dur), + ..Default::default() + } + } + + #[test] + fn cleans_tube_titles() { + assert_eq!( + clean_title("Riley Reid Gets Her Massage (FULL VIDEO) HD 1080p - Pornhub.com"), + "riley reid gets her massage" + ); + assert_eq!(clean_title("A Night In | Brazzers"), "a night in"); + } + + #[test] + fn search_terms_use_performers_and_studio() { + let s = scene( + "Stepsister helps with homework after school 😈", + &["Riley Reid"], + Some("Brazzers"), + 1500, + ); + let terms = search_terms(&s); + assert_eq!(terms[0], "stepsister helps with homework after school"); + assert!(terms[1].starts_with("Riley Reid "), "{terms:?}"); + assert!(terms[1].contains("stepsister"), "{terms:?}"); + assert!(terms[2].starts_with("Brazzers "), "{terms:?}"); + } + + #[test] + fn ranks_the_right_scene_first() { + let s = scene( + "Riley Reid massage turns into more - Brazzers full video", + &[], + Some("Brazzers Official"), + 1830, + ); + let ranked = rank( + &s, + vec![ + cand( + "x", + "Office Party", + &["Someone Else"], + "Reality Kings", + 2400, + ), + cand( + "ok", + "Massage Turns Into More", + &["Riley Reid"], + "Brazzers", + 1801, + ), + cand( + "ok", + "Massage Turns Into More", + &["Riley Reid"], + "Brazzers", + 1801, + ), + cand("y", "Massage Therapy", &["Other Girl"], "Brazzers", 900), + ], + ); + assert_eq!(ranked.len(), 3); + assert_eq!(ranked[0].stash_id, "ok"); + let top = ranked[0].score.unwrap(); + assert!(top > 0.85, "{top} {:?}", ranked[0].match_reasons); + assert!(ranked[0].match_reasons.iter().any(|r| r == "Riley Reid")); + assert!(ranked[1].score.unwrap() < 0.6); + assert!(ranked[2].score.unwrap() < 0.2); + } + + #[test] + fn clip_of_longer_scene_still_scores() { + let s = scene( + "Riley Reid massage turns into more", + &["Riley Reid"], + None, + 600, + ); + let (score, _) = score( + &s, + &cand( + "ok", + "Massage Turns Into More", + &["Riley Reid"], + "Brazzers", + 1800, + ), + ); + assert!(score > 0.75, "{score}"); + } +} diff --git a/src-tauri/src/metadata/mod.rs b/src-tauri/src/metadata/mod.rs index c4e4168..45a5767 100644 --- a/src-tauri/src/metadata/mod.rs +++ b/src-tauri/src/metadata/mod.rs @@ -1 +1,2 @@ +pub mod matcher; pub mod stashbox; diff --git a/src-tauri/src/metadata/stashbox.rs b/src-tauri/src/metadata/stashbox.rs index e12b3e4..af91f8f 100644 --- a/src-tauri/src/metadata/stashbox.rs +++ b/src-tauri/src/metadata/stashbox.rs @@ -25,14 +25,15 @@ pub const STASHDB_ENDPOINT: &str = "https://stashdb.org/graphql"; /// Shared scene selection — every field here exists on stash-box `Scene`: /// `images` (not `image`), `performers` as appearances wrapping `performer`, -/// `date` (present on all box versions; `release_date` only on newer ones). +/// `release_date` (StashDB removed the old `date` field; boxes that only +/// know `date` are retried via [`StashBoxClient::graphql_scenes`]). /// Kept in one place and interpolated into each query so the field lists /// can't drift apart (unknown fields fail the whole call server-side). /// `concat!` can't take a const, so the full documents are built once via /// `LazyLock` instead of per call. const SCENE_SELECTION: &str = r#"id title - date + release_date details duration images { url } @@ -76,6 +77,35 @@ static SEARCH_SCENES_QUERY: std::sync::LazyLock = std::sync::LazyLock::n ) }); +/// Full-text search over titles, performers and studios (what Stash's own +/// Tagger uses). `queryScenes(title:)` above only filters titles, so a +/// performer name or a tube title with extra words never matched. +static SEARCH_SCENES_TERM_QUERY: std::sync::LazyLock = std::sync::LazyLock::new(|| { + format!( + r#"query SearchScenesTerm($term: String!) {{ + searchScenes(term: $term, per_page: 15) {{ + count + scenes {{ + {SCENE_SELECTION} + }} + }} +}} +"# + ) +}); + +/// Older stash-box servers: the deprecated list form. +static SEARCH_SCENE_LEGACY_QUERY: std::sync::LazyLock = std::sync::LazyLock::new(|| { + format!( + r#"query SearchSceneLegacy($term: String!) {{ + searchScene(term: $term, limit: 15) {{ + {SCENE_SELECTION} + }} +}} +"# + ) +}); + const SEARCH_PERFORMERS_QUERY: &str = r#" query SearchPerformers($name: String!) { queryPerformers(input: { name: $name, per_page: 10 }) { @@ -161,6 +191,15 @@ impl StashBoxClient { .filter_map(|e| e.get("message").and_then(|m| m.as_str())) .collect::>() .join("; "); + // StashDB reports a missing/invalid key as a GraphQL error + // with HTTP 200, not a 401. + if msg.to_ascii_lowercase().contains("not authorized") { + return Err(AppError::InvalidInput(format!( + "stash-box rejected the request as not authorized — check the API key \ + for {} in Settings → Metadata", + self.endpoint + ))); + } return Err(AppError::Site(format!( "stash-box GraphQL error: {}", if msg.is_empty() { @@ -174,6 +213,18 @@ impl StashBoxClient { Ok(body.get("data").cloned().unwrap_or(Value::Null)) } + /// Scene queries select `release_date`; older stash-box versions only + /// have `date`, so retry once with that field when the box rejects it. + async fn graphql_scenes(&self, query: &str, variables: Value) -> AppResult { + match self.graphql(query, variables.clone()).await { + Err(AppError::Site(msg)) if msg.contains("release_date") => { + self.graphql(&query.replace("release_date", "date"), variables) + .await + } + other => other, + } + } + /// Test credentials with a cheap query. Returns box name when known. pub async fn test_connection(&self) -> AppResult { let data = self.graphql("query Me { me { name } }", json!({})).await?; @@ -201,7 +252,7 @@ impl StashBoxClient { )); } let data = self - .graphql( + .graphql_scenes( FIND_BY_FINGERPRINTS_QUERY.as_str(), json!({ "fingerprints": [inputs] }), ) @@ -229,7 +280,7 @@ impl StashBoxClient { return Err(AppError::InvalidInput("stash_id is required".into())); } let data = self - .graphql(FIND_SCENE_QUERY.as_str(), json!({ "id": stash_id })) + .graphql_scenes(FIND_SCENE_QUERY.as_str(), json!({ "id": stash_id })) .await?; let raw = data.get("findScene").cloned().unwrap_or(Value::Null); if raw.is_null() { @@ -241,21 +292,43 @@ impl StashBoxClient { Ok(Some(m)) } - /// Title search fallback when no fingerprints match. The box treats - /// `title` as a LIKE query unless quoted. + /// Text search when no fingerprints match: titles, performers and + /// studios (`searchScenes`), falling back to the deprecated + /// `searchScene` and then the title-only `queryScenes` filter on boxes + /// that lack them. pub async fn search_scenes(&self, title: &str) -> AppResult> { - let title = title.trim(); - if title.is_empty() { + let term = search_term(title); + if term.is_empty() { return Err(AppError::InvalidInput("title is required".into())); } - let data = self - .graphql(SEARCH_SCENES_QUERY.as_str(), json!({ "title": title })) - .await?; - let raw = data - .get("queryScenes") - .and_then(|q| q.get("scenes")) - .cloned() - .unwrap_or(Value::Null); + let vars = json!({ "term": term }); + let raw = match self + .graphql_scenes(SEARCH_SCENES_TERM_QUERY.as_str(), vars.clone()) + .await + { + Ok(data) => data + .get("searchScenes") + .and_then(|q| q.get("scenes")) + .cloned() + .unwrap_or(Value::Null), + Err(AppError::Site(msg)) if msg.contains("searchScenes") => { + match self + .graphql_scenes(SEARCH_SCENE_LEGACY_QUERY.as_str(), vars) + .await + { + Ok(data) => data.get("searchScene").cloned().unwrap_or(Value::Null), + Err(AppError::Site(msg)) if msg.contains("searchScene") => self + .graphql_scenes(SEARCH_SCENES_QUERY.as_str(), json!({ "title": term })) + .await? + .get("queryScenes") + .and_then(|q| q.get("scenes")) + .cloned() + .unwrap_or(Value::Null), + Err(e) => return Err(e), + } + } + Err(e) => return Err(e), + }; let mut out = Vec::new(); flatten_scene_values(&raw, &mut out); for m in &mut out { @@ -345,6 +418,25 @@ impl StashBoxClient { /// `FingerprintQueryInput` rows (`{ hash, algorithm }`) for the hashes worth /// sending. Our thumbnail phash is excluded: Stash stores int64 phashes and /// ours (base64, different algorithm) could never match. +/// Tube titles → a search term: emoji and decoration dropped, words kept +/// ("🔥❤️SisLovesMe - Lacy Lennon Fucks Her Stepbro" → +/// "SisLovesMe Lacy Lennon Fucks Her Stepbro"). +fn search_term(title: &str) -> String { + title + .chars() + .map(|c| { + if c.is_alphanumeric() || c == '\'' || c == '.' { + c + } else { + ' ' + } + }) + .collect::() + .split_whitespace() + .collect::>() + .join(" ") +} + fn fingerprint_inputs(md5: Option<&str>, oshash: Option<&str>) -> Vec { let mut inputs = Vec::new(); if let Some(h) = md5.filter(|h| !h.trim().is_empty()) { @@ -474,6 +566,7 @@ fn parse_scene_value(v: &Value) -> Option { image: first_image_url(v), duration: v.get("duration").and_then(|d| d.as_u64()).map(|d| d as u32), endpoint_id: String::new(), + ..Default::default() }) } @@ -500,6 +593,17 @@ fn parse_performer_value(v: &Value) -> Option { #[cfg(test)] mod tests { + + #[test] + fn search_term_strips_decoration() { + assert_eq!( + search_term("🔥❤️SisLovesMe - Lacy Lennon Fucks Her Stepbro"), + "SisLovesMe Lacy Lennon Fucks Her Stepbro" + ); + assert_eq!(search_term(" Lacy Lennon "), "Lacy Lennon"); + assert_eq!(search_term("🔥❤️"), ""); + } + use super::*; // Keeps the shared selection pasted into each query in sync: every query diff --git a/src-tauri/src/mobile/binary_installer.rs b/src-tauri/src/mobile/binary_installer.rs index b3543c7..5c2913b 100644 --- a/src-tauri/src/mobile/binary_installer.rs +++ b/src-tauri/src/mobile/binary_installer.rs @@ -10,9 +10,12 @@ fn yt_dlp_download_url() -> &'static str { "https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp.exe" } else if cfg!(target_os = "macos") { "https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp_macos" + } else if cfg!(target_arch = "aarch64") { + // Linux desktop — never use on Android (install_yt_dlp hard-fails there). + // Standalone builds bundle curl_cffi for `--impersonate`. + "https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp_linux_aarch64" } else { - // Linux desktop — never use on Android (install_yt_dlp hard-fails there) - "https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp" + "https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp_linux" } } diff --git a/src-tauri/src/mobile/ytdlp_bridge.rs b/src-tauri/src/mobile/ytdlp_bridge.rs index e17b434..48aa0e2 100644 --- a/src-tauri/src/mobile/ytdlp_bridge.rs +++ b/src-tauri/src/mobile/ytdlp_bridge.rs @@ -305,6 +305,36 @@ pub fn ensure_media_tools(app: &AppHandle) -> AppResult { } } +/// Open a local file in another Android app (system chooser) through the +/// FileProvider; `file://` URIs are rejected across apps on Android 7+. +#[cfg_attr(not(target_os = "android"), allow(dead_code))] +pub fn open_file(app: &AppHandle, path: &str) -> AppResult<()> { + #[cfg(target_os = "android")] + { + use serde_json::json; + with_plugin_state(app, |state| { + let response: PluginResponse = state + .handle()? + .run_mobile_plugin("openFile", json!({ "path": path, "mime": "" })) + .map_err(|e| AppError::Other(format!("openFile failed: {e}")))?; + if response.ok.unwrap_or(false) { + Ok(()) + } else { + Err(AppError::Other( + response + .message + .unwrap_or_else(|| "no app can open this file".into()), + )) + } + }) + } + #[cfg(not(target_os = "android"))] + { + let _ = (app, path); + Err(AppError::Other("open_file is Android-only".into())) + } +} + /// Run ffmpeg or ffprobe through the Kotlin plugin (Android-native binaries). /// Returns combined stdout/stderr text on success. pub fn execute_media(app: &AppHandle, tool: &str, args: &[String]) -> AppResult { @@ -409,3 +439,87 @@ pub fn cancel_pending_resume(app: &AppHandle) { let _ = app; } } + +/// Immersive video fullscreen on Android: hide the system bars and (for +/// landscape video) lock to sensor-landscape. No-op elsewhere — desktop and +/// browsers use the HTML Fullscreen API. +#[cfg_attr(not(target_os = "android"), allow(dead_code))] +pub fn set_immersive(app: &AppHandle, enabled: bool, landscape: bool) -> AppResult<()> { + #[cfg(target_os = "android")] + { + use serde_json::json; + with_plugin_state(app, |state| { + let response: PluginResponse = state + .handle()? + .run_mobile_plugin( + "setImmersive", + json!({ "enabled": enabled, "landscape": landscape }), + ) + .map_err(|e| AppError::Other(format!("setImmersive failed: {e}")))?; + if response.ok.unwrap_or(false) { + Ok(()) + } else { + Err(AppError::Other( + response + .message + .unwrap_or_else(|| "setImmersive failed".into()), + )) + } + }) + } + #[cfg(not(target_os = "android"))] + { + let _ = (app, enabled, landscape); + Ok(()) + } +} + +/// Android: may ArcHive manage all files in shared storage (needed to delete +/// library files in e.g. Download/ArcHive that this install didn't create)? +/// Other platforms: always true. +#[cfg_attr(not(target_os = "android"), allow(dead_code))] +pub fn storage_access(app: &AppHandle) -> AppResult { + #[cfg(target_os = "android")] + { + use serde_json::json; + with_plugin_state(app, |state| { + let response: PluginResponse = state + .handle()? + .run_mobile_plugin("storageAccess", json!({})) + .map_err(|e| AppError::Other(format!("storageAccess failed: {e}")))?; + Ok(response.ok.unwrap_or(false)) + }) + } + #[cfg(not(target_os = "android"))] + { + let _ = app; + Ok(true) + } +} + +/// Android: open the system "All files access" screen for ArcHive. +#[cfg_attr(not(target_os = "android"), allow(dead_code))] +pub fn open_storage_access_settings(app: &AppHandle) -> AppResult<()> { + #[cfg(target_os = "android")] + { + use serde_json::json; + with_plugin_state(app, |state| { + let response: PluginResponse = state + .handle()? + .run_mobile_plugin("openStorageAccessSettings", json!({})) + .map_err(|e| AppError::Other(format!("openStorageAccessSettings failed: {e}")))?; + if response.ok.unwrap_or(false) { + Ok(()) + } else { + Err(AppError::Other(response.message.unwrap_or_else(|| { + "could not open the settings screen".into() + }))) + } + }) + } + #[cfg(not(target_os = "android"))] + { + let _ = app; + Ok(()) + } +} diff --git a/src-tauri/src/models.rs b/src-tauri/src/models.rs index 1f889d3..6e8e09c 100644 --- a/src-tauri/src/models.rs +++ b/src-tauri/src/models.rs @@ -1,1193 +1,1568 @@ -use serde::{Deserialize, Serialize}; - -#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] -#[serde(rename_all = "lowercase")] -pub enum BrowseKind { - Tag, - Model, - Channel, - Search, - Video, - Category, - Livestream, -} - -#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq, Default)] -#[serde(rename_all = "lowercase")] -pub enum BrowseOrientation { - #[default] - Straight, - Gay, - Lesbian, - Transgender, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct MediaItem { - pub id: String, - pub title: String, - pub url: String, - pub thumbnail: Option, - pub duration: Option, - pub site_id: String, - pub performers: Vec, - pub tags: Vec, - #[serde(default, skip_serializing_if = "Option::is_none")] - pub description: Option, - #[serde(default, skip_serializing_if = "Option::is_none")] - pub channel: Option, - #[serde(default, skip_serializing_if = "Option::is_none")] - pub is_live: Option, - #[serde(default, skip_serializing_if = "Option::is_none")] - pub viewers: Option, - #[serde(default, skip_serializing_if = "Option::is_none")] - pub age: Option, - #[serde(default, skip_serializing_if = "Option::is_none")] - pub gender: Option, - #[serde(default, skip_serializing_if = "Option::is_none")] - pub stream_url: Option, - #[serde(default, skip_serializing_if = "Option::is_none")] - pub embed_url: Option, -} - -#[derive(Debug, Clone, Serialize, Deserialize, Default)] -pub struct BrowsePage { - pub items: Vec, - pub page: u32, - pub has_more: bool, - pub total: Option, - /// True when this page was served from the local browse_cache table. - #[serde(default)] - pub from_cache: bool, - /// Age of the cached payload in seconds when `from_cache` is true. - #[serde(default, skip_serializing_if = "Option::is_none")] - pub cache_age_secs: Option, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct BrowseQuery { - pub kind: BrowseKind, - pub slug: String, - pub page: u32, - #[serde(default, skip_serializing_if = "Option::is_none")] - pub orientation: Option, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct SiteInfo { - pub id: String, - pub display_name: String, - pub base_url: String, - pub supported_kinds: Vec, - pub requires_cookies: bool, -} - -#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] -#[serde(rename_all = "lowercase")] -pub enum DownloadStatus { - Pending, - Active, - Paused, - WaitingForWifi, - Completed, - Failed, - Cancelled, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct BulkImportResult { - pub queued: u32, - pub expanded: u32, - pub skipped: u32, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct DownloadJob { - pub id: String, - pub url: String, - pub adapter: String, - pub status: DownloadStatus, - pub progress: f32, - pub output_path: Option, - pub error: Option, - pub title: Option, - pub created_at: String, - #[serde(default)] - pub retry_count: u32, - #[serde(default, skip_serializing_if = "Option::is_none")] - pub last_retry_at: Option, - /// 1-based position in the active queue (pending + active jobs ahead, - /// plus one). `None` when the backend hasn't computed it yet. Mobile - /// surfaces this as "#N in queue" so users know how long they'll wait. - #[serde(default, skip_serializing_if = "Option::is_none")] - pub queue_position: Option, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct DownloadPlan { - pub url: String, - pub output_template: String, - pub tool: DownloadTool, - pub title: Option, - pub performers: Vec, - pub tags: Vec, - pub adapter_id: String, - #[serde(default, skip_serializing_if = "Option::is_none")] - pub thumbnail_url: Option, - #[serde(default, skip_serializing_if = "Option::is_none")] - pub duration: Option, - #[serde(default, skip_serializing_if = "Option::is_none")] - pub channel: Option, - /// HTTP Referer to send for [`DownloadTool::DirectHttp`] downloads. - /// Needed by Referer-gated CDNs (e.g. PornHub's phncdn). - #[serde(default, skip_serializing_if = "Option::is_none")] - pub referer: Option, - /// Initial download status (e.g., WaitingForWifi). If None, defaults to Pending. - #[serde(default, skip_serializing_if = "Option::is_none")] - pub initial_status: Option, -} - -#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] -#[serde(rename_all = "snake_case")] -pub enum DownloadTool { - YtDlp, - GalleryDl, - DirectHttp, - /// HLS playlist saved to MP4 via ffmpeg stream-copy. Used when only a - /// gated `.m3u8` exists (e.g. PornHub): ffmpeg sends Referer/Cookie - /// headers that plain HTTP and `