feat: more dials for web search providers

CHANGELOG.md

@@ -4,6 +4,36 @@ All notable changes to agentty. Versions follow [SemVer](https://semver.org/).
 
 ## [Unreleased]
 
+### Added
+- **Web Search: more settings per service.** Every new setting changes
+  which results come back, not what a search costs. Tavily (both tiers)
+  gains *Exact phrases*, *Language match*, *Country* and *Safe search*,
+  and Tavily Free gains *Depth*. Brave API gains *Spell correction*.
+  Firecrawl gains *Safe search*, *Sort* (newest first) and *Location*.
+  Exa gains *Safe search* and *Country*, and Perplexity gains *Mode*.
+  Brave API also takes a *Goggle*: the URL of a hosted rules file that
+  re-ranks its results. *Goggle* and *Location* are typed rows, saved only
+  when the service would accept them.
+
+### Changed
+- **Web Search: dials list every documented value.** Past hour on Firecrawl,
+  Serper and Exa. All six Exa modes, including deep-lite and deep-reasoning
+  (deep-reasoning can take 12-40 seconds, longer than the 30-second request
+  limit). Brave's *All regions* country. All 68 DuckDuckGo regions.
+  Firecrawl's categories are now Developer, Research, PDF and Government;
+  GitHub is gone from Firecrawl's API, so a saved GitHub choice is no longer
+  sent.
+- **Web Search: Firecrawl's Research category keeps working after
+  2026-11-16.** From that date Firecrawl answers it with research papers
+  instead of web pages. Each paper is shown with a link to its page on
+  arXiv, PubMed, PubMed Central or doi.org, or to Firecrawl's own page for
+  the paper when it has none of those, and its abstract as the snippet.
+- **Web Search: Never return reaches Perplexity's own domain filter.**
+  Excluded sites used to be added to Perplexity searches as `-site:`
+  words, which Perplexity does not document. They are now sent through its
+  documented exclude filter, as for Tavily, Firecrawl and Exa. Perplexity
+  takes up to 20; any more are still added as `-site:`.
+
 ## [0.9.25] - 2026-10-16
 
 ### Changed

docs/website/configuration.md

@@ -87,7 +87,7 @@ The Smart Mode *feature* toggles (which layers run) live in the `Ctrl+S` overlay
 | `AGENTTY_WEB_SEARCH` | `auto` (default), `on`, or `off`. `auto` advertises `web_search` and leaves the result count to the model, as agentty did before the Web Search pane. `on` applies your `AGENTTY_WEB_SEARCH_COUNT` and `_MAX`. `off` removes `web_search` from every request, from the tool list `agentty mcp-serve` offers to other MCP clients, and refuses any call to it (a replayed thread, a call injected by an ACP client). `0`/`false`/`no` mean `off`; `1`/`true`/`yes` mean `auto` (search available, the model picks the count). |
 | `AGENTTY_WEB_SEARCH_COUNT` | Results when the model does not say how many. Applies only in `on`. Default `10`; range 1–20. |
 | `AGENTTY_WEB_SEARCH_MAX` | Most results the model may ask for in one search; a larger request is clamped, and `0` or a negative count becomes `1`. Applies only in `on`. Default `20`; range 1–50. A search returns at most one result page from the service that answers, and each service caps it: 20 on Brave, Tavily, Serper and Perplexity, 30 on DuckDuckGo, 100 on Firecrawl and Exa. A larger request is cut to that cap. |
-| `AGENTTY_WEB_SEARCH_EXCLUDE` | Domains never to return, space- or comma-separated (`pinterest.com w3schools.com`). Pasted URLs and `www.` are reduced to the host. Sent as `-site:` operators to Brave, DuckDuckGo, Serper and Perplexity, and as the service's own domain filter to Tavily, Firecrawl and Exa, so an excluded site never comes back and its slot goes to another result. |
+| `AGENTTY_WEB_SEARCH_EXCLUDE` | Domains never to return, space- or comma-separated (`pinterest.com w3schools.com`). Pasted URLs and `www.` are reduced to the host. Sent as `-site:` operators to Brave, DuckDuckGo and Serper, and as the service's own domain filter to Tavily, Firecrawl, Exa and Perplexity, so an excluded site never comes back and its slot goes to another result. Up to 20 sites go in the filter; any beyond that are added to the query as `-site:`. |
 | `AGENTTY_WEB_SEARCH_PRIMARY`, `_SECONDARY`, `_FALLBACK` | Which service answers first, second and last; the first to succeed wins. A service id: `brave-free`, `ddg-free`, `tavily-free`, `firecrawl-free` (no account, card or key) or `brave-api`, `tavily-api`, `firecrawl-api`, `exa-api`, `serper-api`, `perplexity-api` (need an API key). `_SECONDARY` and `_FALLBACK` also take `none`. Defaults: `brave-free`, `ddg-free`, `tavily-free`. |
 | `AGENTTY_WEB_SEARCH_LANGUAGE` | `auto` (default) or a two-letter code (`en`, `de`, `ja`, ...) sent to the services that take one: Tavily, Brave API, Serper, Perplexity. The others ignore it. |
 | `AGENTTY_WEB_SEARCH_KEY_<SERVICE>` | An API key, taken from the environment instead of the key store: `AGENTTY_WEB_SEARCH_KEY_EXA_API`, `_BRAVE_API`, `_TAVILY_API`, `_FIRECRAWL_API`, `_SERPER_API`, `_PERPLEXITY_API`. Wins over a stored key and locks that row in the pane. Never written to disk. |
@@ -98,7 +98,7 @@ The **Free** services need no account, card or key, but they are best effort: Br
 
 The **API key** services are the vendors' documented interfaces and do not depend on a page layout. Paste a key into the row under the slot that uses it and press Enter. It is stored encrypted (the OS keystore when `AGENTTY_USE_KEYSTORE` is set, otherwise a machine-bound sealed file) and is never written to `settings.json`; the row then reads `✓ key saved` and stays locked until *Remove key* forgets it. A key from `AGENTTY_WEB_SEARCH_KEY_<SERVICE>` reads `● key from` the variable and is locked for as long as it is set. A keyed service chosen with no key is reported as such in the search result and the next slot is tried.
 
-Each service offers up to three extra settings (freshness, region, category, ...) which appear under its slot, and none for a service that has none. They are saved per service in `settings.json` under `web_search.dials`, so a choice survives moving the service between slots.
+Each service offers up to seven extra settings which appear under its slot, and none for a service that has none. Most are a list to pick from: freshness, region, safe search, spell correction, exact phrases, sort order, and so on. Two are typed: Brave API's *Goggle* takes the `https://` URL of a hosted Goggle (a rules file on GitHub, GitLab or a Gist, registered at [search.brave.com/goggles/create](https://search.brave.com/goggles/create)) that re-ranks Brave's results, and Firecrawl's *Location* takes a place such as `San Francisco,California,United States`. A typed value is saved when you leave the row, and only if it is one the service accepts; otherwise the row goes back to what was saved. Empty the row to clear it. Settings are saved per service in `settings.json` under `web_search.dials`, so a choice survives moving the service between slots. Some only take effect together with another: Tavily's *Country* applies with the General topic, its *Safe search* not with the fast depths, and *Language match* only when a language is set.
 
 Kagi, Mojeek and Linkup are not offered yet: their request and response formats could not be verified against documentation or a real reply.
 

include/agentty/domain/web_search_config.hpp

@@ -30,9 +30,10 @@
 // What is and is not known about `-site:` support: DuckDuckGo documents it;
 // Brave honours it (checked live: a query that returns ten pinterest.com hits
 // returns none with `-site:pinterest.com`; Brave's own docs spell the operator
-// `NOT site:`). Services with a native domain filter (Tavily, Firecrawl, Exa)
-// receive the list as a request field instead, which mcp-cpp does by lifting
-// the `-site:` tokens back out of the query.
+// `NOT site:`). Services with a native domain filter (Tavily, Firecrawl, Exa,
+// Perplexity) receive the list as a request field instead, which mcp-cpp does
+// by lifting the `-site:` tokens back out of the query (at most 20; any more
+// stay in the query).
 //
 // Pure: no IO, no globals. The tool layer installs a value of this type
 // (tool/web_search_policy.hpp); the runtime persists it through the settings
@@ -120,20 +121,24 @@ struct Config {
     std::string language = "auto";
 
     // Per-SERVICE extra settings (freshness, region, ...), keyed
-    // "<service-id>.<dial-id>" -> value id. Per service rather than per slot
+    // "<service-id>.<dial-id>" -> value id, or the typed text for a free-text
+    // dial (a Brave Goggle URL). Per service rather than per slot
     // so choosing Exa for Fallback after trying it as Primary keeps what was
     // set. Only non-default values are stored. Values are not checked here:
-    // mcp-cpp sends a value only if the service lists it.
+    // mcp-cpp sends a value only if it passes web_search_dial_value_ok.
     std::map<std::string, std::string> dials;
 
     [[nodiscard]] bool operator==(const Config&) const = default;
 };
 
 // What a stored dial entry must look like: key "<service>.<dial>" and a value,
-// both short and printable. Applied on load AND on save, so a hand-edited
+// both printable, the key short and the value within kMaxDialValueBytes.
+// Applied on load AND on save, so a hand-edited
 // settings.json cannot hold what the pane could not produce, and junk keys
 // cannot crowd real ones out of the cap.
 inline constexpr std::size_t kMaxDials = 64;
+// Room for the longest free-text dial mcp-cpp declares: a 2048-byte Goggle URL.
+inline constexpr std::size_t kMaxDialValueBytes = 2048;
 [[nodiscard]] bool dial_entry_ok(std::string_view key, std::string_view value) noexcept;
 
 // The typed view of Config::mode_text. Anything unrecognised reads as Auto —

include/agentty/runtime/panel/web_search_form.hpp

@@ -27,7 +27,9 @@ using KeyOrigin = std::function<std::string(std::string_view service)>;
 
 // Row ids beyond the registry's. A service's dials and key hang under the
 // slot that names it (the first slot only, when two name the same service).
-//   web_search.dial.<service>.<dial>   a Choice; value id "" = the service default
+//   web_search.dial.<service>.<dial>   a Choice (value id "" = the service
+//                                      default), or Text for a free-text dial
+//                                      (empty = the default)
 //   web_search.key.<service>           a Secret; always empty, see the .cpp
 inline constexpr std::string_view kDialPrefix = "web_search.dial.";
 inline constexpr std::string_view kKeyPrefix  = "web_search.key.";

src/domain/web_search_config.cpp

@@ -173,7 +173,8 @@ nlohmann::json rewrite_args(const Config& c, nlohmann::json args) {
 }
 
 bool dial_entry_ok(std::string_view key, std::string_view value) noexcept {
-    if (key.empty() || key.size() > 64 || value.empty() || value.size() > 64) return false;
+    if (key.empty() || key.size() > 64 || value.empty() || value.size() > kMaxDialValueBytes)
+        return false;
     const auto dot = key.find('.');
     if (dot == std::string_view::npos || dot == 0 || dot + 1 == key.size()) return false;
     auto printable = [](std::string_view s) {

src/runtime/panel/web_search_form.cpp

@@ -38,6 +38,25 @@ std::string dial_key(std::string_view service, std::string_view dial) {
     return k;
 }
 
+std::string trim(std::string_view s) {
+    constexpr std::string_view kSpace = " \t";
+    const auto b = s.find_first_not_of(kSpace);
+    if (b == std::string_view::npos) return {};
+    return std::string{s.substr(b, s.find_last_not_of(kSpace) - b + 1)};
+}
+
+// Whether `value` may be stored for the text dial `key` ("<service>.<dial>"):
+// mcp-cpp's own rule, so the pane never saves what a search would drop.
+bool text_dial_ok(std::string_view key, std::string_view value) {
+    const auto dot = key.find('.');
+    if (dot == std::string_view::npos) return false;
+    const auto* svc = ::mcp::tools::find_web_search_service(key.substr(0, dot));
+    if (!svc) return false;
+    for (const auto& d : svc->dials)
+        if (d.id == key.substr(dot + 1)) return ::mcp::tools::web_search_dial_value_ok(d, value);
+    return false;
+}
+
 // "Brave Free \xe2\x86\x92 DuckDuckGo Free \xe2\x86\x92 Tavily Free" for the subtitle.
 std::string chain_text(const web_search_cfg::Config& c) {
     std::string out;
@@ -72,14 +91,19 @@ void add_service_rows(form::Builder& b, const web_search_cfg::Config& c,
                       const ::mcp::tools::WebSearchService& svc,
                       const KeyOrigin& key_origin) {
     for (const auto& d : svc.dials) {
-        std::vector<std::string> labels, ids;
-        for (const auto& v : d.values) { labels.push_back(v.label); ids.push_back(v.id); }
         std::string cur;
         if (auto it = c.dials.find(dial_key(svc.id, d.id)); it != c.dials.end())
             cur = it->second;
         std::string help = svc.label + (d.help.empty() ? "" : " \xc2\xb7 " + d.help);
-        b.choice(std::string{kDialPrefix} + dial_key(svc.id, d.id),
-                 "  " + d.label, std::move(labels), std::move(ids), cur, std::move(help));
+        std::string row_id = std::string{kDialPrefix} + dial_key(svc.id, d.id);
+        if (d.text != ::mcp::tools::WebSearchText::None) {
+            b.text(std::move(row_id), "  " + d.label, cur, std::move(help));
+        } else {
+            std::vector<std::string> labels, ids;
+            for (const auto& v : d.values) { labels.push_back(v.label); ids.push_back(v.id); }
+            b.choice(std::move(row_id), "  " + d.label, std::move(labels), std::move(ids),
+                     cur, std::move(help));
+        }
         b.origin(cur.empty() ? "default" : "settings.json");
     }
     if (!svc.needs_key) return;
@@ -214,13 +238,21 @@ void apply_form(const form::Form& f, web_search_cfg::Config& c) {
     reg::read_rows(c, f, reg::Owner::WebSearch);
     // Dial rows. They are not registry rows (which dials exist depends on the
     // services chosen), so they are found by their id prefix. A dial set back
-    // to the service's own default is erased, not stored as "".
+    // to the service's own default (or a text row emptied) is erased, not
+    // stored as "". Typed text mcp-cpp would not send is not stored either:
+    // the rebuild then shows the row with what was on record.
     for (const auto& row : f.fields) {
         if (row.locked || row.id.rfind(kDialPrefix, 0) != 0) continue;
-        const auto* ch = std::get_if<form::field::Choice>(&row.value);
-        if (!ch) continue;
         const std::string key = row.id.substr(kDialPrefix.size());
-        const std::string val{ch->id()};
+        std::string val;
+        if (const auto* ch = std::get_if<form::field::Choice>(&row.value)) {
+            val = std::string{ch->id()};
+        } else if (const auto* tx = std::get_if<form::field::Text>(&row.value)) {
+            val = trim(tx->value);
+            if (!val.empty() && !text_dial_ok(key, val)) continue;
+        } else {
+            continue;
+        }
         if (val.empty()) c.dials.erase(key);
         else             c.dials[key] = val;
     }

tests/web_search_services_test.cpp

@@ -217,14 +217,14 @@ TEST_CASE("web search plan: keys go to keyed services only, and a missing key st
 TEST_CASE("web search plan: dials follow their service, not their slot") {
     sc::Config c;
     c.primary = "firecrawl-free"; c.secondary = "tavily-free"; c.fallback = "none";
-    c.dials["firecrawl-free.category"] = "github";
+    c.dials["firecrawl-free.category"] = "developer";
     c.dials["tavily-free.topic"] = "news";
     c.dials["exa-api.type"] = "fast";             // not in the plan
     auto p = sp::build_plan(c, no_keys);
     REQUIRE(p.slots.size() == 2);
     REQUIRE(p.slots[0].dials.size() == 1);
     CHECK(p.slots[0].dials[0].first == "category");
-    CHECK(p.slots[0].dials[0].second == "github");
+    CHECK(p.slots[0].dials[0].second == "developer");
     REQUIRE(p.slots[1].dials.size() == 1);
     CHECK(p.slots[1].dials[0].first == "topic");
 
@@ -257,6 +257,10 @@ TEST_CASE("web search settings: a dial entry must look like <service>.<dial>") {
     CHECK(!sc::dial_entry_ok("brave-free.freshness", ""));
     CHECK(!sc::dial_entry_ok("brave-free.freshness", "p\nw"));
     CHECK(!sc::dial_entry_ok(std::string(65, 'a') + ".x", "pw"));
+    // A value has room for the longest free-text dial (a Goggle URL), no more.
+    const std::string url = "https://" + std::string(sc::kMaxDialValueBytes - 8, 'g');
+    CHECK(sc::dial_entry_ok("brave-api.goggles", url));
+    CHECK(!sc::dial_entry_ok("brave-api.goggles", url + "g"));
 }
 
 // ── The key store ────────────────────────────────────────────────────────
@@ -480,6 +484,40 @@ TEST_CASE("web search form: a dial round-trips through apply_form, and the defau
     CHECK(c.dials.find("brave-free.freshness") == c.dials.end());
 }
 
+TEST_CASE("web search form: a text dial is a Text row, checked before it is stored") {
+    const std::string goggle =
+        "https://raw.githubusercontent.com/brave/goggles-quickstart/main/goggles/tech_blogs.goggle";
+    sc::Config c;
+    c.primary = "brave-api";
+    auto f = wtf::build_form(c, kNoEnv);
+    auto* row = f.find("web_search.dial.brave-api.goggles");
+    REQUIRE(row != nullptr);
+    auto* tx = std::get_if<form::field::Text>(&row->value);
+    REQUIRE(tx != nullptr);
+    CHECK(tx->value.empty());
+
+    // Typed with stray spaces: stored trimmed, and shown again on rebuild.
+    tx->value = "  " + goggle + " ";
+    wtf::apply_form(f, c);
+    CHECK(c.dials.at("brave-api.goggles") == goggle);
+    auto g = wtf::build_form(c, kNoEnv);
+    CHECK(std::get<form::field::Text>(g.find("web_search.dial.brave-api.goggles")->value).value
+          == goggle);
+
+    // Something a search would not send is not stored; the old value stays.
+    for (const char* bad : {"http://example.com/a.goggle", "$boost=3,site=dev.to",
+                            "https://a.com/x y"}) {
+        std::get<form::field::Text>(g.find("web_search.dial.brave-api.goggles")->value).value = bad;
+        wtf::apply_form(g, c);
+        CHECK(c.dials.at("brave-api.goggles") == goggle, bad);
+    }
+
+    // Emptied: the entry is removed, as a list dial set back to its default is.
+    std::get<form::field::Text>(g.find("web_search.dial.brave-api.goggles")->value).value = "  ";
+    wtf::apply_form(g, c);
+    CHECK(c.dials.find("brave-api.goggles") == c.dials.end());
+}
+
 TEST_CASE("web search form: choosing a slot's service changes the rows under it") {
     sc::Config c;
     c.primary = "exa-api";
@@ -513,6 +551,8 @@ TEST_CASE("web search settings: dials and slots persist, and a key never does")
     s.web_search.secondary = "none";
     s.web_search.language = "de";
     s.web_search.dials["exa-api.type"] = "fast";
+    s.web_search.dials["brave-api.goggles"] =
+        "https://raw.githubusercontent.com/brave/goggles-quickstart/main/goggles/tech_blogs.goggle";
     REQUIRE(ss::store("exa-api", "exa-LEAKCHECK-0123456789"));
     persistence::save_settings(::agentty::IoAccess::grant(), agtest::cap(), s);
 
@@ -527,6 +567,7 @@ TEST_CASE("web search settings: dials and slots persist, and a key never does")
     CHECK(back.web_search.fallback == "tavily-free");        // untouched default
     CHECK(back.web_search.language == "de");
     CHECK(back.web_search.dials.at("exa-api.type") == "fast");
+    CHECK(back.web_search.dials.at("brave-api.goggles").size() > 64);   // over the old cap
 }
 
 TEST_CASE("web search settings: a hand-edited bad value cannot reach a slot") {