[Bug] Google Search/SERP requests return HTTP 429 and the specialist finishes without results

[Ianfr13]

Summary

The google_search specialist is being invoked correctly, but native Google SERP requests are repeatedly blocked with HTTP 429.

The scraper exhausts its retry/IP budget without obtaining SERP HTML. Despite having no usable search results, the specialist task exits with outcome=ok after several minutes.

This makes the feature appear to hang or makes the assistant stop responding without showing a clear error.

Environment

• Project: SurfSense
• Deployment platform: Railway
• Environment: Production
• Specialist: google_search
• Backend deployment: d3ed8a73-65fc-4003-9ac4-885145ce670c
• Approximate occurrence: 2026-08-25 16:51–16:56 UTC
• Relevant implementation areas:
  • surfsense_backend/app/proprietary/platforms/google_search/fetch.py
  • surfsense_backend/app/proprietary/platforms/google_search/scraper.py
  • surfsense_backend/app/proprietary/platforms/google_search/searxng.py
  • surfsense_backend/app/capabilities/google_search/scrape/executor.py

Proxy credentials are intentionally omitted from this issue.

Important finding

This is not a case where the specialist was simply skipped.

The logs confirm that:

• subagent_type='google_search' was invoked.
• The specialist was compiled with its search tools.
• Search queries were submitted.
• The browser fetch layer attempted to retrieve Google SERP pages.
• Google returned repeated HTTP 429 responses.

Representative logs

[google_search][perf] status=429 bytes=6571 has_results=False vet_ms=531 render_ms=4138 from_pool=False pool=0

The same pattern was repeated for multiple attempts:

status=429
has_results=False
from_pool=False
pool=0

The scraper then reported:

[google_search] no SERP HTML for https://www.google.com/search?q=women+over+40+health
[google_search] gave up on https://www.google.com/search?q=women+over+40+health (deadline/24-IP budget)

Other affected queries included searches similar to:

joint pain women over 40
menopause weight loss
menopause weight loss causes

Additional browser failures:

[google_search] no SERP HTML
[google_search] gave up ... (deadline/24-IP budget)

[google_search] render failed: TimeoutError(
  Page.goto: Timeout 30000ms exceeded,
  waiting until "load"
)

Task duration and incorrect success status

The specialist eventually exited with outcome=ok, even though no SERP HTML or usable search results were returned.

Observed invocation durations included approximately:

114 seconds
209 seconds
287 seconds

Examples from the task logs:

atask EXIT subagent_type='google_search'
outcome=ok
ainvoke=114.333s

atask EXIT subagent_type='google_search'
outcome=ok
ainvoke=209.419s

atask EXIT subagent_type='google_search'
outcome=ok
ainvoke=287.233s

Actual behavior

• The google_search specialist starts correctly.
• Google SERP requests receive HTTP 429.
• No valid SERP HTML is obtained.
• The scraper retries across the configured IP/deadline budget.
• Browser navigation may also hit a 30-second timeout.
• The specialist can take several minutes to finish.
• The task is reported as successful despite having no usable results.
• The user receives no clear explanation that Google SERP access failed.

Expected behavior

When Google SERP access is available:

• The specialist should return search results normally.

When Google blocks the request or no SERP HTML can be obtained:

• The request should fail within a bounded timeout.
• The error should be explicit and structured.
• The specialist should report a failure status instead of outcome=ok.
• The UI should show that Google Search is temporarily unavailable.
• The assistant should not silently continue as if the search succeeded.

Suspected cause

Google is actively blocking the automated browser SERP requests with HTTP 429.

The logs show an application-level response from Google, rather than:

• proxy authentication failure;
• DNS failure;
• TLS failure;
• connection refusal;
• backend exception.

The configured proxy may be contributing through IP reputation, rotation behavior, or Google anti-bot detection, but the logs do not prove that the proxy credentials are invalid.

The following values are also significant:

from_pool=False
pool=0

This suggests that no warmed internal SERP IP/session was available for these attempts.

Possible secondary issue: fallback is not active

The code contains a SearXNG fallback path controlled by SEARXNG_URL.

However, there were no corresponding fallback success logs, such as:

[google_search][searxng] ...

This means one of the following should be verified:

• SEARXNG_URL is not configured;
• the fallback is disabled for this request type;
• the fallback is not reached after the Google failure;
• the fallback itself is failing without sufficient logging.

Earlier successful Google News/RSS crawler requests are not evidence that native Google SERP is working. They use a different code path.

Steps to reproduce

1. Deploy SurfSense in Railway production.
2. Configure the Google Search specialist.
3. Run a chat request that requires live Google SERP research.
4. Ask a query that triggers the google_search specialist.
5. Observe the specialist invocation in the backend logs.
6. Observe repeated HTTP 429 responses from Google.
7. Wait for the retry/deadline budget to be exhausted.
8. Observe that the task may finish as outcome=ok without search results.

Impact

• Google Search/SERP specialist is currently unreliable or unusable.
• Requests may take several minutes before returning.
• Users may believe that the model is frozen or has stopped responding.
• The assistant may produce no answer because the research step returned an empty result.
• The current success status makes monitoring and debugging misleading.
• Repeated retries may consume proxy bandwidth and IP quota without producing value.

Suggested investigation

1. Improve error classification

Distinguish at least the following cases:

• HTTP 429 rate limit or anti-bot block;
• CAPTCHA/challenge page;
• empty SERP page;
• browser navigation timeout;
• proxy connection failure;
• proxy authentication failure;
• DNS/TLS failure;
• internal parser failure.

2. Fail fast on repeated HTTP 429

Do not continue the full retry budget when multiple consecutive attempts return the same blocking status.

Return a structured failure containing fields similar to:

{
  "status": "error",
  "provider": "google",
  "error_type": "rate_limited",
  "http_status": 429,
  "results_available": false,
  "retryable": true
}

The exact schema can follow the existing project conventions.

3. Fix task success reporting

The specialist must not finish with outcome=ok when:

has_results=False

and no fallback result was obtained.

The task should either:

• return a structured error;
• return a clearly marked empty result with an error reason;
• or successfully use a fallback provider.

4. Verify proxy/browser-pool behavior

Investigate:

• why pool=0;
• why all attempts report from_pool=False;
• whether sticky sessions are being created;
• whether IP rotation is occurring as expected;
• whether Google is blocking the current proxy IP range;
• whether the browser session has the required cookies;
• whether challenge pages are being detected;
• whether proxy authentication is working independently of Google blocking.

Proxy credentials and secrets should never be logged.

5. Verify SearXNG fallback

If SearXNG is supported as a fallback:

• document the required SEARXNG_URL;
• confirm that it is configured in Railway;
• ensure it is called after Google SERP failure;
• log fallback success/failure without exposing secrets;
• return fallback results when available.

6. Improve observability

Add structured logs containing:

• query hash or opaque query ID;
• provider;
• attempt number;
• HTTP status;
• timeout category;
• whether a proxy pool IP was used;
• whether SERP HTML was detected;
• whether a fallback was attempted;
• final result status.

Do not log raw user queries if they may contain sensitive information.

Acceptance criteria

• A simple Google SERP query returns valid results in production when the provider is available.
• Repeated HTTP 429 responses do not consume several minutes of retries.
• A failed SERP request returns a clear structured error.
• The specialist never reports outcome=ok when no results and no fallback result are available.
• The UI clearly communicates that Google Search is unavailable instead of appearing frozen.
• SearXNG fallback is either verified to work or explicitly documented as unavailable.
• Proxy/IP-pool health is visible through safe structured logs.
• No proxy credentials, tokens, cookies, or authorization headers appear in logs.
• An automated test covers:
  • successful SERP response;
  • HTTP 429;
  • browser timeout;
  • empty SERP response;
  • fallback success;
  • fallback failure.

[meemoprasad]

Answer

When Google blocks scraper requests with HTTP 429 errors or redirects to the reCAPTCHA-Enterprise page, the specialist task runner exits silently with outcome=ok instead of raising an explicit error or returning a failure status.

According to the documentation in surfsense_backend/app/proprietary/platforms/google_search/README.md, a standard Google /search request returns a JS shell that reloads using a sei token, triggering a redirect to a /sorry reCAPTCHA page. If configured, a solver handles this via CAPTCHA_SOLVING_ENABLED=TRUE and CAPTCHA_SOLVER_API_KEY using captcha.py (located in app/utils/captcha/). It harvest an Enterprise token to acquire a GOOGLE_ABUSE_EXEMPTION cookie which it caches inside _exemption_jar.

If this solver is missing, disabled, or if the browser fetch layer encounters timeouts (such as the 30-second TimeoutError on Page.goto in the logs), the scraper exhausts its retry and IP budget and returns no SERP HTML. Although no results are retrieved, the task execution layer (likely inside surfsense_backend/app/capabilities/google_search/scrape/executor.py) swallows this failure, reporting success (outcome=ok) and leaving the assistant hanging without notifying the user of the block.

Evidence

• surfsense_backend/app/proprietary/platforms/google_search/README.md [INFERENCE]: Outlines Google's security behavior (the JS shell redirecting to /sorry), and details how captcha.py and the solver environment variables (CAPTCHA_SOLVING_ENABLED and CAPTCHA_SOLVER_API_KEY) bypass the block by caching the exemption cookie in _exemption_jar.
• .cursor/skills/technical-seo-checker/references/http-status-codes.md [INFERENCE]: Explains that HTTP status code 429 (Too Many Requests) is returned during aggressive crawling or when rate limits are triggered.
• Bug Report Logs [INFERENCE]: Show multiple scraper attempts resulting in status=429 has_results=False and ultimately logging gave up on https://www.google.com/search?q=... (deadline/24-IP budget). The task runner logs atask EXIT subagent_type='google_search' outcome=ok regardless of the failure to acquire SERP HTML.

Confidence & Gaps

Our investigation coverage is 77%.

• UNKNOWN: The specific implementation, entry points, and methods within surfsense_backend/app/capabilities/google_search/scrape/executor.py, scraper.py, and fetch.py were not directly retrieved in the codebase search.
• HYPOTHESIS: It is hypothesized that executor.py catches empty search results and returns a default success status instead of propagating an exception, but this cannot be confirmed as a fact without reading the source file directly.