fix: keep first fetch when shadow-DOM prerender falls back

[cursor]

Bug and impact

Auto-mode shadow-DOM upgrade can replace a working first fetch with a later, worse 2xx body and still report success.

Trigger:

1. First fetch returns 200 HTML that contains custom elements (shadow DOM).
2. The auto-mode retry prerenders; page.content() is a 4xx WAF/challenge page.
3. #276 falls back to the parallel fetch. That second request can also be 200 — a soft-block interstitial, a rate-limit page, or any other divergent snapshot.
4. #277 only checked statusCode < 400, so the later fetch overwrote the known-good first snapshot.

Microlink then extracts metadata / captures from the wrong document while advertising statusCode: 200.

Root cause

#276 returns { mode: 'fetch', statusCode: 200 } when prerender itself is unusable. #277 treated any 2xx retry result as a successful upgrade, not only a successful prerender.

Fix

Require prerenderResult.mode === 'prerender' before replacing the first fetch. Direct { prerender: true } still falls back to fetch as #276 intended.

Validation

• New regression: second fetch is 200 soft block while prerender is 403 — first snapshot is kept.
• Existing shadow-DOM and prerender-mode tests still pass.

  

---

Note

Low Risk
Single guard in the shadow-DOM retry path plus a focused test; behavior change is narrower (keeps first fetch more often) with no auth or data-layer impact.

Overview
Fixes auto-mode shadow-DOM upgrade replacing a good first fetch snapshot with a worse document while still reporting success.

When the prerender retry is unusable (e.g. WAF/challenge page.content() at 4xx), inner prerender logic can fall back to a parallel fetch and return { mode: 'fetch', statusCode: 200 } — sometimes a soft-block or divergent 2xx body. The shadow-DOM path only checked html and statusCode < 400, so that fallback could overwrite the known-good first fetch.

The upgrade now requires prerenderResult.mode === 'prerender' before swapping in the retry result, matching the intent that only a successful prerender should replace fetch.

Adds a regression test: prerender mocked as 403 while the second fetch is 200 “soft block”; the first snapshot with shadow-DOM content is kept.

^{Reviewed by Cursor Bugbot for commit d0a6ff1. Bugbot is set up for automated code reviews on this repo. Configure here.}