Skip to content

match-funders: move grant page discovery out of the request (Trello #285) - #300

Merged
SpikeyCoder merged 17 commits into
mainfrom
claude/match-funders-discovery-budget
Oct 13, 2026
Merged

match-funders: move grant page discovery out of the request (Trello #285)#300
SpikeyCoder merged 17 commits into
mainfrom
claude/match-funders-discovery-budget

Conversation

@SpikeyCoder

@SpikeyCoder SpikeyCoder commented Oct 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Fixes Trello #285 ([SLA] Funder matching: 2 of 3 checks failed in the last 3 hours).

Findings

  • Latency: over 14 days, median match time was ~12.5 s against the 20 s SLA. On Oct 11, 4 synthetic checks timed out at 30 s (01:06, 04:06, 13:06, 14:06 UTC) and 1 took 21.8 s. Real visitors also waited ~21.5 s.
  • Not the database: no Postgres errors or statement timeouts. Every DB query finished by ~6 s; after that the function went quiet until it answered, for ~6 s in healthy runs and 24 s+ in failed ones.
  • Cause: that quiet stretch was discoverGrantUrlsInBatch scanning funder homepages for grant-application links inside the request.
    • It ran batches of 6 sites one after another, each allowed 3 s.
    • The 3 s timer was cleared once headers arrived, so reading a page body had no limit. Reproduced: the old helper was still hanging after 4 s with a 500 ms timeout.
    • Only hits were stored, so the 5,293 of 5,696 funders with a website but no grant link were fetched again on every search.

Changes

  • Discovery runs after the response, via EdgeRuntime.waitUntil, so matching never waits on homepage fetches. Hits go to discovered_apply_url, and later searches read them from there. Cached mission results also run discovery in the background and pick up links found by any search; changes are written back into search_cache, guarded by created_at.
  • discovery.ts (new):
    • Budget: 8 s for the whole batch, 8 sites at a time, 4 s per site, covering headers and body; at most 500K chars read per page.
    • Order: best-ranked results first, and each website is fetched once.
    • Scope: links must be on the funder's site or a subdomain of it. A homepage that redirects off-site yields nothing. One that redirects up to a parent domain (foundation.bigcorp.com → bigcorp.com/foundation/) is limited to the landing path.
    • Scoring: on the link's path, not its host name.
    • Outcomes:
      Outcome Recheck
      Miss: no link, 404/410, not HTML, or a private address 30 days
      Failure: network error, 5xx/429/403, DNS failure, over 4 s 1 day
      Cut off by the batch budget not recorded, tried next time
  • Migration (20261013014519_funder_apply_url_recheck_at.sql):
    • adds funders.apply_url_recheck_at;
    • adds a trigger: moving a funder's website to another page clears that column and the old site's discovered_apply_url, unless the same update sets them. Rewriting the same page (adding https://, www. or a trailing slash) doesn't trigger it.
  • _shared/safe_fetch.ts (homepages are now fetched through it):
    • Security fix: private IPv4 addresses written as IPv6 are refused. URL parsing turns [::ffff:169.254.169.254] into [::ffff:a9fe:a9fe], which the old check let through to the metadata endpoint; reproduced in a test. IPv4-mapped, IPv4-translated, IPv4-compatible, NAT64 and 6to4 forms are now checked, and site-local, local-use NAT64 and Teredo are refused.
    • A DNS lookup cut short by the caller's signal now throws the abort rather than "did not resolve".
    • Redirect bodies are cancelled.
  • Tests: match-funders/discovery_test.ts and _shared/safe_fetch_test.ts (24 Deno tests), both added to CI.

Deploy

  1. Apply the migration: done, applied to production as 20261013014519 on 2026-10-13. The file is named to match.
  2. Deploy match-funders and fetch-grant-deadline (shared safe_fetch.ts), both with --no-verify-jwt as now.

The function works with or without the migration: hits are saved, fresh searches scan, and cached searches skip discovery until the column exists.

Review

This went through repeated /code-review and /security-review rounds. Every security review came back clean; the one security issue, the IPv6 SSRF bypass above, was caught by code review and fixed. Code review converged on these, left as is deliberately:

  • Shared-host websites: a funder whose website is a page on a shared host (facebook.com/x) accepts links anywhere on that host. That was already true on main. Each attempt to limit links to the website's path rejected real links somewhere (/home, /en/, /about.html, /pages/default.aspx).
  • Search-driven discovery: discovery still happens because of searches. A scheduled sweep over apply_url_recheck_at would give coverage independent of traffic, and is a good follow-up once this ships.
  • DNS failures: treated as transient (1-day recheck), because safeFetch can't tell NXDOMAIN from a resolver failure.
  • No claim before scanning: overlapping searches can scan the same homepage. The cost is background-only and bounded by the budget.
  • Write fan-out: one PATCH per website, each guarded by that website. Hits are ~5% of scans.
  • DNS rebinding: safeFetch checks DNS and then lets fetch resolve again. That gap predates this PR.

Testing

  • Deno CI suite plus the new tests: 99 passed. deno check on match-funders shows the same 2 errors as main; fetch-grant-deadline is clean.
  • Migration: run twice on a local Postgres 16, with trigger cases checked, then applied to production; column, trigger and key function verified there.
  • New PostgREST filters checked with read-only queries against production.
  • npm run lint, npm run build and npm test pass.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UPYZwv6X3kmDoJN7g4RaJm

@claude
match-funders: bound grant page discovery to 3 s per request (Trello #…
748200e 
…285)

Funder matching missed its 20 s SLA (4 timeouts at 30 s on Oct 11; real
visitors at ~21.5 s). DB work finished in ~6 s; the rest was live homepage
scanning for grant application links:

- chunks of 6 homepages ran one after another, each allowed 3 s, so 30
  sites added ~15 s;
- the 3 s timer was cleared once headers arrived, so a slow body read had
  no limit at all (reproduced: the old helper still hung after 4 s with a
  500 ms timeout);
- only hits were stored, so the 93% of homepages without a grant link were
  fetched again on every search.

Discovery moves to discovery.ts: one 3 s budget for the whole batch (8 at
a time) that covers headers and body, reads at most 500K chars, and scans
the matched funders only. Homepages scanned (hit or miss) are recorded in
the new funders.apply_url_checked_at and skipped for 30 days. That lookup
fails open, so the function works before the migration is applied.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPYZwv6X3kmDoJN7g4RaJm
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 12, 2026 •
edited
Loading

Copy link
Copy Markdown

Deploying funder-finder with  Cloudflare Pages  Cloudflare Pages

Latest commit: 58f6e92
Status: ✅  Deploy successful!
Preview URL: https://c5482d49.funder-finder.pages.dev
Branch Preview URL: https://claude-match-funders-discove.funder-finder.pages.dev

View logs

claude added 4 commits October 12, 2026 23:12
@claude
match-funders discovery: review fixes
ecac86e 
- Discover over the returned results, best-ranked first, after filtering
  and sorting (and before the mission-path cache write), so the budget
  isn't spent on funders that are dropped; hits update the row's next step.
- Cap discovery at what's left of the request budget; skip it under 250 ms.
- Recently-checked lookup: batches in parallel, inside the discovery budget.
- Only definite outcomes count as checked (a page scanned, 404/410, not
  HTML); network errors, 5xx, 429 and bot walls are retried next request.
- Resolve links against the page's URL after redirects.
- Same-site check by host suffix, not substring (notfund.org, fund.org.x.com).
- Save discovered_apply_url in its own PATCH, so hits are saved before the
  apply_url_checked_at migration; writes go through sbFetch (timeout, keys).
- Flush the streaming TextDecoder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPYZwv6X3kmDoJN7g4RaJm
@claude
match-funders discovery: second review round
7196625 
- A spent budget (even negative) returns no discovery instead of throwing
  from AbortSignal.timeout and failing the request.
- Re-derive a hit's next step from the funder row (its DB next_step), not
  the result row, whose next_step already holds derived text.
- Column is now apply_url_recheck_at (migration not yet applied): 30 days
  after a miss, 1 day after a failure or budget cutoff, so homepages that
  always fail don't take the budget on every request.
- A homepage that redirects off the funder's site counts as a miss; links
  must be on the funder's host or a subdomain of it (not its parent).
- Discovery runs alongside grantee EIN enrichment; record writes are kept
  alive with EdgeRuntime.waitUntil and go through sbFetchBatched (which
  gains a timeout and request init) instead of a copied batching loop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPYZwv6X3kmDoJN7g4RaJm
@claude
match-funders discovery: run after the response
93218f8 
Three review rounds kept finding edge cases in budgeting a homepage scrape
inside the visitor's request (cutoffs, the recheck lookup's latency, a
negative budget). Discovery now runs after the response is sent
(EdgeRuntime.waitUntil) with an 8 s budget, and later searches read its
hits from discovered_apply_url. Matching pays nothing for it.

- Links must be on the funder's original site (or a subdomain), resolved
  against the page's URL after redirects, so fund.org -> home.fund.org
  linking www.fund.org/apply is found and off-site redirects find nothing.
- Sites cut off by the budget aren't recorded; real load failures retry
  after a day, misses after 30 days.
- Each website is fetched once even when several funders share it.
- Changing a funder's website clears apply_url_recheck_at (trigger).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPYZwv6X3kmDoJN7g4RaJm
@claude
match-funders discovery: fourth review round
bd8f1b1 
- Hits found after the response update the rows' next step (re-derived from
  the funder) and, on the mission path, the cached results, so cached
  searches pick up grant links found after they were cached.
- Background discovery starts after EIN enrichment, not alongside it.
- A site still loading after 5 s when the budget runs out counts as failed
  (retried in a day), so sites that always hang don't hold a slot on every
  search; sites cut off sooner are left for next time.
- A response without a Content-Type is read as HTML, not a 30-day miss.
- Changing a funder's website also clears the discovered_apply_url found on
  the old site, unless the same update sets a new one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPYZwv6X3kmDoJN7g4RaJm
@SpikeyCoder SpikeyCoder changed the title match-funders: bound grant page discovery to 3 s per request (Trello #285) match-funders: move grant page discovery out of the request (Trello #285) Oct 12, 2026
claude added 11 commits October 12, 2026 23:33
@claude
match-funders discovery: fifth review round
1796c01 
- abortable(): with an already-aborted signal, handle the dropped promise's
  rejection, which was unhandled and ended the isolate (reproduced in a test).
- A homepage that redirects up to a parent domain (foundation.bigcorp.com ->
  www.bigcorp.com/foundation/) is scanned for links under the landing path;
  only http(s) links count.
- Hits are saved only while the funder's website is still the one scanned.
- The mission path's cache update only replaces the results it wrote
  (created_at guard), and re-derives next steps there; the peer path,
  which has no cache, no longer re-derives rows nobody reads.
- The website trigger compares hosts (funder_website_host), so adding
  https:// or www. doesn't clear discovered_apply_url.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPYZwv6X3kmDoJN7g4RaJm
@claude
match-funders discovery: sixth review round
c0e39d5 
- Scan websites as the rest of the function links them (toExternalUrl):
  trimmed, legacy github.io prefix stripped, relative values skipped.
- Miss and failure recheck writes, like hits, apply only while the funder's
  website is still the one scanned (one PATCH per website).
- bareHost() helper for the host normalization funderScope repeated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPYZwv6X3kmDoJN7g4RaJm
@claude
match-funders discovery: seventh review round
6c02ae0 
- Score a link's path and query, not its host: every link on
  fundingfutures.org or grants.fund.org isn't a grant page.
- Never pick the page scanned (e.g. the /foundation/ landing page after a
  redirect) as its own grant page.
- Fetch homepages through _shared/safe_fetch.ts, which blocks private and
  metadata hosts and re-checks every redirect hop.
- The website trigger compares host and path (funder_website_key), so a
  move to another page on the same host (bigcorp.com -> /foundation) resets
  discovery; adding https://, www. or a trailing slash still doesn't.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPYZwv6X3kmDoJN7g4RaJm
@claude
match-funders discovery: eighth review round
192f105 
- Cached mission results pick up grant pages any search has discovered
  since (one funders lookup on a cache hit, 1.5 s cap, best effort).
- The search cache is written after grantee EIN enrichment, so cached rows
  are the same whether or not discovery later finds anything.
- Each homepage gets 4 s (DISCOVERY_SITE_TIMEOUT_MS); slower ones count as
  failed and the batch moves on. Replaces the "hung after 5 s" rule.
- A host safeFetch refuses (no DNS answer, private address) is a 30-day
  miss, not a daily retry.
- safeFetch cancels the unread body of each redirect response it follows.
- The website trigger keeps apply_url_recheck_at when the same update sets it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPYZwv6X3kmDoJN7g4RaJm
@claude
match-funders discovery: ninth review round
46e96dd 
- safeFetch refusals: only private/metadata addresses are a 30-day miss;
  DNS failures and redirect loops retry in a day.
- A website stored as a site section (bigcorp.com/foundation) keeps links
  to that section; home and language pages (/home, /index.html, /en/) are
  still the whole site.
- A page that finished loading as the batch budget ran out is still scanned.
- applyDiscoveredLink() shared by cache hits and the background onHits;
  cache hits that pick up new links save them back (created_at guard).
- Hits sharing a URL are saved in one PATCH; the background closure keeps
  only each funder's DB next_step, not the candidate rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPYZwv6X3kmDoJN7g4RaJm
@claude
match-funders discovery: tenth review round
0057bba 
- Cached mission searches run discovery in the background too, and any
  grant pages found (by them or any search) are written into the cache
  entry; the cache-hit response no longer waits on a lookup.
- After discovery, both paths pick up hits by re-reading them from funders
  (withDiscoveredLinks), replacing the duplicated next-step update.
- A homepage's section comes from where it was served after redirects; a
  file (about.html) stands for its folder rather than a section no link can
  be under.
- safeFetch refusals are read from the message's fixed wording, not words
  that can also appear in a host name (privatefoundation.org).
- Map.groupBy for grouping by website; a malformed cached results value is
  served as is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPYZwv6X3kmDoJN7g4RaJm
@claude
match-funders discovery: eleventh review round
f07434e 
- Drop the same-site "section" rule: each variant (stored paths, landing
  paths, file names) shut out real links somewhere. Links anywhere on the
  funder's site count again, as before this PR; only a redirect up to a
  parent domain still limits them to the landing path.
- Don't rescan funders given a link since the results were built: the
  recheck lookup also skips rows with apply_url or discovered_apply_url
  set, and cached results pick up known links before discovery runs.
- A grant portal's own root (https://grants.fund.org/) can be the link;
  only the funder's homepage and the page scanned are skipped.
- Content-Type compared case-insensitively.
- safeFetch looks up A and AAAA records together.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPYZwv6X3kmDoJN7g4RaJm
@claude
safeFetch: refuse private IPv4 addresses written as IPv6
74ecdab 
URL parsing rewrites http://[::ffff:169.254.169.254]/ as [::ffff:a9fe:a9fe],
and isPrivateIPv6 only recognised the dotted IPv4-mapped form, so a
redirect to that address reached the cloud metadata endpoint (reproduced
in the new test). Match-funders discovery now fetches through safeFetch,
as fetch-grant-deadline already did.

isPrivateIPv6 now expands the address to its 8 groups and checks the
IPv4 address embedded in IPv4-mapped, IPv4-compatible and NAT64 forms
however it's spelled; ranges compared numerically; unparseable refused.
safe_fetch_test.ts added to CI.

Also: fresh mission searches skip the found-links lookup before
discovery, since their rows were just built from funders.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPYZwv6X3kmDoJN7g4RaJm
@claude
match-funders discovery: twelfth review round
7520957 
- Scan funders with their website as it is now: the due-for-scan lookup
  returns id and website for funders still without a link, so a cached
  result's stale website isn't fetched (and its writes don't miss).
- Before the apply_url_recheck_at migration, cache hits skip discovery
  (nothing could be recorded, so every hit would rescan); fresh searches
  scan as before.
- Cache refresh re-reads links only when discovery found some.
- safeFetch passes the abort signal to its DNS lookups, and also refuses
  site-local fec0::/10, local-use NAT64 64:ff9b:1::/48, IPv4-translated
  ::ffff:0:0:0/96 and 6to4 2002::/16 carrying a private IPv4 address.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPYZwv6X3kmDoJN7g4RaJm
@claude
match-funders discovery: final review fixes
b043ad3 
- Scan in result rank order: the due-for-scan lookup returns rows in
  database order, which decided who the 8 s budget reached.
- safeFetch: a DNS lookup cut short by the caller's signal throws the
  abort, not SSRFBlockedError('did not resolve') (fetch-grant-deadline
  would have answered a slow lookup with a misleading 400).
- A grant portal on its own subdomain scores by its name
  (grants.fund.org/ reads as /grants/), as before path-only scoring.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPYZwv6X3kmDoJN7g4RaJm
@claude
match-funders discovery: last review round
7a51ba2 
- abortable(): a response that arrives after the abort has its body
  cancelled instead of holding the connection until GC.
- Anchor text is capped at 2,000 characters in the scan, so pages with
  unclosed <a> tags can't make it quadratic.
- needsGrantLink() shared by discovery and the found-links lookup.
- safeFetch also refuses Teredo (2001::/32).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPYZwv6X3kmDoJN7g4RaJm
@SpikeyCoder
SpikeyCoder marked this pull request as ready for review October 13, 2026 01:07
@claude
Migration: match production version 20261013014519
58f6e92 
Applied to production as 20261013014519. CREATE OR REPLACE TRIGGER in place
of DROP TRIGGER IF EXISTS + CREATE TRIGGER (same result, re-runnable).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPYZwv6X3kmDoJN7g4RaJm
@SpikeyCoder
SpikeyCoder merged commit b96270a into main Oct 13, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@SpikeyCoder @claude