Web Tools

A self-hosted web toolkit providing fourteen tools for search, content extraction, and archival. Available as an MCP server, REST API, and CLI, powered by SearXNG, Crawl4AI, Scrapling, Camoufox, and the Wayback Machine.

Architecture

graph LR
    MCP["MCP Client<br/>(Claude, Cursor, etc.)"] -->|POST /mcp| Server["Web Tools Server"]
    API["REST Client"] -->|POST /api/v0/*| Server
    CLI["CLI"] -->|direct call| Toolkit["@web-tools/toolkit"]
    Server --> Toolkit
    Toolkit --> SearXNG
    SearXNG --> Redis
    Toolkit --> Scrapling
    Toolkit --> Camoufox
    Toolkit --> Crawl4AI
    Toolkit --> Wayback["Wayback Machine"]

Why three fetchers

They are not redundant. Each reaches pages the others cannot, and the split is measured rather than aesthetic:

	Crawl4AI	Scrapling	Camoufox
Browser	Chromium	Patchright Chromium	Firefox
Egress	this host's IP only¹	rotating US residential	rotating Italian residential
JS challenges	no	yes (`solve`)	n/a (coherent fingerprint)
LinkedIn profiles	decays to 0/6, HTTP 999	94% (34/36)	not measured
Trustpilot reviews	luck-of-the-IP	2/2 via challenge solve	not measured
Italian bot-gated sites	blocked	wrong country	the point
Binary / PDF fetch	no	no	yes (`web_bytes`)
Anti-bot sensor sessions	no	no	yes (`web_spa_fetch`)
Ordinary pages	~2-5s	~0.7-1.9s (`fast`)	~8-12s

Camoufox is Firefox on purpose: stealth-patched headless Chrome was flagged by Akamai even through an Italian residential IP, while Camoufox's fingerprint is internally coherent: its locale and timezone are derived from the exit IP, so web_eval on an Italian site reports Europe/Rome. Italian sources either bot-gate datacenter IPs outright or score the exit country as part of a sensor decision, and a US residential exit is not a milder version of the right answer.

¹ Crawl4AI >= 0.9 treats every HTTP request body as Provenance.UNTRUSTED and lists proxy_config in UNTRUSTED_FORBIDDEN_FIELDS, so passing a proxy is a hard 400. It also pins Chromium to its own localhost egress proxy, so a server-side proxy is overwritten. There is no supported way to give Crawl4AI a proxy, which is why residential egress lives in Scrapling.

Fetch strategy

Callers never choose an engine. web_fetch and web_html take a URL; how to reach it is decided internally, in three tiers:

Tier	Egress	Chosen when
`fast`	direct, no challenge solving	the default
`stealth`	rotating residential proxy	host is known to wall datacenter IPs (LinkedIn's HTTP 999)
`solve`	direct, solves the JS challenge	the `fast` response looks like a challenge page

fast is the default rather than solve even though solve is functionally a superset: solving roughly doubles latency on ordinary pages, and on a challenge it cannot solve it blocks for the whole timeout instead of failing fast. So solving is paid for only on evidence. A small body carrying a known interstitial title with a 403/429/503 gets retried once in solve, and the response reports escalated: true.

So: Scrapling fetches, Crawl4AI renders and does the browser work. web_fetch and web_html fetch through Scrapling; web_fetch then renders that HTML to markdown through Crawl4AI's markdown pipeline (via its raw:// input) so the f filter keeps working. web_crawl, web_execute_js, web_screenshot and web_pdf stay on Crawl4AI. If Scrapling is unreachable, web_fetch falls back to fetching through Crawl4AI directly.

The project is structured as a monorepo with three packages:

packages/toolkit: Core business logic: Zod schemas, tool definitions, SearXNG/Crawl4AI/Wayback clients. Framework-agnostic.
packages/api: Express HTTP server exposing MCP (POST /mcp) and REST (POST /api/v0/{tool_name}) endpoints.
packages/cli: Commander.js CLI for terminal usage.

The full stack deploys as 6 services: Redis, SearXNG, Crawl4AI, Scrapling, Camoufox, and the Web Tools server.

Tools

The server exposes fourteen tools:

`web_search`

Lightweight web search via SearXNG with parallel request strategy for reliability.

Parameter	Type	Description
`query`	string (required)	The search query
`limit`	number (optional)	Max results to return (default: 10, max: 20)
`engines`	string (optional)	Comma-separated engines (e.g. "google,brave")

Returns a JSON array of { url, title, description } results.

`web_fetch`

Fetch a single URL and return its content as clean markdown. Fetched via Scrapling, rendered to markdown by Crawl4AI.

Parameter	Type	Description
`url`	string (required)	URL to fetch
`f`	enum (optional)	Content-filter strategy: `raw`, `fit`, `bm25`, or `llm` (default: `fit`)
`q`	string (optional)	Query string for BM25/LLM filters
`delay`	number (optional)	Seconds to settle before extraction (default: 2)

Returns the page content as markdown.

There is no engine or mode parameter. Which fetcher runs, whether it goes out through the residential proxy, and whether it solves a JS challenge are all decided under the hood. See Fetch strategy.

`web_html`

Fetch a URL and return the raw HTML as served, plus the upstream status. Use this rather than web_fetch when you need markup that markdown conversion destroys: JSON-LD, meta tags, attributes.

Parameter	Type	Description
`url`	string (required)	URL to fetch
`network_idle`	boolean (optional)	Wait for the network to go quiet (default: false)
`timeout_ms`	number (optional)	Upstream fetch timeout (default: 60000)

Returns a JSON object: { status, url, mode, escalated, size, html }. A non-2xx upstream status is reported in status rather than raised as an error, so callers can branch on 999 vs 404 themselves.

`web_screenshot`

Capture a full-page PNG screenshot of a URL via Crawl4AI.

Parameter	Type	Description
`url`	string (required)	URL to screenshot
`screenshot_wait_for`	number (optional)	Seconds to wait before capture (default: 2)

Returns a base64-encoded PNG image.

`web_pdf`

Generate a PDF document of a URL via Crawl4AI.

Parameter	Type	Description
`url`	string (required)	URL to convert to PDF

Returns a base64-encoded PDF.

`web_execute_js`

Execute JavaScript snippets on a URL via Crawl4AI and return the full crawl result.

Parameter	Type	Description
`url`	string (required)	URL to execute scripts on
`scripts`	string[] (required)	List of JavaScript snippets to execute in order

Returns the full CrawlResult JSON including markdown, links, media, and JS execution results.

`web_crawl`

Crawl one or more URLs and extract their content using Crawl4AI.

Parameter	Type	Description
`urls`	string[] (required)	List of URLs to crawl
`browser_config`	object (optional)	Crawl4AI browser configuration
`crawler_config`	object (optional)	Crawl4AI crawler configuration

Returns the extracted content from each URL.

`web_snapshots`

List Wayback Machine snapshots for a URL.

Parameter	Type	Description
`url`	string (required)	URL to check for snapshots
`from`	string (optional)	Start date in YYYYMMDD format
`to`	string (optional)	End date in YYYYMMDD format
`limit`	number (optional)	Max number of snapshots to return (default: 100)
`match_type`	enum (optional)	URL matching: `exact`, `prefix`, `host`, or `domain` (default: `exact`)
`filter`	string[] (optional)	CDX API filters (e.g. `["statuscode:200", "mimetype:text/html"]`)

Returns a JSON array of snapshots with timestamps, status codes, and archive URLs.

`web_archive`

Retrieve an archived page from the Wayback Machine.

Parameter	Type	Description
`url`	string (required)	URL of the page to retrieve
`timestamp`	string (required)	Timestamp in YYYYMMDDHHMMSS format
`original`	boolean (optional)	Get original content without Wayback Machine banner (default: false)

Returns the archived page content.

`web_bytes`

Download a URL's raw bytes through the residential exit, base64-encoded. Use for PDFs and other binaries behind a bot-gated or geo-sensitive origin, where rendering the page as text would lose the document.

Parameter	Type	Description
`url`	string (required)	URL of the binary to download
`timeout_ms`	number (optional)	Fetch timeout (default: 60000)

Returns { status, url, size_b64, b64 }. A non-2xx arrives in status rather than raised, so a caller fetching a PDF that 404s still learns what happened.

`web_eval`

Evaluate JavaScript in a residential browser page and return its JSON result. Use for driving or inspecting a JS app (open a facet, read the codes behind it) on a site that bot-gates this host's own IP, which web_execute_js cannot reach because it runs from that IP.

Parameter	Type	Description
`url`	string (required)	URL to open
`js`	string (required)	Expression or IIFE evaluated in the page; must return JSON
`wait_until`	enum (optional)	`load`, `domcontentloaded`, `networkidle`, `commit`
`wait_ms`	number (optional)	Extra settle time after load (default: 6000)
`timeout_ms`	number (optional)	Navigation timeout (default: 90000)
`fresh_ip`	boolean (optional)	Serve from a new context on a new exit IP, with clean cookies

Returns { status, url, result }.

`web_spa_fetch`

Perform a same-origin in-page fetch on a warmed browser session, for origins that gate requests on an anti-bot sensor cookie.

Parameter	Type	Description
`base_url`	string (required)	Origin to warm and fetch against
`path`	string (required)	Same-origin path for the in-page fetch
`warm_path`	string (optional)	Path navigated to warm the sensor (default: `/`)
`method`	string (optional)	HTTP method (default: `GET`)
`body`	object (optional)	JSON body, sent as a JSON string
`accept`	string (optional)	Accept header (default: `application/json`)
`sensor_wait_ms`	number (optional)	Time spent seeding the sensor on a warm (default: 20000)
`mature_probe`	object (optional)	`{method,path,body,accept}` replayed until it stops returning 403
`mature_max_tries`	number (optional)	Maturation attempts (default: 6)
`timeout_ms`	number (optional)	Client timeout (default: 180000)

Returns { status, text }, where status is the in-page fetch's own HTTP status. A 403 means the sensor has not cleared, so the caller should re-mature or recycle.

This is the only stateful tool here. The sidecar keeps one warmed page per (base_url, warm_path), pins it to a sticky residential exit and feeds the sensor on a keepalive so the cookie stays validated. Treat the session as shared: web_recycle, or anything that tears the browser down, costs whoever is mid-crawl their maturation.

`web_recycle`

Drop the warmed session and the render browser, and take a fresh exit IP. No parameters.

Expensive (a full browser relaunch) and disruptive to any crawl in flight, so reach for it only when an exit IP has been rate-hardened by a target and will not recover on its own. For a fresh IP on a single request, pass fresh_ip to web_eval instead, which costs about a second.

`web_usage_stats`

Process-local usage counters: per-tool call counts, approximate proxy bandwidth and an estimated cost. No parameters.

In-memory only, so it resets on container restart; the started_at field lets a caller detect that. Only the proxied tools accrue bandwidth, and their byte counts are an upper bound rather than a measurement, since web_fetch and web_html fall back to the unproxied browser when a sidecar is unreachable.

Interfaces

MCP

All MCP-compatible clients can connect via HTTP:

Claude Code (CLI)

claude mcp add web_tools \
  --transport http \
  https://your-server.up.railway.app/mcp \
  --header "Authorization: Bearer your-api-key"

Project-level config (`.mcp.json`)

{
  "mcpServers": {
    "web_tools": {
      "type": "http",
      "url": "https://your-server.up.railway.app/mcp",
      "headers": {
        "Authorization": "Bearer your-api-key"
      }
    }
  }
}

Claude Desktop (`claude_desktop_config.json`)

{
  "mcpServers": {
    "web_tools": {
      "type": "http",
      "url": "https://your-server.up.railway.app/mcp",
      "headers": {
        "Authorization": "Bearer your-api-key"
      }
    }
  }
}

REST API

Every tool is also available as a REST endpoint:

# Discovery: list all tools
curl https://your-server.up.railway.app/api/v0 \
  -H "Authorization: Bearer your-api-key"

# Search
curl -X POST https://your-server.up.railway.app/api/v0/web_search \
  -H "Authorization: Bearer your-api-key" \
  -H "Content-Type: application/json" \
  -d '{"query": "railway deployment"}'

# Fetch
curl -X POST https://your-server.up.railway.app/api/v0/web_fetch \
  -H "Authorization: Bearer your-api-key" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

CLI

# Search
web-tools search "railway deployment" --limit 5

# Fetch page as markdown
web-tools fetch https://example.com

# Screenshot
web-tools screenshot https://example.com

# Crawl multiple URLs
web-tools crawl https://a.com https://b.com --magic

# Wayback Machine
web-tools snapshots https://example.com --from 20200101
web-tools archive https://example.com --timestamp 20200101120000

Replace Claude Code's Built-in Web Search & Web Fetch (Optional)

1. Add the MCP server globally:

claude mcp add web_tools --scope user \
  --transport http \
  https://your-server.up.railway.app/mcp \
  --header "Authorization: Bearer your-api-key"

2. Disable the built-in tools by editing ~/.claude/settings.json:

{
  "permissions": {
    "deny": ["WebSearch", "WebFetch"]
  }
}

3. Guide Claude via ~/.claude/CLAUDE.md so it uses your tools:

## Search & Fetch

- Use the web_search MCP tool for all web searches
- Use the web_fetch MCP tool to fetch and read web pages
- Do not attempt to use the built-in WebSearch or WebFetch tools

Deployment (Railway)

Click Deploy on Railway: you'll see all 4 services listed (Redis, SearXNG, Crawl4AI, Web Tools Server)
Click Deploy: Railway provisions everything and wires the services together automatically
An API_KEY is auto-generated during deployment. Find it in your Web Tools service's Variables tab and use it as your Bearer token

Railway Configuration

The Web Tools Server service uses the root Dockerfile, so no config changes are needed.

The SearXNG and Scrapling services build from the repo instead of a Docker image, and each one must have its Root Directory set:

Service	Root Directory	Env
SearXNG	`services/searxng`	`PROXY_URL` (optional), the proxy for outgoing search requests
Scrapling	`services/scrapling`	`PROXY_URL` (US-geo, for the residential path), `PORT=8000`
Camoufox	`services/camoufox`	`PROXY_URL` (IT-geo), `PORT=8000`, `WORKERS=1`

Camoufox keeps WORKERS=1, because one warmed anti-bot session per container cannot be shared across processes. Scale it with replicas, not workers: railway service scale --service camoufox eu-west=2. Keep it in EU West; a US container reaches an Italian exit and an Italian target across the Atlantic twice. Note scale ADDS to existing regions, so pass us-east=0 to move rather than spread.

Set Root Directory before connecting the repo. Railway resolves a service's build config by walking up from its Root Directory, so a subfolder service without one inherits the repo root's Dockerfile, which is the Node server. The symptom is confusing: the build goes green, then the container crashes on ZodError: API_KEY Required, because it is running the API server instead of the sidecar. It also repeats on every push, so a service deployed correctly by hand will replace itself with the API server the next time the repo changes.

railway up cannot fix this: it uploads the right files but leaves the stored config pointing at /. Root Directory is not exposed by the CLI either; set it in the dashboard, or via the public API:
curl https://backboard.railway.com/graphql/v2 \
  -H "Authorization: Bearer $RAILWAY_TOKEN" -H "Content-Type: application/json" \
  -d '{"query":"mutation($s:String!,$e:String,$i:ServiceInstanceUpdateInput!){serviceInstanceUpdate(serviceId:$s,environmentId:$e,input:$i)}",
       "variables":{"s":"<serviceId>","e":"<environmentId>",
       "i":{"rootDirectory":"/services/scrapling",
            "dockerfilePath":"/services/scrapling/Dockerfile",
            "watchPatterns":["/services/scrapling/**"]}}}'
Do not pass builder, because the Builder enum has no DOCKERFILE value (only HEROKU/NIXPACKS/PAKETO/RAILPACK) and the whole mutation fails with a generic "Problem processing request". Railway detects the Dockerfile from the path.

The anchored watchPatterns is worth setting too: without it every push to the repo rebuilds the sidecar, including pushes that do not touch it.

Point the server at its siblings with reference variables rather than hardcoded hostnames, so renaming or moving a service does not silently break private networking:

CRAWL4AI_URL       = http://${{Crawl4AI.RAILWAY_PRIVATE_DOMAIN}}:11235
CRAWL4AI_API_TOKEN = ${{Crawl4AI.CRAWL4AI_API_TOKEN}}
SCRAPLING_URL      = http://${{Scrapling.RAILWAY_PRIVATE_DOMAIN}}:${{Scrapling.PORT}}
CAMOUFOX_URL       = http://${{Camoufox.RAILWAY_PRIVATE_DOMAIN}}:${{Camoufox.PORT}}
SEARXNG_URL        = http://${{SearXNG.RAILWAY_PRIVATE_DOMAIN}}:8080

Reference ${{Service.PORT}} only where the service actually binds it and has no default of its own to diverge from. That holds for the two images in this repo: their CMD is uvicorn --port ${PORT} and they deliberately ship no ENV PORT, so the Railway variable is the single source of truth for both the bind and the URL, and a missing one stops the container at boot rather than yielding http://host:.

It does not hold for the two third-party images, whose URLs keep literal ports: SearXNG hardcodes --port 8080 in its entrypoint and Crawl4AI reads port: 11235 from its own config.yml, so PORT is decoration on both and a reference to it is a guess that fails open. Crawl4AI's read 8000 while the app listened on 11235, which pointed every fetch at a closed port.

Service names are case-sensitive: ${{camoufox.…}} against a service named Camoufox resolves to an empty string rather than erroring, giving http://:8000.

Quick Start (Local)

1. Clone and install

git clone https://github.com/arnaudjnn/web-tools
cd web-tools
pnpm install

2. Configure environment

cp .env.example .env.local

3. Run the sidecars you need, locally

Only the Tools service has a public domain. The four backing services are private, reachable at *.railway.internal from inside the project and from nowhere else, so a laptop cannot point at them. That is deliberate (see Exposure).

For most work you do not need them. Run the server against whichever sidecars you build locally; each is self-contained, and every URL is optional:

docker build -t searxng services/searxng && docker run -d -p 8080:8080 \
  -e SEARXNG_SECRET_KEY=dev -e SEARXNG_REDIS_URL=redis://host.docker.internal:6379/0 searxng
docker run -d -p 6379:6379 redis:7-alpine
docker run -d -p 11235:11235 -e CRAWL4AI_API_TOKEN=dev unclecode/crawl4ai:0.9.2

API_KEY=any-local-value \
SEARXNG_URL=http://localhost:8080 \
CRAWL4AI_URL=http://localhost:11235 \
CRAWL4AI_API_TOKEN=dev \
pnpm run start

The server is at http://localhost:3000. API_KEY is required but arbitrary locally, since it only guards your own endpoint.

Leave a URL out and that path degrades rather than fails: SEARXNG_URL alone gives you web_search; CRAWL4AI_URL gives web_crawl / web_screenshot / web_pdf and markdown rendering; web_fetch and web_html fall back to Crawl4AI when the stealth sidecars are absent. The two stealth sidecars each bake a browser into their image (~200MB Chromium for Scrapling, Camoufox's Firefox plus a GeoIP database), and their residential paths need a PROXY_URL you supply, so build them only when you are working on those paths specifically.

If you genuinely need to reach a deployed sidecar from your machine, add a service domain temporarily (railway domain --service Crawl4AI) and delete it when you are done. Do not leave one on: an exposed SearXNG is an open search proxy that spends your metered residential bandwidth.

Exposure

Only Tools should have a public domain. It is the authenticated front door (API_KEY as a Bearer token); everything behind it talks over Railway's private network:

Service	Public domain	Why
Tools	yes	the API surface: MCP + REST, API-key guarded
SearXNG	no	it has no authentication of its own, so a public domain is an open search proxy, and its outgoing requests egress through your metered `PROXY_URL`
Crawl4AI	no	`CRAWL4AI_API_TOKEN` is the only thing between a public domain and free use of your browser fleet
Scrapling	no	residential egress; nothing should reach it but Tools
Camoufox	no	residential egress + warmed anti-bot sessions

SEARXNG_SECRET_KEY is not an access credential. It is SearXNG's internal signing secret, needed whether or not the service is exposed. Removing a public domain is what makes a service private; deleting its credentials just makes it broken or open.

Environment Variables

Variable	Required	Description
`API_KEY`	Yes	Bearer token for authentication (auto-generated on Railway)
`SEARXNG_URL`	No	SearXNG URL (default: `http://searxng.railway.internal:8080`)
`CRAWL4AI_URL`	No	Crawl4AI URL (default: `http://crawl4ai.railway.internal:11235`)
`CRAWL4AI_API_TOKEN`	No	API token for Crawl4AI authentication
`SCRAPLING_URL`	No	Scrapling URL (default: `http://scrapling.railway.internal:8000`)
`SEARXNG_ENGINES`	No	Default engines (e.g. `"brave,bing"`)
`PROXY_URL`	No	Rotating residential proxy. Set on the SearXNG and Scrapling services, not the server. Required for `mode=stealth`.

Authentication

The API_KEY environment variable is required.

On Railway, the key is auto-generated at deploy time (via ${{secret()}}). For local development, set it in your .env.local file.

Clients provide the key as a Bearer token in the Authorization header or as an ?api_key= query parameter. The /health endpoint is unauthenticated.

License

MIT

Name		Name	Last commit message	Last commit date
.claude/skills/tools-health		.claude/skills/tools-health
assets		assets
packages		packages
services		services
.env.example		.env.example
.gitignore		.gitignore
Dockerfile		Dockerfile
LICENSE		LICENSE
RAILWAY.md		RAILWAY.md
README.md		README.md
package.json		package.json
pnpm-lock.yaml		pnpm-lock.yaml
pnpm-workspace.yaml		pnpm-workspace.yaml
tsconfig.json		tsconfig.json

Latest commit

History

Folders and files

Repository files navigation

Web Tools

Architecture

Why three fetchers

Fetch strategy

Tools

web_search

web_fetch

web_html

web_screenshot

web_pdf

web_execute_js

web_crawl

web_snapshots

web_archive

web_bytes

web_eval

web_spa_fetch

web_recycle

web_usage_stats