[Bug]: All HTTP endpoints return 500 after 0.9.0 upgrade — `_IncludedRouter` has no attribute `path`

[Suquir0]

crawl4ai version

0.9.0 (Docker: unclecode/crawl4ai:latest)

Expected Behavior

POST /crawl, POST /md, POST /html and all other HTTP endpoints should return
normal crawl results (HTTP 200 with JSON response), same as in 0.8.x.

Current Behavior

All HTTP endpoints (except /health, /crawl/stream and /token) return
HTTP 500 with:

{"error": "Internal server error", "correlation_id": "..."}

Container logs show the following repeated for every request:

server - ERROR - unhandled exception [cid=...]
AttributeError: '_IncludedRouter' object has no attribute 'path'
File "starlette/middleware/errors.py", line 164, in call
await self.app(scope, receive_or_disconnect, send_no_error)

The SSE endpoint /crawl/stream works correctly and returns crawl results
without issues.

Is this reproducible?

Yes

Inputs Causing the Bug

- URL(s): any (tested with https://example.com, https://httpbin.org/ip)
- Settings used: CRAWL4AI_API_TOKEN set via environment variable.
  Container exposed on 0.0.0.0:11235 behind a reverse proxy (Zoraxy) with TLS.
- Input data: minimal valid JSON request: {"urls": ["https://example.com"]}

Steps to Reproduce

1. Deploy unclecode/crawl4ai:latest (0.9.0) with CRAWL4AI_API_TOKEN set
2. Wait for container to fully start
3. Send an HTTP request to any affected endpoint:

   curl -X POST http://localhost:11235/crawl \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"urls":["https://example.com"]}'

4. Observe HTTP 500

Code snippets

# Reproduce the bug
TOKEN="***"
curl -s -X POST "http://localhost:11235/crawl" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://httpbin.org/ip"]}'
# → 500: {"error":"Internal server error","correlation_id":"..."}

# Streaming DOES work
curl -s -X POST "http://localhost:11235/crawl/stream" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://httpbin.org/ip"]}'
# → 200: {"url":"...","success":true,...}

OS

Linux (Docker, python:3.12-slim based image). Host: Ubuntu 24.04.

Python version

3.12 (as shipped in the container)

Browser

N/A — the bug occurs on direct API calls, not in the browser

Browser version

No response

Error logs & Screenshots (if applicable)

Root cause: known incompatibility between prometheus_fastapi_instrumentator
and FastAPI >= 0.137.

Starting with FastAPI 0.137, app.routes contains _IncludedRouter objects
(introduced by include_router) which have no .path attribute. When
Starlette's middleware or the instrumentator try to read .path on a matched
route, they raise AttributeError.

References:

• app.routes contains _IncludedRouter with no .path since 0.137 fastapi/fastapi#15791
• [Bug]: fastapi error '_IncludedRouter' object has no attribute 'path' vllm-project/vllm#45597

Possible fixes:

1. Pin fastapi<0.137 in the Dockerfile/requirements.txt until
   prometheus_fastapi_instrumentator is updated
2. Update prometheus_fastapi_instrumentator to a version that
   handles _IncludedRouter

Relevant container logs (repeated for every request):
2026-06-20 14:23:44,771 - server - ERROR - unhandled exception [cid=c141fd77dbd9]
File "starlette/middleware/errors.py", line 164, in call
await self.app(scope, receive_or_disconnect, send_no_error)
AttributeError: '_IncludedRouter' object has no attribute 'path'

[nightcityblade]

Hi, I'd like to work on this. I'll submit a PR shortly.