Skip to content

Trim whitespace around href values before resolving them - #222

Open
bytestream wants to merge 1 commit into
crwlrsoft:mainfrom
bytestream:trim-href
Open

bytestream wants to merge 1 commit into
crwlrsoft:mainfrom
bytestream:trim-href

Conversation

@bytestream

Copy link
Copy Markdown

Problem

Browsers strip leading and trailing ASCII whitespace from href attribute values (HTML spec), but the crawler passes them to the URL resolver verbatim. Markup like this exists in the wild:

<a href="
https://www.example.com/announcements/article">Announcement</a>

PHP's parse_url() doesn't see a scheme because the string starts with a newline, and it rewrites the control character as _, so Validator::absoluteUrl() returns null and the resolver treats the href as a relative path:

Url::parse('https://docs.example.com/2.6.1/')
    ->resolve("\nhttps://www.example.com/announcements/article")
    ->toString();
// https://docs.example.com/2.6.1/_https://www.example.com/announcements/article

Two knock-on effects: the crawler requests a URL that doesn't exist (404), and because the mangled URL lives on the crawled host, onSameHost() / onSameDomain() filters don't exclude what was actually an external link.

Fix

Trim href values before resolving them in every place the library reads one:

  • GetLink / GetLinks (both getLinkUrl() and isSpecialNonHttpLink())
  • Http::crawl() link discovery
  • SimpleWebsitePaginator
  • <link rel="canonical"> in Document
  • HtmlDocument::getBaseHref() (which also covers <base> handling in GetLink, Document and the paginator)

DomQuery already trimmed attribute values, so Dom::cssSelector('a')->link() was unaffected.

Tests

Added regression tests for each of the above. composer test passes (1284 tests). composer cs reports nothing. composer stan reports the same 133 pre-existing errors as on main.

🤖 Generated with Claude Code

Browsers strip leading and trailing ASCII whitespace from href attribute
values, but the crawler passed them to the URL resolver verbatim. An
href such as "\nhttps://example.com" therefore wasn't recognised as an
absolute URL and was resolved as a relative path against the page URL,
yielding URLs like https://www.example.com/foo/_https://example.com
(PHP's parse_url() replaces the control character with an underscore).

Trim hrefs in GetLink/GetLinks, Http::crawl(), SimpleWebsitePaginator,
the canonical link in Document, and HtmlDocument::getBaseHref().

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant