diff --git a/CHANGELOG.md b/CHANGELOG.md index 808a5a6..620b9cc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,13 +12,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/). - HTML page metadata: `extract_document` fills `Extraction::title` (Open Graph, JSON-LD or ``) and the new `Extraction::metadata` (description, author, published date, language, canonical URL, site name, lead image) from `<meta>`, Open Graph, `<link rel=canonical>` and - JSON-LD. `markkit extract --json` includes `format`, `title` and `metadata` for files and stdin. + JSON-LD. `markkit extract --json` includes `format`, `title` and `metadata`. - IR v2: footnotes (`[^1]`), definition lists (`Term` / `: definition`), task lists (`- [x]`), figures with captions, table column alignment (`:---:`) and `colspan`/`rowspan` (laid out on a grid; GFM has no spans). HTML maps GitHub and Pandoc footnote markup, `<dl>`, checkbox list items, `<figure>`/`<figcaption>` and table spans/alignment. Row-header `<th>` cells in table bodies are no longer dropped. - `Document.ir_version` (`markkit::ir::IR_VERSION = 2`) is serialized with the IR JSON. +- Relative URL resolution: links, images and metadata URLs resolve against the new + `MarkkitBuilder::base_url` (CLI `--base-url`) and the page's `<base href>`; `extract_url` uses + the final URL after redirects. Absolute and fragment-only URLs are left unchanged. - Math: `$tex$` and `$$` blocks from MathML (`application/x-tex` annotation), KaTeX and MathJax output. - IR transforms: `MarkkitBuilder::transform(..)` runs `Transform` passes after parsing and before rendering, for every format. Built-ins `transform::NormalizeHeadings` and `transform::DropImages`; diff --git a/Cargo.lock b/Cargo.lock index ae8295d..4b8e4c9 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1208,6 +1208,7 @@ dependencies = [ "tokio", "tower", "tracing", + "url", "zip", ] diff --git a/Cargo.toml b/Cargo.toml index 5fa6158..62dc15f 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -41,6 +41,7 @@ clap = { version = "4", features = ["derive"] } reqwest = { version = "0.13", default-features = false, features = ["rustls", "json"] } # HTML parsing +url = "2" scraper = "0.29" ego-tree = "0.11" schemars = "1" diff --git a/README.md b/README.md index 074107a..b39c035 100644 --- a/README.md +++ b/README.md @@ -77,8 +77,11 @@ markkit extract --file document.html # With CSS selector filtering markkit extract https://example.com --include "article" --exclude "nav,footer" -# JSON output; files and stdin also carry format, title and page metadata -markkit extract --file page.html --json +# JSON output with format, title and page metadata +markkit extract https://example.com --json + +# Resolve relative links in a saved page +markkit extract --file page.html --base-url https://example.com/blog/ # From stdin cat page.html | markkit extract - diff --git a/crates/markkit-cli/src/main.rs b/crates/markkit-cli/src/main.rs index 206b454..9541120 100644 --- a/crates/markkit-cli/src/main.rs +++ b/crates/markkit-cli/src/main.rs @@ -45,6 +45,11 @@ enum Commands { /// Output as JSON with metadata #[arg(long)] json: bool, + + /// Resolve relative links and images in files / stdin against this URL + /// (URLs use the final response URL) + #[arg(long)] + base_url: Option<String>, }, } @@ -61,6 +66,7 @@ async fn main() -> Result<()> { exclude_selectors, max_size, json: output_json, + base_url, } => { let read_stdin = stdin || source.as_deref() == Some("-"); @@ -77,6 +83,10 @@ async fn main() -> Result<()> { builder = builder.exclude_selector(sel); } + if let Some(base) = base_url { + builder = builder.base_url(base); + } + let markkit = builder.build(); let source_label = source.clone().unwrap_or_else(|| "<stdin>".to_string()); @@ -84,36 +94,30 @@ async fn main() -> Result<()> { let extraction = if read_stdin { let mut buf = Vec::new(); std::io::stdin().read_to_end(&mut buf)?; - Some(markkit.extract_document_sync( - &buf, - markkit::DocumentHint::content_type("text/html"), - )?) + markkit + .extract_document_sync(&buf, markkit::DocumentHint::content_type("text/html"))? } else if file { let src = source.as_deref().unwrap(); let bytes = tokio::fs::read(src).await?; // Format detected from the extension, then magic bytes. - Some(markkit.extract_document_sync(&bytes, markkit::DocumentHint::file_name(src))?) + markkit.extract_document_sync(&bytes, markkit::DocumentHint::file_name(src))? } else { - None - }; - let markdown = match &extraction { - Some(e) => e.markdown.clone(), - None => markkit.extract_url(source.as_deref().unwrap()).await?, + markkit + .extract_url_document(source.as_deref().unwrap()) + .await? }; if output_json { - let mut output = serde_json::json!({ + let output = serde_json::json!({ "source": source_label, - "markdown": markdown, + "markdown": extraction.markdown, + "format": extraction.format, + "title": extraction.title, + "metadata": extraction.metadata, }); - if let Some(e) = &extraction { - output["format"] = serde_json::to_value(e.format)?; - output["title"] = serde_json::to_value(&e.title)?; - output["metadata"] = serde_json::to_value(&e.metadata)?; - } println!("{}", serde_json::to_string_pretty(&output)?); } else { - print!("{markdown}"); + print!("{}", extraction.markdown); } } } diff --git a/crates/markkit/Cargo.toml b/crates/markkit/Cargo.toml index b40ede5..fbd8176 100644 --- a/crates/markkit/Cargo.toml +++ b/crates/markkit/Cargo.toml @@ -15,7 +15,7 @@ readme = "../../README.md" [features] default = ["html", "http_client"] -html = ["dep:scraper", "dep:html5ever", "dep:ego-tree"] +html = ["dep:scraper", "dep:html5ever", "dep:ego-tree", "dep:url"] pdf = [] docx = ["dep:zip", "dep:quick-xml"] spreadsheet = ["dep:calamine", "dep:zip", "dep:quick-xml", "dep:cfb"] @@ -43,6 +43,7 @@ schemars = { workspace = true, optional = true } scraper = { workspace = true, optional = true } ego-tree = { workspace = true, optional = true } html5ever = { workspace = true, optional = true } +url = { workspace = true, optional = true } # Office documents (optional): DOCX + spreadsheets share the ZIP/XML stack zip = { workspace = true, optional = true } diff --git a/crates/markkit/src/config.rs b/crates/markkit/src/config.rs index 87d1bca..0f3eb3b 100644 --- a/crates/markkit/src/config.rs +++ b/crates/markkit/src/config.rs @@ -40,6 +40,10 @@ pub struct MarkkitBuilder { pub allowed_hosts: Vec<String>, /// Maximum redirects `extract_url` follows. Default: 5. pub max_redirects: usize, + /// Absolute URL that relative HTML links, images and metadata URLs resolve + /// against (a document `<base href>` applies on top). `extract_url` uses the + /// final response URL instead. Default: none (relative URLs kept as is). + pub base_url: Option<String>, /// Record per-block provenance ([`crate::ir::Source`]) in the IR: /// HTML element XPath, DOCX block index, spreadsheet sheet and range. /// Default: false (no cost, no `source` in IR JSON). @@ -75,6 +79,7 @@ impl Default for MarkkitBuilder { allow_private_networks: false, allowed_hosts: Vec::new(), max_redirects: 5, + base_url: None, track_sources: false, transforms: Default::default(), } @@ -177,6 +182,13 @@ impl MarkkitBuilder { self } + /// Resolve relative HTML URLs against this absolute URL. Ignored when it + /// does not parse or cannot be a base (`data:`, `javascript:`, ...). + pub fn base_url(mut self, url: impl Into<String>) -> Self { + self.base_url = Some(url.into()); + self + } + pub fn build(self) -> Markkit { Markkit::new(self) } diff --git a/crates/markkit/src/extract.rs b/crates/markkit/src/extract.rs index 73db1b9..f12a7a7 100644 --- a/crates/markkit/src/extract.rs +++ b/crates/markkit/src/extract.rs @@ -350,6 +350,8 @@ impl Markkit { /// [`MarkkitBuilder::allow_host`] restricts hosts; redirects capped by /// `max_redirects`; body capped by `max_input_size`; whole fetch capped /// by `max_processing_secs` ([`Error::ResourceLimit`]). + /// + /// Relative links and images resolve against the final URL after redirects. #[cfg(feature = "http_client")] pub async fn extract_url(&self, url: &str) -> crate::Result<String> { self.extract_url_document(url).await.map(|e| e.markdown) @@ -358,10 +360,13 @@ impl Markkit { /// [`Markkit::extract_url`] with metadata (format, title, warnings, stats). /// /// The response `Content-Type` is the format hint; same URL policy and limits. + /// Relative HTML URLs resolve against the final URL after redirects. #[cfg(feature = "http_client")] pub async fn extract_url_document(&self, url: &str) -> crate::Result<Extraction> { let fetched = crate::fetch::fetch(url, &self.config).await?; - self.extract_document_sync( + let mut config = self.config.clone(); + config.base_url = Some(fetched.url); + Markkit::new(config).extract_document_sync( &fetched.bytes, DocumentHint { content_type: fetched.content_type.as_deref(), diff --git a/crates/markkit/src/fetch.rs b/crates/markkit/src/fetch.rs index bc4a127..b7b1276 100644 --- a/crates/markkit/src/fetch.rs +++ b/crates/markkit/src/fetch.rs @@ -31,6 +31,8 @@ use crate::error::Error; pub(crate) struct Fetched { pub bytes: Vec<u8>, pub content_type: Option<String>, + /// Final URL after redirects. + pub url: String, } /// Per-call slot for the first block reason seen inside reqwest. @@ -63,6 +65,7 @@ pub(crate) async fn fetch(url: &str, cfg: &MarkkitBuilder) -> crate::Result<Fetc .get(reqwest::header::CONTENT_TYPE) .and_then(|v| v.to_str().ok()) .map(str::to_string); + let final_url = response.url().to_string(); let mut bytes = Vec::new(); while let Some(chunk) = response.chunk().await? { if bytes.len() + chunk.len() > limit { @@ -75,6 +78,7 @@ pub(crate) async fn fetch(url: &str, cfg: &MarkkitBuilder) -> crate::Result<Fetc Ok(Ok(Fetched { bytes, content_type, + url: final_url, })) }; diff --git a/crates/markkit/src/html/metadata.rs b/crates/markkit/src/html/metadata.rs index 22296fd..6b3e0ff 100644 --- a/crates/markkit/src/html/metadata.rs +++ b/crates/markkit/src/html/metadata.rs @@ -12,6 +12,7 @@ use scraper::{ElementRef, Html, Selector}; use serde_json::Value; +use url::Url; use crate::extract::Metadata; @@ -32,7 +33,7 @@ pub(crate) struct PageInfo { } /// Read title and metadata from a parsed document. -pub(crate) fn extract(dom: &Html) -> PageInfo { +pub(crate) fn extract(dom: &Html, base: Option<&Url>) -> PageInfo { let meta = MetaTags::collect(dom); let ld = JsonLd::collect(dom); @@ -86,6 +87,11 @@ pub(crate) fn extract(dom: &Html) -> PageInfo { .or_else(|| ld.image()), }; + let mut metadata = metadata; + let resolve = |u: Option<String>| u.map(|u| super::urls::resolve(base, &u)); + metadata.canonical_url = resolve(metadata.canonical_url); + metadata.image = resolve(metadata.image); + PageInfo { title, metadata } } @@ -355,7 +361,7 @@ mod tests { use super::*; fn info(html: &str) -> PageInfo { - extract(&Html::parse_document(html)) + extract(&Html::parse_document(html), None) } #[test] diff --git a/crates/markkit/src/html/mod.rs b/crates/markkit/src/html/mod.rs index 85b6099..4755900 100644 --- a/crates/markkit/src/html/mod.rs +++ b/crates/markkit/src/html/mod.rs @@ -4,5 +4,6 @@ mod metadata; mod parser; +mod urls; pub use parser::HtmlParser; diff --git a/crates/markkit/src/html/parser.rs b/crates/markkit/src/html/parser.rs index 19cbb05..01c24ff 100644 --- a/crates/markkit/src/html/parser.rs +++ b/crates/markkit/src/html/parser.rs @@ -29,7 +29,10 @@ use crate::config::MarkkitBuilder; use crate::error::Error; use crate::ir::{Alignment, DefinitionItem, Document, ListItem, Node, Source, TableCell}; +use url::Url; + use super::metadata::PageInfo; +use super::urls; /// Maximum recursion depth for nested elements. const MAX_DEPTH: usize = 200; @@ -42,6 +45,8 @@ pub struct HtmlParser<'a> { config: &'a MarkkitBuilder, /// Footnote target id (without `#`) -> label, filled by references. footnote_labels: RefCell<HashMap<String, String>>, + /// Base for relative URLs (caller base, then the document `<base href>`). + base: Option<Url>, /// Cached 1-based index of an element among same-name element siblings. sibling_index: RefCell<HashMap<NodeId, usize>>, } @@ -51,6 +56,7 @@ impl<'a> HtmlParser<'a> { Self { config, footnote_labels: RefCell::new(HashMap::new()), + base: urls::parse_base(config.base_url.as_deref()), sibling_index: RefCell::new(HashMap::new()), } } @@ -63,10 +69,28 @@ impl<'a> HtmlParser<'a> { /// Parse an HTML string into a [`Document`] plus page title and metadata. pub(crate) fn parse_page(&self, html: &str) -> Result<(Document, PageInfo), Error> { let dom = Html::parse_document(html); - Ok((self.parse_dom(&dom)?, super::metadata::extract(&dom))) + let this = self.for_document(&dom); + Ok(( + this.convert_dom(&dom)?, + super::metadata::extract(&dom, this.base.as_ref()), + )) } fn parse_dom(&self, dom: &Html) -> Result<Document, Error> { + self.for_document(dom).convert_dom(dom) + } + + /// Parser whose base includes the document's `<base href>`. + fn for_document(&self, dom: &Html) -> Self { + Self { + config: self.config, + footnote_labels: RefCell::new(HashMap::new()), + sibling_index: RefCell::new(HashMap::new()), + base: urls::document_base(dom, self.base.clone()), + } + } + + fn convert_dom(&self, dom: &Html) -> Result<Document, Error> { let root = dom.root_element(); let content_nodes: Vec<scraper::ElementRef<'_>> = @@ -389,7 +413,7 @@ impl<'a> HtmlParser<'a> { }) }; } - let url = el.value().attr("href").unwrap_or("").to_string(); + let url = urls::resolve(self.base.as_ref(), el.value().attr("href").unwrap_or("")); let title = el.value().attr("title").map(|s| s.to_string()); let children = self.collect_inline(el, exclude, depth); if children.is_empty() { @@ -406,7 +430,7 @@ impl<'a> HtmlParser<'a> { if !self.config.preserve_images { return None; } - let src = el.value().attr("src").unwrap_or("").to_string(); + let src = urls::resolve(self.base.as_ref(), el.value().attr("src").unwrap_or("")); let alt = el.value().attr("alt").unwrap_or("").to_string(); Some(Node::Image { src, alt }) } diff --git a/crates/markkit/src/html/urls.rs b/crates/markkit/src/html/urls.rs new file mode 100644 index 0000000..f9cced7 --- /dev/null +++ b/crates/markkit/src/html/urls.rs @@ -0,0 +1,134 @@ +//! Relative URL resolution for links, images and metadata URLs. +//! +//! Decisions: +//! - Base = `MarkkitBuilder::base_url` (or the final `extract_url` response URL), with the +//! first document `<base href>` resolved on top, as browsers do. A page-supplied base is +//! accepted only when the result is `http(s)`, so untrusted markup cannot rebase links onto +//! `file:` or other local schemes. A caller base must parse and be able to act as a base +//! (`data:` / `javascript:` cannot); otherwise it is ignored. +//! - Only scheme-less references are rewritten. Absolute URLs (any scheme, including +//! `javascript:`) are returned byte-for-byte, so resolution never changes what a scheme +//! filter downstream sees. Empty and fragment-only (`#id`) references stay in-document. +//! - Without a base nothing changes. + +use scraper::{Html, Selector}; +use url::Url; + +/// Parse a caller-supplied base URL. +pub(crate) fn parse_base(raw: Option<&str>) -> Option<Url> { + Url::parse(raw?.trim()) + .ok() + .filter(|u| !u.cannot_be_a_base()) +} + +/// Effective base for a document: `base` with the first `<base href>` applied on top. +pub(crate) fn document_base(dom: &Html, base: Option<Url>) -> Option<Url> { + let sel = Selector::parse("base[href]").unwrap(); + let Some(href) = dom + .select(&sel) + .next() + .and_then(|el| el.value().attr("href")) + else { + return base; + }; + let href = href.trim(); + let doc_base = match &base { + Some(b) => b.join(href).ok(), + None => Url::parse(href).ok(), + } + .filter(|u| matches!(u.scheme(), "http" | "https")); + doc_base.or(base) +} + +/// Resolve `raw` against `base` when it is a relative reference; otherwise return it unchanged. +pub(crate) fn resolve(base: Option<&Url>, raw: &str) -> String { + let Some(base) = base else { + return raw.to_string(); + }; + let trimmed = raw.trim(); + if trimmed.is_empty() || trimmed.starts_with('#') || has_scheme(trimmed) { + return raw.to_string(); + } + base.join(trimmed) + .map(String::from) + .unwrap_or_else(|_| raw.to_string()) +} + +/// RFC 3986 scheme: `ALPHA *( ALPHA / DIGIT / "+" / "-" / "." ) ":"`. +fn has_scheme(s: &str) -> bool { + let Some((scheme, _)) = s.split_once(':') else { + return false; + }; + let mut chars = scheme.chars(); + chars.next().is_some_and(|c| c.is_ascii_alphabetic()) + && chars.all(|c| c.is_ascii_alphanumeric() || matches!(c, '+' | '-' | '.')) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn base(s: &str) -> Option<Url> { + parse_base(Some(s)) + } + + #[test] + fn resolves_relative_forms() { + let b = base("https://ex.com/a/b/page.html?q=1"); + let r = |s| resolve(b.as_ref(), s); + assert_eq!(r("img.png"), "https://ex.com/a/b/img.png"); + assert_eq!(r("../c"), "https://ex.com/a/c"); + assert_eq!(r("/root"), "https://ex.com/root"); + assert_eq!(r("//cdn.ex/x.js"), "https://cdn.ex/x.js"); + assert_eq!(r("?p=2"), "https://ex.com/a/b/page.html?p=2"); + assert_eq!(r("rel?q=a:b"), "https://ex.com/a/b/rel?q=a:b"); + } + + #[test] + fn leaves_absolute_fragment_and_empty() { + let b = base("https://ex.com/"); + let r = |s| resolve(b.as_ref(), s); + assert_eq!(r("https://other/x"), "https://other/x"); + assert_eq!(r("JavaScript:alert(1)"), "JavaScript:alert(1)"); + assert_eq!(r("mailto:a@b"), "mailto:a@b"); + assert_eq!(r("data:image/png;base64,AA"), "data:image/png;base64,AA"); + assert_eq!(r("#top"), "#top"); + assert_eq!(r(""), ""); + assert_eq!(resolve(None, "x.png"), "x.png"); + } + + #[test] + fn invalid_or_opaque_caller_base_ignored() { + assert!(base("not a url").is_none()); + assert!(base("javascript:alert(1)").is_none()); + assert!(base("data:text/html,x").is_none()); + assert!(base("file:///tmp/").is_some()); + } + + #[test] + fn document_base_rules() { + let dom = |h: &str| Html::parse_document(h); + let d = dom(r#"<base href="/docs/">"#); + assert_eq!( + document_base(&d, base("https://ex.com/a")) + .unwrap() + .as_str(), + "https://ex.com/docs/" + ); + let d = dom(r#"<base href="https://cdn.ex/v2/">"#); + assert_eq!( + document_base(&d, None).unwrap().as_str(), + "https://cdn.ex/v2/" + ); + // A relative <base> without a caller base has nothing to resolve against. + assert!(document_base(&dom(r#"<base href="/docs/">"#), None).is_none()); + // Page-supplied non-http(s) bases are refused; the caller base stays. + for evil in ["javascript:alert(1)", "file:///etc/", "data:text/html,x"] { + let d = dom(&format!(r#"<base href="{evil}">"#)); + assert_eq!( + document_base(&d, base("https://ex.com/")).unwrap().as_str(), + "https://ex.com/" + ); + } + } +} diff --git a/crates/markkit/tests/html_urls.rs b/crates/markkit/tests/html_urls.rs new file mode 100644 index 0000000..a03989e --- /dev/null +++ b/crates/markkit/tests/html_urls.rs @@ -0,0 +1,105 @@ +//! Relative URL resolution (`MarkkitBuilder::base_url`, `<base href>`). + +#![cfg(feature = "html")] + +use markkit::{DocumentHint, Markkit}; + +fn convert(base: Option<&str>, html: &str) -> String { + let mut b = Markkit::builder(); + if let Some(base) = base { + b = b.base_url(base); + } + b.build().extract_html_sync(html).unwrap() +} + +const PAGE: &str = r##"<p><a href="b.html">b</a> <a href="/root">r</a> <a href="//cdn.ex/x">c</a> +<a href="#sec">s</a> <a href="https://other.ex/">o</a> <img src="img/p.png" alt="p"></p>"##; + +#[test] +fn relative_links_and_images_resolve_against_base() { + assert_eq!( + convert(Some("https://ex.com/a/page"), PAGE), + "[b](https://ex.com/a/b.html) [r](https://ex.com/root) [c](https://cdn.ex/x) \ + [s](#sec) [o](https://other.ex/) ![p](https://ex.com/a/img/p.png)\n" + ); +} + +#[test] +fn without_base_urls_unchanged() { + assert_eq!( + convert(None, PAGE), + "[b](b.html) [r](/root) [c](//cdn.ex/x) [s](#sec) [o](https://other.ex/) ![p](img/p.png)\n" + ); +} + +#[test] +fn document_base_href_applies_on_top_of_caller_base() { + let html = r#"<head><base href="/v2/"></head><body><p><a href="x">x</a></p></body>"#; + assert_eq!( + convert(Some("https://ex.com/a/b"), html), + "[x](https://ex.com/v2/x)\n" + ); + let html = + r#"<head><base href="https://docs.ex/"></head><body><p><a href="x">x</a></p></body>"#; + assert_eq!(convert(None, html), "[x](https://docs.ex/x)\n"); +} + +#[test] +fn metadata_urls_resolve() { + let e = Markkit::builder() + .base_url("https://ex.com/post/1") + .build() + .extract_document_sync( + br#"<head><link rel=canonical href="/post/1"><meta property=og:image content="lead.jpg"></head>"#, + DocumentHint::content_type("text/html"), + ) + .unwrap(); + assert_eq!( + e.metadata.canonical_url.as_deref(), + Some("https://ex.com/post/1") + ); + assert_eq!( + e.metadata.image.as_deref(), + Some("https://ex.com/post/lead.jpg") + ); +} + +// ─── Untrusted input ──────────────────────────────────────────────── + +#[test] +fn absolute_dangerous_schemes_pass_through_untouched() { + // Resolution never rewrites (or normalizes) an absolute URL. + let md = convert( + Some("https://ex.com/"), + r#"<p><a href="JaVaScRiPt:alert(1)">j</a> <img src="data:image/png;base64,AA" alt="d"></p>"#, + ); + assert_eq!( + md, + "[j](JaVaScRiPt:alert(1)) ![d](data:image/png;base64,AA)\n" + ); +} + +#[test] +fn page_base_cannot_switch_to_local_or_script_scheme() { + for evil in ["file:///etc/", "javascript:alert(1)//", "data:text/html,"] { + let html = format!( + r#"<head><base href="{evil}"></head><body><p><a href="passwd">x</a></p></body>"# + ); + assert_eq!( + convert(Some("https://ex.com/"), &html), + "[x](https://ex.com/passwd)\n", + "{evil}" + ); + assert_eq!(convert(None, &html), "[x](passwd)\n", "{evil}"); + } +} + +#[test] +fn invalid_caller_base_ignored() { + for bad in ["not a url", "javascript:alert(1)", ""] { + assert_eq!( + convert(Some(bad), r#"<p><a href="x">x</a></p>"#), + "[x](x)\n" + ); + } +} diff --git a/crates/markkit/tests/url_security.rs b/crates/markkit/tests/url_security.rs index a7657d4..45771e1 100644 --- a/crates/markkit/tests/url_security.rs +++ b/crates/markkit/tests/url_security.rs @@ -144,6 +144,31 @@ async fn private_networks_reachable_when_allowed() { assert_eq!(private_ok().extract_url(&base).await.unwrap(), "# Hi\n"); } +#[tokio::test] +async fn relative_urls_resolve_against_final_url_after_redirect() { + let (base, _) = serve(|path| { + if path == "/start" { + redirect("/docs/guide/page") + } else { + html( + r#"<head><title>T +

n i

"#, + ) + } + }) + .await; + let e = private_ok() + .extract_url_document(&format!("{base}/start")) + .await + .unwrap(); + assert_eq!( + e.markdown, + format!("[n]({base}/docs/guide/next) ![i]({base}/docs/i.png)\n") + ); + assert_eq!(e.title.as_deref(), Some("T")); + assert_eq!(e.metadata.canonical_url, Some(format!("{base}/c"))); +} + // ─── Host allowlist ───────────────────────────────────────────────── #[tokio::test] diff --git a/knowledge/formats/html-extraction.md b/knowledge/formats/html-extraction.md index cef2a4a..620e0d9 100644 --- a/knowledge/formats/html-extraction.md +++ b/knowledge/formats/html-extraction.md @@ -183,17 +183,30 @@ JSON-LD nodes are the top-level value, array members and `@graph` members; artic `@type`s (`Article`, `NewsArticle`, `BlogPosting`, ...) are consulted first. Untrusted-input rules: values are whitespace-collapsed, control characters dropped, -capped at 2048 chars. URL fields keep only `http(s)` or scheme-less (relative, not yet -resolved) values. JSON-LD: first 16 scripts, each at most 1 MiB, at most 256 nodes; +capped at 2048 chars. URL fields keep only `http(s)` or scheme-less values; the latter +are resolved like links (below). JSON-LD: first 16 scripts, each at most 1 MiB, at most 256 nodes; serde_json's 128-level recursion limit bounds nesting; invalid JSON is ignored. +## Relative URL Resolution + +`html/urls.rs`. Applies to `Link.url`, `Image.src`, `metadata.canonical_url`, `metadata.image`. + +- Base: `MarkkitBuilder::base_url` (CLI `--base-url`); `extract_url` / `extract_url_document` + use the final response URL after redirects. The first `` resolves on top of it. +- Caller base must parse and be able to act as a base (`data:`, `javascript:` cannot), else + ignored. Page `` accepted only when the result is `http(s)` (no `file:` rebasing + by untrusted markup); a relative `` without a caller base is ignored. +- Rewritten: scheme-less references only (`x.png`, `../a`, `/r`, `//cdn/x`, `?q`). +- Untouched, byte-for-byte: absolute URLs of any scheme (incl. `javascript:`, `data:`, + `mailto:`), empty, fragment-only `#id`. No base → nothing changes. + ## Edge Cases - Nested lists: supported via recursive `collect_list_items` with depth tracking - Tables within tables: outer table parsed; inner table content may appear as text - Inline SVG: stripped entirely (non-content element) - Data URIs: preserved as-is in `src`/`href` attributes (no execution in markdown) -- Relative URLs: preserved as-is (no resolution) +- Relative URLs: resolved only when a base is known, see Relative URL Resolution - Character encoding: `String::from_utf8_lossy` for byte input; html5ever handles encoding within HTML - Null bytes: replaced with U+FFFD by html5ever - Unclosed/mismatched tags: html5ever auto-repairs per HTML5 spec diff --git a/knowledge/foundations/architecture.md b/knowledge/foundations/architecture.md index 1e602e7..ca43cfa 100644 --- a/knowledge/foundations/architecture.md +++ b/knowledge/foundations/architecture.md @@ -67,6 +67,7 @@ Via `MarkkitBuilder`: - Link and image preservation - Office-document limits (archive entries/sizes, sheet rows/cols, grid cells, hidden sheets) - URL fetch policy (`allow_private_networks`, `allow_host`, `max_redirects`) +- Base URL for relative links, images and metadata URLs (`base_url`) - Provenance (`track_sources`): per-block `Source` in the IR (off by default) ## Public entry points diff --git a/knowledge/log.md b/knowledge/log.md index ba70360..63963b1 100644 --- a/knowledge/log.md +++ b/knowledge/log.md @@ -2,6 +2,7 @@ ## 2026-10-26 +* **Relative URLs**: links, images and metadata URLs resolve against `base_url` (or the final `extract_url` response URL) plus ``. Only scheme-less references change; page bases are limited to `http(s)`. New `extract_url_document`. See [HTML Extraction](formats/html-extraction.md) and [Threat Model](security/threat-model.md) T1. * **Transforms**: `transform::Transform` passes between parse and render (`MarkkitBuilder::transform`), built-ins `NormalizeHeadings` / `DropImages`, `from_fn`, IR walkers. `Error` is now `#[non_exhaustive]` with `Error::Transform`. See [Markkit Architecture](foundations/architecture.md), [Examples](operations/examples.md). * **Math + IR JSON Schema**: `Math` / `MathBlock` nodes (TeX), mapped from MathML annotations and KaTeX / MathJax wrappers; `schema/ir.schema.json` generated by `schemars` (`json-schema` feature) and checked in tests. See [IR Specification](foundations/ir.md), [HTML Extraction](formats/html-extraction.md). diff --git a/knowledge/security/threat-model.md b/knowledge/security/threat-model.md index 66496af..6e322d2 100644 --- a/knowledge/security/threat-model.md +++ b/knowledge/security/threat-model.md @@ -40,6 +40,8 @@ tags: - Form elements (`
`, ``, `