From cd23c0a40d33157b75818a754bbccb56e85e67a1 Mon Sep 17 00:00:00 2001 From: Mykhailo Chalyi Date: Mon, 26 Oct 2026 23:48:38 +0000 Subject: [PATCH] feat(features)!: make http_client opt-in Default markkit build no longer links reqwest/hyper. CLI and Python/Node bindings already enable http_client explicitly. CI asserts the default dependency tree stays network-free. --- .github/workflows/ci.yml | 4 ++++ AGENTS.md | 2 +- CHANGELOG.md | 3 +++ README.md | 11 ++++++----- SECURITY.md | 2 +- crates/markkit/Cargo.toml | 3 ++- crates/markkit/src/lib.rs | 5 +++-- knowledge/log.md | 1 + knowledge/operations/maintenance.md | 3 ++- knowledge/security/threat-model.md | 4 ++-- 10 files changed, 25 insertions(+), 13 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 647799b..e9f1895 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -78,6 +78,10 @@ jobs: run: cargo test --workspace --all-features - name: Run tests (office formats, no HTTP client) run: | + if cargo tree -p markkit -e normal | grep -E 'reqwest|hyper'; then + echo "HTTP client leaked into the default feature set" + exit 1 + fi cargo test -p markkit --no-default-features --features docx,spreadsheet if cargo tree -p markkit --no-default-features --features docx,spreadsheet -e normal | grep -E 'reqwest|hyper'; then echo "HTTP client leaked into the docx,spreadsheet feature set" diff --git a/AGENTS.md b/AGENTS.md index 9204780..9ee127f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -87,7 +87,7 @@ just pre-pr # Pre-PR checks - Stable Rust, pinned in `rust-toolchain.toml` - `cargo fmt`, `cargo clippy --workspace --all-targets -- -D warnings` - Licenses/advisories: `cargo deny check` (`deny.toml`), `cargo audit` -- Format deps stay feature-gated; `docx,spreadsheet` build must not link reqwest/hyper (CI checks) +- Format deps stay feature-gated; `http_client` opt-in (never default); default and `docx,spreadsheet` builds must not link reqwest/hyper (CI checks) ### Pre-PR Checklist diff --git a/CHANGELOG.md b/CHANGELOG.md index 6eb370e..94dc357 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -27,6 +27,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ### Changed +- **Breaking (features):** `http_client` is no longer a default feature; the default build pulls in + no HTTP/TLS stack. Library users calling `extract_url` add `features = ["http_client"]`. The CLI and + the Python/Node packages still enable it, so their `extract_url` is unchanged. - **Breaking (IR API):** `Node`, `Document`, `ListItem` and the new IR structs are `#[non_exhaustive]`; matches need a `_` arm and structs are built with constructors. `Node::Table` cells are now `TableCell` (`children`, `colspan`, `rowspan`) with an `alignments` field. DOCX `gridSpan` is diff --git a/README.md b/README.md index 0d9e55a..4da976f 100644 --- a/README.md +++ b/README.md @@ -150,20 +150,21 @@ let markkit = Markkit::builder() | Feature | Default | Description | |---------|---------|-------------| | `html` | Yes | HTML parsing and conversion | -| `http_client` | Yes | URL fetching via reqwest (`extract_url`) | +| `http_client` | No | URL fetching via reqwest (`extract_url`) | | `docx` | No | DOCX conversion (zip + quick-xml) | | `spreadsheet` | No | XLSX/XLSM, XLS, ODS conversion (calamine) | | `pdf` | No | Placeholder; PDF currently returns `UnsupportedFormat` | | `all_formats` | No | `html`, `pdf`, `docx`, `spreadsheet` | | `logging` | No | Reserved: pulls in `tracing`, emits no events yet | -`http_client` is optional. Embedders that fetch content themselves can build without -reqwest (no HTTP/TLS stack in the dependency tree): +`http_client` is opt-in: the default build has no HTTP/TLS stack in the dependency tree. +Enable it to get `extract_url` (the CLI and the Python/Node bindings ship with it on): ```toml [dependencies] -markkit = { version = "0.1", default-features = false, features = ["docx", "spreadsheet"] } -# add "html" if you also convert HTML +markkit = { version = "0.1", features = ["http_client"] } +# office formats only, no HTML, no network: +# markkit = { version = "0.1", default-features = false, features = ["docx", "spreadsheet"] } ``` Office-document limits are configurable on the builder: `max_archive_entries`, diff --git a/SECURITY.md b/SECURITY.md index 93dc1d3..b6f0864 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -32,7 +32,7 @@ Markkit processes untrusted input (HTML, DOCX, spreadsheets) and converts it to - **No code execution**: scripts, styles, forms, iframes and SVG are stripped; macros and formulas are never executed; XML entities are never expanded - **Resource limits**: input size, HTML/XML nesting depth, archive entry count and decompressed size, spreadsheet grid cells, rows and columns - **Explicit failure**: malformed input returns errors, never panics; unsupported formats fail explicitly -- **URL fetching**: enabled by the default `http_client` feature; http/https only, non-public addresses refused by default (after DNS and on every redirect), optional host allowlist, bounded redirects, body size and fetch time. Behind an environment HTTP proxy, the proxy resolves names, so restrict egress there too +- **URL fetching**: opt-in `http_client` feature (off by default; CLI and Python/Node bindings enable it); http/https only, non-public addresses refused by default (after DNS and on every redirect), optional host allowlist, bounded redirects, body size and fetch time. Behind an environment HTTP proxy, the proxy resolves names, so restrict egress there too Full analysis: [threat model](knowledge/security/threat-model.md). diff --git a/crates/markkit/Cargo.toml b/crates/markkit/Cargo.toml index f6f5d96..0bb24f7 100644 --- a/crates/markkit/Cargo.toml +++ b/crates/markkit/Cargo.toml @@ -14,11 +14,12 @@ description = "Extract clean markdown from any format – HTML, PDF, DOCX, and m readme = "../../README.md" [features] -default = ["html", "http_client"] +default = ["html"] html = ["dep:scraper", "dep:html5ever"] pdf = [] docx = ["dep:zip", "dep:quick-xml"] spreadsheet = ["dep:calamine", "dep:zip", "dep:quick-xml", "dep:cfb"] +# Opt-in: the default build has no network stack (no reqwest/hyper/TLS). http_client = ["dep:reqwest", "tokio/net"] logging = ["dep:tracing"] all_formats = ["html", "pdf", "docx", "spreadsheet"] diff --git a/crates/markkit/src/lib.rs b/crates/markkit/src/lib.rs index d6453e2..861eaef 100644 --- a/crates/markkit/src/lib.rs +++ b/crates/markkit/src/lib.rs @@ -35,9 +35,10 @@ //! - **DOCX** (`docx`): Convert Word documents //! - **Spreadsheet** (`spreadsheet`): Convert XLSX/XLSM, XLS and ODS workbooks to tables //! - **PDF** (`pdf`): Planned; currently returns [`Error::UnsupportedFormat`] -//! - **URL fetching** (`http_client`, default): Fetch and extract from URLs via reqwest +//! - **URL fetching** (`http_client`, opt-in): Fetch and extract from URLs via reqwest //! -//! Embedders that do their own fetching can drop reqwest entirely: +//! The default build has no network stack; enable `http_client` for +//! `Markkit::extract_url`. Office formats only, no HTML: //! `markkit = { version = "0.1", default-features = false, features = ["docx", "spreadsheet"] }`. #[cfg(any(feature = "docx", feature = "spreadsheet"))] diff --git a/knowledge/log.md b/knowledge/log.md index ea6e8bb..c558693 100644 --- a/knowledge/log.md +++ b/knowledge/log.md @@ -2,6 +2,7 @@ ## 2026-10-26 +* **HTTP client opt-in**: `http_client` dropped from the `markkit` default features; the default build has no reqwest/hyper. CLI and bindings enable it explicitly; CI asserts the default tree stays network-free. See [Threat Model](security/threat-model.md) T3 and [Maintenance](operations/maintenance.md). * **IR v2**: footnotes, definition lists, task lists, figures, table alignment and spans; `Document.ir_version` (`IR_VERSION = 2`), `#[non_exhaustive]` IR types, `Node::is_block`/`is_inline`. HTML maps GitHub/Pandoc footnotes, `
`, checkbox items, captioned figures, `colspan`/`rowspan`/alignment and row-header ``. New threat T2e and invariant S13 (span grid budget, footnote label sanitizing). See [IR Specification](foundations/ir.md), [HTML Extraction](formats/html-extraction.md), [Threat Model](security/threat-model.md). * **URL fetch hardening**: T3 SSRF is mitigated. `extract_url` blocks non-public addresses by default (URL, resolver and redirect layers), enforces scheme, allowlist, redirect, body and time limits; new invariant S12 and `url_security` tests. Proxy-resolved names are the documented residual. See [Threat Model](security/threat-model.md) and [Security Model](security/security-model.md). diff --git a/knowledge/operations/maintenance.md b/knowledge/operations/maintenance.md index e975a47..d5466dc 100644 --- a/knowledge/operations/maintenance.md +++ b/knowledge/operations/maintenance.md @@ -38,7 +38,8 @@ Preserve security contracts; never weaken limits, tests, or audit policy to ship needed and still justified - `cargo tree --duplicates` reviewed; no unused dependencies - Format dependencies stay behind their feature (`docx`, `spreadsheet`, `http_client`); - CI asserts the `docx,spreadsheet` build links no reqwest/hyper + and `http_client` is never default; CI asserts the default and `docx,spreadsheet` + builds link no reqwest/hyper - Dependabot (`.github/dependabot.yml`) covers cargo and github-actions weekly; review and merge its open PRs diff --git a/knowledge/security/threat-model.md b/knowledge/security/threat-model.md index d97296e..fb1b9b2 100644 --- a/knowledge/security/threat-model.md +++ b/knowledge/security/threat-model.md @@ -100,8 +100,8 @@ tags: **Impact**: High -- access to internal infrastructure. -**Likelihood**: Medium -- `extract_url()` exists whenever `http_client` is enabled, -and `http_client` is a **default** feature. +**Likelihood**: Medium -- `extract_url()` exists only when the opt-in `http_client` +feature is enabled (off by default; the CLI and Python/Node bindings enable it). **Status**: Mitigated (`crates/markkit/src/fetch.rs`).