diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a814768..f9a9f5f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -78,6 +78,10 @@ jobs: run: cargo test --workspace --all-features - name: Run tests (office formats, no HTTP client) run: | + if cargo tree -p markkit -e normal | grep -E 'reqwest|hyper'; then + echo "HTTP client leaked into the default feature set" + exit 1 + fi cargo test -p markkit --no-default-features --features docx,spreadsheet if cargo tree -p markkit --no-default-features --features docx,spreadsheet -e normal | grep -E 'reqwest|hyper'; then echo "HTTP client leaked into the docx,spreadsheet feature set" diff --git a/AGENTS.md b/AGENTS.md index f656360..6579a71 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -96,7 +96,7 @@ just pre-pr # Pre-PR checks - Stable Rust, pinned in `rust-toolchain.toml` - `cargo fmt`, `cargo clippy --workspace --all-targets -- -D warnings` - Licenses/advisories: `cargo deny check` (`deny.toml`), `cargo audit` -- Format deps stay feature-gated; `docx,spreadsheet` build must not link reqwest/hyper (CI checks) +- Format deps stay feature-gated; `http_client` opt-in (never default); default and `docx,spreadsheet` builds must not link reqwest/hyper (CI checks) ### Pre-PR Checklist diff --git a/CHANGELOG.md b/CHANGELOG.md index 620b9cc..d90454d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -62,6 +62,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ### Changed +- **Breaking (features):** `http_client` is no longer a default feature; the default build pulls in + no HTTP/TLS stack. Library users calling `extract_url` add `features = ["http_client"]`. The CLI and + the Python/Node packages still enable it, so their `extract_url` is unchanged. - **Breaking:** `Error` is `#[non_exhaustive]` (new `Error::Transform`); matches need a `_` arm. - **Breaking (IR API):** `Node`, `Document`, `ListItem` and the new IR structs are `#[non_exhaustive]`; matches need a `_` arm and structs are built with constructors. `Node::Table` cells are now diff --git a/README.md b/README.md index b39c035..0a4c332 100644 --- a/README.md +++ b/README.md @@ -174,7 +174,7 @@ let markkit = Markkit::builder() | Feature | Default | Description | |---------|---------|-------------| | `html` | Yes | HTML parsing and conversion | -| `http_client` | Yes | URL fetching via reqwest (`extract_url`) | +| `http_client` | No | URL fetching via reqwest (`extract_url`) | | `docx` | No | DOCX conversion (zip + quick-xml) | | `spreadsheet` | No | XLSX/XLSM, XLS, ODS conversion (calamine) | | `pdf` | No | Placeholder; PDF currently returns `UnsupportedFormat` | @@ -182,13 +182,14 @@ let markkit = Markkit::builder() | `tool` | No | `markkit::tool::MarkkitTool` LLM agent tool (Everruns toolkit contract) | | `logging` | No | Reserved: pulls in `tracing`, emits no events yet | -`http_client` is optional. Embedders that fetch content themselves can build without -reqwest (no HTTP/TLS stack in the dependency tree): +`http_client` is opt-in: the default build has no HTTP/TLS stack in the dependency tree. +Enable it to get `extract_url` (the CLI and the Python/Node bindings ship with it on): ```toml [dependencies] -markkit = { version = "0.1", default-features = false, features = ["docx", "spreadsheet"] } -# add "html" if you also convert HTML +markkit = { version = "0.1", features = ["http_client"] } +# office formats only, no HTML, no network: +# markkit = { version = "0.1", default-features = false, features = ["docx", "spreadsheet"] } ``` Office-document limits are configurable on the builder: `max_archive_entries`, diff --git a/SECURITY.md b/SECURITY.md index 93dc1d3..b6f0864 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -32,7 +32,7 @@ Markkit processes untrusted input (HTML, DOCX, spreadsheets) and converts it to - **No code execution**: scripts, styles, forms, iframes and SVG are stripped; macros and formulas are never executed; XML entities are never expanded - **Resource limits**: input size, HTML/XML nesting depth, archive entry count and decompressed size, spreadsheet grid cells, rows and columns - **Explicit failure**: malformed input returns errors, never panics; unsupported formats fail explicitly -- **URL fetching**: enabled by the default `http_client` feature; http/https only, non-public addresses refused by default (after DNS and on every redirect), optional host allowlist, bounded redirects, body size and fetch time. Behind an environment HTTP proxy, the proxy resolves names, so restrict egress there too +- **URL fetching**: opt-in `http_client` feature (off by default; CLI and Python/Node bindings enable it); http/https only, non-public addresses refused by default (after DNS and on every redirect), optional host allowlist, bounded redirects, body size and fetch time. Behind an environment HTTP proxy, the proxy resolves names, so restrict egress there too Full analysis: [threat model](knowledge/security/threat-model.md). diff --git a/crates/markkit/Cargo.toml b/crates/markkit/Cargo.toml index fbd8176..6e90639 100644 --- a/crates/markkit/Cargo.toml +++ b/crates/markkit/Cargo.toml @@ -14,11 +14,12 @@ description = "Extract clean markdown from any format – HTML, PDF, DOCX, and m readme = "../../README.md" [features] -default = ["html", "http_client"] +default = ["html"] html = ["dep:scraper", "dep:html5ever", "dep:ego-tree", "dep:url"] pdf = [] docx = ["dep:zip", "dep:quick-xml"] spreadsheet = ["dep:calamine", "dep:zip", "dep:quick-xml", "dep:cfb"] +# Opt-in: the default build has no network stack (no reqwest/hyper/TLS). http_client = ["dep:reqwest", "tokio/net"] logging = ["dep:tracing"] # LLM tool contract (`markkit::tool`): Everruns toolkit-library shape. diff --git a/crates/markkit/src/lib.rs b/crates/markkit/src/lib.rs index 7f9788b..31aaa5d 100644 --- a/crates/markkit/src/lib.rs +++ b/crates/markkit/src/lib.rs @@ -35,12 +35,13 @@ //! - **DOCX** (`docx`): Convert Word documents //! - **Spreadsheet** (`spreadsheet`): Convert XLSX/XLSM, XLS and ODS workbooks to tables //! - **PDF** (`pdf`): Planned; currently returns [`Error::UnsupportedFormat`] -//! - **URL fetching** (`http_client`, default): Fetch and extract from URLs via reqwest +//! - **URL fetching** (`http_client`, opt-in): Fetch and extract from URLs via reqwest //! - **Agent tool** (`tool`): `markkit::tool::MarkkitTool`, an LLM tool following the //! Everruns toolkit-library contract (schemas, OpenAI definition, `tower::Service`) //! - **JSON Schema** (`json-schema`): `ir::json_schema()` for the serialized IR //! -//! Embedders that do their own fetching can drop reqwest entirely: +//! The default build has no network stack; enable `http_client` for +//! `Markkit::extract_url`. Office formats only, no HTML: //! `markkit = { version = "0.1", default-features = false, features = ["docx", "spreadsheet"] }`. #[cfg(any(feature = "docx", feature = "spreadsheet"))] diff --git a/knowledge/log.md b/knowledge/log.md index 63963b1..5817f32 100644 --- a/knowledge/log.md +++ b/knowledge/log.md @@ -14,6 +14,7 @@ * **IR v2**: footnotes, definition lists, task lists, figures, table alignment and spans; `Document.ir_version` (`IR_VERSION = 2`), `#[non_exhaustive]` IR types, `Node::is_block`/`is_inline`. HTML maps GitHub/Pandoc footnotes, `
`, checkbox items, captioned figures, `colspan`/`rowspan`/alignment and row-header ``. New threat T2e and invariant S13 (span grid budget, footnote label sanitizing). See [IR Specification](foundations/ir.md), [HTML Extraction](formats/html-extraction.md), [Threat Model](security/threat-model.md). * **Agent tool**: new [Tool Contract](integrations/tool-contract.md) concept in a new `integrations/` domain. `MarkkitTool` (opt-in `tool` feature) follows the Everruns toolkit-library contract; tool arguments are a new untrusted surface in the [Threat Model](security/threat-model.md). * **URL fetch hardening**: T3 SSRF is mitigated. `extract_url` blocks non-public addresses by default (URL, resolver and redirect layers), enforces scheme, allowlist, redirect, body and time limits; new invariant S12 and `url_security` tests. Proxy-resolved names are the documented residual. See [Threat Model](security/threat-model.md) and [Security Model](security/security-model.md). +* **HTTP client opt-in**: `http_client` dropped from the `markkit` default features; the default build has no reqwest/hyper. CLI and bindings enable it explicitly; CI asserts the default tree stays network-free. See [Threat Model](security/threat-model.md) T3 and [Maintenance](operations/maintenance.md). * **Dependency trim**: unused dependencies removed (lockfile shrinks by ~460 lines; the `instant`/`paste` unmaintained advisories and their `deny.toml` ignores are gone with the stub bench deps). quick-xml moves to 0.42, one version shared with calamine; markkit's XML walkers now read `&str` names and values. See [Threat Model](security/threat-model.md) T6d and [Maintenance](operations/maintenance.md). * **Inter-block whitespace**: indented HTML lost blank lines between blocks and leaked a leading space into every following line, because whitespace-only text between blocks stayed in the block list as `Text(" ")`. The parser now normalizes block-context lists; golden fixtures, which had recorded the broken output, are regenerated. See [HTML Extraction](formats/html-extraction.md) and [IR Specification](foundations/ir.md). * **OKF migration**: `specs/` became this OKF v0.2 bundle, grouped into `foundations/`, `formats/`, `security/`, `runtimes/`, `operations/`. Every concept gained frontmatter and a See also section; `just check-okf` and CI enforce conformance. See [Knowledge Maintenance Contract](knowledge-contract.md). diff --git a/knowledge/operations/maintenance.md b/knowledge/operations/maintenance.md index d6fe86e..8dfe26d 100644 --- a/knowledge/operations/maintenance.md +++ b/knowledge/operations/maintenance.md @@ -39,7 +39,8 @@ Preserve security contracts; never weaken limits, tests, or audit policy to ship needed and still justified - `cargo tree --duplicates` reviewed; no unused dependencies - Format dependencies stay behind their feature (`docx`, `spreadsheet`, `http_client`); - CI asserts the `docx,spreadsheet` build links no reqwest/hyper + and `http_client` is never default; CI asserts the default and `docx,spreadsheet` + builds link no reqwest/hyper - Dependabot (`.github/dependabot.yml`) covers cargo and github-actions weekly; review and merge its open PRs diff --git a/knowledge/security/threat-model.md b/knowledge/security/threat-model.md index 6e322d2..b960847 100644 --- a/knowledge/security/threat-model.md +++ b/knowledge/security/threat-model.md @@ -113,8 +113,8 @@ tags: **Impact**: High -- access to internal infrastructure. -**Likelihood**: Medium -- `extract_url()` exists whenever `http_client` is enabled, -and `http_client` is a **default** feature. +**Likelihood**: Medium -- `extract_url()` exists only when the opt-in `http_client` +feature is enabled (off by default; the CLI and Python/Node bindings enable it). **Status**: Mitigated (`crates/markkit/src/fetch.rs`).