Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/).

### Security

- Resource limits on every aspect, in one `markkit::Limits` struct (`MarkkitBuilder::limits(..)` or
per-field setters). New: `timeout` (default 60 s) now bounds parsing and rendering, not just URL
fetch; `max_output_size` (128 MiB) stops output amplification (e.g. one wide table row padding
thousands of rows); `max_nodes` (5 000 000) caps parsed HTML DOM / DOCX XML nodes;
`max_nesting_depth` (256) is configurable, with a hard ceiling of 512 that keeps deep input off
the host stack. Violations return `Error::ResourceLimit` naming the limit.

- `extract_url` (CLI, Python and Node `extract_url` too) now refuses non-public destinations by default:
loopback, private, link-local / cloud metadata and other reserved addresses, checked on the URL,
after DNS resolution and on every redirect. Only `http`/`https` are fetched. Redirects are capped
Expand All @@ -66,6 +73,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/).

### Changed

- **Breaking**: limit fields moved from `MarkkitBuilder` to `MarkkitBuilder::limits`
(`config().max_input_size` → `config().limits.max_input_size`); `max_processing_secs` field is
now `limits.timeout: Duration`. Builder setters are unchanged, plus new `timeout`,
`max_output_size`, `max_nodes`, `max_nesting_depth`, `limits`. HTML nesting now flattens at depth
256 (was 200).
- **Breaking (features):** `http_client` is no longer a default feature; the default build pulls in
no HTTP/TLS stack. Library users calling `extract_url` add `features = ["http_client"]`. The CLI and
the Python/Node packages still enable it, so their `extract_url` is unchanged.
Expand Down Expand Up @@ -114,6 +126,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/).

### Changed

- **Breaking**: limit fields moved from `MarkkitBuilder` to `MarkkitBuilder::limits`
(`config().max_input_size` → `config().limits.max_input_size`); `max_processing_secs` field is
now `limits.timeout: Duration`. Builder setters are unchanged, plus new `timeout`,
`max_output_size`, `max_nodes`, `max_nesting_depth`, `limits`. HTML nesting now flattens at depth
256 (was 200).

- Release automation: merging `chore(release): prepare vX.Y.Z` to main tags, creates the
GitHub Release and publishes to crates.io (idempotent); manual dispatch from main also works

Expand Down Expand Up @@ -160,6 +178,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/).

### Changed

- **Breaking**: limit fields moved from `MarkkitBuilder` to `MarkkitBuilder::limits`
(`config().max_input_size` → `config().limits.max_input_size`); `max_processing_secs` field is
now `limits.timeout: Duration`. Builder setters are unchanged, plus new `timeout`,
`max_output_size`, `max_nodes`, `max_nesting_depth`, `limits`. HTML nesting now flattens at depth
256 (was 200).

- `docx` feature now uses `zip` + `quick-xml` directly; `docx-rs` dependency removed
- `calamine` upgraded to 0.36 (pulls quick-xml 0.41, clearing RUSTSEC-2026-0194/0195); `zip` upgraded to 8 (deflate only), shared with calamine
- PDF extraction returns `Error::UnsupportedFormat` instead of an empty string
Expand Down
36 changes: 31 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -195,10 +195,36 @@ markkit = { version = "0.1", features = ["http_client"] }
# markkit = { version = "0.1", default-features = false, features = ["docx", "spreadsheet"] }
```

Office-document limits are configurable on the builder: `max_archive_entries`,
`max_archive_entry_size`, `max_archive_total_size`, `max_spreadsheet_cells`,
`max_sheet_rows`, `max_sheet_cols`, `include_hidden_sheets`. URL fetching:
`allow_private_networks`, `allow_host`, `max_redirects`.
### Resource limits

Every resource an input can drive is bounded by `markkit::Limits`. Defaults are
generous; violations return `Error::ResourceLimit` naming the limit.

| Limit | Default | Bounds |
|-------|---------|--------|
| `max_input_size` | 50 MiB | Input bytes, fetched URL body |
| `max_output_size` | 128 MiB | Markdown output |
| `timeout` | 60 s | Fetch + parse + render, per call |
| `max_nesting_depth` | 256 (hard max 512) | HTML flattens deeper content; deeper DOCX XML is rejected |
| `max_nodes` | 5 000 000 | HTML DOM nodes, DOCX XML nodes |
| `max_archive_entries` | 10 000 | ZIP entries (DOCX/XLSX/ODS) |
| `max_archive_entry_size` | 64 MiB | Decompressed bytes per ZIP entry |
| `max_archive_total_size` | 256 MiB | Decompressed bytes per ZIP |
| `max_spreadsheet_cells` | 4 000 000 | XLS/ODS grid before parsing |
| `max_sheet_rows` / `max_sheet_cols` | 10 000 / 256 | Rendered per sheet (truncated with a warning) |
| `max_redirects` | 5 | `extract_url` redirects |

```rust
use markkit::{Limits, Markkit};
use std::time::Duration;

let markkit = Markkit::builder()
.limits(Limits::default().timeout(Duration::from_secs(10)).max_output_size(8 << 20))
.build();
```

Each limit also has a builder setter (`.max_nodes(..)`, `.timeout(..)`, ...).
Other options: `include_hidden_sheets`, `allow_private_networks`, `allow_host`.

## Performance

Expand Down Expand Up @@ -244,7 +270,7 @@ XML entities are never expanded; macros and formulas are never executed; hidden
sheets are skipped by default. `extract_url` accepts only http/https, refuses
loopback, private, link-local and cloud-metadata addresses (also after DNS
resolution and on every redirect), caps redirects, and bounds the body by
`max_input_size` and the fetch by `max_processing_secs`. Opt in to internal
`max_input_size` and the fetch by `timeout`. Opt in to internal
hosts with `allow_private_networks(true)`; restrict hosts with `allow_host(..)`.

See the [threat model](knowledge/security/threat-model.md) and the
Expand Down
8 changes: 4 additions & 4 deletions crates/markkit/src/archive.rs
Original file line number Diff line number Diff line change
Expand Up @@ -41,11 +41,11 @@ impl<'a> ZipGuard<'a> {
/// Open the archive and check the entry count.
pub fn open(bytes: &'a [u8], config: &MarkkitBuilder) -> Result<Self, ArchiveError> {
let archive = ZipArchive::new(Cursor::new(bytes))?;
if archive.len() > config.max_archive_entries {
if archive.len() > config.limits.max_archive_entries {
return Err(ArchiveError::Limit(format!(
"archive has {} entries, limit is {}",
archive.len(),
config.max_archive_entries
config.limits.max_archive_entries
)));
}
let names = archive
Expand All @@ -55,8 +55,8 @@ impl<'a> ZipGuard<'a> {
Ok(Self {
archive,
names,
max_entry: config.max_archive_entry_size,
max_total: config.max_archive_total_size,
max_entry: config.limits.max_archive_entry_size,
max_total: config.limits.max_archive_total_size,
total: 0,
})
}
Expand Down
82 changes: 45 additions & 37 deletions crates/markkit/src/config.rs
Original file line number Diff line number Diff line change
@@ -1,12 +1,14 @@
use std::time::Duration;

use crate::extract::Markkit;
use crate::limits::Limits;

/// Builder for configuring a Markkit instance.
#[derive(Debug, Clone)]
pub struct MarkkitBuilder {
/// Maximum input size in bytes (default: 50MB).
pub max_input_size: usize,
/// Maximum processing time in seconds (default: 60).
pub max_processing_secs: u64,
/// Resource limits (input/output size, time, nesting, nodes, archives,
/// spreadsheets, redirects). See [`Limits`].
pub limits: Limits,
/// Whether to preserve link URLs in output.
pub preserve_links: bool,
/// Whether to preserve image references in output.
Expand All @@ -17,19 +19,6 @@ pub struct MarkkitBuilder {
pub include_selectors: Vec<String>,
/// CSS selectors for content to exclude.
pub exclude_selectors: Vec<String>,
/// Maximum number of entries in a ZIP container (DOCX/XLSX/ODS). Default: 10 000.
pub max_archive_entries: usize,
/// Maximum decompressed size of a single ZIP entry in bytes. Default: 64 MiB.
pub max_archive_entry_size: u64,
/// Maximum total decompressed size of a ZIP container in bytes. Default: 256 MiB.
pub max_archive_total_size: u64,
/// Maximum data rows rendered per sheet (header excluded). Default: 10 000.
pub max_sheet_rows: usize,
/// Maximum columns rendered per sheet. Default: 256.
pub max_sheet_cols: usize,
/// Maximum cell grid a workbook may expand to while parsing (XLS/ODS
/// pre-scan, summed over sheets). Default: 4 000 000.
pub max_spreadsheet_cells: u64,
/// Render hidden / very hidden sheets. Default: false (skipped with a warning).
pub include_hidden_sheets: bool,
/// Let `extract_url` reach loopback, private, link-local and other
Expand All @@ -38,8 +27,6 @@ pub struct MarkkitBuilder {
/// Hosts `extract_url` may contact, matched exactly and case-insensitively
/// on every hop (empty = any public host).
pub allowed_hosts: Vec<String>,
/// Maximum redirects `extract_url` follows. Default: 5.
pub max_redirects: usize,
/// Absolute URL that relative HTML links, images and metadata URLs resolve
/// against (a document `<base href>` applies on top). `extract_url` uses the
/// final response URL instead. Default: none (relative URLs kept as is).
Expand All @@ -59,8 +46,7 @@ pub struct MarkkitBuilder {
impl Default for MarkkitBuilder {
fn default() -> Self {
Self {
max_input_size: 50 * 1024 * 1024,
max_processing_secs: 60,
limits: Limits::default(),
preserve_links: true,
preserve_images: true,
extract_code_blocks: true,
Expand All @@ -73,16 +59,9 @@ impl Default for MarkkitBuilder {
"style".into(),
"noscript".into(),
],
max_archive_entries: 10_000,
max_archive_entry_size: 64 * 1024 * 1024,
max_archive_total_size: 256 * 1024 * 1024,
max_sheet_rows: 10_000,
max_sheet_cols: 256,
max_spreadsheet_cells: 4_000_000,
include_hidden_sheets: false,
allow_private_networks: false,
allowed_hosts: Vec::new(),
max_redirects: 5,
base_url: None,
track_sources: false,
transforms: Default::default(),
Expand All @@ -93,12 +72,41 @@ impl Default for MarkkitBuilder {

impl MarkkitBuilder {
pub fn max_input_size(mut self, size: usize) -> Self {
self.max_input_size = size;
self.limits.max_input_size = size;
self
}

/// Replace all limits at once.
pub fn limits(mut self, limits: Limits) -> Self {
self.limits = limits;
self
}

/// Wall-clock budget per call (fetch + parse + render).
pub fn timeout(mut self, timeout: Duration) -> Self {
self.limits.timeout = timeout;
self
}

/// [`MarkkitBuilder::timeout`] in whole seconds.
pub fn max_processing_secs(mut self, secs: u64) -> Self {
self.max_processing_secs = secs;
self.limits.timeout = Duration::from_secs(secs);
self
}

pub fn max_output_size(mut self, bytes: usize) -> Self {
self.limits.max_output_size = bytes;
self
}

/// Clamped to [`crate::limits::HARD_MAX_NESTING_DEPTH`].
pub fn max_nesting_depth(mut self, depth: usize) -> Self {
self.limits.max_nesting_depth = depth;
self
}

pub fn max_nodes(mut self, nodes: usize) -> Self {
self.limits.max_nodes = nodes;
self
}

Expand Down Expand Up @@ -135,32 +143,32 @@ impl MarkkitBuilder {
}

pub fn max_archive_entries(mut self, n: usize) -> Self {
self.max_archive_entries = n;
self.limits.max_archive_entries = n;
self
}

pub fn max_archive_entry_size(mut self, bytes: u64) -> Self {
self.max_archive_entry_size = bytes;
self.limits.max_archive_entry_size = bytes;
self
}

pub fn max_archive_total_size(mut self, bytes: u64) -> Self {
self.max_archive_total_size = bytes;
self.limits.max_archive_total_size = bytes;
self
}

pub fn max_sheet_rows(mut self, rows: usize) -> Self {
self.max_sheet_rows = rows;
self.limits.max_sheet_rows = rows;
self
}

pub fn max_sheet_cols(mut self, cols: usize) -> Self {
self.max_sheet_cols = cols;
self.limits.max_sheet_cols = cols;
self
}

pub fn max_spreadsheet_cells(mut self, cells: u64) -> Self {
self.max_spreadsheet_cells = cells;
self.limits.max_spreadsheet_cells = cells;
self
}

Expand All @@ -183,7 +191,7 @@ impl MarkkitBuilder {
}

pub fn max_redirects(mut self, n: usize) -> Self {
self.max_redirects = n;
self.limits.max_redirects = n;
self
}

Expand Down
Loading
Loading