Repository navigation
feat(cli): embeddable CLI library + optional reqwest TLS - #189
Merged
Merged
Conversation
reqwest default features off; new default `reqwest-tls` feature carries TLS, HTTP/2, proxy discovery and decompression for ReqwestTransport, so embedders with their own HttpTransport get no second TLS stack. clap without color/suggestions to match bashkit. Markkit 0.2 API updates.
Move the fetchkit CLI into `fetchkit_cli::run(args, &dyn CliHost)`; the binary is a thin wrapper. Hosts own stdout/stderr, env, MCP stdio and egress, and may refuse host-only flags. Lets bashkit ship a `fetchkit` builtin with the identical interface.
chaliy
added a commit
to everruns/markkit
that referenced
this pull request
Nov 1, 2026
<!-- claude-projects-attribution: {"github_login":"chaliy"} -->
_Requested by **Михайло** · [project
thread](https://claude.ai/code/project/chan_019YEkBJuZMHJdXrjom86cGX?thread=cmsg_019YEkBJuZMHJdXrjom86cGXFSAmywce8BSJgNQ4YNsrks)_
## What changed
Prep so bashkit can ship a `markkit` builtin with the exact same CLI as
the standalone binary.
- **CLI is now a library.** `markkit_cli::run(args, &impl CliHost) ->
i32`; the `markkit` binary is a thin wrapper with `NativeHost`. The host
owns stdout/stderr, stdin, file reads (`--file`, `--rules`), URL
extraction and MCP stdio. `run` never exits the process or touches the
host filesystem.
- **URL extraction is a host method** that gets the configured
`Markkit`. `NativeHost` keeps calling `extract_url_document` (markkit's
SSRF policy). A sandbox fetches through its own egress, sets `base_url`
to the final URL and calls `extract_document`.
- **`native-http` feature** (default) carries `markkit/http_client` for
`NativeHost`. Without it the CLI library links no reqwest/TLS.
- **Hosts can refuse `--ocr`** (host Tesseract binary, host model
files). It still parses and shows in help; exit 1 with a one-line error
before any input is read. `mcp` is refused by any host that doesn't
implement `serve_mcp`.
- **clap without color/suggestions**, matching bashkit's clap set
(features unify).
- Errors keep anyhow's `Error: …` / `Caused by:` layout but no longer
append a backtrace when `RUST_BACKTRACE` is set (output must not depend
on env; embedders must not leak internals). A closed stdout pipe now
exits 0 instead of panicking in `print!`.
## Why
Design: markkit + fetchkit "preinstalled" in bashkit sessions with an
identical CLI. Sharing the code (not copying clap structs) keeps the
interface from drifting. Companion: everruns/fetchkit#189.
## Before / After
Binary output unchanged apart from the backtrace/broken-pipe notes
above. New `tests/host.rs`:
- `--help`, `extract --help`, `mcp --help`, `--version`: `run()` stdout
== binary stdout, byte for byte
- usage errors (unknown flag, no subcommand, `--layout` without
`--json`): same stderr as binary, exit 2
- runtime error (`extract` with no source): same stderr as binary, exit
1
- stdin (`--stdin`, `-`), `--file` + `--rules` read from an in-memory
host fs
- URL via host: relative links resolve against the host-reported final
URL
- refused `--ocr`: exit 1, no fetch; `mcp` refused by default host;
broken pipe exits 0
```
cargo tree -p markkit-cli --no-default-features --features all_formats -e normal -i reqwest
error: package ID specification `reqwest` did not match any packages
```
`just pre-pr`: fmt, clippy, test, check-okf, script tests, examples pass
locally. `cargo deny` is not installed here, so CI runs it (no new
dependencies).
## Risk
- Low
- **Breaking:** `markkit-cli` with `default-features = false` must
enable `native-http` for URL sources (CHANGELOG).
- CLI help/errors lose ANSI color on terminals.
4 tasks done
chaliy
added a commit
that referenced
this pull request
Nov 1, 2026
<!-- claude-projects-attribution: {"github_login":"chaliy"} -->
_Requested by **Михайло** · [project
thread](https://claude.ai/code/project/chan_019YEkBJuZMHJdXrjom86cGX?thread=cmsg_019YEkBJuZMHJdXrjom86cGXFSAmywce8BSJgNQ4YNsrks)_
## What changed
Release v0.7.0: workspace version 0.7.0, CHANGELOG section with
Highlights / Breaking Changes / What's Changed, knowledge log entry.
Two release-blocking fixes for embedding, found while wiring bashkit:
- **Markkit 0.3** (just published) so bashkit, which also embeds markkit
0.3, links one copy.
- **`fetchkit-cli` depends on `fetchkit` with `default-features =
false`** and gains matching `reqwest-tls` / `docs-pdf` features (default
on). Before, any `fetchkit-cli` dependency forced reqwest TLS
(aws-lc-sys) back on.
### Highlights
- Embed the exact `fetchkit` CLI with `fetchkit_cli::run(args, &dyn
CliHost)` (#189)
- TikTok, Vimeo, SoundCloud oEmbed metadata; YouTube Shorts/live/embed
URLs (#188)
- Build without a TLS stack when supplying your own `HttpTransport`
## Before / After
```
cargo tree -p fetchkit-cli --no-default-features -e normal -i aws-lc-sys
error: package ID specification `aws-lc-sys` did not match any packages
```
Default builds unchanged. `cargo test --workspace`, clippy (`-D
warnings`, also `fetchkit-cli --no-default-features`), fmt and OKF check
pass locally.
## Risk
- Low. Merging runs `release.yml` (tag, GitHub Release) and
`publish.yml` (`fetchkit`, then `fetchkit-cli`).
- Breaking (minor bump): `reqwest-tls` split, documented in CHANGELOG.
### Checklist
- [x] Unit tests are passed
- [x] Smoke tests are passed
- [x] Documentation is updated
- [x] Knowledge is up to date and not in conflict
---
_Generated by [Claude
Code](https://claude.ai/code/session_01Wy7MByPXsW4jHxfLeTfGEc)_
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Михайло · project thread
What changed
Prep so bashkit can ship a
fetchkitbuiltin with the exact same CLI as the standalone binary.fetchkit_cli::run(args, &dyn CliHost) -> i32. Thefetchkitbinary is a 6-line wrapper over it withNativeHost. The host owns stdout/stderr, env lookups (CLOUDFLARE_*), MCP stdio, and egress policy (configure_tool, e.g. a customHttpTransport).runnever callsprocess::exit, prints directly, or readsstd::env.--allow-env-proxy,--render-rakers,--bot-auth-key/--bot-auth-agent,--allow-remote-conversion;mcpis refused by any host that doesn't implementserve_mcp). Flags still parse and show in--help; the command fails before any request withError: <flag> <reason>, exit 1.reqwest-tlsfeature carries TLS, HTTP/2, proxy discovery and gzip/brotli/deflate forReqwestTransport. Withdefault-features = falsefetchkit links no TLS stack and no aws-lc-sys (bashkit is ring-only because aws-lc-sys breaks its aarch64 manylinux cross builds).limits.*, typed warning codes).StdinLockacross.await(needed for aSendfuture; same behavior).Why
Design: markkit + fetchkit "preinstalled" in bashkit sessions with an identical CLI. Sharing the code (not copying clap structs) is the only way to keep the interface from drifting. Companion: everruns/markkit#73.
Before / After
Binary behavior unchanged. New
tests/host.rsproves parity and the host contract:--help,fetch --help,mcp --help,--version:run()output == binary stdout, byte for byte, exit 0fetchthrough a host transport: frontmatter markdown + JSON output, every request (page and agent-discovery probes) went through the hostRisk
default-features = falseusers who rely on HTTPS throughReqwestTransport: enablereqwest-tls. CHANGELOG marks it.Checklist
cargo test --workspace, clippy-D warnings, fmt, docs-D warnings, release build)