diff --git a/CHANGELOG.md b/CHANGELOG.md index 0271b9e..f0102a2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Public TikTok, Vimeo, and SoundCloud metadata fetching through oEmbed, available automatically in the library, CLI, MCP server, and Python bindings. - YouTube Shorts, live, embed, and privacy-enhanced embed URL support, with validated video IDs. +- `fetchkit_cli` library: `run(args, &dyn CliHost)` runs the exact `fetchkit` CLI inside another program (used by bashkit's `fetchkit` builtin), with stdout/stderr, environment, MCP, and egress supplied by the host. + +### Changed + +- **Breaking:** reqwest TLS, HTTP/2, proxy discovery, and body decompression are behind the new default `reqwest-tls` feature. Builds with `default-features = false` must enable it to use the built-in `ReqwestTransport` over HTTPS. +- Require Markkit 0.2. +- clap is built without color and suggestions; CLI help and errors are plain text. ## [0.6.0] - 2026-10-23 diff --git a/Cargo.lock b/Cargo.lock index 12eddcd..388cf9c 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -950,8 +950,11 @@ dependencies = [ name = "fetchkit-cli" version = "0.6.0" dependencies = [ + "async-trait", + "bytes", "clap", "fetchkit", + "futures", "serde", "serde_json", "tokio", @@ -1739,25 +1742,16 @@ checksum = "c41e0c4fef86961ac6d6f8a82609f55f31b05e4fce149ac5710e439df7619ba4" [[package]] name = "markkit" -version = "0.1.1" +version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "04c6e21845c0a20a0001b0688c5d3207201ad7daa4208ab3eb25b556533e22cd" +checksum = "1d8fce8c0d6eda79af05de89944ed828693ca61ac117b19d79030f52e3f23ae8" dependencies = [ - "anyhow", - "async-trait", - "base64", "calamine", "cfb", - "encoding_rs", - "futures", - "mime", - "quick-xml 0.41.0", - "regex", + "quick-xml 0.42.0", "serde", "serde_json", "thiserror", - "tokio", - "unicode-segmentation", "zip", ] diff --git a/Cargo.toml b/Cargo.toml index feec92e..8912a21 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -16,8 +16,11 @@ categories = ["web-programming", "text-processing"] # Async runtime tokio = { version = "1.54.1", features = ["rt-multi-thread", "macros", "time", "sync"] } -# HTTP client -reqwest = { version = "0.13", features = ["gzip", "brotli", "deflate", "stream", "json"] } +# HTTP client. Default features off: TLS, HTTP/2, proxy discovery and body +# decompression are opted into by fetchkit's `reqwest-tls` feature (on by +# default), so embedders that supply their own `HttpTransport` (e.g. bashkit, +# ring-only rustls) do not get a second TLS stack or aws-lc-sys. +reqwest = { version = "0.13", default-features = false, features = ["stream", "json"] } # Serialization serde = { version = "1", features = ["derive"] } @@ -36,8 +39,11 @@ tracing = "0.1" # Error handling thiserror = "2" -# CLI -clap = { version = "4", features = ["derive"] } +# CLI. Default features off (no color, no did-you-mean suggestions) to match +# bashkit's clap set: bashkit embeds this CLI as a builtin, and clap features +# unify, so color/suggestions here would change bashkit's output and make help +# and error text differ between the standalone binary and the builtin. +clap = { version = "4", default-features = false, features = ["derive", "std", "help", "usage", "error-context"] } # Async streaming futures = "0.3" @@ -55,7 +61,7 @@ tower = { version = "0.5", features = ["util"] } # Binary document extraction pdf-inspector = "0.1.8" -markkit = { version = "0.1.1", default-features = false, features = ["docx", "spreadsheet"] } +markkit = { version = "0.2", default-features = false, features = ["docx", "spreadsheet"] } calamine = { version = "0.37", default-features = false } zip = { version = "9", default-features = false, features = ["deflate"] } quick-xml = "0.41" diff --git a/crates/fetchkit-cli/Cargo.toml b/crates/fetchkit-cli/Cargo.toml index 647ec30..5732629 100644 --- a/crates/fetchkit-cli/Cargo.toml +++ b/crates/fetchkit-cli/Cargo.toml @@ -10,6 +10,10 @@ keywords.workspace = true categories.workspace = true readme = "../../README.md" +[lib] +name = "fetchkit_cli" +path = "src/lib.rs" + [[bin]] # Keep CLI docs off so rustdoc doesn't collide with the library's `fetchkit` output. name = "fetchkit" @@ -32,6 +36,10 @@ tokio = { workspace = true } clap = { workspace = true } serde = { workspace = true } serde_json = { workspace = true } +async-trait = { workspace = true } [dev-dependencies] serde_json = { workspace = true } +bytes = { workspace = true } +futures = { workspace = true } +tokio = { workspace = true } diff --git a/crates/fetchkit-cli/src/lib.rs b/crates/fetchkit-cli/src/lib.rs new file mode 100644 index 0000000..a7796d8 --- /dev/null +++ b/crates/fetchkit-cli/src/lib.rs @@ -0,0 +1,748 @@ +//! Fetchkit CLI as a library. +//! +//! The `fetchkit` binary is a thin wrapper around [`run`]. Embedders (e.g. the +//! bashkit `fetchkit` builtin) call the same [`run`] with their own [`CliHost`], +//! so the command line interface (flags, help, version, errors, output) is the +//! same code everywhere. +//! +//! # Usage +//! +//! ```text +//! fetchkit fetch [--output md|json] [--user-agent ] +//! fetchkit mcp +//! fetchkit --llmtxt +//! ``` + +// Decisions: +// - All process-level effects go through `CliHost`: stdout/stderr, environment +// lookups, MCP stdio serving, and outbound policy (`configure_tool`). `run` +// never calls `std::process::exit`, `println!`, or `std::env`; it returns an +// exit code. This is what lets a sandbox (bashkit VFS + network allowlist) +// embed the CLI without the CLI touching the host process. +// - Flags a host cannot honor stay in the parser (help text is identical in +// every host) and are refused at run time via `CliHost::refuse`, with exit +// code 1 and a one-line error. +// - clap errors are rendered to a string and routed through the host. Exit +// codes match `clap::Error::exit`: 0 for --help/--version, 2 for usage errors. + +pub mod mcp; + +use async_trait::async_trait; +use clap::{Args, Parser, Subcommand, ValueEnum}; +use fetchkit::{ + CrawlPage, DocumentConversionSettings, FetchRequest, Tool, ToolBuilder, TOOL_LLMTXT, +}; +use std::ffi::OsString; +use std::io::{self, Write}; + +/// Optional behavior a host may refuse. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[non_exhaustive] +pub enum HostFeature { + /// `--allow-env-proxy`: honor HTTP(S)_PROXY / NO_PROXY. + EnvProxy, + /// `--render-rakers`: rendered HTML via the rakers JS engine. + RenderRakers, + /// `--bot-auth-key` / `--bot-auth-agent`: request signing inside fetchkit. + BotAuth, + /// `--allow-remote-conversion`: upload documents to a third-party converter. + RemoteConversion, +} + +impl HostFeature { + /// The command line flag that requested this feature. + pub fn flag(self) -> &'static str { + match self { + HostFeature::EnvProxy => "--allow-env-proxy", + HostFeature::RenderRakers => "--render-rakers", + HostFeature::BotAuth => "--bot-auth-key", + HostFeature::RemoteConversion => "--allow-remote-conversion", + } + } +} + +/// Environment the CLI runs in. +/// +/// [`NativeHost`] maps this to the real process. Sandboxed hosts map it to +/// their own output buffers, environment, and egress policy. +#[async_trait] +pub trait CliHost: Send + Sync { + /// Write to standard output. + fn write_stdout(&self, text: &str) -> io::Result<()>; + + /// Write to standard error. + fn write_stderr(&self, text: &str); + + /// Look up an environment variable. + fn env_var(&self, name: &str) -> Option; + + /// Apply host policy to the tool after the CLI flags were applied, e.g. a + /// custom [`fetchkit::HttpTransport`] or DNS policy. + fn configure_tool(&self, builder: ToolBuilder) -> ToolBuilder { + builder + } + + /// Return `Some(reason)` to refuse a feature requested by a flag. + fn refuse(&self, feature: HostFeature) -> Option { + let _ = feature; + None + } + + /// Serve the tool over MCP. Hosts without a process stdio refuse. + async fn serve_mcp(&self, tool: Tool) -> Result<(), String> { + let _ = tool; + Err("mcp is not available in this environment".to_string()) + } +} + +/// [`CliHost`] for the standalone `fetchkit` binary: real stdio and env. +#[derive(Debug, Default, Clone, Copy)] +pub struct NativeHost; + +#[async_trait] +impl CliHost for NativeHost { + fn write_stdout(&self, text: &str) -> io::Result<()> { + let stdout = io::stdout(); + let mut handle = stdout.lock(); + handle.write_all(text.as_bytes())?; + handle.flush() + } + + fn write_stderr(&self, text: &str) { + let _ = io::stderr().lock().write_all(text.as_bytes()); + } + + fn env_var(&self, name: &str) -> Option { + std::env::var(name).ok() + } + + async fn serve_mcp(&self, tool: Tool) -> Result<(), String> { + mcp::run_server(tool).await; + Ok(()) + } +} + +/// The clap command tree, for help/argument parity checks in embedders. +pub fn command() -> clap::Command { + ::command() +} + +/// Parse `args` (including the program name) and run the command. +/// +/// Returns the process exit code. +pub async fn run(args: I, host: &dyn CliHost) -> i32 +where + I: IntoIterator, + T: Into + Clone, +{ + let cli = match Cli::try_parse_from(args) { + Ok(cli) => cli, + Err(e) => { + let text = e.render().to_string(); + if e.use_stderr() { + host.write_stderr(&text); + } else if let Some(code) = write_out(host, &text) { + return code; + } + return e.exit_code(); + } + }; + match execute(cli, host).await { + Ok(()) => 0, + Err(Exit(code)) => code, + } +} + +/// Early exit carrying an exit code; the message was already written. +struct Exit(i32); + +fn fail(host: &dyn CliHost, message: impl std::fmt::Display) -> Exit { + host.write_stderr(&format!("Error: {message}\n")); + Exit(1) +} + +fn check(host: &dyn CliHost, feature: HostFeature, requested: bool) -> Result<(), Exit> { + if !requested { + return Ok(()); + } + match host.refuse(feature) { + Some(reason) => Err(fail(host, format!("{} {reason}", feature.flag()))), + None => Ok(()), + } +} + +/// Write to stdout. Broken pipe ends quietly with 0; other errors with 1. +fn write_out(host: &dyn CliHost, text: &str) -> Option { + match host.write_stdout(text) { + Ok(()) => None, + Err(e) if e.kind() == io::ErrorKind::BrokenPipe => Some(0), + Err(e) => { + host.write_stderr(&format!("Error writing to stdout: {e}\n")); + Some(1) + } + } +} + +fn writeln_out(host: &dyn CliHost, text: &str) -> Result<(), Exit> { + match write_out(host, &format!("{text}\n")) { + None => Ok(()), + Some(code) => Err(Exit(code)), + } +} + +async fn execute(cli: Cli, host: &dyn CliHost) -> Result<(), Exit> { + if cli.llmtxt { + return writeln_out(host, &TOOL_LLMTXT); + } + + match cli.command { + Some(Commands::Mcp { + hardened, + allow_env_proxy, + bot_auth_key, + bot_auth_agent, + documents, + }) => { + let tool = build_tool( + host, + ToolFlags { + user_agent: None, + hardened, + allow_env_proxy, + bot_auth_key, + bot_auth_agent, + render_rakers: false, + }, + &documents, + )?; + host.serve_mcp(tool).await.map_err(|e| fail(host, e)) + } + Some(Commands::Fetch { + url, + output, + user_agent, + hardened, + allow_env_proxy, + bot_auth_key, + bot_auth_agent, + content_focus, + crawl, + max_pages, + render_rakers, + documents, + }) => { + let flags = ToolFlags { + user_agent, + hardened, + allow_env_proxy, + bot_auth_key, + bot_auth_agent, + render_rakers, + }; + run_fetch( + host, + &url, + flags, + &documents, + output, + content_focus, + crawl, + max_pages, + ) + .await + } + None => { + host.write_stderr( + "Usage: fetchkit fetch \n or: fetchkit mcp\n or: fetchkit --help\n", + ); + Err(Exit(1)) + } + } +} + +/// Tool-shaping flags shared by `fetch` and `mcp`. +struct ToolFlags { + user_agent: Option, + hardened: bool, + allow_env_proxy: bool, + bot_auth_key: Option, + bot_auth_agent: Option, + render_rakers: bool, +} + +fn build_tool( + host: &dyn CliHost, + flags: ToolFlags, + documents: &DocumentArgs, +) -> Result { + check(host, HostFeature::EnvProxy, flags.allow_env_proxy)?; + check(host, HostFeature::RenderRakers, flags.render_rakers)?; + check( + host, + HostFeature::BotAuth, + flags.bot_auth_key.is_some() || flags.bot_auth_agent.is_some(), + )?; + check( + host, + HostFeature::RemoteConversion, + documents.allow_remote_conversion, + )?; + + let mut builder = Tool::builder().enable_markdown(true); + + let settings = documents.settings(flags.allow_env_proxy, host); + if !settings.is_default() { + let conversion = settings.build().map_err(|e| fail(host, e))?; + builder = builder.document_conversion(conversion); + } + + if flags.hardened { + builder = builder.hardened(); + } + + if flags.allow_env_proxy { + builder = builder.use_env_proxy(true); + } + + if let Some(ua) = flags.user_agent { + builder = builder.user_agent(ua); + } + + #[cfg(feature = "bot-auth")] + if let Some(ref key) = flags.bot_auth_key { + let config = fetchkit::BotAuthConfig::from_base64_seed(key).map_err(|e| fail(host, e))?; + let config = if let Some(ref fqdn) = flags.bot_auth_agent { + config.with_agent_fqdn(fqdn) + } else { + config + }; + builder = builder.bot_auth(config); + } + + #[cfg(not(feature = "bot-auth"))] + if flags.bot_auth_key.is_some() { + return Err(fail( + host, + "--bot-auth-key requires the bot-auth feature (rebuild with --features bot-auth)", + )); + } + + #[cfg(feature = "render-rakers")] + if flags.render_rakers { + builder = builder.enable_render_rakers(true); + } + + #[cfg(not(feature = "render-rakers"))] + if flags.render_rakers { + return Err(fail( + host, + "--render-rakers requires the render-rakers feature (rebuild with --features render-rakers)", + )); + } + + Ok(host.configure_tool(builder).build()) +} + +#[allow(clippy::too_many_arguments)] +async fn run_fetch( + host: &dyn CliHost, + url: &str, + flags: ToolFlags, + documents: &DocumentArgs, + output: OutputFormat, + content_focus: Option, + crawl: bool, + max_pages: usize, +) -> Result<(), Exit> { + // Build request with markdown conversion + let mut request = FetchRequest::new(url).as_markdown(); + if let Some(focus) = content_focus { + request = request.content_focus(focus); + } + if crawl { + request = request.crawl(true).max_pages(max_pages); + } + if flags.render_rakers { + request = request.render_rakers(); + } + + let tool = build_tool(host, flags, documents)?; + + match tool.execute(request).await { + Ok(response) => match output { + OutputFormat::Md => writeln_out(host, &format_md_with_frontmatter(&response)), + OutputFormat::Json => { + let json = serde_json::to_string_pretty(&response).map_err(|e| { + host.write_stderr(&format!("Error serializing response: {e}\n")); + Exit(1) + })?; + writeln_out(host, &json) + } + }, + Err(e) => Err(fail(host, e)), + } +} + +/// Output format for fetch subcommand +#[derive(Debug, Clone, Copy, Default, ValueEnum)] +enum OutputFormat { + /// Markdown with YAML frontmatter + #[default] + Md, + /// JSON format + Json, +} + +/// Document (PDF, Office) conversion options shared by `fetch` and `mcp`. +#[derive(Args, Debug, Default, Clone)] +struct DocumentArgs { + /// Document formats to convert, comma-separated (e.g. pdf,docx,xlsx) + #[arg(long = "convert", value_delimiter = ',')] + formats: Option>, + + /// Converter order, comma-separated (pdf-inspector, markkit, office, cloudflare). Cloudflare + /// reads CLOUDFLARE_ACCOUNT_ID and CLOUDFLARE_API_TOKEN. + #[arg(long = "converter", value_delimiter = ',')] + converters: Option>, + + /// Allow converters that upload documents to a third-party service + #[arg(long)] + allow_remote_conversion: bool, + + /// Maximum document download size in bytes (default 25 MB) + #[arg(long)] + max_document_bytes: Option, +} + +impl DocumentArgs { + fn settings(&self, respect_proxy_env: bool, host: &dyn CliHost) -> DocumentConversionSettings { + DocumentConversionSettings { + formats: self.formats.clone(), + converters: self.converters.clone(), + allow_remote: self.allow_remote_conversion, + max_document_bytes: self.max_document_bytes, + cloudflare_account_id: host.env_var("CLOUDFLARE_ACCOUNT_ID"), + cloudflare_api_token: host.env_var("CLOUDFLARE_API_TOKEN"), + respect_proxy_env, + } + } +} + +/// Fetchkit - AI-friendly web content fetching tool +#[derive(Parser, Debug)] +#[command(name = "fetchkit")] +#[command(author, version, about, long_about = None)] +struct Cli { + #[command(subcommand)] + command: Option, + + /// Print full help with examples (llmtxt) + #[arg(long)] + llmtxt: bool, +} + +#[derive(Subcommand, Debug)] +enum Commands { + /// Run as MCP (Model Context Protocol) server over stdio + Mcp { + /// Apply the hardened outbound policy profile + #[arg(long)] + hardened: bool, + + /// Allow HTTP_PROXY/HTTPS_PROXY/NO_PROXY from the environment + #[arg(long)] + allow_env_proxy: bool, + + /// Ed25519 secret key seed (base64url, 32 bytes) for Web Bot Auth signing + #[arg(long)] + bot_auth_key: Option, + + /// Agent FQDN for Signature-Agent header (requires --bot-auth-key) + #[arg(long)] + bot_auth_agent: Option, + + #[command(flatten)] + documents: DocumentArgs, + }, + /// Fetch URL and output as markdown with metadata frontmatter + Fetch { + /// URL to fetch + url: String, + + /// Output format + #[arg(long, short, default_value = "md")] + output: OutputFormat, + + /// Custom User-Agent + #[arg(long)] + user_agent: Option, + + /// Apply the hardened outbound policy profile + #[arg(long)] + hardened: bool, + + /// Allow HTTP_PROXY/HTTPS_PROXY/NO_PROXY from the environment + #[arg(long)] + allow_env_proxy: bool, + + /// Ed25519 secret key seed (base64url, 32 bytes) for Web Bot Auth signing + #[arg(long)] + bot_auth_key: Option, + + /// Agent FQDN for Signature-Agent header (requires --bot-auth-key) + #[arg(long)] + bot_auth_agent: Option, + + /// Extraction focus: full, main, readable, or agent + #[arg(long)] + content_focus: Option, + + /// Discover and fetch a bounded set of same-origin pages + #[arg(long)] + crawl: bool, + + /// Maximum crawl pages, including the seed + #[arg(long, default_value_t = 5)] + max_pages: usize, + + /// Render HTML with the rakers backend before conversion + #[arg(long)] + render_rakers: bool, + + #[command(flatten)] + documents: DocumentArgs, + }, +} + +fn yaml_quote(value: &str) -> String { + serde_json::to_string(value).unwrap_or_else(|_| "\"\"".to_string()) +} + +/// Format response as markdown with YAML frontmatter +pub fn format_md_with_frontmatter(response: &fetchkit::FetchResponse) -> String { + let mut output = String::new(); + + // Build frontmatter + output.push_str("---\n"); + output.push_str(&format!("url: {}\n", yaml_quote(&response.url))); + output.push_str(&format!("status_code: {}\n", response.status_code)); + if let Some(ref ct) = response.content_type { + output.push_str(&format!("source_content_type: {}\n", yaml_quote(ct))); + } + if let Some(size) = response.size { + output.push_str(&format!("source_size: {}\n", size)); + } + if let Some(ref lm) = response.last_modified { + output.push_str(&format!("last_modified: {}\n", yaml_quote(lm))); + } + if let Some(ref filename) = response.filename { + output.push_str(&format!("filename: {}\n", yaml_quote(filename))); + } + if let Some(truncated) = response.truncated { + if truncated { + output.push_str("truncated: true\n"); + } + } + if let Some(ref quality) = response.quality { + output.push_str(&format!("quality_score: {:.2}\n", quality.score)); + if !quality.warnings.is_empty() { + let warnings = + serde_json::to_string(&quality.warnings).unwrap_or_else(|_| "[]".to_string()); + output.push_str(&format!("quality_warnings: {}\n", warnings)); + } + if let Some(ref method) = quality.extraction_method { + output.push_str(&format!("extraction_method: {}\n", yaml_quote(method))); + } + if let Some(ref action) = quality.suggested_next_action { + output.push_str(&format!("suggested_next_action: {}\n", yaml_quote(action))); + } + } + if let Some(ref crawl) = response.crawl { + output.push_str(&format!("crawl_pages: {}\n", crawl.pages.len())); + if crawl.truncated.unwrap_or(false) { + output.push_str("crawl_truncated: true\n"); + } + } + output.push_str("---\n"); + + // Append content, or error as body for unsupported content + if let Some(ref content) = response.content { + output.push_str(content); + } else if let Some(ref err) = response.error { + output.push_str(err); + } + append_crawl_summary(&mut output, response); + + output +} + +fn append_crawl_summary(output: &mut String, response: &fetchkit::FetchResponse) { + let Some(ref crawl) = response.crawl else { + return; + }; + if crawl.pages.is_empty() { + return; + } + + output.push_str("\n\n## Crawl Discovery\n\n"); + for page in &crawl.pages { + output.push_str(&format!("- {}\n", format_crawl_page(page))); + } +} + +fn format_crawl_page(page: &CrawlPage) -> String { + let title = page.title.as_deref().unwrap_or(page.url.as_str()); + let mut summary = format!("[{}]({})", title.replace(['[', ']'], ""), page.url); + if let Some(status_code) = page.status_code { + summary.push_str(&format!(" - status {status_code}")); + } + if let Some(score) = page.quality_score { + summary.push_str(&format!(", quality {score:.2}")); + } + if let Some(word_count) = page.word_count { + summary.push_str(&format!(", {word_count} words")); + } + if let Some(ref error) = page.error { + summary.push_str(&format!(", error: {error}")); + } + summary +} + +#[cfg(test)] +mod tests { + use super::*; + use fetchkit::{CrawlPage, CrawlResult, FetchResponse, PageQuality}; + + #[test] + fn test_format_md_basic() { + let response = FetchResponse { + url: "https://example.com".to_string(), + status_code: 200, + content_type: Some("text/html".to_string()), + content: Some("# Hello World".to_string()), + ..Default::default() + }; + + let output = format_md_with_frontmatter(&response); + + assert!(output.starts_with("---\n")); + assert!(output.contains("url: \"https://example.com\"\n")); + assert!(output.contains("status_code: 200\n")); + assert!(output.contains("source_content_type: \"text/html\"\n")); + assert!(output.contains("---\n# Hello World")); + } + + #[test] + fn test_format_md_with_all_fields() { + let response = FetchResponse { + url: "https://example.com/page".to_string(), + status_code: 200, + content_type: Some("text/html".to_string()), + size: Some(1234), + last_modified: Some("Wed, 01 Jan 2025 00:00:00 GMT".to_string()), + filename: Some("page.html".to_string()), + truncated: Some(true), + quality: Some(PageQuality { + score: 0.72, + warnings: vec!["low_content".to_string()], + extraction_method: Some("agent_main".to_string()), + suggested_next_action: Some("retry_with_agent_focus_or_crawl".to_string()), + ..Default::default() + }), + content: Some("Content here".to_string()), + ..Default::default() + }; + + let output = format_md_with_frontmatter(&response); + + assert!(output.contains("source_size: 1234\n")); + assert!(output.contains("last_modified: \"Wed, 01 Jan 2025 00:00:00 GMT\"\n")); + assert!(output.contains("filename: \"page.html\"\n")); + assert!(output.contains("truncated: true\n")); + assert!(output.contains("quality_score: 0.72\n")); + assert!(output.contains("quality_warnings: [\"low_content\"]\n")); + assert!(output.contains("extraction_method: \"agent_main\"\n")); + assert!(output.contains("suggested_next_action: \"retry_with_agent_focus_or_crawl\"\n")); + } + + #[test] + fn test_format_md_error_as_body() { + let response = FetchResponse { + url: "https://example.com/file.pdf".to_string(), + status_code: 200, + content_type: Some("application/pdf".to_string()), + error: Some("Binary content not supported".to_string()), + ..Default::default() + }; + + let output = format_md_with_frontmatter(&response); + + // Error should appear as body, not in frontmatter + assert!(!output.contains("error:")); + assert!(output.ends_with("---\nBinary content not supported")); + } + + #[test] + fn test_format_md_truncated_false_omitted() { + let response = FetchResponse { + url: "https://example.com".to_string(), + status_code: 200, + truncated: Some(false), + content: Some("Content".to_string()), + ..Default::default() + }; + + let output = format_md_with_frontmatter(&response); + + // truncated: false should not appear + assert!(!output.contains("truncated")); + } + + #[test] + fn test_format_md_quotes_untrusted_scalars() { + let response = FetchResponse { + url: "https://example.com/a\nforged: true".to_string(), + status_code: 200, + filename: Some("*alias".to_string()), + content: Some("ok".to_string()), + ..Default::default() + }; + + let output = format_md_with_frontmatter(&response); + + assert!(output.contains("url: \"https://example.com/a\\nforged: true\"\n")); + assert!(output.contains("filename: \"*alias\"\n")); + assert!(!output.contains("\nforged: true\n")); + } + + #[test] + fn test_format_md_includes_crawl_summary() { + let response = FetchResponse { + url: "https://example.com".to_string(), + status_code: 200, + content: Some("# Home".to_string()), + crawl: Some(CrawlResult { + seed_url: "https://example.com".to_string(), + max_pages: 2, + pages: vec![CrawlPage { + url: "https://example.com/docs".to_string(), + status_code: Some(200), + title: Some("Docs".to_string()), + word_count: Some(42), + quality_score: Some(0.91), + ..Default::default() + }], + truncated: Some(true), + }), + ..Default::default() + }; + + let output = format_md_with_frontmatter(&response); + + assert!(output.contains("crawl_pages: 1\n")); + assert!(output.contains("crawl_truncated: true\n")); + assert!(output.contains("## Crawl Discovery")); + assert!(output + .contains("[Docs](https://example.com/docs) - status 200, quality 0.91, 42 words")); + } +} diff --git a/crates/fetchkit-cli/src/main.rs b/crates/fetchkit-cli/src/main.rs index 4225c61..6fe353a 100644 --- a/crates/fetchkit-cli/src/main.rs +++ b/crates/fetchkit-cli/src/main.rs @@ -1,589 +1,8 @@ -//! Fetchkit CLI - Command-line interface for fetching web content -//! -//! Provides the `fetchkit` binary with subcommands for fetching URLs -//! and running an MCP server. -//! -//! # Usage -//! -//! ```text -//! fetchkit fetch [--output md|json] [--user-agent ] -//! fetchkit mcp -//! fetchkit --llmtxt -//! ``` - -mod mcp; - -use clap::{Args, Parser, Subcommand, ValueEnum}; -use fetchkit::{CrawlPage, DocumentConversionSettings, FetchRequest, Tool, TOOL_LLMTXT}; -use std::io::{self, Write}; - -/// Output format for fetch subcommand -#[derive(Debug, Clone, Copy, Default, ValueEnum)] -enum OutputFormat { - /// Markdown with YAML frontmatter - #[default] - Md, - /// JSON format - Json, -} - -/// Document (PDF, Office) conversion options shared by `fetch` and `mcp`. -#[derive(Args, Debug, Default, Clone)] -struct DocumentArgs { - /// Document formats to convert, comma-separated (e.g. pdf,docx,xlsx) - #[arg(long = "convert", value_delimiter = ',')] - formats: Option>, - - /// Converter order, comma-separated (pdf-inspector, markkit, office, cloudflare). Cloudflare - /// reads CLOUDFLARE_ACCOUNT_ID and CLOUDFLARE_API_TOKEN. - #[arg(long = "converter", value_delimiter = ',')] - converters: Option>, - - /// Allow converters that upload documents to a third-party service - #[arg(long)] - allow_remote_conversion: bool, - - /// Maximum document download size in bytes (default 25 MB) - #[arg(long)] - max_document_bytes: Option, -} - -impl DocumentArgs { - fn settings(&self, respect_proxy_env: bool) -> DocumentConversionSettings { - DocumentConversionSettings { - formats: self.formats.clone(), - converters: self.converters.clone(), - allow_remote: self.allow_remote_conversion, - max_document_bytes: self.max_document_bytes, - cloudflare_account_id: std::env::var("CLOUDFLARE_ACCOUNT_ID").ok(), - cloudflare_api_token: std::env::var("CLOUDFLARE_API_TOKEN").ok(), - respect_proxy_env, - } - } -} - -/// Fetchkit - AI-friendly web content fetching tool -#[derive(Parser, Debug)] -#[command(name = "fetchkit")] -#[command(author, version, about, long_about = None)] -struct Cli { - #[command(subcommand)] - command: Option, - - /// Print full help with examples (llmtxt) - #[arg(long)] - llmtxt: bool, -} - -#[derive(Subcommand, Debug)] -enum Commands { - /// Run as MCP (Model Context Protocol) server over stdio - Mcp { - /// Apply the hardened outbound policy profile - #[arg(long)] - hardened: bool, - - /// Allow HTTP_PROXY/HTTPS_PROXY/NO_PROXY from the environment - #[arg(long)] - allow_env_proxy: bool, - - /// Ed25519 secret key seed (base64url, 32 bytes) for Web Bot Auth signing - #[arg(long)] - bot_auth_key: Option, - - /// Agent FQDN for Signature-Agent header (requires --bot-auth-key) - #[arg(long)] - bot_auth_agent: Option, - - #[command(flatten)] - documents: DocumentArgs, - }, - /// Fetch URL and output as markdown with metadata frontmatter - Fetch { - /// URL to fetch - url: String, - - /// Output format - #[arg(long, short, default_value = "md")] - output: OutputFormat, - - /// Custom User-Agent - #[arg(long)] - user_agent: Option, - - /// Apply the hardened outbound policy profile - #[arg(long)] - hardened: bool, - - /// Allow HTTP_PROXY/HTTPS_PROXY/NO_PROXY from the environment - #[arg(long)] - allow_env_proxy: bool, - - /// Ed25519 secret key seed (base64url, 32 bytes) for Web Bot Auth signing - #[arg(long)] - bot_auth_key: Option, - - /// Agent FQDN for Signature-Agent header (requires --bot-auth-key) - #[arg(long)] - bot_auth_agent: Option, - - /// Extraction focus: full, main, readable, or agent - #[arg(long)] - content_focus: Option, - - /// Discover and fetch a bounded set of same-origin pages - #[arg(long)] - crawl: bool, - - /// Maximum crawl pages, including the seed - #[arg(long, default_value_t = 5)] - max_pages: usize, - - /// Render HTML with the rakers backend before conversion - #[arg(long)] - render_rakers: bool, - - #[command(flatten)] - documents: DocumentArgs, - }, -} +//! Fetchkit CLI binary. All logic lives in the library (`fetchkit_cli::run`) +//! so embedders such as bashkit expose the exact same interface. #[tokio::main] async fn main() { - let cli = Cli::parse(); - - // Handle --llmtxt flag - if cli.llmtxt { - writeln_safe(&TOOL_LLMTXT); - std::process::exit(0); - } - - match cli.command { - Some(Commands::Mcp { - hardened, - allow_env_proxy, - bot_auth_key, - bot_auth_agent, - documents, - }) => { - mcp::run_server(build_tool( - None, - hardened, - allow_env_proxy, - bot_auth_key, - bot_auth_agent, - false, - &documents, - )) - .await; - } - Some(Commands::Fetch { - url, - output, - user_agent, - hardened, - allow_env_proxy, - bot_auth_key, - bot_auth_agent, - content_focus, - crawl, - max_pages, - render_rakers, - documents, - }) => { - let options = FetchCommandOptions { - documents, - output, - user_agent, - hardened, - allow_env_proxy, - bot_auth_key, - bot_auth_agent, - content_focus, - crawl, - max_pages, - render_rakers, - }; - run_fetch(&url, options).await; - } - None => { - eprintln!("Usage: fetchkit fetch "); - eprintln!(" or: fetchkit mcp"); - eprintln!(" or: fetchkit --help"); - std::process::exit(1); - } - } -} - -fn build_tool( - user_agent: Option, - hardened: bool, - allow_env_proxy: bool, - bot_auth_key: Option, - bot_auth_agent: Option, - render_rakers: bool, - documents: &DocumentArgs, -) -> Tool { - let mut builder = Tool::builder().enable_markdown(true); - - let settings = documents.settings(allow_env_proxy); - if !settings.is_default() { - let conversion = settings.build().unwrap_or_else(|e| { - eprintln!("Error: {e}"); - std::process::exit(1); - }); - builder = builder.document_conversion(conversion); - } - - if hardened { - builder = builder.hardened(); - } - - if allow_env_proxy { - builder = builder.use_env_proxy(true); - } - - if let Some(ua) = user_agent { - builder = builder.user_agent(ua); - } - - #[cfg(feature = "bot-auth")] - if let Some(ref key) = bot_auth_key { - let config = fetchkit::BotAuthConfig::from_base64_seed(key).unwrap_or_else(|e| { - eprintln!("Error: {e}"); - std::process::exit(1); - }); - let config = if let Some(ref fqdn) = bot_auth_agent { - config.with_agent_fqdn(fqdn) - } else { - config - }; - builder = builder.bot_auth(config); - } - - #[cfg(not(feature = "bot-auth"))] - if bot_auth_key.is_some() { - eprintln!("Error: --bot-auth-key requires the bot-auth feature (rebuild with --features bot-auth)"); - std::process::exit(1); - } - - let _ = bot_auth_agent; // suppress unused warning without feature - - #[cfg(feature = "render-rakers")] - if render_rakers { - builder = builder.enable_render_rakers(true); - } - - #[cfg(not(feature = "render-rakers"))] - if render_rakers { - eprintln!("Error: --render-rakers requires the render-rakers feature (rebuild with --features render-rakers)"); - std::process::exit(1); - } - - builder.build() -} - -struct FetchCommandOptions { - documents: DocumentArgs, - output: OutputFormat, - user_agent: Option, - hardened: bool, - allow_env_proxy: bool, - bot_auth_key: Option, - bot_auth_agent: Option, - content_focus: Option, - crawl: bool, - max_pages: usize, - render_rakers: bool, -} - -async fn run_fetch(url: &str, options: FetchCommandOptions) { - // Build request with markdown conversion - let mut request = FetchRequest::new(url).as_markdown(); - if let Some(focus) = options.content_focus { - request = request.content_focus(focus); - } - if options.crawl { - request = request.crawl(true).max_pages(options.max_pages); - } - if options.render_rakers { - request = request.render_rakers(); - } - let tool = build_tool( - options.user_agent, - options.hardened, - options.allow_env_proxy, - options.bot_auth_key, - options.bot_auth_agent, - options.render_rakers, - &options.documents, - ); - - // Execute request - match tool.execute(request).await { - Ok(response) => match options.output { - OutputFormat::Md => print_md_with_frontmatter(&response), - OutputFormat::Json => { - let json = serde_json::to_string_pretty(&response).unwrap_or_else(|e| { - eprintln!("Error serializing response: {}", e); - std::process::exit(1); - }); - writeln_safe(&json); - } - }, - Err(e) => { - eprintln!("Error: {}", e); - std::process::exit(1); - } - } -} - -fn print_md_with_frontmatter(response: &fetchkit::FetchResponse) { - writeln_safe(&format_md_with_frontmatter(response)); -} - -fn yaml_quote(value: &str) -> String { - serde_json::to_string(value).unwrap_or_else(|_| "\"\"".to_string()) -} - -/// Format response as markdown with YAML frontmatter -fn format_md_with_frontmatter(response: &fetchkit::FetchResponse) -> String { - let mut output = String::new(); - - // Build frontmatter - output.push_str("---\n"); - output.push_str(&format!("url: {}\n", yaml_quote(&response.url))); - output.push_str(&format!("status_code: {}\n", response.status_code)); - if let Some(ref ct) = response.content_type { - output.push_str(&format!("source_content_type: {}\n", yaml_quote(ct))); - } - if let Some(size) = response.size { - output.push_str(&format!("source_size: {}\n", size)); - } - if let Some(ref lm) = response.last_modified { - output.push_str(&format!("last_modified: {}\n", yaml_quote(lm))); - } - if let Some(ref filename) = response.filename { - output.push_str(&format!("filename: {}\n", yaml_quote(filename))); - } - if let Some(truncated) = response.truncated { - if truncated { - output.push_str("truncated: true\n"); - } - } - if let Some(ref quality) = response.quality { - output.push_str(&format!("quality_score: {:.2}\n", quality.score)); - if !quality.warnings.is_empty() { - let warnings = - serde_json::to_string(&quality.warnings).unwrap_or_else(|_| "[]".to_string()); - output.push_str(&format!("quality_warnings: {}\n", warnings)); - } - if let Some(ref method) = quality.extraction_method { - output.push_str(&format!("extraction_method: {}\n", yaml_quote(method))); - } - if let Some(ref action) = quality.suggested_next_action { - output.push_str(&format!("suggested_next_action: {}\n", yaml_quote(action))); - } - } - if let Some(ref crawl) = response.crawl { - output.push_str(&format!("crawl_pages: {}\n", crawl.pages.len())); - if crawl.truncated.unwrap_or(false) { - output.push_str("crawl_truncated: true\n"); - } - } - output.push_str("---\n"); - - // Append content, or error as body for unsupported content - if let Some(ref content) = response.content { - output.push_str(content); - } else if let Some(ref err) = response.error { - output.push_str(err); - } - append_crawl_summary(&mut output, response); - - output -} - -fn append_crawl_summary(output: &mut String, response: &fetchkit::FetchResponse) { - let Some(ref crawl) = response.crawl else { - return; - }; - if crawl.pages.is_empty() { - return; - } - - output.push_str("\n\n## Crawl Discovery\n\n"); - for page in &crawl.pages { - output.push_str(&format!("- {}\n", format_crawl_page(page))); - } -} - -fn format_crawl_page(page: &CrawlPage) -> String { - let title = page.title.as_deref().unwrap_or(page.url.as_str()); - let mut summary = format!("[{}]({})", title.replace(['[', ']'], ""), page.url); - if let Some(status_code) = page.status_code { - summary.push_str(&format!(" - status {status_code}")); - } - if let Some(score) = page.quality_score { - summary.push_str(&format!(", quality {score:.2}")); - } - if let Some(word_count) = page.word_count { - summary.push_str(&format!(", {word_count} words")); - } - if let Some(ref error) = page.error { - summary.push_str(&format!(", error: {error}")); - } - summary -} - -/// Write to stdout, exit silently on broken pipe -fn writeln_safe(s: &str) { - let stdout = io::stdout(); - let mut handle = stdout.lock(); - if let Err(e) = writeln!(handle, "{}", s) { - if e.kind() == io::ErrorKind::BrokenPipe { - std::process::exit(0); - } - eprintln!("Error writing to stdout: {}", e); - std::process::exit(1); - } -} - -#[cfg(test)] -mod tests { - use super::*; - use fetchkit::{CrawlPage, CrawlResult, FetchResponse, PageQuality}; - - #[test] - fn test_format_md_basic() { - let response = FetchResponse { - url: "https://example.com".to_string(), - status_code: 200, - content_type: Some("text/html".to_string()), - content: Some("# Hello World".to_string()), - ..Default::default() - }; - - let output = format_md_with_frontmatter(&response); - - assert!(output.starts_with("---\n")); - assert!(output.contains("url: \"https://example.com\"\n")); - assert!(output.contains("status_code: 200\n")); - assert!(output.contains("source_content_type: \"text/html\"\n")); - assert!(output.contains("---\n# Hello World")); - } - - #[test] - fn test_format_md_with_all_fields() { - let response = FetchResponse { - url: "https://example.com/page".to_string(), - status_code: 200, - content_type: Some("text/html".to_string()), - size: Some(1234), - last_modified: Some("Wed, 01 Jan 2025 00:00:00 GMT".to_string()), - filename: Some("page.html".to_string()), - truncated: Some(true), - quality: Some(PageQuality { - score: 0.72, - warnings: vec!["low_content".to_string()], - extraction_method: Some("agent_main".to_string()), - suggested_next_action: Some("retry_with_agent_focus_or_crawl".to_string()), - ..Default::default() - }), - content: Some("Content here".to_string()), - ..Default::default() - }; - - let output = format_md_with_frontmatter(&response); - - assert!(output.contains("source_size: 1234\n")); - assert!(output.contains("last_modified: \"Wed, 01 Jan 2025 00:00:00 GMT\"\n")); - assert!(output.contains("filename: \"page.html\"\n")); - assert!(output.contains("truncated: true\n")); - assert!(output.contains("quality_score: 0.72\n")); - assert!(output.contains("quality_warnings: [\"low_content\"]\n")); - assert!(output.contains("extraction_method: \"agent_main\"\n")); - assert!(output.contains("suggested_next_action: \"retry_with_agent_focus_or_crawl\"\n")); - } - - #[test] - fn test_format_md_error_as_body() { - let response = FetchResponse { - url: "https://example.com/file.pdf".to_string(), - status_code: 200, - content_type: Some("application/pdf".to_string()), - error: Some("Binary content not supported".to_string()), - ..Default::default() - }; - - let output = format_md_with_frontmatter(&response); - - // Error should appear as body, not in frontmatter - assert!(!output.contains("error:")); - assert!(output.ends_with("---\nBinary content not supported")); - } - - #[test] - fn test_format_md_truncated_false_omitted() { - let response = FetchResponse { - url: "https://example.com".to_string(), - status_code: 200, - truncated: Some(false), - content: Some("Content".to_string()), - ..Default::default() - }; - - let output = format_md_with_frontmatter(&response); - - // truncated: false should not appear - assert!(!output.contains("truncated")); - } - - #[test] - fn test_format_md_quotes_untrusted_scalars() { - let response = FetchResponse { - url: "https://example.com/a\nforged: true".to_string(), - status_code: 200, - filename: Some("*alias".to_string()), - content: Some("ok".to_string()), - ..Default::default() - }; - - let output = format_md_with_frontmatter(&response); - - assert!(output.contains("url: \"https://example.com/a\\nforged: true\"\n")); - assert!(output.contains("filename: \"*alias\"\n")); - assert!(!output.contains("\nforged: true\n")); - } - - #[test] - fn test_format_md_includes_crawl_summary() { - let response = FetchResponse { - url: "https://example.com".to_string(), - status_code: 200, - content: Some("# Home".to_string()), - crawl: Some(CrawlResult { - seed_url: "https://example.com".to_string(), - max_pages: 2, - pages: vec![CrawlPage { - url: "https://example.com/docs".to_string(), - status_code: Some(200), - title: Some("Docs".to_string()), - word_count: Some(42), - quality_score: Some(0.91), - ..Default::default() - }], - truncated: Some(true), - }), - ..Default::default() - }; - - let output = format_md_with_frontmatter(&response); - - assert!(output.contains("crawl_pages: 1\n")); - assert!(output.contains("crawl_truncated: true\n")); - assert!(output.contains("## Crawl Discovery")); - assert!(output - .contains("[Docs](https://example.com/docs) - status 200, quality 0.91, 42 words")); - } + let code = fetchkit_cli::run(std::env::args_os(), &fetchkit_cli::NativeHost).await; + std::process::exit(code); } diff --git a/crates/fetchkit-cli/src/mcp.rs b/crates/fetchkit-cli/src/mcp.rs index 8882f8b..8d69e4a 100644 --- a/crates/fetchkit-cli/src/mcp.rs +++ b/crates/fetchkit-cli/src/mcp.rs @@ -3,7 +3,7 @@ use fetchkit::{CrawlPage, Tool}; use serde::{Deserialize, Serialize}; use serde_json::{json, Value}; -use std::io::{self, BufRead, Write}; +use std::io::{self, Write}; /// JSON-RPC 2.0 request #[derive(Debug, Deserialize)] @@ -291,20 +291,26 @@ pub async fn run_server(tool: Tool) { let stdin = io::stdin(); let mut stdout = io::stdout(); - for line in stdin.lock().lines() { - let line = match line { - Ok(l) => l, + // Lock stdin per line (`Stdin::read_line`), never across the `.await` + // below: a held `StdinLock` would make this future `!Send`. + let mut buf = String::new(); + loop { + buf.clear(); + match stdin.read_line(&mut buf) { + Ok(0) => break, + Ok(_) => {} Err(e) => { eprintln!("Error reading stdin: {}", e); continue; } - }; + } + let line = buf.trim_end_matches('\n').trim_end_matches('\r'); if line.is_empty() { continue; } - let request: JsonRpcRequest = match serde_json::from_str(&line) { + let request: JsonRpcRequest = match serde_json::from_str(line) { Ok(req) => req, Err(e) => { let response = JsonRpcResponse::error(None, -32700, format!("Parse error: {}", e)); diff --git a/crates/fetchkit-cli/tests/host.rs b/crates/fetchkit-cli/tests/host.rs new file mode 100644 index 0000000..6b333dd --- /dev/null +++ b/crates/fetchkit-cli/tests/host.rs @@ -0,0 +1,266 @@ +//! `fetchkit_cli::run` with a custom `CliHost`: the embedding contract. +//! +//! Proves an embedder (e.g. the bashkit `fetchkit` builtin) gets the same CLI +//! as the binary, with output, env, transport, and refused flags all routed +//! through the host. No test here touches the network. + +use async_trait::async_trait; +use bytes::Bytes; +use fetchkit::{HttpTransport, ToolBuilder, TransportError, TransportRequest, TransportResponse}; +use fetchkit_cli::{run, CliHost, HostFeature}; +use std::io; +use std::process::Command; +use std::sync::{Arc, Mutex}; + +/// Records stdout/stderr; refuses features listed in `refused`. +#[derive(Default)] +struct TestHost { + stdout: Mutex, + stderr: Mutex, + refused: Vec, + transport: Option>, + broken_pipe: bool, +} + +impl TestHost { + fn out(&self) -> String { + self.stdout.lock().unwrap().clone() + } + fn err(&self) -> String { + self.stderr.lock().unwrap().clone() + } +} + +#[async_trait] +impl CliHost for TestHost { + fn write_stdout(&self, text: &str) -> io::Result<()> { + if self.broken_pipe { + return Err(io::Error::from(io::ErrorKind::BrokenPipe)); + } + self.stdout.lock().unwrap().push_str(text); + Ok(()) + } + fn write_stderr(&self, text: &str) { + self.stderr.lock().unwrap().push_str(text); + } + fn env_var(&self, _name: &str) -> Option { + None + } + fn configure_tool(&self, builder: ToolBuilder) -> ToolBuilder { + match &self.transport { + // The host owns egress policy; no host DNS resolution in tests. + Some(t) => builder.transport(t.clone()).block_private_ips(false), + None => builder, + } + } + fn refuse(&self, feature: HostFeature) -> Option { + self.refused + .contains(&feature) + .then(|| "is not available in this environment".to_string()) + } +} + +/// Serves one HTML page for every request and counts calls. +struct PageTransport { + calls: Mutex>, +} + +#[async_trait] +impl HttpTransport for PageTransport { + async fn execute(&self, req: TransportRequest) -> Result { + self.calls.lock().unwrap().push(req.url.to_string()); + let body = Bytes::from_static(b"Hi

Hello

From the host transport.

"); + Ok(TransportResponse { + status: 200, + url: req.url, + headers: vec![("content-type".into(), "text/html".into())], + body: Box::pin(futures::stream::once(async move { Ok(body) })), + }) + } +} + +async fn run_with(host: &TestHost, args: &[&str]) -> i32 { + run( + std::iter::once("fetchkit").chain(args.iter().copied()), + host, + ) + .await +} + +fn binary(args: &[&str]) -> std::process::Output { + Command::new(env!("CARGO_BIN_EXE_fetchkit")) + .args(args) + .output() + .expect("run fetchkit binary") +} + +#[tokio::test] +async fn help_and_version_match_binary_byte_for_byte() { + for args in [ + &["--help"][..], + &["fetch", "--help"], + &["mcp", "--help"], + &["--version"], + ] { + let host = TestHost::default(); + assert_eq!(run_with(&host, args).await, 0, "{args:?}"); + let bin = binary(args); + assert_eq!(bin.status.code(), Some(0), "{args:?}"); + assert_eq!(host.out(), String::from_utf8_lossy(&bin.stdout), "{args:?}"); + assert!(host.err().is_empty()); + } +} + +#[tokio::test] +async fn usage_error_matches_binary_and_exits_2() { + let args = ["fetch", "--no-such-flag", "https://example.com"]; + let host = TestHost::default(); + assert_eq!(run_with(&host, &args).await, 2); + let bin = binary(&args); + assert_eq!(bin.status.code(), Some(2)); + assert_eq!(host.err(), String::from_utf8_lossy(&bin.stderr)); + assert!(host.out().is_empty()); +} + +#[tokio::test] +async fn no_subcommand_prints_usage_and_exits_1() { + let host = TestHost::default(); + assert_eq!(run_with(&host, &[]).await, 1); + assert!(host.err().starts_with("Usage: fetchkit fetch ")); +} + +#[tokio::test] +async fn llmtxt_goes_to_host_stdout() { + let host = TestHost::default(); + assert_eq!(run_with(&host, &["--llmtxt"]).await, 0); + assert!(!host.out().is_empty()); +} + +#[tokio::test] +async fn fetch_uses_host_transport_and_writes_frontmatter() { + let transport = Arc::new(PageTransport { + calls: Mutex::new(Vec::new()), + }); + let host = TestHost { + transport: Some(transport.clone()), + ..Default::default() + }; + assert_eq!( + run_with(&host, &["fetch", "http://site.test/page"]).await, + 0, + "stderr: {}", + host.err() + ); + let out = host.out(); + assert!( + out.starts_with("---\nurl: \"http://site.test/page\"\n"), + "{out}" + ); + assert!(out.contains("# Hello"), "{out}"); + // Every request (page + agent discovery probes) went through the host. + let calls = transport.calls.lock().unwrap(); + assert_eq!(calls[0], "http://site.test/page"); + assert!(calls.iter().all(|u| u.starts_with("http://site.test/"))); +} + +#[tokio::test] +async fn fetch_json_output_through_host() { + let host = TestHost { + transport: Some(Arc::new(PageTransport { + calls: Mutex::new(Vec::new()), + })), + ..Default::default() + }; + assert_eq!( + run_with(&host, &["fetch", "http://site.test/", "-o", "json"]).await, + 0 + ); + let json: serde_json::Value = serde_json::from_str(&host.out()).unwrap(); + assert_eq!(json["status_code"], 200); +} + +#[tokio::test] +async fn refused_flags_fail_before_any_request() { + let cases: [(&[&str], HostFeature); 4] = [ + (&["--allow-env-proxy"], HostFeature::EnvProxy), + (&["--render-rakers"], HostFeature::RenderRakers), + (&["--bot-auth-key", "x"], HostFeature::BotAuth), + ( + &["--allow-remote-conversion"], + HostFeature::RemoteConversion, + ), + ]; + for (flags, feature) in cases { + let transport = Arc::new(PageTransport { + calls: Mutex::new(Vec::new()), + }); + let host = TestHost { + refused: vec![feature], + transport: Some(transport.clone()), + ..Default::default() + }; + let mut args = vec!["fetch", "http://site.test/"]; + args.extend_from_slice(flags); + assert_eq!(run_with(&host, &args).await, 1, "{flags:?}"); + assert_eq!( + host.err(), + format!( + "Error: {} is not available in this environment\n", + feature.flag() + ) + ); + assert!(host.out().is_empty()); + assert!(transport.calls.lock().unwrap().is_empty(), "{flags:?}"); + } +} + +#[tokio::test] +async fn bot_auth_agent_alone_is_refused_too() { + let host = TestHost { + refused: vec![HostFeature::BotAuth], + ..Default::default() + }; + let code = run_with( + &host, + &["fetch", "http://site.test/", "--bot-auth-agent", "a.test"], + ) + .await; + assert_eq!(code, 1); + assert!(host.err().contains("--bot-auth-key")); +} + +#[tokio::test] +async fn mcp_is_refused_by_default_host() { + let host = TestHost::default(); + assert_eq!(run_with(&host, &["mcp"]).await, 1); + assert_eq!( + host.err(), + "Error: mcp is not available in this environment\n" + ); +} + +#[tokio::test] +async fn broken_pipe_on_stdout_exits_0() { + let host = TestHost { + broken_pipe: true, + ..Default::default() + }; + assert_eq!(run_with(&host, &["--help"]).await, 0); + assert!(host.err().is_empty()); +} + +#[tokio::test] +async fn fetch_error_is_reported_once_with_exit_1() { + // Default DNS policy + unresolvable host: fails inside fetchkit, no panic. + let host = TestHost::default(); + assert_eq!(run_with(&host, &["fetch", "not a url"]).await, 1); + assert!(host.err().starts_with("Error: "), "{}", host.err()); +} + +#[test] +fn run_future_is_send() { + // Embedders run the CLI inside `Send` async contexts (bashkit builtins). + fn assert_send(_: T) {} + let host = TestHost::default(); + assert_send(run(["fetchkit", "--help"], &host)); +} diff --git a/crates/fetchkit/Cargo.toml b/crates/fetchkit/Cargo.toml index d5f900c..b31dd8d 100644 --- a/crates/fetchkit/Cargo.toml +++ b/crates/fetchkit/Cargo.toml @@ -11,7 +11,20 @@ categories.workspace = true readme = "../../README.md" [features] -default = ["docs-pdf", "docs-markkit"] +default = ["reqwest-tls", "docs-pdf", "docs-markkit"] +# TLS (rustls, reqwest's default provider), HTTP/2, system proxy discovery and +# gzip/brotli/deflate decoding for the built-in `ReqwestTransport`. Embedders +# that route all traffic through their own `HttpTransport` disable default +# features and leave this off (see knowledge/foundations/fetchers.md). +reqwest-tls = [ + "reqwest/default-tls", + "reqwest/charset", + "reqwest/http2", + "reqwest/system-proxy", + "reqwest/gzip", + "reqwest/brotli", + "reqwest/deflate", +] # Document conversion backends (see knowledge/foundations/fetchers.md) docs-pdf = ["dep:pdf-inspector"] # Office documents (DOCX, XLSX/XLSM, XLS, ODS) via markkit. diff --git a/crates/fetchkit/src/documents/markkit.rs b/crates/fetchkit/src/documents/markkit.rs index 9e827e1..1e64052 100644 --- a/crates/fetchkit/src/documents/markkit.rs +++ b/crates/fetchkit/src/documents/markkit.rs @@ -100,7 +100,7 @@ impl DocumentConverter for MarkkitConverter { } fn max_input_bytes(&self) -> Option { - Some(self.markkit.config().max_input_size) + Some(self.markkit.config().limits.max_input_size) } async fn convert(&self, input: DocumentInput) -> Result { @@ -131,11 +131,7 @@ impl DocumentConverter for MarkkitConverter { let mut warnings: Vec = Vec::new(); for warning in &extraction.warnings { - let code = warning - .split_once(':') - .map_or(warning.as_str(), |(code, _)| code) - .trim() - .to_string(); + let code = warning.code.as_str().to_string(); if !code.is_empty() && !warnings.contains(&code) { warnings.push(code); } @@ -373,7 +369,10 @@ mod tests { ConvertError::Unsupported(_) )); assert!(matches!( - map_error(markkit::Error::ResourceLimit("too big".into())), + map_error(markkit::Error::ResourceLimit(markkit::LimitExceeded::new( + markkit::LimitKind::MaxInputSize, + "too big", + ))), ConvertError::Failed(message) if message.contains("too big") )); } diff --git a/crates/fetchkit/src/documents/tables.rs b/crates/fetchkit/src/documents/tables.rs index 93a6eb7..7052acc 100644 --- a/crates/fetchkit/src/documents/tables.rs +++ b/crates/fetchkit/src/documents/tables.rs @@ -128,12 +128,12 @@ pub(super) fn check_archive( ) }; let mut archive = zip::ZipArchive::new(Cursor::new(body)).map_err(|_| fail())?; - if archive.len() > config.max_archive_entries.min(10_000) { + if archive.len() > config.limits.max_archive_entries.min(10_000) { return Err(fail()); } let mut total = 0u64; - let part_limit = config.max_archive_entry_size.min(64 * 1024 * 1024); - let total_limit = config.max_archive_total_size.min(256 * 1024 * 1024); + let part_limit = config.limits.max_archive_entry_size.min(64 * 1024 * 1024); + let total_limit = config.limits.max_archive_total_size.min(256 * 1024 * 1024); for index in 0..archive.len() { let entry = archive.by_index_raw(index).map_err(|_| fail())?; total = total @@ -207,7 +207,7 @@ pub(super) fn check_archive( parts.join("/") }; paths.insert(path); - if paths.len() > config.max_archive_entries.min(10_000) { + if paths.len() > config.limits.max_archive_entries.min(10_000) { return Err(fail()); } } diff --git a/knowledge/foundations/fetchers.md b/knowledge/foundations/fetchers.md index de22057..82faefb 100644 --- a/knowledge/foundations/fetchers.md +++ b/knowledge/foundations/fetchers.md @@ -407,7 +407,11 @@ checks, bot-auth signing, and body-size/timeout caps; only the socket-level send is delegated. `FetchOptions.transport` selects the implementation (`None` => default -`ReqwestTransport`). A host application can supply its own transport to route +`ReqwestTransport`). reqwest is built with default features off; the default +`reqwest-tls` feature adds rustls (reqwest's default provider), HTTP/2, system +proxy discovery, and gzip/brotli/deflate decoding for `ReqwestTransport`. +Embedders that supply their own transport (bashkit: ring-only rustls) disable +default features so fetchkit adds no second TLS stack or aws-lc-sys. A host application can supply its own transport to route fetchkit through a dedicated egress boundary without weakening any security policy. When `pinned_addrs` is non-empty a transport MUST connect only to those addresses (TM-SSRF-001, TM-SSRF-005). diff --git a/knowledge/foundations/tool-contract.md b/knowledge/foundations/tool-contract.md index 94b9af2..7eea23e 100644 --- a/knowledge/foundations/tool-contract.md +++ b/knowledge/foundations/tool-contract.md @@ -180,6 +180,23 @@ Provide a builder to configure tool options, including: - Binary name: `fetchkit`. - CLI provides a convenient interface optimized for LLM consumption. +- The CLI is a library: `fetchkit_cli::run(args, &dyn CliHost) -> i32` (crate + `fetchkit-cli`, lib name `fetchkit_cli`). The binary is a thin wrapper over + `run` with `NativeHost`. Embedders (bashkit's `fetchkit` builtin) call the same + `run` with their own host, so flags, help, version, errors, output, and exit + codes are identical everywhere. +- `CliHost` owns every process-level effect: stdout/stderr, env lookups + (`CLOUDFLARE_*`), MCP stdio (`serve_mcp`, refused by default), and egress policy + (`configure_tool`, e.g. a custom `HttpTransport`). `run` never exits the process, + prints directly, or reads `std::env`. +- Hosts may refuse flags via `CliHost::refuse(HostFeature)`: `--allow-env-proxy`, + `--render-rakers`, `--bot-auth-key`/`--bot-auth-agent`, + `--allow-remote-conversion`. Refused flags still parse and show in help; the + command fails before any request with `Error: ` and exit 1. +- Exit codes: 0 success / `--help` / `--version` / broken stdout pipe; 1 runtime + error or refused flag; 2 usage error (clap). +- clap is built with default features off (no color, no suggestions), matching + bashkit's clap set so feature unification does not change either side's output. - Subcommands: - `fetch ` - Fetch URL and convert to markdown - `mcp` - Run as MCP server over stdio diff --git a/knowledge/log.md b/knowledge/log.md index e0d4cd1..3aa14eb 100644 --- a/knowledge/log.md +++ b/knowledge/log.md @@ -1,5 +1,9 @@ # Fetchkit Knowledge Update Log +## 2026-11-01 + +* **Embeddable CLI**: The CLI moved into the `fetchkit_cli` library (`run(args, &dyn CliHost)`) so bashkit can ship a `fetchkit` builtin with the identical interface. Hosts own stdio, env, MCP, and egress, and can refuse host-only flags. reqwest TLS/compression moved behind the default `reqwest-tls` feature; clap default features off; Markkit 0.2. + ## 2026-10-27 * **Social media**: Add public TikTok/Vimeo/SoundCloud oEmbed metadata fetching with fixed endpoints, policy enforcement, bounded output and explicit media limits. Extend YouTube URL matching to Shorts/live/embeds and validate IDs before API query construction.