diff --git a/README.md b/README.md index 26959ace4..9e8cc1e01 100644 --- a/README.md +++ b/README.md @@ -112,6 +112,17 @@ Drop this into your project's `CLAUDE.md` or equivalent: For any file search or grep in the current git-indexed directory, use fff tools. ``` +### Definition search + +MCP and Pi expose only the optional `definitions_only` boolean to the model. MCP accepts `{ "query": "AudioMap", "definitions_only": true }`; Pi's `ffgrep` uses `pattern` instead of `query`. Single-query grep tries these stages in order: + +1. Matching declarations. +2. Ordinary exact matches, with a notice that no matching declarations were found. +3. Fuzzy declarations, labeled “Maybe you meant these definitions”. +4. Fuzzy ordinary matches, labeled “Maybe you meant these other matches”. + +A stage advances only after its complete search returns no matches. A partial page keeps its current stage, and a stage that returned matches never broadens when pagination ends. Cursors preserve the stage, and all stages keep the original file constraints. The multi-pattern tools accept the same flag but use only the first two stages; their patterns remain literal. + ### What changes - Frecency memory. Files you actually open rank higher next time. Warm-up from git touch history runs automatically. @@ -153,6 +164,8 @@ Env vars: `PI_FFF_MODE`, `FFF_FRECENCY_DB`, `FFF_HISTORY_DB`. Flags: `--fff-mode - `ffgrep`. Content search. Accepts `path`, `exclude` (comma, space, or array; leading `!` optional), `caseSensitive`, `context`, and cursor pagination. Auto-detects regex, falls back to fuzzy on zero exact matches, rejects `.*`-style wildcard-only patterns up front. - `fffind`. Path and filename search. Matches the whole repo-relative path, not just the filename. Frecency-aware. The weak-match detector flags scattered fuzzy noise before it floods the agent's context. +Set `definitions_only: true` on `ffgrep` to try declarations first, then exact references, fuzzy declarations, and fuzzy references. Each fallback includes a notice. See the [extension documentation](packages/pi-fff/README.md) for pagination and multi-pattern behavior. + ### Commands - `/fff-mode [tools-and-ui | tools-only | override]`. Show or switch the mode. @@ -491,27 +504,9 @@ require('fff').setup({ `` cycles between `plain`, `regex`, `fuzzy`, and `definition_only`. The list is configurable via `grep.modes`, and single-mode setups hide the indicator entirely. `definition_only` uses plain substring matching and returns only matched declarations, regardless of `grep.classify_definitions`. Its status label uses `hl.grep_definition_active` (default: `DiagnosticOk`). -Set `grep.classify_definitions = true` to show matching definition lines first within each page, grouped under their file names. Each page uses one ordinary grep search; classification does not scan ahead for definitions or search the files again for usages. It defaults to `false`, which skips definition classification and grouping. A **Definitions** divider separates the sections when both are on the same page. With a bottom prompt, the order reverses so definitions stay nearest the prompt. Files with both kinds of matches appear in both sections, with each match shown once. - -Each core grep call uses one `classify_definitions` mode. Rust stores it in `GrepSearchOptions` as `DefinitionMode`, defaulting to `Off`. The JavaScript SDKs expose the same setting as `classifyDefinitions`. - -| Mode | Rust variant | C integer | Result | -| --- | --- | --- | --- | -| `off` | `Off` | `0` | All matches, without classification | -| `classify` | `Classify` | `1` | All matches, with definition labels | -| `definition_only` | `DefinitionOnly` | `2` | Only matches on declared names | -| `usage_only` | `UsageOnly` | `3` | Only other matches | - -JavaScript and Neovim also accept the existing booleans: `false` means `off`, and `true` means `classify`. Neovim groups each page of `classify` results into definitions first, then usages. The SDK default is `off`. Use the same mode when continuing with a cursor. The C argument is a `uint8_t`, preserving the byte representation of existing boolean calls (`0` and `1`). Modes `2` and `3` require the updated native library; never send them to a library that still expects `bool`. - -```typescript -finder.grep('AudioMap', { classifyDefinitions: 'definition_only' }); -finder.multiGrep({ patterns: ['AudioMap', 'audio_map'], classifyDefinitions: 'definition_only' }); -``` - -MCP and Pi expose only the optional `definitions_only` boolean to the model. MCP accepts `{ "query": "AudioMap", "definitions_only": true }`; Pi's `ffgrep` uses `pattern` instead of `query`. If the completed search finds no matching declarations, the tool reports that and returns ordinary matches. Cursors preserve the definition search or fallback state, so the model does not need to repeat a mode. The multi-pattern tools accept the same flag. +Set `grep.classify_definitions = true` to show matching definition lines first within each page, grouped under their file names. Each page uses one ordinary grep search; classification does not scan ahead for definitions or search the files again for usages. It defaults to `false`, which skips definition classification and grouping. It also accepts the strings `'classify'`, `'definition_only'`, and `'usage_only'`. A **Definitions** divider separates the sections when both are on the same page. With a bottom prompt, the order reverses so definitions stay nearest the prompt. Files with both kinds of matches appear in both sections, with each match shown once. -SDK definition filtering occurs before page and per-file result limits. Cursors resume at the next file; page size is a soft limit because the current file is returned together. Repeat the same query, mode, and constraints for each page. A timed-out page can be empty with a continuation cursor. In Neovim, move down (top prompt) or up (bottom prompt) to continue an empty page. Context lines can still contain ordinary code. +Definition filtering occurs before page and per-file result limits. A timed-out page can be empty; move down (top prompt) or up (bottom prompt) to continue searching. Context lines can still contain ordinary code. Definition classification accepts source lines up to 256 bytes, including indentation and excluding line endings. Longer lines stay ordinary matches and are excluded from `definition_only`. The limit uses the original source bytes, regardless of display truncation or whitespace trimming. For multiline regex matches, it applies to the first physical line. @@ -672,6 +667,26 @@ finder.value.destroy(); Every method returns a `Result` (`{ ok: true, value } | { ok: false, error }`). Full type reference: [`packages/fff-node/src/fff-api.ts`](./packages/fff-node/src/fff-api.ts). +### Definition modes + +Each core grep call uses one `classify_definitions` mode. Rust stores it in `GrepSearchOptions` as `DefinitionMode`, defaulting to `Off`. The JavaScript SDKs expose the same setting as `classifyDefinitions`. + +| Mode | Rust variant | C integer | Result | +| --- | --- | --- | --- | +| `off` | `Off` | `0` | All matches, without classification | +| `classify` | `Classify` | `1` | All matches, with definition labels | +| `definition_only` | `DefinitionOnly` | `2` | Only matches on declared names | +| `usage_only` | `UsageOnly` | `3` | Only other matches | + +JavaScript also accepts the existing booleans: `false` means `off`, and `true` means `classify`. The SDK default is `off`. Use the same mode when continuing with a cursor. The C argument is a `uint8_t`, preserving the byte representation of existing boolean calls (`0` and `1`). Modes `2` and `3` require the updated native library; never send them to a library that still expects `bool`. + +```typescript +finder.grep('AudioMap', { classifyDefinitions: 'definition_only' }); +finder.multiGrep({ patterns: ['AudioMap', 'audio_map'], classifyDefinitions: 'definition_only' }); +``` + +SDK definition filtering occurs before page and per-file result limits. `definition_only` retains file constraints when no matches exist. `usage_only` keeps the file-path and literal retries of ordinary grep. Cursors resume at the next file; page size is a soft limit because the current file is returned together. Repeat the same query, mode, and constraints for each page. A timed-out page can be empty with a continuation cursor. Context lines can still contain ordinary code. + TypeScript wrapper over the C library for nodejs and bun. Build custom agent tools, CLIs, or IDE integrations on top of FFF. diff --git a/crates/fff-core/src/grep/grep_tests.rs b/crates/fff-core/src/grep/grep_tests.rs index b70bd0f53..ce8539ad5 100644 --- a/crates/fff-core/src/grep/grep_tests.rs +++ b/crates/fff-core/src/grep/grep_tests.rs @@ -147,6 +147,56 @@ fn definition_only_keeps_unknown_language_files_out_of_fallbacks() { } } +#[test] +fn definition_only_keeps_empty_scopes_while_usage_only_retries() { + let dir = tempfile::tempdir().unwrap(); + std::fs::write( + dir.path().join("a.rs"), + "fn Needle() {}\nNeedle();\n*.txt Needle\n", + ) + .unwrap(); + std::fs::write(dir.path().join("b.txt"), "unrelated\n").unwrap(); + let mut picker = FilePicker::new(FilePickerOptions { + base_path: dir.path().to_string_lossy().into_owned(), + enable_content_indexing: false, + enable_mmap_cache: false, + watch: false, + ..Default::default() + }) + .unwrap(); + picker.collect_files().unwrap(); + let parser = fff_query_parser::QueryParser::new(fff_query_parser::AiGrepConfig); + for mode in [GrepMode::PlainText, GrepMode::Regex, GrepMode::Fuzzy] { + let options = GrepSearchOptions { + mode, + classify_definitions: DefinitionMode::DefinitionOnly, + time_budget_ms: 0, + ..Default::default() + }; + for raw in ["missing.rs Needle", "*.txt Needle", "!*.rs Needle"] { + let result = picker.grep(&parser.parse(raw), &options); + assert!(result.matches.is_empty(), "{mode:?}: {raw}"); + assert_eq!(result.next_file_offset, 0); + assert!(!result.literal_fallback); + } + } + + // usage_only is public API: it keeps the path and literal retries of unfiltered grep. + let options = GrepSearchOptions { + classify_definitions: DefinitionMode::UsageOnly, + time_budget_ms: 0, + ..Default::default() + }; + let lines = |raw: &str| { + let result = picker.grep(&parser.parse(raw), &options); + let lines: Vec = result.matches.iter().map(|m| m.line_number).collect(); + (lines, result.literal_fallback) + }; + assert_eq!(lines("missing.rs Needle"), (vec![2, 3], false)); + assert_eq!(lines("*.txt Needle"), (vec![3], true)); + assert_eq!(lines("!*.rs Needle"), (vec![], false)); +} + #[test] fn pagination_keeps_unreturned_files_before_abort() { let file = crate::types::FileItem::new_raw(0, 1, 0, None, false); diff --git a/crates/fff-mcp/src/cursor.rs b/crates/fff-mcp/src/cursor.rs index a0ec9b3d1..ecf5fae54 100644 --- a/crates/fff-mcp/src/cursor.rs +++ b/crates/fff-mcp/src/cursor.rs @@ -10,8 +10,58 @@ const MAX_CURSORS: usize = 20; #[derive(Clone, Copy, Default)] pub(crate) struct GrepCursor { pub file_offset: usize, - pub definitions_only: bool, - pub found_definitions: bool, + pub stage: GrepStage, + pub found_matches: bool, + pub missing_path: bool, +} + +#[derive(Clone, Copy, Default, PartialEq, Eq, Debug)] +pub(crate) enum GrepStage { + #[default] + Normal, + Broadened, + Definitions, + References, + FuzzyDefinitions, + FuzzyMatches, +} + +impl GrepStage { + pub fn next(self) -> Option { + match self { + Self::Definitions => Some(Self::References), + Self::References => Some(Self::FuzzyDefinitions), + Self::FuzzyDefinitions => Some(Self::FuzzyMatches), + _ => None, + } + } + + pub fn is_fuzzy(self) -> bool { + matches!(self, Self::FuzzyDefinitions | Self::FuzzyMatches) + } + + pub fn definition_mode(self) -> fff::DefinitionMode { + match self { + Self::Definitions | Self::FuzzyDefinitions => fff::DefinitionMode::DefinitionOnly, + // A file-path retry can widen these stages, so keep definitions found elsewhere. + Self::References | Self::FuzzyMatches | Self::Normal | Self::Broadened => { + fff::DefinitionMode::Classify + } + } + } + + pub fn notice(self) -> Option<&'static str> { + match self { + Self::References => Some("No matching definitions found. Showing other matches:"), + Self::FuzzyDefinitions => { + Some("No exact matches found. Maybe you meant these definitions:") + } + Self::FuzzyMatches => Some( + "No exact matches or approximate definitions found. Maybe you meant these other matches:", + ), + _ => None, + } + } } /// Stores cursor state for paginated grep results. diff --git a/crates/fff-mcp/src/main.rs b/crates/fff-mcp/src/main.rs index ee9d033a4..0bef0bf2f 100644 --- a/crates/fff-mcp/src/main.rs +++ b/crates/fff-mcp/src/main.rs @@ -51,7 +51,7 @@ pub const MCP_INSTRUCTIONS: &str = concat!( "Grep matches single lines. Search for ONE identifier per query:\n", " + 'InProgressQuote' -> finds definition + all usages\n", " + 'ActorAuth' -> finds enum, struct, all call sites\n", - "Set definitions_only to true to find declarations. If none are found, the tool reports that and returns other matches. Cursors preserve this behavior.\n", + "Set definitions_only to true to find declarations. Grep tries declarations, exact matches, fuzzy declarations, then fuzzy matches; it reports fallbacks. Multi-grep stays literal. Cursors keep the current stage; fallback starts only after a completed search with no matches.\n", " x 'load.*metadata.*InProgressQuote' -> regex spanning multiple tokens, 0 results\n", " x 'ctx.data::' -> code syntax, too specific, 0 results\n", " x 'struct ActorAuth' -> adding keywords narrows results, misses enums/traits/type aliases\n", diff --git a/crates/fff-mcp/src/server.rs b/crates/fff-mcp/src/server.rs index 594d2de26..75c767a9b 100644 --- a/crates/fff-mcp/src/server.rs +++ b/crates/fff-mcp/src/server.rs @@ -1,9 +1,9 @@ -use crate::cursor::{CursorStore, GrepCursor}; +use crate::cursor::{CursorStore, GrepCursor, GrepStage}; use crate::output::{GrepFormatter, OutputMode, file_suffix}; use fff::grep::{GrepMode, GrepSearchOptions, has_regex_metacharacters}; use fff::types::{FileItem, PaginationArgs}; use fff::{FuzzySearchOptions, QueryParser, SharedFilePicker}; -use fff_query_parser::AiGrepConfig; +use fff_query_parser::{AiGrepConfig, Constraint, FFFQuery}; use rmcp::handler::server::wrapper::Parameters; use rmcp::model::*; use rmcp::{ServerHandler, schemars, tool, tool_handler, tool_router}; @@ -15,6 +15,8 @@ const SCAN_READY_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(3 const MAX_CONTEXT_LINES: usize = 100; +const FUZZY_STAGE_BUDGET_MS: u64 = 2_000; + fn normalize_max_results(raw: Option, default: usize) -> usize { match raw { None => default, @@ -116,7 +118,7 @@ pub struct GrepParams { pub output_mode: Option, /// Context lines before/after each match. pub context: Option, - /// Return matching declarations. If none are found, report that and return other matches. + /// Try declarations, exact matches, fuzzy declarations, then fuzzy matches. Report fallbacks. #[serde(default)] pub definitions_only: bool, } @@ -287,16 +289,24 @@ impl FffServer { .as_deref() .and_then(|id| self.cursor_store.lock().ok()?.get_grep(id)) .unwrap_or(GrepCursor { - definitions_only: params.definitions_only, + stage: if params.definitions_only { + GrepStage::Definitions + } else { + GrepStage::Normal + }, ..Default::default() }); - let prefer_definitions = cursor.definitions_only; + let prefer_definitions = !matches!(cursor.stage, GrepStage::Normal | GrepStage::Broadened); + let broadened_query = (cursor.stage == GrepStage::Broadened).then(|| { + query + .split_whitespace() + .skip(1) + .collect::>() + .join(" ") + }); let (mut options, auto_expand) = make_grep_options(output_mode, mode, cursor.file_offset, context); - if cursor.definitions_only { - options.classify_definitions = fff::DefinitionMode::DefinitionOnly; - } let ctx_lines = options.before_context; // Acquire picker lock once for the entire operation. @@ -308,23 +318,82 @@ impl FffServer { .ok_or_else(|| ErrorData::internal_error("File picker not initialized", None))?; let parser = QueryParser::new(AiGrepConfig); - let parsed = parser.parse(query); - let mut result = picker.grep(&parsed, &options); - let definitions_fallback = cursor.definitions_only - && !cursor.found_definitions - && result.matches.is_empty() - && result.next_file_offset == 0; - if definitions_fallback { - cursor = GrepCursor::default(); - options.file_offset = 0; - options.classify_definitions = fff::DefinitionMode::Classify; - result = picker.grep(&parsed, &options); + let mut original = parser.parse(query); + if let Some(shorter) = &broadened_query { + original.fuzzy_query = parser.parse(shorter).fuzzy_query; + original.raw_query = ""; + } + let mode = if broadened_query.is_some() && has_regex_metacharacters(&original.grep_text()) { + GrepMode::Regex + } else { + mode + }; + let fresh = cursor.stage == GrepStage::Definitions && cursor.file_offset == 0; + let missing_path = file_path_constraint(&original) + .filter(|path| cursor.missing_path || (fresh && !indexes_file_path(picker, path))); + if missing_path.is_some() { + original + .constraints + .retain(|c| !matches!(c, Constraint::FilePath(_))); + cursor.missing_path = true; } + let fuzzy_query = prefer_definitions.then(|| cleanup_fuzzy_query(&original.grep_text())); + let mut fuzzy = None; + let result = loop { + options.classify_definitions = cursor.stage.definition_mode(); + options.file_offset = cursor.file_offset; + options.mode = if cursor.stage.is_fuzzy() { + GrepMode::Fuzzy + } else { + mode + }; + // Fuzzy stages read every candidate file, so bound each one on large indexes. + (options.time_budget_ms, options.enforce_time_budget) = if cursor.stage.is_fuzzy() { + (FUZZY_STAGE_BUDGET_MS, true) + } else { + (0, false) + }; + let parsed = if cursor.stage.is_fuzzy() { + fuzzy.get_or_insert_with(|| { + let mut parsed = parser.parse(fuzzy_query.as_deref().unwrap_or(query)); + parsed.constraints = original.constraints.clone(); + parsed + }) + } else { + &original + }; + let mut result = picker.grep(parsed, &options); + if prefer_definitions && result.literal_fallback { + // A literal retry drops exclusions, so treat it as an empty stage. + result.matches.clear(); + result.files.clear(); + result.next_file_offset = 0; + result.literal_fallback = false; + } + if cursor.stage.is_fuzzy() && result.matches.is_empty() { + // A budget spent without matches ends the stage like an empty full search. + result.next_file_offset = 0; + } + if result.matches.is_empty() + && result.next_file_offset == 0 + && !cursor.found_matches + && let Some(next) = cursor.stage.next() + { + cursor.stage = next; + cursor.file_offset = 0; + continue; + } + break result; + }; - if result.matches.is_empty() && result.next_file_offset == 0 && options.file_offset == 0 { + if cursor.stage == GrepStage::Normal + && result.matches.is_empty() + && result.next_file_offset == 0 + && options.file_offset == 0 + { // Auto-retry: try broadening multi-word queries by dropping first non-constraint word let parts: Vec<&str> = query.split_whitespace().collect(); - if parts.len() >= 2 && !prefer_definitions { + if parts.len() >= 2 { let first_word = parts[0]; let is_valid_constraint = first_word.starts_with('!') || first_word.starts_with('*') @@ -341,12 +410,13 @@ impl FffServer { mode }; - let (mut retry_options, _) = - make_grep_options(output_mode, retry_mode, 0, context); - retry_options.classify_definitions = options.classify_definitions; + let mut retry_options = options.clone(); + retry_options.mode = retry_mode; + retry_options.file_offset = 0; let retry_result = picker.grep(&rest_parsed, &retry_options); if !retry_result.matches.is_empty() && retry_result.matches.len() <= 10 { + cursor.stage = GrepStage::Broadened; let mut cs = self.lock_cursors()?; let text = &GrepFormatter { matches: &retry_result.matches, @@ -368,25 +438,18 @@ impl FffServer { "0 matches for '{}'. Auto-broadened to '{}':\n{}", query, rest_query, text ), - definitions_fallback, + false, )); } } } // Fuzzy fallback for typo tolerance - let fuzzy_query = if prefer_definitions { - cleanup_fuzzy_query(&parsed.grep_text()) - } else { - cleanup_fuzzy_query(query) - }; + let fuzzy_query = cleanup_fuzzy_query(query); let (mut fuzzy_options, _) = make_grep_options(output_mode, GrepMode::Fuzzy, 0, Some(0)); fuzzy_options.classify_definitions = options.classify_definitions; - let mut fuzzy_parsed = parser.parse(&fuzzy_query); - if prefer_definitions { - fuzzy_parsed.constraints = parsed.constraints.clone(); - } + let fuzzy_parsed = parser.parse(&fuzzy_query); let fuzzy_result = picker.grep(&fuzzy_parsed, &fuzzy_options); if !fuzzy_result.matches.is_empty() { @@ -405,11 +468,11 @@ impl FffServer { } lines.push(format!(" {}: {}", m.line_number, m.line_content)); } - return Ok(grep_response(lines.join("\n"), definitions_fallback)); + return Ok(grep_response(lines.join("\n"), false)); } // File path fallback: if query looks like a path, suggest the matching file - if query.contains('/') && !prefer_definitions { + if query.contains('/') { let file_parser = QueryParser::default(); let file_query = file_parser.parse(query); let file_opts = FuzzySearchOptions { @@ -435,21 +498,32 @@ impl FffServer { "0 content matches. But there is a relevant file path: {}", top.relative_path(picker) ), - definitions_fallback, + false, )); } } } - return Ok(grep_response("0 matches.".into(), definitions_fallback)); + return Ok(grep_response("0 matches.".into(), false)); } + let path_notice = missing_path + .map(|path| format!("No indexed file matches '{path}'. Searched all files instead.")); if result.matches.is_empty() && result.next_file_offset == 0 { - return Ok(grep_response("0 matches.".into(), definitions_fallback)); + let response = grep_response( + "0 matches.".into(), + prefer_definitions && !cursor.found_matches, + ); + return Ok(with_notice(response, path_notice)); } + let notice = if !cursor.found_matches || cursor.stage.is_fuzzy() { + cursor.stage.notice() + } else { + None + }; cursor.file_offset = result.next_file_offset; - cursor.found_definitions |= cursor.definitions_only && !result.matches.is_empty(); + cursor.found_matches |= !result.matches.is_empty(); let mut cs = self.lock_cursors()?; let text = GrepFormatter { @@ -465,7 +539,14 @@ impl FffServer { } .format(&mut cs); - Ok(grep_response(text, definitions_fallback)) + let text = match (broadened_query, notice) { + (Some(shorter), _) => { + format!("0 matches for '{query}'. Auto-broadened to '{shorter}':\n{text}") + } + (_, Some(notice)) => format!("{notice}\n{text}"), + _ => text, + }; + Ok(with_notice(grep_response(text, false), path_notice)) } } @@ -598,7 +679,7 @@ impl FffServer { /// Prefer plain text over regex. Filter files with constraints. #[tool( name = "grep", - description = "Search file contents. Search for bare identifiers (e.g. 'InProgressQuote', 'ActorAuth'), NOT code syntax or regex. Set definitions_only to true to find declarations; if none are found, return other matches with a notice. Filter files with constraints (e.g. '*.rs query', 'src/ query'). Use filename, directory (ending with /) or glob expressions to prefilter. See server instructions for constraint syntax and core rules.", + description = "Search file contents. Search for bare identifiers (e.g. 'InProgressQuote', 'ActorAuth'), NOT code syntax or regex. Set definitions_only to true to try declarations, exact matches, fuzzy declarations, then fuzzy matches; report fallbacks. Filter files with constraints (e.g. '*.rs query', 'src/ query'). Use filename, directory (ending with /) or glob expressions to prefilter. See server instructions for constraint syntax and core rules.", annotations( read_only_hint = true, destructive_hint = false, @@ -661,7 +742,11 @@ impl FffServer { .as_deref() .and_then(|id| self.cursor_store.lock().ok()?.get_grep(id)) .unwrap_or(GrepCursor { - definitions_only: params.definitions_only, + stage: if params.definitions_only { + GrepStage::Definitions + } else { + GrepStage::Normal + }, ..Default::default() }); @@ -671,7 +756,7 @@ impl FffServer { cursor.file_offset, context, ); - if cursor.definitions_only { + if cursor.stage == GrepStage::Definitions { options.classify_definitions = fff::DefinitionMode::DefinitionOnly; } @@ -689,12 +774,15 @@ impl FffServer { let constraints = parser.parse_constraints(constraint_query); let mut result = picker.multi_grep(&patterns_refs, &constraints, &options); - let definitions_fallback = cursor.definitions_only - && !cursor.found_definitions + let definitions_fallback = cursor.stage == GrepStage::Definitions + && !cursor.found_matches && result.matches.is_empty() && result.next_file_offset == 0; if definitions_fallback { - cursor = GrepCursor::default(); + cursor = GrepCursor { + stage: GrepStage::References, + ..Default::default() + }; options.file_offset = 0; options.classify_definitions = fff::DefinitionMode::Classify; result = picker.multi_grep(&patterns_refs, &constraints, &options); @@ -706,7 +794,7 @@ impl FffServer { } cursor.file_offset = result.next_file_offset; - cursor.found_definitions |= cursor.definitions_only && !result.matches.is_empty(); + cursor.found_matches |= !result.matches.is_empty(); let mut cs = self.lock_cursors()?; let text = GrepFormatter { matches: &result.matches, @@ -752,6 +840,37 @@ fn grep_response(text: String, definitions_fallback: bool) -> CallToolResult { CallToolResult::success(vec![Content::text(text)]) } +fn with_notice(mut response: CallToolResult, notice: Option) -> CallToolResult { + if let Some(notice) = notice + && let Some(RawContent::Text(text)) = response.content.first_mut().map(|c| &mut c.raw) + { + text.text = format!("{notice}\n{}", text.text); + } + response +} + +fn file_path_constraint<'a>(query: &FFFQuery<'a>) -> Option<&'a str> { + query.constraints.iter().find_map(|c| match c { + Constraint::FilePath(path) => Some(*path), + _ => None, + }) +} + +// Case-insensitive like core's file-path constraint, so `readme.md` finds README.md. +fn indexes_file_path(picker: &fff::FilePicker, suffix: &str) -> bool { + let suffix = suffix.replace('\\', "/").to_ascii_lowercase(); + picker.get_files().iter().any(|file| { + let path = file + .relative_path(picker) + .replace('\\', "/") + .to_ascii_lowercase(); + !file.is_deleted() + && path + .strip_suffix(suffix.as_str()) + .is_some_and(|rest| rest.is_empty() || rest.ends_with('/')) + }) +} + #[cfg(test)] mod tests { use super::*; @@ -895,7 +1014,7 @@ mod tests { assert!(text.contains("No matching definitions found"), "{text}"); let id = text.split("cursor: ").nth(1).unwrap().trim(); let cursor = server.cursor_store.lock().unwrap().get_grep(id).unwrap(); - assert!(!cursor.definitions_only); + assert_eq!(cursor.stage, GrepStage::References); assert!(cursor.file_offset > 0); let params = serde_json::from_value(serde_json::json!({ @@ -908,28 +1027,343 @@ mod tests { } #[test] - fn completed_definition_pages_do_not_restart_as_usages() { - let (_dir, server) = - definition_server(&[("a.rs".into(), "Needle();"), ("b.rs".into(), "Needle();")]); - let id = server.cursor_store.lock().unwrap().store_grep(GrepCursor { - file_offset: 1, - definitions_only: true, - found_definitions: true, - }); + fn exact_references_precede_fuzzy_definitions() { + let (_dir, server) = definition_server(&[ + ("definition.rs".into(), "fn Needle() {}"), + ("reference.txt".into(), "Nedle();"), // spellchecker:disable-line + ]); let params = serde_json::from_value(serde_json::json!({ - "query": "Needle", "cursor": id, + "query": "Nedle", "definitions_only": true, // spellchecker:disable-line + })) + .unwrap(); + let text = response_text(server.perform_grep(¶ms, GrepMode::PlainText).unwrap()); + assert!(text.contains("Showing other matches:"), "{text}"); + assert!(text.contains("reference.txt"), "{text}"); + assert!(!text.contains("definition.rs"), "{text}"); + assert!(!text.contains("Maybe you meant"), "{text}"); + } + + #[test] + fn fuzzy_definitions_precede_fuzzy_references_and_keep_scope() { + let (_dir, server) = definition_server(&[ + ("definition.rs".into(), "fn Needle() {}"), + ("reference.txt".into(), "Needle();"), + ]); + for (query, expected, absent, notice) in [ + ( + "Nedle", // spellchecker:disable-line + "definition.rs", + "reference.txt", + "Maybe you meant these definitions:", + ), + ( + "reference.txt Nedle", // spellchecker:disable-line + "reference.txt", + "definition.rs", + "Maybe you meant these other matches:", + ), + ( + "!*.rs Nedle", // spellchecker:disable-line + "reference.txt", + "definition.rs", + "Maybe you meant these other matches:", + ), + ] { + let params = serde_json::from_value(serde_json::json!({ + "query": query, "definitions_only": true, + })) + .unwrap(); + let text = response_text(server.perform_grep(¶ms, GrepMode::PlainText).unwrap()); + assert!(text.contains(notice), "{text}"); + assert!(text.contains(expected), "{text}"); + assert!(!text.contains(absent), "{text}"); + } + } + + #[test] + fn fuzzy_fallback_cursors_keep_their_stage() { + for definitions in [true, false] { + let files: Vec<_> = (0..60) + .map(|i| { + ( + format!("file_{i:02}.rs"), + if definitions { + "fn Needle() {}" + } else { + "Needle();" + }, + ) + }) + .collect(); + let (_dir, server) = definition_server(&files); + let params = serde_json::from_value(serde_json::json!({ + "query": "Nedle", "definitions_only": true, "maxResults": 50, // spellchecker:disable-line + })) + .unwrap(); + let text = response_text(server.perform_grep(¶ms, GrepMode::PlainText).unwrap()); + assert!(text.contains("Maybe you meant"), "{text}"); + let id = text.split("cursor: ").nth(1).unwrap().trim(); + let cursor = server.cursor_store.lock().unwrap().get_grep(id).unwrap(); + assert_eq!( + cursor.stage, + if definitions { + GrepStage::FuzzyDefinitions + } else { + GrepStage::FuzzyMatches + } + ); + assert!(cursor.found_matches); + let params = serde_json::from_value(serde_json::json!({ + "query": "Nedle", "cursor": id, "maxResults": 50, // spellchecker:disable-line + })) + .unwrap(); + let next = response_text(server.perform_grep(¶ms, GrepMode::PlainText).unwrap()); + assert!(next.contains("Needle"), "{next}"); + assert!(next.contains("Maybe you meant"), "{next}"); + assert!(!next.contains("cursor:"), "{next}"); + } + } + + #[test] + fn empty_definition_fallback_reports_no_matches() { + let (_dir, server) = definition_server(&[("a.rs".into(), "fn unrelated() {}")]); + let params = serde_json::from_value(serde_json::json!({ + "query": "Needle", "definitions_only": true, })) .unwrap(); assert_eq!( response_text(server.perform_grep(¶ms, GrepMode::PlainText).unwrap()), - "0 matches." + "No matching definitions or other matches found." ); } + #[test] + fn definition_fallback_keeps_exclusions_out_of_literal_retries() { + let (_dir, server) = definition_server(&[ + ("a.rs".into(), "fn Needle() {}\nNeedle();\n*.txt Needle\n"), + ("b.txt".into(), "unrelated\n"), + ]); + for query in ["*.txt Needle", "!*.rs Needle"] { + let params = serde_json::from_value(serde_json::json!({ + "query": query, "definitions_only": true, + })) + .unwrap(); + assert_eq!( + response_text(server.perform_grep(¶ms, GrepMode::PlainText).unwrap()), + "No matching definitions or other matches found.", + "{query}" + ); + } + } + + #[test] + fn definition_search_reports_and_drops_a_missing_file_path() { + let (_dir, server) = definition_server(&[ + ("src/a.rs".into(), "fn Needle() {}\nNeedle();\n"), + ("b.rs".into(), "unrelated();\n"), + ]); + let notice = "No indexed file matches 'missing.rs'. Searched all files instead."; + let params = serde_json::from_value(serde_json::json!({ + "query": "missing.rs Needle", "definitions_only": true, + })) + .unwrap(); + let text = response_text(server.perform_grep(¶ms, GrepMode::PlainText).unwrap()); + assert!(text.starts_with(notice), "{text}"); + assert!( + text.contains("src/a.rs") && text.contains("fn Needle()"), + "{text}" + ); + + for query in ["a.rs Needle", "src/a.rs Needle"] { + let params = serde_json::from_value(serde_json::json!({ + "query": query, "definitions_only": true, + })) + .unwrap(); + let text = response_text(server.perform_grep(¶ms, GrepMode::PlainText).unwrap()); + assert!(!text.contains("No indexed file"), "{query}: {text}"); + assert!(text.contains("fn Needle()"), "{query}: {text}"); + } + + let id = server.cursor_store.lock().unwrap().store_grep(GrepCursor { + stage: GrepStage::References, + missing_path: true, + ..Default::default() + }); + let params = serde_json::from_value(serde_json::json!({ + "query": "missing.rs Needle", "cursor": id, + })) + .unwrap(); + let text = response_text(server.perform_grep(¶ms, GrepMode::PlainText).unwrap()); + assert!(text.starts_with(notice), "{text}"); + assert!(text.contains("Needle();"), "{text}"); + } + + #[test] + fn completed_pages_do_not_restart_a_fallback_after_returning_matches() { + let (_dir, server) = definition_server(&[ + ("a.rs".into(), "fn Needle() {}"), + ("b.rs".into(), "unrelated();"), + ]); + for stage in [ + GrepStage::Definitions, + GrepStage::References, + GrepStage::FuzzyDefinitions, + GrepStage::FuzzyMatches, + ] { + let id = server.cursor_store.lock().unwrap().store_grep(GrepCursor { + file_offset: 1, + stage, + found_matches: true, + ..Default::default() + }); + let params = serde_json::from_value(serde_json::json!({ + "query": "Needle", "cursor": id, + })) + .unwrap(); + assert_eq!( + response_text(server.perform_grep(¶ms, GrepMode::PlainText).unwrap()), + "0 matches.", + "{stage:?}" + ); + } + } + + #[test] + fn empty_resumed_stages_restart_the_next_stage_at_the_first_file() { + for (stage, query, source, notice) in [ + ( + GrepStage::Definitions, + "Needle", + "Needle();", + "Showing other matches:", + ), + ( + GrepStage::References, + "Nedle", // spellchecker:disable-line + "fn Needle() {}", + "Maybe you meant these definitions:", + ), + ( + GrepStage::FuzzyDefinitions, + "Nedle", // spellchecker:disable-line + "Needle();", + "Maybe you meant these other matches:", + ), + ] { + let (_dir, server) = + definition_server(&[("a.rs".into(), source), ("b.rs".into(), "unrelated();")]); + let id = server.cursor_store.lock().unwrap().store_grep(GrepCursor { + file_offset: 1, + stage, + found_matches: false, + ..Default::default() + }); + let params = + serde_json::from_value(serde_json::json!({"query": query, "cursor": id})).unwrap(); + let text = response_text(server.perform_grep(¶ms, GrepMode::PlainText).unwrap()); + assert!(text.contains(notice), "{stage:?}: {text}"); + assert!(text.contains(source), "{stage:?}: {text}"); + } + } + + #[test] + fn broadened_cursors_resume_the_shortened_query() { + let (_dir, server) = definition_server(&[ + ("a.rs".into(), "before\nuse Needle1;\nafter\n"), + ("b.rs".into(), "before\nuse Needle2;\nafter\n"), + ("excluded.txt".into(), "Needle outside scope"), + ]); + for query in ["unused Needle *.rs", "unused Needle[0-9] *.rs"] { + let id = server.cursor_store.lock().unwrap().store_grep(GrepCursor { + file_offset: 1, + stage: GrepStage::Broadened, + found_matches: true, + ..Default::default() + }); + let params = serde_json::from_value(serde_json::json!({ + "query": query, "cursor": id, "context": 1, + })) + .unwrap(); + let text = response_text(server.perform_grep(¶ms, GrepMode::PlainText).unwrap()); + assert!(text.starts_with("0 matches for 'unused"), "{text}"); + assert!(text.contains("Auto-broadened to"), "{text}"); + assert!(text.contains("use Needle"), "{text}"); + assert!(text.contains("before"), "{text}"); + assert!(text.contains("after"), "{text}"); + assert!(!text.contains("excluded.txt"), "{text}"); + } + } + + #[test] + fn widened_reference_stage_keeps_definitions_found_elsewhere() { + let (_dir, server) = indexed_server(&[ + ("a.rs", "fn other() {}\n"), + ( + "src/b.rs", + "pub struct NeedleWidget {}\nimpl NeedleWidget {}\n", + ), + ]); + let params = serde_json::from_value(serde_json::json!({ + "query": "a.rs NeedleWidget", "definitions_only": true, + })) + .unwrap(); + let text = response_text(server.perform_grep(¶ms, GrepMode::PlainText).unwrap()); + assert!(text.contains("pub struct NeedleWidget {}"), "{text}"); + } + + #[test] + fn file_guesses_match_indexed_paths_case_insensitively() { + let (_dir, server) = definition_server(&[("README.md".into(), "fn Needle() {}\n")]); + let params = serde_json::from_value(serde_json::json!({ + "query": "readme.md Needle", "definitions_only": true, + })) + .unwrap(); + let text = response_text(server.perform_grep(¶ms, GrepMode::PlainText).unwrap()); + assert!(!text.contains("No indexed file matches"), "{text}"); + } + + fn indexed_server(files: &[(&str, &str)]) -> (tempfile::TempDir, FffServer) { + let dir = tempfile::tempdir().unwrap(); + for (name, content) in files { + let path = dir.path().join(name); + std::fs::create_dir_all(path.parent().unwrap()).unwrap(); + std::fs::write(path, content).unwrap(); + } + let shared = SharedFilePicker::default(); + fff::FilePicker::new_with_shared_state( + shared.clone(), + fff::SharedFrecency::default(), + fff::FilePickerOptions { + base_path: dir.path().to_string_lossy().into_owned(), + enable_mmap_cache: false, + enable_content_indexing: true, + watch: false, + ..Default::default() + }, + ) + .unwrap(); + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(10); + while !shared + .read() + .unwrap() + .as_ref() + .is_some_and(|p| !p.is_scan_active() && p.bigram_index().is_some()) + { + assert!( + std::time::Instant::now() < deadline, + "bigram index was not built" + ); + std::thread::sleep(std::time::Duration::from_millis(20)); + } + (dir, FffServer::new(shared)) + } + fn definition_server(files: &[(String, &str)]) -> (tempfile::TempDir, FffServer) { let dir = tempfile::tempdir().unwrap(); for (name, content) in files { - std::fs::write(dir.path().join(name), content).unwrap(); + let path = dir.path().join(name); + std::fs::create_dir_all(path.parent().unwrap()).unwrap(); + std::fs::write(path, content).unwrap(); } let mut picker = fff::FilePicker::new(fff::FilePickerOptions { base_path: dir.path().to_string_lossy().into_owned(), diff --git a/doc/fff.nvim.txt b/doc/fff.nvim.txt index 9354562db..ea2ee93a4 100644 --- a/doc/fff.nvim.txt +++ b/doc/fff.nvim.txt @@ -370,52 +370,9 @@ so definitions stay nearest the prompt. Files with both kinds of matches appear in both sections, with each match shown once. -Each core grep call uses one `classify_definitions` mode. Rust stores it in -`GrepSearchOptions` as `DefinitionMode`, defaulting to `Off`. The JavaScript -SDKs expose the same setting as `classifyDefinitions`. - - ----------------------------------------------------------------------- - Mode Rust variant C integer Result - ----------------- ----------------- ----------------- ----------------- - off Off 0 All matches, - without - classification - - classify Classify 1 All matches, with - definition labels - - definition_only DefinitionOnly 2 Only matches on - declared names - - usage_only UsageOnly 3 Only other - matches - ----------------------------------------------------------------------- -JavaScript and Neovim also accept the existing booleans: `false` means `off`, -and `true` means `classify`. Neovim groups each page of `classify` results into -definitions first, then usages. The SDK default is `off`. Use the same mode -when continuing with a cursor. The C argument is a `uint8_t`, preserving the byte -representation of existing boolean calls (`0` and `1`). Modes `2` and `3` -require the updated native library; never send them to a library that still -expects `bool`. - ->typescript - finder.grep('AudioMap', { classifyDefinitions: 'definition_only' }); - finder.multiGrep({ patterns: ['AudioMap', 'audio_map'], classifyDefinitions: 'definition_only' }); -< - -MCP and Pi expose only the optional `definitions_only` boolean to the model. -MCP accepts `{ "query": "AudioMap", "definitions_only": true }`; Pi’s -`ffgrep` uses `pattern` instead of `query`. If the completed search finds no -matching declarations, the tool reports that and returns ordinary matches. -Cursors preserve the definition search or fallback state, so the model does not -need to repeat a mode. The multi-pattern tools accept the same flag. - -SDK definition filtering occurs before page and per-file result limits. Cursors -resume at the next file; page size is a soft limit because the current file is -returned together. Repeat the same query, mode, and constraints for each page. -A timed-out page can be empty with a continuation cursor. In Neovim, move down -(top prompt) or up (bottom prompt) to continue an empty page. Context lines can -still contain ordinary code. +Definition filtering occurs before page and per-file result limits. A timed-out +page can be empty; move down (top prompt) or up (bottom prompt) to continue +searching. Context lines can still contain ordinary code. Definition classification accepts source lines up to 256 bytes, including indentation and excluding line endings. Longer lines stay ordinary matches and diff --git a/packages/pi-fff/README.md b/packages/pi-fff/README.md index 501c724bf..281c7bc79 100644 --- a/packages/pi-fff/README.md +++ b/packages/pi-fff/README.md @@ -83,11 +83,11 @@ This extension registers FFF-powered tools (`fffind`, `ffgrep`, `fff-multi-grep` Search file contents. Smart case, plain text by default, regex optional. -Set `definitions_only: true` to search for matching declarations. If the completed search finds none, the tool reports that and returns ordinary matches. Cursors remember the search or fallback state. The optional `fff-multi-grep` tool accepts the same flag. Omitting the flag keeps ordinary search behavior. +Set `definitions_only: true` to try matching declarations, ordinary exact matches, fuzzy declarations, then fuzzy ordinary matches. Each stage runs only if the previous stage completed with zero matches. Fallback results carry a notice; fuzzy results say “Maybe you meant”. Cursors keep the current stage and file constraints, including through empty partial pages. Once a stage returns matches, pagination stays in that stage. The optional `fff-multi-grep` tool accepts the same flag but only tries declarations and ordinary literal matches. Omitting the flag keeps ordinary search behavior. Parameters: - `pattern` — search text or regex -- `definitions_only` — prefer declarations; fall back to ordinary matches with a notice when none exist +- `definitions_only` — try declarations, exact matches, fuzzy declarations, then fuzzy matches; report fallbacks - `path` — directory/file constraint (e.g. `src/`, `*.ts`) - `ignoreCase` — force case-insensitive - `literal` — treat as literal string (default: true) @@ -110,7 +110,7 @@ OR-logic multi-pattern content search. SIMD-accelerated Aho-Corasick. Parameters: - `patterns` — array of literal patterns (OR logic) -- `definitions_only` — prefer declarations with the same fallback behavior +- `definitions_only` — prefer declarations; fall back to ordinary literal matches with a notice - `constraints` — file constraints (e.g. `*.{ts,tsx} !test/`) - `context` — context lines - `limit` — max matches (default: 100) diff --git a/packages/pi-fff/src/index.ts b/packages/pi-fff/src/index.ts index a50406bac..40c442183 100644 --- a/packages/pi-fff/src/index.ts +++ b/packages/pi-fff/src/index.ts @@ -53,6 +53,7 @@ const MENTION_MAX_RESULTS = 20; // If we exceed 10 seconds for indexed grep - something is definitely off const GREP_TIME_BUDGET_MS = 10_000; +const FUZZY_STAGE_BUDGET_MS = 2_000; const HOME_SCAN_STATUS_KEY = "fff"; const HOME_SCAN_POLL_MS = 1_000; @@ -89,11 +90,18 @@ function toolNameList(names: ToolNames): string[] { // Cursor store — simple bounded Map for pagination cursors // --------------------------------------------------------------------------- +type GrepStage = + | "normal" + | "definitions" + | "references" + | "fuzzy_definitions" + | "fuzzy_matches"; + interface ToolGrepCursor { cursor: GrepCursor; finder: FileFinderApi; - definitionsOnly: boolean; - foundDefinitions: boolean; + stage: GrepStage; + foundMatches: boolean; mode?: GrepMode; query?: string; } @@ -1013,7 +1021,7 @@ export default function fffExtension(pi: ExtensionAPI) { promptSnippet: "Grep contents", promptGuidelines: (names) => [ `${names.grep}: prefer bare identifiers as patterns. Literal queries are most efficient.`, - `${names.grep}: use definitions_only: true to find declarations; if none are found, the tool returns other matches with a notice.`, + `${names.grep}: use definitions_only: true to try declarations, exact matches, fuzzy declarations, then fuzzy matches. Each fallback starts only after a completed search with no matches and includes a notice.`, `${names.grep}: use path for include ('src/', '*.ts') and exclude for noise ('test/,*.min.js').`, `${names.grep}: caseSensitive: true when you need exact case (smart-case otherwise).`, `${names.grep}: after 1-2 greps, read the top match instead of more greps.`, @@ -1083,41 +1091,43 @@ export default function fffExtension(pi: ExtensionAPI) { // (case-insensitive when pattern is all lowercase). const smartCase = params.caseSensitive !== true; - let definitionsOnly = resumed?.definitionsOnly ?? params.definitions_only ?? false; - const preferDefinitions = definitionsOnly; - const foundDefinitions = resumed?.foundDefinitions ?? false; + let stage = resumed?.stage ?? (params.definitions_only ? "definitions" : "normal"); + const preferDefinitions = stage !== "normal"; + const foundMatches = resumed?.foundMatches ?? false; const options = { - mode, smartCase, maxMatchesPerFile: GREP_MAX_MATCHES_PER_FILE, pageSize, - cursor: resumed?.cursor ?? null, beforeContext: context, afterContext: context, timeBudgetMs: GREP_TIME_BUDGET_MS, }; - const grepResult = picker.grep(query, { - ...options, - classifyDefinitions: definitionsOnly ? "definition_only" : "classify", - }); - - if (!grepResult.ok) throw new Error(grepResult.error); - - let result = grepResult.value; - const definitionsFallback = - definitionsOnly && - !foundDefinitions && - result.items.length === 0 && - !result.nextCursor; - if (definitionsFallback) { - const fallback = picker.grep(query, { + let result: GrepResult; + let nativeCursor = resumed?.cursor ?? null; + for (;;) { + if (signal?.aborted) throw new Error("Operation aborted"); + if (isFuzzyGrepStage(stage)) mode = "fuzzy"; + const page = picker.grep(query, { ...options, - cursor: null, - classifyDefinitions: "classify", + mode, + cursor: nativeCursor, + classifyDefinitions: + stage === "definitions" || stage === "fuzzy_definitions" + ? "definition_only" + : "classify", + // Fuzzy stages read every candidate file, so bound each one on large indexes. + ...(isFuzzyGrepStage(stage) + ? { timeBudgetMs: FUZZY_STAGE_BUDGET_MS, enforceTimeBudget: true } + : {}), }); - if (!fallback.ok) throw new Error(fallback.error); - result = fallback.value; - definitionsOnly = false; + if (!page.ok) throw new Error(page.error); + result = page.value; + if (isFuzzyGrepStage(stage) && result.items.length === 0) + result.nextCursor = null; + const next = nextGrepStage(stage); + if (result.items.length > 0 || result.nextCursor || foundMatches || !next) break; + stage = next; + nativeCursor = null; } let fuzzyNotice: string | null = null; @@ -1126,7 +1136,8 @@ export default function fffExtension(pi: ExtensionAPI) { if ( result.items.length === 0 && !result.nextCursor && - (!params.cursor || definitionsFallback) && + !params.cursor && + !preferDefinitions && mode !== "regex" ) { // When the caller pinned a specific file (path has an extension), the @@ -1136,7 +1147,7 @@ export default function fffExtension(pi: ExtensionAPI) { // excluded / out-of-scope directories. const lastSeg = params.path?.split(/[\\/]/).pop() ?? ""; const pathTargetsFile = /\.[a-zA-Z][a-zA-Z0-9]{0,9}$/.test(lastSeg); - const fuzzyQuery = pathTargetsFile && !preferDefinitions ? pattern : query; + const fuzzyQuery = pathTargetsFile ? pattern : query; const fuzzy = picker.grep(fuzzyQuery, { mode: "fuzzy", smartCase, @@ -1145,7 +1156,7 @@ export default function fffExtension(pi: ExtensionAPI) { cursor: null, beforeContext: 0, afterContext: 0, - classifyDefinitions: definitionsOnly ? "definition_only" : "classify", + classifyDefinitions: "classify", timeBudgetMs: GREP_TIME_BUDGET_MS, }); @@ -1166,9 +1177,8 @@ export default function fffExtension(pi: ExtensionAPI) { const cursor = storeCursor({ cursor: result.nextCursor, finder: picker, - definitionsOnly, - foundDefinitions: - foundDefinitions || (definitionsOnly && result.items.length > 0), + stage, + foundMatches: foundMatches || result.items.length > 0, mode, query, }); @@ -1177,12 +1187,12 @@ export default function fffExtension(pi: ExtensionAPI) { if (notices.length > 0) output += `\n\n[${notices.join(". ")}]`; if (fuzzyNotice) output = `[${fuzzyNotice}]\n${output}`; - if (definitionsFallback) { + if (preferDefinitions && (!foundMatches || isFuzzyGrepStage(stage))) { const notice = - result.items.length > 0 || result.nextCursor - ? "No matching definitions found. Showing other matches:" - : "No matching definitions or other matches found."; - output = `[${notice}]\n${output}`; + result.items.length === 0 && !result.nextCursor && !foundMatches + ? "No matching definitions or other matches found." + : grepStageNotice(stage); + if (notice) output = `[${notice}]\n${output}`; } return { @@ -1411,9 +1421,9 @@ export default function fffExtension(pi: ExtensionAPI) { const pageSize = Math.min(effectiveLimit, GREP_PAGE_SIZE_MAX); const context = clampContext(params.context); - let definitionsOnly = - resumed?.definitionsOnly ?? params.definitions_only ?? false; - const foundDefinitions = resumed?.foundDefinitions ?? false; + let stage = + resumed?.stage ?? (params.definitions_only ? "definitions" : "normal"); + const foundMatches = resumed?.foundMatches ?? false; const options = { patterns: params.patterns, constraints: params.constraints, @@ -1426,15 +1436,15 @@ export default function fffExtension(pi: ExtensionAPI) { }; const grepResult = f.multiGrep({ ...options, - classifyDefinitions: definitionsOnly ? "definition_only" : "off", + classifyDefinitions: stage === "definitions" ? "definition_only" : "off", }); if (!grepResult.ok) throw new Error(grepResult.error); let result = grepResult.value; const definitionsFallback = - definitionsOnly && - !foundDefinitions && + stage === "definitions" && + !foundMatches && result.items.length === 0 && !result.nextCursor; if (definitionsFallback) { @@ -1445,7 +1455,7 @@ export default function fffExtension(pi: ExtensionAPI) { }); if (!fallback.ok) throw new Error(fallback.error); result = fallback.value; - definitionsOnly = false; + stage = "references"; } let output = formatGrepOutput(result); @@ -1456,9 +1466,8 @@ export default function fffExtension(pi: ExtensionAPI) { const cursor = storeCursor({ cursor: result.nextCursor, finder: f, - definitionsOnly, - foundDefinitions: - foundDefinitions || (definitionsOnly && result.items.length > 0), + stage, + foundMatches: foundMatches || result.items.length > 0, }); notices.push(`More available. cursor="${cursor}" to continue`); } @@ -1623,3 +1632,33 @@ function sessionModes(entries: unknown): FffMode[] { } return modes; } + +function nextGrepStage(stage: GrepStage): GrepStage | undefined { + switch (stage) { + case "definitions": + return "references"; + case "references": + return "fuzzy_definitions"; + case "fuzzy_definitions": + return "fuzzy_matches"; + default: + return undefined; + } +} + +function isFuzzyGrepStage(stage: GrepStage): boolean { + return stage === "fuzzy_definitions" || stage === "fuzzy_matches"; +} + +function grepStageNotice(stage: GrepStage): string | undefined { + switch (stage) { + case "references": + return "No matching definitions found. Showing other matches:"; + case "fuzzy_definitions": + return "No exact matches found. Maybe you meant these definitions:"; + case "fuzzy_matches": + return "No exact matches or approximate definitions found. Maybe you meant these other matches:"; + default: + return undefined; + } +} diff --git a/packages/pi-fff/test/extension.test.ts b/packages/pi-fff/test/extension.test.ts index cf0d835f4..429987c95 100644 --- a/packages/pi-fff/test/extension.test.ts +++ b/packages/pi-fff/test/extension.test.ts @@ -1007,6 +1007,7 @@ describe("ffgrep per-file cap (#825)", () => { for (const name of ["ffgrep", "fff-multi-grep"]) { describe(`${name} definitions_only`, () => { const normal = name === "ffgrep" ? "classify" : "off"; + const fallbackMode = name === "ffgrep" ? "classify" : "off"; const params = name === "ffgrep" ? { pattern: "Needle", path: "src/" } @@ -1072,8 +1073,8 @@ for (const name of ["ffgrep", "fff-multi-grep"]) { ); expect(calls.map((call) => call.classifyDefinitions)).toEqual([ "definition_only", - normal, - normal, + fallbackMode, + fallbackMode, ]); expect(calls[2].cursor).toEqual(next); expect(second.content[0].text).not.toContain("No matching definitions"); @@ -1103,7 +1104,7 @@ for (const name of ["ffgrep", "fff-multi-grep"]) { expect(calls.map((call) => call.classifyDefinitions)).toEqual([ "definition_only", "definition_only", - normal, + fallbackMode, ]); expect(calls[2].cursor).toBeNull(); expect(second.content[0].text).toContain("No matching definitions found"); @@ -1151,34 +1152,171 @@ for (const name of ["ffgrep", "fff-multi-grep"]) { }); } -test("definition fallback keeps fuzzy query and mode in the cursor", async () => { - const calls: { query: string; options: any }[] = []; - grepImpl = (query, options) => { - calls.push({ query, options }); - if (calls.length <= 2) return grepPage(); - return grepPage("Needle();", calls.length === 3 ? { _offset: 9 } : null); - }; - const setup = await start("tools-and-ui"); - const tool = setup.pi.registerTool.mock.calls - .map(([tool]) => tool) - .find((tool) => tool.name === "ffgrep"); - const params = { - pattern: "Nedle", // spellchecker:disable-line - path: "src/", - definitions_only: true, - }; - const first = await tool.execute("typo", params, abortOptions().signal); - const cursor = first.content[0].text.match(/cursor="([^"]+)"/)[1]; - await tool.execute("next", { ...params, cursor }, abortOptions().signal); - expect(calls.map((call) => call.options.mode)).toEqual([ - "plain", - "plain", - "fuzzy", - "fuzzy", - ]); - expect(calls[3].query).toBe(calls[2].query); - expect(calls[3].options.classifyDefinitions).toBe("classify"); - await shutdown(setup); +for (const fuzzyDefinitions of [true, false]) { + test(`definition fallback preserves fuzzy ${fuzzyDefinitions ? "definitions" : "matches"} across pages`, async () => { + const calls: { query: string; options: any }[] = []; + const firstHit = fuzzyDefinitions ? 3 : 4; + grepImpl = (query, options) => { + calls.push({ query, options }); + if (calls.length < firstHit) return grepPage(); + return grepPage( + fuzzyDefinitions ? "fn Needle() {}" : "Needle();", + calls.length === firstHit ? { _offset: 9 } : null, + fuzzyDefinitions, + ); + }; + const setup = await start("tools-and-ui"); + const tool = setup.pi.registerTool.mock.calls + .map(([tool]) => tool) + .find((tool) => tool.name === "ffgrep"); + const params = { + pattern: "Nedle", // spellchecker:disable-line + path: "src/", + exclude: "test/", + definitions_only: true, + }; + const first = await tool.execute("typo", params, abortOptions().signal); + expect(first.content[0].text).toContain( + fuzzyDefinitions + ? "Maybe you meant these definitions:" + : "Maybe you meant these other matches:", + ); + const cursor = first.content[0].text.match(/cursor="([^"]+)"/)[1]; + const second = await tool.execute( + "next", + { pattern: params.pattern, cursor }, + abortOptions().signal, + ); + expect(calls.map((call) => call.options.mode)).toEqual([ + "plain", + "plain", + ...Array(firstHit - 1).fill("fuzzy"), + ]); + expect(calls.map((call) => call.options.classifyDefinitions)).toEqual([ + "definition_only", + "classify", + "definition_only", + ...(fuzzyDefinitions ? ["definition_only"] : ["classify", "classify"]), + ]); + expect(new Set(calls.map((call) => call.query))).toEqual( + new Set(["src/ !test/ " + params.pattern]), + ); + expect(calls[firstHit].options.cursor).toEqual({ _offset: 9 }); + expect(second.content[0].text).toContain("Maybe you meant"); + await shutdown(setup); + }); +} + +for (const stage of [3, 4]) { + test(`definition fallback moves past a fuzzy stage that spent its budget at stage ${stage}`, async () => { + const calls: any[] = []; + grepImpl = (_query, options) => { + calls.push(options); + if (calls.length === stage) return grepPage(undefined, { _offset: 7 }); + return grepPage(); + }; + const setup = await start("tools-and-ui"); + const tool = setup.pi.registerTool.mock.calls + .map(([tool]) => tool) + .find((tool) => tool.name === "ffgrep"); + const result = await tool.execute( + "budget", + { pattern: "Needle", definitions_only: true }, + abortOptions().signal, + ); + expect(calls).toHaveLength(4); + expect(calls[stage - 1]).toMatchObject({ mode: "fuzzy", enforceTimeBudget: true }); + expect(result.content[0].text).toContain( + "No matching definitions or other matches found.", + ); + expect(result.content[0].text).not.toContain("cursor="); + await shutdown(setup); + }); +} + +for (const stage of [1, 2, 3, 4]) { + if (stage <= 2) + test(`definition fallback waits for an empty partial page at stage ${stage}`, async () => { + const calls: any[] = []; + grepImpl = (_query, options) => { + calls.push(options); + if (calls.length === stage) return grepPage(undefined, { _offset: 7 }); + return grepPage(); + }; + const setup = await start("tools-and-ui"); + const tool = setup.pi.registerTool.mock.calls + .map(([tool]) => tool) + .find((tool) => tool.name === "ffgrep"); + const params = { pattern: "Needle", definitions_only: true }; + const first = await tool.execute("partial", params, abortOptions().signal); + expect(calls).toHaveLength(stage); + const cursor = first.content[0].text.match(/cursor="([^"]+)"/)[1]; + const last = await tool.execute( + "finish", + { pattern: params.pattern, cursor }, + abortOptions().signal, + ); + expect(calls).toHaveLength(5); + expect(calls[stage].cursor).toEqual({ _offset: 7 }); + expect(calls[stage].mode).toBe(calls[stage - 1].mode); + expect(calls[stage].classifyDefinitions).toBe(calls[stage - 1].classifyDefinitions); + for (const call of calls.slice(stage + 1)) expect(call.cursor).toBeNull(); + expect(last.content[0].text).toContain( + "No matching definitions or other matches found.", + ); + await shutdown(setup); + }); + + test(`definition fallback stops after matches were returned at stage ${stage}`, async () => { + const calls: any[] = []; + grepImpl = (_query, options) => { + calls.push(options); + return calls.length === stage ? grepPage("Needle", { _offset: 7 }) : grepPage(); + }; + const setup = await start("tools-and-ui"); + const tool = setup.pi.registerTool.mock.calls + .map(([tool]) => tool) + .find((tool) => tool.name === "ffgrep"); + const first = await tool.execute( + "found", + { pattern: "Needle", definitions_only: true }, + abortOptions().signal, + ); + expect(calls).toHaveLength(stage); + const cursor = first.content[0].text.match(/cursor="([^"]+)"/)[1]; + await tool.execute("finish", { pattern: "Needle", cursor }, abortOptions().signal); + expect(calls).toHaveLength(stage + 1); + expect(calls[stage].cursor).toEqual({ _offset: 7 }); + await shutdown(setup); + }); +} + +test("definition fallback stops on cancellation or a native error", async () => { + for (const cancel of [false, true]) { + let calls = 0; + const controller = new AbortController(); + grepImpl = () => { + calls++; + if (cancel) { + controller.abort(); + return grepPage(); + } + return { ok: false, error: "Search failed" }; + }; + const setup = await start("tools-and-ui"); + const tool = setup.pi.registerTool.mock.calls + .map(([tool]) => tool) + .find((tool) => tool.name === "ffgrep"); + await expect( + tool.execute( + "failed", + { pattern: "Needle", definitions_only: true }, + controller.signal, + ), + ).rejects.toThrow(cancel ? "Operation aborted" : "Search failed"); + expect(calls).toBe(1); + await shutdown(setup); + } }); test("grep cursors keep the original finder when path changes or is omitted", async () => {