Read the user's own instrument manuals (#120)

README.md

@@ -44,6 +44,7 @@ tools the model can call to discover instruments and exchange SCPI / IEEE-488.2
   - [Manual test from a terminal](#manual-test-from-a-terminal)
 - [Logging](#logging)
 - [MCP transports (stdio & HTTP)](#mcp-transports-stdio--http)
+- [Your own manual library](#your-own-manual-library)
 - [Protocol revisions](#protocol-revisions)
 - [Long-running calls: progress and tasks](#long-running-calls-progress-and-tasks)
 - [Structured results](#structured-results)
@@ -74,6 +75,9 @@ tools the model can call to discover instruments and exchange SCPI / IEEE-488.2
 - **SRQ-based operation completion** — wait for an operation to *truly* finish via the bus
   service-request event (data-driven from the model's `statusModel`), instead of guessing with
   a fixed timeout.
+- **Reads your own manuals** — point `GPIB_MCP_MANUALS` at a folder of instrument manuals and the server
+  can search them when the command database falls short, returning the passage with the file and page to
+  cite (PDFs via `pdftotext` or a text sidecar).
 - **Measurements come back as data, not prose** — the query, sweep and setting tools declare an
   `outputSchema` and return `structuredContent`, so a reading arrives as a number and a unit (the unit
   taken from the database's audited tokens, never guessed off the wire).
@@ -916,6 +920,50 @@ so the single-threaded instrument access is preserved regardless of transport.
 $env:GPIB_MCP_TRANSPORT = "http"; $env:GPIB_MCP_HTTP_TOKEN = "<secret>"; .\GpibMcp.exe
 ```
 
+## Your own manual library
+
+Point the server at a folder of instrument manuals and it can read them when the command database falls
+short (issue #120):
+
+```powershell
+$env:GPIB_MCP_MANUALS = "C:\Users\me\Documents\Manuals"
+```
+
+That registers one tool, **`manual_search`**, which returns the matching **passages with the file and page
+they came from** — not an answer. Deriving "the command is `CF`" from a page of prose is the model's job,
+done in front of you, with the quoted text visible. A server that synthesised commands out of manuals would
+be guessing with far more confidence than the evidence supports, and the thing on the other end of a wrong
+guess is your hardware.
+
+Resolution order is unchanged: **`instrument_reference` first** (instant, structured, already carries the
+audited unit tokens), then the manuals, then whatever the client can do on its own. When a manual yields a
+command the database lacks, the result says to offer `instrument_db_save` — so the catalogue grows from your
+own documents, and the next lookup is instant.
+
+| Variable | Purpose |
+|---|---|
+| `GPIB_MCP_MANUALS` | folder of manuals (searched recursively). Unset = the tool isn't registered at all |
+| `GPIB_MCP_PDFTOTEXT` | full path to `pdftotext`, if it isn't on `PATH` |
+| `GPIB_MCP_MANUAL_CACHE` | where extracted text is cached (default `%LOCALAPPDATA%\GpibMcp\manual-text`) |
+
+**Reading PDFs.** .NET Framework can't, and bundling a PDF engine into a server whose whole shape is "no
+external dependencies" is a poor trade for a feature that's off by default. So there are three routes, tried
+in order: the file is already `.txt`/`.md`; a sidecar `<name>.txt` sits beside the PDF; or **`pdftotext`**
+(Poppler/xpdf) is on `PATH`, run with `-layout` so command tables keep their columns. If none applies, the
+result says *which* file couldn't be read and how to fix it — a manual that can't be extracted must never
+look like a manual with no match. Extracted text is cached, keyed by path, size and modification time.
+
+**How it finds the right manual.** By filename first — a library is hundreds of large PDFs and extracting
+them all would take minutes and return noise. Pass `model=` and it reads only that instrument's manuals.
+Two behaviours worth knowing, both found by running it against a real 570-PDF library:
+
+- **Series manuals count.** An 8563E's programming manual is often filed as `8560E Programming Guide.pdf`. A
+  human would reach for it, so the search does too — at a much lower rank, and the result is flagged
+  `familyMatchOnly` so the substitution gets stated rather than hidden.
+- **If nothing is named for the model, nothing is searched.** It reports that, and lists the closest names it
+  does have. Reading whichever files happened to be smallest would produce "searched 12 files, no match",
+  which reads as *your library doesn't have this* when the truth is *I never opened the right file*.
+
 ## Protocol revisions
 
 The server implements MCP **2026-07-28** and speaks **`2026-07-28`, `2025-06-18`, `2025-03-26` and
@@ -1102,6 +1150,10 @@ src/GpibMcp.Core/                  backend-neutral core (no driver dependency; b
       ServerTask.cs                one task's state + its CreateTaskResult / tasks/get shapes
       TaskStore.cs                 the live tasks, TTL-bounded
       TaskRunner.cs                single worker thread - keeps the GPIB bus serial
+  Manuals/                         the user's own manual folder (#120)
+    ManualLibrary.cs               which files in it match a question (filename-first)
+    ManualText.cs                  text extraction (sidecar / pdftotext) + on-disk cache
+    ManualSearch.cs                passage search with page-numbered citations
   Instruments/
     IInstrumentManager.cs          tool-facing instrument abstraction (enables testing)
     InstrumentManager.cs           backend-neutral manager (history, errors, capture, the bus lock)
@@ -1119,6 +1171,7 @@ src/GpibMcp.Core/                  backend-neutral core (no driver dependency; b
   Tools/
     ToolArgs.cs                    shared JSON-Schema + argument helpers
     MeasurementValue.cs            reply -> number, and its unit from the audited DB tokens (#113)
+    ManualTools.cs                 manual_search over the user's own manual folder (#120)
     InstrumentIo.cs                resolves a model's IoSpec (terminators + bounded read)
     InstrumentTools.cs             VISA / native-GPIB + serial-poll / wait-SRQ tools
     DatabaseTools.cs               command-database + assignment + set_termination tools

src/GpibMcp.Core/Instruments/InstrumentPaths.cs

@@ -119,7 +119,8 @@ public static void EnsureUserDatabaseSeeded(string exeDir)
             }
         }
 
-        private static string AppDataDir() =>
+        /// <summary>The server's per-user data folder: %LOCALAPPDATA%\GpibMcp.</summary>
+        public static string AppDataDir() =>
             Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData), "GpibMcp");
     }
 }

src/GpibMcp.Core/Manuals/ManualLibrary.cs

@@ -0,0 +1,274 @@
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+
+namespace GpibMcp.Manuals
+{
+    /// <summary>
+    /// A user's folder of instrument manuals (#120), and the rules for finding which files in it are worth
+    /// reading for a given question.
+    ///
+    /// Narrowing by filename first is not an optimisation detail, it is the design. A real library is
+    /// hundreds of large PDFs - extracting all of them to answer one question would take minutes and produce
+    /// mostly noise. Manuals are almost always named by model ("8560E Programming Guide.pdf", or a folder
+    /// "3458A" holding its set), so the model name is a strong, cheap filter that gets to the right handful
+    /// before any file is opened.
+    ///
+    /// The feature is off unless a folder is configured: no folder, no tool.
+    /// </summary>
+    public sealed class ManualLibrary
+    {
+        /// <summary>Extensions worth reading. Text needs no extraction; PDF needs <see cref="ManualText"/>.</summary>
+        private static readonly string[] ReadableExtensions = { ".pdf", ".txt", ".md", ".text" };
+
+        /// <summary>
+        /// Ceiling on files opened for one search when the caller gave no model to narrow by. Bounded so a
+        /// vague question cannot walk a 5 GB library; the search reports when it hits this, because a
+        /// silently truncated search reads as "not in the manuals" when it means "we stopped looking".
+        /// </summary>
+        public const int MaxFilesPerSearch = 12;
+
+        private readonly string _root;
+
+        private ManualLibrary(string root) { _root = root; }
+
+        /// <summary>The configured folder, or null when the feature is off.</summary>
+        public string Root => _root;
+
+        /// <summary>
+        /// Opens the library named by <c>GPIB_MCP_MANUALS</c>, or returns null when it is unset or points
+        /// nowhere. A configured-but-missing folder is worth a warning rather than silence - it is almost
+        /// always a typo, and the alternative is a tool that quietly never finds anything.
+        /// </summary>
+        public static ManualLibrary FromEnvironment()
+        {
+            string configured = Environment.GetEnvironmentVariable("GPIB_MCP_MANUALS");
+            if (string.IsNullOrWhiteSpace(configured)) return null;
+
+            string root = configured.Trim().Trim('"');
+            if (!Directory.Exists(root))
+            {
+                Diagnostics.Log.Warn("GPIB_MCP_MANUALS points at '" + root + "', which does not exist; " +
+                                     "manual lookup is disabled.");
+                return null;
+            }
+            return new ManualLibrary(root);
+        }
+
+        /// <summary>Opens a library at an explicit path (tests, and callers that configure it directly).</summary>
+        public static ManualLibrary At(string root) =>
+            string.IsNullOrWhiteSpace(root) || !Directory.Exists(root) ? null : new ManualLibrary(root.Trim());
+
+        /// <summary>
+        /// The files worth reading for this question, best candidates first. When a model is given, files
+        /// whose name or folder mentions it come first and nothing else is opened unless there are none;
+        /// otherwise the query's own words are matched against filenames.
+        /// </summary>
+        public IReadOnlyList<ManualFile> Candidates(string model, string query, int limit = MaxFilesPerSearch)
+        {
+            bool haveModel = !string.IsNullOrWhiteSpace(model);
+
+            return Enumerate()
+                .Select(f => new { File = f, Score = NameScore(f, model, query) })
+                // With a model in hand, a file must be related to THAT instrument. Otherwise one shared word
+                // in a filename - "frequency" appears in half a library - drags in application notes for
+                // other instruments, which cost seconds to extract and answer a different question.
+                .Where(x => x.Score > 0 && (!haveModel || ModelScore(x.File, model) > 0))
+                .OrderByDescending(x => x.Score)
+                .ThenBy(x => x.File.SizeBytes)      // a short manual is likelier to be the relevant one
+                .Select(x => x.File)
+                .Take(Math.Max(1, limit))
+                .ToList();
+
+            // Deliberately no fallback to "read some files anyway". Reading arbitrary manuals to answer a
+            // question about an instrument they do not cover costs seconds and returns noise - and worse,
+            // it reports "searched 12 files, no match", which reads as "your library does not have this"
+            // when the truth is "I never looked at the right file". Better to say nothing matched and show
+            // what is there.
+        }
+
+        /// <summary>A sample of what the library holds, so a caller told "nothing matched" can retry usefully.</summary>
+        public IReadOnlyList<string> SampleNames(string model, int limit = 20)
+        {
+            IEnumerable<ManualFile> files = Enumerate();
+
+            // With a model in hand, prefer names that at least share its leading digits - "no 8563E manual,
+            // but here are the 856x ones" is a far better prompt than an alphabetical slice of the library.
+            string stem = FamilyStem(model);
+            if (stem != null)
+            {
+                var near = files.Where(f => f.RelativePath.ToLowerInvariant().Contains(stem)).ToList();
+                if (near.Count > 0) files = near;
+            }
+
+            return files.Select(f => f.RelativePath).OrderBy(n => n, StringComparer.OrdinalIgnoreCase)
+                        .Take(Math.Max(1, limit)).ToList();
+        }
+
+        /// <summary>Total readable files, for telling the caller the size of what was not searched.</summary>
+        public int Count() => Enumerate().Count();
+
+        /// <summary>Every readable file in the library.</summary>
+        public IEnumerable<ManualFile> Enumerate()
+        {
+            IEnumerable<string> paths;
+            try { paths = Directory.EnumerateFiles(_root, "*", SearchOption.AllDirectories); }
+            catch (Exception ex)
+            {
+                Diagnostics.Log.Warn("Could not read the manual library at '" + _root + "': " + ex.Message);
+                yield break;
+            }
+
+            foreach (string path in paths)
+            {
+                string ext = Path.GetExtension(path);
+                if (Array.IndexOf(ReadableExtensions, ext.ToLowerInvariant()) < 0) continue;
+
+                ManualFile file = null;
+                try { file = new ManualFile(path, _root); }
+                catch (Exception) { /* vanished or unreadable between enumerate and stat */ }
+
+                // A zero-byte file is a failed download, not a manual.
+                if (file != null && file.SizeBytes > 0) yield return file;
+            }
+        }
+
+        /// <summary>
+        /// Resolves a caller-supplied path against the library, refusing anything outside it. The path comes
+        /// from a tool argument, so "../../../secrets.txt" has to bounce off something.
+        /// </summary>
+        public ManualFile Resolve(string relativeOrFullPath)
+        {
+            if (string.IsNullOrWhiteSpace(relativeOrFullPath)) return null;
+
+            string candidate = relativeOrFullPath.Trim().Trim('"');
+            string full;
+            try
+            {
+                full = Path.GetFullPath(Path.IsPathRooted(candidate)
+                    ? candidate
+                    : Path.Combine(_root, candidate));
+            }
+            catch (Exception) { return null; }
+
+            string rootFull = Path.GetFullPath(_root).TrimEnd(Path.DirectorySeparatorChar) +
+                              Path.DirectorySeparatorChar;
+            if (!full.StartsWith(rootFull, StringComparison.OrdinalIgnoreCase)) return null;
+            if (!File.Exists(full)) return null;
+
+            return new ManualFile(full, _root);
+        }
+
+        /// <summary>
+        /// How well a file's name matches the question. The model is worth far more than a query word: a
+        /// filename containing "8563E" is almost certainly that instrument's manual, whereas one containing
+        /// "frequency" is barely evidence at all.
+        /// </summary>
+        private static int NameScore(ManualFile file, string model, string query)
+        {
+            string haystack = file.RelativePath.Replace('\\', ' ').Replace('/', ' ').ToLowerInvariant();
+            int score = ModelScore(file, model);
+
+            foreach (string word in Words(query))
+                if (word.Length >= 3 && haystack.Contains(word)) score += 5;
+
+            return score;
+        }
+
+        /// <summary>
+        /// How strongly a file's name ties it to <paramref name="model"/>: named for it, named for it without
+        /// a trailing option letter, or merely of its family - in descending order of confidence, and zero
+        /// for no relation at all.
+        /// </summary>
+        private static int ModelScore(ManualFile file, string model)
+        {
+            if (string.IsNullOrWhiteSpace(model)) return 0;
+
+            string haystack = file.RelativePath.Replace('\\', ' ').Replace('/', ' ').ToLowerInvariant();
+            string m = model.Trim().ToLowerInvariant();
+
+            if (haystack.Contains(m)) return 100;
+
+            // Models are often written with a trailing option letter the file omits (54622D -> 54622).
+            if (m.Length > 3 && haystack.Contains(m.Substring(0, m.Length - 1))) return 60;
+
+            // Family match, worth much less: an 8563E's programming manual is often filed as the series -
+            // "8560E Programming Guide" - and a human looking for it would reach for that too. Low score so
+            // a genuine model match always wins, and the result says the substitution happened.
+            string stem = FamilyStem(m);
+            return stem != null && haystack.Contains(stem) ? 25 : 0;
+        }
+
+        /// <summary>True when this file is named for the model itself, not merely its family.</summary>
+        public static bool MatchesModel(ManualFile file, string model) =>
+            file != null && ModelScore(file, model) >= 60;
+
+        /// <summary>
+        /// The leading digits that identify an instrument family - "8563E" and "8560E" share "856". Null when
+        /// the model has no usable numeric stem, in which case there is no family to guess at.
+        /// </summary>
+        public static string FamilyStem(string model)
+        {
+            if (string.IsNullOrWhiteSpace(model)) return null;
+
+            string digits = new string(model.Trim().TakeWhile(char.IsDigit).ToArray());
+            return digits.Length >= 3 ? digits.Substring(0, 3) : null;
+        }
+
+        /// <summary>True when a candidate was found only by family resemblance, not by naming the model.</summary>
+        public static bool IsFamilyMatchOnly(ManualFile file, string model)
+        {
+            if (file == null || string.IsNullOrWhiteSpace(model)) return false;
+
+            string haystack = file.RelativePath.ToLowerInvariant();
+            string m = model.Trim().ToLowerInvariant();
+            if (haystack.Contains(m)) return false;
+            if (m.Length > 3 && haystack.Contains(m.Substring(0, m.Length - 1))) return false;
+
+            string stem = FamilyStem(m);
+            return stem != null && haystack.Contains(stem);
+        }
+
+        /// <summary>Splits a query into lower-cased words worth matching.</summary>
+        public static IEnumerable<string> Words(string query)
+        {
+            if (string.IsNullOrWhiteSpace(query)) yield break;
+            foreach (string raw in query.Split(new[] { ' ', '\t', '\r', '\n', ',', ';', ':', '(', ')', '"', '\'' },
+                                               StringSplitOptions.RemoveEmptyEntries))
+            {
+                string word = raw.Trim().ToLowerInvariant();
+                if (word.Length > 0) yield return word;
+            }
+        }
+    }
+
+    /// <summary>One file in the manual library.</summary>
+    public sealed class ManualFile
+    {
+        public ManualFile(string fullPath, string root)
+        {
+            FullPath = fullPath;
+            var info = new FileInfo(fullPath);
+            SizeBytes = info.Length;
+            ModifiedUtc = info.LastWriteTimeUtc;
+
+            string rootFull = Path.GetFullPath(root).TrimEnd(Path.DirectorySeparatorChar) +
+                              Path.DirectorySeparatorChar;
+            RelativePath = fullPath.StartsWith(rootFull, StringComparison.OrdinalIgnoreCase)
+                ? fullPath.Substring(rootFull.Length)
+                : Path.GetFileName(fullPath);
+        }
+
+        public string FullPath { get; }
+
+        /// <summary>Path within the library - what a citation shows, and what the caller passes back.</summary>
+        public string RelativePath { get; }
+
+        public long SizeBytes { get; }
+        public DateTime ModifiedUtc { get; }
+
+        public bool IsPdf =>
+            string.Equals(Path.GetExtension(FullPath), ".pdf", StringComparison.OrdinalIgnoreCase);
+    }
+}