Copying — or é through pbcopy lands intact without a UTF-8 locale

AGENTS.md

@@ -153,7 +153,7 @@ What follows is how TuiCode implements it today.
 ## Multiple cursors and undo (#106)
 
 - `Alt+Up`/`Alt+Down` move, and `Alt+Shift+Up`/`Alt+Shift+Down` duplicate, the lines under every cursor or selection (`EditorTextView.MoveLines`/`DuplicateLines`). As in VS Code, a selection that ends at column 0 leaves that last line out, and adjacent cursors' lines move as one block. `Ctrl+Alt+Up`/`Ctrl+Alt+Down` add a cursor above/below every cursor, keeping the column across shorter lines. `Alt+Click` adds or removes a cursor, and a plain click or `Esc` goes back to one. `Esc` is `RemoveSecondaryCursors`, `Editor`-scoped and enabled only with extra cursors; the find bar's layer sits above it, so Esc closes find first. These are workbench commands like any other, so they act on the active tab whatever has focus, then focus the editor.
-- **Carets.** The primary caret is TG's own insertion point and selection. The others are `Caret`s in `EditorTextView._secondary` (`EditorTextView.Carets.cs`). A key at several carets (`AtEachCaret`) loads each caret into TG's private fields in turn, last in the buffer first, and runs TG's own command there, so TG's movement and editing semantics all carry over. A caret already visited is held as its distance from the end of the buffer, which edits before it don't change. Carets that meet afterwards merge. Only the commands in `CaretCommands` run at every caret. Copy, cut and paste are our own (paste spreads the clipboard one line per caret when the counts match). Copy and cut at one caret go through the same path, `VerifiedClipboard.Write`, which reads the clipboard back to check the text arrived; `IClipboard.IsSupported` is false on macOS while the clipboard works, so it isn't consulted. A cut the clipboard refused deletes nothing, and the workbench reports either outcome in the status bar (#318). When the read-back doesn't match, `ClipboardTools` copies again through the platform's own program (`pbcopy`, `wl-copy`/`xclip`, `clip.exe`) and reads that back too (#319); tests swap `EditorTextView.ClipboardFallback` for one over a fake `IProcessRunner`, so none spawns a real process. TG 2.1.0's `TextView.OnKeyDown` reads `SelectedLength`, which rebuilds the selection by repeated string concatenation, so every key after select-all on a large file took minutes; `EditorTextView.OnKeyDown` calls it only while autocomplete has suggestions, which is never, since TuiCode doesn't use autocomplete. Anything else, such as select-all, first drops back to the primary caret. A terminal draws only one cursor, and anything we paint fills a whole cell, so in terminals that support kitty's [multiple cursors protocol](https://sw.kovidgoyal.net/kitty/multiple-cursors-protocol/) (kitty 0.43+), `TerminalCursors` has the terminal draw the focused editor's secondary carets in the main cursor's shape. `WorkbenchHost` queries support at startup (`QueueAnsiRequest`; a terminal that doesn't answer is dropped after a second). After each `LayoutAndDrawComplete` (i.e. after the frame is flushed) it sends their screen cells, only when they've changed. It clears them when focus leaves the editor: they aren't tied to the main cursor's visibility, so they'd otherwise show through dialogs. Elsewhere, and while that query is outstanding, `EditorTextView` hides the terminal cursor and paints `EditorTextView.Bar` (`▏`) in the cell after every caret, the primary included, so they all look like the terminal's own bar (#223). A painted caret fills a cell, so it covers the character it sits before — which is why only carets a terminal can't draw are painted, and a single caret anywhere, the editor's or a dialog's, is left to the terminal cursor. The cursor style comes back once there's a single caret. Cursor moves while visiting carets aren't reported to `EditorTab.CursorMoved`, so navigation history sees only the primary caret.
+- **Carets.** The primary caret is TG's own insertion point and selection. The others are `Caret`s in `EditorTextView._secondary` (`EditorTextView.Carets.cs`). A key at several carets (`AtEachCaret`) loads each caret into TG's private fields in turn, last in the buffer first, and runs TG's own command there, so TG's movement and editing semantics all carry over. A caret already visited is held as its distance from the end of the buffer, which edits before it don't change. Carets that meet afterwards merge. Only the commands in `CaretCommands` run at every caret. Copy, cut and paste are our own (paste spreads the clipboard one line per caret when the counts match). Copy and cut at one caret go through the same path, `VerifiedClipboard.Write`, which reads the clipboard back to check the text arrived; `IClipboard.IsSupported` is false on macOS while the clipboard works, so it isn't consulted. A cut the clipboard refused deletes nothing, and the workbench reports either outcome in the status bar (#318). When the read-back doesn't match, `ClipboardTools` copies again through the platform's own program (`pbcopy`, `wl-copy`/`xclip`, `clip.exe`) and reads that back too (#319). `pbcopy` and `pbpaste` run with `LC_ALL=en_US.UTF-8`: in any other locale they garble non-ASCII text on the way in and undo it on the way out, so the read-back can't catch it (#347); tests swap `EditorTextView.ClipboardFallback` for one over a fake `IProcessRunner`, so none spawns a real process. TG 2.1.0's `TextView.OnKeyDown` reads `SelectedLength`, which rebuilds the selection by repeated string concatenation, so every key after select-all on a large file took minutes; `EditorTextView.OnKeyDown` calls it only while autocomplete has suggestions, which is never, since TuiCode doesn't use autocomplete. Anything else, such as select-all, first drops back to the primary caret. A terminal draws only one cursor, and anything we paint fills a whole cell, so in terminals that support kitty's [multiple cursors protocol](https://sw.kovidgoyal.net/kitty/multiple-cursors-protocol/) (kitty 0.43+), `TerminalCursors` has the terminal draw the focused editor's secondary carets in the main cursor's shape. `WorkbenchHost` queries support at startup (`QueueAnsiRequest`; a terminal that doesn't answer is dropped after a second). After each `LayoutAndDrawComplete` (i.e. after the frame is flushed) it sends their screen cells, only when they've changed. It clears them when focus leaves the editor: they aren't tied to the main cursor's visibility, so they'd otherwise show through dialogs. Elsewhere, and while that query is outstanding, `EditorTextView` hides the terminal cursor and paints `EditorTextView.Bar` (`▏`) in the cell after every caret, the primary included, so they all look like the terminal's own bar (#223). A painted caret fills a cell, so it covers the character it sits before — which is why only carets a terminal can't draw are painted, and a single caret anywhere, the editor's or a dialog's, is left to the terminal cursor. The cursor style comes back once there's a single caret. Cursor moves while visiting carets aren't reported to `EditorTab.CursorMoved`, so navigation history sees only the primary caret.
 - **Undo is ours, not TG's** (`EditorTextView.History.cs`). TG 2.1.0's history can't hold an edit at several carets as one step. It also mis-undoes its own kill commands and edits over a selection, and interleaving undo with new edits corrupts it. Every edit is diffed (`LineDiff.Hunks`) against `_recorded`, a copy of the lines as of the last recorded edit, and pushed as one `EditGroup` with the carets before and after. Undo and redo splice those lines back and restore every caret and selection. TG's own history is cleared after each record, and `Undo`/`Redo` (the commands and the methods, via `new`) are ours. Edits reach the recorder through `Edit` (several carets, line moves), `EditAtPrimary` (a key, or `EditorTab.Replace`), or `OnContentsChanged` as a catch-all for anything else TG edits. The diff scans the whole buffer, costing ~0.2 ms per typed character at 50,000 lines (`EditorTypingBenchmarkTests`).
 - **Selecting occurrences (#113)**: `sno`/`spo`/`sao` (`EditorTextView.Occurrences.cs`) have no default keys. From a caret with no selection, they first select the word (letters, digits, `_`) or whitespace run at every caret; after that, occurrences of that text match only whole words or whole whitespace runs. Starting from a selection matches it anywhere, even across lines. Matching is case-sensitive, and the pure `Occurrences` class does it over graphemes, so columns are cells. Next/previous skip text that's already selected, wrap around the buffer, and make the new selection the primary caret so it scrolls into view. Select all replaces every caret, keeping the primary's selection as the primary.
 - TG 2.1.0's kill commands ignore a selection and then throw reading it, even with one cursor. `InvokeEditCommands` deletes the selection instead, as VS Code does.

src/TuiCode.Editor/ClipboardTools.cs

@@ -5,7 +5,11 @@
 namespace TuiCode.Editor;
 
 /// <summary>A platform's own clipboard program, e.g. <c>pbcopy</c>, and the one that reads the clipboard back.</summary>
-internal sealed record ClipboardTool(string Name, string[] Copy, string[] Paste, Encoding InputEncoding);
+internal sealed record ClipboardTool(string Name, string[] Copy, string[] Paste, Encoding InputEncoding)
+{
+    /// <summary>Variables set for both programs on top of TuiCode's own environment.</summary>
+    public IReadOnlyDictionary<string, string> Environment { get; init; } = new Dictionary<string, string>();
+}
 
 internal abstract record ToolRun
 {
@@ -18,7 +22,7 @@ public sealed record Exited(int Code, string Output, string Error) : ToolRun;
 
 internal interface IProcessRunner
 {
-    ToolRun Run(string[] command, byte[] input, bool readOutput, TimeSpan timeout);
+    ToolRun Run(string[] command, IReadOnlyDictionary<string, string> environment, byte[] input, bool readOutput, TimeSpan timeout);
 }
 
 /// <summary>Copies through the platform's clipboard program when Terminal.Gui's clipboard didn't take the text (#319).</summary>
@@ -30,7 +34,11 @@ internal sealed class ClipboardTools(IReadOnlyList<ClipboardTool> candidates, st
         OperatingSystem.IsMacOS(), OperatingSystem.IsWindows(),
         !string.IsNullOrEmpty(Environment.GetEnvironmentVariable("WAYLAND_DISPLAY")), new ProcessRunner());
 
-    private static readonly ClipboardTool PbCopy = new("pbcopy", ["pbcopy"], ["pbpaste"], new UTF8Encoding(false));
+    // Without a UTF-8 locale pbcopy stores the bytes as Mac Roman and pbpaste undoes it, so the read-back can't tell (#347).
+    private static readonly ClipboardTool PbCopy = new("pbcopy", ["pbcopy"], ["pbpaste"], new UTF8Encoding(false))
+    {
+        Environment = new Dictionary<string, string> { ["LC_ALL"] = "en_US.UTF-8" },
+    };
     private static readonly ClipboardTool WlCopy = new("wl-copy", ["wl-copy"], ["wl-paste", "--no-newline"], new UTF8Encoding(false));
     private static readonly ClipboardTool XClip = new("xclip", ["xclip", "-selection", "clipboard"], ["xclip", "-selection", "clipboard", "-o"], new UTF8Encoding(false));
     // clip.exe reads UTF-16 only when it starts with a byte order mark; otherwise it uses the console code page.
@@ -51,11 +59,11 @@ public static ClipboardTools For(bool macOS, bool windows, bool wayland, IProces
         foreach (var tool in candidates)
         {
             var input = tool.InputEncoding.GetPreamble().Concat(tool.InputEncoding.GetBytes(text)).ToArray();
-            var copied = runner.Run(tool.Copy, input, readOutput: false, Timeout);
+            var copied = runner.Run(tool.Copy, tool.Environment, input, readOutput: false, Timeout);
             if (copied is ToolRun.NotFound) continue;
             if (Failure(tool, copied) is { } copyFailed) return copyFailed;
 
-            var pasted = runner.Run(tool.Paste, [], readOutput: true, Timeout);
+            var pasted = runner.Run(tool.Paste, tool.Environment, [], readOutput: true, Timeout);
             if (pasted is ToolRun.NotFound) return new CopyOutcome.Failed($"{tool.Paste[0]} not found ({tool.Name})");
             if (Failure(tool, pasted) is { } pasteFailed) return pasteFailed;
             return ((ToolRun.Exited)pasted).Output.ReplaceLineEndings("\n") == text.ReplaceLineEndings("\n")
@@ -78,7 +86,7 @@ public static ClipboardTools For(bool macOS, bool windows, bool wayland, IProces
 
 internal sealed class ProcessRunner : IProcessRunner
 {
-    public ToolRun Run(string[] command, byte[] input, bool readOutput, TimeSpan timeout)
+    public ToolRun Run(string[] command, IReadOnlyDictionary<string, string> environment, byte[] input, bool readOutput, TimeSpan timeout)
     {
         var info = new ProcessStartInfo(command[0])
         {
@@ -91,6 +99,7 @@ public ToolRun Run(string[] command, byte[] input, bool readOutput, TimeSpan tim
             StandardErrorEncoding = Encoding.UTF8,
         };
         foreach (var argument in command.Skip(1)) info.ArgumentList.Add(argument);
+        Override(info.Environment, environment);
 
         Process process;
         try
@@ -138,4 +147,9 @@ public ToolRun Run(string[] command, byte[] input, bool readOutput, TimeSpan tim
             return new ToolRun.Exited(process.ExitCode, output.Result, error.Result);
         }
     }
+
+    internal static void Override(IDictionary<string, string?> inherited, IReadOnlyDictionary<string, string> environment)
+    {
+        foreach (var (name, value) in environment) inherited[name] = value;
+    }
 }

tests/TuiCode.Tests/ClipboardToolsTests.cs

@@ -101,6 +101,40 @@ public void Write_hands_clip_exe_UTF_16_with_a_byte_order_mark()
 
         Assert.Equal([0xFF, 0xFE, 0xE9, 0x00], program.Input);
     }
+
+    // #347
+    [Fact]
+    public void Write_runs_pbcopy_and_pbpaste_in_a_UTF_8_locale()
+    {
+        var program = new ClipboardProgram();
+
+        ClipboardTools.For(true, false, false, program).Write("a — b");
+
+        Assert.Equal(["LC_ALL=en_US.UTF-8", "LC_ALL=en_US.UTF-8"], program.Environments);
+    }
+
+    [Theory]
+    [InlineData(false, true)]
+    [InlineData(false, false)]
+    [InlineData(true, false)]
+    public void Write_leaves_the_locale_of_byte_transparent_programs_alone(bool wayland, bool windows)
+    {
+        var program = new ClipboardProgram();
+
+        ClipboardTools.For(false, windows, wayland, program).Write("a — b");
+
+        Assert.All(program.Environments, environment => Assert.Equal("", environment));
+    }
+
+    [Fact]
+    public void The_programs_keep_the_rest_of_TuiCode_s_environment()
+    {
+        var inherited = new Dictionary<string, string?> { ["LC_ALL"] = "C", ["DISPLAY"] = ":0", ["WAYLAND_DISPLAY"] = "wayland-0" };
+
+        ProcessRunner.Override(inherited, new Dictionary<string, string> { ["LC_ALL"] = "en_US.UTF-8" });
+
+        Assert.Equal(new Dictionary<string, string?> { ["LC_ALL"] = "en_US.UTF-8", ["DISPLAY"] = ":0", ["WAYLAND_DISPLAY"] = "wayland-0" }, inherited);
+    }
 }
 
 /// <summary>Stands in for every clipboard program: a copy stores its input as UTF-8, a paste returns it; <see cref="Answer"/> overrides either.</summary>
@@ -110,11 +144,13 @@ internal sealed class ClipboardProgram : IProcessRunner
     public byte[] Input { get; private set; } = [];
     public TimeSpan Timeout { get; private set; }
     public List<string> Ran { get; } = [];
+    public List<string> Environments { get; } = [];
     public Func<string[], ToolRun?> Answer { get; init; } = _ => null;
 
-    public ToolRun Run(string[] command, byte[] input, bool readOutput, TimeSpan timeout)
+    public ToolRun Run(string[] command, IReadOnlyDictionary<string, string> environment, byte[] input, bool readOutput, TimeSpan timeout)
     {
         Ran.Add(string.Join(' ', command));
+        Environments.Add(string.Join(' ', environment.Select(variable => $"{variable.Key}={variable.Value}")));
         Timeout = timeout;
         if (Answer(command) is { } answer) return answer;
         if (readOutput) return new ToolRun.Exited(0, Stored, "");