From dba8cd7018b4487b45ad64581512fa619b151acc Mon Sep 17 00:00:00 2001 From: Sekers <46898253+Sekers@users.noreply.github.com> Date: Wed, 23 Sep 2026 15:50:25 -0500 Subject: [PATCH 1/2] Make the PSScriptAnalyzer pin hold, and catch syntax errors The Tests workflow installed PSScriptAnalyzer 1.25.0 so that a new rule could not fail a build that passed before, but nothing made the test use that copy. The windows-latest image installs its own, the newest when the image is built (1.25.0 in 20260907.229.1), and Windows PowerShell 5.1 cannot see the copy pwsh installs, so the 5.1 run always used the image's version. An image update would have changed the rules without any change here. - Tests.yml sets the version once, in PSSCRIPTANALYZER_VERSION, and installs that version. - The test imports exactly that version when the variable is set, and the newest installed copy otherwise. PowerShell itself would load the copy in the first PSModulePath folder that has one, whatever its version. With CI=true, a missing analyzer or pinned version fails the test instead of skipping it. - The test runs under PowerShell 7 only, and names the reason when it skips. 1.25.0 has the same 75 rules in both editions, and syntax only PowerShell 7 accepts already fails every 5.1 test file that imports the module. - Analyze with -Severity Error, ParseError. A syntax error has its own ParseError severity, so filtering on Error alone reported nothing for a planted missing parenthesis. - Research_Notes/File-Encoding-And-Line-Endings.md section 1 now measures a module manifest as well as a script, which is what the AGENTS.md rule for saving a non-ASCII manifest with a BOM cites. Checked: with a fake newer PSScriptAnalyzer first on PSModulePath, the pinned run loads 1.25.0 and passes; a pinned version that is not installed skips locally and fails with CI=true; a planted syntax error fails the test naming the file, line and rule. --- .github/workflows/Tests.yml | 9 ++-- .../File-Encoding-And-Line-Endings.md | 20 +++++++-- Tests/ScriptMessage.Module.Tests.ps1 | 42 +++++++++++++++++-- 3 files changed, 61 insertions(+), 10 deletions(-) diff --git a/.github/workflows/Tests.yml b/.github/workflows/Tests.yml index 280109c..b431f6f 100644 --- a/.github/workflows/Tests.yml +++ b/.github/workflows/Tests.yml @@ -34,14 +34,17 @@ jobs: name: Pester # Windows, because Windows PowerShell 5.1 exists nowhere else. Invoke-Tests.ps1 runs both editions itself. runs-on: windows-latest + env: + # Pinned so a new analyzer rule cannot fail a build that passed before. The runner image ships its own copy, + # so the module tests read this to use exactly this version rather than whichever copy PowerShell finds first. + PSSCRIPTANALYZER_VERSION: '1.25.0' steps: - uses: actions/checkout@v7 - # The module tests run PSScriptAnalyzer when it is installed. Pinned so a new analyzer rule cannot fail a - # build that passed before. + # The module tests run PSScriptAnalyzer under PowerShell 7, and fail in CI rather than skipping without it. - name: Install PSScriptAnalyzer - run: Install-PSResource -Name PSScriptAnalyzer -Version 1.25.0 -TrustRepository + run: Install-PSResource -Name PSScriptAnalyzer -Version $env:PSSCRIPTANALYZER_VERSION -TrustRepository shell: pwsh - name: Run tests diff --git a/Research_Notes/File-Encoding-And-Line-Endings.md b/Research_Notes/File-Encoding-And-Line-Endings.md index 08332d2..ece809d 100644 --- a/Research_Notes/File-Encoding-And-Line-Endings.md +++ b/Research_Notes/File-Encoding-And-Line-Endings.md @@ -6,7 +6,8 @@ happened when this repository was normalized to LF. The rules themselves live in Sections 1 to 4 were measured on **2026-09-15** on Windows 11 (10.0.26200), with Windows PowerShell **5.1.26100.9444**, PowerShell **7.6.6**, and Git **2.54.0.windows.1** with `core.autocrlf=true` set in the -system-wide Git configuration. Section 5 gives its own date. +system-wide Git configuration. Section 5, and the module manifest measurement in section 1, give their own +dates. **Claims are labelled with their evidence.** @@ -30,8 +31,21 @@ A two-line script that assigns one em dash (U+2014) to a string and prints the s Windows PowerShell 5.1 read the em dash's three UTF-8 bytes as three characters, with no error or warning. PowerShell 7 read both files correctly. This is why PowerShell files in this repository are ASCII only rather than UTF-8 with a BOM: with no non-ASCII characters, whether a file has a BOM makes no difference to either -edition. The row saved with a BOM is the fallback for a module manifest that genuinely needs a non-ASCII -character, since a `.psd1` cannot contain an escape such as `[char]0x00E9`. +edition. + +**Measured on 2026-09-23**, on the same machine and PowerShell versions, with ANSI code page 1252: the same +question for a module manifest, the one PowerShell file that may need a BOM, since a `.psd1` cannot contain an +escape such as `[char]0x00E9`. A minimal manifest whose `Author` was `Caf` followed by U+00E9 was saved as UTF-8 +and read with `Import-PowerShellDataFile`, `Test-ModuleManifest`, and `Import-Module`: + +| Saved as | Windows PowerShell 5.1 | PowerShell 7 | +| --- | --- | --- | +| UTF-8 without BOM | 5 characters, `Caf` then U+00C3 U+00A9 | 4 characters, `Caf` then U+00E9 | +| UTF-8 with BOM | 4 characters, `Caf` then U+00E9 | 4 characters, `Caf` then U+00E9 | + +All three commands gave the same result in every case, and none raised an error or warning. So a manifest that +genuinely needs a non-ASCII character is saved as UTF-8 with a BOM, which both editions read correctly. Only the +`Author` value and code page 1252 were tested. Section 5 measures the same question for the JSON configuration file. diff --git a/Tests/ScriptMessage.Module.Tests.ps1 b/Tests/ScriptMessage.Module.Tests.ps1 index 364e491..04c63e8 100644 --- a/Tests/ScriptMessage.Module.Tests.ps1 +++ b/Tests/ScriptMessage.Module.Tests.ps1 @@ -18,10 +18,44 @@ Describe 'ScriptMessage module' { ($ExportedFunctions -join ', ') | Should -Be ($PublicFunctions -join ', ') } - # PSScriptAnalyzer is optional locally. The CI workflow installs it for PowerShell 7, so the check always runs - # there. - It 'has no PSScriptAnalyzer errors' -Skip:(-not (Get-Module -ListAvailable -Name PSScriptAnalyzer)) { - $Findings = @(Invoke-ScriptAnalyzer -Path $ModuleRoot -Recurse -Severity Error) + # PSScriptAnalyzer is optional locally. The check runs under PowerShell 7 only, because Windows PowerShell 5.1 + # cannot see the copy the CI workflow installs; the analyzer has the same rules in both editions, and syntax + # that only PowerShell 7 accepts already fails the 5.1 run when the module imports. + # + # PSSCRIPTANALYZER_VERSION, when set, names the exact version to use. The workflow sets it to the version it + # installs, because the runner image ships its own copy and PowerShell would otherwise load whichever copy it + # finds first. Without it, the newest installed version is used. In CI, where GitHub sets CI=true, a missing + # analyzer fails rather than skips, so the check cannot quietly disappear if the install step breaks. + # + # ParseError is its own severity, so without it a syntax error would report nothing. + It 'has no PSScriptAnalyzer errors' { + if ($PSVersionTable.PSEdition -ne 'Core') + { + Set-ItResult -Skipped -Because 'the analyzer check runs under PowerShell 7 only' + } + + $Installed = @(Get-Module -ListAvailable -Name PSScriptAnalyzer) + if ($env:PSSCRIPTANALYZER_VERSION) + { + $Analyzer = $Installed | Where-Object { $_.Version -eq [version]$env:PSSCRIPTANALYZER_VERSION } | Select-Object -First 1 + $Missing = "PSScriptAnalyzer $env:PSSCRIPTANALYZER_VERSION is not installed" + } + else + { + $Analyzer = $Installed | Sort-Object -Property Version -Descending | Select-Object -First 1 + $Missing = 'PSScriptAnalyzer is not installed' + } + if (-not $Analyzer) + { + if ($env:CI -eq 'true') + { + throw "$Missing, and CI must run this check." + } + Set-ItResult -Skipped -Because $Missing + } + + Import-Module -Name PSScriptAnalyzer -RequiredVersion $Analyzer.Version + $Findings = @(Invoke-ScriptAnalyzer -Path $ModuleRoot -Recurse -Severity Error, ParseError) @($Findings | ForEach-Object { '{0}:{1} {2}' -f $_.ScriptName, $_.Line, $_.RuleName }) | Should -BeNullOrEmpty } From 37ccac387cf2ae88c86ea4c6f8d90b91447e10d7 Mon Sep 17 00:00:00 2001 From: Sekers <46898253+Sekers@users.noreply.github.com> Date: Wed, 23 Sep 2026 15:59:16 -0500 Subject: [PATCH 2/2] Record the measurements behind the PSScriptAnalyzer test The analyzer test and the Tests workflow rest on tool behavior that was measured but only written down in a commit message, and several of those facts change with the analyzer version or the runner image. Add Research_Notes/Test-Tooling-Behavior.md: syntax errors appear only under the ParseError severity; the analyzer's rules match across editions but its parsing does not; PowerShell loads the first copy of a module on PSModulePath, not the newest, and a 5.1 process started from pwsh cannot see pwsh's folders; and the runner image installs whatever analyzer version is newest when it is built. The note says what to repeat when PSSCRIPTANALYZER_VERSION changes, and the test and the workflow point at it. --- .github/workflows/Tests.yml | 1 + Research_Notes/Test-Tooling-Behavior.md | 125 ++++++++++++++++++++++++ Tests/ScriptMessage.Module.Tests.ps1 | 2 + 3 files changed, 128 insertions(+) create mode 100644 Research_Notes/Test-Tooling-Behavior.md diff --git a/.github/workflows/Tests.yml b/.github/workflows/Tests.yml index b431f6f..1cf9408 100644 --- a/.github/workflows/Tests.yml +++ b/.github/workflows/Tests.yml @@ -37,6 +37,7 @@ jobs: env: # Pinned so a new analyzer rule cannot fail a build that passed before. The runner image ships its own copy, # so the module tests read this to use exactly this version rather than whichever copy PowerShell finds first. + # Research_Notes/Test-Tooling-Behavior.md says what to recheck when changing it. PSSCRIPTANALYZER_VERSION: '1.25.0' steps: diff --git a/Research_Notes/Test-Tooling-Behavior.md b/Research_Notes/Test-Tooling-Behavior.md new file mode 100644 index 0000000..7bec16e --- /dev/null +++ b/Research_Notes/Test-Tooling-Behavior.md @@ -0,0 +1,125 @@ +# Test tooling behavior + +How PSScriptAnalyzer, PowerShell's module loading, and the GitHub Actions runner image behave where this +repository's tests depend on them. None of this is module behavior; it constrains how the test suite and the Tests +workflow are written, chiefly the `has no PSScriptAnalyzer errors` test in `Tests/ScriptMessage.Module.Tests.ps1` +and the `pester` job in `.github/workflows/Tests.yml`. + +Measured on **2026-09-23** on Windows 11 (10.0.26200), with Windows PowerShell **5.1.26100.9444**, PowerShell +**7.6.6**, PSScriptAnalyzer **1.25.0**, and Pester **6.1.0**. Nothing connected to a tenant. + +**Claims are labelled with their evidence.** + +- **Measured** means observed on the date given. +- **From source** means read from a file in this repository. +- **From documentation** means read from the linked page on the date given. +- **Unverified** means nobody has tested it. + +> **The one-line summary:** PSScriptAnalyzer reports a syntax error only when asked for `ParseError`, it parses +> with the edition running it, and PowerShell loads the first copy of a module it finds rather than the newest. +> Name the analyzer version explicitly, because the runner image ships its own copy. + +**When changing `PSSCRIPTANALYZER_VERSION`** in `.github/workflows/Tests.yml`, repeat sections 1 and 2 with the new +version. + +## 1. Measured: syntax errors are reported only under the `ParseError` severity + +`Invoke-ScriptAnalyzer -ScriptDefinition 'function f { if ($true { } }'` gave the same result in both editions: + +| `-Severity` | Findings | +| --- | --- | +| `Error` | none | +| `Error, ParseError` | `UnexpectedToken` and `MissingEndParenthesisAfterStatement`, both with severity `ParseError` | +| not given | the same two | + +Under PowerShell 7, with a missing `)` planted in an `if` statement in +`ScriptMessage/Private/ConvertTo-ScriptMessageBodyObject.ps1`, `Invoke-ScriptAnalyzer -Path ScriptMessage -Recurse` +reported nothing with `-Severity Error` and `MissingEndParenthesisAfterStatement` on that line with +`-Severity Error, ParseError`. Without the plant, the module had no finding of either severity. + +This is why the analyzer test passes `-Severity Error, ParseError`. + +## 2. Measured: the analyzer's rules are the same in both editions, but its parsing is not + +- `Get-ScriptAnalyzerRule` listed 75 rules in each edition, with identical names. +- `Invoke-ScriptAnalyzer -ScriptDefinition '$a = $true ? 1 : 2' -Severity Error, ParseError` returned nothing under + PowerShell 7 and `UnexpectedToken` under Windows PowerShell 5.1. The analyzer accepts whatever syntax the edition + running it accepts (**Unverified:** that it uses that edition's parser; its source was not read). + +So an analyzer run under PowerShell 7 alone cannot flag syntax that only PowerShell 7 accepts. The rest of the +suite covers that gap: with the ternary `return $true ? $null : $null` planted in +`ConvertTo-ScriptMessageBodyObject`, `.\Tests\Invoke-Tests.ps1 -Edition Desktop` failed 79 tests and passed 7, and +six of the eight test files failed as a whole. + +This is one of the two reasons the analyzer test runs under PowerShell 7 only; section 3 gives the other. + +## 3. Measured: which copy of a module PowerShell loads + +Two folders were put at the front of `PSModulePath` in a fresh process of each edition. The first held a module +`FakeMod` at version 1.0.0 and the second held `FakeMod` 2.0.0, each exporting one function that returned its +version. + +| Load | Windows PowerShell 5.1 | PowerShell 7 | +| --- | --- | --- | +| Calling the function (command autoloading) | 1.0.0 | 1.0.0 | +| `Import-Module -Name FakeMod` | 1.0.0 | 1.0.0 | + +PowerShell loaded the copy in the first `PSModulePath` folder that held the module, not the newest version. + +The folders each edition searched on the test machine, in order: + +- PowerShell 7: the user's `Documents\PowerShell\Modules`, `C:\Program Files\PowerShell\Modules`, + `$PSHOME\Modules`, `C:\Program Files\WindowsPowerShell\Modules`, and + `C:\Windows\system32\WindowsPowerShell\v1.0\Modules`. +- Windows PowerShell 5.1, started from PowerShell 7 the way `Tests/Invoke-Tests.ps1` starts it: the user's + `Documents\WindowsPowerShell\Modules`, `C:\Program Files\WindowsPowerShell\Modules`, and + `C:\Windows\system32\WindowsPowerShell\v1.0\Modules`. None of PowerShell 7's own folders. + +**From documentation** ([Install-PSResource](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.psresourceget/install-psresource), +read 2026-09-23): the default scope is `CurrentUser`, which installs to a folder such as +`$home\Documents\PowerShell\Modules`, and `AllUsers` installs to a folder such as +`$env:ProgramFiles\PowerShell\Modules`. A bare version such as `1.25.0` is treated as the required version, not +as a minimum. + +So the copy the Tests workflow installs with `Install-PSResource` lands in one of PowerShell 7's own folders, where +a Windows PowerShell 5.1 test run cannot load it. That is the other reason the analyzer test runs under PowerShell +7 only. It imports the version named in `PSSCRIPTANALYZER_VERSION` with `Import-Module -RequiredVersion` rather +than relying on which copy PowerShell finds first. With a fake PSScriptAnalyzer 9.9.9 placed first on +`PSModulePath`, the test loaded the fake when the variable was not set (it then takes the newest installed copy) +and 1.25.0 when the variable was `1.25.0`. + +## 4. The runner image's own copy + +**From documentation**, read 2026-09-23: + +- The [actions/runner-images README](https://github.com/actions/runner-images) maps `windows-latest` to the Windows + Server 2025 with Visual Studio 2026 image, described in + [Windows2025-VS2026-Readme.md](https://github.com/actions/runner-images/blob/main/images/windows/Windows2025-VS2026-Readme.md). + For image version 20260907.229.1, that readme lists `PSScriptAnalyzer: 1.25.0` and `Pester: 3.4.0, 5.9.0`. +- The image's + [toolset-2025-vs2026.json](https://github.com/actions/runner-images/blob/main/images/windows/toolsets/toolset-2025-vs2026.json) + lists PSScriptAnalyzer with no version, and + [Install-PowerShellModules.ps1](https://github.com/actions/runner-images/blob/main/images/windows/scripts/build/Install-PowerShellModules.ps1) + installs a module listed that way with `Install-Module -Scope AllUsers` and no version. So each image build takes + whatever PSScriptAnalyzer version is newest on the PowerShell Gallery at the time. + +**Measured** from the logs of two Tests workflow runs, on 2026-09-22 at commit 453ce45 and on 2026-09-23 at +dba8cd7: the `Set up job` step printed `Image: windows-2025-vs2026` and `Version: 20260907.229.1`. In the +2026-09-22 run, before the analyzer test chose its own version, the test passed under both editions. Since Windows +PowerShell 5.1 cannot see the copy the workflow installs (section 3), that 5.1 run must have loaded the image's +copy. + +So the analyzer version the image provides can change with any image update, with no change in this repository. +`PSSCRIPTANALYZER_VERSION` is what holds the version the test uses. + +## What this does not establish + +- Only PSScriptAnalyzer 1.25.0 was tested. Another version may have different rules, or may report syntax errors + differently; that is why sections 1 and 2 need repeating when the version changes. +- Section 2 tested one construct that only PowerShell 7 accepts (a ternary), planted in one private helper file. +- Section 3 tested loading by name with no version parameter, in processes that had no copy loaded yet. +- The runner's own `PSModulePath` was not measured, and the image may add folders to it. So, before the analyzer + test chose its version, whether the PowerShell 7 run in CI loaded the workflow's copy or the image's is + **Unverified**. So is whether the install step installed a second copy of 1.25.0 or skipped: it printed nothing. +- Which folder the image's copy is in was not established; the 2026-09-22 run shows only that Windows PowerShell + 5.1 could load it. diff --git a/Tests/ScriptMessage.Module.Tests.ps1 b/Tests/ScriptMessage.Module.Tests.ps1 index 04c63e8..ab77fc5 100644 --- a/Tests/ScriptMessage.Module.Tests.ps1 +++ b/Tests/ScriptMessage.Module.Tests.ps1 @@ -28,6 +28,8 @@ Describe 'ScriptMessage module' { # analyzer fails rather than skips, so the check cannot quietly disappear if the install step breaks. # # ParseError is its own severity, so without it a syntax error would report nothing. + # + # Research_Notes/Test-Tooling-Behavior.md records the measurements behind these choices. It 'has no PSScriptAnalyzer errors' { if ($PSVersionTable.PSEdition -ne 'Core') {