Skip to content

Retire the superseded evaluator, and stop the install scripts failing on a shared API quota - #719

Open
awsmadi wants to merge 10 commits into
aws-cloudformation:mainfrom
awsmadi:pr/installers-lint-and-legacy-cleanup
Open

Retire the superseded evaluator, and stop the install scripts failing on a shared API quota#719
awsmadi wants to merge 10 commits into
aws-cloudformation:mainfrom
awsmadi:pr/installers-lint-and-legacy-cleanup

Conversation

@awsmadi

@awsmadi awsmadi commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Base: main. Independent of #717 — nothing here touches evaluator semantics.

Three unrelated-looking pieces that share one theme: things CI was not actually checking. A dead
evaluator nobody had removed, an installer whose failures could not surface, and a PowerShell
script with no static analysis at all.

9 commits, 35 files, +632 / −6,036.

What prompted it

The Windows install job failed on an unrelated pull request:

{"message": "API rate limit exceeded for 20.9.183.48."}

20.9.183.48 is a shared GitHub Actions runner. The anonymous GitHub API allows 60 requests an
hour per source IP, counted across everyone behind it, and install-guard.ps1 resolved the
latest release through it unauthenticated. The same ceiling reaches real users: a corporate NAT, a
VPN, or a colleague installing from the same office is enough.

Pulling that thread turned up two further problems, both of which had been invisible precisely
because nothing was looking.

The commits

Retiring the superseded evaluator

1. 6dc98ba Retire the superseded evaluator and its query engine — deletes evaluate.rs,
evaluate_tests.rs, the old path_value query engine and its tests, and aws_meta_appender.
−4,693 lines. This code was replaced by the eval module and had no remaining callers.

2. 144baac Make verbose output independent of the compiler that built it — verbose output
varied with build configuration, so what an operator saw depended on how the binary was compiled
rather than on the rules being evaluated.

3. e3fea72 Apply rustfmt — formatting only, from the deletions above.

4. 4cab372 Remove the dead Reporter::report method — the trait method had no live
implementations left once the old evaluator was gone. −349 lines across nine reporters.

5. 448317b Delete the two unreachable validate reporterscfn_reporter and
console_reporter were no longer reachable from any command path. −472 lines.

6. 2e0fc4c Delete the rest of the legacy reporting cluster; zero dead-code warnings — the
remaining fragments, taking the crate from 43 dead-code warnings to none.

Making the test suite able to fail

7. 3445508 Anchor test path sanitisation on the crate dir, not $HOME — the integration
harness rewrote absolute paths by substituting $HOME, which only works when the checkout lives
under the home directory. Anywhere else the substitution missed and 15 expected-output tests failed
on path noise alone. Anchored on CARGO_MANIFEST_DIR instead.

The installers

8. 0df9eae Stop the install scripts failing on an anonymous GitHub API quota — the substance
of the fix, covering several distinct problems:

  • Resolution order. gh CLI first when installed and authenticated, since it reuses
    credentials the caller already has; then the REST API with GITHUB_TOKEN if the environment
    supplies one; then anonymously, the only path the limit applies to. An explicit version skips the
    lookup entirely, and install-guard.ps1 gains -Version for that, matching -v in
    install-guard.sh.
  • Backoff from the API's own signals, not guesses: retry-after on a secondary limit,
    x-ratelimit-reset when the primary one is exhausted, exponential backoff only when neither
    header is readable. Blind doubling would retry straight into an empty quota and report a network
    error for what is really a quota problem.
  • A five-minute ceiling on waiting. A primary reset can be an hour away, and an installer that
    appears hung for an hour is worse than one that explains itself. Past the cap it stops and names
    GITHUB_TOKEN, gh auth login and -Version as the ways out.
  • install-guard.sh could not fail. err() exits, but it was reached from the left side of a
    pipeline feeding a while read loop, so exit 1 ended only that subshell and the pipeline took
    its status from the loop, which had read nothing. A failed lookup left the script exiting 0
    with nothing installed. That is why only the Windows job went red: all three platforms hit the
    same wall and only the one without this bug reported it. get_version's result is assigned now.
  • Get-WmiObject was removed in PowerShell 6, and this workflow runs pwsh 7. Architecture
    detection reads OSArchitecture from RuntimeInformation instead — part of the framework,
    present in every supported host, and exercisable off Windows, which the WMI and CIM cmdlets are
    not.
  • The token is passed to curl through a config file on stdin, not argv. An Authorization
    header on a command line is readable from ps by anyone else on the host for the life of the
    request. It is only ever sent to api.github.com; the release archive redirects to a separate
    download host.
  • The install jobs now build from the branch under test, package the binary into the release
    layout, install that, and assert the installed binary's checksum matches the one just built.
    They previously resolved the latest release and installed it, so they tested the installer
    against a binary unrelated to the change under review, over the API that failed above.

Static analysis for the PowerShell script

9. 583c3b8 Gate install-guard.ps1 on PSScriptAnalyzerinstall-guard.sh has been guarded
by shellcheck since it was written; the PowerShell script had nothing. That is how the
Get-WmiObject call survived: the script referenced a cmdlet its own CI shell does not have, and
nothing in the repository was looking.

Configured by .github/PSScriptAnalyzerSettings.psd1 at Error, Warning and Information, so a new
finding fails the build. PSAvoidUsingWriteHost is the single exclusion, with the reasoning
recorded beside it: Get-ArchType and Get-GuardVersion return their values through the
pipeline
, so routing progress commentary to Write-Output as the rule advises would mix that text
into their return values and break them.

Run against the script as it stood before commit 8, the analyser reports two real problems; run
against it now, none. The gate starts clean rather than with a backlog to grandfather in.

Verification

Check Result
cargo fmt --check pass
cargo clippy -- -D warnings pass
cargo clippy --all-targets -- -D warnings pass
typos pass
shellcheck install-guard.sh pass
PSScriptAnalyzer 0 findings
Test suite 620 passed, 0 failed; validate 96/0
Install job, run end to end locally installed binary's SHA256 equals the branch build

Two failure paths were exercised rather than assumed: a missing archive exits 1, and an unreachable
API exits 1 after retrying with the guidance above. Both exited 0 before.

The PowerShell logic that cannot run on a Linux runner was unit-tested separately — architecture
mapping, -Version bypassing the API, retry-after precedence, reset-based waiting, header
fallback, and the gh path — 9 checks, all passing.

Merge order against #717

These two touch 17 files in common and both are individually mergeable, so whichever lands second
will need a rebase. One file is worth knowing about before that happens:
guard/src/commands/reporters/validate/cfn_reporter.rs, which this PR deletes as unreachable and
#717 modifies.

The resolution is to take the deletion. #717's change to that file is a mechanical signature update,
HashSet<String> to SkippedRules, made so the file kept compiling when that type changed, plus a
comment noting that this path carries no record tree and so reports skips without reasons. It is not
a fix, and it has no effect on behaviour, because neither CfnReporter nor SingleLineReporter is
ever constructed. Every other overlapping file is modified by both and conflicts only where the two
edits sit near each other.

Measured with git merge-tree rather than inferred from GitHub's mergeable flag, which reports on
each PR against main separately and so says nothing about this.

awsmadi added 10 commits August 17, 2026 18:00
guard/src/rules/evaluate.rs was the pre-eval.rs evaluator. Nothing on the CLI
path reached it: the compiler reported 15 items in it as never used, RootScope::new
was called only from evaluate_tests.rs, and its one apparent outside consumer --
path_value.rs's QueryResolver::select -- takes &dyn EvaluationContext, the old
context trait. select's only non-test callers were itself, evaluate.rs, and
MetadataAppender, which is constructed only in its own test. One connected dead
component, not a scattering.

Removed:
  - evaluate.rs (1342) and evaluate_tests.rs (2345)
  - QueryResolver and PathAwareValue::select, plus the four helpers only select
    used: map_error_or_empty, map_some_or_error_all, retrieve_index, accumulate.
    eval_context.rs already has its own retrieve_index and accumulate as free
    functions, so the new engine loses nothing.
  - aws_meta_appender.rs and its test

Kept: EvaluationType and StatusContext, which live reporter code still reads --
GenericSummary is constructed at helper.rs:66 and validate.rs:704.

Display for GuardNamedRuleClause moved to exprs.rs beside the Display impls for
its sibling clause types. It lived in evaluate.rs but eval.rs formats it, so it
was the one piece of that file the live evaluator depended on.

evaluate_tests.rs was also a near-duplicate suite: of five test names sampled,
all five exist in eval_tests.rs against the live evaluator. The 39 tests dropped
here (32 evaluate_tests, 4 path_value_tests, 2 values_tests, 1 appender) all
exercised only the deleted engine; the same semantics stay covered by
eval_context_tests for query retrieval, eval_tests::filter_based_* and
test_map_keys_function for filters, and rule_test_type_blocks for type blocks.

358 -> 319 lib tests, 0 failed. 4693 lines deleted.
eval_conjunction_clauses built a user-visible string from
std::any::type_name::<T>(): `context` becomes the `Context=` label on the
Disjunction node in --verbose output, and four fixtures under guard/resources pin
it exactly.

type_name's output is explicitly unspecified -- std documents that it "must not
be considered to uniquely identify a type" and may change between compiler
versions, and it did. Newer rustc renders the elided lifetime, so
cfn_guard::rules::exprs::GuardClause became ...::GuardClause<'_>, and
test_data_file_verbose and test_with_rules_dir_verbose fail on any toolchain
other than the 1.77.2 pinned in rust-toolchain.toml. Verified failing at both
320251c and 57bbdbf (upstream main) under rustc 1.97, so it is pre-existing and
not introduced by this branch.

Truncating at the first `<` restores the pre-change spelling for every T and is
stable under further rendering changes, since only the generic/lifetime portion
varies. Deliberately not changing the strings themselves: that a Rust module path
is user-facing output at all is a real wart, but rewriting it is a visible output
change that belongs with the fixtures.

Also removes the EvaluationContext and Evaluate traits, which the previous commit
left with no production implementations. `cargo clippy -- -D warnings` -- the
gate at pr.yml:94 -- rejects `trait EvaluationContext is never used`, so leaving
them would have failed CI. Differential clippy against upstream main showed this
as the only new lint the branch introduces. Removing them takes with them:

  - StackTracker, the old evaluator's recorder, and StatusContext::new, its only
    caller. StatusContext itself stays: the validate reporters still destructure
    it and generic_summary.rs is live.
  - common_test_helpers.rs, whose only content was a DummyEval implementing the
    trait
  - a DummyEval in parser_tests.rs, constructed once into `let _dummy` and never
    used

324 lib tests, 0 failed, 3 ignored. test_command 19/19 -- both *_verbose tests
now pass, where they were 17/2 before.
Formatting only, no behaviour change. `cargo fmt --check` is a CI gate
(pr.yml:46-54, actions-rust-lang/rustfmt@v1) and this branch was failing it.

Verified this is genuinely unformatted branch code and not a rustfmt version
artifact: upstream main at 57bbdbf passes `cargo fmt --check` cleanly under the
same rustfmt 1.97, so the only files it can rewrite are ones this stack changed.

Two of the six files -- eval_context.rs and outcome_tests.rs -- come from
feat/status-type-migration rather than from this branch, so that branch is
failing the same gate on its own.

324 lib tests, 0 failed. cargo fmt --check exit 0.
`Reporter` had two methods: `report`, taking `&[&StatusContext]`, and `report_eval`,
taking an `EventRecord`. Only the second is called. `report` was the old evaluator's
reporting entry point and nothing has invoked it since the new recorder replaced
it -- several implementations were already `_`-prefixed stubs returning `Ok(())`,
which is what a vestigial required method decays into.

Removed the trait method and its eight implementations. Two traits in the same
files also have a `report`, and both are live: `GenericReporter::report` and the
`report(&mut self) -> Result<i32>` on the structured reporters. They are kept, and
the removal was filtered on whether the parameter list mentions `StatusContext`
rather than on the method name, because name-matching removed live impls twice while
writing this.

This orphans the legacy reporting cluster rather than removing it: StatusContext and
EvaluationType are now unreferenced except by each other, along with
find_all_failing_clauses, extract_name_info, print_partition,
print_compliant_skipped_info, pprint_failed_sub_tree, and the CfnReporter /
SingleLineReporter / ConsoleReporter / StructuredSummary / DataOutput /
DataOutputNewForm / StructureType / SarifRule set. Deleting that cluster is the
follow-up; splitting it out keeps this commit to one reviewable question.

329 lib tests, 0 failed, 1 ignored. cargo fmt --check clean. Zero new clippy lints
against upstream main.
cfn_reporter.rs and console_reporter.rs existed only to implement the
Reporter::report method removed in the previous commit. Neither CfnReporter,
SingleLineReporter nor ConsoleReporter was ever constructed, and after that removal
the files were referenced by nothing but their own `pub mod` declarations in
reporters/validate/mod.rs -- verified by search before deleting rather than inferred
from the dead-code warnings, since those warnings say an item is unused and not that
its file is unreferenced.

The live validate reporting path is unaffected: GenericSummary is constructed at
helper.rs and validate.rs and implements report_eval, which is the method the
evaluator actually calls.

Dead-code warnings 20 -> 15. What remains is the second half of the same cluster --
StructureType, StructuredSummary, DataOutput and DataOutputNewForm in common.rs,
SarifRule, the print_partition / print_compliant_skipped_info /
pprint_failed_sub_tree / extract_name_info / find_all_failing_clauses helpers, and
finally StatusContext and EvaluationType once nothing names them. Left for a
follow-up because each needs its own check that no live reporter destructures it,
and this commit is already one reviewable question.

329 lib tests, 0 failed, 1 ignored. cargo fmt --check clean. Zero new clippy lints
against upstream main.
Finishes what the previous two commits started. Removing Reporter::report left a
connected set of items reachable only from each other, and deleting them in
dependency order collapses it entirely:

  common.rs      extract_name_info, find_all_failing_clauses,
                 print_compliant_skipped_info, StructuredSummary and its impls,
                 StructureType, DataOutput, DataOutputNewForm
  summary_table  print_partition
  sarif.rs       SarifRule
  tracker.rs     deleted -- StatusContext was the whole file once StackTracker went
  mod.rs         EvaluationType and its Display impl, freed by StatusContext going;
                 add_variable_capture_index, a never-called default trait method
  operators.rs   UnaryComparator, a trait with no implementors
  exprs.rs       WhenGuardBlockClause, never constructed by the parser

Dead-code warnings 15 -> 0. Upstream main has 43, so this branch and its two
parents account for all of them.

Order mattered and was followed rather than guessed: the four dead functions were
the last things naming StatusContext, StatusContext was the last thing naming
EvaluationType, and each step was verified by search before deleting rather than
inferred from the warning list -- a warning says an item is unused, not that
removing it is safe.

The live validate path is untouched. GenericSummary is still constructed at
helper.rs and validate.rs, GenericReporter and the structured reporters keep their
own `report` methods, and report_eval remains the evaluator's entry point.

329 lib tests, 0 failed, 1 ignored. All other targets green except validate, which
fails 81/15 identically on upstream main -- a path artifact where the fixture
comparison strips a directory prefix and a /local/... checkout defeats it. cargo fmt
--check clean. Clippy 85 -> 20 errors under 1.97, zero of them new against main.
compare_write_buffer_with_file normalises captured output before comparing it to a
fixture, and did so by rewriting $HOME to ~ and then reducing ~/…/file.yaml to the
bare filename. Two bugs in that:

  - $HOME was substituted by plain substring match, so a checkout whose path merely
    *contains* $HOME was corrupted rather than normalised. With HOME=/home/u, the
    path /local/home/u/repo/tests/resources/x.yaml became
    /local~/repo/tests/resources/x.yaml, and reducing the tail then left /localx.yaml
    welded together. /local/home is a real layout, not a hypothetical.
  - a checkout outside $HOME produced no ~ at all, so the reduction never fired and
    every comparison saw a full absolute path.

Either way the failures looked like product bugs. On this checkout 15 of the 96
validate tests failed for this reason alone, which is why that whole target had been
written off as environmental.

Now anchored on CARGO_MANIFEST_DIR, which is where get_full_path_for_resource_file
roots every resource path, so no assumption about the checkout location survives.

Anchoring there rather than matching bare absolute paths is deliberate: the SARIF
fixtures contain //docs.oasis-open.org/…/sarif-schema-2.1.0.json, and a regex over
absolute paths ending in .json would reduce that URL to its basename. It is the only
slash-bearing file reference in guard/resources, so the distinction matters for
exactly one fixture and is easy to lose.

replace_home_directory_with_tilde is deleted rather than fixed. Nothing else called
it, and no fixture in guard/resources contains a tilde, so the substitution existed
only as a sentinel for the regex that no longer needs one.

validate target 81 passed/15 failed -> 95 passed/1 failed. The remaining failure,
test_validate_with_failing_complex_rule, is a real output difference that the
sanitisation bug was masking, not a path artifact; it is diagnosed separately.
The Windows install job failed on an unrelated pull request with

    {"message": "API rate limit exceeded for 20.9.183.48."}

from install-guard.ps1's release lookup. That address is a shared GitHub Actions
runner, and the anonymous API allows 60 requests an hour per source IP, counted
across everyone behind it. The same limit reaches real users: a corporate NAT, a
VPN, or a second person installing from the same office is enough.

Resolution now prefers whatever needs least from the caller. The gh CLI first,
when it is installed and authenticated, because it reuses credentials the caller
already has. Then the REST API with GITHUB_TOKEN when the environment supplies
one. Then anonymously, which is the only path the limit applies to. An explicit
version skips the lookup entirely, and install-guard.ps1 gains -Version for that,
matching -v in install-guard.sh.

Retries honour what the API says rather than guessing: retry-after on a secondary
limit, x-ratelimit-reset when the primary one is exhausted, exponential backoff
only when neither header is readable. Total waiting is capped at five minutes,
after which it stops and names GITHUB_TOKEN, gh auth login and -Version as the
ways out. Waiting for a primary reset can mean an hour, and an installer that
looks hung for an hour is worse than one that explains itself.

The token is passed to curl through a config file on stdin rather than argv. An
Authorization header on a command line is readable from ps by anyone else on the
host for the life of the request. It is only ever sent to api.github.com; the
release archive redirects to a separate download host.

install-guard.sh could not fail. Its err() exits, but it was reached from the left
side of a pipeline feeding a while-read loop, so exit 1 ended only that subshell
and the pipeline took its status from the loop, which had read nothing. A failed
lookup left the script exiting 0 with nothing installed -- which is why only the
Windows job went red when all three platforms hit the same wall. get_version's
result is assigned now, so the status propagates.

Get-ArchType read Win32_Processor through Get-WmiObject, a cmdlet PowerShell 6
removed; this workflow runs pwsh 7. It reads OSArchitecture from
RuntimeInformation instead, which is part of the framework, present in every
supported host, and exercisable outside Windows -- the WMI and CIM cmdlets are
both Windows-only, so neither could be tested before CI ran.

The install jobs now build cfn-guard from the branch under test, package it into
the release layout, and install that, asserting the installed binary's checksum
matches the one just built. They previously resolved the latest release and
installed it, so they tested the installer against a binary unrelated to the
change under review, and depended on the API that failed above.
install-guard.sh has been guarded by shellcheck since it was written;
install-guard.ps1 had no static analysis at all. That is how a Get-WmiObject call
survived in it: PowerShell 6 removed the WMI cmdlets and this workflow runs pwsh
7, so the script referenced a cmdlet its own CI shell does not have, and nothing
in the repository was looking.

PSScriptAnalyzer is the PowerShell counterpart, and it reports that call as
PSAvoidUsingWMICmdlet. Run against the script as it stood before the preceding
commit it finds two real problems and fourteen instances of one deliberate
choice; run against it now it finds none, so the gate starts clean rather than
with a backlog to grandfather in.

Configured by .github/PSScriptAnalyzerSettings.psd1 at Error, Warning and
Information, so a new finding fails the build. PSAvoidUsingWriteHost is the one
exclusion, with the reasoning recorded next to it: Get-ArchType and
Get-GuardVersion return their values through the pipeline, so routing progress
commentary to Write-Output as the rule advises would mix that text into their
return values and break them.

Runs on ubuntu rather than windows: the analyser is platform independent, and a
Linux runner is cheaper.

Ordering: this depends on the script fixes in the preceding commit. Applied to
main as it stands today the gate fails, on the WMI cmdlet and on Get-Versions
using a plural noun.
Two horizontal-ellipsis characters in comments this branch added, in a file whose merge-base has no
non-ASCII byte anywhere. `...` says the same thing and does not depend on the reader's terminal or
the diff viewer's encoding.

Found by scanning for non-ASCII rather than by reading, which is the only way this class shows up:
it does not fail a build, `typos` does not flag it, and a diff renders it as an ordinary character.
The same scan found 53 em dashes and a stray CJK character on the sibling PR.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant