Retire the superseded evaluator, and stop the install scripts failing on a shared API quota - #719
Open
awsmadi wants to merge 10 commits into
Open
Conversation
guard/src/rules/evaluate.rs was the pre-eval.rs evaluator. Nothing on the CLI
path reached it: the compiler reported 15 items in it as never used, RootScope::new
was called only from evaluate_tests.rs, and its one apparent outside consumer --
path_value.rs's QueryResolver::select -- takes &dyn EvaluationContext, the old
context trait. select's only non-test callers were itself, evaluate.rs, and
MetadataAppender, which is constructed only in its own test. One connected dead
component, not a scattering.
Removed:
- evaluate.rs (1342) and evaluate_tests.rs (2345)
- QueryResolver and PathAwareValue::select, plus the four helpers only select
used: map_error_or_empty, map_some_or_error_all, retrieve_index, accumulate.
eval_context.rs already has its own retrieve_index and accumulate as free
functions, so the new engine loses nothing.
- aws_meta_appender.rs and its test
Kept: EvaluationType and StatusContext, which live reporter code still reads --
GenericSummary is constructed at helper.rs:66 and validate.rs:704.
Display for GuardNamedRuleClause moved to exprs.rs beside the Display impls for
its sibling clause types. It lived in evaluate.rs but eval.rs formats it, so it
was the one piece of that file the live evaluator depended on.
evaluate_tests.rs was also a near-duplicate suite: of five test names sampled,
all five exist in eval_tests.rs against the live evaluator. The 39 tests dropped
here (32 evaluate_tests, 4 path_value_tests, 2 values_tests, 1 appender) all
exercised only the deleted engine; the same semantics stay covered by
eval_context_tests for query retrieval, eval_tests::filter_based_* and
test_map_keys_function for filters, and rule_test_type_blocks for type blocks.
358 -> 319 lib tests, 0 failed. 4693 lines deleted.
eval_conjunction_clauses built a user-visible string from std::any::type_name::<T>(): `context` becomes the `Context=` label on the Disjunction node in --verbose output, and four fixtures under guard/resources pin it exactly. type_name's output is explicitly unspecified -- std documents that it "must not be considered to uniquely identify a type" and may change between compiler versions, and it did. Newer rustc renders the elided lifetime, so cfn_guard::rules::exprs::GuardClause became ...::GuardClause<'_>, and test_data_file_verbose and test_with_rules_dir_verbose fail on any toolchain other than the 1.77.2 pinned in rust-toolchain.toml. Verified failing at both 320251c and 57bbdbf (upstream main) under rustc 1.97, so it is pre-existing and not introduced by this branch. Truncating at the first `<` restores the pre-change spelling for every T and is stable under further rendering changes, since only the generic/lifetime portion varies. Deliberately not changing the strings themselves: that a Rust module path is user-facing output at all is a real wart, but rewriting it is a visible output change that belongs with the fixtures. Also removes the EvaluationContext and Evaluate traits, which the previous commit left with no production implementations. `cargo clippy -- -D warnings` -- the gate at pr.yml:94 -- rejects `trait EvaluationContext is never used`, so leaving them would have failed CI. Differential clippy against upstream main showed this as the only new lint the branch introduces. Removing them takes with them: - StackTracker, the old evaluator's recorder, and StatusContext::new, its only caller. StatusContext itself stays: the validate reporters still destructure it and generic_summary.rs is live. - common_test_helpers.rs, whose only content was a DummyEval implementing the trait - a DummyEval in parser_tests.rs, constructed once into `let _dummy` and never used 324 lib tests, 0 failed, 3 ignored. test_command 19/19 -- both *_verbose tests now pass, where they were 17/2 before.
Formatting only, no behaviour change. `cargo fmt --check` is a CI gate (pr.yml:46-54, actions-rust-lang/rustfmt@v1) and this branch was failing it. Verified this is genuinely unformatted branch code and not a rustfmt version artifact: upstream main at 57bbdbf passes `cargo fmt --check` cleanly under the same rustfmt 1.97, so the only files it can rewrite are ones this stack changed. Two of the six files -- eval_context.rs and outcome_tests.rs -- come from feat/status-type-migration rather than from this branch, so that branch is failing the same gate on its own. 324 lib tests, 0 failed. cargo fmt --check exit 0.
`Reporter` had two methods: `report`, taking `&[&StatusContext]`, and `report_eval`, taking an `EventRecord`. Only the second is called. `report` was the old evaluator's reporting entry point and nothing has invoked it since the new recorder replaced it -- several implementations were already `_`-prefixed stubs returning `Ok(())`, which is what a vestigial required method decays into. Removed the trait method and its eight implementations. Two traits in the same files also have a `report`, and both are live: `GenericReporter::report` and the `report(&mut self) -> Result<i32>` on the structured reporters. They are kept, and the removal was filtered on whether the parameter list mentions `StatusContext` rather than on the method name, because name-matching removed live impls twice while writing this. This orphans the legacy reporting cluster rather than removing it: StatusContext and EvaluationType are now unreferenced except by each other, along with find_all_failing_clauses, extract_name_info, print_partition, print_compliant_skipped_info, pprint_failed_sub_tree, and the CfnReporter / SingleLineReporter / ConsoleReporter / StructuredSummary / DataOutput / DataOutputNewForm / StructureType / SarifRule set. Deleting that cluster is the follow-up; splitting it out keeps this commit to one reviewable question. 329 lib tests, 0 failed, 1 ignored. cargo fmt --check clean. Zero new clippy lints against upstream main.
cfn_reporter.rs and console_reporter.rs existed only to implement the Reporter::report method removed in the previous commit. Neither CfnReporter, SingleLineReporter nor ConsoleReporter was ever constructed, and after that removal the files were referenced by nothing but their own `pub mod` declarations in reporters/validate/mod.rs -- verified by search before deleting rather than inferred from the dead-code warnings, since those warnings say an item is unused and not that its file is unreferenced. The live validate reporting path is unaffected: GenericSummary is constructed at helper.rs and validate.rs and implements report_eval, which is the method the evaluator actually calls. Dead-code warnings 20 -> 15. What remains is the second half of the same cluster -- StructureType, StructuredSummary, DataOutput and DataOutputNewForm in common.rs, SarifRule, the print_partition / print_compliant_skipped_info / pprint_failed_sub_tree / extract_name_info / find_all_failing_clauses helpers, and finally StatusContext and EvaluationType once nothing names them. Left for a follow-up because each needs its own check that no live reporter destructures it, and this commit is already one reviewable question. 329 lib tests, 0 failed, 1 ignored. cargo fmt --check clean. Zero new clippy lints against upstream main.
Finishes what the previous two commits started. Removing Reporter::report left a
connected set of items reachable only from each other, and deleting them in
dependency order collapses it entirely:
common.rs extract_name_info, find_all_failing_clauses,
print_compliant_skipped_info, StructuredSummary and its impls,
StructureType, DataOutput, DataOutputNewForm
summary_table print_partition
sarif.rs SarifRule
tracker.rs deleted -- StatusContext was the whole file once StackTracker went
mod.rs EvaluationType and its Display impl, freed by StatusContext going;
add_variable_capture_index, a never-called default trait method
operators.rs UnaryComparator, a trait with no implementors
exprs.rs WhenGuardBlockClause, never constructed by the parser
Dead-code warnings 15 -> 0. Upstream main has 43, so this branch and its two
parents account for all of them.
Order mattered and was followed rather than guessed: the four dead functions were
the last things naming StatusContext, StatusContext was the last thing naming
EvaluationType, and each step was verified by search before deleting rather than
inferred from the warning list -- a warning says an item is unused, not that
removing it is safe.
The live validate path is untouched. GenericSummary is still constructed at
helper.rs and validate.rs, GenericReporter and the structured reporters keep their
own `report` methods, and report_eval remains the evaluator's entry point.
329 lib tests, 0 failed, 1 ignored. All other targets green except validate, which
fails 81/15 identically on upstream main -- a path artifact where the fixture
comparison strips a directory prefix and a /local/... checkout defeats it. cargo fmt
--check clean. Clippy 85 -> 20 errors under 1.97, zero of them new against main.
compare_write_buffer_with_file normalises captured output before comparing it to a
fixture, and did so by rewriting $HOME to ~ and then reducing ~/…/file.yaml to the
bare filename. Two bugs in that:
- $HOME was substituted by plain substring match, so a checkout whose path merely
*contains* $HOME was corrupted rather than normalised. With HOME=/home/u, the
path /local/home/u/repo/tests/resources/x.yaml became
/local~/repo/tests/resources/x.yaml, and reducing the tail then left /localx.yaml
welded together. /local/home is a real layout, not a hypothetical.
- a checkout outside $HOME produced no ~ at all, so the reduction never fired and
every comparison saw a full absolute path.
Either way the failures looked like product bugs. On this checkout 15 of the 96
validate tests failed for this reason alone, which is why that whole target had been
written off as environmental.
Now anchored on CARGO_MANIFEST_DIR, which is where get_full_path_for_resource_file
roots every resource path, so no assumption about the checkout location survives.
Anchoring there rather than matching bare absolute paths is deliberate: the SARIF
fixtures contain //docs.oasis-open.org/…/sarif-schema-2.1.0.json, and a regex over
absolute paths ending in .json would reduce that URL to its basename. It is the only
slash-bearing file reference in guard/resources, so the distinction matters for
exactly one fixture and is easy to lose.
replace_home_directory_with_tilde is deleted rather than fixed. Nothing else called
it, and no fixture in guard/resources contains a tilde, so the substitution existed
only as a sentinel for the regex that no longer needs one.
validate target 81 passed/15 failed -> 95 passed/1 failed. The remaining failure,
test_validate_with_failing_complex_rule, is a real output difference that the
sanitisation bug was masking, not a path artifact; it is diagnosed separately.
The Windows install job failed on an unrelated pull request with
{"message": "API rate limit exceeded for 20.9.183.48."}
from install-guard.ps1's release lookup. That address is a shared GitHub Actions
runner, and the anonymous API allows 60 requests an hour per source IP, counted
across everyone behind it. The same limit reaches real users: a corporate NAT, a
VPN, or a second person installing from the same office is enough.
Resolution now prefers whatever needs least from the caller. The gh CLI first,
when it is installed and authenticated, because it reuses credentials the caller
already has. Then the REST API with GITHUB_TOKEN when the environment supplies
one. Then anonymously, which is the only path the limit applies to. An explicit
version skips the lookup entirely, and install-guard.ps1 gains -Version for that,
matching -v in install-guard.sh.
Retries honour what the API says rather than guessing: retry-after on a secondary
limit, x-ratelimit-reset when the primary one is exhausted, exponential backoff
only when neither header is readable. Total waiting is capped at five minutes,
after which it stops and names GITHUB_TOKEN, gh auth login and -Version as the
ways out. Waiting for a primary reset can mean an hour, and an installer that
looks hung for an hour is worse than one that explains itself.
The token is passed to curl through a config file on stdin rather than argv. An
Authorization header on a command line is readable from ps by anyone else on the
host for the life of the request. It is only ever sent to api.github.com; the
release archive redirects to a separate download host.
install-guard.sh could not fail. Its err() exits, but it was reached from the left
side of a pipeline feeding a while-read loop, so exit 1 ended only that subshell
and the pipeline took its status from the loop, which had read nothing. A failed
lookup left the script exiting 0 with nothing installed -- which is why only the
Windows job went red when all three platforms hit the same wall. get_version's
result is assigned now, so the status propagates.
Get-ArchType read Win32_Processor through Get-WmiObject, a cmdlet PowerShell 6
removed; this workflow runs pwsh 7. It reads OSArchitecture from
RuntimeInformation instead, which is part of the framework, present in every
supported host, and exercisable outside Windows -- the WMI and CIM cmdlets are
both Windows-only, so neither could be tested before CI ran.
The install jobs now build cfn-guard from the branch under test, package it into
the release layout, and install that, asserting the installed binary's checksum
matches the one just built. They previously resolved the latest release and
installed it, so they tested the installer against a binary unrelated to the
change under review, and depended on the API that failed above.
install-guard.sh has been guarded by shellcheck since it was written; install-guard.ps1 had no static analysis at all. That is how a Get-WmiObject call survived in it: PowerShell 6 removed the WMI cmdlets and this workflow runs pwsh 7, so the script referenced a cmdlet its own CI shell does not have, and nothing in the repository was looking. PSScriptAnalyzer is the PowerShell counterpart, and it reports that call as PSAvoidUsingWMICmdlet. Run against the script as it stood before the preceding commit it finds two real problems and fourteen instances of one deliberate choice; run against it now it finds none, so the gate starts clean rather than with a backlog to grandfather in. Configured by .github/PSScriptAnalyzerSettings.psd1 at Error, Warning and Information, so a new finding fails the build. PSAvoidUsingWriteHost is the one exclusion, with the reasoning recorded next to it: Get-ArchType and Get-GuardVersion return their values through the pipeline, so routing progress commentary to Write-Output as the rule advises would mix that text into their return values and break them. Runs on ubuntu rather than windows: the analyser is platform independent, and a Linux runner is cheaper. Ordering: this depends on the script fixes in the preceding commit. Applied to main as it stands today the gate fails, on the WMI cmdlet and on Get-Versions using a plural noun.
Two horizontal-ellipsis characters in comments this branch added, in a file whose merge-base has no non-ASCII byte anywhere. `...` says the same thing and does not depend on the reader's terminal or the diff viewer's encoding. Found by scanning for non-ASCII rather than by reading, which is the only way this class shows up: it does not fail a build, `typos` does not flag it, and a diff renders it as an ordinary character. The same scan found 53 em dashes and a stray CJK character on the sibling PR.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Base:
main. Independent of #717 — nothing here touches evaluator semantics.Three unrelated-looking pieces that share one theme: things CI was not actually checking. A dead
evaluator nobody had removed, an installer whose failures could not surface, and a PowerShell
script with no static analysis at all.
9 commits, 35 files, +632 / −6,036.
What prompted it
The Windows install job failed on an unrelated pull request:
20.9.183.48is a shared GitHub Actions runner. The anonymous GitHub API allows 60 requests anhour per source IP, counted across everyone behind it, and
install-guard.ps1resolved thelatest release through it unauthenticated. The same ceiling reaches real users: a corporate NAT, a
VPN, or a colleague installing from the same office is enough.
Pulling that thread turned up two further problems, both of which had been invisible precisely
because nothing was looking.
The commits
Retiring the superseded evaluator
1.
6dc98baRetire the superseded evaluator and its query engine — deletesevaluate.rs,evaluate_tests.rs, the oldpath_valuequery engine and its tests, andaws_meta_appender.−4,693 lines. This code was replaced by the
evalmodule and had no remaining callers.2.
144baacMake verbose output independent of the compiler that built it — verbose outputvaried with build configuration, so what an operator saw depended on how the binary was compiled
rather than on the rules being evaluated.
3.
e3fea72Apply rustfmt — formatting only, from the deletions above.4.
4cab372Remove the deadReporter::reportmethod — the trait method had no liveimplementations left once the old evaluator was gone. −349 lines across nine reporters.
5.
448317bDelete the two unreachable validate reporters —cfn_reporterandconsole_reporterwere no longer reachable from any command path. −472 lines.6.
2e0fc4cDelete the rest of the legacy reporting cluster; zero dead-code warnings — theremaining fragments, taking the crate from 43 dead-code warnings to none.
Making the test suite able to fail
7.
3445508Anchor test path sanitisation on the crate dir, not$HOME— the integrationharness rewrote absolute paths by substituting
$HOME, which only works when the checkout livesunder the home directory. Anywhere else the substitution missed and 15 expected-output tests failed
on path noise alone. Anchored on
CARGO_MANIFEST_DIRinstead.The installers
8.
0df9eaeStop the install scripts failing on an anonymous GitHub API quota — the substanceof the fix, covering several distinct problems:
ghCLI first when installed and authenticated, since it reusescredentials the caller already has; then the REST API with
GITHUB_TOKENif the environmentsupplies one; then anonymously, the only path the limit applies to. An explicit version skips the
lookup entirely, and
install-guard.ps1gains-Versionfor that, matching-vininstall-guard.sh.retry-afteron a secondary limit,x-ratelimit-resetwhen the primary one is exhausted, exponential backoff only when neitherheader is readable. Blind doubling would retry straight into an empty quota and report a network
error for what is really a quota problem.
appears hung for an hour is worse than one that explains itself. Past the cap it stops and names
GITHUB_TOKEN,gh auth loginand-Versionas the ways out.install-guard.shcould not fail.err()exits, but it was reached from the left side of apipeline feeding a
while readloop, soexit 1ended only that subshell and the pipeline tookits status from the loop, which had read nothing. A failed lookup left the script exiting 0
with nothing installed. That is why only the Windows job went red: all three platforms hit the
same wall and only the one without this bug reported it.
get_version's result is assigned now.Get-WmiObjectwas removed in PowerShell 6, and this workflow runs pwsh 7. Architecturedetection reads
OSArchitecturefromRuntimeInformationinstead — part of the framework,present in every supported host, and exercisable off Windows, which the WMI and CIM cmdlets are
not.
Authorizationheader on a command line is readable from
psby anyone else on the host for the life of therequest. It is only ever sent to
api.github.com; the release archive redirects to a separatedownload host.
layout, install that, and assert the installed binary's checksum matches the one just built.
They previously resolved the latest release and installed it, so they tested the installer
against a binary unrelated to the change under review, over the API that failed above.
Static analysis for the PowerShell script
9.
583c3b8Gateinstall-guard.ps1on PSScriptAnalyzer —install-guard.shhas been guardedby shellcheck since it was written; the PowerShell script had nothing. That is how the
Get-WmiObjectcall survived: the script referenced a cmdlet its own CI shell does not have, andnothing in the repository was looking.
Configured by
.github/PSScriptAnalyzerSettings.psd1at Error, Warning and Information, so a newfinding fails the build.
PSAvoidUsingWriteHostis the single exclusion, with the reasoningrecorded beside it:
Get-ArchTypeandGet-GuardVersionreturn their values through thepipeline, so routing progress commentary to
Write-Outputas the rule advises would mix that textinto their return values and break them.
Run against the script as it stood before commit 8, the analyser reports two real problems; run
against it now, none. The gate starts clean rather than with a backlog to grandfather in.
Verification
cargo fmt --checkcargo clippy -- -D warningscargo clippy --all-targets -- -D warningstyposshellcheck install-guard.shvalidate96/0Two failure paths were exercised rather than assumed: a missing archive exits 1, and an unreachable
API exits 1 after retrying with the guidance above. Both exited 0 before.
The PowerShell logic that cannot run on a Linux runner was unit-tested separately — architecture
mapping,
-Versionbypassing the API,retry-afterprecedence, reset-based waiting, headerfallback, and the
ghpath — 9 checks, all passing.Merge order against #717
These two touch 17 files in common and both are individually mergeable, so whichever lands second
will need a rebase. One file is worth knowing about before that happens:
guard/src/commands/reporters/validate/cfn_reporter.rs, which this PR deletes as unreachable and#717 modifies.
The resolution is to take the deletion. #717's change to that file is a mechanical signature update,
HashSet<String>toSkippedRules, made so the file kept compiling when that type changed, plus acomment noting that this path carries no record tree and so reports skips without reasons. It is not
a fix, and it has no effect on behaviour, because neither
CfnReporternorSingleLineReporterisever constructed. Every other overlapping file is modified by both and conflicts only where the two
edits sit near each other.
Measured with
git merge-treerather than inferred from GitHub's mergeable flag, which reports oneach PR against
mainseparately and so says nothing about this.