You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(validation): validate findings after a scan - #1007
Add optional standalone validation after a completed source scan, with supplemental assessments that remain tied to the scanned target and evidence.
Changes
Add cs scan [repository] --validate, disabled by default. Preserve the chosen plugin directory or ZIP, Python runtime, Cyber access program, and captured knowledge inputs.
Save completed assessments and a supplemental report across interruption and retry without rewriting the scan's findings or severity policy.
Bind cached assessments to the recorded target, source, evidence, request settings, and the exact standalone instructions used by validation.
Merge the reviewed main snapshot, reuse its shared parsers/runtime helpers, and retain read-only, offline standalone validation and current source-scan outcomes.
Keep validation failures and interruptions in full-output JSON/JSONL error envelopes while retaining completed scan data and diagnostic text.
Keep completed target checks and original validation failures intact when temporary plugin cleanup or later client shutdown fails. Mandatory credential/runtime cleanup failures remain errors when there is no earlier validation failure.
Testing
All five portable plugin checks, SDK types and formatting, and the bundled plugin build passed.
Focused API validation/workflow tests: 92 passed. Both affected CLI test files: 218 passed, including validation failure/interruption envelopes and cleanup-failure regressions. Five new regression cases failed before the cleanup fixes and passed afterward. The inline-instruction cache regression and all three affected empty-target cases also passed.
Python finding-workflow/scan-validation tests: 37 passed. MCP executor/artifact-recovery tests, including fresh and resumed workers: 182 passed.
SDK/MCP types, full formatting, and the CI helper build were repeated after the cleanup fixes: passed. Three fresh native reviews and an independent verifier found no introduced actionable issues on the corrected head.
Final-head full SDK runs with seed 12345 and explicit randomization are running in separate neutral test environments. An earlier run was stopped after 3,641 passes and 11 checkout-layout-dependent authentication fixture failures; all 12 focused authentication controls passed from an isolated working directory. A separate unchanged TUI fixture fails when the neutral working directory resolves different dependency copies; its whole test file passes 9/9 with the frozen SDK dependencies. The final randomized run uses those frozen dependencies. No full-suite or live-model completion is claimed here.
Risk and rollout
Existing scans are unchanged unless --validate is selected. Validation is supplemental and not cost-tracked, so it cannot be combined with --dry-run, --mock, or a scan cost limit such as --max-cost. A validation failure returns exit code 2 while preserving completed scan data and assessments, and prevents subsequent patch generation. The cache contract changes once to invalidate assessments made under older instruction binding.
Public disclosure review
Reviewed the current branch, title, description, commits, changes, PR discussion, available Actions logs and artifacts, and current Code Scanning reports. No sensitive additions were found. Historical Code Quality analysis bodies remain outside this review, so the full-disclosure attestations remain unchecked.
No customer, partner, prospect, or user identities, data, or identifying details are included.
No credentials, personal data, private source, scan findings, or nonpublic links or tickets are included.
I reviewed the branch name, title, description, commits, changes, comments, logs, screenshots, attachments, and links for public disclosure.
mldangelo-oai
changed the title
feat(validation): add optional follow-up validation for scan findings
feat(validation): validate findings after a scan
Oct 12, 2026
Cached implicit evidence bypasses the output-directory ownership and permission checks. sourceSnapshot(..., { evidence: true }) fingerprints descendants but not the root directory's mode, UID, or secure ancestry, so changing a default evidence directory to 0777 (or changing its owner/ancestor permissions) still allows this assessment to be reused. Explicit outputs are protected because #prepareLocalInputs calls validateOutputDir, but inputs.outputDir === null never validates the cached assessment.outputDir. Validate that cached path with the existing output-directory checks before accepting its digest; for an invalid implicit path, treat the cache as unusable and regenerate into a private directory.
This evidence fingerprint does not detect a Windows directory junction being retargeted. The helper stops traversal at reparse-point directories, but directory_content_digest() only records os.readlink() when stat.S_ISLNK() is true; a junction is reported as a directory with a reparse tag, so its target is omitted from the digest. A cached assessment can therefore be reused after an evidence link changes destinations; hash the reparse target itself while continuing not to traverse it.
Sparse checkout can bypass committed-diff validation
The target check can accept a committed-diff checkout that was made sparse after the scan. worktree_content_digest() hashes git diff HEAD and untracked files, but Git reports no diff for omitted skip-worktree files, so the HEAD and digest still match while validation runs with tracked source files missing. Recheck for S entries from git ls-files -t -z (or otherwise bind the full-checkout state) before accepting committed-diff validation.
The reason will be displayed to describe this comment to others. Learn more.
Reviewed the changes since the previous reviewed head in two passes, using exact source at 783128384d1d and prior review evidence. Unchanged portions of the large branch were not exhaustively re-audited.
Approved. No new actionable findings in the reviewed scope.
Validation: CI-helper TypeScript compilation and plugin bundling passed locally; native binaries came from CI with matching compiled source inputs. Focused Python tests passed: test_workbench_scan_validation.py (10 passed in 12.22s). The full cross-platform suite was not run locally; Windows execution remains unverified here.
The reason will be displayed to describe this comment to others. Learn more.
Reviewed the actual diff at 783128384d1d32b400b78828e9862e280b6caee5 in three independent passes, including the API/CLI tests and the saved-source and target checks. Optional validation keeps the target and evidence tied to the recorded scan, preserves completed supplemental assessments on partial failure, and prevents patch generation after a validation failure. Budget, dry-run, and mock restrictions are explicit; validation cost is documented separately.
No blocking findings. Static review only; no model calls, scan execution, or suites rerun.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
zcrab-oai
left a comment
alandelong-oai
left a comment
Summary
Add optional standalone validation after a completed source scan, with supplemental assessments that remain tied to the scanned target and evidence.
Changes
cs scan [repository] --validate, disabled by default. Preserve the chosen plugin directory or ZIP, Python runtime, Cyber access program, and captured knowledge inputs.Testing
12345and explicit randomization are running in separate neutral test environments. An earlier run was stopped after 3,641 passes and 11 checkout-layout-dependent authentication fixture failures; all 12 focused authentication controls passed from an isolated working directory. A separate unchanged TUI fixture fails when the neutral working directory resolves different dependency copies; its whole test file passes 9/9 with the frozen SDK dependencies. The final randomized run uses those frozen dependencies. No full-suite or live-model completion is claimed here.Risk and rollout
Existing scans are unchanged unless
--validateis selected. Validation is supplemental and not cost-tracked, so it cannot be combined with--dry-run,--mock, or a scan cost limit such as--max-cost. A validation failure returns exit code 2 while preserving completed scan data and assessments, and prevents subsequent patch generation. The cache contract changes once to invalidate assessments made under older instruction binding.Public disclosure review
Reviewed the current branch, title, description, commits, changes, PR discussion, available Actions logs and artifacts, and current Code Scanning reports. No sensitive additions were found. Historical Code Quality analysis bodies remain outside this review, so the full-disclosure attestations remain unchecked.