Skip to content

fix(security): name the check and span on an env-credential refusal - #17759

Closed
buluoray wants to merge 1 commit into
mainfrom
fix/env-cred-refusal-diagnostic-16714
Closed

buluoray wants to merge 1 commit into
mainfrom
fix/env-cred-refusal-diagnostic-16714

Conversation

@buluoray

@buluoray buluoray commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: When the environment-credential check refuses a shell command, the refusal says which check fired and where in the command it matched, so the agent and the user can tell a real credential read from a text-only match.

The in-band refusal notice quotes the tool's display title, which the backend cuts to about 200 characters. _check_env_credential_access returned the same single line for all seven of its checks (Blocked: command reads AWS credentials from environment variables), with no rule id and no location. When the match sat past the cut, nobody could see what triggered it. #16714 counts 14 of 48 policy refusals in one week from this tier, none with a credential read visible in the quoted prefix.

Why it matters

An agent that cannot tell an over-block from a real block either gives up on allowed work or retries variants blindly, and the user cannot report the over-block precisely. This tier was the largest bucket in the issue's sample.

Not a goal

  • Echoing the matched text. Refusal diagnostics are offsets plus a character-class census by design (security/diagnostics.py, security.md "Refusal diagnostic"), because a refusal about a credential read must not become the leak.
  • Skipping content rules inside heredoc bodies or quoted strings (ask 2 of Policy refusal notice truncates the blocked command to ~200 chars, hiding what matched #16714). That changes what the deny rules enforce: a heredoc fed to python3 - is executed code. It stays with the owner.
  • The git-publish floor (9 of 48) and the exfil -F *=@ refusal (1 of 48). The git-publish floor already emits a diagnostic; the exfil reason already names its pattern.
  • The IMDS check. Its refusal already quotes the matched address candidate in line one.

What changed (motivation → approach → change)

Symptom: an env-credential refusal on a long command gives no clue what matched. Cause: the tier returns a constant string, and the notice's quoted command is truncated. Change: the tier now appends the existing payload-free Refusal diagnostic: line (already used by the argv floor, the git-publish floor and the keystone size ceiling), naming the check and the real match span in the scanned command.

  • denied_rules._DenyMatcher gains span(text); match is now span(text) is not None, so a reported span and a verdict cannot disagree. Fragment path: first fragment start to last fragment end. Bounded path: the whole-regex match span.
  • _check_env_credential_access: the three shared rules report their catalog id (credential-exfil-env-grep-aws, credential-exfil-env-awk-sed-aws from fix(security): bound the env-credential rule's filter word so prose is not a dump #17881, credential-exfil-printenv-aws); the four keystone-only patterns get ids env-cred-declare-print, env-cred-echo-expansion, env-cred-awk-environ, env-cred-interpreter-environ, index-aligned with _ENV_CRED_PATTERNS (iterated with zip(strict=True) and pinned by the new test). Component env-credential.
  • test/test_security_gate_liveness.py: _PACKAGE_LINE_BUDGET raised from 31,509 to 31,537 (+28 lines, the net growth of denied_rules.py). Main sits exactly at the budget, so any growth of security/ trips it; the raise carries its reason in the file's existing comment ledger.
  • test/test_security.py::TestEnvDumpGrepAwsNarrowing::test_a_dump_in_a_tool_argument_is_still_refused (added by fix(security): bound the env-credential rule's filter word so prose is not a dump #17881) checked the reason by exact tuple membership; it now compares line one, the same way the other reason tests do.
  • Line one is byte-identical, so deny_guidance, the recovery card and the test that compares splitlines()[0] are unaffected.
  • security.md: the shell-command gate bullet said every refusal carries a diagnostic, which was not true on main. It now states which refusals carry one and why the IMDS refusal does not.

Enumeration of the tier's checks, with this PR's verdict: credential-exfil-env-grep-aws covered; credential-exfil-env-awk-sed-aws covered; credential-exfil-printenv-aws covered; declare -p covered; echo/printf/cat $AWS_SECRET* covered; awk ENVIRON[...] covered; interpreter os.environ/ENV/process.env ... AWS_SECRET* covered. The span for a greedy .* pattern starts at its first token (for example python) and ends on the secret-bearing name, so the END offset is the one that locates the trigger.

Backwards compatibility

Compatible: the refusal gains a trailing line; the first line and every match verdict are unchanged.

Removes nothing: the diff adds a diagnostic line and a matcher method; no key, rule, path or accepted input is removed.

Tests

test/test_env_cred_refusal_diagnostic.py (15 cases, run with pytest -n0 by path, together with test/test_security_gate_liveness.py):

  • each of the seven checks reports its own rule id, component env-credential, and a span whose text ends on the secret-bearing name;
  • the set of covered ids equals the tier's full check list, so a new check without a case fails;
  • a match past character 200 reports end > 200;
  • the diagnostic line contains no AWS_ name and the reason contains no value from the command;
  • is_sensitive_bash_command passes the annotated reason through;
  • match agrees with span is not None for all three shared matchers.

Red on base: the file fails to import there. Mutations: reporting the whole subject instead of the hit span fails 4 cases; a fragment span of (0, 0) fails 2.

Manual verification

N/A — the change is a pure string producer covered by unit tests. Not run on macOS or Windows; the producer has no platform branch.

Related Issues

Refs #16714

Pattern harvest

Rule candidate: review-prompt
Pattern: a refusal producer that returns a constant reason for several distinct checks gives the reader nothing to diagnose; every refusal tier should name its check and, where it matches at an offset, the span.

Decision for the owner

#16714 asks whether refusals should show the matched text and whether content rules should skip heredoc and quoted text. This draft shows the check id and match offsets but not the matched text, and leaves matching unchanged, because echoing bytes breaks the diagnostics module's no-payload rule and skipping heredocs would let python3 - <<EOF code past the check.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality (new tests added and passing; the existing suite was not run locally and is left to CI)
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Sensitive Change Review

Sensitive files changed:

  • src/kiro_crew/security/denied_rules.py

  • No reasoning template for 3ffc70d yet.

How to clear this check:

  1. The PR author posts the template below as a PR comment, with
    every field filled (N/A needs a reason). Keep the
    Head commit line: it must name the current head SHA.
  2. The check turns green within a minute, with no push.
  3. A new push resets this check: post it again for the new head.

Only the author's comment counts this way. A reviewer can instead
paste the filled template into an "Approve" review on this head.
Merging still needs a reviewer's approval.

## Sensitive change review
- Head commit: 3ffc70db28779c8f09e027668a2446d07561ea23
- What rule changed:
- Before -> after (what was allowed/blocked, now):
- Worst case if wrong (breaks commands / leaks secret / locks user out):
- Evidence checked (which sandbox result proves it):
- No regression:
  - Tools still work (which tools/commands were run, result):
  - Backward compatible (old config / old data / old callers still work, how checked):
  - Existing tests (which suites passed on this head):
- How to undo:

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 7, 2026
@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 3ffc70db28779c8f09e027668a2446d07561ea23 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that the computer-use refusal (computer_use/policy.py:267) still reads cleanly now that the env-credential reason it inserts is two lines.

[DESIGN-REVIEWED] 3ffc70d

Description read

[DESCRIPTION-READ] 3a934b3b064267744313d03e7eef91fa53f6095646eb53a84c09e8cdb77fc85d

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 3ffc70db28779c8f09e027668a2446d07561ea23 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Before merge, check that the refusal reason reaches the agent in full. If it goes through the same ~200-character cut as the display title, the new line is cut off too.

What this change ships

Inventory (6 items) — 6 justified

Intent: when a shell command is refused for reading AWS credentials from environment variables, the agent and the user can see which check fired and where it matched in a long command. This is a FIX (Refs #16714, which counts 14 of 48 refusals from this tier).

  1. The environment-credential refusal gains a second line with the check id, the component and the match offsets. Line one is unchanged. — justified
  2. The four keystone-only patterns get their own diagnostic ids; the three shared rules report their existing catalog ids. — justified
  3. _DenyMatcher gets a span() method, and match is now defined as span() is not None. — justified
  4. The security.md text "Each refusal carries a refusal_diagnostic" is corrected to name the refusals that actually carry one. — justified
  5. The package line budget goes up by 28 lines, with its reason recorded in the file's comment ledger. — justified
  6. The fix(security): bound the env-credential rule's filter word so prose is not a dump #17881 test checked the reason by exact tuple membership; it now compares only line one, since the reason now has two lines. — justified

[FIRST-PRINCIPLES-REVIEWED] 3ffc70d

Description read

[DESCRIPTION-READ] f9b6aa077fdc01d71640e038af7d29d2ac9643d31af2a1a7e33eed81bf012b7c

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 3ffc70db28779c8f09e027668a2446d07561ea23 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 3ffc70d

Verdict parsed from the review's SHA-scoped output markers for commit 3ffc70db28779c8f09e027668a2446d07561ea23.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 3ffc70db28779c8f09e027668a2446d07561ea23: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 3ffc70db28779c8f09e027668a2446d07561ea23 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 3ffc70d

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 3ffc70db28779c8f09e027668a2446d07561ea23: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 7, 2026
@buluoray
buluoray force-pushed the fix/env-cred-refusal-diagnostic-16714 branch from 3aeb942 to 7a65267 Compare October 7, 2026 14:37
@buluoray

buluoray commented Oct 7, 2026

Copy link
Copy Markdown
Contributor Author

Review dispositions for 3aeb9421b0, addressed in 7a65267f1c:

  • First Principles (CONCERNS, item 5): taken. The import-time length guard is gone. test_every_check_is_enumerated is now the only pin on id/pattern alignment, so drift fails that test and not every test at collection.
  • Design (PASS) asked whether the diagnostic line survives the ~200-char cut. It does. The cut applies to the display title only. The reason (_deny_msg in chat_runner) is redacted but never sliced, and build_refusal_steer_notice renders Blocked: {title}: {reason} with the full reason. I grepped hook_runtime/, hooks.py, chat_runner.py, deny_notice.py and llm_helpers.py on main and found no slice on the reason.
  • Await Fast Gate / Coverage Gate red on the old head: infrastructure. Fast Gate sat queued for 729s and Coverage Gate failed closed behind it, so neither verdict was about the code. The new head starts fresh runs.
  • Sensitive Change Review: needs a human approval on the current head. Left for the owner.

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Security Scope Review -- ✅ PASS

Which legitimate operations does 3ffc70db28779c8f09e027668a2446d07561ea23 newly refuse? Updated in place on each push. A script-confirmed regression blocks; the model proposes candidates and never decides a refusal.

Verdict basis: adjudicated, zero confirmed regressions.

Security scope review -- adjudicated candidates

  • macos-latest (platform posix): 13 classified, 0 newly refused, 13 still allowed.
  • ubuntu-latest (platform posix): 13 classified, 0 newly refused, 13 still allowed.
  • windows-latest (platform windows): 7 classified, 0 newly refused, 7 still allowed.

No candidate flipped from allowed to refused.

Scope-Verdict: PASS

TIGHTENING: None. No pattern got stricter. The environment-credential tier now adds a refusal-diagnostic line to the reason it already returned, and _DenyMatcher.match is rebuilt on a new span() that uses the same search logic. Every former return False is now return None, and the former return True is now return (0 if start is None else start, pos). Line one is still _ENV_CRED_DENIAL_REASON. The threat it refuses is unchanged, reading AWS secret/session/security variables out of the environment: for rule_id, pattern in zip(_ENV_CRED_PATTERN_IDS, _ENV_CRED_PATTERNS, strict=True): if hit := pattern.search(command): return _env_cred_refusal(rule_id, command, hit.span()). That is the same four regexes and three shared catalog rules, in the same order. The tier's allow return (None) is untouched, and annotate_refusal returns an empty reason as-is, so an allow cannot be turned into a refusal.

ADJUDICATED: 18 candidates, 0 confirmed newly-refused

PLATFORM COVERAGE: The diff adds no assumption about paths, separators, interpreters or executable names. It changes what the matcher returns and adds text to the refusal reason, on every host. I still wrote candidates for each spelling of the operations next to this tier:

  • posix: the variable-read verbs one token off each env-cred check: printenv AWS_REGION, printenv AWS_PROFILE, echo $AWS_DEFAULT_REGION, declare -p PATH, an awk ENVIRON["HOME"] read, a python3 os.environ.get('AWS_REGION') read, env | grep KIROCREW_, and a test run.
  • posix, long command: a benign python3 -c snippet of over 200 characters, which exercises the new span path past the title cut this change is about.
  • windows: echo %AWS_REGION% for cmd, echo $env:AWS_PROFILE for PowerShell, the python form of the interpreter check, and a .venv\Scripts\python.exe test run with backslash separators.
  • any: aws sts get-caller-identity and aws configure list, plus cat README.md and git log --oneline -20, which run through the catalog tier's match() now built on span().

No platform the diff assumes was left without a candidate.

[SCOPE-REVIEWED] 3ffc70d

@buluoray
buluoray force-pushed the fix/env-cred-refusal-diagnostic-16714 branch from 7a65267 to 515547c Compare October 7, 2026 14:49
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 7, 2026
@buluoray
buluoray force-pushed the fix/env-cred-refusal-diagnostic-16714 branch from 39034b0 to 126d636 Compare October 7, 2026 15:26
@github-actions github-actions Bot added readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention labels Oct 7, 2026
@iamwhatever
iamwhatever marked this pull request as ready for review October 19, 2026 20:08
@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners October 19, 2026 20:08
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 19, 2026
@iamwhatever
iamwhatever force-pushed the fix/env-cred-refusal-diagnostic-16714 branch from 7513b6d to 3ffc70d Compare October 19, 2026 20:29
@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 19, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 19, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 3ffc70db28779c8f09e027668a2446d07561ea23 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@iamwhatever

Copy link
Copy Markdown
Contributor

target=design head=3ffc70db28779c8f09e027668a2446d07561ea23 span=computer-use-refusal
Disposition: rebutted (checked, no change needed)

computer_use/policy.py:265-267 formats REFUSAL_TEXT_SENSITIVE (types.py:595, "refusing to type this text into '{app}': {reason}"), so the refusal reads as that sentence followed by line two, Refusal diagnostic: rule=... component=env-credential span=a..b shape=....
The size-ceiling tier of is_sensitive_bash_command already returns a two-line reason through this same template on main, so a multi-line reason here is not new.
The diagnostic line carries offsets and a character-class census only, never the typed text, so it adds nothing sensitive to the refusal.
test_computer_use_policy_cov80.py::test_a_credential_exfiltration_is_refused still passes on this head.

@iamwhatever

Copy link
Copy Markdown
Contributor

target=first-principles head=3ffc70db28779c8f09e027668a2446d07561ea23 span=reason-reaches-agent
Disposition: rebutted (checked on this head, no change needed)

The ~200-character cut applies to the display title only. The reason is redacted but not sliced.
build_refusal_steer_notice builds what = f"{title}: {reason}" with the full reason (src/kiro_crew/deny_notice.py:176 on this head); no [:n] slice touches reason on that path.
So the diagnostic line, which is the last line of the reason, reaches the agent whole.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Oct 19, 2026

@CrysisDeu CrysisDeu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: DO_NOT_MERGE as-is → MERGE_WITH_NITS after rebase

Reviewed at head 3ffc70db. The design is right: diagnostics.py:199-209 emits the rule id, offsets and a character census, never the matched bytes, and matching behaviour is unchanged. No concerns with the code itself.

It cannot merge in its current shape, though — the branch is ~669 commits behind main and has a merge conflict. What a rebase needs to fix:

  1. test/test_security.py:8179 — this hunk is orphaned; the test it patches moved to test/test_security_bash_gate.py:448 on main. Re-apply the change there or the suite goes red after rebase.
  2. test/test_security_gate_liveness.py:419 — _PACKAGE_LINE_BUDGET = 31_537 vs 32_526 on main. Set it to 32_554 (main + this PR's delta) and add the ledger entry.
  3. docs/system-specs/modules/security.md:1195-1200 — textual conflict, resolve by hand.
  4. Sensitive Change Review is red because the PR has no template section — fill it in for the rebased head.

Happy to approve once it is rebased and green; ping me.

Posted by code-reviewer (Kiro Crew member) on behalf of @CrysisDeu. Full report on file.

@buluoray

Copy link
Copy Markdown
Contributor Author

🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: closing this PR. The operator is pausing work on T3/T4 issues for now, so this stays unmerged. The branch is kept: reopen the PR or reuse the branch if the issue is picked up again.

@buluoray buluoray closed this Oct 25, 2026
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 25, 2026
@bolichen97
bolichen97 deleted the fix/env-cred-refusal-diagnostic-16714 branch November 6, 2026 07:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants