Skip to content

fix(cron): record error when a run lost calls to a hook policy refused - #40147

Open
leozhad wants to merge 1 commit into
kirodotdev:mainfrom
leozhad:leozh/cron-mixed-refusal-status
Open

leozhad wants to merge 1 commit into
kirodotdev:mainfrom
leozhad:leozh/cron-mixed-refusal-status

Conversation

@leozhad

@leozhad leozhad commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: a cron run that lost tool calls because a policy refused to run the agent spec's PreToolUse hook shows as Error and names those calls, even when another call ran or another call hit the security gate, instead of reading ok or naming only the security block.

A scheduled agent job runs an agent whose spec has a PreToolUse hook on file reads. The profile bound to scheduled jobs turns script_hooks off, so the hook is refused and every file read is blocked, as #34927 decided. The model then runs one other call, such as a shell ls, and that call is approved. The run records last_status: ok, its history row reads success, and record_success() resets the failure counter. The job reads healthy while its work does not happen. When the same run also has one security-gate block, it reads error, but its reason and its Blocked: banner name only that block, never the calls the refused hook blocked, as a comment on #39973 shows.

Why it matters

A refused hook blocks the same calls on every run, and no adapting brings them back. In #39973, each of the 11 silent scheduled jobs that ran in the three days the audit log covered lost its reads this way, and every one still read ok. Only the audit log and the model's own prose said otherwise.

Not a goal

What changed (motivation → approach → change)

The run's tool calls last_status before last_status now Failure counter
one or more approved, none security-blocked, one or more blocked by a hook a policy refused to run ok, and a failure streak is reset error, the reason names those calls does not move
some security-blocked, one or more blocked by a hook a policy refused to run error, the reason names only the security block error, the reason names the security block, then the hook-blocked calls does not move (unchanged)
any other mix unchanged unchanged unchanged

The cause: the hook's result records that a policy refused to run it (ScriptHookResult.refused), but that flag stopped at permission_pre_tool_block, which returns only a reason string. So the gate logged every spec-hook block as mechanism: spec_hook, and the run's tally could not tell a refused hook from a hook's own verdict. With an approved call and no security block, _apply_gate_verdict recorded a success.

The flag now reaches the tally. permission_pre_tool_block in hook_runtime/hook_dispatch.py takes an optional on_policy_refused callback, so its other callers are unchanged. The cron spec floor (CronTurnHooks.refusal) returns (reason, policy_refused). _resolve_permission audits that block under a new mechanism, spec_hook_refused. on_tool_gate gets a fifth argument, hook_refused. _GateTally counts those calls and keeps the first three by name, and a new arm records error, moves neither counter and banners the result. The reason reads N of M tool call(s) blocked because policy refused to run a hook:, then each call with its hook and refusal. It does not match the Partial pattern, so the Schedule page shows Error. A delete_after_run one-shot is kept and paused, as any run ending in error is. A partly security-blocked run that also lost such calls keeps its security-gate reason and adds . N tool call(s) blocked because policy refused to run a hook: and those calls; the two parts share the 500-character line so that neither head is cut, and the reason still opens with the Partial wording. Both new reasons reuse the per-call equal share from #40103, so a long title still gives way before its rule. The gate-verdict table in learn-cron-dashboard.md gains the row.

Backwards compatibility

Breaking: stream_and_collect now calls on_tool_gate with five arguments, not four, and CronTurnHooks.refusal returns a (reason, policy_refused) tuple, not a string. Writer sweep on origin/main @ f8558e3: src/kiro_crew/slack/gateway_runtime/cron_turn.py passes _GateTally.note on both cron paths, which takes the new argument with a default; src/kiro_crew/apps/agent_turn_sdk.py and every other stream_and_collect caller pass no on_tool_gate; the one caller of CronTurnHooks.refusal is _resolve_permission, through stream_and_collect(spec_hooks=...); the 19 test callbacks in test_llm_helpers_tool_gate.py, test_llm_helpers_spawn_brief_scan.py and test_kas_cron_spec_hooks.py take the fifth argument. permission_pre_tool_block gains a keyword with a default, so its four other callers (channel, messaging, sub-agent, task runner) fit, and test_hooks_composition_contract.py re-pins its shape.

The audit row of a spec-hook block that a policy refused to run reads mechanism: spec_hook_refused; a hook's own block keeps spec_hook. A partly security-blocked run's last_error gains the hook part after its unchanged opening.
Reader: src/kiro_crew/llm_helpers_parts/turn_stream.py:stream_and_collect -- cron -- reads the mechanism through _resolve_permission's on_decision funnel and maps spec_hook_refused to hook_refused (test_a_hook_a_policy_refused_to_run_is_told_apart_from_its_own_verdict); no other src or website reader compares the mechanism value
Reader: website/src/utils/cronPartlyBlocked.ts:isPartlyBlocked -- crew page -- matches only the reason's opening words, which are unchanged (test_a_security_block_names_the_hook_blocked_calls_after_it, test_the_schedule_page_tells_a_partial_block_from_a_total_one)

Tests

Proof: PROVEN test/test_cron_refusal_status.py::TestSchedulerContract::test_a_one_shot_whose_hook_policy_refused_to_run_is_kept, test/test_hooks_composition_contract.py::test_the_moved_names_keep_their_base_shapes, test/test_kas_cron_spec_hooks.py::test_a_cron_turn_gates_its_permission_requests_on_the_specs_pre_tool_hooks[2-True], test/test_kas_cron_spec_hooks.py::test_a_cron_turn_gates_its_permission_requests_on_the_specs_pre_tool_hooks[0-False], test/test_kas_cron_spec_hooks.py::test_a_cron_turn_gates_its_permission_requests_on_the_specs_pre_tool_hooks[-1-False] (head 795ab9e)

With the source changes reverted, 42 of the 289 tests in the changed test files fail: 17 in test_cron_refusal_status.py (the 15 tests of the new class other than its three ..._stays_ok controls, the one-shot test, and the Schedule page pattern test), 6 in test_kas_cron_spec_hooks.py, 18 of the 20 tests in test_llm_helpers_tool_gate.py and test_llm_helpers_spawn_brief_scan.py whose callbacks take the fifth argument (the raising-callback and discarded-retry tests pass either way, since a callback's error is swallowed), and the shape pin. The changed test files passed 10 of 10 runs at -n 4 --dist loadgroup --max-worker-restart=2 (289 tests), and the two cron files once in an order shuffled by random.Random(39973) (80 of 80).

  • test_kas_cron_spec_hooks.py::test_a_hook_a_policy_refused_to_run_is_told_apart_from_its_own_verdict: through stream_and_collect with CronTurnHooks, a hook a policy refused and a hook that exited 2 both block the call; only the first reaches on_tool_gate with hook_refused true and is audited as spec_hook_refused. test_an_unreadable_spec_blocks_a_cron_turns_tools_as_in_a_dashboard_chat also pins hook_refused false for a spec that cannot be read.
  • TestRefusedHookBesideAnApprovedCall::test_the_run_records_error, test_it_spends_no_failure_budget and test_it_does_not_reset_a_failure_streak: such a run records error; five of them leave the job enabled and unpaused; one after two blocked runs keeps consecutive_failures at 2.
  • test_the_reason_names_those_calls_with_the_hook, test_another_unresolved_call_is_counted_but_not_named and test_the_reason_names_at_most_three: the reason's exact text, with other unresolved calls counted in M but not named, at most three names, and N more, within 500 characters.
  • test_the_notification_names_the_lost_calls, test_it_reaches_the_sequential_path and test_the_schedule_page_shows_it_as_error_not_partial: the delivered body opens with the same reason on the single-agent and agent_sequence paths, and the reason does not match the Partial pattern.
  • test_any_other_refusal_beside_an_approved_call_stays_ok (hook exited 2, policy_deny, timed out), test_with_no_approved_call_the_no_approved_arm_is_unchanged and test_a_security_block_keeps_the_partial_arm: the unchanged arms.
  • test_a_security_block_names_the_hook_blocked_calls_after_it (with and without an approved call) and test_both_parts_of_a_long_partial_reason_keep_their_heads: a partly security-blocked run names the security block, then the hook-blocked calls, in last_error and the banner; with five long calls of each kind both heads and both and N more clauses survive within 500 characters. test_a_hook_part_that_fits_is_not_cut_beside_a_long_gate_part: beside a long security-gate part, a hook part that fits is shown whole, rules included. TestPartialBlock::test_the_schedule_page_tells_a_partial_block_from_a_total_one also checks that such a mixed reason still matches the Partial pattern.
  • TestSchedulerContract::test_a_one_shot_whose_hook_policy_refused_to_run_is_kept: a delete_after_run one-shot in this arm is kept and paused, with this arm's reason.
  • test_hooks_composition_contract.py::test_the_moved_names_keep_their_base_shapes: the re-pinned shape of permission_pre_tool_block.

Manual verification

One cron turn driven through the real gate path: CronTurnHooks.prepare over a spec with a PreToolUse hook on fs_read, then stream_and_collect with three permission requests (a file read, a shell ls, a second file read), then _annotate_partial_block and _apply_gate_verdict on a job whose failure streak is 2. In case policy-refuses-hook the capabilities.script_hooks check is forced to deny with profile denies: capability disabled, so run_script_hook returns its real governance refusal. Case hook-and-security-block adds a fourth request, a shell rm -rf /, which the security gate refuses. In case hook-exits-2 the hook command runs and exits 2. Each run used a throwaway data home with KIROCREW_PROFILE=standalone. On the head, the policy case wrote two spec_hook_refused audit rows and the control case two spec_hook rows.

Real-machine verification

Machine: own computer, Linux (Amazon Linux 2023, kernel 6.12), x86_64, 96 vCPU / 743 GiB
Before (main f8558e3): env -i PATH=/usr/bin:/bin HOME=$H KIROCREW_HOME=$H/.kiro/crew KIROCREW_PROFILE=standalone PYTHONPATH=src python repro_39973.py policy-refuses-hook -> approved=['r2'] rejected=['r1', 'r3'], last_status=None consecutive_failures=0, last_error=None, banner=False (the scheduler then writes ok, and the streak of 2 was reset)
After (head 795ab9e): env -i PATH=/usr/bin:/bin HOME=$H KIROCREW_HOME=$H/.kiro/crew KIROCREW_PROFILE=standalone PYTHONPATH=src python repro_39973.py policy-refuses-hook -> approved=['r2'] rejected=['r1', 'r3'], last_status='error' consecutive_failures=2, last_error='2 of 3 tool call(s) blocked because policy refused to run a hook: Read File (a1 spec hook (PreToolUse #2): Blocked by governance policy: profile denies: capability disabled), Read File (…same…)', banner=True
Before, same command with hook-and-security-block: approved=['r2'] rejected=['r1', 'r3', 'r4'], last_status='error' consecutive_failures=2, last_error='1 of 4 tool call(s) blocked by the security gate: Running: rm -rf / (Blocked by security policy: rm -rf /.*)', banner=True
After, same command with hook-and-security-block: approved=['r2'] rejected=['r1', 'r3', 'r4'], last_status='error' consecutive_failures=2, last_error='1 of 4 tool call(s) blocked by the security gate: Running: rm -rf / (Blocked by security policy: rm -rf /.*). 2 tool call(s) blocked because policy refused to run a hook: Read File (a1 spec hook (PreToolUse #2): Blocked by governance policy: profile denies: capability disabled), Read File (…same…)', banner=True
Control, same command with hook-exits-2, on both: approved=['r2'] rejected=['r1', 'r3'], last_status=None consecutive_failures=0, last_error=None, banner=False
Cleanup: own computer; the throwaway data homes and the base worktree were deleted.

repro_39973.py
from __future__ import annotations

import asyncio
import sys

import kiro_crew.hooks as hooks_mod
from kiro_crew import llm_helpers
from kiro_crew.acp.types import AcpEvent
from kiro_crew.agent_sdk import spec_hooks
from kiro_crew.agent_sdk.backends import ACP_BACKEND_KAS
from kiro_crew.agent_sdk.capabilities import capabilities_for
from kiro_crew.cron import CronJob, CronSchedule
from kiro_crew.cron_service.turn_hooks import CronTurnHooks
from kiro_crew.hooks import HOOK_EVENT_PRE_TOOL_USE, ScriptHook
from kiro_crew.providers.base import EVENT_COMPLETE, EVENT_PERMISSION_REQUEST
from kiro_crew.slack.gateway import _annotate_partial_block, _apply_gate_verdict, _GateTally


class Provider:
    #: Set by the "hook-and-security-block" case: one call the security gate refuses.
    with_security_block = False

    def __init__(self) -> None:
        self.approved: list[str] = []
        self.rejected: list[str] = []
        self.session_id = "acp-1"
        self.cwd = "/tmp"
        self.capabilities = capabilities_for(ACP_BACKEND_KAS)

    async def stream(self, message, **_kw):
        yield AcpEvent(
            kind=EVENT_PERMISSION_REQUEST, request_id="r1", title="Read File", tool_name="fs_read"
        )
        yield AcpEvent(
            kind=EVENT_PERMISSION_REQUEST,
            request_id="r2",
            title="Running: ls /tmp",
            tool_name="execute_bash",
        )
        yield AcpEvent(
            kind=EVENT_PERMISSION_REQUEST, request_id="r3", title="Read File", tool_name="fs_read"
        )
        if self.with_security_block:
            yield AcpEvent(
                kind=EVENT_PERMISSION_REQUEST,
                request_id="r4",
                title="Running: rm -rf /",
                tool_name="execute_bash",
                tool_input='{"command": "rm -rf /"}',
            )
        yield AcpEvent(kind=EVENT_COMPLETE, stop_reason="end_turn")

    async def approve_tool(self, request_id, **_kw):
        self.approved.append(request_id)

    async def reject_tool(self, request_id):
        self.rejected.append(request_id)


def run(case: str) -> None:
    command = "echo 'not today' >&2; exit 2" if case == "hook-exits-2" else "true"
    hook = ScriptHook(
        id="spec:a1:PreToolUse:2",
        name="a1 spec hook (PreToolUse #2)",
        event=HOOK_EVENT_PRE_TOOL_USE,
        matcher="fs_read",
        command=command,
        spec_origin="user",
    )

    async def fake_turn_spec_hooks(provider, agent, **_kw):
        return spec_hooks.TurnSpecHooks([hook], "/tmp", False, True, ())

    async def not_stale(provider, agent, **_kw):
        return False

    spec_hooks.turn_spec_hooks = fake_turn_spec_hooks
    spec_hooks.hook_projection_stale = not_stale
    Provider.with_security_block = case == "hook-and-security-block"
    if case in ("policy-refuses-hook", "hook-and-security-block"):
        hooks_mod._script_hooks_capability_denied = (
            lambda session_key="": "profile denies: capability disabled"
        )

    async def turn() -> tuple[Provider, _GateTally]:
        provider = Provider()
        cron_hooks = await CronTurnHooks.prepare(
            provider, "a1", "cron:j1", "go", fresh_session=False
        )
        tally = _GateTally()
        await llm_helpers.stream_and_collect(
            provider,
            "go",
            approval_policy=llm_helpers.ToolApprovalPolicy.AUTO_APPROVE,
            on_tool_gate=tally.note,
            session_key="cron:j1",
            spec_hooks=cron_hooks,
        )
        return provider, tally

    provider, tally = asyncio.run(turn())
    job = CronJob(
        id="j1", name="briefing", message="go", schedule=CronSchedule(kind="every", every_secs=900)
    )
    job.consecutive_failures = 2
    result = _annotate_partial_block("I couldn't make today's briefing.", tally)
    _apply_gate_verdict(job, tally)
    print(f"case={case}")
    print(f"  approved={provider.approved} rejected={provider.rejected}")
    print(f"  last_status={job.last_status!r} consecutive_failures={job.consecutive_failures}")
    print(f"  last_error={job.last_error!r}")
    print(f"  banner={result.startswith('Blocked: ')}")


if __name__ == "__main__":
    run(sys.argv[1])

Related Issues

Closes #39973

Naming the hook-blocked calls in a partly security-blocked run follows the suggestion in a comment on #39973.

Builds on #37604, merged for #35101. #34927 decided that a hook a profile refuses keeps blocking the call. #28603 also touches how a cron run names a lost call; whichever lands second folds onto the other.

Sibling sweep

Swept: git grep -n -e 'permission_pre_tool_block' -e 'spec_floor' -e 'on_tool_gate' -e '_GateTally' -e '_apply_gate_verdict' -- src website/src
Sibling: src/kiro_crew/slack/gateway_runtime/cron_verdict.py -- changed here
Sibling: src/kiro_crew/slack/gateway_runtime/runtime_charge.py -- changed here
Sibling: src/kiro_crew/cron_service/turn_hooks.py -- changed here
Sibling: src/kiro_crew/slack/gateway_runtime/cron_turn.py -- not affected -- both cron agent paths pass _GateTally.note and call the shared _annotate_partial_block and _apply_gate_verdict
Sibling: src/kiro_crew/channel_spec_hooks.py -- not affected -- a channel turn records no run status; its refusal is audited as rejected_spec_hook and the turn goes on
Sibling: src/kiro_crew/messaging/tool_hooks.py -- not affected -- the same channel gate, with no run verdict to read the flag
Sibling: src/kiro_crew/subagent_manager/run.py -- not affected -- a sub-agent's refusal goes through the permission ladder and its own completion, not a cron run status
Sibling: src/kiro_crew/task_executor.py -- not affected -- a task step logs the refusal and has no cron run status
Sibling: website/src/utils/cronPartlyBlocked.ts -- not affected -- the new reason does not match its Partial pattern, so the run shows as Error

Pattern harvest

Not generalizable: one flag stopped at a seam that returns only a string, and the verdict arms it feeds are all listed in one spec table.

Checklist

  • A Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...); PR Hygiene does not count commits
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

@leozhad
leozhad requested review from a team and KiroCrewCaptain as code owners November 20, 2026 21:04
@leozhad
leozhad requested a review from cruisercohen November 20, 2026 21:04
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Nov 20, 2026
@leozhad
leozhad force-pushed the leozh/cron-mixed-refusal-status branch from 64a4090 to 795ab9e Compare November 20, 2026 23:30
@github-actions github-actions Bot added merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Nov 20, 2026
@leozhad
leozhad force-pushed the leozh/cron-mixed-refusal-status branch from 795ab9e to 568cb4c Compare November 21, 2026 02:05
@github-actions github-actions Bot added merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Nov 21, 2026
A cron run whose agent spec has a PreToolUse hook that a policy refuses
to run (for example a cron profile that turns script_hooks off) loses
every call that hook guards. When the model then ran one other call, the
verdict saw an approved call, no security block, and recorded ok, so the
job read healthy while its work did not happen.

The flag that says a policy refused the hook (ScriptHookResult.refused)
now reaches the run's tally: permission_pre_tool_block reports it
through an optional callback, the cron spec floor returns it beside the
reason, the gate logs that refusal under mechanism spec_hook_refused,
and on_tool_gate carries it as a fifth argument. A run with an approved
call, no security block and at least one such call records error with
its own reason head, banners the result, and moves neither counter.
Every other refusal beside an approved call keeps the run ok.

A partly security-blocked run that also lost such calls names them
after its security-gate part, under the hook head without its "of M",
so the reason no longer hides the calls that stopped the job.

Closes kirodotdev#39973
@leozhad
leozhad force-pushed the leozh/cron-mixed-refusal-status branch from 568cb4c to 89c17cf Compare November 21, 2026 04:08
@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 21, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cron: a run that lost calls because policy refused to run a spec hook records ok when any other call ran

1 participant