Skip to content

test(monitor): pin the shared provider-error vocabulary - #14079

Merged
iamwhatever merged 2 commits into
mainfrom
fix/provider-error-classifier-pins-14074
Sep 26, 2026
Merged

iamwhatever merged 2 commits into
mainfrom
fix/provider-error-classifier-pins-14074

Conversation

@chenmingwei23

Copy link
Copy Markdown
Contributor

What is the problem?

classify_provider_error_text in src/kiro_crew/monitoring/pull_request.py is the
one classifier that turns a provider CLI's stderr into a ProviderErrorKind. Two
call sites raise PullRequestProviderError carrying its answer:

  • src/kiro_crew/monitoring/gitlab_merge_request.py:214
  • src/kiro_crew/monitoring/azure_devops_pull_request.py:196

No test in the tree references the function. On be067a3503,
grep -rn classify_provider_error_text test/ returns nothing, so every marker in
its four buckets and its TRANSIENT fall-through are asserted nowhere. The
per-provider error tests that do exist cover one kind each -- GitLab HTTP 404
and Azure DevOps HTTP 403 -- and neither provider has a rate-limit case.

Two details of the issue as filed do not hold against current main, and they
narrow what this change does:

  • The rate-limit markers on main are
    ("http 429", "rate limit", "too many requests", "throttled"). secondary rate
    and abuse detection are not among them, and
    git log -S'secondary rate' -- src/kiro_crew/monitoring/pull_request.py is
    empty. That widening lives on the open branch of feat(monitor): the gh-pr reader fetches, the wake judge decides #13759, which edits this same
    function.
  • src/kiro_crew/probes/gh_pr.py does not call this classifier on main. The
    GitHub monitors classify through src/kiro_crew/monitoring/github_provider_errors.py
    into src/kiro_crew/taskq/adapters/github.py, whose own marker tuple carries
    secondary rate and abuse detection and is pinned in
    test/test_dependency_signals.py. The shared classifier therefore has two
    consumers, not three.

The coverage gap is real either way, and it is why the widening on #13759 can land
or be reverted with nothing going red.

Why this issue matters to the user

The kind is what a GitLab or Azure DevOps watch reports and retries on. TRANSIENT
is the fall-through, so a marker that stops matching does not fail loudly: the
refusal is reported as a generic transient error and retried on that basis. With no
assertion anywhere, a marker removed or re-bucketed for one provider's benefit
changes how the other providers report refusals, and the regression surfaces only
as a mis-reported error on a provider nobody was testing.

How our fix solves it

Test-only, no src/ change -- which also keeps this clear of #13759's edits to the
same function.

Pinned once at the classifier, where every consumer meets it:

  • each of the 18 markers across the four buckets asserts the kind it yields;
  • an unrecognised diagnostic asserts the TRANSIENT fall-through.

The table is spelled out as literals rather than read back from the module, because
a table derived from the set under test agrees with any set.

On top, one throttling-refusal case per provider, through that provider's own call
site: test_gitlab_throttling_refusal_is_rate_limited and
test_azure_throttling_refusal_is_rate_limited. Both use wording that carries no
status code and no rate limit phrase, so the shared throttled marker is the
only thing that can yield RATE_LIMITED, and the case fails if that marker goes
away. The Azure case also has to survive that call site's missing-extension branch,
which reads the same text first.

Deliberately out of scope: the pins catch a marker removed or re-bucketed, which is
the regression named. A marker ADDED without a pin still does not go red; catching
that needs the buckets promoted to named constants in src/, which would collide
with #13759. The retryability question the issue raises is untouched.

What tests we did

Both changed files, -n0, isolated KIROCREW_HOME:
test/test_pull_request_readiness.py test/test_source_provider_pull_request_monitors.py
-> 119 passed. 21 of those nodes are new: 18 vocabulary rows, the fall-through,
and the two provider cases.

Nine mutations of src/kiro_crew/monitoring/pull_request.py, each caught. Every
mutation ran behind a control run of the same two files reporting 119 passed
unmutated, and behind a git hash-object comparison proving the file actually
changed; the file was restored from a pristine copy held outside the repository, not
with git checkout.

Mutation Failures Caught by
drop throttled 3 throttled row + both provider cases
drop rate limit 1 API rate limit exceeded row
drop http 429 1 HTTP 429 row
drop too many requests 1 Too Many Requests row
rate-limit bucket answers TRANSIENT 6 all 4 rate rows + both provider cases
drop revoked token 1 revoked token supplied row
drop does not exist 1 does not exist row
drop access denied 1 access denied row
fall-through answers AUTHORIZATION 1 the fall-through case

Every one of those failures is a new test; no pre-existing test fired for any
mutation, which is the coverage gap measured directly.

Also clean: black --check (2 files unchanged), flake8 (no output),
scripts/check_comment_history.py against base be067a3503 (passed, after its own
--test selftest passed), and prepare-pr preflight.py reporting STATUS: READY.

Any other suggestions on the work

  • feat(monitor): the gh-pr reader fetches, the wake judge decides #13759 widens this function's rate-limit set with secondary rate and
    abuse detection. With these pins in place that widening arrives as a declared
    change to a covered function, and wants two more rows in the table here.
  • The GitHub path keeps its own vocabulary in src/kiro_crew/taskq/adapters/github.py.
    Two marker sets now answer the same question in two places; whether they should
    converge is a design question this change does not take.
  • The retryability subtraction the issue mentions -- transport retry keyed on a
    private regex rather than on these kinds -- is a behaviour change needing its own
    pins, and stays deferred.

Closes #14074

…sumer

classify_provider_error_text decides the ProviderErrorKind that the GitLab and
Azure DevOps call sites raise, and no test in the tree references the function.
That leaves the marker set asserted nowhere, so narrowing it for one provider's
benefit is green for the others.

Pin each marker's kind once at the classifier, pin the unrecognised-text
fall-through to TRANSIENT, and add one throttling-refusal case per provider
through its own call site.
@chenmingwei23
chenmingwei23 requested review from a team and CrysisDeu September 26, 2026 01:48
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 26, 2026
@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of bbb7f7367116dadf4eaeba1e4aa2732e6d7f844c and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] bbb7f73

False positive or not applicable? A repository writer can comment:
/ai-review override gpt bbb7f7367116dadf4eaeba1e4aa2732e6d7f844c: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of bbb7f7367116dadf4eaeba1e4aa2732e6d7f844c — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

When #13759 lands its secondary rate/abuse detection widening, confirm two matching rows are added to this vocabulary table.

[DESIGN-REVIEWED] bbb7f73

@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed bbb7f7367116dadf4eaeba1e4aa2732e6d7f844c — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] bbb7f73

Verdict parsed from the review's SHA-scoped output markers for commit bbb7f7367116dadf4eaeba1e4aa2732e6d7f844c.

False positive or not applicable? A repository writer can comment:
/ai-review override fable bbb7f7367116dadf4eaeba1e4aa2732e6d7f844c: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 26, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

CI disclosure: the single red lane on this head is inherited from main, not produced by this diff.

Backend Tests (Windows) (6) fails with:

test/test_spawn_approval_channel_delivery_2381.py::TestASilentSendFailureFallsThroughAtOnce::test_it_registers_no_wait_so_the_window_is_never_spent
AssertionError: assert ['wait'] == []

That is #14060, which records the same deterministic assertion on a pristine origin/main checkout, on Linux and Windows shard 6 alike. The same lane fails with byte-identical assertion text and identical values on open PRs #14080, #14078 and #14077.

This PR is test-only and changes two files -- test/test_pull_request_readiness.py and test/test_source_provider_pull_request_monitors.py. Neither is under messaging/ and neither reaches that test, so no change here can affect it. Rebasing cannot help either while the failure reproduces on main itself.

Board on 80cb7fc368c68fcdd132fce9e6c87008898d0634, deduplicated to the newest attempt per lane name and per (name, external_id) -- both views agree -- 91 lanes: 80 success, 10 skipped, 1 failure (the lane above), 0 cancelled, 0 still in flight. GPT 5.6, Opus 5 and Design Review each left a marker naming this exact head, none with a blocking finding. PR Readiness is red because it reads that one lane.

CI checks out the pull-request merge ref, so re-running that single lane once #14060's fix is on main clears it without any change to this branch.

Brings in 694f619 (#14056), which fixes the test that made Backend Tests
(Windows) (6) red on this pull request. push_guard requires origin/main to be an
ancestor of HEAD before a push, and a rebase would need a force push, so the base
is taken as a merge.
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 26, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

The second commit is a merge of origin/main, not a code change.

The lanes on the previous head resolved against the pull-request merge commit for the OLD base be067a3503, whose tree still holds the test that #14060 fixed. The Windows job log says so outright: HEAD is now at 94ced51 Merge 80cb7fc368... into be067a350358.... That is why Backend Tests (Windows) (6) kept failing on test_it_registers_no_wait_so_the_window_is_never_spent, a name current main no longer has. Neither a job re-run nor the base branch advancing refreshes a merge ref -- only a new pull-request event does -- and prepare-pr's push_guard requires origin/main to be an ancestor of HEAD before any push, which a rebase could satisfy only with a force push. So main is taken as a merge.

The reviewed content is unchanged. git diff origin/main...HEAD is still exactly test/test_pull_request_readiness.py and test/test_source_provider_pull_request_monitors.py, and both files are byte-identical to 80cb7fc368. On the merged tree: 119 passed with -n0, black --check and flake8 clean, and the comment-history gate passes against base 694f61983e.

For the record, none of the previous head's other reds were findings:

  • Resolve desktop matrix -- API rate limit exceeded for installation, 02:35:22 UTC.
  • Design Review, First Principles Review, UX Review, Opus 5 Review -- all failed closed on "Could not read this PR's comments after 3 attempts", in that same rate-limit window. Design Review's own annotation calls its missing verdict "errored / overloaded / incomplete -- NOT blocking".
  • GPT 5.6 Review -- its annotation says it produced a verdict for that head but the slot already held one from the earlier run, so the published [GPT-REVIEWED] 80cb7fc368... verdict with no blocking findings still stood.
  • Backend Tests (3.12, 4) -- test_aws_control_backup.py::TestALostRunWriteDoesNotReUploadForever::test_run_identity_concurrent_failure_then_recovery, which also failed on fix(artifacts): refuse stale content writes with 409 instead of silently overwriting #14075 and passes on main (run 36210419640 on 694f61983e is green).

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 26, 2026
@iamwhatever
iamwhatever enabled auto-merge (squash) September 26, 2026 06:23
@iamwhatever
iamwhatever merged commit daa8472 into main Sep 26, 2026
95 checks passed
@iamwhatever
iamwhatever deleted the fix/provider-error-classifier-pins-14074 branch September 26, 2026 06:24
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

provider errors: the shared classifier's rate-limit vocabulary was widened for GitHub, changing GitLab and Azure DevOps kinds with no pin

2 participants