Skip to content

fix(workflows): return structured 500 on service-layer exceptions - #17347

Merged
iamwhatever merged 1 commit into
kirodotdev:mainfrom
chenmingwei23:fix/al2-glibc-renameat2-missing
Oct 7, 2026
Merged

iamwhatever merged 1 commit into
kirodotdev:mainfrom
chenmingwei23:fix/al2-glibc-renameat2-missing

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Make a deterministic failure in the workflow run/author endpoints return a diagnosable JSON error instead of an opaque HTTP 500.

On Amazon Linux 2 (glibc 2.26) a deterministic platform fault surfaced as an opaque aiohttp 500 from the workflow run endpoints. The renameat2 glibc wrapper symbol was absent (added in glibc 2.28), the workflow memory binding failed closed with NotImplementedError("atomic no-replace rename is unavailable"), and that exception propagated out of the handler with no JSON body — so the caller could not tell a deterministic fault from a transient one and backed off as if the service were unhealthy.

The underlying renameat2 syscall fallback already shipped in #11018, so the primary fault no longer reproduces. But the error-handling gap remains and is platform-independent: api_workflow_run, api_workflow_run_intent, api_workflow_run_rerun, and api_workflow_author call their service methods with no try/except, unlike the sibling api_workflow_run_definition.

Why it matters

Any future service-layer exception in these four handlers still becomes an undiagnosable 500 with no body. A structured {"error", "code"} response lets a caller log and classify the failure instead of guessing whether the service is transiently unhealthy.

Not a goal

What changed (motivation → approach → change)

Observed symptom: a deterministic fault escaped api_workflow_run_intent as a bodyless 500. Root cause: the four workflow mutation handlers call their service method with no exception boundary, unlike api_workflow_run_definition which wraps its call. Change: wrapped the service call in all four handlers (api_workflow_run, api_workflow_run_intent, api_workflow_run_rerun, api_workflow_author) with the same try/except Exception → logger.exception(...) → return _error(message, code, 500) pattern the sibling already uses. Each handler returns its own stable code (workflow_run_failed, workflow_run_intent_failed, workflow_rerun_failed, workflow_author_failed).

Backwards compatibility

Compatible: no behaviour change on the success path or on the existing validation (400/403/404/409/503) paths. The only difference is that an exception that previously escaped as a bare 500 now returns a structured JSON 500 with the same status code.

Removes nothing: the change only adds an exception boundary; no field, path, record, kind, key, or function is removed, renamed, or tightened.

Tests

Added test_mutation_handlers_return_structured_500_on_service_error to test/test_workflow_handler_json_contract.py: parametrized over all four handlers, each service method is made to raise, and the test asserts a 500 with a JSON {"error", "code"} body carrying the handler's stable code. The existing success-path and reject-non-object tests continue to pass. Full file: 17 passed.

Manual verification

PYTHONPATH=src pytest test/test_workflow_handler_json_contract.py -n 2 → 17 passed. N/A for UI — this change touches no frontend surface.

Related Issues

Fixes #17337

Pattern harvest

Rule candidate: review-prompt
Pattern: an aiohttp handler calling a service method with no try/except lets a deterministic fault escape as a bodyless 500 — a sibling handler in the same module already had the boundary, so this is a drift-from-established-pattern class, not a one-off.

@chenmingwei23
chenmingwei23 requested a review from a team as a code owner October 6, 2026 11:21
@chenmingwei23
chenmingwei23 requested a review from buluoray October 6, 2026 11:21
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running labels Oct 6, 2026
@github-actions

github-actions Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of f3daaaa734fd9549a72d7a82df524b4139b07749 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that the frontend's error handling for these four endpoints reads the new code. Otherwise "a caller can classify the failure" holds only for the server log.

[DESIGN-REVIEWED] f3daaaa

Description read

[DESCRIPTION-READ] b503918b82ffbb8eb62f629bd6287365648a2343be81e630c148a162a3695deb

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — 🟡 CONCERNS

Premise-level review of f3daaaa734fd9549a72d7a82df524b4139b07749 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

Point patch: a per-handler try/except in four handlers, while api_workflow_cancel's svc.cancel (another write) and two read calls in the same module stay unwrapped.

Not justified as shipped

  • Item 1 — symptom-level: the description calls the cause "the four workflow mutation handlers call their service method with no exception boundary". The deeper cause is that the dashboard chain (middleware_chain.py:203-238) has no /api/ exception→JSON layer. So each handler has to remember its own boundary, and this PR fixes four of them. I grepped svc\. in dashboard/handlers/workflows.py and found 3 call sites still unwrapped after this PR: svc.cancel (712), svc.result (605) and svc.list_runs (590). One of them is the mutation endpoint cancel, which the "four mutation handlers" framing leaves out. A chain-level fix is larger than this PR, so it is accepted as deferred.
    Clears when: ## What changed says the fix is per-handler (mechanism level), names cancel/result/list_runs as still unwrapped, and names the missing chain-level boundary as what is left.

What this change ships

Inventory (3 items) — 2 justified

Intent: give callers a JSON error they can diagnose, instead of a bare 500, when a workflow run or author service call raises. FIX (#17337, plus a test that fails on base).

  1. The author, run, run_intent and rerun endpoints now return a JSON {"error","code"} 500 when the service call fails — symptom-level, cancel/result/list_runs left unwrapped
  2. Each of those failures now writes a stack trace to the server log — justified
  3. Four new stable error codes appear in API responses — justified

[FIRST-PRINCIPLES-REVIEWED] f3daaaa

Description read

[DESCRIPTION-READ] 8cc892de309aeb439236b4e1b225391f0df3b637e4f395ba45117a16774f01d4

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed f3daaaa734fd9549a72d7a82df524b4139b07749 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] f3daaaa

@github-actions

github-actions Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed f3daaaa734fd9549a72d7a82df524b4139b07749 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] f3daaaa

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 6, 2026
The api_workflow_run, api_workflow_run_intent, api_workflow_run_rerun,
and api_workflow_author handlers called their service methods with no
try/except, so any exception escaped as an opaque aiohttp 500 with no
JSON body. A caller could not tell a deterministic platform fault (e.g.
the glibc < 2.28 "atomic no-replace rename is unavailable"
NotImplementedError, before the syscall fallback in kirodotdev#11018) from a
transient failure, and backed off as if the service were unhealthy.

Wrap all four handlers in the same try/except pattern the sibling
api_workflow_run_definition already uses: log via logger.exception and
return a structured {"error", "code"} 500. Each handler gets its own
stable code so the failure class is diagnosable.

Fixes kirodotdev#17337
@chenmingwei23
chenmingwei23 force-pushed the fix/al2-glibc-renameat2-missing branch from c869c99 to f3daaaa Compare October 6, 2026 16:16
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 6, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving (crew-pr-reviewer). The service-layer calls for workflow author and run are wrapped. An exception is logged with its traceback and returns a structured JSON 500 with a stable code, instead of an opaque 500. The success paths are unchanged. The client sees a fixed message, not exception text. CI is green and the base is main.

@iamwhatever
iamwhatever enabled auto-merge (squash) October 7, 2026 06:46
@iamwhatever
iamwhatever merged commit c49678f into kirodotdev:main Oct 7, 2026
197 of 199 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Workflow run/author handlers return opaque 500 on service-layer exceptions

2 participants