Repository navigation
fix(issue-radar): decode az output as utf-8, not the host code page - #7560
Conversation
Open PR relationship auditThis is a consolidated, point-in-time code-level audit note. It compares complete merge-base diffs and current/merged code; it does not treat a shared topic as duplication or partial coverage as completion. Relationship findings
No PR, Issue, label, branch, or review state was changed by the relationship-note portion of this audit. |
4fd46db to
4f19def
Compare
|
Rebased onto main Conflicts: none, clean rebase. The Gates run locally (changed files only): Please review the rebased branch. Note that a maintainer push makes the maintainer the last pusher, so under this repo's last-push rule a second approver is needed before merge. Reply here if anything looks wrong. |
Opus 4.8 Review (fork) — ✅ no blocking findingsReviewed Review detailsConfirmed: No findings. [OPUS-REVIEWED] 3f1ccf9 |
GPT 5.6 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Design Review (Fable 5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify on a real Windows host that Suggestions
[DESIGN-REVIEWED] 3f1ccf9 |
First Principles Review (Fable 5, fork) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The inline Not justified as shipped
What this change shipsInventory (4 items) — 3 justifiedIntent: FIX — non-ASCII Azure DevOps work items crash the route or garble the board on non-UTF-8 hosts; pin UTF-8 at the single az spawn chokepoint. Defect provenance: the added tests fail on base (verified against
[FIRST-PRINCIPLES-REVIEWED] 3f1ccf9 |
|
🤖 Kiro Crew [operator: NicholasRBowers#a942f9ca]: This PR has been inactive for 7+ days. I reviewed the blockers but they require your input:
When you've addressed these, the pipeline will re-assess on its next cycle. |
`_az_run` is Issue Radar's single spawn chokepoint for every `az` call, and what comes back through it is `az devops invoke` JSON -- work-item titles, descriptions and comments written by people. Non-ASCII is the normal case there, not an edge one. The call passed `text=True` with no `encoding=`, so that JSON was decoded with `locale.getpreferredencoding(False)`: cp950 / cp932 / cp1252 on a Windows console code page. A `UnicodeDecodeError` is neither `FileNotFoundError` nor `TimeoutExpired`, so it escapes both handlers in `_az_run` and takes the chokepoint's own error contract with it -- the caller never sees a `ProviderCliError` and the route answers an opaque 500 instead of its 502. Measured: `'cp950' codec can't decode byte 0xe3`. cp1252 fails the other way and inconsistently, since five of its byte values are undefined: the same title can raise on one work item and arrive mojibaked on the next. Name the encoding at the chokepoint, which is what `platform/update_capability.py` and `pod/runtime.py` already do on their own spawns. That takes this file off the subprocess-encoding baseline. The regressions drive the real `_az_run` with an Azure DevOps work item through a `subprocess.run` stand-in that decodes the way `subprocess` documents -- the `encoding=` keyword when given, the locale codec otherwise -- so the code page is a test parameter rather than a property of the runner. cp950 and ascii cover the raising case from both directions, cp1252 the inconsistent one, and an ASCII payload pins that the ordinary case is unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
4f19def to
0540c0b
Compare
|
Rebased onto current Conflict resolvedOne conflict, in Since this branch was written, main pruned 15 other rows from that baseline (including Resolution: kept main's pruned set verbatim and removed only this PR's own row. None of the rows main deleted were resurrected. The diff against main for that file is now exactly one line:
Verified before pushing
Gates and tests run locally
The |
|
The parent's encoding= says how the pipe is READ; it cannot change what the child WRITES. az is a Python program, and a Python child writing to a pipe encodes stdout with locale.getpreferredencoding(False) -- the ANSI code page on Windows -- unless PYTHONIOENCODING says otherwise. That key is not on minimal_env's safe list, so the child inherited nothing and fell back to the code page: a work-item title outside it is replaced with a question mark before the parent ever sees a byte, which no parent-side codec can recover. _az_env now pins PYTHONIOENCODING=utf-8 beside the AZURE_* knobs it already sets, so both ends of the chokepoint agree by construction. The assignment is unconditional, so an operator's exported code page cannot put the child back on it either. Same pairing the repo already uses in pod/runtime provisioning and the dev-fleet sync relay. Two tests, both red before this change: one drives the real _az_run and asserts the env that reaches the spawn, the other feeds a hostile inherited value to the transport helper and asserts it is overridden.
|
|
bolichen97
left a comment
There was a problem hiding this comment.
Approved after a live re-verification: readiness: passed label, PR Readiness status success on this head, no required check pending/failed/cancelled, mergeable, no effective CHANGES_REQUESTED, and no unresolved human blocking request found in comments or review threads. Native auto-merge (squash) enabled; no admin override used.
…irodotdev#7560) * fix(issue-radar): decode az output as utf-8, not the host code page `_az_run` is Issue Radar's single spawn chokepoint for every `az` call, and what comes back through it is `az devops invoke` JSON -- work-item titles, descriptions and comments written by people. Non-ASCII is the normal case there, not an edge one. The call passed `text=True` with no `encoding=`, so that JSON was decoded with `locale.getpreferredencoding(False)`: cp950 / cp932 / cp1252 on a Windows console code page. A `UnicodeDecodeError` is neither `FileNotFoundError` nor `TimeoutExpired`, so it escapes both handlers in `_az_run` and takes the chokepoint's own error contract with it -- the caller never sees a `ProviderCliError` and the route answers an opaque 500 instead of its 502. Measured: `'cp950' codec can't decode byte 0xe3`. cp1252 fails the other way and inconsistently, since five of its byte values are undefined: the same title can raise on one work item and arrive mojibaked on the next. Name the encoding at the chokepoint, which is what `platform/update_capability.py` and `pod/runtime.py` already do on their own spawns. That takes this file off the subprocess-encoding baseline. The regressions drive the real `_az_run` with an Azure DevOps work item through a `subprocess.run` stand-in that decodes the way `subprocess` documents -- the `encoding=` keyword when given, the locale codec otherwise -- so the code page is a test parameter rather than a property of the runner. cp950 and ascii cover the raising case from both directions, cp1252 the inconsistent one, and an ASCII payload pins that the ordinary case is unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(issue-radar): tell az to write utf-8, not just read it as utf-8 The parent's encoding= says how the pipe is READ; it cannot change what the child WRITES. az is a Python program, and a Python child writing to a pipe encodes stdout with locale.getpreferredencoding(False) -- the ANSI code page on Windows -- unless PYTHONIOENCODING says otherwise. That key is not on minimal_env's safe list, so the child inherited nothing and fell back to the code page: a work-item title outside it is replaced with a question mark before the parent ever sees a byte, which no parent-side codec can recover. _az_env now pins PYTHONIOENCODING=utf-8 beside the AZURE_* knobs it already sets, so both ends of the chokepoint agree by construction. The assignment is unconditional, so an operator's exported code page cannot put the child back on it either. Same pairing the repo already uses in pod/runtime provisioning and the dev-fleet sync relay. Two tests, both red before this change: one drives the real _az_run and asserts the env that reaches the spawn, the other feeds a hostile inherited value to the transport helper and asserts it is overridden. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Joe Guo <zejiangg@amazon.com>
…irodotdev#7560) * fix(issue-radar): decode az output as utf-8, not the host code page `_az_run` is Issue Radar's single spawn chokepoint for every `az` call, and what comes back through it is `az devops invoke` JSON -- work-item titles, descriptions and comments written by people. Non-ASCII is the normal case there, not an edge one. The call passed `text=True` with no `encoding=`, so that JSON was decoded with `locale.getpreferredencoding(False)`: cp950 / cp932 / cp1252 on a Windows console code page. A `UnicodeDecodeError` is neither `FileNotFoundError` nor `TimeoutExpired`, so it escapes both handlers in `_az_run` and takes the chokepoint's own error contract with it -- the caller never sees a `ProviderCliError` and the route answers an opaque 500 instead of its 502. Measured: `'cp950' codec can't decode byte 0xe3`. cp1252 fails the other way and inconsistently, since five of its byte values are undefined: the same title can raise on one work item and arrive mojibaked on the next. Name the encoding at the chokepoint, which is what `platform/update_capability.py` and `pod/runtime.py` already do on their own spawns. That takes this file off the subprocess-encoding baseline. The regressions drive the real `_az_run` with an Azure DevOps work item through a `subprocess.run` stand-in that decodes the way `subprocess` documents -- the `encoding=` keyword when given, the locale codec otherwise -- so the code page is a test parameter rather than a property of the runner. cp950 and ascii cover the raising case from both directions, cp1252 the inconsistent one, and an ASCII payload pins that the ordinary case is unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(issue-radar): tell az to write utf-8, not just read it as utf-8 The parent's encoding= says how the pipe is READ; it cannot change what the child WRITES. az is a Python program, and a Python child writing to a pipe encodes stdout with locale.getpreferredencoding(False) -- the ANSI code page on Windows -- unless PYTHONIOENCODING says otherwise. That key is not on minimal_env's safe list, so the child inherited nothing and fell back to the code page: a work-item title outside it is replaced with a question mark before the parent ever sees a byte, which no parent-side codec can recover. _az_env now pins PYTHONIOENCODING=utf-8 beside the AZURE_* knobs it already sets, so both ends of the chokepoint agree by construction. The assignment is unconditional, so an operator's exported code page cannot put the child back on it either. Same pairing the repo already uses in pod/runtime provisioning and the dev-fleet sync relay. Two tests, both red before this change: one drives the real _az_run and asserts the env that reaches the spawn, the other feeds a hostile inherited value to the transport helper and asserts it is overridden. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Joe Guo <zejiangg@amazon.com>
Problem / Motivation
_az_runis Issue Radar's single spawn chokepoint for everyazcall — its owndocstring says so, and
test/test_spawn_audit.pypins it as such. Every Azure DevOpsrequest in the app funnels through this one
subprocess.run:What comes back through it is
az devops invokeJSON: work-item titles,descriptions and comments, written by people. Non-ASCII is the normal case there,
not an edge one — and
_az_invokefeedsproc.stdoutstraight intojson.loads.text=Truewith noencoding=decodes those bytes withlocale.getpreferredencoding(False), which on a Windows console code page iscp950 / cp932 / cp1252, not UTF-8.
Why it matters
The chokepoint loses its own error contract.
_az_runcatches exactlyFileNotFoundErrorandsubprocess.TimeoutExpired, translating each into a typedProviderSetupError/ProviderCliErrorso the route can answer a 502 with a realreason. A
UnicodeDecodeErroris neither, so it escapes both handlers, skips thefailureaudit line, and surfaces as an opaque 500. Measured with a real Azure DevOpswork item:
cp1252 fails the other way, and inconsistently — which is harder to diagnose than
a clean crash. It decodes most bytes to something, but five byte values are
undefined in it, so the same UTF-8 title can raise on one work item and arrive
mojibaked on the next. A user sees a board that is sometimes garbled and sometimes
broken, with no pattern.
The repository already gates this class (
scripts/check_subprocess_encoding.py), andthis file was on its baseline.
What changed (motivation → approach → change)
Symptom: non-ASCII work items crash or garble the board on non-UTF-8 hosts. Root
cause:
az's UTF-8 JSON decoded with the host code page. Fix: name the encoding, atthe chokepoint, so every caller inherits it.
One line, at the one place every
azcall already passes through — no new plumbing,and no per-call-site sweep to keep in sync. It is not a new convention either:
platform/update_capability.pyandpod/runtime.pyalready pin UTF-8 on their ownspawns, the latter through the shared
kiro_crew.subprocess_utf8.UTF8_TEXT.errors=is deliberately not added: it would convert the raising case into thesilent-mojibake case, which the tests below treat as equally wrong.
azwrites UTF-8,so the correct move is to say so.
The subprocess-encoding baseline row for this file is pruned rather than lowered — it
goes to zero unpinned calls.
The child end, not just the parent end
encoding=says how the parent reads the pipe. It cannot change what the childwrites — and
azis a Python program, so its stdout to a pipe is encoded withlocale.getpreferredencoding(False)unlessPYTHONIOENCODINGsays otherwise. A titlecharacter the child's code page cannot encode is replaced with
?before the parentsees a byte, and no parent-side codec recovers it.
PYTHONIOENCODINGis not onminimal_env's safe-key list, so the child inherited nothing at all and fell back tothe code page.
_az_envnow pins it beside theAZURE_*knobs it already sets:Unconditional, not a default: an operator's exported code page cannot put the child
back on it. Same pairing already used by
pod/provision.pyand the dev-fleet syncrelay (
utf-8:replacethere; strict here, for the reasonerrors=is refused above).Tests
New file
test/test_issue_radar_az_encoding.py.The harness drives the real
_az_runwith asubprocess.runstand-in that decodesthe way
subprocessdocuments: theencoding=keyword when one is given,locale.getpreferredencoding(False)otherwise. That makes the interesting code page atest parameter rather than a property of the runner — deterministic on a UTF-8 Linux CI
box and on a cp950 laptop alike, with no console code page to switch, no timing and no
skip. The payload is an Azure DevOps work item with prose in it, not a synthetic byte
string.
..._work_item_json_survives_a_non_utf8_host_codepage[cp950]/[ascii]LC_ALL=Cone. Asserts the JSON is byte-identical and thatjson.loadsrecovers the exact title, so the consequence is pinned, not just the string..._a_codepage_that_silently_mojibakes_is_also_refusederrors="replace"would leave behind, and the same assertion refuses both of its outcomes..._ascii_output_is_unaffected..._the_spawn_environment_tells_az_to_write_utf8_az_runis driven with the real_az_envand the env that reaches the spawn must carryPYTHONIOENCODING=utf-8. Without it the parent'sencoding=is unfalsifiable..._a_hostile_inherited_encoding_does_not_reach_azAn autouse fixture stubs the chokepoint's preflight (
_resolve_host,_az_bin,_az_env,_audit). None of it touches decoding and all of it needs a configuredhost, so stubbing it keeps each test measuring one thing.
Red-before, with
azure_client.pyrestored to currentorigin/main(
5603ae744) and nothing else changed — 3 failed, 1 passed:The third line is the cp1252 case, and it is why this PR's description says
"inconsistently" rather than "always mojibakes": on this payload cp1252 happens to hit
one of its five undefined bytes. The one test that passed is the ASCII over-fix guard,
which must pass on both sides.
The two env tests are red-before on their own: with the
PYTHONIOENCODINGlineremoved and nothing else changed,
..._tells_az_to_write_utf8fails on its own messageand
..._a_hostile_inherited_encoding_does_not_reach_azraisesKeyError, while thefour decoding tests stay green.
Green after: the new file plus
test/test_spawn_audit.py, which is the existingsuite that pins
_az_runas the sole chokepoint — 22 passed, 6 skipped (the skipsare the Windows-only launcher measurements in
test_azure_launcher_reparse.py).Gates on the touched files:
scripts/check_subprocess_encoding.pypassed (116files remain on the baseline, down from 117),
scripts/check_black_formatting.pypassed in scope,
flake8andisort --check-onlyclean,git diff --checkclean,and
mypy src/kiro_crew/apps/builtins/issue_radar/backend/azure_client.py— Success:no issues found in 1 source file. No repository-wide suite; server CI owns that.
Manual verification
N/A — unit coverage sufficient: the defect is a decoding choice at one spawn, and the
tests drive the real chokepoint with a real work-item payload, which is closer to the
failure than pointing one machine at an Azure DevOps org would be.
Related Issues
None. Found by auditing spawns whose output is user-authored prose and which decode
with the host locale, against
scripts/check_subprocess_encoding.py's own baseline.Pattern harvest
Rule candidate: review-prompt
Pattern: a typed error contract wrapped around a spawn, where the decode step sits
inside the
trybut its exception type is not in theexcept.The narrow version of this ("pass
encoding=") is already a gate. The generalizablehalf is not:
_az_runis careful to translate every failure it expects into a providererror, and the decode is the one failure that walks straight through that translation —
so the symptom is not "wrong characters" but "this endpoint stopped honouring its own
error contract". A reviewer looking at any
except (FileNotFoundError, TimeoutExpired)around a
text=Truespawn can ask whetherUnicodeDecodeErrorwas meant to be in thatlist, or meant to be impossible.
Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement