Skip to content

fix(issue-radar): decode az output as utf-8, not the host code page - #7560

Merged
bolichen97 merged 2 commits into
kirodotdev:mainfrom
leonlaiyc:fix/issue-radar-az-utf8
Sep 22, 2026
Merged

bolichen97 merged 2 commits into
kirodotdev:mainfrom
leonlaiyc:fix/issue-radar-az-utf8

Conversation

@leonlaiyc

@leonlaiyc leonlaiyc commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

_az_run is Issue Radar's single spawn chokepoint for every az call — its own
docstring says so, and test/test_spawn_audit.py pins it as such. Every Azure DevOps
request in the app funnels through this one subprocess.run:

proc = subprocess.run(
    [az, *argv[1:]],
    capture_output=True,
    text=True,          # <- no encoding=
    timeout=timeout,
    env=_az_env(resolved_host),
)

What comes back through it is az devops invoke JSON: work-item titles,
descriptions and comments, written by people.
Non-ASCII is the normal case there,
not an edge one — and _az_invoke feeds proc.stdout straight into json.loads.

text=True with no encoding= decodes those bytes with
locale.getpreferredencoding(False), which on a Windows console code page is
cp950 / cp932 / cp1252, not UTF-8.

Why it matters

The chokepoint loses its own error contract. _az_run catches exactly
FileNotFoundError and subprocess.TimeoutExpired, translating each into a typed
ProviderSetupError / ProviderCliError so the route can answer a 502 with a real
reason. A UnicodeDecodeError is neither, so it escapes both handlers, skips the
failure audit line, and surfaces as an opaque 500. Measured with a real Azure DevOps
work item:

UnicodeDecodeError: 'cp950' codec can't decode byte 0xe3 in position 47

cp1252 fails the other way, and inconsistently — which is harder to diagnose than
a clean crash. It decodes most bytes to something, but five byte values are
undefined in it, so the same UTF-8 title can raise on one work item and arrive
mojibaked on the next. A user sees a board that is sometimes garbled and sometimes
broken, with no pattern.

The repository already gates this class (scripts/check_subprocess_encoding.py), and
this file was on its baseline.

What changed (motivation → approach → change)

Symptom: non-ASCII work items crash or garble the board on non-UTF-8 hosts. Root
cause: az's UTF-8 JSON decoded with the host code page. Fix: name the encoding, at
the chokepoint, so every caller inherits it.

 proc = subprocess.run(
     [az, *argv[1:]],
     capture_output=True,
     text=True,
+    encoding="utf-8",
     timeout=timeout,
     env=_az_env(resolved_host),
 )

One line, at the one place every az call already passes through — no new plumbing,
and no per-call-site sweep to keep in sync. It is not a new convention either:
platform/update_capability.py and pod/runtime.py already pin UTF-8 on their own
spawns, the latter through the shared kiro_crew.subprocess_utf8.UTF8_TEXT.

errors= is deliberately not added: it would convert the raising case into the
silent-mojibake case, which the tests below treat as equally wrong. az writes UTF-8,
so the correct move is to say so.

The subprocess-encoding baseline row for this file is pruned rather than lowered — it
goes to zero unpinned calls.

The child end, not just the parent end

encoding= says how the parent reads the pipe. It cannot change what the child
writes — and az is a Python program, so its stdout to a pipe is encoded with
locale.getpreferredencoding(False) unless PYTHONIOENCODING says otherwise. A title
character the child's code page cannot encode is replaced with ? before the parent
sees a byte
, and no parent-side codec recovers it. PYTHONIOENCODING is not on
minimal_env's safe-key list, so the child inherited nothing at all and fell back to
the code page.

_az_env now pins it beside the AZURE_* knobs it already sets:

 passthrough["NO_COLOR"] = "1"
+passthrough["PYTHONIOENCODING"] = "utf-8"

Unconditional, not a default: an operator's exported code page cannot put the child
back on it. Same pairing already used by pod/provision.py and the dev-fleet sync
relay (utf-8:replace there; strict here, for the reason errors= is refused above).

Tests

New file test/test_issue_radar_az_encoding.py.

The harness drives the real _az_run with a subprocess.run stand-in that decodes
the way subprocess documents: the encoding= keyword when one is given,
locale.getpreferredencoding(False) otherwise. That makes the interesting code page a
test parameter rather than a property of the runner — deterministic on a UTF-8 Linux CI
box and on a cp950 laptop alike, with no console code page to switch, no timing and no
skip. The payload is an Azure DevOps work item with prose in it, not a synthetic byte
string.

Test Locks in
..._work_item_json_survives_a_non_utf8_host_codepage[cp950] / [ascii] the raising case from both directions — a real shipped Windows DBCS code page, and the degenerate POSIX LC_ALL=C one. Asserts the JSON is byte-identical and that json.loads recovers the exact title, so the consequence is pinned, not just the string
..._a_codepage_that_silently_mojibakes_is_also_refused cp1252, the inconsistent case. This is what errors="replace" would leave behind, and the same assertion refuses both of its outcomes
..._ascii_output_is_unaffected over-fix guard: the ordinary case is unchanged. Passes on both sides, which is the point
..._the_spawn_environment_tells_az_to_write_utf8 the child end, through the real chokepoint: _az_run is driven with the real _az_env and the env that reaches the spawn must carry PYTHONIOENCODING=utf-8. Without it the parent's encoding= is unfalsifiable
..._a_hostile_inherited_encoding_does_not_reach_az the assignment wins over an inherited value. Driven at the transport helper so the hostile value is a parameter, not a property of the runner

An autouse fixture stubs the chokepoint's preflight (_resolve_host, _az_bin,
_az_env, _audit). None of it touches decoding and all of it needs a configured
host, so stubbing it keeps each test measuring one thing.

Red-before, with azure_client.py restored to current origin/main
(5603ae744) and nothing else changed — 3 failed, 1 passed:

UnicodeDecodeError: 'cp950'   codec can't decode byte 0xe3 in position 47
UnicodeDecodeError: 'ascii'   codec can't decode byte 0xc3 in position 44
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 48

The third line is the cp1252 case, and it is why this PR's description says
"inconsistently" rather than "always mojibakes": on this payload cp1252 happens to hit
one of its five undefined bytes. The one test that passed is the ASCII over-fix guard,
which must pass on both sides.

The two env tests are red-before on their own: with the PYTHONIOENCODING line
removed and nothing else changed, ..._tells_az_to_write_utf8 fails on its own message
and ..._a_hostile_inherited_encoding_does_not_reach_az raises KeyError, while the
four decoding tests stay green.

Green after: the new file plus test/test_spawn_audit.py, which is the existing
suite that pins _az_run as the sole chokepoint — 22 passed, 6 skipped (the skips
are the Windows-only launcher measurements in test_azure_launcher_reparse.py).

Gates on the touched files: scripts/check_subprocess_encoding.py passed (116
files remain on the baseline, down from 117), scripts/check_black_formatting.py
passed in scope, flake8 and isort --check-only clean, git diff --check clean,
and mypy src/kiro_crew/apps/builtins/issue_radar/backend/azure_client.py — Success:
no issues found in 1 source file
. No repository-wide suite; server CI owns that.

Manual verification

N/A — unit coverage sufficient: the defect is a decoding choice at one spawn, and the
tests drive the real chokepoint with a real work-item payload, which is closer to the
failure than pointing one machine at an Azure DevOps org would be.

Related Issues

None. Found by auditing spawns whose output is user-authored prose and which decode
with the host locale, against scripts/check_subprocess_encoding.py's own baseline.

Pattern harvest

Rule candidate: review-prompt
Pattern: a typed error contract wrapped around a spawn, where the decode step sits
inside the try but its exception type is not in the except.

The narrow version of this ("pass encoding=") is already a gate. The generalizable
half is not: _az_run is careful to translate every failure it expects into a provider
error, and the decode is the one failure that walks straight through that translation —
so the symptom is not "wrong characters" but "this endpoint stopped honouring its own
error contract". A reviewer looking at any except (FileNotFoundError, TimeoutExpired)
around a text=True spawn can ask whether UnicodeDecodeError was meant to be in that
list, or meant to be impossible.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

@leonlaiyc
leonlaiyc requested review from a team and pepmach September 1, 2026 07:50
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Sep 1, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 1, 2026
@bolichen97

Copy link
Copy Markdown
Contributor

Open PR relationship audit

This is a consolidated, point-in-time code-level audit note. It compares complete merge-base diffs and current/merged code; it does not treat a shared topic as duplication or partial coverage as completion.

Relationship findings

  • PR #7544 is OVERLAPPING relative to this PR. The goals differ or the implementations can complement each other; this is not a duplicate claim. Recommended action for PR #7544: KEEP. Two independent instances of the same class in different modules; both should land. Sequence them and let the second rebase the one-row baseline hunk. Files: .github/subprocess-encoding-baseline.txt.

No PR, Issue, label, branch, or review state was changed by the relationship-note portion of this audit.

@bolichen97

Copy link
Copy Markdown
Contributor

Rebased onto main 41dcadf2c by a maintainer as part of the 2026-09-08 open-PR audit.

Conflicts: none, clean rebase. The _az_run hunk still applied cleanly after #7604 rewrote azure_client.py, and the .github/subprocess-encoding-baseline.txt row deletion applied without conflict.

Gates run locally (changed files only): black --check, isort --check-only, flake8 all clean; pytest test/test_issue_radar_az_encoding.py 4 passed; scripts/check_subprocess_encoding.py passed with the baseline row removed.

Please review the rebased branch. Note that a maintainer push makes the maintainer the last pusher, so under this repo's last-push rule a second approver is needed before merge. Reply here if anything looks wrong.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 8, 2026
@github-actions

github-actions Bot commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review (fork) — ✅ no blocking findings

Reviewed 3f1ccf977cd877e1a0b3bf1c3376d935316a937d via the fork AI-review pipeline; updated in place on each push.

Review details

Confirmed: minimal_env(**extra) does env.update(extra), so PYTHONIOENCODING="utf-8" lands unconditionally regardless of the safe-key filter. The parent-side encoding="utf-8" pins the read, and the baseline prune is correct since azure_client.py's single text-mode call is now encoding-pinned. No grounded defect on the changed lines.

No findings.

[OPUS-REVIEWED] 3f1ccf9

@github-actions

github-actions Bot commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed 3f1ccf977cd877e1a0b3bf1c3376d935316a937d via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 3f1ccf9

@github-actions

github-actions Bot commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — ✅ PASS

Design-level review of 3f1ccf977cd877e1a0b3bf1c3376d935316a937d via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify on a real Windows host that az.cmd's wrapped Python honors the injected PYTHONIOENCODING end-to-end; CI cannot exercise that hop.

Suggestions

  • Add except UnicodeDecodeError → ProviderCliError beside the two existing handlers in _az_run: the PR's own "pattern harvest" names the decode as the one failure that escapes the typed contract, and pinning the codec makes that escape rare, not impossible.

[DESIGN-REVIEWED] 3f1ccf9

@github-actions

github-actions Bot commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — 🟡 CONCERNS

Premise-level review of 3f1ccf977cd877e1a0b3bf1c3376d935316a937d via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The inline encoding="utf-8" re-spells the shared UTF8_TEXT mechanism, and the stated reason for refusing it is contradicted by this PR's own tests.

Not justified as shipped

  • Item 1 — duplicate of src/kiro_crew/subprocess_utf8.py: UTF8_TEXT is the repo's one spelling of "decode this child as UTF-8" (grepped UTF8_TEXT: ~40 splat sites in src/kiro_crew), and the description's ground for refusing it — "errors= … would convert the raising case into the silent-mojibake case, which the tests below treat as equally wrong" — does not hold: the test harness decodes with the encoding= kwarg and never reads errors=, so every new test passes unchanged under **UTF8_TEXT; the mojibake the tests refuse came from the wrong codec, not from replacement. Strict decode also leaves a malformed az byte raising the same untyped, contract-escaping error the description names as the harm, against the module's own recorded decision ("must degrade to U+FFFD in that spot, not … crash the caller").
    Subtraction: replace text=True, encoding="utf-8" in _az_run with **UTF8_TEXT.
    Clears when: the splat replaces the inline kwargs, or a test shows a payload where errors="replace" under the correct codec produces the wrong outcome.

What this change ships

Inventory (4 items) — 3 justified

Intent: FIX — non-ASCII Azure DevOps work items crash the route or garble the board on non-UTF-8 hosts; pin UTF-8 at the single az spawn chokepoint. Defect provenance: the added tests fail on base (verified against _az_run on the base tree).

  1. Non-ASCII work-item titles no longer crash or garble the board on non-UTF-8 hosts — duplicate of src/kiro_crew/subprocess_utf8.py (second spelling; divergence unsupported)
  2. az itself is now told to write UTF-8, overriding any operator-exported encoding — justified
  3. This file leaves the subprocess-encoding debt baseline (117 → 116) — justified
  4. New tests pin both decode ends and fail on base — justified

[FIRST-PRINCIPLES-REVIEWED] 3f1ccf9

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Sep 8, 2026
@NicholasRBowers NicholasRBowers added needs-pr-triage PR scanner: awaiting automated triage needs-author-decision PR blocked on author input and removed needs-pr-triage PR scanner: awaiting automated triage labels Sep 12, 2026
@NicholasRBowers

Copy link
Copy Markdown
Contributor

🤖 Kiro Crew [operator: NicholasRBowers#a942f9ca]: This PR has been inactive for 7+ days. I reviewed the blockers but they require your input:

  • The one real blocker is a maintainer-only gate. Fork workflow-change guard is red because this fork PR edits .github/subprocess-encoding-baseline.txt, and it stays red until a maintainer applies the allow-fork-workflow-change label. The edit cannot simply be dropped: scripts/check_subprocess_encoding.py requires a baselined file whose count shrank to be pruned in the same change, so the -1 .../azure_client.py line is mandated by the ratchet. Please ask a maintainer for that label (or for the fix to be re-homed as a same-repo branch, where the guard does not apply). The code change itself still applies: main's _az_run() in src/kiro_crew/apps/builtins/issue_radar/backend/azure_client.py still runs subprocess.run(..., text=True) with no encoding=, and main's baseline still lists that file.
  • Design Review and First Principles Review are both 🟡 CONCERNS on 4f19def1f with no disposition posted; each item needs its own target=design / target=first-principles disposition comment.
  • Everything else is mechanical and can be driven once the label lands: the merge conflict is confined to .github/subprocess-encoding-baseline.txt (main pruned other lines since your base), and the sole CI red, Backend Tests (Windows) (3), is test/test_playwright_cli_installer.py::test_powershell_installer_parses (pwsh TimeoutExpired, untouched by this diff). GPT and Opus are clean on the current head.

When you've addressed these, the pipeline will re-assess on its next cycle.

`_az_run` is Issue Radar's single spawn chokepoint for every `az` call, and what
comes back through it is `az devops invoke` JSON -- work-item titles,
descriptions and comments written by people. Non-ASCII is the normal case there,
not an edge one.

The call passed `text=True` with no `encoding=`, so that JSON was decoded with
`locale.getpreferredencoding(False)`: cp950 / cp932 / cp1252 on a Windows console
code page. A `UnicodeDecodeError` is neither `FileNotFoundError` nor
`TimeoutExpired`, so it escapes both handlers in `_az_run` and takes the
chokepoint's own error contract with it -- the caller never sees a
`ProviderCliError` and the route answers an opaque 500 instead of its 502.
Measured: `'cp950' codec can't decode byte 0xe3`. cp1252 fails the other way and
inconsistently, since five of its byte values are undefined: the same title can
raise on one work item and arrive mojibaked on the next.

Name the encoding at the chokepoint, which is what `platform/update_capability.py`
and `pod/runtime.py` already do on their own spawns. That takes this file off the
subprocess-encoding baseline.

The regressions drive the real `_az_run` with an Azure DevOps work item through a
`subprocess.run` stand-in that decodes the way `subprocess` documents -- the
`encoding=` keyword when given, the locale codec otherwise -- so the code page is
a test parameter rather than a property of the runner. cp950 and ascii cover the
raising case from both directions, cp1252 the inconsistent one, and an ASCII
payload pins that the ordinary case is unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@iamwhatever
iamwhatever force-pushed the fix/issue-radar-az-utf8 branch from 4f19def to 0540c0b Compare September 21, 2026 21:13
@iamwhatever

Copy link
Copy Markdown
Contributor

Rebased onto current origin/main (0140fdd57) by a maintainer, since this branch had gone stale. New head: 0540c0b89. No behaviour change — the shape is byte-for-byte the same work as before the rebase: 3 files, +122/-1.

Conflict resolved

One conflict, in .github/subprocess-encoding-baseline.txt.

Since this branch was written, main pruned 15 other rows from that baseline (including src/kiro_crew/apps/builtins/file_explorer/server.py, which sat adjacent to this PR's row) and re-counted src/kiro_crew/pod/runtime.py from 2 to 1. Git therefore saw the two edits as competing on the same region.

Resolution: kept main's pruned set verbatim and removed only this PR's own row. None of the rows main deleted were resurrected. The diff against main for that file is now exactly one line:

-1 src/kiro_crew/apps/builtins/issue_radar/backend/azure_client.py

src/kiro_crew/apps/builtins/issue_radar/backend/azure_client.py and test/test_issue_radar_az_encoding.py merged without conflict.

Verified before pushing

  • git diff --stat origin/main...HEAD (three dots): 3 files, +122/-1 — matches the pre-rebase shape, so no upstream work was reverted.
  • Symbol check on azure_client.py: the set of def/async def names on main minus the set on this head is empty — no function main added was dropped.

Gates and tests run locally

Gate Result
scripts/check_subprocess_encoding.py pass — nothing in scope decodes with the locale code page outside the baseline (209 known calls in 99 files still listed)
scripts/check_black_formatting.py pass — nothing in scope unformatted outside the baseline
isort --check-only (changed files) pass
flake8 (changed files) pass
mypy src/.../azure_client.py pass (CI scopes mypy to src/kiro_crew/)
pytest test/test_issue_radar_az_encoding.py test/test_subprocess_encoding_gate.py 36 passed

The PR Hygiene red still visible above is a stale September run for a missing ## Pattern harvest section; the body now carries that section, and fresh runs on this head supersede it.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 21, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

Backend Tests (Windows) (1) is not this PR

The red is the known flake tracked in #12376, not this diff.

Failing test (run 35655937793 / job 106522037502, head 0540c0b89):

FAILED test/test_decisions_tool_risk_end_to_end.py::test_a_trusted_tool_call_is_flagged_on_its_card_and_takes_a_verdict - AssertionError: the card must carry the annotation the oracle answered

Why it cannot be this diff

  • This PR adds encoding="utf-8", errors="replace" to one subprocess.run in src/kiro_crew/apps/builtins/issue_radar/backend/azure_client.py, drops one row from .github/subprocess-encoding-baseline.txt, and adds one test file. The failing test neither imports nor reaches any of that.
  • test/test_decisions_tool_risk_end_to_end.py, test/test_decisions_tool_risk_card.py, src/kiro_crew/dashboard/chat_runner.py and the whole src/kiro_crew/decisions/ tree are byte-identical to origin/main on this head (same git blob/tree SHAs).
  • The same test fails on fix(dashboard): serialize the auto-update toggle against both config writers #7167, whose diff is disjoint from this one (one line in dashboard/handlers/updates.py). Two unrelated diffs, one failure: the cause is shared, not either diff.
  • Shard membership is a stable SHA-256 hash of the file path (scripts/ci_file_shards.py), so adding a test file does not reshuffle shards.
  • The file passes locally on Linux on this head, 3/3.

The mechanism

_record_outcome in src/kiro_crew/decisions/points/tool_risk.py bounds the outcome-row write at LOG_BUDGET_SECS = 0.05. On expiry it returns None by design, the tool card gets no decisions_tool_risk meta, and the only trace is a logger.debug line — which is exactly the assertion that fails. The module's autouse _generous_append_deadline fixture raises platform_log_append._APPEND_TIMEOUT_SECONDS to 10 s but leaves this 50 ms caller-side budget at its production value.

Reproduced on Linux by setting tool_risk.LOG_BUDGET_SECS = 0.0001 and changing nothing else: the same assertion fails with the same message.

Load correlates. Backend Tests (Windows) (1) on this head took 12m52s, above the slowest of the nine unrelated PRs where that shard passes (5m15s–12m31s).

Posting evidence only — nothing pushed. The Coverage Gate red is also failing on unrelated PRs and is out of scope here.

The parent's encoding= says how the pipe is READ; it cannot change what the
child WRITES. az is a Python program, and a Python child writing to a pipe
encodes stdout with locale.getpreferredencoding(False) -- the ANSI code page on
Windows -- unless PYTHONIOENCODING says otherwise. That key is not on
minimal_env's safe list, so the child inherited nothing and fell back to the
code page: a work-item title outside it is replaced with a question mark before
the parent ever sees a byte, which no parent-side codec can recover.

_az_env now pins PYTHONIOENCODING=utf-8 beside the AZURE_* knobs it already
sets, so both ends of the chokepoint agree by construction. The assignment is
unconditional, so an operator's exported code page cannot put the child back on
it either. Same pairing the repo already uses in pod/runtime provisioning and
the dev-fleet sync relay.

Two tests, both red before this change: one drives the real _az_run and asserts
the env that reaches the spawn, the other feeds a hostile inherited value to the
transport helper and asserts it is overridden.
@iamwhatever

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — finding accepted and fixed

The blocking finding on head 0540c0b89 was right, so it is implemented rather than answered:

BLOCKING -- azure_client.py:514 -- UTF-8 decoding does not make the child emit UTF-8
Fix: Set PYTHONIOENCODING=utf-8 in _az_env so the child and parent use the same encoding.

encoding="utf-8" on the parent says how the pipe is read; it cannot change what the child writes. az is a Python program, so its stdout to a pipe is encoded with locale.getpreferredencoding(False) unless PYTHONIOENCODING says otherwise — and that key is not on minimal_env's safe-key list, so the child inherited nothing and fell back to the code page. A character the child's code page cannot encode is replaced with ? before the parent sees a byte, and no parent-side codec recovers it.

New head 3f1ccf977cd877e1a0b3bf1c3376d935316a937d (commit 2 of 2):

 passthrough["NO_COLOR"] = "1"
+passthrough["PYTHONIOENCODING"] = "utf-8"

in azure_transport._az_env, which is the function azure_client._az_env delegates to. Unconditional rather than a default, so an operator's exported code page cannot put the child back on it. Strict utf-8, not utf-8:replace, for the same reason this PR refuses errors= on the parent: a replacement character is the silent-corruption case. The repo already pairs both ends this way in pod/provision.py and the dev-fleet sync relay.

Two new tests, both red with that one line removed and nothing else changed:

  • test_the_spawn_environment_tells_az_to_write_utf8 — drives the real _az_run with the real _az_env and asserts the env that reaches the spawn carries the key. Fails on its own message without the fix.
  • test_a_hostile_inherited_encoding_does_not_reach_az — feeds PYTHONIOENCODING=cp1252 to the transport helper and asserts it is overridden. Raises KeyError without the fix.

Local gates on this head: test_issue_radar_az_encoding.py + test_azure_launcher_reparse.py + test_spawn_audit.py → 22 passed, 6 skipped; black --check, isort --check-only, flake8, mypy clean on both touched files; scripts/check_subprocess_encoding.py passed in scope. Shape: 4 files, +183/-1, 2 commits, single-parent on origin/main.

The PR body's "What changed" and test table were updated to describe the child end too, so the description matches the diff. ## Pattern harvest is unchanged.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 22, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

First Principles Review — CONCERNS answered (advisory, no push)

Head 3f1ccf977. Item 1 asks to replace text=True, encoding="utf-8" in _az_run with **UTF8_TEXT.

Conceded, on the part that is checkable: the description's ground for refusing errors= — "the tests below treat [mojibake] as equally wrong" — is not pinned by these tests. The subprocess.run stand-in decodes with payload.decode(codec) and never reads errors=, so every test in test_issue_radar_az_encoding.py passes unchanged under **UTF8_TEXT. What those tests refuse is the wrong codec, not replacement. The description overstated what the harness proves.

What stays, and why it is a decision rather than a defect: with the child now pinned to UTF-8 as well (PYTHONIOENCODING=utf-8 in _az_env, commit 2), both ends of this chokepoint are declared, so a malformed byte on the pipe is no longer an expected input — it means az broke its own contract. On that input the two policies differ in kind:

  • strict (encoding="utf-8") raises UnicodeDecodeError, which escapes _az_run's typed handlers and surfaces as a 500. That is the harm the PR itself names.
  • errors="replace" puts U+FFFD in the affected span, json.loads still parses, and the board renders a title with a visible replacement character — which is subprocess_utf8's own recorded policy ("degrade to U+FFFD in that spot, not … crash the caller").

Read that way the reviewer's subtraction is the better default, and the argument against it in the body does not carry. I am not pushing it on this head for two reasons, both about cost rather than correctness:

  1. It is a policy change to what the route does on a byte az should never emit, not a fix to a defect in the diff. **UTF8_TEXT also changes what the typed error contract promises on that input, which is the thing this PR is about — that belongs to the author's call, and the PR already carries needs-author-decision.
  2. Every new head on a fork PR re-dispatches the full review wave. Tonight's waves went red across this repo on API rate limit exceeded for installation; four lanes here had to be re-run one at a time to clear stubs that never read the diff. Re-rolling all of them for an advisory whose verdict is explicitly non-blocking is not a good trade.

If the author or a maintainer wants it, the change is mechanical and stays inside commit 2: text=True, encoding="utf-8" → **UTF8_TEXT (import from kiro_crew.subprocess_utf8), plus one test that pins the replacement outcome on a deliberately malformed byte, which is the "clears when" the lane names. Say the word and I will amend rather than add a third commit.

Lane state on this head

All blocking lanes green: GPT 5.6 Review (no blocking findings, after the PYTHONIOENCODING fix), Opus 4.8 Review (no blocking findings), Design Review (PASS), UX Review (skipped — no UI surface), Security Scope Review (nothing to scope), Internal Content Scan (clean), PR Hygiene, and the full CI matrix including Backend Tests (Windows) (1). First Principles Review is the only non-green lane and is advisory.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 22, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) September 22, 2026 04:44

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved after a live re-verification: readiness: passed label, PR Readiness status success on this head, no required check pending/failed/cancelled, mergeable, no effective CHANGES_REQUESTED, and no unresolved human blocking request found in comments or review threads. Native auto-merge (squash) enabled; no admin override used.

@bolichen97
bolichen97 merged commit acf1478 into kirodotdev:main Sep 22, 2026
107 of 115 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 22, 2026
amadsalmon pushed a commit to amadsalmon/KiroCrew that referenced this pull request Sep 22, 2026
…irodotdev#7560)

* fix(issue-radar): decode az output as utf-8, not the host code page

`_az_run` is Issue Radar's single spawn chokepoint for every `az` call, and what
comes back through it is `az devops invoke` JSON -- work-item titles,
descriptions and comments written by people. Non-ASCII is the normal case there,
not an edge one.

The call passed `text=True` with no `encoding=`, so that JSON was decoded with
`locale.getpreferredencoding(False)`: cp950 / cp932 / cp1252 on a Windows console
code page. A `UnicodeDecodeError` is neither `FileNotFoundError` nor
`TimeoutExpired`, so it escapes both handlers in `_az_run` and takes the
chokepoint's own error contract with it -- the caller never sees a
`ProviderCliError` and the route answers an opaque 500 instead of its 502.
Measured: `'cp950' codec can't decode byte 0xe3`. cp1252 fails the other way and
inconsistently, since five of its byte values are undefined: the same title can
raise on one work item and arrive mojibaked on the next.

Name the encoding at the chokepoint, which is what `platform/update_capability.py`
and `pod/runtime.py` already do on their own spawns. That takes this file off the
subprocess-encoding baseline.

The regressions drive the real `_az_run` with an Azure DevOps work item through a
`subprocess.run` stand-in that decodes the way `subprocess` documents -- the
`encoding=` keyword when given, the locale codec otherwise -- so the code page is
a test parameter rather than a property of the runner. cp950 and ascii cover the
raising case from both directions, cp1252 the inconsistent one, and an ASCII
payload pins that the ordinary case is unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(issue-radar): tell az to write utf-8, not just read it as utf-8

The parent's encoding= says how the pipe is READ; it cannot change what the
child WRITES. az is a Python program, and a Python child writing to a pipe
encodes stdout with locale.getpreferredencoding(False) -- the ANSI code page on
Windows -- unless PYTHONIOENCODING says otherwise. That key is not on
minimal_env's safe list, so the child inherited nothing and fell back to the
code page: a work-item title outside it is replaced with a question mark before
the parent ever sees a byte, which no parent-side codec can recover.

_az_env now pins PYTHONIOENCODING=utf-8 beside the AZURE_* knobs it already
sets, so both ends of the chokepoint agree by construction. The assignment is
unconditional, so an operator's exported code page cannot put the child back on
it either. Same pairing the repo already uses in pod/runtime provisioning and
the dev-fleet sync relay.

Two tests, both red before this change: one drives the real _az_run and asserts
the env that reaches the spawn, the other feeds a hostile inherited value to the
transport helper and asserts it is overridden.

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Joe Guo <zejiangg@amazon.com>
wknight1 pushed a commit to wknight1/KiroCrew that referenced this pull request Oct 5, 2026
…irodotdev#7560)

* fix(issue-radar): decode az output as utf-8, not the host code page

`_az_run` is Issue Radar's single spawn chokepoint for every `az` call, and what
comes back through it is `az devops invoke` JSON -- work-item titles,
descriptions and comments written by people. Non-ASCII is the normal case there,
not an edge one.

The call passed `text=True` with no `encoding=`, so that JSON was decoded with
`locale.getpreferredencoding(False)`: cp950 / cp932 / cp1252 on a Windows console
code page. A `UnicodeDecodeError` is neither `FileNotFoundError` nor
`TimeoutExpired`, so it escapes both handlers in `_az_run` and takes the
chokepoint's own error contract with it -- the caller never sees a
`ProviderCliError` and the route answers an opaque 500 instead of its 502.
Measured: `'cp950' codec can't decode byte 0xe3`. cp1252 fails the other way and
inconsistently, since five of its byte values are undefined: the same title can
raise on one work item and arrive mojibaked on the next.

Name the encoding at the chokepoint, which is what `platform/update_capability.py`
and `pod/runtime.py` already do on their own spawns. That takes this file off the
subprocess-encoding baseline.

The regressions drive the real `_az_run` with an Azure DevOps work item through a
`subprocess.run` stand-in that decodes the way `subprocess` documents -- the
`encoding=` keyword when given, the locale codec otherwise -- so the code page is
a test parameter rather than a property of the runner. cp950 and ascii cover the
raising case from both directions, cp1252 the inconsistent one, and an ASCII
payload pins that the ordinary case is unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(issue-radar): tell az to write utf-8, not just read it as utf-8

The parent's encoding= says how the pipe is READ; it cannot change what the
child WRITES. az is a Python program, and a Python child writing to a pipe
encodes stdout with locale.getpreferredencoding(False) -- the ANSI code page on
Windows -- unless PYTHONIOENCODING says otherwise. That key is not on
minimal_env's safe list, so the child inherited nothing and fell back to the
code page: a work-item title outside it is replaced with a question mark before
the parent ever sees a byte, which no parent-side codec can recover.

_az_env now pins PYTHONIOENCODING=utf-8 beside the AZURE_* knobs it already
sets, so both ends of the chokepoint agree by construction. The assignment is
unconditional, so an operator's exported code page cannot put the child back on
it either. Same pairing the repo already uses in pod/runtime provisioning and
the dev-fleet sync relay.

Two tests, both red before this change: one drives the real _az_run and asserts
the env that reaches the spawn, the other feeds a hostile inherited value to the
transport helper and asserts it is overridden.

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Joe Guo <zejiangg@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) needs-author-decision PR blocked on author input

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants