Skip to content

fix(tailnet): decode the tailscale CLI as utf-8, not the host code page - #14648

Merged
kyleseaman merged 1 commit into
mainfrom
fix/tailnet-cli-utf8-inrepo
Sep 28, 2026
Merged

kyleseaman merged 1 commit into
mainfrom
fix/tailnet-cli-utf8-inrepo

Conversation

@iamwhatever

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The tailnet origin is derived correctly for operators whose Tailscale device names use non-ASCII characters, on every platform.

Both tailscale CLI spawns in the dashboard run in text mode with no encoding=, so the child's output is decoded with locale.getpreferredencoding() — UTF-8 on POSIX, but the legacy ANSI code page on Windows. Device names are operator-chosen free text and tailscale ... --json is JSON (UTF-8 by RFC 8259 §8.1), so a device named 測試-desk silently decodes to different characters on a cp950 host and _valid_magicdns_name rejects a name that was valid — the tailnet origin is never derived. Bytes that decode under no code page raise UnicodeDecodeError (a ValueError) that escapes the except (OSError, subprocess.SubprocessError) guard on POSIX, or kills the capture helper thread on Windows and leaves proc.stdout as None.

Why it matters

An operator on a Windows host with a non-ASCII device name loses the tailnet origin silently: no error they can see, the feature just does not work. On some byte sequences the gateway leaks a traceback to stderr or mishandles a None stdout instead of the documented "None on any failure" contract.

Not a goal

  • Does not change the encoding of any other subprocess spawn; only the two tailscale CLI sites.
  • Does not add Windows-specific code paths; the fix is one shared UTF8_TEXT mapping used repo-wide.

What changed (motivation → approach → change)

  • Symptom: non-ASCII Tailscale device names fail to resolve, or malformed bytes escape the failure contract, on Windows.
  • Root cause: both spawns pass text=True with no encoding=, deferring to the host code page.
  • Change: both sites now splat **UTF8_TEXT (the repo's existing mapping for children whose encoding is knowable, carrying errors="replace"), so a malformed byte costs one character rather than the payload. tailnet.py and tailnet_serve.py carry the identical spawn and are both fixed. The two files are removed from .github/subprocess-encoding-baseline.txt because they no longer rely on the host code page.

Backwards compatibility

Compatible: no field, type, or contract is rejected, renamed or removed; the change only fixes how the two spawns decode their own child output.

Tests

test/test_tailnet_e2e.py adds coverage that the tailscale CLI output is decoded as UTF-8 regardless of the host locale, including a non-ASCII device name and a malformed-byte input that must not escape the failure contract.

Manual verification

N/A — unit coverage sufficient; the behavior is deterministic given the mocked CLI output.

Related Issues

Supersedes #9351 (the same fix authored by @leonlaiyc on a fork; opened in-repo so the .github/ baseline edit does not trip the fork workflow-change guard).

Pattern harvest

Rule candidate: review-prompt
Pattern: a subprocess.run(..., text=True) with no encoding= on a child whose output is a known-UTF-8 protocol (JSON) or free text defers to the host code page and mis-decodes on Windows; prefer the repo's UTF8_TEXT mapping.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Both tailscale spawns run in text mode with no `encoding=`, so the child's
output is decoded with `locale.getpreferredencoding()` -- UTF-8 on POSIX, but
the legacy ANSI code page on Windows. Tailnet device names are operator-chosen
free text, and `tailscale ... --json` is JSON, which is defined as UTF-8
(RFC 8259 s8.1), so the correct decode is known rather than guessed. Both sites
now splat `UTF8_TEXT`, the mapping this repository already uses for children
whose encoding is knowable.

Two distinct failures, measured rather than assumed:

* Bytes that are also legal in the host code page decode to *different
  characters*, silently. A device named `測試-desk` comes back as other CJK
  characters on a cp950 host, and `_valid_magicdns_name` then rejects a name
  that was valid -- the tailnet origin is simply never derived.
* Bytes that decode under neither take a platform-specific path, and neither
  one is caught. On POSIX `subprocess.run` raises `UnicodeDecodeError`, which
  is a `ValueError` and so passes straight through
  `except (OSError, subprocess.SubprocessError)`. On Windows `capture_output`
  reads on a helper thread, so the same error kills that thread, prints a
  traceback to the gateway's stderr, and leaves `proc.stdout` as `None`.

Both break the contract `_run_json_detail` documents in its own docstring --
"``None`` on ANY failure ... every failure mode here means the same thing" --
by either answering wrongly or escaping. `errors="replace"` (carried by
`UTF8_TEXT`) restores it: a malformed byte costs one character, not the payload,
matching the policy #3669 established.

`tailnet_serve.py` carries the identical spawn and is fixed in the same commit,
because its own docstring says a second, subtly different copy of this spawn is
how one comes to be hardened and its sibling not.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Co-authored-by: leonlaiyc <276034861+leonlaiyc@users.noreply.github.com>
@iamwhatever
iamwhatever requested a review from a team as a code owner September 28, 2026 07:20
@iamwhatever iamwhatever added the drive-to-green PR claimed by drive-to-green pipeline label Sep 28, 2026
@iamwhatever iamwhatever self-assigned this Sep 28, 2026
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 28, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 0c8930280985a2c82736c1f64e3921a2a45d2e55 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the new .bat fixture branch of _raw_cli runs green on the Windows CI lane, since only that lane executes it.

[DESIGN-REVIEWED] 0c89302

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 0c8930280985a2c82736c1f64e3921a2a45d2e55 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Only the Windows-half claim ("kills the capture helper thread, leaves proc.stdout as None — measured") rests on a host CI never runs; confirm once on a real Windows box.

What this change ships

Inventory (5 items) — 5 justified

Intent: FIX — tailnet origin derivation stops mis-decoding operator device names (and stops letting malformed bytes escape the None-on-failure contract) by pinning both tailscale CLI spawns to UTF-8. The defect has checkable provenance: the undecodable-bytes test fails on base, where UnicodeDecodeError (a ValueError) passes the except (OSError, subprocess.SubprocessError) guard in tailnet.py:215.

  1. Non-ASCII Tailscale device names now resolve on legacy-code-page hosts (tailnet.py) — justified
  2. Same decode pin on the serve-side spawn (tailnet_serve.py, identical spawn) — justified
  3. Malformed CLI bytes now cost one U+FFFD character instead of crashing or voiding the payload, on every platform — justified
  4. Both files leave the encoding lint baseline, so the gate keeps them clean — justified
  5. New tests pin the UTF-8 decode with raw-byte fakes, including a guard that the CJK fixture really defeats cp950 — justified

Reuses the repo's existing UTF8_TEXT mapping (its docstring names this exact failure class); the remaining baseline entries are governed by the shrink-only ratchet, not left siblings of this fix.

[FIRST-PRINCIPLES-REVIEWED] 0c89302

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 0c8930280985a2c82736c1f64e3921a2a45d2e55 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 0c89302

Verdict parsed from the review's SHA-scoped output markers for commit 0c8930280985a2c82736c1f64e3921a2a45d2e55.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 0c8930280985a2c82736c1f64e3921a2a45d2e55: <one-sentence reason>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 0c8930280985a2c82736c1f64e3921a2a45d2e55 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 0c89302

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 0c8930280985a2c82736c1f64e3921a2a45d2e55: <one-sentence reason>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants