Repository navigation
fix: route stt and wheel downloads through the shared redirect policy - #32318
Conversation
The whisper model and ffmpeg decoder fetch (stt.models.stream_pinned_payload) and the update engine's manifest and wheel fetch (wheel_engine_transfer's _read_bounded) opened their connection with urllib.request.urlopen, whose default redirect handler follows a hop to any host, plaintext included. Both now open it through asset_downloader.build_opener, the one place that answers which redirects a pinned transfer follows. The stt urls are source constants or the operator's environment override and their upstreams redirect to a download host of their own, so they take the cross-host, https-only policy. The release CDN serves the manifest and wheel itself, so that transfer keeps the same-host policy, except while an operator's KIROCREW_CDN_BASE mirror is set. Each loop keeps its own bounds (pre-write size ceiling, cancel, total deadline, mkstemp staging), which download_to does not carry, so only the connection moves. Part of #22230
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS No design-level concerns: both fetches now use the shared redirect policy, and each is given the policy matching where its url comes from. Notes
[DESIGN-REVIEWED] 9a3ef70 Description read[DESCRIPTION-READ] 5593637ebe27f4763f5f00892fb32f947ad042cda2aa381ff7fd45ef927578b3 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS One sibling the sweep missed: Notes
What this change shipsInventory (7 items) — 6 justifiedIntent: stop pinned stt and update downloads from following redirects to plaintext or off the CDN host. FIX.
[FIRST-PRINCIPLES-REVIEWED] 9a3ef70 Description read[DESCRIPTION-READ] c37a9f3f08d749d2f48e70ddf604f92e6ce60d5fd271063d0216f1d8d45a1dd8 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Approved for the on-call. Review lanes pass on 9a3ef70, no conflict, no open change request.
…#32318) The whisper model and ffmpeg decoder fetch (stt.models.stream_pinned_payload) and the update engine's manifest and wheel fetch (wheel_engine_transfer's _read_bounded) opened their connection with urllib.request.urlopen, whose default redirect handler follows a hop to any host, plaintext included. Both now open it through asset_downloader.build_opener, the one place that answers which redirects a pinned transfer follows. The stt urls are source constants or the operator's environment override and their upstreams redirect to a download host of their own, so they take the cross-host, https-only policy. The release CDN serves the manifest and wheel itself, so that transfer keeps the same-host policy, except while an operator's KIROCREW_CDN_BASE mirror is set. Each loop keeps its own bounds (pre-write size ceiling, cancel, total deadline, mkstemp staging), which download_to does not carry, so only the connection moves. Part of #22230 (cherry picked from commit 9ea1ed8)
Problem / Motivation
Goal: The whisper model, ffmpeg decoder and update-engine downloads follow redirects under the shared download policy, so a hop can no longer go to plaintext, or off the release CDN's host.
The speech model and ffmpeg decoder fetch (
stt/models.py) and the update engine's manifest and wheel fetch (platform/wheel_engine_transfer.py) still opened their own connection withurllib.request.urlopen. Its default redirect handler follows a hop to any host and down to plainhttp://. #10018 listed both sites; #21683 moved only the two app installs ontoasset_downloader.Why it matters
asset_downloaderis the one place that should answer which redirects a pinned download may follow. The update engine runs inside the gateway, which can reach the operator's internal network, so a redirect it follows to another host is a request nobody approved. A hop to plaintext lets anyone on the path watch and alter that leg before the digest check. The digest still guarded the installed bytes, but these two copies did not follow the shared rule.Not a goal
asset_downloader.download_to, which Route the stt model and wheel_engine downloads through asset_downloader.download_to #22230 also asks for and which stays open there. Each loop has bounds thatdownload_tolacks and that their tests pin: the speech fetch refuses an oversized chunk before writing it, can be cancelled, and stages throughmkstemp; the wheel fetch has a total deadline and anApplyCancelthat shuts the socket.download_towould need all of them first, so only the connection moves here.urlopencallers that fetch no sha256-pinned artifact (see Sibling sweep).asset_downloader's two policies themselves.What changed (motivation → approach → change)
Both sites now open their connection through
asset_downloader.build_opener, each with the policy that module defines for its kind of url:stream_pinned_payload)KIROCREW_WHISPER_MODEL_BASE_URL_read_bounded)KIROCREW_CDN_BASEis setA refused hop fails the transfer as before, with a message naming both hosts. The facade
platform/wheel_engine.pyre-exportsasset_downloader, because its facade test requires every project name an owner imports to stay bound on the facade. The stt spec, the update RFC andbuild_opener's list of callers inasset_downloader.pydescribe the new route. Tests answer the connection through a sharedtest/opener_helpers.pyinstead of patchingurlopen.Backwards compatibility
Breaking: a redirect to plaintext from any of these downloads, and a redirect off the release CDN's host while no
KIROCREW_CDN_BASEmirror is set, is now refused. Writer sweep on origin/main @ f80c7b8 (the redirect sources are the endpoints these urls name):huggingface.comodel urls answer 302 to another https host, still followed; GitHub release decoder urls answer 302 to another https host, still followed;updates.crew.kiro.devanddownload.crew.kiro.devanswer the manifest and wheel with a 200; an operator mirror underKIROCREW_WHISPER_MODEL_BASE_URLorKIROCREW_CDN_BASEmay still redirect to another https host.Reader: src/kiro_crew/stt/models.py:ModelStore.ensure -- chat -- the model host's redirect to its download host is still followed (test_the_model_hosts_redirect_to_its_download_host_is_followed)
Reader: src/kiro_crew/stt/decoder.py:maybe_autofetch -- chat -- the GitHub release redirect is a cross-host https hop, which stream_pinned_payload still follows
Reader: src/kiro_crew/platform/wheel_engine.py:download_verified_wheel -- release -- the release CDN serves the wheel with no redirect (test_a_redirect_off_the_cdn_host_is_refused, test_an_operator_mirror_may_redirect_to_another_https_host)
Reader: src/kiro_crew/platform/wheel_engine.py:fetch_verified_manifest -- release -- the feed serves the manifest with no redirect
Tests
test_stt_engine_download.py: the model host's redirect to another https host is followed and installed; a redirect tohttp://is refused with nothing requested from it and nothing left on disk. Both run the real opener over a fake transport.test_wheel_engine.py: a redirect off the release CDN's host is refused and nothing is installed; underKIROCREW_CDN_BASEa mirror's redirect to another https host is followed and the verified wheel lands.opener_helpers.answer_with, with their assertions unchanged.Determinism: the four new tests passed in five runs, each in a different shuffled order (
-n 0). They use no clock, thread or network: the transport is replaced per test withmonkeypatch.Proof: PROVEN test/test_stt_engine_download.py::test_a_redirect_to_plaintext_is_refused, test/test_stt_engine_download.py::test_a_present_model_is_verified_against_its_pin_not_just_its_size, test/test_stt_engine_download.py::test_the_pin_check_is_not_cached_against_forgeable_metadata, test/test_stt_engine_download.py::test_a_failed_download_is_reported_not_raised, test/test_wheel_engine.py::TestWheelDownload::test_sha_mismatch_refused_and_file_removed (head 9a3ef70)
Manual verification
Checked the live endpoints:
https://updates.crew.kiro.dev/feed/stable/latest-cli.jsonand the wheel url it names both answer 200 with no redirect;https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.binanswers 302 to another https host.Related Issues
Part of #22230
Refs #10018, #21683.
Sibling sweep
Swept: git grep -n "urlopen(" -- src/ (each hit read for a sha256-pinned fetch)
Sibling: src/kiro_crew/stt/decoder.py -- not affected -- it fetches through stream_pinned_payload, which this PR moves onto the shared opener
Sibling: src/kiro_crew/eval/bench/datasets.py:249 -- not affected -- a developer's benchmark command, never run by the gateway, fetching source-constant urls that the shared policy lets change host anyway; the sha256 pin decides
Pattern harvest
Rule candidate: semgrep
Pattern: a sha256-pinned download that calls urllib.request.urlopen directly instead of asset_downloader.build_opener
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributed under the project's Apache-2.0 license.