Repository navigation
feat: host feature-video clips on a signed CDN manifest - #9972
Conversation
|
Intent: Let a feature-intro clip ship without cutting a release, by serving clips from a signed per-release CDN manifest and caching them locally — with the signature, a per-file sha256 pin, and a host pin from that signed document as the three things that make CDN bytes safe to act on, and one fail-closed governance scope that can withdraw the whole fetch. |
0413697 to
047cbce
Compare
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS Sound, well-argued design; one seam is off — the model-download redirect pin on the agent-writable config knob breaks real mirrors while protecting nothing. WatchThe consolidation gives Suggestions
[DESIGN-REVIEWED] 0b091e1 |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of All evidence gathered. Emitting the review. First-Principles-Verdict: CONCERNS The random-draw selection rewrite rides along — it deletes a deliberate ordered-walk pin on a preference whose zero option, by the diff's own words, costs nothing. Not justified as shipped
What this change shipsInventory (10 items) — 9 justifiedIntent: ship feature-intro clips from a signed, per-release CDN manifest that the gateway downloads, verifies and serves locally, instead of bundling media in the wheel — ADDITION.
Watch
Subtractions
[FIRST-PRINCIPLES-REVIEWED] 0b091e1 |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 4.8 Review — ✅ no blocking findingsReviewed Review detailsI've reviewed the security-critical paths of this PR: the signature verification core ( The discovery pass produced no candidates. Working through the changed code myself, every place a real defect could hide — nested-payload canonicalization vs. the flat feed shape, No findings. [OPUS-REVIEWED] 0b091e1 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
047cbce to
d4ca87d
Compare
|
self-added: yes
|
|
self-added: yes
|
|
self-added: yes
|
|
self-added: yes
|
|
self-added: yes
|
|
self-added: yes
|
|
self-added: yes
|
|
self-added: yes
|
|
self-added: no
|
|
self-added: yes
|
|
self-added: yes
|
|
self-added: no
|
|
self-added: no
|
|
self-added: no
|
|
self-added: no
|
|
self-added: no
|
|
self-added: no
|
|
self-added: no
|
|
self-added: yes
|
|
self-added: no
|
|
self-added: no
|
|
self-added: no
|
|
self-added: yes
|
|
self-added: no
|
|
self-added: no
|
|
self-added: no
|
|
self-added: no
|
|
self-added: no
|
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsFINDING -- src/kiro_crew/feature_videos_cache.py:550 -- False positive or not applicable? A repository writer can comment: |
|
self-added: yes
|
Feature-intro clips were a static tuple compiled into the package, with the media riding in the wheel. That is the right shape for two clips and the wrong one for a library: every added video needed a release. Clips now come from a manifest published per release folder on the CDN, signed with the same offline key cli.sh pins for the update feed under a distinct schema string, and cached in ~/.kiro/crew/feature-videos/<release>/ by a background pass at gateway boot. The static catalog stays as the fallback for an install that never fetched a manifest. Three properties make it safe to act on bytes from a CDN. The manifest is verified and an unverified one is discarded WHOLE, because the signature covers the document. Every clip is sha256-pinned by that signed manifest, so a tampered object can only fail verification. And the one case where the browser fetches media off this origin -- an eligible clip that is not cached yet -- is pinned to the host the signed manifest declares. Fetching any of it is governed by capabilities.feature_videos_download, fail-closed, consulted at four chokepoints: the manifest fetch, the clip download, the src handed to the browser, and POST /api/feature-videos/ fetch-all. Already-cached clips keep playing under a denial. Selection now draws uniformly among eligible clips instead of taking the first in catalog order. Order stopped being a statement about the user: with a hosted library, catalog order is publication order, so the first entry would always win and the rest would only be reachable by retiring the ones ahead of it. Eligibility is unchanged and still deterministic. The streamed-and-verified transfer itself moves out of embeddings.py into asset_downloader.py, shared by both callers, with resume added: a partial stays in a .part staging file nobody serves and the next boot continues it. embeddings.py keeps everything that is about the model. Its too-small branch also stops mis-reporting -- the staged size is now read before the unlink, where the old inline copy read it after and degraded every undersized download to a generic transport error. New: GET /api/feature-videos/probe (server-side reachability, because the modal's <video> is preload="none" and a client cannot discover a dead src before the user's "Got it" records a permanent verdict), POST /api/feature- videos/fetch-all, and GET /feature-videos/<release>/<file> -- a dynamic route, never add_static and never a listing, validating both URL components against the manifest's own rules and re-checking containment after resolve(). dashboard.feature_videos_enabled keeps its default of false.
|
self-added: yes
|
|
self-added: no
|
|
self-added: no
|
|
self-added: no
|
|
self-added: no
|
|
self-added: no
|
|
self-added: no
|
|
self-added: no
|
|
self-added: no
|
|
self-added: no
|
|
self-added: no
|
|
self-added: no
|
Problem / Motivation
A feature-intro clip could only ship inside the wheel. The catalog was a static
tuple compiled into the package and the media rode along with it, so adding one
video meant cutting a release — and every install paid for every clip, watched or
not. That is the right shape for two clips and the wrong one for a library.
Why it matters
Hosting the clips decouples recording one from cutting a release, and takes the
media out of the wheel: an install with the feature off (the default) never
fetches a clip, and an install with it on fetches its own release's clips in the
background, once, and plays them from disk after that. But a hosted clip is a file
the gateway fetches from the open internet, so acting on it safely is the whole
problem this PR solves.
What changed (motivation → approach → change)
Clips now come from a manifest published per release folder on the CDN, signed
with the same offline key
cli.shpins for the update feed under a distinctschemastring, and cached in~/.kiro/crew/feature-videos/<release>/by abackground pass at gateway boot. The static catalog stays as the fallback for an
install that has never fetched a manifest.
Three properties make it safe to act on bytes from a CDN.
"the bad entries are dropped", because the signature covers the document.
platform/feed_trust.py's verification core is now shared: the newverify_document_signaturediffers from the CLI feed's entry point only inallowing a nested payload and taking its own size cap. One trust root, and the
two document kinds stay non-interchangeable because each requires its own
schemainside the signed payload. One key with a third consumer is arotation fact, so the release runbook (
docs/build/release.md) and thesigning README name it beside the existing two.
trusted. A tampered object can only fail verification; it cannot reach the cache.
opener that verifies TLS, refuses any redirect off the exact origin (hostname
and port), and checks the sha256 before a file is installed. A hosted clip
that is not on disk yet is not offered — not from the CDN, not from anywhere —
so every byte a
<video>plays was verified here first. A manifest replacesthe bundled list rather than adding to it, so on the launch where a manifest has
arrived but no clip has landed yet, nothing plays; the pass runs in the
background and the next launch shows what it landed. That one quiet launch is
the trade, chosen over ever offering a bundled clip and its hosted successor as
two videos. This is a validation invariant (
validate_asset_pathadmits onlythis origin), not a fourth governance consult — the three chokepoints below are
the whole governance surface.
Governance. New
capabilities.feature_videos_downloadscope (fail-closed, inthe egress family with
telemetry/publish), consulted at three chokepointsbecause any one alone is a half-control: each manifest request (the fallback walk
can make up to four, and re-asks before every one, so a withdrawal between two
candidates stops the next), each clip request (the poster and the clip each take
their own audited answer, so a withdrawal landing during one transfer is not
followed by the next request on the old permit), and
POST /api/feature-videos/fetch-all(403, rather than a task that would denyitself). All three are server-side, and that is the whole surface: the browser is
never handed a CDN url, so there is no client-side fetch for the ceiling to miss.
The operator's kill switch is re-read per entry in the same loop, so turning the
feature off mid-pass stops the pass the same way a withdrawn ceiling does. The 60s
memo (
download_permitted_cached) has exactly one reader, thedownload_enabledfield on
/status, which the settings panel polls while a transfer runs. Already-cached clips keep playing under a denial: withdrawing bytes already on disk is a
separate decision this row does not make. No CSP change:
media-srcstays'self' blob:. The answer is deliberately not repeated onGET /api/dashboard/config, which every install fetches on every load and where anaudited read would spend a governance decision on a readout nobody consumes.
Read, write and delete under the cache share one anchor.
checked_cache_root()refuses a cache root that is itself a symlink and returns its canonical path; every
cache write (
ensure_cache_dir, which the manifest store and both clip transfers askfor the folder), the serving route (
resolve_served_path) and the eviction pass(
release_foldersand thermtreein_remove_release) derive their containmentfrom it, and each additionally refuses a symlinked release folder and re-checks that
the folder it is about to write, serve or delete resolves to
<canonical root>/<release>. Without that, a planted<root>/<release>link would make themanifest store and clip staging land wherever the link points, and a linked ROOT
would hand the eviction pass real directories outside the cache to delete. A link is
refused rather than followed even when its target is inside the cache; the write
side opens the root no-follow and proves the descriptor's real path is the canonical
root BEFORE creating anything under it, creates the release folder relative to that
held root, and applies the owner-only mode through the folder's own descriptor
(
fchmod; on Windows the DACL by name under a handle that blocks the rename) — amkdirorchmodby name after a by-name check would follow a link planted inbetween, and the chmod has no undo; and the delete side re-checks immediately before
rmtree, because that call is the irreversible one.The check is then settled on the descriptor, not the name. A by-name check
followed by a by-name open leaves a window: rename the checked release folder (or
the root) away, put a link at its name, and the open — however carefully the LEAF is
handled — follows the link and finds an ordinary file or folder at the end. Both
sides now ask the kernel about the thing they already hold open. Reading: the
serving route opens root, release folder and file as a chain of no-follow opens,
each relative to the descriptor before it (the root on its own name, so a root
swapped for a link is refused rather than followed — a by-name
resolve()of theroot would have handed back the link's target as if it were the root), and requires
the open file's real path (
pinned_fs.fd_real_path—/proc/self/fdon Linux,F_GETPATHon macOS,GetFinalPathNameByHandleWon Windows) to be the held root'sreal path plus
<release>/<name>, and its link count to be 1, so a root or folderswapped for a link after the resolver's checks is a 404 and a hard link into the
cache (a second name for an outside inode, invisible to every path rule) is not
served. Writing: every write under the
cache — both transfers, the install rename, the receipts, the manifest store — runs
inside
feature_videos_manifest.pinned_release_dir, which creates and checks thefolder by name, then holds it open (
asset_downloader.pin_target_dir: the ancestorchain pinned one
openatat a time on POSIX, a handle that blocks renames of thefolder and everything above it on Windows), proves the DESCRIPTOR's real path is
<canonical root>/<release>, and hands the downloader aPinnedTargetDirso thestaging open, the resume hash, the install rename and the lockdown are
descriptor-relative on POSIX and by name under the held handle on Windows. The real
path is what makes the pin sufficient: a swap landing before the open resolves is
followed by the resolve and pinned perfectly, and only the kernel's answer for the
inode already held can tell. A platform that cannot read it refuses. Deleting:
eviction removes a release folder through
pinned_fs.remove_tree_pinned— thefolder is opened with its ancestor chain pinned, the removal is approved only when
that open directory's real path is
<canonical root>/<release>, and every entryunder it is removed against the inodes one scan found; on Windows the folder is
held open (the handle blocks a rename of it and of every ancestor), proven the same
way, emptied under that hold and
rmdired last. Reading back: the cached manifestis read no-follow through a pinned parent and capped at the 1 MiB the network fetch
already enforces, so a link planted at its name is refused rather than read to
exhaustion. And every link check under the cache refuses a Windows junction the way
it refuses a symlink (
platform_compat.is_link_or_junction): a junction lstat's asa plain directory, and a non-admin user can make one. Pinned by tests that rename
the pinned folder away and plant a link at its name mid-transfer (bytes land in the
moved folder, the link's target receives nothing), swap the root between the check
and the open (refused by the real path — on the write side and on the delete side),
swap the release folder after the serving route's resolution (404), swap it after
the eviction listing (nothing deleted, the link's target intact), hard-link a file
into the cache (404), plant a link at the cached manifest's name (no manifest), and
pad a cached manifest past the cap (no manifest).
download_tokeeps its by-name behaviour for a caller that passes notarget_dir(the model download;#10018tracks the siblings), namedTargetDirso which callers hold which guarantee is visible at the call site.A re-published clip is fetched again. Each verified install leaves a receipt
beside the file (
.<name>.sha256) naming the pin it passed, andis_cachedrequirespresence, the declared clip size and a receipt equal to the pin the manifest in
force carries. Without the receipt, a signed re-publish that keeps a basename and
changes its bytes — same clip size, new content; or any re-encoded poster, which has
no size to compare — would read as cached and be served stale for the life of the
release, with nothing downstream re-hashing. The receipt answers the question a
re-hash would (was THIS content verified?) at the cost of one tiny read on the polled
route; a file with no receipt is never counted as cached; the receipt's own name is
not servable (dot prefix,
.sha256suffix). A receipt that cannot be written failsthat ENTRY with a stated reason — the media stays, reads as not cached, and the next
pass re-fetches it — rather than raising out of the background pass and leaving the
status stuck on
downloadingwith every later entry skipped.Selection draws uniformly among eligible clips instead of taking the first in
catalog order. Order stopped being a statement about the user: with a hosted
library, catalog order is publication order, so the first entry would always win
and the rest would only be reachable by retiring the ones ahead of it.
Eligibility is unchanged and still deterministic — a retired clip, or one for a
feature you already use, still cannot appear. Every candidate is already on disk,
so the pick costs no egress and no spinner.
The transfer engine moved out of
embeddings.pyintoasset_downloader.py,shared by both callers, with resume added: a partial stays in a
.partstagingfile nobody serves and the next boot continues it.
embeddings.pykeeps everythingthat is about the model (which url, the pins, the Ollama salvage, the retry ladder,
the progress dict), and
redact_model_urlkeeps its own path-preserving redactionbecause
kirocrew doctorprints it to name which model file resolved. Onebehaviour change for the model transfer, intended: it now goes through the shared
opener, so its redirect policy is the shared one. The old inline
urlopenfollowedany redirect from any url. Now: the CDN default and the agent-writable
memory.embed_model_urlconfig knob may redirect only within their own host andport; the
KIROCREW_EMBED_MODEL_URLenvironment override may redirect toanother
httpshost, never to plaintext. The split is about who chose the url. Anenv value was set by whoever launched the gateway, and their own mirror commonly
answers with a 302 to a storage host (an artifact store fronting a bucket, a bucket
redirecting to its regional endpoint) — refusing that would break the ordinary
mirror shape for no gain, since the sha256 pin decides what is installed. A config
value can be written by the agent's own tools and the CDN is not ours to trust with a
destination, so both keep the host pin. All four cases are pinned by tests. A second private
downloader would be a second place for "did we verify this before installing it?"
to be answered differently. This
consolidation is partial on purpose.
papyrus/backend/tectonic.pyandpptx_maker/backend/engine_source.pykeep their own stream-hash-verify-installloops and adopt only
make_ssl_context()from the shared module (abehaviour-preserving dedup of their SSL-context copies — the reason those two
app-backend files appear in this diff).
stt/models.pyis not touched at all: itkeeps its bare
urlopenand its own loop, as doesplatform/wheel_engine.py::_download_to_file. Moving all four ontodownload_toistracked in #10018 and is not
this PR's job: each has its own install contract to re-verify, and folding four
more call sites into a feature-video change would bury the review of both.
The staging file is never written through a link, on any platform, with no
check-to-open window. A fresh transfer removes whatever sits at the staging name
(a link is removed as a link; a directory is refused) and creates it exclusively
(
O_CREAT | O_EXCL), so a link planted in between fails the create itself. A resumedtransfer opens the partial ONCE, for read and append, and then compares the
descriptor to the name —
fstatagainstlstat, device and file index — andrefuses when they are not the same object.
O_NOFOLLOWalready refuses a link on POSIX; the descriptor comparisonis what covers Windows, where that flag does not exist and
os.openfollows areparse point. The descriptor must also be the inode's only name (
st_nlink == 1):a hard link at the staging name is not a link to follow and IS the named inode, so
it passes both rules above, yet appending to it would write into a file that also
answers to another name and a digest failure would unlink only the staging name.
And what reaches the final name is the inode that was hashed, or nothing: the
digest covers the bytes written to one descriptor, the rename is by name, so after
the rename the installed name is opened no-follow and its identity compared to that
descriptor's — a file swapped in at the staging name between the last write and
the rename is unlinked and the transfer fails instead of landing under a digest it
never met. On POSIX that descriptor stays open until the check is done, because an
inode with an open descriptor cannot be handed back to a file recreated at the
same name (Windows must close it before the rename, and its file ids carry a
reuse sequence number instead). A resumed transfer is held to the same rule from
the other side: the prefix is hashed through the very descriptor the rest of the
body is then appended through, and that descriptor is the one the install checks
— the partial is never re-opened by name after it was hashed, so a same-size file
swapped in at the name between the hash and the append is neither appended to nor
installed. All of it is pinned by tests.
A redirect is followed only to the same origin — hostname AND effective port —
unless the url is the operator's own.
_SameHostRedirectHandler, the default forevery caller and the only policy for a url from a CDN, a signed manifest or a config
file, refuses a redirect off https, to another hostname, or to another port on the
same hostname (
:8443on the trusted name is a different service, and on a box thatco-locates an internal listener with that name it is the same SSRF one port over).
An explicit
:443is the same origin as no port. A non-numeric port is refusedrather than raised.
build_opener(allow_cross_host_redirects=True)is the oneopt-in: it swaps in
_HttpsOnlyRedirectHandler, which still refuses plaintext butlets the host change, and only a url from the gateway's process environment
(
KIROCREW_EMBED_MODEL_URL,KIROCREW_FEATURE_VIDEOS_MANIFEST_URL) is passedwith it — see the model paragraph above for why that line is drawn at "who set it".
A refused redirect is a named failure, not a bare
HTTPError 302: both handlersraise
RedirectRefused, whose reason carries the two hostnames and the policy inforce (never a path, query or credential), and
download_toreturns it verbatim —so the dashboard's model-download status and the feature-video failure state read
"redirect from to refused: … honoured only when set through the
environment override" instead of a transport error that looks like a broken CDN.
The production CDN itself answers the model GET with a direct 200 (checked with a
HEADagainstd3j0sthz5doyui.cloudfront.net); no install depends on a hop.The manifest mirror override is an environment variable, not a config key.
KIROCREW_FEATURE_VIDEOS_MANIFEST_URLis the only override channel. There is nodashboard.feature_videos_manifest_url:config.jsonis writable by the agent's owntools, and a url the agent can choose is a GET the gateway makes to any https host on
the agent's behalf — before the signature check has anything to check. That is a
request target, not a preference, so it belongs to whoever launches the gateway. (The
existing
memory.embed_model_urlconfig knob has the same shape and predates this PR;it is recorded in #10018 alongside the other downloader siblings.) It relocates the
MANIFEST fetch and nothing else:
cdn_baseis inside the signed document, so a copiedmanifest still names the public CDN, and serving clips from another host needs a
manifest signed with that host in it; the user doc says exactly that, and does not
call this an air-gap story. The override names
the manifest directly, so it is fetched once, and the
releaseit declares need notmatch the running version — but it must be one the running build reads back (this
release, this minor's
.0, or the.0of up to three earlier minors). A documentoutside that window is refused with a logged reason instead of being cached under a
folder the next start would never open.
/nextanswers{video, enabled}and nothing else. There is nosourcefield — every offer is played from this origin, so it could only ever say
local,and a value with one possible reading has no reader — and no
download_enabled:whether bytes may be pulled decides what the background pass lands, never what this
route offers, and the settings panel reads it from
/status. The merged frontend(#9995) declares both optional and defines absent as local / off, so this is
contract-safe. There is also no probe route: every offer's files were proved on
disk by the request that offered it, so the dialog opens on the
/nextansweralone. The frontend's
featureVideoProbeclient and itssource === 'remote'branch are now unreachable code on
main; removing them is a frontend-onlyfollow-up and is not attempted from this branch.
New endpoints:
POST /api/feature-videos/fetch-all— runs the pass now, rate limit lifted. Itrefuses instead of starting a task that would no-op: 409
feature_disabledwhenthe kill switch is off, 403
governance_deniedwhen the ceiling forbids thefetch, so the settings panel never polls for bytes that are not coming.
GET /feature-videos/<release>/<file>— the cache reader. A dynamic route, neveradd_staticand never a listing: it validates both URL components against thesame rules the manifest applies — including the suffix rule, so only a name the
parser could have admitted for a clip (
.mp4) or a poster (.jpg/.jpeg/.png/.webp) is served, theContent-Typecomes from a fixed table keyed bythat suffix, and
X-Content-Type-Options: nosniffis set. The release folder isin the user's data home, so a file another local process plants there is
reachable by name; with the suffix rule a planted
x.htmlis a 404, and nothingthis route answers is a type the browser executes on the dashboard's origin. It
anchors containment to the canonical cache root and refuses a symlink at the
release folder and at the file alike. A manifest names files, so a link has no
legitimate reader — it is refused rather than followed, even when its target
sits inside the cache. The body is then streamed from a descriptor opened ONCE
with no-follow semantics (
O_NOFOLLOW; the reparse-point flag on Windows) andtype-checked with
fstat— not throughFileResponse, which reopens by pathafter the check and would follow a link planted in between.
Rangeis honoured(206 +
Content-Range, 416 for an unsatisfiable one) so the player can seek, andHEADcarries the headers with no body./feature-videos/is inSPA_FALLBACK_EXCLUDED_PREFIXES: it is a data namespace, so a request with nosession gets the auth refusal, never
index.htmlwith a 200 that a<video>wouldrender as nothing.
dashboard.feature_videos_enabledkeeps its default offalse.Contract
This is the backend half of a three-way split. The publishing script is
PR #9964; the frontend is
PR #9995, already on
main,— this branch is rebased on it and its
FeatureVideo/FeatureVideoNext/FeatureVideoStatustypes were checked field by field against what the handlersemit.
/statuscarries every fieldFeatureVideoStatusnames;/nextomits thetwo optional ones (
source,download_enabled) whose absent reading the typesdefine.
Schema agreement with #9964 — verified field by field against its
scripts/feature-videos/_manifest.py: sameschemastring, same top-levelfields (
schema,release,cdn_base,generated_at,entries,signature),same entry fields (
id,feature,title,description,file,poster,sha256,poster_sha256,bytes,duration_s,doc,used_when,min_version), same canonical byte form (sorted keys, compact separators, ASCII,trailing newline, nested values included), same trust root.
Four consumer-side decisions worth stating, all of them "accept what the publisher
can emit":
key_idis optional here, and pinned when present. feat: sign feature-video release folders for the CDN #9964 records it on theKMS path and omits it for a local staging key. The key is pinned, so
key_idnever established trust — requiring it would make a publisher that omits the hint
indistinguishable from a forgery.
discard a valid release, and the discard is whole-document — the failure would be
"no videos", not "one clip missing". The fetch cap sits above it at 1 MiB, since
the document is the payload plus a signature.
Every bound, by name
Stated together so neither side has to infer one from the other. Each is also
documented at its definition in
src/kiro_crew/feature_videos_manifest.py._SIGNED_PAYLOAD_MAX_BYTESDEFAULT_MAX_PAYLOAD_BYTES_MANIFEST_MAX_BYTESbytes_MAX_ENTRY_BYTES_MAX_ENTRIESduration_s_MAX_DURATION_S0.0("unknown") instead of ending the parse —float()of a 310-digit integer raises, and one signed manifest must not take every entry down with it64 KiB is not the cap. It was the value before I read #9964, and it is not what
this PR ships — the code went to 256 KiB in the same change that reconciled the
schema. A publisher that self-limits to 64 KiB is safe (anything it emits verifies)
and needs no change on either side, but it must not document 64 KiB as the consumer's
limit, and the outer bound is 1 MiB rather than 256 KiB. The relationship is the part
worth holding onto: the consumer's payload cap must be ≥ the publisher's, because
exceeding it discards the whole manifest rather than one entry.
is what feat: sign feature-video release folders for the CDN #9964 accepts. The fetch path still only ever asks for
major.minor.patchfolders.cap already allows, so a document the publisher would emit cannot trip it instead.
MAX_CACHE_BYTES,8 PiB). A configured megabyte figure is compared against that ceiling before the
multiply, because the multiply alone overflows a finite 1e308 to
inf. A value thatis not a finite positive number reads as "no size cap", the same answer a missing key
gives — a ceiling nobody can act on must not become a reason to delete clips.
Signature envelope — unchanged, and now written down
Confirming for the publish-tool session:
verify_document_signatureadds no newcanonicalization rule. It and the CLI feed's
verify_manifest_signatureare twothin entry points over one
_verify_signaturecore, and the only two things eitherpasses in are is a nested payload allowed and what is the size cap. The key, the
canonical-JSON shape, the trailing newline, the ASCII encoding and the openssl
invocation are not parameters.
The exact bytes to sign are now spelled out as a numbered five-step block in
src/kiro_crew/platform/feed_trust.py's module docstring — copy from thererather than reading it off the code. One step is worth repeating here because it is
the one that fails silently and only for some entries: the payload is serialized
ensure_ascii=Trueand encoded ASCII, so a tool that emitted UTF-8 withensure_ascii=Falseproduces a document that verifies for every ASCII-only entryand never for one carrying a non-ASCII title.
Shared fixture
The fixture manifest deliberately exercises what two independent encoders disagree
about: two entries (so nested-list order is pinned), keys written out of
alphabetical order (so
sort_keysis doing real work), a non-ASCII title, andone entry with
used_when/min_versionset beside one with both empty.test/feature_video_fixture.pyis the seam to import, and it carries both halves:load_fixture_manifest()+pin_fixture_key(monkeypatch)pointsfeed_trust's pins at the committed public key, so verifying the committed bytesneeds nothing else;
mint_throwaway_key(dir)andsign_document(private, doc, scratch),the latter being exactly the canonical byte form above, for the signing half of a
round-trip test.
One deviation from the ask, and the reason: the PRIVATE key is not committed. A
private-key PEM in the diff is what
p/secretsexists to catch, and the SAST jobruns
--config p/secrets --erroron the diff — so committing one reds the gate, andthe only way to make it pass would be a
.semgrepignoreentry carving out our ownfile.
mint_throwaway_key()covers the signing half instead, and a test asserts thefixture directory contains no private key so this cannot regress. Two further guards
keep the fixture from ever becoming an authority: a test asserts the fixture key id
and PEM differ from the real
PINNED_KEY_ID/PINNED_PUBLIC_KEY_B64, andregenerate.pyrotates the key freely because nothing pins its identity — thefixture is proof of a byte format, never of trust.
Regenerating (only needed if the schema changes):
TestSharedFixtureintest/test_feature_videos_hosted.pycovers: the committedfixture verifies and parses; the non-ASCII entry survives; re-serializing the file
with different indentation still verifies (the signature covers the canonical form,
not the stored bytes — a verifier that hashed the file as stored would pass this
fixture and fail every real release); tampering fails; the helper's local
canonicalization agrees with the verifier byte for byte while a UTF-8 /
unsorted / pretty-printed payload signed with the same key does not; and the two
authority guards above.
One deliberate deviation from the work item's wording:
/statuskeepsstateas the
{id: {status, ts}}engagement map and reports the transfer's step as a newdownload_state. Renamingstatewould have been the smaller diff and the largerbreak — the settings panel already reads it.
Release fallback, as documented: this release, then this minor line's
.0whenthe running patch is not 0, then the
.0of up to three earlier minors. Neveracross a major boundary, which is where a stale clip is most likely to show UI that
no longer exists.
Flagged signals
! config/infra file changed— one newdashboard.feature_videos_cache_max_mbkey (the manifest mirror override is env-only, see above),
config-baseline.jsonregenerated from the dataclass registry, and one line in
conftest.pyextendingthe existing no-download floor to
KIROCREW_SKIP_FEATURE_VIDEO_DOWNLOADso no test can reach a CDN. No defaultbehaviour changes: the feature stays off.
Tests
test_asset_downloader.py(39) — resume from a partial, a server that ignores theRange(restarts rather than appending), a partial at/past the declared size, acorrupt prefix caught end-to-end by the digest, sha mismatch, size overshoot,
too-small, non-https and
file://refusal, rate-limit pacing, owner-only install,progress/verify callbacks, url redaction.
test_feature_videos_hosted.py(189) — real signatures, via a generated RSApair with
feed_trust's pins repointed at it (the same fixture shapetest_feed_trust.pyuses), so the positive path runs genuine openssl verificationrather than a stub that would pass for a manifest nobody signed. Covers: tampering
one entry discarding the whole manifest, another key, a disagreeing
key_id, thetwo document kinds not being interchangeable, every unsafe entry shape dropped
individually while good siblings survive (a reused file or poster basename included,
case-folded), release-candidate order and the major
boundary, fetch fallback and the location-mismatch refusal, the on-disk cache being
re-verified (an edited
cdn_baseis refused), owner-only permissions, evictionorder / running-release immunity, the download pass, the shared containment anchor (a
symlinked release folder or cache root is refused for writes and nothing lands
outside; the clip transfer is refused through the same gate; a linked root lists no
eviction candidates,
_remove_releaserefuses even when handed the path, and theserving route answers nothing), the override manifest rule (a release in
the fallback window is fetched once and read back after a restart; one outside it is
refused), a withdrawal between two fallback requests stopping the second, the
serving route's suffix rule (a planted
.html/.svg/.js/ extensionlessfile under the release folder is a 404 even though it is a regular file on disk;
the served type comes from the suffix table, not the bytes; the table equals the
parser's admitted suffixes), the install receipt (a re-published clip of the same
size, or a re-published poster, is not cached; files without a receipt and a
malformed receipt are not cached; the receipt is never served; an unwritable receipt
fails the entry, not the pass), the serving route's streaming (full body, four
Rangeshapes withContent-Range, four 416 shapes,HEADwithout a body, and thecheck-to-open window: a file swapped for a symlink or a directory after resolution
is a 404 and the target is never read; the release folder swapped for a link after
resolution is a 404; a hard link into the cache is a 404; a descriptor whose real
path cannot be read is a 404), the pinned write side (a transfer through a pinned
folder lands in that folder after its name is swapped for a link, and the link's
target receives nothing; a link or a file at the folder name cannot be pinned; the
root swapped between the check and the open is refused by the descriptor's real
path; no real path, no write — the entry fails with a reason and a path-addressed
receipt goes through the same pin), a non-string
docdropped rather than crashingthe parse, static-route path
hardening including a symlink
escape, every off-origin
srcrefused (the manifest's own CDN host included), anuncached hosted clip never offered over the network (the module imports no url
client at all), a manifest with nothing landed yet offering nothing that launch,
governance denial at each chokepoint including between an entry's poster and its
clip, and
/nextanswering exactly{video, enabled}with no governance read (acached clip still plays locally under a denial).
test_feature_videos.py— four existing tests updated where the change reshapesthem (first-in-order → an eligible pick; stability → whole-set reachability plus a
seeded uniformity check; the grown
/statusbody; a verdict withdrawing whicheverclip was shown). Every other assertion's intent is untouched.
test_embeddings.py— theurlopenmonkeypatch target follows the transfer intoasset_downloader, plus four tests driving the production redirect handlers over afake transport: the CDN default and the config-knob url refuse a cross-host 302 (the
second host is never requested, nothing installed, and the status error names the
refused hop and the remedy); the env override follows one to another https host and
installs the pinned bytes; the env override still refuses a 302 to plaintext. One
assertion changed on purpose: the too-small branch used to
read the staged size after unlinking it, so every undersized download surfaced as
a generic transport error; the shared engine reads it before, and now says "too
small". The test's own docstring called that a known quirk.
Local gates green: 356 tests across the four feature-video/downloader files, plus
test_embeddings.py,test_feed_trust.py,test_social_share_governance.py,test_dashboard_security_headers.py,test_route_registry.py, the governancesuite, and the dashboard-config knob suites (~1,900 tests).
black(baselinerespected),
isort,flake8,mypy,docs-lint, brand, harness-parity,feature-map, changelog-history, sync-io-in-async, comment-history,
agent-sdk-boundary, lockdown-before-publish, testpaths-coverage,
subprocess-encoding.
Docs
Owning specs updated in this commit:
governance.md(the new scope, its threechokepoints, and why no CSP change is needed),
learn-cron-dashboard.md(the endpoints, the selection rule, the cache and theserving route),
cli.md(one key, two signed document kinds),memory-skills-hooks.md(where the transfer now lives),docs/feature-map/README.md. User-facing: a "Hosted Clips" section insrc/kiro_crew/docs/feature-videos.mdand one row inconfiguration.md.Note
src/kiro_crew/docs/feature-videos.mdis also touched by #9964 — both editsare additive to different sections; whichever lands second rebases.
Linked issue
no linked issue: this is one work item of an internally-tracked goal, dispatched
with its own written spec. PR #9964
is a sibling work item of the same goal, not an issue this closes.