Skip to content

feat(app-kit): add a batch embed to ctx.embed - #36450

Open
bolichen97 wants to merge 1 commit into
mainfrom
fix/23174-app-batch-embed
Open

bolichen97 wants to merge 1 commit into
mainfrom
fix/23174-app-batch-embed

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Let App Kit apps embed many texts in one call with the gateway's shared embedding model, and get back the vector space's identity and a clear "not ready" answer.

ctx.embed embeds one text at a time and reports only model_id. An app that groups, deduplicates or shortlists its records has no batch call, no width or space signature to tell when its stored vectors went stale, no way to tell "the model is still downloading" from "the embed failed", and no instruction prefix for the instruction-tuned bundled model. Today it either imports the private kiro_crew.embeddings API or loads a second ~700 MB copy of the model.

Why it matters

Apps that need embeddings either lean on a private API that breaks on a wheel upgrade, ship their own model (a second resident copy, in a different vector space), or skip embeddings. Serving batches from the one shared model removes both, and running them behind the host's own memory work keeps an app's corpus loop from slowing memory recall.

Not a goal

  • No new HTTP route and no new listener. An app that wants to serve vectors over HTTP does it from its own routes, behind the existing app auth.
  • No coverage for processes outside the gateway (a script cron, an app backend with its own entry point): they have no ctx, and serving them needs a local route, which is a separate request.
  • No manifest permission. The single-text members already load the same model ungated, so a grant on the batch path alone would gate nothing; fair use comes from the bulk priority and the batch cap.
  • No change to the single-text ctx.embed members (available, model_id, embed_sync, run).

What changed (motivation → approach → change)

An app can now embed up to 64 texts in one call with the gateway's own model, learn which vector space the vectors live in, and hear "not ready" instead of guessing while the model downloads. Approach: extend the existing ctx.embed seam rather than add a route, because in-gateway app code already holds the context. Change: two new members on EmbedSDK (src/kiro_crew/apps/embed_sdk.py).

Part What the app gets
await ctx.embed.embed_batch(texts, *, instruction=None) Up to 64 texts in one call, on the host's background memory pool (mc-membg), one app batch at a time under an admission and a thread of its own (_APP_EMBED_BATCH_WORKERS in src/kiro_crew/executors.py), so a long batch never holds a worker that chat turns or the host's memory writes wait on
ctx.embed.embed_batch_sync(...) The same call, blocking, for the app's own worker thread
EmbedBatchResult ready, one vector per text in input order, model_id, dim, space_signature
reason when not ready invalid_input, too_many_texts, on_event_loop, not_ready (model downloading, loading, swapping, absent, or the bulk queue full), failed
Fair use The shared model's bulk priority (behind memory recall and explicit memory writes, level with the host's own bulk work) and the 64-text cap
instruction Prefixed to every text as Instruct: <instruction>\nQuery:

Nothing raises into the app. The batch members join the open App Kit seam level 10 (apps/app_sdk_level.py), which they share with the agent tool gate's fence on an installed app's credentialMasks, with permissions.chatLaunch and with permissions.browserSites, so APP_SDK_LEVEL stays 10 and an app feature-detects the batch members on ctx.embed, never the level alone. The app guide (docs/app-kit/api-reference.md), the App Kit spec and the memory spec's pool section (memory-skills-hooks.md) describe the call.

Builds on #31077 by @JLargent42.

Backwards compatibility

Compatible: the change only adds EmbedSDK members; every existing member, EmbedSDK() with no arguments and APP_SDK_LEVEL behave as before.

Removes nothing: no path, key, function or validator is removed or tightened.

Tests

test/test_embed_sdk.py adds cases for: vectors with model id, width and space signature; bulk priority behind memory recall; any app context batch-embeds without a grant; the 64-text cap; malformed texts and instruction; the instruction prefix; not_ready for an absent, cold or unconstructible backend; failed for a raise or malformed vectors, never raising; the blocking form refusing on the event loop; the async form running on mc-membg while consolidation's two admission slots are both held.

Fail-before: test_async_batch_embed_runs_on_the_background_memory_pool fails with TimeoutError when embed_batch goes back to consolidation's admission (run_in_memory_background_pool), and passes with this code (an Ubuntu Linux host run on base af5ca42876, with this embed_sdk.py, executors.py and test file byte for byte).

Determinism: test/test_embed_sdk.py passed 20 runs in a row at -n0 and a shuffled run (seed 206958309). test/test_flake_pattern_ratchet.py 71 passed; semgrep 1.78.0 on the diff, 0 findings.

Gates, run on an Ubuntu Linux host on this patch's code and test files on base 09e5482863: black, isort, flake8, mypy src/kiro_crew/, the comment-history, brand, harness-parity, feature-map, file-line-cap, sync-io, loop-bound-lock, docs-lint and memory-store-seam checks all exit 0. The related tests (test_embed_sdk.py, test_app_sdk_level_mirror.py, test_app_backend_sdk.py, every test file that mentions sdk_level, APP_SDK_LEVEL, Bot grants or app memory providers, the App Kit manifest, credential-mask and adoption-pin tests, the embedding preemption tests, the memory kind-routing, origin-gate and tier-policy tests, and every test file that drives memory_background) pass: 1,524 passed in 33 files. scripts/local-gate.py ran its three backend batches and its frontend batch: 68,636 backend tests passed and 218 skipped, and 15,731 frontend tests passed in 530 files. Its 5 backend reds (test_find_ui_auto, test_host_git_repo_exec_neutralization, and three cases of test_kas_orchestrator_memory.py::TestTheRealSessionNewRequest::test_an_unconfirmed_resume_is_evicted_without_deleting_its_record) fail the same way alone on a clean checkout of the base on that host. On this head, whose embed_sdk.py, executors.py and test/test_embed_sdk.py are byte for byte the ones that host ran, black, isort and flake8 on the changed Python files and the light repository checks (comment-history, brand, harness-parity, feature-map, docs-lint, memory-store-seam and the rest) exit 0, and the seam-level assertions of test_app_sdk_level_mirror.py, test_app_backend_sdk.py and the shipped adoption-pin level test, re-run as a script, hold at level 10.

Manual verification

Smoke test on an Ubuntu Linux host, on this patch's code on base 09e5482863, against a real isolated gateway (throwaway KIROCREW_HOME and KIRO_HOME, loopback port, owner session) with a throwaway user app whose routes call await ctx.embed.embed_batch(...). The gateway downloaded the pinned model itself (its sha256 matches the pin) and served it in-process:

Check Result
65 texts reason: too_many_texts, ctx.sdk_level 10
Before POST /api/memory/enable-embeddings, 3 texts reason: not_ready
After it, 3 texts ready: true, 3 vectors, dim 1024, model_id qwen3-embedding:0.6b, space_signature efb9254d4de91136, ctx.sdk_level 10
asyncio.gather of three 3-text calls three ready results, 3 vectors each, one space
64 texts; 3 texts with instruction; 0 texts ready with 64 vectors; ready, a different vector than without the instruction; reason: invalid_input

The model finished loading inside the enable call, so this run did not catch the cold-load not_ready window; test_batch_embed_reports_not_ready covers it. ctx.sdk_level is 10 on this head too, where the batch members share main's level 10.

POST /api/apps/embed-smoke/batch -> HTTP 200
{"sdk_level": 10, "result": {"ready": true, "model_id": "qwen3-embedding:0.6b", "dim": 1024, "space_signature": "efb9254d4de91136", "reason": "", "n_vectors": 3, "first_vector_head": [2.213590621948242, -3.4879093170166016, -1.252328872680664]}}

Screenshots / video

N/A — the diff changes no frontend file, so nothing on screen changes.

Related Issues

Closes #23174

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 2789d03dc6c848b9ca0b4c90fb7bf159bd278b1b touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 2789d03dc6c848b9ca0b4c90fb7bf159bd278b1b — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the change only adds members to the existing ctx.embed seam, removes nothing, and gives batch work its own admission so it does not take turn-context or consolidation slots.

Notes

  • embed_batch imports the private _run_admitted and _APP_EMBED_BATCH_WORKERS from executors.py and spells out its admission key in embed_sdk.py. A run_in_app_embed_batch_pool in executors.py, shaped like run_in_memory_background_pool, would keep the pool, its slot count and its key with their one owner.
  • The admission semaphore exists once per event loop, but the extra mc-membg thread is shared by the whole process. If app code ever awaits embed_batch from more than one loop, those batches would compete with consolidation for its two threads.

[DESIGN-REVIEWED] 2789d03

Description read

[DESCRIPTION-READ] 2710875a09061b12b2a234f73617bf2bbb77f716aa1e26518c31f49d9a866877

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 2789d03dc6c848b9ca0b4c90fb7bf159bd278b1b — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. Only the instruction parameter is in question: it is string concatenation the app can do itself, and the host never uses that prefix.

Notes

  • Item 5 — inherited: the Instruct: … \nQuery: form appears in 1 file (embed_sdk.py, grep Instruct:|query_instruction over src/kiro_crew). The host's own recall never uses it, so it rests only on what the vendor says the model expects. The Goal line never asks for it. Subtraction: delete the instruction keyword and its _prepare prefix branch, and let the app prepend its own string to each text.

What this change ships

Inventory (7 items) — 6 justified

Intent: let App Kit apps batch-embed texts with the gateway's shared model and learn the vector space and readiness, without the private API or a second model copy. ADDITION.

  1. Apps can embed up to 64 texts in one awaited call (ctx.embed.embed_batch) — justified
  2. A blocking form for an app's own worker thread (embed_batch_sync) — justified
  3. Each batch answer carries model_id, dim and space_signature — justified
  4. A failed call returns a named reason (not_ready, too_many_texts, invalid_input, on_event_loop, failed) and never raises — justified
  5. An optional instruction prefix for every text — inherited: vendor-format concatenation the app can do itself
  6. The background memory pool gains one thread and app batches get an admission of their own — justified
  7. Spec wording changed from "Level 7 (APP_SDK_LEVEL = 7)" to "Seam level 7" — undeclared, rides along

[FIRST-PRINCIPLES-REVIEWED] 2789d03

Description read

[DESCRIPTION-READ] 6afd87956a00970a73783c82d61c832c87d46049e1cd8b9f721cf18d5c9ba11b

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@bolichen97
bolichen97 enabled auto-merge (squash) November 13, 2026 06:20
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 2789d03dc6c848b9ca0b4c90fb7bf159bd278b1b — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 2789d03

Verdict parsed from the review's SHA-scoped output markers for commit 2789d03dc6c848b9ca0b4c90fb7bf159bd278b1b.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 2789d03dc6c848b9ca0b4c90fb7bf159bd278b1b false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 9a26e777eff6a83b80e0d6bb778af000b75041e1 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 9a26e77

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 9a26e777eff6a83b80e0d6bb778af000b75041e1 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 13, 2026
@bolichen97
bolichen97 force-pushed the fix/23174-app-batch-embed branch from 7242cb2 to 61f2648 Compare November 13, 2026 09:04
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/app-kit/api-reference.md, docs/system-specs/modules/app-kit-platform.md, src/kiro_crew/apps/app_sdk_level.py and website/src/app-sdk/sdkLevel.ts since this head was judged (af5ca42 -> 187b4da); CI re-running.

Main's #33589 took App Kit seam level 7 for agentReadOnlyDirs, which made this PR conflict, so the batch members are now seam level 8 on both sides. The new base also carries #36457, the fix for the mypy red on main that failed Backend Lint & Type Check on the previous head. The embed code and its tests are unchanged.

[operator: bolichen97]

@github-actions github-actions Bot added merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Nov 13, 2026
@KiroCrewCaptain

KiroCrewCaptain commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 421 lines, more than the 400 it may approve on its own.

Additive change: EmbedSDK gains embed_batch and embed_batch_sync. They return an EmbedBatchResult carrying model_id, dim, space_signature and a reason. They call the backend's existing embed_batch at PRIORITY_BULK and run on mc-membg under an admission of their own with one slot, plus one added pool thread, so consolidation and the backfill and repair loops never wait on app batches. The backend signatures (llama and Ollama) and their None semantics match how the PR handles them. The seam level rises to 8 on both sides, kept equal by the mirror test, and no reader hard-codes 7. Specs are updated in the same commit. Tests cover each reason and the pool placement, and would fail without the change. No regression on the existing members. The CI red looks like an unrelated timing flake, and the merge conflict is a process item.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 9a26e777ef · 421 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@bolichen97
bolichen97 force-pushed the fix/23174-app-batch-embed branch from 61f2648 to 9a26e77 Compare November 13, 2026 15:25
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/app-kit/api-reference.md and docs/system-specs/modules/app-kit-platform.md since this head's CI ran (187b4da -> c000f31); CI re-running.

Main's #36341 (the backend SDK) and #36338 (the desktop window keys) both joined App Kit seam level 7, which made this PR conflict in those two docs. Level 7 now lists all of main's seams, and the batch members stay seam level 8. The new base also carries #36504, the fix for the docs-lint red on main that failed Source Gates on the previous head; that head's other reds were GitHub API rate limits inside CI. The embed code and its tests are unchanged.

[operator: bolichen97]

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Nov 13, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] The only red on this head (9a26e77) is Backend Tests (3.12, 7), and Coverage Gate fails closed behind it. The failing test is test/test_denied_commands_security_cost.py::TestSelfModuleIndexIsLinear::test_the_scan_is_linear_not_quadratic: 2k tokens took 0.512s against its 0.5s wall-clock bound (job).

This PR does not touch the deny gate. The same test fails its bound on many unrelated heads, and shard 7 is also red on main's latest completed runs. The fix for that test is open in #28293, so this PR waits for it to land rather than re-running the shard. Every review lane passes on this head.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 14, 2026
An app that groups or deduplicates its records needs many vectors at
once, the identity of the space they live in, and a clear answer when
the model cannot serve yet. ctx.embed only offered one text at a time
at normal priority, which puts an app's corpus loop beside the host's
own memory writes.

ctx.embed gains embed_batch and embed_batch_sync: up to 64 texts per
call at the shared model's bulk priority, an EmbedBatchResult with the
vectors, model id, width and space signature, a reason when it carries
none (not_ready whenever the backend has no embedding available), and
an optional instruction prefix. The async form runs on the background
memory pool, because paced bulk work would hold an embed pool worker
that turn context waits on for minutes, under a one-slot admission and
a thread of its own, so consolidation never waits behind an app's
queue of batches. No manifest permission and no route, like the rest
of the seam. The batch members join the open App Kit seam level 10,
so an app feature-detects them, never the level alone.

Closes #23174

Co-authored-by: Jacob Largent <65189184+JLargent42@users.noreply.github.com>
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Rebased onto current main to resolve the merge conflict in docs/app-kit/api-reference.md, docs/system-specs/modules/app-kit-platform.md and src/kiro_crew/apps/app_sdk_level.py.

@bolichen97
bolichen97 force-pushed the fix/23174-app-batch-embed branch from 9a26e77 to 2789d03 Compare November 18, 2026 04:17
@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: action required A blocking check or review needs attention

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expose the shared embedding model to apps (SDK function or local route)

2 participants