Repository navigation
Admin - Add App Builder project detail page with session traces link - #6
Merged
Merged
Conversation
RSO
approved these changes
Feb 4, 2026
iscekic
added a commit
that referenced
this pull request
Feb 6, 2026
## Summary Adds a new `cloudflare-o11y` Cloudflare Worker for LLM API observability, and wires the Kilo Gateway to emit metrics to it on every request. ## What it does ### New `cloudflare-o11y` worker - **Metrics ingestion** (`POST /ingest/api-metrics`): Accepts per-request metrics (provider, model, timing, status code, tokens, tools used, etc.) authenticated via Cloudflare Secrets Store. - **PostHog forwarding**: Captures every ingested metric as a PostHog event for analytics, forwarding the user's real IP for GeoIP resolution. - **Analytics Engine storage**: Writes each data point to Cloudflare Analytics Engine (provider, model, client, error flag, TTFB, complete request duration, status code) for time-series alerting queries. - **SLO-based alerting**: Cron-triggered (every minute) multi-window burn-rate alerting following [Google SRE Workbook approach #6](https://sre.google/workbook/alerting-on-slos/): - Three windows: 5m/1m page (14.4x), 30m/3m page (6x), 360m/30m ticket (1x) - Evaluates error rate (99.9% SLO) and latency (p50 at 5s, p90 at 15s) - Pages only fire for recommended models on `kilo-gateway`; everything else is downgraded to ticket - KV-based dedup with severity-aware suppression (pages suppress tickets for the same dimension) - Slack notifications via separate page/ticket webhooks ### Gateway integration (Next.js app) - The OpenRouter proxy route now emits metrics after each upstream response, including TTFB, tools available/used, token counts, user/org context, and IP address. - The response is cloned and drained asynchronously (via `after()`) to measure full request duration without blocking the client. - New `GET /api/recommended-models` endpoint (ISR, 1h revalidate) exposes the preferred models list for the o11y worker to determine page eligibility. ### CI/CD - New `deploy-o11y.yml` reusable workflow for manual and automated deploys. - Production deploy workflow auto-deploys the o11y worker when `cloudflare-o11y/**` changes. - Adds `fetch-depth: 2` to existing checkout steps so `dorny/paths-filter` works correctly. ## Changed files outside `cloudflare-o11y/` | File | Change | |------|--------| | `.github/workflows/deploy-o11y.yml` | New deploy workflow | | `.github/workflows/deploy-production.yml` | Auto-deploy o11y on prod, fix fetch-depth | | `src/app/api/openrouter/[...path]/route.ts` | Emit metrics on every proxied request | | `src/app/api/recommended-models/route.ts` | New endpoint for recommended models list | | `pnpm-workspace.yaml` | Register `cloudflare-o11y` as workspace package | | `pnpm-lock.yaml` | Lockfile update |
kilo-code-bot Bot
added a commit
that referenced
this pull request
Feb 22, 2026
…ation The old cloud-agent flow's processAnalysisStream() writes the raw markdown analysis content to the analysis field in security_findings on stream completion. This field powers the user-facing summary shown when clicking on an auto-dismissed finding. The callback-based cloud-agent-next flow must explicitly replicate this. Updates the plan to: - Add a 'Critical requirement' section under Background explaining the gap - Add step 4a in Phase 2.2 (handleAnalysisCompleted) to write raw markdown to the analysis field before running Tier 3 - Add risk item #6 about the analysis field migration gap - Add verification step for the analysis field in PR 2
jeanduplessis
pushed a commit
that referenced
this pull request
Feb 23, 2026
…ation The old cloud-agent flow's processAnalysisStream() writes the raw markdown analysis content to the analysis field in security_findings on stream completion. This field powers the user-facing summary shown when clicking on an auto-dismissed finding. The callback-based cloud-agent-next flow must explicitly replicate this. Updates the plan to: - Add a 'Critical requirement' section under Background explaining the gap - Add step 4a in Phase 2.2 (handleAnalysisCompleted) to write raw markdown to the analysis field before running Tier 3 - Add risk item #6 about the analysis field migration gap - Add verification step for the analysis field in PR 2
Closed
42 tasks
markijbema
pushed a commit
that referenced
this pull request
May 19, 2026
Review fix #6. SYSTEM_PROMPT_CAP_BYTES / REQUEST_BODY_CAP_BYTES were checked with `string.length` and trimmed with `string.slice`, which both operate on UTF-16 code units. For non-ASCII content the actual UTF-8 byte payload uploaded to R2 could be up to ~3× the intended cap (a single CJK char is 1 UTF-16 unit but 3 UTF-8 bytes; an emoji is 2 UTF-16 units but 4 UTF-8 bytes), so the cap was not bounding what the plan claimed it bounded. Slicing UTF-16 code units could also split a surrogate pair, leading the next UTF-8 encode to emit U+FFFD and breaking the deterministic-prefix promise (truncated content would hash differently across runtimes). Adds `truncateToUtf8Bytes(s, maxBytes)` which encodes once, walks back to a UTF-8 codepoint boundary using the 0b10xxxxxx continuation-byte rule, and returns valid UTF-8 with bytes <= maxBytes. Replaces the three call sites in `buildExperimentPromptCapture` to measure with `Buffer.byteLength` and truncate via the new helper.
markijbema
pushed a commit
that referenced
this pull request
May 20, 2026
…ing 'unreachable' comment Two small clarity fixes from #3325 review: - Rename the local variable bound from `pickAllocationSubject(input)` from `subjectPick` to `allocationSubject`. Matches the field name on the result and answers the reviewer's 'is this the user?' question inline. Pure rename. - Replace the 'Unreachable: getRandomNumber returns [0, totalWeight). Fail closed.' comment with an honest one. The branch is defensive fail-closed insurance for the case where `getRandomNumber`'s contract breaks (NaN, negative, weight rounding), not actually unreachable code. The captureMessage + 'unavailable' return is the right behavior; only the comment was wrong. Addresses comments #5 and #6 on #3325.
markijbema
added a commit
that referenced
this pull request
May 25, 2026
* feat(gateway): capture x-kilo-request and x-kilo-session headers
Phase 2 of the model-experiments rollout. Captures the per-message
client request id (used as the join key for PostHog feedback and for
`model_experiment_request.client_request_id` on experimented traffic)
and accepts `x-kilo-session` as a fallback session id when
`x-kilocode-taskid` is absent.
Also extends MicrodollarUsageContext with the experiment routing
fields (`modelExperimentVariantVersionId`,
`modelExperimentAllocationSubject`, `experimentPromptCapture`,
`clientRequestId`) so later phases can plumb them through without
touching every usage-context construction site.
* refactor(gateway): extract buildDirectProvider used by custom_llm path
Phase 3a of the model-experiments rollout. Pulls the kilo-internal/...
(custom_llm2) Provider construction into a shared
`buildDirectProvider` helper under
`apps/web/src/lib/ai-gateway/experiments/`. The experiment routing
path added in Phase 3b will consume the same helper, so direct-to-
upstream traffic for custom LLMs and experiments share one
implementation. Behavior is unchanged for custom_llm callers.
* feat(gateway): add experiment variant picker and routing-cache loader
Phase 3b of the model-experiments rollout. Adds
`apps/web/src/lib/ai-gateway/experiments/pick-variant.ts` with:
- `isPublicIdExperimented(publicId)` — fast Redis-backed membership
check with Postgres fallback, returning `unavailable` when both
cache and DB are inaccessible so the gateway can fail closed.
- `getRoutingExperimentForPublicId(publicId)` — Redis-cached
resolved experiment for a public id, including a single Postgres
`DISTINCT ON (variant_id)` query that picks each variant's current
`model_experiment_variant_version` and decrypts `encrypted_api_key`
once at cache-build time. Returns `none` (no routing-relevant
experiment), `experiment` (active or paused), or `unavailable`
(config/decrypt/validation failure or BYOK_ENCRYPTION_KEY missing).
- `pickModelExperimentVariant(input)` — deterministic picker using
`getRandomNumber`. Allocation subject precedence is user → machine
→ ip; missing all subjects returns `unavailable`. Variants are
walked by id-asc cumulative weight so live label edits never
rebucket users.
The `upstream` JSONB blob is validated by ExperimentUpstreamSchema
before use; partner-issued api keys live only in the encrypted column
and the cached payload, never in tRPC responses.
* feat(gateway): route experimented public ids to direct upstream
Phase 3c + 3d of the model-experiments rollout.
`getProvider` now returns a discriminated `GetProviderResult`
union (`provider` | `not-found` | `unavailable`) and adds a
new branch for experimented public ids that runs after the BYOK
checks and before kilo-internal/custom_llm and the
kiloExclusiveModels lookup. Active experiments produce a direct
provider built via `buildDirectProvider` plus experiment routing
metadata; paused experiments map to a local model-unavailable
response so traffic doesn't silently fall through to default
routing; cache/DB/config failures map to a 503
temporarily-unavailable response.
Experiment traffic uses three new flags on the provider result:
`skipBalanceCheck` (preview traffic is provider-funded),
`skipProviderPin` (the partner endpoint is selected by the
variant version, not by the org config), and
`skipKiloExclusiveModelSettings` (which
`applyProviderSpecificLogic` honors so the registry does not
override the variant's `internal_id`). Server-side organization
model/data-collection policy still runs against the public id.
Also excludes experimented preview public ids from `kilo-auto/free`
candidate sets so they are never silently selected on a user's
behalf — users only ever reach the experiment path by opting in
to the dedicated preview model id.
The route handler captures `x-kilocode-machineid` once into
`machineIdHeader` and threads it (plus the resolved client IP)
into `getProvider` so the variant picker can fall back from
userId to machineId to clientIp.
* feat(r2): add experiment-prompts content-addressed storage helper
Phase 4a of the model-experiments rollout. New helper module
`apps/web/src/lib/r2/experiment-prompts.ts` exposes
`putPromptIfAbsent(content)` (HEAD-then-PUT under a sha256 hex key
for automatic dedup) and `getPromptByHash(sha)` (admin out-of-band
reads). The bucket comes from a new env var
`R2_EXPERIMENT_PROMPTS_BUCKET_NAME` exported from the existing R2
client module; per-environment buckets are
`kilo-experiment-prompts-dev` and `kilo-experiment-prompts-prod`.
The helper is wired up but not yet called from the request hot
path — Phase 4b will add the bounded prompt capture and
attribution writer that consumes it.
* feat(gateway): persist experiment attribution after microdollar write
Phase 4b + 4c of the model-experiments rollout.
Adds `apps/web/src/lib/ai-gateway/experiments/persist.ts` with two
exports:
- `buildExperimentPromptCapture(request)` produces a bounded
capture of the canonical post-`transformRequest` upstream body,
splitting out the leading `role: 'system'` message for
chat_completions and tail-truncating either side at 4 MB. The
capture is stored on `MicrodollarUsageContext` so the async
`after()` path never retains a reference to the full uncapped
request body.
- `persistExperimentAttribution(input)` writes one row into
`model_experiment_request` after running two parallel
content-addressed R2 puts via `putPromptIfAbsent`, using
`__absent__` / `__failed__` sentinels for missing/failed
sides. Errors are reported to Sentry and swallowed — the billing
write must succeed independently of analytics.
`processUsage.ts` now surfaces the inserted `{ usageId, createdAt }`
out of `logMicrodollarUsage` and `processTokenData` so the
`accountForMicrodollarUsage` after-hook can chain
`persistExperimentAttribution` on the same row. The route handler
calls `buildExperimentPromptCapture` after provider-specific
transforms and stashes the result on the usage context only for
experimented requests.
* chore(experiments): drop unused MicrodollarUsageContext import
* docs(plan): update experimental-models-1 status for PR #3325
Marks Phases 2 + 3 as merged-pending against PR #3325 and Phase 4
as in-progress with 4a/4b/4c on the same PR. 4d (the
`model_experiment_request_stats` view) is now explicitly deferred
until a real report consumer needs it. Also records the concrete
output of each phase as it landed.
* fix(experiments): zero billable cost for provider-funded experiment traffic
Review fix #1. `skipBalanceCheck` only skipped the preflight
zero-balance rejection; it did not mark the usage as free, so
`processTokenData` would still record full upstream cost into
`microdollar_usage.cost` and the insert CTE would increment
`kilocode_users.microdollars_used`. A zero-balance user could be
charged real upstream cost on a preview experiment and pushed
negative.
Adds an explicit `provider_funded` flag to
`MicrodollarUsageContext` and treats it the same as `user_byok`
in the cost-zeroing branch (zeroes `cost_mUsd` and
`cacheDiscount_mUsd` while preserving `market_cost` for
reporting). The route handler sets the flag for experimented
requests, alongside the existing `skipBalanceCheck`.
* docs(plan): record stale membership cache as next-step follow-up
Captures the review-#4 finding from PR #3325: no TTL on the
EXPERIMENTED_PUBLIC_IDS_REDIS_KEY plus best-effort/swallowed
invalidation can leave a stale list indefinitely. Documents the
agreed fix (add a TTL so the cache self-heals via the existing
DB fallback path) and defers it to a follow-up PR.
* fix(experiments): refuse experiment routing when org disables data collection
Review fix #2. Direct experiment upstreams bypass the
gateway-pinned `data_collection` setting — that setting only
applies when the request is forwarded through OpenRouter or Vercel
with a pinned provider config. For experimented public ids the
upstream is selected by the variant version, so the org's
preference would be silently ignored while we still capture
prompts to R2 for partner evaluation.
When an org sets `data_collection: 'deny'` and the request
resolves to an experiment, return the existing
`dataCollectionRequiredResponse()` instead of routing. The
existing model deny list still runs first via
`checkOrganizationModelRestrictions`. Anonymous users and
non-org authenticated users have no settings and continue to
route normally.
* fix(experiments): cache encrypted upstream; decrypt only on pick
Review fix #3. The Redis per-public-id experiment cache previously
held decrypted partner-issued api keys for up to 10 minutes. That
defeated the BYOK_ENCRYPTION_KEY fail-closed behavior: removing or
rotating the env var did nothing for warm caches because
`getRoutingExperimentForPublicId` returned the cached payload
straight to the picker. The PR verification claim ('disabling
BYOK_ENCRYPTION_KEY returns 503') only held on cold caches.
Refactors the cache to hold CIPHERTEXT exclusively. Decryption +
the missing-key check now run inside
`pickModelExperimentVariant`, on the chosen variant only, on
every request. Three knock-on benefits:
1. Key rotation is effective immediately on the next request;
warm caches no longer serve stale plaintext keys.
2. We decrypt one variant per request instead of N at cache-build
time.
3. A Redis dump no longer leaks usable api keys.
Per-request cost is one AES-256-GCM decrypt (microseconds), which
is negligible against an LLM round-trip.
* fix(experiments): cap prompt capture by UTF-8 bytes, not UTF-16 units
Review fix #6. SYSTEM_PROMPT_CAP_BYTES / REQUEST_BODY_CAP_BYTES
were checked with `string.length` and trimmed with
`string.slice`, which both operate on UTF-16 code units. For
non-ASCII content the actual UTF-8 byte payload uploaded to R2
could be up to ~3× the intended cap (a single CJK char is 1 UTF-16
unit but 3 UTF-8 bytes; an emoji is 2 UTF-16 units but 4 UTF-8
bytes), so the cap was not bounding what the plan claimed it
bounded. Slicing UTF-16 code units could also split a surrogate
pair, leading the next UTF-8 encode to emit U+FFFD and breaking
the deterministic-prefix promise (truncated content would hash
differently across runtimes).
Adds `truncateToUtf8Bytes(s, maxBytes)` which encodes once, walks
back to a UTF-8 codepoint boundary using the 0b10xxxxxx
continuation-byte rule, and returns valid UTF-8 with bytes
<= maxBytes. Replaces the three call sites in
`buildExperimentPromptCapture` to measure with
`Buffer.byteLength` and truncate via the new helper.
* test(experiments): cover picker, fail-closed paths, and UTF-8 truncation
Adds two test files covering the new experiment routing surface:
- `persist.test.ts` (14 cases) — `truncateToUtf8Bytes` boundary
cases (multi-byte CJK, surrogate-pair emoji, exact alignment,
zero cap, determinism); `buildExperimentPromptCapture`
extraction (leading vs mid-conversation system message,
serialization of structured content, UTF-8-byte cap enforcement
for both system and body sides).
- `pick-variant.test.ts` (13 cases) — `isPublicIdExperimented`
positive/negative + paused-also-counted; picker determinism for
the same userId; api-key decryption returns the right partner
secret per variant; allocation-subject precedence
user > machine > ip; `unavailable` when no allocation subject;
`not-found` for paused so traffic doesn't silently fall
through; hot-swap preserves the variant_id slot while serving
the new variant_version_id; weighted-distribution sanity (1:3
split lands ~25%); historical attribution via the old version
row survives a hot-swap; completed experiments stop routing.
Full suite: 373 passed (up from 371).
* fix(experiments): honor request-level data-collection opt-out
Follow-up to review fix #2. The previous commit only refused
experiment routing on the org-level setting
(`settings.data_collection === 'deny'`). A caller can also opt a
single request out of training/data-retention via OpenRouter's
request-body fields `provider.data_collection: 'deny'` or
`provider.zdr: true`. Direct experiment upstreams ignore those
fields entirely (we don't go through OpenRouter), but the
prompt-capture path was still uploading to R2 for partner
evaluation, violating the caller's stated intent.
Refuses experiment routing with `dataCollectionRequiredResponse()`
when `!isFreePromptTrainingAllowed(body.provider)` regardless of
org/anon/BYOK status. The check runs unconditionally right after
`getProvider` returns so it covers anonymous and non-org
authenticated users too.
Adds a unit test for `isFreePromptTrainingAllowed` covering all
opt-out shapes (`data_collection: 'deny'`, `zdr: true`,
`zdr: false`, both, neither, undefined) so a regression cannot
silently allow capture for users who explicitly opted out at the
request scope. Also tightens the `GatewayRequest` cast in
`persist.test.ts` per the test-file as-cast convention.
* chore(experiments): drop plan references from code comments
Plans are ephemeral and get removed when the work is done. Comments
that pointed readers at `.plans/experimental-models-1.md` or
referenced specific plan sections now stand on their own — the
relevant context (sentinel meanings, why the api key column is
separate, scope of preview routing) is inlined where it matters
instead of forwarded to a doc that won't exist long-term.
Touched:
- apps/web/src/lib/r2/experiment-prompts.ts (header)
- apps/web/src/lib/ai-gateway/experiments/persist.ts (sentinel docs)
- apps/web/src/lib/ai-gateway/experiments/upstream-schema.ts (api_key)
- apps/web/src/app/admin/model-experiments/ModelExperimentsContent.tsx
(admin UI subtitle: dropped trailing 'See .plans/...' sentence)
- packages/db/src/schema.ts (three model_experiment* table headers)
* chore(experiments): leave shared-with-main files for separate chore PR
The previous commit (b6b1ebc) cleaned plan references from five
files, three of which already exist on main from earlier merged
PRs (Phase 1 schema, Phase 5 admin UI). Touching cross-package
files unrelated to the gateway work blew up this PR's diff and
risked merge churn against PR #3329, which carries those same
three edits independently.
Restore the on-main copies of:
- packages/db/src/schema.ts
- apps/web/src/lib/ai-gateway/experiments/upstream-schema.ts
- apps/web/src/app/admin/model-experiments/ModelExperimentsContent.tsx
The branch-only files (persist.ts, experiment-prompts.ts) keep
their cleanups since they were authored on this branch and
land/ship together with the gateway work.
* revert(gateway): drop kilo-auto/free experimented-id safeguard
Reverts the `filterOutExperimentedPublicIds` filter added in
`72dd51171` for `kilo-auto/free` candidate selection.
The safeguard added a per-candidate `isPublicIdExperimented`
Redis check on every `kilo-auto/free` request — material extra
latency on a hot path — to defend against an invariant that holds
by construction: `autoFreeModels` and the frontier preset list
are hand-curated and never include dedicated preview experiment
ids. The explicit-opt-in property of preview models is preserved
without any runtime guard.
`getProvider`'s experiment-membership check is unchanged —
explicit requests to a preview public id still resolve through the
experiment routing path.
Plan updated to reflect the no-guard decision and rationale.
* chore(experiments): drop unused barrel index.ts
Every consumer in the repo deep-imports the specific file
(`./pick-variant`, `./persist`, `./upstream-schema`,
`./build-direct-provider`); nothing imported from the barrel.
Most `index.ts` files under `ai-gateway/` are real module
implementations, not pure re-exports — drop this one to match.
* refactor(experiments): drop system-prompt split, plumb request_kind through capture
Adapts the gateway path to the simpler model_experiment_request schema
landed in #3358. Addresses comments #2 + #3 on #3325.
- ExperimentPromptCapture: remove `systemPromptContent`, add
`requestKind`. The full post-`transformRequest` body is now stored as
a single content-addressed blob; the upstream API shape is recorded
alongside it for downstream reporting.
- buildExperimentPromptCapture: collapse the chat_completions-only
system-extraction branch; one body capture path for all kinds.
- persistExperimentAttribution: one R2 PUT instead of two, one hash
written, no `__absent__` sentinel. `PROMPT_HASH_ABSENT` removed;
`PROMPT_HASH_FAILED` and `PROMPT_HASH_DELETED` retained.
- Tests: drop the system-extraction cases (no longer applicable), add
coverage for the three request kinds and a determinism check (content
addressing relies on byte-stable serialization).
The dedup-on-shared-system-prompt micro-optimization had no measured
benefit. If we ever want it back, content-addressing the full body is
the better starting point anyway.
* refactor(gateway): make balance <= 0 the primary condition on the usage-limit guard
Reorders the conjuncts on the usage-limit guard so the salient check
reads first and `skipBalanceCheck` is clearly an override flag, not
the gating condition. Pure reorder — boolean evaluation is unchanged.
Addresses comment #1 on #3325.
* fix(experiments): backfill the experimented-public-ids redis cache on miss
Previously a Redis miss / corrupt cache / Redis error in
`isPublicIdExperimented` fell through to Postgres for a single-id
lookup and returned the answer without populating Redis. Result: every
request on a cold cache hammered Postgres until something else
happened to write the key (admin mutation, etc.).
On the cache-miss path we now rebuild the full routing-eligible public
id set from `model_experiment` (matching what
`recomputeExperimentedPublicIds` writes from the admin router) and
SET it back to Redis with the same TTL. Subsequent requests hit Redis
again. The backfill is fire-and-forget — if Redis is genuinely down
we still return the correct answer for this request.
Also drops the per-id-only DB query in favor of the full-set query
because the cache contract is the complete list, not per-id presence.
Addresses comment #4 on #3325.
* refactor(experiments): rename subjectPick → allocationSubject, fix lying 'unreachable' comment
Two small clarity fixes from #3325 review:
- Rename the local variable bound from `pickAllocationSubject(input)`
from `subjectPick` to `allocationSubject`. Matches the field name
on the result and answers the reviewer's 'is this the user?' question
inline. Pure rename.
- Replace the 'Unreachable: getRandomNumber returns [0, totalWeight).
Fail closed.' comment with an honest one. The branch is defensive
fail-closed insurance for the case where `getRandomNumber`'s
contract breaks (NaN, negative, weight rounding), not actually
unreachable code. The captureMessage + 'unavailable' return is the
right behavior; only the comment was wrong.
Addresses comments #5 and #6 on #3325.
* fix(gateway): return usage identity for zero-cost inserts
* fix(gateway): use JS-side identity values to avoid Postgres timestamp format
* refactor(gateway): extract experiment variant types
* refactor(gateway): narrow experiment attribution catch
* refactor(gateway): move prompt storage fallback
* refactor(gateway): simplify prompt hash fallback
* refactor(gateway): inline experiment attribution insert
* refactor(gateway): remove prompt truncation breadcrumb
* refactor(gateway): inline prompt capture serialization
* refactor(gateway): simplify experiment attribution guard
* fix(web): require BYOK encryption key
* refactor(gateway): remove missing BYOK key branch
* refactor(gateway): cache experimented ids from Redis only
* refactor(gateway): load experiment routing from Postgres
* test(gateway): avoid silent Redis membership skips
* feat(gateway): expose active experiment models
* feat(admin): show experiment request log
* fix(gateway): allow anonymous experiment models
* refactor(gateway): treat experiment models as free models
Resolve isPublicIdExperimented once per request and OR the result into the existing free-model gates: rate limit, anonymous-access, and balance check. Drop the now-redundant skipBalanceCheck flag from GetProviderProviderResult.
* chore(gateway): remove unrelated review changes
* refactor(gateway): fold experiment check into isFreeModel
Now that isFreeModel is async (PR #3393), let it answer for experimented public ids directly. Drops the route's local isExperimentedModel boolean from the auth and balance gates and removes the provider_funded flag — cost zeroing is handled by the isFreeModel check in processUsage.ts.
* chore(gateway): revert unnecessary comment edits
* fix(gateway): refuse experiments under enterprise allow-list
Enterprise provider allow-list is enforced via OpenRouter's body.provider.only field, which doesn't reach a direct partner upstream. Refuse experiment routing rather than silently bypassing the org's allow-list. Note Option 3 (unify direct-upstream routing abstraction) in the plan as the proper followup.
* fix(gateway): remove duplicate unfiltered experiment models
Experiment models were already added to restrictionCandidates and folded into the org-restrictions filter, then appended again unfiltered at the end. The duplicate append created two issues: catalog entries appeared twice, and Enterprise model_deny_list entries were silently re-exposed. Drop the duplicate append; the filter at line 32-42 covers experiments and matches the gateway's behavior (experiments do not set bypassAccessCheck and go through checkOrganizationModelRestrictions on the request path).
* fix(gateway): split membership and isFreeModel out of models.ts
models.ts is reachable from client bundles via constants like PRIMARY_DEFAULT_MODEL and preferredModels. Folding the experiment-membership check into isFreeModel inside models.ts dragged in the Redis client (and via pick-variant.ts the Postgres client), breaking the production build with 'Module not found' errors for net/tls/dns/promises and pg.
Split along two seams that already existed: pick-variant.ts now contains only the drizzle-using routing (getRoutingExperimentForPublicId, pickModelExperimentVariant), while a new membership.ts holds the Redis-only isPublicIdExperimented. The async isFreeModel itself moves to a new is-free-model.ts that imports the sync helpers from models.ts and the membership check from membership.ts. Net result: models.ts has no I/O dependencies, the fold is preserved for server callers, and the experiment-membership check still lives once in the codebase.
* docs(experiments): note response model-id leak as v1 followup
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
/admin/app-builder/[id])sessionIdURL parameter in Session Traces page for shareable links