Skip to content

feat(gateway): expose active experiment models - #3386

Merged
markijbema merged 2 commits into
mainfrom
mark/expose-active-experiment-models
May 21, 2026
Merged

markijbema merged 2 commits into
mainfrom
mark/expose-active-experiment-models

Conversation

@markijbema

Copy link
Copy Markdown
Contributor

Summary

  • Exposes active model experiments as selectable OpenRouter-style models in the model list endpoints.
  • Makes active preview IDs available to every caller, independent of organization model restrictions, so local/preview experiment IDs can be selected by clients.
  • Adds focused coverage that active experiments are listed while draft and paused experiments stay hidden.

Verification

  • Activated a local kilo/preview-henk experiment and confirmed it appeared in the local model selector.

Visual Changes

N/A

Reviewer Notes


filteredModels.push(...(await getDirectByokModelsForOrganization(organizationId)));
filteredModels.push(...(await listAvailableCustomLlms(organizationId)));
filteredModels.push(...(await listAvailableExperimentModels()));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: Experiment models bypass enterprise model restrictions

Experiment models are appended to filteredModels after the hasActiveModelRestrictions block (lines 31–39), meaning they are never evaluated against modelDenyList or providerAllowList. This is consistent with the PR description ("independent of organization model restrictions"), but it means enterprise customers with a strict provider allow-list or model deny-list will still see and can select active experiment models.

If this is intentional, a comment here would make the policy explicit for future maintainers — e.g.:

// Experiment models are intentionally exempt from org-level restrictions so
// preview experiments are always reachable during the experiment period.

If it is not intentional, the isAllowed predicate should be applied to experiment models the same way it is applied to the main responseData.data.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll look into it, for now i dont think it matters much either way , model usage is enforced when using

@kilo-code-bot

kilo-code-bot Bot commented May 21, 2026 •

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Executive Summary

The previously flagged WARNING — experiment models bypassing enterprise modelDenyList/providerAllowList restrictions — has been resolved. Experiment models are now included in restrictionCandidates before policy filtering in getAvailableModelsForOrganization, and a new integration test verifies that a denied experiment model is correctly excluded.

Resolved Issues
File Line Issue Status
apps/web/src/lib/organizations/organization-models.ts 44 (old) Experiment models bypassed enterprise model restrictions ✅ Fixed
Files Reviewed (5 files)
  • apps/web/src/app/api/openrouter/models/route.ts
  • apps/web/src/lib/ai-gateway/experiments/list-available-experiment-models.ts
  • apps/web/src/lib/ai-gateway/experiments/list-available-experiment-models.test.ts
  • apps/web/src/lib/organizations/organization-models.ts
  • apps/web/src/routers/organizations/organization-settings-router.test.ts

Reviewed by claude-sonnet-4.6 · 241,536 tokens

Review guidance: REVIEW.md from base branch main

@markijbema
markijbema merged commit f18adff into main May 21, 2026
17 checks passed
@markijbema
markijbema deleted the mark/expose-active-experiment-models branch May 21, 2026 11:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants