Repository navigation
Support multiple ordered sandbox pools per installation #397
Copy link
Copy link
Open
Description
Problem
An installation can select exactly one Sandbox Provider (docker, microsandbox or e2b). Administrators cannot combine compute sources, for example Docker nodes plus microsandbox nodes, or self-hosted nodes that overflow to E2B. Switching providers requires a full sandbox reset that clears all hosted resources.
Multiple nodes of the same provider already work: DecidePlacement picks an online, ready node with free capacity, preferring the highest ready generation and then the fewest active sandboxes.
Where the single-provider assumption lives
- Contract:
contracts/agents-api/sandbox-deployment.mdsays PostgreSQL holds "one active selection per installation",provideris "exactly one ofdocker,microsandbox,e2b", and changing to a different backend requires a reset first. - Schema:
runtime_deploymentis a singleton table (singleton boolean PRIMARY KEY CHECK (singleton)) holding oneprovider_kind,mode,generationand specification.runtime_deployment_generationskeeps retained generations only so that existing allocations can still be cleaned up. - Placement:
services/core/internal/deployment/placement/placement.goreturns no placement indirectmode and otherwise chooses among the nodes of the single provider. - Node configuration:
providers.Loadrejects configuring bothdockerandmicrosandbox("only one sandbox provider can be configured"). - API and Web:
/core/v1/sandbox/deploymentGET/POST/PUT take one selection object, andSandboxSetupWizardasks the administrator to pick one provider.
Proposal
Replace the singleton deployment with an ordered list of sandbox pools. Each pool has its own provider, specification, Runtime release, generation, credential and priority.
- Data model: pools replace the
runtime_deploymentsingleton. Placements and allocations record their pool, including direct (E2B) allocations, which have no placement record today. - Placement: try pools in priority order and use the first one with admissible capacity. Within a node pool, keep the current node selection rules. Direct pools need a declared capacity limit so that lower-priority pools can be reached.
- Lifecycle: generation rollout, reset, owner epoch, suspension policy and node enrollment become per-pool. Removing or replacing a pool clears only that pool's resources.
- API and contract: replace the single deployment object with pool CRUD and ordering under
/core/v1/sandbox/. Update the OpenAPI,sandbox-deployment.mdand itszhtranslation, and the@oac/agents-clientvalidators. - Web: show a pool list with add, reorder, and per-pool nodes and capacity.
Constraints
- Ordering applies only when a new Session is placed, based on capacity and availability. A failed or unknown
Createis never retried in another pool: the Sandbox Provider contract forbids replayingCreate, and Core never substitutes another implementation. - An existing placement, allocation or suspended Environment stays in its original pool and node. Restores never move to another pool.
- No provider-name branches in placement or lifecycle code. Pool behavior comes from the registered adapter declarations.
- Pre-release: replace the singleton outright, with no compatibility layer for the old single-deployment API.
Open questions
- Should a pool's capacity limit be declared by the adapter, set by the administrator, or both?
- Can pools of the same provider kind coexist, for example two E2B pools with different templates?
- How should Session-level selection (for example, by resources or Harness) interact with pool order, if at all?
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels
Activity
LingYi-Liang commentedon Oct 4, 2026
I'd like to pick this up. I'd start with configuring and ordering multiple pools, assigning sessions, and the corresponding Core API and basic management UI. New sessions would use the first pool with capacity; existing sessions would keep their pool for restore and cleanup. I'd leave Project/Template pool selection and #418's image selection for later. Does that scope make sense?
For capacity, could each pool have
max_active/max_retained, including direct mode, and could multiple pools use the same provider? Node limits would still apply. I'd commit the reservation together with session creation to avoid overbooking. If Create times out or its outcome is unknown, I'd keep the original ownership and capacity reservation, without repeating Create or switching pools. I'd follow the final suspension and restore behavior from #297 rather than adding a separate flow.