diff --git a/apps/web/DESIGN.md b/apps/web/DESIGN.md index 522e23168..a1567df99 100644 --- a/apps/web/DESIGN.md +++ b/apps/web/DESIGN.md @@ -474,3 +474,5 @@ Motion reports state and never makes anyone wait: the navigation chip and segmen Use Lucide throughout. Navigation icons are 18px with a 1.75px stroke; shared tool icons use the same stroke at their existing compact sizes. Default navigation icons use secondary ink, selected icons use primary ink. The selected row has a neutral background and a narrow indigo marker. Getting started emphasizes the first confirmed incomplete step with the primary button while leaving every step independently available. Only authoritative, successful zero-Session reads replace the activity chart with first-use guidance. Missing and failed reads retain their existing uncertainty and recovery controls. + +Configuration controls show concise labels, values and actions. Put necessary explanatory policy text behind a `HelpTip` question mark; do not add persistent small-print helper text or note paragraphs. Validation and request failures remain visible beside the affected action or field. diff --git a/apps/web/PRODUCT.md b/apps/web/PRODUCT.md index 9fadea9a0..535b42427 100644 --- a/apps/web/PRODUCT.md +++ b/apps/web/PRODUCT.md @@ -38,14 +38,14 @@ The console runs beside the administrator's own Core, with execution, files and - **Online sandbox configuration** changes the same backend's resources, Runtime or E2B template without retiring existing nodes or changing existing Sessions' resource ownership. New placement follows Core's qualified capacity; saving a target does not promise immediate placement on it. Configuration rollout shows Core's target preparation and retained previous-generation sandbox count. A settled rollout can still have failed, update-required or unknown nodes and old resources. An offline node stays offline even when it has a recorded serving generation. Node and allocation detail distinguish the serving pin, target preparation and each resource's configuration generation. - **E2B credential replacement** uses the same configuration form. Setup requires a key; leaving it blank during an update keeps the saved key. An explicit key, even the same value, is verified as a replacement and advances the target generation after successful verification. Another backend or E2B team requires a deliberate reset. A rejected or uncertain replacement never clears the committed configuration or replays the write. - **E2B deployments** have no machines: Nodes offers a link to System's sandbox configuration. Overview and Sandbox metrics show the sandboxes Core holds in E2B's cloud (running, starting, size, template build) instead of node capacity, with no node column or Add node action; a sandbox's dialog adds its disk use. -- **Automatic suspension** appears when the deployment declares a suspension policy. Nodes and Sandbox metrics show Core's active/retained placement counts and limits across online nodes, suspended placements across all registered nodes, and the read-only idle and snapshot-retention policy. These are Core records, not measured CPU or memory savings. Node allocations show the recorded compute phase and elapsed phase time, never an inferred snapshot-expiry countdown. Sandbox metrics retains Runtime observation status separately from the matching allocation's recorded phase. Independent workspace persistence is explained only when the deployment declares it. Nodes refreshes its inventory every 30 seconds while visible; reset and active preparation retain their faster polling. +- **Automatic suspension** appears when the deployment declares a suspension policy. Nodes and Sandbox metrics show Core's active/retained placement counts and limits across online nodes, suspended placements across all registered nodes, and the current idle and snapshot-retention policy. Nodes owns one **Capacity and suspension** area: deployment-wide idle seconds are editable there alongside per-node active and retained limits; snapshot retention remains read-only. Necessary policy explanations belong in help tips, not persistent helper paragraphs. These are Core records, not measured CPU or memory savings. Node allocations show the recorded compute phase and elapsed phase time, never an inferred snapshot-expiry countdown. Sandbox metrics retains Runtime observation status separately from the matching allocation's recorded phase. Independent workspace persistence is explained only when the deployment declares it. Nodes refreshes its inventory every 30 seconds while visible; reset and active preparation retain their faster polling. - **Getting started**: signing in opens the console on the Overview; nothing is forced first. While a step is to do, a Getting started checklist on the Overview shows four steps, in any order, each with its state and one action: sandboxes ready (a saved deployment and a node online and ready, or a saved E2B deployment whose template build is not reported as not ready), a default model provider on the default harness (on any enabled harness when none is default), a project with an active key, and a first Session, whose action opens the call samples of the newest active project, preferring one with an active key. Completion comes from reads the console already makes. It can be hidden; Show Getting started in the sidebar opens it again, and it ends with a brief "You're set". While it is open, Add node ends with the next step once its node is ready: the default model provider while that is to do, otherwise back to the checklist. The optional three-chapter tour of the console (Monitor, Resources, Platform) opens from it, on the sign-in stage. - Terminology: API terms stay in English in the Chinese UI (Agent, Session, Turn, Skill, Vault, Credential, API key). The sign-in credential is the Core key ("Core Key"); keys issued in a project for applications are project API keys ("项目 API Key"). Provider readiness is "Provider not ready / 提供方未就绪"; revoked credentials and keys use "Revoked / 已撤销". A default model provider is a service address and write-only key; the application's Agent `model` chooses the provider-supported model name. A sandbox is Core-managed compute; Runtime names Core's execution observations, and Environment is the Session's API execution environment. These are distinct counts and resources, not synonyms. ## Capabilities and Constraints - **Projects and keys.** A project owns an isolated set of assets shared by all of its named API keys; projects do not see each other's assets. Issuing or revoking a key never touches assets. Archiving a project revokes every key and keeps its assets viewable and deletable; it can't be undone, so its confirmation says so, counts the active keys it revokes and, when there are any, asks for the project's name. Key plaintext is shown once, at issuance, and never stored by the console. Beside it, and without the key on an active project's page, the console tells developers to set `OPENAI_BASE_URL` (the installation's API base URL) and `OPENAI_API_KEY` (a key of the project), with curl and Python samples that list Agents and create a Session. -- **One home for each setting.** System owns sandbox configuration through its Sandbox configuration secondary page. This is the only place to set up, update, reset or inspect deployment rollout. Nodes owns the node list and individual node operations. Overview and metrics link to these owners instead of repeating their configuration or rollout panels. Resource editing uses a dialog; rollout counts and generations appear in its details dialog. +- **One home for each setting.** System owns sandbox configuration through its Sandbox configuration secondary page. This is the only place to set up, update, reset or inspect deployment rollout. Nodes owns capacity and automatic suspension policy, the node list and individual node operations. Overview and metrics link to these owners instead of repeating their configuration or rollout panels. Resource editing uses a dialog; rollout counts and generations appear in its details dialog. - **Web API only.** Every read and write goes through `/core/v1/**`. The console holds no API key and sends nothing to `/v1`. - **No asset writes except delete.** Assets are created and changed only by a project's keys through the Agents API. The console does not create or edit Agents or Templates, upload Skills or Files, create or replace Credentials, start Sessions, send input or cancel work. Deletion follows the public deletion rules; a busy Session is not deletable and the console never cancels work to make it so. - **Secrets stay write-only.** Credential tokens, Template environment variables and setup commands are never returned, to the administrator included. An Agent's saved model provider shows its protocol, base URL, limits and whether a key is configured, never the key. diff --git a/apps/web/e2e/fixture-console.mjs b/apps/web/e2e/fixture-console.mjs index 69c03a8f5..cf363921a 100644 --- a/apps/web/e2e/fixture-console.mjs +++ b/apps/web/e2e/fixture-console.mjs @@ -372,6 +372,13 @@ async function sandboxRoute(request, response, path, url) { const data = state.admin.runtimeObservations().map((entry) => ({ ...entry, observation: { ...entry.observation, disk: e2b && entry.observation.status === "observed" ? { usage_bytes: 3 * 2 ** 30, limit_bytes: 10 * 2 ** 30 } : null } })); return send(response, 200, { object: "list", data, has_more: false, first_id: data[0]?.observation.id ?? null, last_id: data.at(-1)?.observation.id ?? null }); } + if (path === "/deployment" && request.method === "PATCH") { + const input = await body(request); + if (!state.deployment.suspension || state.deployment.reset) return error(response, 409, "Suspension policy is unavailable.", "sandbox_deployment_conflict"); + if (!Number.isInteger(input.idle_seconds) || input.idle_seconds < 30 || input.idle_seconds > 86400) return error(response, 400, "Invalid idle seconds.", "invalid_request_error"); + state.deployment.suspension.idle_seconds = input.idle_seconds; + return send(response, 200, state.deployment); + } if (path === "/deployment" && (request.method === "POST" || request.method === "PUT")) { const input = await body(request); const initialize = request.method === "POST"; diff --git a/apps/web/e2e/nodes.spec.ts b/apps/web/e2e/nodes.spec.ts index 6c9bb2c4d..71582264d 100644 --- a/apps/web/e2e/nodes.spec.ts +++ b/apps/web/e2e/nodes.spec.ts @@ -352,18 +352,17 @@ test("keeps the page usable when Core refuses a sandbox change, and shows a safe }); test("renames a node and sets how many sandboxes run on it at once", async ({ page, request }) => { - await openConsole(page, request, "nodes?id=node-local"); - // A Docker node shows its limit too, and the edit shows what the host holds. - const capacity = page.getByRole("region", { name: "Capacity" }); + await openConsole(page, request, "nodes"); + // The Nodes policy area owns all per-node capacity edits. + const capacity = page.getByRole("region", { name: "Capacity and suspension" }); await expect(capacity).toContainText("Active / limit"); await expect(capacity).toContainText("5 / 8"); - await page.getByRole("button", { name: "Edit node" }).click(); + await capacity.getByRole("row").filter({ hasText: "core-01" }).getByRole("button", { name: "Edit node" }).click(); const edit = page.getByRole("dialog", { name: "Edit node" }); - await expect(edit.getByText("Host: 16 CPU · 64 GiB. Each sandbox: 2 CPU · 4 GiB. Suggested: at most 8 at once.")).toBeVisible(); await edit.getByLabel("Name").fill("core-01-large"); await edit.getByLabel("Sandboxes at once").fill("6"); await edit.getByRole("button", { name: "Save" }).click(); await expect(edit).toBeHidden(); - await expect(page.getByRole("heading", { name: "core-01-large", level: 1 })).toBeVisible(); + await expect(capacity).toContainText("core-01-large"); await expect(capacity).toContainText("5 / 6"); }); diff --git a/apps/web/e2e/sandbox-lifecycle.spec.ts b/apps/web/e2e/sandbox-lifecycle.spec.ts index db9454ce7..d82164a31 100644 --- a/apps/web/e2e/sandbox-lifecycle.spec.ts +++ b/apps/web/e2e/sandbox-lifecycle.spec.ts @@ -1,5 +1,5 @@ import { expect, test } from "@playwright/test"; -import { openConsole, expectManagementBoundary } from "./console"; +import { openConsole, expectManagementBoundary, setDeployment, writes } from "./console"; test.afterEach(async ({ request }) => expectManagementBoundary(request)); @@ -15,9 +15,9 @@ test("shows automatic suspension policy, retained capacity and elapsed phase tim await route.fulfill({ response, json: { data: data.map((node: Record) => ({ ...node, provider: "microsandbox", retained: Number(node.active) + 2 })) } }); }); await openConsole(page, request, "nodes"); - await expect(page.getByRole("heading", { name: "Automatic suspension" })).toBeVisible(); + await expect(page.getByRole("heading", { name: "Capacity and suspension" })).toBeVisible(); await expect(page.getByText("5 minutes", { exact: true })).toBeVisible(); - await expect(page.getByRole("columnheader", { name: "Retained / limit" })).toBeVisible(); + await expect(page.getByRole("region", { name: "Capacity and suspension" }).getByRole("columnheader", { name: "Retained / limit" })).toBeVisible(); await page.screenshot({ path: testInfo.outputPath("nodes-en.png"), fullPage: true }); await page.getByRole("button", { name: "Open core-01", exact: true }).click(); const allocations = page.getByRole("table", { name: "Sandbox allocations" }); @@ -41,3 +41,27 @@ test("hides suspension policy when the provider declares no suspension support", await expect(page.getByRole("heading", { name: "Automatic suspension" })).toHaveCount(0); await expect(page.getByRole("columnheader", { name: "Retained / limit" })).toHaveCount(0); }); + + +test("edits deployment idle time beside node limits without changing generation", async ({ page, request }) => { + await openConsole(page, request, "nodes"); + const deployment = await setDeployment(request, { suspension: { idle_seconds: 300, retention_seconds: 86400 } }); + await setDeployment(request, { provider: "microsandbox", specification: { ...deployment.specification!, resources: { ...deployment.specification!.resources, root_disk_mib: 8192, environment_disk_mib: 8192 } } }); + await page.getByRole("button", { name: "Refresh sandbox state" }).click(); + const area = page.getByRole("region", { name: "Capacity and suspension" }); + const idle = area.getByLabel("Idle before suspension (seconds)"); + await expect(idle).toHaveValue("300"); + await idle.fill("29"); + await expect(area.getByRole("button", { name: "Save", exact: true })).toBeDisabled(); + await expect(area.getByText("Enter a whole number from 30 to 86,400.")).toBeVisible(); + await idle.fill("60"); + const response = page.waitForResponse((response) => response.request().method() === "PATCH" && response.url().endsWith("/sandbox/deployment")); + await area.getByRole("button", { name: "Save", exact: true }).click(); + const saved = await (await response).json(); + expect(saved.generation).toBe(deployment.generation); + expect(saved.suspension).toEqual({ idle_seconds: 60, retention_seconds: 86400 }); + await expect(idle).toHaveValue("60"); + await page.reload(); + await expect(idle).toHaveValue("60"); + expect((await writes(request)).filter((write) => write === "PATCH /core/v1/sandbox/deployment")).toHaveLength(1); +}); diff --git a/apps/web/src/features/sandbox/NodeEditDialog.tsx b/apps/web/src/features/sandbox/NodeEditDialog.tsx index 8740159f6..074857727 100644 --- a/apps/web/src/features/sandbox/NodeEditDialog.tsx +++ b/apps/web/src/features/sandbox/NodeEditDialog.tsx @@ -94,10 +94,9 @@ export function NodeEditDialog({ client, node, size, suspends, onClose, onSaved {nameError ? {nameError} : null}
- {t("The most sandboxes Core places on this node at the same time.")} - { setActive(event.target.value); setError(null); }} aria-invalid={Boolean(activeError)} aria-errormessage={activeError ? `${id}-active-error` : undefined} aria-describedby={hostFacts ? `${id}-host` : undefined} /> + {t("The most sandboxes Core places on this node at the same time.")}{hostFacts ? <> {hostFacts} : null} + { setActive(event.target.value); setError(null); }} aria-invalid={Boolean(activeError)} aria-errormessage={activeError ? `${id}-active-error` : undefined} /> {activeError ? {activeError} : null} - {hostFacts ? {hostFacts} : null}
{suspends ? (
diff --git a/apps/web/src/features/sandbox/NodeList.tsx b/apps/web/src/features/sandbox/NodeList.tsx index 80bf6dfe1..90ba85e9d 100644 --- a/apps/web/src/features/sandbox/NodeList.tsx +++ b/apps/web/src/features/sandbox/NodeList.tsx @@ -56,7 +56,7 @@ export function OldAddressHint() { return {t("Remove and add again")}; } -export function NodeList({ nodes, allocations, coreUrl, stale, disabled, suspends = false, onOpen, onRemove }: { +export function NodeList({ nodes, allocations, coreUrl, stale, disabled, suspends = false, onOpen, onRemove, onEdit }: { nodes: readonly SandboxNode[]; /** The deployment's address; a node enrolled with another one is on an old address. */ coreUrl: string; @@ -67,6 +67,7 @@ export function NodeList({ nodes, allocations, coreUrl, stale, disabled, suspend disabled: boolean; onOpen: (node: SandboxNode) => void; onRemove: (node: SandboxNode) => void; + onEdit?: (node: SandboxNode) => void; }) { const { t, i18n } = useTranslation("sandbox"); const locale = i18n.resolvedLanguage; @@ -113,6 +114,7 @@ export function NodeList({ nodes, allocations, coreUrl, stale, disabled, suspend {formatDateTime(epochSeconds(node.created_at), locale)} + {onEdit ? : null} diff --git a/apps/web/src/features/sandbox/SandboxLifecycle.test.tsx b/apps/web/src/features/sandbox/SandboxLifecycle.test.tsx index 2e0a726db..680287b85 100644 --- a/apps/web/src/features/sandbox/SandboxLifecycle.test.tsx +++ b/apps/web/src/features/sandbox/SandboxLifecycle.test.tsx @@ -16,6 +16,18 @@ describe("sandbox lifecycle evidence", () => { expect(html).toContain("5 minutes"); expect(html).not.toContain("3 / 14"); }); + it("offers policy editing only when the owner supplies controls", () => { + const nodes = [node("online", { active: 2, max_active: 4, retained: 5, max_retained: 8 })]; + const html = renderToStaticMarkup( true }} />); + expect(html).toContain("Capacity and suspension"); + expect(html).toContain('id="sandbox-idle-seconds"'); + expect(html).toContain('value="300"'); + const readonly = renderToStaticMarkup(); + expect(readonly).not.toContain("sandbox-idle-seconds"); + expect(readonly).not.toContain("Edit node"); + const unsupported = renderToStaticMarkup( true }} />); + expect(unsupported).not.toContain("sandbox-idle-seconds"); + }); it("does not replace missing inventory with zero or claim fresh evidence after failure", () => { const html = renderToStaticMarkup(); expect(html).toContain("current state is unconfirmed"); diff --git a/apps/web/src/features/sandbox/SandboxLifecycle.tsx b/apps/web/src/features/sandbox/SandboxLifecycle.tsx index 8533eff62..8fa98bd9b 100644 --- a/apps/web/src/features/sandbox/SandboxLifecycle.tsx +++ b/apps/web/src/features/sandbox/SandboxLifecycle.tsx @@ -1,24 +1,41 @@ +import { useState, type ReactNode } from "react"; import type { SandboxDeployment, SandboxNode } from "@oac/agents-client"; import { useTranslation } from "react-i18next"; -import { Kpi, KpiStrip, Section } from "../../components/console-ui"; +import { HelpTip, Kpi, KpiStrip, Section } from "../../components/console-ui"; import { formatInteger, formatSpan, MISSING } from "../../lib/format"; import { capacitySummary } from "../fleet/fleet-model"; /** Core accounting and the declared policy, separate from host CPU and memory observations. */ -export function SandboxLifecycle({ deployment, nodes, stale }: { deployment: SandboxDeployment; nodes: readonly SandboxNode[] | null; stale: boolean }) { +export function SandboxLifecycle({ deployment, nodes, stale, controls, children }: { deployment: SandboxDeployment; nodes: readonly SandboxNode[] | null; stale: boolean; controls?: { disabled: boolean; onSave: (seconds: number) => Promise; }; children?: ReactNode }) { const { t, i18n } = useTranslation("sandbox"); const policy = deployment.suspension; - if (!policy) return null; + const [draft, setDraft] = useState(null); + const idle = draft ?? String(policy?.idle_seconds ?? 300); + const valid = /^\d+$/.test(idle) && Number(idle) >= 30 && Number(idle) <= 86400; + if (!policy && !controls) return null; const count = (value: number) => formatInteger(value, i18n.resolvedLanguage); const capacity = nodes ? capacitySummary(nodes) : null; - return
+ return
{stale ?

{t("Showing the last loaded lifecycle counts; current state is unconfirmed.")}

: null} - + {policy ? <> {t("Current policy for new suspensions, not a countdown for existing snapshots. The next Turn requests a resume; capacity and a compatible node must be available.")}{deployment.specification?.workspace ? <> {t("Snapshot expiry releases memory state. With independent workspace storage, files and conversation remain available for a cold continuation; processes restart.")} : null}} /> + : null} + {controls ? <> + {policy ?
{ event.preventDefault(); if (valid && !controls.disabled && await controls.onSave(Number(idle))) setDraft(null); }}> +
+ {t("Deployment-wide: 30–86,400 seconds. Applies to existing and new running sandboxes without restarting them. Suspension already in progress and snapshot retention are unchanged. Capacity pressure may suspend an idle sandbox after 15 seconds.")} + setDraft(event.target.value)} aria-invalid={!valid} aria-describedby={!valid ? "sandbox-idle-error" : undefined} /> + {!valid ? {t("Enter a whole number from 30 to 86,400.")} : null} +
+
+
: null} + + : null} + {children}
; } diff --git a/apps/web/src/features/sandbox/SandboxManagerView.tsx b/apps/web/src/features/sandbox/SandboxManagerView.tsx index e998e231d..33710ef71 100644 --- a/apps/web/src/features/sandbox/SandboxManagerView.tsx +++ b/apps/web/src/features/sandbox/SandboxManagerView.tsx @@ -1,7 +1,7 @@ import { useCallback, useEffect, useRef, useState, type ReactNode, type RefObject } from "react"; import { type SandboxNode } from "@oac/agents-client"; import { useQuery, useQueryClient } from "@tanstack/react-query"; -import { ArrowLeft, Pencil, Plus, Server, Trash2 } from "lucide-react"; +import { ArrowLeft, Plus, Server, Trash2 } from "lucide-react"; import { useTranslation } from "react-i18next"; import { ConfirmDialog } from "../../components/ConfirmDialog"; import { EmptyState, HelpTip, RefreshButton } from "../../components/console-ui"; @@ -10,7 +10,7 @@ import { useFailureToast, useToast } from "../../components/Toast"; import { useConsoleIntent, useConsoleNavigation } from "../../lib/console-navigation"; import { InstallationNotice } from "../../components/InstallationNotice"; import { installationQuery } from "../../lib/installation"; -import { sandboxRequestError } from "../../lib/sandbox-labels"; +import { sandboxRequestError, sandboxWriteUncertain } from "../../lib/sandbox-labels"; import type { SandboxConsoleConfig } from "./console-config"; import { sandboxAdmin, sandboxConsoleConfigQuery } from "./sandbox-queries"; import { useSandboxPageState } from "./use-sandbox-page-state"; @@ -20,6 +20,7 @@ import { SandboxLifecycle } from "./SandboxLifecycle"; import { NodeDetail } from "./NodeDetail"; import { NodeEditDialog } from "./NodeEditDialog"; import { NodeCleanupDialog, type NodeCleanup } from "./NodeCleanupDialog"; +import { writeSandboxDeployment } from "./sandbox-deployment-write"; import { sandboxSize } from "./deployment-specification"; import "./SandboxManagerView.css"; @@ -174,6 +175,34 @@ function SandboxManager({ consoleConfig }: { consoleConfig: SandboxConsoleConfig // Rendered first in both the list and a node's page, so an open command outlives the navigation. const enrollment = hostedNodes && snapshot ? setAdding(false)} onRefresh={refreshNodes} /> : null; + const editDialog = setEditTarget(null)} + onSaved={() => { + const saved = editTarget; + setEditTarget(null); + toast.show(t("Node saved"), { tone: "success" }); + refresh(); + // Other pages read the fleet and the node's detail separately. + void queryClient.invalidateQueries({ queryKey: ["sandbox-fleet"] }); + if (saved) void queryClient.invalidateQueries({ queryKey: ["sandbox-node", saved.id] }); + }} + />; + + async function saveIdle(seconds: number) { + if (busy || loading || !fresh || setupNeedsRefresh) return false; + try { + return await writeSandboxDeployment(queryClient, (signal) => client.updateSuspension({ idle_seconds: seconds }, { signal })) !== null; + } catch (error) { + toast.show(t(sandboxWriteUncertain(error) ? "Couldn't confirm the sandbox change" : "Core rejected the sandbox change"), { tone: "error", detail: sandboxRequestError(error, locale), key: "sandbox-write" }); + return false; + } + } + if (params.id && hostedNodes) { const back = () => goBack("nodes"); return <> @@ -184,7 +213,6 @@ function SandboxManager({ consoleConfig }: { consoleConfig: SandboxConsoleConfig title={selected ? selected.name || selected.id : params.id} actions={<> {refreshButton} - {selected ? : null} {selected ? : null} } /> @@ -196,23 +224,7 @@ function SandboxManager({ consoleConfig }: { consoleConfig: SandboxConsoleConfig
{dialog} {cleanupDialog} - setEditTarget(null)} - onSaved={() => { - const saved = editTarget; - setEditTarget(null); - toast.show(t("Node saved"), { tone: "success" }); - refresh(); - // Other pages read the fleet and the node's detail separately. - void queryClient.invalidateQueries({ queryKey: ["sandbox-fleet"] }); - if (saved) void queryClient.invalidateQueries({ queryKey: ["sandbox-node", saved.id] }); - }} - /> + ; } @@ -233,15 +245,15 @@ function SandboxManager({ consoleConfig }: { consoleConfig: SandboxConsoleConfig {staleNodes.length ?

{staleNodes.length === 1 ? t("{{name}} is still bound to an old Core address. Remove it and add it again.", { name: staleNodes[0] }) : t("{{count}} nodes are still bound to an old Core address: {{names}}. Remove them and add them again.", { count: staleNodes.length, names: new Intl.ListFormat(i18n.resolvedLanguage, { type: "conjunction" }).format(staleNodes) })}

: null} - {hostedNodes && suspends ? 0) ? nodes : null} stale={!nodesConfirmed} /> : null} - {hostedNodes ?
+ {hostedNodes ? 0) ? nodes : null} stale={!nodesConfirmed} controls={{ disabled: busy || loading || !fresh || !nodesConfirmed || setupNeedsRefresh || Boolean(snapshot.deployment.reset), onSave: saveIdle }}> {nodes.length - ? navigate("nodes", { id: node.id })} onRemove={askRemove} /> + ? navigate("nodes", { id: node.id })} onEdit={setEditTarget} onRemove={askRemove} /> : nodesConfirmed ? : null} -
: null} +
: null} : null} {dialog} {cleanupDialog} + {editDialog} ; } diff --git a/apps/web/src/i18n/locales/zh-CN/sandbox.ts b/apps/web/src/i18n/locales/zh-CN/sandbox.ts index f3d07d835..3c21b951b 100644 --- a/apps/web/src/i18n/locales/zh-CN/sandbox.ts +++ b/apps/web/src/i18n/locales/zh-CN/sandbox.ts @@ -1,5 +1,9 @@ import { firstRun } from "./first-run"; export const sandbox = { + "Capacity and suspension": "容量与挂起", + "Idle before suspension (seconds)": "空闲多久后挂起(秒)", + "Deployment-wide: 30–86,400 seconds. Applies to existing and new running sandboxes without restarting them. Suspension already in progress and snapshot retention are unchanged. Capacity pressure may suspend an idle sandbox after 15 seconds.": "整个部署统一生效:30–86,400 秒。适用于现有和新建的运行中沙箱,无需重启。已经开始的挂起和快照保留期不变。容量紧张时,沙箱空闲 15 秒后可能提前挂起。", + "Enter a whole number from 30 to 86,400.": "请输入 30 到 86,400 之间的整数。", "Suspended placements across all registered nodes, including offline nodes. They do not consume an active slot.": "所有注册节点(含离线节点)上的已挂起分配,不占活跃名额。", "Automatic suspension": "自动挂起", "Active and retained limits cover online nodes. Suspended counts include all registered nodes. These are Core placement records, not CPU or memory measurements.": "活跃与保留名额统计在线节点;已挂起数量包含所有注册节点。这些是 Core 分配记录,并非 CPU 或内存实测值。", diff --git a/contracts/agents-api/admin-api.md b/contracts/agents-api/admin-api.md index 44fce17ea..157894f18 100644 --- a/contracts/agents-api/admin-api.md +++ b/contracts/agents-api/admin-api.md @@ -34,7 +34,7 @@ Paths are relative to `/core/v1`. | `projects/{project_id}/environments/{environment_id}/executor-credentials[/{key_id}]` | Executor credentials of a `self_hosted` Environment | [Executor credentials](./environment-executor-credentials.md#core-key-routes) | | `projects/{project_id}/resource-owners`, `projects/{project_id}/write-operations` | Which API key created a resource and each key's writes | [Write provenance](#write-provenance) | | `harnesses`, `harnesses/{harness}/model-configuration` | Enabled Harnesses and each Harness's deployment default model | [Deployment defaults](./model-execution.md#deployment-defaults) | -| `sandbox/deployment`, `sandbox/deployment/reset`, `sandbox/providers/{provider}/discovery` | The sandbox provider, resources and Runtime, reset, and provider configuration discovery such as E2B templates | [Sandbox deployment](./sandbox-deployment.md#routes) | +| `sandbox/deployment`, `sandbox/deployment/reset`, `sandbox/providers/{provider}/discovery` | The sandbox provider, resources and Runtime, idle suspension policy, reset, and provider configuration discovery such as E2B templates | [Sandbox deployment](./sandbox-deployment.md#routes) | | `sandbox/enrollment-tokens`, `sandbox/nodes[/{node_id}[/allocations]]` | Node enrollment tokens, nodes and their allocations and host history | [Nodes guide](../../docs/getting-started/nodes.md), [sandbox deployment](./sandbox-deployment.md), [node host history](./runtime-observability-api.md#node-host-observations-and-history) | | `summary` | Session counts and usage by Project, Agent or key | [Summary](#summary) | | `metrics` | Core's own process, execution, database and job metrics | [Core metrics](./core-metrics.md) | diff --git a/contracts/agents-api/core.openapi.yaml b/contracts/agents-api/core.openapi.yaml index 4d30a2ddc..6ebc87e18 100644 --- a/contracts/agents-api/core.openapi.yaml +++ b/contracts/agents-api/core.openapi.yaml @@ -709,6 +709,15 @@ definitions: description: Docker never suspends, so Core replaces this with max_active; microsandbox uses both limits. type: integer type: object + api.SandboxIdlePolicyInput: + properties: + idle_seconds: + maximum: 86400 + minimum: 30 + type: integer + required: + - idle_seconds + type: object api.SandboxMutationResponse: properties: deleted: @@ -6832,6 +6841,49 @@ paths: summary: Retrieve sandbox deployment tags: - Sandbox Manager + patch: + consumes: + - application/json + description: Changes the Core idle timeout for every running allocation without a restart or generation rollout. Requires checkpoint support and no active reset. Already-started suspensions and snapshot expiry are unchanged. Capacity pressure can still suspend quiet idle compute earlier. + parameters: + - description: Idle timeout in seconds + in: body + name: body + required: true + schema: + $ref: '#/definitions/api.SandboxIdlePolicyInput' + produces: + - application/json + responses: + "200": + description: OK + schema: + $ref: '#/definitions/deployment.View' + "400": + description: Bad Request + schema: + $ref: '#/definitions/api.CoreErrorResponse' + "401": + description: Unauthorized + schema: + $ref: '#/definitions/api.CoreErrorResponse' + "409": + description: Conflict + schema: + $ref: '#/definitions/api.CoreErrorResponse' + "500": + description: Internal Server Error + schema: + $ref: '#/definitions/api.CoreErrorResponse' + "503": + description: Service Unavailable + schema: + $ref: '#/definitions/api.CoreErrorResponse' + security: + - DeploymentAdminAuth: [] + summary: Update automatic suspension policy + tags: + - Sandbox Manager post: consumes: - application/json diff --git a/contracts/agents-api/sandbox-deployment.md b/contracts/agents-api/sandbox-deployment.md index 4721b0aaa..2ce2a5771 100644 --- a/contracts/agents-api/sandbox-deployment.md +++ b/contracts/agents-api/sandbox-deployment.md @@ -6,6 +6,12 @@ The sandbox deployment selects the Sandbox Provider, the per-sandbox resources a This contract owns the Core API routes below and their semantics. The [nodes guide](../../docs/getting-started/nodes.md) owns the operator workflow, the [machine connection API](./machine-api.md#node-routes) the routes nodes call, and the [sandbox node protocol](./node-generation-protocol.md) the node connection. +## Suspension policy + +`PATCH /core/v1/sandbox/deployment` accepts `{"idle_seconds": 300}`. `idle_seconds` is a required integer from 30 to 86400 seconds, defaulting to 300, stored in Core's database. Only configured Providers declaring checkpoint support accept this change; unconfigured, unsupported or resetting deployments reject it. Success returns the deployment view with the saved `suspension.idle_seconds`. + +Changes apply to existing and new running allocations on the next lifecycle pass, without a Core restart, generation change or node rollout. Suspension already in progress is not cancelled; existing snapshots and the 24-hour retention period are unchanged. [Capacity-pressure reclamation](../../docs/sandbox-provider.md#suspension) can still suspend after 15 seconds of inactivity. Web manages this deployment-wide policy beside per-node capacity limits in **Nodes** → **Capacity and suspension**; the policy belongs to the deployment and capacity limits remain node-owned. + ## Routes Every route requires the Core key. [Web's console server](../../docs/web/console-server.md) adds it on the server side for signed-in requests. @@ -13,6 +19,7 @@ Every route requires the Core key. [Web's console server](../../docs/web/console | Route | Effect | | --- | --- | | `GET /core/v1/sandbox/deployment` | Read the safe active configuration, rollout, reset and resource counts | +| `PATCH /core/v1/sandbox/deployment` | Update deployment-wide idle suspension without changing the configuration generation | | `POST /core/v1/sandbox/deployment` | Select the initial provider, resources and Runtime | | `PUT /core/v1/sandbox/deployment` | Change the same provider's target online | | `POST /core/v1/sandbox/deployment/reset` | Start or escalate a durable clear of hosted resources | diff --git a/contracts/agents-api/zh/admin-api.md b/contracts/agents-api/zh/admin-api.md index 8a238fd60..8e63a671d 100644 --- a/contracts/agents-api/zh/admin-api.md +++ b/contracts/agents-api/zh/admin-api.md @@ -1,7 +1,7 @@ --- title: "Core 管理 API" source: contracts/agents-api/admin-api.md -source_hash: 4538ac64f7aee138b27b98f7f33542230d0b21a4fc183dc375fe0803833ac0d6 +source_hash: 108b88af05b64e9a72f635707bdcd06441328542e114eadc31dd29ba50b45ba1 --- Core 管理 API(`/core/v1`)用于管理安装实例:Project 及其 API 密钥、Project 资源的读取和删除、执行器凭据、部署默认模型、沙箱部署及其节点、监控和审计。Web 的[控制台服务器](../../../docs/zh/web/console-server.md#forwarding-to-core)会为已登录的管理员调用它;运维人员则从 Core 主机上的脚本调用它([编写 Core API 脚本](../../../docs/zh/getting-started/operations.md#script-the-core-api))。生成的架构是 [core.openapi.yaml](../core.openapi.yaml),所有错误都使用 [Core 错误封装](core-errors.md)。 @@ -36,7 +36,7 @@ Core 管理 API(`/core/v1`)用于管理安装实例:Project 及其 API 密 | `projects/{project_id}/environments/{environment_id}/executor-credentials[/{key_id}]` | `self_hosted` Environment 的执行器凭据 | [执行器凭据](environment-executor-credentials.md#core-key-routes) | | `projects/{project_id}/resource-owners`、`projects/{project_id}/write-operations` | 创建资源的 API 密钥以及每个密钥执行的写入 | [写入溯源](#write-provenance) | | `harnesses`、`harnesses/{harness}/model-configuration` | 已启用的 Harness 及各 Harness 的部署默认模型 | [部署默认值](model-execution.md#deployment-defaults) | -| `sandbox/deployment`、`sandbox/deployment/reset`、`sandbox/providers/{provider}/discovery` | 沙箱提供商、资源和 Runtime、重置,以及提供商配置发现,例如 E2B 模板 | [沙箱部署](sandbox-deployment.md#routes) | +| `sandbox/deployment`、`sandbox/deployment/reset`、`sandbox/providers/{provider}/discovery` | 沙箱提供商、资源和 Runtime、空闲挂起策略、重置,以及提供商配置发现,例如 E2B 模板 | [沙箱部署](sandbox-deployment.md#routes) | | `sandbox/enrollment-tokens`、`sandbox/nodes[/{node_id}[/allocations]]` | 节点注册令牌、节点及其分配和主机历史 | [节点指南](../../../docs/zh/getting-started/nodes.md)、[沙箱部署](sandbox-deployment.md)、[节点主机历史](runtime-observability-api.md#node-host-observations-and-history) | | `summary` | 按 Project、Agent 或密钥统计的 Session 数量和使用情况 | [汇总](#summary) | | `metrics` | Core 自身的进程、执行、数据库和作业指标 | [Core 指标](core-metrics.md) | diff --git a/contracts/agents-api/zh/sandbox-deployment.md b/contracts/agents-api/zh/sandbox-deployment.md index a740e85b2..9efa5e355 100644 --- a/contracts/agents-api/zh/sandbox-deployment.md +++ b/contracts/agents-api/zh/sandbox-deployment.md @@ -1,13 +1,19 @@ --- title: "沙箱部署" source: contracts/agents-api/sandbox-deployment.md -source_hash: fabf8ff55b54d16e603a21ece74162ff9cecc441dbcab6216c0a9e807ebe7dce +source_hash: 5794644abc5175e18023e3e113e1a55303828b4b1b5cba4c8dcd95a646de0124 --- 沙箱部署为 Core 管理的 `openai_hosted` 执行选择 Sandbox Provider、每个沙箱的资源以及不可变的 Runtime 发行版。PostgreSQL 为每个安装维护一个当前有效选择;Web 和 Core API 写入同一配置。节点文件保存其已安装副本和特定于主机的路径,且不能覆盖其资源或 Runtime。该选择独立于 Harness。部署可以保持未配置状态,没有节点;此时它拒绝托管准入。 本契约负责下列 Core API 路由及其语义。[节点指南](../../../docs/zh/getting-started/nodes.md)负责操作员工作流,[机器连接 API](machine-api.md#node-routes)负责节点调用的路由,[沙箱节点协议](node-generation-protocol.md)负责节点连接。 +## 挂起策略 {#suspension-policy} + +`PATCH /core/v1/sandbox/deployment` 接受 `{"idle_seconds": 300}`。`idle_seconds` 是必填整数,范围为 30–86400 秒,默认为 300。该设置保存在 Core 的数据库中。只有已配置且声明 checkpoint 支持的 Provider 可修改;未配置、不支持或正在重置的部署拒绝写入。成功返回当前部署视图,`suspension.idle_seconds` 反映已保存的值。 + +变更从下一轮生命周期检查起作用于现有和新建的运行中分配,不重启 Core、不推进配置代次,也不重新部署节点。已经开始的挂起不撤销;现有快照和 24 小时快照保留期不变。[容量压力回收](../../../docs/zh/sandbox-provider.md#suspension)仍可在空闲 15 秒后提前挂起。Web 在 **Nodes** → **Capacity and suspension** 同处管理此部署级策略和各节点容量上限;策略适用于整个部署,名额上限仍属于各个节点。 + ## 路由 {#routes} 每个路由都需要 Core 密钥。[Web 的控制台服务器](../../../docs/zh/web/console-server.md)会在服务器端为已登录请求添加该密钥。 @@ -15,6 +21,7 @@ source_hash: fabf8ff55b54d16e603a21ece74162ff9cecc441dbcab6216c0a9e807ebe7dce | 路由 | 效果 | | --- | --- | | `GET /core/v1/sandbox/deployment` | 读取安全的当前配置、推出、重置和资源计数 | +| `PATCH /core/v1/sandbox/deployment` | 更新整个部署的空闲挂起策略,不改变配置代次 | | `POST /core/v1/sandbox/deployment` | 选择初始提供商、资源和 Runtime | | `PUT /core/v1/sandbox/deployment` | 在线更改同一提供商的目标 | | `POST /core/v1/sandbox/deployment/reset` | 启动托管资源的持久清除或将其升级为持久清除 | diff --git a/docs/api/index.md b/docs/api/index.md index 0a6d3cf8c..d50b8cc6c 100644 --- a/docs/api/index.md +++ b/docs/api/index.md @@ -21,3 +21,5 @@ A credential used in another namespace gets 401: a Project API key on `/core/v1` Nodes, Runtime daemons and the self-hosted installer call `/api/v1` with their own credentials. The [machine connection API](../../contracts/agents-api/machine-api.md) lists every route, caller and credential. Operator scripts configure independent filesystem storage through Core-key-authenticated `GET` and `PUT /core/v1/workspace-storage`; see [Workspace storage](../../contracts/agents-api/admin-api.md#workspace-storage). + +Web and operator scripts update the sandbox idle policy through Core-key-authenticated `PATCH /core/v1/sandbox/deployment`; see [Sandbox deployment](../../contracts/agents-api/sandbox-deployment.md). diff --git a/docs/configuration.md b/docs/configuration.md index 8bebb69cb..625e404d3 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -94,7 +94,8 @@ Runtime settings live in Core's database. Change them in Web; scripts use the sa | --- | --- | --- | --- | | Sandbox backend: Docker, microsandbox or E2B | **System** → **Manage sandbox configuration**: the setup wizard, ending with **Save configuration** | `/core/v1/sandbox/deployment` | One backend per installation, chosen after the first sign-in. Another backend needs **Reset deployment** first; see [change the sandbox configuration](./getting-started/nodes.md#change-the-sandbox-configuration) | | Sandbox size, Runtime release, E2B key and template build | **System** → **Manage sandbox configuration** → **Change resources** | `/core/v1/sandbox/deployment` | Web proposes the [default size the Provider declares](./sandbox-provider.md#register-the-provider-kind). Existing sandboxes keep their size and release. The E2B key is write-only and encrypted | -| Nodes and their capacity | **Nodes**: **Add node**; **Edit node** and **Remove node** on a node's page | `/core/v1/sandbox/enrollment-tokens`, `/core/v1/sandbox/nodes` | See [Node capacity](#node-capacity) and the [nodes guide](./getting-started/nodes.md) | +| Nodes and their capacity | **Nodes**: **Add node**; **Capacity and suspension** → **Edit node**; **Remove node** on a node's page | `/core/v1/sandbox/enrollment-tokens`, `/core/v1/sandbox/nodes` | See [Node capacity](#node-capacity) and the [nodes guide](./getting-started/nodes.md) | +| Idle suspension | **Nodes** → **Capacity and suspension** → **Idle before suspension (seconds)** → **Save** | `PATCH /core/v1/sandbox/deployment` | Deployment-wide; see [suspension policy](../contracts/agents-api/sandbox-deployment.md#suspension-policy) | | Projects and API keys | **Projects and keys**: **Create project**, **Rename**, **Issue key**, **Revoke**, **Archive** | `/core/v1/projects` | Keys are shown once; Core stores digests | | Default model per harness | **System** → **Default model configuration**: **Set** | `/core/v1/harnesses/{harness}/model-configuration` | See [Default models](#default-models) | | Executor credentials of a self-hosted Session | **Session log**, then the **Session** page: **Executor credentials** | `/core/v1/projects/{project_id}/environments/{environment_id}/executor-credentials` | See [self-hosted executors](./getting-started/self-hosted.md) | diff --git a/docs/getting-started/nodes.md b/docs/getting-started/nodes.md index 222fd8553..cced5ac36 100644 --- a/docs/getting-started/nodes.md +++ b/docs/getting-started/nodes.md @@ -100,6 +100,7 @@ Run `sudo journalctl -u oac-node-.service`. In a root shell, om Sandbox settings apply to the whole installation; nodes follow them. - **Size, Runtime release, E2B key or template build.** Open **System** → **Manage sandbox configuration** → **Change resources**, edit and save. Existing sandboxes keep their configuration. Each node prepares the new one while it keeps serving the old one, and **Nodes** shows its progress: **Preparing target**, **Ready for target** or **Preparation failed** with the [reason](#readiness-codes). Core places new Sessions on nodes ready for the new configuration first, and on nodes still serving an older one when those have no room. E2B changes apply at once. +- **Capacity and idle suspension.** In **Nodes** → **Capacity and suspension**, use **Edit node** for each node's active and retained limits, and **Idle before suspension (seconds)** → **Save** for the deployment-wide [idle policy](../../contracts/agents-api/sandbox-deployment.md#suspension-policy). - **Backend.** On the same page, choose **Reset deployment**. Auto reset archives idle hosted Sessions at once and lets running work finish until the deadline you set; force cancels it now. Offline nodes must come back so Core can confirm their cleanup. When the reset completes, Core has retired every node and unused command: set up the new backend, then add nodes again. **Cancel reset** stops the remaining work; archived Sessions stay archived. The [reset contract](../../contracts/agents-api/sandbox-deployment.md#generation-ownership-and-rollout) describes what reset archives and keeps. To archive a single hosted Session, use the [Core API](../../contracts/agents-api/admin-api.md#session-archive). diff --git a/docs/sandbox-provider.md b/docs/sandbox-provider.md index 421c3a8d6..3360f83fa 100644 --- a/docs/sandbox-provider.md +++ b/docs/sandbox-provider.md @@ -184,7 +184,7 @@ The deployment's CPU, memory and disk settings, `max_active`, `max_retained` and ### Suspension -Core suspends the idle work of every provider that declares checkpoint support, with one shared policy: normally it suspends work idle for 5 minutes (300 seconds) and keeps the snapshot for 24 hours (86400 seconds). The deployment's [`suspension`](../contracts/agents-api/sandbox-deployment.md#safe-response) reports these values. Core suspends after initialization completes, when no root or Subagent Turn is queued, in progress or waiting, no input or file operation is pending, and real activity has been idle for that time. A Session need not have run a Turn. The idle clock starts no earlier than the allocation entering the running compute phase, including after a wake. For node allocations Core records the first root or child terminal transition with the database clock in the same transaction. Candidate filtering and the Session-locked recheck compare elapsed database time with the idle duration, and the initial snapshot retention deadline is anchored to the same database observation, so Core and database host clocks need not agree. Native completion timestamps stay unchanged in public history but never drive idle admission, and heartbeats never reset activity. Before acknowledging a planned suspension, the daemon closes admission and drains native cleanup, output receipts and file work. +Core suspends the idle work of every provider that declares checkpoint support, with one shared policy: the [configurable idle timeout](../contracts/agents-api/sandbox-deployment.md#suspension-policy) defaults to 5 minutes (300 seconds), and snapshots are kept for 24 hours (86400 seconds). The deployment's [`suspension`](../contracts/agents-api/sandbox-deployment.md#safe-response) reports these values. Core suspends after initialization completes, when no root or Subagent Turn is queued, in progress or waiting, no input or file operation is pending, and real activity has been idle for that time. A Session need not have run a Turn. The idle clock starts no earlier than the allocation entering the running compute phase, including after a wake. For node allocations Core records the first root or child terminal transition with the database clock in the same transaction. Candidate filtering and the Session-locked recheck compare elapsed database time with the idle duration, and the initial snapshot retention deadline is anchored to the same database observation, so Core and database host clocks need not agree. Native completion timestamps stay unchanged in public history but never drive idle admission, and heartbeats never reset activity. Before acknowledging a planned suspension, the daemon closes admission and drains native cleanup, output receipts and file work. Capacity pressure can suspend an already idle node allocation before the normal timeout, after a 15-second inactivity grace. The Session-locked decision rechecks pending work, wake requests and activity, then takes the shared deployment capacity lock and confirms compatible waiting demand. It never interrupts an active Turn or releases capacity before the ordinary quiesce, capture and source-stop proofs complete. Pressure-triggered reclamation is conservative: an in-flight reclamation delays another only when its node is currently eligible to serve the same demand. Unavailable or incompatible nodes keep their ownership receipts without blocking healthy capacity. If suspension cannot make capacity usable, the waiting request keeps its original deadline. diff --git a/docs/zh/api/index.md b/docs/zh/api/index.md index 8e74b6ee9..ebf280b70 100644 --- a/docs/zh/api/index.md +++ b/docs/zh/api/index.md @@ -1,7 +1,7 @@ --- title: "API 命名空间和凭据" source: docs/api/index.md -source_hash: cb8f94a1fc272ce57b319fa375c9a82cc450092526492e2141d08ecca1cf42f2 +source_hash: 66a80b6c579488fe1412ede51a0f0f9c03205443bc0be935b6a79d56f0e758d4 --- Core 提供三个命名空间。每个命名空间都有一种调用方及其独立凭据,凭据只能在其所属命名空间中使用。 @@ -23,3 +23,5 @@ Core 提供三个命名空间。每个命名空间都有一种调用方及其独 节点、Runtime 守护进程和自托管安装程序使用各自的凭据调用 `/api/v1`。[机器连接 API](../../../contracts/agents-api/zh/machine-api.md) 列出了每个路由、调用方和凭据。 运维脚本通过经 Core key 认证的 `GET` 和 `PUT /core/v1/workspace-storage` 配置独立文件系统存储;参见[工作区存储](../../../contracts/agents-api/zh/admin-api.md#workspace-storage)。 + +Web 和运维脚本通过 Core key 认证的 `PATCH /core/v1/sandbox/deployment` 更新沙箱空闲策略;参见[沙箱部署](../../../contracts/agents-api/zh/sandbox-deployment.md)。 diff --git a/docs/zh/configuration.md b/docs/zh/configuration.md index bed419030..f4b4bb11a 100644 --- a/docs/zh/configuration.md +++ b/docs/zh/configuration.md @@ -1,7 +1,7 @@ --- title: "配置参考" source: docs/configuration.md -source_hash: 506550dd727916c603fa2dfd713eaf190856b0536d6e4b6652a374419f2e7a4e +source_hash: 70593bfb0249b0ff864d2b7a6062e2dad852b544ecf3274cbf8c4c9ba7eec527 --- Core 安装的每项设置都恰好只有一个归属位置,分属以下三类: @@ -98,7 +98,8 @@ Runtime 在发现 Harness 适配器之前解析资源目录。`` | --- | --- | --- | --- | | 沙箱后端:Docker、microsandbox 或 E2B | **System** → **Manage sandbox configuration**:设置向导,最后点击 **Save configuration** | `/core/v1/sandbox/deployment` | 每个安装只能使用一个后端,在首次登录后选择。要改用其他后端,必须先执行 **Reset deployment**;请参阅[更改沙箱配置](getting-started/nodes.md#change-the-sandbox-configuration) | | 沙箱大小、Runtime 发行版、E2B 密钥和模板构建 | **System** → **Manage sandbox configuration** → **Change resources** | `/core/v1/sandbox/deployment` | Web 会推荐 [Provider 声明的默认大小](sandbox-provider.md#register-the-provider-kind)。现有沙箱会保留其大小和发行版。E2B 密钥仅可写入,并且已加密 | -| 节点及其容量 | **Nodes**:**Add node**;在节点页面上使用 **Edit node** 和 **Remove node** | `/core/v1/sandbox/enrollment-tokens`、`/core/v1/sandbox/nodes` | 请参阅[节点容量](#node-capacity)和[节点指南](getting-started/nodes.md) | +| 节点及其容量 | **Nodes**:**Add node**;**Capacity and suspension** → **Edit node**;在节点页面上使用 **Remove node** | `/core/v1/sandbox/enrollment-tokens`、`/core/v1/sandbox/nodes` | 请参阅[节点容量](#node-capacity)和[节点指南](getting-started/nodes.md) | +| 空闲挂起 | **Nodes** → **Capacity and suspension** → **Idle before suspension (seconds)** → **Save** | `PATCH /core/v1/sandbox/deployment` | 整个部署统一生效;参阅[挂起策略](../../contracts/agents-api/zh/sandbox-deployment.md#suspension-policy) | | 项目和 API 密钥 | **Projects and keys**:**Create project**、**Rename**、**Issue key**、**Revoke**、**Archive** | `/core/v1/projects` | 密钥只显示一次;Core 存储其摘要 | | 每个 Harness 的默认模型 | **System** → **Default model configuration**:**Set** | `/core/v1/harnesses/{harness}/model-configuration` | 请参阅[默认模型](#default-models) | | 自托管 Session 的执行器凭据 | **Session log**,然后进入 **Session** 页面:**Executor credentials** | `/core/v1/projects/{project_id}/environments/{environment_id}/executor-credentials` | 请参阅[自托管执行器](getting-started/self-hosted.md) | diff --git a/docs/zh/getting-started/nodes.md b/docs/zh/getting-started/nodes.md index d1160bf81..03dda61c4 100644 --- a/docs/zh/getting-started/nodes.md +++ b/docs/zh/getting-started/nodes.md @@ -1,7 +1,7 @@ --- title: "添加和管理节点" source: docs/getting-started/nodes.md -source_hash: 2549c792c6838d69a210be789bf969ed9e2c49b7da9bc9f0151eaa1298965184 +source_hash: dc80db64e7c4d6a021a929db027cfbca6ca8e877b47126ddaa5f48e493461d6d --- 节点是一台 Linux 主机,在沙箱后端为 microsandbox(推荐)或 Docker 时,为 Core 托管 Session 运行沙箱。Core 将新 Session 分配给有空余容量的节点;节点创建沙箱,沙箱回连 Core。E2B 不需要节点。应用为自己的 Session 连接的机器是[自托管执行器](self-hosted.md),而不是节点。 @@ -102,6 +102,7 @@ root 只准备账号、组和服务单元;其他操作(包括 Docker 网络 沙箱设置适用于整个安装,节点跟随这些设置。 - **规格、Runtime 版本、E2B 密钥或模板构建。** 打开 **System** → **Manage sandbox configuration** → **Change resources**,修改并保存。已有沙箱保持其配置。每个节点继续服务旧配置,同时准备新配置;**Nodes** 展示进度:**Preparing target**、**Ready for target**,或带[原因](#readiness-codes)的 **Preparation failed**。Core 优先在已准备好新配置的节点上放置新 Session;这些节点没有容量时,会使用仍服务旧配置的节点。E2B 变更立即生效。 +- **容量与空闲挂起。** 在 **Nodes** → **Capacity and suspension** 中,通过 **Edit node** 修改各节点的活跃和保留上限;通过 **Idle before suspension (seconds)** → **Save** 修改整个部署的[空闲策略](../../../contracts/agents-api/zh/sandbox-deployment.md#suspension-policy)。 - **后端。** 同一页面选择 **Reset deployment**。自动重置立即归档空闲托管 Session,并让运行中的工作在设定截止时间前完成;强制重置立即取消工作。离线节点必须恢复连接,让 Core 确认清理。重置完成时,Core 已撤销所有节点和未使用命令:配置新后端后重新添加节点。**Cancel reset** 停止剩余工作;已归档 Session 仍保持归档。 [重置协议](../../../contracts/agents-api/zh/sandbox-deployment.md#generation-ownership-and-rollout)描述重置会归档和保留哪些内容。归档单个托管 Session 时,使用 [Core API](../../../contracts/agents-api/zh/admin-api.md#session-archive)。 diff --git a/docs/zh/sandbox-provider.md b/docs/zh/sandbox-provider.md index f7f696b65..d61680e20 100644 --- a/docs/zh/sandbox-provider.md +++ b/docs/zh/sandbox-provider.md @@ -1,7 +1,7 @@ --- title: "添加 Sandbox Provider" source: docs/sandbox-provider.md -source_hash: 30ee130c8e62827662f1d0867da206e911e4dbb57ed5d75fe68cffdfda55e2b3 +source_hash: ade105e47436ca60735ba9f084c370d047403ba2f867f38ff55d798d305ae196 --- **Sandbox Provider** 为 Core 管理的 Environment 提供 Runtime daemon 运行所需的外层计算资源,以及启动 daemon 的有界引导流程。本指南说明如何添加 Provider,并作为 Core 驱动 Provider 的参考。接口为 [`SandboxProvider`](https://github.com/MiniMax-AI/OpenAgentCore/blob/main/services/core/internal/sandbox/sandbox_provider.go)。 @@ -186,7 +186,7 @@ placement 自动完成:Session 创建提交持久的 pending 工作,共同 ### 暂停 {#suspension} -Core 用同一个共享 policy 暂停每个声明 checkpoint 支持的 provider 的空闲工作:通常在工作空闲 5 分钟(300 秒)后暂停,snapshot 保留 24 小时(86400 秒)。部署的 [`suspension`](../../contracts/agents-api/zh/sandbox-deployment.md#safe-response) 报告这两个值。Core 在 initialization 完成、没有 root 或 Subagent Turn 排队、进行中或等待、没有 pending input 或 file operation,且真实 activity 已空闲达到该时间后暂停。Session 无需已运行过 Turn。idle clock 不早于 allocation 进入 running compute phase 的时刻开始,包括 wake 后重新进入该 phase。对于 node allocation,Core 在同一事务中用数据库时钟记录首个 root 或 child terminal transition。candidate filter 和 Session-locked recheck 比较数据库已过时间与 idle duration,初始 snapshot retention deadline 也锚定同一数据库观测,因此 Core 与数据库主机时钟无需一致。原生 completion timestamp 在公开历史中保持不变,但不驱动 idle admission,heartbeat 不重置 activity。确认计划暂停前,daemon 关闭 admission 并排空 native cleanup、output receipt 和 file work。 +Core 用同一个共享 policy 暂停每个声明 checkpoint 支持的 provider 的空闲工作:[可配置的空闲时长](../../contracts/agents-api/zh/sandbox-deployment.md#suspension-policy)默认为 5 分钟(300 秒),snapshot 保留 24 小时(86400 秒)。部署的 [`suspension`](../../contracts/agents-api/zh/sandbox-deployment.md#safe-response) 报告这两个值。Core 在 initialization 完成、没有 root 或 Subagent Turn 排队、进行中或等待、没有 pending input 或 file operation,且真实 activity 已空闲达到该时间后暂停。Session 无需已运行过 Turn。idle clock 不早于 allocation 进入 running compute phase 的时刻开始,包括 wake 后重新进入该 phase。对于 node allocation,Core 在同一事务中用数据库时钟记录首个 root 或 child terminal transition。candidate filter 和 Session-locked recheck 比较数据库已过时间与 idle duration,初始 snapshot retention deadline 也锚定同一数据库观测,因此 Core 与数据库主机时钟无需一致。原生 completion timestamp 在公开历史中保持不变,但不驱动 idle admission,heartbeat 不重置 activity。确认计划暂停前,daemon 关闭 admission 并排空 native cleanup、output receipt 和 file work。 容量压力可以让已经空闲的 node allocation 在通常超时之前暂停,但仍需经过 15 秒无活动宽限期。Session 锁内的决策重新检查 pending work、wake request 和 activity,再获取共享 deployment 容量锁并确认兼容的等待需求。它不会中断活动 Turn,也不会在普通 quiesce、capture 和 source-stop 证明完成前释放容量。压力触发的回收采用保守策略:只有进行中回收所属的节点当前有资格服务同一需求时,它才会推迟另一次回收。不可用或不兼容的节点保留其 ownership receipt,但不会阻挡健康节点的容量。如果暂停无法使容量可用,等待请求仍保留原始期限。 diff --git a/packages/agents-client/src/generated/core-api.ts b/packages/agents-client/src/generated/core-api.ts index fef191a79..b09c56f55 100644 --- a/packages/agents-client/src/generated/core-api.ts +++ b/packages/agents-client/src/generated/core-api.ts @@ -805,6 +805,10 @@ export interface SandboxEnrollmentTokenRequest { } export const sandboxEnrollmentTokenRequestFields = ["max_active", "max_retained"] as const; export const sandboxEnrollmentTokenRequestRequired = [] as const; +export interface SandboxIdlePolicyInput { + idle_seconds: number; +} +export const sandboxIdlePolicyInputFields = ["idle_seconds"] as const; export interface SandboxMutationResponse { deleted?: boolean; id: string; diff --git a/packages/agents-client/src/sandbox-client.test.ts b/packages/agents-client/src/sandbox-client.test.ts index a50ec0e3b..8d53d2cfd 100644 --- a/packages/agents-client/src/sandbox-client.test.ts +++ b/packages/agents-client/src/sandbox-client.test.ts @@ -183,6 +183,20 @@ describe("Core sandbox credential boundaries", () => { expect(fetch).toHaveBeenCalledTimes(1); }); + it("patches idle policy once through the existing deployment resource", async () => { + const fetch = vi.fn().mockResolvedValue(response({ ...docker, suspension: { idle_seconds: 60, retention_seconds: 86400 } })); + const admin = new SandboxAdminClient({ baseUrl: "/core/v1/sandbox", token: "admin-only", fetch }); + expect((await admin.updateSuspension({ idle_seconds: 60 })).suspension?.idle_seconds).toBe(60); + const [url, init] = fetch.mock.calls[0]!; + expect(url).toBe("/core/v1/sandbox/deployment"); + expect(init?.method).toBe("PATCH"); + expect(JSON.parse(String(init?.body))).toEqual({ idle_seconds: 60 }); + expect(new Headers(init?.headers).get("Authorization")).toBe("Bearer admin-only"); + fetch.mockRejectedValue(new TypeError("Network failed")); + await expect(admin.updateSuspension({ idle_seconds: 60 })).rejects.toThrow("Network failed"); + expect(fetch).toHaveBeenCalledTimes(2); + }); + it("initializes using only the explicit provider with cancellation and admin credentials", async () => { const deployment = docker; const fetch = vi.fn().mockResolvedValue(response(deployment)); diff --git a/packages/agents-client/src/sandbox-client.ts b/packages/agents-client/src/sandbox-client.ts index 9820d0015..9b8a9bca3 100644 --- a/packages/agents-client/src/sandbox-client.ts +++ b/packages/agents-client/src/sandbox-client.ts @@ -11,7 +11,7 @@ import { sandboxAllocationListFields, sandboxNodeListFields, sandboxResourcesFields, sandboxResourcesRequired, suspensionFields, type DeploymentSpec, type DeploymentView, type HostHistoryPoint, type Node, type NodeAllocation, type NodeDetail, type NodeDiagnosticCode, type NodeHost, type NodeRollout, type NodeUpdate, type Reset, type ResetOfflineNode, type ResetRequest, type Rollout, type RuntimeRelease, - type SandboxDeploymentInput, type SandboxEnrollmentToken, type SandboxResources as SandboxResourcesResource, + type SandboxIdlePolicyInput, type SandboxDeploymentInput, type SandboxEnrollmentToken, type SandboxResources as SandboxResourcesResource, } from "./generated/core-api"; import type { ReadOptions } from "./types"; @@ -264,6 +264,9 @@ export class SandboxAdminClient { updateDeployment(input: UpdateSandboxDeployment, options?: ReadOptions): Promise { return this.#writeDeployment("PUT", input, options); } + async updateSuspension(input: SandboxIdlePolicyInput, options?: ReadOptions): Promise { + return projectDeployment(await this.#core.json("/deployment", options, "PATCH", input)); + } async startReset(input: StartSandboxReset, options?: ReadOptions): Promise { return projectDeployment(await this.#core.json("/deployment/reset", options, "POST", input)); } diff --git a/services/core/cmd/server/managed_setup.go b/services/core/cmd/server/managed_setup.go index 758ff5db0..a0aee5c87 100644 --- a/services/core/cmd/server/managed_setup.go +++ b/services/core/cmd/server/managed_setup.go @@ -158,8 +158,7 @@ func (s *managedSetup) configuration(setup deployment.Setup) (execution.Prepared CoreURL: s.runtimeAPI, BackendFingerprint: setup.BackendFingerprint, Provider: provider, Resources: setup.Specification.Resources, WorkspaceRequirements: adapter.Policy.Workspace, Workspace: setup.Specification.Workspace} if setup.Suspension != nil { - selected.Suspension = &execution.RuntimeSuspensionPolicy{IdleTimeout: time.Duration(setup.Suspension.IdleSeconds) * time.Second, - Retention: time.Duration(setup.Suspension.RetentionSeconds) * time.Second} + selected.Suspension = &execution.RuntimeSuspensionPolicy{Retention: time.Duration(setup.Suspension.RetentionSeconds) * time.Second} } return execution.PreparedRuntimeDeployment{Config: selected, Publish: s.publish}, nil } diff --git a/services/core/internal/api/fakes_test.go b/services/core/internal/api/fakes_test.go index f7bb22fac..b94f0074b 100644 --- a/services/core/internal/api/fakes_test.go +++ b/services/core/internal/api/fakes_test.go @@ -181,6 +181,7 @@ func (f *fakeConfigurationDiscovery) DiscoverConfiguration(a0 context.Context, a } type fakeDeployment struct { + updateIdleSeconds func(context.Context, int64) (deployment.View, error) t testing.TB view func(context.Context) (deployment.View, error) listNodes func(context.Context) ([]deployment.Node, error) @@ -1171,3 +1172,10 @@ func (f *fakeWriteAudit) ListWriteOperations(a0 context.Context, a1 string, a2 w } return f.listWriteOperations(a0, a1, a2) } + +func (f *fakeDeployment) UpdateIdleSeconds(ctx context.Context, seconds int64) (deployment.View, error) { + if f.updateIdleSeconds == nil { + unexpectedCall(f.t, "UpdateIdleSeconds") + } + return f.updateIdleSeconds(ctx, seconds) +} diff --git a/services/core/internal/api/sandbox_deployment_setup.go b/services/core/internal/api/sandbox_deployment_setup.go index 8b2b8f77a..030fb8251 100644 --- a/services/core/internal/api/sandbox_deployment_setup.go +++ b/services/core/internal/api/sandbox_deployment_setup.go @@ -195,3 +195,36 @@ func parseResetGeneration(r *http.Request) (uint64, error) { } return strconv.ParseUint(query.Get("expected_generation"), 10, 64) } + +// SandboxIdlePolicyInput changes the deployment-wide idle timeout independently of its generation. +type SandboxIdlePolicyInput struct { + IdleSeconds *int64 `json:"idle_seconds" binding:"required" minimum:"30" maximum:"86400"` +} + +// @Summary Update automatic suspension policy +// @Description Changes the Core idle timeout for every running allocation without a restart or generation rollout. Requires checkpoint support and no active reset. Already-started suspensions and snapshot expiry are unchanged. Capacity pressure can still suspend quiet idle compute earlier. +// @Tags Sandbox Manager +// @Produce json +// @Security DeploymentAdminAuth +// @Accept json +// @Param body body api.SandboxIdlePolicyInput true "Idle timeout in seconds" +// @Success 200 {object} deployment.View +// @Failure 400,401,409,500,503 {object} CoreErrorResponse +// @Router /core/v1/sandbox/deployment [patch] +func (h *Handler) updateSandboxIdlePolicy(w http.ResponseWriter, r *http.Request) { + raw, ok := readJSONBody(w, r) + if !ok { + return + } + var input SandboxIdlePolicyInput + if decodeInputObject(raw, &input, "idle_seconds") != nil || input.IdleSeconds == nil { + writeDeploymentError(w, r, deployment.ErrInvalidInput) + return + } + result, err := h.Sandboxes.Deployment.UpdateIdleSeconds(r.Context(), *input.IdleSeconds) + if err != nil { + writeDeploymentError(w, r, err) + return + } + writeJSON(w, http.StatusOK, result) +} diff --git a/services/core/internal/api/sandbox_deployment_setup_test.go b/services/core/internal/api/sandbox_deployment_setup_test.go index c3106ad62..26dcfb126 100644 --- a/services/core/internal/api/sandbox_deployment_setup_test.go +++ b/services/core/internal/api/sandbox_deployment_setup_test.go @@ -47,3 +47,38 @@ func TestSandboxDeploymentSetupRequiresAdministratorAndStrictBody(t *testing.T) } } } + +func TestSandboxIdlePolicyRequiresAdministratorAndStrictBody(t *testing.T) { + deps, fakes := sandboxFakes(t) + calls := 0 + fakes.deployment.updateIdleSeconds = func(_ context.Context, seconds int64) (deployment.View, error) { + calls++ + if seconds < 30 || seconds > 86400 { + return deployment.View{}, deployment.ErrInvalidInput + } + return deployment.View{Suspension: &deployment.Suspension{IdleSeconds: seconds, RetentionSeconds: 86400}}, nil + } + h := newTestHandler(t, deps) + for _, test := range []struct { + token, body string + status, calls int + }{ + {"caller", `{"idle_seconds":60}`, 401, 0}, + {"node-credential", `{"idle_seconds":60}`, 401, 0}, + {"administrator", `{}`, 400, 0}, + {"administrator", `{"idle_seconds":null}`, 400, 0}, + {"administrator", `{"idle_seconds":30.5}`, 400, 0}, + {"administrator", `{"idle_seconds":60,"retention_seconds":1}`, 400, 0}, + {"administrator", `{"idle_seconds":29}`, 400, 1}, + {"administrator", `{"idle_seconds":86401}`, 400, 2}, + {"administrator", `{"idle_seconds":60}`, 200, 3}, + } { + request := httptest.NewRequest(http.MethodPatch, "/core/v1/sandbox/deployment", strings.NewReader(test.body)) + request.Header.Set("Authorization", "Bearer "+test.token) + result := httptest.NewRecorder() + h.ServeHTTP(result, request) + if result.Code != test.status || calls != test.calls { + t.Fatal(result.Code, calls, result.Body.String()) + } + } +} diff --git a/services/core/internal/api/sandbox_manager.go b/services/core/internal/api/sandbox_manager.go index 57713a257..3ce2491ce 100644 --- a/services/core/internal/api/sandbox_manager.go +++ b/services/core/internal/api/sandbox_manager.go @@ -38,6 +38,7 @@ type SandboxEnrollmentTokenRequest struct { // identity, generation configuration, capacity and removal. type Deployment interface { View(ctx context.Context) (deployment.View, error) + UpdateIdleSeconds(context.Context, int64) (deployment.View, error) ListNodes(ctx context.Context) ([]deployment.Node, error) NodeDetail(ctx context.Context, id, window string) (deployment.NodeDetail, error) UpdateNode(ctx context.Context, id string, update deployment.NodeUpdate) error @@ -70,6 +71,7 @@ func (h *Handler) registerSandboxManagerRoutes(r chi.Router) { r.Get("/sandbox/deployment", h.sandboxDeployment) r.Post("/sandbox/providers/{provider}/discovery", h.discoverSandboxConfiguration) r.Post("/sandbox/deployment", h.initializeSandboxDeployment) + r.Patch("/sandbox/deployment", h.updateSandboxIdlePolicy) r.Put("/sandbox/deployment", h.updateSandboxDeployment) r.Post("/sandbox/deployment/reset", h.startSandboxReset) r.Delete("/sandbox/deployment/reset", h.cancelSandboxReset) diff --git a/services/core/internal/db/queries/runtime_deployment.sql b/services/core/internal/db/queries/runtime_deployment.sql index 332d6bbc3..68f6da578 100644 --- a/services/core/internal/db/queries/runtime_deployment.sql +++ b/services/core/internal/db/queries/runtime_deployment.sql @@ -19,3 +19,6 @@ SELECT (SELECT count(*) FROM environment_executor_credentials c LEFT JOIN environments e ON e.id = c.environment_id LEFT JOIN sessions s ON s.id = e.session_id WHERE c.revoked_at IS NULL AND (c.environment_id IS NULL OR s.deleted_at IS NULL))::bigint AS self_hosted_executors; + +-- name: UpdateSandboxIdleSeconds :exec +UPDATE runtime_deployment SET idle_seconds = $1 WHERE singleton = true; diff --git a/services/core/internal/db/sqlc/models.go b/services/core/internal/db/sqlc/models.go index fd8071a64..178b78211 100644 --- a/services/core/internal/db/sqlc/models.go +++ b/services/core/internal/db/sqlc/models.go @@ -262,6 +262,7 @@ type RuntimeDeployment struct { ResetAudit []byte `json:"reset_audit"` ProviderConfig []byte `json:"provider_config"` ProviderMetadata []byte `json:"provider_metadata"` + IdleSeconds int64 `json:"idle_seconds"` } type RuntimeDeploymentGeneration struct { diff --git a/services/core/internal/db/sqlc/runtime_deployment.sql.go b/services/core/internal/db/sqlc/runtime_deployment.sql.go index 8f22aa200..9bc5ad9b3 100644 --- a/services/core/internal/db/sqlc/runtime_deployment.sql.go +++ b/services/core/internal/db/sqlc/runtime_deployment.sql.go @@ -56,7 +56,7 @@ func (q *Queries) CountRuntimeDeploymentResources(ctx context.Context) (CountRun } const lockRuntimeDeployment = `-- name: LockRuntimeDeployment :one -SELECT singleton, installation_id, backend_fingerprint, updated_at, provider_kind, owner_epoch, generation, mode, provider_credential, specification, reset_clear, reset_requested_at, reset_deadline_at, reset_forced_at, reset_audit, provider_config, provider_metadata FROM runtime_deployment WHERE singleton = true FOR UPDATE +SELECT singleton, installation_id, backend_fingerprint, updated_at, provider_kind, owner_epoch, generation, mode, provider_credential, specification, reset_clear, reset_requested_at, reset_deadline_at, reset_forced_at, reset_audit, provider_config, provider_metadata, idle_seconds FROM runtime_deployment WHERE singleton = true FOR UPDATE ` func (q *Queries) LockRuntimeDeployment(ctx context.Context) (RuntimeDeployment, error) { @@ -80,6 +80,16 @@ func (q *Queries) LockRuntimeDeployment(ctx context.Context) (RuntimeDeployment, &i.ResetAudit, &i.ProviderConfig, &i.ProviderMetadata, + &i.IdleSeconds, ) return i, err } + +const updateSandboxIdleSeconds = `-- name: UpdateSandboxIdleSeconds :exec +UPDATE runtime_deployment SET idle_seconds = $1 WHERE singleton = true +` + +func (q *Queries) UpdateSandboxIdleSeconds(ctx context.Context, idleSeconds int64) error { + _, err := q.db.Exec(ctx, updateSandboxIdleSeconds, idleSeconds) + return err +} diff --git a/services/core/internal/db/sqlc/runtime_nodes.sql.go b/services/core/internal/db/sqlc/runtime_nodes.sql.go index 3c1306a11..c57ca09fd 100644 --- a/services/core/internal/db/sqlc/runtime_nodes.sql.go +++ b/services/core/internal/db/sqlc/runtime_nodes.sql.go @@ -119,7 +119,7 @@ func (q *Queries) DisconnectRuntimeNode(ctx context.Context, arg DisconnectRunti } const getRuntimeDeployment = `-- name: GetRuntimeDeployment :one -SELECT singleton, installation_id, backend_fingerprint, updated_at, provider_kind, owner_epoch, generation, mode, provider_credential, specification, reset_clear, reset_requested_at, reset_deadline_at, reset_forced_at, reset_audit, provider_config, provider_metadata FROM runtime_deployment WHERE singleton=true +SELECT singleton, installation_id, backend_fingerprint, updated_at, provider_kind, owner_epoch, generation, mode, provider_credential, specification, reset_clear, reset_requested_at, reset_deadline_at, reset_forced_at, reset_audit, provider_config, provider_metadata, idle_seconds FROM runtime_deployment WHERE singleton=true ` func (q *Queries) GetRuntimeDeployment(ctx context.Context) (RuntimeDeployment, error) { @@ -143,6 +143,7 @@ func (q *Queries) GetRuntimeDeployment(ctx context.Context) (RuntimeDeployment, &i.ResetAudit, &i.ProviderConfig, &i.ProviderMetadata, + &i.IdleSeconds, ) return i, err } diff --git a/services/core/internal/db/sqlc/sandbox_reset.sql.go b/services/core/internal/db/sqlc/sandbox_reset.sql.go index fcc589d64..4b2ac0e79 100644 --- a/services/core/internal/db/sqlc/sandbox_reset.sql.go +++ b/services/core/internal/db/sqlc/sandbox_reset.sql.go @@ -50,7 +50,7 @@ func (q *Queries) ForceSandboxReset(ctx context.Context) error { } const getSandboxDeploymentSnapshot = `-- name: GetSandboxDeploymentSnapshot :one -WITH deployment AS MATERIALIZED (SELECT singleton, installation_id, backend_fingerprint, updated_at, provider_kind, owner_epoch, generation, mode, provider_credential, specification, reset_clear, reset_requested_at, reset_deadline_at, reset_forced_at, reset_audit, provider_config, provider_metadata FROM runtime_deployment WHERE singleton = true LIMIT 1), +WITH deployment AS MATERIALIZED (SELECT singleton, installation_id, backend_fingerprint, updated_at, provider_kind, owner_epoch, generation, mode, provider_credential, specification, reset_clear, reset_requested_at, reset_deadline_at, reset_forced_at, reset_audit, provider_config, provider_metadata, idle_seconds FROM runtime_deployment WHERE singleton = true LIMIT 1), observed AS MATERIALIZED (SELECT clock_timestamp() AS as_of), held AS ( SELECT a.deployment_generation, a.node_id, s.id AS session_id, e.id AS environment_id, false AS pending, @@ -91,7 +91,7 @@ held AS ( ), offline AS ( SELECT node_id, name, count(*)::bigint AS resources FROM classified WHERE offline GROUP BY node_id, name ) -SELECT d.singleton, d.installation_id, d.backend_fingerprint, d.updated_at, d.provider_kind, d.owner_epoch, d.generation, d.mode, d.provider_credential, d.specification, d.reset_clear, d.reset_requested_at, d.reset_deadline_at, d.reset_forced_at, d.reset_audit, d.provider_config, d.provider_metadata, +SELECT d.singleton, d.installation_id, d.backend_fingerprint, d.updated_at, d.provider_kind, d.owner_epoch, d.generation, d.mode, d.provider_credential, d.specification, d.reset_clear, d.reset_requested_at, d.reset_deadline_at, d.reset_forced_at, d.reset_audit, d.provider_config, d.provider_metadata, d.idle_seconds, (SELECT count(*) FROM classified WHERE NOT pending)::bigint AS allocations, (SELECT count(*) FROM classified WHERE pending)::bigint AS pending, jsonb_build_object( @@ -144,6 +144,7 @@ func (q *Queries) GetSandboxDeploymentSnapshot(ctx context.Context) (GetSandboxD &i.RuntimeDeployment.ResetAudit, &i.RuntimeDeployment.ProviderConfig, &i.RuntimeDeployment.ProviderMetadata, + &i.RuntimeDeployment.IdleSeconds, &i.Allocations, &i.Pending, &i.Remaining, diff --git a/services/core/internal/deployment/allocations.go b/services/core/internal/deployment/allocations.go index 002c37914..a1c182b67 100644 --- a/services/core/internal/deployment/allocations.go +++ b/services/core/internal/deployment/allocations.go @@ -251,8 +251,8 @@ func (e *ExecutionOperations) cleanup(ctx context.Context, owner Allocation, evi // revision and the Session lock fence a stale lifecycle observation. Moving // running compute to quiescing requires idle compute and either the idle // timeout or capacity pressure, rechecked in the transaction. -func (e *ExecutionOperations) SetCompute(ctx context.Context, owner Allocation, phase string, state json.RawMessage, retainedUntil *time.Time, idleTimeout time.Duration) (Allocation, error) { - if owner.ComputePhase == "suspended" && phase == "restoring" || !computeTransition(owner.ComputePhase, phase) || !json.Valid(state) || (owner.ComputePhase == "running" && phase == "quiescing" && idleTimeout <= 0) { +func (e *ExecutionOperations) SetCompute(ctx context.Context, owner Allocation, phase string, state json.RawMessage, retainedUntil *time.Time) (Allocation, error) { + if owner.ComputePhase == "suspended" && phase == "restoring" || !computeTransition(owner.ComputePhase, phase) || !json.Valid(state) { return Allocation{}, ErrInvalidInput } if phase != "running" && (retainedUntil == nil || retainedUntil.IsZero()) { @@ -278,7 +278,11 @@ func (e *ExecutionOperations) SetCompute(ctx context.Context, owner Allocation, if current.ComputeActivityAt.After(owner.ComputeActivityAt) { return Allocation{}, ErrAllocationConflict } - if !activity.ReadyToSuspend(idleTimeout) { + seconds, err := tx.LoadIdleSeconds() + if err != nil { + return Allocation{}, err + } + if !activity.ReadyToSuspend(time.Duration(seconds) * time.Second) { // A short quiet period also backs off a rejected daemon quiesce. if !activity.ReadyToSuspend(15*time.Second) || current.NodeID == "" { return Allocation{}, ErrNotIdle diff --git a/services/core/internal/deployment/allocations_test.go b/services/core/internal/deployment/allocations_test.go index 9df8051ac..3d8c6e80b 100644 --- a/services/core/internal/deployment/allocations_test.go +++ b/services/core/internal/deployment/allocations_test.go @@ -93,6 +93,7 @@ func (f *fakeReservationTx) InsertAllocation(allocation NewAllocation) (Allocati } type fakeAllocationTx struct { + idleSeconds int64 canRetainEnvironment func(Allocation) (bool, error) t testing.TB loadActivity func(Allocation) (Activity, error) @@ -461,19 +462,15 @@ func TestSetComputeValidatesBeforeStorage(t *testing.T) { until := time.Now().Add(time.Hour) for name, call := range map[string]func(*ExecutionOperations) error{ "skipped phase": func(o *ExecutionOperations) error { - _, err := o.SetCompute(t.Context(), owner, "suspended", json.RawMessage(`{}`), &until, time.Second) + _, err := o.SetCompute(t.Context(), owner, "suspended", json.RawMessage(`{}`), &until) return err }, "state not object": func(o *ExecutionOperations) error { - _, err := o.SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`[]`), &until, time.Second) + _, err := o.SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`[]`), &until) return err }, "no retention": func(o *ExecutionOperations) error { - _, err := o.SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), nil, time.Second) - return err - }, - "no idle timeout": func(o *ExecutionOperations) error { - _, err := o.SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until, 0) + _, err := o.SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), nil) return err }, } { @@ -684,7 +681,7 @@ func TestSetComputeRechecksIdlePressureAndActivity(t *testing.T) { current.ComputeActivityAt = now } pressureChecks, writes := 0, 0 - tx := &fakeAllocationTx{t: t, + tx := &fakeAllocationTx{t: t, idleSeconds: 300, loadAllocation: func() (Allocation, error) { return current, nil }, loadSessionDevice: func() (SessionDevice, bool, error) { return SessionDevice{ID: owner.DeviceID, EnvironmentID: owner.EnvironmentID}, true, nil @@ -714,7 +711,7 @@ func TestSetComputeRechecksIdlePressureAndActivity(t *testing.T) { t.Fatal(err) } until := now.Add(time.Hour) - _, err = operations.SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until, 5*time.Minute) + _, err = operations.SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until) if !errors.Is(err, test.want) { t.Fatalf("SetCompute: %v, want %v", err, test.want) } @@ -753,7 +750,7 @@ func TestBeginRestoreDirectKeepsAllocationAndRetention(t *testing.T) { if err != nil || restored.ID != owner.ID || restored.NodeID != "" || restored.ComputePhase != "restoring" { t.Fatal("direct restore changed ownership", restored, err) } - if _, err := operations.SetCompute(t.Context(), owner, "restoring", json.RawMessage(`{}`), &until, time.Minute); !errors.Is(err, ErrInvalidInput) { + if _, err := operations.SetCompute(t.Context(), owner, "restoring", json.RawMessage(`{}`), &until); !errors.Is(err, ErrInvalidInput) { t.Fatal("restore bypassed capacity admission", err) } } @@ -762,3 +759,5 @@ func (f *fakeReservationTx) ReleaseFailedPlacement(placement.Reserved) error { unexpected(f.t, "ReleaseFailedPlacement") return nil } + +func (f *fakeAllocationTx) LoadIdleSeconds() (int64, error) { return f.idleSeconds, nil } diff --git a/services/core/internal/deployment/fakes_test.go b/services/core/internal/deployment/fakes_test.go index 5b3e69dfa..814aef4d3 100644 --- a/services/core/internal/deployment/fakes_test.go +++ b/services/core/internal/deployment/fakes_test.go @@ -214,6 +214,7 @@ func (f *fakeNodeReads) GenerationKept(nodeID string, generation uint64) (bool, } type fakeNodeTx struct { + saveIdleSeconds func(int64) error t testing.TB loadDeployment func() (Record, error) loadNode func(string) (StoredNode, error) @@ -644,3 +645,10 @@ func (f *fakeReader) PlacementDemand(context.Context, PlacementDemandCursor) ([] func (f *fakeReader) RetainedNativeHistory(context.Context, AllocationKey) (bool, error) { panic("unexpected RetainedNativeHistory") } + +func (f *fakeNodeTx) SaveIdleSeconds(seconds int64) error { + if f.saveIdleSeconds == nil { + unexpected(f.t, "SaveIdleSeconds") + } + return f.saveIdleSeconds(seconds) +} diff --git a/services/core/internal/deployment/service.go b/services/core/internal/deployment/service.go index 4bc68a72b..1cf87452a 100644 --- a/services/core/internal/deployment/service.go +++ b/services/core/internal/deployment/service.go @@ -65,7 +65,7 @@ func (s *Service) view(snapshot Snapshot) (View, error) { result.Configuration = configurationJSON(record.Public) result.Metadata = configurationJSON(record.Metadata) result.CredentialConfigured = d.CredentialStored - if result.Suspension, err = s.suspension(d.Provider); err != nil { + if result.Suspension, err = s.suspension(d.Provider, d.IdleSeconds); err != nil { return View{}, err } } @@ -108,7 +108,11 @@ func (s *Service) setup(d Record) (Setup, error) { if err != nil { return Setup{}, ErrConflict } - return s.describe(result) + result, err = s.describe(result) + if result.Suspension != nil { + result.Suspension.IdleSeconds = d.IdleSeconds + } + return result, err } // describe adds the provider's declared operations, suspension policy and @@ -119,7 +123,7 @@ func (s *Service) describe(setup Setup) (Setup, error) { return Setup{}, err } setup.Operations = adapter.Operations() - if setup.Suspension, err = s.suspension(setup.Provider); err != nil { + if setup.Suspension, err = s.suspension(setup.Provider, 300); err != nil { return Setup{}, err } setup.UsesCredential, err = s.registry.UsesCredential(setup.Provider) @@ -252,12 +256,12 @@ func (s *Service) SetupForSelection(installationID string, input sandbox.Selecti // suspension returns Core's idle suspension policy for a provider that // declares checkpoint support, and nil for any other provider. -func (s *Service) suspension(provider string) (*Suspension, error) { +func (s *Service) suspension(provider string, idleSeconds int64) (*Suspension, error) { checkpoint, err := s.registry.SupportsCheckpoint(provider) if err != nil || !checkpoint { return nil, err } - return &Suspension{IdleSeconds: 5 * 60, RetentionSeconds: 24 * 60 * 60}, nil + return &Suspension{IdleSeconds: idleSeconds, RetentionSeconds: 24 * 60 * 60}, nil } // validateSelection rejects a selection its provider cannot normalize. @@ -313,3 +317,34 @@ func (s *Service) ValidateWorkspaceConfiguration(ctx context.Context, declaratio } return workspacefs.ValidateCombination(*adapter.Policy.Workspace, declaration, 0) } + +// UpdateIdleSeconds changes only the shared idle policy, without a new generation. +func (s *Service) UpdateIdleSeconds(ctx context.Context, seconds int64) (View, error) { + if seconds < 30 || seconds > 86400 { + return View{}, ErrInvalidInput + } + err := s.storage.WithNodes(ctx, func(tx NodeTx) error { + d, err := tx.LoadDeployment() + if err != nil { + return err + } + if d.Provider == "" { + return ErrNotConfigured + } + if d.Reset != nil { + return ErrResetInProgress + } + supported, err := s.registry.SupportsCheckpoint(d.Provider) + if err != nil { + return err + } + if !supported { + return ErrInvalidInput + } + return tx.SaveIdleSeconds(seconds) + }) + if err != nil { + return View{}, err + } + return s.View(ctx) +} diff --git a/services/core/internal/deployment/storage.go b/services/core/internal/deployment/storage.go index 163daa263..f08eea4e7 100644 --- a/services/core/internal/deployment/storage.go +++ b/services/core/internal/deployment/storage.go @@ -112,6 +112,8 @@ type ReservationTx interface { // current, the allocation LoadAllocation returned, and returns it changed. A // write whose stored guard no longer holds is ErrAllocationConflict. type AllocationTx interface { + // LoadIdleSeconds locks the deployment and reads the current idle policy. + LoadIdleSeconds() (int64, error) // CanRetainEnvironment checks the ready retained filesystem, completed // initialization and selected Harness's qualified history under the Session lock. CanRetainEnvironment(current Allocation) (bool, error) @@ -294,8 +296,10 @@ type NodeReads interface { GenerationKept(nodeID string, generation uint64) (bool, error) } -// NodeTx is one node management transaction. +// NodeTx is one node or live deployment policy management transaction. type NodeTx interface { + // SaveIdleSeconds updates the live policy without changing the generation. + SaveIdleSeconds(int64) error NodeReads ListNodes() ([]NodeRecord, error) // InsertNode returns ErrNodeExists when the ID is in use, including by a @@ -360,6 +364,7 @@ type DeploymentTx interface { // Record is the stored deployment. type Record struct { + IdleSeconds int64 // InstallationID is empty until Web setup claims an installation. InstallationID string Provider string diff --git a/services/core/internal/execution/archive_cancellation_cleanup_test.go b/services/core/internal/execution/archive_cancellation_cleanup_test.go index 3ac684477..6dddee6bc 100644 --- a/services/core/internal/execution/archive_cancellation_cleanup_test.go +++ b/services/core/internal/execution/archive_cancellation_cleanup_test.go @@ -108,7 +108,7 @@ func TestArchiveWaitingCleanupReceiptBarrier(t *testing.T) { currentCompute := sandbox.Compute{ID: uuid.NewString(), Name: owner.ID + "-g0"} if checkpoint { state, _ := json.Marshal(runtimeCompute{Current: currentCompute}) - owner, err = leased.Deployment.SetCompute(t.Context(), owner, "running", state, nil, 0) + owner, err = leased.Deployment.SetCompute(t.Context(), owner, "running", state, nil) if err != nil { t.Fatal(err) } diff --git a/services/core/internal/execution/runtime_compute.go b/services/core/internal/execution/runtime_compute.go index bac1e7e48..e19eb5fdb 100644 --- a/services/core/internal/execution/runtime_compute.go +++ b/services/core/internal/execution/runtime_compute.go @@ -17,11 +17,10 @@ import ( "github.com/google/uuid" ) -// RuntimeSuspensionPolicy is the idle suspension policy of a provider that -// suspends sandboxes. +// RuntimeSuspensionPolicy supplies snapshot retention for a provider that +// suspends sandboxes. Deployment storage owns the mutable idle timeout. type RuntimeSuspensionPolicy struct { - IdleTimeout time.Duration - Retention time.Duration + Retention time.Duration } type runtimeCompute struct { @@ -38,15 +37,7 @@ func (r *runtimeLifecycle) saveCompute(ctx context.Context, owner deployment.All if err != nil { return owner, err } - idleTimeout := time.Duration(0) - if phase == "quiescing" { - policy := r.config.Suspension - if policy == nil { - return owner, sandbox.ErrInvalid - } - idleTimeout = policy.IdleTimeout - } - next, err := r.deployment.SetCompute(ctx, owner, phase, raw, until, idleTimeout) + next, err := r.deployment.SetCompute(ctx, owner, phase, raw, until) if err == nil { log.Ctx(ctx).Info("Runtime compute phase saved", "allocation_id", owner.ID, "node_id", owner.NodeID, "from", owner.ComputePhase, "to", phase) } diff --git a/services/core/internal/execution/runtime_lifecycle.go b/services/core/internal/execution/runtime_lifecycle.go index 5cd4a82d7..6db971788 100644 --- a/services/core/internal/execution/runtime_lifecycle.go +++ b/services/core/internal/execution/runtime_lifecycle.go @@ -112,7 +112,7 @@ func validatedRuntimeProvider(config *RuntimeProvider, registry *runtimegateway. } if config.Suspension != nil { policy := *config.Suspension - if !sandbox.SupportsCheckpoint(config.Provider) || policy.IdleTimeout < time.Second || policy.Retention < time.Second { + if !sandbox.SupportsCheckpoint(config.Provider) || policy.Retention < time.Second { return RuntimeProvider{}, sandbox.ErrInvalid } copied.Suspension = &policy diff --git a/services/core/internal/persistence/postgres/deploymentpg/allocations.go b/services/core/internal/persistence/postgres/deploymentpg/allocations.go index 6ae6d1871..ca26ea929 100644 --- a/services/core/internal/persistence/postgres/deploymentpg/allocations.go +++ b/services/core/internal/persistence/postgres/deploymentpg/allocations.go @@ -668,3 +668,8 @@ func (t *reservationTx) ReleaseFailedPlacement(p placement.Reserved) error { } return t.q.ReleaseFailedRuntimePlacement(t.ctx, sqlc.ReleaseFailedRuntimePlacementParams{EnvironmentID: t.environment, NodeID: node, DeploymentGeneration: pgtype.Int8{Int64: int64(p.Generation), Valid: true}}) } + +func (t *allocationTx) LoadIdleSeconds() (int64, error) { + d, err := t.q.LockRuntimeDeployment(t.ctx) + return d.IdleSeconds, err +} diff --git a/services/core/internal/persistence/postgres/deploymentpg/allocations_test.go b/services/core/internal/persistence/postgres/deploymentpg/allocations_test.go index 0fed81384..f25a04de8 100644 --- a/services/core/internal/persistence/postgres/deploymentpg/allocations_test.go +++ b/services/core/internal/persistence/postgres/deploymentpg/allocations_test.go @@ -116,7 +116,7 @@ func TestAllocationWritesNeedTheLease(t *testing.T) { "SettleCreation": func() error { _, err := closed.SettleCreation(t.Context(), owner); return err }, "RequestCleanup": func() error { _, err := closed.RequestCleanup(t.Context(), owner); return err }, "SetCompute": func() error { - _, err := closed.SetCompute(t.Context(), owner, "running", []byte(`{}`), nil, 0) + _, err := closed.SetCompute(t.Context(), owner, "running", []byte(`{}`), nil) return err }, "ClearWake": func() error { return closed.ClearWake(t.Context(), owner, time.Now()) }, diff --git a/services/core/internal/persistence/postgres/deploymentpg/checkpoint_transfer_test.go b/services/core/internal/persistence/postgres/deploymentpg/checkpoint_transfer_test.go index 919f2dac6..8d3713ef3 100644 --- a/services/core/internal/persistence/postgres/deploymentpg/checkpoint_transfer_test.go +++ b/services/core/internal/persistence/postgres/deploymentpg/checkpoint_transfer_test.go @@ -264,7 +264,7 @@ func TestCheckpointPressureSuspendsOnlyUsableDestination(t *testing.T) { readyCheckpointNode(t, f, other, view, checkpointClass) } until := time.Now().Add(24 * time.Hour) - _, err := changes.SetCompute(t.Context(), active, "quiescing", json.RawMessage(`{}`), &until, 5*time.Minute) + _, err := changes.SetCompute(t.Context(), active, "quiescing", json.RawMessage(`{}`), &until) if scenario == "waiting restore" { if err != nil { t.Fatal("compatible wake cannot trigger pressure suspension", err) diff --git a/services/core/internal/persistence/postgres/deploymentpg/idle_policy_test.go b/services/core/internal/persistence/postgres/deploymentpg/idle_policy_test.go new file mode 100644 index 000000000..1befe0ccf --- /dev/null +++ b/services/core/internal/persistence/postgres/deploymentpg/idle_policy_test.go @@ -0,0 +1,66 @@ +package deploymentpg_test + +import ( + "encoding/json" + "errors" + "testing" + "time" + + "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/deployment" + "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox" +) + +func TestIdlePolicyChangesCurrentAllocationsWithoutRollout(t *testing.T) { + f := newFixture(t) + changes, _ := f.execution(t) + installation, before := f.initialize(t, changes, sandbox.Selection{Provider: "microsandbox", DeploymentSpec: testSpecification("microsandbox")}) + node := f.enroll(t, before, deployment.Capacity{MaxActive: 1, MaxRetained: 4}) + f.connect(t, node.NodeID) + owner := runningPressureOwner(t, f, changes, installation, node.NodeID, before.Generation) + if before.Suspension.IdleSeconds != 300 { + t.Fatal(before.Suspension) + } + until := time.Now().Add(time.Hour) + if _, err := changes.SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until); !errors.Is(err, deployment.ErrNotIdle) { + t.Fatal("default idle policy", err) + } + for _, invalid := range []int64{0, 29, 86401} { + if _, err := f.service.UpdateIdleSeconds(admin(t), invalid); !errors.Is(err, deployment.ErrInvalidInput) { + t.Fatal(invalid, err) + } + } + // A longer saved value is also applied to the existing allocation. + for _, seconds := range []int64{86400, 30} { + after, err := f.service.UpdateIdleSeconds(admin(t), seconds) + if err != nil || after.Suspension.IdleSeconds != seconds || after.Generation != before.Generation || after.SpecificationDigest != before.SpecificationDigest || after.OwnerEpoch != before.OwnerEpoch || after.Suspension.RetentionSeconds != before.Suspension.RetentionSeconds { + t.Fatal(after, err) + } + _, err = changes.SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until) + if seconds == 86400 && !errors.Is(err, deployment.ErrNotIdle) { + t.Fatal("increase ignored", err) + } + if seconds == 30 && err != nil { + t.Fatal("decrease ignored", err) + } + } + // Changing policy after the phase commit does not cancel the suspension. + if _, err := f.service.UpdateIdleSeconds(admin(t), 86400); err != nil { + t.Fatal(err) + } + current, err := f.adapter.EnvironmentAllocation(t.Context(), deployment.AllocationKey{TenantID: owner.TenantID, EnvironmentID: owner.EnvironmentID}) + if err != nil || current.ComputePhase != "quiescing" || current.ComputeRetainedUntil == nil { + t.Fatal(current, err) + } +} + +func TestIdlePolicyRejectsUnsupportedAndUnconfiguredDeployment(t *testing.T) { + f := newFixture(t) + if _, err := f.service.UpdateIdleSeconds(admin(t), 60); !errors.Is(err, deployment.ErrNotConfigured) { + t.Fatal(err) + } + changes, _ := f.execution(t) + f.initialize(t, changes, sandbox.Selection{Provider: "docker", DeploymentSpec: testSpecification("docker")}) + if _, err := f.service.UpdateIdleSeconds(admin(t), 60); !errors.Is(err, deployment.ErrInvalidInput) { + t.Fatal(err) + } +} diff --git a/services/core/internal/persistence/postgres/deploymentpg/records.go b/services/core/internal/persistence/postgres/deploymentpg/records.go index 0fc164c42..9124c05b9 100644 --- a/services/core/internal/persistence/postgres/deploymentpg/records.go +++ b/services/core/internal/persistence/postgres/deploymentpg/records.go @@ -58,7 +58,7 @@ func translate(err error) error { // view never needs it. A credential the key cannot open or authenticate is an // internal decryption error. func record(d sqlc.RuntimeDeployment, cipher *credentialcrypto.Cipher, open bool) deployment.Record { - r := deployment.Record{InstallationID: uuidString(d.InstallationID), Provider: d.ProviderKind, BackendFingerprint: d.BackendFingerprint, + r := deployment.Record{IdleSeconds: d.IdleSeconds, InstallationID: uuidString(d.InstallationID), Provider: d.ProviderKind, BackendFingerprint: d.BackendFingerprint, Generation: uint64(d.Generation), OwnerEpoch: uint64(d.OwnerEpoch), Mode: d.Mode, Specification: d.Specification, Configuration: sandbox.ConfigurationRecord{Public: d.ProviderConfig, Metadata: d.ProviderMetadata}, CredentialStored: len(d.ProviderCredential) > 0} if r.CredentialStored && open { diff --git a/services/core/internal/persistence/postgres/deploymentpg/retention_pressure_test.go b/services/core/internal/persistence/postgres/deploymentpg/retention_pressure_test.go index ac6864145..4ae9b02a5 100644 --- a/services/core/internal/persistence/postgres/deploymentpg/retention_pressure_test.go +++ b/services/core/internal/persistence/postgres/deploymentpg/retention_pressure_test.go @@ -106,7 +106,7 @@ func TestPressurePreservesRetainableOwnerWhenRetainedSlotsAreFull(t *testing.T) } pendingPressureDemand(t, f) until := time.Now().Add(24 * time.Hour) - if _, err = changes.SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until, 5*time.Minute); !errors.Is(err, deployment.ErrNotIdle) { + if _, err = changes.SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until); !errors.Is(err, deployment.ErrNotIdle) { t.Fatal("suspended without usable capacity", err) } } diff --git a/services/core/internal/persistence/postgres/deploymentpg/suspension_pressure_test.go b/services/core/internal/persistence/postgres/deploymentpg/suspension_pressure_test.go index 77fed6a34..e81422e41 100644 --- a/services/core/internal/persistence/postgres/deploymentpg/suspension_pressure_test.go +++ b/services/core/internal/persistence/postgres/deploymentpg/suspension_pressure_test.go @@ -62,7 +62,7 @@ func TestPressureSuspensionRequiresUsefulCapacity(t *testing.T) { f.connect(t, other.NodeID) } until := time.Now().Add(time.Hour) - _, err := changes.SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until, 5*time.Minute) + _, err := changes.SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until) if !errors.Is(err, test.want) { t.Fatalf("quiesce: %v, want %v", err, test.want) } @@ -95,7 +95,7 @@ func TestPressureSuspensionSerializesAcrossNodes(t *testing.T) { for i, owner := range owners { wg.Go(func() { until := time.Now().Add(time.Hour) - _, errs[i] = changes.SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until, 5*time.Minute) + _, errs[i] = changes.SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until) }) } wg.Wait() @@ -143,7 +143,7 @@ func TestPressureSuspensionServesRestoreOnItsOriginalNode(t *testing.T) { t.Fatal(err) } until := time.Now().Add(time.Hour) - if _, err := changes.SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until, 5*time.Minute); err != nil { + if _, err := changes.SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until); err != nil { t.Fatal(err) } var wake bool @@ -168,7 +168,7 @@ func TestPressureDemandScansPastIncompatiblePage(t *testing.T) { } pendingPressureDemand(t, f) until := time.Now().Add(time.Hour) - result, err := changes.SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until, 5*time.Minute) + result, err := changes.SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until) if err != nil { t.Fatal(err) } @@ -205,13 +205,13 @@ func TestPressureDemandRejectsUnqualifiedRetainedReceipt(t *testing.T) { } owner := runningPressureOwner(t, f, changes, installation, node.NodeID, view.Generation) until := time.Now().Add(time.Hour) - if _, err := changes.SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until, 5*time.Minute); !errors.Is(err, deployment.ErrNotIdle) { + if _, err := changes.SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until); !errors.Is(err, deployment.ErrNotIdle) { t.Fatal("unqualified receipt suspended healthy owner", err) } // An empty qualified page must not stop the cursor before a later // valid request, even though the raw first page was nonempty. pendingPressureDemand(t, f) - if _, err := changes.SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until, 5*time.Minute); err != nil { + if _, err := changes.SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until); err != nil { t.Fatal("later valid demand was hidden", err) } }) @@ -276,14 +276,14 @@ func TestPressureReclamationIgnoresUnavailableNodeReceipts(t *testing.T) { until := time.Now().Add(time.Hour) - current, err := changes.SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until, 5*time.Minute) + current, err := changes.SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until) if err != nil { t.Fatal("healthy suspension blocked", err) } // Supply healthy native quiesce/capture/stop receipts through the // existing fenced phase transitions; never settle the stuck owner. for _, next := range []string{"suspending", "suspended"} { - current, err = changes.SetCompute(t.Context(), current, next, json.RawMessage(`{}`), &until, 5*time.Minute) + current, err = changes.SetCompute(t.Context(), current, next, json.RawMessage(`{}`), &until) if err != nil { t.Fatal(err) } diff --git a/services/core/internal/persistence/postgres/deploymentpg/tx.go b/services/core/internal/persistence/postgres/deploymentpg/tx.go index 4441dd7d0..fde99c754 100644 --- a/services/core/internal/persistence/postgres/deploymentpg/tx.go +++ b/services/core/internal/persistence/postgres/deploymentpg/tx.go @@ -361,3 +361,7 @@ func (t *deploymentTx) RecordAudit(action, installationID string) error { func (t *deploymentTx) RecordAuditAs(source adminaudit.Source, action, installationID string) error { return auditpg.RecordDeploymentMutation(adminaudit.WithSource(t.ctx, source), t.q, action, "sandbox_deployment", installationID) } + +func (u unit) SaveIdleSeconds(seconds int64) error { + return u.q.UpdateSandboxIdleSeconds(u.ctx, seconds) +} diff --git a/services/core/migrations/000101_sandbox_idle_policy.sql b/services/core/migrations/000101_sandbox_idle_policy.sql new file mode 100644 index 000000000..65c64bc99 --- /dev/null +++ b/services/core/migrations/000101_sandbox_idle_policy.sql @@ -0,0 +1,5 @@ +-- +goose Up +ALTER TABLE runtime_deployment ADD COLUMN idle_seconds bigint NOT NULL DEFAULT 300 CHECK (idle_seconds BETWEEN 30 AND 86400); + +-- +goose Down +ALTER TABLE runtime_deployment DROP COLUMN idle_seconds; diff --git a/services/core/tests/integration/runtime_compute_lifecycle_test.go b/services/core/tests/integration/runtime_compute_lifecycle_test.go index 4c27a8e40..ce1df45c9 100644 --- a/services/core/tests/integration/runtime_compute_lifecycle_test.go +++ b/services/core/tests/integration/runtime_compute_lifecycle_test.go @@ -288,11 +288,12 @@ func newComputeLifecycleFixture(t *testing.T, maxActive, maxRetained int) *compu } server.Close() }) - f := &computeLifecycleFixture{t: t, store: s, provider: p, key: webDeployment(t, s, "microsandbox"), policy: execution.RuntimeSuspensionPolicy{IdleTimeout: time.Second, Retention: time.Hour}} + f := &computeLifecycleFixture{t: t, store: s, provider: p, key: webDeployment(t, s, "microsandbox"), policy: execution.RuntimeSuspensionPolicy{Retention: time.Hour}} view, err := deploymentService(t, s).View(t.Context()) if err != nil { t.Fatal(err) } + f.sql(`UPDATE runtime_deployment SET idle_seconds=30`) f.node = enrollNode(t, s, view, deployment.Capacity{MaxActive: maxActive, MaxRetained: maxRetained}).NodeID f.start() return f diff --git a/services/core/tests/integration/runtime_file_admission_test.go b/services/core/tests/integration/runtime_file_admission_test.go index 676c9e9d9..0368a101b 100644 --- a/services/core/tests/integration/runtime_file_admission_test.go +++ b/services/core/tests/integration/runtime_file_admission_test.go @@ -93,7 +93,7 @@ func TestRuntimeFileWriteAndQuiesceSerializeBothOrders(t *testing.T) { if first == "quiesce" { _, err = writes.ReserveEnvironmentFileWrite(ctx, owner.TenantID, owner.EnvironmentID, key) } else { - _, err = deploymentExecution(t, w).SetCompute(ctx, owner, "quiescing", json.RawMessage(`{}`), &until, time.Nanosecond) + _, err = deploymentExecution(t, w).SetCompute(ctx, owner, "quiescing", json.RawMessage(`{}`), &until) } done <- err }() diff --git a/services/core/tests/integration/runtime_idle_clock_test.go b/services/core/tests/integration/runtime_idle_clock_test.go index 298b2e082..ce1ca3299 100644 --- a/services/core/tests/integration/runtime_idle_clock_test.go +++ b/services/core/tests/integration/runtime_idle_clock_test.go @@ -40,7 +40,7 @@ func managedIdleClockFixture(t *testing.T) (*Store, *Store, deployment.Allocatio if err != nil { t.Fatal(err) } - owner, err = deploymentExecution(t, w).SetCompute(t.Context(), owner, "running", json.RawMessage(`{"instance":"original"}`), nil, 0) + owner, err = deploymentExecution(t, w).SetCompute(t.Context(), owner, "running", json.RawMessage(`{"instance":"original"}`), nil) if err != nil { t.Fatal(err) } @@ -58,6 +58,7 @@ func runtimeDatabaseTime(t *testing.T, s *Store) time.Time { func verifyManagedIdleClock(t *testing.T, s, w *Store, owner deployment.Allocation, before, after time.Time) deployment.Activity { t.Helper() const idleTimeout = time.Minute + runtimeSuspensionSQL(t, s.pool, "UPDATE runtime_deployment SET idle_seconds=60") // Refresh the owner so the idle policy, rather than the stale-activity fence, // must reject the recent completion. owner, err := deploymentStore(s).EnvironmentAllocation(t.Context(), deployment.AllocationKey{TenantID: owner.TenantID, EnvironmentID: owner.EnvironmentID}) @@ -71,12 +72,12 @@ func verifyManagedIdleClock(t *testing.T, s, w *Store, owner deployment.Allocati t.Fatal("idle clock did not use committed terminal ingestion", activity, before, after, err) } until := runtimeDatabaseTime(t, s).Add(time.Hour) - if _, err := deploymentExecution(t, w).SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until, idleTimeout); !errors.Is(err, deployment.ErrNotIdle) { + if _, err := deploymentExecution(t, w).SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until); !errors.Is(err, deployment.ErrNotIdle) { t.Fatal("new completion admitted premature idle", err) } // Advance only the internal activity age; the remote public timestamp remains unchanged. runtimeSuspensionSQL(t, s.pool, "UPDATE runtime_allocations SET compute_activity_at=clock_timestamp()-interval '2 minutes',compute_phase_changed_at=clock_timestamp()-interval '2 minutes' WHERE id=$1", owner.ID) - if _, err := deploymentExecution(t, w).SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until, idleTimeout); err != nil { + if _, err := deploymentExecution(t, w).SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until); err != nil { t.Fatal("remote timestamp delayed elapsed idle timer", err) } return activity diff --git a/services/core/tests/integration/runtime_node_lifecycle_fixture_test.go b/services/core/tests/integration/runtime_node_lifecycle_fixture_test.go index ca2573598..c1445835f 100644 --- a/services/core/tests/integration/runtime_node_lifecycle_fixture_test.go +++ b/services/core/tests/integration/runtime_node_lifecycle_fixture_test.go @@ -118,11 +118,12 @@ func newNodeIsolationFixture(t *testing.T, mode string) *nodeIsolationFixture { server.Close() }) f := &nodeIsolationFixture{initializationCancel: cancelPreparation, t: t, store: s, nodes: deploymentService(t, s), pool: pool, provider: p, key: webDeployment(t, s, "microsandbox"), nodeA: uuid.NewString(), nodeB: uuid.NewString()} + runtimeSuspensionSQL(t, pool, "UPDATE runtime_deployment SET idle_seconds=60") // Keep restored compute awake throughout the isolation assertions. // The suspension setup explicitly dates its activity two minutes in the past. owner := executionOwner(t, s) f.placement = owner.Deployment - f.worker = startOwnedWorker(t, t.Context(), s, &execution.Dispatcher{Registry: registry, ManagedRuntimes: webRuntimes(t, s, f.key, p, &execution.RuntimeSuspensionPolicy{IdleTimeout: time.Minute, Retention: time.Hour})}, owner) + f.worker = startOwnedWorker(t, t.Context(), s, &execution.Dispatcher{Registry: registry, ManagedRuntimes: webRuntimes(t, s, f.key, p, &execution.RuntimeSuspensionPolicy{Retention: time.Hour})}, owner) t.Cleanup(f.stop) f.epoch = fixtureOwnerEpoch(t, s) f.enroll(f.nodeA) diff --git a/services/core/tests/integration/runtime_suspension_concurrency_test.go b/services/core/tests/integration/runtime_suspension_concurrency_test.go index 381ed3e7f..4c4df51fd 100644 --- a/services/core/tests/integration/runtime_suspension_concurrency_test.go +++ b/services/core/tests/integration/runtime_suspension_concurrency_test.go @@ -124,7 +124,7 @@ func TestRuntimeSuspensionCaptureRechecksNewPendingWork(t *testing.T) { ctx, tx, blocker := runtimeSuspensionLockedSession(t, pool, owner.SessionID) done := make(chan error, 1) go func() { - _, err := deploymentExecution(t, w).SetCompute(ctx, owner, "suspending", json.RawMessage(`{}`), &until, 0) + _, err := deploymentExecution(t, w).SetCompute(ctx, owner, "suspending", json.RawMessage(`{}`), &until) done <- err }() runtimeSuspensionWaitBlocked(t, ctx, pool, blocker, done) @@ -193,7 +193,7 @@ func TestRuntimeSuspensionQuiesceCannotOvertakeClaim(t *testing.T) { ctx, tx, blocker := runtimeSuspensionLockedSession(t, pool, owner.SessionID) done := make(chan error, 1) go func() { - _, err := deploymentExecution(t, w).SetCompute(ctx, owner, "quiescing", json.RawMessage(`{}`), &until, time.Nanosecond) + _, err := deploymentExecution(t, w).SetCompute(ctx, owner, "quiescing", json.RawMessage(`{}`), &until) done <- err }() runtimeSuspensionWaitBlocked(t, ctx, pool, blocker, done) diff --git a/services/core/tests/integration/runtime_suspension_test.go b/services/core/tests/integration/runtime_suspension_test.go index e32df1a8c..06e3dd30e 100644 --- a/services/core/tests/integration/runtime_suspension_test.go +++ b/services/core/tests/integration/runtime_suspension_test.go @@ -31,10 +31,11 @@ func runtimeSuspensionFixture(t *testing.T) (*Store, *Store, *pgxpool.Pool, depl if err != nil { t.Fatal(err) } - owner, err = deploymentExecution(t, w).SetCompute(t.Context(), owner, "running", json.RawMessage(`{"instance":"original"}`), nil, 0) + owner, err = deploymentExecution(t, w).SetCompute(t.Context(), owner, "running", json.RawMessage(`{"instance":"original"}`), nil) if err != nil { t.Fatal(err) } + runtimeSuspensionSQL(t, pool, `UPDATE runtime_allocations SET created_at=clock_timestamp()-interval '10 minutes',compute_activity_at=clock_timestamp()-interval '10 minutes',compute_phase_changed_at=clock_timestamp()-interval '10 minutes' WHERE id=$1`, owner.ID) return s, w, pool, owner } @@ -54,16 +55,12 @@ func runtimeSuspensionCompleted(t *testing.T, pool *pgxpool.Pool, owner deployme func runtimeSuspensionStep(t *testing.T, w *Store, owner deployment.Allocation, phase string, until *time.Time) deployment.Allocation { t.Helper() - idleTimeout := time.Duration(0) - if owner.ComputePhase == "running" && phase == "quiescing" { - idleTimeout = time.Nanosecond - } var next deployment.Allocation var err error if phase == "restoring" { next, err = deploymentExecution(t, w).BeginRestore(t.Context(), owner, sandbox.CheckpointCompatibility{ArtifactDomain: "fixture-store", ExecutionClass: "fixture-runtime"}, json.RawMessage(`{"instance":"original","snapshot":"qualified"}`)) } else { - next, err = deploymentExecution(t, w).SetCompute(t.Context(), owner, phase, json.RawMessage(`{"instance":"original","snapshot":"qualified"}`), until, idleTimeout) + next, err = deploymentExecution(t, w).SetCompute(t.Context(), owner, phase, json.RawMessage(`{"instance":"original","snapshot":"qualified"}`), until) } if err != nil { t.Fatalf("%s -> %s: %v", owner.ComputePhase, phase, err) @@ -104,7 +101,7 @@ func TestRuntimeSuspensionRequiresIdleAndNoPendingWork(t *testing.T) { t.Fatalf("activity lost pending work: %+v", activity) } until := time.Now().Add(time.Hour) - _, err = deploymentExecution(t, w).SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until, time.Nanosecond) + _, err = deploymentExecution(t, w).SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until) if kind == "idle" || kind == "no_completed_turn" { if err != nil { t.Fatal(err) @@ -126,7 +123,7 @@ func TestRuntimeSuspensionCASAndActivityFence(t *testing.T) { wg.Add(1) go func() { defer wg.Done() - _, err := deploymentExecution(t, w).SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{"operation":"same-observation"}`), &until, time.Nanosecond) + _, err := deploymentExecution(t, w).SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{"operation":"same-observation"}`), &until) results <- err }() } @@ -152,7 +149,7 @@ func TestRuntimeSuspensionCASAndActivityFence(t *testing.T) { if current.ComputeRevision != owner.ComputeRevision+1 || current.ComputePhase != "quiescing" { t.Fatal("operation intent not durable", current) } - if _, err := deploymentExecution(t, w).SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{"stale":true}`), &until, time.Nanosecond); !errors.Is(err, deployment.ErrAllocationConflict) { + if _, err := deploymentExecution(t, w).SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{"stale":true}`), &until); !errors.Is(err, deployment.ErrAllocationConflict) { t.Fatal("stale phase overwrite", err) } current = runtimeSuspensionStep(t, w, current, "running", nil) @@ -168,15 +165,15 @@ func TestRuntimeSuspensionCASAndActivityFence(t *testing.T) { if err := deploymentExecution(t, w).ClearWake(t.Context(), latest, latest.ComputeActivityAt); err != nil { t.Fatal(err) } - if _, err := deploymentExecution(t, w).SetCompute(t.Context(), observed, "quiescing", json.RawMessage(`{}`), &until, time.Nanosecond); !errors.Is(err, deployment.ErrAllocationConflict) { + if _, err := deploymentExecution(t, w).SetCompute(t.Context(), observed, "quiescing", json.RawMessage(`{}`), &until); !errors.Is(err, deployment.ErrAllocationConflict) { t.Fatal("newer activity was swallowed", err) } for _, invalid := range []json.RawMessage{json.RawMessage(`[]`), json.RawMessage(`null`), json.RawMessage(`false`), json.RawMessage(`{`)} { - if _, err := deploymentExecution(t, w).SetCompute(t.Context(), latest, "running", invalid, nil, 0); !errors.Is(err, deployment.ErrInvalidInput) { + if _, err := deploymentExecution(t, w).SetCompute(t.Context(), latest, "running", invalid, nil); !errors.Is(err, deployment.ErrInvalidInput) { t.Fatal("non-object compute state accepted", string(invalid), err) } } - if _, err := deploymentExecution(t, w).SetCompute(t.Context(), latest, "suspended", json.RawMessage(`{}`), &until, 0); !errors.Is(err, deployment.ErrInvalidInput) { + if _, err := deploymentExecution(t, w).SetCompute(t.Context(), latest, "suspended", json.RawMessage(`{}`), &until); !errors.Is(err, deployment.ErrInvalidInput) { t.Fatal("running skipped snapshot protocol", err) } } @@ -362,19 +359,13 @@ func TestRuntimeSuspensionRechecksCompletionAgainstIdleTimeout(t *testing.T) { runtimeSuspensionSQL(t, pool, `INSERT INTO environment_file_writes(id,environment_id,device_id,request_sha256,state,created_at,settled_at) VALUES($1,$2,$3,$4,$5,clock_timestamp()-interval '10 minutes',clock_timestamp())`, uuid.NewString(), owner.EnvironmentID, owner.DeviceID, strings.Repeat("a", 64), strings.TrimPrefix(kind, "file_")) } until := time.Now().Add(time.Hour) - if _, err := deploymentExecution(t, w).SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until, idleTimeout); !errors.Is(err, deployment.ErrNotIdle) { + if _, err := deploymentExecution(t, w).SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until); !errors.Is(err, deployment.ErrNotIdle) { t.Fatal("completion after idle observation did not fence quiesce", err) } activity, err := deploymentStore(w).Activity(t.Context(), owner.ID) if err != nil || activity.Busy || activity.WakeRequested || activity.ReadyToSuspend(idleTimeout) { t.Fatal("last completion did not restart idle interval", activity, err) } - if _, err := deploymentExecution(t, w).SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until, 0); !errors.Is(err, deployment.ErrInvalidInput) { - t.Fatal("missing idle timeout accepted", err) - } - if _, err := deploymentExecution(t, w).SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until, time.Nanosecond); err != nil { - t.Fatal("elapsed idle timeout rejected", err) - } }) } } @@ -392,7 +383,7 @@ func TestRuntimeComputePhaseChangedAt(t *testing.T) { return value } entered := changedAt() - owner, err := deploymentExecution(t, w).SetCompute(t.Context(), owner, "running", json.RawMessage(`{"instance":"updated"}`), nil, 0) + owner, err := deploymentExecution(t, w).SetCompute(t.Context(), owner, "running", json.RawMessage(`{"instance":"updated"}`), nil) if err != nil { t.Fatal(err) } @@ -442,7 +433,7 @@ func TestNeverStartedSessionIdleStartsAfterInitialization(t *testing.T) { _, w, pool, owner := runtimeSuspensionFixture(t) const timeout = 5 * time.Minute until := time.Now().Add(time.Hour) - runtimeSuspensionSQL(t, pool, `UPDATE runtime_allocations SET compute_activity_at=clock_timestamp()-interval '1 hour',created_at=clock_timestamp()-interval '1 hour' WHERE id=$1`, owner.ID) + runtimeSuspensionSQL(t, pool, `UPDATE runtime_allocations SET compute_phase_changed_at=clock_timestamp(),compute_activity_at=clock_timestamp()-interval '1 hour',created_at=clock_timestamp()-interval '1 hour' WHERE id=$1`, owner.ID) activity, err := deploymentStore(w).Activity(t.Context(), owner.ID) if err != nil || activity.ReadyToSuspend(timeout) { t.Fatal("creation age bypassed ready-time idle clock", activity, err) @@ -453,16 +444,16 @@ func TestNeverStartedSessionIdleStartsAfterInitialization(t *testing.T) { t.Fatal("never-started initialized Session cannot idle", activity, err) } runtimeSuspensionSQL(t, pool, `UPDATE environments SET initialization='running' WHERE id=$1`, owner.EnvironmentID) - if _, err = deploymentExecution(t, w).SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until, timeout); !errors.Is(err, deployment.ErrAllocationConflict) { + if _, err = deploymentExecution(t, w).SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until); !errors.Is(err, deployment.ErrAllocationConflict) { t.Fatal("unfinished initialization admitted suspension", err) } runtimeSuspensionSQL(t, pool, `UPDATE environments SET initialization='complete' WHERE id=$1`, owner.EnvironmentID) runtimeSuspensionSQL(t, pool, `UPDATE runtime_allocations SET compute_wake_requested=true WHERE id=$1`, owner.ID) - if _, err = deploymentExecution(t, w).SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until, timeout); !errors.Is(err, deployment.ErrAllocationConflict) { + if _, err = deploymentExecution(t, w).SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until); !errors.Is(err, deployment.ErrAllocationConflict) { t.Fatal("wake lost to suspension", err) } runtimeSuspensionSQL(t, pool, `UPDATE runtime_allocations SET compute_wake_requested=false WHERE id=$1`, owner.ID) - if _, err = deploymentExecution(t, w).SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until, timeout); err != nil { + if _, err = deploymentExecution(t, w).SetCompute(t.Context(), owner, "quiescing", json.RawMessage(`{}`), &until); err != nil { t.Fatal("idle Session with no Turn was blocked", err) } } diff --git a/services/core/tests/integration/session_execution_configuration_test.go b/services/core/tests/integration/session_execution_configuration_test.go index 3a047f2f8..c8fd9d99c 100644 --- a/services/core/tests/integration/session_execution_configuration_test.go +++ b/services/core/tests/integration/session_execution_configuration_test.go @@ -261,11 +261,12 @@ func TestSessionExecutionConfigurationSurvivesSuspendResume(t *testing.T) { if err != nil { t.Fatal(err) } - owner, err = deploymentExecution(t, w).SetCompute(t.Context(), owner, "running", json.RawMessage(`{"instance":"original"}`), nil, 0) + owner, err = deploymentExecution(t, w).SetCompute(t.Context(), owner, "running", json.RawMessage(`{"instance":"original"}`), nil) if err != nil { t.Fatal(err) } runtimeSuspensionCompleted(t, pool, owner) + runtimeSuspensionSQL(t, pool, `UPDATE runtime_allocations SET compute_activity_at=clock_timestamp()-interval '10 minutes',compute_phase_changed_at=clock_timestamp()-interval '10 minutes' WHERE id=$1`, owner.ID) until := time.Now().Add(time.Hour) for _, phase := range []string{"quiescing", "suspending", "suspended", "restoring", "waking", "running"} { retained := &until