From d61245e562b1f6afafbafbf5c44c661b16f32eb4 Mon Sep 17 00:00:00 2001 From: SaladDay <1203511142@qq.com> Date: Sun, 11 Oct 2026 08:10:24 +0000 Subject: [PATCH 1/3] Recover confirmed lost compute --- contracts/agents-api/index.md | 2 +- contracts/agents-api/zh/index.md | 4 +- docs/getting-started/nodes.md | 2 +- docs/sandbox-provider.md | 10 +- docs/zh/getting-started/nodes.md | 4 +- docs/zh/sandbox-provider.md | 12 +- .../core/internal/deployment/allocations.go | 38 +- .../internal/deployment/allocations_test.go | 59 +++ .../internal/execution/runtime_compute.go | 53 +++ .../execution/runtime_lost_compute_test.go | 399 ++++++++++++++++++ .../core/internal/sandbox/sandbox_provider.go | 10 +- 11 files changed, 575 insertions(+), 18 deletions(-) create mode 100644 services/core/internal/execution/runtime_lost_compute_test.go diff --git a/contracts/agents-api/index.md b/contracts/agents-api/index.md index bf37d7539..01c6f77c5 100644 --- a/contracts/agents-api/index.md +++ b/contracts/agents-api/index.md @@ -128,7 +128,7 @@ Each item is Core's deliberate or native behavior where the official service beh - Hosted workspace admission is limited by the [tenant capacity policy](../../docs/configuration.md#tenant-capacity) and the existing [input reservation deadline](./environments.md#reservations). - Runtimes do not enforce `disabled` or `restricted` networks, so Sessions that need them are rejected ([restricted network policy](./environments.md#restricted-network)). - `packages.system` is rejected; system packages must be preinstalled. -- Cold continuation does not restore process memory, background processes or temporary system-disk changes; completed setup is not replayed. See [Environment persistence](./environments.md#runtime-capability-preparation). Cross-node memory restoration and takeover of an unconfirmed old writer are not qualified. +- Cold continuation does not restore process memory, background processes or temporary system-disk changes; completed setup is not replayed. See [Environment persistence](./environments.md#runtime-capability-preparation). Cross-node memory restoration and takeover of an unconfirmed old writer are not qualified. After a [confirmed compute loss](../../docs/sandbox-provider.md#confirmed-compute-loss), crash consistency of native history is not qualified for any Harness. **Files and Environment files** diff --git a/contracts/agents-api/zh/index.md b/contracts/agents-api/zh/index.md index c8206cad9..0f35d9a04 100644 --- a/contracts/agents-api/zh/index.md +++ b/contracts/agents-api/zh/index.md @@ -1,7 +1,7 @@ --- title: "Agents API 覆盖台账" source: contracts/agents-api/index.md -source_hash: ddc93473d2f9acf6623072b3245b1dc244109f8fc597ef097e25d7ddf07cf392 +source_hash: d7fff07ff5af2349fe56e951e0ed6e215e9d8ba5a191a251730b8fb1e4b00251 --- Core 旨在以下方固定版本为准支持完整的 OpenAI Agents API（[public API rule](https://github.com/MiniMax-AI/OpenAgentCore/blob/main/AGENTS.md#public-api)）。本台账记录 Core 对各项资源实现了哪些内容、哪些契约保存其详细信息，并列出相对于 OpenAI 服务的所有已知差异和所有未解决缺口。[API namespaces and credentials](../../../docs/zh/api/index.md) 说明谁调用哪些 API；[Agents API guide](../../../docs/zh/api/public-agent-api.md) 介绍使用方法。 @@ -131,7 +131,7 @@ Core 自身字段位于 `x_agents_core` 中（[Core extensions](../../../docs/zh - Hosted 工作区准入受[租户容量策略](../../../docs/zh/configuration.md#tenant-capacity)及既有[输入预留截止时间](environments.md#reservations)限制。 - Runtime 不会实施 `disabled` 或 `restricted` 网络，因此需要这些网络的 Session 会被拒绝（[restricted network policy](environments.md#restricted-network)）。 - `packages.system` 会被拒绝；系统软件包必须预先安装。 -- 冷续接不恢复进程内存、后台进程或临时系统盘修改；已完成的 setup 不会重放。参见 [Environment 持久化](environments.md#runtime-capability-preparation)。跨节点内存恢复及接管未确认停止的旧写入者尚未通过资格验证。 +- 冷续接不恢复进程内存、后台进程或临时系统盘修改；已完成的 setup 不会重放。参见 [Environment 持久化](environments.md#runtime-capability-preparation)。跨节点内存恢复及接管未确认停止的旧写入者尚未通过资格验证。在[已确认的计算资源丢失](../../../docs/zh/sandbox-provider.md#confirmed-compute-loss)后，任何 Harness 的原生历史崩溃一致性均未通过资格验证。 **Files 和 Environment files** diff --git a/docs/getting-started/nodes.md b/docs/getting-started/nodes.md index aa02e573c..e72c4f3fb 100644 --- a/docs/getting-started/nodes.md +++ b/docs/getting-started/nodes.md @@ -153,7 +153,7 @@ The node connects out to Core; Core needs no SSH or Docker TCP access to the hos ## When a node host fails -A restarted node service keeps its identity and finds its existing sandboxes again. Core does not take over an active or unconfirmed writer merely because its node is unreachable or its sandbox is missing: those resources show as **Node disconnected** or **Sandbox resource missing** while ownership remains unresolved. A settled retained checkpoint can follow the [checkpoint transfer contract](../sandbox-provider.md#checkpoint-transfer), and confirmed release can permit [cold replacement](../sandbox-provider.md#cold-replacement-after-checkpoint-retention). A lost node state directory is a recovery incident: restore it from its [backup](./operations.md#back-up) together with the database and the provider storage, rather than registering the host again over existing resources. +A restarted node service keeps its identity and finds its existing sandboxes again. Core does not take over an active or unconfirmed writer merely because its node is unreachable or its sandbox is missing: those resources show as **Node disconnected** or **Sandbox resource missing** while ownership remains unresolved. Restart a crashed or hung host rather than removing its node: after the node reconnects, a running sandbox that the node confirms gone is released under the [confirmed compute loss](../sandbox-provider.md#confirmed-compute-loss) rules, and the Session's next input continues on new compute; any other missing sandbox keeps **Sandbox resource missing**. A host that never returns keeps its Sessions' compute ownership. A settled retained checkpoint can follow the [checkpoint transfer contract](../sandbox-provider.md#checkpoint-transfer), and confirmed release can permit [cold replacement](../sandbox-provider.md#cold-replacement-after-checkpoint-retention). A lost node state directory is a recovery incident: restore it from its [backup](./operations.md#back-up) together with the database and the provider storage, rather than registering the host again over existing resources. ## Troubleshooting diff --git a/docs/sandbox-provider.md b/docs/sandbox-provider.md index f70b826dc..1225ddcc5 100644 --- a/docs/sandbox-provider.md +++ b/docs/sandbox-provider.md @@ -89,7 +89,7 @@ Every call receives a bounded context. Expiry or cancellation ends the caller's `ErrInvalid`, `ErrOwnership`, `ErrExists`, `ErrNotFound`, `ErrComputeUnconfirmed` and `ErrCommandUnconfirmed` keep their defined meanings. An unclassified native or transport error is unknown, never permission to retry a mutation. Core never reads provider diagnostics as lifecycle truth or exposes native error text or credentials; the node transport maps errors to fixed codes, and direct SDK details stay private. -Checkpoint support adds `Compute` generation, name and ID and `SnapshotIdentity`; persist operation IDs and the provider's snapshot provenance unchanged. `ObserveOnly` on suspend or resume observes the previous attempt and never starts another capture or restore. `ResumeCompute` only thaws the retained source and never cold-starts a stopped one. Cleanup targets the exact compute incarnation and snapshot, not whatever instance now has the same name. Read [`runtime_compute.go`](https://github.com/MiniMax-AI/OpenAgentCore/blob/main/services/core/internal/execution/runtime_compute.go) and its failure tests before declaring checkpoint support. +Checkpoint support adds `Compute` generation, name and ID and `SnapshotIdentity`; persist operation IDs and the provider's snapshot provenance unchanged. `ObserveOnly` on suspend or resume observes the previous attempt and never starts another capture or restore. `ResumeCompute` only thaws the retained source and never cold-starts a stopped one. Cleanup targets the exact compute incarnation and snapshot, not whatever instance now has the same name. `GetCompute` returns `ErrNotFound` only when the owning node has verified that the exact incarnation no longer exists; an instance with the same name and another native ID is `ErrOwnership`, and an unreachable node or backend is an unknown result. A `ComputeState.Status` of `stopped` or `crashed` means the adapter verified that the exact incarnation’s process no longer exists; its record or disks can remain until `KillCompute` removes them. `paused`, `running` without a Runtime connection, and unknown states do not establish termination. Read [`runtime_compute.go`](https://github.com/MiniMax-AI/OpenAgentCore/blob/main/services/core/internal/execution/runtime_compute.go) and its failure tests before declaring checkpoint support. ### Four distinct readiness facts @@ -164,7 +164,7 @@ The allocation, its dedicated daemon credential digest and the exact Session bin With a configured provider, the Worker scans committed pending hosted Environments that have no allocation, which covers idle Session creation and recovery after an interruption between commit and bootstrap; an existing allocation never re-enters that path. The scan is bounded and serialized by the lifecycle owner and needs no caller action. An initial reservation without a Turn leaves its Session idle, and a daemon connection is never treated as native readiness. The same scan publishes authenticated connection observations with durable generations, after verifying the exact Session and device binding and a settled bootstrap. -Between Turns, Core checks that connected, observed compute is still its Session's running allocation; the check changes nothing and never revives a cleanup request. Running compute never expires: explicit lifecycle cleanup and snapshot retention govern reclamation. A stopped or missing container never authorizes discarding retained workspace or history or selecting replacement compute. Disabling the provider stops new hosted admission and bootstrap but never blocks cancellation, function results or input retry outcomes of existing Sessions. +Between Turns, Core checks that connected, observed compute is still its Session's running allocation; the check changes nothing and never revives a cleanup request. Running compute never expires: explicit lifecycle cleanup, snapshot retention and [confirmed compute loss](#confirmed-compute-loss) govern reclamation. An unreachable container, or a stopped or missing one without that confirmation, never authorizes discarding retained workspace or history or selecting replacement compute. Disabling the provider stops new hosted admission and bootstrap but never blocks cancellation, function results or input retry outcomes of existing Sessions. Terminal cleanup atomically revokes the device's authority, records the Environment's failure or expiry, settles pending input and requests cancellation, and only then calls `Kill`; original input deadlines and retry outcomes are kept. Temporary provider outages, unknown Create results and stopped compute never prove a permanent failure. After public Session deletion Core keeps the allocation and marks it released only after owned compute and volume cleanup and proof that the original Create settled; an unknown creation keeps cleanup ownership even after an absence observation, and bounded scans continue to catch late resources without another `Create`. @@ -212,6 +212,12 @@ Core confirms the original creation is settled and all owned compute and snapsho New input or live Environment file access reserves a new allocation on a compatible node and bootstraps a fresh Runtime device. Preparation reuses the same filesystem and completed initialization rather than replaying initialization. Execution resumes the same native Session from retained history after validating the replacement Runtime's declaration. Missing or invalid native history fails explicitly; Core never substitutes a new native Session or replays completed input. History and published Artifact reads do not reserve compute. This path does not preserve VM memory, background processes or open connections, and it does not qualify an undeclared Runtime, Harness or filesystem combination. +### Confirmed compute loss + +Running compute has no retention deadline, so a host crash or reboot would otherwise leave its allocation owned indefinitely. After the owning node is online again, `GetCompute` returning `ErrNotFound` or a verified `stopped` or `crashed` status for the exact persisted incarnation releases a checkpoint-capable allocation in compute phase `running` only when all of these hold: the original creation is settled, the incarnation has a native ID, no capture, restore or rollback intent is recorded, no authenticated Runtime connection exists for the allocation's device, and the Session meets the conditions for [cold replacement](#cold-replacement-after-checkpoint-retention). Core then calls `KillCompute` for that exact incarnation. Only its confirmed result permits one Session-locked commit that rechecks the compute revision and those conditions, revokes the device and releases the allocation and placement. The Session, Environment, pending input, native Session identity and filesystem remain, and new input or live file access continues as in cold replacement. + +Every other observation keeps ownership and its diagnostic: a `paused`, `running` or unknown native status, an identity mismatch, a timeout or transport error, an unknown creation, a recorded restore, a live Runtime connection, owned workspace storage, incomplete initialization, a terminal Session or a missing `retained_native_history` declaration. A failed `KillCompute` changes nothing, and a concurrent archive, deletion or compute transition prevails. Core never acts while the node is offline, because it has no authority to stop compute without the node. Continuation retains the filesystem and uses the recorded native Session ID; a Harness resume error fails execution rather than creating another native Session. Core does not replay interrupted or completed input. Completeness of reopened native history relative to acknowledged Turns after a crash is [not qualified](../contracts/agents-api/index.md); identity checks do not detect a truncated but parseable history tail. + ### Reset and archive A [reset](../contracts/agents-api/sandbox-deployment.md#reset) is durable execution state that the runtime manager advances outside its counted work. Start, escalation, cancellation, setup, update and finalization serialize through the mutation gate. Idleness is rechecked with the Session lock and then the deployment lock, never in the reverse order during finalization. Work is keyset-paged and bound to the reset's request time and generation, with the absolute deadline and validated audit provenance persisted. diff --git a/docs/zh/getting-started/nodes.md b/docs/zh/getting-started/nodes.md index 9e7d24e05..0f11f49ab 100644 --- a/docs/zh/getting-started/nodes.md +++ b/docs/zh/getting-started/nodes.md @@ -1,7 +1,7 @@ --- title: "添加和管理节点" source: docs/getting-started/nodes.md -source_hash: 0c62518cc740387ed03035ea4870108edeaaa82331a7509b573fb2b775ae3aba +source_hash: dc02bedb776a48e79850afbefa7ec401a93912186ec294afb8d52d5c6d50d9cb --- 节点是一台 Linux 主机，在沙箱后端为 Docker 或 microsandbox 时，为 Core 托管 Session 运行沙箱。Core 将新 Session 分配给有空余容量的节点；节点创建沙箱，沙箱回连 Core。E2B 不需要节点。应用为自己的 Session 连接的机器是[自托管执行器](self-hosted.md)，而不是节点。 @@ -155,7 +155,7 @@ root 只准备账号、组和服务单元；其他操作（包括 Docker 网络 ## 节点主机故障时 {#when-a-node-host-fails} -重启节点服务会保留身份并重新发现已有沙箱。Core 不会仅因节点不可达或沙箱缺失就接管活动或未确认的 writer：所有权未结清时，这些资源显示 **Node disconnected** 或 **Sandbox resource missing**。已结清的保留检查点可以遵循[检查点转移契约](../sandbox-provider.md#checkpoint-transfer)，确认释放后可以按条件进行[冷替换](../sandbox-provider.md#cold-replacement-after-checkpoint-retention)。丢失节点状态目录属于恢复事件：从[备份](operations.md#back-up)恢复，并同时恢复数据库和提供商存储；不要在已有资源上重新注册主机。 +重启节点服务会保留身份并重新发现已有沙箱。Core 不会仅因节点不可达或沙箱缺失就接管活动或未确认的 writer：所有权未结清时，这些资源显示 **Node disconnected** 或 **Sandbox resource missing**。主机崩溃或挂起时，应重启主机而不是移除其节点：节点重新连接后，节点确认已不存在的运行中沙箱会按[已确认的计算资源丢失](../sandbox-provider.md#confirmed-compute-loss)规则释放，Session 的下一个 input 在新计算资源上续接；其他缺失的沙箱保持 **Sandbox resource missing**。主机始终未恢复时，其 Session 的计算资源归属保持不变。已结清的保留检查点可以遵循[检查点转移契约](../sandbox-provider.md#checkpoint-transfer)，确认释放后可以按条件进行[冷替换](../sandbox-provider.md#cold-replacement-after-checkpoint-retention)。丢失节点状态目录属于恢复事件：从[备份](operations.md#back-up)恢复，并同时恢复数据库和提供商存储；不要在已有资源上重新注册主机。 ## 问题排查 {#troubleshooting} diff --git a/docs/zh/sandbox-provider.md b/docs/zh/sandbox-provider.md index ff6501786..7deb78353 100644 --- a/docs/zh/sandbox-provider.md +++ b/docs/zh/sandbox-provider.md @@ -1,7 +1,7 @@ --- title: "添加 Sandbox Provider" source: docs/sandbox-provider.md -source_hash: 63ad2e39e7536caee9bd43f9b643073d32b32147f1440d188d6d38752628adc7 +source_hash: edaa63076fb1629202fc159bb084124eaf0efef56f094d03a99538e281e74e65 --- **Sandbox Provider** 为 Core 管理的 Environment 提供 Runtime daemon 运行所需的外层计算资源，以及启动 daemon 的有界引导流程。本指南说明如何添加 Provider，并作为 Core 驱动 Provider 的参考。接口为 [`SandboxProvider`](https://github.com/MiniMax-AI/OpenAgentCore/blob/main/services/core/internal/sandbox/sandbox_provider.go)。 @@ -91,7 +91,7 @@ Microsandbox 在返回创建不存在回执前永久关闭初始 Create 准入 `ErrInvalid`、`ErrOwnership`、`ErrExists`、`ErrNotFound`、`ErrComputeUnconfirmed` 和 `ErrCommandUnconfirmed` 保持其定义含义。未分类原生或 transport error 表示未知，不授权重试 mutation。Core 不将 provider diagnostics 读作生命周期事实，也不暴露原生错误文本或凭据；node transport 将错误映射为固定 code，直接 SDK 细节保持私有。 -Checkpoint 支持增加 `Compute` generation、name、ID 和 `SnapshotIdentity`；原样持久化 operation ID 与 provider snapshot provenance。suspend 或 resume 的 `ObserveOnly` 仅观察上次尝试，不启动另一 capture 或 restore。`ResumeCompute` 仅解冻保留源，不冷启动已停止源。清理针对精确 compute incarnation 和 snapshot，不针对当前同名实例。声明 checkpoint 支持前阅读 [`runtime_compute.go`](https://github.com/MiniMax-AI/OpenAgentCore/blob/main/services/core/internal/execution/runtime_compute.go) 及其失败测试。 +Checkpoint 支持增加 `Compute` generation、name、ID 和 `SnapshotIdentity`；原样持久化 operation ID 与 provider snapshot provenance。suspend 或 resume 的 `ObserveOnly` 仅观察上次尝试，不启动另一 capture 或 restore。`ResumeCompute` 仅解冻保留源，不冷启动已停止源。清理针对精确 compute incarnation 和 snapshot，不针对当前同名实例。只有所属节点已验证精确 incarnation 不再存在时，`GetCompute` 才返回 `ErrNotFound`；同名但 native ID 不同的实例为 `ErrOwnership`，节点或后端不可达属于未知结果。`ComputeState.Status` 为 `stopped` 或 `crashed` 表示 adapter 已验证该精确 incarnation 的进程不再存在；其记录或磁盘可保留到 `KillCompute` 将其移除。`paused`、没有 Runtime 连接的 `running` 和未知状态均不证明进程已终止。声明 checkpoint 支持前阅读 [`runtime_compute.go`](https://github.com/MiniMax-AI/OpenAgentCore/blob/main/services/core/internal/execution/runtime_compute.go) 及其失败测试。 ### 四个独立就绪事实 {#four-distinct-readiness-facts} @@ -166,7 +166,7 @@ allocation、专用 daemon credential digest 和精确 Session binding 在 `Crea 配置 provider 后，Worker 扫描已提交且没有 allocation 的 pending hosted Environment，涵盖空闲 Session 创建以及 commit 与 bootstrap 之间中断后的恢复；已有 allocation 不重新进入此路径。scan 有界，由 lifecycle owner 串行化，不需要调用方操作。没有 Turn 的初始预约让 Session 保持空闲，daemon 连接不被当作原生 readiness。同一 scan 在验证精确 Session、device binding 和已结算 bootstrap 后，发布带持久 generation 的认证连接观测。 -Core 在 Turn 之间检查已连接且已观察的计算资源仍是其 Session 正在运行的 allocation；该检查不做任何修改，也不复活 cleanup 请求。正在运行的计算资源不会到期：显式生命周期清理和快照保留策略管理资源回收。停止或缺失 container 不授权丢弃保留工作区或历史，也不授权选择替代计算资源。禁用 provider 停止新 hosted admission 与 bootstrap，但不阻止现有 Session 的取消、function result 或 input retry outcome。 +Core 在 Turn 之间检查已连接且已观察的计算资源仍是其 Session 正在运行的 allocation；该检查不做任何修改，也不复活 cleanup 请求。正在运行的计算资源不会到期：显式生命周期清理、快照保留策略和[已确认的计算资源丢失](#confirmed-compute-loss)管理资源回收。不可达的 container，以及未经该确认的停止或缺失 container，不授权丢弃保留工作区或历史，也不授权选择替代计算资源。禁用 provider 停止新 hosted admission 与 bootstrap，但不阻止现有 Session 的取消、function result 或 input retry outcome。 终结清理原子撤销 device authority、记录 Environment 失败或到期、结算 pending input 并请求取消，然后才调用 `Kill`；原 input deadline 与 retry outcome 保留。临时 provider outage、未知 Create result 和停止的计算资源不证明永久失败。公开 Session 删除后 Core 保留 allocation，仅在所属 compute 与 volume 清理完成且原 Create 已结算的证明成立后标记 released；未知创建即使观察到不存在也保留 cleanup ownership，有界 scan 继续捕捉延迟资源，不再调用 `Create`。 @@ -214,6 +214,12 @@ Core 在释放 allocation 和 placement 归属前，确认原始创建已结算 新 input 或实时 Environment 文件访问会在兼容 node 上预约新 allocation，并引导新的 Runtime device。准备流程复用同一文件系统和已完成的初始化，不重放初始化。验证替代 Runtime 的声明后，执行从保留历史续接同一原生 Session。原生历史缺失或无效时明确失败；Core 不替换为新的原生 Session，也不重放已完成的 input。读取历史和已发布 Artifact 不预约计算资源。此路径不保留 VM 内存、后台进程或开放连接，也不为未声明能力的 Runtime、Harness 或文件系统组合提供资格。 +### 已确认的计算资源丢失 {#confirmed-compute-loss} + +正在运行的计算资源没有保留期限，因此主机崩溃或重启后，其 allocation 原本会无限期保持归属。所属节点重新上线后，对于计算阶段为 `running` 且支持检查点的 allocation，只有同时满足以下条件时，`GetCompute` 对已持久化的精确 incarnation 返回 `ErrNotFound` 或已验证的 `stopped`、`crashed` 状态才会释放该 allocation：原始创建已结算，该 incarnation 有 native ID，未记录 capture、restore 或 rollback 意图，allocation 的 device 没有已认证的 Runtime 连接，且 Session 满足[冷替换](#cold-replacement-after-checkpoint-retention)条件。随后 Core 对该精确 incarnation 调用 `KillCompute`。只有其结果得到确认，才允许一次持有 Session 锁的提交：重新检查计算 revision 与上述条件，撤销 device，并释放 allocation 和 placement。Session、Environment、等待中的 input、原生 Session 身份和文件系统均保留，新 input 或实时文件访问按冷替换续接。 + +其他观察结果均保留归属及其诊断：`paused`、`running` 或未知原生状态、身份不匹配、超时或传输错误、未知创建、已记录的 restore、存在 Runtime 实时连接、自带工作区存储、初始化未完成、Session 已终结，或缺少 `retained_native_history` 声明。`KillCompute` 失败时不做任何修改，并发的归档、删除或计算阶段变更优先。节点离线时 Core 从不执行此操作，因为没有节点就无权停止计算资源。续接保留文件系统并使用已记录的原生 Session ID；Harness 恢复出错时执行失败，不创建另一个原生 Session。Core 不重放已中断或已完成的输入。崩溃后重新打开的原生历史相对于已确认 Turn 的完整性[尚未通过资格验证](../../contracts/agents-api/zh/index.md)；身份检查无法检测已截断但仍可解析的历史尾部。 + ### 重置与归档 {#reset-and-archive} [reset](../../contracts/agents-api/zh/sandbox-deployment.md#reset) 是 runtime manager 在计数工作之外推进的持久 execution state。start、escalation、cancellation、setup、update 和 finalization 通过 mutation gate 串行化。空闲状态先在 Session lock 再在 deployment lock 下复查，finalization 时不反向加锁。工作使用 keyset paging，绑定 reset request time 和 generation，并持久化 absolute deadline 与已验证 audit provenance。 diff --git a/services/core/internal/deployment/allocations.go b/services/core/internal/deployment/allocations.go index 24560832a..002c37914 100644 --- a/services/core/internal/deployment/allocations.go +++ b/services/core/internal/deployment/allocations.go @@ -149,27 +149,50 @@ func (e *ExecutionOperations) ReleaseAllocation(ctx context.Context, owner Alloc // reason unless its compute expired. Cancellation requests do not prove that // native work has stopped. func (e *ExecutionOperations) RequestCleanup(ctx context.Context, owner Allocation) (Allocation, error) { - return e.cleanup(ctx, owner, false) + return e.cleanup(ctx, owner, cleanupRequested) } // ReleaseAbsentCreation consumes provider proof that the original Create is // settled and owns no resources: it requests cleanup and releases the // allocation in one commit. func (e *ExecutionOperations) ReleaseAbsentCreation(ctx context.Context, owner Allocation) (Allocation, error) { - return e.cleanup(ctx, owner, true) + return e.cleanup(ctx, owner, creationAbsent) } -func (e *ExecutionOperations) cleanup(ctx context.Context, owner Allocation, absent bool) (Allocation, error) { +// ReleaseLostCompute consumes the owning node's confirmed cleanup of the +// owner's settled running incarnation after it observed that incarnation +// absent, stopped or crashed. It revokes the device and releases the allocation in one commit +// only while the owner's compute revision is current and its Session can +// continue on new compute; otherwise it changes nothing. The Session, its +// pending input, native identity and filesystem stay. +func (e *ExecutionOperations) ReleaseLostCompute(ctx context.Context, owner Allocation) (Allocation, error) { + return e.cleanup(ctx, owner, computeLost) +} + +// cleanupEvidence is the provider proof a cleanup commit consumes. +type cleanupEvidence int + +const ( + cleanupRequested cleanupEvidence = iota + creationAbsent + computeLost +) + +func (e *ExecutionOperations) cleanup(ctx context.Context, owner Allocation, evidence cleanupEvidence) (Allocation, error) { key, err := owner.Key().parse() if err != nil { return Allocation{}, err } + lost := evidence == computeLost var result Allocation err = e.storage.WithAllocationCleanup(ctx, key, func(tx AllocationCleanupTx) error { - current, err := checkAllocation(tx, owner, false) + current, err := checkAllocation(tx, owner, lost) if err != nil { return err } + if lost && (current.State != "running" || current.ComputePhase != "running" || current.ComputeRevision != owner.ComputeRevision || !current.CreateSettled) { + return ErrAllocationConflict + } if current.State == "released" { result = current return nil @@ -187,7 +210,7 @@ func (e *ExecutionOperations) cleanup(ctx context.Context, owner Allocation, abs err = sessions.CancelWork(ctx, tx) } else { retain := false - if revoked.Expired { + if revoked.Expired || lost { retain, err = tx.CanRetainEnvironment(revoked) if err != nil { return err @@ -196,6 +219,9 @@ func (e *ExecutionOperations) cleanup(ctx context.Context, owner Allocation, abs // Compute retention ends independently of the Session only when its // initialized filesystem and native history can continue on new compute. // Explicit archive/reset makes this locked eligibility check false. + if !retain && lost { + return ErrAllocationConflict + } if !retain { err = sessions.TerminateEnvironment(ctx, tx, revoked.Expired, sessions.ProvisioningFailureReason, nil) } @@ -204,7 +230,7 @@ func (e *ExecutionOperations) cleanup(ctx context.Context, owner Allocation, abs return err } pending, err := tx.RequestCleanup(current) - if err != nil || !absent { + if err != nil || evidence == cleanupRequested { result = pending return err } diff --git a/services/core/internal/deployment/allocations_test.go b/services/core/internal/deployment/allocations_test.go index 7bad4ffa3..9df8051ac 100644 --- a/services/core/internal/deployment/allocations_test.go +++ b/services/core/internal/deployment/allocations_test.go @@ -596,6 +596,65 @@ func TestExpiredQualifiedFilesystemCleanupPreservesSessionWork(t *testing.T) { } } +func TestReleaseLostComputeRetainsOrChangesNothing(t *testing.T) { + owner := Allocation{ID: uuid.NewString(), DeviceID: uuid.NewString(), EnvironmentID: uuid.NewString(), TenantID: uuid.NewString(), ProviderKey: uuid.NewString(), State: "running", ComputePhase: "running", ComputeRevision: 7, CreateSettled: true} + for _, test := range []struct { + name string + stored func(*Allocation) + retain bool + want []string + err error + }{ + {"retained", func(*Allocation) {}, true, []string{"LoadAllocation", "LoadSessionDevice", "RevokeDevice", "LoadAllocation", "CanRetainEnvironment", "RequestCleanup", "SettleCreation", "Release"}, nil}, + {"not retainable", func(*Allocation) {}, false, []string{"LoadAllocation", "LoadSessionDevice", "RevokeDevice", "LoadAllocation", "CanRetainEnvironment"}, ErrAllocationConflict}, + {"newer revision", func(a *Allocation) { a.ComputeRevision++ }, true, []string{"LoadAllocation", "LoadSessionDevice"}, ErrAllocationConflict}, + {"quiescing", func(a *Allocation) { a.ComputePhase = "quiescing" }, true, []string{"LoadAllocation", "LoadSessionDevice"}, ErrAllocationConflict}, + {"unknown creation", func(a *Allocation) { a.CreateSettled = false }, true, []string{"LoadAllocation", "LoadSessionDevice"}, ErrAllocationConflict}, + {"released", func(a *Allocation) { a.State = "released" }, true, []string{"LoadAllocation", "LoadSessionDevice"}, ErrAllocationConflict}, + } { + t.Run(test.name, func(t *testing.T) { + var calls []string + stored := owner + test.stored(&stored) + change := func(name string, apply func(*Allocation)) func(Allocation) (Allocation, error) { + return func(current Allocation) (Allocation, error) { + calls = append(calls, name) + apply(¤t) + return current, nil + } + } + tx := &fakeCleanupTx{ + fakeAllocationTx: &fakeAllocationTx{t: t, + loadAllocation: func() (Allocation, error) { calls = append(calls, "LoadAllocation"); return stored, nil }, + loadSessionDevice: func() (SessionDevice, bool, error) { + calls = append(calls, "LoadSessionDevice") + return SessionDevice{ID: owner.DeviceID, EnvironmentID: owner.EnvironmentID}, true, nil + }, + settleCreation: change("SettleCreation", func(*Allocation) {}), + release: change("Release", func(a *Allocation) { a.State = "released" }), + }, + canRetainEnvironment: func(Allocation) (bool, error) { + calls = append(calls, "CanRetainEnvironment") + return test.retain, nil + }, + revokeDevice: func(Allocation) error { calls = append(calls, "RevokeDevice"); return nil }, + requestCleanup: change("RequestCleanup", func(a *Allocation) { a.State = "cleanup_pending" }), + } + storage := &fakeExecutionStorage{t: t, withAllocationCleanup: func(_ context.Context, _ AllocationKey, apply func(AllocationCleanupTx) error) error { + return apply(tx) + }} + operations, err := NewExecutionOperations(newService(t, &fakeStorage{t: t}, &fakeReader{t: t}, testPublicURL), storage, engine.Catalog{}) + if err != nil { + t.Fatal(err) + } + result, err := operations.ReleaseLostCompute(t.Context(), owner) + if !errors.Is(err, test.err) || !slices.Equal(calls, test.want) || (test.err == nil && result.State != "released") { + t.Fatal("lost compute release", result.State, calls, err) + } + }) + } +} + func (f *fakeReservationTx) LoadGenerationSpecification(uint64) (GenerationSpecification, error) { unexpected(f.t, "LoadGenerationSpecification") return GenerationSpecification{}, nil diff --git a/services/core/internal/execution/runtime_compute.go b/services/core/internal/execution/runtime_compute.go index 80cef237a..bac1e7e48 100644 --- a/services/core/internal/execution/runtime_compute.go +++ b/services/core/internal/execution/runtime_compute.go @@ -10,7 +10,9 @@ import ( "github.com/MiniMax-AI/OpenAgentCore/internal/obs/log" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/deployment" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/providercontract" + "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/runtimegateway" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox" + "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sessions" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/workspacefs" "github.com/google/uuid" ) @@ -123,9 +125,18 @@ func (r *runtimeLifecycle) observeCompute(ctx context.Context, owner deployment. func (r *runtimeLifecycle) idleCompute(ctx context.Context, p sandbox.SandboxProvider, owner deployment.Allocation, state runtimeCompute) (err error) { defer func() { err = withObservationOwner(owner, err) }() compute, err := p.GetCompute(ctx, runtimeReference(owner), state.Current) + if errors.Is(err, sandbox.ErrNotFound) { + return r.releaseLostCompute(ctx, p, owner, state, err) + } if err != nil { return err } + if compute.Terminated() { + if compute.Compute.ID != state.Current.ID || compute.Compute.Name != state.Current.Name || compute.Compute.Generation != state.Current.Generation { + return sandbox.ErrOwnership + } + return r.releaseLostCompute(ctx, p, owner, state, sandbox.ErrComputeUnconfirmed) + } if compute.Status != "running" || !compute.BootstrapComplete { return sandbox.ErrComputeUnconfirmed } @@ -192,6 +203,48 @@ func (r *runtimeLifecycle) idleCompute(ctx context.Context, p sandbox.SandboxPro return r.captureCompute(ctx, p, suspending, state, false) } +// releaseLostCompute settles a settled running incarnation that its owning +// node reports absent, stopped or crashed. Only exact cleanup of that +// incarnation, with no live Runtime, no other compute intent and a Session +// that can continue on new compute, releases ownership; every other case +// returns the observation's error. +func (r *runtimeLifecycle) releaseLostCompute(ctx context.Context, p sandbox.SandboxProvider, owner deployment.Allocation, state runtimeCompute, observed error) error { + if !owner.CreateSettled || owner.State != "running" || state.Current.ID == "" || state.Target != nil || state.Snapshot != nil || state.SuspendID != "" || state.RestoreID != "" || state.Rollback { + return observed + } + if _, err := authorizedRuntimePeer(ctx, r.sessions, r.registry, owner.DeviceID); err == nil { + return sandbox.ErrComputeUnconfirmed + } else if !errors.Is(err, sessions.ErrNotFound) && !errors.Is(err, runtimegateway.ErrDeviceNotRegistered) && !errors.Is(err, runtimegateway.ErrSessionClosed) { + return err + } + current, err := r.reader.EnvironmentAllocation(ctx, owner.Key()) + if err != nil { + return err + } + if current.ID != owner.ID || current.State != owner.State || current.ComputePhase != owner.ComputePhase || current.ComputeRevision != owner.ComputeRevision { + return deployment.ErrAllocationConflict + } + retained, err := r.reader.RetainedNativeHistory(ctx, owner.Key()) + if err != nil { + return err + } + if !retained { + return observed + } + if err := r.lease.CheckOwnership(ctx); err != nil { + return err + } + if err := ignoreComputeAbsent(p.KillCompute(ctx, runtimeReference(owner), state.Current)); err != nil { + return err + } + released, err := r.deployment.ReleaseLostCompute(ctx, owner) + if err != nil { + return err + } + r.clearRuntimeState(released) + return nil +} + func (r *runtimeLifecycle) captureCompute(ctx context.Context, p sandbox.SandboxProvider, owner deployment.Allocation, state runtimeCompute, observeOnly bool) (err error) { defer func() { err = withObservationOwner(owner, err) }() started := time.Now() diff --git a/services/core/internal/execution/runtime_lost_compute_test.go b/services/core/internal/execution/runtime_lost_compute_test.go new file mode 100644 index 000000000..365c47e27 --- /dev/null +++ b/services/core/internal/execution/runtime_lost_compute_test.go @@ -0,0 +1,399 @@ +package execution + +import ( + "context" + "encoding/json" + "errors" + "net/http" + "net/http/httptest" + "net/url" + "slices" + "strings" + "testing" + "time" + + "github.com/MiniMax-AI/OpenAgentCore/internal/agentdaemon/proto" + "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/deployment" + "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/deployment/placement" + "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/runtimegateway" + "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox" + "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sessions" + "github.com/gorilla/websocket" +) + +// lostComputeProvider models the owning node's exact-incarnation observation +// and cleanup. These callbacks are deterministic fixtures, not native evidence. +type lostComputeProvider struct { + retentionProvider + observe func(sandbox.Compute) (sandbox.ComputeState, error) + onKill func() + killed []sandbox.Compute +} + +func (p *lostComputeProvider) GetCompute(_ context.Context, _ sandbox.Reference, c sandbox.Compute) (sandbox.ComputeState, error) { + return p.observe(c) +} +func (p *lostComputeProvider) KillCompute(ctx context.Context, r sandbox.Reference, c sandbox.Compute) error { + p.killed = append(p.killed, c) + if p.onKill != nil { + p.onKill() + } + return p.retentionProvider.KillCompute(ctx, r, c) +} + +var lostCompute = sandbox.Compute{Name: "old-compute", ID: "old-compute-id"} + +func observedStatus(status string) func(sandbox.Compute) (sandbox.ComputeState, error) { + return func(c sandbox.Compute) (sandbox.ComputeState, error) { + return sandbox.ComputeState{Compute: c, Status: status, BootstrapComplete: true}, nil + } +} + +func observedError(err error) func(sandbox.Compute) (sandbox.ComputeState, error) { + return func(sandbox.Compute) (sandbox.ComputeState, error) { return sandbox.ComputeState{}, err } +} + +// lostObservations are the owning node's confirmations of loss: typed +// absence, or a verified terminal status of the exact incarnation. +var lostObservations = map[string]func(sandbox.Compute) (sandbox.ComputeState, error){ + "absent": observedError(sandbox.ErrNotFound), + "stopped": observedStatus("stopped"), + "crashed": observedStatus("crashed"), +} + +type lostComputeFixture struct { + workspaceSettlementFixture + provider *lostComputeProvider + owner deployment.Allocation + bootstrap sandbox.Bootstrap + input sessions.EnvironmentInputReservation +} + +// newLostComputeFixture commits a settled running incarnation of a Session +// whose completed initialization and native identity are retained. +func newLostComputeFixture(t *testing.T, external bool, seed ...string) lostComputeFixture { + t.Helper() + provider := &lostComputeProvider{retentionProvider: retentionProvider{bootstraps: make(chan sandbox.Bootstrap, 2)}, observe: func(sandbox.Compute) (sandbox.ComputeState, error) { + return sandbox.ComputeState{}, sandbox.ErrNotFound + }} + f := lostComputeFixture{workspaceSettlementFixture: workspaceSettlementFixtureWithSetup(t, provider, workspaceNodeSetup(t, external)), provider: provider} + r, session := f.lifecycle, f.session + owner, err := r.provision(t.Context(), session.TenantID, session.Environment.ID, r.config.InstallationID) + if err != nil { + t.Fatal(err) + } + f.bootstrap = <-provider.bootstraps + state, _ := json.Marshal(runtimeCompute{Current: lostCompute}) + bind := strings.NewReplacer("$1", "'"+owner.EnvironmentID+"'", "$2", "'"+owner.DeviceID+"'", "$3", "'"+owner.ID+"'", "$4", "'"+string(state)+"'") + exec := func(statements []string) { + for _, statement := range statements { + if _, err := f.pool.Exec(t.Context(), bind.Replace(statement)); err != nil { + t.Fatal(statement, err) + } + } + } + exec([]string{ + `UPDATE environments SET initialization='complete',status='disconnected' WHERE id=$1`, + `UPDATE devices SET supported_agent_kinds='[{"kind":"codex","available":true,"capabilities":{"retained_native_history":true}}]' WHERE id=$2`, + `UPDATE session_devices SET native_session_id='retained-native-session' WHERE device_id=$2`, + `UPDATE runtime_allocations SET compute_phase='running',compute_state=$4,compute_revision=7 WHERE id=$3`, + }) + if f.input, err = f.sessions.ReserveEnvironmentInput(t.Context(), session.TenantID, session.ID, "after-loss", []sessions.Input{{Kind: "message", Payload: json.RawMessage(`{"input":[{"role":"user","content":[{"type":"input_text","text":"continue"}]}]}`)}}); err != nil { + t.Fatal(err) + } + exec(seed) + r.connections = map[string]*runtimeConnection{} + if f.owner, err = r.reader.EnvironmentAllocation(t.Context(), owner.Key()); err != nil { + t.Fatal(err) + } + return f +} + +func (f lostComputeFixture) current(t *testing.T) deployment.Allocation { + t.Helper() + current, err := f.lifecycle.reader.EnvironmentAllocation(t.Context(), f.owner.Key()) + if err != nil { + t.Fatal(err) + } + return current +} + +func (f lostComputeFixture) revoked(t *testing.T, device string) bool { + t.Helper() + var revoked bool + if err := f.pool.QueryRow(t.Context(), `SELECT revoked_at IS NOT NULL FROM devices WHERE id=$1`, device).Scan(&revoked); err != nil { + t.Fatal(err) + } + return revoked +} + +// assertRetained checks that the Session, its pending input, native identity +// and filesystem survived, whatever happened to compute ownership. +func (f lostComputeFixture) assertRetained(t *testing.T, status string, initialization ...string) { + t.Helper() + r, session := f.lifecycle, f.session + initialization = append(initialization, "complete") + environment, err := r.sessions.GetEnvironment(t.Context(), session.TenantID, session.Environment.ID) + if err != nil || environment.Status != status || environment.Initialization != initialization[0] { + t.Fatal("Environment changed", environment.Status, environment.Initialization, err) + } + pending, err := r.sessions.GetEnvironmentInputReservation(t.Context(), session.TenantID, session.ID, f.input.ID) + if err != nil || pending.State != sessions.EnvironmentInputPending || !pending.Deadline.Equal(f.input.Deadline) { + t.Fatal("pending input changed", pending, err) + } + var nativeID string + if err = f.pool.QueryRow(t.Context(), `SELECT native_session_id FROM session_devices WHERE session_id=$1`, session.ID).Scan(&nativeID); err != nil || nativeID != "retained-native-session" { + t.Fatal("native identity changed", nativeID, err) + } + if _, err = r.workspaces.DeleteBatch(t.Context(), ""); err != nil { + t.Fatal(err) + } + if f.control.deletes != 0 { + t.Fatal("filesystem deleted", f.control.deletes) + } +} + +// demand runs the common placement scan and pending provisioning for +// retained input, as TestRetainedEnvironmentDemandRecreatesCompute does. +func (f lostComputeFixture) demand(t *testing.T) { + t.Helper() + r := f.lifecycle + r.config.Mode = string(sandbox.DeploymentNodes) + manager := &runtimeManager{ctx: t.Context(), config: r.config, setupGate: make(chan struct{}, 1), inventory: make(chan struct{}, 1), nodes: make(map[string]*runtimeNode), workspaces: r.workspaces, sessions: r.sessions, sessionExecution: r.sessionExecution, deployment: r.deployment, deploymentService: r.deployments, deploymentReader: r.reader, lease: r.lease, registry: r.registry} + if err := manager.reservePlacements(t.Context()); err != nil { + t.Fatal(err) + } + if err := r.provisionPending(t.Context()); err != nil { + t.Fatal(err) + } +} + +func TestConfirmedLostRunningComputeContinuesRetainedSession(t *testing.T) { + for name, observe := range lostObservations { + t.Run(name, func(t *testing.T) { + f := newLostComputeFixture(t, true) + f.provider.observe = observe + r := f.lifecycle + binding, err := f.storage.Get(t.Context(), f.session.TenantID, f.owner.EnvironmentID) + if err != nil { + t.Fatal(err) + } + f.provider.killError = sandbox.ErrComputeUnconfirmed + if err = r.observe(t.Context(), f.owner); !errors.Is(err, sandbox.ErrComputeUnconfirmed) { + t.Fatal("unconfirmed cleanup settled lost compute", err) + } + // Retained input cannot reserve another writer before confirmed release. + f.demand(t) + if current := f.current(t); current.ID != f.owner.ID || current.State != "running" || current.ComputePhase != "running" || f.provider.creates != 1 || f.revoked(t, f.owner.DeviceID) { + t.Fatal("replacement admitted before confirmed release", current, f.provider.creates) + } + f.provider.killError = nil + if err = r.observe(t.Context(), f.current(t)); err != nil { + t.Fatal(err) + } + released := f.current(t) + if released.ID != f.owner.ID || released.State != "released" || !released.CreateSettled || !f.revoked(t, f.owner.DeviceID) { + t.Fatal("confirmed loss did not release ownership", released) + } + if !slices.Equal(f.provider.killed, []sandbox.Compute{lostCompute, lostCompute}) || f.provider.snapshots != 0 { + t.Fatal("cleanup did not target the exact incarnation", f.provider.killed, f.provider.snapshots) + } + f.assertRetained(t, "disconnected") + + f.demand(t) + replacement := f.current(t) + if replacement.ID == f.owner.ID || replacement.DeviceID == f.owner.DeviceID || replacement.State != "running" || !replacement.CreateSettled || f.provider.creates != 2 { + t.Fatal("retained input did not continue on new compute", replacement, f.provider.creates) + } + next := <-f.provider.bootstraps + if next.Workspace == nil || next.Workspace.Attachment.Reference != binding.Reference || next.DeviceID != replacement.DeviceID { + t.Fatal("replacement did not attach the retained filesystem", next.Workspace) + } + f.assertRetained(t, "disconnected") + }) + } +} + +func TestUnconfirmedLostComputeKeepsOwnership(t *testing.T) { + type negative struct { + name string + owned bool + seed []string + observe func(sandbox.Compute) (sandbox.ComputeState, error) + killErr error + livePeer bool + want error + kills int + status string + initial string + } + foreign := func(status string) func(sandbox.Compute) (sandbox.ComputeState, error) { + return func(c sandbox.Compute) (sandbox.ComputeState, error) { + c.ID = "foreign-compute-id" + return sandbox.ComputeState{Compute: c, Status: status}, nil + } + } + tests := []negative{ + {name: "paused", observe: observedStatus("paused"), want: sandbox.ErrComputeUnconfirmed}, + {name: "created", observe: observedStatus("created"), want: sandbox.ErrComputeUnconfirmed}, + {name: "starting", observe: observedStatus("starting"), want: sandbox.ErrComputeUnconfirmed}, + {name: "draining", observe: observedStatus("draining"), want: sandbox.ErrComputeUnconfirmed}, + {name: "unknown_status", observe: observedStatus("unknown"), want: sandbox.ErrComputeUnconfirmed}, + {name: "running_without_peer", observe: observedStatus("running"), want: runtimegateway.ErrDeviceNotRegistered}, + {name: "transport_unconfirmed", observe: observedError(sandbox.ErrComputeUnconfirmed), want: sandbox.ErrComputeUnconfirmed}, + {name: "node_unavailable", observe: observedError(placement.ErrNodeUnavailable), want: placement.ErrNodeUnavailable}, + {name: "timeout", observe: observedError(context.DeadlineExceeded), want: context.DeadlineExceeded}, + {name: "identity_mismatch", observe: observedError(sandbox.ErrOwnership), want: sandbox.ErrOwnership}, + {name: "crashed_foreign_identity", observe: foreign("crashed"), want: sandbox.ErrOwnership}, + {name: "stopped_foreign_identity", observe: foreign("stopped"), want: sandbox.ErrOwnership}, + } + // Each guard holds for typed absence and for a verified terminal status; + // a terminal status that cannot be released stays unconfirmed. + for _, guard := range []negative{ + {name: "unknown_creation", seed: []string{`UPDATE runtime_allocations SET create_settled=false WHERE id=$3`}, want: sandbox.ErrNotFound}, + {name: "no_compute_id", seed: []string{`UPDATE runtime_allocations SET compute_state='{"current":{"Name":"old-compute"}}' WHERE id=$3`}, want: sandbox.ErrOwnership}, + {name: "restore_intent", seed: []string{`UPDATE runtime_allocations SET compute_state='{"current":{"Name":"old-compute","ID":"old-compute-id"},"target":{"Name":"next","Generation":1},"restore_id":"attempt"}' WHERE id=$3`}, want: sandbox.ErrNotFound}, + {name: "owned_workspace", owned: true, want: sandbox.ErrNotFound}, + {name: "initialization_incomplete", seed: []string{`UPDATE environments SET initialization='running' WHERE id=$1`}, want: sandbox.ErrNotFound, initial: "running"}, + {name: "terminal_environment", seed: []string{`UPDATE environments SET status='failed' WHERE id=$1`}, want: sandbox.ErrNotFound, status: "failed"}, + {name: "native_history_undeclared", seed: []string{`UPDATE devices SET supported_agent_kinds='[{"kind":"codex","available":true,"capabilities":{}}]' WHERE id=$2`}, want: sandbox.ErrNotFound}, + {name: "live_peer", livePeer: true, want: sandbox.ErrComputeUnconfirmed}, + {name: "cleanup_failure", killErr: sandbox.ErrComputeUnconfirmed, want: sandbox.ErrComputeUnconfirmed, kills: 1}, + } { + tests = append(tests, guard) + for _, status := range []string{"stopped", "crashed"} { + terminal := guard + terminal.name, terminal.observe = status+"_"+guard.name, observedStatus(status) + if errors.Is(guard.want, sandbox.ErrNotFound) { + terminal.want = sandbox.ErrComputeUnconfirmed + } + tests = append(tests, terminal) + } + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + f := newLostComputeFixture(t, !test.owned, test.seed...) + r := f.lifecycle + if test.observe != nil { + f.provider.observe = test.observe + } + f.provider.killError = test.killErr + if test.livePeer { + handler := runtimegateway.NewHandler(runtimegateway.HandlerConfig{Authenticator: runtimegateway.NewAuthenticator(r.sessions), Registry: r.registry}) + server := httptest.NewServer(http.HandlerFunc(handler.WS)) + defer server.Close() + query := url.Values{"device_id": {f.bootstrap.DeviceID}, "version": {proto.Version}} + conn, _, err := websocket.DefaultDialer.Dial("ws"+strings.TrimPrefix(server.URL, "http")+"?"+query.Encode(), http.Header{"Authorization": {"Bearer " + f.bootstrap.Credential}}) + if err != nil { + t.Fatal(err) + } + defer conn.Close() + deadline := time.Now().Add(5 * time.Second) + for { + if _, err := authorizedRuntimePeer(t.Context(), r.sessions, r.registry, f.owner.DeviceID); err == nil { + break + } + if time.Now().After(deadline) { + t.Fatal("peer did not register") + } + time.Sleep(20 * time.Millisecond) + } + } + before := f.current(t) + for range 2 { + if err := r.observe(t.Context(), f.current(t)); !errors.Is(err, test.want) { + t.Fatalf("observe = %v, want %v", err, test.want) + } + } + current := f.current(t) + if current.ID != before.ID || current.State != before.State || current.ComputePhase != "running" || current.ComputeRevision != before.ComputeRevision || current.CreateSettled != before.CreateSettled || string(current.ComputeState) != string(before.ComputeState) { + t.Fatal("ownership changed without confirmed loss", current) + } + if f.revoked(t, f.owner.DeviceID) || len(f.provider.killed) != 2*test.kills || f.provider.creates != 1 { + t.Fatal("unconfirmed loss revoked or cleaned compute", f.provider.killed, f.provider.creates) + } + status := test.status + if status == "" { + status = "disconnected" + } + if test.livePeer { + status = "connected" + } + if test.initial != "" { + f.assertRetained(t, status, test.initial) + return + } + f.assertRetained(t, status) + }) + } +} + +func TestLostComputeReleaseSerializesWithConcurrentIntent(t *testing.T) { + for name, observe := range lostObservations { + t.Run(name, func(t *testing.T) { + t.Run("restart_after_kill", func(t *testing.T) { + f := newLostComputeFixture(t, true) + f.provider.observe = observe + r := f.lifecycle + // The first owner confirms cleanup, then stops before the release commit. + stopped, stop := context.WithCancel(t.Context()) + f.provider.onKill = stop + if err := r.observe(stopped, f.owner); !errors.Is(err, context.Canceled) { + t.Fatal(err) + } + if current := f.current(t); current.State != "running" || f.revoked(t, f.owner.DeviceID) { + t.Fatal("interrupted owner committed release", current) + } + f.provider.onKill = nil + restarted := &runtimeLifecycle{nodeID: r.nodeID, registry: runtimegateway.NewRegistry(), sessions: r.sessions, sessionExecution: r.sessionExecution, deployment: r.deployment, deployments: r.deployments, reader: r.reader, lease: r.lease, workspaces: r.workspaces, config: r.config, connections: map[string]*runtimeConnection{}} + if err := restarted.observe(t.Context(), f.current(t)); err != nil { + t.Fatal(err) + } + released := f.current(t) + if released.State != "released" || !slices.Equal(f.provider.killed, []sandbox.Compute{lostCompute, lostCompute}) { + t.Fatal("restart did not converge on the same incarnation", released, f.provider.killed) + } + // A duplicate observation of the stale owner cannot change the receipt. + if err := r.observeCompute(t.Context(), f.owner); !errors.Is(err, deployment.ErrAllocationConflict) { + t.Fatal("stale duplicate release", err) + } + if current := f.current(t); current.State != "released" || current.ComputeRevision != released.ComputeRevision || len(f.provider.killed) != 2 || f.provider.creates != 1 { + t.Fatal("duplicate changed the released receipt", current, f.provider.killed) + } + f.assertRetained(t, "disconnected") + }) + for _, test := range []struct { + name string + statement string + want error + status string + }{ + {name: "compute_revision", statement: `UPDATE runtime_allocations SET compute_revision=compute_revision+1 WHERE environment_id=$1`, want: deployment.ErrAllocationConflict, status: "disconnected"}, + {name: "archive", statement: `UPDATE environments SET status='expired' WHERE id=$1`, want: deployment.ErrAllocationConflict, status: "expired"}, + {name: "deletion", statement: `UPDATE sessions SET deleted_at=clock_timestamp() WHERE id=(SELECT session_id FROM environments WHERE id=$1)`, want: sessions.ErrNotFound}, + } { + t.Run(test.name, func(t *testing.T) { + f := newLostComputeFixture(t, true) + f.provider.observe = observe + f.provider.onKill = func() { + if _, err := f.pool.Exec(t.Context(), test.statement, f.owner.EnvironmentID); err != nil { + t.Error(err) + } + } + if err := f.lifecycle.observe(t.Context(), f.owner); !errors.Is(err, test.want) { + t.Fatalf("observe = %v, want %v", err, test.want) + } + current := f.current(t) + if current.State != "running" || current.ComputePhase != "running" || f.revoked(t, f.owner.DeviceID) || f.provider.creates != 1 { + t.Fatal("concurrent intent lost to compute release", current) + } + if test.status != "" { + f.assertRetained(t, test.status) + } else if _, err := f.lifecycle.workspaces.DeleteBatch(t.Context(), ""); err != nil || f.control.deletes != 0 { + t.Fatal("lost release deleted the filesystem", err) + } + }) + } + }) + } +} diff --git a/services/core/internal/sandbox/sandbox_provider.go b/services/core/internal/sandbox/sandbox_provider.go index 568c16b07..91c156eee 100644 --- a/services/core/internal/sandbox/sandbox_provider.go +++ b/services/core/internal/sandbox/sandbox_provider.go @@ -109,6 +109,8 @@ type SandboxProvider interface { // Store remain the lifecycle owner. Its operations share one declaration. Initial(context.Context, Reference) (Compute, error) NewCompute(context.Context, Reference, uint64, *SnapshotIdentity) (Compute, error) + // GetCompute returns ErrNotFound only after the owning node verified that + // the exact incarnation no longer exists. GetCompute(context.Context, Reference, Compute) (ComputeState, error) Suspend(context.Context, SuspendRequest) (ComputeState, error) Resume(context.Context, ResumeRequest) (ComputeState, error) @@ -170,7 +172,10 @@ type SnapshotIdentity struct { } type ComputeState struct { - Compute Compute + Compute Compute + // Status is the native state of Compute. "stopped" and "crashed" mean the + // adapter verified that the exact incarnation's process no longer exists; + // its record or disks can remain until KillCompute removes them. Status string BootstrapComplete bool Snapshot *SnapshotIdentity @@ -182,6 +187,9 @@ type ComputeState struct { RestoreAttemptClosed string } +// Terminated reports a verified stopped or crashed incarnation. +func (s ComputeState) Terminated() bool { return s.Status == "stopped" || s.Status == "crashed" } + // ClosesRestoreAttempt validates a never-executed closure against the exact // persisted request. Absence without this operation-bound evidence is unknown. func (s ComputeState) ClosesRestoreAttempt(q ResumeRequest) bool { From 589e12855f11a7945b8afa8e3997c7a77477e7c6 Mon Sep 17 00:00:00 2001 From: SaladDay <1203511142@qq.com> Date: Sun, 11 Oct 2026 08:48:48 +0000 Subject: [PATCH 2/3] Record confirmed compute loss qualification --- contracts/agents-api/index.md | 2 +- contracts/agents-api/zh/index.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/contracts/agents-api/index.md b/contracts/agents-api/index.md index 01c6f77c5..d01902017 100644 --- a/contracts/agents-api/index.md +++ b/contracts/agents-api/index.md @@ -128,7 +128,7 @@ Each item is Core's deliberate or native behavior where the official service beh - Hosted workspace admission is limited by the [tenant capacity policy](../../docs/configuration.md#tenant-capacity) and the existing [input reservation deadline](./environments.md#reservations). - Runtimes do not enforce `disabled` or `restricted` networks, so Sessions that need them are rejected ([restricted network policy](./environments.md#restricted-network)). - `packages.system` is rejected; system packages must be preinstalled. -- Cold continuation does not restore process memory, background processes or temporary system-disk changes; completed setup is not replayed. See [Environment persistence](./environments.md#runtime-capability-preparation). Cross-node memory restoration and takeover of an unconfirmed old writer are not qualified. After a [confirmed compute loss](../../docs/sandbox-provider.md#confirmed-compute-loss), crash consistency of native history is not qualified for any Harness. +- Cold continuation does not restore process memory, background processes or temporary system-disk changes; completed setup is not replayed. See [Environment persistence](./environments.md#runtime-capability-preparation). Cross-node memory restoration and takeover of an unconfirmed old writer are not qualified. On microsandbox with retained NFS storage, Codex and Claude Code passed [confirmed compute loss](../../docs/sandbox-provider.md#confirmed-compute-loss) continuation checks after idle and interrupted Turns and within two seconds of completion acknowledgement. Missing-history checks preserved the native identity and failed explicitly with generic `engine_failed` or `environment_unavailable` outcomes. Host-crash durability and completeness of arbitrary crash-damaged history remain unqualified. **Files and Environment files** diff --git a/contracts/agents-api/zh/index.md b/contracts/agents-api/zh/index.md index 0f35d9a04..40cff93ca 100644 --- a/contracts/agents-api/zh/index.md +++ b/contracts/agents-api/zh/index.md @@ -1,7 +1,7 @@ --- title: "Agents API 覆盖台账" source: contracts/agents-api/index.md -source_hash: d7fff07ff5af2349fe56e951e0ed6e215e9d8ba5a191a251730b8fb1e4b00251 +source_hash: bb527943480bed1ad33d3e8eb330543bdced57b66617e52b5a2b18e2d9f5bcc8 --- Core 旨在以下方固定版本为准支持完整的 OpenAI Agents API（[public API rule](https://github.com/MiniMax-AI/OpenAgentCore/blob/main/AGENTS.md#public-api)）。本台账记录 Core 对各项资源实现了哪些内容、哪些契约保存其详细信息，并列出相对于 OpenAI 服务的所有已知差异和所有未解决缺口。[API namespaces and credentials](../../../docs/zh/api/index.md) 说明谁调用哪些 API；[Agents API guide](../../../docs/zh/api/public-agent-api.md) 介绍使用方法。 @@ -131,7 +131,7 @@ Core 自身字段位于 `x_agents_core` 中（[Core extensions](../../../docs/zh - Hosted 工作区准入受[租户容量策略](../../../docs/zh/configuration.md#tenant-capacity)及既有[输入预留截止时间](environments.md#reservations)限制。 - Runtime 不会实施 `disabled` 或 `restricted` 网络，因此需要这些网络的 Session 会被拒绝（[restricted network policy](environments.md#restricted-network)）。 - `packages.system` 会被拒绝；系统软件包必须预先安装。 -- 冷续接不恢复进程内存、后台进程或临时系统盘修改；已完成的 setup 不会重放。参见 [Environment 持久化](environments.md#runtime-capability-preparation)。跨节点内存恢复及接管未确认停止的旧写入者尚未通过资格验证。在[已确认的计算资源丢失](../../../docs/zh/sandbox-provider.md#confirmed-compute-loss)后，任何 Harness 的原生历史崩溃一致性均未通过资格验证。 +- 冷续接不恢复进程内存、后台进程或临时系统盘修改；已完成的 setup 不会重放。参见 [Environment 持久化](environments.md#runtime-capability-preparation)。跨节点内存恢复及接管未确认停止的旧写入者尚未通过资格验证。在使用保留 NFS 存储的 microsandbox 上，Codex 与 Claude Code 已通过空闲 Turn 后、Turn 中断时以及完成确认后两秒内发生[已确认的计算资源丢失](../../../docs/zh/sandbox-provider.md#confirmed-compute-loss)的续接检查。历史缺失检查保留了原生身份，并以通用 `engine_failed` 或 `environment_unavailable` 结果明确失败。主机崩溃时的持久性及任意崩溃受损历史的完整性仍未通过资格验证。 **Files 和 Environment files** From 7cb3527e1236d8b9c5536a0c7ced68074015513b Mon Sep 17 00:00:00 2001 From: SaladDay <1203511142@qq.com> Date: Sun, 11 Oct 2026 09:01:58 +0000 Subject: [PATCH 3/3] Clarify qualified retained recovery scope --- contracts/agents-api/index.md | 2 +- contracts/agents-api/zh/index.md | 4 ++-- docs/sandbox-provider.md | 2 +- docs/zh/sandbox-provider.md | 4 ++-- 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/contracts/agents-api/index.md b/contracts/agents-api/index.md index d01902017..feaa326bc 100644 --- a/contracts/agents-api/index.md +++ b/contracts/agents-api/index.md @@ -128,7 +128,7 @@ Each item is Core's deliberate or native behavior where the official service beh - Hosted workspace admission is limited by the [tenant capacity policy](../../docs/configuration.md#tenant-capacity) and the existing [input reservation deadline](./environments.md#reservations). - Runtimes do not enforce `disabled` or `restricted` networks, so Sessions that need them are rejected ([restricted network policy](./environments.md#restricted-network)). - `packages.system` is rejected; system packages must be preinstalled. -- Cold continuation does not restore process memory, background processes or temporary system-disk changes; completed setup is not replayed. See [Environment persistence](./environments.md#runtime-capability-preparation). Cross-node memory restoration and takeover of an unconfirmed old writer are not qualified. On microsandbox with retained NFS storage, Codex and Claude Code passed [confirmed compute loss](../../docs/sandbox-provider.md#confirmed-compute-loss) continuation checks after idle and interrupted Turns and within two seconds of completion acknowledgement. Missing-history checks preserved the native identity and failed explicitly with generic `engine_failed` or `environment_unavailable` outcomes. Host-crash durability and completeness of arbitrary crash-damaged history remain unqualified. +- Cold continuation does not restore process memory, background processes or temporary system-disk changes; completed setup is not replayed. See [Environment persistence](./environments.md#runtime-capability-preparation). Codex and Claude Code passed cross-node memory restoration on compatible microsandbox nodes with retained NFS storage under the [checkpoint transfer requirements](../../docs/sandbox-provider.md#checkpoint-transfer). Takeover of an unconfirmed old writer remains unqualified. On microsandbox with retained NFS storage, Codex and Claude Code passed [confirmed compute loss](../../docs/sandbox-provider.md#confirmed-compute-loss) continuation checks after idle and interrupted Turns and within two seconds of completion acknowledgement. Missing-history checks preserved the native identity and failed explicitly with generic `engine_failed` or `environment_unavailable` outcomes. Host-crash durability and completeness of arbitrary crash-damaged history remain unqualified. **Files and Environment files** diff --git a/contracts/agents-api/zh/index.md b/contracts/agents-api/zh/index.md index 40cff93ca..e27ed60f1 100644 --- a/contracts/agents-api/zh/index.md +++ b/contracts/agents-api/zh/index.md @@ -1,7 +1,7 @@ --- title: "Agents API 覆盖台账" source: contracts/agents-api/index.md -source_hash: bb527943480bed1ad33d3e8eb330543bdced57b66617e52b5a2b18e2d9f5bcc8 +source_hash: 842e0f7a2685950efaad106244a101e11f213030387426759206ba0bfd44ab61 --- Core 旨在以下方固定版本为准支持完整的 OpenAI Agents API（[public API rule](https://github.com/MiniMax-AI/OpenAgentCore/blob/main/AGENTS.md#public-api)）。本台账记录 Core 对各项资源实现了哪些内容、哪些契约保存其详细信息，并列出相对于 OpenAI 服务的所有已知差异和所有未解决缺口。[API namespaces and credentials](../../../docs/zh/api/index.md) 说明谁调用哪些 API；[Agents API guide](../../../docs/zh/api/public-agent-api.md) 介绍使用方法。 @@ -131,7 +131,7 @@ Core 自身字段位于 `x_agents_core` 中（[Core extensions](../../../docs/zh - Hosted 工作区准入受[租户容量策略](../../../docs/zh/configuration.md#tenant-capacity)及既有[输入预留截止时间](environments.md#reservations)限制。 - Runtime 不会实施 `disabled` 或 `restricted` 网络，因此需要这些网络的 Session 会被拒绝（[restricted network policy](environments.md#restricted-network)）。 - `packages.system` 会被拒绝；系统软件包必须预先安装。 -- 冷续接不恢复进程内存、后台进程或临时系统盘修改；已完成的 setup 不会重放。参见 [Environment 持久化](environments.md#runtime-capability-preparation)。跨节点内存恢复及接管未确认停止的旧写入者尚未通过资格验证。在使用保留 NFS 存储的 microsandbox 上，Codex 与 Claude Code 已通过空闲 Turn 后、Turn 中断时以及完成确认后两秒内发生[已确认的计算资源丢失](../../../docs/zh/sandbox-provider.md#confirmed-compute-loss)的续接检查。历史缺失检查保留了原生身份，并以通用 `engine_failed` 或 `environment_unavailable` 结果明确失败。主机崩溃时的持久性及任意崩溃受损历史的完整性仍未通过资格验证。 +- 冷续接不恢复进程内存、后台进程或临时系统盘修改；已完成的 setup 不会重放。参见 [Environment 持久化](environments.md#runtime-capability-preparation)。Codex 与 Claude Code 已在使用保留 NFS 存储的兼容 microsandbox 节点间通过跨节点内存恢复验证，适用[检查点转移要求](../../../docs/zh/sandbox-provider.md#checkpoint-transfer)。接管未确认停止的旧写入者仍未通过资格验证。在使用保留 NFS 存储的 microsandbox 上，Codex 与 Claude Code 已通过空闲 Turn 后、Turn 中断时以及完成确认后两秒内发生[已确认的计算资源丢失](../../../docs/zh/sandbox-provider.md#confirmed-compute-loss)的续接检查。历史缺失检查保留了原生身份，并以通用 `engine_failed` 或 `environment_unavailable` 结果明确失败。主机崩溃时的持久性及任意崩溃受损历史的完整性仍未通过资格验证。 **Files 和 Environment files** diff --git a/docs/sandbox-provider.md b/docs/sandbox-provider.md index 1225ddcc5..421c3a8d6 100644 --- a/docs/sandbox-provider.md +++ b/docs/sandbox-provider.md @@ -216,7 +216,7 @@ New input or live Environment file access reserves a new allocation on a compati Running compute has no retention deadline, so a host crash or reboot would otherwise leave its allocation owned indefinitely. After the owning node is online again, `GetCompute` returning `ErrNotFound` or a verified `stopped` or `crashed` status for the exact persisted incarnation releases a checkpoint-capable allocation in compute phase `running` only when all of these hold: the original creation is settled, the incarnation has a native ID, no capture, restore or rollback intent is recorded, no authenticated Runtime connection exists for the allocation's device, and the Session meets the conditions for [cold replacement](#cold-replacement-after-checkpoint-retention). Core then calls `KillCompute` for that exact incarnation. Only its confirmed result permits one Session-locked commit that rechecks the compute revision and those conditions, revokes the device and releases the allocation and placement. The Session, Environment, pending input, native Session identity and filesystem remain, and new input or live file access continues as in cold replacement. -Every other observation keeps ownership and its diagnostic: a `paused`, `running` or unknown native status, an identity mismatch, a timeout or transport error, an unknown creation, a recorded restore, a live Runtime connection, owned workspace storage, incomplete initialization, a terminal Session or a missing `retained_native_history` declaration. A failed `KillCompute` changes nothing, and a concurrent archive, deletion or compute transition prevails. Core never acts while the node is offline, because it has no authority to stop compute without the node. Continuation retains the filesystem and uses the recorded native Session ID; a Harness resume error fails execution rather than creating another native Session. Core does not replay interrupted or completed input. Completeness of reopened native history relative to acknowledged Turns after a crash is [not qualified](../contracts/agents-api/index.md); identity checks do not detect a truncated but parseable history tail. +Every other observation keeps ownership and its diagnostic: a `paused`, `running` or unknown native status, an identity mismatch, a timeout or transport error, an unknown creation, a recorded restore, a live Runtime connection, owned workspace storage, incomplete initialization, a terminal Session or a missing `retained_native_history` declaration. A failed `KillCompute` changes nothing, and a concurrent archive, deletion or compute transition prevails. Core never acts while the node is offline, because it has no authority to stop compute without the node. Continuation retains the filesystem and uses the recorded native Session ID; a Harness resume error fails execution rather than creating another native Session. Core does not replay interrupted or completed input. The [coverage ledger](../contracts/agents-api/index.md) records the qualified continuation cases, including crashes immediately after completion acknowledgement. Completeness of arbitrary crash-damaged native history remains unqualified; identity checks do not detect a truncated but parseable history tail. ### Reset and archive diff --git a/docs/zh/sandbox-provider.md b/docs/zh/sandbox-provider.md index 7deb78353..f7f696b65 100644 --- a/docs/zh/sandbox-provider.md +++ b/docs/zh/sandbox-provider.md @@ -1,7 +1,7 @@ --- title: "添加 Sandbox Provider" source: docs/sandbox-provider.md -source_hash: edaa63076fb1629202fc159bb084124eaf0efef56f094d03a99538e281e74e65 +source_hash: 30ee130c8e62827662f1d0867da206e911e4dbb57ed5d75fe68cffdfda55e2b3 --- **Sandbox Provider** 为 Core 管理的 Environment 提供 Runtime daemon 运行所需的外层计算资源，以及启动 daemon 的有界引导流程。本指南说明如何添加 Provider，并作为 Core 驱动 Provider 的参考。接口为 [`SandboxProvider`](https://github.com/MiniMax-AI/OpenAgentCore/blob/main/services/core/internal/sandbox/sandbox_provider.go)。 @@ -218,7 +218,7 @@ Core 在释放 allocation 和 placement 归属前，确认原始创建已结算 正在运行的计算资源没有保留期限，因此主机崩溃或重启后，其 allocation 原本会无限期保持归属。所属节点重新上线后，对于计算阶段为 `running` 且支持检查点的 allocation，只有同时满足以下条件时，`GetCompute` 对已持久化的精确 incarnation 返回 `ErrNotFound` 或已验证的 `stopped`、`crashed` 状态才会释放该 allocation：原始创建已结算，该 incarnation 有 native ID，未记录 capture、restore 或 rollback 意图，allocation 的 device 没有已认证的 Runtime 连接，且 Session 满足[冷替换](#cold-replacement-after-checkpoint-retention)条件。随后 Core 对该精确 incarnation 调用 `KillCompute`。只有其结果得到确认，才允许一次持有 Session 锁的提交：重新检查计算 revision 与上述条件，撤销 device，并释放 allocation 和 placement。Session、Environment、等待中的 input、原生 Session 身份和文件系统均保留，新 input 或实时文件访问按冷替换续接。 -其他观察结果均保留归属及其诊断：`paused`、`running` 或未知原生状态、身份不匹配、超时或传输错误、未知创建、已记录的 restore、存在 Runtime 实时连接、自带工作区存储、初始化未完成、Session 已终结，或缺少 `retained_native_history` 声明。`KillCompute` 失败时不做任何修改，并发的归档、删除或计算阶段变更优先。节点离线时 Core 从不执行此操作，因为没有节点就无权停止计算资源。续接保留文件系统并使用已记录的原生 Session ID；Harness 恢复出错时执行失败，不创建另一个原生 Session。Core 不重放已中断或已完成的输入。崩溃后重新打开的原生历史相对于已确认 Turn 的完整性[尚未通过资格验证](../../contracts/agents-api/zh/index.md)；身份检查无法检测已截断但仍可解析的历史尾部。 +其他观察结果均保留归属及其诊断：`paused`、`running` 或未知原生状态、身份不匹配、超时或传输错误、未知创建、已记录的 restore、存在 Runtime 实时连接、自带工作区存储、初始化未完成、Session 已终结，或缺少 `retained_native_history` 声明。`KillCompute` 失败时不做任何修改，并发的归档、删除或计算阶段变更优先。节点离线时 Core 从不执行此操作，因为没有节点就无权停止计算资源。续接保留文件系统并使用已记录的原生 Session ID；Harness 恢复出错时执行失败，不创建另一个原生 Session。Core 不重放已中断或已完成的输入。[覆盖台账](../../contracts/agents-api/zh/index.md)记录已通过资格验证的续接场景，包括完成确认后立即崩溃。任意崩溃受损原生历史的完整性仍未通过资格验证；身份检查无法检测已截断但仍可解析的历史尾部。 ### 重置与归档 {#reset-and-archive}