Skip to content

test(judge-node): recover after control-plane disconnect - #17

Open
CN059 wants to merge 1 commit into
agent/operator-cancel-commandfrom
agent/real-microvm-control-disconnect
Open

CN059 wants to merge 1 commit into
agent/operator-cancel-commandfrom
agent/real-microvm-control-disconnect

Conversation

@CN059

@CN059 CN059 commented Aug 31, 2026 •

Copy link
Copy Markdown
Member

Scope

Add one strict real-microVM process regression for the remaining control-channel outage gap. The test kills the control plane while an infinite C workload is running, requires Firecracker API/vsock reclamation and judge-node exit within three seconds, restarts the control plane, waits for expired-lease recovery into Attempt 2, and proves a fresh judge node persists the terminal timeout result.

Refs: FR-SCHED-001, FR-JUDGE-003, ACC-P0-003, ACC-P0-016, ACC-P0-017.

Non-scope

No production behavior, public or guest protocol, schema, migration, runtime image, network policy, authentication, authorization, or audit changes.

Verification

Passed locally:

  • cargo fmt --all -- --check
  • cargo check --workspace --all-targets --locked
  • cargo clippy --workspace --all-targets --locked -- -D warnings
  • focused process E2E compiled and followed the existing non-strict KVM skip path
  • python3 scripts/check-workspace.py
  • bash scripts/check-docs.sh

Unverified: strict real-KVM execution is blocked in the current environment because /dev/kvm is not read-write accessible. A strict KVM runner should execute:

OPENOJ_REQUIRE_KVM=1 OPENOJ_FC_TEST_IMAGES=<verified-runtime-out> cargo test -p openoj-control-plane --test process_e2e real_microvm_control_disconnect_reclaims_and_recovers --locked -- --nocapture --test-threads=1

Risk and rollback

Risk is confined to test runtime/flakiness on KVM hosts; the assertions use the existing three-second reclamation contract. Roll back the single test commit if the environment contract proves invalid. No data rollback is needed.

Stack

This PR is intentionally stacked on #16, which is stacked on #15 and the real-microVM E2E series.

Add a strict real-microVM process regression that kills the control plane during an infinite guest workload, requires bounded Firecracker reclamation and judge-node exit, then restarts the control plane and proves expired-lease recovery through a second Attempt and terminal result.

Refs: FR-SCHED-001, FR-JUDGE-003, ACC-P0-003, ACC-P0-016, ACC-P0-017

Tests: cargo fmt --all -- --check; cargo check --workspace --all-targets --locked; cargo clippy --workspace --all-targets --locked -- -D warnings; focused process E2E compiled and skipped without KVM; python3 scripts/check-workspace.py; bash scripts/check-docs.sh

Unverified: strict real-KVM execution is blocked because /dev/kvm is not read-write accessible in this environment

Security: No production boundary changes; the test asserts fail-closed teardown and lease recovery across a control-plane outage.

Compatibility: No protocol, schema, runtime image, or persisted data changes.

Migration: None.
@CN059
CN059 force-pushed the agent/operator-cancel-command branch from b980fa4 to 528a369 Compare August 31, 2026 00:51
@CN059
CN059 force-pushed the agent/real-microvm-control-disconnect branch from 83ad71f to c8841ca Compare August 31, 2026 00:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant