diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index e696674c..e51b6923 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -39,6 +39,9 @@ jobs: - name: Verify toolchain pin alignment run: bash scripts/verify-toolchain-pins.sh + - name: Verify release manifest generation + run: bash scripts/generate-release-manifest.sh --tag v0.0.0-ci --output /tmp/release-manifest.json + - name: Verify action pins run: bash scripts/verify-action-pins.sh diff --git a/docs/invariants.md b/docs/invariants.md index 664dcec5..ffe0069e 100644 --- a/docs/invariants.md +++ b/docs/invariants.md @@ -76,7 +76,7 @@ by writer and are write-once (`0001_schema.sql`). the inclusion lane's live prediction produce the same accept/reject/ordering decisions for every input. The recovery fold (`fold_replay`) drives `Scheduler` itself, not a copy. (Known, documented - exception: the predicate omits the two structural rejections — self-trust, + exception: the predicate omits the three structural rejections — self-trust, since the simulator only runs over the sequencer's own well-formed batches; the reason is recorded at `scheduler_accepts`, and it is test-pinned by the I1 duality test in `sequencer-core/src/scheduler/mod.rs`, which asserts diff --git a/docs/protocol/scheduler-semantics.md b/docs/protocol/scheduler-semantics.md index 195cf00f..131837f0 100644 --- a/docs/protocol/scheduler-semantics.md +++ b/docs/protocol/scheduler-semantics.md @@ -100,8 +100,11 @@ order; the first that applies decides the outcome: Ages saturate at zero. An empty batch is valid at any age, since it has no first frame (**pending**). Gate d rejects a batch whose frames include a `safe_block` above `inclusion_block`, or a `safe_block` lower than the previous -frame's. It looks only inside the batch: a later batch may start below an -earlier one's last `safe_block`, and its user ops then see the lower value. A +frame's, or a `fee_price > MAX_EXPONENT`. Gate d checks all frames before +any frame drain or user-op execution; the overdue-direct backstop still runs +before the gates. Structural validation looks only inside the batch: a later +batch may start below an earlier one's last `safe_block`, and its user ops then +see the lower value. A stale batch keeps the nonce, so every later batch fails gate b until one reuses that nonce, as a batch rebuilt after sequencer recovery does. A rejected and a skipped batch have the same effect: no state change, nonce @@ -122,9 +125,6 @@ For each frame, in order: `frame.fee_price` and clock `frame.safe_block` ([`validate_and_execute_user_op`](../../sequencer-core/src/application/mod.rs)). -The scheduler does not range-check `fee_price`; the reference wallet halts on -converting an exponent above `MAX_EXPONENT` (**pending**). - ## Outputs and the executed-input count An input's outputs are the application's notices and vouchers in execution diff --git a/docs/review/register.md b/docs/review/register.md index a806c3c6..bb06b137 100644 --- a/docs/review/register.md +++ b/docs/review/register.md @@ -260,9 +260,6 @@ evidence: the spec's links and - **Signature acceptance set.** The guest and `POST /tx` accept every `v ≥ 35` and the high-`s` twin, as alloy does. Proposed: `v ∈ {27, 28}` and low `s` only. -- **Out-of-range frame fee.** A `fee_price` above `MAX_EXPONENT` halts the - machine once an op reaches the wallet's fee conversion. Proposed: reject such - a batch structurally. - **Empty batches** are valid at any age and consume a nonce, though the sequencer never sends one. Reject them only if the port forces a new image anyway. diff --git a/docs/watchdog/README.md b/docs/watchdog/README.md index 800b47e4..40328edf 100644 --- a/docs/watchdog/README.md +++ b/docs/watchdog/README.md @@ -49,6 +49,13 @@ retry: a failed tick's retry is the next scheduled one. ### Canonical execution +External starting checkpoints used by `init` and `replay` pass the pinned +emulator's diagnostic `verify_hash_tree` check before trust checks. It checks +the loaded machine state against its stored hash tree; the real-machine +regression covers modified RAM with unchanged hash sidecars, which a cached +root read misses. This checks storage consistency, not canonical provenance: +a non-genesis checkpoint still needs independently trusted provenance. + The watchdog drives the machine in-process through the `cartesi` Lua module, with the rollup host semantics of Dave's sling node and the CLI's epoch driver ([grounding and pins](../cartesi-machine.md#rollup-host-semantics)). It clones @@ -319,12 +326,13 @@ Successful `init` or an idle tick is not evidence of a comparison. - *Head trust.* Sling asserts every stored boundary against its database row, because it adopts content-addressed directories by name. The watchdog keeps one head, published by its own sync and no-replace rename - and never opened for writing, and checks no root when it opens it: v0.22 - has no supported check of data pages against the stored hash tree (its - `verify_hash_tree` serves the emulator's internal tests), so a recorded - root would catch only a swapped directory or a torn hash tree. A damaged head - surfaces as a latched divergence, which the runbook's replay attributes to - the watchdog. + and never opened for writing. Published heads are trusted local artifacts; + ordinary ticks do not repeat bootstrap verification or compare a recorded + root. A damaged head surfaces as a latched divergence, which the runbook's + replay from an independently trusted checkpoint attributes to the watchdog. + External `init` and `replay` checkpoints use `verify_hash_tree` as described + above. The emulator documents this diagnostic for internal tests; its + stale-sidecar behavior is pinned by our real-machine regression. - *An advance the machine does not take.* The references carry on from it as an unchanged accept, where the watchdog fails loud (above); the InputBox cap keeps every chain input deliverable. diff --git a/docs/watchdog/incident-runbook.md b/docs/watchdog/incident-runbook.md index 3d814abd..90e8c891 100644 --- a/docs/watchdog/incident-runbook.md +++ b/docs/watchdog/incident-runbook.md @@ -162,6 +162,16 @@ collection is `running`, `clear` reports `already locked`; retry once it ends. Clearing is always safe: if the cause persists, the next tick latches again. +## Foreclosure + +The guardian multisig, run by the sequencer's operators, decides foreclosure. +Stop and disable the sequencer as soon as foreclosure is decided: an operation +it confirms that has not landed when `foreclose()` executes never will. On +Kubernetes, scale the Deployment to zero, because Deployments restart regardless +of exit code ([restart policy](operator-deployment.md#sequencer-restart-policy)). +Never restart the sequencer for a foreclosed application: batch posts fail at +gas estimation, and every recovery cycle sends a few no-op flush transactions. + ## Drills | Scenario | Drill | Practices | diff --git a/examples/canonical-test/src/main.rs b/examples/canonical-test/src/main.rs index ab9977b0..aaae8f88 100644 --- a/examples/canonical-test/src/main.rs +++ b/examples/canonical-test/src/main.rs @@ -25,6 +25,39 @@ const INVALID_BATCH_REPORT: &[u8] = b"scheduler dropped invalid batch"; const TEST_CHAIN_ID: u64 = 31337; const TEST_DAPP_ADDRESS: Address = Address::ZERO; +#[testsi::test_dapp(kind("scheduler"))] +pub fn scheduler_fee_bounds_keep_the_guest_live() -> TestResult { + let mut machine = devnet_machine()?; + let config = WalletConfig::devnet(); + let key = signing_key(1); + let owner = address_from_signing_key(&key); + let amount = U256::from(250); + machine.advance_state(portal_input( + 10, + config.supported_erc20_token, + owner, + amount, + ))?; + let rejected = batch_with_frame_fee( + 0, + 10, + sequencer_core::fee::MAX_EXPONENT + 1, + vec![signed_user_op_with_fee(&key, 0, u16::MAX, vec![])], + ); + let (outputs, reports) = machine.advance_state(batch_input(11, rejected))?; + assert_invalid_batch_step(&outputs, &reports); + let (outputs, reports) = + machine.advance_state(batch_input(11, batch_with_safe_blocks(0, &[10])))?; + assert_deposit_notice( + &outputs, + &reports, + config.supported_erc20_token, + owner, + amount, + ); + Ok(()) +} + #[testsi::test_dapp(kind("scheduler"))] pub fn scheduler_reports_invalid_batch_from_guest() -> TestResult { let mut machine = devnet_machine()?; diff --git a/scripts/generate-release-manifest.sh b/scripts/generate-release-manifest.sh index 4b8e9896..b6a62f12 100755 --- a/scripts/generate-release-manifest.sh +++ b/scripts/generate-release-manifest.sh @@ -83,6 +83,11 @@ manifest = { "rust": os.environ["RUST_TOOLCHAIN"], "xgenext2fs": os.environ["XGENEXT2FS_VERSION"], "cartesi_machine": os.environ["CARTESI_MACHINE_VERSION"], + "linux_kernel": { + "release": os.environ["CARTESI_LINUX_RELEASE"], + "filename": os.environ["CARTESI_LINUX_KERNEL_FILENAME"], + "sha512": os.environ["CARTESI_LINUX_KERNEL_SHA512"], + }, }, "artifacts": artifacts, "alignment": { diff --git a/sequencer-core/src/protocol.rs b/sequencer-core/src/protocol.rs index a80b2736..5f672b70 100644 --- a/sequencer-core/src/protocol.rs +++ b/sequencer-core/src/protocol.rs @@ -315,11 +315,12 @@ impl ProtocolTiming { /// `batch_reject_reason_for_block` additionally rejects a batch whose frames /// have a `safe_block` above the inclusion block /// (`SafeBlockAboveInclusionBlock`) or that decrease across frames - /// (`NonMonotonicSafeBlocks`). This predicate does not re-check those: a + /// (`NonMonotonicSafeBlocks`), or whose fee exceeds `MAX_EXPONENT` + /// (`FeePriceOutOfRange`). This predicate does not re-check those: a /// well-formed batch submitter never builds such a batch, so they are /// reachable only under a buggy or forked submitter — already outside the /// duality boundary. The I1 duality test pins this as a *documented* - /// asymmetry; mirror those two checks here if the submitter ever stops being + /// asymmetry; mirror those checks here if the submitter ever stops being /// trusted to be well-formed. pub fn scheduler_accepts( &self, diff --git a/sequencer-core/src/scheduler/mod.rs b/sequencer-core/src/scheduler/mod.rs index 75265dca..dafca910 100644 --- a/sequencer-core/src/scheduler/mod.rs +++ b/sequencer-core/src/scheduler/mod.rs @@ -84,6 +84,7 @@ pub enum BatchRejectReason { WrongNonce { expected: u64, got: u64 }, SafeBlockAboveInclusionBlock, NonMonotonicSafeBlocks, + FeePriceOutOfRange, } #[derive(Debug)] @@ -288,6 +289,13 @@ impl Scheduler { } } + if std::iter::once(head) + .chain(tail) + .any(|frame| frame.fee_price > crate::fee::MAX_EXPONENT) + { + return Some(BatchRejectReason::FeePriceOutOfRange); + } + None } @@ -747,6 +755,71 @@ mod tests { ); } + #[test] + fn fee_bounds_reject_the_whole_batch_and_preserve_the_backstop() { + use crate::fee::MAX_EXPONENT; + + for fee in [MAX_EXPONENT + 1, u16::MAX] { + for invalid_position in [0, 1] { + let mut scheduler = Scheduler::new( + RecordingApp::default(), + SchedulerConfig { + sequencer_address: SEQUENCER, + max_wait_blocks: 10, + }, + ); + process_ok(&mut scheduler, direct_input(1, 1)); + process_ok(&mut scheduler, direct_input(8, 2)); + let key = SigningKey::from_bytes((&[1_u8; 32]).into()).unwrap(); + scheduler.app.credit(address_from_signing_key(&key), 1); + let frame = Frame { + safe_block: 8, + fee_price: 0, + user_ops: vec![sign_wire_user_op( + &test_domain(), + &key, + 0, + u16::MAX, + vec![3], + )], + }; + let mut frames = vec![frame.clone(), frame]; + frames[invalid_position].fee_price = fee; + assert_eq!( + process_ok(&mut scheduler, batch_input(11, Batch { nonce: 0, frames })).outcome, + ProcessOutcome::BatchRejected(BatchRejectReason::FeePriceOutOfRange), + ); + // Only the overdue direct executes: no frame drain or user op. + assert_eq!(scheduler.app.events(), [RecordedTx::Direct(1)]); + assert_eq!(scheduler.queued_direct_len(), 1); + assert_eq!(scheduler.next_expected_batch_nonce(), 0); + assert_eq!( + process_ok( + &mut scheduler, + batch_input( + 11, + Batch { + nonce: 0, + frames: vec![Frame { + safe_block: 8, + fee_price: MAX_EXPONENT, + user_ops: vec![] + }], + } + ) + ) + .outcome, + ProcessOutcome::BatchExecuted, + ); + assert_eq!( + scheduler.app.events(), + [RecordedTx::Direct(1), RecordedTx::Direct(2)] + ); + assert_eq!(scheduler.next_expected_batch_nonce(), 1); + } + } + } + #[test] fn non_monotonic_safe_blocks_invalidate_batch() { let mut scheduler = Scheduler::new( @@ -1459,6 +1532,19 @@ mod tests { offchain={offchain}" ); + let out_of_range_fee = ssz(&Batch { + nonce: 0, + frames: vec![Frame { + fee_price: crate::fee::MAX_EXPONENT + 1, + ..empty_frame(10) + }], + }); + let (canonical, offchain) = duality_run(SEQUENCER, 12, 0, &out_of_range_fee); + assert!( + !canonical && offchain, + "fee range is a documented structural omission" + ); + let frame_above_inclusion = ssz(&Batch { nonce: 0, frames: vec![empty_frame(20)], diff --git a/watchdog/canonical.lua b/watchdog/canonical.lua index 792b447d..9060ddd3 100644 --- a/watchdog/canonical.lua +++ b/watchdog/canonical.lua @@ -16,6 +16,7 @@ local canonical = {} --- the on-chain template hash), so an image from other source or another --- network cannot start a canonical history. function canonical.trusted_start(machine, l1, dir, block) + machine.verify(dir) l1:require_safe_head(block) local input_count = l1:input_count_at(block) if input_count == 0 and machine.root_hash(dir) ~= l1:template_hash() then diff --git a/watchdog/machine.lua b/watchdog/machine.lua index 3aa8db62..e8014966 100644 --- a/watchdog/machine.lua +++ b/watchdog/machine.lua @@ -142,6 +142,15 @@ function machine.new(cartesi) m:rename_stored(from_dir, to_dir) end + function api.verify(dir) + local m = load(dir, SHARING_NONE) + local ok, valid = pcall(m.verify_hash_tree, m) + m:destroy() + if not ok or not valid then + errors.operator("stored machine %s fails hash-tree verification", dir) + end + end + function api.root_hash(dir) local m = load(dir, SHARING_NONE) local root = m:get_root_hash() diff --git a/watchdog/tests/e2e.lua b/watchdog/tests/e2e.lua index d19e08a4..d481e2ab 100644 --- a/watchdog/tests/e2e.lua +++ b/watchdog/tests/e2e.lua @@ -327,6 +327,32 @@ test("an unknown range label is an operator error", function() end) end) +test("a stored machine with stale hash sidecars fails verification", function() + local image = require_image(WALLET_IMAGE) + local dir = support.tmpdir() + local copy = dir .. "/machine" + machine.clone(image, copy) + machine.verify(copy) + -- Replace the clone's RAM file to preserve the source even if storage + -- sharing changes. Leave all cached hash files untouched. + local path = copy .. "/0000000080000000-8000000.bin" + local ram_input = assert(io.open(path, "rb")) + local output = assert(io.open(path .. ".tmp", "wb")) + local byte = assert(ram_input:read(1)):byte() + output:write(string.char(byte ~ 1)) + while true do + local chunk = ram_input:read(1024 * 1024) + if not chunk then break end + output:write(chunk) + end + ram_input:close() + output:close() + assert(os.rename(path .. ".tmp", path)) + eq(hex(machine.root_hash(copy)), hex(machine.root_hash(image)), "cached root alone misses corruption") + raises("operator", "hash-tree verification", function() machine.verify(copy) end) + support.remove_tree(dir) +end) + test("the canonical wallet image answers the inspect query with its golden genesis state", function() local image = require_image(WALLET_IMAGE) local golden = abi.bytes_from_hex(support.read("tests/fixtures/wallet_snapshot_empty.hex"):gsub("%s", "")) diff --git a/watchdog/tests/run.lua b/watchdog/tests/run.lua index 42d954d2..e3a993f5 100644 --- a/watchdog/tests/run.lua +++ b/watchdog/tests/run.lua @@ -38,6 +38,19 @@ end -- ── abi ─────────────────────────────────────────────────────────────────── +test("trusted checkpoints reject corrupt hash trees at genesis and after inputs", function() + local canonical = require("watchdog.canonical") + for _, count in ipairs({ 0, 1 }) do + local machine = support.fake_machine({ corrupt_tree = true }) + raises("operator", "hash-tree verification", function() + canonical.trusted_start(machine, { + require_safe_head = function() end, + input_count_at = function() return count end, + }, "/snapshot", 0) + end) + end +end) + test("abi decodes an InputAdded log and keeps the raw envelope", function() local fixture = dofile("watchdog/tests/fixtures/input_added_evm_advance.lua") local input = abi.decode_input_added_log(fixture.log) diff --git a/watchdog/tests/support.lua b/watchdog/tests/support.lua index 9409b043..4cc96e86 100644 --- a/watchdog/tests/support.lua +++ b/watchdog/tests/support.lua @@ -254,6 +254,11 @@ function support.fake_machine(script) errors.operator("bad source") end end + function fake.verify() + if script.corrupt_tree then + errors.operator("stored machine fails hash-tree verification") + end + end function fake.root_hash() return script.root_hash or string.rep("\171", 32) end