Reject invalid frame fees and verify watchdog checkpoints

.github/workflows/ci.yml

@@ -39,6 +39,9 @@ jobs:
       - name: Verify toolchain pin alignment
         run: bash scripts/verify-toolchain-pins.sh
 
+      - name: Verify release manifest generation
+        run: bash scripts/generate-release-manifest.sh --tag v0.0.0-ci --output /tmp/release-manifest.json
+
       - name: Verify action pins
         run: bash scripts/verify-action-pins.sh
 

docs/invariants.md

@@ -76,7 +76,7 @@ by writer and are write-once (`0001_schema.sql`).
   the inclusion lane's live prediction produce the same
   accept/reject/ordering decisions for every input. The recovery fold
   (`fold_replay`) drives `Scheduler<A>` itself, not a copy. (Known, documented
-  exception: the predicate omits the two structural rejections — self-trust,
+  exception: the predicate omits the three structural rejections — self-trust,
   since the simulator only runs over the sequencer's own well-formed batches;
   the reason is recorded at `scheduler_accepts`, and it is test-pinned by
   the I1 duality test in `sequencer-core/src/scheduler/mod.rs`, which asserts

docs/protocol/scheduler-semantics.md

@@ -100,8 +100,11 @@ order; the first that applies decides the outcome:
 Ages saturate at zero. An empty batch is valid at any age, since it has no
 first frame (**pending**). Gate d rejects a batch whose frames include a
 `safe_block` above `inclusion_block`, or a `safe_block` lower than the previous
-frame's. It looks only inside the batch: a later batch may start below an
-earlier one's last `safe_block`, and its user ops then see the lower value. A
+frame's, or a `fee_price > MAX_EXPONENT`. Gate d checks all frames before
+any frame drain or user-op execution; the overdue-direct backstop still runs
+before the gates. Structural validation looks only inside the batch: a later
+batch may start below an earlier one's last `safe_block`, and its user ops then
+see the lower value. A
 stale batch keeps the nonce, so every later batch fails gate b until one
 reuses that nonce, as a batch rebuilt after sequencer recovery does. A
 rejected and a skipped batch have the same effect: no state change, nonce
@@ -122,9 +125,6 @@ For each frame, in order:
    `frame.fee_price` and clock `frame.safe_block`
    ([`validate_and_execute_user_op`](../../sequencer-core/src/application/mod.rs)).
 
-The scheduler does not range-check `fee_price`; the reference wallet halts on
-converting an exponent above `MAX_EXPONENT` (**pending**).
-
 ## Outputs and the executed-input count
 
 An input's outputs are the application's notices and vouchers in execution

docs/review/register.md

@@ -260,9 +260,6 @@ evidence: the spec's links and
 - **Signature acceptance set.** The guest and `POST /tx` accept every `v ≥ 35`
   and the high-`s` twin, as alloy does. Proposed: `v ∈ {27, 28}` and low `s`
   only.
-- **Out-of-range frame fee.** A `fee_price` above `MAX_EXPONENT` halts the
-  machine once an op reaches the wallet's fee conversion. Proposed: reject such
-  a batch structurally.
 - **Empty batches** are valid at any age and consume a nonce, though the
   sequencer never sends one. Reject them only if the port forces a new image
   anyway.

docs/watchdog/README.md

@@ -49,6 +49,13 @@ retry: a failed tick's retry is the next scheduled one.
 
 ### Canonical execution
 
+External starting checkpoints used by `init` and `replay` pass the pinned
+emulator's diagnostic `verify_hash_tree` check before trust checks. It checks
+the loaded machine state against its stored hash tree; the real-machine
+regression covers modified RAM with unchanged hash sidecars, which a cached
+root read misses. This checks storage consistency, not canonical provenance:
+a non-genesis checkpoint still needs independently trusted provenance.
+
 The watchdog drives the machine in-process through the `cartesi` Lua module,
 with the rollup host semantics of Dave's sling node and the CLI's epoch driver
 ([grounding and pins](../cartesi-machine.md#rollup-host-semantics)). It clones
@@ -319,12 +326,13 @@ Successful `init` or an idle tick is not evidence of a comparison.
   - *Head trust.* Sling asserts every stored boundary against its database
     row, because it adopts content-addressed directories by name. The
     watchdog keeps one head, published by its own sync and no-replace rename
-    and never opened for writing, and checks no root when it opens it: v0.22
-    has no supported check of data pages against the stored hash tree (its
-    `verify_hash_tree` serves the emulator's internal tests), so a recorded
-    root would catch only a swapped directory or a torn hash tree. A damaged head
-    surfaces as a latched divergence, which the runbook's replay attributes to
-    the watchdog.
+    and never opened for writing. Published heads are trusted local artifacts;
+    ordinary ticks do not repeat bootstrap verification or compare a recorded
+    root. A damaged head surfaces as a latched divergence, which the runbook's
+    replay from an independently trusted checkpoint attributes to the watchdog.
+    External `init` and `replay` checkpoints use `verify_hash_tree` as described
+    above. The emulator documents this diagnostic for internal tests; its
+    stale-sidecar behavior is pinned by our real-machine regression.
   - *An advance the machine does not take.* The references carry on from it
     as an unchanged accept, where the watchdog fails loud (above); the
     InputBox cap keeps every chain input deliverable.

docs/watchdog/incident-runbook.md

@@ -162,6 +162,16 @@ collection is `running`, `clear` reports `already locked`; retry once it
 ends. Clearing is always safe: if the cause persists, the next tick latches
 again.
 
+## Foreclosure
+
+The guardian multisig, run by the sequencer's operators, decides foreclosure.
+Stop and disable the sequencer as soon as foreclosure is decided: an operation
+it confirms that has not landed when `foreclose()` executes never will. On
+Kubernetes, scale the Deployment to zero, because Deployments restart regardless
+of exit code ([restart policy](operator-deployment.md#sequencer-restart-policy)).
+Never restart the sequencer for a foreclosed application: batch posts fail at
+gas estimation, and every recovery cycle sends a few no-op flush transactions.
+
 ## Drills
 
 | Scenario | Drill | Practices |

examples/canonical-test/src/main.rs

@@ -25,6 +25,39 @@ const INVALID_BATCH_REPORT: &[u8] = b"scheduler dropped invalid batch";
 const TEST_CHAIN_ID: u64 = 31337;
 const TEST_DAPP_ADDRESS: Address = Address::ZERO;
 
+#[testsi::test_dapp(kind("scheduler"))]
+pub fn scheduler_fee_bounds_keep_the_guest_live() -> TestResult {
+    let mut machine = devnet_machine()?;
+    let config = WalletConfig::devnet();
+    let key = signing_key(1);
+    let owner = address_from_signing_key(&key);
+    let amount = U256::from(250);
+    machine.advance_state(portal_input(
+        10,
+        config.supported_erc20_token,
+        owner,
+        amount,
+    ))?;
+    let rejected = batch_with_frame_fee(
+        0,
+        10,
+        sequencer_core::fee::MAX_EXPONENT + 1,
+        vec![signed_user_op_with_fee(&key, 0, u16::MAX, vec![])],
+    );
+    let (outputs, reports) = machine.advance_state(batch_input(11, rejected))?;
+    assert_invalid_batch_step(&outputs, &reports);
+    let (outputs, reports) =
+        machine.advance_state(batch_input(11, batch_with_safe_blocks(0, &[10])))?;
+    assert_deposit_notice(
+        &outputs,
+        &reports,
+        config.supported_erc20_token,
+        owner,
+        amount,
+    );
+    Ok(())
+}
+
 #[testsi::test_dapp(kind("scheduler"))]
 pub fn scheduler_reports_invalid_batch_from_guest() -> TestResult {
     let mut machine = devnet_machine()?;

scripts/generate-release-manifest.sh

@@ -83,6 +83,11 @@ manifest = {
         "rust": os.environ["RUST_TOOLCHAIN"],
         "xgenext2fs": os.environ["XGENEXT2FS_VERSION"],
         "cartesi_machine": os.environ["CARTESI_MACHINE_VERSION"],
+        "linux_kernel": {
+            "release": os.environ["CARTESI_LINUX_RELEASE"],
+            "filename": os.environ["CARTESI_LINUX_KERNEL_FILENAME"],
+            "sha512": os.environ["CARTESI_LINUX_KERNEL_SHA512"],
+        },
     },
     "artifacts": artifacts,
     "alignment": {

sequencer-core/src/protocol.rs

@@ -315,11 +315,12 @@ impl ProtocolTiming {
     /// `batch_reject_reason_for_block` additionally rejects a batch whose frames
     /// have a `safe_block` above the inclusion block
     /// (`SafeBlockAboveInclusionBlock`) or that decrease across frames
-    /// (`NonMonotonicSafeBlocks`). This predicate does not re-check those: a
+    /// (`NonMonotonicSafeBlocks`), or whose fee exceeds `MAX_EXPONENT`
+    /// (`FeePriceOutOfRange`). This predicate does not re-check those: a
     /// well-formed batch submitter never builds such a batch, so they are
     /// reachable only under a buggy or forked submitter — already outside the
     /// duality boundary. The I1 duality test pins this as a *documented*
-    /// asymmetry; mirror those two checks here if the submitter ever stops being
+    /// asymmetry; mirror those checks here if the submitter ever stops being
     /// trusted to be well-formed.
     pub fn scheduler_accepts(
         &self,

sequencer-core/src/scheduler/mod.rs

@@ -84,6 +84,7 @@ pub enum BatchRejectReason {
     WrongNonce { expected: u64, got: u64 },
     SafeBlockAboveInclusionBlock,
     NonMonotonicSafeBlocks,
+    FeePriceOutOfRange,
 }
 
 #[derive(Debug)]
@@ -288,6 +289,13 @@ impl<A: Application> Scheduler<A> {
             }
         }
 
+        if std::iter::once(head)
+            .chain(tail)
+            .any(|frame| frame.fee_price > crate::fee::MAX_EXPONENT)
+        {
+            return Some(BatchRejectReason::FeePriceOutOfRange);
+        }
+
         None
     }
 
@@ -747,6 +755,71 @@ mod tests {
         );
     }
 
+    #[test]
+    fn fee_bounds_reject_the_whole_batch_and_preserve_the_backstop() {
+        use crate::fee::MAX_EXPONENT;
+
+        for fee in [MAX_EXPONENT + 1, u16::MAX] {
+            for invalid_position in [0, 1] {
+                let mut scheduler = Scheduler::new(
+                    RecordingApp::default(),
+                    SchedulerConfig {
+                        sequencer_address: SEQUENCER,
+                        max_wait_blocks: 10,
+                    },
+                );
+                process_ok(&mut scheduler, direct_input(1, 1));
+                process_ok(&mut scheduler, direct_input(8, 2));
+                let key = SigningKey::from_bytes((&[1_u8; 32]).into()).unwrap();
+                scheduler.app.credit(address_from_signing_key(&key), 1);
+                let frame = Frame {
+                    safe_block: 8,
+                    fee_price: 0,
+                    user_ops: vec![sign_wire_user_op(
+                        &test_domain(),
+                        &key,
+                        0,
+                        u16::MAX,
+                        vec![3],
+                    )],
+                };
+                let mut frames = vec![frame.clone(), frame];
+                frames[invalid_position].fee_price = fee;
+                assert_eq!(
+                    process_ok(&mut scheduler, batch_input(11, Batch { nonce: 0, frames })).outcome,
+                    ProcessOutcome::BatchRejected(BatchRejectReason::FeePriceOutOfRange),
+                );
+                // Only the overdue direct executes: no frame drain or user op.
+                assert_eq!(scheduler.app.events(), [RecordedTx::Direct(1)]);
+                assert_eq!(scheduler.queued_direct_len(), 1);
+                assert_eq!(scheduler.next_expected_batch_nonce(), 0);
+                assert_eq!(
+                    process_ok(
+                        &mut scheduler,
+                        batch_input(
+                            11,
+                            Batch {
+                                nonce: 0,
+                                frames: vec![Frame {
+                                    safe_block: 8,
+                                    fee_price: MAX_EXPONENT,
+                                    user_ops: vec![]
+                                }],
+                            }
+                        )
+                    )
+                    .outcome,
+                    ProcessOutcome::BatchExecuted,
+                );
+                assert_eq!(
+                    scheduler.app.events(),
+                    [RecordedTx::Direct(1), RecordedTx::Direct(2)]
+                );
+                assert_eq!(scheduler.next_expected_batch_nonce(), 1);
+            }
+        }
+    }
+
     #[test]
     fn non_monotonic_safe_blocks_invalidate_batch() {
         let mut scheduler = Scheduler::new(
@@ -1459,6 +1532,19 @@ mod tests {
              offchain={offchain}"
         );
 
+        let out_of_range_fee = ssz(&Batch {
+            nonce: 0,
+            frames: vec![Frame {
+                fee_price: crate::fee::MAX_EXPONENT + 1,
+                ..empty_frame(10)
+            }],
+        });
+        let (canonical, offchain) = duality_run(SEQUENCER, 12, 0, &out_of_range_fee);
+        assert!(
+            !canonical && offchain,
+            "fee range is a documented structural omission"
+        );
+
         let frame_above_inclusion = ssz(&Batch {
             nonce: 0,
             frames: vec![empty_frame(20)],

watchdog/canonical.lua

@@ -16,6 +16,7 @@ local canonical = {}
 --- the on-chain template hash), so an image from other source or another
 --- network cannot start a canonical history.
 function canonical.trusted_start(machine, l1, dir, block)
+    machine.verify(dir)
     l1:require_safe_head(block)
     local input_count = l1:input_count_at(block)
     if input_count == 0 and machine.root_hash(dir) ~= l1:template_hash() then

watchdog/machine.lua

@@ -142,6 +142,15 @@ function machine.new(cartesi)
         m:rename_stored(from_dir, to_dir)
     end
 
+    function api.verify(dir)
+        local m = load(dir, SHARING_NONE)
+        local ok, valid = pcall(m.verify_hash_tree, m)
+        m:destroy()
+        if not ok or not valid then
+            errors.operator("stored machine %s fails hash-tree verification", dir)
+        end
+    end
+
     function api.root_hash(dir)
         local m = load(dir, SHARING_NONE)
         local root = m:get_root_hash()

watchdog/tests/e2e.lua

@@ -327,6 +327,32 @@ test("an unknown range label is an operator error", function()
     end)
 end)
 
+test("a stored machine with stale hash sidecars fails verification", function()
+    local image = require_image(WALLET_IMAGE)
+    local dir = support.tmpdir()
+    local copy = dir .. "/machine"
+    machine.clone(image, copy)
+    machine.verify(copy)
+    -- Replace the clone's RAM file to preserve the source even if storage
+    -- sharing changes. Leave all cached hash files untouched.
+    local path = copy .. "/0000000080000000-8000000.bin"
+    local ram_input = assert(io.open(path, "rb"))
+    local output = assert(io.open(path .. ".tmp", "wb"))
+    local byte = assert(ram_input:read(1)):byte()
+    output:write(string.char(byte ~ 1))
+    while true do
+        local chunk = ram_input:read(1024 * 1024)
+        if not chunk then break end
+        output:write(chunk)
+    end
+    ram_input:close()
+    output:close()
+    assert(os.rename(path .. ".tmp", path))
+    eq(hex(machine.root_hash(copy)), hex(machine.root_hash(image)), "cached root alone misses corruption")
+    raises("operator", "hash-tree verification", function() machine.verify(copy) end)
+    support.remove_tree(dir)
+end)
+
 test("the canonical wallet image answers the inspect query with its golden genesis state", function()
     local image = require_image(WALLET_IMAGE)
     local golden = abi.bytes_from_hex(support.read("tests/fixtures/wallet_snapshot_empty.hex"):gsub("%s", ""))

watchdog/tests/run.lua

@@ -38,6 +38,19 @@ end
 
 -- ── abi ───────────────────────────────────────────────────────────────────
 
+test("trusted checkpoints reject corrupt hash trees at genesis and after inputs", function()
+    local canonical = require("watchdog.canonical")
+    for _, count in ipairs({ 0, 1 }) do
+        local machine = support.fake_machine({ corrupt_tree = true })
+        raises("operator", "hash-tree verification", function()
+            canonical.trusted_start(machine, {
+                require_safe_head = function() end,
+                input_count_at = function() return count end,
+            }, "/snapshot", 0)
+        end)
+    end
+end)
+
 test("abi decodes an InputAdded log and keeps the raw envelope", function()
     local fixture = dofile("watchdog/tests/fixtures/input_added_evm_advance.lua")
     local input = abi.decode_input_added_log(fixture.log)

watchdog/tests/support.lua

@@ -254,6 +254,11 @@ function support.fake_machine(script)
             errors.operator("bad source")
         end
     end
+    function fake.verify()
+        if script.corrupt_tree then
+            errors.operator("stored machine fails hash-tree verification")
+        end
+    end
     function fake.root_hash()
         return script.root_hash or string.rep("\171", 32)
     end