fix(bots): count only broadcast submits and explain skipped plans

.claude/commands/railway.md

@@ -42,8 +42,11 @@ railway status                            # lists services + Online/offline
 ```
 
 `midnight-liquidation` services: `bot`, `rindexer`, `Postgres`. Healthy `bot` ticks each block:
-`event="block.new"` → `event="lens.read"` → `event="tick.end"` (`liquidatable=`/`submitted=`
-counters), plus one `event="daemon.start"` at boot. Trouble = `event="tick.error"` or `[ERROR]`;
+`event="block.new"` → `event="lens.read"` → `event="tick.end"`, plus one `event="daemon.start"` at
+boot. In `tick.end`, `submitted=` counts only real broadcasts and `notSent=` counts submits that sent
+nothing; `liquidatable=` is a market gauge, so `liquidatable>0` with `planSkipped=` equal to it means
+the bot sees positions it cannot size — grep `plan.skipped` for the reason and the sizing numbers.
+`complete=false` marks a tick that aborted part-way, so its counters are partial. Trouble = `event="tick.error"` or `[ERROR]`;
 for a reverting tx grep `tx.dropped` / `tx.replace_failed` / `tx.submit_failed`. Triage by event,
 not the raw line — error lines historically carried a multi-KB calldata dump that shippers truncate.
 

bots/blue-liquidation/README.md

@@ -240,7 +240,17 @@ per-`(id, borrower)` exponential backoff suppresses repeated failures.
 ### Simulation
 
 The exact `Executor.exec_606BaXt(...)` bytes are `eth_call`-simulated from the EOA. Only an `ok`
-result is broadcast (`simulate.ok` gate). The exec is `liquidate` + an in-callback queue (via
+result is broadcast (`simulate.ok` gate).
+
+`tick.end`'s `submitted` counts only real broadcasts: `submit` reports whether a transaction went out,
+and only that clears the position's backoff. A send that failed for this position (`tx.submit_failed`)
+counts `notSent` and backs the position off; the three queue-wide refusals (`tx.send_aborted`,
+`nonce.sync_failed`, `queue.nonce_hole`) count `notSent` only, since they reject every send that tick.
+A tick aborted by a hashless nonce-bearing send still emits `tick.end` with `complete: false`.
+
+A position the chain reports liquidatable can still fail to size; the tick counts it as `planSkipped`
+and emits a sampled `plan.skipped` with the reason (`no_collateral`, `zero_price`,
+`seize_rounds_to_zero`) plus the sizing inputs and derived numbers. The exec is `liquidate` + an in-callback queue (via
 `onMorphoLiquidate`) that runs the plan's steps — a plain collateral is one venue swap; exotic
 collateral is unwrap step(s) then usually a venue swap — followed by the repay-token approval. After
 `liquidate` returns, trailing `skim` sweeps drain both market tokens **plus every intermediate token

bots/blue-liquidation/src/constants.ts

@@ -39,3 +39,10 @@ export const VIRTUAL_ASSETS = 1n
  * terminal, so a brief suppression of an already-acted position is harmless.
  */
 export const SETTLED_COOLDOWN_BLOCKS = 20n
+
+/**
+ * Block cadence bounding the per-position `plan.skipped` diagnostic (~5 min at ~2s Base blocks). An
+ * unsizable position recurs every tick, so one line per position per block would be ~21k
+ * lines/hour/bot — all repeating one fact. The `planSkipped` counter stays exact regardless.
+ */
+export const PLAN_SKIP_SAMPLE_EVERY_BLOCKS = 150n

bots/blue-liquidation/src/index.ts

@@ -5,6 +5,7 @@ import {
   assertContractDeployed,
   createBalanceMonitor,
   createBackoff,
+  createBlockSampler,
   createCooldownStore,
   createDeploylessClient,
   createHeartbeatMonitor,
@@ -34,7 +35,7 @@ import type { MarketParams } from './market'
 import type { LiquidationPlan } from './sizing/plan'
 
 import { loadConfig } from './config'
-import { SETTLED_COOLDOWN_BLOCKS } from './constants'
+import { PLAN_SKIP_SAMPLE_EVERY_BLOCKS, SETTLED_COOLDOWN_BLOCKS } from './constants'
 import { createGraphqlCandidateSource, discoverCandidates } from './discovery/borrowers'
 import { encodeLiquidationExec } from './execution/encode-call'
 import { composeQuoting } from './quotes'
@@ -179,6 +180,9 @@ async function main() {
   // Opt-in per-position cooldown (default disabled): one in-memory store for the process lifetime,
   // complementary to `backoff` (see POSITION_LIQUIDATION_COOLDOWN_MS).
   const cooldown = createCooldownStore({ cooldownMs: config.positionCooldownMs })
+  // Bounds the `plan.skipped` diagnostic to one burst per cadence; process-lifetime state, like the
+  // two stores above.
+  const planSkipSampler = createBlockSampler(PLAN_SKIP_SAMPLE_EVERY_BLOCKS)
 
   // The exec calldata for one liquidation — the same bytes the simulate gate checks and the queue
   // broadcasts, so a sim-ok plan and its broadcast can't drift.
@@ -273,7 +277,9 @@ async function main() {
         }),
       submit: async ({ market, borrower, plan, swapPlan, blockNumber, label }) => {
         const fees = initialFees(await signer.getBaseFee(), config.maxFeeWei)
-        await queue.submit({
+        // Returned, not awaited-and-discarded: the tick needs to know whether a tx actually went out
+        // before it clears this position's backoff.
+        return queue.submit({
           request: {
             to: config.executooorAddress,
             data: encodeExec(market, borrower, plan, swapPlan)
@@ -286,6 +292,7 @@ async function main() {
       },
       backoff,
       cooldown,
+      planSkipSampler,
       inflightLabels: () => queue.inflightLabels(),
       logger
     })