fix(indexer): decode checkpoints carrying the Validity expiration (WALM-668)

[harrymove-ctrl]

Closes the code half of WALM-668. The two relayer fixes that ticket also tracks are already merged (#921) and already on staging (#922); this is the indexer fix, which had no PR.

What broke

Staging's accounts_v1 pipeline could not decode testnet checkpoints:

Failed to convert checkpoint protobuf to checkpoint data:
transaction.bcs: invalid value: integer 3, expected variant index 0 <= i < 3

TransactionExpiration gained a fourth variant, Validity, in testnet-v1.79.0. The indexer was pinned to testnet-v1.75.1, which knows three (None, Epoch, ValidDuring), so variant index 3 is out of range for it.

The indexer decodes whole checkpoints, not individual events, so an unknown transaction field stops the checkpoint rather than being skipped — the pipeline retried the same checkpoint forever. By 12:23 UTC on 2026-09-21 that had built up 48,527 pending commit watermarks, and GET /api/accounts/{owner}/exists — which reads the indexer's accounts table — answered false for accounts that exist on-chain, blocking Dio on Console identity linking.

The fix

Move both Sui pins from 623521008f (testnet-v1.75.1) to ceaaff1cc8 (testnet-v1.80.0), the current testnet release.

Nothing else has to move with it:

• Remote store layout is unchanged between the two revisions — same {checkpoint}.binpb.zst objects, same zstd → protobuf → types decode path — so INDEXER_REMOTE_STORE_URL and the ingestion flags stay as they are.
• Earlier expiration variants keep their indices, so already-indexed history and the replay from the saved watermark stay valid.
• src/ needed no changes: Processor, the postgres Handler, and IndexerCluster are source-compatible across the bump.

Cargo.lock is regenerated; the resolved graph still lands on diesel 2.3 / diesel-async 0.9, matching the comment in Cargo.toml.

Tests

tests/affected_checkpoint.rs decodes testnet checkpoint 384534623 — the sample checkpoint named in the incident, committed as a 9.6 KB fixture in the exact form the remote store serves it — through the real StoreIngestionClient, offline. It also asserts the fixture still carries a Validity transaction, so the test can't quietly stop reproducing the failure.

tests/checkpoint_decoding.rs pins Validity to variant index 3 and holds None/Epoch/ValidDuring at 0/1/2.

running 1 test
test decodes_the_checkpoint_that_stalled_staging ... ok

running 2 tests
test decodes_validity_expiration ... ok
test preserves_earlier_expiration_variant_indices ... ok

On the old pin the first test fails at TryFromProtoError and the second does not compile — Validity does not exist there.

Still to do on WALM-668 (deploy-side, not in this PR)

• Deploy this indexer and let it resume from its saved watermark — preserve accounts and checkpoint state, don't skip gaps.
• Confirm the backlog drains and GET /api/accounts/{owner}/exists returns true for 0x35167fa0….
• Verify a fresh rememberAndWait reaches done with a blob_id and that recall returns it, now that the relayer's ValidDuring and exhausted-retry fixes are on staging.
• Record deployed commits and tell Dio when staging is ready for Console retesting.

Review follow-ups folded into this PR

1. Relayer Validity pin (on PR head cc61f695) — bumped services/server Sui crates sui-sdk-types / sui-rpc / sui-transaction-builder / sui-crypto from 0.3.x → =0.4.0 so TransactionExpiration::Validity (BCS variant 3) decodes. Risk path: security_delete_auth BCS-decodes full Transactions. Added unit test security_delete_auth_bcs_decodes_validity_expiration_transaction (passes locally). cargo check clean.
2. CI indexer coverage (local commit 0327b0f6, not yet on remote) — adds indexer-checks job to .github/workflows/test.yml (libpq-dev + cargo check / cargo test in services/indexer). Push blocked: OAuth token for harrymove-ctrl lacks GitHub workflow scope. Harry needs:

   gh auth refresh -h github.com -s workflow
   git push origin harryphan/walm-668-indexer-checkpoint-decoding

   (commit is already authored as harrymove-ctrl <harry.phan@commandoss.com> on this branch locally / on the agent box).

[harrymove-ctrl]

Local indexer test run (review follow-up)

GitHub CI for this PR only runs services/server (cargo test --bins), so these indexer regression tests are not exercised by the green checks. Ran them locally on this branch:

cd services/indexer
# needed rustc 1.90.0 (lock pulls roaring@0.11.5 which requires 1.90)
cargo test --test affected_checkpoint --test checkpoint_decoding

running 1 test
test decodes_the_checkpoint_that_stalled_staging ... ok

running 2 tests
test decodes_validity_expiration ... ok
test preserves_earlier_expiration_variant_indices ... ok

test result: ok. 3 passed; 0 failed

Breaking-change read (MemWal surface)

Not a product/API breaking change for SDK / apps / relayer API. Diff is scoped to services/indexer only (Cargo.toml/lock + tests + fixture). No changes under packages/sdk, services/server, or apps. JS @mysten/sui pins and server sui-sdk-types are untouched.

What does change (ops / indexer only):

• Build: indexer needs a newer rustc (≥1.90 for current lock) when compiling this pin.
• Runtime: indexer Sui type pin testnet-v1.75.1 → testnet-v1.80.0 so it can decode TransactionExpiration::Validity.
• Deploy risk is resume from watermark / catch-up, not client SDK breakage. Preserve accounts + checkpoint state; don’t wipe DB or skip gaps.
• Residual: large pin jump could still hit another unknown checkpoint field beyond expiration; watch first staging deploy.

Recommendation: safe to merge from a MemWal public-API POV after one more human +ops checklist. Do not ask Console/Dio to retest until this is merged, indexer redeployed, and backlog drained.

[harrymove-ctrl]

Retest (full cargo test in services/indexer)

cargo test
# rustc 1.90.0

running 0 tests (unit bin)
test decodes_the_checkpoint_that_stalled_staging ... ok
test decodes_validity_expiration ... ok
test preserves_earlier_expiration_variant_indices ... ok

3 passed; 0 failed

Local decode path looks good. Still OPEN / not merged — staging Console retest still waits on merge + indexer redeploy + watermark catch-up.

[harrymove-ctrl]

Follow-ups status

Landed on the PR branch (cc61f695):

• Relayer sui-sdk-types / sui-rpc / sui-transaction-builder / sui-crypto → 0.4.0 (Validity BCS variant 3).
• Unit test security_delete_auth_bcs_decodes_validity_expiration_transaction (passes locally).
• cargo check clean for server against the new pin.

Needs Harry once (workflow scope): local commit 0327b0f6 adds indexer-checks to .github/workflows/test.yml. Push is rejected with:
refusing to allow an OAuth App to create or update workflow … without workflow scope

gh auth refresh -h github.com -s workflow
cd /workspace/MemWal   # or your checkout of this branch
git push origin harryphan/walm-668-indexer-checkpoint-decoding

After that push, Actions will cargo check + cargo test the indexer (installs libpq-dev) so the WALM-668 regression tests actually run in CI.