[tee8z]

Follows #9.

#9 bounded replay memory and concurrency correctly, but submit_score took a replay slot as soon as the handler ran. The slot was held across the user and session lookups, base64 decoding and SHA-256 hashing on the async runtime. A submission that could never replay (unknown session, wrong owner, impossible length) occupied one of the two slots. Under load, it got a retryable 503 instead of its 400/403/404.

What changed

• Admission order (domain/games/routes.rs). The handler now does these checks before admit():

  • user lookup, session lookup and ownership check
  • session seed and engine config parsing
  • a new check_encoded_length, which compares the base64 string length with the frame count without decoding. STANDARD requires canonical padding, so the encoded length of a valid log is exact. The frame cap applies here too.
  • with bot detection enabled, the server-side timing block (analyze_server_timing and the client-timing cross-reference). It needs only the session row, the request and the clock.

  The IP-activity query and the client frame-timing analysis still run after the replay, as before. Nothing is written before admission, so a busy 503 stays safe to retry. Every rejection keeps its status and message. Two things changed as a result:

  • When a submission fails more than one check, the one that now runs first answers.
  • server_elapsed is now measured when the submission arrives, not after its replay.

• Decode, hash and replay in the blocking job (verify.rs). ReplayPermit::verify now takes the base64 log and the claimed hash, and decodes, hashes and replays on the blocking worker. The encoded string is freed before the replay starts. The decoded bytes come back with the result, so the handler no longer clones them (input_bytes.clone() is gone). ReplayError gained InvalidEncoding and HashMismatch. They return the same 400 messages as before, and a hash mismatch still counts as a rejected score.

• Client backoff (game_canvas.js). On 503, retries back off exponentially with jitter (1 s, 2 s, then 4 s, each scaled by 0.5–1×). Each retry waits at least Retry-After. Retries stop after the same 11 s of total waiting, so a busy server gets about 4–7 re-uploads instead of 11.

• Metrics (metrics.rs). The new metrics are:

  • proofofscore_replay_rejections_total{reason}, with every reason pre-initialised: busy, invalid_input, invalid_encoding, hash_mismatch, timed_out, worker_failed. It is counted where each rejection happens: admit, check_encoded_length and verify.
  • proofofscore_replay_verification_seconds, a histogram of job time with the default buckets, which fit the 10 s budget.

• docs/score-integrity.md describes the admission order and the client backoff.

Tests

• routes::tests::busy_verification_answers_503_before_writing_anything (router over a real socket and in-memory SQLite, with both replay slots held):
  • A valid submission gets 503 with retry-after: 1, and no rows appear in scores, game_input_logs or score_metadata.
  • After the slots are released, the same submission gets 201 and is saved. This proves the 503 was safe to retry.
• routes::tests::invalid_submissions_get_their_4xx_while_verification_is_busy: with both slots held, an unknown session, another player's session and an impossible length get 404, 403 and 400 with their usual messages, and nothing is written.
• verify::tests::encoded_length_check_rejects_logs_that_cannot_hold_the_frames:
  • Exact encoded lengths pass for 0 frames, odd and even counts, and the cap.
  • One character more or less fails, and so does going over the cap.
• verify::tests::undecodable_or_mismatched_logs_keep_their_400_responses: bad base64 and a wrong hash, now detected inside the job, keep their 400 status and messages.
• verify::tests::admission_survives_cancellation_and_recovers_from_worker_failure: the spin on available_permits() + yield_now is replaced with a bounded timeout(5 s, acquire_many_owned(2)). This proves that both the failed worker's slot and the cancelled caller's slot come back.
• metrics::tests::renders_expected_families covers the new families.
• scripts/test-score-submission.cjs uses a stubbed Math.random and checks:
  • the exact backoff sequence and the 11 s budget
  • that jitter changes the delays
  • that Retry-After sets the shortest wait
  • the backoff when the header is missing
  • that manual retry still keeps the original session

The router tests run over a socket, like the metrics listener tests, because calling the router directly would need tower as a new dev-dependency.

Open questions for the owner

• Session reuse. docs/score-integrity.md says "one input log per session", but save_input_log upserts on session_id, and submit_score inserts a new scores row on every accepted submission. A session can therefore be submitted more than once: each submission adds a score, and the stored log is replaced. This PR does not change that. Should a session accept only one score?
• Server-side timing checks never run. Sessions store start_time as OffsetDateTime::to_string() (2026-10-09 12:00:00.0 +00:00:00). The handler parses it with Iso8601::DEFAULT, which requires a T separator, so the parse fails. As a result, analyze_server_timing and the client-timing cross-reference are skipped and server_elapsed_secs is stored as 0. update_session_activity has the same parse. This PR keeps the block as it is, now before admission, and adds no test that depends on it. Fixing the format would turn on 403s for real players. In particular, the cross-reference rejects when the server saw more than twice the time the client reports, and that includes time spent before the game starts. That needs a deliberate decision.

Deliberately not changed

• The 10 s replay wall-clock budget and the 3,000,000-frame cap. Changing them needs replay timings from real games. The new verification-duration histogram provides that data, so revisit both in a follow-up once it has samples.
• The length pre-check works at base64 granularity. A crafted log whose length is right but whose padding is wrong (for example AAAA for 3 frames) still takes a slot and then gets the same 400 from the decoded-length check inside the job. Only a non-browser client can produce this.