Flaky keychain CI: billing-api.test.ts test #2 intermittently fails with 502 lit_unavailable

[GTC6244]

Summary

The Keychain v2 workflow (keychain-ci.yml) job keychain intermittently fails at the node scripts/test-local.mjs step. lit-agent-keychain/tests/billing-api.test.ts test #2 ("subscriptions, scoped direct execution, cancellation, immutable backups, and adversarial billing requests") throws Request failed (502).

Failing run: https://github.com/LIT-Protocol/chipotle/actions/runs/34896120953/job/104150421151 (observed on PR #616, which does not touch keychain code — see below).

cc @glitch003

Where it fails

tests/billing-api.test.ts throws from OwnerClient.create inside this loop:

// billing-api.test.ts:181-182
for (let i = 0; i < 12; i++)
  await a.create(`GROUP_CAP_${i}`, `group-value-${i}`);

Stack: OwnerClient.create (sdk/src/index.ts:455) -> c.api (billing-api.test.ts:42) -> Error: Request failed (502). Test #1 in the same file passes; only test #2 fails, at a non-deterministic iteration.

Root cause (server-side 502 lit_unavailable)

Each create enrolls a distinct action CID, so the keychain server fetches that CID's key via Chipotle::public_key, which executes the PUBLIC_KEY action against the mock Lit backend (tests/mock-lit.ts, a single Node/tsx process on :55440). When that HTTP call errors, it is mapped to a 502:

// src/registry.rs:25
lit.public_key(&cid)
    .await
    .map_err(|_| api::err(Status::BadGateway, "lit_unavailable"))

One of the 12 back-to-back, distinct-CID enrollments hits a transient failure talking to the mock and surfaces as 502. The underlying error is discarded by map_err(|_| ...), so logs don't reveal whether it was a timeout, connection reset, or a mock hiccup.

Evidence it's flaky, not a real regression

• The identical commit (62b57d28e) passed this job ~50 min before it failed: passing run https://github.com/LIT-Protocol/chipotle/actions/runs/34890911963 vs failing run https://github.com/LIT-Protocol/chipotle/actions/runs/34896120953 (ran on the newer synthetic PR-merge SHA 1aa8d822c).
• The same pass->fail flip-flop appears on an unrelated PR branch (feature/cpl-295-...).
• The heavier test Test PR: README Update #1 passes; only test Simplification: OTLP Sidecar Observability #2 fails, at a varying point in the loop.

Note on trigger scope

The workflow's paths: filter includes lit-actions/**, so PRs that only change lit-actions (e.g. #616, a logging-only change) run the full keychain billing/subscription suite even though the keychain CI never compiles lit-actions. This widens exposure to the flake.

Suggested fixes (for keychain owners)

1. Make the public_key/mock-lit interaction robust under burst — a small retry on transient errors in Chipotle::public_key, and/or harden mock-lit.ts for concurrent requests.
2. Stop swallowing the reqwest error in registry.rs (map_err(|_| ...)) so the real 502 cause is logged/diagnosable.
3. Consider narrowing/splitting the lit-actions/** path trigger so logging-only lit-actions PRs don't run the full billing suite.

Workaround

Re-running the failed job clears it (the same commit already passed once).