From 2b58919e0c6577746f626accc8383557d5d930d8 Mon Sep 17 00:00:00 2001 From: Florent Tapponnier <160007691+Flotapponnier@users.noreply.github.com> Date: Fri, 31 Jul 2026 18:22:17 +0200 Subject: [PATCH] fix: shorten methodology bullets to stay under Zod 500/240-char caps --- benchmarks/pm-rate-limits.yml | 4 ++-- benchmarks/pm-ws-latency.yml | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/benchmarks/pm-rate-limits.yml b/benchmarks/pm-rate-limits.yml index b453116a..834488c6 100644 --- a/benchmarks/pm-rate-limits.yml +++ b/benchmarks/pm-rate-limits.yml @@ -49,13 +49,13 @@ abstract: | second window. methodology: - - "Endpoint classes per venue. Polymarket: CLOB `/book`, `/midpoint`, Gamma `/markets`. Kalshi: `/markets/{ticker}/orderbook`, `/markets/{ticker}`, `/markets`. Limitless: `/markets/{slug}/orderbook`, `/markets/{slug}`, `/markets/active`. Manifold: `/v0/bets`, `/v0/market/{id}`, `/v0/markets`. Myriad: `/markets/{slug}`, `/markets`. PredictIt: `/api/marketdata/markets/{id}`, `/api/marketdata/all` (no book endpoint, 15s interval). Smarkets: `/v3/markets/{id}/quotes/`, `/v3/markets/{id}/`, `/v3/events/`. Metaculus: `/api2/questions/{id}/`, `/api2/questions/` (no book or WebSocket). Myriad and Metaculus expose no order book endpoint; that absence is reported, not worked around." + - "Endpoint classes per venue. Polymarket: CLOB `/book`, `/midpoint`, Gamma `/markets`. Kalshi: `/markets/{ticker}/orderbook`, `/markets/{ticker}`, `/markets`. Limitless: `/markets/{slug}/orderbook`, `/markets/{slug}`, `/markets/active`. Manifold: `/v0/bets`, `/v0/market/{id}`, `/v0/markets`. Myriad, PredictIt, Metaculus: price and list only (no order book). Smarkets: `/v3/markets/{id}/quotes/`, `/v3/markets/{id}/`, `/v3/events/`." - "Pinned market per venue: most liquid, near the money (price between 0.15 and 0.85), expiring more than 24h out. Re pinned daily at 00:00 UTC and immediately on probe_invalid. A degenerate 0.99 book has a latency profile that does not represent real usage." - "Warm probes every 5s (7s on Manifold, whose entire API sits behind max-age=5 plus stale-while-revalidate=10; tighter spacing would only measure the cache). Cold connect probe once per minute with keep alives disabled, TCP plus TLS handshake recorded separately." - "Every sample records a cache flag from cf-cache-status, x-cache and age headers. Kalshi's market list is served from CloudFront with max-age=15, so its list latency measures the edge, not the API. Edge served samples (cache=hit) are excluded from every latency aggregate and reported through the cache label, so no venue ranks on its CDN." - "Outcome classification: ok, timeout (8s), http_4xx, http_5xx, throttled (429), probe_invalid, net_error. Latency aggregates include successful requests only; failures count toward the success rate. probe_invalid covers stale pins, e.g. Limitless serves a CDN cached 400 for four hours once a market expires. Those are our fault, never the venue's." - "Daily ramp: 60s per tier at rising requests per 10s window against the book endpoint, one venue at a time, one region at a time on disjoint UTC hours (02:00 us-east, 04:00 eu-west, 06:00 sgp)." - - "Ramp tier clamps: Polymarket and Kalshi 25/50/100 per 10s (at most 7 percent of Polymarket's documented 1500 per 10s book budget), Manifold 15/30/60 (500 req/min per IP documented), Limitless 10/20/40 (limits undocumented, so conservative), Smarkets 10/25/50 (limits undocumented), Metaculus 5/10/20 (limits undocumented, forecasting API). Kalshi stops at the first 429 since its token bucket is documented. Myriad is excluded: its keyless budget is 30 req per 10s and ramping it would only measure our own quota. PredictIt is excluded: its documented 5 req/min limit means any ramp tier above 4 req/10s exceeds the budget." + - "Ramp tiers: Polymarket + Kalshi 25/50/100 per 10s, Manifold 15/30/60, Limitless 10/20/40, Smarkets 10/25/50, Metaculus 5/10/20. Kalshi stops at first 429 (documented token bucket). Myriad excluded (keyless budget 30/10s). PredictIt excluded (5 req/min published limit)." - "Ramp abort guard: the run stops the moment throttled plus 5xx responses exceed 1 percent of any 10 second window. The headline ramp metric is p50 latency during the tier minus the warm baseline p50 of the previous hour in the same region, because venues behind Cloudflare queue bursts without ever returning 429, and added latency is the only honest throttle signal there." - "WebSocket panel covers venues with a public market WebSocket, which today means Polymarket only. Kalshi requires authentication for WS, Manifold and Myriad have none, Limitless uses socket.io. Absence is shown, not penalized." - "Myriad's origin is a single region US East deployment behind Heroku. Latency from eu-west and sgp is geography, reported as measured." diff --git a/benchmarks/pm-ws-latency.yml b/benchmarks/pm-ws-latency.yml index d5da136f..3f094a15 100644 --- a/benchmarks/pm-ws-latency.yml +++ b/benchmarks/pm-ws-latency.yml @@ -47,7 +47,7 @@ methodology: - "Regions: us-east (Railway us-east4), eu-west (Railway europe-west4), sgp (Railway asia-southeast1). The Polymarket CLOB WS endpoint is globally distributed via Cloudflare, so regional differences reflect edge proximity and routing, not server capacity." - "Keepalive: a background goroutine sends the literal text PING every 10 seconds and the venue responds with PONG. Sessions that stop receiving data are detected by the reconnect logic on the next PING timeout. Disconnect rate is exported as a counter." - "Pin: the harness targets the same market used by the pm-rate-limits bench (highest 24h volume binary market with last trade price between 0.15 and 0.85), refreshed daily at 00:00 UTC. The WS client re-subscribes automatically when the pin changes." - - "Metrics: pmapi_ws_connect_to_snapshot_seconds histogram and pmapi_ws_update_interarrival_seconds histogram, both labeled by venue, region and source. The same pm-rate-limits harness process that probes REST endpoints runs the WS session, so both REST and WS data share one Railway deployment per region. Note on Kalshi: its ticker channel sends incremental price updates only (no state snapshot on subscribe), so the connect-to-snapshot metric measures time to the first price change rather than time to initial data, which depends on market activity." + - "Metrics: pmapi_ws_connect_to_snapshot_seconds and pmapi_ws_update_interarrival_seconds histograms, labeled by venue, region and source. The pm-rate-limits harness runs the WS session alongside REST probes, sharing one Railway deployment per region. Kalshi note: its ticker channel is delta-only (no snapshot on subscribe), so connect-to-snapshot measures time to the first price change, which depends on market activity." - "Other venues: Kalshi WS (wss://api.elections.kalshi.com/trade-api/ws/v2) requires RSA-PSS signed headers from a US KYC account and returns 403 from non-US IPs, so it is measured from us-east only and excluded from the rank matrix. Limitless uses socket.io over HTTP long-polling, not standard WebSocket. Manifold, Myriad, PredictIt, Smarkets and Metaculus have no public WebSocket endpoint." findings: @@ -138,7 +138,7 @@ providers: - slug: kalshi name: Kalshi tag: RSA-PSS auth required, CFTC-regulated, probed from us-east only (403 from non-US IPs) - formula: "p50 of time-to-first-price-update from us-east only (24h). Kalshi's ticker channel is delta-only (no snapshot on subscribe), so this measures time from dial to the next price change — not an initial snapshot delivery. Kalshi WS returns 403 from eu-west and sgp even with valid credentials." + formula: "p50 of time-to-first-price-update from us-east only (24h). Kalshi's ticker channel is delta-only (no snapshot on subscribe), so this measures time from dial to the next price change — not an initial snapshot delivery." queries: p50: 1000 * histogram_quantile(0.50, sum(rate(pmapi_ws_connect_to_snapshot_seconds_bucket{source="direct",venue="kalshi",region="us-east"}[24h])) by (le)) p90: 1000 * histogram_quantile(0.90, sum(rate(pmapi_ws_connect_to_snapshot_seconds_bucket{source="direct",venue="kalshi",region="us-east"}[24h])) by (le))