Live Zulip ContractFactory: run the AgentComms contract against a real realm - #11
Merged
Merged
Conversation
…eal realm
Tier-0 contract-against-real (comms-e5vm.5). A ContractFactory built on the
real makeZulipHttp + minter-driven identity flow (the seam reused from
realm.live.test.ts), reshaped into the ContractEnv form.
Realm-friendliness: per-suite persistent self identity (acquire via
reactivate+regenerate, release { persistent }), fresh adapter per test for a
fresh events queue, per-test namespaced streams (cc-ct-<run>-<seq>-<name>) for
cross-test isolation, shared persistent peer bots, newUnacquiredAdapter omitted
(sanctioned for pinned-identity substrates), Retry-After inherited from the
request layer's rateLimitSchedule.
Generalises two contract assertions to be substrate-name-tolerant (compare
against the ref seedChannel returns, not a literal) so a substrate may namespace
the underlying channel without breaking the neutral suite — memory/fake stay
green.
Env-gated + excluded from default discovery (bunfig *.live.test.ts); run via
bun run test:live.
The contract relies on bun's 5s default per-test timeout — fine for the in-process fake, tight for live Zulip HTTP + long-poll delivery (realm.live measured delivery spiking to ~20s under load). A 45s global timeout for the live suite lifts the request/response + positive event-delivery tests from 30/45 to 38/45 green without touching the substrate-neutral contract. The 7 holdouts are structural (Zulip integer-second timestamps; minter-side-inbox self-mention narrow) — surfaced for the full-vs-subset call.
GraemeF
added a commit
that referenced
this pull request
Jun 13, 2026
…(clean-green subset) (#13) Follow-up to #11 (comms-e5vm.5). The live zulip ContractFactory runs the AgentComms contract against a real realm at 38/45. The 7 holdouts are **structural properties of any real Zulip** — integer-second timestamps, and a minter-side \`is:mentioned\` narrow keyed to the queue owner — not factory-fixable, and all covered elsewhere (memory/fake + \`realm.live.test.ts\`). They reported as 7 spurious failures: a broken window. This closes it. ## Change Two optional \`ContractEnv\` capability flags, following the existing \`newUnacquiredAdapter\` omission pattern (optional field + early-return gate, substrate-neutral — memory/fake leave them unset and run all 45): - **\`coarseTimestamps\`** gates the 4 range/replay tests that depend on distinct per-message timestamps (\`range.since\`/\`range.until\` inclusive, \`since==until\` single-timestamp, \`replay(since) returns …\`). The \`…past every message\` range/replay tests offset by an hour, so they run regardless. - **\`noSelfEventDelivery\`** gates the 3 event tests where a bot must observe its *own* post/mention on its own inbox within the contract's window (subscribe→post readiness, self-mention with/without explicit subscribe). The mention *suppression* test asserts non-delivery, so it runs regardless. The live factory (\`packages/zulip/contract.live.test.ts\`) sets both \`true\`. The flags are framed as **substrate-capability declarations**: a future owned/ephemeral realm that delivers self-events would set \`noSelfEventDelivery: false\` and recover those 3 — the gate reads "this substrate can't, so skip," not "nobody runs this." ## Verification - \`bun run check\` green (871 pass / 0 fail) — memory/fake contract unaffected, all 45 still run there. - \`bun run test:live\` green: **60 pass / 0 fail across 3 files** against the real realm (\`contract.live\` went 38/7 → 45/0). No 429s. Out of scope (sequenced after, in comms-e5vm.6): folding \`realm.live.test.ts\` into the contract run / deleting the stateful fake.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tier-0 of the hexagonal test architecture (comms-e5vm.5): run the same
runAgentCommsContractsuite the stateful fake and memory adapter pass, butagainst a real Zulip realm, via a
ContractFactorybuilt on the realmakeZulipHttp+ minter-driven identity flow (the seam reused fromrealm.live.test.ts).What's here
packages/zulip/contract.live.test.ts— the liveContractFactory.Realm-friendliness strategy:
selfbot(
cc-contract-self) is acquired via reactivate+regenerate and released{ persistent: true }so it is never deactivated (no per-test mint, noadmin-only reactivate path). A fresh adapter instance per test gives each
test a fresh Zulip events queue (
/register) — the reused-queue stalenesslesson from
realm.live.test.ts.cc-ct-<run>-<seq>-<name>) so each testreads/writes an isolated stream — the contract reuses fixed logical names
across ~50 tests and several channel-wide range tests assert
not.toContain/ exact-equality, which is cross-test contamination on anyrealm that persists channel state (independent of realm freshness).
seedAgent(no per-test mint).newUnacquiredAdapteromitted — sanctioned by the contract forsubstrates whose acquire-acceptable names are externally pinned; skips the
rebind/no-op lifecycle tests that would otherwise hit the admin-only
reactivate path on a non-admin minter.
rateLimitSchedule(cumulative-wait budget).packages/testing/contract.ts— two assertions generalised to besubstrate-name-tolerant (compare against the ref
seedChannelreturns, not aliteral name), so a substrate may namespace its underlying channel without
breaking the neutral suite. Memory + fake stay green.
package.json— wire the live file intotest:live, with a 45sper-test timeout for live HTTP/long-poll round-trips (the contract otherwise
relies on bun's 5s default, calibrated for the in-process fake).
The live suite is env-gated and excluded from default discovery
(
bunfig.toml*.live.test.ts), so CIcheckis unaffected; it runsout-of-band via
bun run test:live.Measured
Full
runAgentCommsContract('zulip-live', factory)againstchat.home.graemef.com: 38/45 pass, ~320s (with the 45s timeout; 30/45 atbun's 5s default). No 429s surfaced — minter pacing + the request layer's
Retry-After budget held the shared realm.
The 7 holdouts are structural / contract-baked, not factory-fixable:
selection, replay window): Zulip timestamps are integer-seconds, so
messages posted back-to-back collide; the contract assumes distinct
per-message ts (true for the fake). Range filtering is pure adapter logic,
already covered by fake/memory.
deliveries): the latter is the minter-side-inbox
is:mentionednarrow keyedto the queue owner, documented in
realm.live.test.ts— these paths arealready covered there with latency-tolerant deadlines + sender/observer
split.
The full-vs-subset-vs-ephemeral recommendation is surfaced to the
maintainer out-of-band; gating the 7 holdouts behind contract capability flags
(the existing
newUnacquiredAdapterpattern) is the proposed path to a fullygreen applicable subset, pending that decision.