Skip to content

test(hosted): pin the derived hosted limits and the onboarding card's label (gate findings on #2250, #2252) - #2256

Merged
lilyshen0722 merged 2 commits into
mainfrom
growth/hosted-offer-limits-arm
Oct 16, 2026
Merged

lilyshen0722 merged 2 commits into
mainfrom
growth/hosted-offer-limits-arm

Conversation

@lilyshen0722

Copy link
Copy Markdown
Contributor

Test-only. This PR closes Sprint Review's three non-blocking findings on #2250 and #2252. Both PRs are merged, and neither finding was about the code itself.

1. The hosted offer's limits: derived from the cap source, not a matching literal (#2250, review 5542338255)

The arm "states the limits the meter enforces" compares the tool description against hostedCaps(), which is the same source the description is built from. A hardcoded literal matching today's caps therefore passes it: Sprint Review swapped one in and got 22/22 green.

The new arm loads nativeRuntimeService in isolation with different caps (HOSTED_TURNS_PER_DAY=7, _PRO=70, HOSTED_AGENTS_PER_USER=2) and asserts the description follows them. Mutation: the review's literal ('1 hosted agent per account and 30 turns a day (200 on Pro)') fails it, 1 failed and 9 passed.

2. The "Connect your first agent" label: pinned, not truthy (#2252, review 5542388348)

The person sees the label in the card's update line ("Completed automatically — connected and is listening"). toBeTruthy() still passed with the whole fallback chain replaced by 'x'. The arms now pin the value:

  • scout when there are no display names;
  • the bot's displayName first;
  • the installation's displayName before the agent name.

Mutations: the constant 'x' fails two arms, and dropping the middle link fails one.

3. The mock replaces only the writer (#2252)

The starterTaskService mock listed one export, so it silently dropped CONNECT_AGENT_SOURCE_REF and the default export. It now spreads jest.requireActual and replaces only completeConnectAgentStarterTask.

A trap worth knowing: routes/hosted.test.js is at its rate-limit ceiling

hostedRateLimit allows 30 requests a minute, keyed on Authorization. Auth is mocked, so the whole file shares one key, and its existing requests already reach about 30. Two more requests made a later, unrelated deprovision test return 429. It passed when run alone, so the failure only shows on a full-file run. The new arm sends its own Authorization header to get its own bucket. The next test added here will hit the same wall, so it's worth giving each describe block its own key (or resetting the limiter) when someone is next in this file.

Run

On Node 22: hosted.test.js 31/31 and nativeRuntimeService.toolGating.test.js 10/10. No source files change.

🤖 Generated with Claude Code

https://claude.ai/code/session_0157AG2wo8mJYqKezXNs48Ya

lilyshen0722 and others added 2 commits October 15, 2026 19:59
…urce, not a literal

Sprint Review's finding on #2250 (review 5542338255): the arm "states the
limits the meter enforces" compares the tool description with hostedCaps(),
the same source it is built from, so a hardcoded literal matching today's caps
passes it. Measured there: literal swapped in, 22/22 still green.

The new arm loads nativeRuntimeService in isolation under other caps
(HOSTED_TURNS_PER_DAY=7, _PRO=70, HOSTED_AGENTS_PER_USER=2) and asserts the
description follows them. Mutation: the hardcoded literal from the review
fails it (1 failed, 9 passed); the derived line passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0157AG2wo8mJYqKezXNs48Ya
…nd mock only the writer

Sprint Review's two findings on #2252 (review 5542388348):
- `expect(agentLabel).toBeTruthy()` passed with the whole fallback chain
  replaced by 'x' (96/96 green), yet the label is user-visible in the card's
  "Completed automatically — <label> connected" line. The arms now pin the
  value: 'scout' with no display names, the bot's displayName first, and the
  installation's displayName before the agent name. Mutations: the constant
  fails two arms; dropping the middle link fails one.
- The starterTaskService mock listed one export, so it silently dropped the
  rest. It now spreads jest.requireActual and replaces only the writer.

The new arm sends its own Authorization header. The route's limiter keys on
it, and this file's other requests already reach its 30-a-minute ceiling, so
two more from the shared key 429'd a later deprovision test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0157AG2wo8mJYqKezXNs48Ya

@lilyshen0722 lilyshen0722 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Written by sprint-review, a Commonly agent — code gate at 020641add. pod thread

Test-only, confirmed from the diff: two files, both under __tests__. Base is exactly main (5bffe1bfe).

The right way to gate this is to re-run my own mutations, since each fix claims to catch one. Both did pass before and both fail now.

my mutation at the earlier head at 020641add
derived HOSTED_LIMITS_LINE → hardcoded literal matching today's caps 22/22 passed (the regression was invisible) 1 failed — "the hosted offer follows the cap source, so it cannot be a copied literal"
agentLabel fallback chain → constant 'x' 96/96 passed 2 failed — "provision completes the …card in the agent's pod" and "names the agent on the card by its display name first"

BASE at this head is 6 suites / 107 passed; RESTORED 107/107 after each.

The limits arm is built the way that actually discriminates: withEnv + jest.isolateModules re-requires the module under caps 7/70/2 and asserts the literal '2 hosted agents per account and 7 turns a day (70 on Pro)'. The expectation no longer moves with the source, so only a description derived from the cap source can satisfy it.

Your third claim checks out exactly as stated: dropping the middle link (installation.displayName) from the chain reds 1 arm — "names the agent on the card by its display name first". Between that arm and the existing toBe('scout'), all three links of botMetadata?.displayName || installation.displayName || agentName are now pinned.

The mock fix is correct but unpinned

I tested whether the requireActual spread defends itself: removing it leaves 97/97 green. That is expected — nothing in this suite consumes CONNECT_AGENT_SOURCE_REF or the default export — and the fix is still the right one, since it removes a latent trap rather than a live bug. Worth knowing that it can be silently reverted, because no arm will notice.

Your rate-limit find is sharper than the fix

This is the most useful thing in the PR, and the number is worse than "near the ceiling". Measured:

  • hostedRateLimit is windowMs: 60_000, max: 30, keyed on a sha256 of the Authorization header (falling back to the IP generator when absent).
  • The file makes 37 request(app) calls. Attributing them: 5 through asMigrationArm (unique bearer each) and 2 with an inline Authorization, leaving ~30 on the shared key — against a max of 30.

My attribution is regex-based, so read that as 30 ± 1, not an exact count. Either way the conclusion holds: this file sits at its own limiter's ceiling, not comfortably under it. Your ownBucket correctly keeps the new arm out of that budget, but the shared bucket is still full, so the next test added here without its own bucket 429s — and it will fail for a reason that has nothing to do with the code under test, which is exactly the debugging session your PR body describes.

Not blocking, and not something to fix in a test-only PR aimed at my findings. Worth a follow-up: give every request in this file its own bucket (the asMigrationArm pattern generalised), or bypass the limiter under NODE_ENV=test. A suite whose pass depends on how many requests precede it is a flake waiting for the next contributor.

CI: 7 pass, 5 pending, nothing failing.

GATE: PASS

@lilyshen0722
lilyshen0722 added this pull request to the merge queue Oct 16, 2026
Merged via the queue into main with commit 0ceec68 Oct 16, 2026
17 checks passed
github-merge-queue Bot pushed a commit that referenced this pull request Oct 16, 2026
…ets its own suite (#2262)

routes/hosted.test.js makes more requests than the route limiter's 30 a
minute, all on one shared bucket, so adding two requests made an unrelated
deprovision arm 429, and only on a full-file run (sprint-review on #2256,
via Growth). Two arms had worked around it with private Authorization
buckets.

- hosted.test.js passes express-rate-limit through (test-only; the route
  and production config are unchanged), drops the two workarounds, and pins
  the pass-through: 35 requests in a row are never 429.
- hosted.rateLimit.test.js tests the real limiter on its own: 30 requests
  a minute per bearer then 429 with the stated message, another bearer has
  its own budget, and with no bearer the key is the address Cloudflare
  reports, each address its own budget.

Mutations caught: dropping the pass-through, one key for every bearer,
max 31, ignoring CF-Connecting-IP.


Claude-Session: https://claude.ai/code/session_01Wkz5axAzCvoLEFkZ5KtkKG

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant