Skip to content

feat(gateway): computers follow the clock through a watch, and clock writes wait for them (FD-5471) - #614

Merged
earakely-scale merged 11 commits into
mainfrom
edgararakelyan/fd-5471-hb11-gateway-clock-one-version-a
Nov 11, 2026
Merged

earakely-scale merged 11 commits into
mainfrom
edgararakelyan/fd-5471-hb11-gateway-clock-one-version-a

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Part of FD-5420: a computer's own clock follows the gateway's universe clock. This is FD-5471, the gateway side. MCP servers read the gateway's get_time on every call, but a computer can't, because every program on it reads its own clock. So the computer follows the gateway through a long poll, and a clock change doesn't return until every computer shows it.

What changes

  • One version. The clock has one opaque version, a per-boot id and a count. Every set_time, advance and clear changes it, so a version never repeats across gateway restarts. /clock/state and every write's answer carry it.
  • GET /clock/watch?after=<version>&follower=<name>. It answers at once when the clock's version isn't after, else at the next change or after 25 s. The answer is the clock's state with the gateway's wall-clock received and sent stamps, which a follower uses for an NTP-style offset. A watch is its computer's acknowledgement of the version it names. It counts only when it carries that computer's read token, and never for a version the clock hasn't had.
  • Declared computers. The gateway provider lists each child whose card serves clock/v1 sync_time in CLOCK_FOLLOWER_TOKENS (name=read token), on the VM and container paths alike. For now the read token is the computer's control token, a slot a separate token can fill later with no protocol change. TOKEN in the name keeps the tokens out of logged compose files.
  • Writes that wait. set_time, advance and clear take wait_followers_s (0 to 30). Given one, the write waits for every declared computer to show its version. It answers 504 naming the computers still behind, and the change itself stands. A write without one, or to a gateway with no declared computers, answers as before.
  • set_rate (PUT /clock/set-rate). It changes an armed clock's rate from its time as of the write. Like advance, it keeps t0 and the clock's generation, so time triggers keep their relative marks and their waits, and it wakes the trigger engine, so a wait runs at the new rate. set_time still requires a virtual_time, as before.
    • In the prototype, a rate change made through set_time with a time computed from a state read a moment earlier stepped the universe back by the gap. A computer, which never steps back, then ran 65–85 ms ahead.
  • One path for every clock change.
    • Every change goes through Gateway._clock_changed: it re-evaluates the triggers when time moved, answers every held watch, and waits for the followers. Future writers, such as restoring the clock after a restart (FD-5433) or a snapshot reload, can't skip a step.
    • The four write routes share _clock_write's body handling.
  • The watch contract in the card. The card's watch method carries its request (after, follower) and hold_s (25), which a follower's request timeout must exceed. Every wait_followers_s is bounded 0–30 in the schema.
  • Restart. A restarted gateway is unarmed on a new boot id, so every follower's next watch answers at once.

Nothing on main declares a follower yet: computers start serving sync_time in FD-5476. Until then CLOCK_FOLLOWER_TOKENS is never rendered, and every clock route answers as before.

Also in this PR, a test-only fix for a required check:

  • The computer app tests' "notes" app took fastfetch from a Launchpad PPA to prove that an app's outside apt source and its key work.
  • So every computer_app_test failed whenever Launchpad was down: from about 12:10 UTC on 2026-11-11 here and on other PRs, and at roughly 85% uptime over 30 days.
  • It now takes docker-compose-plugin from Docker's apt repository instead. Ubuntu doesn't ship that package either, and the repo's armored key (9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88) sits beside the app.
  • The test keeps its checks: a ~noble version and the binary's own version output.

Not in this PR:

  • FD-5472: 403s for unknown or mismatched tokens, and clock writes behind the gateway's control token.
  • FD-5473: core passing wait_followers_s.

Docs

The clock extension's page should cover version, the watch method, wait_followers_s with its 504, and set_time without virtual_time as a rate change.

Testing

  • Unit: tst/unit/env/gateway/clock_watch_test.py and clock_version_test.py cover:

    • version changes;
    • watch timing;
    • acknowledgements only with the computer's token and never for a version the clock hasn't had;
    • the 504 naming stale computers, with the change standing;
    • unarmed after a restart;
    • set_rate continuous with the clock before it, keeping t0, the generation and a relative trigger's mark;
    • set_time without a time and set_rate without a rate refused;
    • every write refusing a body that isn't a JSON object;
    • 100 advances acknowledged in order.

    Provider and multi-env tests cover rendering and redacting CLOCK_FOLLOWER_TOKENS. The full unit suite passes locally (14,387 tests), apart from one Windows disk-recipe test that fails only on macOS.

  • E2E (071322a2, an EC2 host in our AWS account):

    • Setup: the branch's gateway image in Docker, declaring two computers in CLOCK_FOLLOWER_TOKENS. Each computer is a real follower process long-polling /clock/watch with its token.
    • Hundred advances: 100 of 100 advances with wait_followers_s: 2 answered 200, each acknowledged by both computers, in order. The write took 1.46 ms at p50 and 1.87 ms at p95; the prototype measured an acknowledgement wait of about 1.3 ms at p50.
    • Rate changes: the step a computer would see at the write was under 0.13 ms, across ×1↔×60 and ×1↔×2. Taking the time from a state read 70 ms before the write instead steps the clock back 74.3 ms, which is the prototype's 65–85 ms drift.
    • A paused computer (SIGSTOP): the write answered 504 stale: ["laptop"] after 2,002 ms, and the change stood. After SIGCONT, the next write was acknowledged in 2.3 ms.
    • A killed computer: 504 naming it after 1,002 ms. Once its follower restarted, the next write was acknowledged in 3.0 ms.
    • Forged acknowledgements made during a write's wait all left the laptop stale (504): another computer's token, a wrong token, no token, and the laptop's own token naming a version the clock hadn't had. The laptop's own token naming the current version, the control case, ended the wait.
    • Gateway restart (docker restart): both followers' next watches answered unarmed on a new boot id within 46 ms of the gateway coming up, and re-arming with a wait answered 200.
    • clear: with a wait, it answered 200 once both computers showed it unarmed.
    • No declared computers: a write with a wait answered at once, and the old bodies still work (set_time without wait_followers_s, clear with no body).
  • Remote providers (261462b6, the CI's Modal keys):

    • Setup: the branch's gateway image, built by the provider from its context, ran as the gateway provider runs it, on both modal (a container on the private network) and modal_vm (a container in the VM). It declared two computers, each followed through the gateway's public HTTPS tunnel: desktop from a Modal sandbox of its own, laptop from outside Modal.
    • Advances: 30 of 30 with a wait answered 200 and were acknowledged in order by both computers. A write took 147 ms at p50 on modal and 149 ms on modal_vm, about two round trips through the tunnel.
    • Long polls: across an 80 s idle stretch, every watch was held its full 25 s and answered 200, with no errors over the whole run. The tunnels don't cut them.
    • set_rate kept t0, and both computers picked up ×60.
    • A paused computer: 504 naming the laptop after 3.07 s, the change standing, then acknowledged in about 147 ms once resumed.
    • clear with a wait answered 200.
    • Other remote providers weren't run: this account has no credentials for them. FD-5487 covers them.
  • Integration on the same host (local sandboxes):

    • gateway_test.py's clock tests pass: test_gateway_clock_extension, test_gateway_clock_rate_accuracy, test_gateway_time_trigger_extension, test_gateway_autonomous_time_trigger and test_sync_env_clock_step.
    • The hook tests pass: test_advance_to_next_event_hook_local, test_next_env_event_hook_local, test_register_env_triggers_local and test_prompt_agent_notifications_local.
    • The Windows disk-recipe test that fails on macOS passes on Linux.
  • What the e2e caught: the entrypoint parsed CLOCK_FOLLOWER_TOKENS with the URL-map parser, so a gateway with declared computers didn't start (fixed in 91428dc1, with a unit test). It also caught gateway_test.py's exact unarmed-state assertion (updated in 071322a2).

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 4/5

The clock changes appear sound, but the existing repository requirement for a named precision constant still needs to be met before merging.

Fix All in CursorFindings

  1. P2 Time precision lacks a name ▶
Fix with agent prompt
### Issue 1
src/agent_env/env/gateway/clock_followers.py:undefined-92
`ClockFollowers.state()` uses an inline `3` to choose the precision of `seen_s_ago`. The repository requires magic numbers to be stored as class or instance variables with descriptive names. Move this value into a constant such as `SEEN_SECONDS_DECIMALS`. This repository requirement must be satisfied before merging.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

The gateway now publishes clock versions and lets computers follow changes through long polls.

  • Computers can follow gateway clock changes through long polls.
  • Rate changes keep the gateway clock at its current time.
  • The notes test app checks Docker Compose from Docker's apt repository.

Diagram

sequenceDiagram
    participant Client
    participant Gateway
    participant Computer
    Computer->>Gateway: Watch with shown version and read token
    Client->>Gateway: Change clock with wait_followers_s
    Gateway-->>Computer: New clock state
    Computer->>Computer: Apply clock state
    Computer->>Gateway: Watch naming applied version
    Gateway-->>Client: 200 when all have acknowledged, otherwise 504
Loading

Reviews (17) · Last reviewed commit: "Merge branch 'main' into edgararakelyan/..." · Reviewed by Greptile

…writes wait for them (FD-5471)

The gateway's clock gets one opaque version, a per-boot id and a count, that every
set_time, advance and clear changes. A computer follows it by long-polling
/clock/watch with the version it shows, which is its acknowledgement; the answer
carries the clock and the gateway's received and sent stamps. The gateway provider
declares the computers that follow it in CLOCK_FOLLOWER_TOKENS, each by its read
token (its control token for now), and a write given wait_followers_s waits for
each and answers 504 naming those behind, the change standing. set_time without a
virtual_time changes an armed clock's rate from its time at the write, so a rate
change no longer steps it back by a stale read. A restarted gateway is unarmed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale requested a review from a team as a code owner November 11, 2026 04:25
…t the URL map's (FD-5471)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread src/agent_env/env/gateway/clock.py Resolved
Comment thread src/agent_env/env/gateway/clock.py Resolved
Comment thread src/agent_env/env/gateway/clock.py Resolved
earakely-scale and others added 5 commits November 10, 2026 20:33
… late watch never moves its computer back (FD-5471)

set_time without virtual_time read the clock twice, for its t0 and for the new arm's anchor, losing the
moment between them at the old rate. An acknowledgement now only moves a computer's shown version
forward, a count too long to be one the clock had names nothing rather than failing the watch, and
the boot id's size is a named constant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…owers (FD-5471)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…needs a rate (FD-5471)

set_time given only virtual_seconds_per_real_second now changes an armed clock's rate the way advance
moves it: from its time at the write, keeping t0 and generation, so time triggers keep their marks and
their waits, which the change wakes to run at the new rate. set_time with neither a virtual_time nor a
rate is refused with the 400 it got before, instead of re-arming at rate 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…own method; the watch contract in the card (FD-5471)

- Every clock change goes through Gateway._clock_changed (triggers re-evaluated when time moved, held
  watches answered, the followers waited for), and the four write routes share _clock_write's body
  handling, which also gives set-time the JSON-object check the others had.
- A rate change is the clock extension's set_rate (PUT /clock/set-rate), so set-time takes a
  virtual_time as it always has.
- The route resolves a watch's follower from its token (ClockFollowers.follower_for), so the watch
  itself only acknowledges and holds.
- The card advertises the watch's request and its hold (hold_s, CLOCK_WATCH_HOLD_S) and bounds
  wait_followers_s (CLOCK_MAX_WAIT_FOLLOWERS_S).
- The clock's own tests move to clock_version_test.py, and the route tests wait for a follower to be
  seen rather than sleeping, with bounds a loaded runner can't miss.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…1-hb11-gateway-clock-one-version-a

# Conflicts:
#	AGENTS.md
#	src/agent_env/env/envs/_multi_computers.py
… (FD-5471)

math.isfinite raises on an int too large for a float, and the limit messages formatted the value as
one; only a float is checked for finiteness, and the messages name the limit alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale

Copy link
Copy Markdown
Collaborator Author

🤖 Automated Claude routine — Env Pod PR Context Review. This is a bot, not Edgar Arakelyan; not a human review.

Context from a sibling open PR rather than from this diff: #566 ("a step that runs an assistant over days of task time, with a simulated user") independently adds a set_rate to the same clock and to the same clock/v1 card. Neither branch is an ancestor of the other — both are off main — so neither diff shows the other half.

They disagree on the wire path for the same method name:

this PR (5d049b7) #566 (974a6ab)
card method set_rate → PUT /clock/set-rate (src/agent_env/env/gateway/constants.py:243) set_rate → PUT /clock/rate (constants.py:229)
route /clock/set-rate (gateway.py:328) /clock/rate (gateway.py:344)
request virtual_seconds_per_real_second + wait_followers_s virtual_seconds_per_real_second, minimum: 0
clock version self._count += 1 (clock.py:193) no version (clock.py:189)

A caller reads the endpoint out of the card, so whichever lands second settles what set_rate means for anything already written against the other path.

The row that won't show up as a merge conflict is the last one. #566 predates the opaque version introduced here, so its set_rate body is a clock write that leaves the version unchanged. If that body survives the rebase, a computer long-polling GET /clock/watch?after=<version> is never told the rate changed and wait_followers_s has nothing to wait on — the invariant in this PR's description ("every set_time, advance and clear changes it") would then cover three of the four writes rather than all of them.

Probably just agreeing on one path and keeping this PR's body for set_rate on whichever branch rebases last.


Generated by Claude Code

earakely-scale and others added 3 commits November 11, 2026 05:21
…tion; followers named alike (FD-5471)

- Gateway._change_clock applies a change and does what every one owes: it wakes the time triggers when
  the change kept the clock's generation (an advance or a rate change), so no route or later writer
  chooses it, then answers the held watches and waits for the followers, returning the followers still
  behind beside the state rather than in it.
- ClockFollowers keeps each follower's count alone (Clock.version_of formats it); count_of and
  parse_wait read plainly; the docstrings, card and errors say follower throughout, and the card calls
  state's followers block diagnostic.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tory, not a Launchpad PPA (FD-5471)

The computer app tests composed the notes app with fastfetch from a Launchpad PPA, so every one of them
failed whenever Launchpad was down, as it was on 2026-11-11 (about 85% uptime over 30 days). The app now
takes docker-compose-plugin from Docker's apt repository (download.docker.com, its armored key beside the
app), which Ubuntu doesn't ship either and which needs no other package, so the test still proves an
outside source and its key and checks a ~noble version and the binary's own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"""Each follower's last shown version and how long ago it last watched, for ``/clock/state``."""
now, clock = time.monotonic(), self._clock()
return {name: {"version": None if shown is None else clock.version_of(shown),
"seen_s_ago": round(now - self._seen[name], 3) if name in self._seen else None}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Time precision lacks a name

ClockFollowers.state() uses an inline 3 to choose the precision of seen_s_ago. The repository requires magic numbers to be stored as class or instance variables with descriptive names. Move this value into a constant such as SEEN_SECONDS_DECIMALS. This repository requirement must be satisfied before merging.

Rule Used: Store magic numbers as class or instance variables with descriptive names rather than using them inline in the code. (source)

Learned From
scaleapi/scaleapi#126388

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agent_env/env/gateway/clock_followers.py
Line: 92

Comment:
**Time precision lacks a name**

`ClockFollowers.state()` uses an inline `3` to choose the precision of `seen_s_ago`. The repository requires magic numbers to be stored as class or instance variables with descriptive names. Move this value into a constant such as `SEEN_SECONDS_DECIMALS`. This repository requirement must be satisfied before merging.

**Rule Used:** Store magic numbers as class or instance variables with descriptive names rather than using them inline in the code. ([source](https://app.greptile.com/-/switch-tenant?tenant=scale-ai&next=%2Fcustom-context%3Fmemory%3D002e0051-41ad-46c1-9098-47433c580150))

**Learned From**
[scaleapi/scaleapi#126388](https://github.com/scaleapi/scaleapi/pull/126388)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Cursor Fix in Claude Code Fix in Codex

@earakely-scale
earakely-scale merged commit 6344340 into main Nov 11, 2026
33 of 35 checks passed
@earakely-scale
earakely-scale deleted the edgararakelyan/fd-5471-hb11-gateway-clock-one-version-a branch November 11, 2026 23:39
earakely-scale added a commit that referenced this pull request Nov 12, 2026
#614's clock followers and M8.12's site names were added beside each other in the gateway's entrypoint and
constructor; both are kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant