Skip to content

feat(computers): agent-env's libfaketime for the computer's clock, gated on x86_64 and arm64 (FD-5474) - #619

Open
earakely-scale wants to merge 1 commit into
mainfrom
edgararakelyan/fd-5474-hb14-patched-libfaketime-for
Open

earakely-scale wants to merge 1 commit into
mainfrom
edgararakelyan/fd-5474-hb14-patched-libfaketime-for

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Part of FD-5420, the computer's own clock following the gateway's universe clock. This is FD-5474: the library every computer's session programs will load to read universe time. Every program on a computer reads its own clock, so it can't ask the gateway's get_time the way an MCP server does. Instead, agent-env's build of libfaketime answers those programs' clock reads and clock-time waits from a page a follower process keeps in step with the gateway. Wiring the library into the base computer (/etc/ld.so.preload, the follower, supervisord) is FD-5476.

What it is

src/agent_env/computers/clock/:

File What it is License
libfaketime.patch agent-env's changes to upstream 0.9.12's src/, applied with patch -p1 GPL-2.0-only
agentenv_clock_page.c settings read once at load, who is faked, the clock page's reader; included by the patched libfaketime.c GPL-2.0-only
agentenv_clock_waits.c universe-time reads and waits, the per-process helper thread; included the same way GPL-2.0-only
COPYING GPL-2.0, libfaketime's license
build.sh builds libfaketimeMT.so.1 from upstream's sha256-pinned tarball and the files above Apache-2.0
clockpage.py the clock page's writer, which the follower will use Apache-2.0
clock.conf the library's default settings Apache-2.0

What the library does:

  • Universe time: clock reads (CLOCK_REALTIME, _COARSE, _ALARM, TAI, gettimeofday, time) and waits for a set realtime clock time (clock_nanosleep TIMER_ABSTIME, realtime futex waits, sem_timedwait and sem_clockwait, realtime condvars, absolute realtime timerfds and POSIX timers). Each wait is planned in real time and planned again whenever the page changes.
  • Real time: the monotonic and boot clocks, and every relative sleep and timeout, at any rate.
  • Who: only the session user is faked. A program started with AGENTENV_REAL_TIME=1 stays real, and its children are faked.
  • Settings: read once as the library loads. Stock libfaketime's getenv raced programs' own setenv; that crashed Chromium on about one launch in five and hangs it outright in headless mode.

The patch and the engine files are the prototype's library, moved out of the two Python scripts that applied it as string replacements (about 900 lines of C inside a Python string). Their preprocessed source was byte-identical to the prototype's before the fix below.

A bug the gates found, fixed here

The gates found it on arm64: with the follower stalled 5 ms in a write, CLOCK_REALTIME stepped back up to 280 ns at a rate change, 3 to 7 times per 2,000 writes.

  • Cause: a clock read took its real time before copying the page. A reader that waited out a write therefore paired a time from before the write with the page the write made. A write is continuous with the page it replaces only at its own real time.
  • Fix: the read now copies the page, then reads the real time, and keeps the pair only if the page is still the one it copied (ft_snap_at). The real time is ordered before that check: a dependent load and dmb ishld on aarch64, lfence on x86_64.
  • Result: 0 backsteps in three runs of about 90 million reads each.
  • Cost: a second vDSO read per clock read.

Where it lives

It's at computers/clock/, beside the bundle's other shared recipes (disk/, vm_bundle/), not at artifacts/ubuntu-24.04/clock/ as the issue said. Inside the base computer's folder it would change the base's build context, and with it the image's tag, though nothing loads the library yet. FD-5476 brings it into the base.

Licensing

The patch and the two engine files are derived from libfaketime, so they carry GPL-2.0-only and upstream's COPYING sits beside them. Everything else in the folder and the repo stays Apache-2.0, and the package's license metadata is unchanged.

THIRD_PARTY_NOTICES.md gets a libfaketime section. I also changed its opening sentence, which said no third-party source is incorporated in the repository; that's no longer true for these three files. That sentence is legal wording, so please check the change. Whether the wheel's license metadata should also list GPL-2.0 for these files is a question for legal.

Tests

tst/clock (make clock-test, .github/workflows/clock.yml on ubuntu-24.04 and ubuntu-24.04-arm, not a required check):

  • Setup: it builds this library and stock 0.9.12 in an Ubuntu 24.04 image, the base computer's own base, and runs each gate in a container. Chrome's archive (the browsers bundle's pin, read from chromium.py) and upstream's tarball are fetched once into a cache, checked by sha256, and kept between CI runs.

The gates:

  • Waits: all 64 wait APIs in C, Python, Java and Node at rates 1, 2 and 60, and frozen with a +1 h advance. Elapsed waits stay real (1.000–1.021 s); clock-time waits follow the rate (0.50 s at ×2, 0.017–0.106 s at ×60) and end at the advance.
  • Rate change and set_time back: ×1 to ×60 ends clock-time waits at 1.15 s, and a set_time 182 days back leaves them waiting. Three single-call condvar waits return once at the change, the spurious return POSIX allows and the design records.
  • Jumps: 30 forward jumps of 8 h, a day or a week. Waits inside the jump end within 1.5 ms of it (the mutex wait within its 100 ms slice). Waits past it are never early. Relative waits stay 3.00 s. Monotonic time never steps back.
  • Stress: realtime condvar handoffs and sem_timedwait posts, with the page changing under them, lose nothing; every handoff is under 2 ms.
  • Stalled writer: 2,000 writes stalled 5 ms each: realtime and monotonic never step back.
  • Torn reads: a black-box test of the library itself. The page is rewritten as fast as possible between two universes while 4 threads read CLOCK_REALTIME: 0 torn reads in about 319 million on arm64.
  • Who is faked, and the marker: only the session user's processes follow the page. A marked program stays real, loses the marker, and its children are faked.
  • Environment race: setenv racing clock reads and timed waits, 3 runs of 10 s each: no crash.
  • Chromium: launched directly 60 times with this library, each launch printing its page and exiting 0; stock 0.9.12 fails all 10 of its launches (it hangs). The whole gate runs in about 20–24 s, under its 30 s budget.

Results:

  • arm64 (this Mac's Docker): all 16 tests pass.
  • x86_64 and arm64 EC2 hosts, and CI: running; results below once in.

Docs

The docs site should cover the computer's clock: what follows universe time and what stays real, who is faked, AGENTENV_REAL_TIME, and clock.conf. That page belongs with FD-5476, when computers load the library.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 0/5

This PR needs fixes before merge because cancellation, timer tracking, fallback waits, and interrupted writes can break programs using the library.

Fix All in CursorFindings

  1. P1 Canceled waits leave stale pointers ▶
  2. P1 Semaphore waiters ignore cancellation ▶
  3. P1 Fallback waits use wrong deadlines ▶
  4. P1 Timer records outlive their descriptors ▶
  5. P1 Restarted writers cannot recover ▶
  6. P2 Documented overrides do nothing ▶
  7. P2 Wait errors pass the gates ▶
Fix with agent prompt
### Issue 1
src/agent_env/computers/clock/agentenv_clock_waits.c:667-670
`ft_u_cond_wait` clears its table entry only after `real_pthread_cond_clockwait` returns. If the thread is canceled during that call, the entry stays active. The caller can then destroy or free the condition variable, but the helper still broadcasts to its saved pointer on clock changes. Add a cancellation cleanup handler that removes the entry before the caller's cleanup runs.

### Issue 2
src/agent_env/computers/clock/agentenv_clock_waits.c:602-606
`ft_u_sem_wait` replaces the cancelable semaphore wait with raw futex calls, without checking for pending thread cancellation. With `futex_waitv` available, canceling a thread blocked in `sem_timedwait` or `sem_clockwait` no longer reliably stops it. A frozen clock can leave shutdown waiting forever in `pthread_join`. Preserve cancellation behavior and remove the semaphore's waiter count when cancellation runs.

### Issue 3
src/agent_env/computers/clock/agentenv_clock_waits.c:539
If the kernel lacks `futex_waitv` or blocks it, `ft_waitv_missing` makes later calls leave `ft_u_futex_rt` without translating their deadlines. The first fallback translates once but also stops watching clock changes. With universe time 182 days behind real time, later waits time out immediately; with it ahead, they wait far too long. Keep translating deadlines and checking page changes in the fallback.

### Issue 4
src/agent_env/computers/clock/agentenv_clock_waits.c:252-255
Timer tracking follows a descriptor number, not the open timer. `dup` does not copy `ft_tfd_clock`, and `close` does not remove the tracked entry. Duplicating an armed timerfd and closing the original leaves the helper updating the closed descriptor while the surviving timer misses clock changes. Reusing that number can also make the helper update another timer. Track duplication and closure along with timer creation.

### Issue 5
src/agent_env/computers/clock/clockpage.py:101-105
`write` assumes the existing `seq` is even. If the writer dies after publishing an odd value, the next writer finishes with `seq + 2`, which is still odd. Every later write stays odd too, so readers never accept the recovered writer's updates. Recover an interrupted page before writing, then publish an even sequence with a matching check without replacing the mapped file.

### Issue 6
src/agent_env/computers/clock/agentenv_clock_page.c:76-78
`clock.conf` says the environment can override settings, but `ft_conf_load` returns immediately after reading the file. Once that file is installed, environment settings such as `AGENTENV_CLOCK_PAGE` and `AGENTENV_WAIT_SLICE_MS` are silently ignored. Either merge environment overrides into the startup snapshot or change the documentation to describe file-only settings.

### Issue 7
tst/clock/waits_test.py:19-23
The wait gates collect return codes but mostly check only elapsed time. The jump gate never checks the reported return codes. A wait that ends at the expected time with an error can therefore pass. Assert the expected result for each API, including timerfd reads and semaphore waits, alongside the timing checks.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds a patched libfaketime build and shared clock page for computer programs.

  • Session programs can read universe time through a shared clock page.
  • A clock test gate runs on amd64 and arm64.

Acknowledged behavior:

  • earakely-scale explicitly describes continuing small backward clock changes as intentional.
  • earakely-scale explicitly accepts spurious returns from single-call condition-variable waits when clock changes move their deadlines.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Writer["Clock page writer"] --> Page["Shared clock page"]
  Page --> Reads["Program clock reads"]
  Page --> Helper["Per-process helper"]
  Helper --> Timers["Rearm absolute timers"]
  Helper --> Conditions["Wake condition-variable waits"]
  Page --> Waits["Recheck absolute waits"]
  Kernel["Real kernel clocks"] --> Relative["Monotonic reads and relative waits"]
Loading

Reviews (1) · Last reviewed commit: "feat(computers): agent-env's libfaketime..." · Reviewed by Greptile

…ted on x86_64 and arm64 (FD-5474)

src/agent_env/computers/clock is the library every computer's session programs will load to read universe
time: upstream libfaketime 0.9.12 from its sha256-pinned tarball with agent-env's changes, as a small
libfaketime.patch and two engine files it includes (agentenv_clock_page.c, agentenv_clock_waits.c;
GPL-2.0-only, as libfaketime is), built by build.sh, with the page writer the follower uses (clockpage.py)
and the library's default settings (clock.conf). Clock reads and waits for a set realtime clock time follow
the universe; monotonic and boot clocks and every relative wait stay real; only the session user is faked;
settings are read once at load.

A clock read now copies the page before reading the real time and keeps the pair only if the page is still
the one copied: a reader that waited out a write paired a time from before it with the page it wrote, which
stepped CLOCK_REALTIME back by up to ~280 ns at a rate change.

tst/clock (make clock-test, .github/workflows/clock.yml on x86_64 and arm64) builds it and stock 0.9.12 in an
Ubuntu 24.04 image and gates it: every wait API in C, Python, Java and Node at several rates, a rate change,
a set_time back and 30 jumps, condvar and semaphore stress, a stalled writer, torn reads, who is faked, the
marker, the environment race and Chromium launched directly 60 times (stock 0.9.12 as the positive control),
in under 30 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale requested a review from a team as a code owner November 11, 2026 17:20
Comment on lines +667 to +670
r = real_pthread_cond_clockwait(cond, mutex, CLOCK_REALTIME, &end);
ft_tab_lock(&old);
woken = w->woken;
w->used = 0;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Canceled waits leave stale pointers

ft_u_cond_wait clears its table entry only after real_pthread_cond_clockwait returns. If the thread is canceled during that call, the entry stays active. The caller can then destroy or free the condition variable, but the helper still broadcasts to its saved pointer on clock changes. Add a cancellation cleanup handler that removes the entry before the caller's cleanup runs.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agent_env/computers/clock/agentenv_clock_waits.c
Line: 667-670

Comment:
**Canceled waits leave stale pointers**

`ft_u_cond_wait` clears its table entry only after `real_pthread_cond_clockwait` returns. If the thread is canceled during that call, the entry stays active. The caller can then destroy or free the condition variable, but the helper still broadcasts to its saved pointer on clock changes. Add a cancellation cleanup handler that removes the entry before the caller's cleanup runs.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Cursor Fix in Claude Code Fix in Codex

Comment on lines +602 to +606
k = ft_wait2((uint32_t *) &sem->data, 0, sem->private_flag == 0, (uint32_t) s.seq, ft_real_of(&s, due));
if (k == FT_W_INTR)
{
err = EINTR;
break;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Semaphore waiters ignore cancellation

ft_u_sem_wait replaces the cancelable semaphore wait with raw futex calls, without checking for pending thread cancellation. With futex_waitv available, canceling a thread blocked in sem_timedwait or sem_clockwait no longer reliably stops it. A frozen clock can leave shutdown waiting forever in pthread_join. Preserve cancellation behavior and remove the semaphore's waiter count when cancellation runs.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agent_env/computers/clock/agentenv_clock_waits.c
Line: 602-606

Comment:
**Semaphore waiters ignore cancellation**

`ft_u_sem_wait` replaces the cancelable semaphore wait with raw futex calls, without checking for pending thread cancellation. With `futex_waitv` available, canceling a thread blocked in `sem_timedwait` or `sem_clockwait` no longer reliably stops it. A frozen clock can leave shutdown waiting forever in `pthread_join`. Preserve cancellation behavior and remove the semaphore's waiter count when cancellation runs.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Cursor Fix in Claude Code Fix in Codex

struct ft_clock_page s;
int64_t due = ft_ns(abs);
*handled = false;
if (ft_waitv_missing || !ft_snap(&s)) return 0;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Fallback waits use wrong deadlines

If the kernel lacks futex_waitv or blocks it, ft_waitv_missing makes later calls leave ft_u_futex_rt without translating their deadlines. The first fallback translates once but also stops watching clock changes. With universe time 182 days behind real time, later waits time out immediately; with it ahead, they wait far too long. Keep translating deadlines and checking page changes in the fallback.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agent_env/computers/clock/agentenv_clock_waits.c
Line: 539

Comment:
**Fallback waits use wrong deadlines**

If the kernel lacks `futex_waitv` or blocks it, `ft_waitv_missing` makes later calls leave `ft_u_futex_rt` without translating their deadlines. The first fallback translates once but also stops watching clock changes. With universe time 182 days behind real time, later waits time out immediately; with it ahead, they wait far too long. Keep translating deadlines and checking page changes in the fallback.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Cursor Fix in Claude Code Fix in Codex

Comment on lines +252 to +255
static struct ft_timer *ft_timer_find_fd(int fd)
{
for (int i = 0; i < ft_timers_top; i++)
if (ft_timers[i].kind == 1 && ft_timers[i].fd == fd) return &ft_timers[i];

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Timer records outlive their descriptors

Timer tracking follows a descriptor number, not the open timer. dup does not copy ft_tfd_clock, and close does not remove the tracked entry. Duplicating an armed timerfd and closing the original leaves the helper updating the closed descriptor while the surviving timer misses clock changes. Reusing that number can also make the helper update another timer. Track duplication and closure along with timer creation.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agent_env/computers/clock/agentenv_clock_waits.c
Line: 252-255

Comment:
**Timer records outlive their descriptors**

Timer tracking follows a descriptor number, not the open timer. `dup` does not copy `ft_tfd_clock`, and `close` does not remove the tracked entry. Duplicating an armed timerfd and closing the original leaves the helper updating the closed descriptor while the surviving timer misses clock changes. Reusing that number can also make the helper update another timer. Track duplication and closure along with timer creation.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Cursor Fix in Claude Code Fix in Codex

Comment on lines +101 to +105
body = LAYOUT.pack(MAGIC, VERSION, seq + 2, universe_ns, anchor_ns, rate, armed, 0, generation,
check(seq + 2, universe_ns, anchor_ns, rate, armed, generation, now_mono, now_mono),
now_mono, now_mono)
m[16:] = body[16:]
m[8:16] = struct.pack("<Q", seq + 2)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Restarted writers cannot recover

write assumes the existing seq is even. If the writer dies after publishing an odd value, the next writer finishes with seq + 2, which is still odd. Every later write stays odd too, so readers never accept the recovered writer's updates. Recover an interrupted page before writing, then publish an even sequence with a matching check without replacing the mapped file.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agent_env/computers/clock/clockpage.py
Line: 101-105

Comment:
**Restarted writers cannot recover**

`write` assumes the existing `seq` is even. If the writer dies after publishing an odd value, the next writer finishes with `seq + 2`, which is still odd. Every later write stays odd too, so readers never accept the recovered writer's updates. Recover an interrupted page before writing, then publish an even sequence with a matching check without replacing the mapped file.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Cursor Fix in Claude Code Fix in Codex

Comment on lines +76 to +78
return;
}
for (char **e = environ; e != NULL && *e != NULL; e++)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Documented overrides do nothing

clock.conf says the environment can override settings, but ft_conf_load returns immediately after reading the file. Once that file is installed, environment settings such as AGENTENV_CLOCK_PAGE and AGENTENV_WAIT_SLICE_MS are silently ignored. Either merge environment overrides into the startup snapshot or change the documentation to describe file-only settings.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agent_env/computers/clock/agentenv_clock_page.c
Line: 76-78

Comment:
**Documented overrides do nothing**

`clock.conf` says the environment can override settings, but `ft_conf_load` returns immediately after reading the file. Once that file is installed, environment settings such as `AGENTENV_CLOCK_PAGE` and `AGENTENV_WAIT_SLICE_MS` are silently ignored. Either merge environment overrides into the startup snapshot or change the documentation to describe file-only settings.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Cursor Fix in Claude Code Fix in Codex

Comment thread tst/clock/waits_test.py
Comment on lines +19 to +23
elapsed = {r["api"]: r["real"] for r in results if r["class"] == "elapsed"}
clock = {r["api"]: r["real"] for r in results if r["class"] == "clock_time"}
assert len(clock) == 19
assert {api: s for api, s in elapsed.items() if not 0.99 <= s <= 1.3} == {}
assert {api: s for api, s in clock.items() if not 1 / rate - 0.01 <= s <= 1 / rate + 0.15} == {}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Wait errors pass the gates

The wait gates collect return codes but mostly check only elapsed time. The jump gate never checks the reported return codes. A wait that ends at the expected time with an error can therefore pass. Assert the expected result for each API, including timerfd reads and semaphore waits, alongside the timing checks.

Prompt To Fix With AI
This is a comment left during a code review.
Path: tst/clock/waits_test.py
Line: 19-23

Comment:
**Wait errors pass the gates**

The wait gates collect return codes but mostly check only elapsed time. The jump gate never checks the reported return codes. A wait that ends at the expected time with an error can therefore pass. Assert the expected result for each API, including timerfd reads and semaphore waits, alongside the timing checks.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Cursor Fix in Claude Code Fix in Codex

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant