Repository navigation
Documentation claims say only what their code, tests and records support - #186
SamMausberg wants to merge 5 commits into
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ccd6d9b130
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| ## Tests and checks | ||
|
|
||
| Run `make lint test` before and after you touch code, or `make lint test JOBS=4` for an agent on a shared machine, as [AGENTS.md](AGENTS.md#development) says. Run `make native` when you touch the runtime or the lowering, `make lean` when you touch a rule the calculus models, and `make gpu` or `make embedded` where the hardware is present. `make gpu`, `make tune-device` and `make calibrate-device` are the only targets that run code on a device. They run one at a time, and only the owner of the machine starts them. | ||
| Run `make lint test` before and after you touch code, or `make lint test JOBS=4` for an agent on a shared machine, as [AGENTS.md](AGENTS.md#development) says. Run `make native` when you touch the runtime or the lowering, together with the native tests under both compilers and the sanitizer that bites, since `make native` sanitizes the clang++ build alone. When you touch a rule a Lean model covers, regenerate the certificates with `tools/checks/export_lean_certificates.py` and run `make lean`, which only checks them and rebuilds `proofs/`. Run and `make gpu` or `make embedded` where the hardware is present. Code runs on a device only through `make gpu`, `make tune-device`, `make calibrate-device`, or a narrow run of the device tests a change touches. Each holds the lock `/tmp/cairn-gpu.lock`, so runs queue one behind another, and where the GPU also drives a display only the owner of the machine starts them ([AGENTS.md](AGENTS.md#development)). |
There was a problem hiding this comment.
Invoke the certificate exporter through Python
When a contributor follows this workflow after changing a Lean-modeled rule, running tools/checks/export_lean_certificates.py directly fails with Permission denied because the script is committed with mode 100644. Since make lean invokes it only with --check, that target cannot perform the required regeneration either; document python3 tools/checks/export_lean_certificates.py or make the script executable.
AGENTS.md reference: AGENTS.md:L126-L126
Useful? React with 👍 / 👎.
Summary
Every documentation page now says only what its code, tests or records under
evidence/support. About eighty sentences across the docs, CONTRIBUTING.md, SECURITY.md and the example, tool, benchmark and demo READMEs claimed more than that: a proof where there is a Lean model or a test, "never" or "every" where a test covers some cases, a speedup with no record, a count that has drifted, or a rule the code does not have. Each now states the claim at the strength its evidence carries, and names that evidence. Prose is otherwise unchanged; the voice pass comes in later pull requests, one page or two at a time.Motivation
The design report's documentation findings named five claims beyond their evidence: a failed guard stopping a program inside a GPU kernel (devices.md, internals.md, numerics.md), "every candidate the checker accepts is correct" (tools.md) and "an accepted plan needs no test to be correct" (agents.md) for a plan rule with no Lean model, language.md's "leaving a guard out never changes what a program does", examples.md's 10x and 85x with no record, and concurrency.md's "proved in Lean" for a device store that Lean does not cover. It asked for the rest to be found. Five read-only audits went through every page against the compiler, the runtime, the tests and
evidence/, and every finding below was checked again before it was changed. The owner asked for this directly; no roadmap issue covers it.Changes
modelledspelledmodeledas the repository mostly spells it (110 to 19), and one definition of a guard used on every page that defines one ("a check the compiler writes before an operation that could fail at run time"), with "traps" and "aborts" used one way.compact, the lane pool, the tensor-core tests' real tolerance, and the GH200 runs where a sentence said a thing ran once or only on the RTX 5070 Ti.docs/project/capability_matrix.json) for one count, regenerated bytools/release/capability_matrix.py.parallelregions" to three, in the Compute Sanitizer sentence. Nothing else in the README changes.make nativeandmake leando and do not do; auto-merge set by the author.gh api repos/SamMausberg/cairn/private-vulnerability-reportinggives{"enabled":false}), so the page no longer offers it.src/cairn/std/draw.cairn,sort.cairn): two module comments, whichmake docsturns intodocs/std/draw.mdanddocs/std/sort.md. The code is unchanged.Behaviour and compatibility
none: documentation and two comments. No program is accepted, refused, emitted or printed differently. One heading is renamed, the guide's "9. Lanes are race free by construction" to "9. Lanes cannot race"; nothing linked to it, and
test_tour.pyfinds its twelve programs.Testing
On this branch merged with main at 4d495bf, on the aarch64 GH200 host, with no device run:
Before the merge the same branch ran the tests that read any page it changes: those above plus test_std, test_projects, test_wordfreq, test_layouts, test_expressiveness, test_review_agent_and_device_layer, test_review_tools, test_ai_bench, test_ai_eval, test_std_system, test_focused, test_release, test_language and test_github_files: 838 passed, 1 skipped. The findings were checked with
cairn check,cairn validate,cairn emit,cairn runanddemos/repair/run.pyon this host; the programs are in the session's scratch directory. The fast suite was not run, since no code changed.Evidence and claims
Withdrawn or narrowed, each with what backs the new wording:
--emulate, no test that makes a guard fail on a GPU has run since 0.8.3 (both recorded sessions skipped the three trap tests), and the one guard that failed on the GH200 was in a kernel NVVM 7.0.1 miscompiled (evidence/v1_2/gpu_gh200). The README's wording is followed. devices.md's region paragraph no longer says the process has "already aborted" when a region returns, which held runs contradict.use gselects an implementation, which could change a result, so tune takes one only once validated.compact(concurrency.md): the seventeen certificates are about the host loop; the device store at the exclusive sum of the flags has no certificate and is tested.--genericsrecord (172 and 84 functions are now 190 and 102), the expression-visit limit (3,200,000,E-AST-LIMIT; agents.md had 200000), the device-perf timing ranges (466 to 591 us, not 550; 338 to 348 us throughcq_, against 334 to 340 us), the profile's median errors in place of "within a quarter", the Compute Sanitizer program's three regions (README and matrix),size's 5411 bytes labelled as the v0.8.0 record's, the trust table's 15,500 compiler lines plus the verifier modules it imports and a row for the build driver.places.py), a borrow's lifetime (a fragment), storage floats (the model comparison runs without sanitizers; nothing on a GPU), the math builtins' bits (clang++ and g++ tested; no device run), the tensor-core device tests (twice the bound against the reference loop, not the bound of the exact sum; the matmul app checks the bound).cairn check --genericscertifies a template whose instance can still beE-EFFECT-ORDER(a trait member without a ceiling on both sides of an operator); aconstfolds in exact arithmetic, so0 - 1 + 1is au80; alinearrecord is not copyable;std.drawcan trap on i64 overflow; a mapSlotfrom another map is only a number;std.sysis not the only module that binds C symbols; the operand-order gapkeep(total(c), c)(internals.md) is now named where memory.md and "Safety and trust" said there was none.cairn verify's native replay runs only where it can and otherwise leaves Z3's verdict (cairn verify keeps each path's storage for an owner the paths move, and runs every smt-equivalent verdict natively before giving it #179), the search could not reuse a check (An edit of one function's body is checked from the walk the last check kept, and a differential test holds it to a whole check #101, An edited body's check puts back the instances, layouts and implementations the walk made first there, so an edit that makes them again is not checked whole #114), the device examples and the numeric demo ran again on a GH200 (evidence/v1_2/gpu_gh200 records the suite's device tests on a GH200, and README and the capability matrix say what ran there #172), and CONTRIBUTING's GPU and language-change rules match AGENTS.md.bench/gpucompares after timing and records a mismatch,bench/devicerecordsrelative_errorand refuses nothing,paired.pyandcodegen_only.pywriteresults/native/themselves, the suite's extra arms are listed, the repair demo'sclampfails only whenhi < loandx < lo, the implement demo's float case shrinks to n = 2048, and each demo says what its test checks.No claim is added beyond its record.
Not changed here, for the pull requests that own them: the capability matrix's device rows and the devices.md and numerics.md sentences that name only the RTX 5070 Ti where the GH200 also ran them, which the device-record work is updating; and roadmap.md, which a later pull request brings up to date.
Found and not fixed, since they are code:
E-AST-LIMIT's message and thelimitsrule card still say 200000 (checking.py:671,teaching.py:86);cairn check --genericsmisses the operand-order case above.Performance
none
Checklist
make editors,make docs):make docsregenerateddocs/std/draw.md,docs/std/sort.mdand the capability matrix