chore(tests): untrack generated evidence, guard against its return (#1610) by drsnuggles8 · Pull Request #1636 · drsnuggles8/OloEngineBase

drsnuggles8 · 2026-10-18T14:01:34Z

Closes #1610.

This removes generated verification output from the tracked tree and adds a pre-commit guard that stops it from coming back. It keeps every golden and fixture a test reads, and every curated doc asset. It measures, but does not rewrite, retained history: that needs its own decision (plan in docs/analysis/repo-history-remediation-1610.md).

Before and after

	Before (`0cc2cd225`)	After
Tracked files	9,440	6,566
Tracked bytes (logical)	2,192.7 MB	1,055.6 MB
`OloEditor/assets/tests/`	2,684 files, 975.6 MB	203 files, 110.5 MB
`docs/testing/evidence/`	610 files, 300.5 MB	212 files, 28.1 MB

All removals are git rm --cached: every worktree keeps its local copies.

Inventory and classification

Family	Bytes before	Class	Reader / generator	Outcome
`visual/` goldens (Water, Atmosphere, Drift, FoliageWind/Interaction incl. `amd/`, DDGI, GroomPreview, MeshVisibility, Morph, Transmission, GPUScene*, ImportedCorpus)	109 MB, 189 files	required golden	the suite named beside each file in the allow-list	kept
`visual/` evidence captures, `*-live/` and `unified-layouts-1181/` campaigns, `vegetation-experiment/`	~865 MB, ~2,450 files	run output	written by evidence tests or captured live; nothing reads them	untracked
`visual/` opt-in records `Dog_Cost`, `Dog_Lighting`, `GroomAnimals_Cost` (.txt)	<0.1 MB	curated record	written only with `OLO_*_EXPORT=1`; cited by guides	kept, allow-listed
`visual/` `DigitalHuman_Timing`, `EyeCornea_Timing`, `HeadOral_Timing` (.txt)	<0.1 MB	run output	rewritten on every GPU run	untracked; guides quote the last figures
`visual/Dog_Cost_Matrix.txt`	0.2 MB	curated record	assembled by hand from opt-in runs	moved to `docs/analysis/dog-cost-matrix.txt`
`exports/` (12 reports), `memory/` (5 reports)	0.06 MB	run output	`RenderTargetExportEvidence`, `RendererMemoryAccountingEvidence`	untracked
`golden/`, `splat/`, `state-machine/`, `imported-corpus/`	1.2 MB	fixtures / goldens	GoldenImageTests, GaussianSplat*, state-machine harness, ImportedCorpus	kept
`docs/testing/evidence/` raw campaign data that no doc links (#1338, #1259, #1593, #1594, #1554, #1211, #1392)	272 MB, 404 files	run output	benchmark/live campaigns	untracked; write-ups say how to regenerate and how to reach `0cc2cd225`
`docs/testing/evidence/` files a doc links, `docs/reviews/dog-breeds-1558/` (approval galleries), `docs/images/`, `docs/analysis/foliage-cost-baseline-1391/` (reproducible baseline), `docs/analysis/evidence/*.json`	~70 MB	curated	the linking doc, named in the allow-list	kept
`*-live/` and `unified-layouts-1181/` README write-ups	<0.1 MB	measurement summary	—	moved to `docs/testing/evidence/`

How the golden set was found: each golden-reading test case was run alone against an empty visual/, also under --olo-golden-vendor=amd, with each reported Missing golden file restored and the failing cases re-run until nothing was missing (13 rounds, 110 → 189 files). No reader writes a missing golden instead of failing, and none gates its comparison on the file existing.

Recurrence guard

.gitignore ignores every file in visual/, exports/ and memory/, and every non-Markdown file under docs/testing/evidence/. The directories themselves are re-included, so git add on a tracked golden exits 0. This replaces the per-campaign globs from VSM: terrain, voxel and foliage casters cast no shadow while Virtual Shadow Maps are on #1524, Deferred MSAA: every late G-Buffer writer resolves all seven attachments again (3x 91 MB per frame) #1553 and Digital-human close-up: the scan head's hair reads as a grey veil, below the film-quality bar #1593.
The generated-evidence pre-commit hook (scripts/check_generated_evidence.py) checks the index against scripts/generated-evidence-allowlist.txt. It rejects four things: an unlisted file under OloEditor/assets/tests/, an unlisted non-Markdown file under docs/, a tracked file that the repository's .gitignore excludes (a force-add; global excludes are not consulted), and an allow-list line that has no reader or allows nothing.
Its built-in fixtures run on every invocation. A capture, a render-target export report, a force-added replay scene, campaign data and a motion capture under docs/ must be rejected. A golden, a fixture, a curated doc image and prose must pass. Eight deliberate mutations of the checker each fail the fixtures.
Live demonstration: git add of a real capture (DogBlink_GL_Forward.png) is refused as ignored. git add -f makes the hook report it. A modified golden stages and passes.
The task loop no longer requires an artefact-backed cell's proof to be a committed PNG. The matrix row cites the test, the file it wrote and the measurement. Guide: docs/agent-rules/generated-evidence-stays-out-of-git.md.

History (measured, not rewritten)

GitHub: 2 branches, 0 tags and 954 refs/pull/* refs. A fresh master clone packs to 3.25 GiB, plus LFS.
Dropping every blob that ever lived in these output locations (and is not a current golden or curated file) would take a master pack from 3.26 GiB to 1.08 GiB (−67%). That is 9,568 blobs, 1,978 MiB of them from visual/, and the VSM: terrain, voxel and foliage casters cast no shadow while Virtual Shadow Maps are on #1524 shadow-family campaign alone is ~480 MB.
refs/pull/* keep those blobs on GitHub's side whatever happens to the branches. The 275 evidence-image links in PR descriptions are all SHA-pinned.
The recommendation (backup, rewrite by blob ID, preserving history-only captures first, GitHub GC, re-cloning every worktree) is in docs/analysis/repo-history-remediation-1610.md. Not executed; awaiting a decision.

Coordination

The sibling worktrees for #1607/#1608 and #1592 may write new evidence PNGs under visual/. Those are now ignored. If either sibling merges first with a modification to a PNG this branch untracks, resolve the modify/delete conflict by keeping the delete. A new golden they add needs git add -f plus an allow-list line, or the hook fails.

Verification

Axis	Cells	Evidence	Result
Golden/fixture consumers, fresh checkout	59 golden-reader cases (24 suites)	detached worktree at `6af78a186`, `OloEngine-Tests` built there (its compiled-in editor root points at the fresh tree)	59/59 pass, 0 missing-golden messages. Later commits add only 3 opt-in `.txt` records under `assets/tests/`
AMD vendor goldens, fresh checkout	10 cases, `--olo-golden-vendor=amd`	same	8 pass. 2 fail on SSIM 0.93–0.96 against a 0.985 floor (AMD set on an NVIDIA box). Identical failures with every original file present, so not caused by this change. 0 missing
Asset/scene/fixture consumers, fresh checkout	GoldenImageTests, GoldenBaselineAudit, GaussianSplat, StateMachine*, AssetSceneLoad, AssetContentValidity, AssetBinaryIntegrity, VulkanShaderPipeline, VulkanPassSuite: 254 cases	same	251 pass. 3 skip for reasons unrelated to files (retired by #906, a Vulkan GPU-timing gate, a command-line-only replay)
Regeneration, one test per removed family	`visual/`: WaterShore + GraphOwnedExports; `exports/`: `RenderTargetExportEvidence.EveryCellReportsItsCopiesBytesTimeAndPeak`; `memory/`: `RendererMemoryAccountingEvidence.EveryPathMsaaAndScaleCellPublishesCapacityVersusDemand`	same fresh checkout	All pass. They wrote 20 + 12 + 5 files, and `git status --short` stayed empty. `docs/testing/evidence/` campaigns are live/benchmark runs that no unit test regenerates; their write-ups name the command
Guard	Windows local	`pre-commit run generated-evidence --all-files` and `docs-consistency`	Passed. CI's Linux pre-commit job runs the same hooks
Scene save (found-and-fixed below)	21 suites that save scenes	Release `OloEngine-Tests`; `OloEditor` and `OloServer` built	245 pass, 2 skip (opt-in export flag, no symlink privilege), no unused-result warning from `Serialize`
Backend / render path	—	not reachable: no renderer code changed	n/a

Found and fixed

a1826543d fix(gitignore): .gitignore ignored itself, and !OloEngine//vendor/CMakeLists.txt had a doubled slash, so it never matched. Found by the guard's tracked-but-ignored rule on the real tree.
6af78a186 fix(gitignore): with visual/ ignored as a directory, git add on a modified golden exited 1. Found by staging a golden during verification.
9d9e2b1ae, e7e7c96bf, c6c7819fc, ba5692bb6 fix(tests): RenderTargetExportEvidence.DeferredResolveMatrix saved a scene into exports/ without creating it. A fresh clone no longer has that directory, and only cell order hid the problem. Its read-back could also see a previous run's file. Found by self-review. Verified: all 12 cases pass with exports/ removed.
b28e3a5c4, 6977f6030 fix(scene): SceneSerializer::Serialize wrote through an unchecked stream and returned void, so autosave, the server's save command and scene-asset saves lost data silently, and a failed write truncated the previous save. It now writes atomically (unique temp file + rename), returns [[nodiscard]] bool and logs failures, and every caller handles the result. Found by self-review of the export test. Pinned by SceneRenderSettingsActivationTest.SerializeReportsAFileItCannotWrite.

Filed, not fixed

AssetSerializer::Serialize cannot report a failed write #1635 AssetSerializer::Serialize cannot report a failed write — too big: the void virtual has 32 overrides in AssetSerializer.h alone. SceneAssetSerializer now logs the failure, but cannot return it.

Review guide

Where I'd look hardest

scripts/generated-evidence-allowlist.txt: the 189 golden lines are the measured set. A golden the discovery missed would show up only as a Missing golden on the next GPU run, because PR CI has no GPU.
OloEngine/src/OloEngine/Scene/SceneSerializer.cpp Serialize: the atomic write. Rename over an open file can fail on Windows where an in-place write used to succeed. That is the same trade-off the engine's other atomic writers make.
.gitignore evidence block: the ** plus !**/ idiom (files ignored, directories re-included) is what keeps git add <golden> at exit 0.

What I verified, and how: a fresh checkout built and run in its own worktree (golden readers, AMD pass, fixture/asset consumers, regeneration with git status clean), the guard's fixtures plus eight mutations, and a live force-add rejection.

Least confident about: the decision not to move goldens into a dedicated tracked directory, which would remove the force-add and the per-file list. I kept them in place because GoldenBaselineAuditTest audits every PNG under golden/ trees by name, and the AMD workflows upload visual/amd/**. Moving them is a test-infrastructure change beyond this issue's "tree + guard" scope.

🤖 Generated with Claude Code

`.gitignore` listed itself, and the negation meant to keep OloEngine/vendor/CMakeLists.txt had a doubled slash (`!OloEngine//vendor/CMakeLists.txt`), so it never matched and the one tracked vendor file was ignored. Both files stayed tracked only because they were added before the rules. Found by the generated-evidence guard's tracked-but-ignored check (#1610). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…urated docs Evidence tests and live campaigns had committed their run output beside the goldens: OloEditor/assets/tests/visual/ held 2,656 files (974 MB), of which 189 are goldens a test reads. docs/testing/evidence/ held 610 files (300 MB), mostly unlinked raw campaign data. Every capture regenerates on each run, so a tracked copy goes stale and each edit adds a full copy to history. - visual/, exports/ and memory/ are ignored as whole directories, replacing the per-campaign globs from #1524, #1553 and #1593. The 189 goldens stay tracked. Each was found by running its reader against an empty visual/, one test case per process, also under --olo-golden-vendor=amd, until no read failed. - docs/testing/evidence/: raw captures, HDRs, reports and logs that no doc links are untracked (404 files, 272 MB). Linked files, the dog-breed approval galleries, docs/images/ and the reproducible foliage baseline stay. Capture and report extensions there are now ignored. - Every affected write-up says how to regenerate its output and how to get the old copies (`git show 0cc2cd2:<path>`). Links to live captures that are no longer tracked become plain file names. The unified-layouts-1181 write-ups move to docs/testing/evidence/. `git rm --cached` only: every worktree keeps its local copies. History is not rewritten. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Per-campaign ignore globs caught each campaign only after it had been committed once. The `generated-evidence` hook makes the policy one rule: scripts/generated-evidence-allowlist.txt names every tracked file under OloEditor/assets/tests/, and every capture/report/scene file under docs/, each with the test or doc that reads it. The hook rejects anything else there, and any tracked file that .gitignore excludes (a force-add). It also rejects an entry with no reader and an entry that matches nothing. Its negative-control fixtures (a capture, a render-target export report, a force-added replay scene, campaign data under docs/; plus a golden, a fixture and a curated doc image that must pass) run on every invocation. All seven deliberate mutations of the checker fail them. The task loop no longer asks for an artefact-backed cell's proof to be a committed PNG. The matrix row cites the test, the file it wrote and the measurement. New guide: generated-evidence-stays-out-of-git.md. The measured history and an unexecuted rewrite plan are in docs/analysis/repo-history-remediation-1610.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

With `visual/` ignored as a directory, `git add` on a modified tracked golden staged it but exited 1 ("paths are ignored"), which breaks the --olo-golden-rebase workflow and any script that checks the exit code. Ignore every path inside the output directories and re-include the directories themselves, so only files are ignored: a tracked golden (also under amd/) stages with exit 0, and new output is still ignored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… its scene DeferredResolveMatrix serialized its Msaa4 Late3 scene into assets/tests/exports/ without creating the directory. The directory used to exist because it held tracked reports. Since #1610 exports/ is ignored output, so a fresh clone has no such directory. The test still passed, because earlier cells' WriteArtefact happens to create it first. Create the directory where the file is written instead of relying on cell order. Verified: the case passes with exports/ removed from the working tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…y-by-default Self-review findings on the guard: - It read tracked-but-ignored files with --exclude-standard, which also consults a developer's core.excludesFile and .git/info/exclude. A global `.vscode/` rule made it reject the tracked .vscode/*.json on that machine only. It now reads only the repository's .gitignore files. - docs/ was guarded by an extension list that missed video and other capture formats (.mp4, .webm, .svg, ...) and differed from the .gitignore extension list. Every non-Markdown file under docs/ is now guarded, and docs/testing/evidence/ ignores everything but Markdown, so the two lists cannot drift. - An allow-list entry that matched only unguarded files was reported as matching nothing. Use is now counted against every tracked file. - The opt-in records in visual/ (Dog_Cost, Dog_Lighting, the timing and cost .txt files) are written only behind an env flag, and docs cite them as the record, so they are curated, not per-run output. They are tracked again and allow-listed with the doc that cites them. New fixtures cover the motion-capture and prose-only-entry cases; all eight checker mutations fail them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…files exist Self-review findings on the docs side of #1610: - The READMEs for the hand-taken live-only acceptance captures (digital-human-1222-live, groom-animals-1223-live, flora-traversal-1224-live) were the only written record of those cells' measurements. Their captures stay untracked, but the write-ups move to docs/testing/evidence/, as unified-layouts-1181 did. Each has a history pointer, and docs/README.md indexes them. - integrated-acceptance-1259, integrated-renderer-budgets-1338 and the unified-layouts integration note still said the raw files "are in" or "are retained" in their directories. They now say the files were written there and are no longer tracked. - The remediation note gives the fresh-clone measurement (3.25 GiB) and the pack estimate (3.26 GiB → 1.08 GiB) as separate figures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…rts/ WriteArtefact and the matrix's scene save each built and created the exports path in their own way. MakeExportsDir now does it for both, and WriteArtefact's comment no longer refers to a committed artefact. Verified: all 12 RenderTargetExportEvidence cases pass with exports/ removed from the working tree, and regenerate its 39 files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…y is stale Second self-review pass: - DigitalHuman_Timing.txt and EyeCornea_Timing.txt are rewritten by their tests on every GPU run (no env gate), so they are run output after all. They are untracked again, and their guides say so. Dog_Cost, Dog_Lighting and GroomAnimals_Cost stay: each is written only behind an OLO_*_EXPORT=1 flag. - Dog_Cost_Matrix.txt has no writer. It is a hand-curated record, so it moves out of the output directory to docs/analysis/dog-cost-matrix.txt, linked from dog-showcase.md. - Counting an allow-list entry as used when it matched only prose let a `dir/**` entry outlive its last capture and silently allow the next force-add. An entry is again in use only while it allows a guarded or ignored file, and the fixture now expects a stale report. - Hook description, allow-list paths and the moved live write-ups' notes match the code and the new layout. The remediation plan preserves history-only live captures before any rewrite drops them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…write SceneSerializer::Serialize reports nothing, so a failed write left the read-back to throw from YAML::LoadFile, or to read a previous run's file and pass. Remove the old file first and assert the new one exists. The exports path is now one ExportsDir() plus a void EnsureExportsDir(), so no caller can continue with an empty path after a failed ASSERT. Verified: all 12 RenderTargetExportEvidence cases pass with exports/ removed from the working tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…history pointer - eye-cornea-iris.md and skin-digital-human.md quote the last tracked EyeCornea and DigitalHuman timing figures, because the files themselves are now per-run output. - The remediation plan's preservation step covers every write-up that points at `git show 0cc2cd2:`, not only the live captures, and step 3's drop list allows for blobs kept in step 1. - dog-cost-matrix.txt names its new location, and its allow-list reason says it was assembled from opt-in test runs. - A self-test fixture pins that an entry naming only an unguarded, unignored file is reported as stale. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Serialize wrote through an unchecked std::ofstream and returned void. A target that could not be opened (missing directory, locked or read-only file) or a failed write left nothing, or a truncated file, with no error. Autosave, the scene-asset save path and tests all went through it. It now returns false and logs an error naming the path. Existing callers compile unchanged. Found while reviewing #1610, where a test had to delete-then-stat its scene file because Serialize could not say whether it had written it. SceneRenderSettingsActivationTest.SerializeReportsAFileItCannotWrite pins both outcomes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

With Serialize reporting failure, the delete-then-stat workaround around it goes. A failed open or write now fails the test with the path, and a successful open truncates any previous run's file, so the read-back sees this run's scene. All 12 RenderTargetExportEvidence cases pass with exports/ removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The eye and digital-human guides quoted the last tracked timing figures without the source files' warning that a negative percentage is noise, not a saving. They also named a GPU that the files do not record. The remediation plan's history-pointer list includes both guides. The guard's self-test table formatting is restored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The eye guide named a GPU that its source file does not record and called all three figures noise without giving a floor. It now says only what the file supports. The digital-human guide stated the negative-figure caveat twice; the quoted figures now point at the paragraph that explains them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Follow-up to making SceneSerializer::Serialize report failure: - It writes to a uniquely named `<file>.<id>.tmp`, closes and checks the stream, then renames over the target, as SceneBinarySerializer and the save-game writer do. A failed or partial write no longer truncates the previous good save. Close errors are checked, the temporary file is removed on failure, and a symlinked scene is saved through to its target. Text mode is kept, so a saved scene's bytes do not change. - Its emitter body was a line-for-line copy of SerializeToYAML (two copies that have drifted before, #1140/#1169). It now writes SerializeToYAML()'s output. - It is [[nodiscard]], and every caller handles the result. OloServer's save command logs an error instead of "Scene saved". Editor auto-save reports the failure, does not then log success, and leaves the previous auto-save in place. The scene-asset serializer logs which asset was not saved. Tests wrap their scene saves in EXPECT_TRUE, so a failed write is reported at the write rather than as a confusing load failure later. SerializeReportsAFileItCannotWrite now also checks that a target that cannot be replaced fails the save and leaves no temporary file. AssetSerializer's void interface still drops the result for other asset types: #1635 (too big for this branch). Verified: 247 cases across the 21 suites that save scenes (2 skips: opt-in export, symlink privilege). OloEditor and OloServer build with no unused-result warning from Serialize. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

coderabbitai · 2026-10-18T14:02:22Z

Walkthrough

Warning

Review details and warnings were omitted to fit the comment limit.

drsnuggles8 · 2026-10-18T14:25:01Z

@coderabbitai full review

coderabbitai · 2026-10-18T14:25:08Z

⚠️ Action not completed

Review skipped: 334 files exceed the limit of 100.

drsnuggles8 and others added 16 commits October 18, 2026 13:39

drsnuggles8 merged commit 4be377f into master Oct 18, 2026
32 checks passed

drsnuggles8 deleted the feature/repo-evidence-bloat-audit-1610 branch October 18, 2026 16:40

Navigation Menu

Repository navigation

chore(tests): untrack generated evidence, guard against its return (#1610) - #1636

chore(tests): untrack generated evidence, guard against its return (#1610)#1636
drsnuggles8 merged 16 commits into
masterfrom
feature/repo-evidence-bloat-audit-1610

drsnuggles8 commented Oct 18, 2026

Uh oh!

coderabbitai Bot commented Oct 18, 2026

Uh oh!

drsnuggles8 commented Oct 18, 2026

Uh oh!

coderabbitai Bot commented Oct 18, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

drsnuggles8 commented Oct 18, 2026

Before and after

Inventory and classification

Recurrence guard

History (measured, not rewritten)

Coordination

Verification

Found and fixed

Filed, not fixed

Review guide

Uh oh!

coderabbitai Bot commented Oct 18, 2026

Walkthrough

Uh oh!

drsnuggles8 commented Oct 18, 2026

Uh oh!

coderabbitai Bot commented Oct 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

coderabbitai Bot commented Oct 18, 2026 •

edited

Loading