bent: isolate compile benchmark caches - #44
Conversation
There was a problem hiding this comment.
Review: Isolate benchmark build cache
Solid, well-scoped change. The core design — prewarm a stdlib-only seed, then hard-link-clone a private cache per measured build with cache setup/teardown kept outside the timed window — is correct and validated by TestIsolatedBuildCacheDoesNotLeak. The trim.txt copy-instead-of-link special case shows good awareness of Go's cache-mutation behavior, and validateBuildCache cleanly rejects the UseBuildCache + BuildCache conflict. Docs largely match behavior.
I confirmed the measurement boundary is respected: the clone happens before start := time.Now() and cleanup is deferred, so neither contaminates build-real-ns/op.
Findings below are improvements, none blocking.
| cmd := exec.Command("go", args...) | ||
| cmd.Dir = bench.BuildDir() | ||
| cmd.Env = env | ||
| output, err := cmd.CombinedOutput() |
There was a problem hiding this comment.
[P2] standardLibraryPackages parses merged stderr as package names
cmd.CombinedOutput() merges stderr into the scanned output, and the scanner only filters empty/unsafe/builtin lines. Any stderr line go list emits on success (toolchain download notices, deprecation warnings, etc.) that looks like an import path gets added to packages and later passed to go install, causing a confusing prewarm failure. Prefer cmd.Output() to parse stdout only, keeping stderr for the error message.
| runsDir := filepath.Join(dirs.buildCaches, "runs") | ||
| if err := os.MkdirAll(runsDir, 0o775); err != nil { | ||
| return "", func() {}, err | ||
| } |
There was a problem hiding this comment.
[P2] Orphaned per-build clone dirs accumulate under build-cache/runs
Each build creates a fresh MkdirTemp(runsDir, "cache-") and relies on the deferred cleanup() to remove it. That cleanup is bypassed on os.Exit paths (e.g. the build-bench-file write error in compileOne) and on any crash/kill, leaving full hard-linked cache trees behind. Unlike per-config seeds (which are RemoveAll'd in prepareCacheSeed), runs/ is never pruned, so orphaned trees accumulate across aborted runs and consume disk. Consider clearing runs/ at startup.
| compiled by an earlier pair. Native Go and LLGo follow the same cache policy; | ||
| the workflow also disables setup-go's cross-run build-cache restore. LLGo's | ||
| package-cache fingerprint still separates the LTO and plugin configurations. |
There was a problem hiding this comment.
[P3] setup-go cache:false not applied to pr-validation job despite README claim
The README states "the workflow also disables setup-go's cross-run build-cache restore." This cache: false is set on the matrix job but the pr-validation job also runs Bent with the stdlib cache policy and reports build times, yet its setup-go step omits cache: false. It's functionally harmless (isolated builds and prewarm use private cloned cache dirs), but the blanket README wording overstates coverage. Either scope the sentence to the matrix job or add cache: false here for consistency.
Summary
BuildCache = "stdlib"policy that prewarms only the standard-library dependencies used by the selected suitesThis keeps downloaded module sources available while preventing compiled third-party packages and target packages from leaking across CI runs or benchmark/configuration pairs. Runtime performance and compatibility jobs are unchanged.
Validation
go test ./cmd/bentgo test -race ./cmd/bentgo test ./...go vet ./cmd/bent