Conversation
Bad installers can hang googet indefinitely, most often by becoming unexpectedly interactive in an unattended context, and downloads can hang on a stalled connection. This change kills only true hangs and stalls, never work that is still progressing, and makes a killed install fail cleanly. Installer supervision (new supervisor package, goolib, system): - Installers run in a Windows Job Object (process group on Unix) with stdin set to the null device. The child is started suspended, assigned to the job and then resumed, so there is no startup race. - Forward progress is a rolling-window check across the whole tree: CPU >= 250ms, read+write I/O >= 64KiB, or installer/MSI log growth within 30s. - Watchdogs: inactivity (no progress for 5m); hard cap (60m, lifted for wusa/dism unless the admin or package set one); modal UI in unattended mode (a #32770 or owned modal-frame window persisting 30s with no progress since it appeared). - Work outside the job counts as progress: the Windows Installer service tree while _MSIExecute is held, the TrustedInstaller/TiWorker tree for wusa/dism, and CBS.log growth. After an msiexec abort, only service-side descendants created after start and observed idle for the inactivity timeout are terminated; the services themselves never are. - No abort decisions are made after the root installer exits. On success the job's kill-on-close limit is cleared so tray apps and updaters survive. Waits are bounded (WaitDelay 30s, post-kill wait 60s). - If the job cannot be created, only the hard cap is enforced on the root process, so a running child is never orphaned. Configuration: - googet.conf: SupervisorMode (enforce|monitor|off, default enforce), InactivityTimeout, InstallTimeout, UIGracePeriod, UIDetection, DownloadStallTimeout. "0" disables the inactivity and install timeouts. - goospec install/uninstall/verify accept timeout and inactivity_timeout overrides on every OS. - monitor mode logs one WOULD_KILL line per reason and never kills, for staged rollout. goopack build commands run with supervision off. Failure semantics (install, cli/install, cli/update): - Installs are transactional: overwritten files are backed up beside themselves (copy fallback when a rename is impossible), new files and directories are tracked, removed empty directories are recorded, and everything is rolled back on failure. The package is never recorded in the database on failure, and installer logs are preserved. - Multi-package install/update continues past a failed package and exits 1. Downloads (client, download): - ResponseHeaderTimeout is 30s; there is no overall timeout, so slow links are never capped. Downloader now uses its own http.Client instead of mutating http.DefaultClient. - An idle-read StallReader (default 120s, Downloader.StallTimeout) guards package bodies (HTTP and GCS) and repo index bodies. - Retryable failures (stall, reset, EOF, timeouts, http2, 5xx, 429) resume with Range and back off with jitter. Attempts that advance the file are unlimited; the download fails after 4 consecutive attempts with no progress or 20 attempts that each advanced it by less than 1 MiB. - Content-Range is validated. A 416, a mismatched range, or a checksum mismatch after a resume restarts from byte 0 and resets the budget; repeated restarts that never pass the previous best offset fail. - Fixed a GCS index error-shadowing bug that hid non-404 errors. Notes: - Defaults are enforce with a 60m cap; installers that legitimately run longer must set timeout in their goospec. Fleets can stage with SupervisorMode: monitor. - Killing googet now also kills the in-flight installer tree. - A bootstrapper that exits while a child holding its stdout keeps running is considered finished after 30s instead of being waited on forever. - The Windows-specific paths have been cross-compiled and vetted but not yet executed; supervisor_windows_test.go runs in the Windows CI job.
Long-running installers (SQL Server, Visual Studio, Office) routinely exceed 60m while making steady progress. The inactivity (5m) and UI (30s) watchdogs still catch true hangs quickly; the hard cap is only a backstop for spin loops or log spam that masquerade as progress.
- Run watchdog termination messages and install rollback/commit logging under the new progress.Interrupt so the active spinner cannot redraw over error lines that logger writes to stderr. - StallReader now returns ErrDownloadStalled only after cancel has completed, fixing a race that made TestStallReader_ZeroByteReadsDoNotResetTimer flaky under -race. - enrichOptions now discovers InnoSetup /LOG=path and fully quoted "/log:path" arguments, and no longer treats a following switch such as /quiet as the log path. The log-only corner-case test is replaced with assertions. - Port upstream's TestPackageHTTP table, which the merge dropped, as TestPackageHTTP_ResumeFromDisk.
Windows CI ran these code paths for the first time and found: - TestWindows_MessageBoxAborts hung until the job timeout. On a non-interactive window station, such as session 0 where googet runs as SYSTEM, a blocking #32770 dialog reports IsWindowVisible false, so the visibility filter hid every prompt and UI detection never fired. The filter now applies only on an interactive window station, detected via GetProcessWindowStation and the WSF_VISIBLE flag. The test also gets a hard timeout so a detection regression fails instead of hanging. - Three install tests asserted Unix permission bits (0640, 0604, 0750) that Windows does not model. They now compare against the mode the OS applied after Chmod.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A single hung installer or stalled download blocks a googet run indefinitely:
googet install/updatenever returns and every later package in the batch is skipped.Change
Installer supervisor (
supervisor/)Every installer command (
.msi,.exe,.msu, scripts) now runs under a supervisor:/l*v,/log,/LOG=arguments;CBS.logfor.msu), CPU time, or I/O in the process tree. Slow-but-working installers are never killed.KILL_ON_JOB_CLOSE, so killing googet also kills the installer; on Unix a process group is used. Termination targets the whole tree, including MSI service-spawned children.A terminated installer fails only that package: its changes are rolled back, the run continues with the next package, and googet exits non-zero. The package is never recorded as installed.
Download stall detection and resume (
client/,download/)StallReadercancels a read that receives zero bytes forDownloadStallTimeout(default 120s). Any byte of progress resets the timer, so slow links are never aborted.Range(validated againstContent-Range) or a GCS range reader; servers that ignore ranges fall back to a full restart. Partial files left on disk by an earlier run are resumed.Transactional installs (
install/txn.go)Files overwritten during an install are backed up and restored if the install fails or is terminated; installer logs are preserved and their location is logged. Batch updates continue past packages that cannot be resolved.
Configuration
googet.conf:SupervisorModeenforceenforce,monitor(log what would be killed, kill nothing), oroffInactivityTimeout5m0disablesInstallTimeout4h0disablesUIGracePeriod30sUIDetectiontrueDownloadStallTimeout120sPer package,
ExecFilegainstimeoutandinactivityTimeout(Go durations;"0"disables), validated at goospec parse time.SupervisorMode: monitoris intended for staged rollouts: it logs every decision without terminating anything.Progress UI integration
Supervisor child output goes through
progress.Stdout()/Stderr(), and watchdog/rollback errors are written via a newprogress.Interruptso the spinner cannot overwrite them.Behavior changes
timeoutor the host setsInstallTimeout..msu(wusa) installs have no hard cap unless the package sets one.Testing
go vet ./...andGOOS=windows go vet ./...are clean;GOOS=windows go build ./...and Windows test compilation succeed.go test -race ./...passes on Linux for all packages. New tests cover the supervisor state machine, process-tree termination (Unix), stall detection, range resume against misbehaving servers, retry budgeting, transactional rollback, multi-package continuation after a killed install, and config/goospec parsing.SupervisorMode: monitorbefore enforcing is recommended.