Own DPkg::Lock::Timeout in Puppet and bound the boot security upgrade

[akuzminsky]

Puppet-side hardening for dpkg lock contention during provisioning. Root cause of #289 was in the AMI and is already fixed upstream (infrahouse/infrahouse-ubuntu-pro#20); this is the part that belongs here.

Behaviour changes are in environments/development only. Sandbox and modules/ follow after testing there.

profile::apt_lock_timeout (new)

Declared by profile::repos with stage => init, writes /etc/apt/apt.conf.d/99-lock-timeout with an unscoped DPkg::Lock::Timeout.

Ubuntu's shipped default is binary::apt::DPkg::Lock::Timeout "120" — scoped to the apt command only. Puppet's package provider, cloud-init and the AWS agents all shell out to apt-get, which inherits nothing and fails instantly on a held lock. Measured on noble / apt 2.8.3 against a held /var/lib/dpkg/lock-frontend:

	result
apt-get install, no options	exit=100, elapsed 0s
unscoped drop-in, no -o flag	exit=0, elapsed 45s — waited

Value comes from the apt_lock_timeout custom fact, defaulting to the 300 current AMIs ship.

Two details worth reviewing:

• The path deliberately collides with the AMI's. infrahouse-ubuntu-pro's provision.sh writes the same file. Two drop-ins both setting the key would resolve by lexical filename order, so Puppet converges the AMI's file instead of racing a second one.
• The init stage is load-bearing. A drop-in applied halfway through the catalog does not help the Package resources Puppet already evaluated. This is why it is a separate class rather than a file resource inside profile::repos — only the apt class declaration there carries stage => init, not the profile itself.

Exec[gha-boot-security-upgrade] is now bounded

Retry logic moved into gha-boot-security-upgrade.sh with a hard cumulative deadline (480s budget, exec timeout 540) rather than tries/try_sleep.

exec's timeout is per-attempt, so tries multiplies the worst case with no cumulative cap — the previous timeout => 1200 could consume the entire bootstrap lifecycle-hook budget in a single attempt. And an overrun is not merely a slow run: ih-puppet applies with --detailed-exitcodes and exits 4/6 on a resource failure, which trips ih-bootstrap.sh's trap _ih_signal_abandon ERR into an immediate ABANDON.

Retries are still needed because DPkg::Lock::Timeout does not cover /var/lib/apt/lists/lock (verified: apt-get update fails in ~1s with the option set) and unattended-upgrade refuses to run concurrently with itself.

Comment correction (all three copies, no behaviour change)

profile::unattended_upgrades claimed the base AMI ships those units masked. It does not — the masking comes from terraform-aws-cloud-init's bootcmd, on every boot, before runcmd starts Puppet. That misattribution caused a wrong root-cause analysis, so the correction is applied everywhere rather than left to the promotion cycle. The unmask execs are load-bearing until infrahouse/terraform-aws-cloud-init#91 lands.

Testing

puppet-lint --fail-on-warnings clean on all four module dirs; shellcheck clean.

The script's bounding logic is verified with stubbed apt-get/unattended-upgrade, 10/10 checks: gives up at its budget with exit 1 and no marker, retries through transient failure and writes the marker on success, no needless delay on the fast path, and — the important one — a hung command cannot overshoot the deadline (exits at 25s against a 600s sleep).

The apt lock measurements above were taken on live EC2 instances from both the old and rebuilt AMIs.

Not yet tested: the real thing end to end on a provisioned gha_runner. That is what development is for.

Follow-ups

• Promote to sandbox and modules/ after testing
• After Stop masking apt-daily / unattended-upgrades in bootcmd — Puppet now owns unattended-upgrades terraform-aws-cloud-init#91: drop the now-redundant unmask execs

Refs #289

🤖 Generated with Claude Code

[akuzminsky]

test:

make test-puppet
...
Notice: Compiled catalog for ip-10-0-3-146.us-west-1.compute.internal in environment development in 0.65 seconds
Notice: /Stage[init]/Profile::Apt_lock_timeout/File[/etc/apt/apt.conf.d/99-lock-timeout]/ensure: defined content as '{sha256}b3de1e2b878523ef75a25bde2dada39dbfbb89763b58a3b2ed499441231c0a07'
Notice: /Stage[init]/Apt::Update/Exec[apt_update]/returns: executed successfully
Notice: /Stage[main]/Profile::Github_runner/File[/usr/local/bin/gha-boot-security-upgrade.sh]/ensure: defined content as '{sha256}e1ff67b283aa10c35ff8f8a6f969c15497df165bd7841b985a311448f70f8853'
Notice: /Stage[main]/Sudo/Sudo::Conf[ssm-user]/File[10_ssm-user]/ensure: defined content as '{sha256}b6693623961dc58f3f27ae450e0458fcabcbba841f3068b45e9bb9cdae162237'
Notice: /Stage[main]/Sudo/Sudo::Conf[ssm-user]/Exec[sudo-syntax-check for file /etc/sudoers.d/10_ssm-user]: Triggered 'refresh' from 1 event
Notice: Applied catalog in 5.59 seconds
Notice: Compiled catalog for ip-10-0-3-146.us-west-1.compute.internal in environment development in 0.65 seconds
Notice: /Stage[init]/Apt::Update/Exec[apt_update]/returns: executed successfully
Notice: Applied catalog in 5.64 seconds
2026-07-31 23:37:01,307: INFO: root:__init__._run_apply():170: The run succeeded, and some resources were changed.

aleks@ip-10-0-3-146:~/code/puppet-code$ sudo /usr/local/bin/gha-boot-security-upgrade.sh
gha-boot-security-upgrade: attempt 1, 480s of budget left
gha-boot-security-upgrade: succeeded on attempt 1