feat: enable artifact streaming on ACL by rguptar · Pull Request #9613 · Azure/AgentBaker

rguptar · 2026-09-22T17:02:05Z

What this PR does / why we need it:

Enables Artifact Streaming on ACL using the combined ACR mirror, OverlayBD, and overlaybd-snapshotter system extension from mcr.microsoft.com/azurecontainerregistry/v2/artifact-streaming-sysext.

Cache the architecture-specific sysext in the VHD instead of installing the ACR RPM or enabling ACL's separate OverlayBD extension.
Keep streaming units masked until the feature-enabled node reaches nodePrep, including on PIS-cached VHDs. Use the same provisioning path for CSE and node-controller rather than an early Ignition setup service.
Initialize the sysext's writable paths, preserve/materialize containerd configuration, and configure ACR before starting consumers. Propagate setup failures before kubelet starts.
Resolve the exact pinned DALEC revision for both architectures, including network-isolated registries without tag-listing support.
Preserve the existing Ubuntu and regular Azure Linux package paths, and extend ACL scenarios to cover enabled and disabled behavior.

The initial pin is v1.0.0-1-azlinux3, with -x86-64 and -arm64 architecture suffixes. VHD builds depend on those artifacts being published to the repository above.

github-actions · 2026-09-22T17:09:06Z

Windows Unit Test Results

3 files 18 suites 55s ⏱️
662 tests 662 ✅ 0 💤 0 ❌
685 runs 685 ✅ 0 💤 0 ❌

Results for commit 93408b9.

♻️ This comment has been updated with latest results.

Copilot

Copilot review overview

🟡 Changes recommended

The ACL installer and end-to-end artifact-streaming path lack coverage for package extraction, sysext activation, and real service startup.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1

Open (1)

Add ACL installer and artifact-streaming E2E test coverage · New

What changed in this PR

Enables artifact streaming on Azure Container Linux using the OverlayBD system extension and ACL-specific acr-mirror installation.

Changes:

Adds ACL package selection and RPM extraction.
Configures OverlayBD sysext and service lifecycle.
Extends schemas, documentation, and unit fixtures.

File	Description
`vhdbuilder/packer/install-dependencies.sh`	Installs and initializes `acr-mirror` on ACL.
`vhdbuilder/packer/acl-customdata.yaml`	Enables OverlayBD sysext and masks services initially.
`vhdbuilder/packer/acl-customdata.json`	Regenerates ACL Ignition configuration.
`spec/parts/linux/cloud-init/artifacts/test_components.json`	Adds ACL and Flatcar test metadata.
`spec/parts/linux/cloud-init/artifacts/cse_helpers_spec.sh`	Tests ACL metadata selection and fallback.
`spec/parts/linux/cloud-init/artifacts/cse_config_gpu_spec.sh`	Tests ACL service unmasking.
`schemas/components.cue`	Adds ACL component metadata schema.
`parts/linux/cloud-init/artifacts/README-COMPONENTS.md`	Documents ACL metadata behavior.
`parts/linux/cloud-init/artifacts/cse_helpers.sh`	Selects ACL-specific package entries.
`parts/linux/cloud-init/artifacts/cse_config.sh`	Unmasks OverlayBD services during provisioning.
`parts/common/components.json`	Adds the ACL `acr-mirror` package entry.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+  if isACL "$OS" "$OS_VARIANT"; then
+    bsdtar -C / -xf "$MIRROR_DOWNLOAD_PATH" opt/ || exit "$ERR_ARTIFACT_STREAMING_DOWNLOAD"
+    bsdtar -Oxf "$MIRROR_DOWNLOAD_PATH" usr/lib/systemd/system/acr-mirror.service |
+      install -m 0644 /dev/stdin /etc/systemd/system/acr-mirror.service ||
+      exit "$ERR_ARTIFACT_STREAMING_DOWNLOAD"
+    env -C /opt/acr/bin ./acr init --min-init || exit "$ERR_ARTIFACT_STREAMING_DOWNLOAD"


aks-node-assistant · 2026-09-22T18:34:45Z

AgentBaker Linux gate detective

Run: https://msazure.visualstudio.com/CloudNativeCompute/_build/results?buildId=182278061
Failed job/stage/task: build / buildaclarm64fipstlgen2 / Build VHD (same signature appears across ACL artifact-streaming build lanes).

Detective summary: Packer entered install-dependencies.sh for the ACL artifact-streaming image and failed while handling the ACR mirror package because bsdtar was not installed: /home/packer/install-dependencies.sh: line 215: bsdtar: command not found. Packer then exited non-zero (Script exited with non-zero exit status: 152).

Likely cause / signature: $sigBsdtar.

Confidence: High for PR-change-caused dependency/setup regression.

Recommended owner/action: PR #9613 should install or otherwise guarantee �sdtar before artifact-streaming extraction, or avoid the �sdtar dependency in install-dependencies.sh for ACL images.

Strongest alternative: transient package/feed outage; less likely because downloads immediately before the failure succeeded and the primary error is the local command missing, not an HTTP/package fetch failure.

Evidence: Timeline marks ACL Build VHD tasks failed; Build VHD log reaches artifact-streaming downloads then fails with bsdtar: command not found; PR metadata shows changes to install-dependencies.sh, cse_helpers.sh, and ACL customdata/component files.

Wiki signature: $sigBsdtar

Copilot

Copilot review overview

🟡 Changes recommended

Critical ACL build-tool and backward-compatibility failures, plus incorrect package setup logic, must be resolved.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 2 · 1 · 1

Open (4)

Gate ACL artifact streaming on image capability or add fallback · New
Provide bsdtar for ACL builds or use an available extractor · New
Use ACL 1.0.0 setup flow instead of legacy enablement · New
Add ACL installer and artifact-streaming E2E test coverage

aks-node-assistant · 2026-09-22T23:04:25Z

AgentBaker Linux gate detective

Run: https://msazure.visualstudio.com/CloudNativeCompute/_build/results?buildId=182333136
Failed job/stage/task: build / ACL artifact-streaming Build VHD lanes (buildacltlgen2, buildaclfipstlgen2, buildaclarm64*) plus Test/Scan lanes.

Detective summary: ACL artifact-streaming Build VHD failed again when install-dependencies.sh attempted to extract ACR mirror content using bsdtar and hit bsdtar: command not found; Packer exited 152. The same build also repeated the repair-linked FIPS CIS 5.1.18/5.1.6 drift and the noisy 6.1.3.1 CIS drift in ARM64 22.04 Test/Scan.

Likely cause / signature: vhd-acl-artifact-streaming-bsdtar-command-not-found; secondary signatures vhd-ubuntu2204fips-cis-cron-rules-regression and vhd-ubuntu2204fips-cis-rule-6-1-3-1-regression.

Confidence: High for PR-change-caused ACL artifact-streaming dependency/setup regression; high for recurring CIS baseline drift.

Recommended owner/action: PR #9613 should install or guarantee �sdtar before artifact-streaming extraction, or remove the �sdtar dependency; VHD validation owners should continue repair #39802117 for the separate CIS drift.

Strongest alternative: transient package/feed outage; less likely because the primary Build VHD error is the local command missing after downloads succeeded, and the same signature was previously observed on this PR.

Evidence: Timeline marks ACL Build VHD tasks failed; Build VHD logs show bsdtar: command not found followed by Packer exit 152; Test/Scan logs show 5.1.18|pass->fail, 5.1.6|pass->fail, and 6.1.3.1|pass->fail; PR metadata changes artifact-streaming/customdata/install-dependencies paths.

Wiki signatures: $sigBsdtar, $sigCis, $sigRule

Copilot

Copilot review overview

🔵 Needs a closer look

Three moderate provisioning and cross-platform compatibility issues remain unresolved.

Review effort: Balanced
Findings: 2 · 1 · 1

Open (4)

Provide bsdtar for ACL builds or use an available extractor
Gate ACL artifact streaming on image capability or add fallback
Use ACL 1.0.0 setup flow instead of legacy enablement
Add ACL installer and artifact-streaming E2E test coverage

Copilot

Copilot review overview

🟡 Changes recommended

Critical ACL build and provisioning-path issues prevent artifact streaming from being configured reliably.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 4 · 1 · 1

Open (6)

Add Ignition activation to the actual ACL template path · New
Move ACL OverlayBD activation into idempotent nodePrep · New
Avoid unavailable rpm2cpio and cpio tools on ACL · New
Gate ACL artifact streaming on image capability or add fallback
Use ACL 1.0.0 setup flow instead of legacy enablement
Add ACL installer and artifact-streaming E2E test coverage

Resolved since last review (1)

Provide bsdtar for ACL builds or use an available extractor

+  if isACL "$OS" "$OS_VARIANT"; then
+    systemctl unmask overlaybd-tcmu.service overlaybd-snapshotter.service || exit "$ERR_ARTIFACT_STREAMING_INSTALL"
+  fi
+  retrycmd_if_failure 120 5 25 systemctl --quiet enable --now acr-mirror overlaybd-tcmu overlaybd-snapshotter || exit "$ERR_ARTIFACT_STREAMING_INSTALL"


Copilot

Copilot review overview

🟡 Changes recommended

A critical platform-scoping defect and two moderate setup and test-coverage issues remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 5 · 2 · 1

Open (8)

Restrict ACL systemd services to ACL-specific Flatcar configurations · New
Avoid unavailable rpm2cpio and cpio tools on ACL
Move ACL OverlayBD activation into idempotent nodePrep
Add Ignition activation to the actual ACL template path
Gate ACL artifact streaming on image capability or add fallback
Add RPM parser tests with valid and corrupt zstd fixtures · New
Use ACL 1.0.0 setup flow instead of legacy enablement
Add ACL installer and artifact-streaming E2E test coverage

Copilot

Copilot review overview

🔵 Needs a closer look

Four unresolved moderate issues affect production ACL activation paths, Flatcar compatibility, and extraction test coverage.

Review effort: Balanced
Findings: 5 · 1 · 1

Open (7)

Restrict ACL systemd services to ACL-specific Flatcar configurations
Avoid unavailable rpm2cpio and cpio tools on ACL
Move ACL OverlayBD activation into idempotent nodePrep
Add Ignition activation to the actual ACL template path
Gate ACL artifact streaming on image capability or add fallback
Add RPM parser tests with valid and corrupt zstd fixtures
Add ACL installer and artifact-streaming E2E test coverage

Resolved since last review (1)

Use ACL 1.0.0 setup flow instead of legacy enablement

aks-node-assistant · 2026-09-23T01:34:42Z

AgentBaker Linux gate detective

Run: https://msazure.visualstudio.com/CloudNativeCompute/_build/results?buildId=182357334
Failed job/stage/task: e2e / Run AgentBaker E2E / LocalDNSHostsPlugin AzureLinuxV3, Ubuntu2204, and Ubuntu2404; build build2204fipsgen2containerd Test/Scan also repeated the repair-linked CIS baseline drift.

Detective summary: E2E reported 184 scenarios with 4 failed and 0 flaky. LocalDNS restart-budget validation injected a failing ExecStart=/bin/false, but localdns stayed active after 120s instead of reaching terminal failed/start-limit-hit, so unrecoverable faults would not be bounded. The same run also repeated CIS 5.1.18/5.1.6 scan drift under repair #39802117.

Likely cause / signature: e2e-localdns-restart-budget-hungstart-not-bounded; secondary recurring signature vhd-ubuntu2204fips-cis-cron-rules-regression.

Confidence: High for deterministic LocalDNS restart-budget validation regression; high for recurring CIS drift.

Recommended owner/action: PR #9613 owner should inspect LocalDNS restart-budget behavior now that artifact streaming changes touch scenario selection/config; VHD validation owners should continue repair #39802117 for the separate CIS drift.

Strongest alternative: shared systemd timing flake; less likely because three LocalDNSHostsPlugin variants fail the same injected-fault assertion and the suite reported zero flaky tests.

Evidence: E2E log reports localdns is active after 120s ... expected failed and LocalDNS did not reach terminal failed after exhausting its restart budget; final summary shows 0 flaky tests; timeline marks Run AgentBaker E2E failed; Test/Scan log separately reports 5.1.18|pass->fail and 5.1.6|pass->fail.

Wiki signatures: $sigLocal, $sigCis

Copilot

Copilot review overview

🟡 Changes recommended

Critical and moderate issues remain in package integrity/tooling and systemd provisioning order and dependencies.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 7 · 1

Open (8)

Masked services block artifact setup · New
Unsigned RPM extraction enables root code execution · New
Restrict ACL systemd services to ACL-specific Flatcar configurations
Avoid unavailable rpm2cpio and cpio tools on ACL
Move ACL OverlayBD activation into idempotent nodePrep
Add Ignition activation to the actual ACL template path
Gate ACL artifact streaming on image capability or add fallback
Add ACL installer and artifact-streaming E2E test coverage

Resolved since last review (1)

Add RPM parser tests with valid and corrupt zstd fixtures

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Cache mcr.microsoft.com/acr/v2/artifact-streaming-sysext instead of installing the ACR RPM or enabling a separate OverlayBD extension. Keep exact DALEC revision and architecture pins for local and network-isolated delivery. Defer activation to nodePrep for both CSE and node-controller. Initialize writable paths and ACR configuration before unmasking consumers, preserve GPU containerd settings when materializing its symlink, and fail provisioning before kubelet starts if setup fails. Keep other OS package paths unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0d5fc628-3611-41e9-b6d6-55b92081296b

Copilot

🟡 Changes recommended

The sysext pin lacks required Renovate normalization and revision ordering, while streaming activation lacks test coverage for phase gating, pre-kubelet ordering, failure handling, and disabled nodes.

3 open issues

6 resolved since last review

_{Review effort: Balanced}

+    if [ "${ARTIFACT_STREAMING_ENABLED}" = "true" ] && isACL "$OS" "$OS_VARIANT"; then
+        logs_to_events "AKS.CSE.ensureContainerd.ensureArtifactStreaming" ensureArtifactStreaming || exit $ERR_ARTIFACT_STREAMING_INSTALL


Copilot

🟡 Changes recommended

The streaming scenario enables Secure Boot unconditionally, preventing unsigned ACL images from booting, while tests leave provisioning-phase gating, cached-node activation, and setup-failure propagation unverified.

4 open issues

_{Review effort: Balanced}

+			config.EnableArtifactStreaming = true
+		},
+		VMConfigMutator: func(vmss *armcompute.VirtualMachineScaleSet) {
+			vmss.Properties = addTrustedLaunchToVMSS(vmss.Properties)


Check each decoded JSON type assertion and guard the file list before indexing it, satisfying errcheck without suppressing lint findings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0d5fc628-3611-41e9-b6d6-55b92081296b

Use azurecontainerregistry/v2/artifact-streaming-sysext for VHD caching, Renovate metadata, and node provisioning, retaining the pinned version and registry overrides. Update the matching ShellSpec expectations. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0d5fc628-3611-41e9-b6d6-55b92081296b

Copilot

🔵 Needs a closer look

The scenario unconditionally enables Secure Boot, ignoring the unsigned ACL image setting and preventing those images from booting or reaching streaming validation.

4 open issues

_{Review effort: Balanced}

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0d5fc628-3611-41e9-b6d6-55b92081296b

Copilot

🟡 Changes recommended

The scenario forces Secure Boot and prevents unsigned ACL images from booting, while the new sysext repository lacks the Renovate rule needed to match published version tags.

5 open issues

_{Review effort: Balanced}

+                "renovateTag": "OCI_registry=https://mcr.microsoft.com, name=azurecontainerregistry/v2/artifact-streaming-sysext",
+                "latestVersion": "v1.0.0-2-azlinux3"


Use v1.0.0-1-azlinux3 and update the exact-revision resolution and VHD caching tests to match the initial component publication. Validated: 25 focused ShellSpec examples, 0 failures; JSON pin, shell syntax, and whitespace checks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da9d0569-0965-44eb-9e2b-1d2ac995e638

Copilot

🔵 Needs a closer look

Unconditional Secure Boot prevents unsigned ACL images from booting, and the missing DALEC Renovate rule prevents automatic updates to the new sysext pin.

5 open issues

_{Review effort: Balanced}

Jiri Appl (jiria) · 2026-10-28T21:48:07Z


-    if [ "${ARTIFACT_STREAMING_ENABLED}" = "true" ]; then
+    # Defer ACL artifact streaming to nodePrep: keep sysext services masked
+    # during PIS image preparation and activate them on the real node.


Can you explain why we need to defer?

"PIS images already contain base_prep.complete, so real nodes skip basePrep. nodePrep runs on real provisioning but not pre-provision-only image preparation."

Jiri Appl (jiria) · 2026-10-28T21:48:37Z

@@ -635,9 +635,29 @@ EOF
 }

 ensureArtifactStreaming() {


Is this and IPE mutually exclusive?

I don't know the answer to this.

Jiri Appl (jiria) · 2026-10-28T21:53:49Z

@@ -392,29 +392,37 @@ downloadAndInstallCriTools() {
 installAndConfigureArtifactStreaming() {


Is the caller conditioned on artifact streaming enabled and IPE disabled?

no, this will be called for all VHD flavors that have a corresponding acr-mirror entry in components.json at build time - as the comment says this is a caching operation only, usage of acr-mirror is gated at node provisioning time

Jiri Appl (jiria) · 2026-10-28T21:54:46Z

    fi

+    # Keep ACL sysexts masked during PIS image preparation; configure them on the real node.
+    if [ "${ARTIFACT_STREAMING_ENABLED}" = "true" ] && isACL "$OS" "$OS_VARIANT"; then


Just to confirm, is ARTIFACT_STREAMING_ENABLED set to false by default?

it's set and passed down by AKS RP - it will default to false

Remove the unverified enabled-sysext.conf entry from ACL custom data and regenerate Ignition JSON using strict Butane validation. Preserve all remaining configuration and streaming service masks. Validated: strict Butane regeneration, Ignition configuration equivalence apart from the removed entry, 28 focused ShellSpec examples, and whitespace checks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da9d0569-0965-44eb-9e2b-1d2ac995e638

github-actions · 2026-10-28T22:38:00Z

Package	Line Rate	Health
test	0%	❌
windows	27%	❌
staging_cse_windows Package 1	68%	➖
debug	0%	❌
provisioningscripts	2%	❌
parts_windows Package 1	73%	➖
Summary	39% (1819 / 7001)	❌

github-actions · 2026-10-28T22:39:12Z

Package	Line Rate	Health
github.com/Azure/agentbaker/aks-node-controller	73%	➖
github.com/Azure/agentbaker/aks-node-controller/common	100%	✔
github.com/Azure/agentbaker/aks-node-controller/helpers	68%	➖
github.com/Azure/agentbaker/aks-node-controller/parser	88%	✔
github.com/Azure/agentbaker/aks-node-controller/pkg/gpu	100%	✔
github.com/Azure/agentbaker/aks-node-controller/pkg/nodeconfigutils	91%	✔
github.com/Azure/agentbaker/aks-node-controller/utils	0%	❌
github.com/Azure/agentbaker/apiserver	24%	❌
github.com/Azure/agentbaker/cmd	0%	❌
github.com/Azure/agentbaker/cmd/starter	0%	❌
github.com/Azure/agentbaker/fuzz/api	0%	❌
github.com/Azure/agentbaker/hotfix/render-nodecustomdata	35%	❌
github.com/Azure/agentbaker/pkg/agent	78%	➖
github.com/Azure/agentbaker/pkg/agent/datamodel	77%	➖
github.com/Azure/agentbaker/pkg/agent/toggles	0%	❌
github.com/Azure/agentbaker/pkg/vhdbuilder/datamodel	87%	✔
Summary	75% (6037 / 8001)	➖

github-actions · 2026-10-28T22:40:29Z

Package	Line Rate	Health
shellspec spec	29%	❌
Summary	29% (4731 / 16403)	❌

Copilot

🔵 Needs a closer look

The scenario unconditionally enables Secure Boot instead of respecting ACLBaseImageSigned, causing unsigned development images to fail to boot before artifact-streaming checks can run.

5 open issues

_{Review effort: Balanced}

Copilot AI balanced review requested due to automatic review settings September 22, 2026 17:02

rguptar requested review from a team as code owners September 22, 2026 17:02

github-actions Bot added the components This pull request updates cached components on Linux or Windows VHDs label Sep 22, 2026

Copilot started reviewing on behalf of rguptar September 22, 2026 17:03

View session

Copilot AI reviewed Sep 22, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings September 22, 2026 21:54

Copilot started reviewing on behalf of rguptar September 22, 2026 21:56

View session

Copilot AI reviewed Sep 22, 2026

View reviewed changes

Comment thread parts/linux/cloud-init/artifacts/cse_config.sh Outdated

Comment thread vhdbuilder/packer/install-dependencies.sh Outdated

Comment thread aks-node-controller/pkg/nodeconfigutils/utils.go Outdated

Copilot AI review requested due to automatic review settings September 22, 2026 23:04

Copilot started reviewing on behalf of rguptar September 22, 2026 23:05

View session

Copilot AI reviewed Sep 22, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings September 22, 2026 23:24

Copilot started reviewing on behalf of rguptar September 22, 2026 23:25

View session

Copilot AI reviewed Sep 22, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings September 22, 2026 23:36

Copilot started reviewing on behalf of rguptar September 22, 2026 23:37

View session

Copilot AI reviewed Sep 22, 2026

View reviewed changes

Comment thread aks-node-controller/pkg/nodeconfigutils/utils.go Outdated

Comment thread vhdbuilder/packer/install-dependencies.sh Outdated

Copilot AI review requested due to automatic review settings September 23, 2026 00:02

Copilot started reviewing on behalf of rguptar September 23, 2026 00:04

View session

Copilot AI reviewed Sep 23, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings September 23, 2026 02:27

Copilot started reviewing on behalf of rguptar September 23, 2026 02:28

View session

Copilot AI reviewed Sep 23, 2026

View reviewed changes

Comment thread aks-node-controller/pkg/nodeconfigutils/utils.go Outdated

Comment thread vhdbuilder/packer/install-dependencies.sh Outdated

Copilot AI review requested due to automatic review settings September 23, 2026 23:32

Copilot started reviewing on behalf of rguptar September 23, 2026 23:33

View session

rguptar and others added 3 commits October 28, 2026 11:29

fix: use rpm2cpio for ACL acr-mirror

e7ee95c

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

fix: install ACL acr-mirror with dnf

358060c

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

rguptar force-pushed the guptar/acl/artifactstreaming branch from fb68abe to ce65176 Compare October 28, 2026 18:31