feat: initialize Azure clouds natively in ANC by abigailliang-aks-sig-node · Pull Request #9891 · Azure/AgentBaker

Abigail Liang (abigailliang-aks-sig-node) · 2026-10-23T21:06:33Z

Superseded by the five-part series

Review and merge the following smaller PRs instead of this aggregate PR:

[1/5] feat: [1/5] add cloud platform and trust-store capabilities #9910 -- shared runtime, platform support, and certificate trust stores;
based on main.

[2/5] feat: [2/5] add cloud certificate retrieval protocols #9911 -- certificate retrieval protocols; depends on [1/5].

[3/5] feat: [3/5] add cloud certificate refresh orchestration and CLI #9909 -- certificate-refresh orchestration and standalone CLI;
depends on [2/5].

[4/5] feat: [4/5] add native RepoDepot initialization #9903 -- RepoDepot capabilities; depends on [3/5].

[5/5] feat: [5/5] enable native cloud initialization before provisioning #9904 -- native Go activation, refresh scheduling, and paired
shell-call removal; depends on [4/5].

Only [5/5] switches the native provisioning path. The current English
migration and dependency diagrams are in #9910. The earlier certificate
PR #9902 has also been superseded by this finer-grained split.

This aggregate remains closed. Its original branch, commits, discussion,
and implementation description are retained for historical comparison.

What this PR does / why we need it:

Replace shell-based Azure cloud initialization in ANC's native configuration path
with Go initialization that runs before the existing provisioning scripts:

ANC cloud initialization (Go) -> existing provisioning scripts

This ports the initialization responsibilities; it does not relocate the .sh
file into ANC or rewrite the remaining provisioning scripts.

Fetch and install Azure cloud CA certificates through the legacy ussec/usnat
endpoints or the RCV1P opt-in/root/intermediate endpoints.
Register daily certificate refresh using the running ANC binary, including
hotfix binaries, through refresh-cloud-certificates --location <region>.
Preserve unrelated cron entries and use systemd on Flatcar/ACL.
Configure RepoDepot for Ubuntu, Mariner, and Azure Linux before provisioning.
Certificate opt-out does not skip RepoDepot; immutable images do not configure
package repositories.
Add bounded requests/retries, cancellation, guest-agent events, dry-run handling,
regression coverage, and updated native command fixtures/documentation.

Scope and compatibility

Only --provision-config without --nbc-cmd uses native cloud initialization.
Traditional CSE, NBC/Phase 2, and Phase 2.5 retain their existing shell behavior
and NBC precedence. The shell artifact, VHD mappings, and root AgentBaker
CustomData delivery remain unchanged for compatibility with those paths and
older VHD binaries. This PR does not remove the physical shell artifact or
reduce the current root CustomData payload.

Certificate behavior compatibility

A nonempty malformed opt-in response, a missing field, or any value other than
the JSON boolean true skips certificate installation and still permits
RepoDepot initialization. Failed requests and empty opt-in responses remain
fatal, as in the shell.
Downloaded certificate contents are cached and passed to the trust-store tools
with the shell's newline handling, without additional PEM/X.509 validation or
filtering. Public-certificate parsing is diagnostic-only and does not gate
installation.
An individual operation-certificate download failure or empty certificate
response is logged and skipped while processing continues. Missing operation
lists and RCV1P installation with no remaining .crt files still fail.
Legacy certificate retrieval failures remain fatal. Legacy trust-store
installation failures are logged without blocking later initialization.

Refresh clears the certificate staging cache, not existing system trust-store
certificates. Raw responses are never logged; AKS_CLOUD_LOG_CERTIFICATES=true
logs re-encoded public certificates only.

Best-effort cron and cache failure handling

These two failure cases now retain the shell's best-effort behavior:

Failure to read an existing crontab treats it as empty and still attempts
installation. Unexpected read errors and cron installation failures are
logged without blocking later repository initialization or provisioning.
RepoDepot cache refresh retries up to ten times. Exhaustion still returns a
classified cache-operation error and emits an error-level repository event;
the provisioning boundary logs the failure and continues instead of blocking
the provisioning scripts.

Regression tests exercise actual native provisioning entry points and require
exit code 0 after the cron failures and exhausted Ubuntu/Mariner/Azure Linux
cache refreshes. Repository failure events remain observable even when overall
initialization succeeds.

Preserved native boundaries and operational bounds

Nonfatal handling is limited to these ordinary cron/cache failures. Overall
context cancellation or timeout, repository setup errors, invalid refresh
configuration, and systemd refresh errors still stop native initialization.
Existing certificate failure handling is unchanged by this correction.

This is not a claim of complete shell equivalence outside the cases above.
Refresh scheduling requires a nonempty region, and Ubuntu Microsoft sources use
the node's architecture. Initialization uses the configured CSE_TIMEOUT budget,
with 30-second HTTP requests and two-minute command limits. Periodic certificate
refresh has a 15-minute budget and does not reconfigure repositories or reinstall
schedules.

Validation

From aks-node-controller:

go test -race . ./parser -count=1
go run github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.14.0 run --new-from-rev=origin/main --timeout=5m . ./parser
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o /dev/null .
CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -o /dev/null .

Lint uses the CI-pinned golangci-lint v2.14.0. Native command fixtures were
regenerated for the initial template change. Also checked the native shell
template with bash -n and the diff with git diff --check.
Azure/VHD end-to-end provisioning has not been run.

Which issue(s) this PR fixes:

No linked issue.

Run native cloud certificate and RepoDepot initialization before the existing provisioning scripts for --provision-config. Add native periodic certificate refresh while retaining shell initialization for traditional CSE, NBC, and older VHDs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: be499bac-1283-498e-b5b1-59d8c91047a4

github-actions · 2026-10-23T21:09:12Z

Windows Unit Test Results

3 files 18 suites 1m 7s ⏱️
563 tests 563 ✅ 0 💤 0 ❌
585 runs 585 ✅ 0 💤 0 ❌

Results for commit ba9a4aa.

♻️ This comment has been updated with latest results.

+
+func (c *cloudInitializer) refresh(ctx context.Context, platform cloudPlatform, location string) (bool, error) {
+	legacy := strings.HasPrefix(location, "ussec") || strings.HasPrefix(location, "usnat")
+	slog.Info("selecting Azure cloud certificate endpoint", "legacy", legacy, "location", location)


Copilot

🟡 Changes recommended

Unresolved timeout enforcement, OSGuard handling, and package-cache failure detection issues affect provisioning reliability.

3 open issues

What changed in this PR

Moves Azure cloud initialization into Go for ANC’s native --provision-config path, before existing provisioning scripts run. NBC and traditional CSE retain shell initialization.

Changes:

Adds certificate retrieval, validation, trust-store installation, and daily refresh scheduling.
Adds OS-specific RepoDepot setup, request limits, retries, cancellation, and guest-agent events.
Integrates native initialization with provisioning and dry-run handling, with regression tests and updated command fixtures.

File	Description
aks-node-controller/README.md	Documents native initialization and compatibility boundaries.
aks-node-controller/parser/testdata/Compatibility+EmptyConfig/generatedCSECommand	Updates empty-config command expectation.
aks-node-controller/parser/testdata/AzureLinuxv2+Kata+DisableUnattendedUpgrades=false/generatedCSECommand	Updates Azure Linux Kata command expectation.
aks-node-controller/parser/testdata/AKSUbuntu2204+SSHStatusOn/generatedCSECommand	Updates SSH-enabled command expectation.
aks-node-controller/parser/testdata/AKSUbuntu2204+LocalDNS+HostsPlugin/generatedCSECommand	Updates LocalDNS hosts-plugin command expectation.
aks-node-controller/parser/testdata/AKSUbuntu2204+LocalDNS/generatedCSECommand	Updates LocalDNS command expectation.
aks-node-controller/parser/testdata/AKSUbuntu2204+EnablePubkeyAuth/generatedCSECommand	Updates public-key-enabled command expectation.
aks-node-controller/parser/testdata/AKSUbuntu2204+DisablePubkeyAuth/generatedCSECommand	Updates public-key-disabled command expectation.
aks-node-controller/parser/testdata/AKSUbuntu2204+DefaultPubkeyAuth/generatedCSECommand	Updates default public-key command expectation.
aks-node-controller/parser/testdata/AKSUbuntu2204+CustomOSConfig/generatedCSECommand	Updates custom OS command expectation.
aks-node-controller/parser/testdata/AKSUbuntu2204+CustomCloud/generatedCSECommand	Updates custom-cloud command expectation.
aks-node-controller/parser/testdata/AKSUbuntu2204+Containerd+MIG/generatedCSECommand	Updates containerd/MIG command expectation.
aks-node-controller/parser/testdata/AKSUbuntu2204+CloudProviderOverrides/generatedCSECommand	Updates cloud-provider override command expectation.
aks-node-controller/parser/testdata/AKSUbuntu2204+China/generatedCSECommand	Updates China cloud command expectation.
aks-node-controller/parser/templates/cse_cmd.sh.gtpl	Removes native command’s shell initialization call.
aks-node-controller/parser/parser.go	Removes shell-path environment entry and trims command whitespace.
aks-node-controller/parser/parser_test.go	Verifies native commands omit shell initialization.
aks-node-controller/parser/helper.go	Removes unused shell-path helper and template registration.
aks-node-controller/parser/consts.go	Removes unused shell-path constant.
aks-node-controller/embeddednodecustomdata.go	Shares the Mariner OS identifier.
aks-node-controller/cloud_repositories.go	Implements repository configuration and cache refresh.
aks-node-controller/cloud_repositories_test.go	Tests repository dispatch, validation, and retries.
aks-node-controller/cloud_provision_test.go	Tests ordering, failure reporting, compatibility, and refresh CLI.
aks-node-controller/cloud_init.go	Adds initialization orchestration, requests, commands, and scheduling.
aks-node-controller/cloud_init_test.go	Tests certificates, platforms, scheduling, and error handling.
aks-node-controller/cloud_certificates.go	Implements certificate retrieval, validation, and installation.
aks-node-controller/app.go	Integrates initialization and registers the refresh command.
aks-node-controller/app_test.go	Isolates provisioning tests from host initialization.

_{Review effort: Balanced}

+	case strings.Contains(values["NAME"], "Microsoft Azure Linux"), id == osReleaseIDAzureLinux:
+		id = osReleaseIDAzureLinux


+	}
+	ctx, cancel := context.WithTimeout(ctx, cloudCommandTimeout)
+	defer cancel()
+	cmd := exec.CommandContext(ctx, name, args...)


+		if err == nil && name == "dnf" && dnfError.Match(output) {
+			err = errors.New("dnf reported repository metadata errors")
+		}


Copilot

🔵 Needs a closer look

The certificate failure policy needs owner confirmation, and this provisioning-path migration has not undergone Azure/VHD end-to-end validation.

4 open issues

_{Review effort: Balanced}

+		if err := c.writeFile(filepath.Join(destination, targetName), data); err != nil {
+			return err
+		}
+		installed++


Preserve the certificate compatibility fixes already present in the PR. Extract the operation-download test server helper, remove shadowing and slice-append warnings, and normalize empty-response errors without changing their return paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: be499bac-1283-498e-b5b1-59d8c91047a4

Copilot

🔵 Needs a closer look

Descendants retaining output pipes can keep host commands blocked beyond the configured initialization and refresh deadlines.

4 open issues

_{Review effort: Balanced}

…nd improve test clarity

Copilot

🔵 Needs a closer look

The Ubuntu directory probe rejects usable repositories that disable directory listing, causing native provisioning to fail.

4 open issues

1 previously missed issue

In code that hasn't changed since last review

Avoid rejecting usable APT repositories on directory 403

aks-node-controller/cloud_repositories.go:121

🟡 Medium Risk — 🔄 Backward Compatibility: A usable Ubuntu repository can return 403 for its directory URL while serving APT metadata and packages, but this probe now makes native provisioning fail with exit code 240. The shell's check_url did not reject that response; remove the directory probe and rely on the existing package-manager refresh to check repository access. Add a regression case where the base URI returns 403 but metadata is available.

_{Review effort: Balanced}

Keep cron read/install failures and exhausted RepoDepot cache refreshes from blocking native provisioning, matching the shell. Retain failure logging and repository error events without swallowing cancellation, repository setup errors, or systemd failures. Cover successful provisioning exit codes and preserved fatal boundaries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: be499bac-1283-498e-b5b1-59d8c91047a4

github-actions · 2026-10-26T21:15:38Z

Package	Line Rate	Health
test	0%	❌
windows	27%	❌
staging_cse_windows Package 1	65%	➖
debug	0%	❌
provisioningscripts	2%	❌
parts_windows Package 1	73%	➖
Summary	38% (1652 / 6834)	❌

github-actions · 2026-10-26T21:15:52Z

Package	Line Rate	Health
github.com/Azure/agentbaker/aks-node-controller	74%	➖
github.com/Azure/agentbaker/aks-node-controller/common	100%	✔
github.com/Azure/agentbaker/aks-node-controller/helpers	68%	➖
github.com/Azure/agentbaker/aks-node-controller/parser	88%	✔
github.com/Azure/agentbaker/aks-node-controller/pkg/gpu	100%	✔
github.com/Azure/agentbaker/aks-node-controller/pkg/nodeconfigutils	89%	✔
github.com/Azure/agentbaker/aks-node-controller/utils	0%	❌
github.com/Azure/agentbaker/apiserver	24%	❌
github.com/Azure/agentbaker/cmd	0%	❌
github.com/Azure/agentbaker/cmd/starter	0%	❌
github.com/Azure/agentbaker/fuzz/api	0%	❌
github.com/Azure/agentbaker/hotfix/render-nodecustomdata	35%	❌
github.com/Azure/agentbaker/pkg/agent	78%	➖
github.com/Azure/agentbaker/pkg/agent/datamodel	77%	➖
github.com/Azure/agentbaker/pkg/agent/toggles	0%	❌
github.com/Azure/agentbaker/pkg/vhdbuilder/datamodel	87%	✔
Summary	75% (6517 / 8636)	➖

github-actions · 2026-10-26T21:17:32Z

Package	Line Rate	Health
shellspec spec	28%	❌
Summary	28% (4591 / 16084)	❌

Copilot

🔵 Needs a closer look

Security-sensitive, cross-distro provisioning changes require human review, particularly because Azure/VHD end-to-end validation has not been run.

4 open issues

_{Review effort: Balanced}

Copilot AI balanced review requested due to automatic review settings October 23, 2026 21:06

Abigail Liang (abigailliang-aks-sig-node) requested review from a team as code owners October 23, 2026 21:06

Copilot started reviewing on behalf of Abigail Liang (abigailliang-aks-sig-node) October 23, 2026 21:07

View session

github-advanced-security Bot found potential problems Oct 23, 2026

View reviewed changes

Copilot AI reviewed Oct 23, 2026

View reviewed changes

feat: enhance Azure cloud certificate handling and logging

facce1a

Copilot AI balanced review requested due to automatic review settings October 24, 2026 01:22

Copilot started reviewing on behalf of Abigail Liang (abigailliang-aks-sig-node) October 24, 2026 01:22

View session

Copilot AI reviewed Oct 24, 2026

View reviewed changes

Comment thread aks-node-controller/cloud_certificates.go

Comment on lines +269 to +272

if err := c.writeFile(filepath.Join(destination, targetName), data); err != nil {

return err

}

installed++

Copilot AI balanced review requested due to automatic review settings October 24, 2026 01:45

Copilot started reviewing on behalf of Abigail Liang (abigailliang-aks-sig-node) October 24, 2026 01:46

View session

Copilot AI reviewed Oct 24, 2026

View reviewed changes

fix: update certificate cache clearing logic to remove hidden files a…

9967df4

…nd improve test clarity

Copilot AI balanced review requested due to automatic review settings October 24, 2026 02:02

Copilot started reviewing on behalf of Abigail Liang (abigailliang-aks-sig-node) October 24, 2026 02:03

View session

Copilot AI reviewed Oct 24, 2026

View reviewed changes

Copilot AI balanced review requested due to automatic review settings October 26, 2026 21:12

Copilot started reviewing on behalf of Abigail Liang (abigailliang-aks-sig-node) October 26, 2026 21:13

View session

Copilot AI reviewed Oct 26, 2026

View reviewed changes

This was referenced Oct 26, 2026

feat: [1/3] add native cloud certificate capabilities #9902

Closed

feat: [4/5] add native RepoDepot initialization #9903

Draft

Abigail Liang (abigailliang-aks-sig-node) mentioned this pull request Oct 26, 2026

feat: [5/5] enable native cloud initialization before provisioning #9904

Draft

Abigail Liang (abigailliang-aks-sig-node) closed this Oct 26, 2026

This was referenced Oct 27, 2026

feat: [3/5] add cloud certificate refresh orchestration and CLI #9909

Draft

feat: [1/5] add cloud platform and trust-store capabilities #9910

Open

feat: [2/5] add cloud certificate retrieval protocols #9911

Draft

Navigation Menu

Repository navigation

feat: initialize Azure clouds natively in ANC - #9891

feat: initialize Azure clouds natively in ANC#9891
Abigail Liang (abigailliang-aks-sig-node) wants to merge 5 commits into
mainfrom
abigailliang/anc-cloud-init-go

Abigail Liang (abigailliang-aks-sig-node) commented Oct 23, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 23, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions Bot commented Oct 26, 2026

Uh oh!

github-actions Bot commented Oct 26, 2026

Uh oh!

github-actions Bot commented Oct 26, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		case strings.Contains(values["NAME"], "Microsoft Azure Linux"), id == osReleaseIDAzureLinux:
		id = osReleaseIDAzureLinux

Conversation

Abigail Liang (abigailliang-aks-sig-node) commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Windows Unit Test Results

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

github-actions Bot commented Oct 26, 2026

Uh oh!

github-actions Bot commented Oct 26, 2026

Uh oh!

github-actions Bot commented Oct 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Abigail Liang (abigailliang-aks-sig-node) commented Oct 23, 2026 •

edited

Loading

github-actions Bot commented Oct 23, 2026 •

edited

Loading