Skip to content

feat: [5/5] enable native cloud initialization before provisioning - #9904

Draft
Abigail Liang (abigailliang-aks-sig-node) wants to merge 3 commits into
abigailliang/anc-cloud-repodepotfrom
abigailliang/anc-cloud-native-wiring
Draft

Abigail Liang (abigailliang-aks-sig-node) wants to merge 3 commits into
abigailliang/anc-cloud-repodepotfrom
abigailliang/anc-cloud-native-wiring

Conversation

@abigailliang-aks-sig-node

@abigailliang-aks-sig-node Abigail Liang (abigailliang-aks-sig-node) commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

Part [5/5] of the native ANC cloud-initialization series split from #9891.
Depends on [4/5] #9903, following the three smaller certificate layers.
This PR also hosts the integration acceptance harness for the complete stack.

This is the activation step:

ANC cloud initialization (Go) -> existing provisioning scripts

The Go runtime order is certificates, eligible refresh scheduling, then
RepoDepot, followed by the existing provisioning scripts. This is separate
from the PR dependency order.

  • Connect the existing certificate and RepoDepot capabilities to native
    --provision-config provisioning, before the existing scripts.
  • Remove the native template's shell initialization call in this same PR, avoiding
    both duplicate initialization and an intermediate version with no initialization.
  • Register daily certificate refresh through cron or systemd using the running
    ANC binary, including hotfix binaries.
  • Preserve ordinary cron failures and exhausted RepoDepot cache refreshes as
    logged best-effort cases; keep repository error events observable.
  • Preserve cancellation/timeouts, repository setup errors, systemd failures, and
    the existing certificate failure boundaries.
  • Include orchestration, scheduling, NBC/dry-run, failure-reporting,
    provisioning-order, real-system and VM-fixture coverage.

NBC retains precedence, including Phase 2.5. Traditional CSE/NBC, the physical
shell artifact, CustomData delivery, and VHD mappings remain unchanged. This PR
does not rewrite the provisioning scripts or remove the shell artifact.

At the initial five-part split (8d95830), the stack preserved all 107 original
cloud production/test declarations
from #9891's final implementation, with
13 additional direct capability tests: five from the previous split and
eight covering the newly isolated runtime/store/protocol boundaries. Relative
to the previously published three-part stack, all 112 declarations were retained
with eight additive tests. The declaration-equivalence and exact non-cloud
transplant comparisons refer to that initial split, before this integration
follow-up added E2E infrastructure. The follow-up does not change ANC's
production cloud initialization implementation or the first four PRs.

Series and merge order

  1. [1/5] feat: [1/5] add cloud platform and trust-store capabilities #9910: Platform and trust-store capabilities -- based on main; includes the migration/dependency Mermaid diagrams.
  2. [2/5] feat: [2/5] add cloud certificate retrieval protocols #9911: Certificate retrieval protocols -- based on part 1.
  3. [3/5] feat: [3/5] add cloud certificate refresh orchestration and CLI #9909: Certificate refresh and CLI -- based on part 2.
  4. [4/5] feat: [4/5] add native RepoDepot initialization #9903: RepoDepot capabilities -- based on part 3.
  5. [5/5] feat: [5/5] enable native cloud initialization before provisioning #9904: Native activation and complete-stack integration -- this PR, based on part 4.

The current base is abigailliang/anc-cloud-repodepot for an incremental review
diff. After the preceding PRs merge, merge the updated main into this branch
before retargeting to main and running required CI. This preserves the
incremental diff when prerequisites were squash-merged.
Do not merge this PR into the dependency's feature branch.
This PR remains a draft while its prerequisites and real-VM acceptance are pending.

This PR's original ID, discussion, and branch history are retained through
additive commits; no force push was used. The aggregate #9891 and former
certificate PR #9902 are retained for reference.

Integration acceptance for the complete stack

Test the final PR head, not the preparatory branches in isolation.
The nativecloudinit=true selector registers six real-VM scenarios:

Image Native opt-in Native opt-out Both flags / NBC precedence
Ubuntu 22.04 Ubuntu2204_NativeCloudInit_OptedIn Ubuntu2204_NativeCloudInit_NotOptedIn Ubuntu2204_CloudInit_NBCPrecedence
Azure Linux 3 AzureLinuxV3_NativeCloudInit_OptedIn AzureLinuxV3_NativeCloudInit_NotOptedIn AzureLinuxV3_CloudInit_NBCPrecedence
  • Each scenario builds ANC from the checkout, verifies its SHA256 and retains
    actual provisioning scripts plus the normal Node Ready and pod checks.
    The observer wraps the real legacy initializer in rendered nodecustomdata.yml;
    it does not replace provisioning or shell initialization with a success stub.
  • VM provisioning uses the real Azure WireServer and RCV1P opt-in tag. No
    production endpoint override or VM WireServer redirection is added.
    The supported custom-cloud input contract supplies a nonempty RepoDepot
    endpoint while retaining public Azure ARM, identity and registry endpoints.
  • A loopback proxy passes through actual Ubuntu/Microsoft metadata, signatures,
    keys and packages. Successful metadata counters and cache checks prevent a
    skipped or exhausted RepoDepot path from looking like a happy-path success.
  • Opted-in native VMs also run the guarded Linux cloudintegration suite:
    actual system trust/TLS, gpg, apt/dnf, cron, ten-attempt cache exhaustion with
    Error telemetry, ordinary cron/legacy trust permission failures, certificate
    fatal boundaries and observed in-flight HTTP cancellation. On a real VM it
    additionally exercises systemd timer/service execution and authorization failure.
  • The scheduled PR-built refresh command is executed directly and must leave
    repository/keyring contents and the root crontab's contents/inode/timestamp
    unchanged. System tests restore repository files/crontab and remove their
    unique CA; they must never run on shared or production machines.

Manual CI entrypoint: .pipelines/e2e-anc-cloud-init.yaml.
It requires Microsoft.Compute/PlatformSettingsOverride registration and a
subscription without auto-injected RCV1P tags. Strict mode rejects missing
prerequisites, uses current-checkout compilation and explicit native/NBC modes,
disallows missing-VHD/capacity skips, clears skip tags and uses zero retries.
See execution and coverage details.

The YAML has not been registered as an external ADO pipeline or configured
as a GitHub required check.
The pipeline owner must register it, run the final
stack head and require its successful result before merging the series.
Adding the YAML alone does not enforce a merge gate.

Validation

Passed locally for integration commit 9a99152566:

  • ANC/parser/repository-fixture race suites and the complete E2E module race suite.
    Existing E2E artifact tests used GNU coreutils/timeout on macOS.
  • CI-pinned golangci-lint v2.14.0 for both modules; ANC analysis explicitly
    included the Linux cloudintegration build tag.
  • All eight non-systemd real-system cases in disposable AMD64 containers:
    Ubuntu 22.04 95.16s, Azure Linux 3 80.74s. These execute actual
    trust tools, TLS, gpg, apt/dnf and cron rather than mocked command results.
  • Linux AMD64/ARM64 builds of ANC, the repository proxy and the gated test binary;
    six CLI scenario registrations; real AgentBaker bootstrap rendering/observer
    preservation and Azure CustomData size checks; strict pipeline-YAML contract;
    shellcheck and bash -n for the pipeline script.

Not run: Azure VM/VHD end-to-end provisioning, Node Ready/pod acceptance,
the public-endpoint custom-cloud fixture on a real VM, and systemd service/timer
execution. Container success is not VM acceptance. Azure Linux's complete public
vendor-repository fixture currently targets AMD64; the vendor AMD repository
does not publish ARM64 metadata. Mariner and Flatcar/ACL full-native acceptance
remain separate coverage.

The initial split also passed native template syntax and declaration/non-cloud
transplant comparisons. Regression cases retain provisioning exit code 0
after ordinary cron/cache failures while preserving Error events; cancellation
continues to block provisioning.

Which issue(s) this PR fixes:

No linked issue.

Complete the split from #9891 by connecting native Go initialization and removing the shell call together. Add refresh scheduling, preserve cron/cache best-effort handling and fatal boundaries, retain NBC/dry-run behavior, and carry the original integration coverage and generated fixtures. Preserve all original cloud declarations and latest-main status handling.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: be499bac-1283-498e-b5b1-59d8c91047a4
@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Windows Unit Test Results

  3 files   18 suites   1m 12s ⏱️
563 tests 563 ✅ 0 💤 0 ❌
585 runs  585 ✅ 0 💤 0 ❌

Results for commit 9a99152.

♻️ This comment has been updated with latest results.

Preserve the published activation branch with an additive merge onto the smaller certificate layers. Keep Go activation and shell-call removal atomic, with all original failure policies and regression coverage unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: be499bac-1283-498e-b5b1-59d8c91047a4
@abigailliang-aks-sig-node Abigail Liang (abigailliang-aks-sig-node) changed the title feat: [3/3] enable native cloud initialization before provisioning feat: [5/5] enable native cloud initialization before provisioning Oct 27, 2026
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: be499bac-1283-498e-b5b1-59d8c91047a4
@github-actions

Copy link
Copy Markdown
Contributor

Code Coverage

Package Line Rate Health
test 0% ❌
windows 27% ❌
staging_cse_windows Package 1 65% ➖
debug 0% ❌
provisioningscripts 2% ❌
parts_windows Package 1 73% ➖
Summary 38% (1652 / 6834) ❌

@github-actions

Copy link
Copy Markdown
Contributor

Code Coverage

Package Line Rate Health
github.com/Azure/agentbaker/aks-node-controller 74% ➖
github.com/Azure/agentbaker/aks-node-controller/common 100% ✔
github.com/Azure/agentbaker/aks-node-controller/helpers 68% ➖
github.com/Azure/agentbaker/aks-node-controller/parser 88% ✔
github.com/Azure/agentbaker/aks-node-controller/pkg/gpu 100% ✔
github.com/Azure/agentbaker/aks-node-controller/pkg/nodeconfigutils 89% ✔
github.com/Azure/agentbaker/aks-node-controller/testutil/cloudfixture 80% ✔
github.com/Azure/agentbaker/aks-node-controller/utils 0% ❌
github.com/Azure/agentbaker/apiserver 24% ❌
github.com/Azure/agentbaker/cmd 0% ❌
github.com/Azure/agentbaker/cmd/starter 0% ❌
github.com/Azure/agentbaker/fuzz/api 0% ❌
github.com/Azure/agentbaker/hotfix/render-nodecustomdata 35% ❌
github.com/Azure/agentbaker/pkg/agent 78% ➖
github.com/Azure/agentbaker/pkg/agent/datamodel 77% ➖
github.com/Azure/agentbaker/pkg/agent/toggles 0% ❌
github.com/Azure/agentbaker/pkg/vhdbuilder/datamodel 87% ✔
Summary 75% (6631 / 8760) ➖

@github-actions

Copy link
Copy Markdown
Contributor

Code Coverage

Package Line Rate Health
shellspec spec 28% ❌
Summary 28% (4595 / 16103) ❌

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant