Could not associate Load Balancer in VMSS when using az cli 2.87.0+ versions

[rai-m]

Describe the bug

We are observing an issue after upgrading Azure CLI from 2.75.0 to 2.87.0 and higher versions.

Our Azure DevOps BlueGreen deployment pipeline performs traffic cutover between VM Scale Sets by updating Azure Load Balancer backend pool associations. The deployment works successfully when using Azure CLI 2.75.0, but consistently fails when using Azure CLI 2.87.0+.

Related command

Commands used:

az vmss update --resource-group "${RGNAME_APP}" --name "${VMSSNAME}" --add virtualMachineProfile.networkProfile.networkInterfaceConfigurations[0].ipConfigurations[0].loadBalancerBackendAddressPools id="/subscriptions/${SUBID}/resourceGroups/${RGNAME_LB}/providers/Microsoft.Network/loadBalancers/${LBNAME}/backendAddressPools/${VMSSNAME}-pool"

az vmss update --resource-group "${ACTIVE_RGNAME}" --name "${VMSSNAME_ACTIVE}" --remove virtualMachineProfile.networkProfile.networkInterfaceConfigurations[0].ipConfigurations[0].loadBalancerBackendAddressPools

Errors

Errors observed:

ERROR: Couldn't find 'loadBalancerBackendAddressPools' in
'virtualMachineProfile.networkProfile.networkInterfaceConfigurations[0].ipConfigurations[0]'

Issue script & Debug output

With debug flag:

DEBUG: cli.azure.cli.core.azclierror: Traceback (most recent call last):
File "/opt/az/lib/python3.14/site-packages/azure/cli/core/commands/arm.py", line 742, in _update_instance
return instance[part]
~~~~~~~~^^^^^^
KeyError: 'loadBalancerBackendAddressPools'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/az/lib/python3.14/site-packages/knack/cli.py", line 233, in invoke
cmd_result = self.invocation.execute(args)
File "/opt/az/lib/python3.14/site-packages/azure/cli/core/commands/init.py", line 677, in execute
raise ex
File "/opt/az/lib/python3.14/site-packages/azure/cli/core/commands/init.py", line 820, in _run_jobs_serially
results.append(self._run_job(expanded_arg, cmd_copy))
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/az/lib/python3.14/site-packages/azure/cli/core/commands/init.py", line 789, in _run_job
result = cmd_copy(params)
File "/opt/az/lib/python3.14/site-packages/azure/cli/core/commands/init.py", line 335, in call
return self.handler(*args, **kwargs)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/opt/az/lib/python3.14/site-packages/azure/cli/core/commands/command_operation.py", line 214, in handler
add_properties(instance, arg_values, force_string)
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/az/lib/python3.14/site-packages/azure/cli/core/commands/arm.py", line 576, in add_properties
list_to_add_to = _find_property(instance, list_attribute_path)
File "/opt/az/lib/python3.14/site-packages/azure/cli/core/commands/arm.py", line 756, in _find_property
instance = _update_instance(instance, part, path)
File "/opt/az/lib/python3.14/site-packages/azure/cli/core/commands/arm.py", line 751, in _update_instance
throw_and_show_options(instance, part, path)
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
File "/opt/az/lib/python3.14/site-packages/azure/cli/core/commands/arm.py", line 661, in throw_and_show_options
raise CLIError(error_message)
knack.util.CLIError: Couldn't find 'loadBalancerBackendAddressPools' in 'virtualMachineProfile.networkProfile.networkInterfaceConfigurations[0].ipConfigurations[0]'. Available options:['applicationSecurityGroups', 'name', 'privateIPAddressVersion', 'subnet']

ERROR: cli.azure.cli.core.azclierror: Couldn't find 'loadBalancerBackendAddressPools' in 'virtualMachineProfile.networkProfile.networkInterfaceConfigurations[0].ipConfigurations[0]'. Available options: ['applicationSecurityGroups', 'name', 'privateIPAddressVersion', 'subnet']
ERROR: az_command_data_logger: Couldn't find 'loadBalancerBackendAddressPools' in 'virtualMachineProfile.networkProfile.networkInterfaceConfigurations[0].ipConfigurations[0]'. Available options: ['applicationSecurityGroups', 'name', 'privateIPAddressVersion', 'subnet']
DEBUG: cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7f0c44258670>]

Expected behavior

Expected behavior is for the load balancer to be attached successfully to the VMSS

Environment Summary

azure-cli 2.90.0

core 2.90.0
telemetry 1.1.0

Extensions:
azure-devops 1.0.4

Dependencies:
msal 1.36.0
azure-mgmt-resource 24.0.0

Python location '/opt/az/bin/python3'
Config directory '/agent/_work/_temp/.azclitask-2cLsE4'
Extensions directory '/opt/az/azcliextensions'

Python (Linux) 3.14.6 (main, Aug 26 2026, 01:26:36) [GCC 11.4.0]

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.

[x-engineering-agent]

Bug analysis

Triage: sufficient information. The exact command, error/stack trace, expected attachment, and version/context are supplied. The report identifies CLI 2.75.0 as working and 2.87.0+ as failing, with an actual CLI/core 2.90.0 environment on Linux/Python 3.14.6. No additional requirements are needed to investigate this client-side failure.

Routing: core vm module in Azure/azure-cli, target branch dev; no extension tracker is needed. The vmss command group is registered in this module, despite its different command-group name.

Observed and expected behavior

The failing operation is az vmss update --resource-group ... --name ... --add targeting virtualMachineProfile.networkProfile.networkInterfaceConfigurations[0].ipConfigurations[0].loadBalancerBackendAddressPools, with id=<backend-pool-resource-id>. The captured trace reaches GenericUpdateCommandOperation.handler -> add_properties -> _find_property -> _update_instance, then raises Couldn't find 'loadBalancerBackendAddressPools'. The available options list contains only the keys present in that IP-configuration dictionary.

Adding the first backend-pool association should initialize the optional collection and submit the update, rather than requiring the collection to already exist. The companion az vmss update --remove on the same property is part of the cutover workflow and must continue to detach pools correctly. The supplied traceback proves the --add failure; it does not independently establish a separate --remove failure.

Governed current-source evidence and likely cause

Inspected dev at 7bf31a4fd49c252209732a8b0cf874ecaccdf06a:

1. vm/commands.py lines 276-297 registers vmss update as a legacy generic-update command using get_vmss_modified_by_aaz and update_vmss.
2. custom.py lines 4314-4339 returns the AAZ show result, while VMSSShow._output deserializes it into a client-flattened dictionary. The update getter does not restore omitted optional backend-pool properties.
3. arm.add_properties first looks up the complete collection path. Its existing initialization path only runs when lookup returns None. However, _update_instance / _find_property use instance[part] for dictionaries; a missing final key raises before initialization. An unset model attribute or an explicit dictionary None can therefore work where an omitted dictionary key cannot. This directly explains the reported stack and is consistent with a representation-boundary regression.
4. The current generated response schema still defines loadBalancerBackendAddressPools. The handwritten converter already maps it to load_balancer_backend_address_pools when present. This is not evidence that the service removed the field. The failure happens before the generic handler calls the setter, so changing only update_vmss or that converter cannot fix this traceback.

Concrete fix requirements

• Restore absent-terminal-collection handling at the handwritten generic-update boundary: when --add addresses an omitted final dictionary collection key, resolve and validate its existing parent first, then initialize the requested list. Retain support for explicit None and empty lists, and append to populated lists without replacing existing pool IDs. Keep the change confined to the relevant shared helper and VMSS integration/tests.
• Do not make _find_property globally return None for unknown paths or swallow arbitrary KeyErrors. Missing intermediate objects, invalid indexes, non-unique filters, and non-list targets must still fail with useful errors. Preserve existing model-object behavior, generic argument ordering, --set, and unrelated VMSS/instance/no-wait behavior.
• Exercise the companion removal through the complete request/response path. remove_properties deletes dictionary keys but clears model-list attributes to []; verify that the resulting VMSS request actually clears the association rather than treating omission as preservation. Do not redefine all missing-property removals as successful no-ops. A remove/re-add cutover must not lose unrelated IP-configuration fields.
• This evidence points to handwritten client compatibility logic, not a missing generated schema member. Do not hand-edit vm/aaz/latest/vmss/ or regenerate unrelated commands. If implementation uncovers a genuine durable-model/specification defect, use the mandatory source-first Codegen protocol below.

Focused regression and recording recommendations

• Extend src/azure-cli-core/azure/cli/core/tests/test_generic_update.py with a dictionary-shaped VMSS fixture whose final backend-pool collection is omitted. Cover absent, explicit None, empty, and populated collections; ordered remove/add; preserving existing entries; and the negative path/index/type cases above. Existing coverage exercises emptyProp=None/empty lists and missing-property removal errors, which must remain protected.
• Add a VMSS adapter/command regression in src/azure-cli/azure/cli/command_modules/vm/tests/latest/test_custom_vm_commands.py: mock the AAZ getter's flattened IP-configuration dictionary, apply the reported generic update, and capture the setter arguments to confirm the backend-pool ID survives camelCase-to-snake_case conversion. Include multiple IP configurations and an unrelated-field preservation assertion.
• Extend VMSSUpdateTests in src/azure-cli/azure/cli/command_modules/vm/tests/latest/test_vm_commands.py with a re-recordable no-LB -> attach -> detach -> reattach scenario. Use a supported Uniform VMSS and Standard load balancer, test the supplied full backend-pool resource-ID form (including a separate LB resource group), and check both the outgoing request and a fresh vmss show after each change. Resource preparers must supply resource names/IDs; keep recordings sanitized and replayable. Do not rely only on creating a VMSS that already has a load balancer, because that starts with a populated collection and misses this failure.
• During implementation, run the repository's focused generic-update and VM scenario tests plus scoped style/linter checks; re-record only affected scenarios using the approved workflow. No tests, implementation, or Codegen were executed during this triage. The exact first-bad release and live service behavior have not been independently reproduced here.

Mandatory Codegen execution protocol

Before editing implementation files, determine whether the affected vm command is AAZ-generated. Files under aaz/<profile>/ are generated output and must never be patched directly, including by an AI agent. Check out Azure/aaz beside Azure/azure-rest-api-specs, Azure/aaz-dev-tools, and the downstream repository. API-schema defects start in the specification; command naming, grouping, arguments, API-version selection, help, and examples belong in the durable Azure/aaz command model; non-modelable client behavior belongs in a handwritten subclass or wrapper in custom.py, registered from commands.py. X Engineering Agent creates and promotes the corresponding durable Azure/aaz source pull request before it promotes downstream generated output.

Follow the Azure CLI repository's Codegen workflow and the aaz-dev setup documentation. Set up the checked-out repositories with azdev setup. Use generate only when importing or redesigning command models from Swagger/TypeSpec. For an existing module whose durable Azure/aaz model has been updated, render that model with regenerate:

aaz-dev cli regenerate --name vm --cli-path <azure-cli>

# New/imported command model only:
aaz-dev cli generate --spec <specification-name> --module vm

You MUST actually run the generator; do not merely describe it or imitate its output. If the AAZ/specification checkout, local source change, credentials, or generator is unavailable, stop and report the blocker instead of editing generated files. Inspect _aaz_info provenance and the complete regenerated diff, then run focused azdev style, azdev linter, and azdev test validation. For an extension, also update its version and HISTORY.rst, preserve azext_metadata.json compatibility, and let release automation update src/index.json.

PR title & description format (required)

This repo enforces a PR format (guide). Please author the PR exactly as follows or CI's Check the Format of Pull Request Title and Content will fail.

Use this EXACT PR title (copy verbatim, do not reword):

[VM] Fix #34133: `az vmss update`: Restore adding load balancer backend pools when the property is absent

Keep the backticks around the command and the Fix #34133: prefix. You may only adjust the wording after the command (the final summary) if the fix changes; the [VM] prefix, issue link, and backticked command must stay.

Description — follow the PR template and fill in:

• Link the issue — start the Description with a closing keyword so the PR auto-links and closes it: Fixes Azure/azure-cli#34133.
• Related command — the az ... command this affects.
• Description (mandatory) — why the bug happens, what you changed, and the resulting behavior.
• Testing Guide — example command(s) showing the fix works.
• History Notes — leave the title to drive the history note, or add extra lines in the same format (component in brackets + the command in backticks), e.g. [VM] `az <command>`: <note>.
• Keep the template checklist and tick the items you've satisfied.

[jin-trial]

Hello! Any updates on this issue?