[Extension] Fix #34062: az extension add: Fix Windows dependency installation failures

doc/extensions/authoring.md

@@ -39,6 +39,86 @@ Periodically run the following to ensure your extension will pass CI:
 
 Address comments as appropriate and consult the Azure CLI team if something is unclear.
 
+#### Validating extension dependency installation
+
+On Windows, `az extension add` and `az extension update` prefer compatible dependency
+wheels over newer source distributions. The selected versions must still satisfy
+the extension's requirements and the running Python's compatibility tags. This
+does not pin dependencies or disable pip's build isolation. Linux and macOS retain
+pip's default selection behavior.
+
+The Windows packages use embedded Python with a `python*._pth` file. This file
+disables environment-based Python path configuration, including the `PYTHONPATH`
+that pip uses to expose an isolated build environment. Consequently, a source
+distribution can fail with `BackendUnavailable` even after pip successfully
+installs its build backend. Installing that backend globally is not a substitute
+for build isolation. Prefer an available compatible wheel; an explicit source
+requirement or a dependency with no compatible wheel can still require a
+source-build-capable Python installation and the package's native build tools.
+
+In a configured development checkout with pip and a source-build-capable Python
+installation, run the offline installer regressions:
+
+```bash
+python -m pytest \
+  src/azure-cli-core/azure/cli/core/tests/test_extension_pip.py \
+  src/azure-cli/azure/cli/command_modules/extension/tests/latest/test_extension_install.py
+```
+
+These tests use local package fixtures and real pip subprocesses, including an
+embedded-style interpreter with a retained `_pth` file. They cover wheel selection,
+supported isolated source builds, and installation failures without contacting
+PyPI or Azure. The copied-interpreter case requires CPython 3.11 or later on Linux
+or Windows, with a standalone CPython DLL layout on Windows; it is skipped on
+other hosts. This case reproduces path isolation, not the complete packaged
+Windows runtime.
+The existing broader selectors are
+`src/azure-cli-core/azure/cli/core/tests/test_extension.py` and
+`src/azure-cli-core/azure/cli/core/extension/tests/latest/test_extension_commands.py`.
+The command-module test selector is `extension`; the core selector is
+`azure-cli-core`. Repository validation also includes:
+
+```bash
+azdev style extension azure-cli-core
+azdev linter extension
+azdev test extension azure-cli-core --series
+```
+
+For Windows release validation, use clean x86 and x64 candidate MSI installations
+and the x64 ZIP package. First run `az --version` and the bundled interpreter's
+`-m pip debug --verbose` to capture the actual Python version, architecture, pip
+version, and supported wheel tags. Do not infer architecture from the installation
+directory. For example, the report in [#34062](https://github.com/Azure/azure-cli/issues/34062)
+identifies 32-bit Python explicitly; cryptography 50.0.1 publishes Windows wheels
+for `win_amd64`, not `win32`.
+
+In a temporary PowerShell session, use a separate extension directory:
+
+```powershell
+$previousExtensionDir = $env:AZURE_EXTENSION_DIR
+$testExtensionDir = Join-Path $env:TEMP ("az-extension-" + [guid]::NewGuid())
+try {
+    $env:AZURE_EXTENSION_DIR = $testExtensionDir
+    az extension add --name k8s-extension --version 1.8.0 --debug
+    if ($LASTEXITCODE -ne 0) { throw "Extension installation failed" }
+    az extension show --name k8s-extension
+    if ($LASTEXITCODE -ne 0) { throw "Installed extension was not found" }
+    az k8s-extension --help
+    if ($LASTEXITCODE -ne 0) { throw "Installed extension could not load" }
+} finally {
+    $env:AZURE_EXTENSION_DIR = $previousExtensionDir
+    if (Test-Path $testExtensionDir) { Remove-Item -Recurse -Force $testExtensionDir }
+}
+```
+
+Inspect the pip debug output for the selected dependency artifacts. Also exercise
+an update and a system installation on disposable Windows installations. The
+offline tests do not validate native compilation, packaged Windows DLL loading,
+or current package-index contents. In particular, wheel preference cannot supply
+a wheel that a dependency has not published. Candidate-package Windows runs and
+repository CI remain required; this local installation smoke test needs network
+access but does not provision Azure resources.
+
 ### Publish
 
 **For the extension whose source code is hosted in [Azure/azure-cli-extensions](https://github.com/Azure/azure-cli-extensions)**, we will release for you once your code is merged into `main` branch. You must not update [index.json](https://github.com/Azure/azure-cli-extensions/blob/main/src/index.json) manually in this case.

src/azure-cli-core/azure/cli/core/extension/operations.py

@@ -157,6 +157,9 @@ def _add_whl_ext(cli_ctx, source, ext_sha256=None, pip_extra_index_urls=None, pi
     # Install with pip
     extension_path = build_extension_path(extension_name, system)
     pip_args = ['install', '--target', extension_path, ext_file]
+    if IS_WINDOWS:
+        # Embedded Python's ._pth isolation can prevent pip from importing isolated build backends.
+        pip_args.append('--prefer-binary')
 
     if pip_proxy:
         pip_args = pip_args + ['--proxy', pip_proxy]

src/azure-cli-core/azure/cli/core/tests/extension_pip_test_utils.py

@@ -0,0 +1,177 @@
+# --------------------------------------------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT License. See License.txt in the project root for license information.
+# --------------------------------------------------------------------------------------------
+
+"""Small, generated distributions for exercising extension installation with offline pip."""
+
+import base64
+import csv
+import hashlib
+import io
+import json
+import os
+from pathlib import Path
+import tarfile
+import tempfile
+import zipfile
+from unittest import mock
+
+from azure.cli.core import _session, extension
+
+
+EXTENSION_NAME = 'azcli-extension-pip-test'
+EXTENSION_MODULE = 'azext_pip_test'
+DEPENDENCY_NAME = 'azcli-extension-test-dependency'
+DEPENDENCY_MODULE = 'azcli_extension_test_dependency'
+BACKEND_NAME = 'azcli-extension-test-backend'
+BACKEND_MODULE = 'azcli_extension_test_backend'
+BUILD_LOG_ENV = 'AZCLI_EXTENSION_TEST_BUILD_LOG'
+SENTINEL_ENV = 'AZCLI_EXTENSION_TEST_SENTINEL'
+
+# This backend is installed only as a build requirement, never alongside the extension.
+_BACKEND_SOURCE = '''
+import json
+import os
+from pathlib import Path
+import sys
+import zipfile
+
+
+def build_wheel(wheel_directory, config_settings=None, metadata_directory=None):
+    fixture = json.loads(Path('fixture.json').read_text(encoding='utf-8'))
+    observation = {
+        'backend_file': __file__,
+        'executable': sys.executable,
+        'path': sys.path,
+        'pythonpath': os.environ.get('PYTHONPATH'),
+        'sentinel': os.environ.get('AZCLI_EXTENSION_TEST_SENTINEL'),
+        'http_proxy': os.environ.get('HTTP_PROXY'),
+        'https_proxy': os.environ.get('HTTPS_PROXY'),
+    }
+    Path(os.environ['AZCLI_EXTENSION_TEST_BUILD_LOG']).write_text(
+        json.dumps(observation), encoding='utf-8')
+    if fixture['unsupported']:
+        raise RuntimeError('Synthetic source build is unsupported')
+    with zipfile.ZipFile(Path(wheel_directory) / fixture['wheel'], 'w') as wheel:
+        for path in sorted(Path('wheel').rglob('*')):
+            if path.is_file():
+                wheel.writestr(path.relative_to('wheel').as_posix(), path.read_bytes())
+    return fixture['wheel']
+'''
+
+
+def _wheel_files(name, version, files, requirements=()):
+    dist_info = '{}-{}.dist-info'.format(name.replace('-', '_'), version)
+    contents = {path: text.encode('utf-8') for path, text in files.items()}
+    contents[dist_info + '/METADATA'] = (
+        'Metadata-Version: 2.1\nName: {}\nVersion: {}\n'.format(name, version) +
+        ''.join('Requires-Dist: {}\n'.format(requirement) for requirement in requirements) + '\n'
+    ).encode('utf-8')
+    contents[dist_info + '/WHEEL'] = (
+        b'Wheel-Version: 1.0\nGenerator: extension-pip-test\nRoot-Is-Purelib: true\nTag: py3-none-any\n'
+    )
+    record = io.StringIO()
+    writer = csv.writer(record, lineterminator='\n')
+    for path, data in contents.items():
+        digest = base64.urlsafe_b64encode(hashlib.sha256(data).digest()).rstrip(b'=').decode('ascii')
+        writer.writerow((path, 'sha256=' + digest, len(data)))
+    writer.writerow((dist_info + '/RECORD', '', ''))
+    contents[dist_info + '/RECORD'] = record.getvalue().encode('utf-8')
+    return contents
+
+
+def _write_wheel(directory, name, version, files, requirements=()):
+    path = directory / '{}-{}-py3-none-any.whl'.format(name.replace('-', '_'), version)
+    with zipfile.ZipFile(path, 'w') as wheel:
+        for filename, data in _wheel_files(name, version, files, requirements).items():
+            wheel.writestr(filename, data)
+    return path
+
+
+class ExtensionPipFixture:
+    """Own the local packages, extension targets and restored environment for one test."""
+
+    def __init__(self, testcase):
+        directory = tempfile.TemporaryDirectory(prefix='azure-cli-extension-pip-')
+        testcase.addCleanup(directory.cleanup)
+        self.root = Path(directory.name)
+        self.links = self.root / 'links'
+        self.links.mkdir()
+        self.user_dir = self.root / 'user'
+        self.system_dir = self.root / 'system'
+        self.build_log = self.root / 'build.json'
+        scratch = self.root / 'scratch'
+        scratch.mkdir()
+
+        # Ignore caller pip configuration (including constraints and binary policy), not other environment.
+        environment = {key: value for key, value in os.environ.items() if not key.startswith('PIP_')}
+        environment.update({
+            'PIP_NO_INDEX': '1',
+            'PIP_FIND_LINKS': self.links.as_uri(),
+            'PIP_CONFIG_FILE': os.devnull,
+            'PIP_NO_INPUT': '1',
+            'PIP_PROGRESS_BAR': 'off',
+            'PYTHONDONTWRITEBYTECODE': '1',
+            'PYTHONNOUSERSITE': '1',
+            'TMPDIR': str(scratch),
+            'TEMP': str(scratch),
+            'TMP': str(scratch),
+            'AZURE_CONFIG_DIR': str(self.root / 'config'),
+            BUILD_LOG_ENV: str(self.build_log),
+            SENTINEL_ENV: 'inherited-by-pip-and-build-backend',
+        })
+        for patcher in (
+            mock.patch.dict(os.environ, environment, clear=True),
+            mock.patch.object(tempfile, 'tempdir', str(scratch)),
+            mock.patch.object(extension, 'EXTENSIONS_DIR', str(self.user_dir)),
+            mock.patch.object(extension, 'EXTENSIONS_SYS_DIR', str(self.system_dir)),
+            mock.patch.object(extension, 'DEV_EXTENSION_SOURCES', []),
+            mock.patch('azure.cli.core.util.handle_version_update'),
+            mock.patch('azure.cli.core.extension._homebrew_patch.is_homebrew', return_value=False),
+        ):
+            patcher.start()
+            testcase.addCleanup(patcher.stop)
+        for session in vars(_session).values():
+            if isinstance(session, _session.Session):
+                for attribute, value in (('filename', None), ('data', {})):
+                    patcher = mock.patch.object(session, attribute, value)
+                    patcher.start()
+                    testcase.addCleanup(patcher.stop)
+
+        _write_wheel(self.links, BACKEND_NAME, '1.0', {BACKEND_MODULE + '.py': _BACKEND_SOURCE})
+        self.dependency_wheel = _write_wheel(
+            self.links, DEPENDENCY_NAME, '1.0', {DEPENDENCY_MODULE + '/__init__.py': "__version__ = '1.0'\n"})
+        self.make_sdist()
+        self.extension_wheel = self.make_extension()
+
+    def make_extension(self, requirement='>=1.0'):
+        return _write_wheel(self.links, EXTENSION_NAME, '1.0', {
+            EXTENSION_MODULE + '/__init__.py': '',
+            EXTENSION_MODULE + '/azext_metadata.json': '{"azext.isPreview": false}',
+        }, [DEPENDENCY_NAME + requirement])
+
+    def make_sdist(self, unsupported=False):
+        name = DEPENDENCY_NAME.replace('-', '_')
+        files = {
+            'wheel/' + path: data for path, data in _wheel_files(
+                DEPENDENCY_NAME, '2.0', {DEPENDENCY_MODULE + '/__init__.py': "__version__ = '2.0'\n"}
+            ).items()
+        }
+        files['pyproject.toml'] = (
+            '[build-system]\nrequires = ["{}==1.0"]\nbuild-backend = "{}"\n'.format(BACKEND_NAME, BACKEND_MODULE)
+        ).encode('utf-8')
+        files['fixture.json'] = json.dumps({
+            'wheel': name + '-2.0-py3-none-any.whl',
+            'unsupported': unsupported,
+        }).encode('utf-8')
+        path = self.links / (name + '-2.0.tar.gz')
+        with tarfile.open(path, 'w:gz') as sdist:
+            for filename, data in files.items():
+                info = tarfile.TarInfo(name + '-2.0/' + filename)
+                info.size = len(data)
+                sdist.addfile(info, io.BytesIO(data))
+        return path
+
+    def target(self, system=False):
+        return (self.system_dir if system else self.user_dir) / EXTENSION_NAME