diff --git a/utilities/README.md b/utilities/README.md index d966a6dd..8669f7cd 100644 --- a/utilities/README.md +++ b/utilities/README.md @@ -9,6 +9,7 @@ This directory contains standalone helper scripts for operators standing up or m | Utility | What it does | When to run | |---|---|---| | [`onboarding-tool/`](onboarding-tool/) | Five-stage customer onboarding flow for planning, landing-zone preparation, validation, platform deployment and scenario certification. | Start here for a new Microsoft Discovery customer deployment. Follow the [`USER-GUIDE.md`](onboarding-tool/scripts/USER-GUIDE.md); maintainers use the [`scripts/README.md`](onboarding-tool/scripts/README.md). | +| [`regional-deployment/`](regional-deployment/) | Optional Python/Azure CLI/Bicep quota/capacity recommendations and reviewed regional deployment, including retained-compute/storage Workspace recovery with peering and private DNS. | Use after a verified regional blocker; read its own prerequisites and approve the target region/plan before any deployment. It does not change the five-stage tool or automatically move resources. | | [`resource-provider-registration/`](resource-provider-registration/) | Standalone registration of the Azure resource providers Microsoft Discovery needs. Cross-platform PowerShell; auto-installs `Az.Accounts` / `Az.Resources`. | Use when registering providers outside the onboarding tool. Stage 2 already performs this step. | | [`rbac-roles-assignment/`](rbac-roles-assignment/) | Standalone assignment of the Azure RBAC roles for a Discovery Platform Administrator or Scientist persona. | Use for access maintenance outside onboarding. Stage 2 and Stage 4 handle onboarding-time assignments. | | [`dataasset-migration/`](dataasset-migration/) | Migrates a Discovery v1 `DataContainer` + child `DataAssets` to the v2 `StorageContainer` + `StorageAssets` shape (API `2025-07-01-preview` → `2026-02-01-preview`). Control-plane metadata only — does **not** move blobs. | One-time migration for tenants that onboarded before the v2 storage model. | diff --git a/utilities/regional-deployment/.gitignore b/utilities/regional-deployment/.gitignore new file mode 100644 index 00000000..2338c288 --- /dev/null +++ b/utilities/regional-deployment/.gitignore @@ -0,0 +1,5 @@ +out/ +.state/ +evidence/ +__pycache__/ +*.pyc diff --git a/utilities/regional-deployment/README.md b/utilities/regional-deployment/README.md new file mode 100644 index 00000000..341b8586 --- /dev/null +++ b/utilities/regional-deployment/README.md @@ -0,0 +1,326 @@ +# Opt-in regional fallback for Discovery services + +This optional **Python 3.10+ / Azure CLI / Bicep** utility helps operators recover from +verified regional quota or capacity blockers. It does not change the existing +five-stage PowerShell onboarding flow or automatically relocate resources. + +It provides two explicit deployment boundaries: + +| Mode | Placement and preservation | +| --- | --- | +| `fresh` | If the source compute has not succeeded, generate a new complete stack in an explicitly named **different RG** and selected workload region. Preserve the source RG; do not migrate its data or delete partial resources. | +| `workspace` | If the original Supercomputer, node pool and Discovery StorageContainer have succeeded, retain them and the private customer storage. Create only a new Workspace-side network, identity and platform resources in another workload region. The Discovery ARM control-plane region remains unchanged. | + +The Workspace mode moves the **workspace managed dependency unit**, including its +Search/Foundry/Cosmos services. Discovery does not expose an independent Search +region or Search resource ID in this API. No unsupported per-service placement +field is invented here. + +## Safety and prerequisites + +- Use Azure **public cloud**, an enabled Discovery subscription and Discovery API + `2026-06-01`. Current control-plane choices are `eastus`, `uksouth` and + `swedencentral`; actual create support must still be checked. +- Install Python 3.10+, Azure CLI, Bicep and the Azure CLI `quota` extension. + Python uses only its standard library. This utility does not need PowerShell. +- Sign in as the approved interactive user in the explicit subscription/tenant. + Directory reads must resolve the user and Discovery first-party service principal. +- Provider, persona and first-party Reader/NSP prerequisites follow the + [Discovery registration guide](https://learn.microsoft.com/en-us/azure/microsoft-discovery/concept-resource-provider-registration) + and [role assignments](https://learn.microsoft.com/en-us/azure/microsoft-discovery/concept-role-assignments). + `prepare --apply` writes Providers, the confirmed-missing RG and IAM, including a + minimal subscription-scoped custom NSP role. Use approved Owner, or Contributor + plus User Access Administrator, with required PIM access activated. Do not + assume resource-write permission grants directory or role-definition authority. +- Review data residency, cross-region transfer charges, latency, peering and + private DNS before `select --accept-cross-region`. GlobalStandard model + processing is not guaranteed to remain in the resource's region. +- No automatic deletion, quota application, policy exemption, public-storage + enablement, NSP skip tag, agent creation, inference or scientific job is included. +- Keep configuration, reports, user/tenant information and command evidence in + ignored `out/`. Never commit tokens, storage keys, SAS or private evidence. + +All examples below run from `utilities/regional-deployment`. + +```bash +az login --tenant '' +az account set --subscription '' +az bicep version +az extension show --name quota +# If absent, install the quota extension under your approved tooling process: +# az extension add --name quota +``` + +## 1. Generate the original/default-region inputs + +```bash +python3 regional_deploy.py init \ + --subscription '' --tenant '' \ + --resource-group discovery-primary --name-prefix discprimary \ + --control-plane-region eastus --workload-region eastus2 \ + --vnet-cidr 10.20.0.0/16 --output-dir out/primary +``` + +`init` writes a **new local directory only**, without Azure calls or overwrites. +The output is this utility's `deployment.config.json` and `parameters.json`, not +the Stage 1 PowerShell config format. Parameter paths are relative to the config's +directory; Bicep templates are fixed inside this utility. + +For the ordinary new deployment, review the inputs, prepare prerequisites, then +run `preflight`, `validate`, and the explicitly approved deployment: + +```bash +python3 regional_deploy.py prepare --config out/primary/deployment.config.json --apply +python3 regional_deploy.py preflight --config out/primary/deployment.config.json --stage full +python3 regional_deploy.py validate --config out/primary/deployment.config.json --stage full +# Review the saved JSON what-if before any actual deployment. +python3 regional_deploy.py deploy --config out/primary/deployment.config.json --stage full --apply +python3 regional_deploy.py wait --config out/primary/deployment.config.json --timeout 7200 +``` + +When a prerequisite or deployment fails, stop. Do not automatically replay the +mutation or rerun a full foundation template against policy-managed networking. + +### Adopt an existing deployment for recovery + +Generate local source inputs with `init`, then set the **source** config and +parameter names/regions/CIDRs to the actual original resources before requesting +recommendations. Do not run `full` against that source to reconstruct it. + +Important mappings from the PowerShell Stage 1 config: + +| Existing Stage 1 field | This utility's source input | +| --- | --- | +| `subscriptionId`, target tenant, `resourceGroup` | Config `subscriptionId`, `tenantId`, `resourceGroup` | +| `controlPlaneRegion`, `workloadRegion` | Config `location`, `dataPlaneLocation`; parameters `location`, `overridemrgregion` must match | +| `names.supercomputer`, `names.nodePool`, `names.workspace` | Parameters `supercomputerName`, `nodePoolName`, `workspaceName` | +| `names.storageContainer`, `storage.account` | Parameters `storageContainerName`, `storageAccountName` | +| Actual network name/address space/PE subnet prefix | Parameters `vnetName`, `vnetAddressPrefix`, `privateEndpointSubnetPrefix` | + +Workspace reuse currently requires one customer RG containing the retained +compute/store references, original VNet, storage account, Blob PE named +`pe--blob`, and the original +`privatelink.blob.core.windows.net` zone. It verifies a single original Blob A +record. Centralized DNS in another RG, custom/multiple endpoint layouts, +cross-subscription/RG reuse and healthy-workspace migration require a separately +reviewed design; do not rename or recreate existing resources to fit this baseline. + +## 2. Request read-only neighboring-region recommendations + +```bash +python3 regional_deploy.py recommend \ + --config out/primary/deployment.config.json \ + --candidates centralus northcentralus \ + --report out/recommendations.json +``` + +Without `--failure-file`, the default-region probe must find a machine-verified +quota deficit or unavailable Standard Search offering before suggesting a move. +After a real failed allocation, also supply its fresh structured ARM error JSON +or deployment-operations JSON: + +```bash +python3 regional_deploy.py recommend \ + --config out/primary/deployment.config.json \ + --failure-file /path/to/fresh-arm-error.json \ + --candidates centralus northcentralus \ + --report out/recommendations-after-failure.json +``` + +Readiness records for `model-quota` / `vcpu-quota` are accepted only when they +contain a numeric `available < required` deficit. Generic `InternalServerError`, +plain-text guesses, unreadable quota, `InvalidResourceType`, access failures and +policy failures do not justify relocation. A structured allocation failure may +justify fallback even when the catalog currently advertises a tier: catalogs are +not physical capacity reservations. + +The report: + +- Separates quota limits from physical allocation/capacity failures. +- Ranks candidates by Haversine distance using current subscription location + coordinates, not a hard-coded neighbor/paired-region assumption. This distance + is only a geographic heuristic, not a latency, residency or cost guarantee. +- Defaults to the same `geographyGroup`. Broader discovery requires the explicit + `--allow-cross-geography` flag and a separate residency review at selection. + `--candidates` is an optional operator-approved allowlist. `--limit` bounds live + probes to 1–8 locations, default 3; this is not an exhaustive capacity search. +- Checks Standard Search offerings and exact tier-count quota; Container Apps + environment quota; compatible OpenAI GPT-5.4 GlobalStandard catalog/rate-limit + metadata and regional TPM; and VM SKU, regional/family vCPU quota when new + compute is actually needed. Three system nodes and the configured maximum + workload nodes are budgeted, with both VM families checked separately. +- Infers `workspace` scope when the exact source compute is Succeeded; otherwise + it uses `fresh`. In-progress or unknown compute blocks fallback. It will not + silently propose a duplicate of healthy compute. +- Marks unknown evidence as **not eligible**. It does not reserve capacity, + prove all backing-service availability, validate organizational policy, or + authorize a deployment. + +Exit `0` means at least one candidate is eligible for the next preflight; `2` +means no candidate is verified/no actionable blocker. Runtime errors exit `1`. + +Workspace-only recovery also requires a Succeeded node pool and StorageContainer. +It stops for a retained-compute quota/AKS blocker: relocating Workspace cannot fix +the original compute region's quota. Repair/request that quota, or separately +approve a genuinely new compute-stack plan; this utility does not move an existing +Supercomputer/node pool to another region or silently duplicate it. + +## 3A. Select a fresh fallback stack + +When the report scope is `fresh`, explicitly name a different target RG: + +```bash +python3 regional_deploy.py select \ + --config out/primary/deployment.config.json \ + --recommendation out/recommendations.json --mode fresh --region centralus \ + --target-resource-group discovery-fallback --name-prefix discfallback \ + --vnet-cidr 10.60.0.0/16 --accept-cross-region --output-dir out/fallback +``` + +Selection writes local config/parameters only. It requires a recommendation +less than one hour old with unchanged source inputs/utility code and all selected +candidate checks passed. It preserves the control plane and workload shape, +uses new names, and never edits the source group. Review the new storage/account +plan: this is an isolated new stack, **not data migration**. + +Use the same `prepare → preflight --stage full → validate --stage full → +deploy --stage full --apply → wait` sequence from section 1 against +`out/fallback/deployment.config.json`. All writes still require approval. + +## 3B. Select a workspace-only cross-region recovery + +When the report scope is `workspace`: + +```bash +python3 regional_deploy.py select \ + --config out/primary/deployment.config.json \ + --recommendation out/recommendations.json --mode workspace --region centralus \ + --name-prefix discworkspace --vnet-cidr 10.60.0.0/16 \ + --accept-cross-region --output-dir out/workspace +``` + +The plan retains original Supercomputer/node pool/Discovery StorageContainer and +customer storage IDs. It generates a non-overlapping new VNet, workspace/agent/PE +subnets, NAT/IP, UAMI, four scoped workload roles, two global-VNet peerings and a +link from the new VNet to the **original** Blob DNS zone. The Workspace uses the +selected managed-resource region and references the retained compute/store. +There is no second customer storage account, Blob PE or A-record write, nor a PUT +of the retained VNet/compute/store. Service-managed backing accounts/endpoints for +the new Workspace remain service owned and can incur additional costs. +The new Project reuses output storage, not the old Workspace's agents, +conversations or Workbench state; those are not migrated. Retain old data-bearing +managed resources until their owners approve any later migration/cleanup. + +If another Failed workspace owns the retained compute, specify its exact name +with `--source-workspace`. A Succeeded/in-progress source is not migrated. + +### Explicit association release, only when needed + +If the confirmed Failed source still owns the exact compute association, separately +review and approve the filtered source Workspace PUT: + +```bash +python3 regional_deploy.py release-link \ + --config out/workspace/deployment.config.json --release-method put --apply +``` + +This preserves the source location/tags/UAMI/subnets and removes only the exact +compute link. It can retry service processing for the old failed Workspace; it is +not a harmless metadata-only operation. It never deletes the Workspace/compute, +does not detach healthy sources, and does not automatically retry a rejected +PATCH as a PUT. If the operation is ambiguous, GET the source/link and deployment +state before doing anything else. Do not run it when no release is required. + +### New workspace-side foundation and platform + +```bash +python3 regional_deploy.py preflight --config out/workspace/deployment.config.json --stage workspace +python3 regional_deploy.py validate --config out/workspace/deployment.config.json --stage workspace +# Review: exactly the planned 14 new/child resource IDs; no Modify/Delete. +python3 regional_deploy.py deploy --config out/workspace/deployment.config.json --stage workspace --apply +python3 regional_deploy.py wait --config out/workspace/deployment.config.json --timeout 7200 +``` + +If the new foundation already succeeded, confirm its subnets, roles, NAT, peerings +and DNS link, then change only this config's `deploymentName` to a new `-core` +name. Run fresh preflight/validate/deploy with `--stage workspace-core` instead of +replaying the network. That stage writes only the exact Workspace/model/Project +IDs. It defaults to `workspace-core` for reuse configs. Incomplete foundations +must be reconciled with a separately reviewed minimal plan, not a blind full PUT. + +## 4. Verify and initialize access + +For either target config: + +```bash +python3 regional_deploy.py verify --config out/workspace/deployment.config.json +python3 regional_deploy.py postconfigure --config out/workspace/deployment.config.json --apply +python3 regional_deploy.py audit --config out/workspace/deployment.config.json +``` + +For a fresh fallback, substitute `out/fallback/deployment.config.json`. +The templates create the six logical Discovery resources, including the Project +and chat-model deployment. `postconfigure` resolves the actual Workspace MRG and +verifies/adds Foundry User for the current user, reusing inherited coverage. +Audit checks actual states, IDs/relationships, required backing services, +approved endpoints, roles, private storage and, in reuse mode, peering/retention. +The authenticated project check is **GET agents only**, not inference. + +Require both `audit.json` fields `ok=true` and +`authenticatedProjectReadSucceeded=true` for those respective checks. An +unreachable private API is separately recorded as WARN, not passed as reachable. +Private Workbench/Blob clients require an approved VPN/ExpressRoute, routing and +DNS design; this utility does not install a VPN, gateway transit or custom DNS +forwarding. Packet-level compute/tool execution and scientific certification are +separate tests and may be billable. + +## Receipts, partial outcomes and limits + +- Every environment has its own config-relative `.state/` receipts and timestamped + `evidence/`. A new failed preflight/validation invalidates old success. Compiled + template, filtered parameters, inputs, stage and utility-code hashes are bound + to a one-hour deployment receipt. +- Default what-if policy rejects Modify, Delete, Unsupported and unknown results. + Explicit `--allow-modify` permits reviewed changes, never Delete/Unsupported; + the initial reuse foundation still cannot Modify anything. Reuse writes are + additionally exact-ID allowlisted; resume compiled shapes are bounded. +- `status`/`wait` are read-only. An existing Running/Accepted deployment is not + resubmitted; timeouts do not cancel Azure. There is no automatic mutation replay. +- Quota is checked only for missing model allocations on resume (baseline 250K + workspace + 200K chat TPM). Existing Succeeded allocations are not double counted. +- Failed old MRGs/resources may remain billable. Inventory and cleanup are separate + approved work; never delete the customer RG or shared compute/storage/DNS to + clean up one failed Workspace. +- Live checks are necessary, not sufficient: service-selected model versions, + Storage/Cosmos/other backing capacity, policy, role propagation and private data + paths still need validation and post-deployment readback. No scientific or + inference success is inferred from ARM success. + +## Local validation + +From the repository root: + +```bash +python3 utilities/regional-deployment/check.py +# Python-only unit checks when Azure CLI/Bicep is unavailable: +python3 utilities/regional-deployment/check.py --skip-bicep +``` + +Tests use synthetic data, enforce a read-only query allowlist, and cover trigger +classification, distance/geography, quota units, partial-state selection, immutable +inputs, bounded writes and release guards. The default check also builds all four +Bicep templates and verifies the six-resource/14-resource/three-resource shapes. +It does not log in, run ARM what-if, create resources or send inference. Offline +validation is not an end-to-end deployment certification. + +The code/template layout is self-contained: `regional_deploy.py` is the entry, +`capacity.py` handles read-only evidence, `inputs.py` local input generation, +`deploy.py` the guarded workflow, and `workspace.py` the retention workflow. +This utility does not import scripts from an operator's project directory. + +## References + +- [Infrastructure quickstart](https://learn.microsoft.com/en-us/azure/microsoft-discovery/quickstart-infrastructure?tabs=bicep) — baseline resource relationships and managed-region override. +- [Quota reservation](https://learn.microsoft.com/en-us/azure/microsoft-discovery/concept-quota-reservation). +- [Private endpoint DNS integration](https://learn.microsoft.com/en-us/azure/private-link/private-endpoint-dns-integration). +- [Existing PowerShell onboarding user guide](../onboarding-tool/scripts/USER-GUIDE.md) — a separate interface and permission model; its configuration is not interchangeable with this utility. diff --git a/utilities/regional-deployment/capacity.py b/utilities/regional-deployment/capacity.py new file mode 100644 index 00000000..7dabfcf6 --- /dev/null +++ b/utilities/regional-deployment/capacity.py @@ -0,0 +1,268 @@ +"""Read-only regional evidence; quota, offerings and physical capacity are distinct.""" + +from datetime import datetime, timezone +import json +import math +import re +from urllib.parse import urlsplit + +from deploy import search_dedicated_skus + + +READ_PREFIXES = ( + ("account", "show"), ("provider", "show"), + ("vm", "list-skus"), ("vm", "list-usage"), ("quota", "list"), ("quota", "usage", "list"), + ("cognitiveservices", "model", "list"), ("cognitiveservices", "usage", "list"), + ("network", "vnet", "show"), ("network", "private-dns", "record-set", "a", "show"), + ("storage", "account", "show"), + ("group", "show"), +) +QUOTA_CODES = {"insufficientquota", "quotaexceeded", "vmskuquota"} +CAPACITY_CODES = {"insufficientresourcesavailable", "akscapacityheavyusage", "allocationfailed", "zonalallocationfailed"} +BLOCKING_CODES = { + "authorizationfailed", "authenticationfailed", "invalidauthenticationtoken", "forbidden", + "requestdisallowedbypolicy", "invalidresourcetype", "noregisteredproviderfound", + "missingsubscriptionregistration", "regionmismatch", "resourcenotfound", "resourcegroupnotfound", + "linkedauthorizationfailed", "denyassignmentauthorizationfailed", "subscriptiondisabled", "scopelocked", +} + + +def read_query(runner, label, arguments, **kwargs): + args = tuple(arguments) + if args and args[0] == "rest": + if args.count("--url") != 1 or args.count("--method") != 1: + raise RuntimeError("Read-only REST requires one explicit method and URL.") + uri = urlsplit(args[args.index("--url") + 1]) + method = args[args.index("--method") + 1].lower() + if uri.scheme != "https" or uri.netloc.lower() != "management.azure.com": + raise RuntimeError("Read-only evidence is restricted to the public Azure ARM host.") + offerings = uri.path.lower() == "/providers/microsoft.search/fetchofferings" + if method != "get" and not (method == "post" and offerings): + raise RuntimeError("Regional recommendations cannot mutate Azure resources.") + elif not any(args[:len(prefix)] == prefix for prefix in READ_PREFIXES): + raise RuntimeError("Azure command is not in the regional read-only allowlist.") + return runner.az(label, list(args), **kwargs) + + +def classify_failure(payload): + codes, causes = set(), set() + + def visit(value): + if isinstance(value, dict): + code = str(value.get("code", "")).lower() + if code: + codes.add(code) + if code == "operationnotallowed" and re.search(r"exceeding approved .+ quota|insufficient .+ quota", str(value.get("message", "")), re.I): + causes.add("quota") + # Native/readiness quota evidence must contain an actual numeric deficit. + if (value.get("kind") == "quota" or value.get("id") in ("model-quota", "vcpu-quota")) and str(value.get("status", "")).upper() == "FAIL": + data = value.get("data") or {} + if finite_number(data.get("available")) and finite_number(data.get("required")) and data["available"] < data["required"]: + causes.add("quota") + if value.get("kind") == "capacity" and value.get("status") == "FAIL": + data = value.get("data") or {} + if isinstance(data.get("offeredSkus"), list) and data.get("requiredSku") == "standard" and "standard" not in data["offeredSkus"]: + causes.add("capacity") + for child in value.values(): + visit(child) + elif isinstance(value, list): + for child in value: + visit(child) + elif isinstance(value, str) and value.startswith(("{", "[")): + try: + decoded = json.loads(value) + except ValueError: + return + if isinstance(decoded, (dict, list)): + visit(decoded) + visit(payload) + if codes & BLOCKING_CODES: + return {"actionable": False, "kind": "blocked", "codes": sorted(codes), + "detail": "Resolve access, policy, resource/API or topology errors; do not relocate to bypass them."} + if codes & QUOTA_CODES: + causes.add("quota") + if codes & CAPACITY_CODES: + causes.add("capacity") + return {"actionable": bool(causes), "kind": "+".join(sorted(causes)) or "unverified", "codes": sorted(codes), + "detail": "Quota is an allocation limit; capacity is physical availability. Neither is a reservation."} + + +def finite_number(value): + return isinstance(value, (int, float)) and not isinstance(value, bool) and math.isfinite(value) + + +def short_name(value): + if not isinstance(value, str) or not re.fullmatch(r"[a-z][a-z0-9]+", value): + raise ValueError("Use canonical lowercase Azure region short names.") + return value + + +def location_metadata(locations, name): + matches = [item for item in locations if item.get("name") == name] + if len(matches) != 1: + raise ValueError("Region is missing or ambiguous in subscription location metadata: " + name) + meta = matches[0].get("metadata") or {} + try: + lat, lon = float(meta["latitude"]), float(meta["longitude"]) + except (KeyError, TypeError, ValueError) as error: + raise ValueError("Missing region coordinates; proximity is unverified: " + name) from error + if not -90 <= lat <= 90 or not -180 <= lon <= 180 or not meta.get("geographyGroup"): + raise ValueError("Invalid coordinates/geography; proximity is unverified: " + name) + return lat, lon, meta["geographyGroup"] + + +def nearby_regions(locations, origin, candidates=None, allow_cross_geography=False): + lat, lon, geography = location_metadata(locations, short_name(origin)) + selected = candidates if candidates is not None else [x.get("name") for x in locations if x.get("name") != origin] + result = [] + for name in dict.fromkeys(selected): + short_name(name) + if name == origin: + continue + try: + other_lat, other_lon, other_geography = location_metadata(locations, name) + except ValueError: + if candidates is not None: + raise + continue + if other_geography != geography and not allow_cross_geography: + continue + a, b = math.radians(lat), math.radians(other_lat) + haversine = math.sin((b-a)/2)**2 + math.cos(a)*math.cos(b)*math.sin(math.radians(other_lon-lon)/2)**2 + distance = 6371 * 2 * math.asin(math.sqrt(min(1, max(0, haversine)))) + result.append({"region": name, "distanceKm": round(distance, 1), "geographyGroup": other_geography, + "crossGeography": other_geography != geography}) + return sorted(result, key=lambda row: (row["distanceKm"], row["region"])) + + +def quota_name(row): + name = row.get("name", "") + return str(name.get("value", "") if isinstance(name, dict) else name).lower() + + +def unique_row(rows, name): + found = [row for row in rows if quota_name(row) == name.lower()] + if len(found) != 1: + raise ValueError("Quota row missing or ambiguous: " + name) + return found[0] + + +def quota_values(row): + limit, used = row.get("limit"), row.get("currentValue") + def native_number(value): + if isinstance(value, str) and re.fullmatch(r"\d+(?:\.\d+)?", value): + return float(value) + return value + limit, used = native_number(limit), native_number(used) + if not all(finite_number(value) for value in (limit, used)) or not 0 <= used <= limit: + raise ValueError("Quota values are unknown, malformed or inconsistent.") + return limit, used + + +def model_quota_units(catalog, name): + matches = [] + for row in catalog: + model = row.get("model") or {} + if row.get("kind") not in ("OpenAI", "AIServices") or model.get("name") != name or model.get("format") != "OpenAI": + continue + if model.get("lifecycleStatus") != "GenerallyAvailable": + continue + for sku in model.get("skus", []): + if sku.get("name") != "GlobalStandard" or sku.get("usageName") != "OpenAI.GlobalStandard." + name: + continue + for rate in sku.get("rateLimits", []): + labels = " ".join(str(rate.get(key) or "") for key in ("key", "name", "metric", "type", "unit")).strip().lower() + if labels and "token" not in labels: + continue + try: + renewal, count = float(rate["renewalPeriod"]), float(rate["count"]) + except (KeyError, TypeError, ValueError): + continue + if renewal == 60 and count == 1000: + matches.append(sku["usageName"]) + if not matches: + raise ValueError("No compatible GA AIServices/OpenAI GlobalStandard offering with verified 1K TPM quota units.") + return matches[0] + + +def remaining_model_tpm(workspace_state, chat_state, workspace_tpm=250000, chat_tpm=200000): + return (0 if workspace_state == "Succeeded" else workspace_tpm) + (0 if chat_state == "Succeeded" else chat_tpm) + + +class RegionProbe: + def __init__(self, runner): + self.runner = runner + self.offerings = None + self.skus = {} + + def query(self, label, args, **kwargs): + return read_query(self.runner, label, args, **kwargs) + + def probe(self, region, include_compute=True, required_model_tpm=450000, new_workspace=True): + short_name(region) + checks = [] + + def add(name, condition, detail, kind="offering", data=None): + checks.append({"name": name, "status": "PASS" if condition else "FAIL", "kind": kind, + "detail": detail, "data": data or {}}) + + def check_quota(name, row, required, factor=1): + limit, used = quota_values(row) + available = (limit-used)*factor + add(name, available >= required, f"limit={limit}; used={used}; required={required}; units factor={factor}", + "quota", {"available": available, "required": required, "limit": limit, "used": used}) + + try: + if new_workspace: + if self.offerings is None: + self.offerings = self.query("search-offerings", ["rest", "--method", "post", "--url", + "https://management.azure.com/providers/Microsoft.Search/fetchOfferings?api-version=2026-09-01-preview"], scoped=False) + offered = search_dedicated_skus(self.offerings, region) + add("Search Standard offering", "standard" in offered, ", ".join(sorted(offered)) or "No dedicated offering", "capacity", + {"offeredSkus": sorted(offered), "requiredSku": "standard"}) + usage = self.query(region+"-search-usage", ["rest", "--method", "get", "--url", + f"https://management.azure.com{self.runner.scope}/providers/Microsoft.Search/locations/{region}/usages?api-version=2025-05-01"], scoped=False) + check_quota("Search Standard service count", unique_row(usage.get("value", []), "standard"), 1) + scope = f"{self.runner.scope}/providers/Microsoft.App/locations/{region}" + limits = self.query(region+"-aca-quota", ["quota", "list", "--scope", scope]) + usages = self.query(region+"-aca-usage", ["quota", "usage", "list", "--scope", scope]) + limit = unique_row(limits, "managedenvironmentcount")["properties"]["limit"]["value"] + used = unique_row(usages, "managedenvironmentcount")["properties"]["usages"]["value"] + check_quota("Container Apps environments", {"limit": limit, "currentValue": used}, 4) + if required_model_tpm: + catalog = self.query(region+"-model-catalog", ["cognitiveservices", "model", "list", "--location", region]) + usage_name = model_quota_units(catalog, self.runner.values["chatModelName"]) + usage = self.query(region+"-model-usage", ["cognitiveservices", "usage", "list", "--location", region]) + row = unique_row(usage, usage_name) + quota_status = row.get("status") or row.get("quotaStatus") + if quota_status not in (None, "Included"): + raise ValueError("Model quota allocation status is not verified as usable: " + str(quota_status)) + check_quota("Model TPM", row, required_model_tpm, 1000) + if include_compute: + needs = {} + for sku_name, count in ((self.runner.values.get("systemSku", "Standard_D4s_v6"), 3), (self.runner.values["nodePoolVmSize"], self.runner.values["nodePoolMaxNodeCount"])): + if not re.fullmatch(r"Standard_[A-Za-z0-9_]+", sku_name) or not isinstance(count, int) or isinstance(count, bool) or count < 0: + raise ValueError("Invalid VM SKU or node-count input.") + key = (region, sku_name) + if key not in self.skus: + self.skus[key] = self.query(region+"-sku-"+sku_name, ["vm", "list-skus", "--location", region, + "--resource-type", "virtualMachines", "--size", sku_name, "--all"]) + skus = self.skus[key] + matches = [sku for sku in skus if sku.get("name") == sku_name] + if len(matches) != 1: + raise ValueError("Required VM SKU is missing/ambiguous: " + sku_name) + sku = matches[0] + add("VM SKU "+sku_name, not sku.get("restrictions"), "Subscription restrictions: "+str(sku.get("restrictions", []))) + cpu = [x.get("value") for x in sku.get("capabilities", []) if x.get("name") == "vCPUs"] + if len(cpu) != 1 or not str(cpu[0]).isdigit() or int(cpu[0]) < 1 or not sku.get("family"): + raise ValueError("VM family/vCPU metadata is unverified.") + family = sku["family"].lower() + needs[family] = needs.get(family, 0) + int(cpu[0])*count + usage = self.query(region+"-compute-usage", ["vm", "list-usage", "--location", region]) + for family, needed in dict(needs, cores=sum(needs.values())).items(): + check_quota("Compute "+family, unique_row(usage, family), needed) + except (RuntimeError, ValueError, KeyError, TypeError) as error: + checks.append({"name": "Regional evidence", "status": "UNKNOWN", "kind": "unverified", "detail": str(error), "data": {}}) + return {"region": region, "eligibleForPreflight": bool(checks) and all(x["status"] == "PASS" for x in checks), + "time": datetime.now(timezone.utc).isoformat(), "checks": checks, + "capacityReserved": False, "policyAndNetworkApprovalRequired": True} diff --git a/utilities/regional-deployment/check.py b/utilities/regional-deployment/check.py new file mode 100644 index 00000000..669fd978 --- /dev/null +++ b/utilities/regional-deployment/check.py @@ -0,0 +1,47 @@ +#!/usr/bin/env python3 +"""Local tests and Bicep compilation only: never signs in or calls ARM.""" + +import argparse +import json +from pathlib import Path +import py_compile +import subprocess +import sys +import tempfile +import unittest +from unittest.mock import patch + +from deploy import assert_resume_template +from workspace import assert_workspace_template + +ROOT = Path(__file__).resolve().parent + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--skip-bicep", action="store_true", help="Run Python-only checks when Azure CLI/Bicep is not installed.") + args = parser.parse_args() + with patch("subprocess.run", side_effect=AssertionError("Unit tests must not execute external commands or Azure CLI.")): + result = unittest.TextTestRunner(verbosity=2).run(unittest.defaultTestLoader.discover(str(ROOT/"tests"))) + if not result.wasSuccessful(): + return 1 + for source in ROOT.glob("*.py"): + py_compile.compile(str(source), doraise=True) + if not args.skip_bicep: + with tempfile.TemporaryDirectory(prefix="discovery-regional-bicep-") as directory: + compiled = {} + for name in ("main", "discovery", "workspace", "workspace-core"): + output = Path(directory)/(name+".json") + subprocess.run(["az", "bicep", "build", "--file", str(ROOT/"templates"/(name+".bicep")), + "--outfile", str(output), "--only-show-errors"], check=True) + compiled[name] = json.loads(output.read_text()) + assert_resume_template(compiled["discovery"]) + assert_workspace_template(compiled["workspace"]) + assert_workspace_template(compiled["workspace-core"], True) + print("Four local Bicep builds and bounded resource-shape checks passed.") + print("No Azure sign-in, ARM validation, resource deployment or inference was performed by these checks.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/utilities/regional-deployment/deploy.py b/utilities/regional-deployment/deploy.py new file mode 100644 index 00000000..834cc072 --- /dev/null +++ b/utilities/regional-deployment/deploy.py @@ -0,0 +1,649 @@ +#!/usr/bin/env python3 +"""Auditable Azure CLI workflow for the Microsoft Discovery Bicep quickstart.""" + +import argparse +from datetime import datetime, timezone +import hashlib +import ipaddress +import json +import os +from pathlib import Path +import re +import subprocess +import sys +import time +import uuid + + +ROOT = Path(__file__).resolve().parent +CONTROL_PLANE_APP = "92c174ac-8e41-4815-a1b7-d81b19ab03ce" +REQUIRED_PROVIDERS = ( + "Discovery Network Compute Storage ManagedIdentity AlertsManagement Authorization " + "CognitiveServices ContainerInstance ContainerRegistry ContainerService DocumentDB " + "Features KeyVault MachineLearningServices NetApp OperationalInsights ResourceGraph " + "Search Web Insights Resources Sql App Bing" +).split() +DISCOVERY_TYPES = ( + "supercomputers", "supercomputers/nodepools", "workspaces", + "workspaces/chatmodeldeployments", "storagecontainers", "workspaces/projects", +) +ROLES = { + "Owner": "8e3af657-a8ff-443c-a75c-2fe8c4bcb635", + "Reader": "acdd72a7-3385-48ef-bd42-f606fba81ae7", + "Foundry User": "53ca6127-db72-4b80-b1b0-d745d6d5456d", + "Microsoft Discovery Platform Administrator": "7a2b6e6c-472e-4b39-8878-a26eb63d75c6", + "Managed Identity Contributor": "e40ec5ca-96e0-45a2-b4ff-59039f2c2b59", + "Managed Identity Operator": "f1a07417-d97a-45cb-824c-7a7467783830", + "Storage Account Contributor": "17d1049b-9a84-46fb-8f53-869881c3d3ab", + "Storage Blob Data Contributor": "ba92f5b4-2d11-453d-a403-e96b0029c9fe", + "Network Contributor": "4d97b98b-1d4f-4787-a291-c67834d212e7", + "AcrPush": "8311e382-0749-4cb8-b61a-304f252e45ec", + "AcrPull": "7f951dda-4ed3-4680-a7ca-43fe172d538d", + "Microsoft Discovery Platform Contributor": "01288891-85ee-45a7-b367-9db3b752fc65", + "Microsoft Discovery Bookshelf Index Data Reader": "8ec773c5-7ce6-4b78-91d1-182f8faa536d", +} +PERSONA_ROLES = tuple(n for n in ROLES if n not in ( + "Owner", "Reader", "Foundry User", "AcrPull", "Microsoft Discovery Platform Contributor", +)) +NSP_ACTIONS = { + "Microsoft.Network/networkSecurityPerimeters/joinPerimeterRule/action", + "Microsoft.Network/locations/networkSecurityPerimeterOperationStatuses/read", +} + + +def is_minimal_nsp_role(definition): + permissions = definition.get("permissions", []) + return len(permissions) == 1 and set(permissions[0].get("actions", [])) == NSP_ACTIONS and not any( + permissions[0].get(key) for key in ("notActions", "dataActions", "notDataActions")) + + +def write_json(path, value): + Path(path).write_text(json.dumps(value, indent=2) + "\n") + + +def digest(path): + return hashlib.sha256(Path(path).read_bytes()).hexdigest() + + +def assert_fresh_preflight(report, config_hash, parameters_hash, now=None): + now = now or datetime.now(timezone.utc) + age = (now - datetime.fromisoformat(report["time"])).total_seconds() + if report.get("ok") is not True or not 0 <= age <= 3600 or report.get("configHash") != config_hash or report.get("parametersHash") != parameters_hash: + raise RuntimeError("Run a fresh successful preflight for this configuration first; a later failed preflight invalidates an older validation receipt.") + + +def scope_covers(parent, child): + if not parent or not child: + return False + if parent == "/": + return child.startswith("/") + parent, child = parent.lower().rstrip("/"), child.lower().rstrip("/") + return child == parent or child.startswith(parent + "/") + + +def has_role(assignments, role_id, target): + return any( + a.get("roleDefinitionId", "").split("/")[-1].lower() == role_id.lower() + and scope_covers(a.get("scope", ""), target) + and not a.get("condition") + for a in assignments + ) + + +def is_not_found_error(message): + return bool(re.search(r"\b(ResourceGroupNotFound|DeploymentNotFound|ResourceNotFound)\b", message)) + + +def resolve_managed_group(resource, groups, subscription_scope): + props = resource.get("properties", {}) + explicit = props.get("managedResourceGroupId") or props.get("managedResourceGroup") or props.get("managedResourceGroupName") + if isinstance(explicit, dict): + explicit = explicit.get("id") + candidates = [] + if explicit: + candidates.append(explicit if explicit.startswith("/") else subscription_scope + "/resourceGroups/" + explicit) + resource_id = resource.get("id", "").lower().rstrip("/") + if resource_id: + candidates += [g["id"] for g in groups + if (g.get("managedBy") or "").lower().rstrip("/") == resource_id] + unique = {candidate.lower(): candidate for candidate in candidates} + if len(unique) != 1: + raise RuntimeError("Managed resource group missing or ambiguous; inspect resource/group evidence, do not guess its name.") + group_id = next(iter(unique.values())) + pattern = re.escape(subscription_scope) + r"/resourceGroups/[^/]+" + if not re.fullmatch(pattern, group_id, flags=re.IGNORECASE): + raise RuntimeError("Managed resource group is outside the approved subscription or is not a group ID.") + return group_id + + +def filter_parameters(source, compiled): + declared = compiled.get("parameters", {}) + return { + "$schema": source.get("$schema"), + "contentVersion": "1.0.0.0", + "parameters": {k: v for k, v in source["parameters"].items() if k in declared}, + } + + +def search_dedicated_skus(offerings, location): + regions = offerings.get("regions") + if not isinstance(regions, list): + raise RuntimeError("Search offerings response has no regions list; availability is unknown.") + normalize = lambda value: re.sub(r"[^a-z0-9]", "", value.lower()) + matches = [r for r in regions if normalize(r.get("regionName", "")) == normalize(location)] + if len(matches) != 1 or not isinstance(matches[0].get("skus"), list): + raise RuntimeError("Search offerings region missing or ambiguous; availability is unknown.") + names = set() + for item in matches[0]["skus"]: + name = item.get("name") or item.get("sku", {}).get("name") + if not isinstance(name, str): + raise RuntimeError("Search offering has no SKU name; availability is unknown.") + if name.lower() not in ("free", "serverless"): + names.add(name.lower()) + return names + + +def project_agents_url(workspace, workspace_name, project_name, api_version): + uri = workspace.get("properties", {}).get("workspaceApiUri", "").rstrip("/") + if uri.lower() != "https://" + workspace_name.lower() + ".workspace.discovery.azure.com": + raise RuntimeError("Unexpected workspace API host; no authenticated request sent.") + if not re.fullmatch(r"[a-zA-Z0-9-]{3,24}", project_name): + raise RuntimeError("Unexpected project name; no authenticated request sent.") + return uri + "/projects/" + project_name + "/agents?api-version=" + api_version + + +def assert_resume_template(compiled): + expected = {"microsoft.discovery/" + name for name in DISCOVERY_TYPES} + actual = [r.get("type", "").lower() for r in compiled.get("resources", [])] + if len(actual) != len(expected) or set(actual) != expected: + raise RuntimeError("Resume template must contain exactly the six approved Discovery types and no foundation writes.") + + +def inspect_changes(result, discovery_only, allow_modify=False): + if result.get("status") != "Succeeded" or result.get("error"): + raise RuntimeError("What-if did not succeed; inspect the saved response.") + counts = {} + for change in result.get("changes", []): + kind = change["changeType"] + if kind not in ("Create", "Modify", "Delete", "Ignore", "NoChange", "Deploy", "Unsupported"): + raise RuntimeError("Unknown what-if change type; no deployment authorized: " + kind) + counts[kind] = counts.get(kind, 0) + 1 + if kind in ("Delete", "Deploy", "Unsupported") or (kind == "Modify" and not allow_modify): + raise RuntimeError(f"Unapproved what-if change: {kind} {change['resourceId']}") + if discovery_only and kind in ("Create", "Modify"): + if "/providers/microsoft.discovery/" not in change["resourceId"].lower(): + raise RuntimeError("Resume deployment would modify a non-Discovery resource.") + return counts + + +class Workflow: + def __init__(self, args): + self.args = args + self.config_path = Path(args.config).resolve() + self.config = json.loads(self.config_path.read_text()) + self.sub = self.config["subscriptionId"] + uuid.UUID(self.sub) + uuid.UUID(self.config["tenantId"]) + if self.config.get("apiVersion") != "2026-06-01": + raise ValueError("This utility's templates require Discovery API 2026-06-01.") + self.rg = self.config["resourceGroup"] + self.location = self.config["location"] + self.data_location = self.config.get("dataPlaneLocation", self.location) + self.scope = f"/subscriptions/{self.sub}" + self.rg_id = self.scope + "/resourceGroups/" + self.rg + self.parameters_path = self.config_path.parent / self.config["parametersFile"] + self.state_root = self.config_path.parent / ".state" + self.state_root.mkdir(exist_ok=True, mode=0o700) + self.parameters = json.loads(self.parameters_path.read_text()) + self.values = {k: v["value"] for k, v in self.parameters["parameters"].items()} + if not re.fullmatch(r"[A-Za-z0-9_.()-]{1,90}", self.rg) or self.rg.endswith("."): + raise ValueError("Invalid resource group name.") + for key in ("supercomputerName", "workspaceName", "storageContainerName", "projectName", "chatModelDeploymentName"): + if not re.fullmatch(r"[A-Za-z0-9-]{3,24}", self.values[key]): + raise ValueError("Invalid Discovery resource name: " + key) + source_name = self.config.get("sourceWorkspaceName") + if source_name and not re.fullmatch(r"[A-Za-z0-9-]{3,24}", source_name): + raise ValueError("Invalid source workspace name.") + if self.values.get("chatModelName") != "gpt-5.4" or self.values.get("chatModelFormat", "OpenAI") != "OpenAI": + raise ValueError("This baseline budgets the required OpenAI GPT-5.4 service deployments only.") + if self.values["location"] != self.location: + raise RuntimeError("Configuration and Bicep parameter regions differ.") + if self.values.get("overridemrgregion", self.location) != self.data_location: + raise RuntimeError("Configuration and Bicep data-plane override regions differ.") + stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%S%fZ") + self.out = Path(args.evidence_dir).resolve() if args.evidence_dir else ( + self.config_path.parent / "evidence" / f"{args.command}-{stamp}" + ) + self.out.mkdir(parents=True, exist_ok=True, mode=0o700) + self.ledger = [] + self.checks = [] + write_json(self.out / "source-manifest.json", { + "scriptHash": digest(__file__), "configHash": digest(self.config_path), + "parametersHash": digest(self.parameters_path), + "templates": {str(p.relative_to(ROOT)): digest(p) for p in (ROOT / "templates").rglob("*.bicep")}, + }) + + def az(self, label, arguments, *, scoped=True, optional=False, output_json=True, not_found_only=False): + command = ["az", *arguments] + if scoped: + command += ["--subscription", self.sub] + command += ["--only-show-errors"] + if output_json: + command += ["-o", "json"] + print(f"[{datetime.now().strftime('%H:%M:%S')}] {label}", flush=True) + result = subprocess.run(command, cwd=ROOT, text=True, capture_output=True, + env=dict(os.environ, AZURE_EXTENSION_USE_DYNAMIC_INSTALL="no")) + suffix = ".json" if output_json else ".stdout" + (self.out / (label + suffix)).write_text(result.stdout) + (self.out / (label + ".stderr")).write_text(result.stderr) + self.ledger.append({"command": command, "exitCode": result.returncode, + "time": datetime.now(timezone.utc).isoformat()}) + write_json(self.out / "commands.json", self.ledger) + if result.returncode: + if optional: + if not_found_only and not is_not_found_error(result.stderr): + raise RuntimeError(f"{label}: lookup failed, not a confirmed missing resource; no mutation permitted. {result.stderr[-3000:]}") + return None + raise RuntimeError(f"{label}: {result.stderr[-5000:]}") + return json.loads(result.stdout) if output_json and result.stdout.strip() else result.stdout + + def rest(self, label, resource, optional=False): + return self.az(label, ["rest", "--method", "get", "--url", + "https://management.azure.com" + resource], + scoped=False, optional=optional) + + def check(self, name, condition, detail, required=True): + state = "PASS" if condition else ("FAIL" if required else "WARN") + self.checks.append({"name": name, "state": state, "detail": detail}) + print(f" {state}: {name}: {detail}", flush=True) + + def context(self): + account = self.az("account", ["account", "show"]) + if account.get("environmentName") != "AzureCloud": + raise RuntimeError("This utility supports Azure public cloud only; no preparation/deployment permitted.") + if account["id"] != self.sub or account["tenantId"] != self.config["tenantId"]: + raise RuntimeError("Azure subscription/tenant differs from the approved configuration.") + default = self.az("default-account", ["account", "show"], scoped=False) + if default["tenantId"] != self.config["tenantId"]: + raise RuntimeError("Select the approved tenant before directory/REST operations.") + return account + + def principals(self): + user = self.az("signed-in-user", ["ad", "signed-in-user", "show"], scoped=False) + cp = self.az("control-plane-principal", ["ad", "sp", "show", "--id", CONTROL_PLANE_APP], scoped=False) + return user["id"], cp["id"] + + def assignments(self, label, principal, scope=None): + args = ["role", "assignment", "list", "--assignee", principal] + args += ["--scope", scope, "--include-inherited"] if scope else ["--all"] + return self.az(label, args) + + def prepare(self): + self.context() + if not self.args.apply: + raise RuntimeError("Provider/IAM preparation requires --apply.") + providers = self.az("providers", ["provider", "list"]) + states = {x["namespace"].lower(): x["registrationState"] for x in providers} + for name in REQUIRED_PROVIDERS: + namespace = "Microsoft." + name + if states.get(namespace.lower()) != "Registered": + self.az("register-" + name.lower(), ["provider", "register", "--namespace", namespace, "--wait"]) + group = self.az("resource-group", ["group", "show", "--name", self.rg], optional=True, not_found_only=True) + if group is None: + self.az("create-resource-group", ["group", "create", "--name", self.rg, "--location", self.data_location]) + user, cp = self.principals() + cp_roles = self.assignments("control-plane-roles", cp, self.scope) + definitions = self.az("custom-role-definitions", ["role", "definition", "list", "--custom-role-only", "true", "--scope", self.scope]) + nsp = next((d for d in definitions if is_minimal_nsp_role(d)), None) + if nsp is None: + body = {"Name": "Discovery NSP Perimeter Joiner - " + self.sub[:8], + "Description": "Discovery control-plane NSP join permissions.", + "Actions": sorted(NSP_ACTIONS), "NotActions": [], "DataActions": [], + "NotDataActions": [], "AssignableScopes": [self.scope]} + file = self.out / "nsp-role-definition.json" + write_json(file, body) + nsp = self.az("create-nsp-role", ["role", "definition", "create", "--role-definition", str(file)]) + self.assign(cp, "ServicePrincipal", nsp["name"], self.scope, cp_roles) + self.assign(cp, "ServicePrincipal", ROLES["Reader"], self.scope, cp_roles) + user_roles = self.assignments("user-roles", user, self.rg_id) + for name in PERSONA_ROLES: + self.assign(user, "User", ROLES[name], self.rg_id, user_roles) + print("Providers and persona/NSP prerequisites prepared. Run preflight next.") + + def assign(self, principal, principal_type, role, scope, current): + if has_role(current, role, scope): + return + name = str(uuid.uuid5(uuid.NAMESPACE_URL, "|".join((scope.lower(), principal.lower(), role.lower())))) + self.az("assign-" + name, ["role", "assignment", "create", "--name", name, + "--assignee-object-id", principal, "--assignee-principal-type", principal_type, + "--role", role, "--scope", scope]) + + def quota(self, provider): + scope = f"{self.scope}/providers/{provider}/locations/{self.data_location}" + limits = self.az(provider + "-quota", ["quota", "list", "--scope", scope]) + usages = self.az(provider + "-usage", ["quota", "usage", "list", "--scope", scope]) + used = {x["name"].lower(): x.get("properties", {}).get("usages", {}).get("value") for x in usages} + return {x["name"].lower(): (x.get("properties", {}).get("limit", {}).get("value"), + used.get(x["name"].lower())) for x in limits} + + def preflight(self): + self.begin_preflight() + self.context() + self.check("Documented production region", self.location in ("eastus", "uksouth", "swedencentral"), + "creation support is stricter than the provider catalog; data-plane override=" + self.data_location) + provider = self.az("discovery-provider", ["provider", "show", "--namespace", "Microsoft.Discovery"]) + types = {x["resourceType"].lower(): x for x in provider["resourceTypes"]} + for name in DISCOVERY_TYPES: + info = types.get(name, {}) + locations = ["".join(c for c in s.lower() if c.isalnum()) for s in info.get("locations", [])] + self.check("Discovery " + name, self.location in locations and self.config["apiVersion"] in info.get("apiVersions", []), + "region/API advertised by the subscription-scoped catalog") + read = self.rest("supercomputer-read", f"{self.scope}/providers/Microsoft.Discovery/supercomputers?api-version={self.config['apiVersion']}", optional=True) + self.check("Authenticated Discovery read", isinstance(read, dict) and "value" in read, "real ARM GET, not only template validation") + feature = self.az("default-feature", ["feature", "show", "--namespace", "Microsoft.Discovery", "--name", "DefaultFeature"], optional=True) + self.check("DefaultFeature informational", bool(feature and feature.get("properties", {}).get("state") == "Registered"), + "not an access gate when core resource types and authenticated reads are available", required=False) + providers = self.az("providers", ["provider", "list"]) + states = {x["namespace"].lower(): x["registrationState"] for x in providers} + missing = ["Microsoft." + n for n in REQUIRED_PROVIDERS if states.get("microsoft." + n.lower()) != "Registered"] + self.check("Required providers", not missing, "missing: " + ", ".join(missing) if missing else "all registered") + user, cp = self.principals() + user_roles = self.assignments("user-roles", user, self.rg_id) + for name in PERSONA_ROLES: + self.check("User " + name, has_role(user_roles, ROLES[name], self.rg_id), "resource group or inherited scope") + cp_roles = self.assignments("control-plane-roles", cp, self.scope) + self.check("Control-plane Reader", has_role(cp_roles, ROLES["Reader"], self.scope), "subscription scope") + definitions = self.az("custom-role-definitions", ["role", "definition", "list", "--custom-role-only", "true", "--scope", self.scope]) + self.check("Control-plane NSP", any(has_role(cp_roles, d["name"], self.scope) and is_minimal_nsp_role(d) + for d in definitions), "minimal NSP role assigned at subscription scope without conditions") + inventory = self.az("foundation-inventory", ["resource", "list", "--resource-group", self.rg]) + existing = {(x["type"].lower(), x["name"].lower()) for x in inventory} + wanted = {("microsoft.managedidentity/userassignedidentities", self.values["managedIdentityName"].lower()), + ("microsoft.network/virtualnetworks", self.values["vnetName"].lower()), + ("microsoft.storage/storageaccounts", self.values["storageAccountName"].lower())} + present = len(wanted & existing) + if self.args.stage == "full" and present == 0: + self.check("New foundation", True, "all three foundation resources will be created by the full template; verify after deployment") + elif present != 3: + self.check("Foundation completeness", False, "partial/missing foundation; do not resume Discovery against missing dependencies") + else: + self.check_foundation() + self.check_capacity(include_compute=True) + report = {"ok": not any(c["state"] == "FAIL" for c in self.checks), "checks": self.checks, + "time": datetime.now(timezone.utc).isoformat(), "configHash": digest(self.config_path), + "parametersHash": digest(self.parameters_path), "codeHash": self.code_hash(), "evidence": str(self.out)} + write_json(self.out / "preflight.json", report) + write_json(self.state_root / "latest-preflight.json", report) + if not report["ok"]: + raise RuntimeError("Preflight failed; see preflight.json. No deployment submitted.") + + def begin_preflight(self): + write_json(self.state_root / "latest-preflight.json", { + "ok": False, "time": datetime.now(timezone.utc).isoformat(), + "configHash": digest(self.config_path), "parametersHash": digest(self.parameters_path), + "codeHash": self.code_hash(), "reason": "Preflight started; prior evidence invalidated.", + }) + + def code_hash(self): + paths = sorted(list(ROOT.glob("*.py")) + list((ROOT / "templates").rglob("*.bicep"))) + return hashlib.sha256("\n".join(str(p.relative_to(ROOT))+":"+digest(p) for p in paths).encode()).hexdigest() + + def check_capacity(self, include_compute): + from capacity import RegionProbe, remaining_model_tpm, read_query + base = f"https://management.azure.com{self.rg_id}/providers/Microsoft.Discovery/" + def state(path, label): + obj = read_query(self, label, ["rest", "--method", "get", "--url", base+path+"?api-version="+self.config["apiVersion"]], + scoped=False, optional=True, not_found_only=True) + return obj.get("properties", {}).get("provisioningState") if obj else None + workspace = "workspaces/"+self.values["workspaceName"] + ws_state = state(workspace, "capacity-workspace") + chat_state = state(workspace+"/chatModelDeployments/"+self.values["chatModelDeploymentName"], "capacity-chat") if ws_state is not None else None + needed = remaining_model_tpm(ws_state, chat_state) + if include_compute: + sc = "supercomputers/"+self.values["supercomputerName"] + include_compute = state(sc, "capacity-compute") != "Succeeded" or state(sc+"/nodePools/"+self.values["nodePoolName"], "capacity-nodepool") != "Succeeded" + result = RegionProbe(self).probe(self.data_location, include_compute, needed, ws_state != "Succeeded") + for check in result["checks"]: + self.check(check["name"], check["status"] == "PASS", check["detail"]) + self.checks[-1].update(kind=check["kind"], data=check["data"], status=check["status"]) + write_json(self.out / "regional-capacity.json", result) + + def check_foundation(self): + identity = self.az("identity", ["identity", "show", "--name", self.values["managedIdentityName"], "--resource-group", self.rg]) + roles = self.assignments("identity-roles", identity["principalId"]) + vnet = self.az("vnet", ["network", "vnet", "show", "--name", self.values["vnetName"], "--resource-group", self.rg]) + storage = self.az("storage", ["storage", "account", "show", "--name", self.values["storageAccountName"], "--resource-group", self.rg]) + for obj in [identity, vnet, storage]: + self.check(obj["name"] + " region", obj["location"] == self.data_location, obj["location"]) + for name, scope in [("Microsoft Discovery Platform Contributor", self.rg_id), ("AcrPull", self.rg_id), + ("Storage Blob Data Contributor", storage["id"]), ("Network Contributor", vnet["id"])]: + self.check("UAMI " + name, has_role(roles, ROLES[name], scope), scope) + subnets = {s["name"]: s for s in vnet["subnets"]} + expected = ["aksSubnet", "sc-mgmt", "supercomputerNodepoolSubnet", "workspaceSubnet", "agentSubnet", "searchSubnet", "privateEndpointSubnet"] + for name in expected: + self.check("Subnet " + name, name in subnets, "existing subnet; never overwrite governance NSGs") + for name in ["workspaceSubnet", "agentSubnet", "searchSubnet", "sc-mgmt"]: + target = "Microsoft.ContainerService/managedClusters" if name == "sc-mgmt" else "Microsoft.App/environments" + self.check("Delegation " + name, any(d.get("serviceName") == target for d in subnets.get(name, {}).get("delegations", [])), target) + self.check("Private Blob governance", storage.get("publicNetworkAccess") == "Disabled", "public access remains disabled", self.config.get("privateStorageRequired", True)) + pe_name = "pe-" + self.values["storageAccountName"] + "-blob" + pe = self.az("blob-private-endpoint", ["network", "private-endpoint", "show", "--name", pe_name, "--resource-group", self.rg], optional=True) + self.check("Blob Private Endpoint", bool(pe and pe.get("provisioningState") == "Succeeded" and any( + c.get("privateLinkServiceConnectionState", {}).get("status") == "Approved" for c in pe.get("privateLinkServiceConnections", []) + )), "Succeeded and Approved; data-plane reachability is a separate check") + for name in ["aksSubnet", "supercomputerNodepoolSubnet", "workspaceSubnet", "agentSubnet"]: + endpoints = subnets.get(name, {}).get("serviceEndpoints", []) + self.check("Storage service endpoint " + name, any( + e.get("service") in ("Microsoft.Storage", "Microsoft.Storage.Global") for e in endpoints + ), "required by the quickstart") + blob = self.rest("blob-service", storage["id"] + "/blobServices/default?api-version=2023-05-01") + needed = {"https://studio.discovery.microsoft.com", "https://vscode.dev", "https://*.vscode-cdn.net"} + cors = blob.get("properties", {}).get("cors", {}).get("corsRules", []) + origins = {o for rule in cors if {"GET", "HEAD", "PUT", "DELETE"}.issubset(set(rule.get("allowedMethods", []))) + for o in rule.get("allowedOrigins", [])} + self.check("Blob CORS", needed.issubset(origins), "Studio and VS Code origins/methods") + container = self.values.get("blobContainerName", "discoveryoutputs") + result = self.rest("blob-container", storage["id"] + f"/blobServices/default/containers/{container}?api-version=2023-05-01", optional=True) + self.check("Output blob container", bool(result), "ARM metadata read, not a data-plane test") + zone = "privatelink.blob.core.windows.net" + dns = self.az("blob-dns", ["network", "private-dns", "record-set", "a", "show", "--name", + self.values["storageAccountName"], "--zone-name", zone, "--resource-group", self.rg], optional=True) + link = self.az("blob-dns-link", ["network", "private-dns", "link", "vnet", "show", "--name", + "link-" + self.values["vnetName"], "--zone-name", zone, "--resource-group", self.rg], optional=True) + prefix = subnets.get("privateEndpointSubnet", {}).get("addressPrefix") + records = dns.get("aRecords", []) if dns else [] + valid_ips = bool(prefix and records) and all( + ipaddress.ip_address(a["ipv4Address"]) in ipaddress.ip_network(prefix) for a in records + ) + self.check("Blob private DNS", valid_ips and bool(link and link.get("virtualNetworkLinkState") == "Completed"), + "private address in the endpoint subnet and completed VNet link") + + def build(self): + self.template = ROOT / "templates" / ("discovery.bicep" if self.args.stage == "discovery" else "main.bicep") + self.compiled_path = self.out / "template.json" + self.az("bicep-build", ["bicep", "build", "--file", str(self.template), "--outfile", str(self.compiled_path)], scoped=False, output_json=False) + compiled = json.loads(self.compiled_path.read_text()) + if self.args.stage == "discovery": + assert_resume_template(compiled) + self.filtered_path = self.out / "parameters.json" + write_json(self.filtered_path, filter_parameters(self.parameters, compiled)) + + def deployment_args(self): + return ["--resource-group", self.rg, "--name", self.config["deploymentName"], + "--template-file", str(self.compiled_path), "--parameters", "@" + str(self.filtered_path), + "--mode", "Incremental", "--validation-level", "Provider"] + + def validate(self): + write_json(self.state_root / "validated-deployment.json", {"valid": False, "reason": "Validation started; previous receipt invalidated."}) + report = json.loads((self.state_root / "latest-preflight.json").read_text()) + assert_fresh_preflight(report, digest(self.config_path), digest(self.parameters_path)) + if report.get("codeHash") != self.code_hash(): + raise RuntimeError("Utility code changed after preflight; rerun it.") + self.build() + self.az("arm-validation", ["deployment", "group", "validate", *self.deployment_args(), "--no-prompt", "true"]) + result = self.az("what-if", ["deployment", "group", "what-if", *self.deployment_args(), "--no-pretty-print"]) + counts = self.inspect_preview(result) + receipt = {"valid": True, "time": datetime.now(timezone.utc).isoformat(), "configHash": digest(self.config_path), + "template": str(self.template), "templateHash": digest(self.template), + "parametersHash": digest(self.parameters_path), "compiled": str(self.compiled_path), + "compiledHash": digest(self.compiled_path), "filteredParameters": str(self.filtered_path), + "filteredParametersHash": digest(self.filtered_path), "stage": self.args.stage, + "counts": counts, "codeHash": self.code_hash(), "evidence": str(self.out)} + write_json(self.state_root / "validated-deployment.json", receipt) + print("Validated what-if:", counts) + + def inspect_preview(self, result): + return inspect_changes(result, self.args.stage == "discovery", self.args.allow_modify) + + def deploy(self): + self.context() + if not self.args.apply: + raise RuntimeError("Actual deployment requires --apply after reviewing what-if.") + report = json.loads((self.state_root / "latest-preflight.json").read_text()) + assert_fresh_preflight(report, digest(self.config_path), digest(self.parameters_path)) + receipt = json.loads((self.state_root / "validated-deployment.json").read_text()) + if receipt.get("valid") is not True: + raise RuntimeError("Run a fresh successful validation; previous receipt is invalid.") + if receipt.get("codeHash") != self.code_hash() or report.get("codeHash") != self.code_hash(): + raise RuntimeError("Utility code changed; run fresh preflight and validation.") + for field, path in [("configHash", self.config_path), ("parametersHash", self.parameters_path), + ("templateHash", receipt["template"]), ("compiledHash", receipt["compiled"]), + ("filteredParametersHash", receipt["filteredParameters"])]: + if receipt[field] != digest(path): + raise RuntimeError("Validated input changed: " + field) + age = (datetime.now(timezone.utc) - datetime.fromisoformat(receipt["time"])).total_seconds() + if not 0 <= age <= 3600 or receipt["stage"] != self.args.stage: + raise RuntimeError("Validation receipt is stale or belongs to another stage.") + existing = self.az("existing-deployment", ["deployment", "group", "show", "--resource-group", self.rg, + "--name", self.config["deploymentName"]], optional=True, not_found_only=True) + if existing and existing.get("properties", {}).get("provisioningState") in ("Running", "Accepted"): + raise RuntimeError("This deployment is already running. Use status/wait; do not resubmit.") + self.compiled_path = Path(receipt["compiled"]) + self.filtered_path = Path(receipt["filteredParameters"]) + self.az("deployment-submit", ["deployment", "group", "create", *self.deployment_args(), "--no-prompt", "true", "--no-wait"]) + self.status() + + def status(self): + d = self.az("deployment-status", ["deployment", "group", "show", "--resource-group", self.rg, "--name", self.config["deploymentName"]]) + operations = self.az("deployment-operations", ["deployment", "operation", "group", "list", "--resource-group", self.rg, "--name", self.config["deploymentName"]]) + stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%S%fZ") + write_json(self.out / ("status-" + stamp + ".json"), {"deployment": d, "operations": operations}) + print("Deployment:", d["properties"]["provisioningState"], "correlation:", d["properties"].get("correlationId")) + for op in operations: + p = op.get("properties") or {} + target = p.get("targetResource") or {} + print(" ", p.get("provisioningState"), target.get("resourceType"), target.get("resourceName")) + if p.get("provisioningState") == "Failed": + print(json.dumps(p.get("statusMessage"), indent=2)) + return d["properties"]["provisioningState"] + + def wait(self): + deadline = time.monotonic() + self.args.timeout + while True: + state = self.status() + if state == "Succeeded": + return + if state in ("Failed", "Canceled"): + raise RuntimeError("Deployment reached terminal state " + state + "; no automatic replay.") + if time.monotonic() >= deadline: + raise RuntimeError("Wait timed out; cloud deployment was not canceled. Use status/wait to continue.") + time.sleep(self.args.interval) + + def resource_paths(self): + p = self.values + sc = "supercomputers/" + p["supercomputerName"] + ws = "workspaces/" + p["workspaceName"] + return [sc, sc + "/nodePools/" + p["nodePoolName"], ws, + ws + "/chatModelDeployments/" + p["chatModelDeploymentName"], + "storageContainers/" + p["storageContainerName"], ws + "/projects/" + p["projectName"]] + + def verify(self): + self.context() + results = [] + for i, path in enumerate(self.resource_paths()): + obj = self.rest("resource-" + str(i), f"{self.rg_id}/providers/Microsoft.Discovery/{path}?api-version={self.config['apiVersion']}", optional=True) + state = obj.get("properties", {}).get("provisioningState") if obj else None + results.append({"path": path, "state": state, "id": obj.get("id") if obj else None}) + self.check(path, state == "Succeeded", str(state)) + write_json(self.out / "verification.json", {"resources": results, "paidInferenceTested": False}) + if any(r["state"] != "Succeeded" for r in results): + raise RuntimeError("Not all six Discovery resources are Succeeded.") + print("Resource verification passed. No paid inference was run. Check managed-RG roles and Studio access next.") + + def postconfigure(self): + self.context() + if not self.args.apply: + raise RuntimeError("Foundry User assignment requires --apply.") + user, _ = self.principals() + path = f"{self.rg_id}/providers/Microsoft.Discovery/workspaces/{self.values['workspaceName']}?api-version={self.config['apiVersion']}" + workspace = self.rest("workspace", path) + if workspace.get("properties", {}).get("provisioningState") != "Succeeded": + raise RuntimeError("Workspace is not Succeeded; wait for provisioning before postconfiguration.") + groups = self.az("resource-groups", ["group", "list"]) + mrg = resolve_managed_group(workspace, groups, self.scope) + current = self.assignments("managed-rg-user-roles", user, mrg) + self.assign(user, "User", ROLES["Foundry User"], mrg, current) + confirmed = self.assignments("managed-rg-user-roles-readback", user, mrg) + if not has_role(confirmed, ROLES["Foundry User"], mrg): + raise RuntimeError("Foundry User not visible on readback; inspect propagation before retrying any assignment.") + write_json(self.out / "postconfiguration.json", {"workspaceId": workspace["id"], + "managedResourceGroupId": mrg, "foundryUserVerified": True, + "inheritedRoleReused": has_role(current, ROLES["Foundry User"], mrg)}) + print("Foundry User verified on the workspace managed resource group:", mrg) + + def audit(self): + self.verify() + self.check_foundation() + p = self.values + workspace = json.loads((self.out / "resource-2.json").read_text()) + model = json.loads((self.out / "resource-3.json").read_text()) + project = json.loads((self.out / "resource-5.json").read_text()) + compute = json.loads((self.out / "resource-0.json").read_text()) + store = json.loads((self.out / "resource-4.json").read_text()) + sc_id = self.rg_id + "/providers/Microsoft.Discovery/supercomputers/" + p["supercomputerName"] + store_id = self.rg_id + "/providers/Microsoft.Discovery/storageContainers/" + p["storageContainerName"] + self.check("Workspace compute relationship", {x.lower() for x in workspace["properties"].get("supercomputerIds", [])} == {sc_id.lower()}, "expected Supercomputer ID") + self.check("Project storage relationship", store_id.lower() in {x.lower() for x in project["properties"].get("storageContainerIds", [])}, "expected Discovery StorageContainer ID") + account_id = self.rg_id + "/providers/Microsoft.Storage/storageAccounts/" + p["storageAccountName"] + self.check("Project backing storage", store["properties"].get("storageStore", {}).get("storageAccountId", "").lower() == account_id.lower(), "expected customer storage account") + self.check("Supercomputer managed region", compute.get("tags", {}).get("discovery.overridemrgregion") == self.data_location, self.data_location) + self.check("Workspace managed region", workspace.get("tags", {}).get("discovery.overridemrgregion") == self.data_location, self.data_location) + self.check("Chat model configuration", model["properties"].get("modelName") == p["chatModelName"] and + model["properties"].get("modelFormat") == p["chatModelFormat"], p["chatModelName"]) + groups = self.az("resource-groups", ["group", "list"]) + mrg = resolve_managed_group(workspace, groups, self.scope) + inventory = self.az("managed-inventory", ["resource", "list", "--resource-group", mrg.split("/")[-1]]) + required = {"microsoft.search/searchservices", "microsoft.cognitiveservices/accounts", "microsoft.documentdb/databaseaccounts"} + found, backing = set(), [] + for index, item in enumerate(inventory): + kind = item["type"].lower() + if kind not in required: + continue + found.add(kind) + meta = self.az("backing-" + str(index), ["resource", "show", "--ids", item["id"], "--query", + "{id:id,name:name,type:type,location:location,provisioningState:properties.provisioningState,status:properties.status}"]) + backing.append(meta) + ready = (meta.get("provisioningState") or "").lower() == "succeeded" + if kind == "microsoft.search/searchservices": + ready = ready and (meta.get("status") or "").lower() == "running" + self.check("Backing " + item["name"], ready and meta["location"].replace(" ", "").lower() == self.data_location, + str(meta.get("provisioningState")) + "; " + meta["location"]) + self.check("Required backing services", required.issubset(found), "Search, Foundry and Cosmos DB") + endpoints = self.az("managed-private-endpoints", ["network", "private-endpoint", "list", "--resource-group", mrg.split("/")[-1]]) + self.check("Managed private endpoints", bool(endpoints) and all(x.get("provisioningState") == "Succeeded" and any( + c.get("privateLinkServiceConnectionState", {}).get("status") == "Approved" for c in x.get("privateLinkServiceConnections", [])) for x in endpoints), + str(len(endpoints)) + " endpoints; configuration evidence, not packet reachability") + user, _ = self.principals() + roles = self.assignments("managed-rg-user-roles", user, mrg) + self.check("Foundry User on managed group", has_role(roles, ROLES["Foundry User"], mrg), "scoped or inherited") + url = project_agents_url(workspace, p["workspaceName"], p["projectName"], self.config["apiVersion"]) + agents = self.az("project-agents-read", ["rest", "--method", "get", "--resource", "https://discovery.azure.com", "--url", url], + scoped=False, optional=True) + reachable = isinstance(agents, (dict, list)) + self.check("Read-only authenticated project API", reachable, "GET agents only; use a privately connected runner when required", required=False) + report = {"ok": not any(c["state"] == "FAIL" for c in self.checks), "checks": self.checks, + "managedResourceGroupId": mrg, "backing": backing, "workspaceApiUri": workspace["properties"]["workspaceApiUri"], + "authenticatedProjectReadSucceeded": reachable, "paidInferenceTested": False} + write_json(self.out / "audit.json", report) + if not report["ok"]: + raise RuntimeError("Post-deployment audit failed; inspect audit.json before use.") diff --git a/utilities/regional-deployment/inputs.py b/utilities/regional-deployment/inputs.py new file mode 100644 index 00000000..e1855c7c --- /dev/null +++ b/utilities/regional-deployment/inputs.py @@ -0,0 +1,147 @@ +#!/usr/bin/env python3 +"""Generate new Discovery inputs locally, without login, Azure calls or overwrites.""" + +import argparse +import ipaddress +import json +from pathlib import Path +import re +import sys +import uuid + +from deploy import ROOT, write_json + + +CONTROL_REGIONS = ("eastus", "uksouth", "swedencentral") + + +def region(value): + if not re.fullmatch(r"[a-z][a-z0-9]+", value): + raise ValueError("Use a lowercase Azure region short name, for example centralus.") + return value + + +def prefixes(cidr): + net = ipaddress.ip_network(cidr, strict=True) + private_ranges = tuple(ipaddress.ip_network(x) for x in ("10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16")) + if net.version != 4 or net.prefixlen != 16 or not any(net.subnet_of(x) for x in private_ranges): + raise ValueError("This baseline requires a private IPv4 /16 VNet CIDR.") + blocks = list(net.subnets(new_prefix=24)) + return { + "vnetAddressPrefix": str(net), + "supercomputerNodepoolSubnetPrefix": str(blocks[1]), + "aksSubnetPrefix": str(blocks[2]), + "workspaceSubnetPrefix": str(blocks[3]), + "privateEndpointSubnetPrefix": str(blocks[4]), + "agentSubnetPrefix": str(blocks[5]), + "searchSubnetPrefix": str(blocks[6]), + "managementSubnetPrefix": str(next(blocks[7].subnets(new_prefix=28))), + } + + +def validate_prefix(value): + if not re.fullmatch(r"[a-z][a-z0-9]{2,11}", value): + raise ValueError("name-prefix must be 3-12 lowercase alphanumeric characters, starting with a letter.") + return value + + +def portable(path): + path = Path(path).resolve() + return str(path.relative_to(ROOT)) if path.is_relative_to(ROOT) else str(path) + + +def base_config(sub, tenant, group, control, workload, name): + uuid.UUID(sub) + uuid.UUID(tenant) + if not re.fullmatch(r"[A-Za-z0-9_.()-]{1,90}", group) or group.endswith("."): + raise ValueError("Use a 1-90 character Azure resource group name, without a trailing period.") + if control not in CONTROL_REGIONS: + raise ValueError("Select a documented Discovery control-plane region.") + return { + "subscriptionId": sub, "tenantId": tenant, "resourceGroup": group, + "location": control, "dataPlaneLocation": region(workload), + "deploymentMode": "full", + "apiVersion": "2026-06-01", "deploymentName": name, + "privateStorageRequired": True, + } + + +def full_inputs(sub, tenant, group, name, control, workload, cidr): + validate_prefix(name) + cfg = base_config(sub, tenant, group, control, workload, "discovery-" + name) + suffix = uuid.uuid5(uuid.NAMESPACE_URL, sub.lower() + "/" + group.lower() + "/" + name).hex[:8] + values = { + "location": control, "overridemrgregion": workload, + "supercomputerName": name + "-sc", "workspaceName": name + "-ws", + "nodePoolName": "nodepool1", "projectName": name[:8] + "-prj", + "storageContainerName": name + "-outputs", "blobContainerName": "discoveryoutputs", + "vnetName": "vnet-" + name, "managedIdentityName": "id-" + name, + "natGatewayName": "nat-" + name, "natGatewayPublicIpName": "pip-" + name, + "storageAccountName": "st" + name + suffix, "storageAccountSku": "Standard_GRS", + "nodePoolVmSize": "Standard_D4s_v6", "nodePoolMinNodeCount": 0, + "systemSku": "Standard_D4s_v6", + "nodePoolMaxNodeCount": 3, "nodePoolScaleSetPriority": "Regular", + "chatModelDeploymentName": "gpt-5-4", "chatModelFormat": "OpenAI", + "chatModelName": "gpt-5.4", "enableGhcpAiFeatures": True, + "enableExtensions": True, "networkIsolation": True, + **prefixes(cidr), + } + return cfg, values + + +def workspace_inputs(source, values, name, workload, cidr, private_ip): + validate_prefix(name) + region(workload) + if source["location"] not in CONTROL_REGIONS: + raise ValueError("Use a documented Discovery control-plane region.") + if source.get("deploymentMode") == "workspace": + raise ValueError("Use a full-environment source config, not an earlier workspace-recovery config.") + if values["location"] != source["location"] or values["overridemrgregion"] != source["dataPlaneLocation"]: + raise ValueError("Source config and parameter regions differ.") + network = prefixes(cidr) + old_network = ipaddress.ip_network(values.get("vnetAddressPrefix", "10.0.0.0/16")) + new_network = ipaddress.ip_network(cidr) + if old_network.overlaps(new_network): + raise ValueError("New workspace VNet must not overlap the retained VNet.") + address = ipaddress.ip_address(private_ip) + endpoint_net = ipaddress.ip_network(values.get("privateEndpointSubnetPrefix", "10.0.4.0/24")) + if address not in endpoint_net: + raise ValueError("Retained Blob private IP must be in the source private-endpoint subnet; read its actual DNS record first.") + if workload == source["dataPlaneLocation"]: + raise ValueError("Choose a different workspace workload region for a regional fallback.") + cfg = base_config(source["subscriptionId"], source["tenantId"], source["resourceGroup"], + source["location"], workload, "discovery-" + name + "-workspace") + cfg.update({"supercomputerDataPlaneLocation": source["dataPlaneLocation"], + "storageDataPlaneLocation": source["dataPlaneLocation"], + "deploymentMode": "workspace", + "sourceWorkspaceName": values["workspaceName"], "retainedBlobPrivateIp": str(address)}) + output = {key: values[key] for key in ("supercomputerName", "nodePoolName", "storageContainerName", + "storageAccountName", "chatModelDeploymentName", "chatModelName", "projectName")} + output.update({"location": source["location"], "overridemrgregion": workload, + "existingVnetName": values["vnetName"], "workspaceName": name + "-ws", + "vnetName": "vnet-" + name, "managedIdentityName": "id-" + name, + "natGatewayName": "nat-" + name, "natGatewayPublicIpName": "pip-" + name, + **{k: network[k] for k in ("vnetAddressPrefix", "workspaceSubnetPrefix", "privateEndpointSubnetPrefix", "agentSubnetPrefix")}}) + for key in ("workspaceName", "vnetName", "managedIdentityName", "natGatewayName", "natGatewayPublicIpName"): + if output[key].lower() == str(values.get(key, "")).lower(): + raise ValueError("Fallback names must be different from retained names: " + key) + scope = f"/subscriptions/{cfg['subscriptionId']}/resourceGroups/{cfg['resourceGroup']}" + vnet = scope + "/providers/Microsoft.Network/virtualNetworks/" + output["vnetName"] + storage = scope + "/providers/Microsoft.Storage/storageAccounts/" + output["storageAccountName"] + for key, target in (("platformRoleAssignmentName", scope), ("acrPullRoleAssignmentName", scope), + ("blobRoleAssignmentName", storage), ("networkRoleAssignmentName", vnet)): + output[key] = str(uuid.uuid5(uuid.NAMESPACE_URL, "|".join((target.lower(), output["managedIdentityName"], key)))) + return cfg, output + + +def save_bundle(directory, config, values): + directory = Path(directory).resolve() + # A new directory is mandatory: never overwrite current or historical inputs. + directory.mkdir(parents=True, mode=0o700, exist_ok=False) + config = dict(config, parametersFile="parameters.json") + write_json(directory / "deployment.config.json", config) + write_json(directory / "parameters.json", { + "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#", + "contentVersion": "1.0.0.0", "parameters": {k: {"value": v} for k, v in values.items()}, + }) + return directory / "deployment.config.json" diff --git a/utilities/regional-deployment/regional_deploy.py b/utilities/regional-deployment/regional_deploy.py new file mode 100644 index 00000000..15151d64 --- /dev/null +++ b/utilities/regional-deployment/regional_deploy.py @@ -0,0 +1,262 @@ +#!/usr/bin/env python3 +"""Opt-in Discovery regional recommendations and reviewed cross-region deployment.""" + +import argparse +from datetime import datetime, timezone +import json +from pathlib import Path +import re +import sys + +from capacity import RegionProbe, classify_failure, nearby_regions, read_query +from deploy import Workflow, digest, write_json +from inputs import full_inputs, save_bundle, workspace_inputs +from workspace import WorkspaceWorkflow + + +def discovery_get(runner, path, label): + return read_query(runner, label, ["rest", "--method", "get", "--url", + f"https://management.azure.com{runner.rg_id}/providers/Microsoft.Discovery/{path}?api-version={runner.config['apiVersion']}"], + scoped=False, optional=True, not_found_only=True) + + +def compute_owners(runner): + listing = discovery_get(runner, "workspaces", "recommend-ownership-list") + if not listing or not isinstance(listing.get("value"), list) or listing.get("nextLink"): + raise ValueError("Workspace ownership inventory is incomplete; do not propose compute reuse.") + sc_id = (runner.rg_id+"/providers/Microsoft.Discovery/supercomputers/"+runner.values["supercomputerName"]).lower() + owners = [] + for item in listing["value"]: + name = item.get("name", "") + if not re.fullmatch(r"[A-Za-z0-9-]{3,24}", name): + raise ValueError("Workspace inventory contains an invalid name.") + full = discovery_get(runner, "workspaces/"+name, "recommend-owner-"+name) + links = full.get("properties", {}).get("supercomputerIds") if full else None + if not isinstance(links, list) or not all(isinstance(x, str) for x in links): + raise ValueError("A workspace's compute ownership is unverified.") + if sc_id in {value.lower() for value in links}: + owners.append({"name": name, "state": full.get("properties", {}).get("provisioningState")}) + if len(owners) > 1: + raise ValueError("Compute ownership is ambiguous; resolve it without automatic deletion.") + return owners + + +def recommendation(runner, args): + supplied = json.loads(Path(args.failure_file).read_text()) if args.failure_file else None + diagnosis = classify_failure(supplied) if supplied is not None else None + report = {"time": datetime.now(timezone.utc).isoformat(), "sourceConfigHash": digest(runner.config_path), + "sourceParametersHash": digest(runner.parameters_path), "codeHash": runner.code_hash(), + "defaultRegion": runner.data_location, "candidates": [], "capacityReserved": False, + "failureHash": digest(args.failure_file) if args.failure_file else None} + if diagnosis is not None and not diagnosis["actionable"]: + report["diagnosis"] = diagnosis + return report + if runner.config.get("deploymentMode") != "full": + raise ValueError("Use the original full configuration for recommendations, not a previous workspace fallback.") + runner.context() + sc = discovery_get(runner, "supercomputers/"+runner.values["supercomputerName"], "recommend-source-compute") + if sc and sc.get("properties", {}).get("provisioningState") not in ("Succeeded", "Failed", "Canceled"): + raise ValueError("Source compute is in progress or unverified; wait for a terminal state before choosing a fallback.") + source_ws = discovery_get(runner, "workspaces/"+runner.values["workspaceName"], "recommend-source-workspace") + if source_ws and source_ws.get("properties", {}).get("provisioningState") != "Failed": + report["diagnosis"] = {"actionable": False, "kind": "blocked", "codes": [], + "detail": "The source Workspace is healthy, in progress or unverified; this utility does not migrate it."} + return report + succeeded = bool(sc and sc.get("properties", {}).get("provisioningState") == "Succeeded") + scope = args.scope if args.scope != "auto" else ("workspace" if succeeded else "fresh") + if (scope == "workspace") != succeeded: + raise ValueError("Reuse Succeeded compute with workspace scope; fresh scope is only for a source without Succeeded compute.") + if succeeded: + node = discovery_get(runner, "supercomputers/"+runner.values["supercomputerName"]+"/nodePools/"+runner.values["nodePoolName"], "recommend-retained-node") + store = discovery_get(runner, "storageContainers/"+runner.values["storageContainerName"], "recommend-retained-store") + if any(not value or value.get("properties", {}).get("provisioningState") != "Succeeded" for value in (node, store)): + raise ValueError("Workspace reuse requires the node pool and StorageContainer to be Succeeded. Repair the retained unit or explicitly plan a separate new compute stack; moving only Workspace cannot fix compute quota.") + codes = set((diagnosis or {}).get("codes", [])) + if codes & {"vmskuquota", "akscapacityheavyusage"} or (supplied and re.search(r"exceeding approved .+(cores|vcpu).+quota", json.dumps(supplied), re.I)): + raise ValueError("The failure concerns retained compute, not the Workspace region. Request original-region quota or separately review a new compute stack.") + owners = compute_owners(runner) + report["computeOwners"] = owners + if any(owner["state"] != "Failed" for owner in owners): + report["diagnosis"] = {"actionable": False, "kind": "blocked", "codes": [], + "detail": "Another healthy/in-progress workspace owns compute; refresh stale source inputs instead of relocating."} + return report + probe = RegionProbe(runner) + primary = probe.probe(runner.data_location, scope == "fresh") + report.update(scope=scope, primary=primary, sourceComputeSucceeded=succeeded) + report["diagnosis"] = diagnosis or classify_failure(primary) + if not report["diagnosis"]["actionable"]: + return report + location_response = read_query(runner, "subscription-locations", ["rest", "--method", "get", "--url", + f"https://management.azure.com{runner.scope}/locations?api-version=2022-12-01"], scoped=False) + if not isinstance(location_response.get("value"), list) or location_response.get("nextLink"): + raise ValueError("Subscription-scoped location metadata is incomplete.") + locations = location_response["value"] + candidates = nearby_regions(locations, runner.data_location, args.candidates, args.allow_cross_geography) + for candidate in candidates[:args.limit]: + candidate.update(probe.probe(candidate["region"], scope == "fresh")) + report["candidates"].append(candidate) + report["allowCrossGeography"] = args.allow_cross_geography + return report + + +def assert_recommendation(report, runner, region, mode, now=None): + age = ((now or datetime.now(timezone.utc))-datetime.fromisoformat(report["time"])).total_seconds() + if not 0 <= age <= 3600 or not report.get("diagnosis", {}).get("actionable"): + raise ValueError("Recommendation must be recent and have a verified quota/capacity trigger.") + for field, value in (("sourceConfigHash", digest(runner.config_path)), + ("sourceParametersHash", digest(runner.parameters_path)), ("codeHash", runner.code_hash())): + if report.get(field) != value: + raise ValueError("Recommendation inputs or utility code changed: " + field) + if report.get("scope") != mode: + raise ValueError("Selected mode differs from the diagnosed preservation boundary.") + found = [x for x in report.get("candidates", []) if x.get("region") == region] + if len(found) != 1 or found[0].get("eligibleForPreflight") is not True or not found[0].get("checks"): + raise ValueError("Selected region lacks verified candidate evidence.") + if any(check.get("status") != "PASS" for check in found[0]["checks"]): + raise ValueError("Unknown or failed regional evidence cannot authorize a plan.") + return found[0] + + +def select_inputs(runner, args): + if not args.accept_cross_region: + raise ValueError("Review residency, cost and private connectivity, then explicitly use --accept-cross-region.") + report = json.loads(Path(args.recommendation).read_text()) + selected = assert_recommendation(report, runner, args.region, args.mode) + runner.context() + sc = discovery_get(runner, "supercomputers/"+runner.values["supercomputerName"], "select-source-compute") + if sc and sc.get("properties", {}).get("provisioningState") not in ("Succeeded", "Failed", "Canceled"): + raise ValueError("Source compute is now in progress; do not create an overlapping fallback stack.") + sc_ready = bool(sc and sc.get("properties", {}).get("provisioningState") == "Succeeded") + if args.mode == "fresh": + if sc_ready: + raise ValueError("Compute is now Succeeded; preserve it with a fresh workspace-scope recommendation.") + if not args.target_resource_group or args.target_resource_group.lower() == runner.rg.lower(): + raise ValueError("Fresh fallback requires an explicitly named different target RG; the source group is preserved.") + source_ws = discovery_get(runner, "workspaces/"+runner.values["workspaceName"], "select-source-workspace") + if source_ws and source_ws.get("properties", {}).get("provisioningState") != "Failed": + raise ValueError("Source Workspace is now healthy/in progress; refresh the recovery boundary.") + config, values = full_inputs(runner.sub, runner.config["tenantId"], args.target_resource_group, + args.name_prefix, runner.location, args.region, args.vnet_cidr) + if values["workspaceName"].lower() == runner.values["workspaceName"].lower() or values["supercomputerName"].lower() == runner.values["supercomputerName"].lower(): + raise ValueError("A fresh fallback must use new compute/workspace names; choose a different prefix.") + for key in ("systemSku", "nodePoolVmSize", "nodePoolMaxNodeCount", "nodePoolMinNodeCount", "nodePoolScaleSetPriority", + "storageAccountSku", "blobContainerName", "networkIsolation", "enableExtensions", "enableGhcpAiFeatures"): + if key in runner.values: + values[key] = runner.values[key] + else: + if not sc_ready: + raise ValueError("Retained compute is no longer Succeeded; do not generate a reuse plan.") + if args.target_resource_group and args.target_resource_group.lower() != runner.rg.lower(): + raise ValueError("Workspace reuse is currently limited to the same subscription and customer RG.") + source_name = args.source_workspace or runner.values["workspaceName"] + if not re.fullmatch(r"[A-Za-z0-9-]{3,24}", source_name): + raise ValueError("Invalid source workspace name.") + owners = compute_owners(runner) + if owners and (owners[0]["state"] != "Failed" or owners[0]["name"].lower() != source_name.lower()): + raise ValueError("The exact Failed compute owner must match --source-workspace; do not detach another or healthy workspace.") + source = discovery_get(runner, "workspaces/"+source_name, "select-source-workspace") + if source and source.get("properties", {}).get("provisioningState") != "Failed": + raise ValueError("Source workspace is not confirmed Failed; healthy or in-progress workspaces must not be migrated.") + vnet = read_query(runner, "select-retained-vnet", ["network", "vnet", "show", "--resource-group", runner.rg, "--name", runner.values["vnetName"]]) + if vnet["location"] != runner.data_location or sc.get("tags", {}).get("discovery.overridemrgregion") != runner.data_location: + raise ValueError("Source compute/VNet region differs from the original configuration; refresh the inventory.") + dns = read_query(runner, "select-retained-blob-dns", ["network", "private-dns", "record-set", "a", "show", "--resource-group", runner.rg, + "--zone-name", "privatelink.blob.core.windows.net", "--name", runner.values["storageAccountName"]]) + ips = {item["ipv4Address"] for item in dns.get("aRecords", [])} + if len(ips) != 1: + raise ValueError("This baseline requires one retained Blob A record; use a reviewed design for multi-IP/custom DNS.") + config, values = workspace_inputs(runner.config, runner.values, args.name_prefix, args.region, args.vnet_cidr, ips.pop()) + config["sourceWorkspaceName"] = source_name + config["regionalSelection"] = {"defaultRegion": runner.data_location, "selectedRegion": args.region, + "recommendationHash": digest(args.recommendation), "crossRegionApproved": True, + "distanceKm": selected["distanceKm"], "geographyGroup": selected["geographyGroup"], + "capacityReserved": False} + return save_bundle(args.output_dir, config, values) + + +def parser(): + top = argparse.ArgumentParser(description=__doc__) + commands = top.add_subparsers(dest="command", required=True) + init = commands.add_parser("init", help="Generate local source inputs only; no Azure calls.") + for flag in ("subscription", "tenant", "resource-group", "name-prefix", "workload-region", "output-dir"): + init.add_argument("--"+flag, required=True) + init.add_argument("--control-plane-region", default="eastus", choices=("eastus", "uksouth", "swedencentral")) + init.add_argument("--vnet-cidr", default="10.20.0.0/16") + recommend = commands.add_parser("recommend", help="Read-only quota/capacity checks and geographic recommendations.") + recommend.add_argument("--scope", choices=("auto", "fresh", "workspace"), default="auto") + recommend.add_argument("--failure-file", help="Optional fresh structured ARM failure; access/policy errors block fallback.") + recommend.add_argument("--candidates", nargs="+", help="Optional operator-approved candidate allowlist; otherwise use same-geography locations.") + recommend.add_argument("--allow-cross-geography", action="store_true") + recommend.add_argument("--limit", type=int, default=3) + recommend.add_argument("--report", required=True) + select = commands.add_parser("select", help="Generate reviewed target inputs; no Azure writes.") + select.add_argument("--recommendation", required=True) + select.add_argument("--mode", choices=("fresh", "workspace"), required=True) + select.add_argument("--region", required=True) + select.add_argument("--name-prefix", required=True) + select.add_argument("--target-resource-group") + select.add_argument("--source-workspace") + select.add_argument("--vnet-cidr", default="10.60.0.0/16") + select.add_argument("--accept-cross-region", action="store_true") + select.add_argument("--output-dir", required=True) + for name in ("prepare", "preflight", "validate", "deploy", "status", "wait", "verify", "audit", "postconfigure", "release-link"): + command = commands.add_parser(name) + command.add_argument("--stage", choices=("full", "discovery", "workspace", "workspace-core")) + command.add_argument("--apply", action="store_true") + command.add_argument("--allow-modify", action="store_true") + command.add_argument("--release-method", choices=("patch", "put"), default="put") + command.add_argument("--timeout", type=int, default=7200) + command.add_argument("--interval", type=int, default=60) + for name, command in commands.choices.items(): + if name != "init": + command.add_argument("--config", required=True) + command.add_argument("--evidence-dir") + return top + + +def main(argv=None): + cli = parser() + args = cli.parse_args(argv) + try: + if args.command == "init": + config, values = full_inputs(args.subscription, args.tenant, args.resource_group, args.name_prefix, + args.control_plane_region, args.workload_region, args.vnet_cidr) + print(save_bundle(args.output_dir, config, values)) + return 0 + config = json.loads(Path(args.config).read_text()) + workspace = config.get("deploymentMode") == "workspace" + default_stage = "workspace-core" if workspace else "discovery" + args.stage = getattr(args, "stage", None) or ("full" if args.command in ("recommend", "select") else default_stage) + if workspace != args.stage.startswith("workspace"): + raise ValueError("Stage does not match deploymentMode; do not mix full and workspace-only templates.") + runner = (WorkspaceWorkflow if workspace else Workflow)(args) + if args.command == "recommend": + if not 1 <= args.limit <= 8: + raise ValueError("Probe limit must be between 1 and 8 regions.") + report = recommendation(runner, args) + path = Path(args.report) + path.parent.mkdir(parents=True, exist_ok=True, mode=0o700) + with path.open("x") as stream: + json.dump(report, stream, indent=2) + stream.write("\n") + for row in report["candidates"]: + print(row["region"], row["distanceKm"], "km", "eligible for preflight" if row["eligibleForPreflight"] else "NOT VERIFIED") + print("No capacity reserved or Azure writes performed. Report:", path) + return 0 if any(x["eligibleForPreflight"] for x in report["candidates"]) else 2 + if args.command == "select": + print("Created target inputs:", select_inputs(runner, args)) + return 0 + if args.timeout < 1 or args.interval < 15: + raise ValueError("Use a positive timeout and polling interval of at least 15 seconds.") + if args.command == "release-link" and not workspace: + raise ValueError("Association release requires an explicit workspace-reuse configuration.") + getattr(runner, args.command.replace("-", "_"))() + print("Evidence:", runner.out) + return 0 + except (OSError, RuntimeError, ValueError, KeyError, TypeError) as error: + print("ERROR:", error, file=sys.stderr) + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/utilities/regional-deployment/templates/discovery.bicep b/utilities/regional-deployment/templates/discovery.bicep new file mode 100644 index 00000000..975b886e --- /dev/null +++ b/utilities/regional-deployment/templates/discovery.bicep @@ -0,0 +1,149 @@ +@description('Discovery-only resume deployment. Existing foundation resources are never redeployed.') +@allowed([ + 'eastus' + 'swedencentral' + 'uksouth' +]) +param location string +param overridemrgregion string = location +param supercomputerName string +param nodePoolName string = 'nodepool1' +param workspaceName string +param chatModelDeploymentName string = 'gpt-5-4' +param storageContainerName string +param projectName string +param vnetName string +param managedIdentityName string +param storageAccountName string +param nodePoolVmSize string = 'Standard_D4s_v6' + +param systemSku string = 'Standard_D4s_v6' +@minValue(0) +param nodePoolMinNodeCount int = 0 +@minValue(1) +param nodePoolMaxNodeCount int = 3 +@allowed([ + 'Regular' + 'Spot' +]) +param nodePoolScaleSetPriority string = 'Regular' +param chatModelFormat string = 'OpenAI' +param chatModelName string = 'gpt-5.4' +param enableGhcpAiFeatures bool = true +param enableExtensions bool = true +param networkIsolation bool = true + +resource vnet 'Microsoft.Network/virtualNetworks@2024-05-01' existing = { + name: vnetName +} + +resource managedIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2024-11-30' existing = { + name: managedIdentityName +} + +resource storageAccount 'Microsoft.Storage/storageAccounts@2023-05-01' existing = { + name: storageAccountName +} + +resource supercomputer 'Microsoft.Discovery/supercomputers@2026-06-01' = { + name: supercomputerName + location: location + tags: { + version: 'v2' + 'discovery.overridemrgregion': overridemrgregion + } + properties: { + subnetId: resourceId('Microsoft.Network/virtualNetworks/subnets', vnet.name, 'aksSubnet') + systemSku: systemSku + managementSubnetId: resourceId('Microsoft.Network/virtualNetworks/subnets', vnet.name, 'sc-mgmt') + identities: { + clusterIdentity: { + id: managedIdentity.id + } + kubeletIdentity: { + id: managedIdentity.id + } + workloadIdentities: { + '${managedIdentity.id}': {} + } + } + } +} + +resource nodePool 'Microsoft.Discovery/supercomputers/nodePools@2026-06-01' = { + parent: supercomputer + name: nodePoolName + location: location + properties: { + subnetId: resourceId('Microsoft.Network/virtualNetworks/subnets', vnet.name, 'supercomputerNodepoolSubnet') + vmSize: nodePoolVmSize + maxNodeCount: nodePoolMaxNodeCount + minNodeCount: nodePoolMinNodeCount + scaleSetPriority: nodePoolScaleSetPriority + } +} + +resource workspace 'Microsoft.Discovery/workspaces@2026-06-01' = { + name: workspaceName + location: location + tags: { + version: 'v2' + 'discovery.workbench.enableGhcpAiFeatures': string(enableGhcpAiFeatures) + 'discovery.workbench.enableExtensions': string(enableExtensions) + 'discovery.overridemrgregion': overridemrgregion + NetworkIsolation: string(networkIsolation) + } + properties: { + workspaceIdentity: { + id: managedIdentity.id + } + supercomputerIds: [ + supercomputer.id + ] + agentSubnetId: resourceId('Microsoft.Network/virtualNetworks/subnets', vnet.name, 'agentSubnet') + privateEndpointSubnetId: resourceId('Microsoft.Network/virtualNetworks/subnets', vnet.name, 'privateEndpointSubnet') + workspaceSubnetId: resourceId('Microsoft.Network/virtualNetworks/subnets', vnet.name, 'workspaceSubnet') + } +} + +resource chatModelDeployment 'Microsoft.Discovery/workspaces/chatModelDeployments@2026-06-01' = { + parent: workspace + name: chatModelDeploymentName + location: location + properties: { + modelFormat: chatModelFormat + modelName: chatModelName + } +} + +resource discoveryStorageContainer 'Microsoft.Discovery/storageContainers@2026-06-01' = { + name: storageContainerName + location: location + properties: { + storageStore: { + kind: 'AzureStorageBlob' + storageAccountId: storageAccount.id + } + } +} + +resource project 'Microsoft.Discovery/workspaces/projects@2026-06-01' = { + parent: workspace + name: projectName + location: location + dependsOn: [ + chatModelDeployment + ] + properties: { + storageContainerIds: [ + discoveryStorageContainer.id + ] + } +} + +output supercomputerId string = supercomputer.id +output nodePoolId string = nodePool.id +output workspaceId string = workspace.id +output chatModelDeploymentId string = chatModelDeployment.id +output storageContainerId string = discoveryStorageContainer.id +output projectId string = project.id diff --git a/utilities/regional-deployment/templates/main.bicep b/utilities/regional-deployment/templates/main.bicep new file mode 100644 index 00000000..5d1bcd53 --- /dev/null +++ b/utilities/regional-deployment/templates/main.bicep @@ -0,0 +1,589 @@ +@description('Discovery ARM control-plane region. Physical resources use overridemrgregion.') +@allowed([ + 'eastus' + 'swedencentral' + 'uksouth' +]) +param location string = 'uksouth' + +@description('Azure region for all dataplane resources. Defaults to location when not specified.') +param overridemrgregion string = location + +@description('Name of the Microsoft Discovery Supercomputer. Must be 3-24 characters, alphanumeric and hyphens only.') +@minLength(3) +@maxLength(24) +param supercomputerName string = 'sc-${uniqueString(resourceGroup().id)}' + +@description('Name of the Node Pool created under the Supercomputer. Must be 1-12 lowercase alphanumeric characters, starting with a letter.') +@minLength(1) +@maxLength(12) +param nodePoolName string = 'nodepool1' + +@description('Name of the Microsoft Discovery Workspace. Must be 3-24 characters, alphanumeric and hyphens only.') +@minLength(3) +@maxLength(24) +param workspaceName string = 'ws-${uniqueString(resourceGroup().id)}' + +@description('Name of the Chat Model Deployment created under the Workspace.') +@minLength(3) +@maxLength(24) +param chatModelDeploymentName string = 'gpt-5-4' + +@description('Name of the Microsoft Discovery Storage Container resource. Must be 3-24 characters, alphanumeric and hyphens only.') +@minLength(3) +@maxLength(24) +param storageContainerName string = 'stc-${uniqueString(resourceGroup().id)}' + +@description('Name of the Project created under the Workspace. Must be 3-24 characters, alphanumeric and hyphens only.') +@minLength(3) +@maxLength(24) +param projectName string = 'prj-${uniqueString(resourceGroup().id)}' + +@description('Name of the Virtual Network. Must be 2-64 characters: letters, numbers, underscores, periods, or hyphens; must start with a letter or number and end with a letter, number, or underscore.') +@minLength(2) +@maxLength(64) +param vnetName string = 'discovery-vnet' + +@description('Name of the NAT gateway providing explicit outbound connectivity for the workload subnets.') +param natGatewayName string = 'nat-${uniqueString(resourceGroup().id)}' + +@description('Name of the static Standard IPv4 public IP used by the NAT gateway.') +param natGatewayPublicIpName string = 'pip-nat-${uniqueString(resourceGroup().id)}' + +@description('Name of the User-Assigned Managed Identity.') +param managedIdentityName string = 'uami-${uniqueString(resourceGroup().id)}' + +@description('Globally unique name of the Azure Storage Account (3-24 lowercase alphanumeric characters).') +@minLength(3) +@maxLength(24) +param storageAccountName string = 'stg${uniqueString(resourceGroup().id)}' + +@description('Name of the blob container inside the Storage Account used for Discovery outputs.') +param blobContainerName string = 'discoveryoutputs' + +@description('Replication SKU for the Storage Account. Zone/geo-redundant options are recommended over locally-redundant storage for resiliency.') +@allowed([ + 'Standard_ZRS' + 'Standard_GRS' + 'Standard_GZRS' + 'Standard_RAGRS' + 'Standard_RAGZRS' +]) +param storageAccountSku string = 'Standard_GRS' + +@description('Address space for the Virtual Network.') +param vnetAddressPrefix string = '10.0.0.0/16' + +@description('Address prefix for the Supercomputer Node Pool subnet.') +param supercomputerNodepoolSubnetPrefix string = '10.0.1.0/24' + +@description('Address prefix for the AKS system subnet used by the Supercomputer.') +param aksSubnetPrefix string = '10.0.2.0/24' + +@description('Address prefix for the Workspace subnet (delegated to Microsoft.App/environments).') +param workspaceSubnetPrefix string = '10.0.3.0/24' + +@description('Address prefix for the Private Endpoint subnet.') +param privateEndpointSubnetPrefix string = '10.0.4.0/24' + +@description('Address prefix for the Agent subnet.') +param agentSubnetPrefix string = '10.0.5.0/24' + +@description('Address prefix for Search Subnet.') +param searchSubnetPrefix string = '10.0.6.0/24' + +@description('Address prefix for the Supercomputer management subnet used by AKS API Server VNet Integration. Must be /28 or larger.') +param managementSubnetPrefix string = '10.0.7.0/28' + +@description('VM SKU for the Node Pool.') +param nodePoolVmSize string = 'Standard_D4s_v6' + +@description('VM SKU for the managed AKS system pool; quota reserves three nodes.') +param systemSku string = 'Standard_D4s_v6' + +@description('Maximum number of nodes in the Node Pool.') +@minValue(1) +param nodePoolMaxNodeCount int = 3 + +@description('Minimum number of nodes in the Node Pool (0 allows scale-to-zero).') +@minValue(0) +param nodePoolMinNodeCount int = 0 + +@description('Scale set priority for the Node Pool.') +@allowed([ + 'Regular' + 'Spot' +]) +param nodePoolScaleSetPriority string = 'Regular' + +@description('Chat model format.') +param chatModelFormat string = 'OpenAI' + +@description('Chat model name to deploy.') +param chatModelName string = 'gpt-5.4' + +@description('Enable GitHub Copilot and AI features in the Discovery workspace via the discovery.workbench.enableGhcpAiFeatures tag.') +param enableGhcpAiFeatures bool = true + +@description('Enable the VS Code Extension Marketplace in the Discovery workspace via the discovery.workbench.enableExtensions tag.') +param enableExtensions bool = true + +@description('Workspace network isolation mode via the NetworkIsolation tag. Set to false to enable public preview access as documented in the Infrastructure portal quickstart.') +param networkIsolation bool = true + +// Built-in role definition IDs +var storageBlobDataContributorRoleId = 'ba92f5b4-2d11-453d-a403-e96b0029c9fe' +var discoveryPlatformContributorRoleId = '01288891-85ee-45a7-b367-9db3b752fc65' +var acrPullRoleId = '7f951dda-4ed3-4680-a7ca-43fe172d538d' +var networkContributorRoleId = '4d97b98b-1d4f-4787-a291-c67834d212e7' + +resource natGatewayPublicIp 'Microsoft.Network/publicIPAddresses@2024-05-01' = { + name: natGatewayPublicIpName + location: overridemrgregion + sku: { + name: 'Standard' + tier: 'Regional' + } + properties: { + publicIPAllocationMethod: 'Static' + publicIPAddressVersion: 'IPv4' + } +} + +resource natGateway 'Microsoft.Network/natGateways@2024-05-01' = { + name: natGatewayName + location: overridemrgregion + sku: { + name: 'Standard' + } + properties: { + publicIpAddresses: [ + { + id: natGatewayPublicIp.id + } + ] + } +} + +resource vnet 'Microsoft.Network/virtualNetworks@2024-05-01' = { + name: vnetName + location: overridemrgregion + properties: { + addressSpace: { + addressPrefixes: [ + vnetAddressPrefix + ] + } + subnets: [ + { + name: 'supercomputerNodepoolSubnet' + properties: { + addressPrefix: supercomputerNodepoolSubnetPrefix + defaultOutboundAccess: false + natGateway: { + id: natGateway.id + } + serviceEndpoints: [ + { + service: 'Microsoft.Storage' + } + ] + } + } + { + name: 'aksSubnet' + properties: { + addressPrefix: aksSubnetPrefix + defaultOutboundAccess: false + natGateway: { + id: natGateway.id + } + serviceEndpoints: [ + { + service: 'Microsoft.Storage' + } + ] + } + } + { + name: 'workspaceSubnet' + properties: { + addressPrefix: workspaceSubnetPrefix + defaultOutboundAccess: false + natGateway: { + id: natGateway.id + } + delegations: [ + { + name: 'Microsoft.App.environments' + properties: { + serviceName: 'Microsoft.App/environments' + } + } + ] + serviceEndpoints: [ + { + service: 'Microsoft.Storage' + } + ] + } + } + { + name: 'privateEndpointSubnet' + properties: { + addressPrefix: privateEndpointSubnetPrefix + defaultOutboundAccess: false + } + } + { + name: 'agentSubnet' + properties: { + addressPrefix: agentSubnetPrefix + defaultOutboundAccess: false + natGateway: { + id: natGateway.id + } + delegations: [ + { + name: 'Microsoft.App.environments' + properties: { + serviceName: 'Microsoft.App/environments' + } + } + ] + serviceEndpoints: [ + { + service: 'Microsoft.Storage' + } + ] + } + } + { + name: 'searchSubnet' + properties: { + addressPrefix: searchSubnetPrefix + defaultOutboundAccess: false + natGateway: { + id: natGateway.id + } + delegations: [ + { + name: 'Microsoft.App.environments' + properties: { + serviceName: 'Microsoft.App/environments' + } + } + ] + serviceEndpoints: [ + { + service: 'Microsoft.Storage' + } + ] + } + } + { + name: 'sc-mgmt' + properties: { + addressPrefix: managementSubnetPrefix + defaultOutboundAccess: false + delegations: [ + { + name: 'Microsoft.ContainerService.managedClusters' + properties: { + serviceName: 'Microsoft.ContainerService/managedClusters' + } + } + ] + } + } + ] + } +} + +resource managedIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2024-11-30' = { + name: managedIdentityName + location: overridemrgregion + properties: { + isolationScope: 'Regional' + } +} + +resource storageAccount 'Microsoft.Storage/storageAccounts@2023-05-01' = { + name: storageAccountName + location: overridemrgregion + kind: 'StorageV2' + sku: { + name: storageAccountSku + } + dependsOn: [ + vnet + ] + properties: { + accessTier: 'Hot' + allowBlobPublicAccess: false + allowSharedKeyAccess: false + publicNetworkAccess: 'Disabled' + minimumTlsVersion: 'TLS1_2' + supportsHttpsTrafficOnly: true + // Customer storage stays private; Blob access uses the private endpoint. + networkAcls: { + defaultAction: 'Allow' + bypass: 'AzureServices' + virtualNetworkRules: [ + { + id: resourceId('Microsoft.Network/virtualNetworks/subnets', vnetName, 'supercomputerNodepoolSubnet') + action: 'Allow' + } + { + id: resourceId('Microsoft.Network/virtualNetworks/subnets', vnetName, 'aksSubnet') + action: 'Allow' + } + { + id: resourceId('Microsoft.Network/virtualNetworks/subnets', vnetName, 'workspaceSubnet') + action: 'Allow' + } + { + id: resourceId('Microsoft.Network/virtualNetworks/subnets', vnetName, 'agentSubnet') + action: 'Allow' + } + { + id: resourceId('Microsoft.Network/virtualNetworks/subnets', vnetName, 'searchSubnet') + action: 'Allow' + } + ] + } + } +} + +resource blobServices 'Microsoft.Storage/storageAccounts/blobServices@2023-05-01' = { + parent: storageAccount + name: 'default' + properties: { + cors: { + corsRules: [ + { + allowedOrigins: [ + 'https://studio.discovery.microsoft.com' + 'https://*.vscode-cdn.net' + 'https://vscode.dev' + ] + allowedMethods: [ + 'GET' + 'HEAD' + 'DELETE' + 'PUT' + ] + allowedHeaders: [ + '*' + ] + exposedHeaders: [ + '*' + ] + maxAgeInSeconds: 200 + } + ] + } + } +} + +resource blobContainer 'Microsoft.Storage/storageAccounts/blobServices/containers@2023-05-01' = { + parent: blobServices + name: blobContainerName + properties: { + publicAccess: 'None' + } +} + +module storagePrivateAccess 'modules/storage-private-access.bicep' = { + name: 'storage-private-access-${uniqueString(vnet.id)}' + params: { + location: overridemrgregion + storageAccountId: storageAccount.id + vnetId: vnet.id + privateEndpointSubnetId: resourceId('Microsoft.Network/virtualNetworks/subnets', vnetName, 'privateEndpointSubnet') + privateEndpointName: 'pe-${storageAccountName}-blob' + vnetLinkName: 'link-${vnetName}' + } +} + +resource storageBlobDataContributorAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = { + name: guid(storageAccount.id, managedIdentity.id, storageBlobDataContributorRoleId) + scope: storageAccount + properties: { + roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', storageBlobDataContributorRoleId) + principalId: managedIdentity.properties.principalId + principalType: 'ServicePrincipal' + } +} + +resource discoveryPlatformContributorAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = { + name: guid(resourceGroup().id, managedIdentity.id, discoveryPlatformContributorRoleId) + properties: { + roleDefinitionId: subscriptionResourceId( + 'Microsoft.Authorization/roleDefinitions', + discoveryPlatformContributorRoleId + ) + principalId: managedIdentity.properties.principalId + principalType: 'ServicePrincipal' + } +} + +resource acrPullAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = { + name: guid(resourceGroup().id, managedIdentity.id, acrPullRoleId) + properties: { + roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', acrPullRoleId) + principalId: managedIdentity.properties.principalId + principalType: 'ServicePrincipal' + } +} + +resource networkContributorAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = { + name: guid(vnet.id, managedIdentity.id, networkContributorRoleId) + scope: vnet + properties: { + roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', networkContributorRoleId) + principalId: managedIdentity.properties.principalId + principalType: 'ServicePrincipal' + } +} + +resource supercomputer 'Microsoft.Discovery/supercomputers@2026-06-01' = { + name: supercomputerName + location: location + tags: { + version: 'v2' + 'discovery.overridemrgregion': string(overridemrgregion) + } + dependsOn: [ + vnet + networkContributorAssignment + storageBlobDataContributorAssignment + discoveryPlatformContributorAssignment + acrPullAssignment + ] + properties: { + subnetId: resourceId('Microsoft.Network/virtualNetworks/subnets', vnetName, 'aksSubnet') + systemSku: systemSku + managementSubnetId: resourceId('Microsoft.Network/virtualNetworks/subnets', vnetName, 'sc-mgmt') + identities: { + clusterIdentity: { + id: managedIdentity.id + } + kubeletIdentity: { + id: managedIdentity.id + } + workloadIdentities: { + '${managedIdentity.id}': {} + } + } + } +} + +resource nodePool 'Microsoft.Discovery/supercomputers/nodePools@2026-06-01' = { + parent: supercomputer + name: nodePoolName + location: location + dependsOn: [ + vnet + ] + properties: { + subnetId: resourceId('Microsoft.Network/virtualNetworks/subnets', vnetName, 'supercomputerNodepoolSubnet') + vmSize: nodePoolVmSize + maxNodeCount: nodePoolMaxNodeCount + minNodeCount: nodePoolMinNodeCount + scaleSetPriority: nodePoolScaleSetPriority + } +} + +resource workspace 'Microsoft.Discovery/workspaces@2026-06-01' = { + name: workspaceName + location: location + tags: { + version: 'v2' + 'discovery.workbench.enableGhcpAiFeatures': string(enableGhcpAiFeatures) + 'discovery.workbench.enableExtensions': string(enableExtensions) + 'discovery.overridemrgregion': string(overridemrgregion) + NetworkIsolation: string(networkIsolation) + } + dependsOn: [ + vnet + networkContributorAssignment + storagePrivateAccess + ] + properties: { + workspaceIdentity: { + id: managedIdentity.id + } + supercomputerIds: [ + supercomputer.id + ] + agentSubnetId: resourceId('Microsoft.Network/virtualNetworks/subnets', vnetName, 'agentSubnet') + privateEndpointSubnetId: resourceId('Microsoft.Network/virtualNetworks/subnets', vnetName, 'privateEndpointSubnet') + workspaceSubnetId: resourceId('Microsoft.Network/virtualNetworks/subnets', vnetName, 'workspaceSubnet') + } +} + +resource chatModelDeployment 'Microsoft.Discovery/workspaces/chatModelDeployments@2026-06-01' = { + parent: workspace + name: chatModelDeploymentName + location: location + properties: { + modelFormat: chatModelFormat + modelName: chatModelName + } +} + +resource discoveryStorageContainer 'Microsoft.Discovery/storageContainers@2026-06-01' = { + name: storageContainerName + location: location + dependsOn: [ + storagePrivateAccess + ] + properties: { + storageStore: { + kind: 'AzureStorageBlob' + storageAccountId: storageAccount.id + } + } +} + +resource project 'Microsoft.Discovery/workspaces/projects@2026-06-01' = { + parent: workspace + name: projectName + location: location + dependsOn: [ + chatModelDeployment + ] + properties: { + storageContainerIds: [ + discoveryStorageContainer.id + ] + } +} + +@description('Resource ID of the Supercomputer.') +output supercomputerId string = supercomputer.id + +@description('Resource ID of the Node Pool.') +output nodePoolId string = nodePool.id + +@description('Resource ID of the Workspace.') +output workspaceId string = workspace.id + +@description('Resource ID of the Chat Model Deployment.') +output chatModelDeploymentId string = chatModelDeployment.id + +@description('Resource ID of the Discovery Storage Container.') +output storageContainerId string = discoveryStorageContainer.id + +@description('Resource ID of the Project.') +output projectId string = project.id + +@description('Resource ID of the User-Assigned Managed Identity.') +output managedIdentityId string = managedIdentity.id + +@description('Resource ID of the Storage Account.') +output storageAccountId string = storageAccount.id + +@description('Resource ID of the Virtual Network.') +output vnetId string = vnet.id diff --git a/utilities/regional-deployment/templates/modules/storage-private-access.bicep b/utilities/regional-deployment/templates/modules/storage-private-access.bicep new file mode 100644 index 00000000..664a5a8b --- /dev/null +++ b/utilities/regional-deployment/templates/modules/storage-private-access.bicep @@ -0,0 +1,62 @@ +param location string +param storageAccountId string +param vnetId string +param privateEndpointSubnetId string +param privateEndpointName string +param vnetLinkName string + +resource blobPrivateDnsZone 'Microsoft.Network/privateDnsZones@2024-06-01' = { + name: 'privatelink.blob.${environment().suffixes.storage}' + location: 'global' +} + +resource vnetLink 'Microsoft.Network/privateDnsZones/virtualNetworkLinks@2024-06-01' = { + parent: blobPrivateDnsZone + name: vnetLinkName + location: 'global' + properties: { + registrationEnabled: false + virtualNetwork: { + id: vnetId + } + } +} + +resource blobPrivateEndpoint 'Microsoft.Network/privateEndpoints@2024-05-01' = { + name: privateEndpointName + location: location + properties: { + subnet: { + id: privateEndpointSubnetId + } + privateLinkServiceConnections: [ + { + name: 'blob' + properties: { + privateLinkServiceId: storageAccountId + groupIds: [ + 'blob' + ] + } + } + ] + } +} + +resource blobPrivateDnsZoneGroup 'Microsoft.Network/privateEndpoints/privateDnsZoneGroups@2024-05-01' = { + parent: blobPrivateEndpoint + name: 'default' + properties: { + privateDnsZoneConfigs: [ + { + name: 'blob' + properties: { + privateDnsZoneId: blobPrivateDnsZone.id + } + } + ] + } +} + +output privateEndpointId string = blobPrivateEndpoint.id +output privateDnsZoneId string = blobPrivateDnsZone.id diff --git a/utilities/regional-deployment/templates/workspace-core.bicep b/utilities/regional-deployment/templates/workspace-core.bicep new file mode 100644 index 00000000..3ef2258c --- /dev/null +++ b/utilities/regional-deployment/templates/workspace-core.bicep @@ -0,0 +1,47 @@ +param location string +param overridemrgregion string +param supercomputerName string +param storageContainerName string +param workspaceName string +param projectName string +param chatModelDeploymentName string = 'gpt-5-4' +param chatModelName string = 'gpt-5.4' +param vnetName string +param managedIdentityName string + +resource supercomputer 'Microsoft.Discovery/supercomputers@2026-06-01' existing = { name: supercomputerName } +resource storageContainer 'Microsoft.Discovery/storageContainers@2026-06-01' existing = { name: storageContainerName } +resource vnet 'Microsoft.Network/virtualNetworks@2024-05-01' existing = { name: vnetName } +resource identity 'Microsoft.ManagedIdentity/userAssignedIdentities@2024-11-30' existing = { name: managedIdentityName } +resource workspace 'Microsoft.Discovery/workspaces@2026-06-01' = { + name: workspaceName + location: location + tags: { + version: 'v2' + 'discovery.overridemrgregion': overridemrgregion + 'discovery.workbench.enableGhcpAiFeatures': 'True' + 'discovery.workbench.enableExtensions': 'True' + NetworkIsolation: 'True' + } + properties: { + workspaceIdentity: { id: identity.id } + supercomputerIds: [supercomputer.id] + workspaceSubnetId: resourceId('Microsoft.Network/virtualNetworks/subnets', vnet.name, 'workspaceSubnet') + agentSubnetId: resourceId('Microsoft.Network/virtualNetworks/subnets', vnet.name, 'agentSubnet') + privateEndpointSubnetId: resourceId('Microsoft.Network/virtualNetworks/subnets', vnet.name, 'privateEndpointSubnet') + } +} +resource chatModel 'Microsoft.Discovery/workspaces/chatModelDeployments@2026-06-01' = { + parent: workspace + name: chatModelDeploymentName + location: location + properties: { modelFormat: 'OpenAI', modelName: chatModelName } +} +resource project 'Microsoft.Discovery/workspaces/projects@2026-06-01' = { + parent: workspace + name: projectName + location: location + dependsOn: [chatModel] + properties: { storageContainerIds: [storageContainer.id] } +} +output workspaceId string = workspace.id diff --git a/utilities/regional-deployment/templates/workspace.bicep b/utilities/regional-deployment/templates/workspace.bicep new file mode 100644 index 00000000..8c67256b --- /dev/null +++ b/utilities/regional-deployment/templates/workspace.bicep @@ -0,0 +1,192 @@ +@allowed([ + 'eastus' + 'uksouth' + 'swedencentral' +]) +param location string +param overridemrgregion string +param supercomputerName string +param storageContainerName string +param storageAccountName string +param existingVnetName string +param workspaceName string +param projectName string +param chatModelDeploymentName string = 'gpt-5-4' +param chatModelName string = 'gpt-5.4' +param vnetName string +param managedIdentityName string +param natGatewayName string +param natGatewayPublicIpName string +param platformRoleAssignmentName string +param acrPullRoleAssignmentName string +param blobRoleAssignmentName string +param networkRoleAssignmentName string +param vnetAddressPrefix string = '10.10.0.0/16' +param workspaceSubnetPrefix string = '10.10.3.0/24' +param privateEndpointSubnetPrefix string = '10.10.4.0/24' +param agentSubnetPrefix string = '10.10.5.0/24' + +resource supercomputer 'Microsoft.Discovery/supercomputers@2026-06-01' existing = { + name: supercomputerName +} +resource storageContainer 'Microsoft.Discovery/storageContainers@2026-06-01' existing = { + name: storageContainerName +} +resource storageAccount 'Microsoft.Storage/storageAccounts@2023-05-01' existing = { + name: storageAccountName +} +resource retainedVnet 'Microsoft.Network/virtualNetworks@2024-05-01' existing = { + name: existingVnetName +} +resource blobDns 'Microsoft.Network/privateDnsZones@2024-06-01' existing = { + name: 'privatelink.blob.${environment().suffixes.storage}' +} + +resource publicIp 'Microsoft.Network/publicIPAddresses@2024-05-01' = { + name: natGatewayPublicIpName + location: overridemrgregion + sku: { name: 'Standard', tier: 'Regional' } + properties: { publicIPAllocationMethod: 'Static', publicIPAddressVersion: 'IPv4' } +} +resource nat 'Microsoft.Network/natGateways@2024-05-01' = { + name: natGatewayName + location: overridemrgregion + sku: { name: 'Standard' } + properties: { publicIpAddresses: [{ id: publicIp.id }] } +} +resource vnet 'Microsoft.Network/virtualNetworks@2024-05-01' = { + name: vnetName + location: overridemrgregion + properties: { + addressSpace: { addressPrefixes: [vnetAddressPrefix] } + subnets: [ + { + name: 'workspaceSubnet' + properties: { + addressPrefix: workspaceSubnetPrefix + defaultOutboundAccess: false + natGateway: { id: nat.id } + delegations: [{ name: 'aca', properties: { serviceName: 'Microsoft.App/environments' } }] + serviceEndpoints: [{ service: 'Microsoft.Storage' }] + } + } + { + name: 'agentSubnet' + properties: { + addressPrefix: agentSubnetPrefix + defaultOutboundAccess: false + natGateway: { id: nat.id } + delegations: [{ name: 'aca', properties: { serviceName: 'Microsoft.App/environments' } }] + serviceEndpoints: [{ service: 'Microsoft.Storage' }] + } + } + { + name: 'privateEndpointSubnet' + properties: { addressPrefix: privateEndpointSubnetPrefix, defaultOutboundAccess: false } + } + ] + } +} +resource identity 'Microsoft.ManagedIdentity/userAssignedIdentities@2024-11-30' = { + name: managedIdentityName + location: overridemrgregion + properties: { isolationScope: 'Regional' } +} +resource platformRole 'Microsoft.Authorization/roleAssignments@2022-04-01' = { + name: platformRoleAssignmentName + properties: { + principalId: identity.properties.principalId + principalType: 'ServicePrincipal' + roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', '01288891-85ee-45a7-b367-9db3b752fc65') + } +} +resource acrPullRole 'Microsoft.Authorization/roleAssignments@2022-04-01' = { + name: acrPullRoleAssignmentName + properties: { + principalId: identity.properties.principalId + principalType: 'ServicePrincipal' + roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', '7f951dda-4ed3-4680-a7ca-43fe172d538d') + } +} +resource blobRole 'Microsoft.Authorization/roleAssignments@2022-04-01' = { + name: blobRoleAssignmentName + scope: storageAccount + properties: { + principalId: identity.properties.principalId + principalType: 'ServicePrincipal' + roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'ba92f5b4-2d11-453d-a403-e96b0029c9fe') + } +} +resource networkRole 'Microsoft.Authorization/roleAssignments@2022-04-01' = { + name: networkRoleAssignmentName + scope: vnet + properties: { + principalId: identity.properties.principalId + principalType: 'ServicePrincipal' + roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', '4d97b98b-1d4f-4787-a291-c67834d212e7') + } +} +resource toRetainedVnet 'Microsoft.Network/virtualNetworks/virtualNetworkPeerings@2024-05-01' = { + parent: vnet + name: 'peer-to-retained-discovery' + properties: { + remoteVirtualNetwork: { id: retainedVnet.id } + allowVirtualNetworkAccess: true + allowForwardedTraffic: false + allowGatewayTransit: false + useRemoteGateways: false + } +} +resource fromRetainedVnet 'Microsoft.Network/virtualNetworks/virtualNetworkPeerings@2024-05-01' = { + parent: retainedVnet + name: 'peer-to-${vnetName}' + properties: { + remoteVirtualNetwork: { id: vnet.id } + allowVirtualNetworkAccess: true + allowForwardedTraffic: false + allowGatewayTransit: false + useRemoteGateways: false + } +} +// Reuse the original Blob endpoint and A record; a second endpoint could overwrite shared DNS. +resource blobDnsLink 'Microsoft.Network/privateDnsZones/virtualNetworkLinks@2024-06-01' = { + parent: blobDns + name: 'link-${vnetName}' + location: 'global' + properties: { registrationEnabled: false, virtualNetwork: { id: vnet.id } } +} +resource workspace 'Microsoft.Discovery/workspaces@2026-06-01' = { + name: workspaceName + location: location + tags: { + version: 'v2' + 'discovery.overridemrgregion': overridemrgregion + 'discovery.workbench.enableGhcpAiFeatures': 'True' + 'discovery.workbench.enableExtensions': 'True' + NetworkIsolation: 'True' + } + dependsOn: [platformRole, acrPullRole, blobRole, networkRole, toRetainedVnet, fromRetainedVnet, blobDnsLink] + properties: { + workspaceIdentity: { id: identity.id } + supercomputerIds: [supercomputer.id] + workspaceSubnetId: resourceId('Microsoft.Network/virtualNetworks/subnets', vnet.name, 'workspaceSubnet') + agentSubnetId: resourceId('Microsoft.Network/virtualNetworks/subnets', vnet.name, 'agentSubnet') + privateEndpointSubnetId: resourceId('Microsoft.Network/virtualNetworks/subnets', vnet.name, 'privateEndpointSubnet') + } +} +resource chatModel 'Microsoft.Discovery/workspaces/chatModelDeployments@2026-06-01' = { + parent: workspace + name: chatModelDeploymentName + location: location + properties: { modelFormat: 'OpenAI', modelName: chatModelName } +} +resource project 'Microsoft.Discovery/workspaces/projects@2026-06-01' = { + parent: workspace + name: projectName + location: location + dependsOn: [chatModel] + properties: { storageContainerIds: [storageContainer.id] } +} +output workspaceId string = workspace.id +output reusedSupercomputerId string = supercomputer.id +output reusedStorageContainerId string = storageContainer.id diff --git a/utilities/regional-deployment/tests/support.py b/utilities/regional-deployment/tests/support.py new file mode 100644 index 00000000..2b889871 --- /dev/null +++ b/utilities/regional-deployment/tests/support.py @@ -0,0 +1,90 @@ +import argparse +import copy +from pathlib import Path +import sys + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) + +from deploy import Workflow +from inputs import full_inputs, save_bundle + +SUB = "11111111-1111-4111-8111-111111111111" +TENANT = "22222222-2222-4222-8222-222222222222" + + +def workflow(directory, command="recommend", stage="full", apply=False): + config, values = full_inputs(SUB, TENANT, "source-rg", "source", "eastus", "eastus2", "10.20.0.0/16") + path = save_bundle(Path(directory)/"inputs", config, values) + args = argparse.Namespace(config=str(path), evidence_dir=str(Path(directory)/"evidence"), command=command, + stage=stage, apply=apply, allow_modify=False, timeout=30, interval=15) + return Workflow(args) + + +class Backend: + """Synthetic ARM/CLI metadata; no credentials or Azure calls.""" + def __init__(self): + self.calls = [] + self.source_compute = None + self.source_workspace = None + self.offered = {"eastus2": ["standard"], "centralus": ["standard"], "northcentralus": ["standard"]} + self.model_free = 1000 + self.model_status = "Included" + self.model_rate = 1000 + self.fail = False + self.restricted = False + self.compute_limit = 100 + self.locations = [ + {"name": "eastus2", "metadata": {"latitude": "0", "longitude": "0", "geographyGroup": "US"}}, + {"name": "centralus", "metadata": {"latitude": "0", "longitude": "1", "geographyGroup": "US"}}, + {"name": "northcentralus", "metadata": {"latitude": "0", "longitude": "2", "geographyGroup": "US"}}, + {"name": "japaneast", "metadata": {"latitude": "0", "longitude": "0.1", "geographyGroup": "APAC"}}, + ] + self.owners = [] + + def az(self, label, arguments, **kwargs): + self.calls.append((label, arguments, kwargs)) + if self.fail: + raise RuntimeError("AuthorizationFailed: synthetic denied read") + if label == "subscription-locations": + return {"value":copy.deepcopy(self.locations)} + if label == "recommend-ownership-list": + return {"value":[{"name":x["name"]} for x in self.owners]} + if label.startswith("recommend-owner-"): + name=label.removeprefix("recommend-owner-") + owner=next(x for x in self.owners if x["name"]==name) + return {"properties":{"provisioningState":owner["state"],"supercomputerIds":[owner["computeId"]]}} + if label.endswith("source-compute"): + return copy.deepcopy(self.source_compute) + if label.endswith("source-workspace"): + return copy.deepcopy(self.source_workspace) + if label in ("recommend-retained-node", "recommend-retained-store"): + return {"properties":{"provisioningState":"Succeeded"}} + if label == "select-retained-vnet": + return {"location": "eastus2", "addressSpace": {"addressPrefixes": ["10.20.0.0/16"]}} + if label == "select-retained-blob-dns": + return {"aRecords": [{"ipv4Address": "10.20.4.9"}]} + if arguments[:1] == ["rest"]: + url = arguments[arguments.index("--url")+1] + if "fetchOfferings" in url: + return {"regions": [{"regionName": name, "skus": [{"name": sku} for sku in skus]} for name, skus in self.offered.items()]} + if "Microsoft.Search/locations/" in url: + return {"value": [{"name": {"value": "standard"}, "limit": 12, "currentValue": 0}]} + if arguments[:2] == ["quota", "list"]: + return [{"name": "managedenvironmentcount", "properties": {"limit": {"value": 50}}}] + if arguments[:3] == ["quota", "usage", "list"]: + return [{"name": "managedenvironmentcount", "properties": {"usages": {"value": 0}}}] + if arguments[:3] == ["cognitiveservices", "model", "list"]: + return [{"kind": "OpenAI", "model": {"name": "gpt-5.4", "format": "OpenAI", "version": "test-version", + "isDefaultVersion": True, "lifecycleStatus": "GenerallyAvailable", "skus": [{"name": "GlobalStandard", + "usageName": "OpenAI.GlobalStandard.gpt-5.4", "rateLimits": [{"count": self.model_rate, "renewalPeriod": 60}]}]}}] + if arguments[:3] == ["cognitiveservices", "usage", "list"]: + return [{"name": {"value": "OpenAI.GlobalStandard.gpt-5.4"}, "status": self.model_status, "limit": 1000, "currentValue": 1000-self.model_free}] + if arguments[:2] == ["vm", "list-skus"]: + sku = arguments[arguments.index("--size")+1] + family, cpu = ("standardESv5Family", 8) if sku == "Standard_E8s_v5" else ("standardDSv6Family", 4) + return [{"name": sku, "family": family, "restrictions": [{"reasonCode": "NotAvailableForSubscription"}] if self.restricted else [], + "capabilities": [{"name": "vCPUs", "value": str(cpu)}]}] + if arguments[:2] == ["vm", "list-usage"]: + return [{"name": {"value": name}, "limit": self.compute_limit, "currentValue": 0} for name in ("cores", "standardDSv6Family", "standardESv5Family")] + raise AssertionError("Unexpected synthetic query: " + label + " " + str(arguments)) diff --git a/utilities/regional-deployment/tests/test_capacity.py b/utilities/regional-deployment/tests/test_capacity.py new file mode 100644 index 00000000..8fe29eac --- /dev/null +++ b/utilities/regional-deployment/tests/test_capacity.py @@ -0,0 +1,169 @@ +import copy +import tempfile +import unittest +from unittest.mock import Mock + +from support import Backend, workflow +import capacity as c + + +class CapacityTests(unittest.TestCase): + def test_quota_and_capacity_are_different(self): + for code, kind in (("InsufficientQuota", "quota"), ("VmSkuQuota", "quota"), + ("QuotaExceeded", "quota"), ("InsufficientResourcesAvailable", "capacity"), + ("AKSCapacityHeavyUsage", "capacity")): + with self.subTest(code=code): + self.assertEqual(c.classify_failure({"error": {"code": code}})["kind"], kind) + + def test_policy_and_auth_failures_are_not_relocated(self): + for code in c.BLOCKING_CODES: + with self.subTest(code=code): + report = c.classify_failure({"error": {"code": "DeploymentFailed", "details": [{"code": "InsufficientQuota"}, {"code": code}]}}) + self.assertFalse(report["actionable"]) + + def test_generic_internal_error_and_free_text_are_unknown(self): + for value in ({"error": {"code": "InternalServerError"}}, "Quota exceeded, maybe", {"status": "Failed"}): + self.assertFalse(c.classify_failure(value)["actionable"]) + + def test_canonical_compute_operation_not_allowed_quota_error_is_recognized(self): + self.assertEqual(c.classify_failure({"error":{"code":"OperationNotAllowed","message":"Operation results in exceeding approved standardDSv6Family Cores quota."}})["kind"], "quota") + self.assertFalse(c.classify_failure({"error":{"code":"OperationNotAllowed","message":"The operation is not allowed."}})["actionable"]) + + def test_official_readiness_numeric_quota_evidence_is_accepted(self): + self.assertTrue(c.classify_failure({"id":"model-quota","status":"Fail","data":{"available":10,"required":450}})["actionable"]) + + def test_embedded_arm_status_message_is_parsed(self): + result = c.classify_failure([{"properties": {"statusMessage": '{"error":{"code":"InsufficientResourcesAvailable"}}'}}]) + self.assertEqual(result["kind"], "capacity") + + def test_numeric_deficit_is_required_for_readiness_quota_trigger(self): + row = {"kind": "quota", "status": "FAIL", "data": {"available": 100, "required": 450}} + self.assertTrue(c.classify_failure(row)["actionable"]) + for available in (450, 1000, "0", None, False): + row["data"]["available"] = available + self.assertFalse(c.classify_failure(row)["actionable"]) + + def test_read_only_allowlist_rejects_writes_and_tokens(self): + runner = Mock() + for arguments in (["group", "delete"], ["deployment", "group", "create"], ["account", "get-access-token"], + ["rest", "--method", "put", "--url", "https://management.azure.com/resource"], + ["rest", "--method", "post", "--url", "https://management.azure.com/resource"], + ["rest", "--method", "get", "--url", "https://attacker.example/resource"], + ["rest", "--method", "get", "--url", "https://management.azure.com@attacker.example/resource"]): + with self.subTest(arguments=arguments), self.assertRaises(RuntimeError): + c.read_query(runner, "test", arguments) + runner.az.assert_not_called() + + def test_only_offerings_post_is_allowed(self): + runner = Mock() + c.read_query(runner, "test", ["rest", "--method", "post", "--url", "https://management.azure.com/providers/Microsoft.Search/fetchOfferings?api-version=test"]) + runner.az.assert_called_once() + + def test_nearest_same_geography_not_paired_region_assumption(self): + rows = c.nearby_regions(Backend().locations, "eastus2") + self.assertEqual([r["region"] for r in rows], ["centralus", "northcentralus"]) + self.assertAlmostEqual(rows[0]["distanceKm"], 111.2, places=1) + + def test_cross_geography_requires_explicit_option(self): + rows = c.nearby_regions(Backend().locations, "eastus2", allow_cross_geography=True) + self.assertEqual(rows[0]["region"], "japaneast") + self.assertTrue(rows[0]["crossGeography"]) + + def test_explicit_candidate_allowlist_and_unknown_location(self): + self.assertEqual([r["region"] for r in c.nearby_regions(Backend().locations, "eastus2", ["northcentralus"])], ["northcentralus"]) + with self.assertRaises(ValueError): + c.nearby_regions(Backend().locations, "eastus2", ["unknownregion"]) + malformed = copy.deepcopy(Backend().locations) + malformed[0]["metadata"]["latitude"] = "NaN" + with self.assertRaises(ValueError): + c.nearby_regions(malformed, "eastus2") + + def test_model_unit_conversion_requires_matching_metadata(self): + data = Backend().az("catalog", ["cognitiveservices", "model", "list"]) + self.assertEqual(c.model_quota_units(data, "gpt-5.4"), "OpenAI.GlobalStandard.gpt-5.4") + data[0]["model"]["skus"][0]["rateLimits"][0]["unit"] = "Requests" + with self.assertRaises(ValueError): + c.model_quota_units(data, "gpt-5.4") + + def test_resume_counts_only_missing_model_allocations(self): + self.assertEqual(c.remaining_model_tpm(None, None), 450000) + self.assertEqual(c.remaining_model_tpm("Succeeded", None), 200000) + self.assertEqual(c.remaining_model_tpm(None, "Succeeded"), 250000) + self.assertEqual(c.remaining_model_tpm("Succeeded", "Succeeded"), 0) + + def probe(self, backend=None, compute=True, needed=450000): + with tempfile.TemporaryDirectory() as directory: + runner = workflow(directory) + backend = backend or Backend() + runner.az = backend.az + return c.RegionProbe(runner).probe("centralus", compute, needed), backend + + def test_verified_candidate_never_claims_capacity_reservation(self): + result, _ = self.probe() + self.assertTrue(result["eligibleForPreflight"]) + self.assertFalse(result["capacityReserved"]) + self.assertTrue(result["policyAndNetworkApprovalRequired"]) + + def test_search_capacity_failure_with_available_quota(self): + backend = Backend(); backend.offered["centralus"] = [] + result, _ = self.probe(backend) + self.assertFalse(result["eligibleForPreflight"]) + self.assertEqual(c.classify_failure(result)["kind"], "capacity") + self.assertTrue(any(x["name"] == "Search Standard service count" and x["status"] == "PASS" for x in result["checks"])) + + def test_model_quota_deficit_is_not_capacity(self): + backend = Backend(); backend.model_free = 200 + result, _ = self.probe(backend) + self.assertEqual(c.classify_failure(result)["kind"], "quota") + + def test_unknown_or_blocked_model_status_never_passes(self): + for status in ("Blocked", "InOverage", "Other"): + backend = Backend(); backend.model_status = status + result, _ = self.probe(backend) + self.assertFalse(result["eligibleForPreflight"]) + self.assertFalse(c.classify_failure(result)["actionable"]) + + def test_unknown_quota_units_never_pass(self): + backend = Backend(); backend.model_rate = 20 + result, _ = self.probe(backend) + self.assertFalse(result["eligibleForPreflight"]) + self.assertEqual(result["checks"][-1]["status"], "UNKNOWN") + + def test_denied_reads_are_unknown_not_zero_quota(self): + backend = Backend(); backend.fail = True + result, _ = self.probe(backend) + self.assertFalse(result["eligibleForPreflight"]) + self.assertFalse(c.classify_failure(result)["actionable"]) + + def test_workspace_probe_does_not_check_or_allocate_compute(self): + result, backend = self.probe(compute=False) + self.assertTrue(result["eligibleForPreflight"]) + self.assertFalse(any(args[0] == "vm" for _, args, _ in backend.calls)) + + def test_compute_reserves_system_and_workload_nodes(self): + result, backend = self.probe() + family = next(x for x in result["checks"] if x["name"] == "Compute standarddsv6family") + self.assertEqual(family["data"]["required"], 24) + self.assertEqual(sum(args[:2]==["vm","list-skus"] for _,args,_ in backend.calls),1) + + def test_native_numeric_strings_are_normalized_but_unknown_is_not_unlimited(self): + self.assertEqual(c.quota_values({"limit":"100","currentValue":"8"}),(100,8)) + for value in ("unknown","-1","NaN","Infinity",True): + with self.assertRaises(ValueError):c.quota_values({"limit":value,"currentValue":"0"}) + + def test_separate_compute_families_are_both_checked(self): + with tempfile.TemporaryDirectory() as directory: + runner = workflow(directory); runner.values["nodePoolVmSize"] = "Standard_E8s_v5"; runner.az = Backend().az + result = c.RegionProbe(runner).probe("centralus") + needs = {r["name"]: r["data"]["required"] for r in result["checks"] if r["name"].startswith("Compute ")} + self.assertEqual(needs, {"Compute standarddsv6family": 12, "Compute standardesv5family": 24, "Compute cores": 36}) + + def test_restricted_vm_and_insufficient_compute_quota_fail(self): + backend = Backend(); backend.restricted = True; backend.compute_limit = 10 + result, _ = self.probe(backend) + self.assertFalse(result["eligibleForPreflight"]) + self.assertEqual(c.classify_failure(result)["kind"], "quota") + + +if __name__ == "__main__": + unittest.main() diff --git a/utilities/regional-deployment/tests/test_plans.py b/utilities/regional-deployment/tests/test_plans.py new file mode 100644 index 00000000..dc625bf5 --- /dev/null +++ b/utilities/regional-deployment/tests/test_plans.py @@ -0,0 +1,202 @@ +import argparse +import copy +from datetime import datetime, timedelta, timezone +import json +from pathlib import Path +import tempfile +import unittest +from unittest.mock import Mock, patch + +from support import Backend, SUB, TENANT, workflow +from deploy import write_json +from inputs import full_inputs, save_bundle, workspace_inputs +import regional_deploy as cli + + +class PlanTests(unittest.TestCase): + def setup_runner(self, directory, retained=False): + runner = workflow(directory) + runner.context = Mock() + backend = Backend() + backend.offered["eastus2"] = [] + if retained: + backend.source_compute = {"properties": {"provisioningState": "Succeeded"}, "tags": {"discovery.overridemrgregion": "eastus2"}} + runner.az = backend.az + return runner, backend + + def recommendations(self, runner, **kwargs): + args = dict(failure_file=None, scope="auto", candidates=["centralus", "northcentralus"], allow_cross_geography=False, limit=3) + args.update(kwargs) + return cli.recommendation(runner, argparse.Namespace(**args)) + + def selection(self, directory, runner, report, **kwargs): + report_path = Path(directory)/"recommendation.json" + write_json(report_path, report) + args = dict(recommendation=str(report_path), accept_cross_region=True, region="centralus", mode=report["scope"], + name_prefix="fallback", target_resource_group="new-rg" if report["scope"] == "fresh" else None, + source_workspace=None, vnet_cidr="10.60.0.0/16", output_dir=str(Path(directory)/"selected")) + args.update(kwargs) + return cli.select_inputs(runner, argparse.Namespace(**args)) + + def test_init_is_local_and_never_overwrites(self): + with tempfile.TemporaryDirectory() as directory, patch("subprocess.run", side_effect=AssertionError("No Azure commands allowed")): + args = ["init", "--subscription", SUB, "--tenant", TENANT, "--resource-group", "example-rg", "--name-prefix", "example", + "--workload-region", "eastus2", "--output-dir", str(Path(directory)/"source")] + self.assertEqual(cli.main(args), 0) + self.assertEqual(cli.main(args), 1) + + def test_fresh_and_reuse_recommendations_are_scoped_from_actual_compute_state(self): + for retained, expected in ((False, "fresh"), (True, "workspace")): + with self.subTest(retained=retained), tempfile.TemporaryDirectory() as directory: + runner, backend = self.setup_runner(directory, retained) + report = self.recommendations(runner) + self.assertEqual(report["scope"], expected) + self.assertTrue(report["diagnosis"]["actionable"]) + self.assertEqual([r["region"] for r in report["candidates"]], ["centralus", "northcentralus"]) + self.assertTrue(all(r["eligibleForPreflight"] for r in report["candidates"])) + self.assertFalse(any(args[:1] == ["vm"] for _, args, _ in backend.calls) if retained else False) + + def test_no_observed_blocker_does_not_automatically_relocate(self): + with tempfile.TemporaryDirectory() as directory: + runner, backend = self.setup_runner(directory); backend.offered["eastus2"] = ["standard"] + report = self.recommendations(runner) + self.assertFalse(report["diagnosis"]["actionable"]) + self.assertEqual(report["candidates"], []) + + def test_explicit_allocation_failure_can_override_green_catalog_not_access_or_policy(self): + with tempfile.TemporaryDirectory() as directory: + runner, backend = self.setup_runner(directory); backend.offered["eastus2"] = ["standard"] + error = Path(directory)/"failure.json"; write_json(error, {"error": {"code": "InsufficientResourcesAvailable"}}) + self.assertTrue(self.recommendations(runner, failure_file=str(error))["candidates"]) + backend.calls.clear(); write_json(error, {"error": {"code": "RequestDisallowedByPolicy"}}) + self.assertFalse(self.recommendations(runner, failure_file=str(error))["diagnosis"]["actionable"]) + self.assertEqual(backend.calls, []) + + def test_compute_quota_is_not_misrepresented_as_workspace_only_fallback(self): + with tempfile.TemporaryDirectory() as directory: + runner, _ = self.setup_runner(directory, True) + error=Path(directory)/"failure.json";write_json(error,{"error":{"code":"VmSkuQuota"}}) + with self.assertRaises(ValueError): + self.recommendations(runner,failure_file=str(error)) + + def test_cannot_silently_duplicate_succeeded_compute(self): + with tempfile.TemporaryDirectory() as directory: + runner, _ = self.setup_runner(directory, True) + with self.assertRaises(ValueError): + self.recommendations(runner, scope="fresh") + + def test_in_progress_source_compute_blocks_recommendation(self): + with tempfile.TemporaryDirectory() as directory: + runner, backend = self.setup_runner(directory) + backend.source_compute = {"properties":{"provisioningState":"Accepted"}} + with self.assertRaises(ValueError): + self.recommendations(runner) + + def test_healthy_workspace_does_not_generate_false_quota_fallback(self): + with tempfile.TemporaryDirectory() as directory: + runner, backend = self.setup_runner(directory, True) + backend.source_workspace = {"properties":{"provisioningState":"Succeeded"}} + backend.model_free = 0 + report = self.recommendations(runner) + self.assertFalse(report["diagnosis"]["actionable"]) + self.assertEqual(report["candidates"], []) + + def test_stale_missing_source_does_not_hide_another_healthy_compute_owner(self): + with tempfile.TemporaryDirectory() as directory: + runner, backend = self.setup_runner(directory, True) + backend.owners = [{"name":"active-workspace","state":"Succeeded","computeId":runner.rg_id+"/providers/Microsoft.Discovery/supercomputers/"+runner.values["supercomputerName"]}] + report = self.recommendations(runner) + self.assertFalse(report["diagnosis"]["actionable"]) + self.assertEqual(report["candidates"], []) + + def test_location_queries_use_explicit_subscription_scoped_rest_not_cli_default(self): + with tempfile.TemporaryDirectory() as directory: + runner, backend = self.setup_runner(directory) + self.recommendations(runner) + rows=[(args,kw) for label,args,kw in backend.calls if label=="subscription-locations"] + self.assertEqual(len(rows),1) + self.assertIn("https://management.azure.com"+runner.scope+"/locations?api-version=2022-12-01",rows[0][0]) + self.assertFalse(rows[0][1]["scoped"]) + + def test_selected_fresh_stack_uses_an_explicit_different_group(self): + with tempfile.TemporaryDirectory() as directory: + runner, _ = self.setup_runner(directory) + report = self.recommendations(runner) + path = self.selection(directory, runner, report) + config = json.loads(path.read_text()) + self.assertEqual(config["resourceGroup"], "new-rg") + self.assertEqual(config["location"], "eastus") + self.assertEqual(config["dataPlaneLocation"], "centralus") + self.assertEqual(config["parametersFile"], "parameters.json") + self.assertTrue(config["regionalSelection"]["crossRegionApproved"]) + + def test_fresh_plan_rejects_source_group_and_absent_group_approval(self): + for target in (None, "source-rg"): + with self.subTest(target=target), tempfile.TemporaryDirectory() as directory: + runner, _ = self.setup_runner(directory); report = self.recommendations(runner) + with self.assertRaises(ValueError): + self.selection(directory, runner, report, target_resource_group=target) + + def test_workspace_plan_keeps_compute_storage_and_original_network_names(self): + with tempfile.TemporaryDirectory() as directory: + runner, _ = self.setup_runner(directory, True) + path = self.selection(directory, runner, self.recommendations(runner)) + cfg = json.loads(path.read_text()) + values = {k:v["value"] for k,v in json.loads((path.parent/"parameters.json").read_text())["parameters"].items()} + self.assertEqual(cfg["deploymentMode"], "workspace") + self.assertEqual(cfg["supercomputerDataPlaneLocation"], "eastus2") + self.assertEqual(cfg["storageDataPlaneLocation"], "eastus2") + self.assertEqual(cfg["dataPlaneLocation"], "centralus") + self.assertEqual(cfg["retainedBlobPrivateIp"], "10.20.4.9") + for key in ("supercomputerName", "nodePoolName", "storageContainerName", "storageAccountName"): + self.assertEqual(values[key], runner.values[key]) + self.assertEqual(values["existingVnetName"], runner.values["vnetName"]) + self.assertNotEqual(values["vnetName"], runner.values["vnetName"]) + self.assertNotEqual(values["managedIdentityName"], runner.values["managedIdentityName"]) + + def test_healthy_or_in_progress_workspace_is_never_detached_by_planning(self): + for state in ("Succeeded", "Accepted", "Running", "Creating", "Canceled"): + with self.subTest(state=state), tempfile.TemporaryDirectory() as directory: + runner, backend = self.setup_runner(directory, True); report = self.recommendations(runner) + backend.source_workspace = {"properties": {"provisioningState": state}} + with self.assertRaises(ValueError): + self.selection(directory, runner, report) + + def test_selection_requires_explicit_cross_region_acceptance(self): + with tempfile.TemporaryDirectory() as directory: + runner, _ = self.setup_runner(directory); report = self.recommendations(runner) + with self.assertRaises(ValueError): + self.selection(directory, runner, report, accept_cross_region=False) + + def test_stale_changed_future_or_failed_evidence_cannot_authorize_a_plan(self): + with tempfile.TemporaryDirectory() as directory: + runner, _ = self.setup_runner(directory); report = self.recommendations(runner) + for field in ("sourceConfigHash", "sourceParametersHash", "codeHash", "scope"): + changed = copy.deepcopy(report); changed[field] = "changed" + with self.subTest(field=field), self.assertRaises(ValueError): + cli.assert_recommendation(changed, runner, "centralus", "fresh") + now = datetime.now(timezone.utc) + for delta in (-1, 3601): + changed = copy.deepcopy(report); changed["time"] = (now-timedelta(seconds=delta)).isoformat() + with self.assertRaises(ValueError): + cli.assert_recommendation(changed, runner, "centralus", "fresh", now) + changed = copy.deepcopy(report); changed["candidates"][0]["checks"][0]["status"] = "UNKNOWN" + with self.assertRaises(ValueError): + cli.assert_recommendation(changed, runner, "centralus", "fresh") + + def test_compute_state_change_requires_a_new_preservation_plan(self): + with tempfile.TemporaryDirectory() as directory: + runner, backend = self.setup_runner(directory); report = self.recommendations(runner) + backend.source_compute = {"properties": {"provisioningState": "Succeeded"}} + with self.assertRaises(ValueError): + self.selection(directory, runner, report) + + def test_overlap_and_invalid_networks_are_rejected(self): + cfg, values = full_inputs(SUB, TENANT, "source-rg", "source", "eastus", "eastus2", "10.20.0.0/16") + for cidr in ("10.20.0.0/16", "8.8.0.0/16", "10.60.0.0/24", "127.0.0.0/16"): + with self.subTest(cidr=cidr), self.assertRaises(ValueError): + workspace_inputs(cfg, values, "fallback", "centralus", cidr, "10.20.4.9") + + +if __name__ == "__main__": + unittest.main() diff --git a/utilities/regional-deployment/tests/test_safety.py b/utilities/regional-deployment/tests/test_safety.py new file mode 100644 index 00000000..0b26df18 --- /dev/null +++ b/utilities/regional-deployment/tests/test_safety.py @@ -0,0 +1,153 @@ +from datetime import datetime, timezone +import json +from pathlib import Path +import tempfile +import unittest +from unittest.mock import Mock + +from support import ROOT, SUB, TENANT, workflow +import deploy as d +from inputs import full_inputs, workspace_inputs +import workspace as w + + +class SafetyTests(unittest.TestCase): + def test_preview_deletes_unknown_changes_and_foundation_modifies_are_blocked(self): + for kind in ("Delete", "Unsupported", "Unexpected", "Deploy"): + with self.subTest(kind=kind), self.assertRaises(RuntimeError): + d.inspect_changes({"status":"Succeeded", "changes":[{"changeType":kind,"resourceId":"/resource"}]},False) + with self.assertRaises(RuntimeError): + d.inspect_changes({"status":"Failed","changes":[]},False) + + def test_resume_cannot_touch_foundation(self): + result = {"status":"Succeeded", "changes":[{"changeType":"Create","resourceId":"/providers/Microsoft.Network/virtualNetworks/new"}]} + with self.assertRaises(RuntimeError): + d.inspect_changes(result, True) + + def test_workspace_writes_are_exact_id_allowlisted(self): + cfg, values = full_inputs(SUB, TENANT, "source-rg", "source", "eastus", "eastus2", "10.20.0.0/16") + _, p = workspace_inputs(cfg, values, "fallback", "centralus", "10.60.0.0/16", "10.20.4.9") + group = f"/subscriptions/{SUB}/resourceGroups/source-rg" + allowed = w.workspace_write_ids(p, group) + self.assertEqual(len(allowed), 14) + self.assertNotIn((group+"/providers/Microsoft.Network/virtualNetworks/"+p["existingVnetName"]).lower(), allowed) + self.assertEqual(len(w.workspace_write_ids(p, group, True)), 3) + for resource in (group+"/providers/Microsoft.Discovery/supercomputers/"+p["supercomputerName"], + group+"/providers/Microsoft.Storage/storageAccounts/"+p["storageAccountName"], + group+"/providers/Microsoft.Network/privateDnsZones/privatelink.blob.core.windows.net/A/storage"): + preview = {"status":"Succeeded","changes":[{"changeType":"Create","resourceId":resource}]} + with self.subTest(resource=resource), self.assertRaises(RuntimeError): + w.inspect_workspace_preview(preview,p,group) + + def test_workspace_initial_modify_is_blocked_even_with_approval_flag(self): + cfg, values = full_inputs(SUB, TENANT, "source-rg", "source", "eastus", "eastus2", "10.20.0.0/16") + _, p = workspace_inputs(cfg, values, "fallback", "centralus", "10.60.0.0/16", "10.20.4.9") + group = f"/subscriptions/{SUB}/resourceGroups/source-rg" + target = next(iter(w.workspace_write_ids(p, group))) + with self.assertRaises(RuntimeError): + w.inspect_workspace_preview({"status":"Succeeded","changes":[{"changeType":"Modify","resourceId":target}]},p,group,False,True) + + def test_compiled_core_and_resume_shapes_cannot_gain_other_writes(self): + d.assert_resume_template({"resources":[{"type":"Microsoft.Discovery/"+x} for x in d.DISCOVERY_TYPES]}) + w.assert_workspace_template({"resources":[{"type":x} for x in w.CORE_TYPES]},True) + with self.assertRaises(RuntimeError): + w.assert_workspace_template({"resources":[{"type":x} for x in w.CORE_TYPES]+[{"type":"Microsoft.Discovery/supercomputers"}]},True) + + def test_failed_source_put_preserves_identity_network_tags_and_other_links(self): + source = {"location":"eastus", "tags":{"version":"v2","discovery.overridemrgregion":"eastus2"}, + "properties":{"provisioningState":"Failed","workspaceIdentity":{"id":"/identity","clientId":"readonly"}, + "supercomputerIds":["/compute/a","/compute/b"],"agentSubnetId":"/agent","workspaceSubnetId":"/workspace", + "privateEndpointSubnetId":"/pe","workspaceApiUri":"https://readonly.example"}} + payload = w.failed_workspace_detach_put_payload(source,"/compute/a") + self.assertEqual(payload["tags"],source["tags"]) + self.assertEqual(payload["location"],source["location"]) + self.assertEqual(payload["properties"]["workspaceIdentity"],{"id":"/identity"}) + self.assertEqual(payload["properties"]["supercomputerIds"],["/compute/b"]) + for key in ("workspaceApiUri","provisioningState"): + self.assertNotIn(key,payload["properties"]) + source["properties"]["provisioningState"]="Succeeded" + with self.assertRaises(RuntimeError): + w.failed_workspace_detach_put_payload(source,"/compute/a") + + def test_new_preflight_invalidates_old_success_even_before_failed_reads(self): + with tempfile.TemporaryDirectory() as directory: + runner=workflow(directory) + d.write_json(runner.state_root/"latest-preflight.json",{"ok":True}) + runner.begin_preflight() + report=json.loads((runner.state_root/"latest-preflight.json").read_text()) + self.assertFalse(report["ok"]) + with self.assertRaises(RuntimeError): + d.assert_fresh_preflight(report,d.digest(runner.config_path),d.digest(runner.parameters_path)) + + def test_deploy_requires_apply_and_postconfigure_requires_succeeded(self): + with tempfile.TemporaryDirectory() as directory: + runner=workflow(directory,"deploy"); runner.context=Mock(); runner.az=Mock(side_effect=AssertionError("No writes")) + with self.assertRaises(RuntimeError):runner.deploy() + runner.az.assert_not_called() + with self.assertRaises(RuntimeError):runner.postconfigure() + runner.az.assert_not_called() + + def test_nonpublic_cloud_is_rejected_before_preparation(self): + with tempfile.TemporaryDirectory() as directory: + runner=workflow(directory) + runner.az=Mock(return_value={"id":SUB,"tenantId":TENANT,"environmentName":"AzureUSGovernment"}) + with self.assertRaises(RuntimeError):runner.context() + self.assertEqual(runner.az.call_count,1) + + def test_failed_validation_invalidates_a_previous_receipt(self): + with tempfile.TemporaryDirectory() as directory: + runner=workflow(directory) + d.write_json(runner.state_root/"validated-deployment.json",{"valid":True}) + with self.assertRaises((RuntimeError,OSError)):runner.validate() + self.assertFalse(json.loads((runner.state_root/"validated-deployment.json").read_text())["valid"]) + + def test_receipts_bind_compiled_parameters_config_code_and_stage(self): + with tempfile.TemporaryDirectory() as directory: + runner=workflow(directory,"deploy",apply=True); runner.context=Mock() + template=ROOT/"templates/main.bicep"; compiled=Path(directory)/"compiled.json"; params=Path(directory)/"filtered.json" + d.write_json(compiled,{});d.write_json(params,{}) + now=datetime.now(timezone.utc).isoformat() + report={"ok":True,"time":now,"configHash":d.digest(runner.config_path),"parametersHash":d.digest(runner.parameters_path),"codeHash":runner.code_hash()} + d.write_json(runner.state_root/"latest-preflight.json",report) + receipt=dict(report,valid=True,stage="full",template=str(template),templateHash=d.digest(template),compiled=str(compiled),compiledHash=d.digest(compiled), + filteredParameters=str(params),filteredParametersHash=d.digest(params)) + d.write_json(runner.state_root/"validated-deployment.json",receipt) + runner.az=Mock(return_value={"properties":{"provisioningState":"Running"}}) + with self.assertRaisesRegex(RuntimeError,"already running"):runner.deploy() + self.assertEqual(runner.az.call_count,1) + runner.az.reset_mock();compiled.write_text('{"changed":true}') + with self.assertRaisesRegex(RuntimeError,"Validated input changed"):runner.deploy() + runner.az.assert_not_called() + + def test_scope_boundaries_and_conditional_roles_are_not_trusted(self): + self.assertFalse(d.scope_covers("/subscriptions/s/resourceGroups/r","/subscriptions/s/resourceGroups/rogue")) + self.assertFalse(d.has_role([{"roleDefinitionId":"/roles/test","scope":"/","condition":"restricted"}],"test","/target")) + + def test_nsp_preparation_does_not_reuse_privileged_or_restricted_custom_roles(self): + role={"permissions":[{"actions":sorted(d.NSP_ACTIONS)}]} + self.assertTrue(d.is_minimal_nsp_role(role)) + role["permissions"][0]["actions"].append("Microsoft.Authorization/roleAssignments/write") + self.assertFalse(d.is_minimal_nsp_role(role)) + role["permissions"][0]["actions"]=sorted(d.NSP_ACTIONS) + role["permissions"][0]["notActions"]=[next(iter(d.NSP_ACTIONS))] + self.assertFalse(d.is_minimal_nsp_role(role)) + + def test_unsupported_preview_is_blocked_even_with_modify_approval(self): + with self.assertRaises(RuntimeError): + d.inspect_changes({"status":"Succeeded","changes":[{"changeType":"Unsupported","resourceId":"/resource"}]},False,True) + + def test_authenticated_api_rejects_unexpected_hosts_and_project_paths(self): + for uri in ("https://attacker.example","http://test.workspace.discovery.azure.com","https://test.workspace.discovery.azure.com@attacker.example"): + with self.assertRaises(RuntimeError): + d.project_agents_url({"properties":{"workspaceApiUri":uri}},"test","project","2026-06-01") + + def test_workspace_template_never_adds_a_second_customer_blob_endpoint_or_skip_tags(self): + text=(ROOT/"templates/workspace.bicep").read_text() + self.assertNotIn("Microsoft.Network/privateEndpoints@",text) + self.assertNotIn("SkipAssociate",text) + for name in ("storageAccount","retainedVnet","blobDns","supercomputer","storageContainer"): + self.assertRegex(text,r"resource "+name+r" .* existing =") + + +if __name__ == "__main__": + unittest.main() diff --git a/utilities/regional-deployment/workspace.py b/utilities/regional-deployment/workspace.py new file mode 100644 index 00000000..bb82e434 --- /dev/null +++ b/utilities/regional-deployment/workspace.py @@ -0,0 +1,333 @@ +#!/usr/bin/env python3 +"""Workspace-only regional recovery, preserving existing Discovery compute/storage.""" + +import argparse +from collections import Counter +from datetime import datetime, timezone +import ipaddress +import json +import sys +import time +import uuid + +from deploy import (DISCOVERY_TYPES, NSP_ACTIONS, PERSONA_ROLES, + REQUIRED_PROVIDERS, ROOT, ROLES, Workflow, digest, + filter_parameters, has_role, inspect_changes, + resolve_managed_group, write_json, is_minimal_nsp_role) + + +CORE_TYPES = ( + "microsoft.discovery/workspaces", "microsoft.discovery/workspaces/chatmodeldeployments", + "microsoft.discovery/workspaces/projects", +) + + +def failed_workspace_detach_payload(workspace, supercomputer_id): + if workspace.get("properties", {}).get("provisioningState") != "Failed": + raise RuntimeError("Only a confirmed Failed source workspace may release its compute link.") + links = workspace["properties"].get("supercomputerIds", []) + if not isinstance(links, list) or not all(isinstance(x, str) for x in links): + raise RuntimeError("Source workspace compute links are not a valid ID list.") + if not any(x.lower() == supercomputer_id.lower() for x in links): + raise RuntimeError("Source workspace does not own the exact retained compute link; no PATCH permitted.") + return {"properties": {"supercomputerIds": [x for x in links if x.lower() != supercomputer_id.lower()]}} + + +def failed_workspace_detach_put_payload(workspace, supercomputer_id): + patch = failed_workspace_detach_payload(workspace, supercomputer_id) + old = workspace["properties"] + props = {"workspaceIdentity": {"id": old["workspaceIdentity"]["id"]}, + "supercomputerIds": patch["properties"]["supercomputerIds"]} + for key in ["agentSubnetId", "privateEndpointSubnetId", "workspaceSubnetId", "publicNetworkAccess", + "customerManagedKeys", "keyVaultProperties", "logAnalyticsClusterId"]: + if key in old: + props[key] = old[key] + return {"location": workspace["location"], "tags": workspace.get("tags", {}), "properties": props} + + +def assert_workspace_template(compiled, core_only=False): + expected = Counter(CORE_TYPES) + if not core_only: + expected.update({"microsoft.network/publicipaddresses": 1, + "microsoft.network/natgateways": 1, + "microsoft.network/virtualnetworks": 1, + "microsoft.managedidentity/userassignedidentities": 1, + "microsoft.authorization/roleassignments": 4, + "microsoft.network/virtualnetworks/virtualnetworkpeerings": 2, + "microsoft.network/privatednszones/virtualnetworklinks": 1}) + actual = Counter(r.get("type", "").lower() for r in compiled.get("resources", [])) + if actual != expected: + raise RuntimeError("Workspace template has unexpected writes; compute, storage, DNS records and existing foundation must be references only.") + + +def workspace_write_ids(values, group, core_only=False): + p = values + ws = group + "/providers/Microsoft.Discovery/workspaces/" + p["workspaceName"] + result = {ws, ws + "/chatModelDeployments/" + p["chatModelDeploymentName"], + ws + "/projects/" + p["projectName"]} + if core_only: + return {x.lower() for x in result} + vnet = group + "/providers/Microsoft.Network/virtualNetworks/" + p["vnetName"] + retained = group + "/providers/Microsoft.Network/virtualNetworks/" + p["existingVnetName"] + storage = group + "/providers/Microsoft.Storage/storageAccounts/" + p["storageAccountName"] + result.update({vnet, + group + "/providers/Microsoft.Network/natGateways/" + p["natGatewayName"], + group + "/providers/Microsoft.Network/publicIPAddresses/" + p["natGatewayPublicIpName"], + group + "/providers/Microsoft.ManagedIdentity/userAssignedIdentities/" + p["managedIdentityName"], + vnet + "/virtualNetworkPeerings/peer-to-retained-discovery", + retained + "/virtualNetworkPeerings/peer-to-" + p["vnetName"], + group + "/providers/Microsoft.Network/privateDnsZones/privatelink.blob.core.windows.net/virtualNetworkLinks/link-" + p["vnetName"]}) + for key, scope in [("platformRoleAssignmentName", group), ("acrPullRoleAssignmentName", group), + ("blobRoleAssignmentName", storage), ("networkRoleAssignmentName", vnet)]: + uuid.UUID(p[key]) + result.add(scope + "/providers/Microsoft.Authorization/roleAssignments/" + p[key]) + return {x.lower() for x in result} + + +def inspect_workspace_preview(result, values, group, core_only=False, allow_modify=False): + counts = inspect_changes(result, False, allow_modify and core_only) + allowed = workspace_write_ids(values, group, core_only) + for change in result.get("changes", []): + if change["changeType"] not in ("Ignore", "NoChange") and change["resourceId"].lower() not in allowed: + raise RuntimeError("Workspace preview touches an unapproved resource: " + change["resourceId"]) + return counts + + +class WorkspaceWorkflow(Workflow): + @property + def core_only(self): + return self.args.stage == "workspace-core" + + def preflight(self): + self.begin_preflight() + self.context() + self.check("Discovery control plane", self.location in ("eastus", "uksouth", "swedencentral"), self.location) + self.check("Distinct workspace data plane", self.data_location != self.config["supercomputerDataPlaneLocation"], self.data_location) + provider = self.az("discovery-provider", ["provider", "show", "--namespace", "Microsoft.Discovery"]) + types = {x["resourceType"].lower(): x for x in provider["resourceTypes"]} + for name in DISCOVERY_TYPES: + self.check("Discovery API " + name, self.config["apiVersion"] in types.get(name, {}).get("apiVersions", []), self.config["apiVersion"]) + providers = self.az("providers", ["provider", "list"]) + registered = {x["namespace"].lower() for x in providers if x["registrationState"] == "Registered"} + self.check("Required providers", all("microsoft." + n.lower() in registered for n in REQUIRED_PROVIDERS), "registration article/quickstart union") + user, cp = self.principals() + user_roles = self.assignments("user-roles", user, self.rg_id) + for role in PERSONA_ROLES: + self.check("User " + role, has_role(user_roles, ROLES[role], self.rg_id), "existing scoped or inherited assignment") + cp_roles = self.assignments("control-plane-roles", cp, self.scope) + self.check("Control-plane Reader", has_role(cp_roles, ROLES["Reader"], self.scope), self.scope) + definitions = self.az("custom-role-definitions", ["role", "definition", "list", "--custom-role-only", "true", "--scope", self.scope]) + self.check("Control-plane NSP", any(has_role(cp_roles, d["name"], self.scope) and is_minimal_nsp_role(d) + for d in definitions), "existing minimal first-party perimeter permissions") + self.check_retained_resources() + self.check_capacity(include_compute=False) + report = {"time": datetime.now(timezone.utc).isoformat(), "ok": not any(c["state"] == "FAIL" for c in self.checks), + "checks": self.checks, "configHash": digest(self.config_path), + "parametersHash": digest(self.parameters_path), "codeHash": self.code_hash(), "evidence": str(self.out)} + write_json(self.out / "preflight.json", report) + write_json(self.state_root / "latest-preflight.json", report) + if not report["ok"]: + raise RuntimeError("Workspace preflight failed; no deployment submitted.") + + def check_retained_resources(self): + p = self.values + for index in (0, 1, 4): + path = self.resource_paths()[index] + obj = self.rest("retained-discovery-" + str(index), f"{self.rg_id}/providers/Microsoft.Discovery/{path}?api-version={self.config['apiVersion']}") + props = obj.get("properties", {}) + self.check("Retained " + path, props.get("provisioningState") == "Succeeded" and obj.get("location") == self.location, + str(props.get("provisioningState")) + "; existing resource, never PUT") + if index == 0: + self.check("Retained compute region", obj.get("tags", {}).get("discovery.overridemrgregion") == self.config["supercomputerDataPlaneLocation"], + "not compared to the workspace data-plane region") + if index == 4: + self.check("Retained output store", props.get("storageStore", {}).get("storageAccountId", "").lower() == + (self.rg_id + "/providers/Microsoft.Storage/storageAccounts/" + p["storageAccountName"]).lower(), "unchanged backing account") + old_vnet = self.az("retained-vnet", ["network", "vnet", "show", "--resource-group", self.rg, "--name", p["existingVnetName"]]) + planned = ipaddress.ip_network(p["vnetAddressPrefix"]) + self.check("Non-overlapping workspace VNet", not any(planned.overlaps(ipaddress.ip_network(c)) for c in old_vnet["addressSpace"]["addressPrefixes"]), + str(planned) + "; global peering to existing VNet") + storage = self.az("retained-storage", ["storage", "account", "show", "--resource-group", self.rg, "--name", p["storageAccountName"]]) + self.check("Storage remains private", storage.get("publicNetworkAccess") == "Disabled" and storage["location"] == self.config["storageDataPlaneLocation"], "no storage PUT or public-access change") + dns = self.az("retained-blob-dns", ["network", "private-dns", "record-set", "a", "show", "--resource-group", self.rg, + "--zone-name", "privatelink.blob.core.windows.net", "--name", p["storageAccountName"]]) + self.check("Blob DNS record retained", {a["ipv4Address"] for a in dns.get("aRecords", [])} == {self.config["retainedBlobPrivateIp"]}, "reuse existing endpoint; never overwrite A record") + pe = self.az("retained-blob-pe", ["network", "private-endpoint", "show", "--resource-group", self.rg, "--name", "pe-" + p["storageAccountName"] + "-blob"]) + self.check("Retained Blob endpoint", pe.get("provisioningState") == "Succeeded" and any( + c.get("privateLinkServiceConnectionState", {}).get("status") == "Approved" for c in pe.get("privateLinkServiceConnections", [])), "Approved endpoint reused through peering") + new_vnet = self.az("workspace-vnet", ["network", "vnet", "show", "--resource-group", self.rg, "--name", p["vnetName"]], optional=True, not_found_only=True) + identity = self.az("workspace-identity", ["identity", "show", "--resource-group", self.rg, "--name", p["managedIdentityName"]], optional=True, not_found_only=True) + for name, obj in [("workspace VNet", new_vnet), ("workspace UAMI", identity)]: + self.check(name, bool(obj and obj["location"] == self.data_location) if self.core_only else obj is None or obj["location"] == self.data_location, + "existing in target region" if obj else "planned new; required before workspace-core") + if self.core_only and new_vnet: + subnets = {item["name"]: item for item in new_vnet.get("subnets", [])} + for name, prefix in (("workspaceSubnet", "workspaceSubnetPrefix"), ("agentSubnet", "agentSubnetPrefix"), + ("privateEndpointSubnet", "privateEndpointSubnetPrefix")): + self.check("Workspace subnet "+name, subnets.get(name, {}).get("addressPrefix") == p[prefix], "exact reviewed subnet prefix") + if name != "privateEndpointSubnet": + self.check("Workspace delegation "+name, any(x.get("serviceName") == "Microsoft.App/environments" for x in subnets.get(name, {}).get("delegations", [])), "Container Apps delegation") + for i, (vnet_name, peer_name) in enumerate(((p["vnetName"], "peer-to-retained-discovery"), (p["existingVnetName"], "peer-to-"+p["vnetName"]))): + peer = self.az("core-peering-"+str(i), ["network", "vnet", "peering", "show", "--resource-group", self.rg, + "--vnet-name", vnet_name, "--name", peer_name]) + self.check("Core peering "+peer_name, peer.get("peeringState") == "Connected" and peer.get("provisioningState") == "Succeeded", "bidirectional private routing") + link = self.az("core-blob-dns-link", ["network", "private-dns", "link", "vnet", "show", "--resource-group", self.rg, + "--zone-name", "privatelink.blob.core.windows.net", "--name", "link-"+p["vnetName"]]) + self.check("Core Blob DNS link", link.get("virtualNetworkLinkState") == "Completed", "reuse original zone; do not replace its A record") + if identity: + roles = self.assignments("workspace-identity-roles", identity["principalId"]) + for role, scope in [("Microsoft Discovery Platform Contributor", self.rg_id), ("AcrPull", self.rg_id), + ("Storage Blob Data Contributor", storage["id"]), + ("Network Contributor", self.rg_id + "/providers/Microsoft.Network/virtualNetworks/" + p["vnetName"])]: + self.check("Workspace UAMI " + role, has_role(roles, ROLES[role], scope), scope) + target = self.az("target-workspace", ["rest", "--method", "get", "--url", + f"https://management.azure.com{self.rg_id}/providers/Microsoft.Discovery/workspaces/{p['workspaceName']}?api-version={self.config['apiVersion']}"], + scoped=False, optional=True, not_found_only=True) + self.check("Workspace immutable-region consistency", target is None or (target["location"] == self.location and + target.get("tags", {}).get("discovery.overridemrgregion") == self.data_location), "never change region/tags of an existing workspace") + source_name = self.config.get("sourceWorkspaceName") + if source_name and source_name != p["workspaceName"]: + source = self.az("source-workspace-link", ["rest", "--method", "get", "--url", + f"https://management.azure.com{self.rg_id}/providers/Microsoft.Discovery/workspaces/{source_name}?api-version={self.config['apiVersion']}"], + scoped=False, optional=True, not_found_only=True) + sc_id = (self.rg_id + "/providers/Microsoft.Discovery/supercomputers/" + p["supercomputerName"]).lower() + self.check("Retained compute ownership released", source is None or all( + x.lower() != sc_id for x in source.get("properties", {}).get("supercomputerIds", [])), + "old Failed workspace must not retain the exclusive association; use reviewed release_link, never delete compute") + owners = self.rest("compute-owners", f"{self.rg_id}/providers/Microsoft.Discovery/workspaces?api-version={self.config['apiVersion']}") + if not isinstance(owners.get("value"), list) or owners.get("nextLink"): + raise RuntimeError("Workspace ownership inventory is incomplete; resolve it before attaching retained compute.") + sc_id = (self.rg_id+"/providers/Microsoft.Discovery/supercomputers/"+p["supercomputerName"]).lower() + conflicting = [] + for obj in owners["value"]: + name = obj.get("name", "") + if not name or "/" in name: + raise RuntimeError("Workspace ownership metadata has an invalid name.") + full = self.rest("owner-"+name, f"{self.rg_id}/providers/Microsoft.Discovery/workspaces/{name}?api-version={self.config['apiVersion']}") + links = full.get("properties", {}).get("supercomputerIds") + if not isinstance(links, list) or not all(isinstance(value, str) for value in links): + raise RuntimeError("Workspace compute ownership is unverified: " + name) + if name.lower() != p["workspaceName"].lower() and sc_id in {value.lower() for value in links}: + conflicting.append(name) + self.check("No conflicting compute owner", not conflicting, ", ".join(conflicting) or "no other workspace in the customer RG owns the retained compute") + + def build(self): + self.template = ROOT / "templates" / ("workspace-core.bicep" if self.core_only else "workspace.bicep") + self.compiled_path = self.out / "template.json" + self.az("bicep-build", ["bicep", "build", "--file", str(self.template), "--outfile", str(self.compiled_path)], scoped=False, output_json=False) + compiled = json.loads(self.compiled_path.read_text()) + assert_workspace_template(compiled, self.core_only) + self.filtered_path = self.out / "parameters.json" + write_json(self.filtered_path, filter_parameters(self.parameters, compiled)) + write_json(self.out / "workspace-source-manifest.json", {"scriptHash": digest(__file__), "allowedWriteIds": sorted(workspace_write_ids(self.values, self.rg_id, self.core_only))}) + + def inspect_preview(self, result): + return inspect_workspace_preview(result, self.values, self.rg_id, self.core_only, self.args.allow_modify) + + def audit(self): + self.verify() + p = self.values + workspace = json.loads((self.out / "resource-2.json").read_text()) + model = json.loads((self.out / "resource-3.json").read_text()) + project = json.loads((self.out / "resource-5.json").read_text()) + sc_id = self.rg_id + "/providers/Microsoft.Discovery/supercomputers/" + p["supercomputerName"] + storage_id = self.rg_id + "/providers/Microsoft.Discovery/storageContainers/" + p["storageContainerName"] + self.check("Workspace compute relationship", {x.lower() for x in workspace["properties"].get("supercomputerIds", [])} == {sc_id.lower()}, "exact original Supercomputer") + self.check("Project storage relationship", storage_id.lower() in {x.lower() for x in project["properties"].get("storageContainerIds", [])}, "original Discovery StorageContainer") + self.check("Workspace managed-region override", workspace.get("tags", {}).get("discovery.overridemrgregion") == self.data_location, self.data_location) + self.check("Chat model configuration", model["properties"].get("modelName") == p["chatModelName"] and + model["properties"].get("modelFormat") == "OpenAI", "gpt-5-4 resolves to OpenAI GPT-5.4") + groups = self.az("resource-groups", ["group", "list"]) + mrg = resolve_managed_group(workspace, groups, self.scope) + inventory = self.az("managed-inventory", ["resource", "list", "--resource-group", mrg.split("/")[-1]]) + required = {"microsoft.search/searchservices", "microsoft.cognitiveservices/accounts", "microsoft.documentdb/databaseaccounts"} + found = set() + backing = [] + for i, obj in enumerate(inventory): + kind = obj["type"].lower() + if kind not in required: + continue + found.add(kind) + meta = self.az("backing-" + str(i), ["resource", "show", "--ids", obj["id"], "--query", + "{id:id,name:name,type:type,location:location,provisioningState:properties.provisioningState,status:properties.status,sku:sku}"]) + backing.append(meta) + ready = (meta.get("provisioningState") or "").lower() == "succeeded" + if kind == "microsoft.search/searchservices": + ready = ready and (meta.get("status") or "").lower() == "running" + self.check("Backing " + obj["name"], ready and meta["location"].replace(" ", "").lower() == self.data_location, + f"{meta.get('provisioningState')}; {meta['location']}") + self.check("Required backing services", required.issubset(found), "Search, Cosmos DB and Foundry") + pe = self.az("managed-private-endpoints", ["network", "private-endpoint", "list", "--resource-group", mrg.split("/")[-1]]) + approved = bool(pe) and all(x.get("provisioningState") == "Succeeded" and any( + c.get("privateLinkServiceConnectionState", {}).get("status") == "Approved" for c in x.get("privateLinkServiceConnections", [])) for x in pe) + self.check("Managed private endpoints", approved, str(len(pe)) + " endpoints; metadata approval only") + for i, (vnet, name) in enumerate([(p["vnetName"], "peer-to-retained-discovery"), + (p["existingVnetName"], "peer-to-" + p["vnetName"])]): + peer = self.az("peering-" + str(i), ["network", "vnet", "peering", "show", "--resource-group", self.rg, + "--vnet-name", vnet, "--name", name]) + self.check("Peering " + name, peer.get("peeringState") == "Connected" and peer.get("provisioningState") == "Succeeded", "Connected / Succeeded") + self.check_retained_resources() + user, _ = self.principals() + roles = self.assignments("managed-rg-user-roles", user, mrg) + self.check("Foundry User on managed group", has_role(roles, ROLES["Foundry User"], mrg), "scoped or inherited") + uri = workspace["properties"].get("workspaceApiUri", "").rstrip("/") + if uri.lower() != "https://" + p["workspaceName"].lower() + ".workspace.discovery.azure.com": + raise RuntimeError("Workspace API URI is not the expected HTTPS Discovery host; no authenticated request sent.") + agents = self.az("project-agents-read", ["rest", "--method", "get", "--resource", "https://discovery.azure.com", + "--url", uri + "/projects/" + p["projectName"] + "/agents?api-version=" + self.config["apiVersion"]], + scoped=False, optional=True) + reachable = isinstance(agents, (dict, list)) + self.check("Read-only authenticated project API", reachable, "GET agents only; no agent creation/inference", required=False) + report = {"ok": not any(x["state"] == "FAIL" for x in self.checks), "checks": self.checks, + "managedResourceGroupId": mrg, "backing": backing, "workspaceApiUri": uri, + "workspaceUiUri": workspace["properties"].get("workspaceUiUri"), + "authenticatedProjectReadSucceeded": reachable, "paidInferenceTested": False} + write_json(self.out / "audit.json", report) + if not report["ok"]: + raise RuntimeError("Workspace post-deployment audit has failures; inspect audit.json.") + print("Workspace relationship/backing/network/RBAC audit passed. No inference performed.") + + def release_link(self): + self.context() + if not self.args.apply: + raise RuntimeError("Releasing a Failed workspace association requires --apply.") + source = self.config["sourceWorkspaceName"] + if source.lower() == self.values["workspaceName"].lower(): + raise RuntimeError("Source and target workspaces must differ.") + source_path = f"{self.rg_id}/providers/Microsoft.Discovery/workspaces/{source}?api-version={self.config['apiVersion']}" + before = self.rest("source-workspace-before", source_path) + sc_path = f"{self.rg_id}/providers/Microsoft.Discovery/supercomputers/{self.values['supercomputerName']}" + sc = self.rest("supercomputer-before", sc_path + "?api-version=" + self.config["apiVersion"]) + if sc.get("properties", {}).get("provisioningState") != "Succeeded": + raise RuntimeError("Retained Supercomputer is not Succeeded; do not change its workspace association.") + method = self.args.release_method + payload = (failed_workspace_detach_put_payload(before, sc_path) if method == "put" + else failed_workspace_detach_payload(before, sc_path)) + payload_path = self.out / "detach-payload.json" + write_json(payload_path, payload) + self.az("source-workspace-detach-" + method, ["rest", "--method", method, "--url", + "https://management.azure.com" + source_path, "--body", "@" + str(payload_path)], scoped=False) + deadline = time.monotonic() + 300 + while True: + after = self.rest("source-workspace-readback", source_path) + links = after.get("properties", {}).get("supercomputerIds", []) + if not any(x.lower() == sc_path.lower() for x in links): + for key in ["location", "tags"]: + if before.get(key) != after.get(key): + raise RuntimeError("Source workspace immutable field changed unexpectedly: " + key) + for key in ["agentSubnetId", "privateEndpointSubnetId", "workspaceSubnetId"]: + if before["properties"].get(key) != after["properties"].get(key): + raise RuntimeError("Source workspace field changed unexpectedly: " + key) + if before["properties"]["workspaceIdentity"]["id"].lower() != after["properties"]["workspaceIdentity"]["id"].lower(): + raise RuntimeError("Source workspace identity changed unexpectedly.") + check = self.rest("supercomputer-readback", sc_path + "?api-version=" + self.config["apiVersion"]) + if check["properties"].get("provisioningState") != "Succeeded": + raise RuntimeError("Retained compute state changed; inspect evidence before proceeding.") + write_json(self.out / "link-release.json", {"sourceWorkspaceId": before["id"], + "supercomputerId": sc_path, "linkReleased": True, "computePreserved": True, + "remainingLinks": links, "sourceWorkspaceState": after["properties"].get("provisioningState")}) + print("Failed workspace compute association released; Supercomputer remains Succeeded.") + return + if time.monotonic() >= deadline: + raise RuntimeError("Association release not confirmed; do not replay the PATCH blindly.") + time.sleep(15)