[Mahalaxmibejugam]

Problem

Macrobenchmark builds intermittently fail in create-cluster with:

ERROR: (gcloud.container.clusters.create) ... code: GCE_STOCKOUT
Try a different location, or try again later: Google Compute Engine does not have enough resources available to fulfill request: <zone>.

The failure happens in gcloud container clusters create, before the workload node pool is created: GKE can't provision the control plane of the zonal cluster in _ZONE. These stockouts are intermittent and zone-specific, so the workload node pool isn't the constrained resource.

A regional cluster doesn't help: per the GKE docs, regional cluster creation fails if VMs can't be created in any one of its zones.

Fix

• create_cluster.sh: try the control plane in _ZONE, then the region's other zones; delete a failed cluster before the next try. The _MACHINE_TYPE pool stays pinned to _ZONE (--node-locations), so node placement and zonal-bucket co-location don't change. The chosen zone is saved as CLUSTER_LOCATION.
• seed_checkpoint.sh, run_workload.sh, cleanup_helm.sh, cleanup_cluster.sh: use ${CLUSTER_LOCATION:-${_ZONE}} (one line each).

No new substitutions; existing configurations need no changes.

Testing

• Local test with a stubbed gcloud, covering:
  • control plane created in _ZONE;
  • fallback to a second and to a third zone, with the node pool still in _ZONE;
  • every zone failing, which records create-cluster as failed;
  • zone listing failing, so only _ZONE is tried.
• Confirmed that a zonal cluster with its control plane in one zone accepts a node pool located only in another zone of the same region.
• End-to-end macrobenchmark run with the control plane forced into a zone different from _ZONE: all steps succeeded, the workload nodes ran in _ZONE, cleanup deleted the cluster, and benchmark metrics were in line with regular runs.

[gemini-code-assist]

Code Review

This pull request implements a zone fallback mechanism for GKE control plane creation to mitigate capacity errors (such as GCE_STOCKOUT) during macrobenchmarks. It introduces configuration variables for fallback zones, retry rounds, and delays, and updates GKE interaction scripts to dynamically use the determined cluster location while keeping the workload node pool pinned to the primary zone. The review feedback highlights two important improvements in create_cluster.sh: preventing premature script termination under pipefail by executing the gcloud pipeline directly within the if condition, and ensuring the temporary log file is cleaned up when exiting early on non-retryable errors.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.17%. Comparing base (b45114a) to head (9903073).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1132   +/-   ##
=======================================
  Coverage   92.17%   92.17%           
=======================================
  Files          20       20           
  Lines        4011     4011           
=======================================
  Hits         3697     3697           
  Misses        314      314           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[zhixiangli]

Thanks! I verified the approach on real GKE: with the control plane in us-central1-c and the pool created with --node-locations=us-central1-a, the workload node came up Ready in us-central1-a. The fallback loop also behaves correctly in stubbed tests, and create-cluster is recorded exactly once on full failure.

Two cleanup gaps are worth fixing before merge (inline on cleanup_cluster.sh). The rest are small.

Tests: could you commit the stubbed-gcloud tests from the description, e.g. cloudbuild/macrobenchmarks/metrics/tests/test_create_cluster.py? CI already runs that directory, and lib.sh lets tests override FAILED_FILE/BUILD_VARS_FILE. Cases worth locking in:

• success in _ZONE;
• fallback, with --zone=<fallback> --node-locations=<_ZONE>;
• all zones fail, with create-cluster recorded once;
• zones list fails, so only _ZONE is tried;
• if the cleanup suggestion is adopted, cleanup finding a cluster in a fallback zone.

Docs (nit): README.md (the _ZONE row and the create-cluster row) and macrobenchmarks_schema.json (zone) still say _ZONE is the cluster's zone. Suggest: "zone for the workload node pool and zonal buckets; preferred zone for the control plane (may fall back within the region)". Anyone running get-credentials --zone=<_ZONE> with _SKIP_CLEANUP=true won't find a cluster that fell back to another zone.

[Mahalaxmibejugam]

Added cloudbuild/macrobenchmarks/metrics/tests/test_create_cluster.py covering the five cases, plus cleanup skipping when no cluster exists and falling back to the recorded location when the lookup fails. Updated the _ZONE and create-cluster rows in the README; left the schema description unchanged to keep this PR to the scripts.