test: add integration suites that create real GCE instances

.github/workflows/integration.yml

@@ -0,0 +1,122 @@
+---
+name: 'Integration Tests'
+
+# These create real instances and real disks in a real GCP project, so they are
+# never run automatically on a pull request: secrets are not available to forks,
+# and every run costs money. Maintainers trigger them by hand, and they run
+# weekly against main so that a GCE-side change is noticed before a release.
+'on':
+  workflow_dispatch:
+    inputs:
+      suites:
+        description: "Regexp of suites to run, e.g. 'default|windows'. Empty runs all of them."
+        required: false
+        default: ""
+  schedule:
+    - cron: "0 5 * * 1"
+
+# A project has a finite CPU quota per region. Two runs at once exhaust it and
+# both fail, so let an in-flight run finish rather than cancelling it.
+concurrency:
+  group: gce-integration
+  cancel-in-progress: false
+
+permissions:
+  contents: read
+  id-token: write
+
+jobs:
+  integration:
+    runs-on: ubuntu-latest
+    timeout-minutes: 90
+    env:
+      GCE_PROJECT: ${{ secrets.GCE_PROJECT }}
+      KITCHEN_GCE_ZONE: ${{ vars.KITCHEN_GCE_ZONE || 'us-central1-a' }}
+      KITCHEN_GCE_REGION: ${{ vars.KITCHEN_GCE_REGION || 'us-central1' }}
+      KITCHEN_GCE_EMAIL: ${{ secrets.GCE_SERVICE_ACCOUNT }}
+      KITCHEN_SSH_KEY: ${{ github.workspace }}/.ssh/id_kitchen_gce
+      KITCHEN_RUN_ID: ${{ github.run_id }}
+    steps:
+      # Fail immediately, and legibly, rather than after a checkout and a
+      # bundle install, if the repository has not been configured for this.
+      - name: Check the project is configured
+        run: |
+          set -eu
+          if [ -z "${GCE_PROJECT}" ]; then
+            echo "::error::The GCE_PROJECT secret is not set - see integration/README.md"
+            exit 1
+          fi
+
+      - name: Checkout
+        uses: actions/checkout@v7
+
+      - name: Install Ruby
+        uses: ruby/setup-ruby@v1
+        with:
+          ruby-version: "3.4"
+          bundler-cache: true
+
+      # Workload identity federation exchanges GitHub's OIDC token for Google
+      # credentials, so no service account key is stored anywhere. The driver
+      # picks the result up as Application Default Credentials.
+      - name: Authenticate to Google Cloud
+        uses: google-github-actions/auth@v3
+        with:
+          project_id: ${{ secrets.GCE_PROJECT }}
+          workload_identity_provider: ${{ secrets.GCE_WORKLOAD_IDENTITY_PROVIDER }}
+          service_account: ${{ secrets.GCE_SERVICE_ACCOUNT }}
+
+      - name: Install the gcloud CLI
+        uses: google-github-actions/setup-gcloud@v3
+
+      # The driver does not manage SSH keys. The public half travels to the
+      # instance as metadata; see integration/kitchen.yml.
+      - name: Generate an SSH key for the run
+        run: |
+          set -eu
+          mkdir -p "$(dirname "${KITCHEN_SSH_KEY}")"
+          ssh-keygen -t ed25519 -N "" -C "kitchen-google-integration" -f "${KITCHEN_SSH_KEY}"
+
+      # Nothing in the default VPC allows WinRM, and the driver cannot open it:
+      # its startup script runs inside the guest. Without this the windows suite
+      # waits out its timeout on a running instance.
+      - name: Ensure the WinRM firewall rule exists
+        run: |
+          set -eu
+          gcloud compute firewall-rules describe kitchen-google-integration-winrm \
+            --project "${GCE_PROJECT}" >/dev/null 2>&1 && exit 0
+          gcloud compute firewall-rules create kitchen-google-integration-winrm \
+            --project "${GCE_PROJECT}" \
+            --allow tcp:5985 \
+            --target-tags kitchen-google-integration \
+            --source-ranges 0.0.0.0/0
+
+      # Via the environment rather than inline, so the input cannot be read as
+      # part of the command line.
+      - name: Run the integration suites
+        working-directory: integration
+        env:
+          SUITES: ${{ github.event.inputs.suites }}
+        run: bundle exec kitchen test "${SUITES}" --concurrency 4
+
+      # Destroy runs whatever happened above. A suite that leaks an instance on
+      # failure turns a red build into a recurring bill.
+      - name: Destroy everything
+        if: always()
+        working-directory: integration
+        run: bundle exec kitchen destroy --concurrency 4
+
+      - name: Upload logs
+        if: failure()
+        uses: actions/upload-artifact@v4
+        with:
+          name: kitchen-logs
+          path: integration/.kitchen/logs/
+          retention-days: 7
+
+      # Belt and braces: if destroy could not run, say so loudly rather than
+      # leaving instances and disks to be found on the next bill.
+      - name: Warn about anything left behind
+        if: failure()
+        run: |
+          echo "::warning::If 'Destroy everything' did not succeed, look for instances and disks labelled run-id=${KITCHEN_RUN_ID}"

.github/workflows/linters.yml

@@ -8,6 +8,7 @@ jobs:
   lint-unit:
     uses: test-kitchen/.github/.github/workflows/lint-unit.yml@main
     with:
-      # Documentation and debugging gems are not needed to lint or test.
-      # YARD in particular must never gate CI.
-      bundle_without: "development:docs"
+      # Documentation gems are not needed to lint or test, and the integration
+      # group is only for the suites in integration/, which never run on a pull
+      # request. YARD in particular must never gate CI.
+      bundle_without: "development:docs:integration"

.gitignore

@@ -10,3 +10,6 @@ doc/
 
 # RSpec example status persistence
 spec/examples.txt
+
+# Test Kitchen integration run state
+integration/.kitchen/

CONTRIBUTING.md

@@ -50,14 +50,31 @@ bundle exec cookstyle -a
 The unit tests mock the Google Compute Engine API, so they do not create real
 instances and do not require GCP credentials.
 
-### Manual testing against GCE
-
-Changes that touch instance creation should also be exercised against a real
-project, since the unit tests cannot catch API-level regressions. Set up
-[Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials),
-point a `kitchen.yml` at a project you control, and run `kitchen test`.
-Remember that this creates billable resources — confirm the instances are gone
-with `kitchen destroy` and check the GCE console afterwards.
+### Integration tests
+
+Mocking the API proves the driver builds the right request, not that GCE accepts
+it. The suites in [`integration/`](integration/README.md) close that gap: each
+one creates a real instance and asserts, from inside the guest, that the driver
+configured it as asked.
+
+```sh
+export GCE_PROJECT=my-gcp-project
+ssh-keygen -t ed25519 -N "" -f ~/.ssh/id_kitchen_gce
+
+bundle exec rake integration:list
+bundle exec rake integration:test
+bundle exec rake integration:destroy   # after a failed run
+```
+
+They are not part of `bundle exec rake` — they create billable resources — and
+they never run on a pull request. Maintainers run them on demand, and weekly
+against `main`. See [`integration/README.md`](integration/README.md) for what
+each suite covers and how to set the project up.
+
+Changes that touch instance creation should be exercised this way before
+merging, since the unit tests cannot catch an API-level regression. `kitchen
+test` destroys on success but leaves a failed instance running, so confirm with
+`rake integration:destroy` and check the GCE console afterwards.
 
 ## Submitting changes
 

Gemfile

@@ -13,3 +13,11 @@ end
 group :docs do
   gem "yard"
 end
+
+# Only needed to run the suites in integration/, which create real GCE
+# instances. `bundle install --without integration` skips them.
+group :integration do
+  gem "winrm", "~> 2.3"
+  gem "winrm-elevated", "~> 1.2"
+  gem "winrm-fs", "~> 1.3"
+end

Rakefile

@@ -34,6 +34,25 @@ rescue LoadError
   puts "yard is not available. (sudo) gem install yard to generate documentation."
 end
 
+namespace :integration do
+  # Deliberately not part of any default task: these create real instances and
+  # real disks in a real project, and cost real money.
+  desc "Run the integration suites against GCE (requires a GCP project)"
+  task :test do
+    Dir.chdir("integration") { sh "bundle exec kitchen test --concurrency 4" }
+  end
+
+  desc "Destroy anything the integration suites left behind"
+  task :destroy do
+    Dir.chdir("integration") { sh "bundle exec kitchen destroy --concurrency 4" }
+  end
+
+  desc "List the integration suites"
+  task :list do
+    Dir.chdir("integration") { sh "bundle exec kitchen list" }
+  end
+end
+
 # Documentation is intentionally NOT part of the default task: missing YARD
 # comments should never fail CI.
 task default: %i{test style}

integration/README.md

@@ -0,0 +1,142 @@
+# Integration suites
+
+The unit suite stubs the Compute Engine client, so it can prove the driver
+*builds* the right request but never that GCE accepts it. These suites close
+that gap: each one creates a real instance and asserts, from inside the guest,
+that the driver configured it the way the suite asked for.
+
+They are not part of `rake default`. They create real instances and real disks,
+and cost real money.
+
+## What each suite covers
+
+| Suite | What it proves |
+| --- | --- |
+| `default` | Create, converge and destroy from an image family, with one boot disk and an external IP. The driver's own metadata reaches the instance alongside the user's. |
+| `region` | With `region` and no `zone`, the driver lists the region's zones, picks one that is up, and the instance lands there. |
+| `extra-disk` | A second persistent disk is created standalone, waited on until `READY`, attached, and deleted again on destroy. |
+| `local-ssd` | A `local-ssd` disk is attached as `SCRATCH`, at GCE's fixed 375 GB, with no size sent for it. |
+| `legacy-disk` | The deprecated top-level `disk_size` / `disk_type` / `autodelete_disk` options still produce a working boot disk. |
+| `boot-disk-size` | A 10 GB request against a 20 GB image is raised to the image's size rather than rejected by GCE. |
+| `metadata` | Custom metadata, network tags and `service_account_scopes` reach the instance, and short scope aliases such as `storage-ro` are expanded. |
+| `preemptible` | `preemptible: true` is honoured, and auto-restart and live migration are forced off however the suite asks for them. |
+| `long-instance-and-disk-names` | A suite and disk name that overflow GCE's 63-character budget fall back to a UUID instance name, leaving room for a legal disk name. |
+| `windows` | The WinRM path: the guest agent resets the password for a non-builtin account over the serial port, and the driver's startup script opens 5985 inside the guest. |
+
+## Running them
+
+You need a GCP project with the Compute Engine API enabled, credentials with
+rights to create instances, disks and firewall rules (see
+[Authentication](../README.md#authentication)), and enough CPU quota in the
+target region for four small instances at once.
+
+```sh
+bundle install
+export GCE_PROJECT=my-gcp-project
+ssh-keygen -t ed25519 -N "" -f ~/.ssh/id_kitchen_gce
+
+cd integration
+bundle exec kitchen list
+bundle exec kitchen test default-ubuntu-2204
+```
+
+Or from the repository root:
+
+```sh
+bundle exec rake integration:list
+bundle exec rake integration:test      # everything
+bundle exec rake integration:destroy   # clean up after a failed run
+```
+
+`kitchen test` destroys on success. It leaves the instance up on failure so you
+can log in and look, so **run `kitchen destroy` when you are done** — or
+`rake integration:destroy`, which does it for every suite.
+
+### Settings
+
+| Variable | Default | Purpose |
+| --- | --- | --- |
+| `GCE_PROJECT` | *none* | Required. Project to create instances in. |
+| `KITCHEN_GCE_ZONE` | `us-central1-a` | Zone for every suite but `region`. |
+| `KITCHEN_GCE_REGION` | `us-central1` | Region for the `region` suite. |
+| `KITCHEN_GCE_USER` | `kitchen` | Login name, on both Linux and Windows. |
+| `KITCHEN_SSH_KEY` | `~/.ssh/id_kitchen_gce` | Private key to connect with. The matching `.pub` is sent to the instance as `ssh-keys` metadata, so it must exist. |
+| `KITCHEN_GCE_EMAIL` | `kitchen@example.com` | `email` for the Windows password exchange. |
+| `KITCHEN_RUN_ID` | `local` | Written to every instance as a `run-id` label, so a leaked one can be traced back to the run that made it. |
+
+### SSH keys
+
+The driver does not manage SSH keys, so `kitchen.yml` puts the public half into
+`ssh-keys` instance metadata itself. Two things follow:
+
+* The `.pub` file must exist before `kitchen create`, or rendering `kitchen.yml`
+  fails.
+* If the project or the instance enforces **OS Login**, metadata SSH keys are
+  ignored and nothing will connect. Turn `enable-oslogin` off for these
+  instances, or run the suites in a project that does not require it.
+
+### The Windows suite
+
+Nothing in the default VPC allows WinRM, and the driver cannot open it — its
+startup script runs *inside* the guest. Create the rule once per project, which
+is what CI does:
+
+```sh
+gcloud compute firewall-rules create kitchen-google-integration-winrm \
+  --project "${GCE_PROJECT}" \
+  --allow tcp:5985 \
+  --target-tags kitchen-google-integration \
+  --source-ranges 0.0.0.0/0
+```
+
+Narrow `--source-ranges` to the addresses you run Test Kitchen from rather than
+leaving it open to the internet.
+
+## Concurrency and quota
+
+The suites run four at a time. A fresh project is often capped at 8 to 24 CPUs
+in a region, and the Windows suite uses a 2-vCPU machine type, so raising
+`--concurrency` much past four tends to fail with a quota error rather than run
+faster.
+
+## In CI
+
+[`.github/workflows/integration.yml`](../.github/workflows/integration.yml) runs
+these weekly against `main`, and on demand through **Actions → Integration Tests
+→ Run workflow**. It is never triggered by a pull request: secrets are not
+available to forks, and every run costs money.
+
+CI authenticates with workload identity federation, exchanging GitHub's OIDC
+token for Google credentials, so no service account key is stored anywhere.
+
+It needs three repository secrets, and a workload identity pool with a provider
+scoped to this repository:
+
+| Secret | Value |
+| --- | --- |
+| `GCE_PROJECT` | Project to create instances in. |
+| `GCE_WORKLOAD_IDENTITY_PROVIDER` | Full resource name of the provider, `projects/<number>/locations/global/workloadIdentityPools/<pool>/providers/<provider>`. |
+| `GCE_SERVICE_ACCOUNT` | Email of the service account the provider impersonates. Also used as the `email` for the Windows password exchange. |
+
+Two optional repository variables override the location: `KITCHEN_GCE_ZONE` and
+`KITCHEN_GCE_REGION`.
+
+The service account needs `roles/compute.instanceAdmin.v1` to create instances
+and disks, `roles/compute.securityAdmin` (or a narrower custom role) to create
+the WinRM firewall rule once, and `roles/iam.serviceAccountUser` so it can
+attach itself to the instances the `metadata` suite gives scopes to.
+
+## Adding a suite
+
+Add it to `kitchen.yml` with a script in `scripts/`. Assertions live in the
+**provisioner**, not a verifier: the script is transferred over the driver's own
+transport and executed on the instance, so reaching the machine at all is part
+of every assertion, a non-zero exit fails the suite, and there is no verifier
+licence to satisfy.
+
+Anything the script needs to know has to travel as instance metadata — see the
+`region` suite. The provisioner runs on the instance, where nothing of the local
+environment survives.
+
+Keep each suite pointed at one behaviour: when it fails, its name should say
+what broke.