Skip to content

test: cover the branches a mutant could survive - #176

Merged
tas50 merged 1 commit into
mainfrom
spec-coverage
Aug 30, 2026
Merged

test: cover the branches a mutant could survive#176
tas50 merged 1 commit into
mainfrom
spec-coverage

Conversation

@tas50

@tas50 tas50 commented Aug 30, 2026

Copy link
Copy Markdown
Member

Why

The suite has good volume — 3,184 lines of spec against 1,838 of lib — and reads well. But volume is not coverage, so I ran a mutation pass: change one thing in lib/, run the suite, see whether anything notices. Twenty-five changes to the driver left all 344 examples green.

Most fall into two shapes:

  1. A builder is tested exhaustively in isolation and never checked for being wired in. create_instance_object is the worst case: deleting any of its eight assignments — disks, guest_accelerators, metadata, network_interfaces, scheduling, service_accounts, tags, labels — left the suite passing. Every one of those builders has its own thorough describe block; the object they get attached to had three assertions on it, and be_a(Instance) was one of them.

    spec/support/driver_context.rb already had exactly the right helper — created_instance_payload, which runs a real create and returns what insert_instance actually received. Nothing in the suite called it. It does now.

  2. A guard exists and no example takes the branch it guards. parse_event's is_a?(Hash) check, instance_service_accounts' nil check, image_disk_size_gb's ClientError rescue (zero line coverage), disk_size_for_image's nil check.

One of these was actively misleading. response_from_serial_port scans the port newest line first, and in every relevant example the matching response was the last line — so nothing before it was ever parsed. That made "ignores a response generated for a different key" tautological: the stale entry was never read, and the example passed with the modulus/exponent comparison deleted outright. Moving the noise after the response is what makes those examples mean what their names say.

What this changes

lib/ is untouched. Every one of these behaviours was already correct — it just was not asserted.

Was survivable Now asserted by
Any of the eight assignments in create_instance_object instance_payload_spec.rb, "the payload handed to insert_instance"
create running generate_server_name before create_disks_config lifecycle_spec.rb, "with names that only fit if the disks are configured first"
create not calling update_windows_password at all lifecycle_spec.rb, "with a WinRM transport"
destroy deleting disks before the instance holding them lifecycle_spec.rb, "deletes the instance before the disks it is holding"
The instance-name budget off by one either way lifecycle_spec.rb, the two new budget-boundary contexts
A standalone disk attachment sending no autoDelete disk_build_spec.rb, "auto-deletes the standalone disk with the instance"
The modulus/exponent check on the serial port response windows_password_spec.rb, noise moved after the response
parse_event's Hash guard windows_password_spec.rb, "skips a line that is valid JSON but not an object"
A configured Windows username reaching the agent windows_password_spec.rb, "with an account other than the built-in Administrator"
The serial port number windows_password_spec.rb, "reads the serial port the Windows agent writes to"
service_account_scopes: nil instance_payload_spec.rb
disk_size: nil disk_build_spec.rb
disk_size_for_image at the exact-match boundary disk_build_spec.rb
image_disk_size_gb memoization, nil guard, and ClientError rescue disk_build_spec.rb, "reading the image's size"
The auto-migrate warning for guest accelerators validation_spec.rb

Three of these are regressions the driver has already had once, which is why they are worth pinning:

  • create ordering. The instance name is budgeted against the longest disk name, so the disk config has to be settled first. generate_server_name called by hand is always handed an already-normalised config, so only a real create can catch the swap — and swapping them reintroduces the illegal 67-character disk name from fix: leave room for the disk name when generating an instance name #162.
  • autoDelete on standalone disks. Without it, kitchen destroy reports success and the disk keeps billing.
  • The key check on the serial port. Accepting a stale response hands back a password that decryption then fails on.

Testing

Every row above was verified by making the change to lib/ and confirming the suite goes red, then reverting:

$ bundle exec rake test
373 examples, 0 failures     # was 344

$ bundle exec cookstyle --chefstyle
18 files inspected, no offenses detected

Sample of the mutation runs, before and after:

before                                              after
drop tags from payload            => 0 failures     => 1 failure
drop labels from payload          => 0 failures     => 1 failure
drop disks from payload           => 0 failures     => 5 failures
drop modulus/exponent match       => 0 failures     => 3 failures
drop Hash guard in parse_event    => 0 failures     => 1 failure
swap disks config / server name   => 0 failures     => 1 failure
destroy: disks before instance    => 0 failures     => 2 failures
name budget: > becomes >=         => 0 failures     => 1 failure
disk suffix: +1 becomes +0        => 0 failures     => 1 failure
drop auto_delete on attached disk => 0 failures     => 2 failures

Left alone deliberately

  • connection and authorization have no line coverage. That is by design — the shared context stubs connection, and windows_spec.rb asserts a second ComputeService is never built. Covering them would mean asserting on the Google client's constructor, which tests the library rather than this gem.
  • Memoization of image_name and boot_disk_source_image. Removing either only costs an extra API call; there is no behaviour to pin.
  • instance_metadata stubbing the subject's own metadata (instance_payload_spec.rb:284). It works and it is readable; rewriting it is churn without a finding behind it.

The suite has good volume and reads well, but a mutation pass over lib/
found twenty-five changes to the driver that left all 344 examples green.
Most of them are places where a builder is tested thoroughly in isolation
and never checked for being wired in, or where a guard exists and nothing
ever takes the branch it guards.

The largest hole was create_instance_object. Deleting any of the eight
assignments in it - disks, guest accelerators, metadata, network
interfaces, scheduling, service accounts, tags, labels - left the suite
passing, because every builder is asserted on its own and the object they
are attached to never was. spec/support/driver_context.rb already defined
created_instance_payload, which runs a real create and hands back what
insert_instance actually received; nothing called it. It does now.

The rest, each of which used to survive:

- create ran generate_server_name before create_disks_config without
  complaint, which is the reordering the disk-name budget exists to
  prevent, and reintroduces the illegal 67-character disk name
- create no longer calling update_windows_password at all
- destroy deleting disks before the instance still holding them
- the instance-name budget off by one in either direction
- the standalone disk attachment sending no autoDelete, which orphans a
  billing disk after a successful destroy
- the modulus and exponent check on the serial port response, which the
  specs never reached: the port is scanned newest line first and the
  matching response was always the last line, so the noise before it was
  never parsed and "ignores a response generated for a different key" was
  tautological
- parse_event's Hash guard, without which a bare JSON scalar on the port
  raises TypeError mid-create
- a configured Windows username reaching the agent, rather than the
  built-in Administrator that Google's images ship disabled
- the serial port number
- nil service_account_scopes, which is what "service_account_scopes:"
  written bare in kitchen.yml parses to
- nil disk_size, likewise
- disk_size_for_image at the boundary, where the request already matches
  the image exactly
- image_disk_size_gb's memoization, its nil-image guard, and its
  ClientError rescue, which had no line coverage at all
- the auto-migrate warning for guest accelerators

No lib/ changes: every one of these was already correct, and is now
asserted. 344 examples to 373.

Signed-off-by: Tim Smith <tim@mondoo.com>
@tas50
tas50 merged commit b88c504 into main Aug 30, 2026
8 checks passed
@tas50
tas50 deleted the spec-coverage branch August 30, 2026 03:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant