test: cover the branches a mutant could survive - #176
Merged
Merged
Conversation
The suite has good volume and reads well, but a mutation pass over lib/ found twenty-five changes to the driver that left all 344 examples green. Most of them are places where a builder is tested thoroughly in isolation and never checked for being wired in, or where a guard exists and nothing ever takes the branch it guards. The largest hole was create_instance_object. Deleting any of the eight assignments in it - disks, guest accelerators, metadata, network interfaces, scheduling, service accounts, tags, labels - left the suite passing, because every builder is asserted on its own and the object they are attached to never was. spec/support/driver_context.rb already defined created_instance_payload, which runs a real create and hands back what insert_instance actually received; nothing called it. It does now. The rest, each of which used to survive: - create ran generate_server_name before create_disks_config without complaint, which is the reordering the disk-name budget exists to prevent, and reintroduces the illegal 67-character disk name - create no longer calling update_windows_password at all - destroy deleting disks before the instance still holding them - the instance-name budget off by one in either direction - the standalone disk attachment sending no autoDelete, which orphans a billing disk after a successful destroy - the modulus and exponent check on the serial port response, which the specs never reached: the port is scanned newest line first and the matching response was always the last line, so the noise before it was never parsed and "ignores a response generated for a different key" was tautological - parse_event's Hash guard, without which a bare JSON scalar on the port raises TypeError mid-create - a configured Windows username reaching the agent, rather than the built-in Administrator that Google's images ship disabled - the serial port number - nil service_account_scopes, which is what "service_account_scopes:" written bare in kitchen.yml parses to - nil disk_size, likewise - disk_size_for_image at the boundary, where the request already matches the image exactly - image_disk_size_gb's memoization, its nil-image guard, and its ClientError rescue, which had no line coverage at all - the auto-migrate warning for guest accelerators No lib/ changes: every one of these was already correct, and is now asserted. 344 examples to 373. Signed-off-by: Tim Smith <tim@mondoo.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The suite has good volume — 3,184 lines of spec against 1,838 of lib — and reads well. But volume is not coverage, so I ran a mutation pass: change one thing in
lib/, run the suite, see whether anything notices. Twenty-five changes to the driver left all 344 examples green.Most fall into two shapes:
A builder is tested exhaustively in isolation and never checked for being wired in.
create_instance_objectis the worst case: deleting any of its eight assignments —disks,guest_accelerators,metadata,network_interfaces,scheduling,service_accounts,tags,labels— left the suite passing. Every one of those builders has its own thorough describe block; the object they get attached to had three assertions on it, andbe_a(Instance)was one of them.spec/support/driver_context.rbalready had exactly the right helper —created_instance_payload, which runs a realcreateand returns whatinsert_instanceactually received. Nothing in the suite called it. It does now.A guard exists and no example takes the branch it guards.
parse_event'sis_a?(Hash)check,instance_service_accounts' nil check,image_disk_size_gb'sClientErrorrescue (zero line coverage),disk_size_for_image's nil check.One of these was actively misleading.
response_from_serial_portscans the port newest line first, and in every relevant example the matching response was the last line — so nothing before it was ever parsed. That made"ignores a response generated for a different key"tautological: the stale entry was never read, and the example passed with the modulus/exponent comparison deleted outright. Moving the noise after the response is what makes those examples mean what their names say.What this changes
lib/is untouched. Every one of these behaviours was already correct — it just was not asserted.create_instance_objectinstance_payload_spec.rb, "the payload handed to insert_instance"createrunninggenerate_server_namebeforecreate_disks_configlifecycle_spec.rb, "with names that only fit if the disks are configured first"createnot callingupdate_windows_passwordat alllifecycle_spec.rb, "with a WinRM transport"destroydeleting disks before the instance holding themlifecycle_spec.rb, "deletes the instance before the disks it is holding"lifecycle_spec.rb, the two new budget-boundary contextsautoDeletedisk_build_spec.rb, "auto-deletes the standalone disk with the instance"windows_password_spec.rb, noise moved after the responseparse_event's Hash guardwindows_password_spec.rb, "skips a line that is valid JSON but not an object"windows_password_spec.rb, "with an account other than the built-in Administrator"windows_password_spec.rb, "reads the serial port the Windows agent writes to"service_account_scopes: nilinstance_payload_spec.rbdisk_size: nildisk_build_spec.rbdisk_size_for_imageat the exact-match boundarydisk_build_spec.rbimage_disk_size_gbmemoization, nil guard, andClientErrorrescuedisk_build_spec.rb, "reading the image's size"validation_spec.rbThree of these are regressions the driver has already had once, which is why they are worth pinning:
createordering. The instance name is budgeted against the longest disk name, so the disk config has to be settled first.generate_server_namecalled by hand is always handed an already-normalised config, so only a realcreatecan catch the swap — and swapping them reintroduces the illegal 67-character disk name from fix: leave room for the disk name when generating an instance name #162.autoDeleteon standalone disks. Without it,kitchen destroyreports success and the disk keeps billing.Testing
Every row above was verified by making the change to
lib/and confirming the suite goes red, then reverting:Sample of the mutation runs, before and after:
Left alone deliberately
connectionandauthorizationhave no line coverage. That is by design — the shared context stubsconnection, andwindows_spec.rbasserts a secondComputeServiceis never built. Covering them would mean asserting on the Google client's constructor, which tests the library rather than this gem.image_nameandboot_disk_source_image. Removing either only costs an extra API call; there is no behaviour to pin.instance_metadatastubbing the subject's ownmetadata(instance_payload_spec.rb:284). It works and it is readable; rewriting it is churn without a finding behind it.