[katbyte]

A new rule, AZT008, that reports acceptance tests sitting in a file not named after the file declaring the resource or data source they test.

Why: a resource's tests are found by file name. When lb_probe_resource.go is tested from loadbalancer_probe_resource_test.go, nobody reading the resource finds its tests, and anything that pairs the two by name misses them. Run against azurerm it reports 17 of 1,908 acceptance test files. One is a real bug: a test named for the Linux scale set, in the scale set's file, that tests the Linux virtual machine instead, duplicating a test the virtual machine already has.

Tests in a file named for something else. The rule takes what each test builds its test data for, finds the file that declares it, and expects the test file to carry that file's name. One report per test file, on the first test, with a count of the rest.

-loadbalancer_probe_resource_test.go
+lb_probe_resource_test.go

A test that drifted into another resource's file. The same check catches a single test that builds its data for the wrong thing.

 // thing_resource_test.go
 func TestAccThing_probe(t *testing.T) {
-	data := acceptance.BuildTestData(t, "azurerm_lb_probe", "test")
+	data := acceptance.BuildTestData(t, "azurerm_thing", "test")

Splitting one resource's tests across files. By default a part between the declaring file's name and _test.go is accepted, so lb_probe_resource_identity_gen_test.go passes. The suffix option turns that off and requires exactly lb_probe_resource_test.go.

          AZT008:
            suffix: false

What it reports in azurerm. Every report was checked against the files, and a separate check that ignores which resource a test is for found no test file the rule missed.

What is off	Test files
loadbalancer_ file for an lb_ resource	4
a part before _resource, as in kubernetes_cluster_network_resource_test.go	7
_resource missing from the name, as in netapp_backup_vault_test.go	3
one file for a resource and its data source	1
a test building data for a different resource than its file	2

Things that are fine as they are.

• A test that builds test data for several things is fine when its file is right for any one of them.
• A data source test built with the bare resource name is accepted in either the resource's or the data source's file. azurerm has 30 of these.
• Tests for actions and ephemeral resources are not judged, since those register through other methods. azurerm has 25.
• With suffix: false, generated identity tests, list tests and state migration tests count as parts and are reported as well: 414 files in azurerm, so that setting does not suit azurerm today.
• The rule runs on the test package and reads the service package's own files from disk, so a report clears as soon as the file is renamed, cached runs included.

The rule only reports.