Saturday, October 2, 2021

Go: Reference Mutation Testing

In the Go community we do a lot of table-driving testing, where we write tests that iterate over a slice or map of test cases and run them through a single test where the inputs vary. These are great because it reduces code (and usually bugs!) and makes it easier to focus on the data.

I've recently come up with specialization of this that I think helps focus on the exact differences between "good" and "bad" data in test cases. I'm probably not the first person to come up with this and if someone provides me with a link to prior art I'll update the post with the correct terminology and links to sources.

A common idiom for table-driving testing is that each test case provides a unique input and an expected outcome. For each case the full input needs to be provided. For test inputs with many fields this creates a lot of repetition which makes it more difficult to see what the novel changes are from one test case to the next.

In this technique rather than providing the full input for each test case, a single "reference" input is created and used repeatedly. The test cases then include a function that takes a pointer of the reference value type and mutates that value in some way. For each test case the the reference value is copied , the operation under test is applied to the copied value, and the outcome is verified. Then, the test case's mutator is called on the copied value. The operation is then repeated and the outcome is verified.

I call this technique "reference mutation testing". Here's some example code:

package main
import (
"errors"
"testing"
)
type Record struct {
Label string
Value int
}
func (r Record) Validate() error {
if len(r.Label) == 0 {
return errors.New("label cannot be empty")
}
if len(r.Label) > 64 {
return errors.New("label must be less than 64 characters")
}
if r.Value < 0 {
return errors.New("value must be positive")
}
if r.Value > 1023 {
return errors.New("value cannot exceed 1023")
}
return nil
}
 
func TestValidateErrors(t *testing.T) {
t.Parallel()
goodRecord := Record{
Label: "test",
Value: 16,
}
for label, tCase := range map[string]struct {
mutate    func(*Record)
expectErr bool
}{
"empty label": {
mutate: func(r *Record) {
r.Label = ""
},
expectErr: true,
},
"short label": {
mutate: func(r *Record) {
r.Label = "t"
},
expectErr: false,
},
"overlong label": {
mutate: func(r *Record) {
r.Label = "0123456789abcdef" +
"0123456789abcdef" +
"0123456789abcdef" +
"0123456789abcdef" +
"0"
},
expectErr: true,
},
"long label": {
mutate: func(r *Record) {
r.Label = "0123456789abcdef" +
"0123456789abcdef" +
"0123456789abcdef" +
"0123456789abcdef"
},
expectErr: false,
},
"overhigh value": {
mutate: func(r *Record) {
r.Value = 1024
},
expectErr: true,
},
"high value": {
mutate: func(r *Record) {
r.Value = 1023
},
expectErr: false,
},
"negative value": {
mutate: func(r *Record) {
r.Value = -1
},
expectErr: true,
},
"0 value": {
mutate: func(r *Record) {
r.Value = 0
},
expectErr: false,
},
} {
label, tCase := label, tCase
goodRecord := goodRecord
t.Run(label, func(t *testing.T) {
t.Parallel()
if err := goodRecord.Validate(); err != nil {
t.Fatalf("unexpected error: %s", err.Error())
}
tCase.mutate(&goodRecord)
err := goodRecord.Validate()
if tCase.expectErr && err == nil {
t.Fatal("expected error, got none")
} else if !tCase.expectErr && err != nil {
t.Fatalf("got unexpected error: %s", err)
}
})
}
}

Using the mutator function makes it clear what the exact differences are between each test case. Since you're mutating the reference value it's important that you work on a copy. It's tempting to factor testing the reference value out of the test case, but you shouldn't. Even working on a copy of reference value changes can persist; changing the contents of a member slice or map can persist between copies. By testing the reference value each time you're reducing the chances that mutations persisting will go undetected.

This technique isn't ideal for every situation or even every table-driven test but it can be handy where you want to test how behavior changes in response to very specific changes to input.