Version: 1.2.0

# Mutation testing

We assume that the reader is familiar with LIGO's testing framework. A reference can be found here.

## A simple testing example​

To demonstrate how to use the mutation primitives in the testing framework, we will have a look at a basic function that we would like to test. Suppose we want to construct a function that takes an integer argument and doubles it, tentatively the following one:

const twice = (x: int) => x + x;

Assume that we want to make sure that this function works as expected, because it will be used as part of a major development. We could write the following tests:

const simple_tests = (f : ((input: int) => int)) : unit => {
/* Test 1 */
assert (Test.michelson_equal(Test.run(f, 0), Test.eval(0)));
/* Test 2 */
assert (Test.michelson_equal(Test.run(f, 2), Test.eval(4)));
};
const test = simple_tests(twice);

These tests check that `twice`:

• when run on input `0`, it returns `0`.
• when run on input `2`, it returns `4`.

The function implemented (`twice`) above passes the tests:

# Outputs:
# Everything at the top-level was executed.
# - test exited with value ().

The implementation is, in fact, correct. However, it is easy to make a mistake and write the following implementation instead:

const twice = (x: int): int => x * x;

And, in fact, when we run `simple_tests` on this faulty implementation, we will see that it also passes the tests.

This is because `0 * 0 = 0 + 0 = 0` and `2 * 2 = 2 + 2 = 4`. What lessons can we draw from this?

The function was tested, but nothing guaranteed that the tests are complete enough.

Mutation testing tries to help in this area by modifying functions while keeping the same tests fixed, and alerting if some of the modified functions pass all of the tests: in that situation, the tests were not good enough to separate a good implementation from the (possibly) incorrect ones.

We can see now how to do mutation testing in LIGO for the original implementation for `twice` (`x + x`). The primitive from the testing framework that we will use is

val Test.mutation_test : (value: 'a, tester: ('a -> 'b)) => option <['b, mutation]>

which takes a value to mutate and and a function to apply to altered versions of that value (testing function). As soon as the function correctly terminates (i.e. does not fail) in some value mutation, `Test.mutation_test` will stop and return the result of the function application, together with a `mutation` describing the change in the value. If all of the mutations tested fail, then `Test.mutation_test` will return `None`.

Typically, the values to mutate are functions (i.e. `'a` will be a function type), and these functions' return type (i.e. `'b`) will be `unit`.

For the example above, the function that will be applied is `simple_tests`, and the value to mutate is `twice`:

const test_mutation =
match(Test.mutation_test(twice, simple_tests)) {
when(None()): unit;
when(Some(pmutation)): do {
Test.log(pmutation[1]);
Test.println("Some mutation also passes the tests! ^^")
}
};

Running the tests again, the following output is obtained:

# Outputs:
# Mutation at: File "gitlab-pages/docs/advanced/src/mutation-testing/twice.jsligo", line 1, characters 31-36:
# 1 | let twice = (x : int) : int => x + x;
# 2 |
#
# Replacing by: MUL(x ,
# x).
# File "gitlab-pages/docs/advanced/src/mutation-testing/twice.jsligo", line 18, characters 25-77:
# 17 | Some: pmutation => { Test.log(pmutation[1]);
# 18 | failwith ("Some mutation also passes the tests! ^^") }
# 19 | });
#
# Test failed with "Some mutation also passes the tests! ^^"

The primitive `Test.mutation_test` tries out various mutations on `twice`, and sees if they pass all of the tests. In this scenario, it was discovered that the mutation `MUL(x,x)` also passes the tests: this is the precise case we discussed earlier, when the incorrect implementation `x * x` would not be detected by the tests. We need to update the test suite. In this case, we could propose to add a new test:

const simple_tests = (f : ((input: int) => int)) : unit => {
/* Test 1 */
assert (Test.michelson_equal(Test.run(f, 0), Test.eval(0)));
/* Test 2 */
assert (Test.michelson_equal(Test.run(f, 2), Test.eval(4)));
/* Test 3 */
assert (Test.michelson_equal(Test.run(f, 1), Test.eval(2)));
};

this verifies that when input `1` is given, output `2` is returned. Running the mutation testing again after this adjustment, no mutation (among those tried) will pass the tests, giving extra confidence in the tests proposed:

# Outputs:
# Everything at the top-level was executed.
# - test exited with value ().
# - test_mutation exited with value ().

## Mutating a contract​

The following is an example on how to mutate a contract. For that, we will use a variation of the canonical LIGO contract with only two entrypoints `Add` and `Sub`:

// This is mutation-contract.jsligo
export namespace C {
export type storage = int;
// Two entrypoints
@entry const add = (delta: int, store: storage): [list<operation>, storage] => [list([]),store + delta];
@entry const sub = (delta: int, store: storage): [list<operation>, storage] => [list([]),store - delta];
}

Doing mutation testing on a contract with multiple entrypoints can help in finding out entrypoints that are not covered by the tests.

Consider the following test, which deploys a contract passed as an argument, and then tests that the entrypoint `Add(7)` works as intended on an initial storage `5`:

// This is mutation-contract-test.jsligo
type storage = int;
type param = parameter_of MutationContract.C;
const initial_storage = 7;
const tester = (taddr : typed_address<param, storage>, _c : michelson_contract<param, storage> , _ : int) : unit => {
}
const test_original = (() => {
let orig = Test.originate(contract_of(MutationContract.C), initial_storage, 0tez);
})();

For performing mutation testing as before, we write the following test:

const test_mutation =
match(Test.originate_module_and_mutate(contract_of(MutationContract.C), initial_storage, 0tez, tester)) {
when(None()): unit;
when(Some(pmutation)): do {
let _l = Test.log(pmutation[1]);
// In a real program, one would write `failwith "A mutation passes"`
// Since we want to demonstrate the issue without an actual error
// a milder println is used in this document.
let _p = Test.println("A mutation of the contract still passes the tests!");
}
};

Running this test, the following output is obtained:

ligo run test --library . gitlab-pages/docs/advanced/src/mutation-testing/mutation-contract-test.jsligo
# Outputs:
# File "gitlab-pages/docs/advanced/src/mutation-testing/mutation-contract-test.jsligo", line 27, characters 6-68:
# 26 | Test.log(pmutation[1]);
# 27 | failwith("A mutation of the contract still passes the tests!");
# 28 | }
#
# An uncaught error occured:
# Failwith: "A mutation of the contract still passes the tests!"
# Trace:
# File "gitlab-pages/docs/advanced/src/mutation-testing/mutation-contract-test.jsligo", line 27, characters 6-68
# Mutation at: File "gitlab-pages/docs/advanced/src/mutation-testing/mutation-contract.jsligo", line 8, characters 73-86:
# 7 | @entry const add = (delta : int, store : storage) : result => [list([]), store + delta];
# 8 | @entry const sub = (delta : int, store : storage) : result => [list([]), store - delta];
#
# Replacing by: store + delta.

The mutation testing found that the operation `sub` (corresponding to the entrypoint `Sub`) can be changed with no consequences in the test: we take this as a warning signalling that the test above does not cover the `Sub` entrypoint. We can fix this by adding a new call to the `Sub` entrypoint in the test above:

const tester_add_and_sub = (taddr : typed_address<param, storage>, _c : michelson_contract<param, storage>, _i : int) : unit => {
let _xfer2 = Test.transfer_exn(taddr, Sub(3), 1mutez);
}

Running the updated test, we see that this time no mutation on `sub` will give the same result.

## Multiple mutations​

There is an alternative version of `Test.mutation_test` and `Test.originate_module_and_mutate` that will collect all mutants that make the passed function correctly terminate. Its type is similar to that of `Test.mutation_test`, but instead of returning an optional type, it returns a list:

Test.mutation_test_all : (value: 'a, tester: ('a -> 'b)) => list <['b, mutation]>;
Test.originate_and_mutate_all : (contract: module_contract<'p, 's>, init: 's, balance: tez, (tester: (originated_address: typed_address<'p, 's>, code: michelson_contract<'p, 's>, size: int) => 'b)) => list<['b, mutation]>

The example above can be modified to collect first all mutants, and then process the list:

const test_mutation_all =
when([]): unit;
when([hd,...tl]): do {
let ms = list([hd,...tl]);
let _p = Test.println("Some mutations also pass the tests!");
for (const m of ms) {
let [_, mutation] = m;
let path = Test.save_mutation(".", mutation);
let _l = Test.log("saved at:");
let _p = Test.log(path);
};
}
};

In this case, the list of mutants is processed by saving each mutant to a file with the help of:

Test.save_mutation : (path: string, mutation: mutation) => option <string>

where the first argument represents the path where the mutation is to be saved, and the second argument is the mutation. This function returns an optional string, representing either: the name of the file where the mutation was saved or a failure.

## Preventing mutation​

In some cases, it might be a good idea to prevent mutation in certain places. A good example of this can be an assertion that is checking some invariant. To prevent such mutations, the decorator `@no_mutation` can be used:

// This is mutation-contract.mligo
type storage = int;
type result = [list<operation>, storage];
// Two entrypoints
@entry
const add = (delta : int, store : storage) : result => {
@no_mutation let _a = assert (0 == 0);
return [list([]), store + delta];
};
@entry @no_mutation
const sub = (delta : int, store : storage) : result => {
return [list([]), store - delta];
};

In the example, two mutations are prevented. The first one, The second one, it is on the function `sub`, which prevents the mutations presented in the example from the previous sections. is an assertion of a silly invariant, `0` equals `0`, that should not be mutated to things like: `0` less than `0`, `0` equal `1`, etc.