Version: 0.25.0

# Mutation testing

We assume that the reader is familiar with LIGO's testing framework. A reference can be found here.

## A simple testing example#

To demonstrate how to use the mutation primitives in the testing framework, we will have a look at a basic function that we would like to test. Suppose we want to construct a function that takes an integer argument and doubles it, tentatively the following one:

let twice = (x: int): int => x + x;

Assume that we want to make sure that this function works as expected, because it will be used as part of a major development. We could write the following tests:

let simple_tests = (f : ((input: int) => int)) : unit => {
/* Test 1 */
assert (Test.michelson_equal(Test.run(f, 0), Test.eval(0)));
/* Test 2 */
assert (Test.michelson_equal(Test.run(f, 2), Test.eval(4)));
};
let test = simple_tests(twice);

These tests check that `twice`:

• when run on input `0`, it returns `0`.
• when run on input `2`, it returns `2`.

The function implemented (`twice`) above passes the tests:

// Outputs:
// Everything at the top-level was executed.
// - test exited with value ().

The implementation is, in fact, correct. However, it is easy to make a mistake and write the following implementation instead:

let twice = (x: int): int => x * x;

And, in fact, when we run `simple_tests` on this faulty implementation, we will see that it also passes the tests.

This is because `0 * 0 = 0 + 0 = 0` and `2 * 2 = 2 + 2 = 4`. What lessons can we draw from this?

The function was tested, but nothing guaranteed that the tests are complete enough.

Mutation testing tries to help in this area by modifying functions while keeping the same tests fixed, and alerting if some of the modified functions pass all of the tests: in that situation, the tests were not good enough to separate a good implementation from the (possibly) incorrect ones.

We can see now how to do mutation testing in LIGO for the original implementation for `twice` (`x + x`). The primitive from the testing framework that we will use is

Test.mutation_test : (value: 'a, tester: ('a -> 'b)) => option <['b, mutation]>

which takes a value to mutate and and a function to apply to altered versions of that value (testing function). As soon as the function correctly terminates (i.e. does not fail) in some value mutation, `Test.mutation_test` will stop and return the result of the function application, together with a `mutation` describing the change in the value. If all of the mutations tested fail, then `Test.mutation_test` will return `None`.

Typically, the values to mutate are functions (i.e. `'a` will be a function type), and these functions' return type (i.e. `'b`) will be `unit`.

For the example above, the function that will be applied is `simple_tests`, and the value to mutate is `twice`:

let test_mutation =
match(Test.mutation_test(twice, simple_tests), {
None: () => unit,
Some: pmutation => { Test.log(pmutation);
failwith ("Some mutation also passes the tests! ^^") }
});

Running the tests again, the following output is obtained:

// Outputs:
// Mutation at: File "gitlab-pages/docs/advanced/src/mutation.jsligo", line 1, characters 31-36:
// 1 | let twice = (x : int) : int => x + x;
// 2 |
//
// Replacing by: MUL(x ,
// x).
// File "gitlab-pages/docs/advanced/src/mutation.jsligo", line 18, characters 25-77:
// 17 | Some: pmutation => { Test.log(pmutation);
// 18 | failwith ("Some mutation also passes the tests! ^^") }
// 19 | });
//
// Test failed with "Some mutation also passes the tests! ^^"

The primitive `Test.mutation_test` tries out various mutations on `twice`, and sees if they pass all of the tests. In this scenario, it was discovered that the mutation `MUL(x,x)` also passes the tests: this is the precise case we discussed earlier, when the incorrect implementation `x * x` would not be detected by the tests. We need to update the test suite. In this case, we could propose to add a new test:

let simple_tests = (f : ((input: int) => int)) : unit => {
/* Test 1 */
assert (Test.michelson_equal(Test.run(f, 0), Test.eval(0)));
/* Test 2 */
assert (Test.michelson_equal(Test.run(f, 2), Test.eval(4)));
/* Test 3 */
assert (Test.michelson_equal(Test.run(f, 1), Test.eval(2)));
};

this verifies that when input `1` is given, output `2` is returned. Running the mutation testing again after this adjustment, no mutation (among those tried) will pass the tests, giving extra confidence in the tests proposed:

// Outputs:
// Everything at the top-level was executed.
// - test exited with value ().
// - test_mutation exited with value ().

## Mutating a contract#

The following is an example on how to mutate a contract. For that, we will use a variation of the canonical LIGO contract with only two entrypoints `Increment` and `Decrement`:

// This is mutation-contract.jsligo
type storage = int;
type parameter =
["Increment", int]
| ["Decrement", int];
type return_ = [list<operation>, storage];
// Two entrypoints
let add = ([store, delta]: [storage, int]): storage => store + delta;
let sub = ([store, delta]: [storage, int]): storage => store - delta;
/* Main access point that dispatches to the entrypoints according to
the smart contract parameter. */
let main = ([action, store]: [parameter, storage]) : return_ => {
return [
list([]) as list<operation>, // No operations
match(action, {
Increment:(n: int) => add ([store, n]),
Decrement:(n: int) => sub ([store, n])})
]
};

Doing mutation testing on a contract with multiple entrypoints can help in finding out entrypoints that are not covered by the tests.

Consider the following test, which deploys a contract passed as an argument (of the same type as `main` above), and then tests that the entrypoint `Increment(7)` works as intended on an initial storage `5`:

// This continues mutation-contract.jsligo
let originate_and_test = (mainf : ((p: parameter, s: storage) => return_)) : unit => {
let initial_storage = 5 as int;
let [taddr, _, _] = Test.originate(mainf, initial_storage, 0 as tez);
let r = Test.transfer_to_contract_exn(contr, (Increment (7)), 1 as mutez);
assert (Test.get_storage(taddr) == initial_storage + 7);
};
let test = originate_and_test(main);

For performing mutation testing as before, we write the following test:

let test_mutation =
match(Test.mutation_test(main, originate_and_test), {
None: () => unit,
Some: pmutation => { Test.log(pmutation);
failwith ("Some mutation also passes the tests! ^^") }
});

Running this test, the following output is obtained:

// Outputs:
// Mutation at: File "gitlab-pages/docs/advanced/src/mutation-contract.jsligo", line 12, characters 55-68:
// 11 | let add = ([store, delta]: [storage, int]): storage => store + delta;
// 12 | let sub = ([store, delta]: [storage, int]): storage => store - delta;
// 13 |
//
// delta).
// File "gitlab-pages/docs/advanced/src/mutation-contract.jsligo", line 41, characters 25-77:
// 40 | Some: pmutation => { Test.log(pmutation);
// 41 | failwith ("Some mutation also passes the tests! ^^") }
// 42 | });
//
// Test failed with "Some mutation also passes the tests! ^^"

The mutation testing found that the operation `sub` (corresponding to the entrypoint `Decrement`) can be changed with no consequences in the test: we take this as a warning signalling that the test above does not cover the `Decrement` entrypoint. We can fix this by adding a new call to the `Decrement` entrypoint in the test above:

let originate_and_test = (mainf : ((p: parameter, s: storage) => return_)) : unit => {
let initial_storage = 5 as int;
let [taddr, _, _] = Test.originate(mainf, initial_storage, 0 as tez);
Running the updated test, we see that this time no mutation on `sub` will give the same result.