Mutation testing

We assume that the reader is familiar with LIGO's testing framework. A reference can be found here.

A simple testing example

To demonstrate how to use the mutation primitives in the testing framework, we will have a look at a basic function that we would like to test. Suppose we want to construct a function that takes an integer argument and doubles it, tentatively the following one:

const twice = (x: int) => x + x;

Assume that we want to make sure that this function works as expected, because it will be used as part of a major development. We could write the following tests:

const simple_tests = (f : ((input: int) => int)) : unit => {

/* Test 1 */

assert (Test.michelson_equal(Test.run(f, 0), Test.eval(0)));

/* Test 2 */

assert (Test.michelson_equal(Test.run(f, 2), Test.eval(4)));

};

const test = simple_tests(twice);

These tests check that twice:

• when run on input 0, it returns 0.
• when run on input 2, it returns 4.

The function implemented (twice) above passes the tests:

ligo run test gitlab-pages/docs/advanced/src/mutation.jsligo

// Outputs:

// Everything at the top-level was executed.

// - test exited with value ().

The implementation is, in fact, correct. However, it is easy to make a mistake and write the following implementation instead:

const twice = (x: int): int => x * x;

And, in fact, when we run simple_tests on this faulty implementation, we will see that it also passes the tests.

This is because 0 * 0 = 0 + 0 = 0 and 2 * 2 = 2 + 2 = 4. What lessons can we draw from this?

The function was tested, but nothing guaranteed that the tests are complete enough.

Mutation testing tries to help in this area by modifying functions while keeping the same tests fixed, and alerting if some of the modified functions pass all of the tests: in that situation, the tests were not good enough to separate a good implementation from the (possibly) incorrect ones.

We can see now how to do mutation testing in LIGO for the original implementation for twice (x + x). The primitive from the testing framework that we will use is

val Test.mutation_test : (value: 'a, tester: ('a -> 'b)) => option <['b, mutation]>

which takes a value to mutate and and a function to apply to altered versions of that value (testing function). As soon as the function correctly terminates (i.e. does not fail) in some value mutation, Test.mutation_test will stop and return the result of the function application, together with a mutation describing the change in the value. If all of the mutations tested fail, then Test.mutation_test will return None.

Typically, the values to mutate are functions (i.e. 'a will be a function type), and these functions' return type (i.e. 'b) will be unit.

For the example above, the function that will be applied is simple_tests, and the value to mutate is twice:

const test_mutation =

match(Test.mutation_test(twice, simple_tests), {

None: () => unit,

Some: pmutation => { Test.log(pmutation[1]);

failwith ("Some mutation also passes the tests! ^^") }

});

Running the tests again, the following output is obtained:

ligo run test gitlab-pages/docs/advanced/src/mutation.jsligo

// Outputs:

// Mutation at: File "gitlab-pages/docs/advanced/src/mutation.jsligo", line 1, characters 31-36:

// 1 | let twice = (x : int) : int => x + x;

// 2 |

//

// Replacing by: MUL(x ,

// x).

// File "gitlab-pages/docs/advanced/src/mutation.jsligo", line 18, characters 25-77:

// 17 | Some: pmutation => { Test.log(pmutation[1]);

// 18 | failwith ("Some mutation also passes the tests! ^^") }

// 19 | });

//

// Test failed with "Some mutation also passes the tests! ^^"

The primitive Test.mutation_test tries out various mutations on twice, and sees if they pass all of the tests. In this scenario, it was discovered that the mutation MUL(x,x) also passes the tests: this is the precise case we discussed earlier, when the incorrect implementation x * x would not be detected by the tests. We need to update the test suite. In this case, we could propose to add a new test:

const simple_tests = (f : ((input: int) => int)) : unit => {

/* Test 1 */

assert (Test.michelson_equal(Test.run(f, 0), Test.eval(0)));

/* Test 2 */

assert (Test.michelson_equal(Test.run(f, 2), Test.eval(4)));

/* Test 3 */

assert (Test.michelson_equal(Test.run(f, 1), Test.eval(2)));

};

this verifies that when input 1 is given, output 2 is returned. Running the mutation testing again after this adjustment, no mutation (among those tried) will pass the tests, giving extra confidence in the tests proposed:

ligo run test gitlab-pages/docs/advanced/src/mutation.jsligo

// Outputs:

// Everything at the top-level was executed.

// - test exited with value ().

// - test_mutation exited with value ().

Mutating a contract

The following is an example on how to mutate a contract. For that, we will use a variation of the canonical LIGO contract with only two entrypoints Increment and Decrement:

// This is mutation-contract.jsligo

type storage = int;

type parameter =

["Increment", int]

| ["Decrement", int];

type return_ = [list<operation>, storage];

// Two entrypoints

const add = (store: storage, delta: int): storage => store + delta;

const sub = (store: storage, delta: int): storage => store - delta;

/* Main access point that dispatches to the entrypoints according to

the smart contract parameter. */

const main = (action: parameter, store: storage) : return_ => {

return [

list([]) as list<operation>, // No operations

match(action, {

Increment:(n: int) => add (store, n),

Decrement:(n: int) => sub (store, n)})

]

};

Doing mutation testing on a contract with multiple entrypoints can help in finding out entrypoints that are not covered by the tests.

Consider the following test, which deploys a contract passed as an argument (of the same type as main above), and then tests that the entrypoint Increment(7) works as intended on an initial storage 5:

// This continues mutation-contract.jsligo

const originate_and_test = (mainf : (p: parameter) => (s: storage) => return_) : unit => {

let initial_storage = 5 as int;

let [taddr, _, _] = Test.originate(mainf, initial_storage, 0 as tez);

let contr = Test.to_contract(taddr);

let _ = Test.transfer_to_contract_exn(contr, (Increment (7)), 1 as mutez);

assert (Test.get_storage(taddr) == initial_storage + 7);

};

const test = originate_and_test(main);

For performing mutation testing as before, we write the following test:

const test_mutation =

match(Test.mutation_test(main, originate_and_test), {

None: () => unit,

Some: pmutation => { Test.log(pmutation[1]);

failwith ("Some mutation also passes the tests! ^^") }

});

Running this test, the following output is obtained:

ligo run test gitlab-pages/docs/advanced/src/mutation-contract.jsligo

// Outputs:

// Mutation at: File "gitlab-pages/docs/advanced/src/mutation-contract.jsligo", line 12, characters 55-68:

// 11 | let add = (store: storage, delta: int): storage => store + delta;

// 12 | let sub = (store: storage, delta: int): storage => store - delta;

// 13 |

//

// Replacing by: ADD(store ,

// delta).

// File "gitlab-pages/docs/advanced/src/mutation-contract.jsligo", line 41, characters 25-77:

// 40 | Some: pmutation => { Test.log(pmutation[1]);

// 41 | failwith ("Some mutation also passes the tests! ^^") }

// 42 | });

//

// Test failed with "Some mutation also passes the tests! ^^"

The mutation testing found that the operation sub (corresponding to the entrypoint Decrement) can be changed with no consequences in the test: we take this as a warning signalling that the test above does not cover the Decrement entrypoint. We can fix this by adding a new call to the Decrement entrypoint in the test above:

const originate_and_test = (mainf : ((p: parameter, s: storage) => return_)) : unit => {

let initial_storage = 5 as int;

let [taddr, _, _] = Test.originate_uncurried(mainf, initial_storage, 0 as tez);

let contr = Test.to_contract(taddr);

let _ = Test.transfer_to_contract_exn(contr, (Increment (7)), 1 as mutez);

let _ = Test.transfer_to_contract_exn(contr, (Decrement (3)), 1 as mutez);

assert (Test.get_storage(taddr) == initial_storage + 4);

};

Running the updated test, we see that this time no mutation on sub will give the same result.

Multiple mutations

There is an alternative version of Test.mutation_test that will collect all mutants that make the passed function correctly terminate. Its type is similar to that of Test.mutation_test, but instead of returning an optional type, it returns a list:

Test.mutation_test_all : (value: 'a, tester: ('a -> 'b)) => list <['b, mutation]>;

The example above can be modified to collect first all mutants, and then process the list:

const test_mutation =

match(Test.mutation_test_all(main, originate_and_test), list([

([]: list<[unit, mutation]>) => unit,

([hd,...tl]: list<[unit, mutation]>) => {

let ms = [hd,...tl];

for (const m of ms) {

let [_, mutation] = m;

let path = Test.save_mutation(".", mutation);

Test.log("saved at:");

Test.log(path);

};

failwith ("Some mutation also passes the tests! ^^") }

]));

In this case, the list of mutants is processed by saving each mutation to a file with the help of:

Test.save_mutation : (path: string, mutation: mutation) => option <string>

where the first argument represents the path where the mutation is to be saved, and the second argument is the mutation. This function returns an optional string, representing either: the name of the file where the mutation was saved or a failure.

Preventing mutation

In some cases, it might be a good idea to prevent mutation in certain places. A good example of this can be an assertion that is checking some invariant. To prevent such mutations, the attribute @no_mutation can be used:

/ This is mutation-contract.jsligo

type storage = int;

type parameter =

["Increment", int]

| ["Decrement", int];

type return_ = [list<operation>, storage];

// Two entrypoints

const add = (store: storage, delta: int): storage => store + delta;

/* @no_mutation */ const sub = (store: storage, delta: int): storage => store - delta;

/* Main access point that dispatches to the entrypoints according to

the smart contract parameter. */

const main = (action: parameter, store: storage) : return_ => {

/* @no_mutation */ let _ = assert (0 == 0);

return [

list([]) as list<operation>, // No operations

match(action, {

Increment:(n: int) => add (store, n),

Decrement:(n: int) => sub (store, n)})

]

};

In the example, two mutations are prevented. The first one, it is on the function sub, which prevents the mutations presented in the example from the previous sections. The second one, it is an assertion of a silly invariant, 0 equals 0, that should not be mutated to things like: 0 less than 0, 0 equal 1, etc.