We assume that the reader is familiar with LIGO's testing framework. A reference can be found here.
A simple testing example
To demonstrate how to use the mutation primitives in the testing framework, we will have a look at a basic function that we would like to test. Suppose we want to construct a function that takes an integer argument and doubles it, tentatively the following one:
Assume that we want to make sure that this function works as expected, because it will be used as part of a major development. We could write the following tests:
These tests check that
- when run on input
0, it returns
- when run on input
2, it returns
The function implemented (
twice) above passes the tests:
The implementation is, in fact, correct. However, it is easy to make a mistake and write the following implementation instead:
And, in fact, when we run
simple_tests on this faulty
implementation, we will see that it also passes the tests.
This is because
0 * 0 = 0 + 0 = 0 and
2 * 2 = 2 + 2 = 4. What
lessons can we draw from this?
The function was tested, but nothing guaranteed that the tests are complete enough.
Mutation testing tries to help in this area by modifying functions while keeping the same tests fixed, and alerting if some of the modified functions pass all of the tests: in that situation, the tests were not good enough to separate a good implementation from the (possibly) incorrect ones.
We can see now how to do mutation testing in LIGO for the original
x + x). The primitive from the testing
framework that we will use is
which takes a value to mutate and and a function to apply to altered
versions of that value (testing function). As soon as the function
correctly terminates (i.e. does not fail) in some value mutation,
Test.mutation_test will stop and return the result of the function
application, together with a
mutation describing the change in the
value. If all of the mutations tested fail, then
Typically, the values to mutate are functions (i.e.
'a will be
a function type), and these functions' return type (i.e.
'b) will be
For the example above, the function that will be applied is
and the value to mutate is
Running the tests again, the following output is obtained:
Test.mutation_test tries out various mutations on
twice, and sees if they pass all of the tests. In this scenario, it
was discovered that the mutation
MUL(x,x) also passes the tests:
this is the precise case we discussed earlier, when the incorrect
x * x would not be detected by the tests. We need to
update the test suite. In this case, we could propose to add a new
this verifies that when input
1 is given, output
2 is returned.
Running the mutation testing again after this adjustment, no mutation
(among those tried) will pass the tests, giving extra confidence in
the tests proposed:
Mutating a contract
The following is an example on how to mutate a contract. For that, we
will use a variation of the canonical LIGO contract with only two
Doing mutation testing on a contract with multiple entrypoints can help in finding out entrypoints that are not covered by the tests.
Consider the following test, which deploys a contract passed as
an argument (of the same type as
main above), and then tests that
Increment(7) works as intended on an initial storage
For performing mutation testing as before, we write the following test:
Running this test, the following output is obtained:
The mutation testing found that the operation
sub (corresponding to
Decrement) can be changed with no consequences in the
test: we take this as a warning signalling that the test above does not
Decrement entrypoint. We can fix this by adding a new call
Decrement entrypoint in the test above:
Running the updated test, we see that this time no mutation on
will give the same result.
There is an alternative version of
Test.mutation_test that will
collect all mutants that make the passed function correctly terminate.
Its type is similar to that of
Test.mutation_test, but instead of
returning an optional type, it returns a list:
The example above can be modified to collect first all mutants, and then process the list:
In this case, the list of mutants is processed by saving each mutation to a file with the help of:
where the first argument represents the path where the mutation is to be saved, and the second argument is the mutation. This function returns an optional string, representing either: the name of the file where the mutation was saved or a failure.
In some cases, it might be a good idea to prevent mutation in certain
places. A good example of this can be an assertion that is checking
some invariant. To prevent such mutations, the attribute
@no_mutation can be used:
In the example, two mutations are prevented. The first one, it is on
sub, which prevents the mutations presented in the
example from the previous sections. The second one, it is an assertion
of a silly invariant,
0, that should not be mutated to
0 less than