We assume that the reader is familiar with LIGO's testing framework. A reference can be found here.
To demonstrate how to use the mutation primitives in the testing framework, we will have a look at a basic function that we would like to test. Suppose we want to construct a function that takes an integer argument and doubles it, tentatively the following one:
Assume that we want to make sure that this function works as expected, because it will be used as part of a major development. We could write the following tests:
These tests check that
- when run on input
0, it returns
- when run on input
2, it returns
The function implemented (
twice) above passes the tests:
The implementation is, in fact, correct. However, it is easy to make a mistake and write the following implementation instead:
And, in fact, when we run
simple_tests on this faulty
implementation, we will see that it also passes the tests.
This is because
0 * 0 = 0 + 0 = 0 and
2 * 2 = 2 + 2 = 4. What
lessons can we draw from this?
The function was tested, but nothing guaranteed that the tests are complete enough.
Mutation testing tries to help in this area by modifying functions while keeping the same tests fixed, and alerting if some of the modified functions pass all of the tests: in that situation, the tests were not good enough to separate a good implementation from the (possibly) incorrect ones.
We can see now how to do mutation testing in LIGO for the original
x + x). The primitive from the testing
framework that we will use is
which takes a value to mutate and and a function to apply to altered
versions of that value (testing function). As soon as the function
correctly terminates (i.e. does not fail) in some value mutation,
Test.mutation_test will stop and return the result of the function
application, together with a
mutation describing the change in the
value. If all of the mutations tested fail, then
Typically, the values to mutate are functions (i.e.
'a will be
a function type), and these functions' return type (i.e.
'b) will be
For the example above, the function that will be applied is
and the value to mutate is
Running the tests again, the following output is obtained:
Test.mutation_test tries out various mutations on
twice, and sees if they pass all of the tests. In this scenario, it
was discovered that the mutation
MUL(x,x) also passes the tests:
this is the precise case we discussed earlier, when the incorrect
x * x would not be detected by the tests. We need to
update the test suite. In this case, we could propose to add a new
this verifies that when input
1 is given, output
2 is returned.
Running the mutation testing again after this adjustment, no mutation
(among those tried) will pass the tests, giving extra confidence in
the tests proposed:
The following is an example on how to mutate a contract. For that, we
will use a variation of the canonical LIGO contract with only two
Doing mutation testing on a contract with multiple entrypoints can help in finding out entrypoints that are not covered by the tests.
Consider the following test, which deploys a contract passed as
an argument (of the same type as
main above), and then tests that
Increment(7) works as intended on an initial storage
For performing mutation testing as before, we write the following test:
Running this test, the following output is obtained:
The mutation testing found that the operation
sub (corresponding to
Decrement) can be changed with no consequences in the
test: we take this as a warning signalling that the test above does not
Decrement entrypoint. We can fix this by adding a new call
Decrement entrypoint in the test above:
Running the updated test, we see that this time no mutation on
will give the same result.