top of page
  • Writer's pictureCraig Risi

Why you should consider Mutation Testing



In my previous article, I looked at the importance of Static Analysis and how it can benefit the quality of your code. Another common static testing approach that can be extremely valuable to any development team is that of mutation testing. Along with effective unit tests, mutation testing can provide one of the best forms of testing that can help to ensure quality at a code level.


Mutation testing often doesn’t get too much attention drawn to it compared to many of the other forms of testing. And this is perhaps largely because mutation testing doesn’t focus on the core functionality of the software, nor does it target any of the non-functional requirements that often form most of the focus of our quality. What it does do though, is help to identify the gaps in our unit testing approach and ensure that all-important decisions that the software needs to make, are catered for.


What is Mutation Testing

As the name suggests, mutation testing is a software testing type that is based on changes or mutations. Unlike regular testing which tests the source code as it is written, mutation testing rather introduces minuscule changes into the source code to check whether the defined test cases can detect errors in the code.


This adds an extra degree of test coverage to the unit testing effort, as it checks to verify that tests are correctly failing for changes in the code that should force the failure. This will not only test the effectiveness of the existing tests but also help the developers to write additional tests as it will help identify areas of code that their unit tests do not effectively cover.


Mutation testing derives its name from the mutations it brings to the code, which are effectively called mutants. The ideal case is that none of the test cases should pass. If the test passes, then it means that there is an error in the test code or a missing permutation. In mutation testing, this is referred to as a mutant. If the test fails, then there is no error in the code, and the mutant was killed. The goal of this testing is to kill all mutants.


It might seem a little strange to include mutation testing in a chapter on static analysis. However, even if the testing approach doesn’t best conform to the definition of static analysis because code is executed – its purpose is to help improve the code and test coverage at a development level and forms part of a typical developer process. It is best done alongside the other static analysis techniques, and this is why I have placed it in this chapter. This form of testing can be done elsewhere, but it loses its value when done outside of the typical development process.


Mutation Testing Concepts

As mutation testing makes use of some rather unique terms, it’s important to perhaps explain them in more detail so that you don’t get confused.


#1) Mutants: Each time a change is made to the code, it forms a new mutant version of the code. When the test data is run through the mutant, it should ideally give us different results from the original source code. Mutants are also called mutant programs.


There are different types of mutants.


Survived Mutants: These are the mutants that are still alive after running test data through the original and mutated variants of the source code. These should be eliminated in your testing. They are also known as live mutants.


Killed Mutants: These are mutants that are killed after mutation testing. We get these when we get different results from the original and mutated versions of the source code.


Equivalent Mutants: These are closely related to live mutants, in that, they are ‘alive’ even after running test data through them. What differentiates them from others is that they have the same meaning as the original source code, even though they may have different syntax.


Mutators/mutation operators: These are what make mutations possible. I’ve already spoken about the importance of code coverage and ensuring that every decision made by your code is adequately tested. These operators or decision paths are what is changed in the mutation testing process. They can be referred to as faults or mutation rules.


Mutation score: This is a score that is calculated based on the number of mutants that have been identified and subsequently eliminated:



In this calculation, equivalent mutants are not considered when calculating the mutation score. With the goal to ensure all mutants are successively eliminated the goal is to get as high a score as possible.


How To Do Mutation Testing

Well, now that we understand a few specifics about mutation testing, it’s time to briefly discuss how it works. It might perhaps already be obvious that the mutation testing application will change operators in the code and verify that a test fails and catches the change – but to perhaps make the point clearer, I will illustrate it with the following actual example:


We will start with a Unit Test and an actual piece of code and then show how the code will undergo mutations to expose the lack of test coverage:


Unit Test

describe("User", function() {

it("should compare the two numbers from user input", function(){

expect(20).toBeGreaterThan(5);

{)

{);


This unit test is a simple test to compare two numbers and ensure that the code can correctly identify that one is bigger than the other. In this case, that 20 is greater than 5.

Below is the actual code that is called by the test. In it we enter two numbers, and it checks to see if the first number is lower than the second d and if it isn’t, we will notify of the error and request the user to enter a second number value that is higher.


Original code (JavaScript)

const number_compare = () => {

first_num = parseInt(prompt("Enter first number"))

second_num = parseInt(prompt("Enter second number that is higher than first number"))

if (first_num < second_num) {

alert(`Second number is ${second_num}. First number is

${first_num}.`)

} else {

alert(`Second number: ${second_num}, is more than first number: ${first_num}.

Please enter the correct number`)

}

}

number_compare();


Now, when we run this simple unit test, the outcome should be quite straightforward in that it will compare the two numbers and ensure that one is bigger than the other.

What mutation will do is focus on the operator in the code and switch it. Therefore changing the greater-than operator (first_num > second_num) to a lesser than operator (first_num < second_num)


This change will now change the result of the unit test with the number no longer being lower as expected – meaning the test will fail. This is the correct outcome and shows that in a function that in a small function like this, our simple test will provide the right amount of coverage.


Mutation Testing types

As with all forms of testing mutation testing also has multiple testing types that each serve a purpose:


Value Mutation

Here, we introduce a mutation by changing the parameter and/or constant values, usually by +/- 1.


Example: Original code (JavaScript)

let arr = [2,3,4,5]

for(let i=0; i<= 6; i++){

if(i%2===0){

console.log(i*2)

}

}


If the above code was meant to multiply the even numbers where i<4, then value mutation would mean changing the initialization to let i=1.


Statement Mutation

Here, we delete or duplicate a statement in a code block. We could also rearrange statements in a code block.


In an if-else block, for example, we could delete the else part or even the entire if-else block. So, if we take the duplication idea, the above code will look as follows:


Example: Original code (JavaScript)

let arr = [2,3,4,5]

for(let i=0; i<= 6; i++){

if(i%2===0){

console.log(i*2)

console.log(i*2)

}

}


Decision Mutation

This is perhaps the most immediately identifiable type of mutation, where the operators are changed to provide different meanings – and thereby forcing the logic of a test to break accordingly.


Mutation Testing Tools

This is not something that can be simply done with any self-built testing tool and so it’s useful to make use of existing testing tools that can help with the mutation testing approach. While not an exhaustive list, tools like Stryker, Jumble, PIT, and Insure++ are all recommended and capable of performing mutation testing across a wide number of programming languages.


Why Mutation Testing

Hopefully, you can already see the immense value that mutation testing can bring in improving the quality of the code and unit test approach. As with other static analysis approaches, it covers a large portion of your code that may be missed by other functional testing techniques. It also tests specific parts of the code and not just paths, branches, or statements. Allowing for more completeness in the testing effort than would otherwise be considered by typical test analysis.


The caveats to Mutation testing

However, there are risks and the primary issue with mutation testing is that it is incredibly resource-intensive. Each code mutation takes time to process and compile, plus the unit tests then need to be executed in the process. Do this for each possible permutation in a line of code and you can easily see how this adds up in both the cost of computing resources and time.


For this reason, I strongly recommend that mutation testing is not conducted in a testing pipeline like other forms of testing, but rather – again – forms part of a code review and static analysis process. Mutation testing is also best conducted on a local machine so that any form of shared infrastructure or cloud computing is not utilized, as this would prove incredibly expensive for a company.


This should not detract from the use of mutation testing and even if it adds a considerable amount of time to the testing at a code and unit level, the improvement to the overall code quality makes up for it and a software development team will make this up with a reduced testing effort at a component and end-to-end level if all parts of the code have gone through this process effectively.



There may be obvious caveats to mutation testing and it may seem like a laborious and time-consuming process, but combined with proper static analysis adds a significant improvement to both the quality of the code and the underlying unit tests that simply cannot be understated. It's an effort and exercise that all teams should look to implement in their development practices.


Comments


Thanks for subscribing!

bottom of page