For part two of my Agile metrics section (you can read part 1 here), I want to focus on metrics that are testing specific. These are a little more common and easier to track than the previous metrics, but only because they remain so important to assessing and measuring the effectiveness of your testing and ability to automate and respond to fixes quicker.
On their own none of these metrics can tell you much about how effective is team is doing form a quality perspective, but together they paint a good picture at the health of your software testing. What’s better, although the focus of this article is on Agile specific metrics, these testing metrics can be easily applied to any SDLC
Percentage of Automated Test Coverage
This metric measures the percentage of test coverage achieved by automated testing. As time progresses and more tests get automated, you should expect higher test coverage and, as a result, increased software quality. This can be measured as an entire number for the service, but should essentially be module specific and also track the number of code changes in specific modules to identify areas of high risk.
Usefulness for teams:
Test automation represents one of the only ways to achieve high test coverage in Agile teams, because test cases grow with the added functionality at each sprint.
Automated test coverage provides a basic measure of risk to teams. The more test coverage achieved by automation, the lower the chances of production defects in a release.
Defects Found in Production / Escaped Defects
Escaped defects is a simple metric that counts the defects for a given release that were found after the release date. Such defects have been found by the customer as opposed to the Agile development team. Since escaped defects tend to be quite costly, it’s helpful to analyse them carefully, and strive to see this metric decrease.
Usefulness for teams:
Analysing escaped defects helps to ensure continuous improvement in testing and development processes. Defining the root cause of escaped defects helps prevent recurrence of the same issues in subsequent releases.
Agile teams can capture the escaped defects metric per unit of time, per sprint, or release, providing specific insights into what went wrong with development or testing in a specific part of the project.
Defect Categories
Knowing whether a defect has been found is not enough—it’s important to categorize bugs to get qualitative information about defects. You can break software defects down into a number of categories, including:
Functionality errors
Communication errors
Security bugs
Performance defects
You can group defects into categories and make visual representations of the data using Pareto charts.
Usefulness for teams:
Using a Pareto chart and the Pareto principle, you can determine the 20 percent of defect categories that cause 80 percent of the problems with your software.
By highlighting the categories with most defects, the team comes to a better understanding of what they need to work on improving.
Defect Cycle Time
Defect cycle time measures how much time elapses between starting work on fixing a bug and fully resolving that bug. An Agile control chart visually represents cycle time across different Agile tasks. The chart below is a control chart indicating cycle time for a specific bug: The x-axis on the chart shows time and the y-axis indicates the number of hours it takes to resolve defects.
Usefulness for teams:
Rapid resolution of defects is conducive to quicker release times in a fast-paced Agile team.
By measuring defect cycle time against a defined threshold, you gauge exactly how fast Agile teams resolve issues and whether they are showing the expected progress over an increasing number of sprints or iterations.
Defect Spill-over
Defect spill-over measures the number of defects that don’t get resolved in a given sprint or iteration. You can also measure whether defects spilling over from one sprint get resolved in the next sprint.
Usefulness for teams:
The main goal for agile teams is to produce working software when each iteration completes. Measuring spillover minimizes the chances of teams getting stuck in the future because of a build-up of technical debt.
Measuring defect spill-over per sprint helps Agile teams get a clear idea of how efficiently they are dealing with issues.
An important caveat though worth mentioning with the above metrics though is that in every instance the way data is collected should be through the use of tools and automated as much as possible. If you rely too much on your team to manually input this data, then you are defeating the purpose of moving in an agile fashion and probably spending too much effort in capturing your data.
I hope these two articles on measuring quality will help you to focus on the right things when tracking the quality improvement of your teams. Hopefully also saving you time from having to traverse through loads of unnecessary data and rather focusing on these key factors that can essentially tell you all the important things that you need to know to get your teams moving in the right direction.
Comments