Why test intelligence is the future of testing

Oftentimes, clients want us to translate a report with a bunch of metrics into things they care about, such as:

Test intelligence concepts can be used to give us the insights to answer these questions, and nearly unlimited possibilities. Here are two ways predictive modeling can be used:

1. Future test-execution trends

When will testing be complete? Predictive modeling can take into account the defect open rate and defect closure rate — along with team size, historical velocity, application availability, holidays, changes in scope and many other variables — to produce different possible scenarios to understand the impacts on timeline. If a team member leaves, how does that affect the timeline? What if we hire more people? What if we add more tests? What are the best case and worse case scenarios?

2. Defect trend information

Data can identify potential hot spots. For example, if one particular part or functionality has 80% of the defects, we can look into that and find out what’s going on, and then predict when the closure rate will be higher than the open rate.

This helps stakeholders understand the duration of the testing effort and when it’s going to end based on those dynamic drivers and variables. This concept can be applied to an infinite number of use cases.

Using data to ward off the storm

Just as hurricanes pick up a preexisting weather disturbance and gradually gain strength, so, too, do some testing problems. One week, the percentage is at 50. The next, it’s 60. Over time, how does that pattern continue? We call this prediction the hurricane model because it gives us a sense of how the storm is brewing and lets us map out a few different tracks (see figure 1).

By comparing the past several weeks with overall history and looking at the trends from the past four, six or eight weeks, we can start to predict when the proverbial storm will hit. Two to three weeks in, as soon as you start to have a trend and the context of how well you’re doing, you can use historical data to model different scenarios to start monitoring little changes.

After seeing the different paths in the hurricane model, one client recently noted, “Looking at these charts, we can see that if we wait until two weeks before launch, we will have to make huge changes, such as hiring five more people just to finish on time, or change the scope. If we look at the chart on the left and make tiny changes now, that outcome changes a lot, and we can get by with minimal staffing changes.”

When we begin a project, we may not have historical data from that particular client, but we can pull in historical data from similar clients. One of the models is based on a concept called time series linear regression. As long as you have some historical trend information — from previous sprints, historical data in the test management tool, collective knowledge or any number of things — we can use that as a starting point. We keep a rolling update as the sprints and releases move on. As the program continues, the predictive model becomes even more accurate.

Summary

Test intelligence allows users to speed up product deployment with quality assurance. The basic concepts in test intelligence can be applied to projects in many different ways, depending on your needs.