Samuli Heljo | Using all-pairs testing in a Kanban team

The Blog

Using all-pairs testing in a Kanban team

published June 22, 2011

Have you ever worked with software system that contains multiple variables that can be combined in n different ways? Yup, pretty normal situation. My interest has lately drifted towards statistics and especially towards random variation. If you dig into these topics, sooner or later you will encounter orthogonal arrays and Design of Experiments.

This all relates pretty directly also to software testing. There is a method called all-pairs testing which is a way to replace OVAT (one variable at a time) with paired testing. Idea behind paired testing is to test combinations of feature variables and thus radically reduce amount of test cases. Instead of modifying one variable at a time we use the all-pairs algorithm to magically generate feature parameter pairs and test them. To my understanding, all-pairs algorithm does not generate orthogonal arrays and it produces fewer test cases than orthogonal array would. Downside is that it does not support advanced analysis methods such as Analysis of Variance for test data but who cares?

Background
We are building a publishing wizard which users can use to publish Google-maps like applications and use those on their web pages. The wizard is currently going through some modifications. Unfortunately, there are quite a few combinations how map can be constructed and testing all these combinations (browser, map layers, opacities, order, settings…) using OVAT testing would take forever. Therefore, we are now test driving all-pairs testing to see if it really works.

Our team model
When story is selected into development lane in our Kanban board, it is splitted into tasks like in sprint planning. During story analysis or while story is in development we usually construct test cases. In this case we constructed test cases using ALLPAIRS Test Case Generation Tool and used that to verify that story is ready to be pulled to next stage. We do not have any assigned “testers” in team and quality assurance is done by team member cross testing. You test my code, I test yours. I feel that this cross testing is mandatory as developers can be blind for the bugs they introduce and they usually end up testing “happy paths”.

It is actually pretty hard to come up with most of the unique test paths in a complex system as there can be thousands of combinations. Moreover, testing is not necessary a core skill in an agile team and testing can turn into random browsing.

How to use ALLPAIRS?
First, download ALLPAIRS. Next, you should list all variables in tested design and different values for those variables. E.g. we chose “browser”, “map layer opacity”, “amount of map layers”, “order of background maps” and 16 other variables that can be changed in wizard. These variables can be anything from your business domain. Next, we listed values for different variables. For example “browser” values where “IE”, “FF”, “Opera”, “Chrome”, and “Safari”. Once all variables and their values were listed in the text file, we ran all-pairs to generate test cases for us.

This process gave us 35 test cases that actually test 1374 paired combinations. How? It’s magic. Pairs are cleverly chosen.

Results
At this point our tasks were done so we headed to testing using all-pairs test plan (according to our Jira, combined development work effort was about 25 days). We started pretty confident that our random testing would have found most of the bugs already. The results? Um, well.. We found 10 bugs. One thing that we encountered was that all-pairs had generated some test combinations that were actually illegal in our application. In these cases we ended up just choosing some other combination.

All-in-all I will definitely use all-pairs again. It is a great way to reduce test cases. Moreover, it is a consistent way to avoid random testing and it helps developers to break their testing patterns.

Samuli @ 6:15 (3 Comments)

tags: Agile, Kanban, Testing

Trackback URL

3 Comments to Using all-pairs testing in a Kanban team

Samuli says on June 23, 2011 @ 11:44:19:

There was a question in Twitter “Would be interested in whether the 10 bugs were actually pair-related at all?”

Answer:
Few bugs were combinations of browser and functionality. E.g.
- use Safari.
- select more than one background map and publish it.

But mostly the bugs were not interdependent. Real benefits for us were:
1) Reduced amount of testcases. Bugs could have been found using OVAT testing, but it would have increased testing time dramatically.
2) Automatic generation of test case combinations. We could have done this by hand but it felt much better to let ALLPAIRS generate test cases.

Maaret Pyhäjärvi says on June 23, 2011 @ 13:06:23:

For browser + functionality testing, your added comment is pretty much what I expected.

For approaches such as this, I find the learning that follows really relevant: what are the features that need pairing or combinations, and what are ok with “test them just once” to uncover the problems.

Another interesting topic is whether you include these tests now in your upcoming tests (regression) or whether less is enough for that purpose, as in most cases would probably be.

I too believe all-pairs is nice, but I find that it easily can distract, if not followed with the analysis of whether the technique was relevant in this case or not.

Very rarely I’d go for all combinations. Adding dependent variables – identifying those – makes the challenge again just a bit more exciting. :)

Samuli says on June 23, 2011 @ 13:19:42:

Good points. In this case adding all test cases to regression suite would be way too much. Identifying paired combinations is a good advice. I have to try that next time :)

Cheers!