The Blog

Using all-pairs testing in a Kanban team

published June 22, 2011

Have you ever worked with software system that contains multiple variables that can be combined in n different ways? Yup, pretty normal situation. My interest has lately drifted towards statistics and especially towards random variation. If you dig into these topics, sooner or later you will encounter orthogonal arrays and Design of Experiments.

This all relates pretty directly also to software testing. There is a method called all-pairs testing which is a way to replace OVAT (one variable at a time) with paired testing. Idea behind paired testing is to test combinations of feature variables and thus radically reduce amount of test cases. Instead of modifying one variable at a time we use the all-pairs algorithm to magically generate feature parameter pairs and test them. To my understanding, all-pairs algorithm does not generate orthogonal arrays and it produces fewer test cases than orthogonal array would. Downside is that it does not support advanced analysis methods such as Analysis of Variance for test data but who cares?

Background
We are building a publishing wizard which users can use to publish Google-maps like applications and use those on their web pages. The wizard is currently going through some modifications. Unfortunately, there are quite a few combinations how map can be constructed and testing all these combinations (browser, map layers, opacities, order, settings…) using OVAT testing would take forever. Therefore, we are now test driving all-pairs testing to see if it really works.

Our team model
When story is selected into development lane in our Kanban board, it is splitted into tasks like in sprint planning. During story analysis or while story is in development we usually construct test cases. In this case we constructed test cases using ALLPAIRS Test Case Generation Tool and used that to verify that story is ready to be pulled to next stage. We do not have any assigned “testers” in team and quality assurance is done by team member cross testing. You test my code, I test yours. I feel that this cross testing is mandatory as developers can be blind for the bugs they introduce and they usually end up testing “happy paths”.

It is actually pretty hard to come up with most of the unique test paths in a complex system as there can be thousands of combinations. Moreover, testing is not necessary a core skill in an agile team and testing can turn into random browsing.

How to use ALLPAIRS?
First, download ALLPAIRS. Next, you should list all variables in tested design and different values for those variables. E.g. we chose “browser”, “map layer opacity”, “amount of map layers”, “order of background maps” and 16 other variables that can be changed in wizard. These variables can be anything from your business domain. Next, we listed values for different variables. For example “browser” values where “IE”, “FF”, “Opera”, “Chrome”, and “Safari”. Once all variables and their values were listed in the text file, we ran all-pairs to generate test cases for us.

This process gave us 35 test cases that actually test 1374 paired combinations. How? It’s magic. Pairs are cleverly chosen.

Results
At this point our tasks were done so we headed to testing using all-pairs test plan (according to our Jira, combined development work effort was about 25 days). We started pretty confident that our random testing would have found most of the bugs already. The results? Um, well.. We found 10 bugs. One thing that we encountered was that all-pairs had generated some test combinations that were actually illegal in our application. In these cases we ended up just choosing some other combination.

All-in-all I will definitely use all-pairs again. It is a great way to reduce test cases. Moreover, it is a consistent way to avoid random testing and it helps developers to break their testing patterns.

Samuli @ 6:15 (3 Comments)

tags: Agile, Kanban, Testing

Trackback URL