The Blog

Reflections on Kanban vs. Scrum development

published June 23, 2011

Done and done! 1,5 years of hard work at National Land Survey is over. What a great project! This post will be on my personal findings about Scrum team transition to Kanban. What worked and what not? Moreover, I will try to analyze the failures we made and to come up with some solutions.  Let’s get started!

Background
4 person development team was formed about 1,5 years ago. Team members were selected based on a cv, a hourly fee, psychological tests, and an interview.  Two people knew each other from earlier project, but everyone else were complete strangers.  National Land Survey’s “project team” consisted of Product Owner / project manager and about 8 subject-matter experts. Product owner had no previous experience on Scrum but he had been coached on Scrum prior to kick-off. Team chose me to work as a Scrum Master.

We started Scrum by the book. Two week iterations, Definition of Done, retrospectives, you name it. Some of our initial DoD agreements did not materialize immediately but got better while team learned to know each other. Basically steps that team went through in terms of Scrum implementation improvements were:

• Scrum by the book, velocity, definition of done
• Unit testing
• Swarming
• Continuous integration
• Automated UI tests
• No Monday sprint plannings
• Team member cross testing
• Process waste visualization

Team does not jell immediately and this takes time. Even though I knew all these things would be nice and even mandatory to have in place, you should not force them on a team if it is not ready. In addition, team felt that it had to make progress with features and team’s process improvements would have to be made in small steps. I feel that every team will have to discover their own process and you cannot copy-paste team culture. Therefore, first important rule for organizations:

Do not break well functioning teams. If possible, try to move them together to a next project.

Hmm, what did Jurgen say?
In Turku Agile day I listened to Jurgen Appelo talking about team’s emergent goals. Well working team will come up with its own goals in addition to those set by the Product Owner. First of all, it is important to be aware of this behavior but also to encourage team to find their own way. When well functioning team is torn down, a new team will start formation process from “lower level” (Tuckman).  If team’s emergent goal had been a beneficial one, it is also lost.

Group prototype
Once group starts to work together it begins to form a group prototype. This prototype represents “us” and it affects team’s self-image.  If prototype is “energetic and responsible” you are more likely to get good results than when prototype is a “sloth”. This whole prototype thing is actually really interesting as it also means that those who are prototypically central become highly influential. Check out this nice book about the subject. In short, well functioning team is a gem. Try not to break it.

Please, I am busy, do not get side tracked
Ok, back to year 2011. About three months ago we wanted to try out Kanban. Team had been doing Scrum over a year, solution was in production, and now team had trouble with “expedite” type of work. This work that had to be taken care of right away was mostly production maintenance and occasionally some bug fixing. Team (and PO) felt that it could not wait for two weeks just to fit these expedites into a sprint. Furthermore, team began to encounter stories that were really difficult to estimate. “This could be a 1 or a 10, it depends on x”. Team felt that estimation of these tasks was waste because they had to be made anyway. Downside of this of course was the fact that when some story was ten times bigger than originally estimated, sprint goal was pretty much nullified. In addition, team had challenges with work that had to be stopped because sprint was over. This caused situations where everyone knew that a feature needed an improvement but because there was not enough time to do it, team decided to move improvements to next sprint. This is somewhat wasteful because it would be nice to completely finish a feature team is working on if everyone agrees that it lacks some key functionality. Why move work in to the future? “Because we do not have time right now”. This lead us suggesting the use of Kanban to PO. Luckily, he agreed.

After three months of development and 4 version later, our board looked like this:

(click to enlarge)

Development team’s Kanban process went like so:

Development team used the Kanban board, “project team” did not. Product backlog was still maintained by the PO and stories were estimated in product backlog like in Scrum. Sprint plannings were replaced with “pull planning”. When “selected” stage had room, PO chose next tasks on to the board. Once “Analysis” stage had space development team broke user stories into tasks like in sprint planning. The difference was that these planning sessions were much shorter than with Scrum as usually only one story had to be analyzed.  Team had 3 implementation lanes. When there were empty slots, story was moved onto the “development” stage. Then, slowly tasks moved across implementation and once everything was coded, tested and ready, story was deployed to “staging” environment and PO was notified. Next, PO and project team verified that everything was OK after which story was scheduled to a release. One lane was reserved for expedites. Plain and simple! Did that work? Well, that’s a good question.

Start with positive, what worked?
Team’s lead time was painfully visible and pressure to deploy increased as post-its kept piling up.  Our lead time clock was started when user story was placed to the “selected” stage. Clock was stopped when feature was deployed to production. Lead time was measured in calendar days. We noticed improvements in lead time, partly because now PO was more concerned about it. I think this is in line with Lean promises that you get process modifications just by visualizing it.

Lead time is very easy to track with Kanban and in our case lead time was reduced.

In picture below you can see lead time distributions of 82 stories.

In this second picture, an individual moving range chart has been created and it looks pretty steady around 11,5 days. There are some special causes that were caused by “clustering” etc.

In third picture, I have created a boxplot to see how lead time varied between releases.

Based on these images, I would say team’s process was pretty stable. Was this data usable or not? Theory is that once we know our average lead time and standard deviation we can establish a SLA. Then organization can order features and be somewhat confident that it will get them when needed. Problem with this logic is that in our case organization was not particularly interested in this stuff. They were more interested in release level information and Scrum style velocity information was enough for them. That being said, in some other context this could be vital information.

So, what else? Benefits also included the fact that the team members were more aware about the status of a story. “Is this already deployed to test?” The board told you.

Work was nicely organized as everyone in the team could see progress on the board

Also, team’s initial concern about expedite work worked out pretty well. But does this encourage towards “could you fix this for me quickly” –type of behavior that is the one main thing we are trying to avoid with Scrum? I think so.

Team was very responsive to expedites. But this is not necessary a good thing.

What did not work as advertised?
It was pretty big surprise that PO actually felt that visibility to work was lower than it had been with Scrum. Once we thought about this it was pretty clear why. We had thought that informal meetings and a Kanban board would keep PO up to date on progress. It did not. In Scrum, we spent 15-20% of our time in Scrum “meetings” that also served as a way to keep everyone on track. That being said, this is not a fault in Kanban but came out of our work.

PO felt that visibility to work with our Kanban implementation was lower compared to Scrum.

One of the biggest “problems” was caused by the lack of a timebox. We had no predefined release cadence (maybe we should have), nor had we cadence for demos.  But the problem was not only about cadence, it was also about the lack of commitment and “positive pressure”. In Scrum, timeboxed sprint will foster positive buzz as the team will not want to look stupid in demo. I felt that Kanban was lacking this energy.

Lack of commitments caused positive buzz to disappear.

Daily stand-ups where done in front of the Kanban board. Instead of asking what team members did and will do next, we focused on tasks. This created two problems. I think it decreased team member commitment and also caused team to focus on tasks instead of each others. Difference may be subtle but I felt that the team was more present in Daily Scrums and concentrated more to each others with Scrum.

Daily stand-ups were not as focused as they were in Scrum.

We held review when the team or PO “felt like it”. Usually at this point new features were already deployed to production and everyone had already tried them. This somewhat took the edge away from reviews.

Reviews were not as exciting as they were in Scrum.

What was really difficult?
It was not uncommon that one or more tasks were blocked by some external factor. We encountered situations where all free slots were taken and 50% of tasks were blocked. Often it was the case that none of us nor PO could do anything to speed up blocking issue. What to do then? We could increase WIP limits or try to proactively work with upcoming blockers.  We tried both ways even though you should not continuously tinker with WIP limits.

Our initial WIP limits were too small and we had to increase them later.

Mistakes, mistakes, mistakes
It is obvious that we made many Kanban rookie mistakes and David Anderson probably would not give us Professional Kanban Master Certification. Anyways, Kanban by stripping down Scrum did not produce results we had hoped. I feel that Scrum team can benefit from Kanban type work visualization. However Kanban as a method, in my opinion, needs some structure.  There was a nice tweet by Henrik Kniber:

Kanban teams rediscovering value of basic Scrum practices such as sprint reviews & backlog workshops

And I agree. “Kanban bible” recommends release cadences but it does not mandate where daily meetings should be held, how to inform your stakeholders nor does the book require you to get rid of developer commitments. Basically all the things that did not work was our own fault. In that sense Kanban is pretty advanced method because you really have to understand implications it has to psychology, team culture, and visibility. It definitely is easier to introduce Kanban to a company than to do a full scale Scrum transition. “Just do as you always have done, but use this board”. But will this actually change anything?

In overall, Scrum or not, organizations in Finland are in great need of full value stream visualization. It is not only about the development team. Value stream should be more visible in business side also. Then full concept to cash lead time would be very interesting.

Finally, you can also say that we created local optima with our Kanban and our lead time calculations should have included product backlog as it is the real inventory.  Once again, I agree.

Samuli @ 10:35 (24 Comments)

tags: Kanban, Scrum

Trackback URL

Using all-pairs testing in a Kanban team

published June 22, 2011

Have you ever worked with software system that contains multiple variables that can be combined in n different ways? Yup, pretty normal situation. My interest has lately drifted towards statistics and especially towards random variation. If you dig into these topics, sooner or later you will encounter orthogonal arrays and Design of Experiments.

This all relates pretty directly also to software testing. There is a method called all-pairs testing which is a way to replace OVAT (one variable at a time) with paired testing. Idea behind paired testing is to test combinations of feature variables and thus radically reduce amount of test cases. Instead of modifying one variable at a time we use the all-pairs algorithm to magically generate feature parameter pairs and test them. To my understanding, all-pairs algorithm does not generate orthogonal arrays and it produces fewer test cases than orthogonal array would. Downside is that it does not support advanced analysis methods such as Analysis of Variance for test data but who cares?

Background
We are building a publishing wizard which users can use to publish Google-maps like applications and use those on their web pages. The wizard is currently going through some modifications. Unfortunately, there are quite a few combinations how map can be constructed and testing all these combinations (browser, map layers, opacities, order, settings…) using OVAT testing would take forever. Therefore, we are now test driving all-pairs testing to see if it really works.

Our team model
When story is selected into development lane in our Kanban board, it is splitted into tasks like in sprint planning. During story analysis or while story is in development we usually construct test cases. In this case we constructed test cases using ALLPAIRS Test Case Generation Tool and used that to verify that story is ready to be pulled to next stage. We do not have any assigned “testers” in team and quality assurance is done by team member cross testing. You test my code, I test yours. I feel that this cross testing is mandatory as developers can be blind for the bugs they introduce and they usually end up testing “happy paths”.

It is actually pretty hard to come up with most of the unique test paths in a complex system as there can be thousands of combinations. Moreover, testing is not necessary a core skill in an agile team and testing can turn into random browsing.

How to use ALLPAIRS?
First, download ALLPAIRS. Next, you should list all variables in tested design and different values for those variables. E.g. we chose “browser”, “map layer opacity”, “amount of map layers”, “order of background maps” and 16 other variables that can be changed in wizard. These variables can be anything from your business domain. Next, we listed values for different variables. For example “browser” values where “IE”, “FF”, “Opera”, “Chrome”, and “Safari”. Once all variables and their values were listed in the text file, we ran all-pairs to generate test cases for us.

This process gave us 35 test cases that actually test 1374 paired combinations. How? It’s magic. Pairs are cleverly chosen.

Results
At this point our tasks were done so we headed to testing using all-pairs test plan (according to our Jira, combined development work effort was about 25 days). We started pretty confident that our random testing would have found most of the bugs already. The results? Um, well.. We found 10 bugs. One thing that we encountered was that all-pairs had generated some test combinations that were actually illegal in our application. In these cases we ended up just choosing some other combination.

All-in-all I will definitely use all-pairs again. It is a great way to reduce test cases. Moreover, it is a consistent way to avoid random testing and it helps developers to break their testing patterns.

Samuli @ 6:15 (3 Comments)

tags: Agile, Kanban, Testing

Trackback URL