The Blog
Looking for Scrum training in Finland?
published February 2, 2012
Well, you came to the right place. Tomorrow, February 3rd, we are releasing Finnish Scrum guide 2011 and thus it would be a great time for you to gain more Scrum knowledge. I provide following courses:
Professional Scrum Foundations
Scrum Teams succeed best with a solid foundation and this 2 day course prepares students to begin effectively using Scrum immediately. The Scrum framework, mechanics, and roles of Scrum are emphasized with emphasis on practical application. This course is the perfect introduction to Scrum for anyone new to the framework. Whether your team needs a flawless Scrum launch or needs to reboot a struggling Scrum implementation, this class lays the foundation for an effective Scrum team. This course lays the foundation of knowledge needed for other, more advanced training such as Professional Product Owner and Professional Scrum Master.
Professional Scrum Master
The Professional Scrum Master program includes a course, an assessment, and a certification. The Professional Scrum Master course is the first significant update of the Certified ScrumMaster (CSM) course that Ken Schwaber first created in 2002. The Professional Scrum Master course covers Scrum basics, including the framework, mechanics, and roles of Scrum, but it also teaches how to use Scrum to optimize value, productivity, and the total cost of ownership of software products. The program includes two levels of assessment and certification.
Source:
Scrum.org Program overviews.
Interested? Drop me a message!
Stop using story points?
published January 26, 2012
I just read an interesting article about replacing story points with number of stories implemented in a sprint. Check the whole article by Vasco Duarte: “Story Points Considered Harmful – Or why the future of estimation is really in our past… “.
After reading Vasco’s article I returned to my old project that I had some data still available. Was the number of stories completed in a sprint normally distributed? Well, yes.
Data has 22 sprints which were gathered during one year.
Vasco has some really good points, be sure to check his blog out. About the estimation effort he says:
Claim 3: The use of Story points doesn’t take a lot of time:
Having worked with Story Points for several years this is not my experience. Although some progress has been done by people like Ken Power (at Cisco) with the Silent Grouping technique, the fact that we need such technique should dispute any idea that estimating in SP’s “doesn’t take a lot of time”. In fact, as anybody that has tried a non-trivial project knows it can take days of work to estimate the initial backlog for a reasonable size project.
I have had similar experiences and working with planning poker *can* take a lot of time.
Let’s presume that estimation could actually be done using just number of stories. I am wondering if the estimation process still adds some value to the team. Is estimation increasing their understanding about the story? Will some team members understand story differently if size is not fully discussed? If a sprint planning is done using some team average will it have negative effect on team “commitment”? Hmm, I guess we will find out sooner or later.
Anyways, I think that complexity sciences are currently bringing very interesting stuff into Agile development.
Agile vs. Lean Six Sigma
published November 2, 2011
Are these two related in any way? Is Lean Six Sigma some ancient process improvement method that has nothing to do with software development? Now that I’ve covered Kanban vs. Scrum I thought it would be a time to compare Agile to Lean Six Sigma.
Five domains of system categorization
Before we dig into Lean Six Sigma I should clarify some terms that will be used. A leader’s framework for decision making defines basics for Cynefin Framework, which is available at Cognitive Edge. This sense-making framework defines five categories for any system: simple, complicated, complex, chaotic, and disorder. Basically, it says that work in different domains should be managed differently. The fastest way to get a grasp of the ideas is to watch this video:
What is Lean Six Sigma?
As most of you already know what Agile software development is all about, I won’t go into that. However, Lean Six Sigma is something that the majority of people have no idea of. Some people see Lean Six Sigma as a desperate attempt to combine the old-fashioned scientific Six Sigma method to the Lean method as that is now the hot topic. Lean was derived from Toyota Production System and Six Sigma has nothing to do with that, right? Not so fast.
Six Sigma method was originally developed by Motorola in the 1970s and it is a systematic way of improving processes. Later, Lean concepts were added and Lean Six Sigma was born. You can read the full history e.g. here. Jeff Liker comments on Lean vs. Six Sigma like this:
If a company that does not have a history of process improvement has started with lean and used lean tools to work on flow issues, setting up cells and supermarkets, will this be enough? The answer is a resounding no. All they have done is set the stage for identifying problems. They need a problem solving method. Some companies are using six sigma in this way and that is fine. The serious ones learn that the green belt training given to the work groups will solve the large majority of problems. But then you still need to evolve over time into other aspects of the system–team leader role, andon, standardized work, visual management, hoshin kanri, etc. I say evolve because it is a long-term process in which you try something, make mistakes, reflect, adjust, and then as you get good at that you add more to begin to build a system. The underlying framework was also taught by Deming and it is PDCA.
Lean Six Sigma defines an improvement process, tools, and roles that should be present in a Lean Six Sigma project. I think that role belt system (black belt, green belt, etc.) part is somewhat comical and can actually hinder Sig Sigma adoption in organizations. However, the theory is that you can take any process (sales process, production of goods, etc.), check how much defects it’s currently producing, and initiate an improvement project to decrease the defects thus increasing your company’s value. Lean Six Sigma defines 5 phases that improvement project should go through:
Define: What is the problem, how it should be improved, and how the improvement should be seen?
Measure: Measure the process using historical and current data.
Analyze: Analyze the gathered data to find insights.
Improve: Improve the process by finding key factors of the process.
Control: Ensure that modifications are used and deviations from the target are corrected before they result in defects.
These phases form a DMAIC acronym that simply states the order of process improvement activities. In addition to the phases, Lean Six Sigma defines a whole host of tools to be used in every phase. The tools are very much statistical such as ANOVA and Regression analysis. This is basically what you can read in Wikipedia but does not answer to the question: “What is a Lean Six Sigma project like in practice?”.
A manufacturing Lean Six Sigma sample
Your company manufactures tables. You have a problem that 30% of the tables’ surfaces are curved and have to be scrapped. This costs you annually 300.000€. Your employees all have some ideas how to improve the process and these “improvements” have been tried out. However, the long term scrap average remains at 30%. You start digging into the problem and find out that over 200 variables have some influence over the end product. You are wondering whether there are few vital variables and if optimal levels for these could be found, the scrap would be dramatically reduced. As it turns out, there usually is. Your challenge is to find these.
You start reducing the number of variables, you do this by guessing, interviewing experts, and using tools suchs as 5 whys and FMEA. You gather data from the current production and use historical data to find patterns. Usually historical data is completely useless as causal relationship is almost impossible to deduct from it. However, by continuing this guesswork, you can reduce the number of variables to about 10. You are quite confident that some combination of these variables will produce the result you are after. How do you test all combinations of the variables? One at a time? No, you will design an experiment. A factorial experiment. You will make a mathematically clever test plan which allows you to change multiple variables at a time and be able to componentize the effects of each variable using statistical software such as Minitab. You can only run about 40 tests and gather information about very large number of combinations. These tests reveal you that variable A should be set to 10.5, variable B should be set to 3232. You make modifications to the production line and check the results. Defect rate drops to 5%. Great! How long did this take? From weeks to months, it depends on your business.
Err… I am not manufacturing tables.
Right. Now it gets tricky. Six Sigma sees people producing software as a process, and so it could be improved. Agile people respond that they are producing a once-in-a-lifetime solution in a complex domain which requires innovation and creativity, you cannot use that statistical stuff in here. I think both camps are partially right. Why? Please, read on.
Lean concepts fit better together with Agile. Drive down the waste and you are improving. But a Six Sigma approach to improve e.g. Scrum team’s performance would go something like so:
1) Come up with potential process variables that could cause number of defects to reduce. E.g. unit test coverage, ATDD test coverage, and usage of pair programming.
2) Design an experiment where different features will get done using different combination of values for selected variables. A small sample of a test plan could be:
-unit test coverage 100%, ATDD test coverage 0%, no pair programming
-unit test coverage 0%, ATDD test coverage 100%, pair programming…
3) Execute an experiment and find out that 80% of defects could be reduced by pair programming and keeping unit test coverage at 100%. ATDD test coverage has little effect on the end results (Note that I do not know if this is the case, this is just an example!)
The problem is that I can already sense how a Scrum team would react to this suggestion. Are you [insert your favorite word] kidding? “Trust me, I am a Lean Six Sigma Black Belt” will not help starting the experiment either.
Lean Six Sigma works with a Taylorist mindset and tries to find causal relationship between an action and an output. It is a neat way of working when causal relationship can be detected in a complicated domain but falls short when working with complexity. There is nothing wrong with experimentation per se, but if the approach is completely out of the group’s norms there is a slim chance to get the experiment done. Statistical stuff in Six Sigma is quite hard and takes time to learn. If the team cannot fully understand what the statistics behind the method are, acceptance of experimentation approach might not happen.
Romanticized complexity
Not all people agree that software development is all complex. I ran into an article about this stuff. Joseph Pelrine says:
Unfortunately, the typical Agilist perception of complexity is not quite aligned with any of the main scientific definitions of the term. Agile literature abounds with romanticised, subjective interpretations of terms such as complexity, self-organisation, emergence, which can only be understood by remembering that ‘a little knowledge is often a dangerous thing.
He continues the analysis of software projects and state that 38.5% of work is in the complex domain. Many people are saying that “Software project is complex” but Cognitive Edge blog claims:
The activities tend to be weighted more to the complicated and complex domains, with activities related to the coding aspect of software development landing in the complicated (or sometimes simple) domain, and activities associated with project management landing in the complex (sometimes chaotic) domain.
Finally:
Software development can benefit by being treated as a complex domain (and not an ordered one), and taking advantage of the toolbox of social complexity, namely the Cynefin method. The field (as well as many other fields of human endeavour) would benefit even more from a multi-ontological approach, understanding that there are multiple domains involved, taking the best techniques for the various domains, and combining them in an appropriate and flexible manner. This multi-ontological approach would also go a long way towards resolving the infighting and bickering now taking place between the “Agile” and “Lean” communities.
Of course he is boosting Cynefin method, but I am fine with that. I am also aware of “sick stigma” pet name that is used by Dave Snowden, which is actually pretty funny. Anyways, I see no point of using pure Six Sigma when dealing with complex domain. However, for some complicated problems method contains useful stuff.
Understanding variation, measurement and multivariate experiments
For me, these three were the most important aspects that I learned in a Lean Six Sigma Black Belt training. Firstly, by knowing if some variation is random (that is inherently present in all processes) or a special cause is a starting point when improving any process. If decisions are made based on random variation, improvement is headed to a dead end. Secondly, all measurement contains error and if you don’t know how much error your measurement system is causing, you are once a again making wrong decisions. Take a Scrum team’s velocity for an example. If the velocity is increasing sprint by sprint, it doesn’t necessarily mean that team is actually improving. “Improvement” could be caused by the measurement error when team changes how it estimates work. Thirdly, even though multivariate experiments do not fit neatly in to the software development process, they can be used to gain better understanding of a service or a product the team is building. Google Website Optimizer has multivariate testing tools available and Lean Startup movement is advocating usage of experiments. “The scientific method”, as Lean Six Sigma calls itself, has something that is usable also in a software product. Our Lean Six Sigma course instructor said that Lean Sigma Sigma could be seen as design of experiments on steroids, which I completely agree with.
During this week I visited The division of Pharmaceutical Technology to get more insights how Design of Experiments are used in pharmaceutical research. Problem in Finland seems to be that there are only few organizations that are actually teaching Design of Experiments. All-in-all, I try look beyond flame wars and to learn as much as I possibly can from different disciplines. It’s like in MMA:
Since the late 1990s both strikers and grapplers have been successful at MMA though it is rare to see any fighter who is not schooled in both striking and grappling arts reach the highest levels of competition.
Not the best metaphor but you get the point.
I have heard multiple times that creativity needs variation. Without it, creativity is diminished. I understand what people are saying and agree with it but I do not think it is quite this simple. It is not sufficient for an organization to just add variation to development process and collect the innovative fruits. Organization must learn to see patterns emerging in complex domain and have skills to productize them. Unfortunately, variation in many software projects is only seen by the customer in form of bugs and low quality. This has nothing to do with new innovations.
Lean Six Sigma criticism
Even though the method has some interesting tools, its approach to management feels somewhat outdated. Improvement process is driven by black belts that solve (at least in theory) organization’s problems one by one thus increasing organization’s value. Of course stupid workers are only in the way of genius black belts. The problem is that after improvement has been “installed” and black belts leave, workers can return to old ways of working, because they didn’t own the new solution in the first place. Especially in the knowledge work, team must take ownership of its problems and appropriate solutions. I feel that the whole black belt system does not work with Scrum because there should not be specialized titles in the first place. The team should be cross-functional, it should have all the skills needed to turn a backlog into a working solution and not rely on outsiders. Experiments should be owned by the whole team, not one person. Of course, then you encounter the problem that in Finland you have a very small chance of finding anyone who can actually design experiments, should you need one.
One could also state that Six Sigma did not invent Design of Experiments and that is the most valuable part of the whole method. If its management style is not suitable for complex domain, what is left?
Conclusion
Are Agile and Lean Six Sigma friends or foes? Are they compatible? They deal with different problems. Agile is suited for complex work, Lean Six Sigma might be usable in simple and complicated domains. Experimentation and probing mindset is useful in both domains. What I intend to do is to bring understanding of variation and design of experiments as tools for Agile software teams.
Additional Material
A Podcast on complexity by Dave Snowden
Reflections on Kanban vs. Scrum development
published June 23, 2011
Done and done! 1,5 years of hard work at National Land Survey is over. What a great project! This post will be on my personal findings about Scrum team transition to Kanban. What worked and what not? Moreover, I will try to analyze the failures we made and to come up with some solutions. Let’s get started!
Background
4 person development team was formed about 1,5 years ago. Team members were selected based on a cv, a hourly fee, psychological tests, and an interview. Two people knew each other from earlier project, but everyone else were complete strangers. National Land Survey’s “project team” consisted of Product Owner / project manager and about 8 subject-matter experts. Product owner had no previous experience on Scrum but he had been coached on Scrum prior to kick-off. Team chose me to work as a Scrum Master.
We started Scrum by the book. Two week iterations, Definition of Done, retrospectives, you name it. Some of our initial DoD agreements did not materialize immediately but got better while team learned to know each other. Basically steps that team went through in terms of Scrum implementation improvements were:
- Scrum by the book, velocity, definition of done
- Unit testing
- Swarming
- Continuous integration
- Automated UI tests
- No Monday sprint plannings
- Team member cross testing
- Process waste visualization
Team does not jell immediately and this takes time. Even though I knew all these things would be nice and even mandatory to have in place, you should not force them on a team if it is not ready. In addition, team felt that it had to make progress with features and team’s process improvements would have to be made in small steps. I feel that every team will have to discover their own process and you cannot copy-paste team culture. Therefore, first important rule for organizations:
Do not break well functioning teams. If possible, try to move them together to a next project.
Hmm, what did Jurgen say?
In Turku Agile day I listened to Jurgen Appelo talking about team’s emergent goals. Well working team will come up with its own goals in addition to those set by the Product Owner. First of all, it is important to be aware of this behavior but also to encourage team to find their own way. When well functioning team is torn down, a new team will start formation process from “lower level” (Tuckman). If team’s emergent goal had been a beneficial one, it is also lost.
Group prototype
Once group starts to work together it begins to form a group prototype. This prototype represents “us” and it affects team’s self-image. If prototype is “energetic and responsible” you are more likely to get good results than when prototype is a “sloth”. This whole prototype thing is actually really interesting as it also means that those who are prototypically central become highly influential. Check out this nice book about the subject. In short, well functioning team is a gem. Try not to break it.
Please, I am busy, do not get side tracked
Ok, back to year 2011. About three months ago we wanted to try out Kanban. Team had been doing Scrum over a year, solution was in production, and now team had trouble with “expedite” type of work. This work that had to be taken care of right away was mostly production maintenance and occasionally some bug fixing. Team (and PO) felt that it could not wait for two weeks just to fit these expedites into a sprint. Furthermore, team began to encounter stories that were really difficult to estimate. “This could be a 1 or a 10, it depends on x”. Team felt that estimation of these tasks was waste because they had to be made anyway. Downside of this of course was the fact that when some story was ten times bigger than originally estimated, sprint goal was pretty much nullified. In addition, team had challenges with work that had to be stopped because sprint was over. This caused situations where everyone knew that a feature needed an improvement but because there was not enough time to do it, team decided to move improvements to next sprint. This is somewhat wasteful because it would be nice to completely finish a feature team is working on if everyone agrees that it lacks some key functionality. Why move work in to the future? “Because we do not have time right now”. This lead us suggesting the use of Kanban to PO. Luckily, he agreed.
After three months of development and 4 version later, our board looked like this:
(click to enlarge)
Development team’s Kanban process went like so:
Development team used the Kanban board, “project team” did not. Product backlog was still maintained by the PO and stories were estimated in product backlog like in Scrum. Sprint plannings were replaced with “pull planning”. When “selected” stage had room, PO chose next tasks on to the board. Once “Analysis” stage had space development team broke user stories into tasks like in sprint planning. The difference was that these planning sessions were much shorter than with Scrum as usually only one story had to be analyzed. Team had 3 implementation lanes. When there were empty slots, story was moved onto the “development” stage. Then, slowly tasks moved across implementation and once everything was coded, tested and ready, story was deployed to “staging” environment and PO was notified. Next, PO and project team verified that everything was OK after which story was scheduled to a release. One lane was reserved for expedites. Plain and simple! Did that work? Well, that’s a good question.
Start with positive, what worked?
Team’s lead time was painfully visible and pressure to deploy increased as post-its kept piling up. Our lead time clock was started when user story was placed to the “selected” stage. Clock was stopped when feature was deployed to production. Lead time was measured in calendar days. We noticed improvements in lead time, partly because now PO was more concerned about it. I think this is in line with Lean promises that you get process modifications just by visualizing it.
Lead time is very easy to track with Kanban and in our case lead time was reduced.
In picture below you can see lead time distributions of 82 stories.
In this second picture, an individual moving range chart has been created and it looks pretty steady around 11,5 days. There are some special causes that were caused by “clustering” etc.
In third picture, I have created a boxplot to see how lead time varied between releases.
Based on these images, I would say team’s process was pretty stable. Was this data usable or not? Theory is that once we know our average lead time and standard deviation we can establish a SLA. Then organization can order features and be somewhat confident that it will get them when needed. Problem with this logic is that in our case organization was not particularly interested in this stuff. They were more interested in release level information and Scrum style velocity information was enough for them. That being said, in some other context this could be vital information.
So, what else? Benefits also included the fact that the team members were more aware about the status of a story. “Is this already deployed to test?” The board told you.
Work was nicely organized as everyone in the team could see progress on the board
Also, team’s initial concern about expedite work worked out pretty well. But does this encourage towards “could you fix this for me quickly” –type of behavior that is the one main thing we are trying to avoid with Scrum? I think so.
Team was very responsive to expedites. But this is not necessary a good thing.
What did not work as advertised?
It was pretty big surprise that PO actually felt that visibility to work was lower than it had been with Scrum. Once we thought about this it was pretty clear why. We had thought that informal meetings and a Kanban board would keep PO up to date on progress. It did not. In Scrum, we spent 15-20% of our time in Scrum “meetings” that also served as a way to keep everyone on track. That being said, this is not a fault in Kanban but came out of our work.
PO felt that visibility to work with our Kanban implementation was lower compared to Scrum.
One of the biggest “problems” was caused by the lack of a timebox. We had no predefined release cadence (maybe we should have), nor had we cadence for demos. But the problem was not only about cadence, it was also about the lack of commitment and “positive pressure”. In Scrum, timeboxed sprint will foster positive buzz as the team will not want to look stupid in demo. I felt that Kanban was lacking this energy.
Lack of commitments caused positive buzz to disappear.
Daily stand-ups where done in front of the Kanban board. Instead of asking what team members did and will do next, we focused on tasks. This created two problems. I think it decreased team member commitment and also caused team to focus on tasks instead of each others. Difference may be subtle but I felt that the team was more present in Daily Scrums and concentrated more to each others with Scrum.
Daily stand-ups were not as focused as they were in Scrum.
We held review when the team or PO “felt like it”. Usually at this point new features were already deployed to production and everyone had already tried them. This somewhat took the edge away from reviews.
Reviews were not as exciting as they were in Scrum.
What was really difficult?
It was not uncommon that one or more tasks were blocked by some external factor. We encountered situations where all free slots were taken and 50% of tasks were blocked. Often it was the case that none of us nor PO could do anything to speed up blocking issue. What to do then? We could increase WIP limits or try to proactively work with upcoming blockers. We tried both ways even though you should not continuously tinker with WIP limits.
Our initial WIP limits were too small and we had to increase them later.
Mistakes, mistakes, mistakes
It is obvious that we made many Kanban rookie mistakes and David Anderson probably would not give us Professional Kanban Master Certification. Anyways, Kanban by stripping down Scrum did not produce results we had hoped. I feel that Scrum team can benefit from Kanban type work visualization. However Kanban as a method, in my opinion, needs some structure. There was a nice tweet by Henrik Kniber:
Kanban teams rediscovering value of basic Scrum practices such as sprint reviews & backlog workshops
And I agree. “Kanban bible” recommends release cadences but it does not mandate where daily meetings should be held, how to inform your stakeholders nor does the book require you to get rid of developer commitments. Basically all the things that did not work was our own fault. In that sense Kanban is pretty advanced method because you really have to understand implications it has to psychology, team culture, and visibility. It definitely is easier to introduce Kanban to a company than to do a full scale Scrum transition. “Just do as you always have done, but use this board”. But will this actually change anything?
In overall, Scrum or not, organizations in Finland are in great need of full value stream visualization. It is not only about the development team. Value stream should be more visible in business side also. Then full concept to cash lead time would be very interesting.
Finally, you can also say that we created local optima with our Kanban and our lead time calculations should have included product backlog as it is the real inventory. Once again, I agree.
Using all-pairs testing in a Kanban team
published June 22, 2011
Have you ever worked with software system that contains multiple variables that can be combined in n different ways? Yup, pretty normal situation. My interest has lately drifted towards statistics and especially towards random variation. If you dig into these topics, sooner or later you will encounter orthogonal arrays and Design of Experiments.
This all relates pretty directly also to software testing. There is a method called all-pairs testing which is a way to replace OVAT (one variable at a time) with paired testing. Idea behind paired testing is to test combinations of feature variables and thus radically reduce amount of test cases. Instead of modifying one variable at a time we use the all-pairs algorithm to magically generate feature parameter pairs and test them. To my understanding, all-pairs algorithm does not generate orthogonal arrays and it produces fewer test cases than orthogonal array would. Downside is that it does not support advanced analysis methods such as Analysis of Variance for test data but who cares?
Background
We are building a publishing wizard which users can use to publish Google-maps like applications and use those on their web pages. The wizard is currently going through some modifications. Unfortunately, there are quite a few combinations how map can be constructed and testing all these combinations (browser, map layers, opacities, order, settings…) using OVAT testing would take forever. Therefore, we are now test driving all-pairs testing to see if it really works.
Our team model
When story is selected into development lane in our Kanban board, it is splitted into tasks like in sprint planning. During story analysis or while story is in development we usually construct test cases. In this case we constructed test cases using ALLPAIRS Test Case Generation Tool and used that to verify that story is ready to be pulled to next stage. We do not have any assigned “testers” in team and quality assurance is done by team member cross testing. You test my code, I test yours. I feel that this cross testing is mandatory as developers can be blind for the bugs they introduce and they usually end up testing “happy paths”.
It is actually pretty hard to come up with most of the unique test paths in a complex system as there can be thousands of combinations. Moreover, testing is not necessary a core skill in an agile team and testing can turn into random browsing.
How to use ALLPAIRS?
First, download ALLPAIRS. Next, you should list all variables in tested design and different values for those variables. E.g. we chose “browser”, “map layer opacity”, “amount of map layers”, “order of background maps” and 16 other variables that can be changed in wizard. These variables can be anything from your business domain. Next, we listed values for different variables. For example “browser” values where “IE”, “FF”, “Opera”, “Chrome”, and “Safari”. Once all variables and their values were listed in the text file, we ran all-pairs to generate test cases for us.
This process gave us 35 test cases that actually test 1374 paired combinations. How? It’s magic. Pairs are cleverly chosen.
Results
At this point our tasks were done so we headed to testing using all-pairs test plan (according to our Jira, combined development work effort was about 25 days). We started pretty confident that our random testing would have found most of the bugs already. The results? Um, well.. We found 10 bugs. One thing that we encountered was that all-pairs had generated some test combinations that were actually illegal in our application. In these cases we ended up just choosing some other combination.
All-in-all I will definitely use all-pairs again. It is a great way to reduce test cases. Moreover, it is a consistent way to avoid random testing and it helps developers to break their testing patterns.
Tomcat secure sessions with AJP and HTTP protocols using Apache proxy balancer
published May 20, 2011
During this week I have been digging into Apache AJP protocol and how it communicates with Tomcat. Especially, I have been interested in knowing how SSL works in different Tomcat connector setups. Once again in hindsight everything is so clear but had many desperate moments during these days :) Anyways, you should first check mod_proxy balancer configurations, and ajp protocol.
Basically, I have tried two different solutions:
1) Apache proxy balancer using HTTP
<Proxy balancer://liferaycluster>
BalancerMember http://node1 route=node1
BalancerMember http://node2 route=node2
ProxySet stickysession=ROUTEID
</Proxy>
2) Apache proxy balancer using AJP
<Proxy balancer://liferaycluster>
BalancerMember ajp://node1 route=node1 ping=3
BalancerMember ajp://node2 route=node2 ping=3
ProxySet stickysession=ROUTEID
</Proxy>
Benefit of using AJP is the “ping” setting which Apache uses to check whether Tomcat is up or down and does more sophisticated load balancing based on that information. However, things become more tricky in Tomcat connector configurations when checking whether JSESSION will end up to be “secure” or “not”. In HTTP connector you can define secureness of your connector like so:
<Connector port="8443" protocol="HTTP/1.1" SSLEnabled="false" scheme="https" secure="true" proxyPort="443"/>
But if you define connector with AJP protocol:
<Connector port="8443" protocol="AJP/1.3" SSLEnabled="false" scheme="http" secure="false" proxyPort="443"/>
“Secure” setting will not have any effect to JSESSIONID cookie. However, it does have an effect to function call “ServletRequest.isSecure()”. Go figure.
It seems that AJP protocol contains boolean value wheter initial connection to Apache was secure or not and that value is passed to Tomcat which uses that information to create user session cookie.
If you need an AJP setup where JSESSIONID should not be secure even if initial connection was through HTTPS you can do Apache haxing to remove Secure setting from cookie like so:
Header edit "Set-Cookie: JSESSIONID=" Secure " "
This will replace word Secure with empty string from JSESSIONID cookie. This is not the most clever thing to do because now your session is open for hijacking.
One click deployment to clustered Liferay with Jenkins
published April 27, 2011
During last year, we have built pretty cool one click deployment system with Jenkins. Environment has been created little by little, but when decision was made to cluster Liferay and in addition to serve our 15.000 lines of JavaScript code from Apache, it was pretty clear that manual installations would be a pain. So we built environment like this:
Developer can initiate (1) portlet build from the Jenkins CI GUI. Jenkins runs (2) Ant build, unit tests, selenium tests and builds multiple war packages. If build is successful Jenkins first copies (3) our JavaScript framework to Apache and then deploys (4) portlet wars to clustered liferay environment. All this with one click.
Configuration is pretty simple, each copy step is run as a shell script like so:
./deployFrameworkToApache.sh;
./deployPortletsToNode1.sh;
./deployPortletsToNode2.sh;
I have to give credit to Jenkins, it is really easy to configure.
Liferay clustering checklist
published April 21, 2011
Yesterday our team finished clustering of our Liferay portal environment. In case I have to do this again, I decided to write a small checklist what should be taken into a consideration when clustering. Official Liferay clustering guide can be found here.
Updated May 6, 2011: added ajp configurations.
Updated May 20, 2011: Check also how Apache AJP protocol works.
1. Ensure that you have same JRE/JDK on both cluster nodes.
2. Check that all Liferay nodes can access your database (or database cluster) and it allows connections from these hosts.
3. Ensure that load balancer can forward requests to both nodes and firewalls won’t block access.
4. Check that all of cluster nodes are able to send email using your configured SMTP server (e.g. with telnet).
5. Install same version of Liferay to all cluster nodes.
6. If you need session fail over, configure that.
7. Configure tomcat connectors “secure” and “unsecure” to use ajp protocol like so:
<Connector address="xx.xx.xx.xx" port="8009" protocol="AJP/1.3" ... />
<Connector address="xx.xx.xx.xx" port="8443" protocol="AJP/1.3" ... />
8. Configure properties “net.sf.ehcache.configurationResourceName” and “ehcache.multi.vm.config.location” correctly to portal-ext.properties.
9. Move your Lucene index to database or if that is not possible, to shared NFS mount. In case of shared NFS mount, linux UIDs for Liferay user must be equal in all nodes so that Linux file permissions work correctly. Liferay user must be able to write to Lucene directory.
10. Move your document library to shared NFS mount or to database. Same rules apply as above.
11. Ensure that your configured Lucene index and document library paths point to correct location in portal-ext.properties.
12. Start your cluster nodes.
13. Check that mod_proxy_ajp module is loaded by apache.
14. Configure Apache load balancer like so:
Proxy balancers:
<Proxy balancer://liferaycluster>
BalancerMember ajp://node1:8009 route=node1 ping=5 loadfactor=1
BalancerMember ajp://node2:8009 route=node2 ping=5 loadfactor=1
ProxySet stickysession=ROUTEID
</Proxy>
<Proxy balancer://secureliferaycluster>
BalancerMember ajp://node1:8443 route=node1 ping=5 loadfactor=1
BalancerMember ajp://node2:8443 route=node2 ping=5 loadfactor=1
ProxySet stickysession=ROUTEID
</Proxy>
This inside your :80 virtual host:
Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED
ProxyPass /group balancer://secureliferaycluster/group/
ProxyPassReverse /group balancer://secureliferaycluster/group/
ProxyPass / balancer://liferaycluster/
ProxyPassReverse / balancer://liferaycluster/
This inside your :443 virtual host:
ProxyPass /group balancer://secureliferaycluster/group/
ProxyPassReverse /group balancer://secureliferaycluster/group/
ProxyPass / balancer://liferaycluster/
ProxyPassReverse / balancer://liferaycluster/
That’s it!
Building a Kanban board
published March 10, 2011
We have been using Scrum in my current project from the beginning of 2010. Our team is swarming pretty effectively on tasks and we decided that next step would be to try out Kanban as we are currently in “production” mode and see the end of our project closing.
I started this process by reading Kanban book by David J. Anderson which I highly recommend. In his book, David suggest that transition to Kanban should be started small and from the area of team’s “political control”. Board should be designed to fit current process and team should not try to optimize process prematurely. In addition, we knew that in our case input to the Kanban system would be a Scrum backlog. Armed with this information we started to sketch first the version of our board.
Sketching the version one
Layout of your Kanban board will depend on your current situation. There isn’t one right layout and people doing the work will have the best knowledge. We started by sketching our current process and thinking what classes of services we would be providing. Classes of services are basically priorities that will be assigned to kanbans so that critical work can cut in line. In our case, we decided to build 2-tier board, where “larger” cards represent user stories and “smaller” cards represent tasks. Apparently, the best practice (or at least some practice) seems to be that user stories are limited by swimlanes and their tasks are limited by “vertical WIP”. Sounds awful but bear with me.
Get some supplies
After initial board was sketched I headed to bookstore to get some tape and post-its in different colors. I tried to find very narrow black tape, but there was no such thing available so instead I bought some electricity tape and a box cutter.
Building the version one
First things first. kanban systems came from Toyota where 5s (Sorting, straightening, shining, standardizing, sustaining) methodology originated. In our case we decided to do something to our “not so shining” whiteboard.
Because I did not find any narrow tape I used a box cutter to slice tape to more stripes.
This turned out to be a bad choice. Surprisingly, I could not do completely straight slices and almost cut my fingers into half. Next time I will find proper sized tape. Anyhow, our board started to build up. Different colored tape turned out to be a good choice. Below you can see the magical metamorphosis.
And so was first version completed.
So how does this board work?
This Kanban board is designed for a four person team, who has been working with Scrum. Work will be pulled into system from product backlog that is handled by a product owner. The team will then develop features after which they are verified by a product owner or someone from business. After verification, the team is responsible for production deployment. Currently, we favor smaller user stories and layout is designed that in mind.
Click on image below to see larger description how our first version works.
Stay tuned!
This is my first attempt to do a Kanban board and there will surely be something to fix. First thing that we will be checking is our WIP limits. Cannot wait to try how this works in practice…
Scan-agile 2011
published March 3, 2011
Today, I spent my day at Scan-Agile. All-in-all it was a great event with interesting presentations. The first event I participated in was a specification game workshop by Gojko Adzic and Lasse koskela. The main goal of this workshop was to increase knowledge on communication gaps and Gojko really gave good examples why developers and product owners should get end user input earlier.
Second event was about continuous deployment at Elisa corporate webstore and nextdoor.fi. It was interesting to know that both of these teams started with Scrum but later moved to kanban. For them, kanban seemed to work better while continuously deploying new features to production. Shortening lead times seems to be an increasing trend and it is easy to see value it brings to business.
After lunch there was a really interesting presentation by Luca Minudel and Joseph Perlrine on how software is written in formula one team. According to Luca, F1 is very complex domain and no one person can understand the whole codebase of a race car. Interesting point made was that Toyota didn’t ever win constructor championship (although their trend was rising). I am currently reading a book about Toyota product development system and their approach seems to put more value to up front design compared to Scrum approach that was practiced by Ferrari (Luca did not mention Ferrari, but somehow I got the feeling…) I do not completely buy the claim that lean model has no room in complex software development, but in my opinion complex problems will require emergent design made by whole team. Again, one size does not fit all and for example new product derivate (a new Toyota model) is different business than F1 race car.
Last session was about kanban given by Mattias Skarin. Nice presentation, but I was already familiar with most of this stuff. We are currently test driving kanban on paikkatietoikkuna.fi and I am planning on reporting about that later.
Scan-agile was an excellent event and to me it was very clear that all successful implementations very built by cross-functional teams that worked hard to achieve excellence. Also, one process will not fit all purposes and it is important to select right tool for the right job.
Older Posts »Subscribe to RSS feed
The Tag Cloud
Agile Business Coaching Coding horror Conference Customer Design of Experiments Future Group dynamics ITIL It should not be that hard Java EE Kanban Leadership Lean Liferay Methodologies Natural UI Performance tuning Process Productivity Quality Retrospective RIA Scrum Six Sigma Social psychology Software Software architecture Testing This is great TOGAF
WP Cumulus Flash tag cloud by Roy Tanck and Luke Morton requires Flash Player 9 or better.
Samuli's Links
The Blog Archive
February 2012 (1)
January 2012 (1)
November 2011 (1)
June 2011 (2)
May 2011 (1)
April 2011 (2)
March 2011 (2)
February 2011 (1)
January 2011 (1)
December 2010 (1)
November 2010 (1)
October 2010 (3)
September 2010 (3)
August 2010 (5)
July 2010 (2)
June 2010 (3)
May 2010 (4)
April 2010 (2)
March 2010 (6)
February 2010 (7)
January 2010 (3)
December 2009 (7)
November 2009 (6)
