limitedwipsociety.org

Tim Wasson posted the following Experience Report on the kanbandev list:

Since there has been a thread recently about what to do with a failing project, I’ll share my story:

I was hired into this situation:
3 full time devs, 1 part time dev
6 different codeignitor (php web app framework) applications, all hitting the same db, all stored in a single 1GB svn repository
Code written by offshore groups and multiple devs without coherent architecture
Green team (oldest dev was 26)
Fresh team – longest time here was 10 months, then 6 months, then 4 months, then 2 weeks.
No processes of any sort

The “build” was going to be the be-all end-all attempt to fix all the design sins of the previous 12 months in just one month, while continuing to keep the business running with daily bill runs totalling about 20000 bills per month.  Some of the tickets for the release had complete descriptions consisting of “Decide how to clean up the database and do it”

Yeah…. riiiiiight.

The original release date had already passed when I came on board.
85% of the tickets slated to be done in the release had not even been started.
The devs were spending 60-90% of their day on “fires”

And management said “the build” HAD to be finished in December.

Yeah…. riiiight.  I wondered what I had gotten myself into.

One month later, we have dealt with 65 individual “fire” tickets, completed 24 planned tickets, and we will deliver a build that has the most important things the business needed, while reducing error rates on bills from 15% to less than 1%, and 2 of the devs have rolled off onto the next batch of prioritized tickets.

Here’s what I did:
1. Got a white board
2. Got stickies
3. Make Kanban board with regular whiteboard pens.  Figured the process would change, so make it easy to change
4. Started the morning meeting.
5. Identified fires with pink tickets.
6. Focused devs on one task at a time.  Told them they were not to think, plan, or meet with anyone about anything that wasn’t in work or in their immediate queue.
7. Met with mgmt and convinced them of two things:  If we insist on getting everything in the build, it will not ship until february, and two, I can’t promise any dates.
8. Had devs politely but firmly refer all questions from biz people to me.  The devs stopped giving anyone else answers.  Once bugging the devs is no longer effective, people stop doing it.  Told the business that if they want status, come to the morning meeting.  My guys were not going to answer status questions about tickets because that’s what the morning meeting was for.
9. Arranged to handle all “you need to go change this in the db” type problems at one time right after the morning meeting.
10. Convinced mgmt that we had to do a release in December of whatever we had in order to improve morale among the devs and the rest of the company.
11.  Kept refusing to give a date for the first two weeks I was here.
12. After two weeks, I gave a rough idea where I thought we might get to in the queue of tickets if we wanted to release on 12/15.

The biggest things that my devs have appreciated:
1. Focus – only one ticket in work at any given time.
2. Credit – Fixing fires is no longer just time the devs are supposed to eat in addition to their normal working hours.
3. Horizon – They were buried under a mound of hundreds of “MUST HAVES – BLOCKER – CRITICAL – HIGH PRIORITY” tickets.  My devs only worry about tickets that are coming up in their queue.
4. Traction – Moving the tickets at the morning meeting is a moral builder.  Seeing the stacks of fire tickets as real work that has been accomplished rather than things that prevented them from working has totally changed their attitude towards meeting the daily problems the business faces.
5. Root Cause Troubleshooting – If a fire is important enough to fight, then we’re going to fight it.  We’re going to ask for good concrete examples, we’re going to track them down, we’re going to fix them, we’re going to get the testing that we need, we’re going to get the support from the business, and we’re going to put these things to bed.

So take it from me:

Kanban works.
It is so much easier than Scrum.
With Scrum you waste time arguing over how long a ticket will take, you waste time arguing over committing to this one last ticket for the release date, fires still don’t count as “real” work, and the tracking of hours and burndown chart turns into a timecard game each time I’ve seen it.  The hours and hours wasted making up numbers which will are guaranteed to be either too big or too small – it’s waste.  The sprint planning meetings are a beatdown.  You still feel pushed into promising dates when you don’t know.  With Scrum, daily changes in priority are a problem, adding and removing tickets is a problem, because now you have to re-check your dates, renegotiate everything.  That is almost all waste.

You can start Kanban tomorrow.  Just do it.  Learn as you do it.  Don’t be afraid to rewrite the process on the white board.  Mine changes about every 3 days.

Prioritize work
Focus on one thing at a time
Manage interruptions
Create checklists, processes, and rituals
Show progress

Funny story:  When I arrived, the devs were routinely going into the master database and modifying records throughout the day in response to various emails from company people.  This was leading directly to database replication problems, data corruption, and a willingness to continue to fix the data instead of fixing the code.

The entire business was upset that their database was unstable, but was contributing to that with constant requests to just fix the data.  I decided to make touching the master database a bit more expensive:  I created a “Database Ceremony” – In order to touch the master database, three people would be required to participate: me, the developer, and the business person who wants the change, must all participate.  First, we read the famous “Holy Hand Grenade of Antioch” passage from the “book of armaments” from Monty Python’s Quest for the Holy Grail.  I have a special round glass decanter that we take out of a special box, and we put on the desk with us.  Then we recite the reading aloud together.  Then we carefully log each and every query, check the numbers as we go, and at then end we say “amen” and we print the log, it becomes another “chapter” in our own 3-ring binder “book of armaments”, and we put the decanter back in the box.

Yes, it’s silly and unprofessional. So is messing with your master database all the time.  If a biz person really needs the change, we’re perfectly happy to do it, but they’re going to recite Monty Python with the geeks.  And using direct sql in your master database is like the holy hand grenade:  a powerful weapon, useful and appropriate at certain times, but one to be used with an attitude of care and reverence.

But there is a sound psychological reason behind this:  Reading the passage aloud puts us all in to a frame of mind that something important is going on, and the beginning and the end of it signify a period in which we must be extra careful  The attendees are recorded in the log as well, so there is accountability that we actually changed the data that the business wanted changed.  The standard for the log is simple:  You must show ALL your work in the master db.  Do selects and prove that the record you want to change is the right one, then change it, then prove it changed.  The log must allow us to restore the db to a proper condition if needed.

I must admit that after a database ceremony, when I put the decanter back into the box and close the lid, I do breath easier because I know the db is safe and sound.  And there is a “lean” angle to this as well:  The decanter on the desk is an Andon showing that the group is engaged in a particular unusual activity.

Anyway, sorry for the long post.

Tim

Tags: Experience, Tim Wasson