I hesitate to say that I’ve been sitting on this post for a while. Let’s just say I’ve been giving it some thought.
At my current place of work, because we host our product online the testers, beyond testing the code pre-release necessarily have to have to support this code once it goes live.
This leads to a situation where the more that is deployed, the greater the scope of support testing becomes. Pile on top of this the joys and complexities of feature requests and bug reports that inevitably come in along with a weekly deployment cycle and you have a recipe for what recruiters like to call a ‘fast paced work environment’.
Our testing team grew organically, but rapidly from a single person to two, three, then five. Each tester would join the team, skill up on our products, environment and processes and then gravitated to their own areas that they were particularly good with. We had the tendency to do work on those areas fairly exclusively. Having testers who were product area experts was good in that we turned requests for testing around quickly, and were able to troubleshoot problems at lightning speed.
Over time however, problems started to arise. Testers started to miss problems, sometimes very obvious ones. When one of the team was away, we occasionally struggled to do the work that was normally assigned to them. We knew the products we were working with, but new additions and changes might mean that the product was significantly different if we hadn’t looked at that area for a couple of months.
With around sixteen different product areas to support (the majority of which are still being actively developed) it was also difficult to make sure that all these areas were getting the attention they required.
I spent a lot of time thinking about how to solve these issues. Up to that point, the testing team was seen as an highly efficient unit. Of course it only takes a few small issues before people begin asking pointed questions. I was asked some pointed questions and felt some sense of urgency to overcome the situation.
When I broke down the problems into their components, the answer became obvious. Issues that I expected testers to be picking up were being missed due to fatigue through repetition. If you look at something frequently enough, there’s a tendency to skip over things.
It’s like driving a car. When you learn to drive, you are painfully aware of everything. Tyre check, clutch, gearstick, accelerator, speedometer, indicator, head checks, and so on. As you become more accustomed, changing gears goes from a precariously balanced set of bodily maneuvers to a smooth singular action. Same with changing lanes, or overtaking.
You can become blase about it though. Miss the clutch changing a gear – grind the gearbox. Miss the speedo when on the freeway, cop a speeding fine. Miss a head check when changing lanes, kill everyone in your vehicle.
Okay, maybe a little dramatic, but you get the point. Repetition and subtle change don’t go together if that subtle change needs to be noticed immediately.
Another issue was over-specialisation. While we had people who could troubleshoot and test their area of specialty, if they weren’t there, then the testing process took noticeably longer because the other testers had become unfamiliar with that part of the product.
Lastly, because people gravitated toward what they were good at doing, and because the amount that we had to support was growing steadily, there were little things that started to slip. There wasn’t a clear enough delineation of responsibilities for the testers to follow.
When I thought about how to solve the problem of over-familiarity and under-familiarity, and slippage, it seemed that a rotation system was the obvious solution. Rotate the testers through all of the product areas on a regular basis.
There were a few things in terms of logistics to get right. How long should each rotation be? How do you reach an equitable split in the workload between testers? Should each area of the rotation contain related products, or should there be a distribution of different products within each area?
We experimented with a few different options before arriving at a solution that worked well. With our rotation split into four areas, we found that having related items grouped into a single rotation worked better than splitting them up. I suspect that this is because each rotation is quite different, as opposed to having four rotations that are variations of each other.
Four weeks per rotation was too long. Two weeks was too short. We finally settled on three weeks per rotation and this has worked well for us. Of course YMMV depending on how complex the animals you’re testing are and how actively they’re being changed.
It hasn’t all been smooth sailing. There is occasionally confusion for people interacting with testers being thrown by who is in which area of the rotation, but that’s more of an information flow thing, really.
The rotation relies on the testers being able enough to work autonomously and being diligent enough to put their hand up when their workload requires assistance. When testers are absent, it is a reasonably simple matter to distribute their workload amongst the testers either side of that rotation.
It has turned out to be an excellent training tool. With such a vast array of product areas to learn, the rotation breaks it down into manageable chunks that one can get to grips with before moving on.
We review the layout of the rotations every six months or so and occasionally make adjustments where the workload of one has increased at the expense of the other. There are also occasions where projects that finish create an influx of tickets for a particular area. We keep track of the feedback/bug reporting rate around these releases to try and determine whether it looks like being a long-term increase in work load or not.
As rotation time rolls around, the testers brief their replacements on where the product is at, what changes have taken place and what the incoming tester(s) may need to be aware of. Any near-complete tasks are kept by the tester that started them, rather than bringing someone else up to speed.
Overall it has been a massive help to the team, and allows me as a manager to more easily keep tabs on the testers’ workloads. I don’t know how well a rotation would work for shorter projects, but if you’re struggling with some of the issues I have described above for long-term support projects then you might like to consider it.