CAST2011 – The Testing Competition

(co-written with Louise Perold)

BK: I wasn’t sure about entering the testing competition. It looked like a bit of fun, but it was going to take up an entire evening of a 2 day conference. It’s difficult enough to catch up with all the people you want to see without taking up a bunch more time. I noticed that quite a few of the testers I wanted to talk to were also entering though and moreover, I’d not had the chance to test with Louise Perold before and this was a perfect opportunity.

LP: I was at CAST in 2007 in Seattle and entered the tester competition there as an individual. I had a lot of fun the first time, so I was quite keen on taking part again. It is a challenge though when you have been concentrating all day and your head is pretty full and then you have 4 hours that you know you are going to give some serious attention. So much of my job now is attending meetings and I get so little time to test anymore, that I definitely wanted to just break something again. The opportunity to do this with my good friend Ben, was something I could not pass up.

BK: The testing challenge was simple to explain; Install some software created by Dave Gilbert (who was attending the conference and on hand for the competition), then test said software for bugs, submit bug reports and tie things up with a test report at the end. Not so easy in practice, as it turned out. We faced a couple of challenges immediately. I have a thumb drive with testing tools I like to use – only I’d left it in Japan. The other challenge was that both Lou and I had left the power supply for our laptops back at the hotel. I left Lou to download the tool while I raced back to the hotel to pick up said power supplies.

LP: I also regretted not having my proper laptop with me. Testing on a netbook for a four hour stretch is not ideal. I found that I had timesnapper on my machine and got that setup and bookmarked the relevent links – and then started to try and get the app downloaded.

BK: I got back to discover that the low bandwidth of the conference centre was causing problems with the download. We sourced a USB key that was floating around with the app on it and installed from there. It was here I discovered my first bug. Upon launch, the app crashed – didn’t handle non-US date format. Lou and I decided we should go talk to Dave and let him know this would probably affect other users with non-US settings. We also decided to interview him about the product.

LP: At the previous Seattle conference, the “Hey David” team, got a lot of good feedback by speaking to the developer. I also remembered how nice David was, easy to talk to and how much useful information I also got from the last competition. I decided to follow the same strategy and we asked him some questions around the product. I asked my first – something I normally start out with when testing a new product or function – “What was the problem you were trying to solve with this product?”. This would give me some insights into how the product needed to serve his needs and some information on how he interacts with his in his work. I then asked him if there were any areas that he was concerned about where he felt there couldbe bugs as well as complex areas of the code, etc. We got some great information.
We also bought him a beer.
As I recall it, David then came to sit at our table – we had power there and there were only the two of us at first.

BK: Having David at our table made life a lot easier. Being able to test and chat simultaneously with the guy who wrote what you’re testing is really the only way to fly (okay not the only way, but it works well for me). When you have built up a relationship with someone and see what makes them tick, you open up a much wider range of possible communication styles. You can always ask direct questions, but you can also make jokes, you also put a face (or faces) to the bug reports you send in. If something comes across as harsh in print, they may give you the benefit of the doubt if they have other references that show you’re not arbitrarily being a bastard.

Lou and I settled into testing our respective areas, logging bugs, consulting with one another about bug reproduction and logging. 4 hours is actually a really really short amount of time, especially since the app we were testing on was non-trivial. We chatted with Dave as well, not just talking shop, but also music and life in general, following where the conversation naturally went. It was good fun. Some of the bugs we found were cool and difficult to reproduce. The bug reporting mechanism itself left something to be desired, but we rolled with it. I went with a chartered exploratory testing style and here was where I made my first real error.

I took notes in a file as I do with most chartered ET that I do. What I didn’t do was take the time to make my mission explicit. I knew I was doing recon ET in order to find bugs related to handling of varying file types and extensions, but that may not have been clear to anyone trying to debrief me (or judge against my notes). The notes themselves were okay and gave a reasonably good indication as to what I was thinking, but I could have taken a few minutes and made them a lot better.

LP: I had wanted to use Rapid Reporter, and then couldn’t get it downloaded quickly so then decided just to use notepad. I realised once again how much of a skill taking good notes is. If it is not something that you continually practice, it is really easy to get out of the habit and miss things in your notes. I guess also, when you are trying to log bugs quickly (the first team to log an issue would get the credit for the bug), I felt pressure to focus on finding things and getting them logged. I should have also realised (as in real life) how important it is to recall and explain what you have and haven’t covered and exactly how you got to certain issues, why you focused in certain places.

BK: James Bach wandered by and asked about our test strategy. I thought our answer was a reasonably good one (more or less what I just outlined above). Since Dave was simultaneously the developer and product owner, we talked to him about where we could add the most value from his point of view, divided the work between us, got ourselves into a position where we could easily exchange information and got to it.

LP: I didn’t feel like I was able to give enough detail around the specifics of what I had been covering. Maybe because I had encountered a number of crashes that were not always quick to reproduce. Maybe because my notes were really bad. So I felt I was able to talk to our bigger plan, but not really exactly how we were implementing it or where we were in that implementation.

BK: When James wandered back again asking about the test report, it occurred to me we should have gone to him earlier and asked him what he wanted to see. No time like the present though, so we asked him there. Having gained what we thought was a good understanding of what the report should contain, I went about putting it together while Lou kept testing. In hindsight, we should have been building the test report as we went. No sooner had we started putting the test report together, we discovered we had about a half of a testing mission. Okay the competition was a little bit arbitrary and the mission was about finding and reporting bugs, but I was hard pressed to clearly define the specific mission that we had. We knew that we had a hard 4 hour limit, but that was the only stopping heuristic we had. If we’d started our report earlier, it might have occurred to us that there were other factors to consider when answering ‘When will you stop testing?’.

Because the bug logging mechanism essentially made the logged bugs invisible, backtracking to find the important bugs we logged took a lot of time. If we’d logged these in the test report as we went, it would have meant we had more time for other things. As it turned out, we barely had enough time to turn in the report as it was. Even when we did turn in the report it was missing a few things. We described the areas that we tested, but not a whole lot about the actual testing we did. We didn’t really have a clear statement of our overall mission (for reasons already explained). We listed the important bugs we found, both crash bugs and bugs we considered important to look at. We also listed the areas that we hadn’t covered.

We listed our conclusion up-front. I know that some people like to skim, so having the tl;dr (too long; didn’t read) stuff at the start would at least get the point across, with the larger details to follow. We also provided our testing notes – this was both a good thing and a bad thing.

LP: I also realised afterwards that we hadn’t really explained the subtleties of our mission and context. The fact that we were in a “competition” meant certain things – that we would focus on logging bugs as quickly as possible so that we could be credited has a impact on the way you test (and probably also the way you log the bugs). My notes when I looked back on them were just bad. I don’t think we should have included them. Even the mindmap – which I normally use to outline my strategy, was seriously lacking detail. I think again I learned that for me to really focus a lot better and do a better testing job, I should just spend a bit of extra time and energy on defining and outlining my strategy and then showing where I am on and off track of that. I also know that my bug logging was generally not very good.

The fact that you couldn’t go back to the bug after logging it to edit or add information or screenshots meant that you needed to get it done right the first time. I think I was just trying to get them done as fast as possible. Although, our relationship with the developer meant that for most of the bugs, I did also show them to him or discuss them with him. I think that this would have an impact on his ability to process them even if they were maybe not the greatest bugs logged.

Looking back at it now, it probably would have been a good idea to have a mini low tech dashboard, where we could have tracked exactly where we were against the chosen functional focus areas.

BK: A mini-dashboard, yeah nice one. Would have been really handy and really easy to do too.

Lou and I left the venue kinda down about our efforts. On the walk back to the hotel, we chatted about what else we could have done. I think about every second sentence started with ‘Ah shit, you know what else we could have done?’. We kinda figured that if there was a prize for tester/programmer relations we might be in the running. We assumed that there were a bunch of other teams well ahead of us for the overall prize.

BK: We’d like to send a big thank you to Dave Gilbert for putting his app up for testing and for being the focal point for a large number of testers, man I didn’t envy him that at all. Thanks also to James for putting up the cash (literally) from his own wallet as the prize. Thanks to everyone who had to wade through the list of bugs and test reports that all of the teams turned in. That wouldn’t have been a small task. The competition was a lot of fun.

LP: Ditto from me 🙂 Dave was really an awesome developer to work with and test for.

BK: I have some suggestions for the CAST2012 competition (or really for anyone that wants to run one of these at a conference)

  • Consider a competition that runs for the duration of the conference
    LP: Actually I kind of liked the 4 hour competition because I feel like it adds a unique pressure that makes for an excellent learning experience
    A longer comp has its pros and cons, but it allows different dynamics and I reckon it’d better mimic everyday testing situations.

    • Testers can split their time between attending sessions and doing actual testing – they might even put into practice something they’ve just learned.
    • Competition staff (developers, project manager, product owner etc) can selectively be on or off duty (just like in real life) – they may also not be so easy to find – also like in real life.
    • Given that testers are now likely to mob competition staff in order to max out the tester/programmer (or other) relationship thing, it allows said programmer (or other) to escape the love fest if it all becomes a bit much.
  • Use a decent bug tracker.
    LP:I second this one.

    • Something simple like Mantis or Redmine would do the job. Set up separate projects for each testing team, or set up single users for each team so that it’s easier to keep track of who has done what
    • Said projects could be made invisible for non-members of that team (but team members would be able to check against each others’ work to avoid bug duplication)
  • Don’t worry so much about the cash. It’s a nice little motivator to be sure and yet it’s probably less important than the fact you’re up against some of the more talented testers on the planet

For those that are planning to enter next year’s competition, here are some suggestions:

  • Be prepared
    • Have your toolkit ready to go. You might be testing against a specific OS, a web app, mobile app, more than one of these or something different again.
    • Practice – If you’re not at the testing coalface every day, spend some time shaking the rust off your test-jutsu. Not just execution of testing, but information recon and reporting also.
  • Talk to people
    • Find out who matters, and build a relationship with these people.
    • Find out what matters to them
    • Buying them beer probably doesn’t hurt.
  • Use what you know to direct your testing
  • Keep your feedback loops short (short being relative)
  • Don’t overthink it. Plunge in and quit. Get some, go again – however you want to think about it, make sure you’re also doing it.
  • It’s not going to be perfect. It doesn’t have to be. It just has to be useful (to someone that matters)

Leave a Reply

Your email address will not be published. Required fields are marked *