If you’re keen to attend, then pricing details are here and you can sign up here. If you make it, be sure to say hi.

Fake testing is the blind worship of process and procedure (over the sapient and conscientious application of skill). There’s been some talk of its demise recently and while I want to believe we’re heading in the right direction, I also want to keep in mind that this is not an easy problem to solve and if we’re not careful we won’t solve the problem, we’ll just shift it elsewhere.

Let me give you an example.

At GTAC, there were some very interesting tools demo’ed. I looked at BITE (Browser Integrated Testing Environment) and thought – yeah, there’s another nail in the fake testing coffin. I particularly like the concept of making it easy for the end-user to report bugs that contain information that programmers can use to solve issues quickly.

As I continued to watch, something nagged at me. A sense that something was not quite right. Not quite deja vu, but certainly the sense that if it were this easy, we’d have done it before.

Shortly thereafter, I was chatting with good friend and colleague Jared Quinert. I had forgotten how good he is at taking a subject, stripping back all the bullshit and laying its foundations bare. If you ever get the chance to work with, or hire him. Do it. Seriously. As usual, Jared locked onto what I’d been feeling uneasy about without me even having to explain myself. Having shared the link to the talk I’ve linked to at the start of this post, our conversation went something like this:

Jared: Record playback software is apparently central to the life of a manual tester…

Ben: Yeah I chuckled at that too.

Jared: Recording scripts, converting to English doesn’t solve the abstraction problem. Waiting to hear them elaborate on re-recording being cheaper than maintaining…and they’re still just talking about QTP features, but nobody has piped up with this QTP can re-learn missing controls…

Ben: True

Jared: Tool running in the browser probably creates other issues too I would think.

Ben: Likely yes. That and when you have several thousand tests and you need to re-record a couple hundred, the appeal wears thin.

Jared: The abstraction is the problem, not the approach and as always, I’m skeptical that the test can simply be replayed without any issues when there is dynamic content.

Ben: Agreed

Jared: I think I get annoyed by the ignorance they have of other tools, which is why programmers seem to keep reinventing the automation wheel.

Nerds have far too much faith in machines and solving problems through programming. Some good things come out, and some massively misguided efforts come out as well. But ultimately, these guys will decide our future, because they will dangle an appealing, human-free testing future, which will be doggedly pursued by management. If there are two choices for how work is designed, and one gives management more control, they’ll choose the latter.

Effectiveness is secondary. They’re not solving my problems. They’re solving developer problems. And management problems that arise due to poor work design. But they’ll probably win. Those of us who are skilled enough and market well will retain decent jobs, but there may not be many of those roles. Mostly people will forget how to do the skilled work. When they say ‘I wanted to help the manual testers’, I always hear ‘I wanted to help the shit testers (Even though they didn’t ask for any help)’.

I really wanted these tools to be of assistance in helping to put a crapload of fake testers out of a job, but I really think the reality will be different. While the bug reporting side of things might help developers fix problems faster and free up testers to do more valuable testing, the automation side of things will a) be farmed out to low-skilled low-paid workers who know enough to hack a little bit of javascript and b) spawn a bunch of new frameworks that deal with the abstraction problem that this tool has inherited from every other record and playback tool of its ilk. In the case of a) it may eliminate a bunch of drones writing test plans and test cases then attempting to execute them, but it’s shifting the problem into the automation sphere.

The fact that Google has released the client but not the server will be a deterrent for some, but it won’t take long before someone puts together a workable open source server and the crap automation tool + low paid maintenance worker farce will have a new player. It might mean that that you won’t need quite as many fake test drones in your organization, but in order to make a dent in the plague that is fake testing, we need something a little better than crap + 1.

The intended audience seemed to be testers who are enslaved by process and useless dogma, though he pitched it as being for testers who are lacking respect. I suppose the latter comes from being the former. I expect that many of those people will be feeling some trepidation having heard this presentation. That’s probably a good thing, but the doom was layered on pretty thick. I thought it was overkill personally.

In my previous post, I wasn’t shy about criticizing the content as I saw it. I’m not backing away from those criticisms either. They stand. That said, there were some cool and useful things that Mr. Whittaker did talk about. It’s just a shame that the vast majority of the points he made were very blinkered and didn’t really extend beyond web tech (especially web tech for large companies). I spent enough time going over that last time. There’s more nonsense I could have a go at, but it wouldn’t serve any real purpose. If you’re curious you can watch it yourself.

I think it’s a little irresponsible to say ‘you’re users are going to suffer anyway, so just get it out there’. That works in some situations – mostly when users are expecting it. When you have customers paying money for a product that they expect to work well, that strategy isn’t so great. The users (rightly) get upset when the software doesn’t do the job. He did justify this point later with ‘use dogfooders and releases to trusted users’, but that’s not going to work across the board.

At Google, the majority of the tools they produce are free for the end user. Their model is based on building things that people will use (for free) and building advertising into it. If it doesn’t work perfectly, oh well. It’s not like you’re paying money for it.

Try that on with a product where people are forking over money. It doesn’t have to be a lot of money. Even $10 per month is enough for someone to become irate when what they expect to work does not. As a company releasing software, you have a duty of care to make sure that software does the job.

Yes of course there will be bugs. You can’t make it ‘perfect’. That’s not the point. The point is that you have taken reasonable steps to make sure that it does the job intended and doesn’t do what it wasn’t. Would you release CAD sofware for engineers with cursory testing because you can easily patch it later? What about drivers for machines used in radiation therapy? Pacemakers? Traffic control? Problems with these don’t just mean users get pissed of and bitch about it on twitter. It means people die or are maimed horrifically.

Yes some of the stuff that Mr. Whittaker talks about is cool and is made by developers and will serve to bring testers closer to both the end user and to developers. That’s awesome and I’m all for that. That doesn’t mean that testing as a sapient vocation is under threat. Fake testers would appear to have a use by date and I’m all for that too. That day can’t come soon enough for me.

Computers are getting better at assisting us to test and yes of course we have the developers who wrote that code to thank for it. For example, the tools Mr. Whittaker spoke about to help end-users report bugs are very cool indeed. I’d love to put something like that in place for the end users of my company’s products. I also agree with him that we should be embracing our users in order to assist us with testing. Select groups of early adopters, dogfooders, people who understand that what they are using is not perfect yet and that we appreciate their feedback. Why wouldn’t you want to tap into that?

There were other worthwhile truisms that Mr. Whittaker brought up in amongst the hand wringing and gnashing of teeth.

Developers can test.
This is true. Testing is part of a developers job. Being able to look at requirements or a story and think about not only how the code should work, but how it should handle failure is something developers should be doing. They should also be harnessing frameworks out there to build checks that save them time down the track. Agreed. That’s part of their job and if they’re doing it, that’s full of win for both them and the testers.

Stop doing stuff you can build a robot for.
I’m on board with this too. Anything that doesn’t require a human to evaluate and that can reasonably be automated, should be.

Testing is not the sole domain of testers. It doesn’t matter who does it, only that it gets done.
I agree with this one too, by and large. You can get into semantics about what testing is and what skill sets are particular to dedicated testers, but yes as long as testing is done to the extent it needs to be in a given context, then fine.

The thing that irked me the most about this presentation is the fallacious premise that you have to have produced something corporeal and tangible in order to be of any worth. This is simply not true. Testing is a cognitive process throughout which you produce information that (among other things) helps bring a piece of software from conception to fruition.

There are places for testers who code and there are places for testers that don’t. The advances that Mr. Whittaker is talking about will make life difficult for (so called) testers who don’t think; for the those who blindly follow a busted process because someone called it ‘best practice’ and they decided to stop learning when they graduated from whatever school they last graduated from. On the other hand, for testers who love to learn, who enjoy new challenges and are not afraid to get their hands dirty there is a bright future. It may be that their job title might move from ‘tester’ to something else. Who cares? They’ll be bringing the same incisive critical thinking and analysis to whatever you decide to call them and they’ll still be testing. Bring it on.

This one is a response to a blog post by Eric Jacobson, who wrote up some notes from James Whittaker’s keynote at STARwest.

I myself did not attend STARwest, so I have only these notes to go on, but as I read them a few things struck me as being odd. Some of Mr. Whittaker’s assertions I agree with, though I think that his reasoning for them is flawed. From Eric’s blog:

testers do not get respect in their current roles…the reason we lack respect is all the recent software quality “game changers” have been thought of by programmers

The game changers that he listed were strange to me. If I read the post correctly, they are there to help back up the assertion that testers should be fearful for their jobs. Let me list them, then list reasons why after most of them my reaction was either ‘meh’, or ‘huh?’

•  Today’s software is easier to update and fix problems in (easier than software from 10 years ago).

There are some words missing in this statement. I don’t know what they are, so let’s try some variations.

Some of today’s software is potentially easier to update an fix problems in (yadda yadda), provided that it is well designed, well structured, well written, well commented and well tested.

or maybe

Today’s software is easier to update and fix problems in (yadda yadda) due to the advancement of IDE’s and debuggers as well as frameworks built into some modern languages that assist structure and testability.

Well okay, both fair points. Both have the potential to increase software quality. I’m not seeing any particular threat to software testing here though.

Threat rating: Meh (with a side of Huh?).

• Crash recovery – some software can fix itself.

Ok. Couple of things

1. So what?
2. ‘fix itself’ is a pretty nebulous statement. Maybe James said more on this and because I missed it, I can’t speak to it specifically.

Netflix servers have the Chaos Monkey that kills processes at random to help them work out how to make their product more robust. That’s cool. Does that mean their testers are on the way out? Other software that crashes can sometimes recover and or send logs back home so stuff can get traced and fixed. That’s also great, but the software has at that point, already crashed. If it keeps running, it may well be in a state that is detrimental to the end user. Whatever. This point is irrelevant to testing getting in the way of quality.

Threat rating: Huh?

• Reduction of dependencies via standards (e.g., most HTML 5.0 websites work on all browsers now).

Would love to see the reference to the stats backing up this statement. Tell it to the banks and insurance companies and other ultra-conservative behemoths mired in their own red tape and bullshit processes. Those poor bastards are still on IE6 or 7. Aah Standards. Yep. Standards are great. I don’t know if this is an attempt to say that cross-browser testing is no longer required. It may make checking functionality across browsers a little easier, but again I’m not seeing a threat to testing here either.

Threat rating: Meh.

• Continuous Builds – quicker availability of builds makes software easier to test but it has nothing to do with testers.

Eric already commented here that build tools have nothing to do with testers – well, not far beyond lighting up a suite of unit tests and potentially helping deliver builds to testers faster.

Threat rating: Meh

• Initial Code Quality (e.g., TDD, unit tests, peer reviews)

These are all great things and if anything are complementary to software testing. TDD and unit testing can help Developers put better code in place sooner and find out about stuff that is broken at the unit level faster. Code reviews have the potential to help coders in the room become better at their craft (and if there are testers in the room, it can help them understand how coders think, how their code is put together and any bad habits that their colleagues might have).
Yep. It’s a positive for code quality. Not seeing any threat to testing here.

Threat rating: Meh

• Convergence of the User and Test Community (e.g., crowd source testing, dog food testing).  Per James, “Testers have to act like users, users don’t have to act.”

Hmm, having had first-hand experience with crowd-sourced testing, I have to say this has a long, long, long way to go before it is any threat to the profession of software testing. It is a useful supplement to software testing, however using it as a replacement for software testing would be a recipe for pain.

Having end users test a site can be great and the feedback you receive can give you some great insights into how they want to use a product and what they want to achieve. To equate that with ‘Users are better because they don’t have to pretend’ is naive. There are a couple of problems. Firstly they’re not trained to look for problems. They may stumble across them and they may find things that your software testers do not. That doesn’t mean that testing is not a worthwhile activity. Secondly, users aren’t trained to describe problems in a concise or logical manner. There’s only so much troubleshooting you can do with the description ‘It’s shit and it doesn’t work. I paid good money for this, so fucking fix it’ (spelling and grammar corrected for the sake of legibility). They don’t generally have access to logs, or provide screenshots or video of how a problem came to be.

So – to my knowledge, not thought of by programmers and not a threat to testing.

Threat rating: Meh.

Eric then lists 4 additional ‘painful facts’ about testing (I’m assuming these were taken directly from Mr .Whittaker’s presentation)

1. Only the software matters.  People care about what was built, not who tested it.

Depends on who you mean by people (and by ‘matters’). End users? Sure – unless the software doesn’t do what they need, or crashes on them constantly. Then they’re all about who tested it. Funny about that. What about other people?
Again – how is this painful? Hindsight is 20/20 and it’s possible that valuable info from a software tester is less valuable to the audience after the fact.  A software tester’s job is to provide information to decision makers to help them make their decisions. Talented testers do this in such a way that the information is timely, relevant and valuable. Do that consistently and you can bet your arse that they’ll care about who tested it.

2. The value of the testing is the activity of testing, not the artifact.  Stop wasting your time creating bug reports and test cases.  Start harnessing the testing that already exists (e.g., beta testing).

The first sentence here is gold. The second sentence is a bit WTF? Mostly because the example given was ‘beta testing’ – were there no other examples of testing given? I’m all for ditching unnecessary test documentation. Do as much as you need and no more. I’m also all for harnessing existing testing (as distinct from checking, which should also be harnessed, but is not the same thing). It also doesn’t mean ‘when you’re done coding, throw it at your users’.

3.The only important artifact is the code.  Want your tests to matter?  Make them part of the code

I wish I had an analogy to better illustrate how useless this statement is. Important to whom? In relation to what other artefacts? This statement is probably referring to ‘checks‘, but it manages to also dismiss other forms of testing with a sweeping blanket statement that doesn’t actually mean anything. Wow.

If you have a bunch of checks, sure. Automate them. Build testability into your code, absolutely. Your code and probably product quality will be better for it and chances are your testers will thank you.

4. Bugs don’t count unless they get fixed.  Don’t waste time logging bugs.  Instead, keep the testers testing.

Aside from the notion that bugs count in any case, I couldn’t agree more, but again I fail to see how is this painful to testing. Log bugs if your developer can’t fix it when you tell em, be that in a bug tracker or on a post-it attached to a story. Whatever works, just get em fixed (you know, unless they don’t actually matter to the people that count).

I disagree with Mr. Whittaker about the reason that there is a lack of respect for software testers in the industry. The ‘game changers’ that he talks about are largely produced by programmers in order to make life easier for … programmers. Sure they’re also generally good for software quality, that’s not why we testers get a bum rap.

The reason we don’t get a whole lot of respect is because the industry is saturated with fake testers. Not only that, but there are complimentary industries *coughcoughcertificationcough* that make a tidy profit churning out crap material to the gullible and the ignorant, helping to produce the next generation of fake testers.

Now it may be that Mr. Whittaker is taking these testers to task when he says that developers are getting better at testing, but testers aren’t getting better at programming and “It’s only important that testing get done, not who gets it done.” If so, then great. I agree. The faster fake testing dies, the sooner the rest of us can get on with it. I might finally be able to hire someone without having to read through the resumes of 300 oxygen thieves.

I do think that his exhortation to “get a specialty and become an expert in some niche of testing (e.g., Security, Internationalization, Accessibility, Privacy) or learn how to code.” is just scaremongering for any tester who knows their craft, but hey, controversy puts bums on seats, so I suppose I can’t fault him too much for that. Sure, become a specialist if you have an interest in it. Knowing how to code certainly won’t hurt you, but I know a number of world-class testers who do not code and are testing generalists. I do not believe their livelihoods are in danger any time soon. As testers, we bring something different to the table. Some programmers are excellent testers. I like working with them. I encourage them to get better at testing. It helps me to help them look good. For coders, being able to think like a tester is a skill worth developing. That goes beyond being able to write unit tests.

So as far as controversial keynote speeches go, this was probably quite entertaining. As far as it containing any real substance to support an assertion that a skilled software tester should be concerned about losing their job to coders though – I just don’t see it.

To me, effective test reporting is the glue that holds it all together. If you can’t (or don’t) report your findings effectively, in a way that your audience finds useful, then your testing is not effective.

It occurs to me that test reporting seems to fit within two categories. I call them Pull reporting and Push reporting.

Pull Reporting is where your audience comes to you for information. Generally they have questions that they want an answer to, such as ‘When will you be finished testing?’, ‘How stable is Product X?’, ‘Did you find any bugs in interface Y?’ and so on.

Setting aside for the moment the quality of these questions, let’s talk a bit about the sort of situations in which pull testing arises. Stand-up meetings, drive-by situation reports, weekly meetings are all examples of pull reporting. In each case you have an audience that (at least to some degree) has context in mind and is requesting specific information. In my experience it’s the drive-by sit-rep that can be the most challenging, often because I’m in the middle of working on something else and I need to context-switch in order to answer. I don’t know about you, but it takes me time to switch out of something I’m deeply focused on in order to answer something unrelated.

In this situation, there are a few things you can employ to buy yourself some time.

• Remember to breathe. If nothing else, oxygen is good for your brain and taking a deep breath gives you a few more seconds to switch over and consider the question posed.
• Ask your audience to repeat the question. If you were in the middle of something else, it’s possible you heard something akin to Charlie Brown’s teacher rather than a question. Ask them to repeat it.
• If you don’t feel like you can effectively answer the question right this second, ask for time. The question might seem urgent, but it’s rare that the situation is so mission critical that you can’t say ‘Give me 5 minutes, I’ll come to your desk (or call you back etc) with what you’re after.
• If you are ready to answer, go ahead and answer. If you are speculating about something, make sure that you identify that you are speculating. ‘I know x, y and z. It may be that a & b, but I’m not sure’
• Poll them to see if you have given them the info they’re after. It shows them you actually care about what they’re asking and it gives you immediate feedback as to whether your report was effective.

The drive-by report is also a great opportunity for an ambush. Knowing that someone is focused elsewhere and will need to context switch – that’s a great time to go on the offensive if you immediately want someone on the back foot. It’s a dick move, but it works. Hopefully you’re not on the receiving end, but should it happen, then remember that not all questions need to be answered. Beware ‘why’ – especially when it’s accompanied by a pronoun as the third word (why did you… why didn’t you… why aren’t you etc). You may find it more constructive to turn those sorts of why questions into ‘what’ questions. ‘What has happened that is concerning you?’ Answer that, and then you can decide whether the why is worth answering. Suggesting a different venue and time for the discussion is probably also a good idea also.

If you do find yourself in a position where you can reel off a report on request, you might like to consider using Michael Kelly’s MCOASTER heuristic. It’s a handy list of items that your audience may be interested in. You don’t have to report on every single item. Pick the ones you think your audience wants to know about (and check in after to see if they want more).

Push reporting is the opposite in terms of who has context in mind. Push reporting is when you have information that you believe your audience should know, but they are not aware of it. In order to effectively convey the information you have, you need to understand what is important to them, what they know already and how to frame your information in such a way that it is easily digestible and meaningful to your audience.

Examples of Push reporting are Bug reports, Reporting showstoppers or mission-critical issues, Risk reporting. Time is generally on your side (though not always). Take the time to make sure that your audience is properly framed. Brief them on your subject matter before you launch into your report. Even if you have an appointment with an agenda for your report, you can’t assume your audience will immediately be ready to hear what you have to say.

Push reporting is where your ability to tell a compelling testing story comes to the fore. Our job is to present meaningful information to people that matter. If all I do is present that information, then I really feel that I’m not living up to my potential as a tester. I would rather present that information in such a way that it presents a compelling call to action.

In my presentation at CAST I used an example from the movie Hunt for Red October. In this clip, the sonar operator aboard the USS Dallas (Jonsey) presents information to the captain about what he has discovered.

The Hunt for Red October: Jonsey Reports

Keep in mind that the raw information that Jonsey has is

• He knows he has heard a submarine that they don’t have previous records of.
• He lost the sub but picked up a sound the computer tells him his magma displacement
• He has done some testing and believes the sound is man made.
• He has picked up the sound on three separate occasions.

That’s it. That’s the raw info he had. Everything else he presented to the captain he has put together from what he thinks that information probably means.

He could have just presented that information alone. If he had, it’s possible that the captain would have extrapolated the rest. They tend not to put morons at the helm of nuclear powered, nuclear armed naval vessels. Still, if this is all the captain gets, then the onus is on him to do the legwork. What Jonsey has done is taken raw information and put it into a format that greatly helps the captain see the significance of the information. The only thing left for the captain to do is to decide whether he buys Jonsey’s version of events and then act accordingly.

This is about as close to the perfect push report as you’re likely to see. It may be that your own push reports aren’t as high-stakes as hunting a rogue Russian sub, but the principle remains. If you have information that your audience needs to hear, take the time to put it into a format that helps them see the significance of that information.

Ask yourself what you want from revealing your information. Do you think certain action should be taken? How does your information support taking that action? Do you want to highlight risks associated with your information? What are some credible issues that could arise as a result? How does your information support that? It may be that you have information that you simply wish to relay in case it is useful. Nothing wrong with that, but think about the information you are presenting and whether or not there is more to it than what you have.

Remember also that however compelling your case, the final decision is unlikely to be yours. Suggest a recommended course of action. Don’t demand. There may be other factors that you are not aware of that have some bearing on the situation. It’s your job to find and present the information. Let them take responsibility for what happens next. That’s why they get paid the big bucks.

Elicit feedback. Just as with pull reporting, you want to find out whether the information you gave was useful. If not, why not? You can take the response you get and feed his back into your testing.

I think that the distinction betwen pull and push reporting is a useful one. It provides me a framework to think about how to report and what to include. I’m curious to hear your thoughts or how you think about test reporting.

What I love about CAST is that there is a very collegial feel to the event. It feels like a safe place to air new thoughts and ideas (or old ones) and discuss them – even have them shot down, in an environment that promotes and supports learning and cooperation.

There were a number of firsts for me personally. I hadn’t presented at CAST before. My presentation went well, I thought. I had a much fuller room than I was expecting. People seemed to find it interesting and useful and there were a lot of good questions asked. It wasn’t perfect. The room’s air con was LOUD, so the set of speakers I brought to play a snippet of video were woefully inadequate. Along the same lines, I wasn’t using the mic to speak and a few people let me know I could have projected a little more. Strange. I don’t have a problem projecting in the dojo. Probably because I’m barking instructions at people. Presenting to a room of testers is a little different it seems. Something for me to work on, then.

Louise Perold and I entered the CAST testing competition – a 4 hour event to test the abilities and skills of the testers that entered. Louise and I walked back to the hotel lamenting what else we could have done if we’d just had a little longer. We had some awesome competition. People that I know by reputation if not personally and all of them excellent testers. When on the following day James Bach announced we’d won the award for best tester/programmer relations we were happy. That was something we thought we had done well. When he went on to tell us we’d won the overall competition, we were stunned and delighted in equal parts. Louise and I will blog more about this at some point soon.

Lastly, I got chatting to Michael Bolton about my thoughts on test framing and how I thought it related to audience framing (I have another post coming on this topic too). He invited me to present my point of view at his tutorial on Test Framing, which I did. I was grateful for the opportunity and a number of people said they got something out of it, so that was something else I was happy about.

The thing I love most about CAST is the interaction between people. Because I’m somewhat geographically challenged, I get to one testing conference per year. I choose CAST because of the people who attend. Because they challenge me and make me think and remind me of the power of diversity and having skilled, free-thinking people working together.

I was super happy to spend time with the Usual Suspects (the people I generally hang out with at CAST) – Henrik Andersson, Chris Blain, Tim Coulter, Carsten Feilberg, Johan Jonasson, and Lou Perold. I also met and got to know a bunch of other cool people (most of them Swedes – it was a real invasion on their part) Robert Bergqvist, Maria Kedomo, Mattius Gustavsson, Sigge Birgisson.

As always seems to happen, there were a great number of people I would love to have spent more time with. I met in person a number of people I’d only previously interacted with online. There are a number of people that I’d met before, but haven’t spent nearly as much time chatting to as I’d like and a few I wanted to meet, but didn’t get to. If I try and name everyone, I will undoubtedly forget someone. Suffice it to say there were a massive number of testers I’d have loved to spend more time with. I suspect that number will only be bigger this time next year.

I didn’t get to see all of the presentations I wanted to see. Some of them were captured on video (the emerging topics tracks and lightning talks among them, I believe). I plan on taking my camera next year to capture stuff I can’t make it to. If you didn’t make it to cast, I highly recommend you check them out.

The organizers are to be commended. Jon and James Bach were the most conspicuous of the organizers, but Paul Holland and his crew of facilitators and all the volunteers who ran around doing stuff behind the scenes are to be commended on a job well done. Volunteers are the unsung heroes of events like this. When things go well, you tend not to notice them. Without them though, this event would be near impossible.

There are a few improvements that could be made. I’d like to echo some of Michael Hunter’s sentiments. A lot of people are okay with a high-carb diet. I’m not one of them.

The distance between hotels made meeting people who weren’t at my hotel kinda difficult (though we managed to find pubs here and there that did the job).

As far as content goes, I got new ideas from the vast majority of the talks I attended. I must admit to being a little disappointed by the debate between James Bach and Doug Hoffman. I was hoping the format would be more of a formal debate arguing for and against a specific statement, but it turned out to be more of an argument/discussion with the interjections and interplay that tend to go hand-in-hand. I also thought Doug was more gentle with James than he deserved. I think this sort of event has potential. I’d like to see a change to a formal debating format, but having a relevant topic debated by some heavyweights in the industry would be very interesting.

I could create a list of questions and great quotes as Michael Hunter has done, but I think those people tweeting to the #CAST2011 hashtag took care of that most effectively (try searching on it). I’m glad they did, because I barely had time to tweet a thing for the whole event.

I’m hoping very much that I’ll be able to attend (and perhaps even present) at next year’s CAST, but that depends on a number of variables that are somewhat beyond my control. If you are considering attending next year, I recommend that you do in the strongest terms possible.

I might get one in twenty resumes that I don’t immediately bin. Out of those I keep, I’ll probably end up interviewing one in five.

For the sake of my own sanity I’m going to list all the things that will get your resume binned (by me at least) and some things you can do to increase the chance that not only will you be interviewed, but that you will be hired.

I don’t care how well you’ve copied that stock standard resume you googled. Most of the stuff online is fucking awful anyway. Here’s how your standard crappy resume goes:

Name and particulars
Some sort of generic mission statement about how you want to do your best for <insert company name here>
Description of tertiary education
Description of previous positions held describing your responsibilities in bullet points

There are variants which include
a bullet point list of skills with some context-free rating of competence
a list of desired positions that doesn’t include ‘software tester’ (note – if you want to be a project manager when you grow up, I don’t care, but if you’re applying for a software tester role and the ‘desired role’ field doesn’t have ‘software tester’ in it then you get binned)

These painfully generic resumes make my eyes bleed and contain absolutely nothing that tells me whether or not you might be a good fit for the role.
Moreover, these resumes are often riddled with spelling and grammatical errors. If I see these, you get binned. I consider your resume your first work product. I’m looking for people who are detail oriented and have pride in their work. If you can’t be bothered to get this right, then the job I’m offering is not for you.
Don’t tell me about how you know you’re a perfect fit for the role. That’s not for you to decide. There’s being confident and then there’s being a knob. If you’re a knob, you go in the bin.

If you’re going to go to the trouble of making a resume – something that you are hoping will make you stand out from every other person that applies for the role, why would you go out of your way to make your resume as close as you can to every other resume you’ve seen? I recommend in the strongest possible terms that you buy and read a book called ‘Rites of Passage’ by John Lucht. Read it all and pay special attention to the section on putting a resume together.

As for what I want to see in your resume, I don’t want to know what you did, I want to know what you achieved. What did you do that made a difference to the company?
Tell me what skills you have and tell me which ones you regularly use. Even better, point me at examples of your work,  blog posts discussing your skills, posts to stack exchange where you’ve helped people out with the skills you’ve listed. The less work you make me do to convince me you’re worth hiring, the better. If you don’t have a lot of experience, be up-front about it (and show me what you’re doing about it. Tell me about an open source project you’re working on to improve your skills, or a software testing conference you attended recently). If you haven’t done anything to make yourself appealing you shouldn’t be surprised when you’re passed over.

Okay, let’s pretend you’ve sent me a resume that offers me the faintest glimmer that you’re someone I might be able to spend 40-60 hours a week with. The next thing I’m going to do is give you a questionnaire that shows me how you think. It used to surprise me the number of people that plagiarised their answers. Don’t do this. I know how to use the internets too and I will find out. When that happens, I will tell everyone I know and like not to hire a person called <your name here> because s/he is dishonest. Which also sucks for anyone that shares your name. I’ve considered posting a wall of shame on my blog for would be testers who are dishonest and stupid, but I’m not that much of a bastard yet.

So – answer the questions in your own words. Say what you think, not what you think I want to hear. If you think a question makes no sense or is not relevant, say so or ask for qualification. I’ve been known to ask stupid questions on purpose to see if a candidate will call me on it. Almost no one does. Don’t play buzzword bingo and don’t use nebulous, meaningless phrases unless you’re prepared to define what they mean to you.

Okay then. So assuming you’re not a plagiarist and you can think for yourself, we might get to spend some time chatting face-to-face or over the phone. I’m going to ask you a bunch more questions. I’m still looking to make sure you know how to think, and have an aptitude for the kind of testing that I will want you to do. I’m also looking to make sure you’re not a psycho and that you’ll be a good cultural fit for the team. If you’re a dick, I don’t care how mad your skillz are; we won’t be working together. If I like you and the people I work with like you then you’ll probably get an offer. There’s a lot of work in between an application and an offer though. Perhaps I should know better by now, but I live in hope that the next resume across my desk will be from someone that takes this stuff to heart. I won’t be holding my breath though.

EDIT:
Here’s what some other people have to say about hiring testers
Paul Carvalho

I had just such an experience recently over on UTest’s blog where Rex Black of ISTQB fame was interviewed for ‘Testing the Limits’. Mr Black was hopeful of conversation between himself and readers of the UTest blog. Being a reader and having a number of questions I was happy to oblige.

Unfortunately the dialogue has, at least thus far, not been what I was hoping for. Now I don’t want to be unfair. Mr Black is a busy man. He did go into quite some detail on a number of the questions I asked. He indicated his lack of time a number of times during his replies. I had a number of other questions and made several statements though that I was hoping would prompt further dialogue. I’d like to think he will at some point he will take a well earned breather and spare a few minutes to come back and continue.

He was kind enough to point me in the direction of their psychographic analysts. I did put in an a request to them for more information, but have thus far received no response. Should that change, I will be sure to amend this paragraph. (Mr. Black, if you are reading this, perhaps you would be so kind as to have a word on my behalf and have them forward the relevant materials)

In the event that the conversation moves over here, let me list the areas where I would like to continue discussion:

What do you think the value is (to the certificate holder) of a certificate that pretty much anyone can achieve after a multiple choice exam that involves no practical examination of testing?

Would the testing industry not be better served by an examination process that rigorously tests the candidate’s ability to apply in pratice the theory and techniques they have learned, even if such a process was time consuming, difficult and did not scale well?

Wouldn’t the ISTQB serve the industry better by helping busineses realise that excellent testing is difficult to do well and that not everyone is cut out to be a professional tester?

There were a couple of other questions that I was waiting to ask. I wanted to get the current set squared away before I continued, but since I am not sure when this discussion will continue, let me state them here.

You say that ISTQB’s competitors have obvious commercial interests. Is the ISTQB a not-for-profit organisation? If that is the case, what happens with the $100 or so that examinees pay for their certification. 160,000 x $100 is not a small sum of money, even over 10 years.

I should say that these last couple of questions are not only for Mr. Black, but are open to anyone from the ISTQB who can authoratively answer them.

Once again, I’d like to invite you to read through this PDF, at least slides 8 to 14, but the rest is very worthwhile also. There are questions and assertions there that I think you really need to answer, for the sake of your credibility if nothing else.

The floor is yours.

It’s taken me longer than I thought to come up to speed. I’ve been scraping the rust off my unix-fu and with unfettered access to the codebase and the command line, I feel like a long-lost missing limb has been reattached. Between that and the whole ‘learning new stuff’ thing, I’ve not had much time for anything else. There seems to be light at the end of the tunnel though. I’m grokking more of the code base and starting to make some useful code contributions.

One of the things that I’m trying to do at the moment is put a little bit of structure in the way that the business group and the development group communicate. Right now there is lots and lots of communication, which is great. The confusion comes in where the lines blur between the business telling dev what they want and going to the extent of speccing out everything in such minute detail that there’s not much design work left to be done by the dev group. That might not really sound like a problem to some, but in practice I’ve found that it can make the dev guys feel micromanaged or like their opinion is not required/valued – especially when design-level decisions are architecturally difficult to implement. If non-tech design reaches too far and gets it wrong, it can mean wasted time and effort either for them when the developer rejects it, or for both if the developer codes it in order to prove the point. The thing I love about my current place is that there are no passengers, and this stuff tends to get worked out, but I think I can do more.

This is where I am hoping that Cucumber will help out. If I can put together a series of tests that step out the current behaviour with a DSL that both the business side and tech side can build their communication around, then I suspect we will find it a little easier to step through the desired behaviour in enough detail to be clear about functionality while leaving the dev guys to work out the implementation. It’s not the only reason I want to use Cucumber, but I do have high hopes in this area. It probably won’t hurt that people will be able to see this stuff turn into checks that can be executed automatically. The key is still in the communication. I’m looking to make the clarity of that communication easier.

Cucumber is new tech to me as well. Conceptually I get it, but it’s different when you start to get your hands dirty. After a lot of reading, I decided to go with Cucumber, Capybara and at this stage, I think a combination of Selenium and TestUnit (Ruby’s in-built unit testing framework). I’m a Capybara noob as well, so I’m up to my elbows in new stuff and loving it. Mostly. Sorta. Apart from the frustrating bits.

Almost the entirety of the literature I’ve found on this setup is for BDD – which is to be expected, given that’s what its for. All of the setup and how-to stuff is for building stuff that doesn’t exist yet. Great if you’re building from scratch, but not super helpful if you have an existing code base you want to fit tests to.

I had an utter bastard of a time getting Cucumber and Capybara to work at all. Granted that it’s probably because I’m working with code that does a few unusual things, but I didn’t expect quite as much screwing around as I had to do – so, I’m going to detail it here in case there are any other poor bastards out there who are trying something similar. Hopefully this will save you some frustration.

My setup is (as of writing) Ubuntu 10.4, using rails 2. I already have selenium installed, and the installation of that is not arduous, so I’m going to omit that part.

I installed the following gems:

libxml2 libxml2-dev libxslt1-dev
gherkin cucumber cucumber-rails nokogiri capybara database-cleaner

I’ve listed some that should be pulled in automatically by other gems’ dependencies. For instance, gherkin should be installed when installing cucumber, but for some reason I had to do it manually. YMMV.

After installing the requisite gems, it should be a simple matter of generating all of the cucumber goodness from the root of your rails project

(ruby) script/generate cucumber –capybara
(you can add other options if you want to have them built in)

If all your bits and pieces are installed correctly, you should see a bunch of files and directories get created.

Here are some things that tripped me up:
I thought I might do some work with rspec as well, so I pulled those down. If you’re thinking along the same lines, then Cucumber seems to barf if you install anything beyond rspec 1.3.3
I also had to mess around with the dependencies for the json gem in my project as cucumber seems quite particular about wanting 1.4.6

By default, cucumber will clean the database after each run (as is generally considered good automation practice). In my case however, I have enough stuff in the db that it doesn’t make sense right now to load from scratch or mock the bits I need. Maybe down the track but for the moment I’ve just switched this off in the environment file (more on this shortly).

Anyway, moving on. From root, there should now be a features directory. edit features/support/env.rb
You can either make your settings changes in here, or in another file and require it from here. This file is automatically generated, so if you plan on making a lot of changes here or you think you’ll have cause to regenerate this file, the latter might be a better option.

Here are the changes I put in

Capybara.default_selector = :css
Capybara.run_server = false
Capybara.app_host = <url of my testing area>
Capybara.javascript_driver = :selenium

One of the things that really screwed me around was not realising that Cucumber starts its own webserver and runs on that. I tried pointing it at the server instance that Rails kicks off, only to find that things were falling over and there was nothing in my logs to tell me why. I felt like a complete moron once I did find out, but then again, the docco in this area appears to be awfully light so I don’t feel too bad. Anyway – be warned. If you don’t want Cucumber to use its own webserver, you need to tell it so explicitly.

As I said, there’s bugger all out there right now on retrofitting scripts for an existing code base. It sounds like it’d be easy, right? The code and functionality is already there, so skip the red/green/refactor bit and go straight to green, right? Not really.

Rather than conceptually pre-fabricating your functionality, you get to look through your existing functionality, decide what is most important to test, work out where you need to retroactively install test hooks and build use cases that serve a business purpose, while not getting lost down a rabbit hole around functionality that currently exists, but perhaps doesn’t work quite how it should. Yay! Given that there is the potential for test to ‘just light up green’ when you write them, I’ve also found it a good idea to write checks that fail so you know they’re actually doing something, then correcting them – it sort of keeps the spirit of BDD/TDD even if you’re not driving any development just at this point.

Retrofitting Cucumber tests to your site is not as simple as it might sound. I am hopeful that it will be rewarding. It’s a little too early to tell just yet, but the signs are positive. If there’s any interest, I may follow up with a post about some more of the specifics of the actual hands-on retrofitting bit. If you have questions, fire away.