If you’re keen to attend, then pricing details are here and you can sign up here. If you make it, be sure to say hi.

Fake testing is the blind worship of process and procedure (over the sapient and conscientious application of skill). There’s been some talk of its demise recently and while I want to believe we’re heading in the right direction, I also want to keep in mind that this is not an easy problem to solve and if we’re not careful we won’t solve the problem, we’ll just shift it elsewhere.

Let me give you an example.

At GTAC, there were some very interesting tools demo’ed. I looked at BITE (Browser Integrated Testing Environment) and thought – yeah, there’s another nail in the fake testing coffin. I particularly like the concept of making it easy for the end-user to report bugs that contain information that programmers can use to solve issues quickly.

As I continued to watch, something nagged at me. A sense that something was not quite right. Not quite deja vu, but certainly the sense that if it were this easy, we’d have done it before.

Shortly thereafter, I was chatting with good friend and colleague Jared Quinert. I had forgotten how good he is at taking a subject, stripping back all the bullshit and laying its foundations bare. If you ever get the chance to work with, or hire him. Do it. Seriously. As usual, Jared locked onto what I’d been feeling uneasy about without me even having to explain myself. Having shared the link to the talk I’ve linked to at the start of this post, our conversation went something like this:

Jared: Record playback software is apparently central to the life of a manual tester…

Ben: Yeah I chuckled at that too.

Jared: Recording scripts, converting to English doesn’t solve the abstraction problem. Waiting to hear them elaborate on re-recording being cheaper than maintaining…and they’re still just talking about QTP features, but nobody has piped up with this QTP can re-learn missing controls…

Ben: True

Jared: Tool running in the browser probably creates other issues too I would think.

Ben: Likely yes. That and when you have several thousand tests and you need to re-record a couple hundred, the appeal wears thin.

Jared: The abstraction is the problem, not the approach and as always, I’m skeptical that the test can simply be replayed without any issues when there is dynamic content.

Ben: Agreed

Jared: I think I get annoyed by the ignorance they have of other tools, which is why programmers seem to keep reinventing the automation wheel.

Nerds have far too much faith in machines and solving problems through programming. Some good things come out, and some massively misguided efforts come out as well. But ultimately, these guys will decide our future, because they will dangle an appealing, human-free testing future, which will be doggedly pursued by management. If there are two choices for how work is designed, and one gives management more control, they’ll choose the latter.

Effectiveness is secondary. They’re not solving my problems. They’re solving developer problems. And management problems that arise due to poor work design. But they’ll probably win. Those of us who are skilled enough and market well will retain decent jobs, but there may not be many of those roles. Mostly people will forget how to do the skilled work. When they say ‘I wanted to help the manual testers’, I always hear ‘I wanted to help the shit testers (Even though they didn’t ask for any help)’.

I really wanted these tools to be of assistance in helping to put a crapload of fake testers out of a job, but I really think the reality will be different. While the bug reporting side of things might help developers fix problems faster and free up testers to do more valuable testing, the automation side of things will a) be farmed out to low-skilled low-paid workers who know enough to hack a little bit of javascript and b) spawn a bunch of new frameworks that deal with the abstraction problem that this tool has inherited from every other record and playback tool of its ilk. In the case of a) it may eliminate a bunch of drones writing test plans and test cases then attempting to execute them, but it’s shifting the problem into the automation sphere.

The fact that Google has released the client but not the server will be a deterrent for some, but it won’t take long before someone puts together a workable open source server and the crap automation tool + low paid maintenance worker farce will have a new player. It might mean that that you won’t need quite as many fake test drones in your organization, but in order to make a dent in the plague that is fake testing, we need something a little better than crap + 1.

The intended audience seemed to be testers who are enslaved by process and useless dogma, though he pitched it as being for testers who are lacking respect. I suppose the latter comes from being the former. I expect that many of those people will be feeling some trepidation having heard this presentation. That’s probably a good thing, but the doom was layered on pretty thick. I thought it was overkill personally.

In my previous post, I wasn’t shy about criticizing the content as I saw it. I’m not backing away from those criticisms either. They stand. That said, there were some cool and useful things that Mr. Whittaker did talk about. It’s just a shame that the vast majority of the points he made were very blinkered and didn’t really extend beyond web tech (especially web tech for large companies). I spent enough time going over that last time. There’s more nonsense I could have a go at, but it wouldn’t serve any real purpose. If you’re curious you can watch it yourself.

I think it’s a little irresponsible to say ‘you’re users are going to suffer anyway, so just get it out there’. That works in some situations – mostly when users are expecting it. When you have customers paying money for a product that they expect to work well, that strategy isn’t so great. The users (rightly) get upset when the software doesn’t do the job. He did justify this point later with ‘use dogfooders and releases to trusted users’, but that’s not going to work across the board.

At Google, the majority of the tools they produce are free for the end user. Their model is based on building things that people will use (for free) and building advertising into it. If it doesn’t work perfectly, oh well. It’s not like you’re paying money for it.

Try that on with a product where people are forking over money. It doesn’t have to be a lot of money. Even $10 per month is enough for someone to become irate when what they expect to work does not. As a company releasing software, you have a duty of care to make sure that software does the job.

Yes of course there will be bugs. You can’t make it ‘perfect’. That’s not the point. The point is that you have taken reasonable steps to make sure that it does the job intended and doesn’t do what it wasn’t. Would you release CAD sofware for engineers with cursory testing because you can easily patch it later? What about drivers for machines used in radiation therapy? Pacemakers? Traffic control? Problems with these don’t just mean users get pissed of and bitch about it on twitter. It means people die or are maimed horrifically.

Yes some of the stuff that Mr. Whittaker talks about is cool and is made by developers and will serve to bring testers closer to both the end user and to developers. That’s awesome and I’m all for that. That doesn’t mean that testing as a sapient vocation is under threat. Fake testers would appear to have a use by date and I’m all for that too. That day can’t come soon enough for me.

Computers are getting better at assisting us to test and yes of course we have the developers who wrote that code to thank for it. For example, the tools Mr. Whittaker spoke about to help end-users report bugs are very cool indeed. I’d love to put something like that in place for the end users of my company’s products. I also agree with him that we should be embracing our users in order to assist us with testing. Select groups of early adopters, dogfooders, people who understand that what they are using is not perfect yet and that we appreciate their feedback. Why wouldn’t you want to tap into that?

There were other worthwhile truisms that Mr. Whittaker brought up in amongst the hand wringing and gnashing of teeth.

Developers can test.
This is true. Testing is part of a developers job. Being able to look at requirements or a story and think about not only how the code should work, but how it should handle failure is something developers should be doing. They should also be harnessing frameworks out there to build checks that save them time down the track. Agreed. That’s part of their job and if they’re doing it, that’s full of win for both them and the testers.

Stop doing stuff you can build a robot for.
I’m on board with this too. Anything that doesn’t require a human to evaluate and that can reasonably be automated, should be.

Testing is not the sole domain of testers. It doesn’t matter who does it, only that it gets done.
I agree with this one too, by and large. You can get into semantics about what testing is and what skill sets are particular to dedicated testers, but yes as long as testing is done to the extent it needs to be in a given context, then fine.

The thing that irked me the most about this presentation is the fallacious premise that you have to have produced something corporeal and tangible in order to be of any worth. This is simply not true. Testing is a cognitive process throughout which you produce information that (among other things) helps bring a piece of software from conception to fruition.

There are places for testers who code and there are places for testers that don’t. The advances that Mr. Whittaker is talking about will make life difficult for (so called) testers who don’t think; for the those who blindly follow a busted process because someone called it ‘best practice’ and they decided to stop learning when they graduated from whatever school they last graduated from. On the other hand, for testers who love to learn, who enjoy new challenges and are not afraid to get their hands dirty there is a bright future. It may be that their job title might move from ‘tester’ to something else. Who cares? They’ll be bringing the same incisive critical thinking and analysis to whatever you decide to call them and they’ll still be testing. Bring it on.

This one is a response to a blog post by Eric Jacobson, who wrote up some notes from James Whittaker’s keynote at STARwest.

I myself did not attend STARwest, so I have only these notes to go on, but as I read them a few things struck me as being odd. Some of Mr. Whittaker’s assertions I agree with, though I think that his reasoning for them is flawed. From Eric’s blog:

testers do not get respect in their current roles…the reason we lack respect is all the recent software quality “game changers” have been thought of by programmers

The game changers that he listed were strange to me. If I read the post correctly, they are there to help back up the assertion that testers should be fearful for their jobs. Let me list them, then list reasons why after most of them my reaction was either ‘meh’, or ‘huh?’

•  Today’s software is easier to update and fix problems in (easier than software from 10 years ago).

There are some words missing in this statement. I don’t know what they are, so let’s try some variations.

Some of today’s software is potentially easier to update an fix problems in (yadda yadda), provided that it is well designed, well structured, well written, well commented and well tested.

or maybe

Today’s software is easier to update and fix problems in (yadda yadda) due to the advancement of IDE’s and debuggers as well as frameworks built into some modern languages that assist structure and testability.

Well okay, both fair points. Both have the potential to increase software quality. I’m not seeing any particular threat to software testing here though.

Threat rating: Meh (with a side of Huh?).

• Crash recovery – some software can fix itself.

Ok. Couple of things

1. So what?
2. ‘fix itself’ is a pretty nebulous statement. Maybe James said more on this and because I missed it, I can’t speak to it specifically.

Netflix servers have the Chaos Monkey that kills processes at random to help them work out how to make their product more robust. That’s cool. Does that mean their testers are on the way out? Other software that crashes can sometimes recover and or send logs back home so stuff can get traced and fixed. That’s also great, but the software has at that point, already crashed. If it keeps running, it may well be in a state that is detrimental to the end user. Whatever. This point is irrelevant to testing getting in the way of quality.

Threat rating: Huh?

• Reduction of dependencies via standards (e.g., most HTML 5.0 websites work on all browsers now).

Would love to see the reference to the stats backing up this statement. Tell it to the banks and insurance companies and other ultra-conservative behemoths mired in their own red tape and bullshit processes. Those poor bastards are still on IE6 or 7. Aah Standards. Yep. Standards are great. I don’t know if this is an attempt to say that cross-browser testing is no longer required. It may make checking functionality across browsers a little easier, but again I’m not seeing a threat to testing here either.

Threat rating: Meh.

• Continuous Builds – quicker availability of builds makes software easier to test but it has nothing to do with testers.

Eric already commented here that build tools have nothing to do with testers – well, not far beyond lighting up a suite of unit tests and potentially helping deliver builds to testers faster.

Threat rating: Meh

• Initial Code Quality (e.g., TDD, unit tests, peer reviews)

These are all great things and if anything are complementary to software testing. TDD and unit testing can help Developers put better code in place sooner and find out about stuff that is broken at the unit level faster. Code reviews have the potential to help coders in the room become better at their craft (and if there are testers in the room, it can help them understand how coders think, how their code is put together and any bad habits that their colleagues might have).
Yep. It’s a positive for code quality. Not seeing any threat to testing here.

Threat rating: Meh

• Convergence of the User and Test Community (e.g., crowd source testing, dog food testing).  Per James, “Testers have to act like users, users don’t have to act.”

Hmm, having had first-hand experience with crowd-sourced testing, I have to say this has a long, long, long way to go before it is any threat to the profession of software testing. It is a useful supplement to software testing, however using it as a replacement for software testing would be a recipe for pain.

Having end users test a site can be great and the feedback you receive can give you some great insights into how they want to use a product and what they want to achieve. To equate that with ‘Users are better because they don’t have to pretend’ is naive. There are a couple of problems. Firstly they’re not trained to look for problems. They may stumble across them and they may find things that your software testers do not. That doesn’t mean that testing is not a worthwhile activity. Secondly, users aren’t trained to describe problems in a concise or logical manner. There’s only so much troubleshooting you can do with the description ‘It’s shit and it doesn’t work. I paid good money for this, so fucking fix it’ (spelling and grammar corrected for the sake of legibility). They don’t generally have access to logs, or provide screenshots or video of how a problem came to be.

So – to my knowledge, not thought of by programmers and not a threat to testing.

Threat rating: Meh.

Eric then lists 4 additional ‘painful facts’ about testing (I’m assuming these were taken directly from Mr .Whittaker’s presentation)

1. Only the software matters.  People care about what was built, not who tested it.

Depends on who you mean by people (and by ‘matters’). End users? Sure – unless the software doesn’t do what they need, or crashes on them constantly. Then they’re all about who tested it. Funny about that. What about other people?
Again – how is this painful? Hindsight is 20/20 and it’s possible that valuable info from a software tester is less valuable to the audience after the fact.  A software tester’s job is to provide information to decision makers to help them make their decisions. Talented testers do this in such a way that the information is timely, relevant and valuable. Do that consistently and you can bet your arse that they’ll care about who tested it.

2. The value of the testing is the activity of testing, not the artifact.  Stop wasting your time creating bug reports and test cases.  Start harnessing the testing that already exists (e.g., beta testing).

The first sentence here is gold. The second sentence is a bit WTF? Mostly because the example given was ‘beta testing’ – were there no other examples of testing given? I’m all for ditching unnecessary test documentation. Do as much as you need and no more. I’m also all for harnessing existing testing (as distinct from checking, which should also be harnessed, but is not the same thing). It also doesn’t mean ‘when you’re done coding, throw it at your users’.

3.The only important artifact is the code.  Want your tests to matter?  Make them part of the code

I wish I had an analogy to better illustrate how useless this statement is. Important to whom? In relation to what other artefacts? This statement is probably referring to ‘checks‘, but it manages to also dismiss other forms of testing with a sweeping blanket statement that doesn’t actually mean anything. Wow.

If you have a bunch of checks, sure. Automate them. Build testability into your code, absolutely. Your code and probably product quality will be better for it and chances are your testers will thank you.

4. Bugs don’t count unless they get fixed.  Don’t waste time logging bugs.  Instead, keep the testers testing.

Aside from the notion that bugs count in any case, I couldn’t agree more, but again I fail to see how is this painful to testing. Log bugs if your developer can’t fix it when you tell em, be that in a bug tracker or on a post-it attached to a story. Whatever works, just get em fixed (you know, unless they don’t actually matter to the people that count).

I disagree with Mr. Whittaker about the reason that there is a lack of respect for software testers in the industry. The ‘game changers’ that he talks about are largely produced by programmers in order to make life easier for … programmers. Sure they’re also generally good for software quality, that’s not why we testers get a bum rap.

The reason we don’t get a whole lot of respect is because the industry is saturated with fake testers. Not only that, but there are complimentary industries *coughcoughcertificationcough* that make a tidy profit churning out crap material to the gullible and the ignorant, helping to produce the next generation of fake testers.

Now it may be that Mr. Whittaker is taking these testers to task when he says that developers are getting better at testing, but testers aren’t getting better at programming and “It’s only important that testing get done, not who gets it done.” If so, then great. I agree. The faster fake testing dies, the sooner the rest of us can get on with it. I might finally be able to hire someone without having to read through the resumes of 300 oxygen thieves.

I do think that his exhortation to “get a specialty and become an expert in some niche of testing (e.g., Security, Internationalization, Accessibility, Privacy) or learn how to code.” is just scaremongering for any tester who knows their craft, but hey, controversy puts bums on seats, so I suppose I can’t fault him too much for that. Sure, become a specialist if you have an interest in it. Knowing how to code certainly won’t hurt you, but I know a number of world-class testers who do not code and are testing generalists. I do not believe their livelihoods are in danger any time soon. As testers, we bring something different to the table. Some programmers are excellent testers. I like working with them. I encourage them to get better at testing. It helps me to help them look good. For coders, being able to think like a tester is a skill worth developing. That goes beyond being able to write unit tests.

So as far as controversial keynote speeches go, this was probably quite entertaining. As far as it containing any real substance to support an assertion that a skilled software tester should be concerned about losing their job to coders though – I just don’t see it.

To me, effective test reporting is the glue that holds it all together. If you can’t (or don’t) report your findings effectively, in a way that your audience finds useful, then your testing is not effective.

It occurs to me that test reporting seems to fit within two categories. I call them Pull reporting and Push reporting.

Pull Reporting is where your audience comes to you for information. Generally they have questions that they want an answer to, such as ‘When will you be finished testing?’, ‘How stable is Product X?’, ‘Did you find any bugs in interface Y?’ and so on.

Setting aside for the moment the quality of these questions, let’s talk a bit about the sort of situations in which pull testing arises. Stand-up meetings, drive-by situation reports, weekly meetings are all examples of pull reporting. In each case you have an audience that (at least to some degree) has context in mind and is requesting specific information. In my experience it’s the drive-by sit-rep that can be the most challenging, often because I’m in the middle of working on something else and I need to context-switch in order to answer. I don’t know about you, but it takes me time to switch out of something I’m deeply focused on in order to answer something unrelated.

In this situation, there are a few things you can employ to buy yourself some time.

• Remember to breathe. If nothing else, oxygen is good for your brain and taking a deep breath gives you a few more seconds to switch over and consider the question posed.
• Ask your audience to repeat the question. If you were in the middle of something else, it’s possible you heard something akin to Charlie Brown’s teacher rather than a question. Ask them to repeat it.
• If you don’t feel like you can effectively answer the question right this second, ask for time. The question might seem urgent, but it’s rare that the situation is so mission critical that you can’t say ‘Give me 5 minutes, I’ll come to your desk (or call you back etc) with what you’re after.
• If you are ready to answer, go ahead and answer. If you are speculating about something, make sure that you identify that you are speculating. ‘I know x, y and z. It may be that a & b, but I’m not sure’
• Poll them to see if you have given them the info they’re after. It shows them you actually care about what they’re asking and it gives you immediate feedback as to whether your report was effective.

The drive-by report is also a great opportunity for an ambush. Knowing that someone is focused elsewhere and will need to context switch – that’s a great time to go on the offensive if you immediately want someone on the back foot. It’s a dick move, but it works. Hopefully you’re not on the receiving end, but should it happen, then remember that not all questions need to be answered. Beware ‘why’ – especially when it’s accompanied by a pronoun as the third word (why did you… why didn’t you… why aren’t you etc). You may find it more constructive to turn those sorts of why questions into ‘what’ questions. ‘What has happened that is concerning you?’ Answer that, and then you can decide whether the why is worth answering. Suggesting a different venue and time for the discussion is probably also a good idea also.

If you do find yourself in a position where you can reel off a report on request, you might like to consider using Michael Kelly’s MCOASTER heuristic. It’s a handy list of items that your audience may be interested in. You don’t have to report on every single item. Pick the ones you think your audience wants to know about (and check in after to see if they want more).

Push reporting is the opposite in terms of who has context in mind. Push reporting is when you have information that you believe your audience should know, but they are not aware of it. In order to effectively convey the information you have, you need to understand what is important to them, what they know already and how to frame your information in such a way that it is easily digestible and meaningful to your audience.

Examples of Push reporting are Bug reports, Reporting showstoppers or mission-critical issues, Risk reporting. Time is generally on your side (though not always). Take the time to make sure that your audience is properly framed. Brief them on your subject matter before you launch into your report. Even if you have an appointment with an agenda for your report, you can’t assume your audience will immediately be ready to hear what you have to say.

Push reporting is where your ability to tell a compelling testing story comes to the fore. Our job is to present meaningful information to people that matter. If all I do is present that information, then I really feel that I’m not living up to my potential as a tester. I would rather present that information in such a way that it presents a compelling call to action.

In my presentation at CAST I used an example from the movie Hunt for Red October. In this clip, the sonar operator aboard the USS Dallas (Jonsey) presents information to the captain about what he has discovered.

The Hunt for Red October: Jonsey Reports

Keep in mind that the raw information that Jonsey has is

• He knows he has heard a submarine that they don’t have previous records of.
• He lost the sub but picked up a sound the computer tells him his magma displacement
• He has done some testing and believes the sound is man made.
• He has picked up the sound on three separate occasions.

That’s it. That’s the raw info he had. Everything else he presented to the captain he has put together from what he thinks that information probably means.

He could have just presented that information alone. If he had, it’s possible that the captain would have extrapolated the rest. They tend not to put morons at the helm of nuclear powered, nuclear armed naval vessels. Still, if this is all the captain gets, then the onus is on him to do the legwork. What Jonsey has done is taken raw information and put it into a format that greatly helps the captain see the significance of the information. The only thing left for the captain to do is to decide whether he buys Jonsey’s version of events and then act accordingly.

This is about as close to the perfect push report as you’re likely to see. It may be that your own push reports aren’t as high-stakes as hunting a rogue Russian sub, but the principle remains. If you have information that your audience needs to hear, take the time to put it into a format that helps them see the significance of that information.

Ask yourself what you want from revealing your information. Do you think certain action should be taken? How does your information support taking that action? Do you want to highlight risks associated with your information? What are some credible issues that could arise as a result? How does your information support that? It may be that you have information that you simply wish to relay in case it is useful. Nothing wrong with that, but think about the information you are presenting and whether or not there is more to it than what you have.

Remember also that however compelling your case, the final decision is unlikely to be yours. Suggest a recommended course of action. Don’t demand. There may be other factors that you are not aware of that have some bearing on the situation. It’s your job to find and present the information. Let them take responsibility for what happens next. That’s why they get paid the big bucks.

Elicit feedback. Just as with pull reporting, you want to find out whether the information you gave was useful. If not, why not? You can take the response you get and feed his back into your testing.

I think that the distinction betwen pull and push reporting is a useful one. It provides me a framework to think about how to report and what to include. I’m curious to hear your thoughts or how you think about test reporting.

It turns out that the disk my VPS was hosted on crashed irrecoverably. Of course, it took my provider 5 days to get back to me with an explanation as to why my site was down. They eventually did give me an explanation, but not before I got sick of my requests for information going unanswered and transferred my site to a new host. I think there is money to be made in seeking out stocks of companies that have awesome customer service and buying that. I have yet to test whether the opposite is true, but if a company has shitty customer service, I just don’t have the patience to deal with it anymore. Moral of this story is – VPSland has shitty customer service, so they get no more of my money. Linode comes highly recommended by people I trust, so I have big expectations of them. Time will tell.

The site is not the only change in my life of late. I went back to Australia last month for final selections for the Australian national kendo team. Unfortunately I was not successful, so I won’t be going to Milan next year to compete. A pity really, I would have loved to have gone. It’s ironic that I came to Japan to do more kendo training, but if anything have ended up doing less over the last 3 years than if I’d stayed in Melbourne. I made a conscious decision to try and strike a better balance between my home life, my work and my passion for kendo. I did that, but it meant that I was not as sharp as I needed to be at selections. I didn’t fight badly, just not well enough. As it stands, I’ve made the Australian team twice. I would have loved to have gone a third time, but it is not to be. Other things in my life are calling. I will not be part of the squad for subsequent campaigns. I thought I’d be more upset about it than I am, but when it’s time, it’s time.

I’m looking forward to spending more time with friends and family. I’m looking forward to spending more time on my Japanese (which is still awful, for the amount of time I’ve spent in the country). I’m looking forward to being more active in the testing community. I’ll still be swinging sticks at people, but I no longer have to drive myself into the ground in order to get ready for competition.  It actually feels pretty good.

To test is to tell two parallel stories: a story of the product, and the story of our testing. Test framing is a key skill that helps us to compose, edit, narrate, and justify the story of our testing in a logical, coherent, and rapid way.  The goal of test framing is to link each testing activity with the testing mission.

This to me is a worthy skill to have. Ask a tester why they execute a certain test at a certain time and often they will struggle to tell you. It doesn’t mean they are testing randomly or that there is no strategy to their approach, but that they are not yet skilled enough to articulate their reasoning from conception to execution; or if you like, they are not yet able to frame the reasoning for their actions in a way that is compelling.

Here’s where my brain nags at me. I like this concept of test framing and yet it seems to be missing something. Something fundamental; something without which the act of framing loses much of its meaning.

I said ‘not yet able to frame the reasoning for their actions in a way that is compelling’, which begs the question:
Compelling to whom?

I think Test framing as James and Michael describe it is part of a larger set of framing skills that are fundamental to being a skillful software tester. From the perspective of training software testers and answering to anyone who asks about the specifics of the testing you are doing, Test framing is a key skill. There is much more to framing than this however.

Much of what I’ve seen written about test framing to date implies an audience, but does not say a great deal about the audience explicitly. Simon Morley has written an excellent blog post on Decision and Analysis Frames in Testing. If you haven’t read it yet, please do. Simon’s writing hit upon the importance of understanding the filters through which your audience views the world (the frame through which they are looking) and that this can be different depending on the role your audience plays. He writes:

…the way we present information can affect its interpretation – depending upon the frame that a stakeholder is adopting.

Think of a frame as a filter through which someone looks at a problem – they’re taking in lots of data but only the data that gets through the filter gets attention (the rest may end up in the subconscious for later or isn’t absorbed), “I have my show-stopper filter on today so I don’t notice the progress the team has made…”

So, being aware of the potential different types of frames that each project member might have as well as some traps associated with frame formulation is important.

He goes on to write about the different kind of frames the audience and the tester might employ (it is worth noting the differences between them). He also writes about problems with framing, such as Functional Blindness (I see parallels here to the Dunning-Kruger effect), the Sunk Cost fallacy, overconfidence and distortion through measurement and numbers.

Understanding the audience’s part in framing information is a key part in effective communication. My presentation at CAST2011 was on telling a compelling testing story, so framing, especially in terms of the audience is a subject that is very interesting to me. Framing of your testing (and the information it reveals) is inseparable from your audience. Even if you are only framing your information for you – you are still your own audience.

Each of us has different filters through which we view the world. Even within a single role (tester, programmer, project manager etc) individuals can have vastly different outlooks on the same subject. Largely this has to do with the way we model the world. We create a mental representation for everything we interact with. Every object, every person – we have a mental representation about what each of them means to us based on our previous experiences and our previous models. Our experiences shape our models and our models become the filters through which we experience the world.

Necessarily then, every individual has different models (and these differences may be subtle or profound), and so we all have different frames of reference through which we interpret the world around us.

To understand frames, you need to understand models. Not just the model that we have of our system under test, but the models we have of the world around us, the model we have of our audience especially. The map is not the territory. How we view ourselves is not how others view us. The reality (who we actually are) is different again. Same goes for our audience and indeed any other object or concept you care to name.

When we present information about something to someone, we are in fact presenting our interpretation of that information through the filter of our models, through the filter of our audience’s models to them.

Understanding the model(s) your audience is using can help you build information that they need to know about. Understanding your audience’s frame of reference when receiving information can help you frame that information in a way that they are receptive to. Moreover, you can decide if your audience needs information that will augment the models they have and whether or not you need to reframe them in order to present it.

You may present information to your audience that you thought was vitally important only to have them react as though it were inconsequential. Why is that? There are lots of possible explanations, but it’s likely that either they are working from a model which renders your information inconsequential, or it could be that you have a richer model than they do and they do not understand the importance of your information because the limited nature of their model does not allow them to.

Here is where reframing can be helpful. If your audience’s current model creates a filter whereby your information is not useful or valid it does not necessarily follow that your information is invalid per se. Were your audience to look at your information from a different perspective it may be that they would find if valid and important after all.

I feel like there is a lot more here to be pondered and discussed, but for the sake of brevity (hah) I will leave off here for now.

BK: I wasn’t sure about entering the testing competition. It looked like a bit of fun, but it was going to take up an entire evening of a 2 day conference. It’s difficult enough to catch up with all the people you want to see without taking up a bunch more time. I noticed that quite a few of the testers I wanted to talk to were also entering though and moreover, I’d not had the chance to test with Louise Perold before and this was a perfect opportunity.

LP: I was at CAST in 2007 in Seattle and entered the tester competition there as an individual. I had a lot of fun the first time, so I was quite keen on taking part again. It is a challenge though when you have been concentrating all day and your head is pretty full and then you have 4 hours that you know you are going to give some serious attention. So much of my job now is attending meetings and I get so little time to test anymore, that I definitely wanted to just break something again. The opportunity to do this with my good friend Ben, was something I could not pass up.

BK: The testing challenge was simple to explain; Install some software created by Dave Gilbert (who was attending the conference and on hand for the competition), then test said software for bugs, submit bug reports and tie things up with a test report at the end. Not so easy in practice, as it turned out. We faced a couple of challenges immediately. I have a thumb drive with testing tools I like to use – only I’d left it in Japan. The other challenge was that both Lou and I had left the power supply for our laptops back at the hotel. I left Lou to download the tool while I raced back to the hotel to pick up said power supplies.

BK: I got back to discover that the low bandwidth of the conference centre was causing problems with the download. We sourced a USB key that was floating around with the app on it and installed from there. It was here I discovered my first bug. Upon launch, the app crashed – didn’t handle non-US date format. Lou and I decided we should go talk to Dave and let him know this would probably affect other users with non-US settings. We also decided to interview him about the product.

BK: Having David at our table made life a lot easier. Being able to test and chat simultaneously with the guy who wrote what you’re testing is really the only way to fly (okay not the only way, but it works well for me). When you have built up a relationship with someone and see what makes them tick, you open up a much wider range of possible communication styles. You can always ask direct questions, but you can also make jokes, you also put a face (or faces) to the bug reports you send in. If something comes across as harsh in print, they may give you the benefit of the doubt if they have other references that show you’re not arbitrarily being a bastard.

Lou and I settled into testing our respective areas, logging bugs, consulting with one another about bug reproduction and logging. 4 hours is actually a really really short amount of time, especially since the app we were testing on was non-trivial. We chatted with Dave as well, not just talking shop, but also music and life in general, following where the conversation naturally went. It was good fun. Some of the bugs we found were cool and difficult to reproduce. The bug reporting mechanism itself left something to be desired, but we rolled with it. I went with a chartered exploratory testing style and here was where I made my first real error.

I took notes in a file as I do with most chartered ET that I do. What I didn’t do was take the time to make my mission explicit. I knew I was doing recon ET in order to find bugs related to handling of varying file types and extensions, but that may not have been clear to anyone trying to debrief me (or judge against my notes). The notes themselves were okay and gave a reasonably good indication as to what I was thinking, but I could have taken a few minutes and made them a lot better.

LP: I had wanted to use Rapid Reporter, and then couldn’t get it downloaded quickly so then decided just to use notepad. I realised once again how much of a skill taking good notes is. If it is not something that you continually practice, it is really easy to get out of the habit and miss things in your notes. I guess also, when you are trying to log bugs quickly (the first team to log an issue would get the credit for the bug), I felt pressure to focus on finding things and getting them logged. I should have also realised (as in real life) how important it is to recall and explain what you have and haven’t covered and exactly how you got to certain issues, why you focused in certain places.

BK: James Bach wandered by and asked about our test strategy. I thought our answer was a reasonably good one (more or less what I just outlined above). Since Dave was simultaneously the developer and product owner, we talked to him about where we could add the most value from his point of view, divided the work between us, got ourselves into a position where we could easily exchange information and got to it.

LP: I didn’t feel like I was able to give enough detail around the specifics of what I had been covering. Maybe because I had encountered a number of crashes that were not always quick to reproduce. Maybe because my notes were really bad. So I felt I was able to talk to our bigger plan, but not really exactly how we were implementing it or where we were in that implementation.

BK: When James wandered back again asking about the test report, it occurred to me we should have gone to him earlier and asked him what he wanted to see. No time like the present though, so we asked him there. Having gained what we thought was a good understanding of what the report should contain, I went about putting it together while Lou kept testing. In hindsight, we should have been building the test report as we went. No sooner had we started putting the test report together, we discovered we had about a half of a testing mission. Okay the competition was a little bit arbitrary and the mission was about finding and reporting bugs, but I was hard pressed to clearly define the specific mission that we had. We knew that we had a hard 4 hour limit, but that was the only stopping heuristic we had. If we’d started our report earlier, it might have occurred to us that there were other factors to consider when answering ‘When will you stop testing?’.

Because the bug logging mechanism essentially made the logged bugs invisible, backtracking to find the important bugs we logged took a lot of time. If we’d logged these in the test report as we went, it would have meant we had more time for other things. As it turned out, we barely had enough time to turn in the report as it was. Even when we did turn in the report it was missing a few things. We described the areas that we tested, but not a whole lot about the actual testing we did. We didn’t really have a clear statement of our overall mission (for reasons already explained). We listed the important bugs we found, both crash bugs and bugs we considered important to look at. We also listed the areas that we hadn’t covered.

We listed our conclusion up-front. I know that some people like to skim, so having the tl;dr (too long; didn’t read) stuff at the start would at least get the point across, with the larger details to follow. We also provided our testing notes – this was both a good thing and a bad thing.

LP: I also realised afterwards that we hadn’t really explained the subtleties of our mission and context. The fact that we were in a “competition” meant certain things – that we would focus on logging bugs as quickly as possible so that we could be credited has a impact on the way you test (and probably also the way you log the bugs). My notes when I looked back on them were just bad. I don’t think we should have included them. Even the mindmap – which I normally use to outline my strategy, was seriously lacking detail. I think again I learned that for me to really focus a lot better and do a better testing job, I should just spend a bit of extra time and energy on defining and outlining my strategy and then showing where I am on and off track of that. I also know that my bug logging was generally not very good.

Looking back at it now, it probably would have been a good idea to have a mini low tech dashboard, where we could have tracked exactly where we were against the chosen functional focus areas.

BK: A mini-dashboard, yeah nice one. Would have been really handy and really easy to do too.

Lou and I left the venue kinda down about our efforts. On the walk back to the hotel, we chatted about what else we could have done. I think about every second sentence started with ‘Ah shit, you know what else we could have done?’. We kinda figured that if there was a prize for tester/programmer relations we might be in the running. We assumed that there were a bunch of other teams well ahead of us for the overall prize.

BK: We’d like to send a big thank you to Dave Gilbert for putting his app up for testing and for being the focal point for a large number of testers, man I didn’t envy him that at all. Thanks also to James for putting up the cash (literally) from his own wallet as the prize. Thanks to everyone who had to wade through the list of bugs and test reports that all of the teams turned in. That wouldn’t have been a small task. The competition was a lot of fun.

LP: Ditto from me Dave was really an awesome developer to work with and test for.

BK: I have some suggestions for the CAST2012 competition (or really for anyone that wants to run one of these at a conference)

• Consider a competition that runs for the duration of the conference
  LP: Actually I kind of liked the 4 hour competition because I feel like it adds a unique pressure that makes for an excellent learning experience
  A longer comp has its pros and cons, but it allows different dynamics and I reckon it’d better mimic everyday testing situations.
  • Testers can split their time between attending sessions and doing actual testing – they might even put into practice something they’ve just learned.
  • Competition staff (developers, project manager, product owner etc) can selectively be on or off duty (just like in real life) – they may also not be so easy to find – also like in real life.
  • Given that testers are now likely to mob competition staff in order to max out the tester/programmer (or other) relationship thing, it allows said programmer (or other) to escape the love fest if it all becomes a bit much.
• Use a decent bug tracker.
  LP:I second this one.
  • Something simple like Mantis or Redmine would do the job. Set up separate projects for each testing team, or set up single users for each team so that it’s easier to keep track of who has done what
  • Said projects could be made invisible for non-members of that team (but team members would be able to check against each others’ work to avoid bug duplication)
• Don’t worry so much about the cash. It’s a nice little motivator to be sure and yet it’s probably less important than the fact you’re up against some of the more talented testers on the planet

For those that are planning to enter next year’s competition, here are some suggestions:

• Be prepared
  • Have your toolkit ready to go. You might be testing against a specific OS, a web app, mobile app, more than one of these or something different again.
  • Practice – If you’re not at the testing coalface every day, spend some time shaking the rust off your test-jutsu. Not just execution of testing, but information recon and reporting also.
• Talk to people
  • Find out who matters, and build a relationship with these people.
  • Find out what matters to them
  • Buying them beer probably doesn’t hurt.
• Use what you know to direct your testing
• Keep your feedback loops short (short being relative)
• Don’t overthink it. Plunge in and quit. Get some, go again – however you want to think about it, make sure you’re also doing it.
• It’s not going to be perfect. It doesn’t have to be. It just has to be useful (to someone that matters)

What I love about CAST is that there is a very collegial feel to the event. It feels like a safe place to air new thoughts and ideas (or old ones) and discuss them – even have them shot down, in an environment that promotes and supports learning and cooperation.

There were a number of firsts for me personally. I hadn’t presented at CAST before. My presentation went well, I thought. I had a much fuller room than I was expecting. People seemed to find it interesting and useful and there were a lot of good questions asked. It wasn’t perfect. The room’s air con was LOUD, so the set of speakers I brought to play a snippet of video were woefully inadequate. Along the same lines, I wasn’t using the mic to speak and a few people let me know I could have projected a little more. Strange. I don’t have a problem projecting in the dojo. Probably because I’m barking instructions at people. Presenting to a room of testers is a little different it seems. Something for me to work on, then.

Louise Perold and I entered the CAST testing competition – a 4 hour event to test the abilities and skills of the testers that entered. Louise and I walked back to the hotel lamenting what else we could have done if we’d just had a little longer. We had some awesome competition. People that I know by reputation if not personally and all of them excellent testers. When on the following day James Bach announced we’d won the award for best tester/programmer relations we were happy. That was something we thought we had done well. When he went on to tell us we’d won the overall competition, we were stunned and delighted in equal parts. Louise and I will blog more about this at some point soon.

Lastly, I got chatting to Michael Bolton about my thoughts on test framing and how I thought it related to audience framing (I have another post coming on this topic too). He invited me to present my point of view at his tutorial on Test Framing, which I did. I was grateful for the opportunity and a number of people said they got something out of it, so that was something else I was happy about.

The thing I love most about CAST is the interaction between people. Because I’m somewhat geographically challenged, I get to one testing conference per year. I choose CAST because of the people who attend. Because they challenge me and make me think and remind me of the power of diversity and having skilled, free-thinking people working together.

I was super happy to spend time with the Usual Suspects (the people I generally hang out with at CAST) – Henrik Andersson, Chris Blain, Tim Coulter, Carsten Feilberg, Johan Jonasson, and Lou Perold. I also met and got to know a bunch of other cool people (most of them Swedes – it was a real invasion on their part) Robert Bergqvist, Maria Kedomo, Mattius Gustavsson, Sigge Birgisson.

As always seems to happen, there were a great number of people I would love to have spent more time with. I met in person a number of people I’d only previously interacted with online. There are a number of people that I’d met before, but haven’t spent nearly as much time chatting to as I’d like and a few I wanted to meet, but didn’t get to. If I try and name everyone, I will undoubtedly forget someone. Suffice it to say there were a massive number of testers I’d have loved to spend more time with. I suspect that number will only be bigger this time next year.

I didn’t get to see all of the presentations I wanted to see. Some of them were captured on video (the emerging topics tracks and lightning talks among them, I believe). I plan on taking my camera next year to capture stuff I can’t make it to. If you didn’t make it to cast, I highly recommend you check them out.

The organizers are to be commended. Jon and James Bach were the most conspicuous of the organizers, but Paul Holland and his crew of facilitators and all the volunteers who ran around doing stuff behind the scenes are to be commended on a job well done. Volunteers are the unsung heroes of events like this. When things go well, you tend not to notice them. Without them though, this event would be near impossible.

There are a few improvements that could be made. I’d like to echo some of Michael Hunter’s sentiments. A lot of people are okay with a high-carb diet. I’m not one of them.

The distance between hotels made meeting people who weren’t at my hotel kinda difficult (though we managed to find pubs here and there that did the job).

As far as content goes, I got new ideas from the vast majority of the talks I attended. I must admit to being a little disappointed by the debate between James Bach and Doug Hoffman. I was hoping the format would be more of a formal debate arguing for and against a specific statement, but it turned out to be more of an argument/discussion with the interjections and interplay that tend to go hand-in-hand. I also thought Doug was more gentle with James than he deserved. I think this sort of event has potential. I’d like to see a change to a formal debating format, but having a relevant topic debated by some heavyweights in the industry would be very interesting.

I could create a list of questions and great quotes as Michael Hunter has done, but I think those people tweeting to the #CAST2011 hashtag took care of that most effectively (try searching on it). I’m glad they did, because I barely had time to tweet a thing for the whole event.

I’m hoping very much that I’ll be able to attend (and perhaps even present) at next year’s CAST, but that depends on a number of variables that are somewhat beyond my control. If you are considering attending next year, I recommend that you do in the strongest terms possible.