Let me give you some friendly advice to help you become the power tester you want to be. I’ll get the boot camp drill sergeant stuff out of the way first.

If you’re anything like the vast majority of new testers I’ve encountered, you’re full of questions. Probably a lot of questions that start with ‘what’s the best way to…‘ or ‘what is the best practice for…‘

Take that combination of words and remove it from your vocab. When I see questions like this, I cannot help but think that the person who asked it is incapable of free thought. That may not be the case, but those questions translate in my head as ‘Please spoonfeed me an answer so I don’t have to think. Please just tell me what to do. Give me a recipe that I can apply wholesale without consideration of any other factors whatsoever’.

Sounds fucking stupid, doesn’t it? Yes it does. Because it is.

Imagine if someone asked you ‘What’s the best way for solving conflict in the Middle East?’ That’s basically what’s happening. Hey, let me ask you for a simple answer to an incredibly complex question without giving you any context whatsoever as to how it applies to me.

Same thing goes for asking about pre-written test cases, answers to interview questions, what the best testing tool is. That’s sheer laziness. Don’t do it. You’ll see lots of other people doing it. Lots. If you do the same thing as everyone else, then you’re no different from anyone else. You’re a commodity and ultimately expendible.

Want to know how to distinguish yourself from the hordes of zombie testers out there? Don’t do what they do. Don’t be like them. If you want to be just another minimum-wage meatbot then fine, just find a different career.

If you want to be good at testing – hell, if you want to be even a competent tester, critical thinking is a skill you’re going to need to practice. I realize that might come as a shock if you thought testing is about following test cases (that’s not really testing btw, but that’s a discussion for another time). The habits you learned in grade school – memorization and subsequent regurgitation of the right answers – that’s not thinking. That stuff doesn’t work for testing.

Testing isn’t about pass or fail. It isn’t about getting the correct answer and turning it in to the teacher for a pat on the head. Testing is about asking questions about software and its artefacts and its people to find out information for the people that matter. It’s about maintaining your sense of uncertainty when everyone else is losing theirs. It’s about exploring issues of risk and the tradeoffs you make between making something better and the cost involved in doing so. It’s about seeing what’s in the gaps. What do people say they want? What do they actually want? What do they say they did? What did they actually do? What does this component connect to? What does it not connect to that it should (or vice versa) – are these the sorts of questions you ask when you test?

How do you know that a bug is a bug? (You might like to read up on oracles) How do you convince other people that a bug is a bug in a way that is tactful and persuasive? (You might like to read up on bug advocacy)

What else can you do to become an awesome tester? Want to know how other awesome testers do their testing? Do testing with them. Have you heard about Weekend Testing? (Yeah there are people who use their spare time to get better at testing, how crazy is that?) Even if you don’t decide to test with them, you can read the transcripts of the testing they did and their testing notes. You can learn from people of all sorts of experience levels.

There is no shortage of excellent testing blogs out there. There are ten times as many that are garbage, but if you’re here, then that’s a start. Any of the links in my blog roll and any of the links in theirs should have more than enough material to keep a new tester’s brain occupied for a very long time indeed.

Still reading? Well, if you haven’t been scared off yet then you might actually be serious about becoming a good tester. Read this stuff. Practice this stuff. Find anyone that knows software testing and learn from them – by which I mean question them. Don’t take anything that anyone tells you as gospel truth. Examine it. Turn it over. Poke holes in it. Good testers welcome challenge and criticism. It’s what helps them become better testers.

There are testers out there that will coach you. They’re good at it too. Contact them. Ask for their help, but understand that the onus is on you to do the hard work.

Becoming a skilled tester isn’t an easy path, but it is a very rewarding one. In the time I’ve been a tester, I’ve met amazing people from around the world and formed a network of peers I can go to when I have questions. I love the work that I do and I get to work with some amazing minds. I wouldn’t have had any of these opportunities had I not made the choice to make myself the best tester I could be.

If you’re a new tester and you want to be a good tester, then you have work to do. Best get to it.

If you haven’t registered for CAST yet, the early bird pricing finishes at the end of February. Get on it.

If you’re keen to attend, then pricing details are here and you can sign up here. If you make it, be sure to say hi.

Fake testing is the blind worship of process and procedure (over the sapient and conscientious application of skill). There’s been some talk of its demise recently and while I want to believe we’re heading in the right direction, I also want to keep in mind that this is not an easy problem to solve and if we’re not careful we won’t solve the problem, we’ll just shift it elsewhere.

Let me give you an example.

At GTAC, there were some very interesting tools demo’ed. I looked at BITE (Browser Integrated Testing Environment) and thought – yeah, there’s another nail in the fake testing coffin. I particularly like the concept of making it easy for the end-user to report bugs that contain information that programmers can use to solve issues quickly.

As I continued to watch, something nagged at me. A sense that something was not quite right. Not quite deja vu, but certainly the sense that if it were this easy, we’d have done it before.

Shortly thereafter, I was chatting with good friend and colleague Jared Quinert. I had forgotten how good he is at taking a subject, stripping back all the bullshit and laying its foundations bare. If you ever get the chance to work with, or hire him. Do it. Seriously. As usual, Jared locked onto what I’d been feeling uneasy about without me even having to explain myself. Having shared the link to the talk I’ve linked to at the start of this post, our conversation went something like this:

Jared: Record playback software is apparently central to the life of a manual tester…

Ben: Yeah I chuckled at that too.

Jared: Recording scripts, converting to English doesn’t solve the abstraction problem. Waiting to hear them elaborate on re-recording being cheaper than maintaining…and they’re still just talking about QTP features, but nobody has piped up with this QTP can re-learn missing controls…

Ben: True

Jared: Tool running in the browser probably creates other issues too I would think.

Ben: Likely yes. That and when you have several thousand tests and you need to re-record a couple hundred, the appeal wears thin.

Jared: The abstraction is the problem, not the approach and as always, I’m skeptical that the test can simply be replayed without any issues when there is dynamic content.

Ben: Agreed

Jared: I think I get annoyed by the ignorance they have of other tools, which is why programmers seem to keep reinventing the automation wheel.

Nerds have far too much faith in machines and solving problems through programming. Some good things come out, and some massively misguided efforts come out as well. But ultimately, these guys will decide our future, because they will dangle an appealing, human-free testing future, which will be doggedly pursued by management. If there are two choices for how work is designed, and one gives management more control, they’ll choose the latter.

Effectiveness is secondary. They’re not solving my problems. They’re solving developer problems. And management problems that arise due to poor work design. But they’ll probably win. Those of us who are skilled enough and market well will retain decent jobs, but there may not be many of those roles. Mostly people will forget how to do the skilled work. When they say ‘I wanted to help the manual testers’, I always hear ‘I wanted to help the shit testers (Even though they didn’t ask for any help)’.

I really wanted these tools to be of assistance in helping to put a crapload of fake testers out of a job, but I really think the reality will be different. While the bug reporting side of things might help developers fix problems faster and free up testers to do more valuable testing, the automation side of things will a) be farmed out to low-skilled low-paid workers who know enough to hack a little bit of javascript and b) spawn a bunch of new frameworks that deal with the abstraction problem that this tool has inherited from every other record and playback tool of its ilk. In the case of a) it may eliminate a bunch of drones writing test plans and test cases then attempting to execute them, but it’s shifting the problem into the automation sphere.

The fact that Google has released the client but not the server will be a deterrent for some, but it won’t take long before someone puts together a workable open source server and the crap automation tool + low paid maintenance worker farce will have a new player. It might mean that that you won’t need quite as many fake test drones in your organization, but in order to make a dent in the plague that is fake testing, we need something a little better than crap + 1.

The intended audience seemed to be testers who are enslaved by process and useless dogma, though he pitched it as being for testers who are lacking respect. I suppose the latter comes from being the former. I expect that many of those people will be feeling some trepidation having heard this presentation. That’s probably a good thing, but the doom was layered on pretty thick. I thought it was overkill personally.

In my previous post, I wasn’t shy about criticizing the content as I saw it. I’m not backing away from those criticisms either. They stand. That said, there were some cool and useful things that Mr. Whittaker did talk about. It’s just a shame that the vast majority of the points he made were very blinkered and didn’t really extend beyond web tech (especially web tech for large companies). I spent enough time going over that last time. There’s more nonsense I could have a go at, but it wouldn’t serve any real purpose. If you’re curious you can watch it yourself.

I think it’s a little irresponsible to say ‘you’re users are going to suffer anyway, so just get it out there’. That works in some situations – mostly when users are expecting it. When you have customers paying money for a product that they expect to work well, that strategy isn’t so great. The users (rightly) get upset when the software doesn’t do the job. He did justify this point later with ‘use dogfooders and releases to trusted users’, but that’s not going to work across the board.

At Google, the majority of the tools they produce are free for the end user. Their model is based on building things that people will use (for free) and building advertising into it. If it doesn’t work perfectly, oh well. It’s not like you’re paying money for it.

Try that on with a product where people are forking over money. It doesn’t have to be a lot of money. Even $10 per month is enough for someone to become irate when what they expect to work does not. As a company releasing software, you have a duty of care to make sure that software does the job.

Yes of course there will be bugs. You can’t make it ‘perfect’. That’s not the point. The point is that you have taken reasonable steps to make sure that it does the job intended and doesn’t do what it wasn’t. Would you release CAD sofware for engineers with cursory testing because you can easily patch it later? What about drivers for machines used in radiation therapy? Pacemakers? Traffic control? Problems with these don’t just mean users get pissed of and bitch about it on twitter. It means people die or are maimed horrifically.

Yes some of the stuff that Mr. Whittaker talks about is cool and is made by developers and will serve to bring testers closer to both the end user and to developers. That’s awesome and I’m all for that. That doesn’t mean that testing as a sapient vocation is under threat. Fake testers would appear to have a use by date and I’m all for that too. That day can’t come soon enough for me.

Computers are getting better at assisting us to test and yes of course we have the developers who wrote that code to thank for it. For example, the tools Mr. Whittaker spoke about to help end-users report bugs are very cool indeed. I’d love to put something like that in place for the end users of my company’s products. I also agree with him that we should be embracing our users in order to assist us with testing. Select groups of early adopters, dogfooders, people who understand that what they are using is not perfect yet and that we appreciate their feedback. Why wouldn’t you want to tap into that?

There were other worthwhile truisms that Mr. Whittaker brought up in amongst the hand wringing and gnashing of teeth.

Developers can test.
This is true. Testing is part of a developers job. Being able to look at requirements or a story and think about not only how the code should work, but how it should handle failure is something developers should be doing. They should also be harnessing frameworks out there to build checks that save them time down the track. Agreed. That’s part of their job and if they’re doing it, that’s full of win for both them and the testers.

Stop doing stuff you can build a robot for.
I’m on board with this too. Anything that doesn’t require a human to evaluate and that can reasonably be automated, should be.

Testing is not the sole domain of testers. It doesn’t matter who does it, only that it gets done.
I agree with this one too, by and large. You can get into semantics about what testing is and what skill sets are particular to dedicated testers, but yes as long as testing is done to the extent it needs to be in a given context, then fine.

The thing that irked me the most about this presentation is the fallacious premise that you have to have produced something corporeal and tangible in order to be of any worth. This is simply not true. Testing is a cognitive process throughout which you produce information that (among other things) helps bring a piece of software from conception to fruition.

There are places for testers who code and there are places for testers that don’t. The advances that Mr. Whittaker is talking about will make life difficult for (so called) testers who don’t think; for the those who blindly follow a busted process because someone called it ‘best practice’ and they decided to stop learning when they graduated from whatever school they last graduated from. On the other hand, for testers who love to learn, who enjoy new challenges and are not afraid to get their hands dirty there is a bright future. It may be that their job title might move from ‘tester’ to something else. Who cares? They’ll be bringing the same incisive critical thinking and analysis to whatever you decide to call them and they’ll still be testing. Bring it on.

This one is a response to a blog post by Eric Jacobson, who wrote up some notes from James Whittaker’s keynote at STARwest.

I myself did not attend STARwest, so I have only these notes to go on, but as I read them a few things struck me as being odd. Some of Mr. Whittaker’s assertions I agree with, though I think that his reasoning for them is flawed. From Eric’s blog:

testers do not get respect in their current roles…the reason we lack respect is all the recent software quality “game changers” have been thought of by programmers

The game changers that he listed were strange to me. If I read the post correctly, they are there to help back up the assertion that testers should be fearful for their jobs. Let me list them, then list reasons why after most of them my reaction was either ‘meh’, or ‘huh?’

•  Today’s software is easier to update and fix problems in (easier than software from 10 years ago).

There are some words missing in this statement. I don’t know what they are, so let’s try some variations.

Some of today’s software is potentially easier to update an fix problems in (yadda yadda), provided that it is well designed, well structured, well written, well commented and well tested.

or maybe

Today’s software is easier to update and fix problems in (yadda yadda) due to the advancement of IDE’s and debuggers as well as frameworks built into some modern languages that assist structure and testability.

Well okay, both fair points. Both have the potential to increase software quality. I’m not seeing any particular threat to software testing here though.

Threat rating: Meh (with a side of Huh?).

• Crash recovery – some software can fix itself.

Ok. Couple of things

1. So what?
2. ‘fix itself’ is a pretty nebulous statement. Maybe James said more on this and because I missed it, I can’t speak to it specifically.

Netflix servers have the Chaos Monkey that kills processes at random to help them work out how to make their product more robust. That’s cool. Does that mean their testers are on the way out? Other software that crashes can sometimes recover and or send logs back home so stuff can get traced and fixed. That’s also great, but the software has at that point, already crashed. If it keeps running, it may well be in a state that is detrimental to the end user. Whatever. This point is irrelevant to testing getting in the way of quality.

Threat rating: Huh?

• Reduction of dependencies via standards (e.g., most HTML 5.0 websites work on all browsers now).

Would love to see the reference to the stats backing up this statement. Tell it to the banks and insurance companies and other ultra-conservative behemoths mired in their own red tape and bullshit processes. Those poor bastards are still on IE6 or 7. Aah Standards. Yep. Standards are great. I don’t know if this is an attempt to say that cross-browser testing is no longer required. It may make checking functionality across browsers a little easier, but again I’m not seeing a threat to testing here either.

Threat rating: Meh.

• Continuous Builds – quicker availability of builds makes software easier to test but it has nothing to do with testers.

Eric already commented here that build tools have nothing to do with testers – well, not far beyond lighting up a suite of unit tests and potentially helping deliver builds to testers faster.

Threat rating: Meh

• Initial Code Quality (e.g., TDD, unit tests, peer reviews)

These are all great things and if anything are complementary to software testing. TDD and unit testing can help Developers put better code in place sooner and find out about stuff that is broken at the unit level faster. Code reviews have the potential to help coders in the room become better at their craft (and if there are testers in the room, it can help them understand how coders think, how their code is put together and any bad habits that their colleagues might have).
Yep. It’s a positive for code quality. Not seeing any threat to testing here.

Threat rating: Meh

• Convergence of the User and Test Community (e.g., crowd source testing, dog food testing).  Per James, “Testers have to act like users, users don’t have to act.”

Hmm, having had first-hand experience with crowd-sourced testing, I have to say this has a long, long, long way to go before it is any threat to the profession of software testing. It is a useful supplement to software testing, however using it as a replacement for software testing would be a recipe for pain.

Having end users test a site can be great and the feedback you receive can give you some great insights into how they want to use a product and what they want to achieve. To equate that with ‘Users are better because they don’t have to pretend’ is naive. There are a couple of problems. Firstly they’re not trained to look for problems. They may stumble across them and they may find things that your software testers do not. That doesn’t mean that testing is not a worthwhile activity. Secondly, users aren’t trained to describe problems in a concise or logical manner. There’s only so much troubleshooting you can do with the description ‘It’s shit and it doesn’t work. I paid good money for this, so fucking fix it’ (spelling and grammar corrected for the sake of legibility). They don’t generally have access to logs, or provide screenshots or video of how a problem came to be.

So – to my knowledge, not thought of by programmers and not a threat to testing.

Threat rating: Meh.

Eric then lists 4 additional ‘painful facts’ about testing (I’m assuming these were taken directly from Mr .Whittaker’s presentation)

1. Only the software matters.  People care about what was built, not who tested it.

Depends on who you mean by people (and by ‘matters’). End users? Sure – unless the software doesn’t do what they need, or crashes on them constantly. Then they’re all about who tested it. Funny about that. What about other people?
Again – how is this painful? Hindsight is 20/20 and it’s possible that valuable info from a software tester is less valuable to the audience after the fact.  A software tester’s job is to provide information to decision makers to help them make their decisions. Talented testers do this in such a way that the information is timely, relevant and valuable. Do that consistently and you can bet your arse that they’ll care about who tested it.

2. The value of the testing is the activity of testing, not the artifact.  Stop wasting your time creating bug reports and test cases.  Start harnessing the testing that already exists (e.g., beta testing).

The first sentence here is gold. The second sentence is a bit WTF? Mostly because the example given was ‘beta testing’ – were there no other examples of testing given? I’m all for ditching unnecessary test documentation. Do as much as you need and no more. I’m also all for harnessing existing testing (as distinct from checking, which should also be harnessed, but is not the same thing). It also doesn’t mean ‘when you’re done coding, throw it at your users’.

3.The only important artifact is the code.  Want your tests to matter?  Make them part of the code

I wish I had an analogy to better illustrate how useless this statement is. Important to whom? In relation to what other artefacts? This statement is probably referring to ‘checks‘, but it manages to also dismiss other forms of testing with a sweeping blanket statement that doesn’t actually mean anything. Wow.

If you have a bunch of checks, sure. Automate them. Build testability into your code, absolutely. Your code and probably product quality will be better for it and chances are your testers will thank you.

4. Bugs don’t count unless they get fixed.  Don’t waste time logging bugs.  Instead, keep the testers testing.

Aside from the notion that bugs count in any case, I couldn’t agree more, but again I fail to see how is this painful to testing. Log bugs if your developer can’t fix it when you tell em, be that in a bug tracker or on a post-it attached to a story. Whatever works, just get em fixed (you know, unless they don’t actually matter to the people that count).

I disagree with Mr. Whittaker about the reason that there is a lack of respect for software testers in the industry. The ‘game changers’ that he talks about are largely produced by programmers in order to make life easier for … programmers. Sure they’re also generally good for software quality, that’s not why we testers get a bum rap.

The reason we don’t get a whole lot of respect is because the industry is saturated with fake testers. Not only that, but there are complimentary industries *coughcoughcertificationcough* that make a tidy profit churning out crap material to the gullible and the ignorant, helping to produce the next generation of fake testers.

Now it may be that Mr. Whittaker is taking these testers to task when he says that developers are getting better at testing, but testers aren’t getting better at programming and “It’s only important that testing get done, not who gets it done.” If so, then great. I agree. The faster fake testing dies, the sooner the rest of us can get on with it. I might finally be able to hire someone without having to read through the resumes of 300 oxygen thieves.

I do think that his exhortation to “get a specialty and become an expert in some niche of testing (e.g., Security, Internationalization, Accessibility, Privacy) or learn how to code.” is just scaremongering for any tester who knows their craft, but hey, controversy puts bums on seats, so I suppose I can’t fault him too much for that. Sure, become a specialist if you have an interest in it. Knowing how to code certainly won’t hurt you, but I know a number of world-class testers who do not code and are testing generalists. I do not believe their livelihoods are in danger any time soon. As testers, we bring something different to the table. Some programmers are excellent testers. I like working with them. I encourage them to get better at testing. It helps me to help them look good. For coders, being able to think like a tester is a skill worth developing. That goes beyond being able to write unit tests.

So as far as controversial keynote speeches go, this was probably quite entertaining. As far as it containing any real substance to support an assertion that a skilled software tester should be concerned about losing their job to coders though – I just don’t see it.

To me, effective test reporting is the glue that holds it all together. If you can’t (or don’t) report your findings effectively, in a way that your audience finds useful, then your testing is not effective.

It occurs to me that test reporting seems to fit within two categories. I call them Pull reporting and Push reporting.

Pull Reporting is where your audience comes to you for information. Generally they have questions that they want an answer to, such as ‘When will you be finished testing?’, ‘How stable is Product X?’, ‘Did you find any bugs in interface Y?’ and so on.

Setting aside for the moment the quality of these questions, let’s talk a bit about the sort of situations in which pull testing arises. Stand-up meetings, drive-by situation reports, weekly meetings are all examples of pull reporting. In each case you have an audience that (at least to some degree) has context in mind and is requesting specific information. In my experience it’s the drive-by sit-rep that can be the most challenging, often because I’m in the middle of working on something else and I need to context-switch in order to answer. I don’t know about you, but it takes me time to switch out of something I’m deeply focused on in order to answer something unrelated.

In this situation, there are a few things you can employ to buy yourself some time.

• Remember to breathe. If nothing else, oxygen is good for your brain and taking a deep breath gives you a few more seconds to switch over and consider the question posed.
• Ask your audience to repeat the question. If you were in the middle of something else, it’s possible you heard something akin to Charlie Brown’s teacher rather than a question. Ask them to repeat it.
• If you don’t feel like you can effectively answer the question right this second, ask for time. The question might seem urgent, but it’s rare that the situation is so mission critical that you can’t say ‘Give me 5 minutes, I’ll come to your desk (or call you back etc) with what you’re after.
• If you are ready to answer, go ahead and answer. If you are speculating about something, make sure that you identify that you are speculating. ‘I know x, y and z. It may be that a & b, but I’m not sure’
• Poll them to see if you have given them the info they’re after. It shows them you actually care about what they’re asking and it gives you immediate feedback as to whether your report was effective.

The drive-by report is also a great opportunity for an ambush. Knowing that someone is focused elsewhere and will need to context switch – that’s a great time to go on the offensive if you immediately want someone on the back foot. It’s a dick move, but it works. Hopefully you’re not on the receiving end, but should it happen, then remember that not all questions need to be answered. Beware ‘why’ – especially when it’s accompanied by a pronoun as the third word (why did you… why didn’t you… why aren’t you etc). You may find it more constructive to turn those sorts of why questions into ‘what’ questions. ‘What has happened that is concerning you?’ Answer that, and then you can decide whether the why is worth answering. Suggesting a different venue and time for the discussion is probably also a good idea also.

If you do find yourself in a position where you can reel off a report on request, you might like to consider using Michael Kelly’s MCOASTER heuristic. It’s a handy list of items that your audience may be interested in. You don’t have to report on every single item. Pick the ones you think your audience wants to know about (and check in after to see if they want more).

Push reporting is the opposite in terms of who has context in mind. Push reporting is when you have information that you believe your audience should know, but they are not aware of it. In order to effectively convey the information you have, you need to understand what is important to them, what they know already and how to frame your information in such a way that it is easily digestible and meaningful to your audience.

Examples of Push reporting are Bug reports, Reporting showstoppers or mission-critical issues, Risk reporting. Time is generally on your side (though not always). Take the time to make sure that your audience is properly framed. Brief them on your subject matter before you launch into your report. Even if you have an appointment with an agenda for your report, you can’t assume your audience will immediately be ready to hear what you have to say.

Push reporting is where your ability to tell a compelling testing story comes to the fore. Our job is to present meaningful information to people that matter. If all I do is present that information, then I really feel that I’m not living up to my potential as a tester. I would rather present that information in such a way that it presents a compelling call to action.

In my presentation at CAST I used an example from the movie Hunt for Red October. In this clip, the sonar operator aboard the USS Dallas (Jonsey) presents information to the captain about what he has discovered.

The Hunt for Red October: Jonsey Reports

Keep in mind that the raw information that Jonsey has is

• He knows he has heard a submarine that they don’t have previous records of.
• He lost the sub but picked up a sound the computer tells him his magma displacement
• He has done some testing and believes the sound is man made.
• He has picked up the sound on three separate occasions.

That’s it. That’s the raw info he had. Everything else he presented to the captain he has put together from what he thinks that information probably means.

He could have just presented that information alone. If he had, it’s possible that the captain would have extrapolated the rest. They tend not to put morons at the helm of nuclear powered, nuclear armed naval vessels. Still, if this is all the captain gets, then the onus is on him to do the legwork. What Jonsey has done is taken raw information and put it into a format that greatly helps the captain see the significance of the information. The only thing left for the captain to do is to decide whether he buys Jonsey’s version of events and then act accordingly.

This is about as close to the perfect push report as you’re likely to see. It may be that your own push reports aren’t as high-stakes as hunting a rogue Russian sub, but the principle remains. If you have information that your audience needs to hear, take the time to put it into a format that helps them see the significance of that information.

Ask yourself what you want from revealing your information. Do you think certain action should be taken? How does your information support taking that action? Do you want to highlight risks associated with your information? What are some credible issues that could arise as a result? How does your information support that? It may be that you have information that you simply wish to relay in case it is useful. Nothing wrong with that, but think about the information you are presenting and whether or not there is more to it than what you have.

Remember also that however compelling your case, the final decision is unlikely to be yours. Suggest a recommended course of action. Don’t demand. There may be other factors that you are not aware of that have some bearing on the situation. It’s your job to find and present the information. Let them take responsibility for what happens next. That’s why they get paid the big bucks.

Elicit feedback. Just as with pull reporting, you want to find out whether the information you gave was useful. If not, why not? You can take the response you get and feed his back into your testing.

I think that the distinction betwen pull and push reporting is a useful one. It provides me a framework to think about how to report and what to include. I’m curious to hear your thoughts or how you think about test reporting.

It turns out that the disk my VPS was hosted on crashed irrecoverably. Of course, it took my provider 5 days to get back to me with an explanation as to why my site was down. They eventually did give me an explanation, but not before I got sick of my requests for information going unanswered and transferred my site to a new host. I think there is money to be made in seeking out stocks of companies that have awesome customer service and buying that. I have yet to test whether the opposite is true, but if a company has shitty customer service, I just don’t have the patience to deal with it anymore. Moral of this story is – VPSland has shitty customer service, so they get no more of my money. Linode comes highly recommended by people I trust, so I have big expectations of them. Time will tell.

The site is not the only change in my life of late. I went back to Australia last month for final selections for the Australian national kendo team. Unfortunately I was not successful, so I won’t be going to Milan next year to compete. A pity really, I would have loved to have gone. It’s ironic that I came to Japan to do more kendo training, but if anything have ended up doing less over the last 3 years than if I’d stayed in Melbourne. I made a conscious decision to try and strike a better balance between my home life, my work and my passion for kendo. I did that, but it meant that I was not as sharp as I needed to be at selections. I didn’t fight badly, just not well enough. As it stands, I’ve made the Australian team twice. I would have loved to have gone a third time, but it is not to be. Other things in my life are calling. I will not be part of the squad for subsequent campaigns. I thought I’d be more upset about it than I am, but when it’s time, it’s time.

I’m looking forward to spending more time with friends and family. I’m looking forward to spending more time on my Japanese (which is still awful, for the amount of time I’ve spent in the country). I’m looking forward to being more active in the testing community. I’ll still be swinging sticks at people, but I no longer have to drive myself into the ground in order to get ready for competition.  It actually feels pretty good.

To test is to tell two parallel stories: a story of the product, and the story of our testing. Test framing is a key skill that helps us to compose, edit, narrate, and justify the story of our testing in a logical, coherent, and rapid way.  The goal of test framing is to link each testing activity with the testing mission.

This to me is a worthy skill to have. Ask a tester why they execute a certain test at a certain time and often they will struggle to tell you. It doesn’t mean they are testing randomly or that there is no strategy to their approach, but that they are not yet skilled enough to articulate their reasoning from conception to execution; or if you like, they are not yet able to frame the reasoning for their actions in a way that is compelling.

Here’s where my brain nags at me. I like this concept of test framing and yet it seems to be missing something. Something fundamental; something without which the act of framing loses much of its meaning.

I said ‘not yet able to frame the reasoning for their actions in a way that is compelling’, which begs the question:
Compelling to whom?

I think Test framing as James and Michael describe it is part of a larger set of framing skills that are fundamental to being a skillful software tester. From the perspective of training software testers and answering to anyone who asks about the specifics of the testing you are doing, Test framing is a key skill. There is much more to framing than this however.

Much of what I’ve seen written about test framing to date implies an audience, but does not say a great deal about the audience explicitly. Simon Morley has written an excellent blog post on Decision and Analysis Frames in Testing. If you haven’t read it yet, please do. Simon’s writing hit upon the importance of understanding the filters through which your audience views the world (the frame through which they are looking) and that this can be different depending on the role your audience plays. He writes:

…the way we present information can affect its interpretation – depending upon the frame that a stakeholder is adopting.

Think of a frame as a filter through which someone looks at a problem – they’re taking in lots of data but only the data that gets through the filter gets attention (the rest may end up in the subconscious for later or isn’t absorbed), “I have my show-stopper filter on today so I don’t notice the progress the team has made…”

So, being aware of the potential different types of frames that each project member might have as well as some traps associated with frame formulation is important.

He goes on to write about the different kind of frames the audience and the tester might employ (it is worth noting the differences between them). He also writes about problems with framing, such as Functional Blindness (I see parallels here to the Dunning-Kruger effect), the Sunk Cost fallacy, overconfidence and distortion through measurement and numbers.

Understanding the audience’s part in framing information is a key part in effective communication. My presentation at CAST2011 was on telling a compelling testing story, so framing, especially in terms of the audience is a subject that is very interesting to me. Framing of your testing (and the information it reveals) is inseparable from your audience. Even if you are only framing your information for you – you are still your own audience.

Each of us has different filters through which we view the world. Even within a single role (tester, programmer, project manager etc) individuals can have vastly different outlooks on the same subject. Largely this has to do with the way we model the world. We create a mental representation for everything we interact with. Every object, every person – we have a mental representation about what each of them means to us based on our previous experiences and our previous models. Our experiences shape our models and our models become the filters through which we experience the world.

Necessarily then, every individual has different models (and these differences may be subtle or profound), and so we all have different frames of reference through which we interpret the world around us.

To understand frames, you need to understand models. Not just the model that we have of our system under test, but the models we have of the world around us, the model we have of our audience especially. The map is not the territory. How we view ourselves is not how others view us. The reality (who we actually are) is different again. Same goes for our audience and indeed any other object or concept you care to name.

When we present information about something to someone, we are in fact presenting our interpretation of that information through the filter of our models, through the filter of our audience’s models to them.

Understanding the model(s) your audience is using can help you build information that they need to know about. Understanding your audience’s frame of reference when receiving information can help you frame that information in a way that they are receptive to. Moreover, you can decide if your audience needs information that will augment the models they have and whether or not you need to reframe them in order to present it.

You may present information to your audience that you thought was vitally important only to have them react as though it were inconsequential. Why is that? There are lots of possible explanations, but it’s likely that either they are working from a model which renders your information inconsequential, or it could be that you have a richer model than they do and they do not understand the importance of your information because the limited nature of their model does not allow them to.

Here is where reframing can be helpful. If your audience’s current model creates a filter whereby your information is not useful or valid it does not necessarily follow that your information is invalid per se. Were your audience to look at your information from a different perspective it may be that they would find if valid and important after all.

I feel like there is a lot more here to be pondered and discussed, but for the sake of brevity (hah) I will leave off here for now.