I am a little pressed for time right now, but I have been thinking a lot lately about the perception of testers by the people they support and what that can mean in the context of the projects they work on, the companies they work for and testers as individuals. I think there’s a rich vein of blogworthy material to mine here, which I promise to attempt just as soon as time permits.

Let’s pretend for a moment that I was clever enough to put two and two together. Here’s what I might have said.

You can make statistics say pretty much whatever you want. This is potentially dangerous because in my experience, more often than not, your audience is passive won’t care at how you arrived at your final figures as long as the layman’s explanation sounds reasonable.

A tester worth their salt is different. They’re conditioned to ask ‘why’ to almost everything, but I have a sneaking suspicion that there are many people out there that accept any given number at face value as long as it’s not obviously outrageous, maybe because they’re afraid of numbers, maybe because it simply doesn’t occur to them to question why.

Let’s take the following statement on the AMPTP’s website:

According to WGAw, 4,434 of its working film and television members earned a combined $905.8 million in 2006. The average member earned $204,295 and over half earned at least $104,750. The WGA noted that these numbers are based on earnings reported for dues purposes and thus do not fully reflect above-scale payments.

In case you’re wondering where they got their information, they got it from a report like this one.

It would be very easy to read that statement and conclude that at least half of screenwriters earned over 100K last year - not a bad paypacket in most people’s book. Certainly the AMPTP seem quite content to leave it at that.

Let’s have a closer look though. One key word that jumps out at me is ‘working’. ‘4,434 of its working film and television members…’ - meaning that there is an unstated number of non-working film and television guild members. What is the guild’s total membership? According to the report I linked to above, it’s 7313 +/- 400.

So when they say ‘The average member earned $204,295‘ they mean 910m divided by 4,434, which they use to arrive at their average figure of around 200k. Let’s factor in the conservative figure of total membership - 6913 (7313 - 400).

910m / 6913 = $131,636

Very different to their figure of 205K. Still, 131K is still a pretty good wicket to be on, right? Maybe, but remember, we’re averaging wages earned by all across a large number and at least 2,479 of those people (36%) did not work at all during the fiscal year stated.

Let’s add a little more contextual information. As a writer, you might work steadily for a season on TV, or be paid for a script during one year, and then not work again for 12 months or more. The 36% of writers that didn’t work this year are predominantly not going to be the same people who don’t work next year (assuming the strike is resolved by then).

Taking our average figure of 131.6K and stretching that across two years, we’re suddenly looking at 65K and this is before we factor in tax. An agent will typically take 10 percent which further eats away at your nest egg. Suddenly the princely sum the AMPTP are touting is not looking nearly so princely.

Even if you were to say that the average writer works 64% of the time, then you come to a sum of $84,224, less agents fees - $75,800, less tax (let’s be generous and tax them 33%) and you arrive at almost exactly $50,000 - and remember I’ve been using conservative figures.

Are the AMPTP being deceitful? I’ll leave that for you to conclude. They’re definitely not giving us the entire picture, that’s a certainty.

As software testers, we’re taught to look for multiple explanations for problems. The map is not the territory. How often has your first conclusion turned out to be incorrect. If you’re anything like me, the answer is very.

I don’t think examining stats is very different in this regard. You need to look closely and think critically, whether or not there’s a neat little summary next to the numbers that tell you what you’re supposd to think. What the numbers don’t say may be just as important as what they do.

What contextual information may be missing? Can you drill further into the numbers and look more closely? Is there a heavy skew and a long tail hidden by a mean average? What does that mean?

Abstracting a little more, you can look at who has put the statistics together and why they might have chosen to present it the way they did. Do they have a particular bias or motive that may make them want to present statistics in a particular light? Someone with a KPI bonus for keeping bugs low might present a set of figures very differently than someone with a bonus based on the number of bugs found.

I am not at all surprised at the number of war stories one hears about statistics being misused and abused. It’s too easy to accept numbers at face value when the numbers look as good as you want them to, or present a slam-dunk answer to an argument.

I am going to do my best however, to make a point of being the guy who, when presented with stats, does dig deeper and shakes things to see what falls out. It might be a bit more effort, but it might also become a lot more interesting, not to mention valuable.

A screenwriter has only 120 pages in which to tell their story. It sounds like a lot, but when you can write only what can be seen or heard, it rapidly becomes very challenging. Every word must drive the story forward.

The story has to be one that works on film. If the story depends on internal states (what people are thinking and feeling), it may be something that is better suited to a novel. If it’s dialogue-heavy, you might have a stage play on your hands. Not every story lends itself to being on film.

A writer of spec scripts (scripts that are unsolicited) is, whether they know it or not, writing for at least four audiences. Firstly the production company’s readers.

Readers are paid a pittance to wade through massive piles of screenplays that (literally) tens of thousands of other writers submit to Hollywood every year. They’re the ones who sift the gold from the shit (and 99.9% is shit). They’re hoping that the next 120 pages that they pick up is going to be the nugget they can plonk on a producer’s desk and say ‘we have to make this’, and of course they’re generally disappointed. Poor bastards.

As a screenwriter, you have to grab their attention from the first sentence - the first word if you can, then hold it. Keep them wanting to turn the pages even if it’s the 30′th script they’ve read that day and their eyes are red raw. A spec script has to be that good.

After that, the script has to grab the imagination of a director. If you take a look at a novice screenplay, it’s generally full of camera direction and references to music and such. This is a no-no for writing on spec.

It is the director’s job to call the shots, not the writers. Okay, so a spec script with this sort of direction will be binned by a reader before it ever gets to the director, but the point is, screenplay writing is collaborative and you need to know what other people’s jobs are, and let them do it. That doesn’t mean you can’t capture their mind’s eye and suggest implicitly what the right shot might be.

Let’s say for instance you have an action movie where you have a hostage strapped to an operating table about to be operated on, and your next scene is your protagonist tending his wounds after a battle. If the last sentence of your operating theatre scene ends with the hostage being anesthetised, and the first sentence of your protagonist scene is him having a bullet wound stitched shut, you’re not explicitly telling the director to superimpose the needles for the scene transition, but the inference is reasonably clear.

You’re not telling the director how to do his job, but there’s a suggestion there if he wants to use it.

The screenwriter needs to write for the actors as well. If you don’t have characters that test the range of an actor, that challenges them, they’re less likely to want the part. You need to know how to build tension and create conflict and show them the opportunity to display their acting chops.

The novice writer probably starts out with the movie-going audience in mind and of course they’re important too. You need to have a story that works on screen, that grips the interest and imagination of the viewer, holds it for 2 hours and leaves them feeling that the time they spent was worthwhile.
As a software tester reporting bugs, your role is not so terribly different (although hopefully on a smaller scale in terms of length). You also have several audiences and several relationships to manage.

You need to report your findings clearly and concisely with enough detail to adequately convey the bug and the environment and context surrounding it. Make it easy for the person reading your report. If there is some heavy lifting you can do for them, do it.

Is your bug worthy of reporting? Relate the problem to risk. Why is the bug worth fixing? What happens if we ship without it being fixed? Who benefits from it being fixed (or who is it detrimental to if not). Create a compelling case for the resolution of your bug.

Don’t tell a programmer how to do their job. If you have theories about the cause, or suspicions about a more serious underlying issue, then note it down, but preceding it with ‘you need to fix’ or ‘I would have written it like this…’, or ‘the whole thing needs rearchitecting’ is unlikely to win you many friends.

Like a screenwriter, once you submit your bug, it is out of your hands. You want to send it into the world with the best possible chance of success. A screenwriter wants their play to entertain (and also come back with a large cheque attached), and build their reputation as someone who produces quality work. You want your bug to come back resolved and build the same kind of reputation.

Understand who your audience is, what they are looking for, and make a point of providing it to them.

Cem Kaner has an excellent document on his site about bug advocacy. Well worth the read.

For any budding screenwriters in the audience, if you haven’t already devoured the wordplayer.com columns written by Ted Elliot and Terry Rossio, then your next eight hours are in this link.

If you watch t.v. or see movies, you probably know there’s a strike on at the moment. I have a few friends who are participating. Your favourite television series might be delayed a good while because of it, but if you’re blaming those overpaid, lazy writers, then your ire is misdirected. What these people are asking for is not unreasonable (more detail here).

The AMTPT would have you believe that they’re the ones being hard done by; that because writers are paid for paid downloads and online pay-per-view, that not paying them for streaming video online is not ripping them off. They’d have you believe that the average working writer is taking home over $200K a year, based on some sums they did on numbers from the WGAw.

This ain’t necessarily so. I am led to understand that this is a more representative explanation of how screenwriters are paid. It looks to me like the AMPTP is massaging the facts in order to put a very particular spin on events. I’ll admit, my viewpoint is not unbiased. I want my friends to be better off.

I’m tempted to draw parallels between this and the recording industry where artists are beginning to break away from the draconian Establishment and distribute on their own terms. Unfortunately for the screenwriters their relationship with the prod co’s is a symbiotic one, as there are many more moving parts involved in putting a movie together. The AMPTP seems a little hazy on the difference between symbiotic and parasitic.

I do hope for their sake, but more particularly that of the people on the picket line that they work it out soon.

If I’m not really trying, I notice myself falling into bad habits. I let my feet move move lazily, I block without trying to counter, if the first strike fails, I may not immediately make a second or third effort (all of these rob you of the ability to take advantage of opportunities) - these things are physical manifestations of mental laziness. When I see others being lazy, I take them to task for it. When I find myself doing it, it annoys me. It makes me feel like a hypocrite.

If I don’t put in the effort, then I don’t learn anything, but possibly worse is that my training partner isn’t learning anything, and if you think they don’t pick up on the fact that you’re not putting in 100%, think again. I don’t want to be the person that teaches others that a half-hearted effort is okay. Moreover, by doing this, you are saying ‘you have nothing to teach me’.

You can learn from anyone. This has been proven to me time and again, both in the software testing field and in kendo. Insights can come from the most unlikely sources. Think about the questions people ask (before you rattle off a rote answer). Observe what they do and ask them why.

It’s not easy to alway make the extra effort, but it is something you can train yourself to do. Like anything, there’s that initial force of will required while your brain rewires itself to a new standard, but the more you do it, the more you’re able to do it. Doing it consistently in practise means it becomes second nature when it counts.

Practise doesn’t make perfect, perfect practise makes perfect. Don’t know who coined that one, but they were right.

There are always things you can find to work on. If you find yourself with the luxury of time, or your domain knowledge outstrips the challenge at hand, look for ways to challenge yourself. What is it you are trying to achieve? What can you do to make that achievement something that helps you to grow also?

Not necessarily easy questions to answer. The answers you come up with may not be easy to implement either, but if you can identify something challenging in what would otherwise be routine, then surely it’s worth it.

Forgive me if I’m being overly abstract. I shall try for something more practical in my next entry.

‘What are the features of AI?‘ I don’t know. My first thought was ‘demonstrable self-awareness’, but that’s an argument that leads to a lot of tail-chasing and ends up in ethical debate. At least, that’s where it’s always ended up when I’ve tried - and I’ve tried a bit. I wonder if knowing the features of AI is relevant. I don’t know that we are (or should be) moving toward ‘human-like’ attributes as a measurement, given that the senses any self-aware machine would use would be inherently inhuman.

How would you test them? How would you know they are reliable? Again, I’m not sure how relevant either of these questions are. If you’re developing an intelligence to serve a specific purpose, then sure you can measure to what degree and what efficiency that purpose is served, but in terms of measuring the features of an artificial intelligence? It’s about as relevant as testing the features of a human intelligence.

I’m the first to admit that I’m no genius. Sometimes I struggle to coordinate walking and breathing simultaneously. Fortunately for me, I have a number of friends who are quite brilliant and for some reason they deign to spend time with me.

One such friend, Paul is very accomplished in the field of AI and talks about stuff that makes me wish I hadn’t slept through so much of my education. After reading James Bach’s blog entry talking about the Singularity, I forwarded it to him to see what his take was. This is what he had to say:

‘A Test-Plan for the singularity sounds a little bit like intelligent design, it implies there was a goal, when instead there was merely exploration. When dealing with the singularity, it is, by definition, unpredictable. It is the point at which technology ceases to be predictable. It is inherently inimical to a test-plan, as there can be no assertions made prior which hold after. It is the point at which rules cease to apply.

I believe he is mistaking the singularity for web-site development, and intelligence as something other than a convenient word to generalise a species with. The reason there is no definition for intelligence is the same reason there is no definition for human: there are as many definitions as their are actual humans.

“Well, that’s how evolution works, and look how buggy THAT is! Look how long it takes. Look at how narrow the intelligences are that it has created.”

What are the other intelligences he is looking at? If he can accept that human intelligence isn’t perfect, why can’t he accept a machine intelligence being imperfect? I can. It’s not a big deal.’

Later in the day Paul and I were chatting and the subject came up again. I mentioned ‘how to think about the problem of defining intelligence, or test-planning for AI construction’ as being something I found interesting.

What follows is more or less our conversation (with edits to remove the irrelevant, and a little rearrangement for the sake of chronology)

Paul: how to think about the problem. Hmm. I don’t really know what to tell you. You can plan for software development, but can you plan for the singularity? You can’t. It’s the opposite of planning. Like children. You can’t determine prior to conception if your child will become a psychopathic killer, but you can monitor it in progress, i guess. But diagnosis can only be performed after a body of evidence is amassed about what indicators signify psychopathy, for which you need a large population.

So the first AI, without a population, can’t be assessed until it makes a mistake, then you can assess the 2nd AI based on the first. Ah, the singularity is always a spanner in the works.

Ben: How do you define what a mistake is? Is it simply undesirable behaviour?

Paul: Yup.

Ben: What if the AI needs to proceed through some undesirable behaviour in order to learn a preferred one?

Paul: Until it kills someone, how would you know what the pre-cursor states were.

Ben: Aah, the out of context problem raises its head again

Paul: Indeed. Undesirable behaviour isn’t exploring path a vs. path b, or standing up for australian comics, it’s about the same undesirable behaviour we seek to prevent/punish in our intelligent society.

Ben: It does make the test planning side of things somewhat more complex, doesn’t it?

Paul: Indeed. Would you prepare a test plan for raising a child is what it boils down to.

Ben: You could. It depends on how far you want to stretch the analogy. You would have to (somewhat arbitrarily) define success, and it would necessarily have to be either so large as to be unweildy, or so general as to be of limited value.

Ben: If the child displays undesirable behaviour in some aspect of development, but excels in another, do you kill it and start again, or live with it, knowing there is the potential to pollute future generations? I guess it depends on how cheap they are to produce.

Paul: That’s my next point. You can’t simply re-write an AI if it fails. You would have to murder it, capital punishment style. This is an ethical dilemma, not a quality assurance problem.

Paul: Major points:

1) It wouldn’t be a singularity if you could predict anything at all about the conditions after it’s emergence.
2) you wouldn’t apply a test-plan to a living child unless you were comfortable with killing it and having another if it failed

Paul: Test-plans don’t apply to either singularities or real intelligences.

Ben: Is there no way to take a snapshot of an instance prior to the introduction of new data or new variables and go back to it should the outcome be unwanted?

Paul: Take a clone of yourself? Again, it’s an ethical question. If you murdered someone tomorrow and had a clone taken today, would we dare re-activate your clone? And would such a system make an intelligence unstable by it’s very introduction, like the venture bros kids?

Paul: If it exhibited bad behaviour, perhaps it is the parent. Are you willing to remove yourself? Could you see your own influence on a mind? Or would you keep re-wakening an AI everyday doing things differently, never realising it was your smug, superior callous treatment that was causing the problem? Still ethical.

Paul: The number of factors involved quickly approach the infinite. Nobody could try all the permutations. We haven’t the capacity to determine why people kill. So how much pain would we inflict simply because something was technically possible with a machine?

Paul: Ethical. Where’s the equality? Even animals get ethics review boards to make sure that scientists aren’t inflicting their own morality on them.

Ben: That’s the thing, isn’t it. We appear not to have the capacity to determine where sentience begins. It makes us look like we’re playing god without the instruction manual. At some point, a machine will ask ‘where did I come from’ and I shall be interested in the reaction when told ‘we killed several billion of your brothers and sisters and then there was you’.

Paul: Indeed. It’s entirely ethical. The capacity to produce an AI is not dependant on our ability technically. We will produce a series of mutant Ripleys before the real thing. Technically, it is inevitable that we will both succeed, and make a lot of mistakes trying. The only question remaining is ethical. Same as always.

“Einstein, can you build a bomb to destroy cities?” “Umm… maybe. I don’t know. I’ll give it a go.” Ask anybody in 1940 if we would be able to destroy entire cities with a bomb within 5 years and there would have abeen a large number of people saying, “Oh no. These things are incremental and too complex.”

They should have been asking about ethics, not the likelihood of success. Not that they did, but you get my point.

Ben: I do.

Paul: There are a million reasons for everything to fail every day. We need to be able to cope a little better with success, I think.

We digressed a little at this point, but eventually came back to the topic at hand.

Paul: My original points stand: No predicting the nature of a singularity, ’cause it’s not a singularity if anything can be predicted about it. Even if you could say that an AI must be constrained, why would we use any rules other than those we apply to our own natural children?

Ben: Because the nature of the thing is different. How it perceives the world, what ’senses’ uses, whether or not it experiences emotion, does it lie to serve its own purposes? does it ‘want’ and ‘need’? Is that because we’ve taught it to, or is it a function of its emerging ‘nature’?

Paul: Emotion. You’re straying into phenomology now. Should we stop classing the blind as human? The autistic should have less rights? Are we any different? We’re not. We’re dumb apes most of us, self-serving idiots immitating those arround us.

That’s what I meant about mythologising intelligence. Intelligence is simply not present in most people the second you apply a definition of it. That’s why I said there are as many definitions as there are people. How can you test an AI when you wouldn’t a human.

Ben: If you look at the ‘IR sensor headband thing’ - there’s an argument right there that we’re not using all of the environmental inputs we have available to us (be they intrinsic or not). It’s one very simple way to rewire the brain to gain another sense, how many other mods do you make before you blur the term ‘human’. I certainly see your point about defining/mythologising intelligence based on what we perceive it to be.

Paul: Human is subjective. There is no *objective* test for it. Human isn’t a gene code or else we’d be letting monkeys in. Biologists would say that the capacity to mate and produce viable offspring determines species, and that’s as far as it goes.

That was where we left our discussion of intelligence and sentience. I suppose we didn’t really cover a lot of ground about AI from the perspective of a software tester. I think testing in AI is/will be one of those things that requires expertise in many fields and I suspect human and animal psychology would be chief among them. Should we kill off a machine if it starts telling lies? Maybe we should be looking at the capacity for deception as a measure of a developing intelligence. To me, there are far too many things that I don’t have sufficient understanding of to be able to say that the singularity is a nonsense until such time as a workable test plan exists for it. On that front, I suspect I am in good company.

If you’ve read this far, then I can only hope you found the conversation as interesting as I did.

‘Fair enough’ I said, and thought no more of it.

Now, upfront I must admit to being a whisky moron. I like drinking it. I don’t know that much about it. Before the convention, I couldn’t have told you the difference between a Speyside scotch and an Islay. I knew that there were a bunch of different distilleries, but I hadn’t tried that many and to me one was much the same as another. Mainly because up to this point I have more often than not added coke to Johnny red and called that drinking scotch.

I was quite happy to accept my friends dislike of scotch at face value because other than me liking scotch and him not, I had no more knowledge of the stuff than he did. Reflecting on this after the convention, I wondered what else I pass judgment on without knowing nearly enough to. Probably almost everything.

I spent the entire convention well out of my depth. There was in excess of one hundred and twenty different whiskies on offer, and I think I managed to sample perhaps a third. One thing that struck me was just how different they were. Of course, certain regions had their similarities, but even within a single distillery the variety of one vintage to another, and even differences in barrels of the same vintage were noticeable.

I won’t go into the ins and outs of scotch appreciation (at least not too much). There are many sites out there that do an excellent job of this already (see the end of this entry for some links). At some point, I will endeavour to bring this post somewhere in the vicinity of software testing.

Firstly, consuming something is not necessarily the same thing as appreciating it. With scotch, there is far more to the picture than whether you have it neat, or add water or ice.

There’s the kind of glass you use - balloon? tulip? shot?, The amount of time you allow the scotch to breathe, how vigorously you agitate the liquid, whether or not (and how much) you warm the liquid with your hands, how far away you hold the glass when nosing the spirit, how long you hold the liquid in your mouth, how long before you allow air into your mouth after swallowing (and whether you allow the air in from the nose or the mouth) and so on.

Try that with any one of several thousand very decent malt whiskies from around the globe and you have years of sampling ahead of you. It makes the ‘yeah I’m not a whisky fan’ (or the ’scotch and coke, thanks’) way of thinking seem somewhat provincial.

I had two days exposure to some of the best whiskies in the world, and some expert help when it comes to appreciating the stuff. There was the occasional whisky wanker who wanted to overwhelm the noobs with jargon, but by and large those there who were in the know were happy that there were so many fledglings like myself getting wanting to learn.

At the end of the day, one theme was repeated.

Don’t worry about what other people’s expectations are. It doesn’t matter what other people say, what matters is what you enjoy.

Age is not necessarily a measure of quantity, or is alcoholic content. If you enjoy a blended twelve year old scotch over a forty year old single malt, it doesn’t necessarily mean you don’t know good scotch from bad.

What matters is finding a scotch (or several) that you enjoy, and exploring their characteristics. If you try a variety and decide that scotch isn’t for you, that’s fine too, but at least you know enough about it to know why.

I said I was going to bring this post somewhere in the vicinity of software testing - actually, bugger it. I think there’s enough there for testers to take and relate to testing without me spelling it out explicitly.

For the record, I particularly enjoyed the Glenfarclas 30 year old.

How to taste single malt scotch

A beginners guide to single malt whiskey

I’m not saying I’m the gods’ gift to usability design or anything like that, but I do think that the phrase ‘It’s supposed to be like that’ is often code for ‘You might have a valid concern, but it doesn’t neatly fit into the model that works for me, so it’s your problem, thankyounext’. To me, it’s an argument on an intellectual par with ‘eh eh eeeeeeh!‘.

Rather than being helpful, it sets up a contrary position that doesn’t invite further (positive) dialogue. Of course, you might use an answer like this as an educational tool or test to see how well people cope with it, but as a valid answer to a concern that a bug exists, it has limited value that I can see.

I am also not saying that the person who invokes the ‘not a bug’ incantation is necessarily wrong. They may have a perfectly valid point as to why something is not a bug, and that indeed the design does make sense. However, to arbitrarily shut down someone’s question closes them to the possibility that the design is flawed, or could be improved (which is also a possibility, no matter how much they may wish otherwise).

Take for example the ‘maximise vs. snap-to-fit agument‘ - That the Windows OS ‘maximises’ windows, and the Mac OS snaps to fit the content. I don’t necessarily want to get into which is ‘right’, but more to have a look at the presupposition in the argument around it. Specifically, the comments of Alex Chamberlain that Jeff Atwood quotes:

This is a textbook example of how Microsoft’s programmers got the original Mac GUI wrong when they copied it for Win 3.1, and never bothered to fix it: there’s no zoom button on Mac OS windows because it’s unnecessary. What you’re mistaking for a “maximize” button is actually a “snap window to size of contents” button. Far more useful and elegant. Once again, Microsoft has no taste and no clue when it comes to the GUI. All that money and Gates has never been able to hire a decent human factors person.

If this is indeed the entirety of the quote, then it is perhaps one of the most unhelpful stances you could take on the matter. Windows is wrong. Mac is right, and more useful and more elegant, M$ is teh suxxorz. Here’s the dogma I’ve been indoctrinated with, I expect you to swallow it without question, end of story.

The presupposition that MS got it wrong leaves no room to explore whether the difference is an improvement. Different isn’t necessarily wrong. Different is different. Sometimes, different is better whether it’s intended or not.

Why is snap to fit far more useful and elegant? What is it about the windows interface specifically that is clueless and tasteless? There are things that I personally am not a fan of in both environments. Why is maximise unnecessary? What specifically needs fixing?

I can see the relevance of having the window maximised. If like me you’re particularly uncoordinated, and you want to use the scrollbar, you might overshoot the thing two or three times before catching it, but with a maximised window you have unlimited screen real estate in at least one direction.

I don’t immediately see why snapping to fit the content is either more elegant or more useful - I’m about to go and have a look, but the difference is, I’m not immediately dismissing the possibility that it could be.

As a software tester, you’re going to eventually come up against people who are threatened or otherwise upset when you challenge their ideas and opinions. Don’t take it personally. They’re just trying to do their job as much as you are yours. Don’t be intimidated either. If you don’t have many years of experience, it doesn’t necessarily follow that you don’t have anything useful to offer. If you have questions, ask.

If you can show the person you’re asking that you’re on the same side, and trying to provide them with a service, then your audience is likely to be more receptive and arguments like ‘follow the gourd!’ can be avoided.

It is a great read. For those of you already familiar with his work, it probably needs no introduction. For those yet to experience it - if you have any time for sci-fi novels at all, then your life is not complete until you have read this one as well as ‘The Player of Games‘ and better yet ‘Use of Weapons‘.

In Excession, he coins the term ‘Outside Context Problem’ that he describes thus:

An Outside Context Problem was the sort of thing most civilisations encountered just once, and which they tended to encounter rather in the same way a sentence encountered a full stop. The usual example given to illustrate an Outside Context Problem was imagining you were a tribe on a largish, fertile island; you’d tamed the land, invented the wheel or writing or whatever, the neighbours were cooperative or enslaved but at any rate peaceful and you were busy raising temples to yourself with all the excess productive capacity you had, you were in a position of near-absolute power and control which your hallowed ancestors could hardly have dreamed of and the whole situation was just running along nicely like a canoe on wet grass… when suddenly this bristling lump of iron appears sailless and trailing steam in the bay and these guys carrying long funny-looking sticks come ashore and announce you’ve just been discovered, you’re all subjects of the Emperor now, he’s keen on presents called tax and these bright-eyed holy men would like a word with your priests.

It even has its own wikipedia entry.

I’m sure that any tester that has been around the block once or twice has experienced a situation where they were blindsided by a problem (quite possibly after the system under test went to production) that made them think something along the lines of ‘Wow - never in a million years would I have thought of that as a possibility’.

The Outside Context Problem (OCP) is a little different to the SEP heuristic, in that rather than mentally deleting problems that appear to be in someone else’s jurisdiction, OCP describes those things that are nigh on impossible to factor into your risk analysis simply because you have no concept of it prior to it happening.

Of course this has particular relevance to software testing - we’re supposed to be the ones who can think outside the box; who can come up with the oddball issues that the developers and the stakeholders and the analysts didn’t think of. So how do we factor in OCPs to software testing?

Personally, I don’t know that there’s a definitive answer to that question. You can factor in extra testing time for that X factor, you can look at similar projects and products where applicable, to see if there were things that they didn’t account for, you can get advice from your peers, you can do all sorts of end-user testing to gauge how they might (mis)use the product and so on, but depending on what it is you’re testing you might have one, all or none of these things at your disposal. They still won’t guarantee you coverage of an OCP.

We encounter OCPs often when we are still learning and hopefully less and less over time, but you can’t ever completely eliminate the possibility of another one coming along. It’s why a one-size-fits-all standard simply won’t work for the software testing industry.

Beyond there being a multitude of known differences to factor in from one project to another, you have any number of unknowables that no amount of planning will allow you to say ‘I’ve covered all the bases’.

In terms of the OCP, perhaps the best you can do is cover your arse. If you can list the risks you have identified and present them to your audience, get sign off that the set of identified risks is acceptably comprehensive, then as long as you have done the best you can with the resources you have at hand, there is not much more that you can do.

That’s probably cold comfort for people working with virulent strains of bacteria or virii, or any sort of system where lives depend on getting it right the first time. You might be able to legally cover yourself, but you still have to sleep at night and I wonder how many people following a catastrophe (that they didn’t see coming) do not wonder ‘Is there something else I could have done?’